[DevOps] Elastic APM

[turnUpTheChill]

#566

• Fixes ["Observability"] Implement APM, Template Example and Create Enhancement for other to contribute. #566

Changelog entry

Added annotations and alterations to methods to allow them to be instrumented with Elastic APM.
There is additional functionality like error message logging, which can be added but this gets the base line instrumentation and metrics

Additional info

• Anything else reviewers and future readers should know
  This is noted in the APM.md, but in order for this to work there must be an active APM server available and configured with environment variables

---

Pull Request Checklist

For Reviewer

• Submitted under the correct tag
• Runs without error or warning
• Documentation revised / included
• Unit tests included
• Issues referenced
• Does not create new bugs / issues

[jlubken]

I think this looks good.

When I run this locally I get a stream of errors where the app is attempting to connect to the metrics server.

Is there a way to prevent this (maybe with an env variable or standard convention) if the metrics server is not expected to be running?

[turnUpTheChill]

@jlubken That should be possible, let me see how I can accomplish that.

We also have adocker-compose.yml file that can allow for a local monitoring stack to be deployed a long side the local app. I can provide instructions for that if it sounds like something that would be useful

[turnUpTheChill]

@jlubken It looks like the capability to do this will be available in a future release with an environment variable. If you want it sooner than that, I could try and do something with the logging object
elastic/apm-agent-python#790

[turnUpTheChill]

@jlubken I have added the configuration to disable APM locally. Should resolve that issue.

I can add some instructions and methods in case Developers do want to run APM locally so they can test out different instrumentation options. Let me know what sounds best to you.

cc: @MooseQuest