Ability to expose how loaded waitress is

[robertlagrant]

Is it possible to grab Waitress metrics (how many channels currently active, how many threads active, how many TCP connections open, for example)?

Is this currently possible? If not, I think it'd be a good feature :-)

(My plan is to shove them in prometheus_client, which can compute averages, monitor for high watermark breaches and trigger Kubernetes pod scaling, etc.)

[digitalresistor]

No, waitress at this point in time doesn't make any of that sort of information available.

[NicolasLM]

As a Prometheus and Waitress user this would be great!

Obviously waitress should not depend on prometheus_client, so I believe that the way to integrate metrics into waitress without being too invasive is by using signals.

[digitalresistor]

Waitress will currently log when there are more requests in the queue than there are threads to process said requests. It is logged under waitress.queue. Additional information and or prometheus support or things of that nature require someone to actively investigate how to implement that in waitress and the performance impact thereof.

[mcdonc]

It might not be a terrible idea to allow people to interrogate for this info via a signal handler and stdout/stderr and let people build on that.

[runyaga]

It would be a great idea. Also would suggest adding traceback for all threads.
Back in the Old World you could send signal USR1 to ZServer and it would dump out to stdout traceback adding the other metrics would be nice bonus.

[nsballmann]

How about making prometheus_client an optional dependency and serving the (configurable) /metrics endpoint (or maybe on a different port) if the package is available?

[robsonpeixoto]

Is safe to check the len(server.task_dispatcher.queue) to monitor the queue length?
If so, is possible to add the metric:

d = Gauge('waitress_task_queue_length', 'Waitress task queue length')
d.set_function(lambda: len(server.task_dispatcher.queue))

How to get the number of connections? The Gauge.set_function method can be very useful to monitor the waitress.

[robsonpeixoto]

Use prometheus to monitor the queue is not a viable solution, because the /metrics route will need to wait process all tasks from the queue until process the /metrics request.

Maybe use statsd is the better solution.

https://statsd.readthedocs.io/en/v3.3/types.html#gauges

[NicolasLM]

It doesn't seem like there is consensus on the technology to use to report metrics. One way to tackle the problem could be to not pick any technology but instead provide a set of signals users could use. By signals I mean something like blinker signals, not unix signals.

Waitress would only need to define a set a signals like "new connection", "started task", "finished task"... and users would handle these signals if they want to derive metrics useful to them. Popular integrations like statsd or Prometheus could be created in a contrib part of waitress or even in a separate repository because signals would act as an API between core waitress and the integration.

This is the approach used by Flask, Django, Celery and many more to provide developers a way to integrate and extend the frameworks.

For a very down to earth example on how this could be used to provide metrics, see how I used signals in a project of mine to create an integration with Datadog.

[pauls-username]

Here is a simple monkey patch solution I have used. https://gist.github.com/pauls-username/56744f5bd43b531a348bb59c36541cd7

There are probably more metrics that you could add using this method. The key is getting the server instance. Maybe adding some sort of a hook to get the server instance (without using monkey patching) would be a good official solution?

[Rudd-O]

We really should have an optional export of open metrics for Waitress.

That said, the question of whether to serve the /metrics handler in the same HTTP port and using the same Waitress threads... is tricky. If your Waitress is overloaded — kind of the moment when you need metrics the most — your Prometheus instance won't be able to scrape anything, because the Prometheus instance is sharing the same inbound queue as everything else. That's not good.

I like the monkeypatch code sample a lot, but — seconding what the previous commenter said — the code should also start a separate HTTP server using the Prometheus routine to serve /metrics on a different port.

Without metrics, Waitress is unobservable and that does not bode well fro production.

[nsballmann]

Practically, this is not a problem. Monitoring is not for the point in time where the server doesn't serve anymore. Then, you know that already. That's the reason you are looking at the monitoring data in the first place. Monitoring data is an indicator what may be the reason and how fast you got there.

Yes, the /metrics handler will consume resources (CPU, Mem, Queue-entries) while serving. But this just means, it's overloaded earlier. But the trade-off is, by spending those resources, you get some insight which resources are consumed most and need most to be added (if possible).

In most implementations it doesn't matter if its in the same HTTP server or a different one. They consume the same resources. The overhead of a different process is probably higher. Serving it in the same process Pythons GIL may play a role in the game.

So my personal assessment is: Metrics consuming a small part of the resources is better than to have no metrics at all.

[Rudd-O]

Agreed on the assessment — note that serving on a different port allows metrics to still be collected if the main server is overloaded in the specific sense that the main port has an enormous queue.

[robertlagrant]

Use prometheus to monitor the queue is not a viable solution, because the /metrics route will need to wait process all tasks from the queue until process the /metrics request.

Maybe use statsd is the better solution.

https://statsd.readthedocs.io/en/v3.3/types.html#gauges

I don't quite understand why this is the case. Can't Prometheus expose a format like that? I'm thinking of https://prometheus.io/docs/tutorials/understanding_metric_types/#gauge

[Rudd-O]

As a Prometheus and Waitress user this would be great!

Obviously waitress should not depend on prometheus_client, so I believe that the way to integrate metrics into waitress without being too invasive is by using signals.

FYI you can create and serve Prometheus metrics without depending on any library. They're simple lines of text your program serves on a port.

Use prometheus to monitor the queue is not a viable solution, because the /metrics route will need to wait process all tasks from the queue until process the /metrics request.

It is viable, you just have to serve them on a different port, and possibly with a priority queue, or maybe on a separate thread entirely.

For the record, this is standard practice at Google. You don't serve your production traffic on the same port as your metrics traffic, because otherwise you don't get any metrics when you are swamped in production. Different buses for different purposes.

statsd

That's push, and therefore it's pretty hard to get right (many corner cases with push based solutions, especially on dynamic service stacks). You miss metrics sometimes, sometimes you don't know that you are supposed to be getting metrics, et cetera. With systems like Prometheus, if the service isn't giving you metrics, you will get a metric that will fire an alert telling you that the service is down, absolutely no exceptins.