Scraping Metrics

Prometheus is a pull-based system. It requires targets from which it will get metrics. They can be exposed from inside your services or as generic exporters acting as intermediaries between Prometheus and other services or systems.

Services can be instrumented to provide metrics using one of the Client Libraries (https://prometheus.io/docs/instrumenting/clientlibs/). Many of the languages are supported. If no library is available for your programming language of choice to instrument by one of Prometheus data formats, or if you don't want to add one more dependency by using Prometheus data format, there is always the option of implementing one of the exposition formats (https://prometheus.io/docs/instrumenting/exposition_formats/).

The alternative to instrumentation of our services is to use exporters. The Exporters and Integrations (https://prometheus.io/docs/instrumenting/exporters/) page lists quite a few official and community maintained solutions.

Having both options in front of us, we should make a decision which type to use. Should we instrument our services or use exporters? The decision does not have to be binary. We can use both.

In some cases, we do not have a choice. With third-party software like, for instance, HAProxy, exporters might be the only option since it does not natively provide metrics in Prometheus format. On the other hand, if there is a very particular set of metrics that should be scraped from one of our services, instrumentation is the best option unless that service already exposes metrics in a different format.

More often than not, we do have a choice between using an exporter or instrumenting a service. In such a case, I have a strong preference towards exporters. Instrumentation, even though it is sometimes unavoidable, leads to undesirable coupling. Our services should do only what they are designed to do. If, for example, we have a service that acts as a shopping cart, adding instrumentation and, probably, dependency on Prometheus library introduces tight coupling. If we do that, our shopping cart is not focused on solving only that single business domain but has side functions as well. You might argue that adding instrumentation is not a significant effort. Still, keeping services focused exclusively on their business domain has multiple benefits, and we should avoid increasing their scope by adding any additional responsibilities. That is, as long as we can avoid doing that.

My advice is to always start with exporters, and instrument your services only if you require metrics that are not provided by one of the existing exporters. That way, your services will have clear responsibilities and be focused on a business domain while all infrastructure type of tasks will be delegated to vertical services like, in this case, exporters.

In this chapter, we'll use exporters as the only mean of providing targets that will be utilized by Prometheus to scrape metrics. If you realize that you do have to instrument your services, please consult Prometheus documentation (https://prometheus.io/docs/instrumenting/clientlibs/) for more information.

Later on in the book, we might instrument our demo service if we do realize that it provides a substantial advantage.

Now that we had a brief overview of the different ways to expose metrics, we can proceed towards a hands-on exploration of the subject.