Re: [PATCH] CONTRIB: contrib/prometheus-exporter: Add a Prometheus exporter for HAProxy
Am 11.02.2019 um 10:40 schrieb Christopher Faulet: > Le 09/02/2019 à 10:47, Aleksandar Lazic a écrit : >> Hi Christopher. >> >> Am 07-02-2019 22:09, schrieb Christopher Faulet: >>> Hi, >>> >>> This patch adds a new component in contrib. It is a Prometheus >>> exporter for HAProxy. >> >> [snipp] >> >>> More details in the README. >>> >>> I'm not especially a Prometheus expert. And I must admit I never use >>> it. So if anyone have comments or suggestions, he is welcome. >> >> Just for my curiosity, what's wrong with the haproxy_exporter especially >> that haproxy_exporter uses the csv format from haproxy ? >> > > Hi Aleks, > > Nothing wrong. haproxy_exporter works pretty well AFAIK. It is just an > external > component and it may seem a bit annoying to deploying it instead of having > such > functionality built-in in HAProxy. Furthermore, haproxy_exporter "only" > exports > proxies and servers statistics. Unlike the built-in exporter, haproxy_exporter > is limited to what the stats applet exposes in HTTP. So, it cannot export > global > information. And with a bit more work, we can imagine to export even more info > from the built-in exporter. You are absolutely right. > However, to mitigate what I said, it is not an aim for HAProxy to support all > monitoring and alerting tools (Prometheus, Graphite, InfluxDB, OpenTSDB...). > So > it was added under contrib and not officially integrated into HAPRoxy. It is a > first step of a reflection on the output format for stats to let all kind of > external tools to retrieve them. But it not a priority either. It was just a > quick development and now we will wait and see if there is a particular demand > to go further and how we could address it, if any. I will add this prometheus-exporter to the dev builds as soon as it's in the source. Maybe we should think about ci with github? https://github.com/marketplace/category/continuous-integration Regards Aleks
Re: [PATCH] CONTRIB: contrib/prometheus-exporter: Add a Prometheus exporter for HAProxy
Le 09/02/2019 à 10:47, Aleksandar Lazic a écrit : Hi Christopher. Am 07-02-2019 22:09, schrieb Christopher Faulet: Hi, This patch adds a new component in contrib. It is a Prometheus exporter for HAProxy. [snipp] More details in the README. I'm not especially a Prometheus expert. And I must admit I never use it. So if anyone have comments or suggestions, he is welcome. Just for my curiosity, what's wrong with the haproxy_exporter especially that haproxy_exporter uses the csv format from haproxy ? Hi Aleks, Nothing wrong. haproxy_exporter works pretty well AFAIK. It is just an external component and it may seem a bit annoying to deploying it instead of having such functionality built-in in HAProxy. Furthermore, haproxy_exporter "only" exports proxies and servers statistics. Unlike the built-in exporter, haproxy_exporter is limited to what the stats applet exposes in HTTP. So, it cannot export global information. And with a bit more work, we can imagine to export even more info from the built-in exporter. However, to mitigate what I said, it is not an aim for HAProxy to support all monitoring and alerting tools (Prometheus, Graphite, InfluxDB, OpenTSDB...). So it was added under contrib and not officially integrated into HAPRoxy. It is a first step of a reflection on the output format for stats to let all kind of external tools to retrieve them. But it not a priority either. It was just a quick development and now we will wait and see if there is a particular demand to go further and how we could address it, if any. -- Christopher
Re: [PATCH] CONTRIB: contrib/prometheus-exporter: Add a Prometheus exporter for HAProxy
Hi Christopher. Am 07-02-2019 22:09, schrieb Christopher Faulet: Hi, This patch adds a new component in contrib. It is a Prometheus exporter for HAProxy. [snipp] More details in the README. I'm not especially a Prometheus expert. And I must admit I never use it. So if anyone have comments or suggestions, he is welcome. Just for my curiosity, what's wrong with the haproxy_exporter especially that haproxy_exporter uses the csv format from haproxy ? Thanks Regards Aleks
Re: [PATCH] CONTRIB: contrib/prometheus-exporter: Add a Prometheus exporter for HAProxy
On 8/2/19 11:11 π.μ., Willy Tarreau wrote:
> Hi Christopher,
>
>
> On Thu, Feb 07, 2019 at 10:09:52PM +0100, Christopher Faulet wrote:
>> Hi,
>>
>> This patch adds a new component in contrib. It is a Prometheus exporter for
>> HAProxy.
> (...)
>
> Thanks for doing this. After reading the whole patch, I measure how
> uninteresting an experience this must have been! After all, using C
> to produce yet-another-format is akin to using python to write
> yet-another-http-proxy :-)
>
> I totally agree with your approach of placing it under contrib/. After
> all we've already done the same with other extensions explicitly
> targetting products (mod_security, mod_defender, systemd, ...).
> We support standards, not products. And products which respect standards
> are naturally supported via the standards, so there is indeed no reason
> for opening a new can of worms by inviting $PRODUCT_OF_THE_MONTH into
> src/ especially when these products change faster than our maintenance
> cycle.
>
> In my opinion the right place for a stats exporter is : outside. However
> I'm well aware that our export formats are not necessarily friendly to
> such exporters. For example, the fact that prometheus uses this funny
> ordering forces a gateway to keep all metrics in memory before being
> able to dump them. It's not cool either. We could have a long-term
> approach consisting in trying to implement multiple tree walk methods
> combined with a few formats so that implementing external exporters in
> various languages becomes trivial. In this case such tools could provide
> high quality agents to collect our metrics by default without having to
> work around some limitations or constraints.
>
> This is probably a design discussion to have for the long term here on
> the list : what are the main stats export mechanisms desired in field.
> I can imagine that most agents will want to poll haproxy and dump the
> whole stats once in a while, some will rely on it to send a full dump
> once in a while (this may already become an issue during reloads), some
> might possibly want to subscribe to change notification of certain
> metrics, or receive a diff from the previous dump once in a while. And
> for all these variations I guess we may have to implement 2 or 3 dump
> styles :
> - Front -> Back -> Server -> metric
> - { Front, Back, Server } -> metric
> - metric: { front, back, server }
>
> I don't know if I'm completely off or not, but I do think that those who
> have experience with such tools should definitely join the discussion to
> share their observations and deployment difficulties met in field.
>
There are mainly two ways to get metrics out of software:
1. Push, where foobar software uploads stats to a remote/local entry point.
Graphite is one of the
most used system for this.
2. Pull, the metrics pipeline/infra scrapes foobar software to fetch metrics.
The above is the easy part of getting metrics, the most challenging is data
types(counters, gauges,
summaries) and format.
Graphite has a very simply format and data type, you send strings over TCP
connections:
metricname value timestamp(epoch)
where metricname looks like a FS tree:
loadbalancers.edge.lb-01.haproxy.frontend.www_haproxy_org.lbtol 124 1549661217
Prometheus uses the pull method and it is a bit more complicated.
You also have "proxy" systems like telegraf/fluentd that can work on any method
and build a bridge
between foobar software and any metric pipeline. All those "proxy" systems
allow you to write any
transformer you want, so the options are countless.
I have to agree with you that supporting all possible combinations is quite
hard.
Some software support both, some only one of them. More and more you see new
software have
"instrumentation" out of the box using the pull method and default to
Prometheus model.
I personally find the cvs data we get out of STATS socket easy to use. I can
easily write a software
to support both models. I have written one to support the Push method and use
Graphite as a metric
pipeline and it would be trivial to write an exporter for Prometheus and for
another X system in two
years.
I prefer the foorbar software to give me raw data and let me decide how I will
use them. I don't
want to have any kind of aggregation at the source of the metrics or any other
funny things, which
can use issues when I do aggregation at the store or during visualization.
My 2cents,
Pavlos
signature.asc
Description: OpenPGP digital signature
Re: [PATCH] CONTRIB: contrib/prometheus-exporter: Add a Prometheus exporter for HAProxy
Hi Christopher,
On Thu, Feb 07, 2019 at 10:09:52PM +0100, Christopher Faulet wrote:
> Hi,
>
> This patch adds a new component in contrib. It is a Prometheus exporter for
> HAProxy.
(...)
Thanks for doing this. After reading the whole patch, I measure how
uninteresting an experience this must have been! After all, using C
to produce yet-another-format is akin to using python to write
yet-another-http-proxy :-)
I totally agree with your approach of placing it under contrib/. After
all we've already done the same with other extensions explicitly
targetting products (mod_security, mod_defender, systemd, ...).
We support standards, not products. And products which respect standards
are naturally supported via the standards, so there is indeed no reason
for opening a new can of worms by inviting $PRODUCT_OF_THE_MONTH into
src/ especially when these products change faster than our maintenance
cycle.
In my opinion the right place for a stats exporter is : outside. However
I'm well aware that our export formats are not necessarily friendly to
such exporters. For example, the fact that prometheus uses this funny
ordering forces a gateway to keep all metrics in memory before being
able to dump them. It's not cool either. We could have a long-term
approach consisting in trying to implement multiple tree walk methods
combined with a few formats so that implementing external exporters in
various languages becomes trivial. In this case such tools could provide
high quality agents to collect our metrics by default without having to
work around some limitations or constraints.
This is probably a design discussion to have for the long term here on
the list : what are the main stats export mechanisms desired in field.
I can imagine that most agents will want to poll haproxy and dump the
whole stats once in a while, some will rely on it to send a full dump
once in a while (this may already become an issue during reloads), some
might possibly want to subscribe to change notification of certain
metrics, or receive a diff from the previous dump once in a while. And
for all these variations I guess we may have to implement 2 or 3 dump
styles :
- Front -> Back -> Server -> metric
- { Front, Back, Server } -> metric
- metric: { front, back, server }
I don't know if I'm completely off or not, but I do think that those who
have experience with such tools should definitely join the discussion to
share their observations and deployment difficulties met in field.
In the mean time I think your patch should be merged, you'll get more
feedback on it this way, and it's not a critical part like HTX that
you want to be certain to be perfect before merging :-)
Thanks,
Willy
[PATCH] CONTRIB: contrib/prometheus-exporter: Add a Prometheus exporter for HAProxy
Hi,
This patch adds a new component in contrib. It is a Prometheus exporter
for HAProxy. It is developed as a "service", i.e. an applet which can be
called from a "use-service" rule. To build HAProxy with this component,
you should use the Makefile variable "EXTRA_OBJS":
$> make TARGET=linux2628
EXTRA_OBJS="contrib/prometheus-exporter/service-prometheus.o"
To be used, it must be enabled in the configuration with an
"http-request" rule and the corresponding HTTP proxy must enable the HTX
support. For instance:
frontend test
mode http
...
option http-use-htx
http-request use-service prometheus-exporter if { path /metrics }
...
More details in the README.
I'm not especially a Prometheus expert. And I must admit I never use it.
So if anyone have comments or suggestions, he is welcome.
Thanks
--
Christopher
>From d7e241786cc7056f28dad8bdd8fd72171c7a4e78 Mon Sep 17 00:00:00 2001
From: Christopher Faulet
Date: Thu, 7 Feb 2019 15:38:42 +0100
Subject: [PATCH] CONTRIB: contrib/prometheus-exporter: Add a Prometheus
exporter for HAProxy
It has been developped as a service applet. Internally, it is called
"promex". To build HAProxy with the promex service, you should use the Makefile
variable "EXTRA_OBJS". To be used, it must be enabled in the configuration with
an "http-request" rule and the corresponding HTTP proxy must enable the HTX
support. For instance:
frontend test
mode http
...
option http-use-htx
http-request use-service prometheus-exporter if { path /metrics }
...
See contrib/prometheus-exporter/README for details.
---
contrib/prometheus-exporter/README| 242 ++
.../prometheus-exporter/service-prometheus.c | 2284 +
2 files changed, 2526 insertions(+)
create mode 100644 contrib/prometheus-exporter/README
create mode 100644 contrib/prometheus-exporter/service-prometheus.c
diff --git a/contrib/prometheus-exporter/README b/contrib/prometheus-exporter/README
new file mode 100644
index 0..f7fda6f07
--- /dev/null
+++ b/contrib/prometheus-exporter/README
@@ -0,0 +1,242 @@
+PROMEX: A Prometheus exporter for HAProxy
+---
+
+Prometheus is a monitoring and alerting system. More and more people use it to
+monitor their environment (this is written February 2019). It collects metrics
+from monitored targets by scraping metrics HTTP endpoints on these targets. For
+HAProxy, The Prometheus team offically supports an exporter written in Go
+(https://github.com/prometheus/haproxy_exporter). But it requires an extra
+software to deploy and monitor. PROMEX, on its side, is a built-in Prometheus
+exporter for HAProxy. It was developed as a service and is directly available in
+HAProxy, like the stats applet.
+
+However, PROMEX is not built by default with HAProxy. It is provided as an extra
+component for everyone want to use it. So you need to explicity build HAProxy
+with the PROMEX service, using the Makefile variable "EXTRA_OBJS". For instance:
+
+> make TARGET=linux2628 EXTRA_OBJS="contrib/prometheus-exporter/service-prometheus.o"
+
+if HAProxy provides the PROMEX service, the following build option will be
+reported by the command "haproxy -vv":
+
+Built with the Prometheus exporter as a service
+
+To be used, it must be enabled in the configuration with an "http-request" rule
+and the corresponding HTTP proxy must enable the HTX support. For instance:
+
+frontend test
+mode http
+...
+option http-use-htx
+http-request use-service prometheus-exporter if { path /metrics }
+...
+
+
+This service has been developed as a third-party component because it could
+become obsolete, depending on how much time Prometheus will remain heavily
+used. This is said with no ulterior motive of course. Prometheus is a great
+software and I hope all the well for it. But we involve in a environment moving
+quickly and a solution may be obvious today could be deprecated the next
+year. And because PROMEX is not integrated by default into the HAProxy codebase,
+it will need some interest to be actively supported. All contribution of any
+kind are welcome.
+
+You must also be careful if you use with huge configurations. Unlike the stats
+applet, all metrics are not grouped by service (proxy, listener or server). With
+PROMEX, all lines for a given metric are provided as one single group. So
+instead of collecting all metrics for a proxy before moving to the next one, we
+must loop on all proxies for each metric. Same for the servers. Thus, it will
+spend much more ressources to produce the Prometheus metrics than the CSV export
+through the stats page. To give a comparison order, quick benchmarks shown that
