Load Balancing Software Users
Good Day, I would like to know if you are interested in reaching out “Load Balancing Software Users“. If you would like to see a few examples I can send you the names of a few firms that use the specific technologies for review. Looking forward to helping you build new revenue streams for your business. Thanks and Regards, Craig Wilson Marketing Manager If you don’t want to receive any more emails from us REPLY “Unsubscribe”.
[PATCH] BUG/MINOR: init: fix set-dumpable when using uid/gid
in mworker mode used with uid/gid settings, it was not possible to get a coredump despite the set-dumpable option. indeed prctl(2) manual page specifies the dumpable attribute is reverted to `/proc/sys/fs/suid_dumpable` in a few conditions such as process effective user and group are changed. this patch moves the whole set-dumpable logic before the polling code in order to catch all possible cases where we could have changed the uid/gid. It however does not cover the possible segfault at startup. this patch should be backported in 2.0. Signed-off-by: William Dauchy --- Hi Willy, Here is the backport for haproxy-20 tree. Thanks, William --- src/haproxy.c | 50 +- 1 file changed, 25 insertions(+), 25 deletions(-) diff --git a/src/haproxy.c b/src/haproxy.c index fa3fbad4..a0e630df 100644 --- a/src/haproxy.c +++ b/src/haproxy.c @@ -2946,31 +2946,6 @@ int main(int argc, char **argv) } } - /* try our best to re-enable core dumps depending on system capabilities. -* What is addressed here : -* - remove file size limits -* - remove core size limits -* - mark the process dumpable again if it lost it due to user/group -*/ - if (global.tune.options & GTUNE_SET_DUMPABLE) { - limit.rlim_cur = limit.rlim_max = RLIM_INFINITY; - -#if defined(RLIMIT_FSIZE) - if (setrlimit(RLIMIT_FSIZE, &limit) == -1) - ha_warning("[%s.main()] Failed to set the raise the maximum file size.\n", argv[0]); -#endif - -#if defined(RLIMIT_CORE) - if (setrlimit(RLIMIT_CORE, &limit) == -1) - ha_warning("[%s.main()] Failed to set the raise the core dump size.\n", argv[0]); -#endif - -#if defined(USE_PRCTL) - if (prctl(PR_SET_DUMPABLE, 1, 0, 0, 0) == -1) - ha_warning("[%s.main()] Failed to set the dumpable flag, no core will be dumped.\n", argv[0]); -#endif - } - /* check ulimits */ limit.rlim_cur = limit.rlim_max = 0; getrlimit(RLIMIT_NOFILE, &limit); @@ -3253,6 +3228,31 @@ int main(int argc, char **argv) fork_poller(); } + /* try our best to re-enable core dumps depending on system capabilities. +* What is addressed here : +* - remove file size limits +* - remove core size limits +* - mark the process dumpable again if it lost it due to user/group +*/ + if (global.tune.options & GTUNE_SET_DUMPABLE) { + limit.rlim_cur = limit.rlim_max = RLIM_INFINITY; + +#if defined(RLIMIT_FSIZE) + if (setrlimit(RLIMIT_FSIZE, &limit) == -1) + ha_warning("[%s.main()] Failed to set the raise the maximum file size.\n", argv[0]); +#endif + +#if defined(RLIMIT_CORE) + if (setrlimit(RLIMIT_CORE, &limit) == -1) + ha_warning("[%s.main()] Failed to set the raise the core dump size.\n", argv[0]); +#endif + +#if defined(USE_PRCTL) + if (prctl(PR_SET_DUMPABLE, 1, 0, 0, 0) == -1) + ha_warning("[%s.main()] Failed to set the dumpable flag, no core will be dumped.\n", argv[0]); +#endif + } + global.mode &= ~MODE_STARTING; /* * That's it : the central polling loop. Run until we stop. -- 2.24.0
Re: [PATCH] BUG/MINOR: init: fix set-dumpable when using uid/gid
вт, 19 нояб. 2019 г. в 14:15, William Dauchy : > in mworker mode used with uid/gid settings, it was not possible to get > a coredump despite the set-dumpable option. > indeed prctl(2) manual page specifies the dumpable attribute is reverted > to `/proc/sys/fs/suid_dumpable` in a few conditions such as process > effective user and group are changed. > small question. `/proc/sys/fs/suid_dumpable` is linux specific. will it work under freebsd, openbsd ? windows ? also, linux might not mount that filesystem. will it work ? > > this patch moves the whole set-dumpable logic before the polling code in > order to catch all possible cases where we could have changed the > uid/gid. It however does not cover the possible segfault at startup. > > this patch should be backported in 2.0. > > Signed-off-by: William Dauchy > > --- > > Hi Willy, > > Here is the backport for haproxy-20 tree. > > Thanks, > > William > > --- > src/haproxy.c | 50 +- > 1 file changed, 25 insertions(+), 25 deletions(-) > > diff --git a/src/haproxy.c b/src/haproxy.c > index fa3fbad4..a0e630df 100644 > --- a/src/haproxy.c > +++ b/src/haproxy.c > @@ -2946,31 +2946,6 @@ int main(int argc, char **argv) > } > } > > - /* try our best to re-enable core dumps depending on system > capabilities. > -* What is addressed here : > -* - remove file size limits > -* - remove core size limits > -* - mark the process dumpable again if it lost it due to > user/group > -*/ > - if (global.tune.options & GTUNE_SET_DUMPABLE) { > - limit.rlim_cur = limit.rlim_max = RLIM_INFINITY; > - > -#if defined(RLIMIT_FSIZE) > - if (setrlimit(RLIMIT_FSIZE, &limit) == -1) > - ha_warning("[%s.main()] Failed to set the raise > the maximum file size.\n", argv[0]); > -#endif > - > -#if defined(RLIMIT_CORE) > - if (setrlimit(RLIMIT_CORE, &limit) == -1) > - ha_warning("[%s.main()] Failed to set the raise > the core dump size.\n", argv[0]); > -#endif > - > -#if defined(USE_PRCTL) > - if (prctl(PR_SET_DUMPABLE, 1, 0, 0, 0) == -1) > - ha_warning("[%s.main()] Failed to set the dumpable > flag, no core will be dumped.\n", argv[0]); > -#endif > - } > - > /* check ulimits */ > limit.rlim_cur = limit.rlim_max = 0; > getrlimit(RLIMIT_NOFILE, &limit); > @@ -3253,6 +3228,31 @@ int main(int argc, char **argv) > fork_poller(); > } > > + /* try our best to re-enable core dumps depending on system > capabilities. > +* What is addressed here : > +* - remove file size limits > +* - remove core size limits > +* - mark the process dumpable again if it lost it due to > user/group > +*/ > + if (global.tune.options & GTUNE_SET_DUMPABLE) { > + limit.rlim_cur = limit.rlim_max = RLIM_INFINITY; > + > +#if defined(RLIMIT_FSIZE) > + if (setrlimit(RLIMIT_FSIZE, &limit) == -1) > + ha_warning("[%s.main()] Failed to set the raise > the maximum file size.\n", argv[0]); > +#endif > + > +#if defined(RLIMIT_CORE) > + if (setrlimit(RLIMIT_CORE, &limit) == -1) > + ha_warning("[%s.main()] Failed to set the raise > the core dump size.\n", argv[0]); > +#endif > + > +#if defined(USE_PRCTL) > + if (prctl(PR_SET_DUMPABLE, 1, 0, 0, 0) == -1) > + ha_warning("[%s.main()] Failed to set the dumpable > flag, no core will be dumped.\n", argv[0]); > +#endif > + } > + > global.mode &= ~MODE_STARTING; > /* > * That's it : the central polling loop. Run until we stop. > -- > 2.24.0 > > >
Re: [PATCH] BUG/MINOR: init: fix set-dumpable when using uid/gid
Hi, On Tue, Nov 19, 2019 at 02:42:23PM +0500, Илья Шипицин wrote: > small question. > `/proc/sys/fs/suid_dumpable` is linux specific. will it work under freebsd, > openbsd ? windows ? > also, linux might not mount that filesystem. will it work ? this code is protected around USE_PRCTL define, so I guess it is not selected on other OS. for the linux part, we do not use this path in proc fs, I was referring to that to explain the behaviour of prctl. hope it helps, -- William
HAProxy question
Hello, As part of a project, I have some questions about HAProxy's abilities. Could you confirm if HAProxy is able to handle the following points? 1. STP Protection (RSTP) 2. VLANs interfaces 3. HA Cluster in Active / Passive mode 4. SNMP for monitoring 5. HealthCheck of LDAP services 6. Round robin and failover load balancing 7. Routing flows to a specific pool based on the source IP address 8. Filtering incoming flow by IP/port 9. Oneconnect" type profile Thanks for your support. Regards Micael Gillet Courriel confidentiel: Ce message est protégé par les règles relatives au secret des correspondances. Il est donc établi à destination exclusive de son destinataire. Celui-ci peut donc contenir des informations confidentielles. La divulgation de ces informations est à ce titre rigoureusement interdite. Si vous avez reçu ce message par erreur, merci de le renvoyer à l'expéditeur dont l'adresse e-mail figure ci-dessus et de détruire le message ainsi que toute pièce jointe. This message is protected by the secrecy of correspondence rules. Therefore, this message is intended solely for the attention of the addressee. This message may contain privileged or confidential information, as such the disclosure of these informations is strictly forbidden. If, by mistake, you have received this message, please return this message to the addressser whose e-mail address is written above and destroy this message and all files attached. [https://www.msa.fr/lfy/documents/98830/d420d3b1-9e7c-05ab-6765-009d1e6c1d1f?t=1572346078784]
Re: [PATCH] BUG/MINOR: init: fix set-dumpable when using uid/gid
On Tue, Nov 19, 2019 at 10:11:36AM +0100, William Dauchy wrote: > Here is the backport for haproxy-20 tree. Now merged, thanks William. Willy
Re: HAProxy question
Hi. Nov 19, 2019 11:05:34 AM Micael Gillet : > Hello, As part of a project, I have some questions about HAProxy's abilities. > Could you confirm if HAProxy is able to handle the following points? > > * STP Protection (RSTP) > * VLANs interfaces This is to low level for HAProxy, IMHO. > * HA Cluster in Active / Passive mode Yes it's possible. > * SNMP for monitoring Not out of the box but with tools possible. > * HealthCheck of LDAP services > * Round robin and failover load balancing > * Routing flows to a specific pool based on the source IP address > * Filtering incoming flow by IP/port Yes it's possible. > * Oneconnect" type profile Is this what you mean with that question? https://support.f5.com/csp/article/K7208 It looks like you want to replace a F5 cluster. I would recommend to get in touch with HAProxy Company for a proposal as I assume that the commercial product will fit in your requirements. > Thanks for your support. Regards Micael Gillet Regards aleks > Courriel confidentiel: > Ce message est protégé par les règles relatives au secret des > correspondances. Il est donc établi à destination exclusive de son > destinataire. Celui-ci peut donc contenir des informations confidentielles. > La divulgation de ces informations est à ce titre rigoureusement interdite. > Si vous avez reçu ce message par erreur, merci de le renvoyer à l'expéditeur > dont l'adresse e-mail figure ci-dessus et de détruire le message ainsi que > toute pièce jointe. > This message is protected by the secrecy of correspondence rules. Therefore, > this message is intended solely for the attention of the addressee. This > message may contain privileged or confidential information, as such the > disclosure of these informations is strictly forbidden. If, by mistake, you > have received this message, please return this message to the addressser > whose e-mail address is written above and destroy this message and all files > attached. > > > [https://www.msa.fr/lfy/documents/98830/d420d3b1-9e7c-05ab-6765-009d1e6c1d1f?t=1572346078784] >
Re: Firewall and Haproxy
On Sun, Nov 17, 2019 at 2:41 PM TomK wrote: > Hey All, > > When adding hosts to a F/W behind a VIP (keepalived for example) to > which Haproxy is bound, should just the VIP be added to the F/W or would > all member hosts behind Haproxy need to be added as well? > > If all member hosts behind haproxy need to be added, why? > > Only reason I can think of adding individual host members is for > troubleshooting purposes. Other then that, can't think of a valid > reason why each member host would connect separately. > > -- > Thx, > TK. > > Hi, You should just open traffic to ports configured on the VIP in HAProxy. Baptiste
http-buffer-request details
hello, how is that supposed to work ? https://github.com/haproxy/haproxy/blob/master/doc/configuration.txt#L6225 does it buffer the entire body ? does it use memory / hdd for buffering ? how are those buffers allocated ? what if I do not have a lot of RAM ? thanks, Ilya Shipitcin
Re: [PATCH] MINOR: contrib/prometheus-exporter: allow to select the exported metrics
Hi William, I missed Pierre's email. I'm CCing him. Le 18/11/2019 à 21:00, William Dauchy a écrit : Thanks. Having a way to filter metrics in the Prometheus exporter was on my todo-list :) Filtering on scopes is pretty simple and it is a good start to solve performance issues for huge configs. It is a good idea. However, IMHO, it is easier to use the query-string to select exported metrics. I don't know if it is compatible with the way Prometheus is used. For instance, based on your idea, to get only metrics about servers, the URI could be "/metric?scope=server". And to get metrics about frontends and backends, it could be "/metric?scope=frontend&scope=backend". Of course, it is extensible. We can imagine to add filters on frontend/backend/server names. Here is a quick patch based on this. What do you think about it ? If you said me your way to select metrics is better from the Prometheus point of view, I'm ok with that. In fact I even did not thought about it because my state of mind was to filter at startup like we were doing before. I like you propostion as it is way more extensible than mine. I was not even thinking it would be possible to access the URL, I now have a lot of ideas of the future ;) example config would look like: - job_name: 'lb-builtin-metrics' metrics_path: '/metrics?scope=backend' consul_sd_configs: - server: localhost:8500 services: - lb-builtin-exporter relabel_configs: - source_labels: ['__meta_consul_dc'] target_label: 'dc' - source_labels: ['__address__', '__meta_consul_service_id'] target_label: 'instance' If you have other things on your todo list regarding that feel free to share so we might share the work around it. Well, for now, there is nothing concrete. But I've planned to add a filter on frontend/backend names and, if possible, a way to list the metrics to export. The scope idea may be used too for this last one, by grouping metrics by categories (http, conn, timers, checks, rates...). In the end, it's more a question of time than anything else :) Also have you seen the message from Pierre? They are a few things we would like to discuss whether it would be possible to aggregate a few things on backend side. https://www.mail-archive.com/haproxy@formilux.org/msg35369.html (we somehow forgot to put you in copy though) Regarding the problem of servers in maintenance, since we parse the query-string, it is possible to add more filters. I may add a parameter to filter out servers in maintenance. For instance, by passing "no-maint" in the query-string, all servers in maintenance could be ignored. This way, it would still be possible to have servers metrics. For others points, I will reply to the previous email. Thank you for this patch. Do you think it could be acceptable to mark it as a possible backport for v2.0.x? It's quite critical on our side as we are dealing with ~130MB on some cases. If not we will do the backport on our wide while waiting for the v2.1. It can safely be backported to 2.0. It is not a critical component and it is independent of other parts. I will push a test based on your patch tomorrow on our side. Ok, let me know if you encounter any issues. Thanks, -- Christopher Faulet
Re: native prometheus exporter: retrieving check_status
Hi Pierre, Sorry I missed you email. Thanks to William for the reminder. Le 15/11/2019 à 15:55, Pierre Cheynier a écrit : We've recently tried to switch to the native prometheus exporter, but went quickly stopped in our initiative given the output on one of our preprod server: $ wc -l metrics.out 1478543 metrics.out $ ls -lh metrics.out -rw-r--r-- 1 pierre pierre 130M nov. 15 15:33 metrics.out This is not only due to a large setup, but essentially related to server lines, since we extensively user server-templates for server addition/deletion at runtime. # backend & servers number $ echo "show stat -1 2 -1" | sudo socat stdio /var/lib/haproxy/stats | wc -l 1309 $ echo "show stat -1 4 -1" | sudo socat stdio /var/lib/haproxy/stats | wc -l 36360 # But a lot of them are actually "waiting to be provisioned" (especially on this preprod environment) $ echo "show stat -1 4 -1" | sudo socat stdio /var/lib/haproxy/stats | grep MAINT | wc -l 34113 We'll filter out the server metrics as a quick fix, and will hopefully submit something to do it natively, but we would also like to get your feedbacks about some use-cases we expected to solve with this native exporter. I addressed this issue based on a William's idea. I also proposed to add a filter to exclude all servers in maintenance from the export. Let me know if you see a better way to do so. For the moment, from the exporter point of view, it is not really hard to do such filtering. Ultimately, one of them would be a great value-added for us: being able to count check_status types (and their values in the L7STS case) per backend. So, there are 3 associated points: * it's great to have new metrics (such as `haproxy_process_current_zlib_memory`), but we also noticed that some very useful ones were not present due to their type, example: [ST_F_CHECK_STATUS] = IST("untyped"), What could be done to be able to retrieve them? (I thought about something similar to `HRSP_[1-5]XX`, where the different check status could be defined and counted). Hum, I can add the check status. Mapping all status on integers is possible. However, having a metric per status is probably not the right solution, because it is not a counter but just a state (a boolean). If we do so, all status would be set to 0 except the current status. It is not really handy. But a mapping is possible. We already do this for the frontend/backend/server status (ST_F_STATUS). * also for `check_status`, there is the case of L7STS and its associated values that are present in another field. Most probably it could benefit from a better representation in a prometheus output (thanks to labels)? We can also export the metrics ST_F_CHECK_CODE. For the use of labels, I have no idea. For now, the labels are static in the exporter. And I don't know if it is pertinent to add dynamic info in labels. If so, what is your idea ? Add a "code" label associated to the check_status metric ? * what about getting some backend-level aggregation of server metrics, such as the one that was previously mentioned, to avoid retrieving all the server metrics but still be able to get some insights? I'm thinking about an aggregation of some fields at backend level, which was not previously done with the CSV output. It is feasible. But only counters may be aggregated. It may be enabled using a parameter in the query-string. However, it is probably pertinent only when the server metrics are filtered out. Because otherwise, Prometheus can handle the aggregation itself. -- Christopher Faulet
master-worker no-exit-on-failure with SO_REUSEPORT and a port being already in use
Hi list, I'm facing some issues with already in use ports and the fallback feature, during a reload. SO_REUSEPORT already makes ist easier/better but not perfect, as there are still cases were it fails. In my test case I've got a Squid running on port 80 and a HAProxy with "master-worker no-exit-on-failure". I am using the shipped (2.0.8) systemd unit file and startup HAProxy with some frontend and a bind on like 1337 or something. I then add another bind for port 80, which is in use by squid already and try to reload HAProxy. It takes some time until it failes: Nov 19 14:39:21 894a0f616fec haproxy[2978]: [WARNING] 322/143921 (2978) : Reexecuting Master process ... Nov 19 14:39:28 894a0f616fec haproxy[2978]: [ALERT] 322/143922 (2978) : Starting frontend somefrontend: cannot bind socket [0.0.0.0:80] ... Nov 19 14:39:28 894a0f616fec systemd[1]: haproxy.service: Main process exited, code=exited, status=1/FAILURE The reload itself is still running (systemd) and will timeout after about 90s. After that, because of the Restart=always, I guess, it ends up in a restart loop. So I would have expected that the master process will fallback to the old process and proceed with the old child until the problem has been fixed. Can anybody confirm that? Is that intended? https://cbonte.github.io/haproxy-dconv/2.0/management.html#4 https://cbonte.github.io/haproxy-dconv/2.0/configuration.html#3.1-master-worker -- Regards, Christian Ruppert
Re: master-worker no-exit-on-failure with SO_REUSEPORT and a port being already in use
On Tue, Nov 19, 2019 at 03:45:09PM +0100, Christian Ruppert wrote: > Hi list, > Hello, > I'm facing some issues with already in use ports and the fallback > feature, during a reload. SO_REUSEPORT already makes ist easier/better > but not perfect, as there are still cases were it fails. > In my test case I've got a Squid running on port 80 and a HAProxy with > "master-worker no-exit-on-failure". The "no-exit-on-failure" option is only useful when you don't want the master to kill all the HAProxy processes when one of the workers was killed by another thing that the master (segv, OOM, bug..). In this case you still need another worker available to do the job. It's mostly used with a configuration with nbproc > 1. > I am using the shipped (2.0.8) > systemd unit file and startup HAProxy with some frontend and a bind on > like 1337 or something. > I then add another bind for port 80, which is in use by squid already > and try to reload HAProxy. It takes some time until it failes: > > Nov 19 14:39:21 894a0f616fec haproxy[2978]: [WARNING] 322/143921 (2978) > : Reexecuting Master process > ... > Nov 19 14:39:28 894a0f616fec haproxy[2978]: [ALERT] 322/143922 (2978) : > Starting frontend somefrontend: cannot bind socket [0.0.0.0:80] > ... > Nov 19 14:39:28 894a0f616fec systemd[1]: haproxy.service: Main process > exited, code=exited, status=1/FAILURE > > The reload itself is still running (systemd) and will timeout after > about 90s. After that, because of the Restart=always, I guess, it ends > up in a restart loop. > > So I would have expected that the master process will fallback to the > old process and proceed with the old child until the problem has been > fixed. > > Can anybody confirm that? Is that intended? > > https://cbonte.github.io/haproxy-dconv/2.0/management.html#4 > https://cbonte.github.io/haproxy-dconv/2.0/configuration.html#3.1-master-worker > Looks like a bug to me, the master should have fallback to the "waitpid mode" in this case. Maybe we don't send the sd_notify OK when we are in waitpid mode and systemd kills the process after the reload timeout. I'll do some tests to check what's going on. -- William Lallemand
Re: http-buffer-request details
Le 19/11/2019 à 13:20, Илья Шипицин a écrit : hello, how is that supposed to work ? https://github.com/haproxy/haproxy/blob/master/doc/configuration.txt#L6225 does it buffer the entire body ? does it use memory / hdd for buffering ? how are those buffers allocated ? what if I do not have a lot of RAM ? Hi Ilya, As mentioned in the documentation, HTTP processing is delayed waiting the whole body is received or the request buffer is full. The condition about the first chunk of a chunked request is only valid for the legacy HTTP mode. It was removed in 2.1, so the documentation is a bit outdated on this point. BTW, this means that HAProxy waits for the entire request's body before processing it, if this one is smaller than a buffer. Otherwise, it waits the request buffer is full. In this case, only a part of the payload will be available. But there is no extra memory allocated to store the entire body. About the buffers allocation, as for most of other objects dynamically allocated in HAProxy, it uses memory pools to do so. The buffer size if 16k by default and may be tuned via the "tune.bufsize" global directive. Hope it helps, -- Christopher Faulet
Re: http-buffer-request details
Christopher, Am 19.11.19 um 16:23 schrieb Christopher Faulet: > As mentioned in the documentation, HTTP processing is delayed waiting > the whole body is received or the request buffer is full. The condition > about the first chunk of a chunked request is only valid for the legacy > HTTP mode. It was removed in 2.1, so the documentation is a bit outdated > on this point. Consider filing a short `type: doc` issue if you realize that the documentation is outdated, but you don't have time to think about a proper wording / creating a patch: https://github.com/haproxy/haproxy/issues?q=is%3Aopen+is%3Aissue+label%3A%22type%3A+doc%22 That way one can come around later and make the necessary adjustments when there's a bit more time. Best regards Tim Düsterhus
Re: [PATCH] MINOR: contrib/prometheus-exporter: allow to select the exported metrics
Le 19/11/2019 à 14:51, Christopher Faulet a écrit : Regarding the problem of servers in maintenance, since we parse the query-string, it is possible to add more filters. I may add a parameter to filter out servers in maintenance. For instance, by passing "no-maint" in the query-string, all servers in maintenance could be ignored. This way, it would still be possible to have servers metrics. William, Here is updated patches with the support for "scope" and "no-maint" parameters. If this solution is good enough for you (and if it works :), I will push it. Thanks, -- Christopher Faulet >From 205c4775ccec914a2a501fe9499a77b96285315b Mon Sep 17 00:00:00 2001 From: Christopher Faulet Date: Mon, 18 Nov 2019 14:47:08 +0100 Subject: [PATCH 1/2] MINOR: contrib/prometheus-exporter: filter exported metrics by scope Now, the prometheus exporter parses the HTTP query-string to filter or to adapt the exported metrics. In this first version, it is only possible select the scopes of metrics to export. To do so, one or more parameters with "scope" as name must be passed in the query-string, with one of those values: global, frontend, backend, server or '*' (means all). A scope parameter with no value means to filter out all scopes (nothing is returned). The scope parameters are parsed in their appearance order in the query-string. So an empty scope will reset all scopes already parsed. But it can be overridden by following scope parameters in the query-string. By default everything is exported. The filtering can also be done on prometheus scraping configuration, but general aim is to optimise the source of data to improve load and scraping time. This is particularly true for huge configuration with thousands of backends and servers. Also note that this configuration was possible on the previous official haproxy exporter but with even more parameters to select the needed metrics. Here we thought it was sufficient to simply avoid a given type of metric. However, more filters are still possible. Thanks to William Dauchy. This patch is based on his work. --- contrib/prometheus-exporter/README| 24 +++ .../prometheus-exporter/service-prometheus.c | 157 +++--- 2 files changed, 156 insertions(+), 25 deletions(-) diff --git a/contrib/prometheus-exporter/README b/contrib/prometheus-exporter/README index 915fc7f54..84ae8e27e 100644 --- a/contrib/prometheus-exporter/README +++ b/contrib/prometheus-exporter/README @@ -50,6 +50,30 @@ spend much more ressources to produce the Prometheus metrics than the CSV export through the stats page. To give a comparison order, quick benchmarks shown that a PROMEX dump is 5x slower and 20x more verbose than a CSV export. + +metrics filtering +--- + +It is possible to dynamically select the metrics to export if you don't use all +of them passing parameters in the query-string. + +* Filtering on scopes + +The metrics may be filtered by scopes. Multiple parameters with "scope" as name +may be passed in the query-string to filter exported metrics, with one of those +values: global, frontend, backend, server or '*' (means all). A scope parameter +with no value means to filter out all scopes (nothing is returned). The scope +parameters are parsed in their appearance order in the query-string. So an empty +scope will reset all scopes already parsed. But it can be overridden by +following scope parameters in the query-string. By default everything is +exported. Here are examples: + + /metrics?scope=server # ==> server metrics will be exported + /metrics?scope=frontend&scope=backend # ==> Frontend and backend metrics will be exported + /metrics?scope=*&scope= # ==> no metrics will be exported + /metrics?scope=&scope=global # ==> global metrics will be exported + + Exported metrics -- diff --git a/contrib/prometheus-exporter/service-prometheus.c b/contrib/prometheus-exporter/service-prometheus.c index 508e6b1fc..c70daa905 100644 --- a/contrib/prometheus-exporter/service-prometheus.c +++ b/contrib/prometheus-exporter/service-prometheus.c @@ -64,6 +64,12 @@ enum { #define PROMEX_FL_METRIC_HDR0x0001 #define PROMEX_FL_INFO_METRIC 0x0002 #define PROMEX_FL_STATS_METRIC 0x0004 +#define PROMEX_FL_SCOPE_GLOBAL 0x0008 +#define PROMEX_FL_SCOPE_FRONT 0x0010 +#define PROMEX_FL_SCOPE_BACK0x0020 +#define PROMEX_FL_SCOPE_SERVER 0x0040 + +#define PROMEX_FL_SCOPE_ALL (PROMEX_FL_SCOPE_GLOBAL|PROMEX_FL_SCOPE_FRONT|PROMEX_FL_SCOPE_BACK|PROMEX_FL_SCOPE_SERVER) /* The max length for metrics name. It is a hard limit but it should be * enougth. @@ -2102,72 +2108,81 @@ static int promex_dump_srv_metrics(struct appctx *appctx, struct htx *htx) static int promex_dump_metrics(struct appctx *appctx, struct stream_interface *si, struct htx *htx) { int ret; + int flags = appctx->ctx.stats.flags; switch (appctx->st1) { case PROMEX_DUMPER_INIT: appctx->c
Re: http-buffer-request details
Le 19/11/2019 à 16:32, Tim Düsterhus a écrit : Christopher, Am 19.11.19 um 16:23 schrieb Christopher Faulet: As mentioned in the documentation, HTTP processing is delayed waiting the whole body is received or the request buffer is full. The condition about the first chunk of a chunked request is only valid for the legacy HTTP mode. It was removed in 2.1, so the documentation is a bit outdated on this point. Consider filing a short `type: doc` issue if you realize that the documentation is outdated, but you don't have time to think about a proper wording / creating a patch: https://github.com/haproxy/haproxy/issues?q=is%3Aopen+is%3Aissue+label%3A%22type%3A+doc%22 That way one can come around later and make the necessary adjustments when there's a bit more time. You're right Tim. But this one is small enough to be fixed immediately :). I will push the patch with other ones I'm working on. But it is already fixed. -- Christopher Faulet
Re: http-buffer-request details
Christopher, Am 19.11.19 um 16:39 schrieb Christopher Faulet: > You're right Tim. But this one is small enough to be fixed immediately > :). I will push the patch with other ones I'm working on. But it is > already fixed. > That's even better of course! Disregard my comment in *this specific* case then :-) Best regards Tim Düsterhus
Re: native prometheus exporter: retrieving check_status
On Tue, Nov 19, 2019 at 03:31:28PM +0100, Christopher Faulet wrote: > > * also for `check_status`, there is the case of L7STS and its associated > > values that are present in another field. Most probably it could benefit > > from a better representation in a prometheus output (thanks to labels)? > > We can also export the metrics ST_F_CHECK_CODE. For the use of labels, I > have no idea. For now, the labels are static in the exporter. And I don't > know if it is pertinent to add dynamic info in labels. If so, what is your > idea ? Add a "code" label associated to the check_status metric ? we need to be very careful here indeed. It's not very clear in my mind how much values we are talking about, but labels trigger the creation of a new metric of each key/value pair. So it can quickly explode your memory on scrapping side. > > * what about getting some backend-level aggregation of server metrics, such > > as the one that was previously mentioned, to avoid retrieving all the > > server metrics but still be able to get some insights? > > I'm thinking about an aggregation of some fields at backend level, which > > was not previously done with the CSV output. > > It is feasible. But only counters may be aggregated. It may be enabled using > a parameter in the query-string. However, it is probably pertinent only when > the server metrics are filtered out. Because otherwise, Prometheus can > handle the aggregation itself. My only fear for this point would be to make the code too complicated and harder to maintain. -- William
Re: [PATCH] MINOR: contrib/prometheus-exporter: allow to select the exported metrics
Hi Christopher, On Tue, Nov 19, 2019 at 04:35:47PM +0100, Christopher Faulet wrote: > Here is updated patches with the support for "scope" and "no-maint" > parameters. If this solution is good enough for you (and if it works :), I > will push it. this looks good to me and the test was conclusive on our side. For now we won't make use of no-maint as the server are completely deactivated. We should however measure how much data does it represent now, in order to see if we could simply use no-maint parameter. I will push the no-maint patch on our production later today. Thank you for your work, -- William
RE: native prometheus exporter: retrieving check_status
> Hi Pierre, Hi!, > I addressed this issue based on a William's idea. I also proposed to add a > filter to exclude all servers in maintenance from the export. Let me know if > you > see a better way to do so. For the moment, from the exporter point of view, > it > is not really hard to do such filtering. Yes, that's a great addition, and should improve a lot, but I'm still not sure if it will be sufficient (meaning that we could end up dropping all servers if the endpoint are still too huge, as we used to do with the old exporter). BTW, we also did that since we're using server-templates, and the naming in templates make the server-name information useless (since we can't modify the name at runtime). So we previously had a sufficient level of info at backend level, thanks to native aggregations. >> [ST_F_CHECK_STATUS] = IST("untyped"), >> What could be done to be able to retrieve them? (I thought about something >> similar to >> `HRSP_[1-5]XX`, where the different check status could be defined and >> counted). >> > > Hum, I can add the check status. Mapping all status on integers is possible. > However, having a metric per status is probably not the right solution, > because > it is not a counter but just a state (a boolean). If we do so, all status > would > be set to 0 except the current status. It is not really handy. But a mapping > is > possible. We already do this for the frontend/backend/server status > (ST_F_STATUS). Yes, it would work perfectly. At the end the goal for us would be to be able to retrieve this state. My idea about a counter was more about backend-level aggregations, if consistent (I'm not sure it is actually, hence the feedback request). >> * also for `check_status`, there is the case of L7STS and its associated >> values that are present >> in another field. Most probably it could benefit from a better >> representation in a prometheus >> output (thanks to labels)? >> > We can also export the metrics ST_F_CHECK_CODE. For the use of labels, I have > no > idea. For now, the labels are static in the exporter. And I don't know if it > is > pertinent to add dynamic info in labels. If so, what is your idea ? Add a > "code" > label associated to the check_status metric ? Here again, my maybe-not-so-good idea was to keep the ability to retrieve all the underlying details at backend level, such as: * 100 servers are L7OK * 1 server is L4TOUT * 2 servers are L4CON * 2 servers are L7STS ** 1 due to a HTTP 429 ** 1 due to a HTTP 503 But this is maybe overkill in terms of complexity, we could maybe push more on our ability to retrieve non-maint servers status. > It is feasible. But only counters may be aggregated. It may be enabled using > a > parameter in the query-string. However, it is probably pertinent only when > the > server metrics are filtered out. Because otherwise, Prometheus can handle the > aggregation itself. Sure, we should rely on this as much as possible. -- Pierre
Re: master-worker no-exit-on-failure with SO_REUSEPORT and a port being already in use
On Tue, Nov 19, 2019 at 04:19:26PM +0100, William Lallemand wrote: > > I then add another bind for port 80, which is in use by squid already > > and try to reload HAProxy. It takes some time until it failes: > > > > Nov 19 14:39:21 894a0f616fec haproxy[2978]: [WARNING] 322/143921 (2978) > > : Reexecuting Master process > > ... > > Nov 19 14:39:28 894a0f616fec haproxy[2978]: [ALERT] 322/143922 (2978) : > > Starting frontend somefrontend: cannot bind socket [0.0.0.0:80] > > ... > > Nov 19 14:39:28 894a0f616fec systemd[1]: haproxy.service: Main process > > exited, code=exited, status=1/FAILURE > > > > The reload itself is still running (systemd) and will timeout after > > about 90s. After that, because of the Restart=always, I guess, it ends > > up in a restart loop. > > > > So I would have expected that the master process will fallback to the > > old process and proceed with the old child until the problem has been > > fixed. > > The patch in attachment fixes a bug where haproxy could reexecute itself in waitpid mode with -sf -1. I'm not sure this is your bug, but if this is the case you should see haproxy in waitpid mode, then the master exiting with the usage message in your logs. -- William Lallemand >From 481a3c62a622974587c731b1bdc1478538fd6527 Mon Sep 17 00:00:00 2001 From: William Lallemand Date: Tue, 19 Nov 2019 17:04:18 +0100 Subject: [PATCH] BUG/MEDIUM: mworker: don't fill the -sf argument with -1 during the reexec Upon a reexec_on_failure, if the process tried to exit after the initialization of the process structure but before it was filled with a PID, the PID in the mworker_proc structure is set to -1. In this particular case the -sf argument is filled with -1 and haproxy will exit with the usage message because of that argument. Should be backported in 2.0. --- src/haproxy.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/src/haproxy.c b/src/haproxy.c index a0e630dfa..1d4771e64 100644 --- a/src/haproxy.c +++ b/src/haproxy.c @@ -673,7 +673,7 @@ void mworker_reload() next_argv[next_argc++] = "-sf"; list_for_each_entry(child, &proc_list, list) { - if (!(child->options & (PROC_O_TYPE_WORKER|PROC_O_TYPE_PROG))) + if (!(child->options & (PROC_O_TYPE_WORKER|PROC_O_TYPE_PROG)) || child->pid <= -1 ) continue; next_argv[next_argc] = memprintf(&msg, "%d", child->pid); if (next_argv[next_argc] == NULL) -- 2.21.0
travis-ci: should we drop openssl-1.1.0 and replace it with 3.0 ?
hello, https://www.openssl.org/source/ says "The 1.1.0 series is currently only receiving security fixes and will go out of support on 11th September 2019" what if we drop it ? and replace with 3.0 ? cheers, Ilya Shipitcin
Re: travis-ci: should we drop openssl-1.1.0 and replace it with 3.0 ?
well, we can actually build bigger matrix by adding builds. I just want to save some electricity on non needed builds. вт, 19 нояб. 2019 г. в 22:41, Илья Шипицин : > hello, > > https://www.openssl.org/source/ says "The 1.1.0 series is currently only > receiving security fixes and will go out of support on 11th September 2019" > > > what if we drop it ? and replace with 3.0 ? > > cheers, > Ilya Shipitcin >
RE: travis-ci: should we drop openssl-1.1.0 and replace it with 3.0 ?
Maybe after they stop security fixes we can drop 1.1.0. I know there are many distributions still in support that use this branch. 3.0 doesn’t exist yet, and won’t until later in 2020 which is unfortunate since that means there will be no FIPS validated branch for several months. From: Илья Шипицин [mailto:chipits...@gmail.com] Sent: Tuesday, November 19, 2019 12:48 PM To: HAProxy Subject: Re: travis-ci: should we drop openssl-1.1.0 and replace it with 3.0 ? well, we can actually build bigger matrix by adding builds. I just want to save some electricity on non needed builds. вт, 19 нояб. 2019 г. в 22:41, Илья Шипицин mailto:chipits...@gmail.com>>: hello, https://www.openssl.org/source/ says "The 1.1.0 series is currently only receiving security fixes and will go out of support on 11th September 2019" what if we drop it ? and replace with 3.0 ? cheers, Ilya Shipitcin Information in this e-mail may be confidential. It is intended only for the addressee(s) identified above. If you are not the addressee(s), or an employee or agent of the addressee(s), please note that any dissemination, distribution, or copying of this communication is strictly prohibited. If you have received this e-mail in error, please notify the sender of the error.
Re: travis-ci: should we drop openssl-1.1.0 and replace it with 3.0 ?
yep, 3.0 stands for openssl master branch. the point is to catch incompatibilities before it is released. вт, 19 нояб. 2019 г. в 22:51, Gibson, Brian (IMS) : > Maybe after they stop security fixes we can drop 1.1.0. I know there are > many distributions still in support that use this branch. 3.0 doesn’t > exist yet, and won’t until later in 2020 which is unfortunate since that > means there will be no FIPS validated branch for several months. > > > > *From:* Илья Шипицин [mailto:chipits...@gmail.com] > *Sent:* Tuesday, November 19, 2019 12:48 PM > *To:* HAProxy > *Subject:* Re: travis-ci: should we drop openssl-1.1.0 and replace it > with 3.0 ? > > > > well, we can actually build bigger matrix by adding builds. I just want to > save some electricity on non needed builds. > > > > вт, 19 нояб. 2019 г. в 22:41, Илья Шипицин : > > hello, > > > > https://www.openssl.org/source/ says "The 1.1.0 series is currently only > receiving security fixes and will go out of support on 11th September 2019" > > > > > > what if we drop it ? and replace with 3.0 ? > > > > cheers, > > Ilya Shipitcin > > > -- > > Information in this e-mail may be confidential. It is intended only for > the addressee(s) identified above. If you are not the addressee(s), or an > employee or agent of the addressee(s), please note that any dissemination, > distribution, or copying of this communication is strictly prohibited. If > you have received this e-mail in error, please notify the sender of the > error. >
Re: travis-ci: should we drop openssl-1.1.0 and replace it with 3.0 ?
Hello, On Tuesday, 19 November 2019, Илья Шипицин wrote: > yep, 3.0 stands for openssl master branch. > the point is to catch incompatibilities before it is released. > I am objecting to this. This can be done WHEN openssl declares that the API is stable. Testing and implementing build fixes for APIs while they are under active development not only takes away precious dev time, it's also causes our own code to be messed up with workarounds possibly only needed for specific openssl development code at one point in time. Lukas