Re: 1.9 external health checks fail suddenly
Hi Lukas,
On Tue, Jul 09, 2019 at 03:59:04PM +0200, Lukas Tribus wrote:
> Hello Veiko,
>
>
> On Tue, 9 Jul 2019 at 15:40, Veiko Kukk wrote:
> >
> > On 2019-07-08 16:06, Lukas Tribus wrote:
> > > The bug you may be affected by is:
> > > https://github.com/haproxy/haproxy/issues/141
> > >
> > > Can you check what happens with:
> > > nbthread 1
> >
> > I'm afraid I can't because those are production systems that won't be
> > able to service with single thread, they have relatively high ssl
> > termination load.
>
> You could probably raise nbproc at that point, if you can get away
> with some stats issues ...
>
> How are you currently working around this issue? Did you disable
> external checks? I'd assume failing checks have negative impact on
> production systems also.
>
>
> Willy, in issue #141 in sounds like you already have an idea how this
> could be fixed, is there a patch that we can ask Veiko to try for
> this?
I didn't have a patch but just did it. It was only compile-tested,
please verify that it works as expected on a non-sensitive machine
first!
Cheers,
Willy
>From 32205189f881b98cb0bbe6ed32178f2929e9a627 Mon Sep 17 00:00:00 2001
From: Willy Tarreau
Date: Tue, 9 Jul 2019 16:27:39 +0200
Subject: WIP/BUG: checks: make sure we isolate the thread doing the fork
---
src/checks.c | 3 +++
1 file changed, 3 insertions(+)
diff --git a/src/checks.c b/src/checks.c
index d3920ce8d..46f93e58f 100644
--- a/src/checks.c
+++ b/src/checks.c
@@ -1977,8 +1977,10 @@ static int connect_proc_chk(struct task *t)
block_sigchld();
+ thread_isolate();
pid = fork();
if (pid < 0) {
+ thread_release();
ha_alert("Failed to fork process for external health check: %s.
Aborting.\n",
strerror(errno));
set_server_check_status(check, HCHK_STATUS_SOCKERR,
strerror(errno));
@@ -2015,6 +2017,7 @@ static int connect_proc_chk(struct task *t)
}
/* Parent */
+ thread_release();
if (check->result == CHK_RES_UNKNOWN) {
if (pid_list_add(pid, t) != NULL) {
t->expire = tick_add(now_ms, MS_TO_TICKS(check->inter));
--
2.20.1
Re: 1.9 external health checks fail suddenly
Hello Veiko, On Tue, 9 Jul 2019 at 15:40, Veiko Kukk wrote: > > On 2019-07-08 16:06, Lukas Tribus wrote: > > The bug you may be affected by is: > > https://github.com/haproxy/haproxy/issues/141 > > > > Can you check what happens with: > > nbthread 1 > > I'm afraid I can't because those are production systems that won't be > able to service with single thread, they have relatively high ssl > termination load. You could probably raise nbproc at that point, if you can get away with some stats issues ... How are you currently working around this issue? Did you disable external checks? I'd assume failing checks have negative impact on production systems also. Willy, in issue #141 in sounds like you already have an idea how this could be fixed, is there a patch that we can ask Veiko to try for this? cheers, lukas
Re: 1.9 external health checks fail suddenly
On 2019-07-08 16:06, Lukas Tribus wrote: The bug you may be affected by is: https://github.com/haproxy/haproxy/issues/141 Can you check what happens with: nbthread 1 I'm afraid I can't because those are production systems that won't be able to service with single thread, they have relatively high ssl termination load. Veiko
Re[2]: The case for changing the documentation syntax
It sounds like restructuredText and Asciidoc are the top choices. They both look capable: http://hyperpolyglot.org/lightweight-markup I can, as a next step, post this as an Issue on the Github project and it can be triaged and tracked. For something like this, it might even make sense to create a new branch so that multiple people can work on it. In that case, splitting the documentation into multiple files would be helpful too. If approved, an empty file for each section of the documentation could be created in order to have the skeleton of the project. Having the documentation split into multiple files may make maintaining the documentation easier in the future too (i.e. someone could change one section without conflicting with a person making a change in another section). How have collaborative efforts like this been done in the past? How would multiple people be able to commit changes to this branch? Other thoughts? -- Original Message -- From: "Pavlos Parissis" To: "Nick Ramirez" Sent: 7/3/2019 10:44:11 AM Subject: Re: The case for changing the documentation syntax On Δευτέρα, 1 Ιουλίου 2019 5:01:33 Μ.Μ. CEST Nick Ramirez wrote: Hello all, [...snip...] The solution I am proposing: Rather than using a home-grown, difficult to parse, not-consistently-used grammar. We should use a standard. We should use reStructuredText: http://docutils.sourceforge.net/rst.html The reStructuredText syntax gives us the following benefits: * It is well documented * Tools exist to parse this and convert it to other formats (such as HTML) * Tools exist that will "error check" the document to ensure that the correct syntax is used throughout configuration.txt (which would become configuration.rst) * Tools such as Jekyll can easily parse reStructuredText and build sophisticated, beautiful webpages that feature search functionality, table-of-contents, images, graphs, links, etc. We could really start to make the documentation shine! * We won't have to worry about updating special tools because reStructuredText syntax will allow us to reliably parse it forever * reStructuredText is still easily human-readable using a terminal, plain-text editor, etc. I and others are fully willing to make the conversion to reStructuredText, too. What do you all think? +1 from me. asciidoctor is something you should have a look at and consider as well. I know that people don't like markdown, but it is very simple to use and that is, sometimes, more important than standards and etc. My cents, Pavlos
Re: DOC: Suggest to replace the netstat commands
On 2019-07-09 10:12, Willy Tarreau wrote: On Tue, Jul 09, 2019 at 10:09:36AM +0200, Klaus Foerster wrote: It might be a good idea to show the netstat and the ss command. netstat is for example no more installed by default on ubuntu systems, whereas ss is. Of course netstat can be installed without issues, but it's not there by default. That's sad, though it's understandable given that ubuntu is not exactly made to be primarily used from the command line for most of their users, so they possibly don't care about end-user's experience in production environments where people like to use the same commands on all of their systems. But indeed, indicating what command to run instead of netstat on Linux (at least as a recommended lower cost solution) would be nice. Hello, I agree with Willy, as a Linux user (Debian) I've only seen my point of view based on my Linux experience. If the 'ss' tool is no present on *bsd by example, we have to leave the doc the more compatible as possible. So forgot my suggestion, sorry for your time. Regards, -- [Alain Belkadi / LinuxBeach]
Re: prometheus service kills ssl handshake
Am 08.07.2019 um 12:37 schrieb Aleksandar Lazic: > Hi Christopher. > > Am 08.07.2019 um 10:30 schrieb Christopher Faulet: >> Le 06/07/2019 à 23:02, Aleksandar Lazic a écrit : >>> Hi. >>> >>> I use HAP 2.0.1 with haproxy service with my image. >>> After some times (~several hours) the ssl handshake stops working for the >>> https >>> frontend which offers the prom service. >>> > > [snipp] > >> >> Hi Aleks, >> >> Could you check with the latest 2.0 snapshot ? An issue about Prometheus was >> fixed (#151 on GitHub). And some others about connections. >> > > Okay I created the image with ss-20190706 . > > https://gitlab.com/aleks001/haproxy20-centos/commit/212ed6f4099dd92c72b426726afdf04022065798 After ~20 hours of running Prometheus scraper with ss-20190706 the ssl handshake errors are gone. From my point of view works the Prometheus exporter now. Regards Aleks
Re: DOC: Suggest to replace the netstat commands
On Tue, Jul 09, 2019 at 10:09:36AM +0200, Klaus Foerster wrote: > It might be a good idea to show the netstat and the ss command. > > netstat is for example no more installed by default on ubuntu systems, > whereas ss is. > Of course netstat can be installed without issues, but it's not there by > default. That's sad, though it's understandable given that ubuntu is not exactly made to be primarily used from the command line for most of their users, so they possibly don't care about end-user's experience in production environments where people like to use the same commands on all of their systems. But indeed, indicating what command to run instead of netstat on Linux (at least as a recommended lower cost solution) would be nice. Willy
Re: DOC: Suggest to replace the netstat commands
It might be a good idea to show the netstat and the ss command. netstat is for example no more installed by default on ubuntu systems, whereas ss is. Of course netstat can be installed without issues, but it's not there by default. On 7/9/19 8:20 AM, Willy Tarreau wrote: On Mon, Jul 08, 2019 at 04:51:24PM +0200, Alain Belkadi wrote: Hello, As the "netstat" command is deprecated since a long time (1), I suggest to replace it with other commands like ss and ip. I disagree with this. netstat is not deprecated at all, it's superseded *on linux* because there we have netlink which provides a much faster and more complete interface than the one used by netstat. But netstat is the only command you'll find on about all systems and its output format is pretty consistent. However it might make sense to add a few lines close to the locations where netstat is mentioned to indicate that on Linux ss is preferred since it's much less resource intensive than netstat. Thanks, Willy
