Re: Connections stuck in CLOSE_WAIT state with h2

On Thu, Aug 02, 2018 at 11:29:17AM +0200, Janusz Dziemidowicz wrote: > pt., 27 lip 2018 o 10:35 Willy Tarreau napisal(a): > > > > On Fri, Jul 27, 2018 at 10:28:36AM +0200, Milan Petruzelka wrote: > > > after 2 days I also have no blocked connections. There's no need to wait > > > until Monday as

Re: Connections stuck in CLOSE_WAIT state with h2

pt., 27 lip 2018 o 10:35 Willy Tarreau napisał(a): > > On Fri, Jul 27, 2018 at 10:28:36AM +0200, Milan Petruzelka wrote: > > after 2 days I also have no blocked connections. There's no need to wait > > until Monday as I suggested yesterday. > > Perfect, many thanks Milan. Sorry for being late,

Re: Connections stuck in CLOSE_WAIT state with h2

On Fri, Jul 27, 2018 at 10:28:36AM +0200, Milan Petruzelka wrote: > after 2 days I also have no blocked connections. There's no need to wait > until Monday as I suggested yesterday. Perfect, many thanks Milan. Willy

Re: Connections stuck in CLOSE_WAIT state with h2

On Fri, 27 Jul 2018 at 10:08, Willy Tarreau wrote: > Hi Olivier, > > On Fri, Jul 27, 2018 at 09:04:04AM +0200, Olivier Doucet wrote: > > 24 hours later, still no issue to be reported. All sessions are expiring > > just fine. I think you can merge :) > > Yes I think you're right, I'll do this, it

Re: Connections stuck in CLOSE_WAIT state with h2

Hi Olivier, On Fri, Jul 27, 2018 at 09:04:04AM +0200, Olivier Doucet wrote: > 24 hours later, still no issue to be reported. All sessions are expiring > just fine. I think you can merge :) Yes I think you're right, I'll do this, it will at least help all the users who don't want to patch their

Re: Connections stuck in CLOSE_WAIT state with h2

Hello, 2018-07-26 11:09 GMT+02:00 Willy Tarreau : > Hi Olivier, > > On Thu, Jul 26, 2018 at 10:53:33AM +0200, Olivier Doucet wrote: > > Previous build: > > https://tof.cx/images/2018/07/26/f31243bfede22e20a7a991ae6c39506d.png > > (we can clearly see when reload happens :p) > > > > New build: >

Re: Connections stuck in CLOSE_WAIT state with h2

Hi Olivier, On Thu, Jul 26, 2018 at 10:53:33AM +0200, Olivier Doucet wrote: > Previous build: > https://tof.cx/images/2018/07/26/f31243bfede22e20a7a991ae6c39506d.png > (we can clearly see when reload happens :p) > > New build: >

Re: Connections stuck in CLOSE_WAIT state with h2

Hello, 2018-07-26 10:33 GMT+02:00 Willy Tarreau : > Hi Milan! > > On Thu, Jul 26, 2018 at 10:31:24AM +0200, Milan Petruzelka wrote: > > my haproxy with last two patches has no blocked connection since it > started > > 24 hours ago. Before last patches it always had at least one. I think you > >

Re: Connections stuck in CLOSE_WAIT state with h2

Hi Milan! On Thu, Jul 26, 2018 at 10:31:24AM +0200, Milan Petruzelka wrote: > my haproxy with last two patches has no blocked connection since it started > 24 hours ago. Before last patches it always had at least one. I think you > finally nailed it. To be on the safe side, I would wait till

Re: Connections stuck in CLOSE_WAIT state with h2

On Wed, 25 Jul 2018 at 18:39, Willy Tarreau wrote: > > This issue is very close to Milan bug, that's why I posted as a reply. If > > I'm wrong, I'll split it in another thread. > > It's definitely the same as Milan's from my point of view. Many thanks > for reporting it as well. > > Hi Willy,

Re: Connections stuck in CLOSE_WAIT state with h2

Hi Olivier, On Wed, Jul 25, 2018 at 05:51:46PM +0200, Olivier Doucet wrote: > It seems I have the same issue as Milan : > We activated HTTP/2 on production a few weeks ago, and on some customers > (not all !) we can observe a very strange behaviour : it seems some > frontend sessions are not

Re: Connections stuck in CLOSE_WAIT state with h2

Hello, 2018-07-25 10:20 GMT+02:00 Willy Tarreau : > Hi Milan, > > On Wed, Jul 25, 2018 at 10:15:50AM +0200, Milan Petruzelka wrote: > > Now I'll add both patches (WIP-h2 and h2-error.diff) and give it a try in > > production. > > Thank you. At first I thought you still had the errors with them

Re: Connections stuck in CLOSE_WAIT state with h2

Hi Milan, On Wed, Jul 25, 2018 at 10:15:50AM +0200, Milan Petruzelka wrote: > Now I'll add both patches (WIP-h2 and h2-error.diff) and give it a try in > production. Thank you. At first I thought you still had the errors with them applied and was despeared, now I understand there's still hope

Re: Connections stuck in CLOSE_WAIT state with h2

Hi Willy, On Tue, 24 Jul 2018 at 14:37, Willy Tarreau wrote: > So I'm having one update to emit the missing info on "show fd" (patch > merged > and pushed already, that I'm attaching here if it's easier for you) I've compiled version 1.8.12-12a4b5-16 with from Git and let it run overnight.

Re: Connections stuck in CLOSE_WAIT state with h2

Hi Milan, On Tue, Jul 24, 2018 at 12:23:37PM +0200, Milan Petruzelka wrote: > Hi Willy, > > Do you *think* that you got less CLOSE_WAITs or that the latest fixes > > didn't change anything ? I suspect that for some reason you might be > > hit by several bugs, which is what has complicated the

Re: Connections stuck in CLOSE_WAIT state with h2

Hi Willy, Do you *think* that you got less CLOSE_WAITs or that the latest fixes > didn't change anything ? I suspect that for some reason you might be > hit by several bugs, which is what has complicated the diagnostic, but > that's just pure guess. > > I'm not sure. I left patched haproxy

Re: Connections stuck in CLOSE_WAIT state with h2

Hi Milan, On Mon, Jul 23, 2018 at 08:41:03AM +0200, Milan Petruzelka wrote: > After weekend CLOSE_WAIT connections are still there. Ah bad :-( > What > does cflg=0x80203300 in "show fd" mean? These are the connection flags. You can figure them with contrib/debug/flags : $ ./flags 0x80203300 |

Re: Connections stuck in CLOSE_WAIT state with h2

Hi, I've compiled latest haproxy 1.8.12 from Git repo (HAProxy version 1.8.12-5e100b-15, released 2018/07/20) with latest h2 patches and extended h2 debug info. And after some time I caught one CLOSE_WAIT connection. Here is extended show fd debug: 25 : st=0x20(R:pra W:pRa) ev=0x00(heopi)

Re: Connections stuck in CLOSE_WAIT state with h2

On Fri, 20 Jul 2018 at 14:36, Milan Petruželka wrote: > > I've applied both patches to vanilla haproxy 1.8.12. I'll leave it running > and report back. > > Hi, After weekend CLOSE_WAIT connections are still there. What does cflg=0x80203300 in "show fd" mean? FDs with cflg=0x80203300 are

Re: Connections stuck in CLOSE_WAIT state with h2

On Fri, 20 Jul 2018 at 13:37, Willy Tarreau wrote: > Thank you Janusz for testing. I've also merged the other patch that could > lead to some CLOSE_WAIT that we tested a few months ago. It was different > but maybe you were suffering from the two causes. > I've applied both patches to vanilla

Re: Connections stuck in CLOSE_WAIT state with h2

On Fri, Jul 20, 2018 at 01:30:00PM +0200, Janusz Dziemidowicz wrote: > I've been running 1.8.12 with this patch for an hour. It seems that it > helped somewhat, but not entirely. After an hour I still see about 10 > CLOSE_WAIT sockets. The number seems to grow a lot slower, but still > grows (and

Re: Connections stuck in CLOSE_WAIT state with h2

czw., 19 lip 2018 o 11:14 Willy Tarreau napisał(a): > > Hi Milan, Janusz, > > I suspect I managed to reliably trigger the issue you were facing and > found a good explanation for it. It is caused by unprocessed bytes at > the end of the H1 stream. I manage to reproduce it if I chain two layers >

Re: Connections stuck in CLOSE_WAIT state with h2

Hi Milan, Janusz, I suspect I managed to reliably trigger the issue you were facing and found a good explanation for it. It is caused by unprocessed bytes at the end of the H1 stream. I manage to reproduce it if I chain two layers of haproxy with the last one sending stats. It does not happen

Re: Connections stuck in CLOSE_WAIT state with h2

> > 20180629.1347 mpeh2 fd25 h2c_error - st04 fl0002 err05 > Just hit h2c_error - H2_ERR_STREAM_CLOSED > After adding more debug I found following pattern around h2c_error in hanging connections: ... everything OK until now 20180629.1826 e901:backend.srvrep[000e:001a]: HTTP/1.1 200 OK

Re: Connections stuck in CLOSE_WAIT state with h2

On Fri, 29 Jun 2018 at 11:19, Milan Petruželka wrote: > I've added more debug into h2s_close to see not only h2s state and flags > but also h2c state and flags. My only way to reproduce the bug is to let > Haproxy run until some of its FD falls into CLOSE_WAIT. After I catch some, > I'll report

Re: Connections stuck in CLOSE_WAIT state with h2

Hi Willy, I'm back at work after 2 weeks on the beach in Dalmatia. I've patched my Haproxy 1.8.11 with all three patches discussed here in last two weeks. It didn't help. Then tried to run Haproxy with debug enabled. The last logs from FD hanging in CLOSE_WAIT looks like this:

Re: Connections stuck in CLOSE_WAIT state with h2

2018-06-15 11:21 GMT+02:00 Willy Tarreau : >> I've tried with all three patches, still no luck. I had to revert >> native h2 shortly because I've started getting ERR_SPDY_PROTOCOL_ERROR >> in Chrome. The error was always on POST request. > > Too bad, have to dig again then :-/. > Thank you Janusz!

Re: Connections stuck in CLOSE_WAIT state with h2

On Fri, Jun 15, 2018 at 11:19:04AM +0200, Janusz Dziemidowicz wrote: > 2018-06-14 19:49 GMT+02:00 Willy Tarreau : > > On Thu, Jun 14, 2018 at 07:22:34PM +0200, Janusz Dziemidowicz wrote: > >> 2018-06-14 18:56 GMT+02:00 Willy Tarreau : > >> > >> > If you'd like to run a test, I'm attaching the

Re: Connections stuck in CLOSE_WAIT state with h2

2018-06-14 19:49 GMT+02:00 Willy Tarreau : > On Thu, Jun 14, 2018 at 07:22:34PM +0200, Janusz Dziemidowicz wrote: >> 2018-06-14 18:56 GMT+02:00 Willy Tarreau : >> >> > If you'd like to run a test, I'm attaching the patch. >> >> Sure, but you forgot to attach it :) > > Ah, that's because I'm stupid

Re: Connections stuck in CLOSE_WAIT state with h2

On Thu, Jun 14, 2018 at 07:22:34PM +0200, Janusz Dziemidowicz wrote: > 2018-06-14 18:56 GMT+02:00 Willy Tarreau : > > > If you'd like to run a test, I'm attaching the patch. > > Sure, but you forgot to attach it :) Ah, that's because I'm stupid :-) Here it comes this time. Willy diff --git

Re: Connections stuck in CLOSE_WAIT state with h2

2018-06-14 18:56 GMT+02:00 Willy Tarreau : > If you'd like to run a test, I'm attaching the patch. Sure, but you forgot to attach it :) -- Janusz Dziemidowicz

Re: Connections stuck in CLOSE_WAIT state with h2

On Thu, Jun 14, 2018 at 01:51:20PM +0200, Janusz Dziemidowicz wrote: > 2018-06-14 11:46 GMT+02:00 Willy Tarreau : > >> Will try. > > I've tried the seconds path, together with the first one, no change at all. > > However, I was able to catch it on my laptop finally. I still can't > easily

Re: Connections stuck in CLOSE_WAIT state with h2

2018-06-14 11:46 GMT+02:00 Willy Tarreau : >> Will try. I've tried the seconds path, together with the first one, no change at all. However, I was able to catch it on my laptop finally. I still can't easily reproduce this, but at least that's something. Little background, my company makes online

Re: Connections stuck in CLOSE_WAIT state with h2

On Thu, Jun 14, 2018 at 11:29:39AM +0200, Janusz Dziemidowicz wrote: > 2018-06-14 11:14 GMT+02:00 Willy Tarreau : > > Yep it's really not easy and probably once we find it I'll be ashamed > > saying "I thought this code was not merged"... By the way yesterday I > > found another suspect but I'm

Re: Connections stuck in CLOSE_WAIT state with h2

2018-06-14 11:14 GMT+02:00 Willy Tarreau : > Yep it's really not easy and probably once we find it I'll be ashamed > saying "I thought this code was not merged"... By the way yesterday I > found another suspect but I'm undecided on it ; the current architecture > of the H2 mux complicates the code

Re: Connections stuck in CLOSE_WAIT state with h2

On Wed, Jun 13, 2018 at 08:01:30PM +0200, Janusz Dziemidowicz wrote: > 2018-06-13 19:14 GMT+02:00 Willy Tarreau : > > On Wed, Jun 13, 2018 at 07:06:58PM +0200, Janusz Dziemidowicz wrote: > >> 2018-06-13 14:42 GMT+02:00 Willy Tarreau : > >> > Hi Milan, hi Janusz, > >> > > >> > thanks to your

Re: Connections stuck in CLOSE_WAIT state with h2

2018-06-13 19:14 GMT+02:00 Willy Tarreau : > On Wed, Jun 13, 2018 at 07:06:58PM +0200, Janusz Dziemidowicz wrote: >> 2018-06-13 14:42 GMT+02:00 Willy Tarreau : >> > Hi Milan, hi Janusz, >> > >> > thanks to your respective traces, I may have come up with a possible >> > scenario explaining the

Re: Connections stuck in CLOSE_WAIT state with h2

On Wed, Jun 13, 2018 at 07:06:58PM +0200, Janusz Dziemidowicz wrote: > 2018-06-13 14:42 GMT+02:00 Willy Tarreau : > > Hi Milan, hi Janusz, > > > > thanks to your respective traces, I may have come up with a possible > > scenario explaining the CLOSE_WAIT you're facing. Could you please > > try the

Re: Connections stuck in CLOSE_WAIT state with h2

2018-06-13 14:42 GMT+02:00 Willy Tarreau : > Hi Milan, hi Janusz, > > thanks to your respective traces, I may have come up with a possible > scenario explaining the CLOSE_WAIT you're facing. Could you please > try the attached patch ? Unfortunately there is no change for me. CLOSE_WAIT sockets

Re: Connections stuck in CLOSE_WAIT state with h2

Hi Milan, hi Janusz, thanks to your respective traces, I may have come up with a possible scenario explaining the CLOSE_WAIT you're facing. Could you please try the attached patch ? Thanks, Willy >From fc527303f1391849ae5880b304486e328010b5ff Mon Sep 17 00:00:00 2001 From: Willy Tarreau Date:

Re: Connections stuck in CLOSE_WAIT state with h2

Hi Milan, On Fri, Jun 08, 2018 at 01:26:41PM +0200, Milan Petruzelka wrote: > On Wed, 6 Jun 2018 at 11:20, Willy Tarreau wrote: > > > Hi Milan, > > > > On Wed, Jun 06, 2018 at 11:09:19AM +0200, Milan Petruzelka wrote: > > > Hi Willy, > > > > > > I've tracked one of connections hanging in

Re: Connections stuck in CLOSE_WAIT state with h2

On Wed, 6 Jun 2018 at 11:20, Willy Tarreau wrote: > Hi Milan, > > On Wed, Jun 06, 2018 at 11:09:19AM +0200, Milan Petruzelka wrote: > > Hi Willy, > > > > I've tracked one of connections hanging in CLOSE_WAIT state with tcpdump > > over last night. It started at 17:19 like this: > > > >

Re: Connections stuck in CLOSE_WAIT state with h2

Hi Milan, On Wed, Jun 06, 2018 at 11:09:19AM +0200, Milan Petruzelka wrote: > Hi Willy, > > I've tracked one of connections hanging in CLOSE_WAIT state with tcpdump > over last night. It started at 17:19 like this: > > "Packet No.","Time in >

Re: Connections stuck in CLOSE_WAIT state with h2

Hi Willy, I've tracked one of connections hanging in CLOSE_WAIT state with tcpdump over last night. It started at 17:19 like this: "Packet No.","Time in seconds","Source","Destination","Protocol","Length","Info" "1","0.00","ip_client","ip_haproxy_server","TCP","62","64311 > 443

Re: Connections stuck in CLOSE_WAIT state with h2

Hi Willy On Thu, 31 May 2018 at 21:07, Willy Tarreau wrote: > Thank you. It's pretty constant, so it doesn't really seem load-dependant > (unless your load is reasonably stable of course). Here is a 3-day sample from the "grow" period - https://pasteboard.co/HnQu44t.png. Overall trend is

Re: Connections stuck in CLOSE_WAIT state with h2

Hi Milan, On Thu, May 31, 2018 at 10:54:11AM +0200, Milan Petruzelka wrote: > I've got the same problem described by Janusz Dziemidowicz. After enabling > h2 on Haproxy 1.8.8 it slowly accumulates frontend connections. Reload or > restart cuts them down, but they start to grow again. Upgrade to

Re: Connections stuck in CLOSE_WAIT state with h2

I've got the same problem described by Janusz Dziemidowicz. After enabling h2 on Haproxy 1.8.8 it slowly accumulates frontend connections. Reload or restart cuts them down, but they start to grow again. Upgrade to Haproxy 1.8.9 did not help. See a 40-day graph here -

Re: Connections stuck in CLOSE_WAIT state with h2

On Fri, May 25, 2018 at 03:43:16PM +0200, Janusz Dziemidowicz wrote: > 2018-05-24 23:26 GMT+02:00 Willy Tarreau : > >> Anyway, I'll do another round of experiments (without tfo) tomorrow. > > > > Much appreciated, thank you. > > I've removed tfo, curves and tls-ticket-keys from bind

Re: Connections stuck in CLOSE_WAIT state with h2

On Thu, May 24, 2018 at 11:20:13PM +0200, Janusz Dziemidowicz wrote: > 2018-05-24 22:26 GMT+02:00 Willy Tarreau : > >> This kinda seems like the socket was closed on the writing side, but > >> the client has already sent something and everything is stuck. I was > >> not able to

Re: Connections stuck in CLOSE_WAIT state with h2

2018-05-24 22:26 GMT+02:00 Willy Tarreau : >> This kinda seems like the socket was closed on the writing side, but >> the client has already sent something and everything is stuck. I was >> not able to reproduce the problem by myself. Any ideas how to debug >> this further? > > For

Re: Connections stuck in CLOSE_WAIT state with h2

Hi Janusz, On Thu, May 24, 2018 at 01:49:52PM +0200, Janusz Dziemidowicz wrote: > Recently I've moved several servers from haproxy 1.7.x to 1.8.x I have > a setup with nghttpx handling h2 (haproxy connects to nghttpx via unix > socket which handles h2 and connects back to haproxy with plain >

Connections stuck in CLOSE_WAIT state with h2

Recently I've moved several servers from haproxy 1.7.x to 1.8.x I have a setup with nghttpx handling h2 (haproxy connects to nghttpx via unix socket which handles h2 and connects back to haproxy with plain http/1.1 also through unix socket). After the upgrade I wanted to switch to native h2