Hi Nick, it's a bit late now and I'm not sure I will be able to test this tomorrow... I have quite a lot to do. Will try asap, if not sooner, by Monday. But I'll try to do it tomorrow.
> Thanks for your help debugging this. Thanks for your help in solving this. :D Thanks, Anze On Saturday 13 March 2010, Nick Chalk wrote: > Hello Anze. > > On 13 March 2010 20:08, Anze <anzen...@volja.net> wrote: > >> Interesting. Have you run haproxy -d from the command line to check > >> the debugging output? "option log-health-checks" is also useful. > > > > I was running it from the command line (too) because logging was not > > working (rsyslogd's UDP support was disabled). That was the output I have > > sent in the first mails. > > Ah, missed that email. > > >> It's a half-close. HAProxy only sends one request, and then waits for > >> the response, so we might as well close the transmit channel. > > > > Are you sure nothing is being sent anymore through this socket? Never? > > Not in my packet traces. HAProxy sends one request, receives the > response from the real server, real server closes its channel, HAProxy > replies with RST. Future tests bind a new socket. > > >> When tcpdump'ing the problem system that led me to work on this, I > >> found that HAProxy was not closing the transmit channel properly. The > >> real servers were sending FIN to close their side, and HAProxy was > >> responding with RST. So I put in the half-close to make sure the > >> connection was cleanly shut down. > > > > I think this is not the proper way to deal with this, but as I said, I am > > no expert in sockets programming. > > Neither am I - my background is in System/Network Admin. I'm also just > learning the HAProxy code, so there may be a better way to ensure that > the check connection is closed properly. > > > I would say that haproxy still tries to sent RST but can't - which leads > > to an error somewhere. > > The RST should not be sent - that's a response to an already-closed > socket. However, HAProxy is not signalling that the connection should > be closed by sending a FIN. > > >> It's interesting that this has caused problems in your system. As well > >> as running HAProxy in debug mode, would you be able to do a tcpdump on > >> the system running haproxy? If you could do that both with and without > >> the half-close, that would be very useful. > > > > Hmmm, I am a bit out of depths here. I would love to help out, but please > > send me more specific instructions on what you need. > > > > What arguments should I use with tcpdump so it helps you? Also, if you > > don't mind, I'd like to send it privately to you and Willy (so I don't > > need to clean up too much private / security sensitive info from it). > > I have no problem with you sending it to me directly. > > My usual tcpdump invocation is: > tcpdump -i any -s 0 -w <output_file> > Substitute an appropriate filename for <output_file>. You won't want > to leave this running for too long, as it listens on all interfaces > and captures the whole of every packet. I'd suggest starting the > tcpdump, then start HAProxy, wait for a few health checks to be sent, > then stop the tcpdump with Ctrl-C. > > >> If required, I can send you a version of checks.c with debugging code > >> in it - that will dump a lot of data on the checks if you run it in > >> debug. > > > > Sure, bring it on. ;) > > I'll send that directly to you, rather than the list... > > >> Those two will not check the content - I'll need to trawl through the > >> config code to check whether it defaults to checking the HTTP return > >> code in these cases. > > > > I have tested GET with the fixed version (no half-closed socket) and it > > works. > > Interesting. I'll test that locally. > > Thanks for your help debugging this. > > Nick. >
