RE: HAproxy and instant messagers (Ym)
Anyone willing to help? Also, if I can get some help interpreting the following lines extracted from haproxy.log: balance roundrobin/ proxy 0/0/1/26004/6 200 2054 - - CD-- 470/470/456/456/0 0/0 POST http://98.139.60.242/ HTTP/1.1 balance roundrobin/ proxy 0/0/1/-1/50003 504 194 - - sH-- 490/490/489/489/0 0/0 POST http://98.139.60.242/ HTTP/1.1 balance roundrobin/ proxy 0/0/0/2/57935 407 47897 - - cD-- 531/531/528/528/0 0/0 GET http://i1.ytimg.com/i/ HTTP/1.1 specifically, what do those flags mean? e.g. CD, sH, cD Thank you. Sent: Friday 11, February 02, 2011 13:09 To: 'haproxy@formilux.org' Subject: HAproxy and instant messagers (Ym) Hello, I'm using Haproxy to balance HTTP requests to Squid servers for web access. Http requests seem to work fine but when I connect to Yahoo messenger, I am randomly kicked out and logged in again. If I connect directly through Squid, I do not encounter these disconnects, so I am assuming it has something to do with my Haproxy config. Maybe it timeouts the connection.. I can't figure it out. Here is my config: defaults log global modehttp option httplog option dontlognull option redispatch option abortonclose retries 3 maxconn 9000 contimeout 5000 clitimeout 5 srvtimeout 5 frontend front bind 0.0.0.0: mode http option forwardfor default_backend back backend roundrobin mode http option httpclose balance roundrobin server proxy 192.168.100.5:3128 check inter 2000 fall 3 server proxy_bkp 192.168.100.6:3128 check inter 2000 fall 3 backup I also tried using server proxy 192.168.100.5:3128 cookie 2 check. Any suggestions (adding / removing options from the config)? Scope: being used by several hundred people for http browsing, voip and IM. The information contained herein is intended for its addressee(s) only and it is privileged or otherwise confidential. Any unauthorized distribution, amendment or disclosure hereof is strictly forbidden by the law. Please find complete and translated versions at http://www.rompetrol.com/disclaimer.html
Re: HAproxy and instant messagers (Ym)
Hi, On Mon, Feb 14, 2011 at 10:35:15AM +0200, Florescu, Dan Alexandru wrote: Anyone willing to help? It's possible that people are not much skilled in this application :-/ Also, if I can get some help interpreting the following lines extracted from haproxy.log: It's a good point you got the logs. Here are the explanations : balance roundrobin/ proxy 0/0/1/26004/6 200 2054 - - CD-- 470/470/456/456/0 0/0 POST http://98.139.60.242/ HTTP/1.1 balance roundrobin/ proxy 0/0/1/-1/50003 504 194 - - sH-- 490/490/489/489/0 0/0 POST http://98.139.60.242/ HTTP/1.1 balance roundrobin/ proxy 0/0/0/2/57935 407 47897 - - cD-- 531/531/528/528/0 0/0 GET http://i1.ytimg.com/i/ HTTP/1.1 specifically, what do those flags mean? e.g. CD, sH, cD CD means the Client closed the connection during the Data phase. I don't know how Ym works but it's possible that both sides exchange small chunks, and the client has actively stopped doing so (eg: click Stop, close the browser window, ...). sH means the server has failed to respond to a request within the configured timeout. In your case, the server did not say anything within 50 seconds. Since you're connecting through Squid, I suspect that in fact squid failed to establish a connection to the server (server may be overloaded) and that haproxy timed out first. You can increase your haproxy timeouts if you want, so that you get the Squid's error in the response. cD means the client timed out during the data phase. Most likely there were data to read in the buffer but the client did not care to read them, and haproxy timed out after 58 seconds. Since haproxy is between a browser and a squid, it needs to be configured to look for the Proxy-Connection header instead of the Connection header. I think it is what is causing these issues, because the client and squid can't agree on what scheme to use (close/keep-alive). You have to enable option http-use-proxy-header for that. Also, you'd better use option http-server-close than httpclose, as it will maintain the connection alive on the client side. Regards, Willy
RE: HAproxy and instant messagers (Ym)
Hi Willy, Thank you for the explanations. In fact I had assumed that CD/cD was something related to timing out and connection closing, and so I increased contimeout from 5000 (5ms) to 5 (5s) (BTW, I think that was a typo in my config). This has solved my problem. Although, looking at the timers from those three lines, nothing is below 5 seconds (6, 50003, 57935). Maybe I missed the specific lines in the log. Anyway, it works now. Cheers. -Original Message- From: Willy Tarreau Sent: Monday 14, February 02, 2011 12:41 Subject: Re: HAproxy and instant messagers (Ym) Hi, On Mon, Feb 14, 2011 at 10:35:15AM +0200, Florescu, Dan Alexandru wrote: Anyone willing to help? It's possible that people are not much skilled in this application :-/ Also, if I can get some help interpreting the following lines extracted from haproxy.log: It's a good point you got the logs. Here are the explanations : balance roundrobin/ proxy 0/0/1/26004/6 200 2054 - - CD-- 470/470/456/456/0 0/0 POST http://98.139.60.242/ HTTP/1.1 balance roundrobin/ proxy 0/0/1/-1/50003 504 194 - - sH-- 490/490/489/489/0 0/0 POST http://98.139.60.242/ HTTP/1.1 balance roundrobin/ proxy 0/0/0/2/57935 407 47897 - - cD-- 531/531/528/528/0 0/0 GET http://i1.ytimg.com/i/ HTTP/1.1 specifically, what do those flags mean? e.g. CD, sH, cD CD means the Client closed the connection during the Data phase. I don't know how Ym works but it's possible that both sides exchange small chunks, and the client has actively stopped doing so (eg: click Stop, close the browser window, ...). sH means the server has failed to respond to a request within the configured timeout. In your case, the server did not say anything within 50 seconds. Since you're connecting through Squid, I suspect that in fact squid failed to establish a connection to the server (server may be overloaded) and that haproxy timed out first. You can increase your haproxy timeouts if you want, so that you get the Squid's error in the response. cD means the client timed out during the data phase. Most likely there were data to read in the buffer but the client did not care to read them, and haproxy timed out after 58 seconds. Since haproxy is between a browser and a squid, it needs to be configured to look for the Proxy-Connection header instead of the Connection header. I think it is what is causing these issues, because the client and squid can't agree on what scheme to use (close/keep-alive). You have to enable option http-use-proxy-header for that. Also, you'd better use option http-server-close than httpclose, as it will maintain the connection alive on the client side. Regards, Willy The information contained herein is intended for its addressee(s) only and it is privileged or otherwise confidential. Any unauthorized distribution, amendment or disclosure hereof is strictly forbidden by the law. Please find complete and translated versions at http://www.rompetrol.com/disclaimer.html
Re: HAproxy and instant messagers (Ym)
On Mon, Feb 14, 2011 at 05:54:08PM +0200, Florescu, Dan Alexandru wrote: Hi Willy, Thank you for the explanations. In fact I had assumed that CD/cD was something related to timing out and connection closing, and so I increased contimeout from 5000 (5ms) to 5 (5s) (BTW, I think that was a typo in my config). This has solved my problem. Just for the record, 5000 is 5000ms = 5s. 5 is 50s. In theory you should not have to increase the contimeout. The only reason you might want to do so is if you're queuing when connecting to the server, in which case you'd better increase the queue timeout. But the server should always immediately accept new incoming connections. Although, looking at the timers from those three lines, nothing is below 5 seconds (6, 50003, 57935). Maybe I missed the specific lines in the log. Anyway, it works now. Keep an eye on that anyway, timeouts are something that can appear and disappear due to various environmental issues. Regards, Willy
Haproxy and Plesk
Hi all, I am looking for way to configure same VIP to support multiples ports on haproxy for one physical box. Same physical box has multiple applications (80,443 [SSL],8443 [SSL]). I would like use one VIP to support all the ports. Is there way to do on haproxy? Please kindly advice. Thanks Regards Dennis Devota Savvis Inc 201-472-2821 203-687-8619 This message contains information which may be confidential and/or privileged. Unless you are the intended recipient (or authorized to receive for the intended recipient), you may not read, use, copy or disclose to anyone the message or any information contained in the message. If you have received the message in error, please advise the sender by reply e-mail and delete the message and any attachment(s) thereto without retaining any copies.
Re: balance (hdr) problem (maybe bug?)
Hi, Thanks to all your tests and observations, I managed to spot the bug and to fix it. The headers are linked in a list whose tail is known as hdr_idx-tail. This pointer is used when adding new headers. Unfortunately the header removal function did not update it when it removed the last header. The effect is that when adding the x-forwarded-for header, it's added just after the previously removed Connection header so it does not appear in the list when walked from the beginning. It's really when you said that it broke when Connection was the last header that I understood what to look for. Thanks ! I've confirmed it's working fine now, and successfully deployed the new version yesterday in our live environment. Many thanks for fixing it! :) Sidenote to list-readers: Do not use stunnel with the out-of-tree x-forwarded-for patch for SSL termination if you want to balance based on that header with two haproxy frontends and one backend. Stunnel will add an additional X-Forwarded-For header even if your clients send one. Expected behaviour: Case a) should not jump between servers. An empty x-forwarded-for header means that always the same header (an empty one) should be hashed, you should always end up on the same server. This is not true in haproxy, and there is an explicit test for this. If you try to hash on an empty or non-existing header or parameter, then it falls back to round robin. This is very important because there are many conditions where you want to optimize stickiness when the information is present, but not send all visitors to the same random server if they don't have the info. Oh right, it really has to be like that! That was just a (too) quick thought and I was misleaded... Well thanks again for this great piece of software! :) Best regards, Craig
Re: Haproxy and Plesk
For sure you can use a listen like this; frontend DISPATCH-lb10 bind :80,:81,:8001,:8002,:8881,:8882 This will tell haproxy to listen on all IP for those specific ports If you check the doc you should have some complete usage examples Good luck. Devota a écrit : Hi all, I am looking for way to configure same VIP to support multiples ports on haproxy for one physical box. Same physical box has multiple applications (80,443 [SSL],8443 [SSL]). I would like use one VIP to support all the ports. Is there way to do on haproxy? Please kindly advice. Thanks Regards **Dennis Devota** **Savvis Inc** **201-472-2821** **203-687-8619** This message contains information which may be confidential and/or privileged. Unless you are the intended recipient (or authorized to receive for the intended recipient), you may not read, use, copy or disclose to anyone the message or any information contained in the message. If you have received the message in error, please advise the sender by reply e-mail and delete the message and any attachment(s) thereto without retaining any copies. -- Guillaume Bourque, B.Sc., consultant, infrastructures technologiques libres ! Logisoft Technologies inc. http://www.logisoftech.com 514 576-7638, http://ca.linkedin.com/in/GuillaumeBourque/fr
does http-server-close close idle client sockets when needed?
I can't find in the documentation anything about how haproxy handles client keep-alive (using http-server-close) when the maximum number of client connections has been reached. If there are idle client connections, will the proxy close them to allow new connections to be established? Or, will new connections be refused while keeping idle connections around? It seems like it would be best to begin closing idle connections when the connection limit is approaching thus degrading to non-keep-alive behavior under heavy load. What does haproxy 1.4 do? -Bryan
Re: balance (hdr) problem (maybe bug?)
On Tue, Feb 15, 2011 at 12:26:54AM +0100, Craig wrote: Hi, Thanks to all your tests and observations, I managed to spot the bug and to fix it. The headers are linked in a list whose tail is known as hdr_idx-tail. This pointer is used when adding new headers. Unfortunately the header removal function did not update it when it removed the last header. The effect is that when adding the x-forwarded-for header, it's added just after the previously removed Connection header so it does not appear in the list when walked from the beginning. It's really when you said that it broke when Connection was the last header that I understood what to look for. Thanks ! I've confirmed it's working fine now, and successfully deployed the new version yesterday in our live environment. Many thanks for fixing it! :) And thanks for confirming :-) Sidenote to list-readers: Do not use stunnel with the out-of-tree x-forwarded-for patch for SSL termination if you want to balance based on that header with two haproxy frontends and one backend. Stunnel will add an additional X-Forwarded-For header even if your clients send one. Yes and that's expected, otherwise you'd get random addresses, sometimes the one added by stunnel, sometimes the one the client's sets. The real issue in fact is that balance hdr() was never meant to be used to hash a header that is allowed to appear multiple times. Right now it takes the first occurrence. Ideally we should improve it so that it can work as usesrc hdr_ip() and that we can pass it the occurrence number. That way we'd be able to ask for the last occurrence only. But some changes are still required to be able to do that with any header. I'd say that the real issue with the stunnel x-forwarded-for patch is that it is not compatible with keep-alive. It will only modify the first request in the connection, so that can become a problem when hashing as you do if keep-alive is enabled. Regards, Willy
Re: does http-server-close close idle client sockets when needed?
Hi Bryan, On Mon, Feb 14, 2011 at 08:08:25PM -0800, Bryan Talbot wrote: I can't find in the documentation anything about how haproxy handles client keep-alive (using http-server-close) when the maximum number of client connections has been reached. If there are idle client connections, will the proxy close them to allow new connections to be established? Or, will new connections be refused while keeping idle connections around? It seems like it would be best to begin closing idle connections when the connection limit is approaching thus degrading to non-keep-alive behavior under heavy load. What does haproxy 1.4 do? It only relies on the keep-alive timeout to close the connections. There is no point keeping that timeout high, normally it should only cover the time separating two objects fetched on the same page. Add a small delay to that in order to cover some TCP retransmits, and I'd say you're fine with something around 3-4 seconds. At those numbers there's no point trying to close some idle connections to try to gain a few more, because the same delays could be caused by your application servers slowing down on the very same connections, so the proxy really has to be able to deal with that many connections anyway. Also, actively closing them early can cause nasty side effects because they can be closed too fast when the client expects to reuse them, causing resets in response to requests, and connection reopenings. It can also change the client's mind when it wanted to pipeline requests. In fact, I'd rather proceed differently. We could say that we prefer to switch to close in the negociation when the frontend has more than XX% of connections in use, and similarly for the global number of connections. That would help the system self-regulate without causing issues to clients since we'd announce to them the connection they're using will be closed. We just need this to be user-configurable so that users with correct settings are not impacted by this unexpected behaviour. Regards, Willy
