Re: unexpected server disconnect (SH--) when using HAProxy 1.5x
Hi Cory, On Fri, Mar 25, 2011 at 12:30:15PM -0400, Cory Forsyth wrote: I don't think the number of seconds is significant. I tried it again now, reloading several times and each time the number of milliseconds in that SH-- line is different. I think that just indicates how long the long-polling connection was open when the reload severed it. OK. Here's the config: (...) Thank you for this config. I see nothing fancy here. I tried various test patterns with both 1.4.13 and 1.5-dev4 to try to reproduce errors 502, but I could not find a way to get a different behaviour between both of them :-/ There are two possible reasons to get a 502 : - server closes - server returns an invalid response The SH log tends to indicate the former, as an invalid response would have had flags PH instead (and I have rechecked that they're correctly produced). Still there is something I'm suspecting (which should not differ between both versions). You don't have any option which affects the Connection header. Thus, once the response is received, the connection becomes a tunnel and any data may flow between the client and the server. If the client sends more data than expected to the server and the server closes without reading those data, it may emit a reset. On Linux, it will emit a reset even before the emitted data has been sent or acknowledged, causing the response to be destroyed and to never reach haproxy. By specifying option http-server-close in your defaults section, you'll ensure that no excess data will be sent to the server, so it will only get what it expects to receive. If doing so changes something, then you should have a look at the data sent by the client, it is possible that sometimes it is too large. Have you tried to correlate haproxy's logs with the server's logs, to see if the server accepted the request or spotted anything wrong in it ? Regards, Willy
Re: unexpected server disconnect (SH--) when using HAProxy 1.5x
I don't think the number of seconds is significant. I tried it again now,
reloading several times and each time the number of milliseconds in that
SH-- line is different. I think that just indicates how long the
long-polling connection was open when the reload severed it.
Here's the config:
global
maxconn 3
stats socket /var/run/haproxy.stat mode 600 level admin
defaults
mode http
timeout connect 5000ms
timeout client 6ms
timeout server 6ms
backend stats_be
stats uri /stats
stats refresh 5s
backend the_backend
server the_backend1 x.x.x.x:80
frontend stats_fe
bind *:8080
default_backend stats_be
frontend http_proxy
log 127.0.0.1local0
log 127.0.0.1local1 err
bind *:80
mode http
option forwardfor
option httplog
option log-separate-errors
# log the name of the virtual server
capture request header Host len 20
# log the amount of data uploaded during a POST
capture request header Content-Length len 10
# log the beginning of the referrer
capture request header Referer len 20
capture request header User-Agent len 20
# server name (useful for outgoing proxies only)
capture response header Server len 20
# logging the content-length is useful with option logasap
capture response header Content-Length len 10
default_backend the_backend
Here are a few more examples of the SH-- lines:
Mar 25 12:22:35 localhost haproxy-1.5v4[28922]:
96.246.63.146:63288[25/Mar/2011:12:22:34.341] http_proxy
the_backend/the_backend1 6/0/0/-1/1166
502 204 - - SH-- 0/0/0/0/0 0/0 {the-proxy.etc|63|http://the-proxy|Mozilla/5.0
(Macinto} {|} POST / HTTP/1.1
Mar 25 12:22:40 localhost haproxy-1.5v4[28922]:
96.246.63.146:63296[25/Mar/2011:12:22:36.945] http_proxy
the_backend/the_backend1 6/0/0/-1/3199
502 204 - - SH-- 0/0/0/0/0 0/0 {the-proxy.etc|63|http://the-proxy|Mozilla/5.0
(Macinto} {|} POST / HTTP/1.1
Mar 25 12:22:52 localhost haproxy-1.5v4[28922]:
96.246.63.146:63302[25/Mar/2011:12:22:41.440] http_proxy
the_backend/the_backend1
6/0/0/-1/10988 502 204 - - SH-- 0/0/0/0/0 0/0 {the-proxy.etc|63|
http://the-proxy|Mozilla/5.0 (Macinto} {|} POST / HTTP/1.1
Mar 25 12:24:56 localhost haproxy-1.5v4[30407]:
96.246.63.146:63609[25/Mar/2011:12:24:49.936] http_proxy
the_backend/the_backend1 6/0/0/-1/6924
502 204 - - SH-- 0/0/0/0/0 0/0 {the-proxy.etc|63|http://the-proxy|Mozilla/5.0
(Macinto} {|} POST / HTTP/1.1
(I replaced some identifying info in those headers with the-proxy.etc)
thanks,
-Cory
On Thu, Mar 24, 2011 at 4:06 PM, Willy Tarreau w...@1wt.eu wrote:
Hi Cory,
On Thu, Mar 24, 2011 at 01:10:49PM -0400, Cory Forsyth wrote:
I'm not sure if this is a bug in HAProxy, or if upgrading HAProxy to 1.4
has
uncovered a bug in my application.
I am developing a web app that uses long polling. I'm proxying its
requests
through haproxy. I've noticed that when reload the page in the browser
(which presumably severs the connection that the javascript is holding
open
to the server through haproxy), I get an http status 502 error logged by
haproxy. It looks something like this:
Mar 23 12:43:45 localhost haproxy-1.5v4[5969]:
96.246.63.146:61899[23/Mar/2011:12:43:25.276] http_proxy
backend1/backendserver1 6/0/0/-1/20014
502 204 - - SH-- 0/0/0/0/0 0/0
{proxy.myserver|65|http://proxy.myserver|Mozilla/5.0
(Macinto} {|} POST / HTTP/1.1
This happens in Haproxy version 1.5-dev3 and 1.5-dev4, but it does not
happen in haproxy version 1.4.13. Is there some additional investigation
I
can do to determine if it's a bug in 1.5x or just something that my web
app
shouldn't have been doing anyway that is now (correctly) being logged as
an
error by 1.5x?
What I understand from the logs is that the server suddenly closed
after exactly 20 seconds. Maybe this is not what happened, but we
have to find why it was logged like this. Could you please post your
config ?
Thanks,
Willy
unexpected server disconnect (SH--) when using HAProxy 1.5x
I'm not sure if this is a bug in HAProxy, or if upgrading HAProxy to 1.4 has
uncovered a bug in my application.
I am developing a web app that uses long polling. I'm proxying its requests
through haproxy. I've noticed that when reload the page in the browser
(which presumably severs the connection that the javascript is holding open
to the server through haproxy), I get an http status 502 error logged by
haproxy. It looks something like this:
Mar 23 12:43:45 localhost haproxy-1.5v4[5969]:
96.246.63.146:61899[23/Mar/2011:12:43:25.276] http_proxy
backend1/backendserver1 6/0/0/-1/20014
502 204 - - SH-- 0/0/0/0/0 0/0
{proxy.myserver|65|http://proxy.myserver|Mozilla/5.0
(Macinto} {|} POST / HTTP/1.1
This happens in Haproxy version 1.5-dev3 and 1.5-dev4, but it does not
happen in haproxy version 1.4.13. Is there some additional investigation I
can do to determine if it's a bug in 1.5x or just something that my web app
shouldn't have been doing anyway that is now (correctly) being logged as an
error by 1.5x?
thanks,
Cory
Re: unexpected server disconnect (SH--) when using HAProxy 1.5x
Hi Cory,
On Thu, Mar 24, 2011 at 01:10:49PM -0400, Cory Forsyth wrote:
I'm not sure if this is a bug in HAProxy, or if upgrading HAProxy to 1.4 has
uncovered a bug in my application.
I am developing a web app that uses long polling. I'm proxying its requests
through haproxy. I've noticed that when reload the page in the browser
(which presumably severs the connection that the javascript is holding open
to the server through haproxy), I get an http status 502 error logged by
haproxy. It looks something like this:
Mar 23 12:43:45 localhost haproxy-1.5v4[5969]:
96.246.63.146:61899[23/Mar/2011:12:43:25.276] http_proxy
backend1/backendserver1 6/0/0/-1/20014
502 204 - - SH-- 0/0/0/0/0 0/0
{proxy.myserver|65|http://proxy.myserver|Mozilla/5.0
(Macinto} {|} POST / HTTP/1.1
This happens in Haproxy version 1.5-dev3 and 1.5-dev4, but it does not
happen in haproxy version 1.4.13. Is there some additional investigation I
can do to determine if it's a bug in 1.5x or just something that my web app
shouldn't have been doing anyway that is now (correctly) being logged as an
error by 1.5x?
What I understand from the logs is that the server suddenly closed
after exactly 20 seconds. Maybe this is not what happened, but we
have to find why it was logged like this. Could you please post your
config ?
Thanks,
Willy
