Re: Keep-alive and websocket connections
Thanks for your suggestion, Lukas. For my own understanding, are you saying that there is no difference between having "http-keep-alive" and having "http-server-close" to a backend server once websocket connection to that server is establish, and both settings allow for establishing websocket connection perfectly. So is there any advantage of having "http-keep-alive" to a websocket backend? On Thu, Feb 6, 2014 at 4:56 PM, Lukas Tribus wrote: > Hi, > > >> In the latest HAProxy 1.5 release (dev22), it is indicated that >> keep-alive is now enabled by default for both client and server sides. >> I have some questions regarding its use in the following scenario. >> >> I use HAProxy in front of an array of servers: one nginx for >> delivering static files, and the others being application servers. One >> of the application servers exclusively deals with websocket (or in the >> event of ws failure, switching to streaming) connections. Currently, I >> am using 'http-server-close' by default for all servers, but I think >> it'd be better to switch to 'http-keep-alive' for the nginx and keep >> 'http-server-close' for the websockets server. > > You can just default to http-keep-alive everywhere. > > HAProxy recognizes the upgrade headers and switches to TCP mode > automatically [1]. > > Recognizing the upgrade in a HTTP transaction is possible with all modes > expect tcp mode (of course) and (the pre-dev22 default) http tunnel mode [2]. > > > >> Is this a correct setup? Thanks. > > It is, but you may as well simplify it with http-keep-alive on all sections. > > I don't see any advantage by configuring http-server-close on the websocket > backend. > > Of course you should test this, before putting it in production. > > > > Regards, > > Lukas > > > > [1] > http://blog.exceliance.fr/2012/11/07/websockets-load-balancing-with-haproxy/ > [2] http://cbonte.github.io/haproxy-dconv/configuration-1.5.html#4
Keep-alive and websocket connections
Dear all, In the latest HAProxy 1.5 release (dev22), it is indicated that keep-alive is now enabled by default for both client and server sides. I have some questions regarding its use in the following scenario. I use HAProxy in front of an array of servers: one nginx for delivering static files, and the others being application servers. One of the application servers exclusively deals with websocket (or in the event of ws failure, switching to streaming) connections. Currently, I am using 'http-server-close' by default for all servers, but I think it'd be better to switch to 'http-keep-alive' for the nginx and keep 'http-server-close' for the websockets server. Is this a correct setup? Thanks. Best, Chris
Re: HAProxy maintains connection to proxied server after client disconnects
Hi Lukas, Thank you for your suggestion. It works! I guess this is a shortcoming of streaming and eventsource protocols, because there are two separate connections involved for each client, whereas for websocket and long-polling haproxy can tell for sure the connection breaks given there is only one connection. I guess *option abortonclose* is also good to have in general, since it helps defend against DDoS attacks? And I do not see much side-effect for turning it on. Correct me if I'm wrong here. Thanks again! Regards, Chris On Sun, Jun 9, 2013 at 11:51 AM, Lukas Tribus wrote: > Hi Chris, > > you probably need to set the following option to achieve this behavior: > option abortonclose > > > > http://cbonte.github.io/haproxy-dconv/configuration-1.5.html#4-option%20abortonclose > > > > Regards, > > Lukas
HAProxy maintains connection to proxied server after client disconnects
Hi, I'm using HAProxy (1.5 dev18) in front of tornado servers to load-balancing websockets and xhr-streaming connections. I notice something strange but consistent with xhr-streaming connection (details here: https://groups.google.com/forum/?fromgroups#!topic/sockjs/LwfgT2qwIT0). In short, even after the client closes its connection, HAProxy still maintains the xhr-streaming connection to the proxied backend, until the timeout kicks in (i.e. timeout server xxx). I've tested this with both tornado 2.4.1 and the latest 3.0.2, and the results are consistent. Other transports (e.g. websockets, polling) do not have this problem, but evensource transport shares the same issue. Any idea what causes this? Thanks! Regards, Chris
Re: XHR-streaming and Websocket connections in 1.5-dev17
Hi Baptiste,
Thanks for your suggestion. I have read that article (a few times
actually), and except for a few timeout settings I lack comparing to the
config in the article, everything else (that matters for websocket) seems
to be the same with my configuration. The following are the timeout setting
I don't have:
timeout http-keep-alive 1s
> timeout http-request15s
> timeout queue 30s
> timeout tarpit 60s
Do you think these matter for the websocket and xhr-streaming connections?
Many thanks.
Best,
Chris
On Thu, Feb 28, 2013 at 2:40 AM, Baptiste wrote:
> Hi Chris,
>
> Please read the article below:
>
> http://blog.exceliance.fr/2012/11/07/websockets-load-balancing-with-haproxy/
>
> You may find answers ;)
>
> cheers
>
> On Wed, Feb 27, 2013 at 4:35 PM, Chris Yang wrote:
> >
> > Hello again,
> >
> > Regarding the second problem (failure to establishing websocket
> connections
> > in supported browsers), I can confirm that it happens on both Chrome and
> > Firefox. And below is what I see for the '101 Switching Protocols'
> request
> > in Firefox when it happened:
> >
> >> Request URL: https://www.myexample.com/sockjs/673/nw56jyoj/websocket
> >> Request Method: GET
> >> Status Code: HTTP/1.1 101 Switching Protocols
> >> Request Headers
> >> User-Agent:Mozilla/5.0 (Windows NT 6.1; rv:18.0) Gecko/20100101
> >> Firefox/18.0
> >> Upgrade:websocket
> >> Sec-WebSocket-Version:13
> >> Sec-WebSocket-Key:Fg8Z/EASgn6AlVh0hMYLtw==
> >> Pragma:no-cache
> >> Origin:https://www.playmymodel.com
> >> Host:www.playmymodel.com
> >> Connection:keep-alive, Upgrade
> >> Cache-Control:no-cache
> >> Accept-Language:en-US,en;q=0.5
> >> Accept-Encoding:gzip, deflate
> >> Accept:text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8
> >> Response Headers
> >> Upgrade:websocket
> >> Sec-WebSocket-Accept:NxU0HFQKbhwr7VBCHf23qLXH/Ic=
> >> Connection:Upgrade
> >
> >
> > and below is what follows from the SockJS side:
> >
> >> Closed transport: websocket SimpleEvent(type=close, code=2007,
> >> reason=Transport timeouted, wasClean=false)
> >
> >
> > This failure of establishing websocket connection happen relatively often
> > now, with 4 - 5 Chrome processes running on my laptop, I can almost
> always
> > get at least 1 Chrome which encounters this. In addition, I have also
> tested
> > this with HAProxy 1.5-dev15, and the problem persists. For completeness,
> I
> > attach the updated config as below:
> >
> >> global
> >> log 127.0.0.1 local1 info
> >> log 127.0.0.1 local1 notice
> >> #log loghostlocal0 info
> >> maxconn 4096
> >> chroot /usr/share/haproxy
> >> uid 99
> >> gid 99
> >> daemon
> >> #debug
> >> #quiet
> >>
> >> defaults
> >> log global
> >> modehttp
> >> option httplog
> >> option dontlognull
> >> retries 3
> >> option redispatch
> >> maxconn 500
> >> timeout connect 6s
> >> timeout client 600s
> >> timeout server 600s
> >> timeout tunnel 1200s
> >>
> >> frontend public
> >> modehttp
> >> bind0.0.0.0:80
> >> bind0.0.0.0:443 ssl crt /etc/haproxy/ssl_wc/mysite.pem
> ciphers
> >> HIGH:!aNULL:!MD5 no-tlsv10
> >> timeout client 300s
> >> option http-server-close
> >> option forwardfor except 127.0.0.1
> >> reqadd X-Forwarded-Proto:\ https if { ssl_fc }
> >> reqadd X-Forwarded-Proto:\ http if !{ ssl_fc }
> >> # define ACLs below
> >> acl is_https ssl_fc
> >> redirect scheme https if !is_https
> >> # and other ACLs and rules (omitted here)
> >>
> >> backend sockjs_1
> >> #timeout server 60s
> >> server sock1 127.0.0.1:8000
> >>
> >> backend sockjs_2
> >> #timeout server 60s
> >> server sock2 127.0.0.1:8060
> >>
> >> backend nginx
> >> #timeout server 20s
> >> server nginx 127.0.0.1:8484
> >>
> >> backend stats
> >
Re: XHR-streaming and Websocket connections in 1.5-dev17
Hi Willy, Thanks for your comments. It's unusual to see the client write both keep-alive and Upgrade in the > connection header. Thus, I suspect that something between the client and > the server (eg: a proxy on the client side or the application server > itself) > has added it. If it's a proxy on the client side, this proxy might be > responsible for this. Not all proxies support WebSocket yet. You should > try to take a capture of the front traffic (between client and haproxy), > and isolate a session which exhibits this behaviour (based on the IP > address > logged as reporting the problem). In such a capture, we'll see if the > server > response is valid, and if it is the client which closes. If we see the > client > close, we'll know that there's something between the client and haproxy > which > doesn't support WebSocket. The extra keep-alive in connection header seems to be a Firefox thing. For example, SockJS mentions about it here: http://sockjs.github.com/sockjs-protocol/sockjs-protocol-0.3.3.html#section-73. The strange thing is that when I was testing using Firefox, sometimes it connects websocket fine, but sometimes not. I will try to capture the traffic as you've suggested, and compare the captures when it works and when it does not. I am using Hmmm if you can easily reproduce it, this becomes more of a concern. Are > you sure you're directly connected ? Would you happen to be running windows > on your laptop with some anti-virus/anti-malware or such software ? We've > seen that in the past, it was Trend Micro's OfficeScan. It would let > traffic > to port 8000 pass without issue, but 80 was improperly analysed and the > connection was frozen. It would be possible that a similar type of software > is causing the issue. > There's nothing abnormal in your configuration. I'm seeing a redirect for > http->https, have you tried without SSL ? Indeed, I am using SSL exclusively, because encrypted traffic actually increases the chance of websocket (and other supported protocols in SockJS) working, since those annoying proxies and antivirus may just let the traffic pass through without modifying them. In fact, given the same network environment I was in for testing, if I don't do https, things barely work, websocket and xhr-streaming just hang. I also read this article: http://www.infoq.com/articles/Web-Sockets-Proxy-Servers, which seems to suggest that, due to the fact many proxies do not support websocket, such connection failure is not uncommon. The article also suggests to use HTTPS to increase the success rate, which is exactly what I've done. Still, even though I haven't count precisely, the overall success rate is definitely below 90% for me. What else can I try here? Thanks! Best, Chris On Fri, Mar 1, 2013 at 1:39 AM, Willy Tarreau wrote: > Hi Chris, > > I'm quickly jumping into this thread because I'm seeing something strange : > > On Wed, Feb 27, 2013 at 10:35:16AM -0500, Chris Yang wrote: > > Hello again, > > > > Regarding the second problem (failure to establishing websocket > connections > > in supported browsers), I can confirm that it happens on both Chrome and > > Firefox. And below is what I see for the '101 Switching Protocols' > request > > in Firefox when it happened: > > > > Request URL: https://www.myexample.com/sockjs/673/nw56jyoj/websocket > > > <https://www.playmymodel.com/sockjs/chat/673/nw56jyoj/websocket> > > > Request Method: GET > > > Status Code: HTTP/1.1 101 Switching Protocols > > > > *Request Headers > > > *User-Agent:Mozilla/5.0 (Windows NT 6.1; rv:18.0) Gecko/20100101 > > > Firefox/18.0 > > > Upgrade:websocket > > > Sec-WebSocket-Version:13 > > > Sec-WebSocket-Key:Fg8Z/EASgn6AlVh0hMYLtw== > > > Pragma:no-cache > > > Origin:https://www.playmymodel.com > > > Host:www.playmymodel.com > > > Connection:keep-alive, Upgrade > > > Cache-Control:no-cache > > > Accept-Language:en-US,en;q=0.5 > > > Accept-Encoding:gzip, deflate > > > Accept:text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8 > > It's unusual to see the client write both keep-alive and Upgrade in the > connection header. Thus, I suspect that something between the client and > the server (eg: a proxy on the client side or the application server > itself) > has added it. If it's a proxy on the client side, this proxy might be > responsible for this. Not all proxies support WebSocket yet. You should > try to take a capture of the front traffic (between client and haproxy), > and isolate a session which exhibits this behaviour (based on the IP > add
Re: XHR-streaming and Websocket connections in 1.5-dev17
Hello again,
Regarding the second problem (failure to establishing websocket connections
in supported browsers), I can confirm that it happens on both Chrome and
Firefox. And below is what I see for the '101 Switching Protocols' request
in Firefox when it happened:
Request URL: https://www.myexample.com/sockjs/673/nw56jyoj/websocket
> <https://www.playmymodel.com/sockjs/chat/673/nw56jyoj/websocket>Request
> Method: GET
> Status Code: HTTP/1.1 101 Switching Protocols
> *Request Headers
> *User-Agent:Mozilla/5.0 (Windows NT 6.1; rv:18.0) Gecko/20100101
> Firefox/18.0
> Upgrade:websocket
> Sec-WebSocket-Version:13
> Sec-WebSocket-Key:Fg8Z/EASgn6AlVh0hMYLtw==
> Pragma:no-cache
> Origin:https://www.playmymodel.com
> Host:www.playmymodel.com
> Connection:keep-alive, Upgrade
> Cache-Control:no-cache
> Accept-Language:en-US,en;q=0.5
> Accept-Encoding:gzip, deflate
> Accept:text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8
> *Response Headers
> *Upgrade:websocket
> Sec-WebSocket-Accept:NxU0HFQKbhwr7VBCHf23qLXH/Ic=
> Connection:Upgrade
and below is what follows from the SockJS side:
Closed transport: websocket SimpleEvent(type=close, code=2007,
> reason=Transport timeouted, wasClean=false)
This failure of establishing websocket connection happen relatively often
now, with 4 - 5 Chrome processes running on my laptop, I can almost always
get at least 1 Chrome which encounters this. In addition, I have also
tested this with HAProxy 1.5-dev15, and the problem persists. For
completeness, I attach the updated config as below:
global
> log 127.0.0.1 local1 info
> log 127.0.0.1 local1 notice
> #log loghostlocal0 info
> maxconn 4096
> chroot /usr/share/haproxy
> uid 99
> gid 99
> daemon
> #debug
> #quiet
>
> defaults
> log global
> modehttp
> option httplog
> option dontlognull
> retries 3
> option redispatch
> maxconn 500
> timeout connect 6s
> timeout client 600s
> timeout server 600s
> timeout tunnel 1200s
>
> frontend public
> modehttp
> bind0.0.0.0:80
> bind0.0.0.0:443 ssl crt /etc/haproxy/ssl_wc/mysite.pem
> ciphers HIGH:!aNULL:!MD5 no-tlsv10
> timeout client 300s
> option http-server-close
> option forwardfor except 127.0.0.1
> reqadd X-Forwarded-Proto:\ https if { ssl_fc }
> reqadd X-Forwarded-Proto:\ http if !{ ssl_fc }
> # define ACLs below
> acl is_https ssl_fc
> redirect scheme https if !is_https
> # and other ACLs and rules (omitted here)
>
> backend sockjs_1
> #timeout server 60s
> server sock1 127.0.0.1:8000
>
> backend sockjs_2
> #timeout server 60s
> server sock2 127.0.0.1:8060
>
> backend nginx
> #timeout server 20s
> server nginx 127.0.0.1:8484
>
> backend stats
> stats enable
> stats uri /haproxy/stats
> stats realm Haproxy\ Statistics
>
> backend www
> #timeout server 300s
> #option forwardfor header X-Client-IP
> #no option httpclose
> #option http-server-close
> server content 127.0.0.1:8080
Again, backends sockjs_1 and sockjs_2 are what is being affected here.
Could anyone comment on what may cause this undesirable websocket failures
here? Thanks a lot!
Best,
Chris
On Fri, Feb 22, 2013 at 8:40 AM, Chris Yang wrote:
>
> Hello,
>
> I upgraded to dev17 from dev15. I am running Tornado servers behind
> HAProxy with SockJS support. Comparing to before the upgrade, I've noticed
> two problematic behaviors:
>
> 1. When a client is using IE with xhr-streaming protocol to connect to
> servers, such connections are closed (seeing 'Connection get reset by peer'
> error on the Python side) more often than before. Sometimes disconnection
> happened soon after connection (half a minute). I guess it might be related
> to timeout, but since I haven't changed any configuration on HAProxy, I
> wonder if this behavior is related to the upgrade of HAProxy itself.
>
> 2. From time to time, Chrome failed to finish the websocket handshake with
> the server, and instead used xhr-streaming. This didn't happen before the
> upgrade, but I've noticed at least 5+ such cases during the testing with
> dev17.
>
> I can confirm that both client-side library (sockjs-client) and
> server-side library (sockjs-tornado) have stayed the same since their last
> respective upgrade (mont
XHR-streaming and Websocket connections in 1.5-dev17
Hello,
I upgraded to dev17 from dev15. I am running Tornado servers behind HAProxy
with SockJS support. Comparing to before the upgrade, I've noticed two
problematic behaviors:
1. When a client is using IE with xhr-streaming protocol to connect to
servers, such connections are closed (seeing 'Connection get reset by peer'
error on the Python side) more often than before. Sometimes disconnection
happened soon after connection (half a minute). I guess it might be related
to timeout, but since I haven't changed any configuration on HAProxy, I
wonder if this behavior is related to the upgrade of HAProxy itself.
2. From time to time, Chrome failed to finish the websocket handshake with
the server, and instead used xhr-streaming. This didn't happen before the
upgrade, but I've noticed at least 5+ such cases during the testing with
dev17.
I can confirm that both client-side library (sockjs-client) and server-side
library (sockjs-tornado) have stayed the same since their last respective
upgrade (months before dev17 came out). So the only variable which might
have produced the observations above could be HAProxy.
I attach below my HAProxy config (excluding most of the acls), and I'd much
appreciate that if someone could take a quick look and let me know if some
setting(s) may not play well with dev17 which might have caused the
behavior above, or some new setting(s) should be added. Notice that
backends sockjs_1 and sockjs_2 are the ones experiencing the problems.
global
log 127.0.0.1 local1 info
log 127.0.0.1 local1 notice
#log loghostlocal0 info
maxconn 4096
chroot /usr/share/haproxy
uid 99
gid 99
daemon
#debug
#quiet
defaults
log global
modehttp
option httplog
option dontlognull
retries 3
option redispatch
maxconn 500
timeout connect 6s
frontend public
modehttp
bind0.0.0.0:80
bind0.0.0.0:443 ssl crt /etc/haproxy/ssl_wc/mysite.pem ciphers
HIGH:!aNULL:!MD5 no-tlsv10
timeout client 300s
option http-server-close
option forwardfor except 127.0.0.1
reqadd X-Forwarded-Proto:\ https if { ssl_fc }
reqadd X-Forwarded-Proto:\ http if !{ ssl_fc }
#option http-pretend-keepalive
# define ACLs and use them below
acl is_https ssl_fc
redirect scheme https if !is_https
# other acls omitted here
backend sockjs_1
timeout server 60s
server sock_1 127.0.0.1:8000
backend sockjs_2
timeout server 60s
server sock_2 127.0.0.1:8060
backend nginx
timeout server 20s
server nginx 127.0.0.1:8484
backend stats
stats enable
stats uri /haproxy/stats
stats realm Haproxy\ Statistics
backend www
timeout server 300s
#option forwardfor header X-Client-IP
#no option httpclose
#option http-server-close
server content 127.0.0.1:8080
Thanks very much!
Best,
Chris
