Re: abortonclose for established connections?
Hi Ludovico, On Mon, Apr 27, 2015 at 10:54:43AM -0700, Ludovico Cavedon wrote: > Hi Willy, > > On Wed, Apr 22, 2015 at 10:58 AM, Ludovico Cavedon > wrote: > > > I will let you know, thanks again! > > > > > I reproduced your test case and it turned out the abortonclose option was > only in the listener section (and not on backend/defautls). > I moved it to "defaults" and now haproxy is behaving as expected. Great! Thanks for your feedback! Cheers, Willy
Re: abortonclose for established connections?
Hi Willy, On Wed, Apr 22, 2015 at 10:58 AM, Ludovico Cavedon wrote: > I will let you know, thanks again! > > I reproduced your test case and it turned out the abortonclose option was only in the listener section (and not on backend/defautls). I moved it to "defaults" and now haproxy is behaving as expected. Thanks for your help! Ludovico
Re: abortonclose for established connections?
Hi Willy, thank you for the very detailed information. On Wed, Apr 22, 2015 at 8:57 AM, Willy Tarreau wrote: > That's normal, this is httpterm and it doesn't monitor the connection while > it's waiting. But in your case it should definitely work. Or it means that > your server ignores the client abort. It would be nice if you could double- > check this. > This could very well be the case. I will check and let you know. If your server does indeed not support client-abort, then there's a > workaround. Haproxy 1.5 supports half-closed timeouts. You can simply > add a directive "timeout server-fin 1s" and it will automatically close > the server connection 1 second after sending it the close if the server > doesn't respond (and then report a server error) : > My server is nginx, I will see if I can configure it to "abortonclose" as well. Otherwise the server-fin option looks viable. I suspect it could match your needs if the normal solution doesn't work. > This *does* require abortonclose since you want the shutdown() to have been > done first. Please keep us updated. > I will let you know, thanks again! Ludovico
Re: abortonclose for established connections?
Hi again Ludovico,
so I ran some tests here with latest 1.5. I telnet to haproxy, send
"GET /?t=1 HTTP/1.1" then quit. It forwards to a server which
waits 10s before responding.
First, without "option abortonclose" :
17:47:40.174784 accept4(7, {sa_family=AF_INET, sin_port=htons(53833),
sin_addr=inet_addr("127.0.0.1")}, [16], SOCK_NONBLOCK) = 12
17:47:40.174879 setsockopt(12, SOL_TCP, TCP_NODELAY, [1], 4) = 0
17:47:40.174927 accept4(7, 0x7fff89a6c8d0, [128], SOCK_NONBLOCK) = -1 EAGAIN
(Resource temporarily unavailable)
17:47:40.174994 recvfrom(12, 0x743234, 7000, 0, 0, 0) = -1 EAGAIN (Resource
temporarily unavailable)
17:47:40.175043 epoll_ctl(3, EPOLL_CTL_ADD, 12, {EPOLLIN|0x2000, {u32=12,
u64=12}}) = 0
17:47:40.175075 epoll_wait(3, {}, 200, 1000) = 0
17:47:41.176196 epoll_wait(3, {}, 200, 1000) = 0
17:47:42.177325 epoll_wait(3, {}, 200, 1000) = 0
17:47:43.178458 epoll_wait(3, {{EPOLLIN, {u32=12, u64=12}}}, 200, 1000) = 1
17:47:44.071505 recvfrom(12, "GET /?t=1 HTTP/1.1\r\n", 7000, 0, NULL, NULL)
= 24
17:47:44.071564 setsockopt(12, SOL_TCP, TCP_QUICKACK, [1], 4) = 0
17:47:44.071612 epoll_wait(3, {{EPOLLIN, {u32=12, u64=12}}}, 200, 1000) = 1
17:47:44.411719 recvfrom(12, "\r\n", 6976, 0, NULL, NULL) = 2
17:47:44.411809 socket(PF_INET, SOCK_STREAM, IPPROTO_TCP) = 13
17:47:44.411841 fcntl(13, F_SETFL, O_RDONLY|O_NONBLOCK) = 0
17:47:44.411864 setsockopt(13, SOL_TCP, TCP_NODELAY, [1], 4) = 0
17:47:44.411891 connect(13, {sa_family=AF_INET, sin_port=htons(8080),
sin_addr=inet_addr("127.0.0.1")}, 16) = -1 EINPROGRESS (Operation now in
progress)
17:47:44.411969 epoll_wait(3, {}, 200, 0) = 0
17:47:44.411998 sendto(13, "GET /?t=1 HTTP/1.1\r\n\r\n", 26,
MSG_DONTWAIT|MSG_NOSIGNAL, NULL, 0) = 26
17:47:44.412047 epoll_wait(3, {}, 200, 0) = 0
17:47:44.412071 recvfrom(13, 0x745224, 7000, 0, 0, 0) = -1 EAGAIN (Resource
temporarily unavailable)
17:47:44.412096 epoll_ctl(3, EPOLL_CTL_ADD, 13, {EPOLLIN|0x2000, {u32=13,
u64=13}}) = 0
17:47:44.412121 epoll_wait(3, {}, 200, 1000) = 0
17:47:45.413235 epoll_wait(3, {{EPOLLIN|0x2000, {u32=12, u64=12}}}, 200, 1000)
= 1
17:47:46.288142 recvfrom(12, "", 7000, 0, NULL, NULL) = 0
=> the close was received
17:47:46.288210 epoll_ctl(3, EPOLL_CTL_DEL, 12, {0, {u32=12, u64=12}}) = 0
17:47:46.288242 epoll_wait(3, {}, 200, 1000) = 0
17:47:47.289369 epoll_wait(3, {}, 200, 1000) = 0
... but not forwarded, as expected.
Now with "option abortonclose" :
17:50:30.406154 accept4(7, {sa_family=AF_INET, sin_port=htons(53838),
sin_addr=inet_addr("127.0.0.1")}, [16], SOCK_NONBLOCK) = 12
17:50:30.406247 setsockopt(12, SOL_TCP, TCP_NODELAY, [1], 4) = 0
17:50:30.406297 accept4(7, 0x7fff407d3740, [128], SOCK_NONBLOCK) = -1 EAGAIN
(Resource temporarily unavailable)
17:50:30.406352 recvfrom(12, 0x743234, 7000, 0, 0, 0) = -1 EAGAIN (Resource
temporarily unavailable)
17:50:30.406415 epoll_ctl(3, EPOLL_CTL_ADD, 12, {EPOLLIN|0x2000, {u32=12,
u64=12}}) = 0
17:50:30.406444 epoll_wait(3, {}, 200, 1000) = 0
17:50:31.407562 epoll_wait(3, {}, 200, 1000) = 0
17:50:32.408696 epoll_wait(3, {}, 200, 1000) = 0
17:50:33.409768 epoll_wait(3, {{EPOLLIN, {u32=12, u64=12}}}, 200, 1000) = 1
17:50:33.451660 recvfrom(12, "GET /?t=1 HTTP/1.1\r\n", 7000, 0, NULL, NULL)
= 24
17:50:33.451715 setsockopt(12, SOL_TCP, TCP_QUICKACK, [1], 4) = 0
17:50:33.451762 epoll_wait(3, {{EPOLLIN, {u32=12, u64=12}}}, 200, 1000) = 1
17:50:33.736088 recvfrom(12, "\r\n", 6976, 0, NULL, NULL) = 2
17:50:33.736186 socket(PF_INET, SOCK_STREAM, IPPROTO_TCP) = 13
17:50:33.736222 fcntl(13, F_SETFL, O_RDONLY|O_NONBLOCK) = 0
17:50:33.736246 setsockopt(13, SOL_TCP, TCP_NODELAY, [1], 4) = 0
17:50:33.736272 connect(13, {sa_family=AF_INET, sin_port=htons(8080),
sin_addr=inet_addr("127.0.0.1")}, 16) = -1 EINPROGRESS (Operation now in
progress)
17:50:33.736349 epoll_wait(3, {}, 200, 0) = 0
17:50:33.736378 sendto(13, "GET /?t=1 HTTP/1.1\r\n\r\n", 26,
MSG_DONTWAIT|MSG_NOSIGNAL, NULL, 0) = 26
17:50:33.736427 epoll_wait(3, {}, 200, 0) = 0
17:50:33.736451 recvfrom(13, 0x745224, 7000, 0, 0, 0) = -1 EAGAIN (Resource
temporarily unavailable)
17:50:33.736475 epoll_ctl(3, EPOLL_CTL_ADD, 13, {EPOLLIN|0x2000, {u32=13,
u64=13}}) = 0
17:50:33.736500 epoll_wait(3, {{EPOLLIN|0x2000, {u32=12, u64=12}}}, 200, 1000)
= 1
17:50:34.378879 recvfrom(12, "", 7000, 0, NULL, NULL) = 0
==> close was received
17:50:34.378929 shutdown(13, 1 /* send */) = 0
==> and immediately forwarded.
17:50:34.378992 epoll_ctl(3, EPOLL_CTL_DEL, 12, {0, {u32=12, u64=12}}) = 0
17:50:34.379043 epoll_wait(3, {}, 200, 1000) = 0
17:50:35.380170 epoll_wait(3, {}, 200, 1000) = 0
17:50:36.381305 epoll_wait(3, {}, 200, 1000) = 0
==> the server didn't consider it, as is confirmed here :
$ netstat -atn|grep CLO
tcp0 0 127.0.0.1:8080 127.0.0.1:36255 CLOSE_WAIT
tcp0 0 127.0.0.1:18080 127.0.0.1:53838 CLOSE_WAIT
That's normal, this is httpterm and it doesn't monitor the connection while
it's waiting. But in your cas
Re: abortonclose for established connections?
Hi Ludovico, On Fri, Apr 17, 2015 at 08:24:43PM -0700, Ludovico Cavedon wrote: > Hi, > > I am trying to find a solution to the following issue. > > I have a client A that sends hundreds of HTTP requests per second to > server B running ha-proxy 1.5.3. > Server B/haproxy forwards them to server C. > > Some of these request are long-polling: server C will receive the > request and hang for a very long time (even hours potentially). > Client A may decide to give up on long-polling request, close the > connection and forget about it. > However, haproxy will keep the connection state half-open, assuming > the client is still there waiting for data. > > On client A the connection state will go to FIN_WAIT2, and will be > forgotten after 60 seconds. > > After a while, client A will happen reuse the source port for a new > connection, and send a SYN packet. On server B, however, the > connection is still there in CLOSE_WAIT. > > I am trying to find a way to avoid this from happening. Ideally I > would like haproxy to close the connection completely if the client > closes its side. > "abortonclose" seemed the right option, but it looks like it works > only if the FIN arrives before the connection to server C has not been > established yet. > > The statement from [1] sounded promising: > > In 1.5 we have even improved that a bit further for users of long-polling > > requests. When "option abortonclose" is set, if the client closes the send > > side, then haproxy forwards this closing event to the server. > > however it does not seem to happen in my case. > > What am I doing wrong? Does it have to do with the fact that > connection to server C are reused/persistent? > Is there maybe a way to tell haproxy to close the client connection if > there is no keepalive? You're not necessarily doing anything wrong, it's fairly possible there's a bug or an inconsistency between multiple options. I'll have to retry here with your config, because the statement you quoted indeed indicates that we should get a better behaviour. I won't do it immediately because I'm still stuck fixing the things I recently broke before merging that into 1.6. In the mean time I would appreciate it if you could retry with 1.5.11 to verify if you see the close being forwarded or not, because maybe it's due to a bug that was fixed since 1.5.3. Thanks, Willy
abortonclose for established connections?
Hi, I am trying to find a solution to the following issue. I have a client A that sends hundreds of HTTP requests per second to server B running ha-proxy 1.5.3. Server B/haproxy forwards them to server C. Some of these request are long-polling: server C will receive the request and hang for a very long time (even hours potentially). Client A may decide to give up on long-polling request, close the connection and forget about it. However, haproxy will keep the connection state half-open, assuming the client is still there waiting for data. On client A the connection state will go to FIN_WAIT2, and will be forgotten after 60 seconds. After a while, client A will happen reuse the source port for a new connection, and send a SYN packet. On server B, however, the connection is still there in CLOSE_WAIT. I am trying to find a way to avoid this from happening. Ideally I would like haproxy to close the connection completely if the client closes its side. "abortonclose" seemed the right option, but it looks like it works only if the FIN arrives before the connection to server C has not been established yet. The statement from [1] sounded promising: > In 1.5 we have even improved that a bit further for users of long-polling > requests. When "option abortonclose" is set, if the client closes the send > side, then haproxy forwards this closing event to the server. however it does not seem to happen in my case. What am I doing wrong? Does it have to do with the fact that connection to server C are reused/persistent? Is there maybe a way to tell haproxy to close the client connection if there is no keepalive? Thank you in advance for your help! Ludovico Here is my haproxy configuration: global log /dev/loglocal2 chroot /var/lib/haproxy stats socket /run/haproxy/admin.sock mode 660 level admin stats timeout 30s user haproxy group haproxy daemon defaults log global modehttp option httplog option dontlognull timeout connect 5s timeout client 700s timeout client-fin 30s timeout server 700s timeout tunnel 72h listen http-internal bind 10.A.B.C:80 mode http cookie SERVERID insert nocache indirect option http-keep-alive option forwardfor option redispatch option abortonclose capture request header X-Real-IP len 50 default_backend user backend user balance hdr(User-Agent) hash-type consistent mode http server serverC01 10.A.B.X:80 check server serverC01 10.A.B.Y:80 check [1] http://permalink.gmane.org/gmane.comp.web.haproxy/3862
