Re: abortonclose for established connections?

[Ludovico Cavedon]

Hi Willy,

On Wed, Apr 22, 2015 at 10:58 AM, Ludovico Cavedon cave...@lastline.com
wrote:

 I will let you know, thanks again!


I reproduced your test case and it turned out the abortonclose option was
only in the listener section (and not on backend/defautls).
I moved it to defaults and now haproxy is behaving as expected.

Thanks for your help!
Ludovico

Re: abortonclose for established connections?

[Willy Tarreau]

Hi Ludovico,

On Mon, Apr 27, 2015 at 10:54:43AM -0700, Ludovico Cavedon wrote:
 Hi Willy,
 
 On Wed, Apr 22, 2015 at 10:58 AM, Ludovico Cavedon cave...@lastline.com
 wrote:
 
  I will let you know, thanks again!
 
 
 I reproduced your test case and it turned out the abortonclose option was
 only in the listener section (and not on backend/defautls).
 I moved it to defaults and now haproxy is behaving as expected.

Great! Thanks for your feedback!

Cheers,
Willy

Re: abortonclose for established connections?

[Willy Tarreau]

Hi again Ludovico,

so I ran some tests here with latest 1.5. I telnet to haproxy, send
GET /?t=1 HTTP/1.1 then quit. It forwards to a server which
waits 10s before responding.

First, without option abortonclose :

17:47:40.174784 accept4(7, {sa_family=AF_INET, sin_port=htons(53833), 
sin_addr=inet_addr(127.0.0.1)}, [16], SOCK_NONBLOCK) = 12
17:47:40.174879 setsockopt(12, SOL_TCP, TCP_NODELAY, [1], 4) = 0
17:47:40.174927 accept4(7, 0x7fff89a6c8d0, [128], SOCK_NONBLOCK) = -1 EAGAIN 
(Resource temporarily unavailable)
17:47:40.174994 recvfrom(12, 0x743234, 7000, 0, 0, 0) = -1 EAGAIN (Resource 
temporarily unavailable)
17:47:40.175043 epoll_ctl(3, EPOLL_CTL_ADD, 12, {EPOLLIN|0x2000, {u32=12, 
u64=12}}) = 0
17:47:40.175075 epoll_wait(3, {}, 200, 1000) = 0
17:47:41.176196 epoll_wait(3, {}, 200, 1000) = 0
17:47:42.177325 epoll_wait(3, {}, 200, 1000) = 0
17:47:43.178458 epoll_wait(3, {{EPOLLIN, {u32=12, u64=12}}}, 200, 1000) = 1
17:47:44.071505 recvfrom(12, GET /?t=1 HTTP/1.1\r\n, 7000, 0, NULL, NULL) 
= 24
17:47:44.071564 setsockopt(12, SOL_TCP, TCP_QUICKACK, [1], 4) = 0
17:47:44.071612 epoll_wait(3, {{EPOLLIN, {u32=12, u64=12}}}, 200, 1000) = 1
17:47:44.411719 recvfrom(12, \r\n, 6976, 0, NULL, NULL) = 2
17:47:44.411809 socket(PF_INET, SOCK_STREAM, IPPROTO_TCP) = 13
17:47:44.411841 fcntl(13, F_SETFL, O_RDONLY|O_NONBLOCK) = 0
17:47:44.411864 setsockopt(13, SOL_TCP, TCP_NODELAY, [1], 4) = 0
17:47:44.411891 connect(13, {sa_family=AF_INET, sin_port=htons(8080), 
sin_addr=inet_addr(127.0.0.1)}, 16) = -1 EINPROGRESS (Operation now in 
progress)
17:47:44.411969 epoll_wait(3, {}, 200, 0) = 0
17:47:44.411998 sendto(13, GET /?t=1 HTTP/1.1\r\n\r\n, 26, 
MSG_DONTWAIT|MSG_NOSIGNAL, NULL, 0) = 26
17:47:44.412047 epoll_wait(3, {}, 200, 0) = 0
17:47:44.412071 recvfrom(13, 0x745224, 7000, 0, 0, 0) = -1 EAGAIN (Resource 
temporarily unavailable)
17:47:44.412096 epoll_ctl(3, EPOLL_CTL_ADD, 13, {EPOLLIN|0x2000, {u32=13, 
u64=13}}) = 0
17:47:44.412121 epoll_wait(3, {}, 200, 1000) = 0
17:47:45.413235 epoll_wait(3, {{EPOLLIN|0x2000, {u32=12, u64=12}}}, 200, 1000) 
= 1
17:47:46.288142 recvfrom(12, , 7000, 0, NULL, NULL) = 0

= the close was received

17:47:46.288210 epoll_ctl(3, EPOLL_CTL_DEL, 12, {0, {u32=12, u64=12}}) = 0
17:47:46.288242 epoll_wait(3, {}, 200, 1000) = 0
17:47:47.289369 epoll_wait(3, {}, 200, 1000) = 0

... but not forwarded, as expected.

Now with option abortonclose :

17:50:30.406154 accept4(7, {sa_family=AF_INET, sin_port=htons(53838), 
sin_addr=inet_addr(127.0.0.1)}, [16], SOCK_NONBLOCK) = 12
17:50:30.406247 setsockopt(12, SOL_TCP, TCP_NODELAY, [1], 4) = 0
17:50:30.406297 accept4(7, 0x7fff407d3740, [128], SOCK_NONBLOCK) = -1 EAGAIN 
(Resource temporarily unavailable)
17:50:30.406352 recvfrom(12, 0x743234, 7000, 0, 0, 0) = -1 EAGAIN (Resource 
temporarily unavailable)
17:50:30.406415 epoll_ctl(3, EPOLL_CTL_ADD, 12, {EPOLLIN|0x2000, {u32=12, 
u64=12}}) = 0
17:50:30.406444 epoll_wait(3, {}, 200, 1000) = 0
17:50:31.407562 epoll_wait(3, {}, 200, 1000) = 0
17:50:32.408696 epoll_wait(3, {}, 200, 1000) = 0
17:50:33.409768 epoll_wait(3, {{EPOLLIN, {u32=12, u64=12}}}, 200, 1000) = 1
17:50:33.451660 recvfrom(12, GET /?t=1 HTTP/1.1\r\n, 7000, 0, NULL, NULL) 
= 24
17:50:33.451715 setsockopt(12, SOL_TCP, TCP_QUICKACK, [1], 4) = 0
17:50:33.451762 epoll_wait(3, {{EPOLLIN, {u32=12, u64=12}}}, 200, 1000) = 1
17:50:33.736088 recvfrom(12, \r\n, 6976, 0, NULL, NULL) = 2
17:50:33.736186 socket(PF_INET, SOCK_STREAM, IPPROTO_TCP) = 13
17:50:33.736222 fcntl(13, F_SETFL, O_RDONLY|O_NONBLOCK) = 0
17:50:33.736246 setsockopt(13, SOL_TCP, TCP_NODELAY, [1], 4) = 0
17:50:33.736272 connect(13, {sa_family=AF_INET, sin_port=htons(8080), 
sin_addr=inet_addr(127.0.0.1)}, 16) = -1 EINPROGRESS (Operation now in 
progress)
17:50:33.736349 epoll_wait(3, {}, 200, 0) = 0
17:50:33.736378 sendto(13, GET /?t=1 HTTP/1.1\r\n\r\n, 26, 
MSG_DONTWAIT|MSG_NOSIGNAL, NULL, 0) = 26
17:50:33.736427 epoll_wait(3, {}, 200, 0) = 0
17:50:33.736451 recvfrom(13, 0x745224, 7000, 0, 0, 0) = -1 EAGAIN (Resource 
temporarily unavailable)
17:50:33.736475 epoll_ctl(3, EPOLL_CTL_ADD, 13, {EPOLLIN|0x2000, {u32=13, 
u64=13}}) = 0
17:50:33.736500 epoll_wait(3, {{EPOLLIN|0x2000, {u32=12, u64=12}}}, 200, 1000) 
= 1
17:50:34.378879 recvfrom(12, , 7000, 0, NULL, NULL) = 0

== close was received

17:50:34.378929 shutdown(13, 1 /* send */) = 0

== and immediately forwarded.

17:50:34.378992 epoll_ctl(3, EPOLL_CTL_DEL, 12, {0, {u32=12, u64=12}}) = 0
17:50:34.379043 epoll_wait(3, {}, 200, 1000) = 0
17:50:35.380170 epoll_wait(3, {}, 200, 1000) = 0
17:50:36.381305 epoll_wait(3, {}, 200, 1000) = 0

== the server didn't consider it, as is confirmed here :

$ netstat -atn|grep CLO
tcp0  0 127.0.0.1:8080  127.0.0.1:36255 CLOSE_WAIT 
tcp0  0 127.0.0.1:18080 127.0.0.1:53838 CLOSE_WAIT 

That's normal, this is httpterm and it doesn't monitor the connection while
it's waiting. But in your case it should definitely work. Or 

Re: abortonclose for established connections?

[Ludovico Cavedon]

Hi Willy,

thank you for the very detailed information.

On Wed, Apr 22, 2015 at 8:57 AM, Willy Tarreau w...@1wt.eu wrote:

 That's normal, this is httpterm and it doesn't monitor the connection while
 it's waiting. But in your case it should definitely work. Or it means that
 your server ignores the client abort. It would be nice if you could double-
 check this.


This could very well be the case. I will check and let you know.

If your server does indeed not support client-abort, then there's a
 workaround. Haproxy 1.5 supports half-closed timeouts. You can simply
 add a directive timeout server-fin 1s and it will automatically close
 the server connection 1 second after sending it the close if the server
 doesn't respond (and then report a server error) :


My server is nginx, I will see if I can configure it to abortonclose as
well. Otherwise the server-fin option looks viable.

I suspect it could match your needs if the normal solution doesn't work.
 This *does* require abortonclose since you want the shutdown() to have been
 done first. Please keep us updated.


I will let you know, thanks again!

Ludovico

Re: abortonclose for established connections?

[Willy Tarreau]

Hi Ludovico,

On Fri, Apr 17, 2015 at 08:24:43PM -0700, Ludovico Cavedon wrote:
 Hi,
 
 I am trying to find a solution to the following issue.
 
 I have a client A that sends hundreds of HTTP requests per second to
 server B running ha-proxy 1.5.3.
 Server B/haproxy forwards them to server C.
 
 Some of these request are long-polling: server C will receive the
 request and hang for a very long time (even hours potentially).
 Client A may decide to give up on long-polling request, close the
 connection and forget about it.
 However, haproxy will keep the connection state half-open, assuming
 the client is still there waiting for data.
 
 On client A the connection state will go to FIN_WAIT2, and will be
 forgotten after 60 seconds.
 
 After a while, client A will happen reuse the source port for a new
 connection, and send a SYN packet. On server B, however, the
 connection is still there in CLOSE_WAIT.
 
 I am trying to find a way to avoid this from happening. Ideally I
 would like haproxy to close the connection completely if the client
 closes its side.
 abortonclose seemed the right option, but it looks like it works
 only if the FIN arrives before the connection to server C has not been
 established yet.
 
 The statement from [1] sounded promising:
  In 1.5 we have even improved that a bit further for users of long-polling
  requests. When option abortonclose is set, if the client closes the send
  side, then haproxy forwards this closing event to the server.
 
 however it does not seem to happen in my case.
 
 What am I doing wrong? Does it have to do with the fact that
 connection to server C are reused/persistent?
 Is there maybe a way to tell haproxy to close the client connection if
 there is no keepalive?

You're not necessarily doing anything wrong, it's fairly possible there's
a bug or an inconsistency between multiple options. I'll have to retry
here with your config, because the statement you quoted indeed indicates
that we should get a better behaviour. I won't do it immediately because
I'm still stuck fixing the things I recently broke before merging that
into 1.6.

In the mean time I would appreciate it if you could retry with 1.5.11 to
verify if you see the close being forwarded or not, because maybe it's due
to a bug that was fixed since 1.5.3.

Thanks,
Willy