Hi,
We're running a haproxy as a TLS unwrapping daemon (socket to socket) and
are running into some cases where processes will spin at 100% CPU for 5-30
seconds.
It looks related to half-closed or resetting TCP connections out to end
users, and always self-recovers after some amount of time.
The symptoms differ slightly:
Usually it looks like this:
epoll_wait(0, {{EPOLLIN|EPOLLHUP|0x2000, {u32=8881, u64=8881}}}, 200, 14) =
1
read(8881, 0x870ff53, 5) = -1 EAGAIN (Resource temporarily
unavailable)
epoll_wait(0, {{EPOLLIN|EPOLLHUP|0x2000, {u32=8881, u64=8881}}}, 200, 14) =
1
read(8881, 0x870ff53, 5) = -1 EAGAIN (Resource temporarily
unavailable)
epoll_wait(0, {{EPOLLIN|EPOLLHUP|0x2000, {u32=8881, u64=8881}}}, 200, 14) =
1
read(8881, 0x870ff53, 5) = -1 EAGAIN (Resource temporarily
unavailable)
epoll_wait(0, {{EPOLLIN|EPOLLHUP|0x2000, {u32=8881, u64=8881}}}, 200, 14) =
1
read(8881, 0x870ff53, 5) = -1 EAGAIN (Resource temporarily
unavailable)
epoll_wait(0, {{EPOLLIN|EPOLLHUP|0x2000, {u32=8881, u64=8881}}}, 200, 13) =
1
EPOLLIN is set, EPOLLHUP is set, and EPOLLRDHUP (0x2000, which is also
mapped to POLL_HUP internally). The read() always fails as EAGAIN and it
drops immediately back into the epoll loop.
Occasionally the syscalls are recvfrom() instead of read().
Occasionally the EPOLLERR flag is *also* set, yet it still loops. So the
connection has a fatal problem.
Occasionally it'll call epoll_wait() in a tight loop, with a combination of
the above options, but never make a syscall.
...and I've seen it with just EPOLLIN|RDHUP and no HUP.
The src/ssl_sock.c:ssl_sock_to_buf func appears to be most of the problem.
Compared to the raw socket function, nothing is checking for POLL_ERR.
However I'm not confident I know where to place the HUP detection, or how
it's looping with epoll_wait() without doing any syscalls.
OpenSSL internally buffers partial packets, but if the remote end is HUP'ed
there's probably no way you'll get the rest of the read.
I'm also not really sure how the sockets can end up in this state without
haproxy immediately closing them (IN + HUP + EAGAIN -> socket was closed,
right?). It's an obvious enough weirdness that I feel like I'm reading this
wrong. It does also eventually wiggle into a state where haproxy closes the
conn completely, which is why it doesn't spin forever. It's not clear if
that's from a timeout or a state change in the socket.
Anything else I can dig up on this? We have some small modifications to
haproxy, but largely in unrelated areas of the code. I can share the
changes privately with someone if necessary. Haven't figured out a test
case yet, hoping the description makes the issue obvious to the authors.
thanks!
-Dormando
(not mailing from my usual address since I got RBL'ed :/ user on the same
IP had a spam field day)
