>
> Could you please confirm that most of the CLOSE_WAIT are on the front side
> and the ESTABLISHED on the backend side ? If that's the case, can you also
> please verify if there are pending data in the send queue for CLOSE_WAIT
> sockets (3rd column in netstat) ?
>
Most of the close_wait/established are on the haproxy side, between haproxy
and cdn, not haproxy and nginx.
what we see from netstat/ss is the recv queue is full, not the send-q,
below is our haproxy accept queue, which is 40000, when haproxy down, the
recv-Q soon become full like 40000 or 40001:
State Recv-Q Send-Q Local Address:Port Peer Address:Port
LISTEN 0 40000 *:443 *:*
users:(("haproxy",pid=152006,fd=8))
I will give you more details after new benchmark to reproduce.
On Fri, Apr 21, 2017 at 1:23 PM, Willy Tarreau <w...@1wt.eu> wrote:
> On Thu, Apr 20, 2017 at 11:03:42PM +0800, jaseywang wrote:
> > 1. The backlog of haproxy soon become full and begin to drop new tcp
> > connection since peak traffic begin, before CDN, our net.core.somaxconn
> is
> > 1024, and use default backlog of haproxy, everything performs well. After
> > using CDN, even we increase the somaxconn and backlog parameter to 40000,
> > it soon becomes full and deny to accept new connection, so no normal http
> > request to haproxy, and our client can't open the webpage.
> >
> > 2. Before CDN, the normal TCP connections like:
> > 7k estab, 700 finwait1, 5k finwai2, 14k timewait,10 closewait, 230
> lastack.
> > Now, the TCP connections like:
> > 22k estab, 260 finwai1, 1.6k finwai2, 23k timewait, 25k closewait, 5k
> > lastack.
> > The most weird thing is the abnormal increasement of closewait, and after
> > the packet capture, most of the closewait is between haproxy and cdn.
>
> This really sounds like the CDN tries to close some connections at certain
> moments when it's not appropriate (eg: in the middle of a transfer),
> leaving
> a CLOSE_WAIT between haproxy and the CDN, and an ESTABLISHED between
> haproxy
> and nginx. The problem is to understand why it remains in this situation
> instead of either resetting the received data or finishing to drain them.
>
> Could you please confirm that most of the CLOSE_WAIT are on the front side
> and the ESTABLISHED on the backend side ? If that's the case, can you also
> please verify if there are pending data in the send queue for CLOSE_WAIT
> sockets (3rd column in netstat) ?
>
> It would also help a lot to provide the output from this :
>
> echo "show sess all" | socat /var/run/haproxy.sock -
>
> as well as the output from "netstat -nato" to compare them. Be careful,
> this
> may reveal some confidential information such as some private IP addresses,
> so you may prefer not to post to the list, it's up to you.
>
> Otherwise your config looks pretty clean and safe, so I don't think it's
> a config issue you're facing.
>
> Willy
>
