Hi Willy On Thu, 31 May 2018 at 21:07, Willy Tarreau <w...@1wt.eu> wrote:
> Thank you. It's pretty constant, so it doesn't really seem load-dependant > (unless your load is reasonably stable of course). Here is a 3-day sample from the "grow" period - https://pasteboard.co/HnQu44t.png. Overall trend is steady growth with somewhat varying speed. There are also short periods when the number of frontend connections decreases. These periods seem to match times when request rate drops below 5000/hour. > I'm seeing that the number of entries in "show fd" is not that much, did > you truncate it or > is it all you had ?I'm asking because suddenly that makes me think that > it could be possible that we're leaking sockets that are not correctly > closed (for whatever reason). That would explain why they're not visible > in "show fd" but remain in CLOSE_WAIT. It's absolutely not what I was > expecting last time and I could have overlooked this possibility. > "show fd" list was not truncated. It's really shorter than the number of CLOSE_WAIT netstat entries. Extra netstat "CLOSE_WAIT" connections invisible in "show fg" may come from socket leak. See below for current sample - 111 CLOSE_WAIT vs. 64 "show fd". Why are there so many "show fd" sockets with cflg=0x80203300? They are the one which keep growing. Diff between my first "show fd" from yesterday and new from today clearly shows that a lot of them did not change and new appeared. cat show-fd-2018-06-01.txt | cut -c128-142 | grep cflg | sort | uniq -c | sort -n 1 cflg=0x00202306 1 cflg=0x00241300 10 cflg=0x80203306 45 cflg=0x80203300 !!!!! wc -l * 42 show-fd-2018-05-31.txt 64 show-fd-2018-06-01.txt diff -u * --- show-fd-2018-05-31.txt 2018-06-01 10:58:21.354207477 +0200 +++ show-fd-2018-06-01.txt 2018-06-01 10:58:50.886461366 +0200 @@ -12,7 +12,7 @@ 17 : st=0x20(R:pra W:pRa) ev=0x00(heopi) [nlc] cache=0 owner=0x2012970 iocb=0x4d2c80(conn_fd_handler) tmask=0x1 umask=0x0 cflg=0x80203300 fe=fe-http mux=H2 mux_ctx=0x22a7480 18 : st=0x20(R:pra W:pRa) ev=0x00(heopi) [nlc] cache=0 owner=0x21b7e30 iocb=0x4d2c80(conn_fd_handler) tmask=0x1 umask=0x0 cflg=0x80203300 fe=fe-http mux=H2 mux_ctx=0x210b610 19 : st=0x20(R:pra W:pRa) ev=0x00(heopi) [nlc] cache=0 owner=0x1efa590 iocb=0x4d2c80(conn_fd_handler) tmask=0x1 umask=0x0 cflg=0x80203300 fe=fe-http mux=H2 mux_ctx=0x1f02610 - 20 : st=0x25(R:PrA W:pRa) ev=0x00(heopi) [nlc] cache=0 owner=0x21b83a0 iocb=0x4d2c80(conn_fd_handler) tmask=0x1 umask=0x0 cflg=0x80203306 fe=fe-http mux=H2 mux_ctx=0x22a2800 + 20 : st=0x20(R:pra W:pRa) ev=0x00(heopi) [nlc] cache=0 owner=0x1fe8a90 iocb=0x4d2c80(conn_fd_handler) tmask=0x1 umask=0x0 cflg=0x80203300 fe=fe-http mux=H2 mux_ctx=0x2311bd0 21 : st=0x20(R:pra W:pRa) ev=0x00(heopi) [nlc] cache=0 owner=0x20a0f10 iocb=0x4d2c80(conn_fd_handler) tmask=0x1 umask=0x0 cflg=0x80203300 fe=fe-http mux=H2 mux_ctx=0x216ecf0 22 : st=0x20(R:pra W:pRa) ev=0x00(heopi) [nlc] cache=0 owner=0x1f02a40 iocb=0x4d2c80(conn_fd_handler) tmask=0x1 umask=0x0 cflg=0x80203300 fe=fe-http mux=H2 mux_ctx=0x20d03b0 23 : st=0x20(R:pra W:pRa) ev=0x00(heopi) [nlc] cache=0 owner=0x21b8e80 iocb=0x4d2c80(conn_fd_handler) tmask=0x1 umask=0x0 cflg=0x80203300 fe=fe-http mux=H2 mux_ctx=0x22d2ff0 @@ -25,18 +25,40 @@ 30 : st=0x20(R:pra W:pRa) ev=0x00(heopi) [nlc] cache=0 owner=0x217cac0 iocb=0x4d2c80(conn_fd_handler) tmask=0x1 umask=0x0 cflg=0x80203300 fe=fe-http mux=H2 mux_ctx=0x22cd210 31 : st=0x20(R:pra W:pRa) ev=0x00(heopi) [nlc] cache=0 owner=0x2163540 iocb=0x4d2c80(conn_fd_handler) tmask=0x1 umask=0x0 cflg=0x80203300 fe=fe-http mux=H2 mux_ctx=0x209b2e0 32 : st=0x20(R:pra W:pRa) ev=0x00(heopi) [nlc] cache=0 owner=0x209b750 iocb=0x4d2c80(conn_fd_handler) tmask=0x1 umask=0x0 cflg=0x80203300 fe=fe-http mux=H2 mux_ctx=0x20a02b0 - 33 : st=0x25(R:PrA W:pRa) ev=0x00(heopi) [nlc] cache=0 owner=0x20ce280 iocb=0x4d2c80(conn_fd_handler) tmask=0x1 umask=0x0 cflg=0x80203306 fe=fe-http mux=H2 mux_ctx=0x2321d60 + 33 : st=0x20(R:pra W:pRa) ev=0x00(heopi) [nlc] cache=0 owner=0x20c07c0 iocb=0x4d2c80(conn_fd_handler) tmask=0x1 umask=0x0 cflg=0x80203300 fe=fe-http mux=H2 mux_ctx=0x2328430 34 : st=0x20(R:pra W:pRa) ev=0x00(heopi) [nlc] cache=0 owner=0x2162d40 iocb=0x4d2c80(conn_fd_handler) tmask=0x1 umask=0x0 cflg=0x80203300 fe=fe-http mux=H2 mux_ctx=0x1fbe580 + 35 : st=0x20(R:pra W:pRa) ev=0x00(heopi) [nlc] cache=0 owner=0x1f914d0 iocb=0x4d2c80(conn_fd_handler) tmask=0x1 umask=0x0 cflg=0x80203300 fe=fe-http mux=H2 mux_ctx=0x1ed2da0 36 : st=0x20(R:pra W:pRa) ev=0x00(heopi) [nlc] cache=0 owner=0x1f05a70 iocb=0x4d2c80(conn_fd_handler) tmask=0x1 umask=0x0 cflg=0x80203300 fe=fe-http mux=H2 mux_ctx=0x20c0d80 37 : st=0x20(R:pra W:pRa) ev=0x00(heopi) [nlc] cache=0 owner=0x20e43a0 iocb=0x4d2c80(conn_fd_handler) tmask=0x1 umask=0x0 cflg=0x80203300 fe=fe-http mux=H2 mux_ctx=0x1f18200 38 : st=0x20(R:pra W:pRa) ev=0x00(heopi) [nlc] cache=0 owner=0x20a0120 iocb=0x4d2c80(conn_fd_handler) tmask=0x1 umask=0x0 cflg=0x80203300 fe=fe-http mux=H2 mux_ctx=0x20ea4e0 39 : st=0x20(R:pra W:pRa) ev=0x00(heopi) [nlc] cache=0 owner=0x2082a90 iocb=0x4d2c80(conn_fd_handler) tmask=0x1 umask=0x0 cflg=0x80203300 fe=fe-http mux=H2 mux_ctx=0x22a1e00 40 : st=0x20(R:pra W:pRa) ev=0x00(heopi) [nlc] cache=0 owner=0x21b8740 iocb=0x4d2c80(conn_fd_handler) tmask=0x1 umask=0x0 cflg=0x80203300 fe=fe-http mux=H2 mux_ctx=0x22a5ca0 - 41 : st=0x25(R:PrA W:pRa) ev=0x00(heopi) [nlc] cache=0 owner=0x215a730 iocb=0x4d2c80(conn_fd_handler) tmask=0x1 umask=0x0 cflg=0x80203306 fe=fe-http mux=H2 mux_ctx=0x236f300 - 42 : st=0x25(R:PrA W:pRa) ev=0x00(heopi) [nlc] cache=0 owner=0x210b440 iocb=0x4d2c80(conn_fd_handler) tmask=0x1 umask=0x0 cflg=0x80203306 fe=fe-http mux=H2 mux_ctx=0x2326220 + 41 : st=0x20(R:pra W:pRa) ev=0x00(heopi) [nlc] cache=0 owner=0x1fa7b10 iocb=0x4d2c80(conn_fd_handler) tmask=0x1 umask=0x0 cflg=0x80203300 fe=fe-http mux=H2 mux_ctx=0x230db80 + 42 : st=0x20(R:pra W:pRa) ev=0x00(heopi) [nlc] cache=0 owner=0x2152460 iocb=0x4d2c80(conn_fd_handler) tmask=0x1 umask=0x0 cflg=0x80203300 fe=fe-http mux=H2 mux_ctx=0x1f2bf70 43 : st=0x20(R:pra W:pRa) ev=0x00(heopi) [nlc] cache=0 owner=0x20d8110 iocb=0x4d2c80(conn_fd_handler) tmask=0x1 umask=0x0 cflg=0x80203300 fe=fe-http mux=H2 mux_ctx=0x1faa770 - 44 : st=0x25(R:PrA W:pRa) ev=0x00(heopi) [nlc] cache=0 owner=0x216f020 iocb=0x4d2c80(conn_fd_handler) tmask=0x1 umask=0x0 cflg=0x80203306 fe=fe-http mux=H2 mux_ctx=0x20e7850 - 45 : st=0x24(R:Pra W:pRa) ev=0x00(heopi) [nlc] cache=0 owner=0x20816b0 iocb=0x4d2c80(conn_fd_handler) tmask=0x1 umask=0x1 cflg=0x00201300 fe=GLOBAL mux=PASS mux_ctx=0x1fa6600 - 46 : st=0x25(R:PrA W:pRa) ev=0x00(heopi) [nlc] cache=0 owner=0x2011390 iocb=0x4d2c80(conn_fd_handler) tmask=0x1 umask=0x0 cflg=0x80203306 fe=fe-http mux=H2 mux_ctx=0x1f2bf70 + 44 : st=0x20(R:pra W:pRa) ev=0x00(heopi) [nlc] cache=0 owner=0x1fbe0a0 iocb=0x4d2c80(conn_fd_handler) tmask=0x1 umask=0x0 cflg=0x80203300 fe=fe-http mux=H2 mux_ctx=0x22e52a0 + 45 : st=0x20(R:pra W:pRa) ev=0x00(heopi) [nlc] cache=0 owner=0x1f23ad0 iocb=0x4d2c80(conn_fd_handler) tmask=0x1 umask=0x0 cflg=0x80203300 fe=fe-http mux=H2 mux_ctx=0x22a2800 + 46 : st=0x20(R:pra W:pRa) ev=0x00(heopi) [nlc] cache=0 owner=0x20ccdd0 iocb=0x4d2c80(conn_fd_handler) tmask=0x1 umask=0x0 cflg=0x80203300 fe=fe-http mux=H2 mux_ctx=0x236d720 + 47 : st=0x20(R:pra W:pRa) ev=0x00(heopi) [nlc] cache=0 owner=0x2083130 iocb=0x4d2c80(conn_fd_handler) tmask=0x1 umask=0x0 cflg=0x80203300 fe=fe-http mux=H2 mux_ctx=0x236eb00 + 48 : st=0x20(R:pra W:pRa) ev=0x00(heopi) [nlc] cache=0 owner=0x1f90e00 iocb=0x4d2c80(conn_fd_handler) tmask=0x1 umask=0x0 cflg=0x80203300 fe=fe-http mux=H2 mux_ctx=0x2397bd0 49 : st=0x20(R:pra W:pRa) ev=0x00(heopi) [nlc] cache=0 owner=0x1f8de50 iocb=0x4d2c80(conn_fd_handler) tmask=0x1 umask=0x0 cflg=0x80203300 fe=fe-http mux=H2 mux_ctx=0x2327030 - 50 : st=0x25(R:PrA W:pRa) ev=0x00(heopi) [nlc] cache=0 owner=0x1f8aea0 iocb=0x4d2c80(conn_fd_handler) tmask=0x1 umask=0x0 cflg=0x80203306 fe=fe-http mux=H2 mux_ctx=0x230db80 + 50 : st=0x20(R:pra W:pRa) ev=0x00(heopi) [nlc] cache=0 owner=0x217a360 iocb=0x4d2c80(conn_fd_handler) tmask=0x1 umask=0x0 cflg=0x80203300 fe=fe-http mux=H2 mux_ctx=0x236d8b0 + 51 : st=0x20(R:pra W:pRa) ev=0x00(heopi) [nlc] cache=0 owner=0x215a730 iocb=0x4d2c80(conn_fd_handler) tmask=0x1 umask=0x0 cflg=0x80203300 fe=fe-http mux=H2 mux_ctx=0x238bb70 + 52 : st=0x20(R:pra W:pRa) ev=0x00(heopi) [nlc] cache=0 owner=0x209fb60 iocb=0x4d2c80(conn_fd_handler) tmask=0x1 umask=0x0 cflg=0x80203300 fe=fe-http mux=H2 mux_ctx=0x20e7850 + 53 : st=0x25(R:PrA W:pRa) ev=0x00(heopi) [nlc] cache=0 owner=0x20a1230 iocb=0x4d2c80(conn_fd_handler) tmask=0x1 umask=0x0 cflg=0x80203306 fe=fe-http mux=H2 mux_ctx=0x25a9470 + 54 : st=0x20(R:pra W:pRa) ev=0x00(heopi) [nlc] cache=0 owner=0x1f90730 iocb=0x4d2c80(conn_fd_handler) tmask=0x1 umask=0x0 cflg=0x80203300 fe=fe-http mux=H2 mux_ctx=0x1ff82d0 + 55 : st=0x25(R:PrA W:pRa) ev=0x00(heopi) [nlc] cache=0 owner=0x24d1f10 iocb=0x4d2c80(conn_fd_handler) tmask=0x1 umask=0x0 cflg=0x80203306 fe=fe-http mux=H2 mux_ctx=0x230e930 + 56 : st=0x20(R:pra W:pRa) ev=0x00(heopi) [nlc] cache=0 owner=0x21647d0 iocb=0x4d2c80(conn_fd_handler) tmask=0x1 umask=0x0 cflg=0x80203300 fe=fe-http mux=H2 mux_ctx=0x22e8b60 + 57 : st=0x20(R:pra W:pRa) ev=0x00(heopi) [nlc] cache=0 owner=0x1f8c310 iocb=0x4d2c80(conn_fd_handler) tmask=0x1 umask=0x0 cflg=0x80203300 fe=fe-http mux=H2 mux_ctx=0x1ff1b00 + 58 : st=0x20(R:pra W:pRa) ev=0x00(heopi) [nlc] cache=0 owner=0x1f90060 iocb=0x4d2c80(conn_fd_handler) tmask=0x1 umask=0x0 cflg=0x80203300 fe=fe-http mux=H2 mux_ctx=0x2326220 + 59 : st=0x25(R:PrA W:pRa) ev=0x00(heopi) [nlc] cache=0 owner=0x1ff80d0 iocb=0x4d2c80(conn_fd_handler) tmask=0x1 umask=0x0 cflg=0x80203306 fe=fe-http mux=H2 mux_ctx=0x24df3e0 + 60 : st=0x25(R:PrA W:pRa) ev=0x00(heopi) [nlc] cache=0 owner=0x21b81d0 iocb=0x4d2c80(conn_fd_handler) tmask=0x1 umask=0x0 cflg=0x80203306 fe=fe-http mux=H2 mux_ctx=0x236f300 + 61 : st=0x25(R:PrA W:pRa) ev=0x00(heopi) [nlc] cache=0 owner=0x20f3230 iocb=0x4d2c80(conn_fd_handler) tmask=0x1 umask=0x0 cflg=0x80203306 fe=fe-http mux=PASS mux_ctx=0x2163e90 + 62 : st=0x20(R:pra W:pRa) ev=0x00(heopi) [nlc] cache=0 owner=0x2158150 iocb=0x4d2c80(conn_fd_handler) tmask=0x1 umask=0x0 cflg=0x80203300 fe=fe-http mux=H2 mux_ctx=0x23004c0 + 63 : st=0x25(R:PrA W:pRa) ev=0x00(heopi) [nlc] cache=0 owner=0x21122f0 iocb=0x4d2c80(conn_fd_handler) tmask=0x1 umask=0x0 cflg=0x80203306 fe=fe-http mux=PASS mux_ctx=0x1f88ba0 + 64 : st=0x25(R:PrA W:pRa) ev=0x00(heopi) [nlc] cache=0 owner=0x21b8ae0 iocb=0x4d2c80(conn_fd_handler) tmask=0x1 umask=0x0 cflg=0x80203306 fe=fe-http mux=H2 mux_ctx=0x254fd10 + 65 : st=0x25(R:PrA W:pRa) ev=0x00(heopi) [nlc] cache=0 owner=0x210b440 iocb=0x4d2c80(conn_fd_handler) tmask=0x1 umask=0x0 cflg=0x00202306 sv=nginx01/be-nginx01 mux=PASS mux_ctx=0x1fddaf0 + 66 : st=0x22(R:pRa W:pRa) ev=0x00(heopi) [Nlc] cache=0 owner=0x21b9050 iocb=0x4d2c80(conn_fd_handler) tmask=0x1 umask=0x0 cflg=0x00241300 fe=GLOBAL mux=PASS mux_ctx=0x1f92230 + 68 : st=0x25(R:PrA W:pRa) ev=0x00(heopi) [nlc] cache=0 owner=0x21b8910 iocb=0x4d2c80(conn_fd_handler) tmask=0x1 umask=0x0 cflg=0x80203306 fe=fe-http mux=H2 mux_ctx=0x2321d60 + 69 : st=0x25(R:PrA W:pRa) ev=0x00(heopi) [nlc] cache=0 owner=0x2109130 iocb=0x4d2c80(conn_fd_handler) tmask=0x1 umask=0x0 cflg=0x80203306 fe=fe-http mux=H2 mux_ctx=0x2543fc0 + 71 : st=0x25(R:PrA W:pRa) ev=0x00(heopi) [nlc] cache=0 owner=0x1f8d780 iocb=0x4d2c80(conn_fd_handler) tmask=0x1 umask=0x0 cflg=0x80203306 fe=fe-http mux=H2 mux_ctx=0x236fdd0 Thanks, Milan ######################################################################## date ; netstat -aptn|grep haproxy|cut -c 69-80|sort|uniq -c|sort -n Fri Jun 1 10:02:07 CEST 2018 4 LISTEN 22 ESTABLISHED 111 CLOSE_WAIT ######################################################################## date ; echo "show fd" | socat - $HASOCK | wc -l Fri Jun 1 10:02:07 CEST 2018 64 ######################################################################## date ; echo "show fd" | socat - $HASOCK Fri Jun 1 10:02:07 CEST 2018 4 : st=0x05(R:PrA W:pra) ev=0x01(heopI) [nlC] cache=0 owner=0x1eb4bf0 iocb=0x4be950(listener_accept) tmask=0xffffffffffffffff umask=0xfffffffffffffffe l.st=RDY fe=GLOBAL 5 : st=0x05(R:PrA W:pra) ev=0x01(heopI) [nlC] cache=0 owner=0x1eb6df0 iocb=0x4be950(listener_accept) tmask=0xffffffffffffffff umask=0xfffffffffffffffe l.st=RDY fe=fe-http 6 : st=0x05(R:PrA W:pra) ev=0x01(heopI) [nlC] cache=0 owner=0x1eb7300 iocb=0x4be950(listener_accept) tmask=0xffffffffffffffff umask=0xfffffffffffffffe l.st=RDY fe=fe-http 8 : st=0x05(R:PrA W:pra) ev=0x01(heopI) [nlC] cache=0 owner=0x1ed1b10 iocb=0x4be950(listener_accept) tmask=0xffffffffffffffff umask=0xfffffffffffffffe l.st=RDY fe=fe-service 9 : st=0x05(R:PrA W:pra) ev=0x00(heopi) [nlC] cache=0 owner=0x1ed2020 iocb=0x4be950(listener_accept) tmask=0xffffffffffffffff umask=0xfffffffffffffffe l.st=RDY fe=fe-service 10 : st=0x05(R:PrA W:pra) ev=0x00(heopi) [nlc] cache=0 owner=0x754758 iocb=0x48cdf0(unknown) tmask=0xffffffffffffffff umask=0xfffffffffffffffe 11 : st=0x05(R:PrA W:pra) ev=0x00(heopi) [nlc] cache=0 owner=0x4e7200 iocb=0x4e7200(thread_sync_io_handler) tmask=0xffffffffffffffff umask=0xfffffffffffffffe 13 : st=0x20(R:pra W:pRa) ev=0x00(heopi) [nlc] cache=0 owner=0x1f05c30 iocb=0x4d2c80(conn_fd_handler) tmask=0x1 umask=0x0 cflg=0x80203300 fe=fe-http mux=H2 mux_ctx=0x1f055a0 14 : st=0x20(R:pra W:pRa) ev=0x00(heopi) [nlc] cache=0 owner=0x20ea060 iocb=0x4d2c80(conn_fd_handler) tmask=0x1 umask=0x0 cflg=0x80203300 fe=fe-http mux=H2 mux_ctx=0x22a1580 15 : st=0x20(R:pra W:pRa) ev=0x00(heopi) [nlc] cache=0 owner=0x1ff94d0 iocb=0x4d2c80(conn_fd_handler) tmask=0x1 umask=0x0 cflg=0x80203300 fe=fe-http mux=H2 mux_ctx=0x20fa360 16 : st=0x20(R:pra W:pRa) ev=0x00(heopi) [nlc] cache=0 owner=0x21b7a90 iocb=0x4d2c80(conn_fd_handler) tmask=0x1 umask=0x0 cflg=0x80203300 fe=fe-http mux=H2 mux_ctx=0x1fc58f0 17 : st=0x20(R:pra W:pRa) ev=0x00(heopi) [nlc] cache=0 owner=0x2012970 iocb=0x4d2c80(conn_fd_handler) tmask=0x1 umask=0x0 cflg=0x80203300 fe=fe-http mux=H2 mux_ctx=0x22a7480 18 : st=0x20(R:pra W:pRa) ev=0x00(heopi) [nlc] cache=0 owner=0x21b7e30 iocb=0x4d2c80(conn_fd_handler) tmask=0x1 umask=0x0 cflg=0x80203300 fe=fe-http mux=H2 mux_ctx=0x210b610 19 : st=0x20(R:pra W:pRa) ev=0x00(heopi) [nlc] cache=0 owner=0x1efa590 iocb=0x4d2c80(conn_fd_handler) tmask=0x1 umask=0x0 cflg=0x80203300 fe=fe-http mux=H2 mux_ctx=0x1f02610 20 : st=0x20(R:pra W:pRa) ev=0x00(heopi) [nlc] cache=0 owner=0x1fe8a90 iocb=0x4d2c80(conn_fd_handler) tmask=0x1 umask=0x0 cflg=0x80203300 fe=fe-http mux=H2 mux_ctx=0x2311bd0 21 : st=0x20(R:pra W:pRa) ev=0x00(heopi) [nlc] cache=0 owner=0x20a0f10 iocb=0x4d2c80(conn_fd_handler) tmask=0x1 umask=0x0 cflg=0x80203300 fe=fe-http mux=H2 mux_ctx=0x216ecf0 22 : st=0x20(R:pra W:pRa) ev=0x00(heopi) [nlc] cache=0 owner=0x1f02a40 iocb=0x4d2c80(conn_fd_handler) tmask=0x1 umask=0x0 cflg=0x80203300 fe=fe-http mux=H2 mux_ctx=0x20d03b0 23 : st=0x20(R:pra W:pRa) ev=0x00(heopi) [nlc] cache=0 owner=0x21b8e80 iocb=0x4d2c80(conn_fd_handler) tmask=0x1 umask=0x0 cflg=0x80203300 fe=fe-http mux=H2 mux_ctx=0x22d2ff0 24 : st=0x20(R:pra W:pRa) ev=0x00(heopi) [nlc] cache=0 owner=0x1f8f2c0 iocb=0x4d2c80(conn_fd_handler) tmask=0x1 umask=0x0 cflg=0x80203300 fe=fe-http mux=H2 mux_ctx=0x1ee8030 25 : st=0x20(R:pra W:pRa) ev=0x00(heopi) [nlc] cache=0 owner=0x1f88c90 iocb=0x4d2c80(conn_fd_handler) tmask=0x1 umask=0x0 cflg=0x80203300 fe=fe-http mux=H2 mux_ctx=0x1fa1e20 26 : st=0x20(R:pra W:pRa) ev=0x00(heopi) [nlc] cache=0 owner=0x21b8cb0 iocb=0x4d2c80(conn_fd_handler) tmask=0x1 umask=0x0 cflg=0x80203300 fe=fe-http mux=H2 mux_ctx=0x1f2d260 27 : st=0x20(R:pra W:pRa) ev=0x00(heopi) [nlc] cache=0 owner=0x20d0b40 iocb=0x4d2c80(conn_fd_handler) tmask=0x1 umask=0x0 cflg=0x80203300 fe=fe-http mux=H2 mux_ctx=0x1f04870 28 : st=0x20(R:pra W:pRa) ev=0x00(heopi) [nlc] cache=0 owner=0x20a3670 iocb=0x4d2c80(conn_fd_handler) tmask=0x1 umask=0x0 cflg=0x80203300 fe=fe-http mux=H2 mux_ctx=0x1f02790 29 : st=0x20(R:pra W:pRa) ev=0x00(heopi) [nlc] cache=0 owner=0x21639c0 iocb=0x4d2c80(conn_fd_handler) tmask=0x1 umask=0x0 cflg=0x80203300 fe=fe-http mux=H2 mux_ctx=0x1f21dd0 30 : st=0x20(R:pra W:pRa) ev=0x00(heopi) [nlc] cache=0 owner=0x217cac0 iocb=0x4d2c80(conn_fd_handler) tmask=0x1 umask=0x0 cflg=0x80203300 fe=fe-http mux=H2 mux_ctx=0x22cd210 31 : st=0x20(R:pra W:pRa) ev=0x00(heopi) [nlc] cache=0 owner=0x2163540 iocb=0x4d2c80(conn_fd_handler) tmask=0x1 umask=0x0 cflg=0x80203300 fe=fe-http mux=H2 mux_ctx=0x209b2e0 32 : st=0x20(R:pra W:pRa) ev=0x00(heopi) [nlc] cache=0 owner=0x209b750 iocb=0x4d2c80(conn_fd_handler) tmask=0x1 umask=0x0 cflg=0x80203300 fe=fe-http mux=H2 mux_ctx=0x20a02b0 33 : st=0x20(R:pra W:pRa) ev=0x00(heopi) [nlc] cache=0 owner=0x20c07c0 iocb=0x4d2c80(conn_fd_handler) tmask=0x1 umask=0x0 cflg=0x80203300 fe=fe-http mux=H2 mux_ctx=0x2328430 34 : st=0x20(R:pra W:pRa) ev=0x00(heopi) [nlc] cache=0 owner=0x2162d40 iocb=0x4d2c80(conn_fd_handler) tmask=0x1 umask=0x0 cflg=0x80203300 fe=fe-http mux=H2 mux_ctx=0x1fbe580 35 : st=0x20(R:pra W:pRa) ev=0x00(heopi) [nlc] cache=0 owner=0x1f914d0 iocb=0x4d2c80(conn_fd_handler) tmask=0x1 umask=0x0 cflg=0x80203300 fe=fe-http mux=H2 mux_ctx=0x1ed2da0 36 : st=0x20(R:pra W:pRa) ev=0x00(heopi) [nlc] cache=0 owner=0x1f05a70 iocb=0x4d2c80(conn_fd_handler) tmask=0x1 umask=0x0 cflg=0x80203300 fe=fe-http mux=H2 mux_ctx=0x20c0d80 37 : st=0x20(R:pra W:pRa) ev=0x00(heopi) [nlc] cache=0 owner=0x20e43a0 iocb=0x4d2c80(conn_fd_handler) tmask=0x1 umask=0x0 cflg=0x80203300 fe=fe-http mux=H2 mux_ctx=0x1f18200 38 : st=0x20(R:pra W:pRa) ev=0x00(heopi) [nlc] cache=0 owner=0x20a0120 iocb=0x4d2c80(conn_fd_handler) tmask=0x1 umask=0x0 cflg=0x80203300 fe=fe-http mux=H2 mux_ctx=0x20ea4e0 39 : st=0x20(R:pra W:pRa) ev=0x00(heopi) [nlc] cache=0 owner=0x2082a90 iocb=0x4d2c80(conn_fd_handler) tmask=0x1 umask=0x0 cflg=0x80203300 fe=fe-http mux=H2 mux_ctx=0x22a1e00 40 : st=0x20(R:pra W:pRa) ev=0x00(heopi) [nlc] cache=0 owner=0x21b8740 iocb=0x4d2c80(conn_fd_handler) tmask=0x1 umask=0x0 cflg=0x80203300 fe=fe-http mux=H2 mux_ctx=0x22a5ca0 41 : st=0x20(R:pra W:pRa) ev=0x00(heopi) [nlc] cache=0 owner=0x1fa7b10 iocb=0x4d2c80(conn_fd_handler) tmask=0x1 umask=0x0 cflg=0x80203300 fe=fe-http mux=H2 mux_ctx=0x230db80 42 : st=0x20(R:pra W:pRa) ev=0x00(heopi) [nlc] cache=0 owner=0x2152460 iocb=0x4d2c80(conn_fd_handler) tmask=0x1 umask=0x0 cflg=0x80203300 fe=fe-http mux=H2 mux_ctx=0x1f2bf70 43 : st=0x20(R:pra W:pRa) ev=0x00(heopi) [nlc] cache=0 owner=0x20d8110 iocb=0x4d2c80(conn_fd_handler) tmask=0x1 umask=0x0 cflg=0x80203300 fe=fe-http mux=H2 mux_ctx=0x1faa770 44 : st=0x20(R:pra W:pRa) ev=0x00(heopi) [nlc] cache=0 owner=0x1fbe0a0 iocb=0x4d2c80(conn_fd_handler) tmask=0x1 umask=0x0 cflg=0x80203300 fe=fe-http mux=H2 mux_ctx=0x22e52a0 45 : st=0x20(R:pra W:pRa) ev=0x00(heopi) [nlc] cache=0 owner=0x1f23ad0 iocb=0x4d2c80(conn_fd_handler) tmask=0x1 umask=0x0 cflg=0x80203300 fe=fe-http mux=H2 mux_ctx=0x22a2800 46 : st=0x20(R:pra W:pRa) ev=0x00(heopi) [nlc] cache=0 owner=0x20ccdd0 iocb=0x4d2c80(conn_fd_handler) tmask=0x1 umask=0x0 cflg=0x80203300 fe=fe-http mux=H2 mux_ctx=0x236d720 47 : st=0x20(R:pra W:pRa) ev=0x00(heopi) [nlc] cache=0 owner=0x2083130 iocb=0x4d2c80(conn_fd_handler) tmask=0x1 umask=0x0 cflg=0x80203300 fe=fe-http mux=H2 mux_ctx=0x236eb00 48 : st=0x20(R:pra W:pRa) ev=0x00(heopi) [nlc] cache=0 owner=0x1f90e00 iocb=0x4d2c80(conn_fd_handler) tmask=0x1 umask=0x0 cflg=0x80203300 fe=fe-http mux=H2 mux_ctx=0x2397bd0 49 : st=0x20(R:pra W:pRa) ev=0x00(heopi) [nlc] cache=0 owner=0x1f8de50 iocb=0x4d2c80(conn_fd_handler) tmask=0x1 umask=0x0 cflg=0x80203300 fe=fe-http mux=H2 mux_ctx=0x2327030 50 : st=0x20(R:pra W:pRa) ev=0x00(heopi) [nlc] cache=0 owner=0x217a360 iocb=0x4d2c80(conn_fd_handler) tmask=0x1 umask=0x0 cflg=0x80203300 fe=fe-http mux=H2 mux_ctx=0x236d8b0 51 : st=0x20(R:pra W:pRa) ev=0x00(heopi) [nlc] cache=0 owner=0x215a730 iocb=0x4d2c80(conn_fd_handler) tmask=0x1 umask=0x0 cflg=0x80203300 fe=fe-http mux=H2 mux_ctx=0x238bb70 52 : st=0x20(R:pra W:pRa) ev=0x00(heopi) [nlc] cache=0 owner=0x209fb60 iocb=0x4d2c80(conn_fd_handler) tmask=0x1 umask=0x0 cflg=0x80203300 fe=fe-http mux=H2 mux_ctx=0x20e7850 53 : st=0x25(R:PrA W:pRa) ev=0x00(heopi) [nlc] cache=0 owner=0x20a1230 iocb=0x4d2c80(conn_fd_handler) tmask=0x1 umask=0x0 cflg=0x80203306 fe=fe-http mux=H2 mux_ctx=0x25a9470 54 : st=0x20(R:pra W:pRa) ev=0x00(heopi) [nlc] cache=0 owner=0x1f90730 iocb=0x4d2c80(conn_fd_handler) tmask=0x1 umask=0x0 cflg=0x80203300 fe=fe-http mux=H2 mux_ctx=0x1ff82d0 55 : st=0x25(R:PrA W:pRa) ev=0x00(heopi) [nlc] cache=0 owner=0x24d1f10 iocb=0x4d2c80(conn_fd_handler) tmask=0x1 umask=0x0 cflg=0x80203306 fe=fe-http mux=H2 mux_ctx=0x230e930 56 : st=0x20(R:pra W:pRa) ev=0x00(heopi) [nlc] cache=0 owner=0x21647d0 iocb=0x4d2c80(conn_fd_handler) tmask=0x1 umask=0x0 cflg=0x80203300 fe=fe-http mux=H2 mux_ctx=0x22e8b60 57 : st=0x20(R:pra W:pRa) ev=0x00(heopi) [nlc] cache=0 owner=0x1f8c310 iocb=0x4d2c80(conn_fd_handler) tmask=0x1 umask=0x0 cflg=0x80203300 fe=fe-http mux=H2 mux_ctx=0x1ff1b00 58 : st=0x20(R:pra W:pRa) ev=0x00(heopi) [nlc] cache=0 owner=0x1f90060 iocb=0x4d2c80(conn_fd_handler) tmask=0x1 umask=0x0 cflg=0x80203300 fe=fe-http mux=H2 mux_ctx=0x2326220 59 : st=0x25(R:PrA W:pRa) ev=0x00(heopi) [nlc] cache=0 owner=0x1ff80d0 iocb=0x4d2c80(conn_fd_handler) tmask=0x1 umask=0x0 cflg=0x80203306 fe=fe-http mux=H2 mux_ctx=0x24df3e0 60 : st=0x25(R:PrA W:pRa) ev=0x00(heopi) [nlc] cache=0 owner=0x21b81d0 iocb=0x4d2c80(conn_fd_handler) tmask=0x1 umask=0x0 cflg=0x80203306 fe=fe-http mux=H2 mux_ctx=0x236f300 61 : st=0x25(R:PrA W:pRa) ev=0x00(heopi) [nlc] cache=0 owner=0x20f3230 iocb=0x4d2c80(conn_fd_handler) tmask=0x1 umask=0x0 cflg=0x80203306 fe=fe-http mux=PASS mux_ctx=0x2163e90 62 : st=0x20(R:pra W:pRa) ev=0x00(heopi) [nlc] cache=0 owner=0x2158150 iocb=0x4d2c80(conn_fd_handler) tmask=0x1 umask=0x0 cflg=0x80203300 fe=fe-http mux=H2 mux_ctx=0x23004c0 63 : st=0x25(R:PrA W:pRa) ev=0x00(heopi) [nlc] cache=0 owner=0x21122f0 iocb=0x4d2c80(conn_fd_handler) tmask=0x1 umask=0x0 cflg=0x80203306 fe=fe-http mux=PASS mux_ctx=0x1f88ba0 64 : st=0x25(R:PrA W:pRa) ev=0x00(heopi) [nlc] cache=0 owner=0x21b8ae0 iocb=0x4d2c80(conn_fd_handler) tmask=0x1 umask=0x0 cflg=0x80203306 fe=fe-http mux=H2 mux_ctx=0x254fd10 65 : st=0x25(R:PrA W:pRa) ev=0x00(heopi) [nlc] cache=0 owner=0x210b440 iocb=0x4d2c80(conn_fd_handler) tmask=0x1 umask=0x0 cflg=0x00202306 sv=nginx01/be-nginx01 mux=PASS mux_ctx=0x1fddaf0 66 : st=0x22(R:pRa W:pRa) ev=0x00(heopi) [Nlc] cache=0 owner=0x21b9050 iocb=0x4d2c80(conn_fd_handler) tmask=0x1 umask=0x0 cflg=0x00241300 fe=GLOBAL mux=PASS mux_ctx=0x1f92230 68 : st=0x25(R:PrA W:pRa) ev=0x00(heopi) [nlc] cache=0 owner=0x21b8910 iocb=0x4d2c80(conn_fd_handler) tmask=0x1 umask=0x0 cflg=0x80203306 fe=fe-http mux=H2 mux_ctx=0x2321d60 69 : st=0x25(R:PrA W:pRa) ev=0x00(heopi) [nlc] cache=0 owner=0x2109130 iocb=0x4d2c80(conn_fd_handler) tmask=0x1 umask=0x0 cflg=0x80203306 fe=fe-http mux=H2 mux_ctx=0x2543fc0 71 : st=0x25(R:PrA W:pRa) ev=0x00(heopi) [nlc] cache=0 owner=0x1f8d780 iocb=0x4d2c80(conn_fd_handler) tmask=0x1 umask=0x0 cflg=0x80203306 fe=fe-http mux=H2 mux_ctx=0x236fdd0
