Hello, Willy. As you said, take a look :
getsockopt(0x12e,0xffff,0x1007,0x7fffffffdb94,0x7fffffffdb90,0x0) = 0 (0x0) sendto(302,"\^D\0\^V0\0\0^z\M-L-\a\0d8\0\0"...,926,0x80,NULL,0x0) = 926 (0x39e) recvfrom(682,"\^S\0W0\0\0\M-,\^?\M-L-\^P\0\^E@"...,8030,0x0,NULL,0x0) = 988 (0x3dc) recvfrom(682,0x801f3545c,7042,0x0,0x0,0x0) ERR#35 'Resource temporarily unavailable' getsockopt(0x2a9,0xffff,0x1007,0x7fffffffdb94,0x7fffffffdb90,0x0) = 0 (0x0) sendto(681,"\^S\0W0\0\0\M-,\^?\M-L-\^P\0\^E@"...,988,0x80,NULL,0x0) = 988 (0x3dc) recvfrom(1428,"\^N\0!\M-0\0\0\M-\\M^_\M-H-\^AoU"...,8030,0x0,NULL,0x0) = 444 (0x1bc) recvfrom(1428,0x8011b523c,7586,0x0,0x0,0x0) ERR#35 'Resource temporarily unavailable' getsockopt(0x593,0xffff,0x1007,0x7fffffffdb94,0x7fffffffdb90,0x0) = 0 (0x0) sendto(1427,"\^N\0!\M-0\0\0\M-\\M^_\M-H-\^AoU"...,444,0x80,NULL,0x0) = 444 (0x1bc) recvfrom(201,"\b\0\\0\0\0\M-=\M-]\M-G-\^O\0\0"...,8030,0x0,NULL,0x0) = 2627 (0xa43) recvfrom(201,0x800ec5ac3,5403,0x0,0x0,0x0) ERR#35 'Resource temporarily unavailable' getsockopt(0xbf,0xffff,0x1007,0x7fffffffdb94,0x7fffffffdb90,0x0) = 0 (0x0) sendto(191,"\b\0\\0\0\0\M-=\M-]\M-G-\^O\0\0"...,2627,0x80,NULL,0x0) = 2627 (0xa43) recvfrom(888,"\^S\0W0\0\0\M-,\^?\M-L-\^P\0\^E@"...,8030,0x0,NULL,0x0) = 1226 (0x4ca) recvfrom(888,0x801ee354a,6804,0x0,0x0,0x0) ERR#35 'Resource temporarily unavailable' getsockopt(0x377,0xffff,0x1007,0x7fffffffdb94,0x7fffffffdb90,0x0) = 0 (0x0) sendto(887,"\^S\0W0\0\0\M-,\^?\M-L-\^P\0\^E@"...,1226,0x80,NULL,0x0) = 1226 (0x4ca) recvfrom(674,"\f\0\M-=\M-0\0\0\M^K}\M-#-d\r\0"...,8030,0x0,NULL,0x0) = 982 (0x3d6) recvfrom(674,0x800f6f456,7048,0x0,0x0,0x0) ERR#35 'Resource temporarily unavailable' getsockopt(0x2a1,0xffff,0x1007,0x7fffffffdb94,0x7fffffffdb90,0x0) = 0 (0x0) sendto(673,"\f\0\M-=\M-0\0\0\M^K}\M-#-d\r\0"...,982,0x80,NULL,0x0) = 982 (0x3d6) recvfrom(1032,"\^S\0W0\0\0\M-,\^?\M-L-\^P\0\^E@"...,8030,0x0,NULL,0x0) = 1205 (0x4b5) recvfrom(1032,0x801ddb535,6825,0x0,0x0,0x0) ERR#35 'Resource temporarily unavailable' getsockopt(0x407,0xffff,0x1007,0x7fffffffdb94,0x7fffffffdb90,0x0) = 0 (0x0) sendto(1031,"\^S\0W0\0\0\M-,\^?\M-L-\^P\0\^E@"...,1205,0x80,NULL,0x0) = 1205 (0x4b5) recvfrom(1339,"\v\0tpDa\^A\^DV \0\0\^A\M^R\M^K"...,8030,0x0,NULL,0x0) = 68 (0x44) recvfrom(1339,0x8011790c4,7962,0x0,0x0,0x0) ERR#35 'Resource temporarily unavailable' getsockopt(0x53c,0xffff,0x1007,0x7fffffffdb94,0x7fffffffdb90,0x0) = 0 (0x0) sendto(1340,"\v\0tpDa\^A\^DV \0\0\^A\M^R\M^K"...,68,0x80,NULL,0x0) = 68 (0x44) recvfrom(913,"\v\0tpj\M-h\^A\^D\M-Q\^]\0\0\^A"...,8030,0x0,NULL,0x0) = 108 (0x6c) recvfrom(913,0x8019090ec,7922,0x0,0x0,0x0) ERR#35 'Resource temporarily unavailable' getsockopt(0x392,0xffff,0x1007,0x7fffffffdb94,0x7fffffffdb90,0x0) = 0 (0x0) sendto(914,"\v\0tpj\M-h\^A\^D\M-Q\^]\0\0\^A"...,108,0x80,NULL,0x0) = 108 (0x6c) recvfrom(166,"\^D\0\^V0\0\0\M-$\M^@\M-L-\^T\0p"...,8030,0x0,NULL,0x0) = 643 (0x283) recvfrom(166,0x800f13303,7387,0x0,0x0,0x0) ERR#35 'Resource temporarily unavailable' So yes, a lot of recv/send calls as you said before. -----Mensagem original----- De: Willy Tarreau [mailto:w...@1wt.eu] Enviada em: segunda-feira, 28 de outubro de 2013 03:37 Para: Fred Pedrisa Cc: 'Lukas Tribus'; haproxy@formilux.org Assunto: Re: RES: RES: RES: RES: RES: High CPU Usage (HaProxy) Hi Fred, On Mon, Oct 21, 2013 at 08:41:16PM -0200, Fred Pedrisa wrote: > Hello, > > Ok. > > This is the top output : > > 2748 root 1 87 0 30396K 21656K CPU8 8 28.0H 49.17% haproxy > 2726 root 1 45 0 38588K 32128K CPU24 16 21.1H 33.79% haproxy > 2718 root 1 39 0 26300K 17464K kqread 28 807:21 29.98% haproxy > 2752 root 1 38 0 30396K 21748K kqread 30 859:13 25.39% haproxy > 2738 root 1 32 0 22204K 14896K kqread 11 796:36 20.65% haproxy > 2740 root 1 31 0 34492K 27404K kqread 10 451:19 18.26% haproxy > 2780 root 1 31 0 18108K 9416K kqread 31 568:38 17.77% haproxy > 2732 root 1 29 0 34492K 27840K kqread 9 405:50 16.16% haproxy > 2730 root 1 28 0 18108K 10868K kqread 15 463:21 15.38% haproxy > 2764 root 1 29 0 18108K 10752K CPU15 15 441:34 14.60% haproxy > 2760 root 1 27 0 18108K 11620K kqread 30 353:48 12.89% haproxy > 2778 root 1 26 0 14012K 8360K kqread 29 407:07 12.16% haproxy > 2756 root 1 26 0 34492K 26280K kqread 8 502:13 9.57% haproxy > 2746 root 1 26 0 22204K 13036K kqread 29 350:32 9.57% haproxy > 47408 root 1 25 0 158M 103M kqread 11 434:37 9.08% haproxy > 2734 root 1 23 0 22204K 13704K kqread 15 384:14 6.69% haproxy > 2722 root 1 23 0 14012K 5052K kqread 10 203:38 6.30% haproxy > 2782 root 1 22 0 14012K 6352K kqread 13 208:07 4.98% haproxy > 2744 root 1 21 0 18108K 12496K kqread 28 170:59 3.27% haproxy > 2758 root 1 21 0 14012K 8320K kqread 29 71:17 2.69% haproxy > 2768 root 1 20 0 14012K 5700K kqread 28 53:16 1.46% haproxy > 2766 root 1 21 0 14012K 4868K kqread 21 88:39 1.27% haproxy > 2724 root 1 20 0 14012K 7136K kqread 14 89:32 1.17% haproxy > 2728 root 1 20 0 14012K 6520K kqread 30 65:21 1.17% haproxy > 2716 root 1 20 0 14012K 5216K kqread 28 67:38 0.98% haproxy > 2762 root 1 20 0 9916K 3936K kqread 30 39:16 0.68% haproxy > 2720 root 1 20 0 14012K 8564K kqread 23 104:37 0.39% haproxy > 2754 root 1 20 0 14012K 6312K kqread 22 80:37 0.39% haproxy > 2736 root 1 20 0 14012K 5884K kqread 25 59:06 0.20% haproxy > 2772 root 1 20 0 14012K 6984K kqread 10 73:54 0.10% haproxy > 2770 root 1 20 0 34492K 25516K kqread 31 111:38 0.00% haproxy > > Right now, the load is around 12.45, sometimes going up to 16.00 +/- I suspect something different. What type of protocol are you relaying ? Very often, people working in pure TCP mode transfer a lot of very small packets. And if you have 300 Mbps with many smal packets, it can mean a lot of wakeups/sleep cycles with a very high syscall rate. You could check using "strace -c" on one of the highly loaded processes : strace -c -p 1248 Type Ctrl-C after one second, and check the numbers. I'd bet that you'll see a lot of send/recv calls. How is the user vs system CPU usage ? If you're seeing a lot of user time, you may want to give a try to 1.5-dev19, it avoids calling process_session() as much as possible, saving a lot of CPU cycles in user space. If your CPU usage is mostly system, then it means that only tuning the system will help. Regards, Willy
