Hi Willy, Op 9-11-2017 om 5:45 schreef Willy Tarreau:
Hi Pieter,On Thu, Nov 09, 2017 at 02:28:46AM +0100, PiBa-NL wrote:Actually haproxy has been running for a few weeks with 100% and i didnt notice.. it does keep working it seems.. Anyhow thought i would try and capture the next event if it would happen again. It did after a few hours.. After the truss output below the last line keeps repeating fast lots and lots of times. kevent(0,0x0,0,{ },7,{ 1.000000000 }) = 0 (0x0) kevent(0,0x0,0,{ },7,{ 1.000000000 }) = 0 (0x0) kevent(0,0x0,0,{ },7,{ 1.000000000 }) = 0 (0x0) kevent(0,0x0,0,{ 1,EVFILT_READ,EV_EOF,0x0,0x0,0x0 },7,{ 0.994000000 }) = 1 (0x1) recvfrom(1,0x8024ed972,16290,0,NULL,0x0) = 0 (0x0) kevent(0,{ 1,EVFILT_READ,EV_DELETE,0x0,0x0,0x0 },1,0x0,0,0x0) = 0 (0x0) kevent(0,0x0,0,{ },7,{ 0.000000000 }) = 0 (0x0) kevent(0,0x0,0,{ },7,{ 0.000000000 }) = 0 (0x0) kevent(0,0x0,0,{ },7,{ 0.000000000 }) = 0 (0x0) kevent(0,0x0,0,{ },7,{ 0.000000000 }) = 0 (0x0) kevent(0,0x0,0,{ },7,{ 0.000000000 }) = 0 (0x0)We had something similar on Linux in relation with TCP splicing and the fd cache, for which a fix was emitted. But yesterday Christopher explained me that the fix has an impact on the way applets are scheduled in 1.8, so actually it could mean that the initial bug might possibly cover a larger scope than splicing only, and that recv+send might also be affected. If you're interested in testing, the commit in 1.7 is c040c1f ("BUG/MAJOR: stream-int: don't re-arm recv if send fails") and is present in the latest snapshot (we really need to emit 1.7.10 BTW). I'd be curious to know if it fixes it or not. At least it will tell us if that's related to this fd cache thing or to something completely different such as Lua. I also need to check with Thierry if we could find a way to add some stats about the time spent in Lua to "show info" to help debugging such cases where Lua is involved. By the way, thanks for your dump, we'll check the sessions' statuses. There are not that many, and maybe it will give us a useful indication! Cheers, Willy
Thanks for your time, i didn't think the 'splice' problem mentioned on mailing-list would be relevant for my case so i didn't see a need to try latest snapshot. Couldn't find much other recent cpu issues there. But ill try and compile haproxy 1.7 latest snapshot or perhaps just 1.7.9 with this extra patch and see if it keeps running with low cpu usage for a few days.. I have not compiled haproxy for a while, ill see what works the easiest for me, assuming can make it work build on a separate FreeBSD machine and packaged/copied to the actual 'problem machine' that doesn't have compilation tools on it.. hopefully my build binary will be 'compatible'..
Will report back in a few day's.. Thanks, PiBa-NL / Pieter
