Hi Sander,
On Mon, Jul 08, 2019 at 02:44:44PM +0200, Sander Klein wrote:
> Hi,
>
> I'm having an issue with HAProxy causing CPU spikes with certain traffic.
We've actually fixed quite a number of issues causing this over the last
few years, though most of them are already addressed by the versions you're
running.
> We have a client who is downloading lots of URL's during the night. When the
> download starts there is not much other traffic going on and there doesn't
> seem to be any problem. But, when the morning comes, 'normal' traffic starts
> hitting HAProxy and every 10 seconds or so, HAProxy starts eating 100% of
> CPU while network traffic drops. When HAProxy stops eating CPU after 10
> seconds, network traffic rises again. When the crawler is finished
> everything returns to normal. So it looks like some kind of mix of traffic
> which causes it.
What's you CPU like between the peaks ? 1%, 10%, 50% ? Just to get a rough
estimate of whether it's something reaching a critical point or if it's
something doing its mess alone in its corner.
> I've tested it with HAProxy 1.8.20, 1.9.8 (which I am running by default)
> and 2.0.1. They all show the same behaviour. I also tried with 2 different
> kernels to see if anything happens there. With kernel 4.9 top show HAProxy
> using 100% CPU where 50% is user and 50% is system. With kernel 4.19 I see
> 100% CPU usage with 70% user and 50% system.
In fact once something stats to loop, all calls are so short that it's very
difficult for the system to measure an accurate time spent in user/sys, so
I am not surprised that it changes with the kernel.
> I also tried with disabling H2, splicing, and some regexes I use. Even tried
> new hardware, and moved it to a VM just to see if I could find any
> difference, but none...
Are you using threads ? I'm asking because I'm currently working on an
issue which I found could cause exactly this behaviour. I'm fairly certain
we've met it in the past without being able to attribute it to exactly
this.
> Does anyone have a good idea how to troubleshoot this any further?
If you're using threads, attaching gdb to the process and issuing "info
threads" will tell us where they are. If many of them are in
fd_update_events() or fd_may_recv(), you're likely on the one I've been
working on.
Other possibilities (due to the regularity of your observation) are :
- timeouts (check in your conf if a 10s timeout appears somewhere,
maybe it triggers and is improperly caught)
- health checks (maybe you have 10s checks, or 2s checks with 4
retries or I don't know what, which causes a special event to
occur after 10s)
In any case you're clearly facing a bug, but it's always difficult to
tell.
It could be useful to issue "show activity" twice 1 second apart when
this happens, and maybe even "show fd" and "show sess all" if you don't
have too many connections.
Thanks,
Willy
