I think you're just attempting to reload haproxy too fast. There are race conditions in getting the list of running pids and passing them into haproxy -- that list changes before the next proxy is started.
Your test case is reloading haproxy about 10 times per second. There are several reports on this list about that being an issue. AFAIK, the issue isn't with the proxy code itself, but just with the way that the list of running processes are collected and signaled. Anyway, for your case, adding a -wait=5s option to consul-template makes the problem go away for me. Maybe you can get away with reloading every 1 second, but multiple times per second is not likely to work reliably. -Bryan On Fri, Feb 5, 2016 at 6:11 PM, Maciej Katafiasz <mkatafi...@purestorage.com > wrote: > I will try with explicit -reap, but we have actually investigated that > as one possible cause, and consul-template reports automatically > engaging reap mode as it sees itself as PID 1, and then reaps the > processes that actually exit correctly. The problem is that old > haproxy processes continue to run and process requests when they're > not supposed to, which is different to dead, but unreaped processes > that you'd see with faulty PID 1. As I said, most of the "aha!" > moments we've had with this problem turned out to be spurious > correlations and the issue ultimately came back. It's just that > sometimes, for no articulable reason, it works fine, and then the next > time it doesn't. > > Cheers, > > On 5 February 2016 at 16:59, Cyril Bonté <cyril.bo...@free.fr> wrote: > > Hi, > > > > > > Le 06/02/2016 01:03, Maciej Katafiasz a écrit : > >> > >> On 5 February 2016 at 16:02, Maciej Katafiasz > >> <mkatafi...@purestorage.com> wrote: > >>> > >>> Link to the tarball: > >>> https://purestorage.app.box.com/s/nnzqueais46plzd9xfisnmkeab7j9s0y > >>> > >>> I will be sending it as an attachment in a separate mail as a followup > >>> to this one, in case the mailing list software scrubs attachments > >>> and/or considers them spam. > >> > >> > >> And here's the tarball as an attachment > > > > > > This looks to be concern more consul inside a docker container than a > > haproxy itself. But this may explain some similar reports made by other > > users recently. > > > > Use the -reap option with consul-template, and haproxy will reload > > correctly. > > > > As quick example : > > $ docker run --name=haproxy -d --net=host haproxy-bugtest consul-template > > -config=/tmp/haproxy.ctmpl.cfg -log-level=debug -reap > > > > $ docker exec -ti haproxy sh > > # ps aux > > ... > > 16 root 0:00 haproxy -f /etc/haproxy/haproxy.cfg -d -p > > /var/run/haproxy.pid -sf > > ... > > > > # haproxy -f /etc/haproxy/haproxy.cfg -d -p /var/run/haproxy.pid -sf 16 > &> > > /tmp/debug.log& > > > > # ps aux > > ... > > 27 root 0:00 haproxy -f /etc/haproxy/haproxy.cfg -d -p > > /var/run/haproxy.pid -sf 16 > > ... > > => No more PID 16 > > > > > > > > > > -- > > Cyril Bonté > >
