Hi Willy, I tried manually sending SIGUSR1 and SIGTTOU as you suggested but I had mixed results. Sometimes the procs would do what I expected and sometimes they wouldn't.
> Looks similar indeed. RHEL has selinux enabled by default I believe, I > don't know if that could prevent haproxy from sending a signal to > another process. Maybe you can try to stop it (if enabled at all). selinux is disabled on these hosts. I tried with an infinite while loop like in the github post but I can't force it to happen. I tried rapid fire, with a 1 second pause, and with a small random rewrite to haproxy.cfg before the reload (to mimic a change written out by consul-template) and no matter how long I let the while loop run, the previous process always cleanly exists - with and without traffic passing through it (stats page continually refreshing) while :; do service haproxy reload; done while :; do service haproxy reload; sleep 1; done while :; do /etc/haproxy/test.sh; service haproxy reload; sleep 1; done test.sh: #!/bin/bash sed "s/<<testrand>>/$RANDOM/" /etc/haproxy/haproxy.cfg.tmpl > /etc/haproxy/haproxy.cfg I'm going to take a closer look at the source for consul-template ( https://github.com/hashicorp/consul-template) to see if it manipulates the command provided in the template configuration or if it blindly passes it to the shell. This is what the config looks like for consul-template: template { source = "/etc/consul-template/templates/haproxy.cfg.ctmpl" destination = "/etc/haproxy/haproxy.cfg" command = "/sbin/service haproxy reload || true" } I will let you know if I find anything of interest. Regards, Chris On Fri, Oct 30, 2015 at 1:20 PM, Willy Tarreau <[email protected]> wrote: > On Fri, Oct 30, 2015 at 12:23:57PM -0400, Chris Riley wrote: > > Hi Willy, > > > > Thanks for your quick reply. > > > > > It should work better but will very likely hide the root cause. I > suspect > > > you'll find two processes running after a reload because the old one > > > doesn't stop then. > > > > Yep, that's exactly what I'm seeing with 3.10. I've got a bunch of > haproxy > > processes stacked up, all with an -sf flag and each being passed the PID > of > > the previous process. The only one not in the process list is the initial > > haproxy instance created when 'service haproxy start' was first run. > > > > lb-01 ~ [qa] # ps ax | grep hap > > 2834 ? Ss 0:00 /usr/sbin/haproxy -D -f > /etc/haproxy/haproxy.cfg > > -p /var/run/haproxy.pid -sf 2822 > > 2871 ? Ss 0:00 /usr/sbin/haproxy -D -f > /etc/haproxy/haproxy.cfg > > -p /var/run/haproxy.pid -sf 2859 > > 2883 ? Ss 0:00 /usr/sbin/haproxy -D -f > /etc/haproxy/haproxy.cfg > > -p /var/run/haproxy.pid -sf 2871 > > 2910 ? Ss 0:00 /usr/sbin/haproxy -D -f > /etc/haproxy/haproxy.cfg > > -p /var/run/haproxy.pid -sf 2896 > > 2922 ? Ss 0:00 /usr/sbin/haproxy -D -f > /etc/haproxy/haproxy.cfg > > -p /var/run/haproxy.pid -sf 2910 > > 2947 ? Ss 0:00 /usr/sbin/haproxy -D -f > /etc/haproxy/haproxy.cfg > > -p /var/run/haproxy.pid -sf 2934 > > 2959 ? Ss 0:00 /usr/sbin/haproxy -D -f > /etc/haproxy/haproxy.cfg > > -p /var/run/haproxy.pid -sf 2947 > > 2962 pts/1 S+ 0:00 grep --colour=auto hap > > These ones don't all match, in fact only the last one is fine. That's > strange. > > > > That would confirm the possibility that the signal is not sent at all, > > > or at least not to the right process. Could you check the exact command > > > that is started, to ensure the pids are correct (or present at all) ? > > > Can you also try by hand to first send SIGUSR1 to the old process, > > > then perform the reload, then send SIGTTOU by hand to the old one ? > > > If it works, it would confirm an issue with the ability to send a > > > signal to the old process from the new one. > > > > 'service haproxy start' invokes this: > > > > daemon $exec -D -f /etc/$prog/$prog.cfg -p /var/run/$prog.pid > > > > which produces: > > > > usr/sbin/haproxy -D -f /etc/haproxy/haproxy.cfg -p /var/run/haproxy.pid > > > > ps ax shows: > > > > 2822 ? Ss 0:00 /usr/sbin/haproxy -D -f > /etc/haproxy/haproxy.cfg > > -p /var/run/haproxy.pid > > > > cat /var/run/haproxy.pid shows: > > 2822 > > > > 'service haproxy reload' invokes this: > > > > $exec -D -f /etc/$prog/$prog.cfg -p /var/run/$prog.pid -sf $(cat > > /var/run/$prog.pid) > > OK so there is no reason for it not to work. > > > which produces: > > > > 2834 ? Ss 0:00 /usr/sbin/haproxy -D -f > /etc/haproxy/haproxy.cfg > > -p /var/run/haproxy.pid -sf 2822 > > > > cat /var/run/haproxy.pid then shows: > > 2834 > > > > I'll try manually sending SIGUSR1 and SIGTTOU as you suggested and see > if I > > can determine what's happening. > > Yes that will definitely help. > > > > Any chance that this issue is related to or that same as this one? > > > > https://github.com/haproxy/haproxy/issues/48 > > Looks similar indeed. RHEL has selinux enabled by default I believe, I > don't know if that could prevent haproxy from sending a signal to another > process. Maybe you can try to stop it (if enabled at all). > > Regards, > Willy > >
