On Thu, May 24, 2018 at 10:07:23AM +0200, William Dauchy wrote:
> On Wed, May 23, 2018 at 08:45:04PM +0200, William Dauchy wrote:
> > More details which could help understand what is going on:
> >
> > ps output:
> >
> > root 15928 0.3 0.0 255216 185268 ? Ss May21 10:11
> > /usr/sbin/haproxy -Ws -f /etc/haproxy/haproxy.cfg -p /run/haproxy.pid -sf
> > 16988 16912 6340 28271 30590 30334 -x /var/lib/haproxy/stats
> > haproxy 6340 2.0 0.0 526172 225476 ? Ssl May22 35:03 \_
> > /usr/sbin/haproxy -Ws -f /etc/haproxy/haproxy.cfg -p /run/haproxy.pid -sf
> > 6328 6315 -x /var/lib/haproxy/stats
> > haproxy 28271 1.8 0.0 528720 229508 ? Ssl May22 27:13 \_
> > /usr/sbin/haproxy -Ws -f /etc/haproxy/haproxy.cfg -p /run/haproxy.pid -sf
> > 28258 28207 28232 6340 -x /var/lib/haproxy/stats
> > haproxy 30590 265 0.0 527268 225032 ? Rsl 04:35 2188:55 \_
> > /usr/sbin/haproxy -Ws -f /etc/haproxy/haproxy.cfg -p /run/haproxy.pid -sf
> > 30578 28271 6340 -x /var/lib/haproxy/stats
> > haproxy 30334 197 0.0 526704 224544 ? Rsl 09:17 1065:59 \_
> > /usr/sbin/haproxy -Ws -f /etc/haproxy/haproxy.cfg -p /run/haproxy.pid -sf
> > 30322 30295 27095 6340 28271 30590 -x /var/lib/haproxy/stats
> > haproxy 16912 1.7 0.0 527544 216552 ? Ssl 18:14 0:03 \_
> > /usr/sbin/haproxy -Ws -f /etc/haproxy/haproxy.cfg -p /run/haproxy.pid -sf
> > 16899 28271 30590 30334 6340 -x /var/lib/haproxy/stats
> > haproxy 17001 2.2 0.0 528392 214656 ? Ssl 18:17 0:00 \_
> > /usr/sbin/haproxy -Ws -f /etc/haproxy/haproxy.cfg -p /run/haproxy.pid -sf
> > 16988 16912 6340 28271 30590 30334 -x /var/lib/haproxy/stats
> >
> >
> > lsof output:
> >
> > haproxy 6340 haproxy 5u unix 0xffff883feec97000 0t0
> > 679289634 /var/lib/haproxy/stats.15928.tmp
> > haproxy 6340 6341 haproxy 5u unix 0xffff883feec97000 0t0
> > 679289634 /var/lib/haproxy/stats.15928.tmp
> > haproxy 6340 6342 haproxy 5u unix 0xffff883feec97000 0t0
> > 679289634 /var/lib/haproxy/stats.15928.tmp
> > haproxy 6340 6343 haproxy 5u unix 0xffff883feec97000 0t0
> > 679289634 /var/lib/haproxy/stats.15928.tmp
> > haproxy 17020 haproxy 5u unix 0xffff883feec97000 0t0
> > 679289634 /var/lib/haproxy/stats.15928.tmp
> > haproxy 17020 17021 haproxy 5u unix 0xffff883feec97000 0t0
> > 679289634 /var/lib/haproxy/stats.15928.tmp
> > haproxy 17020 17022 haproxy 5u unix 0xffff883feec97000 0t0
> > 679289634 /var/lib/haproxy/stats.15928.tmp
> > haproxy 17020 17023 haproxy 5u unix 0xffff883feec97000 0t0
> > 679289634 /var/lib/haproxy/stats.15928.tmp
> > haproxy 28271 haproxy 5u unix 0xffff883feec97000 0t0
> > 679289634 /var/lib/haproxy/stats.15928.tmp
> > haproxy 28271 28272 haproxy 5u unix 0xffff883feec97000 0t0
> > 679289634 /var/lib/haproxy/stats.15928.tmp
> > haproxy 28271 28273 haproxy 5u unix 0xffff883feec97000 0t0
> > 679289634 /var/lib/haproxy/stats.15928.tmp
> > haproxy 28271 28274 haproxy 5u unix 0xffff883feec97000 0t0
> > 679289634 /var/lib/haproxy/stats.15928.tmp
> >
> > (So on unhealthy nodes, I find old processes which are still linked to
> > the socket.)
> >
> > The provisioning part is also seeing data which are supposed to be
> > already updated through the runtime API. I suspect I am getting old
> > data when connecting to the unix socket. The later being still attached
> > to an old process?
> > Indeed, if I try
> > for i in {1..500}; do sudo echo "show info" | sudo socat stdio
> > /var/lib/haproxy/stats | grep Pid; done
> >
> > I get "Pid: 17001" most of the time, which is the last process
> > but I sometimes get: "Pid: 28271"(!) which is a > 24 hours old
> > process.
> >
> > Is there something we are doing wrongly?
>
> After some more testing, I don't have this issue using haproxy v1.8.8
> (rollbacked for > 12 hours). I hope I don't speak too fast.
>
Hi,
I managed to reproduce something similar with the 1.8.8 version. It looks like
letting a socat connected to the socket helps.
I'm looking into the code to see what's happening.
--
William Lallemand
