Re: haproxy 2.0.14 failing to bind peer sockets
On Mon, Apr 06, 2020 at 03:26:22PM -0700, James Brown wrote: > It seems to also fail on cold start when there's a `peers` block that is > unused. It's very mysterious! > > We aren't actually using the peers for anything in this config any more, so > I'm going to strip it out for now and proceed with testing 2.0.14. > Hello James, What the patch you reverted does, is binding on the peers socket even if it's not used in your configuration. Which was not the case before. So it's possible that another daemon is using this port, but you never had this problem because haproxy was not trying to bind with it. The error you observe is common when you try to launch HAProxy with a port which is already used. Please check with netstat or ss if the port used in your peer configuration is already in use. Regards, -- William Lallemand
Re: haproxy 2.0.14 failing to bind peer sockets
It seems to also fail on cold start when there's a `peers` block that is unused. It's very mysterious! We aren't actually using the peers for anything in this config any more, so I'm going to strip it out for now and proceed with testing 2.0.14. On Mon, Apr 6, 2020 at 1:56 PM Willy Tarreau wrote: > On Mon, Apr 06, 2020 at 01:50:56PM -0700, James Brown wrote: > > I actually messed up testing last week; reverting Tim's commit appears to > > fix it. > > OK that's very useful, thanks! However you didn't respond to my other > question: > > > > > James, just to confirm, does it fail to start from a cold start or > only > > > > on reloads ? > > Willy > -- James Brown Engineer
Re: haproxy 2.0.14 failing to bind peer sockets
On Mon, Apr 06, 2020 at 01:50:56PM -0700, James Brown wrote: > I actually messed up testing last week; reverting Tim's commit appears to > fix it. OK that's very useful, thanks! However you didn't respond to my other question: > > > James, just to confirm, does it fail to start from a cold start or only > > > on reloads ? Willy
Re: haproxy 2.0.14 failing to bind peer sockets
I actually messed up testing last week; reverting Tim's commit appears to fix it. On Fri, Apr 3, 2020 at 5:41 AM Willy Tarreau wrote: > On Fri, Apr 03, 2020 at 02:27:05PM +0200, Willy Tarreau wrote: > > On Thu, Apr 02, 2020 at 12:32:32PM -0700, James Brown wrote: > > > I reverted that commit, but it doesn't appear to have fixed the issue. > > > > > > I also tried adding a stick-table using this peers group to my config > (this > > > test cluster didn't actually have any stick-tables), but it still > fails at > > > startup with the same error. > > > > James, just to confirm, does it fail to start from a cold start or only > > on reloads ? > > I'm trying with this config and this command: > >global > stats socket /tmp/sock1 mode 666 level admin expose-fd listeners > stats timeout 1d > >peers p > peer peer1 127.0.0.1:8521 > peer peer2 127.0.0.1:8522 > >listen l > mode http > bind 127.0.0.1:2501 > timeout client 10s > timeout server 10s > timeout connect 10s > stick-table size 200 expire 10s type ip peers p store server_id > stick on src > server s 127.0.0.1:8000 > >$ ./haproxy -D -L peer1 -f peers.cfg -p /tmp/haproxy.pid >$ ./haproxy -D -L peer1 -f peers.cfg -p /tmp/haproxy.pid -sf $(pidof > haproxy) -x /tmp/sock1 >$ ./haproxy -D -L peer1 -f peers.cfg -p /tmp/haproxy.pid -sf $(pidof > haproxy) -x /tmp/sock1 >$ ./haproxy -D -L peer1 -f peers.cfg -p /tmp/haproxy.pid -sf $(pidof > haproxy) -x /tmp/sock1 > > For now I can't figure how to reproduce it :-/ If you manage to modify > this config to trigger the issue that would be great! > > Willy > -- James Brown Engineer
Re: haproxy 2.0.14 failing to bind peer sockets
On Fri, Apr 03, 2020 at 02:27:05PM +0200, Willy Tarreau wrote: > On Thu, Apr 02, 2020 at 12:32:32PM -0700, James Brown wrote: > > I reverted that commit, but it doesn't appear to have fixed the issue. > > > > I also tried adding a stick-table using this peers group to my config (this > > test cluster didn't actually have any stick-tables), but it still fails at > > startup with the same error. > > James, just to confirm, does it fail to start from a cold start or only > on reloads ? I'm trying with this config and this command: global stats socket /tmp/sock1 mode 666 level admin expose-fd listeners stats timeout 1d peers p peer peer1 127.0.0.1:8521 peer peer2 127.0.0.1:8522 listen l mode http bind 127.0.0.1:2501 timeout client 10s timeout server 10s timeout connect 10s stick-table size 200 expire 10s type ip peers p store server_id stick on src server s 127.0.0.1:8000 $ ./haproxy -D -L peer1 -f peers.cfg -p /tmp/haproxy.pid $ ./haproxy -D -L peer1 -f peers.cfg -p /tmp/haproxy.pid -sf $(pidof haproxy) -x /tmp/sock1 $ ./haproxy -D -L peer1 -f peers.cfg -p /tmp/haproxy.pid -sf $(pidof haproxy) -x /tmp/sock1 $ ./haproxy -D -L peer1 -f peers.cfg -p /tmp/haproxy.pid -sf $(pidof haproxy) -x /tmp/sock1 For now I can't figure how to reproduce it :-/ If you manage to modify this config to trigger the issue that would be great! Willy
Re: haproxy 2.0.14 failing to bind peer sockets
On Thu, Apr 02, 2020 at 12:32:32PM -0700, James Brown wrote: > I reverted that commit, but it doesn't appear to have fixed the issue. > > I also tried adding a stick-table using this peers group to my config (this > test cluster didn't actually have any stick-tables), but it still fails at > startup with the same error. James, just to confirm, does it fail to start from a cold start or only on reloads ? Willy
Re: haproxy 2.0.14 failing to bind peer sockets
I reverted that commit, but it doesn't appear to have fixed the issue. I also tried adding a stick-table using this peers group to my config (this test cluster didn't actually have any stick-tables), but it still fails at startup with the same error. On Thu, Apr 2, 2020 at 11:28 AM Tim Düsterhus wrote: > James, > > Am 02.04.20 um 19:53 schrieb James Brown: > > I'm upgrading one of our test clusters from 2.0.13 to 2.0.14 and our > > regular graceful-restart process is failing with: > > > > [ALERT] 092/174647 (114374) : [/usr/sbin/haproxy.main()] Some protocols > > failed to start their listeners! Exiting. > > I suppose this commit might be at fault here: > > https://github.com/haproxy/haproxy/commit/a2cfd7e356f4d744294b510b05d88bf58304db25 > > Try reverting it to see whether it fixes the issue. > > Best regards > Tim Düsterhus > -- James Brown Engineer
Re: haproxy 2.0.14 failing to bind peer sockets
James, Am 02.04.20 um 19:53 schrieb James Brown: > I'm upgrading one of our test clusters from 2.0.13 to 2.0.14 and our > regular graceful-restart process is failing with: > > [ALERT] 092/174647 (114374) : [/usr/sbin/haproxy.main()] Some protocols > failed to start their listeners! Exiting. I suppose this commit might be at fault here: https://github.com/haproxy/haproxy/commit/a2cfd7e356f4d744294b510b05d88bf58304db25 Try reverting it to see whether it fixes the issue. Best regards Tim Düsterhus
haproxy 2.0.14 failing to bind peer sockets
I'm upgrading one of our test clusters from 2.0.13 to 2.0.14 and our regular graceful-restart process is failing with: [ALERT] 092/174647 (114374) : [/usr/sbin/haproxy.main()] Some protocols failed to start their listeners! Exiting. Looking at strace, it looks like the bind(2) call for the peer socket is failing. Did something change about the order in which peer sockets are bound? Our peers block is pretty straightforward and hasn't changed in several years. peers lb peer devlb1west 10.132.46.130:7778 peer devlb2west 10.132.37.135:7778 Our graceful restart command looks like /usr/sbin/haproxy -f /path/to/haproxy.config -p /home/srvelb/run/haproxy.pid -sf 70409 -x /path/to/admin/mode/socket and also hasn't changed since the addition of domain-socket FD passing in 1.8. I notice a bunch of peer-related commits got pulled into 2.0.14... Anyone else seen this? -- James Brown Engineer
