HI James Thank you for your reply.
I do not see how the old haproxy being on a separate PID could do anything with a socket created by a new PID. ​Do you bring up your new instance with real servers in a maintenance state? this seems to be required to do a correct handover before making them live and active/ready to handle connections. Also there is a SYN_BLOCK firewall rule required during the reload? I ask because we have had no reports of such a race condition. ​ Regards Andrew Smalley Loadbalancer.org Ltd. On 12 April 2017 at 23:34, James Brown <jbr...@easypost.com> wrote: > Hi Andrew: > > Thanks for you feedback, but I'm describing a very specific bug wherein > the old haproxy will unlink the new haproxy's bound unix domain socket upon > reload due to a race condition in the domain socket cleanup code if a > listen overflow occurs while the graceful is in process. > > On Wed, Apr 12, 2017 at 11:39 AM, Andrew Smalley < > asmal...@loadbalancer.org> wrote: > >> HI James >> >> When you do a graceful reload of haproxy this is what happens. >> >> 1. the old process will accept no more connections and the stats page is >> stopped and so is the socket >> 2. a new haproxy instance is started where new clients get connected to, >> and this has the live socket >> 3. when the old haproxy instance has no more clients left it dies >> silently leaving all the clients on the new haproxy instance. >> >> This is expected behavior as you want the first haproxy to die when the >> last client leaves. >> >> >> Regards >> >> Andrew Smalley >> >> Loadbalancer.org Ltd. >> >> >> >> On 12 April 2017 at 19:32, James Brown <jbr...@easypost.com> wrote: >> >>> This just hit us again on a different set of load balancers... if >>> there's a listen socket overflow on a domain socket during graceful, >>> haproxy completely deletes the domain socket and becomes inaccessible. >>> >>> On Tue, Feb 21, 2017 at 6:47 PM, James Brown <jbr...@easypost.com> >>> wrote: >>> >>>> Under load, we're sometimes seeing a situation where HAProxy will >>>> completely delete a bound unix domain socket after a reload. >>>> >>>> The "bad flow" looks something like the following: >>>> >>>> >>>> - haproxy is running on pid A, bound to /var/run/domain.sock (via a >>>> bind line in a frontend) >>>> - we run `haproxy -sf A`, which starts a new haproxy on pid B >>>> - pid B binds to /var/run/domain.sock.B >>>> - pid B moves /var/run/domain.sock.B to /var/run/domain.sock (in >>>> uxst_bind_listener) >>>> - in the mean time, there are a zillion connections to >>>> /var/run/domain.sock and pid B isn't started up yet; backlog is >>>> exhausted >>>> - pid B signals pid A to shut down >>>> - pid A runs the destroy_uxst_socket function and tries to connect >>>> to /var/run/domain.sock to see if it's still in use. The connection >>>> fails >>>> (because the backlog is full). Pid A unlinks /var/run/domain.sock. >>>> Everything is sad forever now. >>>> >>>> I'm thinking about just commenting out the call to destroy_uxst_socket >>>> since this is all on a tmpfs and we don't really care if spare sockets are >>>> leaked when/if we change configuration in the future. Arguably, the >>>> solution should be something where we don't overflow the listen socket at >>>> all; I'm thinking about also binding to a TCP port on localhost and just >>>> using that for the few seconds it takes to reload (since otherwise we run >>>> out of ephemeral sockets to 127.0.0.1); it still seems wrong for haproxy to >>>> unlink the socket, though. >>>> >>>> This has proven extremely irritating to reproduce (since it only occurs >>>> if there's enough load to fill up the backlog on the socket between when >>>> pid B starts up and when pid A shuts down), but I'm pretty confident that >>>> what I described above is happening, since periodically on reloads the >>>> domain socket isn't there and this code fits. >>>> >>>> Our configs are quite large, so I'm not reproducing them here. The >>>> reason we bind on a domain socket at all is because we're running two sets >>>> of haproxies — one in multi-process mode doing TCP-mode SSL termination >>>> pointing back over a domain socket to a single-process haproxy applying all >>>> of our actual config. >>>> >>>> -- >>>> James Brown >>>> Systems ​ >>>> Engineer >>>> >>> >>> >>> >>> -- >>> James Brown >>> Engineer >>> >> >> > > > -- > James Brown > Engineer >
