Hi Conrad, I agree that under normal circumstances, the intermediate process shouldn't be abnormally terminated. However, I was interested in understanding the behaviour in case of such an occurrence (though unlikely). The only "fix" I can imagine is reaping of such orphan processes. Thanks for the explanation.
-Bharath On Mon, Feb 29, 2016 at 5:18 PM, Conrad Hoffmann <con...@soundcloud.com> wrote: > Hi, > > On 02/27/2016 02:23 AM, BR Kumar wrote: > > Hi Conrad, > > > > Thanks for responding. Regarding point (2), I'm not sure the last level > of > > child processes are awaiting connection closure and would exit > afterwards. > > In my test, netstat showed no connection against the haproxy port. Yet, > the > > orphaned child process continued indefinitely. To reiterate, if the > > intermediate haproxy process (highlighted through ** in the subsequent > > illustration) were to die, its child processes linger on forever > regardless > > of connections. In addition, there doesn't appear to be a way to reap > them > > when the wrapper re-spawns the new hierarchy. This can be simulated > easily > > in a test environment. I believe this shortcoming needs a fix. Thoughts? > > This depends on your definition of "terminate". If you just kill the > process, yes, the child processes will be reparented to init and stick > around. There is not much to be fixed about this, you just shoudn't be oing > this. > > But if you send the proper signal then the parent process will in turn send > signals to the child processes before it exits, basically telling the child > processes to finish their existing connections, then terminate. In this > case, they might linger around for a few minutes but will eventually > terminate. > > If this doesn't happen in your setup then I assume the service handling is > not properly set up, e.g. systemd does not understand how to properly > start/stop/reload the service. These things are configurable and haproxy is > not quite the "default case" here. > > Hope that helps, > Conrad > > > > > Thanks, > > Bharath > > > > ---- > > systemd > > L haproxy-systemd-wrapper > > L ***haproxy** <---terminate this* > > L haproxy \ > > L haproxy } nbproc processes > > L haproxy } > > L haproxy / > > > > > > On Thu, Feb 25, 2016 at 3:02 PM, Conrad Hoffmann <con...@soundcloud.com> > > wrote: > > > >> Hi Bharat, > >> > >> On 02/24/2016 03:04 AM, BR Kumar wrote: > >>> Couple of questions related to the systemd wrapper: > >>> > >>> 1) I noticed that it spawns a 2 level hierarchy of haproxy processes > >>> instead of a single child process. Can someone help understand why? > >> > >> It's a little complex, but I can try: > >> > >> It is probably easier to understand when thinking about a scenario where > >> nbproc is > 1, let's say 4. When running under supervision of systemd > (or > >> any other service manager, e.g. runit), you need a single application > >> process that does not daemonize. Systemd monitors this process to e.g. > to > >> restart it if it dies. But most service managers are designed to > supervise > >> one process per service. So if we want 4 processes, the application > itself > >> has to manage this. This is why you would see a process hierarchy like > >> this: > >> > >> systemd > >> L haproxy-systemd-wrapper > >> L haproxy > >> L haproxy \ > >> L haproxy } nbproc processes > >> L haproxy } > >> L haproxy / > >> > >> Now, you might say, but there is already the systemd-wrapper , why do we > >> need the one additional haproxy instance? It is related to the special > >> (carefully designed) restart mechanism that haproxy implements. Going > into > >> details would make this a pretty long mail, but the gist it that _all_ > >> haproxy process will be replaced by new ones in the process, including > the > >> parent one. Thus, the moment that the parent process were to exit, > systemd > >> would think the service needs to be restarted and fire up yet another > >> instance of haproxy (even though there is already a replacement, but > >> systemd doesn't know about it). So the sole purpose of the wrapper is to > >> prevent systemd from interfering with th restart mechanism implement in > the > >> application itself, basically. > >> Sorry, this leaves out a lot of details, but I hope it does make a > little > >> sense. > >> > >>> 2) The problem arises when the intermediate haproxy process dies for > any > >>> reason and the child process is adopted by pid 1. When the systemd > >> wrapper > >>> is restarted, this results in a total of 3 haproxy processes. Is this > >>> behavior expected? Are there fixes/workaround to ensure that only > >>> systemd-wrapper controlled processes are retained? > >> > >> This is intentional. The restart mechanism is designed to not interrupt > >> existing connections. As such, the child processes will be detached from > >> their parent, keep running until all connections they are processing are > >> closed by the client side and then exit. > >> > >> Cheers, > >> Conrad > >> -- > >> Conrad Hoffmann > >> Traffic Engineer > >> > >> SoundCloud Ltd. | Rheinsberger Str. 76/77, 10115 Berlin, Germany > >> > >> Managing Director: Alexander Ljung | Incorporated in England & Wales > >> with Company No. 6343600 | Local Branch Office | AG Charlottenburg | > >> HRB 110657B > >> > > > > -- > Conrad Hoffmann > Traffic Engineer > > SoundCloud Ltd. | Rheinsberger Str. 76/77, 10115 Berlin, Germany > > Managing Director: Alexander Ljung | Incorporated in England & Wales > with Company No. 6343600 | Local Branch Office | AG Charlottenburg | > HRB 110657B >
