Hi all,
On 22:01 Wed 04 Oct , Lukas Tribus wrote:
> Hello Moemen,
>
>
> Am 04.10.2017 um 19:21 schrieb Moemen MHEDHBI:
> >
> > I am wondering if this is actually an expected behaviour and if maybe
> > that restart/stop should just shutdown the process and its open connections.
> > I have made the following tests:
> > 1/ keep an open connection then do a restart will work correctly without
> > waiting for existing connections to be closed.
>
> You're right, I got confused there.
>
> Stop or restart is supposed to kill existing connections without any
> timeouts, and
> systemd would signal it with a SIGTERM to the systemd-wrapper:
>
> https://cbonte.github.io/haproxy-dconv/1.7/management.html#4
>
>
>
> > I think it makes more sense to say that restart will not wait for
> > established connections.
>
> Correct, that's the documented behavior as per haproxy documentation and
> really the implicit assumption when talking about stopping/restarting in a
> process
> management context (I need more coffee).
>
>
>
> > Otherwise there will be no difference between
> > reload and restart unless there is something else am not aware of.
> > If we need to fix 2/, a possible solution would be:
> > - Set killmode to "control-group" rather than "mixed" (the current
> > value) in systemd unit file.
>
> Indeed the mixed killmode was a conscious choice:
> https://marc.info/?l=haproxy&m=141277054505608&w=2
>
>
> I guess the problem is that when a reload happens before a restart and
> pre-reload
> systemd-wrapper process is still alive, systemd gets confused by that old
> process
> and therefor, refrains from starting up the new instance.
The biggest issue here is that we are using a signal to trigger the
reload (which is a complex, non-atomic operation) and let things settle
on their own. Systemd assumes that as soon as the signal is delivered
(i.e. the ExecReload command is done), the reload is finished, while in
our case the reload is finished when the old haproxy process is really
dead. Using a signal to trigger the reload is handy, so we could keep
that, but the wrapper would need some changes to make reloads more
robust:
1. It should use sd_notify(3) to communicate the start/stop/reload
status to systemd (that would also mean converting the actual
service to Type=notify). This way no other operation will be
performed on the unit until the reload is finished and the process
group is in a known-good state.
2. It should handle the old process better: apart from relying on the
new haproxy process for killing the old one, it should explicitly
SIGKILL it after a given timeout if it's not dead yet and make sure
reloads are timeboxed.
IIUC, in 1.8 the wrapper has been replaced by the master process which
seems to do point 2 above, but point 1 is something that should still be
handled IMHO.
Regards,
Apollon
