[systemd-devel] option to wait for pid file to appear
Mantas Mikulėnas: Maybe socket-activation would work for you? (With Nginx it's also a hack though.) Accept=No Environment=NGINX=3; It is not terrifically complex. The documented way to stop the forking is "daemon off", but an inherited listening socket also does it. * https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=728015 * https://github.com/nginx/nginx/blob/8e8734ec82cde91a02d0cbfaae0d0df6b5aaab14/src/core/nginx.c#L347 ___ systemd-devel mailing list systemd-devel@lists.freedesktop.org https://lists.freedesktop.org/mailman/listinfo/systemd-devel
Re: [systemd-devel] option to wait for pid file to appear
On 7 June 2018 at 10:35, Lennart Poettering wrote: > Nah, daemon() does not write PID files, you have to do that in your > own code. As daemon() calls _exit() (not even exit()) in the parent after the fork, the only way to synchronize the pid writing is not to use the daemon() at all and inline daemon() functionality. I doubt that it will ever be done for the wast majority of sources using the function. And under systemd there are zero incentives to do that as the workaround with waiting for the pid file to appear is good enough to make things reliable. So as such the warning that currently systemd produces is just a distraction that a user cannot do nothing about. As a side note the usage of daemon() in 3 from 4 sources I randomly picked up on github is wrong as the code calls the function and then writes the pid file right after parsing the config before any initialization of sockets, but that just pointed out how lack of simple service initialization API lead to everybody inventing own buggy code... ___ systemd-devel mailing list systemd-devel@lists.freedesktop.org https://lists.freedesktop.org/mailman/listinfo/systemd-devel
Re: [systemd-devel] option to wait for pid file to appear
On Do, 07.06.18 09:50, Igor Bukanov (i...@mir2.org) wrote: > On 18 May 2018 at 19:37, Lennart Poettering wrote: > > On Do, 17.05.18 22:54, Igor Bukanov (i...@mir2.org) wrote: > > Well, no. The protocol is clear, and what we do is pretty close to > > black magic, and still racy in many ways. > > > > I mean, broken behaviour is still broken behaviour, even if we > > nowadays handle it pretty nicely. Really, nginx should be fixed, > > everybody benefits, sysvinit as much as we do. > > It turned out it is not only nginx that behaves this way. Any program > that is using the daemon(3) call available both in Linux and BSD > writes the pid from the child without the parent waiting. So this > behaviour must be widespread and as such I think must be accepted as a > de-facto standard (even if it is unfortunate) and systemd should not > even warn about it. Nah, daemon() does not write PID files, you have to do that in your own code. The fact that much code doesn't synchronize properly around that is really a bug in that code. I mean, you can get away with using daemon() as long as you synchronize around it, for example by first allocating some pipe(), then waiting on it in the parent, and signalling it from the child after having written the PID file... Again, not synchronizing properly is broken for sysvinit as much as it is broken for systemd. We deal with it to some point, but the only fully safe and correct way to handle this is to fix the programs in question. Lennart -- Lennart Poettering, Red Hat ___ systemd-devel mailing list systemd-devel@lists.freedesktop.org https://lists.freedesktop.org/mailman/listinfo/systemd-devel
Re: [systemd-devel] option to wait for pid file to appear
On Thu, 7 Jun 2018, Igor Bukanov wrote: > On 18 May 2018 at 19:37, Lennart Poettering wrote: > > On Do, 17.05.18 22:54, Igor Bukanov (i...@mir2.org) wrote: > > Well, no. The protocol is clear, and what we do is pretty close to > > black magic, and still racy in many ways. > > > > I mean, broken behaviour is still broken behaviour, even if we > > nowadays handle it pretty nicely. Really, nginx should be fixed, > > everybody benefits, sysvinit as much as we do. > > It turned out it is not only nginx that behaves this way. Any program > that is using the daemon(3) call available both in Linux and BSD > writes the pid from the child without the parent waiting. So this > behaviour must be widespread and as such I think must be accepted as a > de-facto standard (even if it is unfortunate) and systemd should not > even warn about it. I would agree. There's nothing the end user can really do about it, and they're more likely to complain here about it than the problematic software. For what it's worth, I took a look at the Type=pid-file idea again... and I'm not sure if I'm going to be able to do it. The problem is primarily the way systemd only tracks up to one "control" process and one "main" process, and that with Type=pid-file it's not clear at the outset whether the process spawned by systemd is one or the other. I don't think starting it off as a control process and then converting it to a main process -- if and only if it writes its own PID out -- is the right approach. So I might have to leave this idea for now, unless I can think of some way around this. Sorry if you were waiting on it. ___ systemd-devel mailing list systemd-devel@lists.freedesktop.org https://lists.freedesktop.org/mailman/listinfo/systemd-devel
Re: [systemd-devel] option to wait for pid file to appear
On 18 May 2018 at 19:37, Lennart Poettering wrote: > On Do, 17.05.18 22:54, Igor Bukanov (i...@mir2.org) wrote: > Well, no. The protocol is clear, and what we do is pretty close to > black magic, and still racy in many ways. > > I mean, broken behaviour is still broken behaviour, even if we > nowadays handle it pretty nicely. Really, nginx should be fixed, > everybody benefits, sysvinit as much as we do. It turned out it is not only nginx that behaves this way. Any program that is using the daemon(3) call available both in Linux and BSD writes the pid from the child without the parent waiting. So this behaviour must be widespread and as such I think must be accepted as a de-facto standard (even if it is unfortunate) and systemd should not even warn about it. ___ systemd-devel mailing list systemd-devel@lists.freedesktop.org https://lists.freedesktop.org/mailman/listinfo/systemd-devel
Re: [systemd-devel] option to wait for pid file to appear
On Do, 17.05.18 22:54, Igor Bukanov (i...@mir2.org) wrote: > On 17 May 2018 at 19:23, Lennart Poettering wrote: > > > So yes, this is a bug in nginx. They really should fix that. And this > > is not only broken when you use systemd, but on sysvinit too, as a > > command like this would likely fail there too: "service nginx start > > service nginx status", as the start would return before the PID file > > is written, and then status would claim the service to be down... > > Given that systemd deals with this situation in a very reasonable way, > nginx must not be alone in doing this. And if this a common way to > write such code, perhaps it should not be considered a bug but rather > one more peculiarity of application services the service manager > should deal Well, no. The protocol is clear, and what we do is pretty close to black magic, and still racy in many ways. I mean, broken behaviour is still broken behaviour, even if we nowadays handle it pretty nicely. Really, nginx should be fixed, everybody benefits, sysvinit as much as we do. Or, to turn this around: if nginx is happy that all only works somewhat reliably on systemd systems, then they might as well just do this properly, and rework this to Type=notify. Because right now, systemd can deal with the broken behaviour, but *only* systemd really can, and this will trip up on other systems, such as sysvinit or upstart. Lennart -- Lennart Poettering, Red Hat ___ systemd-devel mailing list systemd-devel@lists.freedesktop.org https://lists.freedesktop.org/mailman/listinfo/systemd-devel
Re: [systemd-devel] option to wait for pid file to appear
On 05/17/2018 03:54 PM, Igor Bukanov wrote: Given that systemd deals with this situation in a very reasonable way, nginx must not be alone in doing this. And if this a common way to write such code, perhaps it should not be considered a bug but rather one more peculiarity of application services the service manager should deal IIRC, the Python daemon library does this. -- Ian Pilcher arequip...@gmail.com "I grew up before Mark Zuckerberg invented friendship" ___ systemd-devel mailing list systemd-devel@lists.freedesktop.org https://lists.freedesktop.org/mailman/listinfo/systemd-devel
Re: [systemd-devel] option to wait for pid file to appear
Am 17.05.2018 um 22:54 schrieb Igor Bukanov: > On 17 May 2018 at 19:23, Lennart Poettering wrote: > >> So yes, this is a bug in nginx. They really should fix that. And this >> is not only broken when you use systemd, but on sysvinit too, as a >> command like this would likely fail there too: "service nginx start >> service nginx status", as the start would return before the PID file >> is written, and then status would claim the service to be down... > > Given that systemd deals with this situation in a very reasonable way, > nginx must not be alone in doing this. And if this a common way to > write such code, perhaps it should not be considered a bug but rather > one more peculiarity of application services the service manager > should deal why? either fix the pidfile stuff in nginx or simply start supporting "Type=notify" as a lot of other services already do and NO i don't grasp why "live updates" need that at all instead have main-pid as a tiny shim-controller which *never* exits at all ___ systemd-devel mailing list systemd-devel@lists.freedesktop.org https://lists.freedesktop.org/mailman/listinfo/systemd-devel
Re: [systemd-devel] option to wait for pid file to appear
On 17 May 2018 at 19:23, Lennart Poettering wrote: > So yes, this is a bug in nginx. They really should fix that. And this > is not only broken when you use systemd, but on sysvinit too, as a > command like this would likely fail there too: "service nginx start > service nginx status", as the start would return before the PID file > is written, and then status would claim the service to be down... Given that systemd deals with this situation in a very reasonable way, nginx must not be alone in doing this. And if this a common way to write such code, perhaps it should not be considered a bug but rather one more peculiarity of application services the service manager should deal ___ systemd-devel mailing list systemd-devel@lists.freedesktop.org https://lists.freedesktop.org/mailman/listinfo/systemd-devel
Re: [systemd-devel] option to wait for pid file to appear
On Do, 17.05.18 13:18, Igor Bukanov (i...@mir2.org) wrote: > Notice that there are 2 master processes, the old one with pid 2408 > and the one 2418. Initiate the shutdown of the initial master: nginx can let PID 1 know about the new main PID by sending a MAINPID= sd_notify() message. Or even better, it could just push its fds into the fdstore and let systemd take care of restarts without losing connections... Lennart -- Lennart Poettering, Red Hat ___ systemd-devel mailing list systemd-devel@lists.freedesktop.org https://lists.freedesktop.org/mailman/listinfo/systemd-devel
Re: [systemd-devel] option to wait for pid file to appear
On Do, 17.05.18 11:33, Reindl Harald (h.rei...@thelounge.net) wrote: > > but that busy waits. Is there any option to replace this hack via a > > native systemd solution, like explicitly waiting for the pid file to > > appear before considering the unit ready? > > this is a bug in nginx > > however, after many years with systemd on Fedora one needs yet to show > me a service where you can't simply remove the PIDFile line and you are done systemd is able to guess the main PID of a process in many cases, but this cannot work in all cases, because from the outside we have no clue what might be the main process and what is not. In particular HTTP servers which fork off multiple worker processes are candidates where systemd won't guess right, because there's ambiguity as there are likely multiple processes around after start. Hence, yes, ideally all services would use Type=simple or Type=notify. But if they don't they should use Type=forking and that only in combination with PIDFile=, otherwise service management becomes black magic... Lennart -- Lennart Poettering, Red Hat ___ systemd-devel mailing list systemd-devel@lists.freedesktop.org https://lists.freedesktop.org/mailman/listinfo/systemd-devel
Re: [systemd-devel] option to wait for pid file to appear
On Do, 17.05.18 11:25, Igor Bukanov (i...@mir2.org) wrote: > Hi, > > I have a service unit for nginx that uses Type=forking and PIDFile. > That works, but occasionally I see in the log a message like > > nginx.service: PID file /run/nginx/nginx.pid not readable (yet?) after > start: No such file or directory So yes, this is a bug in nginx. They really should fix that. And this is not only broken when you use systemd, but on sysvinit too, as a command like this would likely fail there too: "service nginx start service nginx status", as the start would return before the PID file is written, and then status would claim the service to be down... Hence, please file a bug against nginx asking them to wait in the parent until the PID file is fully written before exiting. It's how this works on SysV systems, and systemd wants that too. > but that busy waits. Is there any option to replace this hack via a > native systemd solution, like explicitly waiting for the pid file to > appear before considering the unit ready? As others indicated, the best approach would be to do Type=notify, and patch nginx to send out the right ready message after it wrote the PID file. That'd be the native systemd solution. PID files suck (as they have no clear clean-up regime), but if services use them, then they should really implement them properly, and thus write them before exiting in the forking parent. Lennart -- Lennart Poettering, Red Hat ___ systemd-devel mailing list systemd-devel@lists.freedesktop.org https://lists.freedesktop.org/mailman/listinfo/systemd-devel
Re: [systemd-devel] option to wait for pid file to appear
On Thu, May 17, 2018 at 12:12:10PM +0200, Igor Bukanov wrote: > On 17 May 2018 at 11:58, Mantas Mikulėnas wrote: > > Have you tried without the PIDFile= setting at all? > > As far as I can see that breaks live updates that nginx supports where > it starts a new process and workers and then gracefully terminates the > old main. FTR, it *is* possible to do live updates like that. We added support for that with FD_STORE in pid1, and systemd-journald and systemd-logind now support restart without losing connections. It would be great if nginx did that too. Zbyszek ___ systemd-devel mailing list systemd-devel@lists.freedesktop.org https://lists.freedesktop.org/mailman/listinfo/systemd-devel
Re: [systemd-devel] option to wait for pid file to appear
On 17 May 2018 at 11:58, Mantas Mikulėnas wrote: > this would be anyways broken because systemd reads the PIDFile once at > start to dtermine MAINPID and if MAINPID goes away the service fails This is not true as with PIDFile systemd does pick up the new process as a new main without restarting the unit. This is an example from my development VM: systemctl status nginx ... Active: active (running) since Thu 2018-05-17 11:02:57 UTC; 3min 18s ago Docs: man:nginx(8) Process: 2407 ExecStart=/usr/sbin/nginx ... Main PID: 2408 (nginx) CGroup: /system.slice/nginx.service ├─2408 nginx: master process ... ├─2409 nginx: worker process └─2410 nginx: worker process Initiate a graceful restart of nginx: kill -s USR2 2408 systemctl status nginx ... Active: active (running) since Thu 2018-05-17 11:02:57 UTC; 5min ago Docs: man:nginx(8) Process: 2407 ExecStart=/usr/sbin/nginx ... Main PID: 2408 (nginx) CGroup: /system.slice/nginx.service ├─2408 nginx: master process /usr/sbin/nginx ... ├─2409 nginx: worker process ├─2410 nginx: worker process ├─2418 nginx: master process /usr/sbin/nginx ... ├─2419 nginx: worker process └─2420 nginx: worker process Notice that there are 2 master processes, the old one with pid 2408 and the one 2418. Initiate the shutdown of the initial master: kill -s QUIT 2408 systemctl status nginx Active: active (running) since Thu 2018-05-17 11:02:57 UTC; 7min ago Docs: man:nginx(8) Process: 2407 ExecStart=/usr/sbin/nginx -c /vol/opt/act/webapp/nginx/nginx.conf -g daemon on; master_process on; worker_processes auto; pid /run/nginx/nginx.pid; error_log sys Main PID: 2418 (nginx) CGroup: /system.slice/nginx.service ├─2418 nginx: master process /usr/sbin/nginx -c /vol/opt/act/webapp/nginx/nginx.conf -g daemon on; master_process on; worker_processes auto; pid /run/nginx/nginx.pid; ├─2419 nginx: worker process └─2420 nginx: worker process Here systemd correctly recognized 2418 as the new master without restarting the unit as seen by active time. ___ systemd-devel mailing list systemd-devel@lists.freedesktop.org https://lists.freedesktop.org/mailman/listinfo/systemd-devel
Re: [systemd-devel] option to wait for pid file to appear
On Thu, 17 May 2018, Igor Bukanov wrote: > On 17 May 2018 at 12:07, Michael Chapman wrote: > > It _is_ better for the PID file to be written out before the initial > > process exits, but systemd will handle things correctly even if they > > happen the other way around. Essentially the service won't be considered > > to have completed activation until both events occur. If one or the other > > takes too long (i.e. longer than the TimeoutStartSec= setting), the unit > > will fail. > > Does it means that even ExecStartPost and especially ExecReload can > always safely use $MAINPID? I.e. does systemd call these Execs only > after it managed to read the the pid file even when the file is > written after the initial ExecStart process exited? So it looks like systemd will add the MAINPID environment variable any time it knows what the main PID is. It learns this by reading the PID file, or if PIDFile= is omitted by guessing. It forgets it when it sees the SIGCHLD from that PID and no new PID file is available. So by the time ExecStartPost runs, the main PID may or may not be known. Indeed, it's permitted for the PID file to be written out by ExecStartPost itself. For ExecReload, though, there definitely should be a known main PID. I can't see any way you can get systemd to reload a service whose main PID hasn't been determined (or guessed). ExecReload may cause systemd to forget the main PID, if it makes the process exit, but it will be run with $MAINPID set. > And yes, Type=pid-file will be very useful. I have exactly a shell > script that will benefit from it nicely. I'll dig it out and see what I can do. It's probably bitrotted quite a bit since I wrote it -- I never completed it because my own need for it went away. ___ systemd-devel mailing list systemd-devel@lists.freedesktop.org https://lists.freedesktop.org/mailman/listinfo/systemd-devel
Re: [systemd-devel] option to wait for pid file to appear
On 17 May 2018 at 12:07, Michael Chapman wrote: > It _is_ better for the PID file to be written out before the initial > process exits, but systemd will handle things correctly even if they > happen the other way around. Essentially the service won't be considered > to have completed activation until both events occur. If one or the other > takes too long (i.e. longer than the TimeoutStartSec= setting), the unit > will fail. Does it means that even ExecStartPost and especially ExecReload can always safely use $MAINPID? I.e. does systemd call these Execs only after it managed to read the the pid file even when the file is written after the initial ExecStart process exited? And yes, Type=pid-file will be very useful. I have exactly a shell script that will benefit from it nicely. ___ systemd-devel mailing list systemd-devel@lists.freedesktop.org https://lists.freedesktop.org/mailman/listinfo/systemd-devel
Re: [systemd-devel] option to wait for pid file to appear
On Thu, 17 May 2018, Igor Bukanov wrote: > Hi, > > I have a service unit for nginx that uses Type=forking and PIDFile. > That works, but occasionally I see in the log a message like > > nginx.service: PID file /run/nginx/nginx.pid not readable (yet?) after > start: No such file or directory > > After investigating this father I see the reason for this is that > nginx creates the pid file in a child process, not in the parent (at > least this is how I interpret their code). As the parent may exit and > systemd may respond to it before the child writes the pid, that leads > to the above message. The message is essentially harmless, at least when you're talking about initial service startup. It _is_ better for the PID file to be written out before the initial process exits, but systemd will handle things correctly even if they happen the other way around. Essentially the service won't be considered to have completed activation until both events occur. If one or the other takes too long (i.e. longer than the TimeoutStartSec= setting), the unit will fail. I think the only time this is actually a problematic situation is if the service is in a fully activated state, and the main process goes away before its replacement writes out a new PID file. There's a window there when systemd can think that the main process has simply decided to exit itself. In this case it will stop the unit. On the other hand, if this sequence of events was initiated because the admin explicitly reloaded the service through systemd, then again systemd will, as far as I know, be happy with these two events happening in the wrong order. I'd need to check the code more thoroughly to be sure of this though. For what it's worth, there is an "introduce Type=pid-file" item in the systemd TODO that seems like it would help with these kinds of problematic services. I see value in having something that simply waits for a PID file to appear, and doesn't care whether any processes exited in the meantime. As an example use-case, authors of scripts currently need to use systemd-notify and NotifyAccess= to implement reliable service startup notification. With Type=pid-file, however, they could just do: #!/bin/sh # Run various setup stuff, possibly "exit 1" # if there's a problem ... # All good, tell systemd we're active echo $$ >/run/some-script.pid # ... As it happens, I actually have some code to implement Type=pid-file, but I never got around to finish cleaning it up and raising a pull request. I should probably dig it out and raise a PR. ___ systemd-devel mailing list systemd-devel@lists.freedesktop.org https://lists.freedesktop.org/mailman/listinfo/systemd-devel
Re: [systemd-devel] option to wait for pid file to appear
Am 17.05.2018 um 12:12 schrieb Igor Bukanov: > On 17 May 2018 at 11:58, Mantas Mikulėnas wrote: >> Have you tried without the PIDFile= setting at all? > > As far as I can see that breaks live updates that nginx supports where > it starts a new process and workers and then gracefully terminates the > old main this would be anyways broken because systemd reads the PIDFile once at start to dtermine MAINPID and if MAINPID goes away the service fails ___ systemd-devel mailing list systemd-devel@lists.freedesktop.org https://lists.freedesktop.org/mailman/listinfo/systemd-devel
Re: [systemd-devel] option to wait for pid file to appear
On 17 May 2018 at 11:58, Mantas Mikulėnas wrote: > Have you tried without the PIDFile= setting at all? As far as I can see that breaks live updates that nginx supports where it starts a new process and workers and then gracefully terminates the old main. Regards, Igor ___ systemd-devel mailing list systemd-devel@lists.freedesktop.org https://lists.freedesktop.org/mailman/listinfo/systemd-devel
Re: [systemd-devel] option to wait for pid file to appear
On Thu, May 17, 2018, 12:26 Igor Bukanov wrote: > Hi, > > I have a service unit for nginx that uses Type=forking and PIDFile. > That works, but occasionally I see in the log a message like > > nginx.service: PID file /run/nginx/nginx.pid not readable (yet?) after > start: No such file or directory > > After investigating this father I see the reason for this is that > nginx creates the pid file in a child process, not in the parent (at > least this is how I interpret their code). As the parent may exit and > systemd may respond to it before the child writes the pid, that leads > to the above message. > > I can workaround that via replacing ExecStart=/usr/sbin/nginx with > something like: > > ExecStart=/bin/sh -c "set -e; /usr/sbin/nginx; while ! test -s > /run/nginx/nginx.pid; do sleep 0.1; done' > > but that busy waits. Is there any option to replace this hack via a > native systemd solution, like explicitly waiting for the pid file to > appear before considering the unit ready? > A "native systemd solution" would be to get rid of pidfiles completely. Have you tried without the PIDFile= setting at all? Maybe socket-activation would work for you? (With Nginx it's also a hack though.) > -- Mantas Mikulėnas ___ systemd-devel mailing list systemd-devel@lists.freedesktop.org https://lists.freedesktop.org/mailman/listinfo/systemd-devel
Re: [systemd-devel] option to wait for pid file to appear
Am 17.05.2018 um 11:25 schrieb Igor Bukanov: > I have a service unit for nginx that uses Type=forking and PIDFile. > That works, but occasionally I see in the log a message like > > nginx.service: PID file /run/nginx/nginx.pid not readable (yet?) after > start: No such file or directory > > After investigating this father I see the reason for this is that > nginx creates the pid file in a child process, not in the parent (at > least this is how I interpret their code). As the parent may exit and > systemd may respond to it before the child writes the pid, that leads > to the above message. > > I can workaround that via replacing ExecStart=/usr/sbin/nginx with > something like: > > ExecStart=/bin/sh -c "set -e; /usr/sbin/nginx; while ! test -s > /run/nginx/nginx.pid; do sleep 0.1; done' > > but that busy waits. Is there any option to replace this hack via a > native systemd solution, like explicitly waiting for the pid file to > appear before considering the unit ready? this is a bug in nginx however, after many years with systemd on Fedora one needs yet to show me a service where you can't simply remove the PIDFile line and you are done ___ systemd-devel mailing list systemd-devel@lists.freedesktop.org https://lists.freedesktop.org/mailman/listinfo/systemd-devel
[systemd-devel] option to wait for pid file to appear
Hi, I have a service unit for nginx that uses Type=forking and PIDFile. That works, but occasionally I see in the log a message like nginx.service: PID file /run/nginx/nginx.pid not readable (yet?) after start: No such file or directory After investigating this father I see the reason for this is that nginx creates the pid file in a child process, not in the parent (at least this is how I interpret their code). As the parent may exit and systemd may respond to it before the child writes the pid, that leads to the above message. I can workaround that via replacing ExecStart=/usr/sbin/nginx with something like: ExecStart=/bin/sh -c "set -e; /usr/sbin/nginx; while ! test -s /run/nginx/nginx.pid; do sleep 0.1; done' but that busy waits. Is there any option to replace this hack via a native systemd solution, like explicitly waiting for the pid file to appear before considering the unit ready? Regards, Igor Bukanov ___ systemd-devel mailing list systemd-devel@lists.freedesktop.org https://lists.freedesktop.org/mailman/listinfo/systemd-devel
