Re: [systemd-devel] failing unmounts during reboot
Am 25.07.19 um 13:29 schrieb Frank Steiner: > Silvio Knizek wrote: > >> the proper approach would be to define the dependency of the generated >> .service to the mount point with a drop-in and RequiresMountsFor=. See >> man:systemd.unit for more information. > > How would that work for init.d scripts? excatly the same way i guess - remember that your sysvinit script is in reality a generated service as you can see in "systemctl status sysv-servicename" /etc/systemd/system/servicename.service.d/ dropins ___ systemd-devel mailing list systemd-devel@lists.freedesktop.org https://lists.freedesktop.org/mailman/listinfo/systemd-devel
Re: [systemd-devel] failing unmounts during reboot
Lennart Poettering wrote: Things like ssh when run as a sysv script: people expect that shutting down ssh doesn't kill all your children, and it didn't in sysv... And there are similar things. That's a reasonable example, thanks! -- Dipl.-Inform. Frank Steiner Web: http://www.bio.ifi.lmu.de/~steiner/ Lehrstuhl f. BioinformatikMail: http://www.bio.ifi.lmu.de/~steiner/m/ LMU, Amalienstr. 17 Phone: +49 89 2180-4049 80333 Muenchen, Germany Fax: +49 89 2180-99-4049 * Rekursion kann man erst verstehen, wenn man Rekursion verstanden hat. * ___ systemd-devel mailing list systemd-devel@lists.freedesktop.org https://lists.freedesktop.org/mailman/listinfo/systemd-devel
Re: [systemd-devel] failing unmounts during reboot
Am 25.07.19 um 13:15 schrieb Mantas Mikulėnas: > IIRC the idea is that user sessions are killed during shutdown before > unmounting anything anyway, and services have implicit > After=local-fs.target so they get stopped before local filesystems? The > shutdown process is one area that I still know very little about. the shutdown process is pretty simple: the exactly opposite order based on dependencies like the startup and this without inter dependencies are stopped in parallel as they are started in parallel at boot ___ systemd-devel mailing list systemd-devel@lists.freedesktop.org https://lists.freedesktop.org/mailman/listinfo/systemd-devel
Re: [systemd-devel] failing unmounts during reboot
On Do, 25.07.19 13:46, Frank Steiner (fsteiner-ma...@bio.ifi.lmu.de) wrote: > Mantas Mikulėnas wrote: > > > I think that's a deliberate decision made in the > > systemd-sysv-generator. Note how the generated .service files have > > "KillMode=process" (and even "RemainAfterExit=yes"). The default for > > native services is to kill the entire cgroup, and IIRC that even was > > one of the main reasons for using cgroups. > > Most likely it's there to retain compatibility with some of the > > weirder init.d scripts – those which don't start any daemons; those > > which start several; and so on and so on. > > Thanks a lot! I found the service file in /run/systemd/generator.late. > Anyway, I cannot think of any init.d script where sub processes or > anything should be left running when the script is stopped, so > KillMode=process seems strange. Maybe Lennart knows why this decision > was taken. Things like ssh when run as a sysv script: people expect that shutting down ssh doesn't kill all your children, and it didn't in sysv... And there are similar things. Lennart -- Lennart Poettering, Berlin ___ systemd-devel mailing list systemd-devel@lists.freedesktop.org https://lists.freedesktop.org/mailman/listinfo/systemd-devel
Re: [systemd-devel] failing unmounts during reboot
Lennart Poettering wrote: When generating native unit files from legacy sysv scripts, we use KillMode=process, which means we'll only kill the main process, and nothing else. This choice was made since its behaviour comes closest to classic SysV behaviour (since there the init system didn't kill any auxiliary processes either). Given it's 2019 it might be wise to just write a native unit file if you want better control of this. Note that for native unit files we use different defaults: there we kill everything by default. You can also reuse the generated unit, but only change the KillMode= setting, by creating a drop-in in /etc/systemd/system/.service.d/.conf, and then adding there: [Service] KillMode=control-group Yes, Mantas answer brought me to the same solution. For me that's everything I need to solve the problems with our init.d scripts I guess, so thanks! But let me underline: a SysV script which leaves processes around is simply buggy. In sysv it was expected that scripts would clean up properly on "stop", on their own. If you don't do that in your script, then maybe fix that... The only correct way to shut things down is with individual ordering between units really, but that falls apart if you stick to historic sysv semantics too much. Totally agreed, but the problem are scripts like the ones for grid or matlab that have 200 lines or do such weird things that you couldn't do anything but write a unit more or less like the one systemd-sysv-generator creates. I just have to deal with those, but the KillMode drop-in does help enough. Thanks for all the explanations! cu, Frank -- Dipl.-Inform. Frank Steiner Web: http://www.bio.ifi.lmu.de/~steiner/ Lehrstuhl f. BioinformatikMail: http://www.bio.ifi.lmu.de/~steiner/m/ LMU, Amalienstr. 17 Phone: +49 89 2180-4049 80333 Muenchen, Germany Fax: +49 89 2180-99-4049 * Rekursion kann man erst verstehen, wenn man Rekursion verstanden hat. * ___ systemd-devel mailing list systemd-devel@lists.freedesktop.org https://lists.freedesktop.org/mailman/listinfo/systemd-devel
Re: [systemd-devel] failing unmounts during reboot
Mantas Mikulėnas wrote: I think that's a deliberate decision made in the systemd-sysv-generator. Note how the generated .service files have "KillMode=process" (and even "RemainAfterExit=yes"). The default for native services is to kill the entire cgroup, and IIRC that even was one of the main reasons for using cgroups. Most likely it's there to retain compatibility with some of the weirder init.d scripts – those which don't start any daemons; those which start several; and so on and so on. Thanks a lot! I found the service file in /run/systemd/generator.late. Anyway, I cannot think of any init.d script where sub processes or anything should be left running when the script is stopped, so KillMode=process seems strange. Maybe Lennart knows why this decision was taken. But now that I found the service file, I created a drop-in /etc/systemd/system/bla.service.d/kill.conf with [Service] KillMode=control-group and now, after reloading everything, the sleep is killed. So for all init.d scripts that I get from somewhere I've now way to ensure that all their processes are cleaned up by a simple drop-in. This will likely solve most of my problems. Thanks for pointing me to the right direction :-) cu, Frank -- Dipl.-Inform. Frank Steiner Web: http://www.bio.ifi.lmu.de/~steiner/ Lehrstuhl f. BioinformatikMail: http://www.bio.ifi.lmu.de/~steiner/m/ LMU, Amalienstr. 17 Phone: +49 89 2180-4049 80333 Muenchen, Germany Fax: +49 89 2180-99-4049 * Rekursion kann man erst verstehen, wenn man Rekursion verstanden hat. * ___ systemd-devel mailing list systemd-devel@lists.freedesktop.org https://lists.freedesktop.org/mailman/listinfo/systemd-devel
Re: [systemd-devel] failing unmounts during reboot
On Do, 25.07.19 13:29, Frank Steiner (fsteiner-ma...@bio.ifi.lmu.de) wrote: > Silvio Knizek wrote: > > > the proper approach would be to define the dependency of the generated > > .service to the mount point with a drop-in and RequiresMountsFor=. See > > man:systemd.unit for more information. > > How would that work for init.d scripts? > > > Also the systemd-sysv-generator can only do so much. > > 1) If the systemd-sysv-generator creates wrapper .service files on-the-fly, >is it possible to hook into those with dropins? The man page didn't >give much information. Yes. > 2) If init.d scripts are wrapped by .service files, shouldn't the >processes spawned by these scripts be killed when shutting down? No, for compat with sysv that's not what we do. "/etc/init.d/fooobar stop" didn't do any such clean-up and neither do we hence. Lennart -- Lennart Poettering, Berlin ___ systemd-devel mailing list systemd-devel@lists.freedesktop.org https://lists.freedesktop.org/mailman/listinfo/systemd-devel
Re: [systemd-devel] failing unmounts during reboot
On Do, 25.07.19 11:40, Frank Steiner (fsteiner-ma...@bio.ifi.lmu.de) wrote: > Hi, > > I'm currently discussing a problem with the SuSE support about failing > unmounts during reboot. Tyring to debug this I realized that systemd > is not killing processes left over by some init.d script. E.g. use > the following script in /etc/init.d/ > > #!/bin/sh > # > ### BEGIN INIT INFO > # Provides: bla > # Required-Start: $network $remote_fs sshd > # Required-Stop: $network $remote_fs sshd > # Default-Start: 2 3 5 > # Description:test > ### END INIT INFO > case "$1" in > start) cd /test; /usr/bin/sleep 99d & ;; > stop) true;; > esac When generating native unit files from legacy sysv scripts, we use KillMode=process, which means we'll only kill the main process, and nothing else. This choice was made since its behaviour comes closest to classic SysV behaviour (since there the init system didn't kill any auxiliary processes either). Given it's 2019 it might be wise to just write a native unit file if you want better control of this. Note that for native unit files we use different defaults: there we kill everything by default. You can also reuse the generated unit, but only change the KillMode= setting, by creating a drop-in in /etc/systemd/system/.service.d/.conf, and then adding there: [Service] KillMode=control-group But let me underline: a SysV script which leaves processes around is simply buggy. In sysv it was expected that scripts would clean up properly on "stop", on their own. If you don't do that in your script, then maybe fix that... > On shutdown, unmounting /test will fail because the sleep process is > not killed. Shouldn't there be a mechanism in system to kill processes > spawned by LSB script when shutting these down? Well, quite frankly, either way we do it people will be upset. If we kill all processes of a service on stop, people tell us "sysv didn't kill all processes on service stop, why do you"? Now you say the opposite: "why don't you kill all service processes on stop, you should!", but there's no way out. If you ask me, just forget about SysV init scripts in 2019, and spent the 15min to just put together a native unit. It will save you frustration in the long run, and fixes all these issues. Also note that we live in a world where various kinds of storage (including much of NFS) requires local services running to operate. Because of that we can't just decide "oh, its time to tear down NFS, let's kill *every* process by PID 1", because then the whole system will be borked. The only correct way to shut things down is with individual ordering between units really, but that falls apart if you stick to historic sysv semantics too much. > > And moreover, wouldn't it make sense to have a mechanism to at least > try to kill all processes using a filesystem before unmounting it? There's no sensbible APi for that. Moreover this should be entirely unnecessary with correctly behaving services. It's just that you wrote a broken one... > We often see failing unmounts of several local or iscsi fs during > reboot, and in the support case we are currently working on with SuSE > failing iscsi fs even cause xfs I/O errors. So it might be a good idea > to have sth. like a lsof + kill before unmounting a filesystem, maybe > configurable with a flag to enable or disable it. Even if lsof or kill > failed, it wouldn't be worse than now. lsof is a) slow (it searches all of /proc), b) racy (because it won't properly grok fds coming and going), and c) incomplete (we live in a world if pidns these days). This is a hack on top of a hack really, let's not do that. > As far as I see there is no way to write a drop-in for a mount unit > that allows to execute commands before the unmount happens, is that > right? Sth. like "ExecPreUmount=" would help here, especially if there > was sth. like a umount@.service that would be called for every umount > with e.g. the mounpoint accessable with a variable. We didn't add that on purpose, since we wanted to make sure that what systemd does is mostly reproducible with a plain "mount" command on the shell... You can manually do this though, but it's a hack really: just write a service, order it After= the specific mount, and Before=local-fs.target. But it's going to be super racy, and a poor hack against missing ordering deps. Long story short: fix your deps, write proper units. Lennart -- Lennart Poettering, Berlin ___ systemd-devel mailing list systemd-devel@lists.freedesktop.org https://lists.freedesktop.org/mailman/listinfo/systemd-devel
Re: [systemd-devel] failing unmounts during reboot
Silvio Knizek wrote: the proper approach would be to define the dependency of the generated .service to the mount point with a drop-in and RequiresMountsFor=. See man:systemd.unit for more information. How would that work for init.d scripts? Also the systemd-sysv-generator can only do so much. 1) If the systemd-sysv-generator creates wrapper .service files on-the-fly, is it possible to hook into those with dropins? The man page didn't give much information. 2) If init.d scripts are wrapped by .service files, shouldn't the processes spawned by these scripts be killed when shutting down? For my example init.d script I see this with "sytemctl status bla": CGroup: /system.slice/bla.service └─9997 /usr/bin/sleep 99d But after "systemctl stop bla", the sleep process is still there. When I write a .service unit of type oneshot with RemainAfterExit=yes that calls a script that spawns another sleep process, I see the same structure: CGroup: /system.slice/blu.service └─24351 sleep 44d But after "systemctl stop blu" this sleep is gone. So why doesn't the .service wrapper for the init.d script kill the processes it has spawned? Is that a bug? That would likely solve most of the problems with init.d scripts even for iscsi mounts due to the ordering that can be done with $remote_fs. Please write yourself proper units if you encounter problems. I'm not writing init.d script, I always use units for my own stuff. I'm talking about init.d scripts that are still delivered with many software packages and that I wouldn't like or try to rewrite. cu, Frank -- Dipl.-Inform. Frank Steiner Web: http://www.bio.ifi.lmu.de/~steiner/ Lehrstuhl f. BioinformatikMail: http://www.bio.ifi.lmu.de/~steiner/m/ LMU, Amalienstr. 17 Phone: +49 89 2180-4049 80333 Muenchen, Germany Fax: +49 89 2180-99-4049 * Rekursion kann man erst verstehen, wenn man Rekursion verstanden hat. * ___ systemd-devel mailing list systemd-devel@lists.freedesktop.org https://lists.freedesktop.org/mailman/listinfo/systemd-devel
Re: [systemd-devel] failing unmounts during reboot
On Thu, Jul 25, 2019 at 12:49 PM Frank Steiner < fsteiner-ma...@bio.ifi.lmu.de> wrote: > Hi, > > I'm currently discussing a problem with the SuSE support about failing > unmounts during reboot. Tyring to debug this I realized that systemd > is not killing processes left over by some init.d script. E.g. use > the following script in /etc/init.d/ > > #!/bin/sh > # > ### BEGIN INIT INFO > # Provides: bla > # Required-Start: $network $remote_fs sshd > # Required-Stop: $network $remote_fs sshd > # Default-Start: 2 3 5 > # Description:test > ### END INIT INFO > case "$1" in > start) cd /test; /usr/bin/sleep 99d & ;; >stop) true;; > esac > > > On shutdown, unmounting /test will fail because the sleep process is > not killed. Shouldn't there be a mechanism in system to kill processes > spawned by LSB script when shutting these down? > I think that's a deliberate decision made in the systemd-sysv-generator. Note how the generated .service files have "KillMode=process" (and even "RemainAfterExit=yes"). The default for native services is to kill the entire cgroup, and IIRC that even was one of the main reasons for using cgroups. Most likely it's there to retain compatibility with some of the weirder init.d scripts – those which don't start any daemons; those which start several; and so on and so on. > > And moreover, wouldn't it make sense to have a mechanism to at least > try to kill all processes using a filesystem before unmounting it? > We often see failing unmounts of several local or iscsi fs during > reboot, and in the support case we are currently working on with SuSE > failing iscsi fs even cause xfs I/O errors. So it might be a good idea > to have sth. like a lsof + kill before unmounting a filesystem, maybe > configurable with a flag to enable or disable it. Even if lsof or kill > failed, it wouldn't be worse than now. > IIRC the idea is that user sessions are killed during shutdown before unmounting anything anyway, and services have implicit After=local-fs.target so they get stopped before local filesystems? The shutdown process is one area that I still know very little about. > > As far as I see there is no way to write a drop-in for a mount unit > that allows to execute commands before the unmount happens, is that > right? Sth. like "ExecPreUmount=" would help here, especially if there > was sth. like a umount@.service that would be called for every umount > with e.g. the mounpoint accessable with a variable. > No, there isn't. -- Mantas Mikulėnas ___ systemd-devel mailing list systemd-devel@lists.freedesktop.org https://lists.freedesktop.org/mailman/listinfo/systemd-devel
Re: [systemd-devel] failing unmounts during reboot
Am 25.07.19 um 12:10 schrieb Silvio Knizek: > the proper approach would be to define the dependency of the generated > .service to the mount point with a drop-in and RequiresMountsFor=. See > man:systemd.unit for more information. > Also the systemd-sysv-generator can only do so much. Please write > yourself proper units if you encounter problems FWIW: and do it with dropins in /etc/systemd/system/servicename.service.d/ something which was completly impossible with initscripts and even if they had some place to customize it had no sane concept of ordering ___ systemd-devel mailing list systemd-devel@lists.freedesktop.org https://lists.freedesktop.org/mailman/listinfo/systemd-devel
Re: [systemd-devel] failing unmounts during reboot
Am Donnerstag, den 25.07.2019, 11:40 +0200 schrieb Frank Steiner: > Hi, > > I'm currently discussing a problem with the SuSE support about > failing > unmounts during reboot. Tyring to debug this I realized that systemd > is not killing processes left over by some init.d script. E.g. use > the following script in /etc/init.d/ > > #!/bin/sh > # > ### BEGIN INIT INFO > # Provides: bla > # Required-Start: $network $remote_fs sshd > # Required-Stop: $network $remote_fs sshd > # Default-Start: 2 3 5 > # Description:test > ### END INIT INFO > case "$1" in > start) cd /test; /usr/bin/sleep 99d & ;; >stop) true;; > esac > > > On shutdown, unmounting /test will fail because the sleep process is > not killed. Shouldn't there be a mechanism in system to kill > processes > spawned by LSB script when shutting these down? > > And moreover, wouldn't it make sense to have a mechanism to at least > try to kill all processes using a filesystem before unmounting it? > We often see failing unmounts of several local or iscsi fs during > reboot, and in the support case we are currently working on with SuSE > failing iscsi fs even cause xfs I/O errors. So it might be a good > idea > to have sth. like a lsof + kill before unmounting a filesystem, maybe > configurable with a flag to enable or disable it. Even if lsof or > kill > failed, it wouldn't be worse than now. > > As far as I see there is no way to write a drop-in for a mount unit > that allows to execute commands before the unmount happens, is that > right? Sth. like "ExecPreUmount=" would help here, especially if > there > was sth. like a umount@.service that would be called for every umount > with e.g. the mounpoint accessable with a variable. > > cu, > Frank > Hi, the proper approach would be to define the dependency of the generated .service to the mount point with a drop-in and RequiresMountsFor=. See man:systemd.unit for more information. Also the systemd-sysv-generator can only do so much. Please write yourself proper units if you encounter problems. BR Silvio ___ systemd-devel mailing list systemd-devel@lists.freedesktop.org https://lists.freedesktop.org/mailman/listinfo/systemd-devel
[systemd-devel] failing unmounts during reboot
Hi, I'm currently discussing a problem with the SuSE support about failing unmounts during reboot. Tyring to debug this I realized that systemd is not killing processes left over by some init.d script. E.g. use the following script in /etc/init.d/ #!/bin/sh # ### BEGIN INIT INFO # Provides: bla # Required-Start: $network $remote_fs sshd # Required-Stop: $network $remote_fs sshd # Default-Start: 2 3 5 # Description:test ### END INIT INFO case "$1" in start) cd /test; /usr/bin/sleep 99d & ;; stop) true;; esac On shutdown, unmounting /test will fail because the sleep process is not killed. Shouldn't there be a mechanism in system to kill processes spawned by LSB script when shutting these down? And moreover, wouldn't it make sense to have a mechanism to at least try to kill all processes using a filesystem before unmounting it? We often see failing unmounts of several local or iscsi fs during reboot, and in the support case we are currently working on with SuSE failing iscsi fs even cause xfs I/O errors. So it might be a good idea to have sth. like a lsof + kill before unmounting a filesystem, maybe configurable with a flag to enable or disable it. Even if lsof or kill failed, it wouldn't be worse than now. As far as I see there is no way to write a drop-in for a mount unit that allows to execute commands before the unmount happens, is that right? Sth. like "ExecPreUmount=" would help here, especially if there was sth. like a umount@.service that would be called for every umount with e.g. the mounpoint accessable with a variable. cu, Frank -- Dipl.-Inform. Frank Steiner Web: http://www.bio.ifi.lmu.de/~steiner/ Lehrstuhl f. BioinformatikMail: http://www.bio.ifi.lmu.de/~steiner/m/ LMU, Amalienstr. 17 Phone: +49 89 2180-4049 80333 Muenchen, Germany Fax: +49 89 2180-99-4049 * Rekursion kann man erst verstehen, wenn man Rekursion verstanden hat. * ___ systemd-devel mailing list systemd-devel@lists.freedesktop.org https://lists.freedesktop.org/mailman/listinfo/systemd-devel
