Bug#785714: kexec-tools is broken when using systemd, danger of filesystem corruption
Hi, some additional information/thoughts: Systemds builtin kexec.target is indeed the indended way to do kexec reboots by the developers. See man systemctl: kexec Shut down and reboot the system via kexec. This is mostly equivalent to start kexec.target --irreversible, but also prints a wall message to all users. If combined with --force, shutdown of all running services is skipped, however all processes are killed and all file systems are unmounted or mounted read-only, immediately followed by the reboot. Thus I suggest using the intended way to do kexec reboots on systemd enabled installations. This also means breaking the old SysV behavior - when the current target is kexec, imho you should _always_ load a new kernel image, regardless of the LOAD_KEXEC value in /etc/default/kexec. As a first step, the SysV LSB loader script could be modified (with a slightly modified patch from my first report): --- debian/kexec-load.init.d.org2014-11-24 05:02:10.0 +0100 +++ debian/kexec-load.init.d2015-06-25 17:16:34.152465942 +0200 @@ -102,12 +102,14 @@ ;; stop) # If running systemd, we want kexec reboot only if current - # command is reboot + # target is kexec (determined via systemd-exec.service) if [ -d /run/systemd/system ]; then - systemctl list-jobs systemd-reboot.service | grep -q systemd-reboot.service + systemctl list-jobs systemd-kexec.service | grep -q systemd-kexec.service if [ $? -ne 0 ]; then exit 0 fi + # Override LOAD_EXEC option, because kexec target always implies kexec reboot + LOAD_KEXEC="true" fi do_stop ;; The SysV LSB kexec script also has to be modified, kexec invocation is done by systemd (when everything has been properly terminated/mounted read only), so it should be skipped in the LSB script when running systemd. If you don't skip it, you will end up with race conditions and possible data corruption (as explained earlier). +++ debian/kexec.init.d 2015-06-25 17:21:02.405402935 +0200 @@ -36,7 +36,10 @@ exit 3 ;; stop) - do_stop + # Systemd has its own kexec service (which will call the kexec binary), so skip, if running with systemd + if [ ! -d /run/systemd/system ]; then + do_stop + fi ;; *) echo "Usage: $0 start|stop" >&2 Both modifications should make kexec work reliably on systemd enabled installations, kexec reboots will have to be performed/triggered with 'systemctl kexec' (or 'systemctl start kexec.target'), other reboot methods will result in cold reboots, the value of LOAD_KEXEC in /etc/default/kexec is ignored. The old behavior will not change when running SysV as init system. As a second step, I would suggest creating a proper systemd unit file to do the kernel image loading, instead of relying on systemd-sysv to autogenerate it from the SysV LSB init script. Cheers, Thomas -- To UNSUBSCRIBE, email to debian-bugs-rc-requ...@lists.debian.org with a subject of "unsubscribe". Trouble? Contact listmas...@lists.debian.org
Bug#785714: kexec-tools is broken when using systemd, danger of filesystem corruption
Hi Khalid, first of all, thanks for looking into the issue. I am by far no systemd expert, so if I get anything wrong, please correct me :) I do think however, that I know enough to be certain that there is a problem with the package when utilizing systemd as init system. > Have you seen any evidence of corruption on a filesystem? Yes, I can reproduce unclean (un-)mounts on my testing machine (Xeon E3-1245, SSD) when using the original kexec-tools package on every reboot with kexec enabled: [...] [Sat Jun 20 00:05:32 2015] Adding 4194300k swap on /dev/mapper/vg-swap. Priority:-1 extents:1 across:4194300k FS [Sat Jun 20 00:05:32 2015] EXT4-fs (dm-4): 1 orphan inode deleted [Sat Jun 20 00:05:32 2015] EXT4-fs (dm-4): recovery complete [Sat Jun 20 00:05:32 2015] EXT4-fs (dm-4): mounted filesystem with ordered data mode. Opts: (null) [Sat Jun 20 00:05:32 2015] EXT4-fs (dm-3): 5 orphan inodes deleted [Sat Jun 20 00:05:32 2015] EXT4-fs (dm-3): recovery complete [...] Whereas I cannot reproduce this when using my patched package: [...] [Sat Jun 20 00:07:39 2015] EXT4-fs (dm-3): mounted filesystem with ordered data mode. Opts: usrjquota=aquota.user,jqfmt=vfsv1 [Sat Jun 20 00:07:39 2015] EXT4-fs (dm-4): mounted filesystem with ordered data mode. Opts: (null) [Sat Jun 20 00:07:39 2015] Adding 4194300k swap on /dev/mapper/vg-swap. Priority:-1 extents:1 across:4194300k FS [Sat Jun 20 00:07:39 2015] EXT4-fs (dm-5): mounted filesystem with ordered data mode. Opts: (null) [...] Reboot times are roughly the same, only with the current official Jessie package, I get unclean mounts all the time, which in turn suggests, that the filesystem has not been properly unmounted before kexec loaded the new kernel image. > /etc/init.d/kexec script header says: > > # X-Stop-After:umountroot > > which means stop target for /etc/init.d/kexec will not be called until > root fs has been unmounted, at which point all other filesystems must > have already been unmounted. When using SysV init, this is indeed true - however systemd seems to mask the umountroot/umountfs LSB init scripts, which means that they are not considered for dependencies at all. Thus the kexec LSB init script dependency is rendered useless: lrwxrwxrwx 1 root root 9 May 26 08:07 /lib/systemd/system/umountroot.service -> /dev/null lrwxrwxrwx 1 root root 9 May 26 08:07 /lib/systemd/system/umountfs.service -> /dev/null This can also bee seen when checking the unit file (auto-)generated by systemd-sysv: /run/systemd/generator.late/kexec-load.service: # Automatically generated by systemd-sysv-generator [Unit] SourcePath=/etc/init.d/kexec-load Description=LSB: Load kernel image with kexec Before=runlevel2.target runlevel3.target runlevel4.target runlevel5.target shutdown.target After=local-fs.target remote-fs.target kexec.service Conflicts=shutdown.target [Service] Type=forking Restart=no TimeoutSec=5min IgnoreSIGPIPE=no KillMode=process GuessMainPID=no RemainAfterExit=yes SysVStartPriority=2 ExecStart=/etc/init.d/kexec-load start ExecStop=/etc/init.d/kexec-load stop /run/systemd/generator.late/kexec.service: # Automatically generated by systemd-sysv-generator [Unit] SourcePath=/etc/init.d/kexec Description=LSB: Execute the kexec -e command to reboot system Before=runlevel2.target runlevel3.target runlevel4.target runlevel5.target shutdown.target Conflicts=shutdown.target [Service] Type=forking Restart=no TimeoutSec=5min IgnoreSIGPIPE=no KillMode=process GuessMainPID=no RemainAfterExit=yes SysVStartPriority=1 ExecStart=/etc/init.d/kexec start ExecStop=/etc/init.d/kexec stop So basically kexec.service depends on kexec-load.service to be stopped before itself, which is okay, because the kernel image needs to be loaded before doing the actual kexec reboot. However there are no other dependencies whatsoever on the filesystem or other services, which basically means that kexec.service will be in a race with other services being shut down at the same time. If you get lucky the other services have been stopped before kexec gets executed, otherwise you are not (and the observed inconsistencies may occur) :( > Rebooting the system with "reboot" calls the target reboot. That is correct. >If one invokes the target kexec directly, their intent is to kexec immediately > (same as exceuting "kexec -e" which does not do an orderly shutdown). > Orderly shutdown is meant to happen when reboot is called, not when > kernel is kexec'd explicitly by the user. I do not agree with this change. This however, is not. I guess you are confusing kexec.service (which is the autogenerated unit for your LSB kexec script) with the systemds' builtin kexec.target (which will be used when issuing 'systemctl kexec') - the according unit file can be found here: /lib/systemd/system/kexec.target: # This file is part of systemd. # # systemd is free software; you can redistribute it and/or modify it # under the terms of the GNU Lesser General Public License as published by #
Bug#785714: kexec-tools is broken when using systemd, danger of filesystem corruption
Package: kexec-tools Version: 1:2.0.7-5 Severity: grave Tags: patch Rebooting via kexec is currently broken when using the default init system of Jessie (systemd) and may result in filesystem corruption or other unwanted effects of improperly shutdown services. This bug does not apply when using the SysV init system. Reboots (e.g. via 'reboot', 'systemctl kexec' or 'systemctl reboot') will cause the LSB SysV scripts of kexec-tools (/etc/init.d/kexec-load and /etc/init.d/kexec) to be stopped BEFORE reaching the systemd kexec target (and other systemd targets), which means that other LSB init scripts and/or systemd services will be in a race condition with the LSB kexec script and the filesystems may not be properly unmounted beforce /etc/init.d/kexec calls /sbin/kexec -e. As an effect of this, after rebooting via kexec, one may often observe orphaned inodes or other filesystem incosistencies, especially on a busy machine with heavy IO. There also seems to be logical error in the /etc/init.d/kexec-load LSB script, because the kexec image will only get loaded when the systemd target is reboot - when the target is kexec, the image won't be loaded and thus a normal reboot will occur. The following patch for the source package should fix this and make rebooting via 'systemctl kexec' possible (when /etc/default/kexec has LOAD_KEXEC enabled), 'reboot' or 'systemctl reboot' will result in normal reboots. --- debian/kexec.init.d.org 2014-07-25 19:03:25.0 +0200 +++ debian/kexec.init.d 2015-05-19 14:26:40.680256999 +0200 @@ -36,7 +36,10 @@ exit 3 ;; stop) - do_stop + # Only execute, when running with SysV, systemd has its own kexec target + if [ ! -d /run/systemd/system ] ; then + do_stop + fi ;; *) echo "Usage: $0 start|stop" >&2 --- debian/kexec-load.init.d.org2014-11-24 05:02:10.0 +0100 +++ debian/kexec-load.init.d2015-05-19 14:27:43.431537728 +0200 @@ -102,9 +102,9 @@ ;; stop) # If running systemd, we want kexec reboot only if current - # command is reboot + # target is kexec if [ -d /run/systemd/system ]; then - systemctl list-jobs systemd-reboot.service | grep -q systemd-reboot.service + systemctl list-jobs systemd-kexec.service | grep -q systemd-kexec.service if [ $? -ne 0 ]; then exit 0 fi -- System Information: Debian Release: 8.0 APT prefers stable-updates APT policy: (500, 'stable-updates'), (500, 'stable') Architecture: amd64 (x86_64) Foreign Architectures: i386 Kernel: Linux 3.16.0-4-amd64 (SMP w/8 CPU cores) Locale: LANG=en_US.UTF-8, LC_CTYPE=en_US.UTF-8 (charmap=UTF-8) Shell: /bin/sh linked to /bin/dash Init: systemd (via /run/systemd/system) Versions of packages kexec-tools depends on: ii debconf [debconf-2.0] 1.5.56 ii libc6 2.19-18 kexec-tools recommends no packages. kexec-tools suggests no packages. -- debconf information: * kexec-tools/load_kexec: true kexec-tools/use_grub_config: false -- To UNSUBSCRIBE, email to debian-bugs-rc-requ...@lists.debian.org with a subject of "unsubscribe". Trouble? Contact listmas...@lists.debian.org
Bug#715569: Official mod_vroot lib broken, lib built from source works
Package: proftpd-mod-vroot Version: 0.9.2-2+b2 Severity: grave We are currently using the amd64 Debian Squeeze version of mod_vroot for ProFTPD on some of our servers and wanted to upgrade to Wheezy, however we encountered problems - the Wheezy Version of mod_vroot did not seem to work at all, it verifiably gets loaded, but something is broken, so the modules vroot restrictions do not apply at all. We tried to debug the issue and ran into an oddity regarding the current official Debian package for proftpd-mod-vroot: If we build the package 1:1 from Debian sources via dpkg-buildpackage, we get a slightly different/larger binary than the official binary, and this self-built binary then works as expected. Our steps to build the package (on a plain Wheezy amd64 box with installed/ up-to-date proftpd-dev package): wget http://ftp.de.debian.org/debian/pool/main/p/proftpd-mod-vroot/proftpd-mod-vroot_0.9.2-2.dsc http://ftp.de.debian.org/debian/pool/main/p/proftpd-mod-vroot/proftpd-mod-vroot_0.9.2.orig.tar.gz http://ftp.de.debian.org/debian/pool/main/p/proftpd-mod-vroot/proftpd-mod-vroot_0.9.2-2.debian.tar.gz dpkg-source -x *.dsc; cd proftpd-mod-vroot-0.9.2 dpkg-buildpackage -us -uc -rfakeroot The resulting binary size of mod_vroot.so is 28160, current official (0.9.2-2+b2) binary size: 28080 Is it possible that the officially shipped binary is broken or was built with an older ProFTPD version? -- System Information: Debian Release: 7.1 APT prefers stable APT policy: (500, 'stable') Architecture: amd64 (x86_64) Foreign Architectures: i386 Kernel: Linux 3.2.0-4-amd64 (SMP w/8 CPU cores) Locale: LANG=en_US.UTF-8, LC_CTYPE=en_US.UTF-8 (charmap=UTF-8) Shell: /bin/sh linked to /bin/bash Versions of packages proftpd-mod-vroot depends on: ii libc6 2.13-38 ii proftpd-basic [proftpd-abi-1.3.4a] 1.3.4a-4+nmu1 proftpd-mod-vroot recommends no packages. proftpd-mod-vroot suggests no packages. -- no debconf information -- To UNSUBSCRIBE, email to debian-bugs-rc-requ...@lists.debian.org with a subject of "unsubscribe". Trouble? Contact listmas...@lists.debian.org