Here is a recap of my work and current status : #1) when a system has a separated /var, the unit will hang since it is looking for /var/run to be present and it has been unmounted.
#2) when using Unattended-Upgrade::InstallOnShutdown "true"; the upgrade never completes as the query to the online archive fails since the network is no longer available. #3) it is impossible to enable the unattended-upgrades.service unit. Here is an example : > $ systemctl status unattended-upgrades.service > ● unattended-upgrades.service - Unattended Upgrades Shutdown > Loaded: loaded (/lib/systemd/system/unattended-upgrades.service; enabled; > vendor preset: enabled) > Active: inactive (dead) > Docs: man:unattended-upgrade(8) According to the doc[2], during shutdown, even if Before= or After= is used, the unit being started will only start after the Shutdown of its dependencies. Prior to the recent addition of local-fs.target network.target, the unit ran quickly enough for the /var to be still mounted. But even without these dependencies, InstallOnShutdown would fail with the following : > 2017-03-10 13:40:42,803 INFO Starting unattended upgrades script > 2017-03-10 13:40:42,803 INFO Allowed origins are: ['o=Ubuntu,a=zesty', > 'o=Ubuntu,a=zesty-security'] > 2017-03-10 13:41:40,554 ERROR An error occurred: 'Cannot initiate the > connection to 192.168.200.3:8000 (192.168.200.3). - connect (101: Network is > unreachable)' > 2017-03-10 13:41:40,555 ERROR The URI > 'http://fr.archive.ubuntu.com/ubuntu/pool/main/i/init-system-helpers/init-system-helpers_1.47_all.deb' > failed to download, aborting When trying to switch the unit to an ExecStop=, we find that the Stop never runs. This is caused by the fact that the unit is disabled (#3). Trying to enable the unit leads to : > # systemctl enable unattended-upgrades > Synchronizing state of unattended-upgrades.service with SysV service script > with /lib/systemd/systemd-sysv-install. > Executing: /lib/systemd/systemd-sysv-install enable unattended-upgrades > update-rc.d: error: unattended-upgrades Default-Start contains no runlevels, > aborting. Adding runlevels 2 3 4 5 fixes this then the unit can be enabled. So we get to a unit that looks like this : > [Unit] > Description=Unattended Upgrades Shutdown > DefaultDependencies=no > After=network.target local-fs.target > RequiresMountsFor=/var/run /var/log > Documentation=man:unattended-upgrade(8) > > [Service] > Type=oneshot > RemainAfterExit=yes > ExecStop=/usr/share/unattended-upgrades/unattended-upgrade-shutdown > TimeoutStopSec=900 > > [Install] > WantedBy=multi-user.target Before= is replaced by After= as, during shutdown otherwise the unit does not run. RequiresMountsFor= is added as both /var/log and /var/run are needed in order to run correctly. RemainAfterExit=yes is added so the unit appears as started. There is no longer a requirement to have an ExecStart present. WantedBy is switched to multi-user.target as on the way up, we do nothing and we are no longer depending on anything related to shutdown. Now this only works IF /var is a separate FS. The reason for that is the presence of DefaultDependencies=no. I don't think that it is required but was there in the initial unit. Removing it in the final unit fixes the only remaining issue. The unit is now : > [Unit] > Description=Unattended Upgrades Shutdown > After=network.target local-fs.target > RequiresMountsFor=/var/run /var/log > Documentation=man:unattended-upgrade(8) > > [Service] > Type=oneshot > RemainAfterExit=yes > ExecStop=/usr/share/unattended-upgrades/unattended-upgrade-shutdown > TimeoutStopSec=900 > > [Install] > WantedBy=multi-user.target This works correctly and has been tested on : - Xenial with and without /var as a separate FS - Zesty with and without /var as a separate FS InstallOnShutdown now also works as advertized. I am now getting confirmation on that change since it is a rather sensible modification. -- You received this bug notification because you are a member of Ubuntu Touch seeded packages, which is subscribed to unattended-upgrades in Ubuntu. https://bugs.launchpad.net/bugs/1654600 Title: unattended-upgrade-shutdown hangs when /var is a separate filesystem Status in unattended-upgrades package in Ubuntu: In Progress Status in unattended-upgrades source package in Xenial: Triaged Status in unattended-upgrades source package in Yakkety: Triaged Status in unattended-upgrades package in Debian: New Bug description: [SRU justification] This fix is needed to make sure that the system does not hang on shutdown when /var is a speparate file system [Impact] System can hang up to 10 minutes if not fixed. [Fix] Change the systemd unit's ExecStart to an ExecStop so the unit is correctly sequenced. [Test Case] In a VM with /var separated from the / file system, shutdown the VM. It will hang for 10 minutes [Regression] None to be expected. A ExecStop unit will be sequenced before the Before= units which is earlier than previously. [Original description of the problem] The systemd unit file unattended-upgrades.service is used to stop a running unattended-upgrade process during shutdown. This unit file is running together with all filesystem unmount services. The unattended-upgrades service checks if the lockfile for unattended-upgrade (in /var/run) exists, and if it does, there is an unattended-upgrade in progress and the service will wait until it finishes (and therefore automatically wait at shutdown). However, if /var is a separate filesystem, it will get unmounted even though /var/run is a tmpfs that's still mounted on top of the /var/run directory in the /var filesystem. The unattended-upgrade script will fail to find lockfile, sleeps for 5 seconds, and tries to check the lockfile again. After 10 minutes (the default timeout), it will finally exit and the system will continue shutdown. The problem is the error handling in /usr/share/unattended-upgrades/unattended-upgrade-shutdown where it tries to lock itself: while True: res = apt_pkg.get_lock(options.lock_file) logging.debug("get_lock returned %i" % res) # exit here if there is no lock if res > 0: logging.debug("lock not taken") break lock_was_taken = True The function apt_pkg.get_lock() either returns a file descriptor, or -1 on an error. File descriptors are just C file descriptors, so they are always positive integers. The code should check the result to be negative, not positive. I have attached a patch to reverse the logic. Additional information: 1) Description: Ubuntu 16.04.1 LTS Release: 16.04 2) unattended-upgrades: Installed: 0.90ubuntu0.3 Candidate: 0.90ubuntu0.3 Version table: *** 0.90ubuntu0.3 500 500 http://nl.archive.ubuntu.com/ubuntu xenial-updates/main amd64 Packages 500 http://nl.archive.ubuntu.com/ubuntu xenial-updates/main i386 Packages 100 /var/lib/dpkg/status 0.90 500 500 http://nl.archive.ubuntu.com/ubuntu xenial/main amd64 Packages 500 http://nl.archive.ubuntu.com/ubuntu xenial/main i386 Packages 3) Fast reboot 4) Very slow reboot (after a 10 minutes timeout) To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/unattended-upgrades/+bug/1654600/+subscriptions -- Mailing list: https://launchpad.net/~touch-packages Post to : touch-packages@lists.launchpad.net Unsubscribe : https://launchpad.net/~touch-packages More help : https://help.launchpad.net/ListHelp
