[Bug 1924494] Re: Systemd shutdown order unmounts filesystem used by qemu/kvm guest before shutting down the guest
> And by the way, I just tested the old setup with your full > recommendation, adding the BEFORE= statement and it finally stopped the > I/O errors. I can now take the time to transition properly. Awesome, glad I could help! And it might help others who find this bug if affected by a similar case. ** Changed in: libvirt (Ubuntu) Status: Incomplete => Invalid -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1924494 Title: Systemd shutdown order unmounts filesystem used by qemu/kvm guest before shutting down the guest To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/libvirt/+bug/1924494/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1924494] Re: Systemd shutdown order unmounts filesystem used by qemu/kvm guest before shutting down the guest
And by the way, I just tested the old setup with your full recommendation, adding the BEFORE= statement and it finally stopped the I/O errors. I can now take the time to transition properly. Thank you! Pedro -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1924494 Title: Systemd shutdown order unmounts filesystem used by qemu/kvm guest before shutting down the guest To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/libvirt/+bug/1924494/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1924494] Re: Systemd shutdown order unmounts filesystem used by qemu/kvm guest before shutting down the guest
** Attachment added: "systemd-analyze dot graph" https://bugs.launchpad.net/ubuntu/+source/libvirt/+bug/1924494/+attachment/5492912/+files/USRgraph.svg -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1924494 Title: Systemd shutdown order unmounts filesystem used by qemu/kvm guest before shutting down the guest To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/libvirt/+bug/1924494/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1924494] Re: Systemd shutdown order unmounts filesystem used by qemu/kvm guest before shutting down the guest
** Attachment added: "Services, hook, and journal after adding AFTER= and REQUIRES=" https://bugs.launchpad.net/ubuntu/+source/libvirt/+bug/1924494/+attachment/5492904/+files/buglog.txt -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1924494 Title: Systemd shutdown order unmounts filesystem used by qemu/kvm guest before shutting down the guest To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/libvirt/+bug/1924494/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1924494] Re: Systemd shutdown order unmounts filesystem used by qemu/kvm guest before shutting down the guest
Christian I set this up back when without properly researching systemd.mount, particularly the automatic dependencies. I make half of the changes you recommended (Requires=virt-guest-shutdown.target; After=virt-guest- shutdown.target) and it made no change, the I/O errors still showed up. I'm attaching the custom systemd services and the libvirt hook for your information. But, instead of following in the direction of a custom setup I will transition this to /etc/fstab parent/child mounts (the child mount is a .vhd file on the parent fs that will now be mounted using the loop device instead of nbd) and a simpler libvirt hook with mount/umount functionality. Sorry for making this more complicated than they should have been. Regards Pedro -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1924494 Title: Systemd shutdown order unmounts filesystem used by qemu/kvm guest before shutting down the guest To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/libvirt/+bug/1924494/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1924494] Re: Systemd shutdown order unmounts filesystem used by qemu/kvm guest before shutting down the guest
For your experiments, here a few examples 1. using a mount unit for ntfs https://brunofontes.net/enu/post/mounting-partitions-systemd 2. doing disk setup with a service https://unix.stackexchange.com/questions/608431/systemd-fix-ordering-cycle-when-mounting-an-ntfs-volume-on-a-legacy-raid-array General help on mount units https://www.freedesktop.org/software/systemd/man/systemd.mount.html And as I said, for this case once you have such unites you can control them fine grained via: a) be completely shut down before libvirt can end (not what you want actually as that can be running concurrent to libvirt-guest service - this is better for other things to do before guests shut down - like swap scrubbing for secrecy or such) Requires=virt-guest-shutdown.target After=virt-guest-shutdown.target b) Ensure to be started before libvirt and even more important, only end after them on shutdown (reverse order) Before=libvirtd.service libvirt-guests.service -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1924494 Title: Systemd shutdown order unmounts filesystem used by qemu/kvm guest before shutting down the guest To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/libvirt/+bug/1924494/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1924494] Re: Systemd shutdown order unmounts filesystem used by qemu/kvm guest before shutting down the guest
Thanks, In your log I see that stopping remote file systems and libvirtd properly was ordered AFTER libvirt-guests tried to shut down the guest. I've checked this in a few ways 1. libvirt/libvirt-guests vs remote fs (e.g. NFS) 2. libvirt/libvirt-guests vs NBD #1 Remote-FS - well ordered Apr 23 14:37:02 ps01ubx libvirt-guests.sh[51404]: Starting shutdown on guest: psw10x64 .. Apr 23 14:39:00 ps01ubx libvirt-guests.sh[51071]: Waiting for 1 guests to shut down, 5 seconds left Apr 23 14:39:04 ps01ubx libvirt-guests.sh[52102]: Timeout expired while shutting down domains Then libvirtd.service stops/fails and eventually Apr 23 14:39:05 ps01ubx systemd[1]: Stopped target Remote File Systems. So this isn't totally looking as I first assumed which was 1. NFS with the NTFS images on it disconnected 2. shutdown guest (fails due to missing NFS) Since the shutdown of libvirt-guests has a 120s timeout that is like a bit of breathing room in the shutdown. A lot happens before, a lot happens after - but we see that there is nothing concurrently going on while waiting for the guest shutdown to complete. What we see before is the I/O errors that you mentioned. 19221 Apr 23 14:37:04 ps01ubx kernel: block nbd0: NBD_DISCONNECT 19222 Apr 23 14:37:04 ps01ubx kernel: block nbd0: Disconnected due to user request. 19223 Apr 23 14:37:04 ps01ubx kernel: block nbd0: shutting down sockets ... 19229 Apr 23 14:37:04 ps01ubx kernel: blk_update_request: I/O error, dev nbd0, sector 1637875585 op 0x0:(READ) flags 0x0 phys_seg 1 prio class 0 #2 NBD - too early Since the rest is behaving well it might be more the question "why is NTFS exiting early". # Set up via: Apr 22 12:00:02 ps01ubx ntfs-3g[2059]: Mounted /dev/nbd0p2 (Read-Write, label "psData", NTFS 3.1) # guest shutdown starts Apr 23 14:37:02 ps01ubx libvirt-guests.sh[51404]: Starting shutdown on guest: psw10x64 # NBD shut down via Apr 23 14:37:04 ps01ubx kernel: block nbd0: NBD_DISCONNECT # Then I/O errors happen Apr 23 14:37:04 ps01ubx kernel: blk_update_request: I/O error, dev nbd0, sector 1637875584 op 0x0:(READ) flags 0x80700 phys_seg 1 prio class 0 So it could be that NBD disconnect just waited until the guest shutdown was "kicked" but not until it completed? The log does not contain via which config/service you have configured your NTFS/NBD. It is just a old style /etc/fstab entry or anything more explicit in regard to ordering control like a .mount target or a custom .service? Could you please outline in detail the config that you use for that NBD/NTFS setup? If you have a target/unit/service for the NTFS . If you have any kind of such could you add to your config Requires=virt-guest-shutdown.target After=virt-guest-shutdown.target That would be what you add to anything that you want to have done/ordered into libvirt shutdown. That should ensure correct order. Waiting to see how your NTFS setup looks like ... -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1924494 Title: Systemd shutdown order unmounts filesystem used by qemu/kvm guest before shutting down the guest To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/libvirt/+bug/1924494/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
Re: [Bug 1924494] Re: Systemd shutdown order unmounts filesystem used by qemu/kvm guest before shutting down the guest
Hi Christian, Thanks for your attention. I replicated the issue and here is the journal log for today. Regards Pedro Serrano On Wed, 2021-04-21 at 11:08 +, Christian Ehrhardt wrote: > Hi Pedro, > > > Wouldn't it be beneficial for all guests to be shutdown before > unmounting all filesystems? > > Yes on first sight that looks like a compelling suggestion. > But i've learned the hard way that usually adding dependencies > eventually breaks more things than it fixes - so let us first try to > understand what to expect and what happened. > > > The ordering at the moment includes: > - libvirt-guests.service > - virt-guest-shutdown.target > - libvirtd.service > > Of these the "big one" is libvirtd.service which has > After=network.target > After=local-fs.target > After=remote-fs.target > And many more which imply further lateness on startup. > > libvirt-guests.service is what shuts down the guests and starts as > After=libvirtd.service > and thereby after all of the above. > > virt-guest-shutdown.target exists to allow people to hook arbitrary > things into this chain in case any thing needs to be done. > Note: In most cases it is better t identify what a local system has > that makes it special and set it up to go down before this. See the > discussion and references from > https://bugzilla.redhat.com/show_bug.cgi?id=1401054 for this. > > On shutdown usually the order of these is reversed - I'd expect that > libvirt-guests.service needs to stop before libvirtd.service stops > (which also is reasonable as it needs to interact with libvirt). > And libvirt.service should need to be passed to leave the > local/remote-FS targets on the way out, but targets might be more > special in this regard. > > Therefore the question becomes - which FS or FS-target was > stopped/cut in your case before libvirt.service was stopped. > Could you maybe share the journal of that boot so that we can try to > dissect what was going down beofore it? > > You can access those with > $ journalctl --list-boots > And then select the right one and redirect it into a file like > $ journalctl -b > logfile > > ** Bug watch added: Red Hat Bugzilla #1401054 > https://bugzilla.redhat.com/show_bug.cgi?id=1401054 > > ** Changed in: libvirt (Ubuntu) > Status: New => Incomplete > ** Attachment added: "ps01ubx-journalctl-2021-04-23-1441.txt" https://bugs.launchpad.net/bugs/1924494/+attachment/5491583/+files/ps01ubx-journalctl-2021-04-23-1441.txt -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1924494 Title: Systemd shutdown order unmounts filesystem used by qemu/kvm guest before shutting down the guest To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/libvirt/+bug/1924494/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
Re: [Bug 1924494] Re: Systemd shutdown order unmounts filesystem used by qemu/kvm guest before shutting down the guest
Sure can. I'm away but should have access to that laptop tomorrow afternoon. Would it matter if I re-created the conditions and sent you that journal? On Apr 21, 2021, 07:21, at 07:21, "Christian Ehrhardt " <1924...@bugs.launchpad.net> wrote: >Hi Pedro, > >> Wouldn't it be beneficial for all guests to be shutdown before >unmounting all filesystems? > >Yes on first sight that looks like a compelling suggestion. >But i've learned the hard way that usually adding dependencies >eventually breaks more things than it fixes - so let us first try to >understand what to expect and what happened. > > >The ordering at the moment includes: >- libvirt-guests.service >- virt-guest-shutdown.target >- libvirtd.service > >Of these the "big one" is libvirtd.service which has > After=network.target > After=local-fs.target > After=remote-fs.target >And many more which imply further lateness on startup. > >libvirt-guests.service is what shuts down the guests and starts as > After=libvirtd.service >and thereby after all of the above. > >virt-guest-shutdown.target exists to allow people to hook arbitrary >things into this chain in case any thing needs to be done. >Note: In most cases it is better t identify what a local system has >that makes it special and set it up to go down before this. See the >discussion and references from >https://bugzilla.redhat.com/show_bug.cgi?id=1401054 for this. > >On shutdown usually the order of these is reversed - I'd expect that >libvirt-guests.service needs to stop before libvirtd.service stops >(which also is reasonable as it needs to interact with libvirt). >And libvirt.service should need to be passed to leave the >local/remote-FS targets on the way out, but targets might be more >special in this regard. > >Therefore the question becomes - which FS or FS-target was stopped/cut >in your case before libvirt.service was stopped. >Could you maybe share the journal of that boot so that we can try to >dissect what was going down beofore it? > >You can access those with >$ journalctl --list-boots >And then select the right one and redirect it into a file like >$ journalctl -b > logfile > >** Bug watch added: Red Hat Bugzilla #1401054 > https://bugzilla.redhat.com/show_bug.cgi?id=1401054 > >** Changed in: libvirt (Ubuntu) > Status: New => Incomplete > >-- >You received this bug notification because you are subscribed to the >bug >report. >https://bugs.launchpad.net/bugs/1924494 > >Title: > Systemd shutdown order unmounts filesystem used by qemu/kvm guest > before shutting down the guest > >Status in libvirt package in Ubuntu: > Incomplete > >Bug description: > Last night I started a reboot without manually shutting down a > qemu/kvm guest first. After X went down a bunch of I/O errors > associated with a network block device that contains a NTFS filesystem > showed up on the console. I noticed that the umount.target had been > reached but the virt-guest-shutdown.target was still waiting for the > guest to shut down, until it timed out. > > Wouldn't it be beneficial for all guests to be shutdown before > unmounting all filesystems? > > ProblemType: Bug > DistroRelease: Ubuntu 20.10 > Package: systemd 246.6-1ubuntu1.3 > ProcVersionSignature: Ubuntu 5.8.0-49.55-generic 5.8.18 > Uname: Linux 5.8.0-49-generic x86_64 > ApportVersion: 2.20.11-0ubuntu50.5 > Architecture: amd64 > CasperMD5CheckResult: skip > Date: Thu Apr 15 08:35:48 2021 > InstallationDate: Installed on 2021-01-17 (88 days ago) >InstallationMedia: Ubuntu 20.04.1 LTS "Focal Fossa" - Release amd64 >(20200731) > MachineType: HP HP ZBook 15u G6 >ProcKernelCmdLine: BOOT_IMAGE=/boot/vmlinuz-5.8.0-49-generic >root=UUID=25df7fed-1baf-421e-941e-7f7c280db6e8 ro iommu=pt >intel_iommu=on default_hugepagesz=1G hugepagesz=1G hugepages=6 >pcie_aspm.policy=performance > SourcePackage: systemd > UpgradeStatus: Upgraded to groovy on 2021-04-08 (6 days ago) > dmi.bios.date: 01/08/2021 > dmi.bios.release: 8.1 > dmi.bios.vendor: HP > dmi.bios.version: R70 Ver. 01.08.01 > dmi.board.name: 8549 > dmi.board.vendor: HP > dmi.board.version: KBC Version 52.67.00 > dmi.chassis.type: 10 > dmi.chassis.vendor: HP > dmi.ec.firmware.release: 82.103 >dmi.modalias: >dmi:bvnHP:bvrR70Ver.01.08.01:bd01/08/2021:br8.1:efr82.103:svnHP:pnHPZBook15uG6:pvr:rvnHP:rn8549:rvrKBCVersion52.67.00:cvnHP:ct10:cvr: > dmi.product.family: 103C_5336AN HP ZBook > dmi.product.name: HP ZBook 15u G6 > dmi.product.sku: 9AP56LP#ABA > dmi.sys.vendor: HP > >To manage notifications about this bug go to: >https://bugs.launchpad.net/ubuntu/+source/libvirt/+bug/1924494/+subscriptions -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1924494 Title: Systemd shutdown order unmounts filesystem used by qemu/kvm guest before shutting down the guest To manage notifications about this bug go to:
[Bug 1924494] Re: Systemd shutdown order unmounts filesystem used by qemu/kvm guest before shutting down the guest
Hi Pedro, > Wouldn't it be beneficial for all guests to be shutdown before unmounting all filesystems? Yes on first sight that looks like a compelling suggestion. But i've learned the hard way that usually adding dependencies eventually breaks more things than it fixes - so let us first try to understand what to expect and what happened. The ordering at the moment includes: - libvirt-guests.service - virt-guest-shutdown.target - libvirtd.service Of these the "big one" is libvirtd.service which has After=network.target After=local-fs.target After=remote-fs.target And many more which imply further lateness on startup. libvirt-guests.service is what shuts down the guests and starts as After=libvirtd.service and thereby after all of the above. virt-guest-shutdown.target exists to allow people to hook arbitrary things into this chain in case any thing needs to be done. Note: In most cases it is better t identify what a local system has that makes it special and set it up to go down before this. See the discussion and references from https://bugzilla.redhat.com/show_bug.cgi?id=1401054 for this. On shutdown usually the order of these is reversed - I'd expect that libvirt-guests.service needs to stop before libvirtd.service stops (which also is reasonable as it needs to interact with libvirt). And libvirt.service should need to be passed to leave the local/remote-FS targets on the way out, but targets might be more special in this regard. Therefore the question becomes - which FS or FS-target was stopped/cut in your case before libvirt.service was stopped. Could you maybe share the journal of that boot so that we can try to dissect what was going down beofore it? You can access those with $ journalctl --list-boots And then select the right one and redirect it into a file like $ journalctl -b > logfile ** Bug watch added: Red Hat Bugzilla #1401054 https://bugzilla.redhat.com/show_bug.cgi?id=1401054 ** Changed in: libvirt (Ubuntu) Status: New => Incomplete -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1924494 Title: Systemd shutdown order unmounts filesystem used by qemu/kvm guest before shutting down the guest To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/libvirt/+bug/1924494/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1924494] Re: Systemd shutdown order unmounts filesystem used by qemu/kvm guest before shutting down the guest
** Package changed: systemd (Ubuntu) => libvirt (Ubuntu) -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1924494 Title: Systemd shutdown order unmounts filesystem used by qemu/kvm guest before shutting down the guest To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/libvirt/+bug/1924494/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
