This is output after reinstating the fix journalctl -b 0 | grep -i -e "\[drm\] Initialized" -e "Starting libvirtd.service" Jul 18 20:31:17 black kernel: [drm] Initialized simpledrm 1.0.0 20200625 for simple-framebuffer.0 on minor 0 Jul 18 20:31:21 black kernel: [drm] Initialized amdgpu 3.57.0 20150101 for 0000:03:00.0 on minor 1 Jul 18 20:31:21 black kernel: [drm] Initialized amdgpu 3.57.0 20150101 for 0000:6d:00.0 on minor 0 Jul 18 20:31:36 black systemd[1]: Starting libvirtd.service - libvirt legacy monolithic daemon...
the delay is very large; there is no longer a sleep added. tim@black:~$ sudo systemctl status libvirtd ● libvirtd.service - libvirt legacy monolithic daemon Loaded: loaded (/usr/lib/systemd/system/libvirtd.service; enabled; preset: enabled) Drop-In: /etc/systemd/system/libvirtd.service.d └─override.conf Active: active (running) since Thu 2024-07-18 20:31:36 AEST; 2min 34s ago TriggeredBy: ● libvirtd-admin.socket ● libvirtd-ro.socket ● libvirtd.socket Docs: man:libvirtd(8) https://libvirt.org/ Main PID: 8954 (libvirtd) Tasks: 27 (limit: 32768) Memory: 34.5M (peak: 39.8M) CPU: 358ms CGroup: /system.slice/libvirtd.service ├─8954 /usr/sbin/libvirtd --timeout 120 ├─9079 /usr/sbin/dnsmasq --conf-file=/var/lib/libvirt/dnsmasq/default.conf --leasefile-ro --dhcp-script=/usr/lib/libvirt/libvirt_leaseshelper ├─9080 /usr/sbin/dnsmasq --conf-file=/var/lib/libvirt/dnsmasq/default.conf --leasefile-ro --dhcp-script=/usr/lib/libvirt/libvirt_leaseshelper └─9189 /usr/libexec/virtiofsd --fd=34 -o source=/home/tim/Downloads Jul 18 20:31:36 black systemd[1]: Started libvirtd.service - libvirt legacy monolithic daemon. Jul 18 20:31:36 black dnsmasq[9079]: started, version 2.90 cachesize 150 Jul 18 20:31:36 black dnsmasq[9079]: compile time options: IPv6 GNU-getopt DBus no-UBus i18n IDN2 DHCP DHCPv6 no-Lua TFTP conntrack ipset nftset auth cryptohas> Jul 18 20:31:36 black dnsmasq-dhcp[9079]: DHCP, IP range 192.168.122.2 -- 192.168.122.254, lease time 1h Jul 18 20:31:36 black dnsmasq-dhcp[9079]: DHCP, sockets bound exclusively to interface virbr0 Jul 18 20:31:36 black dnsmasq[9079]: reading /etc/resolv.conf Jul 18 20:31:36 black dnsmasq[9079]: using nameserver 127.0.0.53#53 Jul 18 20:31:36 black dnsmasq[9079]: read /etc/hosts - 8 names Jul 18 20:31:36 black dnsmasq[9079]: read /var/lib/libvirt/dnsmasq/default.addnhosts - 0 names Jul 18 20:31:36 black dnsmasq-dhcp[9079]: read /var/lib/libvirt/dnsmasq/default.hostsfile On Thu, 18 Jul 2024 at 20:27, Tim Richardson <t...@tim-richardson.net> wrote: > This is a 22.04 upgraded to 24.04. I never once saw this behaviour prior > to the upgrade (I always use this VM, it is my development environment, so > I would have noticed). > > However, I was not using Ubuntu kernels with 22.04, I was using liquorix. > Currently I am on a stock Ubuntu 24.04 kernel. > > The ten second delay is not needed, it seems. I was just being very > cautious (or silly). I removed it. > > > It is not passthrough, it is an Ubuntu guest using libvirt graphics, > shared memory, 3d acceleration. > The hardware is a PC with AMD Ryzen4 7700X and one AMD graphics card. > > > this is the case without the fix, when autostart failed: > > > journalctl -b 0 | grep -i -e "\[drm\] Initialized" -e "Starting > libvirtd.service" > Jul 18 20:10:20 black kernel: [drm] Initialized simpledrm 1.0.0 20200625 > for simple-framebuffer.0 on minor 0 > Jul 18 20:10:23 black systemd[1]: Starting libvirtd.service - libvirt > legacy monolithic daemon... > Jul 18 20:10:25 black kernel: [drm] Initialized amdgpu 3.57.0 20150101 for > 0000:03:00.0 on minor 1 > Jul 18 20:10:25 black kernel: [drm] Initialized amdgpu 3.57.0 20150101 for > 0000:6d:00.0 on minor 0 > > > A second run: > > tim@black:~$ journalctl -b 0 | grep -i -e "\[drm\] Initialized" -e > "Starting libvirtd.service" > Jul 18 20:19:54 black kernel: [drm] Initialized simpledrm 1.0.0 20200625 > for simple-framebuffer.0 on minor 0 > Jul 18 20:19:57 black systemd[1]: Starting libvirtd.service - libvirt > legacy monolithic daemon... > Jul 18 20:19:59 black kernel: [drm] Initialized amdgpu 3.57.0 20150101 for > 0000:03:00.0 on minor 1 > Jul 18 20:19:59 black kernel: [drm] Initialized amdgpu 3.57.0 20150101 for > 0000:6d:00.0 on minor 0 > > > DRI > > tim@black:~$ ll /dev/dri > total 0 > drwxr-xr-x 3 root root 140 Jul 18 20:19 ./ > drwxr-xr-x 22 root root 6680 Jul 18 20:20 ../ > drwxr-xr-x 2 root root 120 Jul 18 20:19 by-path/ > crw-rw----+ 1 root video 226, 0 Jul 18 20:19 card0 > crw-rw----+ 1 root video 226, 1 Jul 18 20:19 card1 > crw-rw----+ 1 root render 226, 128 Jul 18 20:19 renderD128 > crw-rw----+ 1 root render 226, 129 Jul 18 20:19 renderD129 > > > $ systemctl list-units --all --type device | grep dri > <nothing> [same are yours] > > > > VM spec: > > <domain type="kvm"> > <name>ubuntu24.04</name> > <uuid>50a13bea-17be-440d-88cf-63086538e1a5</uuid> > <metadata> > <libosinfo:libosinfo xmlns:libosinfo=" > http://libosinfo.org/xmlns/libvirt/domain/1.0"> > <libosinfo:os id="http://ubuntu.com/ubuntu/24.04"/> > </libosinfo:libosinfo> > </metadata> > <memory unit="KiB">16777216</memory> > <currentMemory unit="KiB">16777216</currentMemory> > <memoryBacking> > <hugepages> > <page size="2048" unit="KiB"/> > </hugepages> > <source type="memfd"/> > <access mode="shared"/> > </memoryBacking> > <vcpu placement="static">16</vcpu> > <os firmware="efi"> > <type arch="x86_64" machine="pc-q35-8.2">hvm</type> > <firmware> > <feature enabled="yes" name="enrolled-keys"/> > <feature enabled="yes" name="secure-boot"/> > </firmware> > <loader readonly="yes" secure="yes" > type="pflash">/usr/share/OVMF/OVMF_CODE_4M.ms.fd</loader> > <nvram > template="/usr/share/OVMF/OVMF_VARS_4M.ms.fd">/var/lib/libvirt/qemu/nvram/ubuntu24.04_VARS.fd</nvram> > <boot dev="hd"/> > </os> > <features> > <acpi/> > <apic/> > <vmport state="off"/> > <smm state="on"/> > </features> > <cpu mode="host-passthrough" check="none" migratable="on"/> > <clock offset="utc"> > <timer name="rtc" tickpolicy="catchup"/> > <timer name="pit" tickpolicy="delay"/> > <timer name="hpet" present="no"/> > </clock> > <on_poweroff>destroy</on_poweroff> > <on_reboot>restart</on_reboot> > <on_crash>destroy</on_crash> > <pm> > <suspend-to-mem enabled="no"/> > <suspend-to-disk enabled="no"/> > </pm> > <devices> > <emulator>/usr/bin/qemu-system-x86_64</emulator> > <disk type="file" device="disk"> > <driver name="qemu" type="qcow2"/> > <source file="/opt/virtual_machines/ubuntu-dev-2024.qcow2"/> > <target dev="vda" bus="virtio"/> > <address type="pci" domain="0x0000" bus="0x04" slot="0x00" > function="0x0"/> > </disk> > <controller type="usb" index="0" model="qemu-xhci" ports="15"> > <address type="pci" domain="0x0000" bus="0x02" slot="0x00" > function="0x0"/> > </controller> > <controller type="pci" index="0" model="pcie-root"/> > <controller type="pci" index="1" model="pcie-root-port"> > <model name="pcie-root-port"/> > <target chassis="1" port="0x10"/> > <address type="pci" domain="0x0000" bus="0x00" slot="0x02" > function="0x0" multifunction="on"/> > </controller> > <controller type="pci" index="2" model="pcie-root-port"> > <model name="pcie-root-port"/> > <target chassis="2" port="0x11"/> > <address type="pci" domain="0x0000" bus="0x00" slot="0x02" > function="0x1"/> > </controller> > <controller type="pci" index="3" model="pcie-root-port"> > <model name="pcie-root-port"/> > <target chassis="3" port="0x12"/> > <address type="pci" domain="0x0000" bus="0x00" slot="0x02" > function="0x2"/> > </controller> > <controller type="pci" index="4" model="pcie-root-port"> > <model name="pcie-root-port"/> > <target chassis="4" port="0x13"/> > <address type="pci" domain="0x0000" bus="0x00" slot="0x02" > function="0x3"/> > </controller> > <controller type="pci" index="5" model="pcie-root-port"> > <model name="pcie-root-port"/> > <target chassis="5" port="0x14"/> > <address type="pci" domain="0x0000" bus="0x00" slot="0x02" > function="0x4"/> > </controller> > <controller type="pci" index="6" model="pcie-root-port"> > <model name="pcie-root-port"/> > <target chassis="6" port="0x15"/> > <address type="pci" domain="0x0000" bus="0x00" slot="0x02" > function="0x5"/> > </controller> > <controller type="pci" index="7" model="pcie-root-port"> > <model name="pcie-root-port"/> > <target chassis="7" port="0x16"/> > <address type="pci" domain="0x0000" bus="0x00" slot="0x02" > function="0x6"/> > </controller> > <controller type="pci" index="8" model="pcie-root-port"> > <model name="pcie-root-port"/> > <target chassis="8" port="0x17"/> > <address type="pci" domain="0x0000" bus="0x00" slot="0x02" > function="0x7"/> > </controller> > <controller type="pci" index="9" model="pcie-root-port"> > <model name="pcie-root-port"/> > <target chassis="9" port="0x18"/> > <address type="pci" domain="0x0000" bus="0x00" slot="0x03" > function="0x0" multifunction="on"/> > </controller> > <controller type="pci" index="10" model="pcie-root-port"> > <model name="pcie-root-port"/> > <target chassis="10" port="0x19"/> > <address type="pci" domain="0x0000" bus="0x00" slot="0x03" > function="0x1"/> > </controller> > <controller type="pci" index="11" model="pcie-root-port"> > <model name="pcie-root-port"/> > <target chassis="11" port="0x1a"/> > <address type="pci" domain="0x0000" bus="0x00" slot="0x03" > function="0x2"/> > </controller> > <controller type="pci" index="12" model="pcie-root-port"> > <model name="pcie-root-port"/> > <target chassis="12" port="0x1b"/> > <address type="pci" domain="0x0000" bus="0x00" slot="0x03" > function="0x3"/> > </controller> > <controller type="pci" index="13" model="pcie-root-port"> > <model name="pcie-root-port"/> > <target chassis="13" port="0x1c"/> > <address type="pci" domain="0x0000" bus="0x00" slot="0x03" > function="0x4"/> > </controller> > <controller type="pci" index="14" model="pcie-root-port"> > <model name="pcie-root-port"/> > <target chassis="14" port="0x1d"/> > <address type="pci" domain="0x0000" bus="0x00" slot="0x03" > function="0x5"/> > </controller> > <controller type="sata" index="0"> > <address type="pci" domain="0x0000" bus="0x00" slot="0x1f" > function="0x2"/> > </controller> > <controller type="virtio-serial" index="0"> > <address type="pci" domain="0x0000" bus="0x03" slot="0x00" > function="0x0"/> > </controller> > <filesystem type="mount" accessmode="passthrough"> > <driver type="virtiofs"/> > <source dir="/home/tim/Downloads"/> > <target dir="host_downloads"/> > <address type="pci" domain="0x0000" bus="0x07" slot="0x00" > function="0x0"/> > </filesystem> > <interface type="network"> > <mac address="52:54:00:9c:2a:df"/> > <source network="br0"/> > <model type="virtio"/> > <address type="pci" domain="0x0000" bus="0x01" slot="0x00" > function="0x0"/> > </interface> > <serial type="pty"> > <target type="isa-serial" port="0"> > <model name="isa-serial"/> > </target> > </serial> > <console type="pty"> > <target type="serial" port="0"/> > </console> > <channel type="unix"> > <target type="virtio" name="org.qemu.guest_agent.0"/> > <address type="virtio-serial" controller="0" bus="0" port="1"/> > </channel> > <channel type="spicevmc"> > <target type="virtio" name="com.redhat.spice.0"/> > <address type="virtio-serial" controller="0" bus="0" port="2"/> > </channel> > <input type="tablet" bus="usb"> > <address type="usb" bus="0" port="1"/> > </input> > <input type="mouse" bus="ps2"/> > <input type="keyboard" bus="ps2"/> > <tpm model="tpm-crb"> > <backend type="emulator" version="2.0"/> > </tpm> > <graphics type="spice"> > <listen type="none"/> > <image compression="off"/> > <gl enable="yes"/> > </graphics> > <sound model="ich9"> > <address type="pci" domain="0x0000" bus="0x00" slot="0x1b" > function="0x0"/> > </sound> > <audio id="1" type="spice"/> > <video> > <model type="virtio" heads="1" primary="yes"> > <acceleration accel3d="yes"/> > </model> > <address type="pci" domain="0x0000" bus="0x00" slot="0x01" > function="0x0"/> > </video> > <redirdev bus="usb" type="spicevmc"> > <address type="usb" bus="0" port="2"/> > </redirdev> > <redirdev bus="usb" type="spicevmc"> > <address type="usb" bus="0" port="3"/> > </redirdev> > <watchdog model="itco" action="reset"/> > <memballoon model="virtio"> > <address type="pci" domain="0x0000" bus="0x05" slot="0x00" > function="0x0"/> > </memballoon> > <rng model="virtio"> > <backend model="random">/dev/urandom</backend> > <address type="pci" domain="0x0000" bus="0x06" slot="0x00" > function="0x0"/> > </rng> > </devices> > </domain> > > On Thu, 18 Jul 2024 at 17:05, Christian Ehrhardt < > 2073...@bugs.launchpad.net> wrote: > >> Hi again Tim, >> this is an even more interesting case - as it depends on kernel module >> loading will be different between systems. It will differ in: >> 1. Need: only if gpu passthrough or GL rendernodes are configured this >> will matter >> 2. Availability: systems might have more GPUs, which one do we wait on. >> Or they have none with this path never being populated >> >> So this is an interesting effort for a sysadmin - to decide I configured >> and have the need #1, but I know it is available #2 so now how do I tune >> my system to cope with that. >> >> But at the same time tricky for a generic fix to not negatively affect >> those that do not need it or have systems which never have it. >> >> I have no solution yet, but some thoughts and questions. >> >> >> ## 1 Ordering >> >> What you describe is gladly AFAIK the uncommon case, you are describing >> that libvirt starts and then starts the guest before the kernel >> initialized dri/drm. >> >> On the systems I could quickly check that was not the case, I have >> always seen things like: >> >> $ journalctl -b 0 | grep -i -e "\[drm\] Initialized" -e "Starting >> libvirtd.service" >> Mai 21 08:11:03 Keschdeichel kernel: [drm] Initialized simpledrm 1.0.0 >> 20200625 for simple-framebuffer.0 on minor 0 >> Mai 21 08:11:05 Keschdeichel kernel: [drm] Initialized i915 1.6.0 >> 20230929 for 0000:00:02.0 on minor 1 >> Mai 21 08:11:05 Keschdeichel kernel: [drm] Initialized evdi 1.14.4 >> 20240410 for evdi.0 on minor 0 >> Mai 21 08:11:05 Keschdeichel kernel: [drm] Initialized evdi 1.14.4 >> 20240410 for evdi.1 on minor 2 >> Mai 21 08:11:05 Keschdeichel kernel: [drm] Initialized evdi 1.14.4 >> 20240410 for evdi.2 on minor 3 >> Mai 21 08:11:05 Keschdeichel kernel: [drm] Initialized evdi 1.14.4 >> 20240410 for evdi.3 on minor 4 >> Mai 21 08:11:10 Keschdeichel systemd[1]: Starting libvirtd.service - >> libvirt legacy monolithic daemon... >> >> >> I'm just curious how to match this fail onto your case. >> How does this look for you? >> I assume that without your change you get libvirt starting before (all) >> drm - is that what you see? >> >> >> ## 2 Waiting >> >> [Service] >> ExecStartPre=/bin/sleep 10 >> >> >> I understand that this fixes your issue and keep it until we've found >> something better. >> But that can not be a generic change we'd apply. >> It is 10 seconds for you, maybe someone needs 12 or 123456 - we could >> never set this right and would slow everyone not even needing it down for >> nothing. >> >> Yet, as documented workaround it is nice >> >> >> ## 3 The Unit >> >> [Unit] >> After=multi-user.target dev-dri.device >> >> This looks much better, but AFAICS it should do nothing. >> My system has dri entries >> >> $ ll /dev/dri/ >> total 0 >> drwxr-xr-x 3 root root 180 Mai 21 08:11 ./ >> drwxr-xr-x 22 root root 6060 Jul 18 02:07 ../ >> drwxr-xr-x 2 root root 160 Mai 21 08:11 by-path/ >> crw-rw----+ 1 root video 226, 0 Mai 24 03:46 card0 >> crw-rw----+ 1 root video 226, 1 Mai 24 03:46 card1 >> crw-rw----+ 1 root video 226, 2 Mai 24 03:46 card2 >> crw-rw----+ 1 root video 226, 3 Mai 24 03:46 card3 >> crw-rw----+ 1 root video 226, 4 Mai 24 03:46 card4 >> crw-rw----+ 1 root render 226, 128 Mai 24 03:46 renderD128 >> >> But there are no such devices defined >> >> $ systemctl list-units --all --type device | grep dri >> <nothing> >> >> That is because the closest to a matchin udev rule is >> $ cat /lib/udev/rules.d/60-drm.rules >> # do not edit this file, it will be overwritten on update >> >> ACTION!="remove", SUBSYSTEM=="drm", SUBSYSTEMS=="pci|usb|platform", >> IMPORT{builtin}="path_id" >> >> # by-path >> KERNEL=="card*", ENV{ID_PATH}=="?*", >> SYMLINK+="dri/by-path/$env{ID_PATH}-card" >> KERNEL=="card*", ENV{ID_PATH_WITH_USB_REVISION}=="?*", >> SYMLINK+="dri/by-path/$env{ID_PATH_WITH_USB_REVISION}-card" >> KERNEL=="controlD*", ENV{ID_PATH}=="?*", >> SYMLINK+="dri/by-path/$env{ID_PATH}-control" >> KERNEL=="controlD*", ENV{ID_PATH_WITH_USB_REVISION}=="?*", >> SYMLINK+="dri/by-path/$env{ID_PATH_WITH_USB_REVISION}-control" >> KERNEL=="renderD*", ENV{ID_PATH}=="?*", >> SYMLINK+="dri/by-path/$env{ID_PATH}-render" >> KERNEL=="renderD*", ENV{ID_PATH_WITH_USB_REVISION}=="?*", >> SYMLINK+="dri/by-path/$env{ID_PATH_WITH_USB_REVISION}-render" >> >> And there is nothing that would create dev-dri.device >> They would need a TAG+="systemd" entry. >> Similar to discussions [1][2] >> But even if we'd have that, AFAIU there would be dev-dri-card0.device but >> not just dev-dri.device >> And even if we'd have a particular guest config might need >> dev-dri-card7.device and that initializes even later - so just waiting on >> DRI sounds neat. >> >> Yet on the other hand, just like selecting the right timeout on the >> sleep - this is dependent on the system config, hardware and needs :-/ >> >> >> ## 4 What now? >> >> I'd appreciate if you could: >> - share the initialization order your system really has >> - explain if you found something about dev-dri.device that I do not know >> yet >> - explain in more details which HW general and the GPUs you wait on are >> - share how you configured the guest (is it passthrough, is it gl >> rendering ...?) >> >> That would help us to understand the situation a bit better, if we are >> lucky it might even allow to recreate it. >> >> Still, my expectation is that with all that we eventually need to reach >> out to the project at [3] or [4]. >> Due to the "at system config file time we'd never know if and what we >> need to wait on" problem described above I'd expect that this might need >> something completely else. Like libvirt internally (it knows what a guest >> needs as it knows its definition) waiting for that if and only as needed. >> Or I might overlook something obvious which the subject matter experts >> there might know and share. >> >> But for now, I'd appreciate if you could help my curiosity by providing >> the above. >> >> [1]: https://github.com/systemd/systemd/issues/25408 >> [2]: https://github.com/joukewitteveen/xlogin/issues/15 >> [3]: https://gitlab.com/libvirt/libvirt >> [4]: https://listman.redhat.com/mailman/listinfo/libvir-list >> >> P.S. @Sergio who usually looks at these - sorry for stealing those >> interesting cases in my morning, bad timezone luck for you :-P. Do not >> be concerned, you might deal with is long enough down the road :-) >> >> ** Bug watch added: github.com/systemd/systemd/issues #25408 >> https://github.com/systemd/systemd/issues/25408 >> >> ** Bug watch added: github.com/joukewitteveen/xlogin/issues #15 >> https://github.com/joukewitteveen/xlogin/issues/15 >> >> ** Changed in: libvirt (Ubuntu) >> Status: New => Incomplete >> >> -- >> You received this bug notification because you are subscribed to the bug >> report. >> https://bugs.launchpad.net/bugs/2073442 >> >> Title: >> Failed to autostart VM: cannot open directory '/dev/dri': No such file >> or directory >> >> To manage notifications about this bug go to: >> >> https://bugs.launchpad.net/ubuntu/+source/libvirt/+bug/2073442/+subscriptions >> >> > > -- > Tim Richardson > -- Tim Richardson -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/2073442 Title: Failed to autostart VM: cannot open directory '/dev/dri': No such file or directory To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/libvirt/+bug/2073442/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs