[Kernel-packages] [Bug 1838575] Re: passthrough devices cause >17min boot delay
** Changed in: linux (Ubuntu) Assignee: Colin Ian King (colin-king) => (unassigned) -- You received this bug notification because you are a member of Kernel Packages, which is subscribed to linux in Ubuntu. https://bugs.launchpad.net/bugs/1838575 Title: passthrough devices cause >17min boot delay Status in linux package in Ubuntu: Confirmed Status in qemu package in Ubuntu: Invalid Bug description: Adding passthrough devices to a guest introduces a boot delay of > 17 minutes. During this time libvirt reports the guest to be "paused". The delay does not seem to scale with the # of hostdevs - that is, 1 hostdev causes about a 17min delay, while 16 only bumps that up to ~18min. Removing all hostdevs avoids the delay. To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1838575/+subscriptions -- Mailing list: https://launchpad.net/~kernel-packages Post to : kernel-packages@lists.launchpad.net Unsubscribe : https://launchpad.net/~kernel-packages More help : https://help.launchpad.net/ListHelp
[Kernel-packages] [Bug 1838575] Re: passthrough devices cause >17min boot delay
As outlined in the past conceptually there is nothing that qemu can do. The kernel can in theory get memory zeroing to become concurrent and thereby scale with CPUs but that is an effort that was already started twice and didn't get into the kernel yet. Workarounds are known to shrink that size massively (HP). Also this is nothing the kernel Team would usually "do" they would follow upstream on it, so I'm leaving it open but low for them. I realized those discussions where going on but not linked here. => https://lore.kernel.org/lkml/20181105165558.11698-1-daniel.m.jor...@oracle.com/T/ => https://lwn.net/Articles/728023/ => https://blogs.oracle.com/linux/making-kernel-tasks-faster-with-ktask,-an-update Is one example. I'd ask the kernel Team to keep an eye open for that, be encouraging in discussions. This really has a huge benefit in those use cases. ** Changed in: linux (Ubuntu) Importance: Medium => Low ** Changed in: qemu (Ubuntu) Status: Triaged => Invalid -- You received this bug notification because you are a member of Kernel Packages, which is subscribed to linux in Ubuntu. https://bugs.launchpad.net/bugs/1838575 Title: passthrough devices cause >17min boot delay Status in linux package in Ubuntu: Confirmed Status in qemu package in Ubuntu: Invalid Bug description: Adding passthrough devices to a guest introduces a boot delay of > 17 minutes. During this time libvirt reports the guest to be "paused". The delay does not seem to scale with the # of hostdevs - that is, 1 hostdev causes about a 17min delay, while 16 only bumps that up to ~18min. Removing all hostdevs avoids the delay. To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1838575/+subscriptions -- Mailing list: https://launchpad.net/~kernel-packages Post to : kernel-packages@lists.launchpad.net Unsubscribe : https://launchpad.net/~kernel-packages More help : https://help.launchpad.net/ListHelp
[Kernel-packages] [Bug 1838575] Re: passthrough devices cause >17min boot delay
As qemu (seems) to be unable to do much I'll set it to triaged (we understand what is going on) and low (can't do much). ** Changed in: qemu (Ubuntu) Status: Incomplete => Triaged ** Changed in: qemu (Ubuntu) Importance: Medium => Low -- You received this bug notification because you are a member of Kernel Packages, which is subscribed to linux in Ubuntu. https://bugs.launchpad.net/bugs/1838575 Title: passthrough devices cause >17min boot delay Status in linux package in Ubuntu: Confirmed Status in qemu package in Ubuntu: Triaged Bug description: Adding passthrough devices to a guest introduces a boot delay of > 17 minutes. During this time libvirt reports the guest to be "paused". The delay does not seem to scale with the # of hostdevs - that is, 1 hostdev causes about a 17min delay, while 16 only bumps that up to ~18min. Removing all hostdevs avoids the delay. To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1838575/+subscriptions -- Mailing list: https://launchpad.net/~kernel-packages Post to : kernel-packages@lists.launchpad.net Unsubscribe : https://launchpad.net/~kernel-packages More help : https://help.launchpad.net/ListHelp
[Kernel-packages] [Bug 1838575] Re: passthrough devices cause >17min boot delay
This is a silly but useful distribution check with log10 of the allocation sizes: Fast: 108 3 1293 4 12133 5 113330 6 27794 7 1119 8 Slow: 194 3 1738 4 17375 5 143411 6 55 7 3 8 I got no warnings about missed calls, but always be aware that some numbers might be off - ususally they are ok for relative comparisons. So yeah, the slow case just needs to map more smaller pieces as that is all it can find. That explains the "getting worse with system runtime", and I don#t think there is much that can be done here. I'll discuss if there is any gain in splitting this from one thread into many (for very huge memory sizes). P.S. Finally I just want to re-iterate that using hugepages due to their pre-allocation, less fragmentation, less mapping behavior really is the "config" way out of this. -- You received this bug notification because you are a member of Kernel Packages, which is subscribed to linux in Ubuntu. https://bugs.launchpad.net/bugs/1838575 Title: passthrough devices cause >17min boot delay Status in linux package in Ubuntu: Confirmed Status in qemu package in Ubuntu: Incomplete Bug description: Adding passthrough devices to a guest introduces a boot delay of > 17 minutes. During this time libvirt reports the guest to be "paused". The delay does not seem to scale with the # of hostdevs - that is, 1 hostdev causes about a 17min delay, while 16 only bumps that up to ~18min. Removing all hostdevs avoids the delay. To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1838575/+subscriptions -- Mailing list: https://launchpad.net/~kernel-packages Post to : kernel-packages@lists.launchpad.net Unsubscribe : https://launchpad.net/~kernel-packages More help : https://help.launchpad.net/ListHelp
[Kernel-packages] [Bug 1838575] Re: passthrough devices cause >17min boot delay
I modified the kernel to have a few functions non-inlined to be better tracable: vfio_dma_do_map vfio_dma_do_unmap mutex_lock mutex_unlock kzalloc vfio_link_dma vfio_pin_map_dma vfio_pin_pages_remote vfio_iommu_map Then run tracing on this load with limited to the functions in my focus: $ sudo trace-cmd record -p function_graph -l vfio_iommu_type1_ioctl -l vfio_pci_ioctl -l vfio_set_irqs_validate_and_prepare -l vfio_pci_set_irqs_ioctl -l pci_probe_reset_slot -l pci_probe_reset_bus -l vfio_pci_count_devs -l vfio_pci_fill_devs -l vfio_pci_validate_devs -l vfio_group_get_external_user -l vfio_group_put_external_user -l vfio_external_user_iommu_id -l vfio_dma_do_map -l vfio_dma_do_unmap -l mutex_lock -l mutex_unlock -l vfio_link_dma -l vfio_pin_map_dma -l vfio_pin_pages_remote -l vfio_iommu_map This I traced twice to compare it: - after a clean boot; duration ~12 seconds - right afterwards (rerun from gdb); duration 175~ seconds It shows that the locks are interesting for concurrency considerations, but here mostly noisy. OTOH it is interesting that the slow case has much more of them, maybe they need to unlock for interactiveness in between more often. I'll count them in postprocessing like: The result shows that the locking involved scaled much more. Duration is ~*14.5 and locks are ~*34 the average lock activity between vfio calls is even more with 1.52 -> 6.32. But the calls to vfio_pin_pages_remote / vfio_iommu_map went up as well. 64020 -> 327852; which means smaller chunks I guess. The timing of these functions changes slightly, initially it is much different than later in the trace. Lets take a clean shot at just those durations and sizes: -- You received this bug notification because you are a member of Kernel Packages, which is subscribed to linux in Ubuntu. https://bugs.launchpad.net/bugs/1838575 Title: passthrough devices cause >17min boot delay Status in linux package in Ubuntu: Confirmed Status in qemu package in Ubuntu: Incomplete Bug description: Adding passthrough devices to a guest introduces a boot delay of > 17 minutes. During this time libvirt reports the guest to be "paused". The delay does not seem to scale with the # of hostdevs - that is, 1 hostdev causes about a 17min delay, while 16 only bumps that up to ~18min. Removing all hostdevs avoids the delay. To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1838575/+subscriptions -- Mailing list: https://launchpad.net/~kernel-packages Post to : kernel-packages@lists.launchpad.net Unsubscribe : https://launchpad.net/~kernel-packages More help : https://help.launchpad.net/ListHelp
[Kernel-packages] [Bug 1838575] Re: passthrough devices cause >17min boot delay
(systemtap)
probe module("vfio_iommu_type1").function("vfio_iommu_type1_ioctl") {
printf("New vfio_iommu_type1_ioctl\n");
start_stopwatch("vfioioctl");
}
probe module("vfio_iommu_type1").function("vfio_iommu_type1_ioctl").return {
timer=read_stopwatch_ns("vfioioctl")
printf("Completed vfio_iommu_type1_ioctl: %d\n", timer);
stop_stopwatch("vfioioctl");
}
probe module("vfio_iommu_type1").function("vfio_pin_pages_remote") {
timer=read_stopwatch_ns("vfioioctl")
printf("%ld: %s\n", timer, $$parms);
}
The overhead was significant enough, the fast case this time was 132 seconds.
The following slow case had 209 seconds (still slower but not more by factors).
While the overhead might have influenced the absolute timing of this too much.
It still might help indicate the size/frequency distribution of these calls
that we wanted to know.
--
You received this bug notification because you are a member of Kernel
Packages, which is subscribed to linux in Ubuntu.
https://bugs.launchpad.net/bugs/1838575
Title:
passthrough devices cause >17min boot delay
Status in linux package in Ubuntu:
Confirmed
Status in qemu package in Ubuntu:
Incomplete
Bug description:
Adding passthrough devices to a guest introduces a boot delay of > 17
minutes. During this time libvirt reports the guest to be "paused".
The delay does not seem to scale with the # of hostdevs - that is, 1
hostdev causes about a 17min delay, while 16 only bumps that up to
~18min. Removing all hostdevs avoids the delay.
To manage notifications about this bug go to:
https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1838575/+subscriptions
--
Mailing list: https://launchpad.net/~kernel-packages
Post to : kernel-packages@lists.launchpad.net
Unsubscribe : https://launchpad.net/~kernel-packages
More help : https://help.launchpad.net/ListHelp
[Kernel-packages] [Bug 1838575] Re: passthrough devices cause >17min boot delay
The iommu is locked in there early and the iommu element is what is passed from userspace. That represents the vfio container for this device (container->fd) qemu: if (ioctl(container->fd, VFIO_IOMMU_MAP_DMA, ) == 0 kernel: static long vfio_iommu_type1_ioctl(void *iommu_data, unsigned int cmd, unsigned long arg) struct vfio_iommu *iommu = iommu_data; [... into vfio_dma_do_map ...] mutex_lock(>lock); There isn't much divide and conquer splitting that seems easily possible for now :-/ Down there while this lock is held all the memory size must be pinned -> vfio_pin_pages_remote Which gets the biggest chunk it can to then map it -> vfio_iommu_map This is repeated until all of the requested size is handled. Establishing iommu maps is known to be expensive, an assumption would be that in the semi-fast cases is either: - memory is still non-fragmented so we only need a few calls - the iommu is sort of async-busy from the former work (same calls, but longer) That should be visible in the amount of vfio_pin_pages_remote if we don't miss some. -- You received this bug notification because you are a member of Kernel Packages, which is subscribed to linux in Ubuntu. https://bugs.launchpad.net/bugs/1838575 Title: passthrough devices cause >17min boot delay Status in linux package in Ubuntu: Confirmed Status in qemu package in Ubuntu: Incomplete Bug description: Adding passthrough devices to a guest introduces a boot delay of > 17 minutes. During this time libvirt reports the guest to be "paused". The delay does not seem to scale with the # of hostdevs - that is, 1 hostdev causes about a 17min delay, while 16 only bumps that up to ~18min. Removing all hostdevs avoids the delay. To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1838575/+subscriptions -- Mailing list: https://launchpad.net/~kernel-packages Post to : kernel-packages@lists.launchpad.net Unsubscribe : https://launchpad.net/~kernel-packages More help : https://help.launchpad.net/ListHelp
[Kernel-packages] [Bug 1838575] Re: passthrough devices cause >17min boot delay
Each qemu (version) is slightly different in the road to this, but then seems to behave. This one is slightly better to get "in front" of the slow call to map all the memory. $ virsh nodedev-detach pci__21_00_1 --driver vfio $ gdb /usr/bin/qemu-system-x86_64 (gdb) b vfio_dma_map (gdb) command 1 Type commands for breakpoint(s) 1, one per line. End with a line saying just "end". >silent >if size != 134217728000 >cont >end >end (gdb) run -m 131072 -smp 1 -no-user-config -device vfio-pci,host=21:00.1,id=hostdev0,bus=pci.0,addr=0x7 -enable-kvm (gdb) catch syscall 16 (gdb) c Also correct the link to qemu - it is the one from vfio_listener_region_add: https://git.qemu.org/?p=qemu.git;a=blob;f=hw/vfio/common.c;h=3e03c495d868f154b26a33c0c2d870273a0b4198;hb=HEAD#l664 -- You received this bug notification because you are a member of Kernel Packages, which is subscribed to linux in Ubuntu. https://bugs.launchpad.net/bugs/1838575 Title: passthrough devices cause >17min boot delay Status in linux package in Ubuntu: Confirmed Status in qemu package in Ubuntu: Incomplete Bug description: Adding passthrough devices to a guest introduces a boot delay of > 17 minutes. During this time libvirt reports the guest to be "paused". The delay does not seem to scale with the # of hostdevs - that is, 1 hostdev causes about a 17min delay, while 16 only bumps that up to ~18min. Removing all hostdevs avoids the delay. To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1838575/+subscriptions -- Mailing list: https://launchpad.net/~kernel-packages Post to : kernel-packages@lists.launchpad.net Unsubscribe : https://launchpad.net/~kernel-packages More help : https://help.launchpad.net/ListHelp
[Kernel-packages] [Bug 1838575] Re: passthrough devices cause >17min boot delay
I could next build a test kernel with some debug around the vfio iommu dma map to check how time below that call is spent. I'm sure that data already is hidden in some of my trace data, but to eventually change/experiment I need to build one anyway. I expect anyway to summarize and go into a discussion at some point, as even if one would split the map calls iommus are full of quirks and constraints that might bit us later. -- You received this bug notification because you are a member of Kernel Packages, which is subscribed to linux in Ubuntu. https://bugs.launchpad.net/bugs/1838575 Title: passthrough devices cause >17min boot delay Status in linux package in Ubuntu: Confirmed Status in qemu package in Ubuntu: Incomplete Bug description: Adding passthrough devices to a guest introduces a boot delay of > 17 minutes. During this time libvirt reports the guest to be "paused". The delay does not seem to scale with the # of hostdevs - that is, 1 hostdev causes about a 17min delay, while 16 only bumps that up to ~18min. Removing all hostdevs avoids the delay. To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1838575/+subscriptions -- Mailing list: https://launchpad.net/~kernel-packages Post to : kernel-packages@lists.launchpad.net Unsubscribe : https://launchpad.net/~kernel-packages More help : https://help.launchpad.net/ListHelp
[Kernel-packages] [Bug 1838575] Re: passthrough devices cause >17min boot delay
Reference: this is the call from qemu that I think we see above (on x86) is at [1]. If this time the assumption is correct the kernel place would be at vfio_iommu_type1_ioctl. For debugging: $ gdb qemu/x86_64-softmmu/qemu-system-x86_64 (gdb) catch syscall 16 (gdb) run -m 131072 -smp 1 -no-user-config -device vfio-pci,host=21:00.1,id=hostdev0,bus=pci.0,addr=0x7 (gdb) continue 22 # Now we are "in front" of the bad call. https://git.qemu.org/?p=qemu.git;a=blob;f=hw/vfio/common.c;h=3e03c495d868f154b26a33c0c2d870273a0b4198;hb=HEAD#l480 -- You received this bug notification because you are a member of Kernel Packages, which is subscribed to linux in Ubuntu. https://bugs.launchpad.net/bugs/1838575 Title: passthrough devices cause >17min boot delay Status in linux package in Ubuntu: Confirmed Status in qemu package in Ubuntu: Incomplete Bug description: Adding passthrough devices to a guest introduces a boot delay of > 17 minutes. During this time libvirt reports the guest to be "paused". The delay does not seem to scale with the # of hostdevs - that is, 1 hostdev causes about a 17min delay, while 16 only bumps that up to ~18min. Removing all hostdevs avoids the delay. To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1838575/+subscriptions -- Mailing list: https://launchpad.net/~kernel-packages Post to : kernel-packages@lists.launchpad.net Unsubscribe : https://launchpad.net/~kernel-packages More help : https://help.launchpad.net/ListHelp
[Kernel-packages] [Bug 1838575] Re: passthrough devices cause >17min boot delay
Many ioctls (as expected) but they are all fast and match what we knew from strace. Thread 1 "qemu-system-x86" hit Catchpoint 1 (call to syscall ioctl), 0x772fae0b in ioctl () at ../sysdeps/unix/syscall-template.S:78 78 in ../sysdeps/unix/syscall-template.S (gdb) bt #0 0x772fae0b in ioctl () at ../sysdeps/unix/syscall-template.S:78 #1 0x55914217 in vfio_init_container (container=0x57608430, group_fd=11, errp=0x7fffdda8) at /home/ubuntu/qemu/hw/vfio/common.c:1172 #2 0x5591445e in vfio_connect_container (group=0x576083b0, as=0x565ca5a0 , errp=0x7fffdda8) at /home/ubuntu/qemu/hw/vfio/common.c:1257 #3 0x55914f35 in vfio_get_group (groupid=45, as=0x565ca5a0 , errp=0x7fffdda8) at /home/ubuntu/qemu/hw/vfio/common.c:1482 #4 0x55920575 in vfio_realize (pdev=0x57600570, errp=0x7fffdda8) at /home/ubuntu/qemu/hw/vfio/pci.c:2759 #5 0x55b853d7 in pci_qdev_realize (qdev=0x57600570, errp=0x7fffde20) at hw/pci/pci.c:2096 #6 0x55aac178 in device_set_realized (obj=0x57600570, value=true, errp=0x7fffdff0) at hw/core/qdev.c:835 #7 0x55cb857a in property_set_bool (obj=0x57600570, v=0x57602f00, name=0x55f3b5d2 "realized", opaque=0x574e79e0, errp=0x7fffdff0) at qom/object.c:2080 #8 0x55cb6731 in object_property_set (obj=0x57600570, v=0x57602f00, name=0x55f3b5d2 "realized", errp=0x7fffdff0) at qom/object.c:1272 #9 0x55cb9965 in object_property_set_qobject (obj=0x57600570, value=0x57602dc0, name=0x55f3b5d2 "realized", errp=0x7fffdff0) at qom/qom-qobject.c:26 #10 0x55cb6a2a in object_property_set_bool (obj=0x57600570, value=true, name=0x55f3b5d2 "realized", errp=0x7fffdff0) at qom/object.c:1338 #11 0x55a12cb1 in qdev_device_add (opts=0x566a0770, errp=0x5661e790 ) at qdev-monitor.c:635 #12 0x55a1b11f in device_init_func (opaque=0x0, opts=0x566a0770, errp=0x5661e790 ) at vl.c:2206 #13 0x55e1c7fd in qemu_opts_foreach (list=0x56468ee0 , func=0x55a1b0f4 , opaque=0x0, errp=0x5661e790 ) at util/qemu-option.c:1170 #14 0x55a2009d in main (argc=14, argv=0x7fffe3d8, envp=0x7fffe450) at vl.c:4387 The slow one is again the full iommu mapping of the guest memory: (gdb) bt #0 0x772fae0b in ioctl () at ../sysdeps/unix/syscall-template.S:78 #1 0x55911a9e in vfio_dma_map (container=0x57608430, iova=4294967296, size=134217728000, vaddr=0x7fe01fe0, readonly=false) at /home/ubuntu/qemu/hw/vfio/common.c:349 #2 0x5591299b in vfio_listener_region_add (listener=0x57608440, section=0x7fffcad0) at /home/ubuntu/qemu/hw/vfio/common.c:664 #3 0x5587dfb0 in listener_add_address_space (listener=0x57608440, as=0x565ca5a0 ) at /home/ubuntu/qemu/memory.c:2714 #4 0x5587e3dd in memory_listener_register (listener=0x57608440, as=0x565ca5a0 ) at /home/ubuntu/qemu/memory.c:2781 #5 0x5591499b in vfio_connect_container (group=0x576083b0, as=0x565ca5a0 , errp=0x7fffdda8) at /home/ubuntu/qemu/hw/vfio/common.c:1365 #6 0x55914f35 in vfio_get_group (groupid=45, as=0x565ca5a0 , errp=0x7fffdda8) at /home/ubuntu/qemu/hw/vfio/common.c:1482 #7 0x55920575 in vfio_realize (pdev=0x57600570, errp=0x7fffdda8) at /home/ubuntu/qemu/hw/vfio/pci.c:2759 #8 0x55b853d7 in pci_qdev_realize (qdev=0x57600570, errp=0x7fffde20) at hw/pci/pci.c:2096 #9 0x55aac178 in device_set_realized (obj=0x57600570, value=true, errp=0x7fffdff0) at hw/core/qdev.c:835 #10 0x55cb857a in property_set_bool (obj=0x57600570, v=0x57602f00, name=0x55f3b5d2 "realized", opaque=0x574e79e0, errp=0x7fffdff0) at qom/object.c:2080 #11 0x55cb6731 in object_property_set (obj=0x57600570, v=0x57602f00, name=0x55f3b5d2 "realized", errp=0x7fffdff0) at qom/object.c:1272 #12 0x55cb9965 in object_property_set_qobject (obj=0x57600570, value=0x57602dc0, name=0x55f3b5d2 "realized", errp=0x7fffdff0) at qom/qom-qobject.c:26 #13 0x55cb6a2a in object_property_set_bool (obj=0x57600570, value=true, name=0x55f3b5d2 "realized", errp=0x7fffdff0) at qom/object.c:1338 #14 0x55a12cb1 in qdev_device_add (opts=0x566a0770, errp=0x5661e790 ) at qdev-monitor.c:635 #15 0x55a1b11f in device_init_func (opaque=0x0, opts=0x566a0770, errp=0x5661e790 ) at vl.c:2206 #16 0x55e1c7fd in qemu_opts_foreach (list=0x56468ee0 , func=0x55a1b0f4 , opaque=0x0, errp=0x5661e790 ) at util/qemu-option.c:1170 #17 0x55a2009d in main (argc=14, argv=0x7fffe3d8, envp=0x7fffe450) at vl.c:4387 (gdb) c Continuing. In particular: vfio_dma_map (container=0x57608430, iova=4294967296, size=134217728000,
[Kernel-packages] [Bug 1838575] Re: passthrough devices cause >17min boot delay
Just when I thought I understood the pattern. Sixth run (again kill and restart) 6384 9.826097 <... ioctl resumed> , 0x7ffcc8ed6e20) = 0 <19.495688> So for now lets summarize that it varies :-/ But it always seems slow. -- You received this bug notification because you are a member of Kernel Packages, which is subscribed to linux in Ubuntu. https://bugs.launchpad.net/bugs/1838575 Title: passthrough devices cause >17min boot delay Status in linux package in Ubuntu: Confirmed Status in qemu package in Ubuntu: Incomplete Bug description: Adding passthrough devices to a guest introduces a boot delay of > 17 minutes. During this time libvirt reports the guest to be "paused". The delay does not seem to scale with the # of hostdevs - that is, 1 hostdev causes about a 17min delay, while 16 only bumps that up to ~18min. Removing all hostdevs avoids the delay. To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1838575/+subscriptions -- Mailing list: https://launchpad.net/~kernel-packages Post to : kernel-packages@lists.launchpad.net Unsubscribe : https://launchpad.net/~kernel-packages More help : https://help.launchpad.net/ListHelp
[Kernel-packages] [Bug 1838575] Re: passthrough devices cause >17min boot delay
The above was through libvirt, doing that directly in qemu now to throw it into debugging more easily: $ virsh nodedev-detach pci__21_00_1 --driver vfio $ qemu/x86_64-softmmu/qemu-system-x86_64 -name guest=test-vfio-slowness -m 131072 -smp 1 -no-user-config -drive file=/var/lib/uvtool/libvirt/images/eoan.qcow,format=qcow2,if=none,id =drive-virtio-disk0 -device virtio-blk- pci,scsi=off,bus=pci.0,addr=0x4,drive=drive-virtio-disk0,id=virtio- disk0,bootindex=1 -device vfio- pci,host=21:00.1,id=hostdev0,bus=pci.0,addr=0x7 -display curses The speed differs a lot between retries: First run (after boot): 5204 3.793765 <... ioctl resumed> , 0x7a6687b0) = 0 <13.464192> Second run (after aborting the former qemu) 5251 179.361434 <... ioctl resumed> , 0x7ffed49fbc70) = 0 <189.030433> Third run (after aborting the former qemu) 5728 703.886935 <... ioctl resumed> , 0x7fff9f6a7120) = 0 <713.556876> Maybe a former unclean stop makes it take extra time? Or there is still cleanup going on in the iommu which stalls the follow on mappings to be established. Forth run (after proper guest shutdown + 10 min wait) 6171 11.167221 <... ioctl resumed> , 0x7ffe597f03e0) = 0 <20.838401> Fifth run (again kill and restart) 62151182.244403 <... ioctl resumed> , 0x7fff20a49b10) = 0 <1191.917291> 10 sec iommu counters while in the hang: 34330395280 cycles 16 cmd_processed (33.33%) 6 cmd_processed_inv (33.37%) 0 ign_rd_wr_mmio_1ff8h (33.41%) 45 int_dte_hit (33.44%) 0 int_dte_mis (33.44%) 345 mem_dte_hit (33.44%) 907 mem_dte_mis (33.44%) 111 mem_iommu_tlb_pde_hit (33.44%) 102 mem_iommu_tlb_pde_mis (33.40%) 7927 mem_iommu_tlb_pte_hit (33.37%) 1761 mem_iommu_tlb_pte_mis (33.33%) 0 mem_pass_excl (33.30%) 0 mem_pass_pretrans (33.29%) 0 mem_pass_untrans (33.29%) 0 mem_target_abort (33.29%) 8492 mem_trans_total (33.30%) 0 page_tbl_read_gst (33.30%) 168 page_tbl_read_nst (33.30%) 129 page_tbl_read_tot (33.30%) 0 smi_blk (33.30%) 0 smi_recv (33.31%) 0 tlb_inv (33.31%) 0 vapic_int_guest (33.31%) 24 vapic_int_non_guest (33.31%) 10.011867065 seconds time elapsed But then the slowness doesn't have to be visible in these counters. Next step, check the kernel if we can find (on x86) any place to parallelize this. -- You received this bug notification because you are a member of Kernel Packages, which is subscribed to linux in Ubuntu. https://bugs.launchpad.net/bugs/1838575 Title: passthrough devices cause >17min boot delay Status in linux package in Ubuntu: Confirmed Status in qemu package in Ubuntu: Incomplete Bug description: Adding passthrough devices to a guest introduces a boot delay of > 17 minutes. During this time libvirt reports the guest to be "paused". The delay does not seem to scale with the # of hostdevs - that is, 1 hostdev causes about a 17min delay, while 16 only bumps that up to ~18min. Removing all hostdevs avoids the delay. To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1838575/+subscriptions -- Mailing list: https://launchpad.net/~kernel-packages Post to : kernel-packages@lists.launchpad.net Unsubscribe : https://launchpad.net/~kernel-packages More help : https://help.launchpad.net/ListHelp
[Kernel-packages] [Bug 1838575] Re: passthrough devices cause >17min boot delay
On x86 this looks pretty similar and at the place we have seen before:
45397 0.73 readlink("/sys/bus/pci/devices/:21:00.1/iommu_group",
"../../../../kernel/iommu_groups/"..., 4096) = 34 <0.20>
45397 0.53 openat(AT_FDCWD, "/dev/vfio/45", O_RDWR|O_CLOEXEC) = 31
<0.33>
45397 0.60 ioctl(31, VFIO_GROUP_GET_STATUS, 0x7ffd774f80f8) = 0
<0.15>
45397 0.41 openat(AT_FDCWD, "/dev/vfio/vfio", O_RDWR|O_CLOEXEC) = 32
<0.19>
45397 0.46 ioctl(32, VFIO_GET_API_VERSION, 0x80002) = 0 <0.11>
45397 0.38 ioctl(32, VFIO_CHECK_EXTENSION, 0x3) = 1 <0.12>
45397 0.35 ioctl(31, VFIO_GROUP_SET_CONTAINER, 0x55758e6acec8) = 0
<0.11>
45397 0.33 ioctl(32, VFIO_SET_IOMMU, 0x3) = 0 <0.000174>
45397 0.000199 ioctl(32, VFIO_DEVICE_GET_PCI_HOT_RESET_INFO or
VFIO_IOMMU_GET_INFO or VFIO_IOMMU_SPAPR_TCE_GET_INFO, 0x7ffd774f8100) = 0
<0.12>
45397 0.38 ioctl(14, KVM_CREATE_DEVICE, 0x7ffd774f8094) = 0 <0.15>
45397 0.39 ioctl(33, KVM_SET_DEVICE_ATTR, 0x7ffd774f80a0) = 0
<0.28>
45397 0.54 ioctl(32, VFIO_DEVICE_PCI_HOT_RESET or VFIO_IOMMU_MAP_DMA,
0x7ffd774f7f20) = 0 <0.000287>
45397 0.000312 ioctl(32, VFIO_DEVICE_PCI_HOT_RESET or VFIO_IOMMU_MAP_DMA,
0x7ffd774f7f20) = 0 <0.35>
45397 0.58 ioctl(32, VFIO_DEVICE_PCI_HOT_RESET or VFIO_IOMMU_MAP_DMA,
0x7ffd774f7f20) = 0 <0.37>
45397 0.60 ioctl(32, VFIO_DEVICE_PCI_HOT_RESET or VFIO_IOMMU_MAP_DMA,
0x7ffd774f7f20) = 0 <0.334458>
45397 0.334593 ioctl(32, VFIO_DEVICE_PCI_HOT_RESET or VFIO_IOMMU_MAP_DMA,
0x7ffd774f7f20) = 0 <0.55>
45397 0.88 ioctl(32, VFIO_DEVICE_PCI_HOT_RESET or VFIO_IOMMU_MAP_DMA
45402 9.654538 <... futex resumed> ) = -1 ETIMEDOUT (Connection timed out)
<10.000112>
45402 0.000144 madvise(0x7fd51236f000, 8368128, MADV_DONTNEED) = 0
<0.36>
45402 0.76 exit(0) = ?
45402 0.74 +++ exited with 0 +++
45397 45.903415 <... ioctl resumed> , 0x7ffd774f7f20) = 0 <55.558237>
--
You received this bug notification because you are a member of Kernel
Packages, which is subscribed to linux in Ubuntu.
https://bugs.launchpad.net/bugs/1838575
Title:
passthrough devices cause >17min boot delay
Status in linux package in Ubuntu:
Confirmed
Status in qemu package in Ubuntu:
Incomplete
Bug description:
Adding passthrough devices to a guest introduces a boot delay of > 17
minutes. During this time libvirt reports the guest to be "paused".
The delay does not seem to scale with the # of hostdevs - that is, 1
hostdev causes about a 17min delay, while 16 only bumps that up to
~18min. Removing all hostdevs avoids the delay.
To manage notifications about this bug go to:
https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1838575/+subscriptions
--
Mailing list: https://launchpad.net/~kernel-packages
Post to : kernel-packages@lists.launchpad.net
Unsubscribe : https://launchpad.net/~kernel-packages
More help : https://help.launchpad.net/ListHelp
[Kernel-packages] [Bug 1838575] Re: passthrough devices cause >17min boot delay
I built qemu head from git
$ export CFLAGS="-O0 -g"
$ ./configure --disable-user --disable-linux-user --disable-docs
--disable-guest-agent --disable-sdl --disable-gtk --disable-vnc --disable-xen
--disable-brlapi --enable-fdt --disable-bluez --disable-vde --disable-rbd
--disable-libiscsi --disable-libnfs --disable-libusb --disable-usb-redir
--disable-seccomp --disable-glusterfs --disable-tpm --disable-numa
--disable-slirp --disable-blobs --target-list=ppc64-softmmu
$ make -j
$ virsh nodedev-detach pci_0005_01_00_0 --driver vfio
$ virsh nodedev-detach pci_0005_01_00_1 --driver vfio
$ virsh nodedev-detach pci_0005_01_00_2 --driver vfio
$ virsh nodedev-detach pci_0005_01_00_3 --driver vfio
$ virsh nodedev-detach pci_0005_01_00_4 --driver vfio
$ virsh nodedev-detach pci_0005_01_00_5 --driver vfio
$ sudo ppc64-softmmu/qemu-system-ppc64 -machine
pseries-4.1,accel=kvm,usb=off,dump-guest-core=off,cap-cfpc=broken,cap-
sbbc=broken,cap-ibs=broken -name guest=test-vfio-slowness -m 131072 -smp
1 -no-user-config -device spapr-pci-host-bridge,index=1,id=pci.1 -drive
file=/var/lib/uvtool/libvirt/images/test-huge-mem-
init.qcow,format=qcow2,if=none,id=drive-virtio-disk0 -device virtio-blk-
pci,scsi=off,bus=pci.0,addr=0x3,drive=drive-virtio-disk0,id=virtio-
disk0,bootindex=1 -device vfio-
pci,host=0005:01:00.0,id=hostdev0,bus=pci.1.0,addr=0x1 -device vfio-
pci,host=0005:01:00.1,id=hostdev1,bus=pci.1.0,addr=0x2 -device vfio-
pci,host=0005:01:00.2,id=hostdev2,bus=pci.1.0,addr=0x3 -device vfio-
pci,host=0005:01:00.3,id=hostdev3,bus=pci.1.0,addr=0x4 -device vfio-
pci,host=0005:01:00.4,id=hostdev4,bus=pci.1.0,addr=0x5 -device vfio-
pci,host=0005:01:00.5,id=hostdev5,bus=pci.1.0,addr=0x6 -msg timestamp=on
-display curses
I found VFIO_IOMMU_SPAPR_REGISTER_MEMORY:
96783 0.88 readlink("/sys/bus/pci/devices/0005:01:00.0/iommu_group",
"../../../../kernel/iommu_groups/"..., 4096) = 33 <0.22>
96783 0.66 openat(AT_FDCWD, "/dev/vfio/8", O_RDWR|O_CLOEXEC) = 16
<0.25>
96783 0.59 ioctl(16, VFIO_GROUP_GET_STATUS, 0x7fffe3fd6e20) = 0
<0.18>
96783 0.50 openat(AT_FDCWD, "/dev/vfio/vfio", O_RDWR|O_CLOEXEC) = 17
<0.14>
96783 0.49 ioctl(17, VFIO_GET_API_VERSION, 0) = 0 <0.08>
96783 0.39 ioctl(17, VFIO_CHECK_EXTENSION, 0x3) = 0 <0.11>
96783 0.40 ioctl(17, VFIO_CHECK_EXTENSION, 0x1) = 0 <0.08>
96783 0.37 ioctl(17, VFIO_CHECK_EXTENSION, 0x7) = 1 <0.08>
96783 0.37 ioctl(16, VFIO_GROUP_SET_CONTAINER, 0x65690e1bb48) = 0
<0.08>
96783 0.37 ioctl(17, VFIO_SET_IOMMU, 0x7) = 0 <0.39>
96783 0.70 ioctl(17, VFIO_IOMMU_SPAPR_REGISTER_MEMORY
96785 10.019032 <... futex resumed> ) = -1 ETIMEDOUT (Connection timed out)
<10.022751>
96785 0.053520 madvise(0x7fabd1f6, 8257536, MADV_DONTNEED) = 0
<0.20>
96785 0.007283 exit(0) = ?
96785 0.72 +++ exited with 0 +++
96783276.894553 <... ioctl resumed> , 0x7fffe3fd6b70) = 0 <286.974436>
96783 0.000107 --- SIGWINCH {si_signo=SIGWINCH, si_code=SI_KERNEL} ---
This is repeatable and explains why I haven't seen exactly the same that I saw
on x86.
VFIO_IOMMU_SPAPR_REGISTER_MEMORY is ppc specific (but with the same long hang
behavior).
Already there at least for ppc the documentation (kernel) says:
460- VFIO_IOMMU_SPAPR_REGISTER_MEMORY/VFIO_IOMMU_SPAPR_UNREGISTER_MEMORY
ioctls
461 receive a user space address and size of the block to be pinned.
462 Bisecting is not supported and VFIO_IOMMU_UNREGISTER_MEMORY is
expected to
463 be called with the exact address and size used for registering
464 the memory block. The userspace is not expected to call these often.
465 The ranges are stored in a linked list in a VFIO container.
The size seems to be all memory as I see:
reg.vaddr = (uintptr_t) vfio_prereg_gpa_to_vaddr(section, gpa);
reg.size = end - gpa;
And GDB confirms that is ALL of the guests memory (which explains the scaling
with memory size)
78 ret = ioctl(container->fd, VFIO_IOMMU_SPAPR_REGISTER_MEMORY, );
(gdb) p reg.size/1024/1024
$3 = 131072
"non bisectable" is the bad flag here.
It might be splittable in the kernel, but for this qemu can't do a lot as it
has to be a single range.
Now that we have this confirmed, lets search the same on x86.
--
You received this bug notification because you are a member of Kernel
Packages, which is subscribed to linux in Ubuntu.
https://bugs.launchpad.net/bugs/1838575
Title:
passthrough devices cause >17min boot delay
Status in linux package in Ubuntu:
Confirmed
Status in qemu package in Ubuntu:
Incomplete
Bug description:
Adding passthrough devices to a guest introduces a boot delay of > 17
minutes. During this time libvirt reports the guest to be "paused".
The delay does not seem to scale with the # of hostdevs - that is, 1
hostdev causes about a 17min delay, while 16
[Kernel-packages] [Bug 1838575] Re: passthrough devices cause >17min boot delay
Hmm, with strace showing almost a hang on a single of those ioctl calls you'D think that is easy to spot :-/ But this isn't as clear as expected: sudo trace-cmd record -p function_graph -l vfio_pci_ioctl -O graph-time Disable all but 1 CPUs to have less concurrency in the trace. => Not much better But it is the handler: 1276 »···.ioctl»·»···= vfio_pci_ioctl, There seem to be a lot of calls to vfio_pci_ioctl and not just a few slow ones. Whatever strace thought, that is not how it looks like. Starting again with wider traces to check what the entry might be. sys_splice is high on the radar atm. As a bonus now the test env broke and can't passthrough anymore. The emulex firmware breaks and fails. Maybe abort for now and focus on other things and resetup the testbed another day. -- You received this bug notification because you are a member of Kernel Packages, which is subscribed to linux in Ubuntu. https://bugs.launchpad.net/bugs/1838575 Title: passthrough devices cause >17min boot delay Status in linux package in Ubuntu: Confirmed Status in qemu package in Ubuntu: Incomplete Bug description: Adding passthrough devices to a guest introduces a boot delay of > 17 minutes. During this time libvirt reports the guest to be "paused". The delay does not seem to scale with the # of hostdevs - that is, 1 hostdev causes about a 17min delay, while 16 only bumps that up to ~18min. Removing all hostdevs avoids the delay. To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1838575/+subscriptions -- Mailing list: https://launchpad.net/~kernel-packages Post to : kernel-packages@lists.launchpad.net Unsubscribe : https://launchpad.net/~kernel-packages More help : https://help.launchpad.net/ListHelp
[Kernel-packages] [Bug 1838575] Re: passthrough devices cause >17min boot delay
On this platform strace still confirms the same paths: And perf as well (slight arch differences, but still mem setup). 46.85% [kernel] [k] lruvec_lru_size 16.89% [kernel] [k] clear_user_page 5.74% [kernel] [k] inactive_list_is_low.isra.0 4.48% [kernel] [k] find_next_bit 3.74% [kernel] [k] cpumask_next 2.72% [kernel] [k] copypage_power7 2.40% [kernel] [k] kvm_age_rmapp 1.94% [kernel] [k] snapshot_refaults 1.28% [kernel] [k] shrink_page_list Note: This is 5.2.0-10-generic Also cross arch, as soon as no hostdev is passed the init can be skipped. the 75 seconds above drop down to the ~7 seconds I have seen before the single thread init starts. -- You received this bug notification because you are a member of Kernel Packages, which is subscribed to linux in Ubuntu. https://bugs.launchpad.net/bugs/1838575 Title: passthrough devices cause >17min boot delay Status in linux package in Ubuntu: Confirmed Status in qemu package in Ubuntu: Incomplete Bug description: Adding passthrough devices to a guest introduces a boot delay of > 17 minutes. During this time libvirt reports the guest to be "paused". The delay does not seem to scale with the # of hostdevs - that is, 1 hostdev causes about a 17min delay, while 16 only bumps that up to ~18min. Removing all hostdevs avoids the delay. To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1838575/+subscriptions -- Mailing list: https://launchpad.net/~kernel-packages Post to : kernel-packages@lists.launchpad.net Unsubscribe : https://launchpad.net/~kernel-packages More help : https://help.launchpad.net/ListHelp
[Kernel-packages] [Bug 1838575] Re: passthrough devices cause >17min boot delay
As assumed this really seems to be cross arch and for all sizes. Here 16 PU, 128G on ppc64el: #1: 54 seconds #2: 7 seconds #3: 23 seconds Upped to 192GB this has: #1: 75 seconds #2: 5 seconds #3: 23 seconds As a note, in this case I checked there are ~7 seconds before it does into this "single thread, kernel busy" load. Those 7 seconds might be other/usual init in qemu/kvm while the long load is the Just the linear scaling is what breaks it, in case the same speed applies 10 times as much +1.28TB would be in the 10 minutes as well. The remaining difference might be dimm speed and bandwidth. -- You received this bug notification because you are a member of Kernel Packages, which is subscribed to linux in Ubuntu. https://bugs.launchpad.net/bugs/1838575 Title: passthrough devices cause >17min boot delay Status in linux package in Ubuntu: Confirmed Status in qemu package in Ubuntu: Incomplete Bug description: Adding passthrough devices to a guest introduces a boot delay of > 17 minutes. During this time libvirt reports the guest to be "paused". The delay does not seem to scale with the # of hostdevs - that is, 1 hostdev causes about a 17min delay, while 16 only bumps that up to ~18min. Removing all hostdevs avoids the delay. To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1838575/+subscriptions -- Mailing list: https://launchpad.net/~kernel-packages Post to : kernel-packages@lists.launchpad.net Unsubscribe : https://launchpad.net/~kernel-packages More help : https://help.launchpad.net/ListHelp
[Kernel-packages] [Bug 1838575] Re: passthrough devices cause >17min boot delay
You can do so even per-size via e.g. /sys/kernel/mm/hugepages/hugepages-1048576kB/nr_hugepages As discussed the later the allocation the higher the chance to fail, so re-check the sysfs file after each change if it actually got that much memory. The default size is only a boot time parameter. But you can specify explicit sizes in libvirt xml. -- You received this bug notification because you are a member of Kernel Packages, which is subscribed to linux in Ubuntu. https://bugs.launchpad.net/bugs/1838575 Title: passthrough devices cause >17min boot delay Status in linux package in Ubuntu: Confirmed Status in qemu package in Ubuntu: Incomplete Bug description: Adding passthrough devices to a guest introduces a boot delay of > 17 minutes. During this time libvirt reports the guest to be "paused". The delay does not seem to scale with the # of hostdevs - that is, 1 hostdev causes about a 17min delay, while 16 only bumps that up to ~18min. Removing all hostdevs avoids the delay. To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1838575/+subscriptions -- Mailing list: https://launchpad.net/~kernel-packages Post to : kernel-packages@lists.launchpad.net Unsubscribe : https://launchpad.net/~kernel-packages More help : https://help.launchpad.net/ListHelp
[Kernel-packages] [Bug 1838575] Re: passthrough devices cause >17min boot delay
** Changed in: linux (Ubuntu) Importance: Undecided => Medium ** Changed in: linux (Ubuntu) Assignee: (unassigned) => Colin Ian King (colin-king) -- You received this bug notification because you are a member of Kernel Packages, which is subscribed to linux in Ubuntu. https://bugs.launchpad.net/bugs/1838575 Title: passthrough devices cause >17min boot delay Status in linux package in Ubuntu: Confirmed Status in qemu package in Ubuntu: Incomplete Bug description: Adding passthrough devices to a guest introduces a boot delay of > 17 minutes. During this time libvirt reports the guest to be "paused". The delay does not seem to scale with the # of hostdevs - that is, 1 hostdev causes about a 17min delay, while 16 only bumps that up to ~18min. Removing all hostdevs avoids the delay. To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1838575/+subscriptions -- Mailing list: https://launchpad.net/~kernel-packages Post to : kernel-packages@lists.launchpad.net Unsubscribe : https://launchpad.net/~kernel-packages More help : https://help.launchpad.net/ListHelp
[Kernel-packages] [Bug 1838575] Re: passthrough devices cause >17min boot delay
Naive question: can we tweak the hugepage file settings at run time via /proc/sys/vm/nr_hugepages and not require the kernel parameters? -- You received this bug notification because you are a member of Kernel Packages, which is subscribed to linux in Ubuntu. https://bugs.launchpad.net/bugs/1838575 Title: passthrough devices cause >17min boot delay Status in linux package in Ubuntu: Confirmed Status in qemu package in Ubuntu: Incomplete Bug description: Adding passthrough devices to a guest introduces a boot delay of > 17 minutes. During this time libvirt reports the guest to be "paused". The delay does not seem to scale with the # of hostdevs - that is, 1 hostdev causes about a 17min delay, while 16 only bumps that up to ~18min. Removing all hostdevs avoids the delay. To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1838575/+subscriptions -- Mailing list: https://launchpad.net/~kernel-packages Post to : kernel-packages@lists.launchpad.net Unsubscribe : https://launchpad.net/~kernel-packages More help : https://help.launchpad.net/ListHelp
[Kernel-packages] [Bug 1838575] Re: passthrough devices cause >17min boot delay
Summary: As I mentioned before (on the other bug that I referred). The problem is that with a PT device it needs to reset and map the VDIO devices. So with >0 PT devices attached it needs an init that scales with memory size of the guest (see my fast results with PT but small guest memory). As I outlined before the "fix" for that is either: - (admin) use Huge pages (1G if possible) for the very huge guest which will make this way more efficient. - (kernel development) get the kernel to use more cores for this initialization (currently 1) Last time we disussed this Nvidia said they consider 1G Hugepages as recommended setup for huge guests (due to the benefits). But not for guests in general (due to the complexity managing the memory). I don't know where it ended, btu I expect it to be in some guidelines. That is why I wondered about th elack of hueg pages in the spec to test qemu passthrough perf if you remember. I don't think there is much more qemu can do at this point (at least I'd not know), but we can use this re-discovery of this issue as a chance. If you want to dive (or ask one of your teammates) into the kernel side to check if it could be made multi-cpu initialization for better scaling. I'm adding a kernel task for your consideration. ** Also affects: linux (Ubuntu) Importance: Undecided Status: New ** Changed in: linux (Ubuntu) Status: New => Confirmed ** Changed in: qemu (Ubuntu) Assignee: Rafael David Tinoco (rafaeldtinoco) => (unassigned) ** Changed in: qemu (Ubuntu) Status: Confirmed => Incomplete -- You received this bug notification because you are a member of Kernel Packages, which is subscribed to linux in Ubuntu. https://bugs.launchpad.net/bugs/1838575 Title: passthrough devices cause >17min boot delay Status in linux package in Ubuntu: Confirmed Status in qemu package in Ubuntu: Incomplete Bug description: Adding passthrough devices to a guest introduces a boot delay of > 17 minutes. During this time libvirt reports the guest to be "paused". The delay does not seem to scale with the # of hostdevs - that is, 1 hostdev causes about a 17min delay, while 16 only bumps that up to ~18min. Removing all hostdevs avoids the delay. To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1838575/+subscriptions -- Mailing list: https://launchpad.net/~kernel-packages Post to : kernel-packages@lists.launchpad.net Unsubscribe : https://launchpad.net/~kernel-packages More help : https://help.launchpad.net/ListHelp
