Related Merge Proposals: - qemu: https://code.launchpad.net/~paelzer/ubuntu/+source/qemu/+git/qemu/+merge/380467 - libvirt: https://code.launchpad.net/~paelzer/ubuntu/+source/libvirt/+git/libvirt/+merge/380469
I have already mailed Ubuntu-security on this in the past and now added a review slot for them on the MPs to feel safe for the "load .so from other place". I think it is as safe as the normal apth in /usr/lib/... but they are the experts to decide. Finally I have converted this (in the description) to also be an FFe and subscribed ubuntu-release to check and ack on it. ** Description changed: + [Feature Freeze Exception] + + Hi, + this is IMHO a just a bugfix. But since it involves some bigger changes I wanted to be on the safe side and get an ack by the release Team. + + Problem: + - on upgrade qemu processes are left running as they + represent a guest VM + - later trying to add features e.g. ceph disk hot-add will + need to load .so files e.g. from qemu-block-extra package + - those modules can on ly be loaded from the same build, but those are + gone after upgrade + + Solution: + - If qemu fails to load from its usual paths it will + now also look in /var/run/<version/ + - package upgrade code will place the .so's there + - things will be cleaned on reboot which is much simpler + and error-proof than trying to detect which versions + binaries are running + - libvirt has a change to allow just reading and + mapping from that path (apparmor) + + @Release team it would be great if you would agree to this being safe + for an FFe. + --- initial report --- Upgrading qemu binaries causes the on-disk versions to change, but the in-memory running instances still attempt to dynamically load a library which matches its same version. This can cause running instances to fail actions like hotplugging devices. This can be alleviated by migrating the instance to a new host or restarting the instance, however in cloud type environments there may be instances that cannot be migrated (sriov, etc) or the cloud operator does not have permission to reboot. This may be resolvable for many situations by changing the packaging to keep older versions of qemu libraries around on disk (similar to how the kernel package keeps older kernel versions around). --- testcase --- $ apt install uvtool-libvirt $ uvt-simplestreams-libvirt --verbose sync --source http://cloud-images.ubuntu.com/daily arch=amd64 label=daily release=bionic $ uvt-kvm create --password ubuntu lateload arch=amd64 release=bionic label=daily cat > curldisk.xml << EOF <disk type='network' device='disk'> <driver name='qemu' type='raw'/> <source protocol="http" name="ubuntu/dists/bionic-updates/main/installer-amd64/current/images/netboot/mini.iso"> <host name="archive.ubuntu.com" port="80"/> </source> <target dev='vdc' bus='virtio'/> <readonly/> </disk> EOF # Here up or downgrade the installed packages, even a minor # version or a rebuild of the same version $ virsh attach-device lateload curldisk.xml Reported issue happens on attach: root@b:~# virsh attach-device lateload cdrom-curl.xml error: Failed to attach device from cdrom-curl.xml error: internal error: unable to execute QEMU command 'device_add': Property 'virtio-blk-device.drive' can't find value 'drive-virtio-disk2' In the log we can see: Failed to initialize module: /usr/lib/x86_64-linux-gnu/qemu/block-curl.so Note: only modules from the same build can be loaded. --- solution options (WIP) --- For a packaging solution we would need: - qemu-block-extra / qemu-system-gui binary packages would need sort of a -$buildid in the name. That could be the version string (sanitized for package name) - /usr/lib/x86_64-linux-gnu/qemu/*.so would need a -$buildid - loading of modules from qemu would need to consider $buildid when creating module names. util/module.c in module_load_one / module_load_file It already searches in multiple dirs, maybe it could insert the $buildid there - We'd need a way of detecting running versions of qemu binaries and only make them uninstallable once the binaries are all gone. I have not seen something like that in apt yet (kernel is easy in comparison as only one can be loaded at a time). ALTERNATIVES: - disable loadable module support - add an option to load all modules in advance (unlikely to be liked upstream) and not desirable for many setups using qemu (especially not as default) - add an option to load a module (e.g via QMP/HMP) which would allow an admin to decide doing so for the few setups that benefit. - that could down the road then even get a libvirt interface for easier consumption Heads up - None of the above would be SRUable --- mitigation options --- - live migrate for upgrades - prohibited by SR-IOV usage - Tech to get SR-IOV migratable is coming (e.g. via net_failover, bonding in DPDK, ...) - load the modules you need in advance - Note: lacking an explicit "load module" command makes this slightly odd for now - but using iscsi or ceph is never spontaneous, a deployment has or hasn't the setup to use those - Create a single small read-only node and attach this to each guest, that will load the driver and render you immune to the issue. While more clunky, this isn't so much different than how it would be with an explicit "load module" command. - Actually the target doesn't have to exist it can fail to attach - and still achieves what is needed comment #17 has an example. + Actually the target doesn't have to exist it can fail to attach + and still achieves what is needed comment #17 has an example. -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1847361 Title: Upgrade of qemu binaries causes running instances not able to dynamically load modules To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/libvirt/+bug/1847361/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
