You have been subscribed to a public bug: ---Problem Description (by far...@us.ibm.com) --- Cannot use vfio-ccw dasd passthrough for KVM guests under Ubuntu 20.04/22.04 Contact Information = Eric Farman <far...@linux.ibm.com> ---uname output--- Linux m34mkvmt5 5.4.0-135-generic #152-Ubuntu SMP Wed Nov 23 21:05:01 UTC 2022 s390x s390x s390x GNU/Linux ---Additional Hardware Info--- ECKD DASD, connected as a mediated device for KVM device passthrough
Machine Type = IBM z14 (3906) LPAR ---Debugger--- A debugger is not configured ---Steps to Reproduce--- Attempting to spawn a guest with a vfio-ccw hostdev device fails with an AppArmor policy restriction on both 20.04 and 22.04, for files that QEMU attempts to open for the device. The failure also occurs when trying to hotplug such a device, which I'll use in these steps to keep the XML simple: eric@kvmhost:~# chzdev -ea ca8b eric@kvmhost:~# echo 0.0.ca8b > /sys/bus/ccw/drivers/dasd-eckd/unbind eric@kvmhost:~# echo 0.0.0b16 > /sys/bus/css/drivers/io_subchannel/unbind eric@kvmhost:~# echo 0.0.0b16 > /sys/bus/css/drivers/vfio_ccw/bind eric@kvmhost:~# echo 11f2d2bc-4083-431d-a023-eff72715c4f0 > /sys/bus/css/devices/0.0.0b16/mdev_supported_types/vfio_ccw-io/create eric@kvmhost:~# cat hostdev.xml <hostdev mode='subsystem' type='mdev' model='vfio-ccw'> <source> <address uuid='11f2d2bc-4083-431d-a023-eff72715c4f0'/> </source> <address type='ccw' cssid='0xfe' ssid='0x0' devno='0xca8b'/> </hostdev> eric@kvmhost:~# virsh attach-device guest hostdev.xml error: Failed to attach device from hostdev.xml error: internal error: unable to execute QEMU command 'device_add': s390_ccw_realize: Failed to build initial schib: Invalid argument eric@kvmhost:~# dmesg | grep DENIED [ 5949.232089] audit: type=1400 audit(1670350246.709:22): apparmor="DENIED" operation="open" profile="libvirt-0e995f6d-f85e-4ffe-b612-e07bfd62116a" name="/sys/devices/css0/0.0.0b16/pimpampom" pid=1497 comm="qemu-system-s39" requested_mask="r" denied_mask="r" fsuid=64055 ouid=0 While the failure occurs with the pimpampom file for the subchannel, there are two others that QEMU would attempt to open after this: eric:qemu$ git grep -B 2 -pn fopen hw/s390x/ hw/s390x/css.c=2577=static int css_sch_get_chpids(SubchDev *sch, CssDevId *dev_id) -- hw/s390x/css.c-2585- fid_path = g_strdup_printf("/sys/bus/css/devices/%x.%x.%04x/chpids", hw/s390x/css.c-2586- dev_id->cssid, dev_id->ssid, dev_id->devid); hw/s390x/css.c:2587: fd = fopen(fid_path, "r"); -- hw/s390x/css.c=2612=static int css_sch_get_path_masks(SubchDev *sch, CssDevId *dev_id) -- hw/s390x/css.c-2619- fid_path = g_strdup_printf("/sys/bus/css/devices/%x.%x.%04x/pimpampom", hw/s390x/css.c-2620- dev_id->cssid, dev_id->ssid, dev_id->devid); hw/s390x/css.c:2621: fd = fopen(fid_path, "r"); -- hw/s390x/css.c=2643=static int css_sch_get_chpid_type(uint8_t chpid, uint32_t *type, -- hw/s390x/css.c-2649- fid_path = g_strdup_printf("/sys/devices/css%x/chp0.%02x/type", hw/s390x/css.c-2650- dev_id->cssid, chpid); hw/s390x/css.c:2651: fd = fopen(fid_path, "r"); The first two directories are links to the third, so I made the following entry in /etc/apparmor.d/local/abstractions/libvirt-qemu which Works For Me: eric@kvmhost:~# cat /etc/apparmor.d/local/abstractions/libvirt-qemu /sys/devices/css0/** r, This is of course a very broad brush, so perhaps there's a better deterministic way to the files in question for the subchannel(s) that are requested. (I apologize if that deterministic logic is tied up in the "hostdev networks" bug I see here: https://bugs.launchpad.net/ubuntu/+source/libvirt/+bug/1677398) For what it's worth, those files are not ones that remain open once the device is connected to the guest: eric@kvmhost:~# cat /etc/apparmor.d/libvirt/libvirt-43b44ca9-d1c2-46f7-a686-2329a5a1d425.files # DO NOT EDIT THIS FILE DIRECTLY. IT IS MANAGED BY LIBVIRT. "/var/log/libvirt/**/guest.log" w, "/var/lib/libvirt/qemu/domain-guest/monitor.sock" rw, "/var/lib/libvirt/qemu/domain-3-guest/*" rw, "/run/libvirt/**/guest.pid" rwk, "/run/libvirt/**/*.tunnelmigrate.dest.guest" rw, "/dev/dasdb" rwk, "/dev/pts/2" rw, "/dev/vhost-net" rw, "/dev/vfio/2" rwk, (The passed through DASD device is /dev/vfio/2 in the above list, not /dev/dasdb. The latter is the guest rootfs, connected via virtio-blk.) ======================= Comment: <far...@us.ibm.com> - 2024-03-06 13:30:45 ===================================================================== Verified that this still misbehaves with 20.04.6 and 22.04.4. Both with the manual sysfs changes described in the initial comment, and the more convenient driverctl and mdevctl tooling. ======================= Comment: <far...@us.ibm.com> - 2024-03-06 13:34:28 ===================================================================== eric@host:~# virsh attach-device guest_3c4c hostdev.xml error: Failed to attach device from hostdev.xml error: internal error: unable to execute QEMU command 'device_add': s390_ccw_realize: Failed to build initial schib: Invalid argument eric@host:~# dmesg | grep 0165 [ 127.558194] vfio_ccw 0.0.0165: MDEV: Registered [ 224.657413] audit: type=1400 audit(1709751698.761:42): apparmor="DENIED" operation="open" profile="libvirt-ab758866-8077-4abb-b662-5f8972f93279" name="/sys/devices/css0/0.0.0165/pimpampom" pid=1813 comm="qemu-system-s39" requested_mask="r" denied_mask="r" fsuid=64055 ouid=0 eric@host:~# cat /etc/apparmor.d/local/abstractions/libvirt-qemu eric@host:~# lsb_release -a No LSB modules are available. Distributor ID: Ubuntu Description: Ubuntu 22.04.4 LTS Release: 22.04 Codename: jammy eric@host:~# uname -a Linux host 5.15.0-97-generic #107-Ubuntu SMP Wed Feb 7 13:27:35 UTC 2024 s390x s390x s390x GNU/Linux eric@host:~# which qemu-system-s390x /usr/bin/qemu-system-s390x eric@host:~# qemu-system-s390x --version QEMU emulator version 6.2.0 (Debian 1:6.2+dfsg-2ubuntu6.17) Copyright (c) 2003-2021 Fabrice Bellard and the QEMU Project developers ** Affects: linux (Ubuntu) Importance: Undecided Assignee: Skipper Bug Screeners (skipper-screen-team) Status: New ** Tags: architecture-s39064 bugnameltc-200694 severity-medium targetmilestone-inin--- -- [UBUNTU 20.04] Cannot use vfio-ccw dasd passthrough for KVM guests under Ubuntu https://bugs.launchpad.net/bugs/2056441 You received this bug notification because you are a member of Kernel Packages, which is subscribed to linux in Ubuntu. -- Mailing list: https://launchpad.net/~kernel-packages Post to : kernel-packages@lists.launchpad.net Unsubscribe : https://launchpad.net/~kernel-packages More help : https://help.launchpad.net/ListHelp