Public bug reported:
---Problem Description (by far...@us.ibm.com) ---
Cannot use vfio-ccw dasd passthrough for KVM guests under Ubuntu 20.04/22.04
Contact Information = Eric Farman <far...@linux.ibm.com>
---uname output---
Linux m34mkvmt5 5.4.0-135-generic #152-Ubuntu SMP Wed Nov 23 21:05:01 UTC 2022
s390x s390x s390x GNU/Linux
---Additional Hardware Info---
ECKD DASD, connected as a mediated device for KVM device passthrough
Machine Type = IBM z14 (3906) LPAR
---Debugger---
A debugger is not configured
---Steps to Reproduce---
Attempting to spawn a guest with a vfio-ccw hostdev device fails with an
AppArmor policy restriction on both 20.04 and 22.04, for files that QEMU
attempts to open for the device. The failure also occurs when trying to hotplug
such a device, which I'll use in these steps to keep the XML simple:
eric@kvmhost:~# chzdev -ea ca8b
eric@kvmhost:~# echo 0.0.ca8b > /sys/bus/ccw/drivers/dasd-eckd/unbind
eric@kvmhost:~# echo 0.0.0b16 > /sys/bus/css/drivers/io_subchannel/unbind
eric@kvmhost:~# echo 0.0.0b16 > /sys/bus/css/drivers/vfio_ccw/bind
eric@kvmhost:~# echo 11f2d2bc-4083-431d-a023-eff72715c4f0 >
/sys/bus/css/devices/0.0.0b16/mdev_supported_types/vfio_ccw-io/create
eric@kvmhost:~# cat hostdev.xml
<hostdev mode='subsystem' type='mdev' model='vfio-ccw'>
<source>
<address uuid='11f2d2bc-4083-431d-a023-eff72715c4f0'/>
</source>
<address type='ccw' cssid='0xfe' ssid='0x0' devno='0xca8b'/>
</hostdev>
eric@kvmhost:~# virsh attach-device guest hostdev.xml
error: Failed to attach device from hostdev.xml
error: internal error: unable to execute QEMU command 'device_add':
s390_ccw_realize: Failed to build initial schib: Invalid argument
eric@kvmhost:~# dmesg | grep DENIED
[ 5949.232089] audit: type=1400 audit(1670350246.709:22): apparmor="DENIED"
operation="open" profile="libvirt-0e995f6d-f85e-4ffe-b612-e07bfd62116a"
name="/sys/devices/css0/0.0.0b16/pimpampom" pid=1497 comm="qemu-system-s39"
requested_mask="r" denied_mask="r" fsuid=64055 ouid=0
While the failure occurs with the pimpampom file for the subchannel,
there are two others that QEMU would attempt to open after this:
eric:qemu$ git grep -B 2 -pn fopen hw/s390x/
hw/s390x/css.c=2577=static int css_sch_get_chpids(SubchDev *sch, CssDevId
*dev_id)
--
hw/s390x/css.c-2585- fid_path =
g_strdup_printf("/sys/bus/css/devices/%x.%x.%04x/chpids",
hw/s390x/css.c-2586- dev_id->cssid, dev_id->ssid,
dev_id->devid);
hw/s390x/css.c:2587: fd = fopen(fid_path, "r");
--
hw/s390x/css.c=2612=static int css_sch_get_path_masks(SubchDev *sch, CssDevId
*dev_id)
--
hw/s390x/css.c-2619- fid_path =
g_strdup_printf("/sys/bus/css/devices/%x.%x.%04x/pimpampom",
hw/s390x/css.c-2620- dev_id->cssid, dev_id->ssid,
dev_id->devid);
hw/s390x/css.c:2621: fd = fopen(fid_path, "r");
--
hw/s390x/css.c=2643=static int css_sch_get_chpid_type(uint8_t chpid, uint32_t
*type,
--
hw/s390x/css.c-2649- fid_path =
g_strdup_printf("/sys/devices/css%x/chp0.%02x/type",
hw/s390x/css.c-2650- dev_id->cssid, chpid);
hw/s390x/css.c:2651: fd = fopen(fid_path, "r");
The first two directories are links to the third, so I made the
following entry in /etc/apparmor.d/local/abstractions/libvirt-qemu which
Works For Me:
eric@kvmhost:~# cat /etc/apparmor.d/local/abstractions/libvirt-qemu
/sys/devices/css0/** r,
This is of course a very broad brush, so perhaps there's a better
deterministic way to the files in question for the subchannel(s) that
are requested. (I apologize if that deterministic logic is tied up in
the "hostdev networks" bug I see here:
https://bugs.launchpad.net/ubuntu/+source/libvirt/+bug/1677398)
For what it's worth, those files are not ones that remain open once the
device is connected to the guest:
eric@kvmhost:~# cat
/etc/apparmor.d/libvirt/libvirt-43b44ca9-d1c2-46f7-a686-2329a5a1d425.files
# DO NOT EDIT THIS FILE DIRECTLY. IT IS MANAGED BY LIBVIRT.
"/var/log/libvirt/**/guest.log" w,
"/var/lib/libvirt/qemu/domain-guest/monitor.sock" rw,
"/var/lib/libvirt/qemu/domain-3-guest/*" rw,
"/run/libvirt/**/guest.pid" rwk,
"/run/libvirt/**/*.tunnelmigrate.dest.guest" rw,
"/dev/dasdb" rwk,
"/dev/pts/2" rw,
"/dev/vhost-net" rw,
"/dev/vfio/2" rwk,
(The passed through DASD device is /dev/vfio/2 in the above list, not
/dev/dasdb. The latter is the guest rootfs, connected via virtio-blk.)
======================= Comment: <far...@us.ibm.com> - 2024-03-06 13:30:45
=====================================================================
Verified that this still misbehaves with 20.04.6 and 22.04.4. Both with the
manual sysfs changes described in the initial comment, and the more convenient
driverctl and mdevctl tooling.
======================= Comment: <far...@us.ibm.com> - 2024-03-06 13:34:28
=====================================================================
eric@host:~# virsh attach-device guest_3c4c hostdev.xml
error: Failed to attach device from hostdev.xml
error: internal error: unable to execute QEMU command 'device_add':
s390_ccw_realize: Failed to build initial schib: Invalid argument
eric@host:~# dmesg | grep 0165
[ 127.558194] vfio_ccw 0.0.0165: MDEV: Registered
[ 224.657413] audit: type=1400 audit(1709751698.761:42): apparmor="DENIED"
operation="open" profile="libvirt-ab758866-8077-4abb-b662-5f8972f93279"
name="/sys/devices/css0/0.0.0165/pimpampom" pid=1813 comm="qemu-system-s39"
requested_mask="r" denied_mask="r" fsuid=64055 ouid=0
eric@host:~# cat /etc/apparmor.d/local/abstractions/libvirt-qemu
eric@host:~# lsb_release -a
No LSB modules are available.
Distributor ID: Ubuntu
Description: Ubuntu 22.04.4 LTS
Release: 22.04
Codename: jammy
eric@host:~# uname -a
Linux host 5.15.0-97-generic #107-Ubuntu SMP Wed Feb 7 13:27:35 UTC 2024 s390x
s390x s390x GNU/Linux
eric@host:~# which qemu-system-s390x
/usr/bin/qemu-system-s390x
eric@host:~# qemu-system-s390x --version
QEMU emulator version 6.2.0 (Debian 1:6.2+dfsg-2ubuntu6.17)
Copyright (c) 2003-2021 Fabrice Bellard and the QEMU Project developers
** Affects: linux (Ubuntu)
Importance: Undecided
Assignee: Skipper Bug Screeners (skipper-screen-team)
Status: New
** Tags: architecture-s39064 bugnameltc-200694 severity-medium
targetmilestone-inin---
** Tags added: architecture-s39064 bugnameltc-200694 severity-medium
targetmilestone-inin---
** Changed in: ubuntu
Assignee: (unassigned) => Skipper Bug Screeners (skipper-screen-team)
** Package changed: ubuntu => linux (Ubuntu)
--
You received this bug notification because you are a member of Kernel
Packages, which is subscribed to linux in Ubuntu.
https://bugs.launchpad.net/bugs/2056441
Title:
[UBUNTU 20.04] Cannot use vfio-ccw dasd passthrough for KVM guests
under Ubuntu
Status in linux package in Ubuntu:
New
Bug description:
---Problem Description (by far...@us.ibm.com) ---
Cannot use vfio-ccw dasd passthrough for KVM guests under Ubuntu 20.04/22.04
Contact Information = Eric Farman <far...@linux.ibm.com>
---uname output---
Linux m34mkvmt5 5.4.0-135-generic #152-Ubuntu SMP Wed Nov 23 21:05:01 UTC
2022 s390x s390x s390x GNU/Linux
---Additional Hardware Info---
ECKD DASD, connected as a mediated device for KVM device passthrough
Machine Type = IBM z14 (3906) LPAR
---Debugger---
A debugger is not configured
---Steps to Reproduce---
Attempting to spawn a guest with a vfio-ccw hostdev device fails with an
AppArmor policy restriction on both 20.04 and 22.04, for files that QEMU
attempts to open for the device. The failure also occurs when trying to hotplug
such a device, which I'll use in these steps to keep the XML simple:
eric@kvmhost:~# chzdev -ea ca8b
eric@kvmhost:~# echo 0.0.ca8b > /sys/bus/ccw/drivers/dasd-eckd/unbind
eric@kvmhost:~# echo 0.0.0b16 > /sys/bus/css/drivers/io_subchannel/unbind
eric@kvmhost:~# echo 0.0.0b16 > /sys/bus/css/drivers/vfio_ccw/bind
eric@kvmhost:~# echo 11f2d2bc-4083-431d-a023-eff72715c4f0 >
/sys/bus/css/devices/0.0.0b16/mdev_supported_types/vfio_ccw-io/create
eric@kvmhost:~# cat hostdev.xml
<hostdev mode='subsystem' type='mdev' model='vfio-ccw'>
<source>
<address uuid='11f2d2bc-4083-431d-a023-eff72715c4f0'/>
</source>
<address type='ccw' cssid='0xfe' ssid='0x0' devno='0xca8b'/>
</hostdev>
eric@kvmhost:~# virsh attach-device guest hostdev.xml
error: Failed to attach device from hostdev.xml
error: internal error: unable to execute QEMU command 'device_add':
s390_ccw_realize: Failed to build initial schib: Invalid argument
eric@kvmhost:~# dmesg | grep DENIED
[ 5949.232089] audit: type=1400 audit(1670350246.709:22): apparmor="DENIED"
operation="open" profile="libvirt-0e995f6d-f85e-4ffe-b612-e07bfd62116a"
name="/sys/devices/css0/0.0.0b16/pimpampom" pid=1497 comm="qemu-system-s39"
requested_mask="r" denied_mask="r" fsuid=64055 ouid=0
While the failure occurs with the pimpampom file for the subchannel,
there are two others that QEMU would attempt to open after this:
eric:qemu$ git grep -B 2 -pn fopen hw/s390x/
hw/s390x/css.c=2577=static int css_sch_get_chpids(SubchDev *sch, CssDevId
*dev_id)
--
hw/s390x/css.c-2585- fid_path =
g_strdup_printf("/sys/bus/css/devices/%x.%x.%04x/chpids",
hw/s390x/css.c-2586- dev_id->cssid,
dev_id->ssid, dev_id->devid);
hw/s390x/css.c:2587: fd = fopen(fid_path, "r");
--
hw/s390x/css.c=2612=static int css_sch_get_path_masks(SubchDev *sch, CssDevId
*dev_id)
--
hw/s390x/css.c-2619- fid_path =
g_strdup_printf("/sys/bus/css/devices/%x.%x.%04x/pimpampom",
hw/s390x/css.c-2620- dev_id->cssid,
dev_id->ssid, dev_id->devid);
hw/s390x/css.c:2621: fd = fopen(fid_path, "r");
--
hw/s390x/css.c=2643=static int css_sch_get_chpid_type(uint8_t chpid, uint32_t
*type,
--
hw/s390x/css.c-2649- fid_path =
g_strdup_printf("/sys/devices/css%x/chp0.%02x/type",
hw/s390x/css.c-2650- dev_id->cssid, chpid);
hw/s390x/css.c:2651: fd = fopen(fid_path, "r");
The first two directories are links to the third, so I made the
following entry in /etc/apparmor.d/local/abstractions/libvirt-qemu
which Works For Me:
eric@kvmhost:~# cat /etc/apparmor.d/local/abstractions/libvirt-qemu
/sys/devices/css0/** r,
This is of course a very broad brush, so perhaps there's a better
deterministic way to the files in question for the subchannel(s) that
are requested. (I apologize if that deterministic logic is tied up in
the "hostdev networks" bug I see here:
https://bugs.launchpad.net/ubuntu/+source/libvirt/+bug/1677398)
For what it's worth, those files are not ones that remain open once
the device is connected to the guest:
eric@kvmhost:~# cat
/etc/apparmor.d/libvirt/libvirt-43b44ca9-d1c2-46f7-a686-2329a5a1d425.files
# DO NOT EDIT THIS FILE DIRECTLY. IT IS MANAGED BY LIBVIRT.
"/var/log/libvirt/**/guest.log" w,
"/var/lib/libvirt/qemu/domain-guest/monitor.sock" rw,
"/var/lib/libvirt/qemu/domain-3-guest/*" rw,
"/run/libvirt/**/guest.pid" rwk,
"/run/libvirt/**/*.tunnelmigrate.dest.guest" rw,
"/dev/dasdb" rwk,
"/dev/pts/2" rw,
"/dev/vhost-net" rw,
"/dev/vfio/2" rwk,
(The passed through DASD device is /dev/vfio/2 in the above list, not
/dev/dasdb. The latter is the guest rootfs, connected via virtio-blk.)
======================= Comment: <far...@us.ibm.com> - 2024-03-06 13:30:45
=====================================================================
Verified that this still misbehaves with 20.04.6 and 22.04.4. Both with the
manual sysfs changes described in the initial comment, and the more convenient
driverctl and mdevctl tooling.
======================= Comment: <far...@us.ibm.com> - 2024-03-06 13:34:28
=====================================================================
eric@host:~# virsh attach-device guest_3c4c hostdev.xml
error: Failed to attach device from hostdev.xml
error: internal error: unable to execute QEMU command 'device_add':
s390_ccw_realize: Failed to build initial schib: Invalid argument
eric@host:~# dmesg | grep 0165
[ 127.558194] vfio_ccw 0.0.0165: MDEV: Registered
[ 224.657413] audit: type=1400 audit(1709751698.761:42): apparmor="DENIED"
operation="open" profile="libvirt-ab758866-8077-4abb-b662-5f8972f93279"
name="/sys/devices/css0/0.0.0165/pimpampom" pid=1813 comm="qemu-system-s39"
requested_mask="r" denied_mask="r" fsuid=64055 ouid=0
eric@host:~# cat /etc/apparmor.d/local/abstractions/libvirt-qemu
eric@host:~# lsb_release -a
No LSB modules are available.
Distributor ID: Ubuntu
Description: Ubuntu 22.04.4 LTS
Release: 22.04
Codename: jammy
eric@host:~# uname -a
Linux host 5.15.0-97-generic #107-Ubuntu SMP Wed Feb 7 13:27:35 UTC 2024
s390x s390x s390x GNU/Linux
eric@host:~# which qemu-system-s390x
/usr/bin/qemu-system-s390x
eric@host:~# qemu-system-s390x --version
QEMU emulator version 6.2.0 (Debian 1:6.2+dfsg-2ubuntu6.17)
Copyright (c) 2003-2021 Fabrice Bellard and the QEMU Project developers
To manage notifications about this bug go to:
https://bugs.launchpad.net/ubuntu/+source/linux/+bug/2056441/+subscriptions
--
Mailing list: https://launchpad.net/~kernel-packages
Post to : kernel-packages@lists.launchpad.net
Unsubscribe : https://launchpad.net/~kernel-packages
More help : https://help.launchpad.net/ListHelp
