Re: [PATCH v13 13/15] s390/vfio-ap: handle host AP config change notification
On Tue, 22 Dec 2020 20:16:04 -0500 Tony Krowiak wrote: > The motivation for config change notification is to enable the vfio_ap > device driver to handle hot plug/unplug of AP queues for a KVM guest as a > bulk operation. For example, if a new APID is dynamically assigned to the > host configuration, then a queue device will be created for each APQN that > can be formulated from the new APID and all APQIs already assigned to the > host configuration. Each of these new queue devices will get bound to their > respective driver one at a time, as they are created. In the case of the > vfio_ap driver, if the APQN of the queue device being bound to the driver > is assigned to a matrix mdev in use by a KVM guest, it will be hot plugged > into the guest if possible. Given that the AP architecture allows for 256 > adapters and 256 domains, one can see the possibility of the vfio_ap > driver's probe/remove callbacks getting invoked an inordinate number of > times when the host configuration changes. Keep in mind that in order to > plug/unplug an AP queue for a guest, the guest's VCPUs must be suspended, > then the guest's AP configuration must be updated followed by the VCPUs > being resumed. If this is done each time the probe or remove callback is > invoked and there are hundreds or thousands of queues to be probed or > removed, this would be incredibly inefficient and could have a large impact > on guest performance. What the config notification does is allow us to > make the changes to the guest in a single operation. > > This patch implements the on_cfg_changed callback which notifies the > AP device drivers that the host AP configuration has changed (i.e., > adapters, domains and/or control domains are added to or removed from the > host AP configuration). > > Adapters added to host configuration: > * The APIDs of the adapters added will be stored in a bitmap contained > within the struct representing the matrix device which is the parent > device of all matrix mediated devices. > * When a queue is probed, if the APQN of the queue being probed is > assigned to an mdev in use by a guest, the queue may get hot plugged > into the guest; however, if the APID of the adapter is contained in the > bitmap of adapters added, the queue hot plug operation will be skipped > until the AP bus notifies the driver that its scan operation has > completed (another patch). I guess, I should be able to find this in patch 14. But I can't. > * When the vfio_ap driver is notified that the AP bus scan has completed, > the guest's APCB will be refreshed by filtering the mdev's matrix by > APID. > > Domains added to host configuration: > * The APQIs of the domains added will be stored in a bitmap contained > within the struct representing the matrix device which is the parent > device of all matrix mediated devices. > * When a queue is probed, if the APQN of the queue being probed is > assigned to an mdev in use by a guest, the queue may get hot plugged > into the guest; however, if the APQI of the domain is contained in the > bitmap of domains added, the queue hot plug operation will be skipped > until the AP bus notifies the driver that its scan operation has > completed (another patch). > > Control domains added to the host configuration: > * The domain numbers of the domains added will be stored in a bitmap > contained within the struct representing the matrix device which is the > parent device of all matrix mediated devices. > > When the vfio_ap device driver is notified that the AP bus scan has > completed, the APCB for each matrix mdev to which the adapters, domains > and control domains added are assigned will be refreshed. If a KVM guest is > using the matrix mdev, the APCB will be hot plugged into the guest to > refresh its AP configuration. > > Adapters removed from configuration: > * Each queue device with the APID identifying an adapter removed from > the host AP configuration will be unlinked from the matrix mdev to which > the queue's APQN is assigned. > * When the vfio_ap driver's remove callback is invoked, if the queue > device is not linked to the matrix mdev, the refresh of the guest's > APCB will be skipped. > > Domains removed from configuration: > * Each queue device with the APQI identifying a domain removed from > the host AP configuration will be unlinked from the matrix mdev to which > the queue's APQN is assigned. > * When the vfio_ap driver's remove callback is invoked, if the queue > device is not linked to the matrix mdev, the refresh of the guest's > APCB will be skipped. > > If any queues with an APQN assigned to a given matrix mdev have been > unlinked or any control domains assigned to a given matrix mdev have been > removed from the host AP configuration, the APCB of the matrix mdev will > be refreshed. If a KVM guest is using the matrix mdev, the APCB will be hot > plugged into the guest to refresh its AP configuration. > >
[PATCH v13 13/15] s390/vfio-ap: handle host AP config change notification
The motivation for config change notification is to enable the vfio_ap device driver to handle hot plug/unplug of AP queues for a KVM guest as a bulk operation. For example, if a new APID is dynamically assigned to the host configuration, then a queue device will be created for each APQN that can be formulated from the new APID and all APQIs already assigned to the host configuration. Each of these new queue devices will get bound to their respective driver one at a time, as they are created. In the case of the vfio_ap driver, if the APQN of the queue device being bound to the driver is assigned to a matrix mdev in use by a KVM guest, it will be hot plugged into the guest if possible. Given that the AP architecture allows for 256 adapters and 256 domains, one can see the possibility of the vfio_ap driver's probe/remove callbacks getting invoked an inordinate number of times when the host configuration changes. Keep in mind that in order to plug/unplug an AP queue for a guest, the guest's VCPUs must be suspended, then the guest's AP configuration must be updated followed by the VCPUs being resumed. If this is done each time the probe or remove callback is invoked and there are hundreds or thousands of queues to be probed or removed, this would be incredibly inefficient and could have a large impact on guest performance. What the config notification does is allow us to make the changes to the guest in a single operation. This patch implements the on_cfg_changed callback which notifies the AP device drivers that the host AP configuration has changed (i.e., adapters, domains and/or control domains are added to or removed from the host AP configuration). Adapters added to host configuration: * The APIDs of the adapters added will be stored in a bitmap contained within the struct representing the matrix device which is the parent device of all matrix mediated devices. * When a queue is probed, if the APQN of the queue being probed is assigned to an mdev in use by a guest, the queue may get hot plugged into the guest; however, if the APID of the adapter is contained in the bitmap of adapters added, the queue hot plug operation will be skipped until the AP bus notifies the driver that its scan operation has completed (another patch). * When the vfio_ap driver is notified that the AP bus scan has completed, the guest's APCB will be refreshed by filtering the mdev's matrix by APID. Domains added to host configuration: * The APQIs of the domains added will be stored in a bitmap contained within the struct representing the matrix device which is the parent device of all matrix mediated devices. * When a queue is probed, if the APQN of the queue being probed is assigned to an mdev in use by a guest, the queue may get hot plugged into the guest; however, if the APQI of the domain is contained in the bitmap of domains added, the queue hot plug operation will be skipped until the AP bus notifies the driver that its scan operation has completed (another patch). Control domains added to the host configuration: * The domain numbers of the domains added will be stored in a bitmap contained within the struct representing the matrix device which is the parent device of all matrix mediated devices. When the vfio_ap device driver is notified that the AP bus scan has completed, the APCB for each matrix mdev to which the adapters, domains and control domains added are assigned will be refreshed. If a KVM guest is using the matrix mdev, the APCB will be hot plugged into the guest to refresh its AP configuration. Adapters removed from configuration: * Each queue device with the APID identifying an adapter removed from the host AP configuration will be unlinked from the matrix mdev to which the queue's APQN is assigned. * When the vfio_ap driver's remove callback is invoked, if the queue device is not linked to the matrix mdev, the refresh of the guest's APCB will be skipped. Domains removed from configuration: * Each queue device with the APQI identifying a domain removed from the host AP configuration will be unlinked from the matrix mdev to which the queue's APQN is assigned. * When the vfio_ap driver's remove callback is invoked, if the queue device is not linked to the matrix mdev, the refresh of the guest's APCB will be skipped. If any queues with an APQN assigned to a given matrix mdev have been unlinked or any control domains assigned to a given matrix mdev have been removed from the host AP configuration, the APCB of the matrix mdev will be refreshed. If a KVM guest is using the matrix mdev, the APCB will be hot plugged into the guest to refresh its AP configuration. Signed-off-by: Tony Krowiak --- drivers/s390/crypto/vfio_ap_drv.c | 3 +- drivers/s390/crypto/vfio_ap_ops.c | 159 +++--- drivers/s390/crypto/vfio_ap_private.h | 13 ++- 3 files changed, 158 insertions(+), 17 deletions(-) diff --git a/drivers/s390/crypto/vfio_ap_drv.c
