Hello, This series can be found on Github:
https://github.com/shvipin/linux vfio/liveupdate/v4 It adds the base support to enable Live Update feature for VFIO devices. In v3 of this series, both core PCI subsystem and VFIO changes were together. Based on the feedbacks on the patches, increasing complexities and the overall number of patches, we have split the effort in two parts, PCI and VFIO. v4 version of PCI work is here: https://lore.kernel.org/linux-pci/[email protected]/ This series is the VFIO part, built on top of PCI v4. Going forward, work on both subsystem will be independent and we may not be making same version releases in sync. However, VFIO changes will always be on the top of some version of PCI changes. Overview -------- This series made following changes to support VFIO live update: 1. VFIO Live Update feature is put behind VFIO_PCI_LIVEUPDATE config option. It also requires that VFIO_DEVICE_CDEV config is enabled. 1. To keep things simple in this series, enabling this option will disable VFIO_PCI_DMABUF. This is temporary. 1. A VFIO cdev FD can be preserved via Live Update session and retrieved after kexec via the same session. 2. Getting the preserved device FD after kexec via legacy VFIO group APIs or by directly opening /dev/vfio/devices/vfioX will error out. User has to get the FD from Live Update subsystem. 3. For now, only FD is preserved, underlying device and its states are not. 4. To keep the next kernel safe, just before the kexec, VFIO resets the device, disables BME and restores the original state of the device. This is temporary and will be removed when we have added full support of the Live Update. 5. New VFIO selftests have been added which interacts with VFIO - vfio_pci_liveupdate_uapi_test - vfio_pci_liveupdate_kexec_test liveupdate. 6. Documentations have been added for kernel docs. Dependencies ------------ This series has following dependencies as shown in the diagram |---------------------------| | VFIO v4 | |---------------------------| | Liveupdate Selftests[1] | |---------------------------| | PCI v4 [2] | |---------------------------| | Liveupdate Refcount [3] | |---------------------------| [1] https://lore.kernel.org/all/[email protected]/ These selftests changes are needed by other selftests also (iommu, kvm), so, I have sent them separately. [2] https://lore.kernel.org/linux-pci/[email protected]/ This is the PCI v4, which adds the live update support in PCI core. [3] https://lore.kernel.org/lkml/[email protected]/ PCI v4 is dependent on this. Github branch I have mentioned at the top has all of these dependencies. Testing ------- Series has been tested using new VFIO live update selftests in both qemu and bare-metal environment. In bare-metal environment, it was ran using the Intel DSA PCIe device. vfio_pci_liveupdate_kexec_test test is executed as following $ ./setup.sh 0000:00:04.0 $ ./vfio_pci_liveupdate_kexec_test --stage 1 0000:6a:01.0 $ kexec --reuse-cmdline -l -s /boot/vmlinuz $ kexec -e $ ./setup.sh 0000:00:04.0 $ ./vfio_pci_liveupdate_kexec_test --stage 2 0000:6a:01.0 Kernel commandline used for booting the VM and testing had following options passed - liveupdate=on - kho=on - vfio_iommu_type1.allow_unsafe_interrupts=Y - vfio_pci.enable_sriov=Y - vfio_pci.disable_denylist=Y - vfio.enable_unsafe_noiommu_mode=Y Changelog --------- v4: - Addressed sashiko feedback regarding issues in resetting the devize in freeze - Dropped the patch which was skipping reset of preserved VFIO device after kexec when user opens the device. This will be added back in future but for now not needed as freeze() is also resetting the device before kexec. - Kernel doc and comment updates based on feedbacks. v3: https://lore.kernel.org/kvm/[email protected]/ v2: https://lore.kernel.org/kvm/[email protected]/ v1: https://lore.kernel.org/kvm/[email protected]/ rfc: https://lore.kernel.org/kvm/[email protected]/ David Matlack (10): vfio/pci: Notify PCI subsystem about devices preserved across Live Update vfio: Enforce preserved devices are retrieved via LIVEUPDATE_SESSION_RETRIEVE_FD vfio/pci: Store incoming Live Update state in struct vfio_pci_core_device docs: liveupdate: Add documentation for VFIO PCI vfio: selftests: Add vfio_pci_liveupdate_uapi_test vfio: selftests: Add Makefile support for TEST_GEN_PROGS_EXTENDED vfio: selftests: Expose iommu_modes to tests vfio: selftests: Expose low-level helper routines for setting up struct vfio_pci_device vfio: selftests: Verify that opening VFIO device fails during Live Update vfio: selftests: Add continuous DMA to vfio_pci_liveupdate_kexec_test Vipin Sharma (6): vfio/pci: Register a file handler with Live Update Orchestrator vfio/pci: Preserve vfio-pci device files across Live Update vfio/pci: Retrieve preserved device files after Live Update vfio: selftests: Build liveupdate library in VFIO selftests vfio: selftests: Initialize vfio_pci_device using a VFIO cdev FD vfio: selftests: Add vfio_pci_liveupdate_kexec_test Documentation/core-api/liveupdate.rst | 1 + .../driver-api/vfio_pci_liveupdate.rst | 23 ++ MAINTAINERS | 2 + drivers/vfio/device_cdev.c | 67 +++- drivers/vfio/group.c | 9 + drivers/vfio/pci/Kconfig | 16 +- drivers/vfio/pci/Makefile | 1 + drivers/vfio/pci/vfio_pci.c | 13 +- drivers/vfio/pci/vfio_pci_core.c | 62 ++-- drivers/vfio/pci/vfio_pci_liveupdate.c | 349 ++++++++++++++++++ drivers/vfio/pci/vfio_pci_priv.h | 19 + drivers/vfio/vfio.h | 18 + drivers/vfio/vfio_main.c | 16 +- include/linux/kho/abi/vfio_pci.h | 43 +++ include/linux/vfio.h | 13 + include/linux/vfio_pci_core.h | 2 + tools/testing/selftests/vfio/Makefile | 23 +- .../vfio/lib/include/libvfio/iommu.h | 2 + .../lib/include/libvfio/vfio_pci_device.h | 8 + tools/testing/selftests/vfio/lib/iommu.c | 4 +- .../selftests/vfio/lib/vfio_pci_device.c | 60 ++- .../vfio/vfio_pci_liveupdate_kexec_test.c | 256 +++++++++++++ .../vfio/vfio_pci_liveupdate_uapi_test.c | 93 +++++ 23 files changed, 1041 insertions(+), 59 deletions(-) create mode 100644 Documentation/driver-api/vfio_pci_liveupdate.rst create mode 100644 drivers/vfio/pci/vfio_pci_liveupdate.c create mode 100644 include/linux/kho/abi/vfio_pci.h create mode 100644 tools/testing/selftests/vfio/vfio_pci_liveupdate_kexec_test.c create mode 100644 tools/testing/selftests/vfio/vfio_pci_liveupdate_uapi_test.c base-commit: 2a4c0c11c0193889446cdb6f1540cc2b9aff97dd -- 2.54.0.563.g4f69b47b94-goog
