On Fri, Feb 11, 2022 at 08:26:10AM +0100, Klaus Jensen wrote: > On Jan 26 18:11, Lukasz Maniak wrote: > > Changes since v3: > > - Addressed comments to review on pcie: Add support for Single Root I/O > > Virtualization (SR/IOV) > > - Fixed issues reported by checkpatch.pl > > > > Knut Omang (2): > > pcie: Add support for Single Root I/O Virtualization (SR/IOV) > > pcie: Add some SR/IOV API documentation in docs/pcie_sriov.txt > > > > Lukasz Maniak (4): > > hw/nvme: Add support for SR-IOV > > hw/nvme: Add support for Primary Controller Capabilities > > hw/nvme: Add support for Secondary Controller List > > docs: Add documentation for SR-IOV and Virtualization Enhancements > > > > Ćukasz Gieryk (9): > > pcie: Add a helper to the SR/IOV API > > pcie: Add 1.2 version token for the Power Management Capability > > hw/nvme: Implement the Function Level Reset > > hw/nvme: Make max_ioqpairs and msix_qsize configurable in runtime > > hw/nvme: Remove reg_size variable and update BAR0 size calculation > > hw/nvme: Calculate BAR attributes in a function > > hw/nvme: Initialize capability structures for primary/secondary > > controllers > > hw/nvme: Add support for the Virtualization Management command > > hw/nvme: Update the initalization place for the AER queue > > > > docs/pcie_sriov.txt | 115 ++++++ > > docs/system/devices/nvme.rst | 36 ++ > > hw/nvme/ctrl.c | 675 ++++++++++++++++++++++++++++++++--- > > hw/nvme/ns.c | 2 +- > > hw/nvme/nvme.h | 55 ++- > > hw/nvme/subsys.c | 75 +++- > > hw/nvme/trace-events | 6 + > > hw/pci/meson.build | 1 + > > hw/pci/pci.c | 100 ++++-- > > hw/pci/pcie.c | 5 + > > hw/pci/pcie_sriov.c | 302 ++++++++++++++++ > > hw/pci/trace-events | 5 + > > include/block/nvme.h | 65 ++++ > > include/hw/pci/pci.h | 12 +- > > include/hw/pci/pci_ids.h | 1 + > > include/hw/pci/pci_regs.h | 1 + > > include/hw/pci/pcie.h | 6 + > > include/hw/pci/pcie_sriov.h | 77 ++++ > > include/qemu/typedefs.h | 2 + > > 19 files changed, 1460 insertions(+), 81 deletions(-) > > create mode 100644 docs/pcie_sriov.txt > > create mode 100644 hw/pci/pcie_sriov.c > > create mode 100644 include/hw/pci/pcie_sriov.h > > > > -- > > 2.25.1 > > > > > > Hi Lukasz, > > Back in v3 you changed this: > > - Secondary controller cannot be set online unless the corresponding VF > is enabled (sriov_numvfs set to at least the secondary controller's VF > number) > > I'm having issues getting this to work now. As I understand it, this now > requires that sriov_numvfs is set prior to onlining the devices, i.e.: > > echo 1 > /sys/bus/pci/devices/0000\:01\:00.0/sriov_numvfs > > However, this causes the kernel to reject it: > > nvme nvme1: Device not ready; aborting initialisation, CSTS=0x2 > nvme nvme1: Removing after probe failure status: -19 > > Is this the expected behavior? Must I manually bind the device again to > the nvme driver? Prior to v3 this worked just fine since the VF was > onlined at this point. > > It would be useful if you added a small "onlining for dummies" section > to the docs ;)
Hi Klaus, Yes, this is the expected behavior and yeah it is less user friendly than in v3. Yet, after re-examining the NVMe specification, we concluded that this is how it should work. This is now the correct minimum flow needed to run a VF-based functional NVMe controller: # Unbind all flexible resources from the primary controller nvme virt-mgmt /dev/nvme0 -c 0 -r 1 -a 1 -n 0 nvme virt-mgmt /dev/nvme0 -c 0 -r 0 -a 1 -n 0 # Reset the primary controller to actually release the resources echo 1 > /sys/bus/pci/devices/0000:01:00.0/reset # Enable VF echo 1 > /sys/bus/pci/devices/0000:01:00.0/sriov_numvfs # Assign flexible resources to VF and set it ONLINE nvme virt-mgmt /dev/nvme0 -c 1 -r 1 -a 8 -n 21 nvme virt-mgmt /dev/nvme0 -c 1 -r 0 -a 8 -n 21 nvme virt-mgmt /dev/nvme0 -c 1 -r 0 -a 9 -n 0 # Bind NVMe driver for VF controller echo 0000:01:00.1 > /sys/bus/pci/drivers/nvme/bind I will update the docs. Thanks, Lukasz