Hi,
This is a follow up to the fifth RFC patchset [0] for supporting multiple
accelerated vSMMU instances. Support for multiple non-accel vSMMUs has
been merged [1], while the iommufd patches have been separated into a
separated series [2].
This patchset implements support for specifying multiple accelerated
vSMMU instances, and is tested with Shameer's v6 qemu series [3] for
accelerated SMMUs.
Moreover, this patchset implements support for Tegra241 CMDQV for
accelerated SMMUs, and is tested with Shameer's v1 qemu RFC [4] for
accelerated SMMU Tegra241 CMDQV support.
For instance, specifying hostdevs associated with multiple
accelerated SMMUs + CMDQV enabled, configured to be routed to
pcie-expander-bus controllers in a way where VFIO device to SMMUv3
associations are matched with the host:
<devices>
...
<controller type='pci' index='1' model='pcie-expander-bus'>
<model name='pxb-pcie'/>
<target busNr='252'/>
<address type='pci' domain='0x0000' bus='0x00' slot='0x01'
function='0x0'/>
</controller>
<controller type='pci' index='2' model='pcie-expander-bus'>
<model name='pxb-pcie'/>
<target busNr='248'/>
<address type='pci' domain='0x0000' bus='0x00' slot='0x02'
function='0x0'/>
</controller>
...
<controller type='pci' index='21' model='pcie-root-port'>
<model name='pcie-root-port'/>
<target chassis='21' port='0x0'/>
<address type='pci' domain='0x0000' bus='0x01' slot='0x00'
function='0x0'/>
</controller>
<controller type='pci' index='22' model='pcie-root-port'>
<model name='pcie-root-port'/>
<target chassis='22' port='0xa8'/>
<address type='pci' domain='0x0000' bus='0x02' slot='0x00'
function='0x0'/>
</controller>
...
<hostdev mode='subsystem' type='pci' managed='no'>
<source>
<address domain='0x0009' bus='0x01' slot='0x00' function='0x0'/>
</source>
<address type='pci' domain='0x0000' bus='0x15' slot='0x00'
function='0x0'/>
</hostdev>
<hostdev mode='subsystem' type='pci' managed='no'>
<source>
<address domain='0x0019' bus='0x01' slot='0x00' function='0x0'/>
</source>
<address type='pci' domain='0x0000' bus='0x16' slot='0x00'
function='0x0'/>
</hostdev>
<iommu model='smmuv3Dev'>
<driver parentIdx='1' accel='on' cmdqv='on' ats='on' ril='on' pasid='on'
oas='44'/>
</iommu>
<iommu model='smmuv3Dev'>
<driver parentIdx='2' accel='on' cmdqv='on' ats='on' ril='on' pasid='on'
oas='44'/>
</iommu>
</devices>
This would get translated to a qemu command line with the arguments
below:
-device
'{"driver":"pxb-pcie","bus_nr":252,"id":"pci.1","bus":"pcie.0","addr":"0x1"}' \
-device
'{"driver":"pxb-pcie","bus_nr":248,"id":"pci.2","bus":"pcie.0","addr":"0x2"}' \
-device
'{"driver":"pcie-root-port","port":0,"chassis":21,"id":"pci.21","bus":"pci.1","addr":"0x0"}'
\
-device
'{"driver":"pcie-root-port","port":168,"chassis":22,"id":"pci.22","bus":"pci.2","addr":"0x0"}'
\
-device
'{"driver":"arm-smmuv3","primary-bus":"pci.1","id":"iommu0","accel":true,"tegra241-cmdqv":true,"ats":true,"ril":true,"pasid":true,"oas":44}'
\
-device
'{"driver":"arm-smmuv3","primary-bus":"pci.2","id":"iommu1","accel":true,"tegra241-cmdqv":true,"ats":true,"ril":true,"pasid":true,"oas":44}'
\
-device
'{"driver":"vfio-pci","host":"0009:01:00.0","id":"hostdev0","bus":"pci.21","addr":"0x0"}'
\
-device
'{"driver":"vfio-pci","host":"0019:01:00.0","id":"hostdev1","bus":"pci.22","addr":"0x0"}'
\
This series is on Github:
https://github.com/NathanChenNVIDIA/libvirt/tree/smmuv3-accel-cmdqv-01-26/
Thanks,
Nathan
[0]
https://lists.libvirt.org/archives/list/[email protected]/thread/F4XIUZCDNMXGW5WBJ47VLVSZMGXO5B6E/
[1]
https://lists.libvirt.org/archives/list/[email protected]/thread/WOJKDS3N23N63SMMDLLTZJRVXWPHK7EM/
[2]
https://lists.libvirt.org/archives/list/[email protected]/thread/WIBZ6HRZGVCDXJBOWYROHYAQY5XBSABW/
[3]
https://lore.kernel.org/all/[email protected]/#r
[4] https://lore.kernel.org/all/[email protected]/
Nathan Chen (4):
qemu: Add support for HW-accelerated nested SMMUv3
tests: qemuxmlconfdata: provide HW-accel smmuv3 sample XML and CLI
args
qemu: add IOMMU attribute "cmdqv" for smmuv3
tests: qemuxmlconfdata: provide cmdqv sample XML and CLI args
docs/formatdomain.rst | 32 +++++++
src/conf/domain_conf.c | 93 ++++++++++++++++++-
src/conf/domain_conf.h | 6 ++
src/conf/domain_validate.c | 24 ++++-
src/conf/schemas/domaincommon.rng | 30 ++++++
src/qemu/qemu_command.c | 12 +++
...v3-pci-bus-accel-cmdqv.aarch64-latest.args | 41 ++++++++
...uv3-pci-bus-accel-cmdqv.aarch64-latest.xml | 62 +++++++++++++
.../iommu-smmuv3-pci-bus-accel-cmdqv.xml | 49 ++++++++++
...u-smmuv3-pci-bus-accel.aarch64-latest.args | 41 ++++++++
...mu-smmuv3-pci-bus-accel.aarch64-latest.xml | 62 +++++++++++++
.../iommu-smmuv3-pci-bus-accel.xml | 49 ++++++++++
...-smmuv3-pci-bus-single.aarch64-latest.args | 2 +-
.../iommu-smmuv3-pci-bus.aarch64-latest.args | 4 +-
tests/qemuxmlconftest.c | 2 +
15 files changed, 502 insertions(+), 7 deletions(-)
create mode 100644
tests/qemuxmlconfdata/iommu-smmuv3-pci-bus-accel-cmdqv.aarch64-latest.args
create mode 100644
tests/qemuxmlconfdata/iommu-smmuv3-pci-bus-accel-cmdqv.aarch64-latest.xml
create mode 100644 tests/qemuxmlconfdata/iommu-smmuv3-pci-bus-accel-cmdqv.xml
create mode 100644
tests/qemuxmlconfdata/iommu-smmuv3-pci-bus-accel.aarch64-latest.args
create mode 100644
tests/qemuxmlconfdata/iommu-smmuv3-pci-bus-accel.aarch64-latest.xml
create mode 100644 tests/qemuxmlconfdata/iommu-smmuv3-pci-bus-accel.xml
--
2.43.0
