Hi,
This is a follow up to the v10 patchset [0] for supporting multiple
hardware-accelerated vSMMU instances.
For instance, specifying hostdevs associated with multiple accelerated
SMMUs, configured to be routed to pcie-expander-bus controllers in a way
where VFIO device to SMMUv3 associations are matched with the host:
<devices>
...
<controller type='pci' index='1' model='pcie-expander-bus'>
<model name='pxb-pcie'/>
<target busNr='252'/>
<address type='pci' domain='0x0000' bus='0x00' slot='0x01'
function='0x0'/>
</controller>
<controller type='pci' index='2' model='pcie-expander-bus'>
<model name='pxb-pcie'/>
<target busNr='248'/>
<address type='pci' domain='0x0000' bus='0x00' slot='0x02'
function='0x0'/>
</controller>
...
<controller type='pci' index='21' model='pcie-root-port'>
<model name='pcie-root-port'/>
<target chassis='21' port='0x0'/>
<address type='pci' domain='0x0000' bus='0x01' slot='0x00'
function='0x0'/>
</controller>
<controller type='pci' index='22' model='pcie-root-port'>
<model name='pcie-root-port'/>
<target chassis='22' port='0xa8'/>
<address type='pci' domain='0x0000' bus='0x02' slot='0x00'
function='0x0'/>
</controller>
...
<hostdev mode='subsystem' type='pci' managed='no'>
<source>
<address domain='0x0009' bus='0x01' slot='0x00' function='0x0'/>
</source>
<address type='pci' domain='0x0000' bus='0x15' slot='0x00'
function='0x0'/>
</hostdev>
<hostdev mode='subsystem' type='pci' managed='no'>
<source>
<address domain='0x0019' bus='0x01' slot='0x00' function='0x0'/>
</source>
<address type='pci' domain='0x0000' bus='0x16' slot='0x00'
function='0x0'/>
</hostdev>
<iommu model='smmuv3'>
<driver pciBus='1' accel='on' ats='on' ril='off' ssidsize='20' oas='44'/>
</iommu>
<iommu model='smmuv3'>
<driver pciBus='2' accel='on' ats='on' ril='off' ssidsize='20' oas='44'/>
</iommu>
</devices>
This would get translated to a qemu command line with the arguments
below:
-device
'{"driver":"pxb-pcie","bus_nr":252,"id":"pci.1","bus":"pcie.0","addr":"0x1"}' \
-device
'{"driver":"pxb-pcie","bus_nr":248,"id":"pci.2","bus":"pcie.0","addr":"0x2"}' \
-device
'{"driver":"pcie-root-port","port":0,"chassis":21,"id":"pci.21","bus":"pci.1","addr":"0x0"}'
\
-device
'{"driver":"pcie-root-port","port":168,"chassis":22,"id":"pci.22","bus":"pci.2","addr":"0x0"}'
\
-device
'{"driver":"arm-smmuv3","primary-bus":"pci.1","id":"iommu0","accel":true,"ats":"on","ril":"off","ssidsize":"20","oas":"44"}'
\
-device
'{"driver":"arm-smmuv3","primary-bus":"pci.2","id":"iommu1","accel":true,"ats":"on","ril":"off","ssidsize":"20","oas":"44"}'
\
-device
'{"driver":"vfio-pci","host":"0009:01:00.0","id":"hostdev0","bus":"pci.21","addr":"0x0"}'
\
-device
'{"driver":"vfio-pci","host":"0019:01:00.0","id":"hostdev1","bus":"pci.22","addr":"0x0"}'
\
Two QEMU series implementing auto support for accel SMMUv3 properties
are now present in the upstream master branch [1][2]. This Libvirt
series leverages the auto support: users can omit SMMUv3 properties
ATS, RIL, OAS, and SSIDSIZE on the domain XML, and they will also be
omitted on the QEMU command line as a result. With these new QEMU series
applied, the feature property values will default to 'auto' to be
resolved based on what the host SMMU supports.
This series also adds documentation fixes for docs/formatdomain.rst and
implements support for QEMU capability checks on device-pluggable
accel smmuv3.
This series is on Github:
https://github.com/NathanChenNVIDIA/libvirt/tree/smmuv3-accel-v11/
Thanks,
Nathan
[0]
https://lists.libvirt.org/archives/list/[email protected]/thread/Z33Q6ALIPXLHCRH4NV5AQIRELJOVOZKR/
[1] https://lore.kernel.org/all/[email protected]/
[2] https://lore.kernel.org/all/[email protected]/
Changes from v10:
- Move QEMU capabilities patches before introducing accel support
- Move qemu_validate.c QEMU cap check for accel to the main accel
support patch
- Use QEMU_CAPS_ARM_SMMUV3_SMMU_PER_BUS instead of
QEMU_CAPS_ARM_SMMUV3
- Remove check for qemu reporting arm-smmuv3 support on non-ARM
architecture
- Revise qemuValidateDomainDeviceDefIOMMU conditional statement format
- Add QEMU_CAPS_DEVICE_SMMUV3 and gate QEMU_CAPS_ARM_SMMUV3_SMMU_PER_BUS
on QEMU_CAPS_DEVICE_SMMUV3
Changes from v9:
- Rename ssidSize to ssidsize
- Update docs/formatdomain.rst to mention `since 12.6.0` for accel
smmuv3 properties and these properties' hypervisor defaults being
used if not specified.
- Separate out virDomainIOMMUDefEquals() check for pci_bus into a
new fix commit
- Include fix commit for `since 11.10.0` for pciBus attribute in
docs/formatdomain.rst
- Use T:accel to propagate accel=off
- Use qemuOnOffAuto() for ats and ril
- Add QEMU capabilities checks for device-pluggable smmuv3 and accel
smmuv3
Changes from v8:
- Changed smmuv3 attribute QEMU CLI values to strings to match QEMU 11.0
Changes from v7:
- Remove support for x-vpasid-cap-offset based on feedback pointing out
the QEMU feature prefixed with x- is experimental and not to be
supported in Libvirt
- Implement support for auto SMMUv3 properties. Omitting these
properties in the XML omits them on the QEMU command line as well,
which results in default auto values being chosen for ATS, RIL,
SSIDSIZE, and OAS
- Add validation check that only allows enabling accel SMMUv3 features
when accel is enabled
- Add tests for accel=on but omitting accel SMMUv3 feature properties
Changes from v6:
- Remove smmuv3 'pasid' attribute
- Add smmuv3 'ssidsize' attribute
- Add support for 'vpasidCapOffset' element
- Modify logic for building accel smmuv3 qemu command line to omit ATS,
RIL, accel flags when not needed (enabled/disabled by default in QEMU
SMMUv3)
Nathan Chen (7):
conf: Include check for pci_bus in virDomainIOMMUDefEquals()
docs: formatdomain: Include Libvirt version for IOMMU pciBus attribute
qemu: introduce QEMU_CAPS_DEVICE_SMMUV3
qemu: introduce QEMU_CAPS_ARM_SMMUV3_SMMU_PER_BUS
qemu: introduce QEMU_CAPS_ARM_SMMUV3_ACCEL
qemu: Add support for HW-accelerated nested SMMUv3
tests: qemuxmlconfdata: provide HW-accel smmuv3 sample XML and CLI
args
docs/formatdomain.rst | 41 ++++-
src/conf/domain_conf.c | 80 ++++++++++
src/conf/domain_conf.h | 5 +
src/conf/domain_validate.c | 27 +++-
src/conf/schemas/domaincommon.rng | 25 +++
src/qemu/qemu_capabilities.c | 12 ++
src/qemu/qemu_capabilities.h | 3 +
src/qemu/qemu_command.c | 15 ++
src/qemu/qemu_validate.c | 20 +++
.../caps_10.0.0_aarch64.replies | 79 ++++++----
.../caps_10.0.0_aarch64.xml | 1 +
.../caps_10.2.0_aarch64.replies | 89 +++++++----
.../caps_10.2.0_aarch64.xml | 2 +
.../caps_11.0.0_aarch64.replies | 132 ++++++++++++----
.../caps_11.0.0_aarch64.xml | 3 +
.../caps_11.1.0_aarch64.replies | 143 ++++++++++++++----
.../caps_11.1.0_aarch64.xml | 3 +
.../caps_8.2.0_aarch64.replies | 79 ++++++----
.../caps_8.2.0_aarch64.xml | 1 +
.../caps_8.2.0_armv7l.replies | 79 ++++++----
.../caps_8.2.0_armv7l.xml | 1 +
.../caps_9.2.0_aarch64+hvf.replies | 75 ++++++---
.../caps_9.2.0_aarch64+hvf.xml | 1 +
...uv3-pci-bus-accel-auto.aarch64-latest.args | 38 +++++
...muv3-pci-bus-accel-auto.aarch64-latest.xml | 63 ++++++++
.../iommu-smmuv3-pci-bus-accel-auto.xml | 53 +++++++
...u-smmuv3-pci-bus-accel.aarch64-latest.args | 38 +++++
...mu-smmuv3-pci-bus-accel.aarch64-latest.xml | 63 ++++++++
.../iommu-smmuv3-pci-bus-accel.xml | 53 +++++++
tests/qemuxmlconftest.c | 2 +
30 files changed, 1038 insertions(+), 188 deletions(-)
create mode 100644
tests/qemuxmlconfdata/iommu-smmuv3-pci-bus-accel-auto.aarch64-latest.args
create mode 100644
tests/qemuxmlconfdata/iommu-smmuv3-pci-bus-accel-auto.aarch64-latest.xml
create mode 100644 tests/qemuxmlconfdata/iommu-smmuv3-pci-bus-accel-auto.xml
create mode 100644
tests/qemuxmlconfdata/iommu-smmuv3-pci-bus-accel.aarch64-latest.args
create mode 100644
tests/qemuxmlconfdata/iommu-smmuv3-pci-bus-accel.aarch64-latest.xml
create mode 100644 tests/qemuxmlconfdata/iommu-smmuv3-pci-bus-accel.xml
--
2.43.0
