RE: [PATCH v5 17/32] hw/arm/tegra241-cmdqv: mmap VINTF Page0 for CMDQV
> -Original Message- > From: Eric Auger > Sent: 26 May 2026 13:32 > To: Shameer Kolothum Thodi ; qemu- > [email protected]; [email protected] > Cc: [email protected]; [email protected]; [email protected]; Nicolin > Chen ; Nathan Chen ; Matt > Ochs ; Jiandi An ; Jason Gunthorpe > ; [email protected]; Krishnakant Jaju > ; [email protected] > Subject: Re: [PATCH v5 17/32] hw/arm/tegra241-cmdqv: mmap VINTF Page0 > for CMDQV > > External email: Use caution opening links or attachments > > > Hi Shameer, > > On 5/22/26 12:01 PM, Shameer Kolothum Thodi wrote: > >>> On 5/19/26 12:36 PM, Shameer Kolothum wrote: > >>> Actually is the whole host passthrough principle is not really explained > >>> anywhere. At least that's my feeling. It would be nice to have a summary > >>> for it, in the coverletter and in individual patch. Or maybe we can link > >>> to another doc. Reading the kernel uapi does not really provide the full > >>> picture, at least that's my own feeling. > >> Fair point. A summary of operation is useful. I am thinking of adding > >> it at the top of hw/arm/tegra241-cmdqv.c: > >> > >> /* > >>* Tegra241 CMDQV - overview > >>* --- > >>*... > >>*/ > >> > >> I will populate the details and share it for review before v6. > > Please find below. Hopefully, I have captured all the important details. > > Please take a look and let me know. > > > > Thanks, > > Shameer > > > > diff --git a/hw/arm/tegra241-cmdqv.c b/hw/arm/tegra241-cmdqv.c > > index ebf12d0597..5a103f37b8 100644 > > --- a/hw/arm/tegra241-cmdqv.c > > +++ b/hw/arm/tegra241-cmdqv.c > > @@ -7,6 +7,84 @@ > > * SPDX-License-Identifier: GPL-2.0-or-later > > */ > > > > +/* > > + * Tegra241 CMDQV - overview > > + * = > > + * > > + * NVIDIA Tegra241 extends SMMUv3 with a Command Queue > Virtualization (CMDQ-V) > > + * block. It lets a guest issue SMMU invalidation commands directly to > > + * dedicated hardware queues (vCMDQs) without trapping into the > hypervisor on > > + * the fast path. vCMDQs are grouped into Virtual Interfaces (VINTFs); the > > + * host kernel allocates one VINTF per emulated SMMUv3 instance via > iommufd. > > + * QEMU emulates the CMDQV MMIO region and drives the host kernel > calls > > + * (VIOMMU_ALLOC, HW_QUEUE_ALLOC, mmap); the actual command > processing happens > > + * on real hardware. > > + * > > + * MMIO layout (64KB pages, total TEGRA241_CMDQV_IO_LEN) > > + * - > > + * The direct vCMDQ apertures (0x1/0x2) are HW aliases of the > VINTF > > + * apertures (0x3/0x4); they expose the same per-vCMDQ > register slots > > + * under different addressing. > > + * > > + * 0x0 CMDQV Config page: QEMU-trapped. > > + * 0x1 Direct vCMDQ Page 0 (control/status): QEMU-trapped and > routed > > + *via vintf_ptr() to either the mmap'd VINTF page (allocated > > + *slot) or a per-vCMDQ register cache (unallocated slot). > > + * 0x2 Direct vCMDQ Page 1 (BASE / DRAM addresses): QEMU- > trapped. > > + * 0x3 VINTF Page 0 (per-VINTF control/status): mmap'd from the > host > > + *via iommufd and installed into guest MMIO as a RAM-device > > + *subregion after the first HW_QUEUE_ALLOC; subsequent accesses > > + *bypass QEMU. > > + * 0x4 VINTF Page 1 (per-VINTF BASE): QEMU-trapped. > > + * > > + * The direct vCMDQ aperture stays trapped (rather than aliased to the > VINTF > > direct vCMDQ aperture page 0 stays trapped as opposed to the VIINTF page0 Ok. > > > + * mmap) to preserve the spec's R/W register semantics for unallocated > > + * vCMDQs: the direct aperture allows programming before VINTF > allocation, > > + * while aliasing would route through the VINTF drop path instead. > see last discussion Replied there. > > + * > > + * Lifecycle (driven by guest events) > > + * -- > > + * 1. First vfio-pci device attach (.set_iommu_device) triggers: > > + *- tegra241_cmdqv_probe(): IOMMU_GET_HW_INFO confirms host > CMDQV support. > > + *- IOMMU_VIOMMU_ALLOC: the kernel allocates a VINTF for this VM, > > + * configures the VM's VMID (from its stage-2 HWPT) in VINTF_CONFIG, > > + * forces HYP_OWN=0, and r
Re: [PATCH v5 17/32] hw/arm/tegra241-cmdqv: mmap VINTF Page0 for CMDQV
Hi Shameer, On 5/22/26 12:01 PM, Shameer Kolothum Thodi wrote: >>> On 5/19/26 12:36 PM, Shameer Kolothum wrote: >>> Actually is the whole host passthrough principle is not really explained >>> anywhere. At least that's my feeling. It would be nice to have a summary >>> for it, in the coverletter and in individual patch. Or maybe we can link >>> to another doc. Reading the kernel uapi does not really provide the full >>> picture, at least that's my own feeling. >> Fair point. A summary of operation is useful. I am thinking of adding >> it at the top of hw/arm/tegra241-cmdqv.c: >> >> /* >>* Tegra241 CMDQV - overview >>* --- >>*... >>*/ >> >> I will populate the details and share it for review before v6. > Please find below. Hopefully, I have captured all the important details. > Please take a look and let me know. > > Thanks, > Shameer > > diff --git a/hw/arm/tegra241-cmdqv.c b/hw/arm/tegra241-cmdqv.c > index ebf12d0597..5a103f37b8 100644 > --- a/hw/arm/tegra241-cmdqv.c > +++ b/hw/arm/tegra241-cmdqv.c > @@ -7,6 +7,84 @@ > * SPDX-License-Identifier: GPL-2.0-or-later > */ > > +/* > + * Tegra241 CMDQV - overview > + * = > + * > + * NVIDIA Tegra241 extends SMMUv3 with a Command Queue Virtualization > (CMDQ-V) > + * block. It lets a guest issue SMMU invalidation commands directly to > + * dedicated hardware queues (vCMDQs) without trapping into the hypervisor on > + * the fast path. vCMDQs are grouped into Virtual Interfaces (VINTFs); the > + * host kernel allocates one VINTF per emulated SMMUv3 instance via iommufd. > + * QEMU emulates the CMDQV MMIO region and drives the host kernel calls > + * (VIOMMU_ALLOC, HW_QUEUE_ALLOC, mmap); the actual command processing > happens > + * on real hardware. > + * > + * MMIO layout (64KB pages, total TEGRA241_CMDQV_IO_LEN) > + * - > + * The direct vCMDQ apertures (0x1/0x2) are HW aliases of the VINTF > + * apertures (0x3/0x4); they expose the same per-vCMDQ register slots > + * under different addressing. > + * > + * 0x0 CMDQV Config page: QEMU-trapped. > + * 0x1 Direct vCMDQ Page 0 (control/status): QEMU-trapped and routed > + *via vintf_ptr() to either the mmap'd VINTF page (allocated > + *slot) or a per-vCMDQ register cache (unallocated slot). > + * 0x2 Direct vCMDQ Page 1 (BASE / DRAM addresses): QEMU-trapped. > + * 0x3 VINTF Page 0 (per-VINTF control/status): mmap'd from the host > + *via iommufd and installed into guest MMIO as a RAM-device > + *subregion after the first HW_QUEUE_ALLOC; subsequent accesses > + *bypass QEMU. > + * 0x4 VINTF Page 1 (per-VINTF BASE): QEMU-trapped. > + * > + * The direct vCMDQ aperture stays trapped (rather than aliased to the VINTF direct vCMDQ aperture page 0 stays trapped as opposed to the VIINTF page0 > + * mmap) to preserve the spec's R/W register semantics for unallocated > + * vCMDQs: the direct aperture allows programming before VINTF allocation, > + * while aliasing would route through the VINTF drop path instead. see last discussion > + * > + * Lifecycle (driven by guest events) > + * -- > + * 1. First vfio-pci device attach (.set_iommu_device) triggers: > + *- tegra241_cmdqv_probe(): IOMMU_GET_HW_INFO confirms host CMDQV > support. > + *- IOMMU_VIOMMU_ALLOC: the kernel allocates a VINTF for this VM, > + * configures the VM's VMID (from its stage-2 HWPT) in VINTF_CONFIG, > + * forces HYP_OWN=0, and returns the mmap offset/length for VINTF Page > 0. what about the v/p SID mapping. How does the kernel know which SIDs are supposed to write into that VINTF? where do we pass this info? > + * > + * 2. Guest writes VINTF_CONFIG.ENABLE = 1: > + *QEMU mmap()s the offset from step 1 into its address space and reports > + *STATUS.ENABLE_OK = 1. The host VINTF was already enabled by > + *IOMMU_VIOMMU_ALLOC; QEMU only acks back. > + * > + * 3. Guest completes vCMDQ setup (BASE, CMDQ_ALLOC_MAP.ALLOC, CMDQV_EN, > + *VINTF.ENABLE, in any order; each precondition write retries the > + *allocation): > + *IOMMU_HW_QUEUE_ALLOC binds the guest BASE GPA (translated through > + *stage-2 and pinned by the kernel) to a host vCMDQ in this VM's VINTF. > + * > + * 4. After the first successful HW_QUEUE_ALLOC, the mmap'd VINTF Page 0 is > + *installed into guest MMIO as a RAM-device subregion. Guest vCMDQ Page 0 > + *accesses (CMDQ_EN, PROD/CONS_INDX, STATUS, GERROR/GERRORN) thereafter > + *go straight to host hardware, bypassing QEMU. > + * > + * Per-VM isolation > + * > + * - Each VM has its own iommufd FD; all iommufd objects (VINTF, hw_queues, > + * mmap regions) belong to that FD. Cross-FD lookups fail, so one VM > + * cannot reach another VM's IDs. > + * - IOMMU_VIOMMU_ALLOC configures the VM's VMI
RE: [PATCH v5 17/32] hw/arm/tegra241-cmdqv: mmap VINTF Page0 for CMDQV
> > On 5/19/26 12:36 PM, Shameer Kolothum wrote: > > Actually is the whole host passthrough principle is not really explained > > anywhere. At least that's my feeling. It would be nice to have a summary > > for it, in the coverletter and in individual patch. Or maybe we can link > > to another doc. Reading the kernel uapi does not really provide the full > > picture, at least that's my own feeling. > > Fair point. A summary of operation is useful. I am thinking of adding > it at the top of hw/arm/tegra241-cmdqv.c: > > /* >* Tegra241 CMDQV - overview >* --- >*... >*/ > > I will populate the details and share it for review before v6. Please find below. Hopefully, I have captured all the important details. Please take a look and let me know. Thanks, Shameer diff --git a/hw/arm/tegra241-cmdqv.c b/hw/arm/tegra241-cmdqv.c index ebf12d0597..5a103f37b8 100644 --- a/hw/arm/tegra241-cmdqv.c +++ b/hw/arm/tegra241-cmdqv.c @@ -7,6 +7,84 @@ * SPDX-License-Identifier: GPL-2.0-or-later */ +/* + * Tegra241 CMDQV - overview + * = + * + * NVIDIA Tegra241 extends SMMUv3 with a Command Queue Virtualization (CMDQ-V) + * block. It lets a guest issue SMMU invalidation commands directly to + * dedicated hardware queues (vCMDQs) without trapping into the hypervisor on + * the fast path. vCMDQs are grouped into Virtual Interfaces (VINTFs); the + * host kernel allocates one VINTF per emulated SMMUv3 instance via iommufd. + * QEMU emulates the CMDQV MMIO region and drives the host kernel calls + * (VIOMMU_ALLOC, HW_QUEUE_ALLOC, mmap); the actual command processing happens + * on real hardware. + * + * MMIO layout (64KB pages, total TEGRA241_CMDQV_IO_LEN) + * - + * The direct vCMDQ apertures (0x1/0x2) are HW aliases of the VINTF + * apertures (0x3/0x4); they expose the same per-vCMDQ register slots + * under different addressing. + * + * 0x0 CMDQV Config page: QEMU-trapped. + * 0x1 Direct vCMDQ Page 0 (control/status): QEMU-trapped and routed + *via vintf_ptr() to either the mmap'd VINTF page (allocated + *slot) or a per-vCMDQ register cache (unallocated slot). + * 0x2 Direct vCMDQ Page 1 (BASE / DRAM addresses): QEMU-trapped. + * 0x3 VINTF Page 0 (per-VINTF control/status): mmap'd from the host + *via iommufd and installed into guest MMIO as a RAM-device + *subregion after the first HW_QUEUE_ALLOC; subsequent accesses + *bypass QEMU. + * 0x4 VINTF Page 1 (per-VINTF BASE): QEMU-trapped. + * + * The direct vCMDQ aperture stays trapped (rather than aliased to the VINTF + * mmap) to preserve the spec's R/W register semantics for unallocated + * vCMDQs: the direct aperture allows programming before VINTF allocation, + * while aliasing would route through the VINTF drop path instead. + * + * Lifecycle (driven by guest events) + * -- + * 1. First vfio-pci device attach (.set_iommu_device) triggers: + *- tegra241_cmdqv_probe(): IOMMU_GET_HW_INFO confirms host CMDQV support. + *- IOMMU_VIOMMU_ALLOC: the kernel allocates a VINTF for this VM, + * configures the VM's VMID (from its stage-2 HWPT) in VINTF_CONFIG, + * forces HYP_OWN=0, and returns the mmap offset/length for VINTF Page 0. + * + * 2. Guest writes VINTF_CONFIG.ENABLE = 1: + *QEMU mmap()s the offset from step 1 into its address space and reports + *STATUS.ENABLE_OK = 1. The host VINTF was already enabled by + *IOMMU_VIOMMU_ALLOC; QEMU only acks back. + * + * 3. Guest completes vCMDQ setup (BASE, CMDQ_ALLOC_MAP.ALLOC, CMDQV_EN, + *VINTF.ENABLE, in any order; each precondition write retries the + *allocation): + *IOMMU_HW_QUEUE_ALLOC binds the guest BASE GPA (translated through + *stage-2 and pinned by the kernel) to a host vCMDQ in this VM's VINTF. + * + * 4. After the first successful HW_QUEUE_ALLOC, the mmap'd VINTF Page 0 is + *installed into guest MMIO as a RAM-device subregion. Guest vCMDQ Page 0 + *accesses (CMDQ_EN, PROD/CONS_INDX, STATUS, GERROR/GERRORN) thereafter + *go straight to host hardware, bypassing QEMU. + * + * Per-VM isolation + * + * - Each VM has its own iommufd FD; all iommufd objects (VINTF, hw_queues, + * mmap regions) belong to that FD. Cross-FD lookups fail, so one VM + * cannot reach another VM's IDs. + * - IOMMU_VIOMMU_ALLOC configures the VM's VMID in VINTF_CONFIG; the CMDQV + * hardware substitutes / checks VMID on every command the guest issues. + * - The kernel allocates the VINTF with HYP_OWN = 0, which restricts the + * guest to a safe subset of commands. + * - Per VINTF, the kernel programs SID_MATCH and SID_REPLACE to restrict + * invalidations to the StreamIDs assigned to this VM. + * - IOMMU_HW_QUEUE_ALLOC binds each vCMDQ to a single VINTF, so a guest + * cannot reach a vCMDQ that belon
Re: [PATCH v5 17/32] hw/arm/tegra241-cmdqv: mmap VINTF Page0 for CMDQV
On 5/21/26 11:44 AM, Shameer Kolothum Thodi wrote: > >> -Original Message- >> From: Eric Auger >> Sent: 21 May 2026 09:00 >> To: Shameer Kolothum Thodi ; qemu- >> [email protected]; [email protected] >> Cc: [email protected]; [email protected]; [email protected]; Nicolin >> Chen ; Nathan Chen ; Matt >> Ochs ; Jiandi An ; Jason Gunthorpe >> ; [email protected]; Krishnakant Jaju >> ; [email protected] >> Subject: Re: [PATCH v5 17/32] hw/arm/tegra241-cmdqv: mmap VINTF Page0 >> for CMDQV >> >> External email: Use caution opening links or attachments >> >> >> Hi Shameer, >> >> On 5/20/26 5:47 PM, Shameer Kolothum Thodi wrote: >>> Hi Eric, >>> >>>> -Original Message- >>>> From: Eric Auger >>>> Sent: 20 May 2026 13:54 >>>> To: Shameer Kolothum Thodi ; qemu- >>>> [email protected]; [email protected] >>>> Cc: [email protected]; [email protected]; [email protected]; >>>> Nicolin Chen ; Nathan Chen ; >>>> Matt Ochs ; Jiandi An ; Jason >>>> Gunthorpe ; [email protected]; Krishnakant >>>> Jaju ; [email protected] >>>> Subject: Re: [PATCH v5 17/32] hw/arm/tegra241-cmdqv: mmap VINTF >> Page0 >>>> for CMDQV >>>> >>>> External email: Use caution opening links or attachments >>>> >>>> >>>> Hi Shameer, >>>> >>>> On 5/19/26 12:36 PM, Shameer Kolothum wrote: >>>>> From: Nicolin Chen >>>>> >>>>> The CMDQ-V CMDQ pages provide a VM wide view of all VCMDQs, while >>>>> the >>>> s/CMDQ pages/VCMDQ pages - yeah that's all very confusing ;-) - >>>>> VINTF pages expose a logical view local to a given VINTF. Although >>>>> real hardware may support multiple VINTFs, the kernel currently >>>>> exposes a single VINTF per VM. >>>>> >>>>> The kernel provides an mmap offset for the VINTF Page0 region during >>>> for the host VINTF page0 allocated for the VM >>>> >>>> Actually is the whole host passthrough principle is not really >>>> explained anywhere. At least that's my feeling. It would be nice to >>>> have a summary for it, in the coverletter and in individual patch. Or >>>> maybe we can link to another doc. Reading the kernel uapi does not >>>> really provide the full picture, at least that's my own feeling. >>> Fair point. A summary of operation is useful. I am thinking of adding >>> it at the top of hw/arm/tegra241-cmdqv.c: >>> >>> /* >>>* Tegra241 CMDQV - overview >>>* --- >>>*... >>>*/ >>> >>> I will populate the details and share it for review before v6. >>> >>>> when a VM allocates a VIOMMU for the CMDQV, the kernel allocates this >>>> and this resource that is then given for exclusive usage of the VM ... >>>> How does it guarantee security between VMs, etc... >>> Isolation is enforced by the kernel and the CMDQV hardware. >>> I think the key points here are: >>> >>> - Each VM has its own iommufd FD. All iommufd objects (VINTF, hw >>> queues, mmap regions) belong to that FD; one VM cannot reach >>> another VM's IDs. >>> >>> - Each FD gets its own VINTF, allocated by IOMMU_VIOMMU_ALLOC. The >>> kernel writes the VM's VMID into the VINTF and the host CMDQV >> s/writes/configures >>> hardware checks it on every command. >>> >>> - The kernel allocates the VINTF with HYP_OWN = 0, which restricts >>> the guest to a safe subset of commands. >>> >>> - Per VINTF, the kernel programs SID_MATCH and SID_REPLACE so only >>> the VM's own vfio-pci devices can issue commands; other SIDs are >>> rejected by the hardware. >>> >>> - IOMMU_HW_QUEUE_ALLOC binds each VCMDQ to one specific VINTF, so >> a >>> guest cannot reach a VCMDQ that belongs to another VM. >> at the moment a single VCMDQ is used by VM, is that a correct >> understanding? > Not quite. QEMU exposes one VINTF (VINTF0) with two VCMDQs to the > guest. The guest decides how many of them to enable. A Linux guest > enables both: > ... > arm-smmu-v3 arm-smmu-v3.0.auto: allocated 524288 entries for vcmdq0 > arm-smmu-v3 arm-smmu-v3.0.auto: allocated 524288 entries for vcmdq1 understood! Eric > ... > > Thanks, > Shameer
RE: [PATCH v5 17/32] hw/arm/tegra241-cmdqv: mmap VINTF Page0 for CMDQV
> -Original Message- > From: Eric Auger > Sent: 21 May 2026 09:00 > To: Shameer Kolothum Thodi ; qemu- > [email protected]; [email protected] > Cc: [email protected]; [email protected]; [email protected]; Nicolin > Chen ; Nathan Chen ; Matt > Ochs ; Jiandi An ; Jason Gunthorpe > ; [email protected]; Krishnakant Jaju > ; [email protected] > Subject: Re: [PATCH v5 17/32] hw/arm/tegra241-cmdqv: mmap VINTF Page0 > for CMDQV > > External email: Use caution opening links or attachments > > > Hi Shameer, > > On 5/20/26 5:47 PM, Shameer Kolothum Thodi wrote: > > Hi Eric, > > > >> -Original Message- > >> From: Eric Auger > >> Sent: 20 May 2026 13:54 > >> To: Shameer Kolothum Thodi ; qemu- > >> [email protected]; [email protected] > >> Cc: [email protected]; [email protected]; [email protected]; > >> Nicolin Chen ; Nathan Chen ; > >> Matt Ochs ; Jiandi An ; Jason > >> Gunthorpe ; [email protected]; Krishnakant > >> Jaju ; [email protected] > >> Subject: Re: [PATCH v5 17/32] hw/arm/tegra241-cmdqv: mmap VINTF > Page0 > >> for CMDQV > >> > >> External email: Use caution opening links or attachments > >> > >> > >> Hi Shameer, > >> > >> On 5/19/26 12:36 PM, Shameer Kolothum wrote: > >>> From: Nicolin Chen > >>> > >>> The CMDQ-V CMDQ pages provide a VM wide view of all VCMDQs, while > >>> the > >> s/CMDQ pages/VCMDQ pages - yeah that's all very confusing ;-) - > >>> VINTF pages expose a logical view local to a given VINTF. Although > >>> real hardware may support multiple VINTFs, the kernel currently > >>> exposes a single VINTF per VM. > >>> > >>> The kernel provides an mmap offset for the VINTF Page0 region during > >> for the host VINTF page0 allocated for the VM > >> > >> Actually is the whole host passthrough principle is not really > >> explained anywhere. At least that's my feeling. It would be nice to > >> have a summary for it, in the coverletter and in individual patch. Or > >> maybe we can link to another doc. Reading the kernel uapi does not > >> really provide the full picture, at least that's my own feeling. > > Fair point. A summary of operation is useful. I am thinking of adding > > it at the top of hw/arm/tegra241-cmdqv.c: > > > > /* > >* Tegra241 CMDQV - overview > >* --- > >*... > >*/ > > > > I will populate the details and share it for review before v6. > > > >> when a VM allocates a VIOMMU for the CMDQV, the kernel allocates this > >> and this resource that is then given for exclusive usage of the VM ... > >> How does it guarantee security between VMs, etc... > > Isolation is enforced by the kernel and the CMDQV hardware. > > I think the key points here are: > > > > - Each VM has its own iommufd FD. All iommufd objects (VINTF, hw > > queues, mmap regions) belong to that FD; one VM cannot reach > > another VM's IDs. > > > > - Each FD gets its own VINTF, allocated by IOMMU_VIOMMU_ALLOC. The > > kernel writes the VM's VMID into the VINTF and the host CMDQV > s/writes/configures > > hardware checks it on every command. > > > > - The kernel allocates the VINTF with HYP_OWN = 0, which restricts > > the guest to a safe subset of commands. > > > > - Per VINTF, the kernel programs SID_MATCH and SID_REPLACE so only > > the VM's own vfio-pci devices can issue commands; other SIDs are > > rejected by the hardware. > > > > - IOMMU_HW_QUEUE_ALLOC binds each VCMDQ to one specific VINTF, so > a > > guest cannot reach a VCMDQ that belongs to another VM. > at the moment a single VCMDQ is used by VM, is that a correct > understanding? Not quite. QEMU exposes one VINTF (VINTF0) with two VCMDQs to the guest. The guest decides how many of them to enable. A Linux guest enables both: ... arm-smmu-v3 arm-smmu-v3.0.auto: allocated 524288 entries for vcmdq0 arm-smmu-v3 arm-smmu-v3.0.auto: allocated 524288 entries for vcmdq1 ... Thanks, Shameer
Re: [PATCH v5 17/32] hw/arm/tegra241-cmdqv: mmap VINTF Page0 for CMDQV
Hi Shameer, On 5/20/26 5:47 PM, Shameer Kolothum Thodi wrote: > Hi Eric, > >> -Original Message- >> From: Eric Auger >> Sent: 20 May 2026 13:54 >> To: Shameer Kolothum Thodi ; qemu- >> [email protected]; [email protected] >> Cc: [email protected]; [email protected]; [email protected]; Nicolin >> Chen ; Nathan Chen ; Matt >> Ochs ; Jiandi An ; Jason Gunthorpe >> ; [email protected]; Krishnakant Jaju >> ; [email protected] >> Subject: Re: [PATCH v5 17/32] hw/arm/tegra241-cmdqv: mmap VINTF Page0 >> for CMDQV >> >> External email: Use caution opening links or attachments >> >> >> Hi Shameer, >> >> On 5/19/26 12:36 PM, Shameer Kolothum wrote: >>> From: Nicolin Chen >>> >>> The CMDQ-V CMDQ pages provide a VM wide view of all VCMDQs, while the >> s/CMDQ pages/VCMDQ pages - yeah that's all very confusing ;-) - >>> VINTF pages expose a logical view local to a given VINTF. Although real >>> hardware may support multiple VINTFs, the kernel currently exposes a >>> single VINTF per VM. >>> >>> The kernel provides an mmap offset for the VINTF Page0 region during >> for the host VINTF page0 allocated for the VM >> >> Actually is the whole host passthrough principle is not really explained >> anywhere. At least that's my feeling. It would be nice to have a summary >> for it, in the coverletter and in individual patch. Or maybe we can link >> to another doc. Reading the kernel uapi does not really provide the full >> picture, at least that's my own feeling. > Fair point. A summary of operation is useful. I am thinking of adding > it at the top of hw/arm/tegra241-cmdqv.c: > > /* >* Tegra241 CMDQV - overview >* --- >*... >*/ > > I will populate the details and share it for review before v6. > >> when a VM allocates a VIOMMU for the CMDQV, the kernel allocates this >> and this resource that is then given for exclusive usage of the VM ... >> How does it guarantee security between VMs, etc... > Isolation is enforced by the kernel and the CMDQV hardware. > I think the key points here are: > > - Each VM has its own iommufd FD. All iommufd objects (VINTF, hw > queues, mmap regions) belong to that FD; one VM cannot reach > another VM's IDs. > > - Each FD gets its own VINTF, allocated by IOMMU_VIOMMU_ALLOC. The > kernel writes the VM's VMID into the VINTF and the host CMDQV s/writes/configures > hardware checks it on every command. > > - The kernel allocates the VINTF with HYP_OWN = 0, which restricts > the guest to a safe subset of commands. > > - Per VINTF, the kernel programs SID_MATCH and SID_REPLACE so only > the VM's own vfio-pci devices can issue commands; other SIDs are > rejected by the hardware. > > - IOMMU_HW_QUEUE_ALLOC binds each VCMDQ to one specific VINTF, so a > guest cannot reach a VCMDQ that belongs to another VM. at the moment a single VCMDQ is used by VM, is that a correct understanding? Yes this general principle explanation sound very useful to me to help the reader understand the passthrough principle. > >> Furthermore, what is not straightforward in this description is the >> sequence that will be executed >> - viommu_alloc done on which trap event? >> - hw_queue_alloc on which trap event? >> - mmap on guest VINT enable >>> vIOMMU allocation. However, the logical-to-physical association between >>> VCMDQs and a VINTF is only established after HW_QUEUE allocation. Prior >>> to that, the mapped Page0 does not back any real VCMDQ state. > Ok. Will capture the sequence of operations and the surrounding behaviour > in the same top of file comment in hw/arm/tegra241-cmdqv.c. > > Thanks, > Shameer > Thanks Eric
RE: [PATCH v5 17/32] hw/arm/tegra241-cmdqv: mmap VINTF Page0 for CMDQV
Hi Eric, > -Original Message- > From: Eric Auger > Sent: 20 May 2026 13:54 > To: Shameer Kolothum Thodi ; qemu- > [email protected]; [email protected] > Cc: [email protected]; [email protected]; [email protected]; Nicolin > Chen ; Nathan Chen ; Matt > Ochs ; Jiandi An ; Jason Gunthorpe > ; [email protected]; Krishnakant Jaju > ; [email protected] > Subject: Re: [PATCH v5 17/32] hw/arm/tegra241-cmdqv: mmap VINTF Page0 > for CMDQV > > External email: Use caution opening links or attachments > > > Hi Shameer, > > On 5/19/26 12:36 PM, Shameer Kolothum wrote: > > From: Nicolin Chen > > > > The CMDQ-V CMDQ pages provide a VM wide view of all VCMDQs, while the > s/CMDQ pages/VCMDQ pages - yeah that's all very confusing ;-) - > > VINTF pages expose a logical view local to a given VINTF. Although real > > hardware may support multiple VINTFs, the kernel currently exposes a > > single VINTF per VM. > > > > The kernel provides an mmap offset for the VINTF Page0 region during > for the host VINTF page0 allocated for the VM > > Actually is the whole host passthrough principle is not really explained > anywhere. At least that's my feeling. It would be nice to have a summary > for it, in the coverletter and in individual patch. Or maybe we can link > to another doc. Reading the kernel uapi does not really provide the full > picture, at least that's my own feeling. Fair point. A summary of operation is useful. I am thinking of adding it at the top of hw/arm/tegra241-cmdqv.c: /* * Tegra241 CMDQV - overview * --- *... */ I will populate the details and share it for review before v6. > > when a VM allocates a VIOMMU for the CMDQV, the kernel allocates this > and this resource that is then given for exclusive usage of the VM ... > How does it guarantee security between VMs, etc... Isolation is enforced by the kernel and the CMDQV hardware. I think the key points here are: - Each VM has its own iommufd FD. All iommufd objects (VINTF, hw queues, mmap regions) belong to that FD; one VM cannot reach another VM's IDs. - Each FD gets its own VINTF, allocated by IOMMU_VIOMMU_ALLOC. The kernel writes the VM's VMID into the VINTF and the host CMDQV hardware checks it on every command. - The kernel allocates the VINTF with HYP_OWN = 0, which restricts the guest to a safe subset of commands. - Per VINTF, the kernel programs SID_MATCH and SID_REPLACE so only the VM's own vfio-pci devices can issue commands; other SIDs are rejected by the hardware. - IOMMU_HW_QUEUE_ALLOC binds each VCMDQ to one specific VINTF, so a guest cannot reach a VCMDQ that belongs to another VM. > Furthermore, what is not straightforward in this description is the > sequence that will be executed > - viommu_alloc done on which trap event? > - hw_queue_alloc on which trap event? > - mmap on guest VINT enable > > vIOMMU allocation. However, the logical-to-physical association between > > VCMDQs and a VINTF is only established after HW_QUEUE allocation. Prior > > to that, the mapped Page0 does not back any real VCMDQ state. Ok. Will capture the sequence of operations and the surrounding behaviour in the same top of file comment in hw/arm/tegra241-cmdqv.c. Thanks, Shameer
Re: [PATCH v5 17/32] hw/arm/tegra241-cmdqv: mmap VINTF Page0 for CMDQV
Hi Shameer,
On 5/19/26 12:36 PM, Shameer Kolothum wrote:
> From: Nicolin Chen
>
> The CMDQ-V CMDQ pages provide a VM wide view of all VCMDQs, while the
s/CMDQ pages/VCMDQ pages - yeah that's all very confusing ;-) -
> VINTF pages expose a logical view local to a given VINTF. Although real
> hardware may support multiple VINTFs, the kernel currently exposes a
> single VINTF per VM.
>
> The kernel provides an mmap offset for the VINTF Page0 region during
for the host VINTF page0 allocated for the VM
Actually is the whole host passthrough principle is not really explained
anywhere. At least that's my feeling. It would be nice to have a summary
for it, in the coverletter and in individual patch. Or maybe we can link
to another doc. Reading the kernel uapi does not really provide the full
picture, at least that's my own feeling.
when a VM allocates a VIOMMU for the CMDQV, the kernel allocates this
and this resource that is then given for exclusive usage of the VM ...
How does it guarantee security between VMs, etc...
Furthermore, what is not straightforward in this description is the
sequence that will be executed
- viommu_alloc done on which trap event?
- hw_queue_alloc on which trap event?
- mmap on guest VINT enable
> vIOMMU allocation. However, the logical-to-physical association between
> VCMDQs and a VINTF is only established after HW_QUEUE allocation. Prior
> to that, the mapped Page0 does not back any real VCMDQ state.
>
> When VINTF is enabled, mmap the kernel provided Page0 region and set
> ENABLE_OK only if the mmap succeeds. Unmap it when VINTF is disabled.
> This prepares the VINTF mapping in advance of subsequent patches that
> add VCMDQ allocation support.
>
> Signed-off-by: Nicolin Chen
> Co-developed-by: Shameer Kolothum
> Signed-off-by: Shameer Kolothum
Thanks
Eric
> ---
> hw/arm/tegra241-cmdqv.h | 3 +++
> hw/arm/tegra241-cmdqv.c | 50 ++---
> 2 files changed, 50 insertions(+), 3 deletions(-)
>
> diff --git a/hw/arm/tegra241-cmdqv.h b/hw/arm/tegra241-cmdqv.h
> index 6b21a407af..f0b031b4dc 100644
> --- a/hw/arm/tegra241-cmdqv.h
> +++ b/hw/arm/tegra241-cmdqv.h
> @@ -37,12 +37,15 @@
> #define CMDQV_VINTF_PAGE1_BASE 0x4
> #define CMDQV_VCMDQ_STRIDE 0x80
>
> +#define VINTF_PAGE_SIZE 0x1
> +
> typedef struct Tegra241CMDQV {
> struct iommu_viommu_tegra241_cmdqv cmdqv_data;
> SMMUv3AccelState *s_accel;
> MemoryRegion mmio_cmdqv;
> qemu_irq irq;
> IOMMUFDVeventq *veventq;
> +void *vintf_page0;
>
> /* CMDQ-V Config page register cache */
> uint32_t config;
> diff --git a/hw/arm/tegra241-cmdqv.c b/hw/arm/tegra241-cmdqv.c
> index 5d2d9d2e26..4d9f094b2a 100644
> --- a/hw/arm/tegra241-cmdqv.c
> +++ b/hw/arm/tegra241-cmdqv.c
> @@ -9,6 +9,8 @@
>
> #include "qemu/osdep.h"
> #include "qemu/log.h"
> +#include "qemu/error-report.h"
> +#include "qapi/error.h"
>
> #include "hw/arm/smmuv3.h"
> #include "hw/arm/smmuv3-common.h"
> @@ -228,8 +230,42 @@ static void
> tegra241_cmdqv_write_vcmdq_page1_64(Tegra241CMDQV *cmdqv,
> trace_tegra241_cmdqv_write_vcmdq_page1(index, offset0, value);
> }
>
> +static bool
> +tegra241_cmdqv_munmap_vintf_page0(Tegra241CMDQV *cmdqv, Error **errp)
> +{
> +if (!cmdqv->vintf_page0) {
> +return true;
> +}
> +
> +if (munmap(cmdqv->vintf_page0, VINTF_PAGE_SIZE) < 0) {
> +error_setg_errno(errp, errno, "Failed to unmap VINTF page0");
> +return false;
> +}
> +cmdqv->vintf_page0 = NULL;
> +return true;
> +}
> +
> +static bool tegra241_cmdqv_mmap_vintf_page0(Tegra241CMDQV *cmdqv, Error
> **errp)
> +{
> +IOMMUFDViommu *viommu = cmdqv->s_accel->viommu;
> +
> +if (cmdqv->vintf_page0) {
> +return true;
> +}
> +
> +if (!iommufd_backend_viommu_mmap(viommu->iommufd, viommu->viommu_id,
> + VINTF_PAGE_SIZE,
> + cmdqv->cmdqv_data.out_vintf_mmap_offset,
> + &cmdqv->vintf_page0, errp)) {
> +return false;
> +}
> +
> +return true;
> +}
> +
> static void tegra241_cmdqv_config_vintf_write(Tegra241CMDQV *cmdqv,
> - hwaddr offset, uint64_t value)
> + hwaddr offset, uint64_t value,
> + Error **errp)
> {
> int i;
>
> @@ -244,8 +280,11 @@ static void
> tegra241_cmdqv_config_vintf_write(Tegra241CMDQV *cmdqv,
>
> cmdqv->vintf_config = value;
> if (value & R_VINTF0_CONFIG_ENABLE_MASK) {
> -cmdqv->vintf_status |= R_VINTF0_STATUS_ENABLE_OK_MASK;
> +if (tegra241_cmdqv_mmap_vintf_page0(cmdqv, errp)) {
> +cmdqv->vintf_status |= R_VINTF0_STATUS_ENABLE_OK_MASK;
> +}
> } else {
> +tegra241_cmdqv_munmap_vintf_page0(cmdqv, errp);
> cmdqv->vintf_status &= ~R_VINTF0_ST
