Hello,
On 7/23/26 05:50, Wencheng Yang wrote:
__
On 2026/7/22 13:00, Alex Williamson wrote:
On Tue, 21 Jul 2026 11:03:39 -0400
"Michael S. Tsirkin"<[email protected]> <mailto:[email protected]> wrote:
On Tue, Jul 21, 2026 at 08:41:04AM -0600, Alex Williamson wrote:
On Tue, Jul 21, 2026, at 7:06 AM, Yang Wencheng wrote:
From: YangWencheng<[email protected]>
<mailto:[email protected]>
A passthrough device's own MMIO/BAR range is registered with the IOMMU
(VFIO_IOMMU_MAP_DMA) when its memory region becomes part of the guest
address space, to support peer-to-peer DMA into that BAR from other
devices. Toggling the guest's PCI_COMMAND memory-decode-enable bit off
and back on -- something PCI enumeration/attribute code does routinely,
sometimes several times for the same device (e.g. disable before
reprogramming a BAR then re-enable, or generic driver probing during
guest OS boot) -- currently tears down and rebuilds this mapping every
single time via vfio_listener_region_del()/region_add(), unconditionally.
For devices with very large BARs this is expensive: mapping a 64GB BAR
into IOMMU page tables measured at ~8.5s per call on this hardware, and
was observed being repeated 3-5 times for the same BAR during a single
guest boot, because nothing changed about the underlying mapping between
the disable and the following re-enable.
Defer the actual VFIO_IOMMU_UNMAP_DMA for "ram device" regions (i.e.
passthrough device BARs used for P2P DMA -- explicitly not regular guest
RAM or RamDiscardManager-backed regions, so migration/ballooning/
virtio-mem are unaffected) instead of issuing it immediately in
region_del(). If a matching region_add() for the exact same region
(same MemoryRegion pointer, iova, size, vaddr, readonly) follows, cancel
the deferred unmap and skip the VFIO_IOMMU_MAP_DMA entirely -- the
host-side mapping was never actually removed, so there is nothing to
redo. A region_add() for anything that doesn't match maps normally, as
before.
Nothing is leaked: any mapping still on the pending list is flushed with
a real unmap in two places -- vfio_container_instance_finalize(), before
the container's fd is closed (VM shutdown / last device in a group removed),
and vfio_bars_finalize(), matched by MemoryRegion pointer to just that
device's own BARs, so a device hot-unplugged while the container stays
alive for other devices can't leave a dangling deferred entry either.
This does not weaken the PCI_COMMAND memory-decode security boundary:
every guest write to PCI_COMMAND is still forwarded synchronously and
unconditionally to the real device's config space in
vfio_pci_write_config() (an entirely separate code path, untouched by
this change), which is what actually gates whether the physical device
claims/responds to transactions targeting its BAR, independent of
whatever the IOMMU's routing table still contains. This change only
avoids redundant bookkeeping of that routing table when nothing about
the mapping has changed.
Seems like you should focus on huge pfnmap support on your platform
rather than hack the VMM to not behave like bare metal. Thanks,
But bare metal does not tweak an iommu during pci enumeration?
Bare metal is not using the IOMMU to map the device into a guest
physical address space, so no, an equivalent operation does not occur
there. The proposal here is intentionally leaving stale gpa mappings
to avoid the unmap/remap overhead, where we've essentially eliminated
that overhead on other platforms with huge pfnmap support. The
specific platform that's being optimized here is also omitted, which
makes it impossible to determine the ongoing need for this code.
The scheme here effectively assumes that a region_del() will either be
followed by a region_add() that cancels the deferred unmap, or the
deferral will be executed on finalize. It does not actually account
for cases where the guest might remap the BAR to a different address,
ex. pci=realloc, leaving a stale mapping in place. The finalize also
appears to be using a different MR from the one deferred, such that the
IOMMU is actually only unmapped due to close().
More importantly though, this entire scheme is relying on a gap in the
vfio type1 IOMMU backend, where clearing the memory enable bit of the
command register zaps the CPU mappings and errors faults while cleared,
but does not generate unmaps in the IOMMU. IOMMUFD resolves that gap,
generating a move-notify/invalidate on the dma-buf, which results in an
IOMMU unmap. IOW, the premise of this workaround is broken in the
direction that VFIO is headed.
Therefore, I'd once again suggest that effort is better redirected to
implement huge pfnmap support for whatever platform this might be in
order to follow a proven solution to this problem for type1. Huge
pfnmap is used both in the CPU fault path as well as the DMA mapping
path, by virtue of generating user faults in the pfnmap. Alternatively,
iommufd may provide better p2p DMA mapping performance than type1
without huge pfnmap, but CPU faults would lag without such support
anyway. Thanks,
Alex
I noticed you provided a kernel patch "PCI: Batch BAR sizing operations",
the patch eliminates the overhead of IOMMU map/unmap operations from
guest kernel. I found OVMF has the same issue, I tried to fix it in edk2 source
code, but the PCI commands are scattered, it's hard to handle it like kernel.
I think it's reasonable to handle it qemu, as the overhead is introduced by
qemu to support PCIe device passthrough.
"we've essentially eliminated that overhead on other platforms with huge
pfnmap support."
Can you provide some key references about this approach?
Check these related commits.
Linux :
commit f9e54c3a2f5b ("vfio/pci: implement huge_fault support")
QEMU:
commit 00b519c0bca0 ("vfio/helpers: Align mmaps")
Thanks,
C.
