Hi Alex,
On 14/10/2026 21:38, Matt Evans wrote:
Hi Alexes,
On 14/10/2026 00:13, Alex Williamson wrote:
On Tue, 6 Oct 2026 20:36:36 +0100
Matt Evans <[email protected]> wrote:
A VFIO DMABUF can export a subset of a BAR to userspace by fd; add
support for mmap() of this fd. This provides another route for a
process to map BARs, in which the process can only map a specific
subset of a BAR represented by the exported DMABUF.
mmap() support enables userspace drivers to safely delegate BAR
sub-ranges to other processes by sharing a DMABUF fd, without having
to share the (omnipotent) VFIO device fd with them.
Since the main VFIO BAR mmap() is now DMABUF-aware, the new mmap()
reuses
the existing vm_ops.
The lifecycle of an exported DMABUF remains decoupled from that of the
device fd it came from, i.e. the device fd could be closed with DMABUF
VMAs present, meaning a fault on a VMA could happen concurrently with
vfio_pci_dma_buf_cleanup().
To deal with this scenario, the fault handler now temporarily takes a
VFIO device registration to ensure the vdev remains valid, and then
vdev locks can be taken.
This sounds like potentially measurable overhead if something is
slamming the fault handler where huge-fault is not supported or active.
Have you considered a fast path for the internal dmabuf where the
lifecycle is constrained?
[snip]
Hmm. I'll hack up a test to try to measure the relative overhead of VMA
faults before/after (modulo access to meaningful HW).
Alex Mastro and I have put together some tests, and measured this
overhead on the v8 series.
https://github.com/evansm7/vfio-playground/tree/master/vfio-barmap-perf
Alex also generated a similar test with a slightly different method:
https://github.com/opsound/vfio-tools/blob/main/vfio_bar_bench.c
Test method:
Time accesses to a VFIO mmap'd BAR immediately after the mmap, to
measure the 'cold' initial fault time. Re-time a second pass for which
the pagetable should now be populated, and subtract. This should
isolate the cost of the fault (uarch exception, generic pagefault
plumbing, VFIO fault handler) from common costs (the MMIO access itself,
TLB misses, the loop iterating across the BAR). Maximise the number of
faults by disabling hugepages (per concern above).
This is done both for a single-threaded case which just iterates across
a new BAR mapping, and a multi-threaded contended case. Rather than
just pthreads, this test fork()s to try to avoid seeing contention on
per-mm locks (e.g. mmap_lock) and to focus on the VMA/VFIO locking.
The default test mmaps before forking, meaning there's one shared
mapping/DMABUF accessed from both processes. The fault handler should
then contend on dmabuf->resv, vdev->memory_lock, and vdev->dmabuf_lock.
For comparison/curiosity, the --mt-map-late option mmaps _after_
forking; this leads to a mapping/DMABUF in each process so I expect no
contention on dmabuf->resv (but still on memory_lock/dmabuf_lock).
Alex's test instead runs threads in-process, and goes further with
supporting a larger number of threads (though with the caveat that they
may also be fighting over intra-process mm locks). It also supports a
mode where threads can access per-thread VMAs (which is comparable to
the 'mt-map-late' option of my test). Both are interesting approaches IMO.
Findings:
On the dusty AMD Ryzen 3950X under my desk (targeting a 256MB BAR) I
found the following averaged over 200 runs:
- A single-threaded/uncontended fault became 5.8% slower
- A two-process contended case became 8.7% slower
- 0.5% of this is the "early mmap" dmabuf->resv contention, i.e. using
mt-map-late (a mapping in each process) reduces by 0.5%.
Alex's runs on fancier EPYC 9334 machines show a similar (slightly
lower) overhead in the 1-2 case, and go on to show fairly high overhead
when many threads are involved (baseline times in ns, relative series v8
behaviour in percentage change):
Threads One shared mapping Mapping per thread
7.3 v8 7.3 v8
--------------------------------------------
1 761 +4.5% 777 +2.7%
2 869 +6.6% 840 +5.6%
8 1465 +78% 1363 +47%
16 4152 +97% 4258 +19%
32 9927 +92% 10158 +12%
Now. The absolute timings of these sub-microsecond faults indicate a
realistic application won't see a human-visible change. Small BARs
mapped with 4k pages are still fast to map; it is reasonable to ask
that applications map huge (GBs/TBs) BARs using huge (MB/GB) pages! Of
course, this is just a one-time cost of warming up the VMA, not ongoing.
Alex, my preference is not to complicate the fault handler with a fast
path just yet. It's easy to think of a contrived scenario where this
matters but in middle-of-the-road application behaviour it would be
surprising if this was noticeable. What are your thoughts?
(In the previous mail I suggested keying off priv->vfile (or a flag
thereof), and it seems ~straightforward to skip the extra
synchronisation. If you feel strongly that the fault handler doesn't
have enough if() statements yet and needs more paths then we could bring
this into v9, but would prefer to follow up later. :) A fastpath diff
appears small but the reasoning overhead is bigger.)
Thanks,
Matt