On 13-08-2026 17:12, Paolo Abeni wrote:
On 8/7/26 10:56 PM, Aditya Garg wrote:
The MANA driver can fail to bring up its queues on systems with high
memory utilization because every GDMA queue ring is allocated as a
single dma_alloc_coherent() of the whole power-of-2 ring size. Under
memory fragmentation these high-order allocations may fail, preventing
the driver from creating queues when opening the interface, after a VF
reset, or when reconfiguring channels, ring parameters or MTU.
Per-queue sizes that are problematic, with depth and size given as
(default, max) over the ethtool ring settings:
ring entry depth size
------------------------------------------------------------
TX completion queue 64 B (256, 16384) (16 KB, 1024 KB)
TX send queue 32 B (256, 16384) ( 8 KB, 512 KB)
RX completion queue 64 B (1024, 8192) (64 KB, 512 KB)
RX receive queue 32 B (1024, 8192) (32 KB, 256 KB)
event queue 16 B 2048 (fixed) 32 KB
This series addresses the issue by:
1. Routing all CPU-side ring access through mana_gd_ring_ptr() and
mana_gd_ring_contig_avail(). On a contiguous ring these reduce to
simple arithmetic, so this patch is a pure refactor.
2. Falling back in mana_gd_alloc_memory() to a vector of scattered
order-0 coherent pages when the contiguous allocation fails. The
device sees the same page-list format either way, as
mana_gd_create_dma_region() already describes a ring as a list of
MANA_PAGE_SIZE addresses. The HW channel stays contiguous, as
advertising a scattered page list needs the HW channel itself.
Throughput testing confirms no regression. Since the fallback only
triggers under memory fragmentation, the scattered-page path was enabled
unconditionally for all eligible GDMA queue rings during testing (iperf3,
Gbit/s):
Baseline Patched Patched
Connections Contiguous Contiguous Scattered
-----------------------------------------------
1 46.1 46.2 46.1
16 182 182 182
32 182 182 182
64 182 182 182
Given the above, I suggest to follow-up and remove entirely the single
dma_alloc_coherent() allocation; you will get simpler code and less
pressure for later dma users.
/P
Thanks for the review, Paolo!
I got these throughput numbers from an idle system running only iperf3,
so we're not sure whether they would remain the same with other
workloads running in parallel. I'll try to get these numbers as well.
Also, our hardware team indicated that this fallback leads to additional
resource consumption on their side. I'll check with them and post a
follow-up.
Regards,
Aditya
