On Wed, Jul 1, 2026 at 12:22 PM Bobby Eshleman <[email protected]> wrote: > > From: Bobby Eshleman <[email protected]> > > Every devmem dmabuf binding today hands the page_pool PAGE_SIZE niovs. > This caps a single RX descriptor at PAGE_SIZE, burning CPU on buffer > churn for large flows. > > Add a bind-time netlink attribute, NETDEV_A_DMABUF_RX_BUF_SIZE, that > lets userspace request a larger niov size. The value must be a power of > two >= PAGE_SIZE. > > Measurements > ------------ > Setup: kperf in devmem RX/TX cuda mode, 4 flows, 64 MB messages, 60s, > dctcp, num-rx-queues=4, dmabuf-rx/tx-size-mb=2048, 10 runs per niov > size, mlx5. > > CPU Util: > > niov net sirq % net idle % app sys % app idle > % > ----- ---------------- ---------------- ---------------- > ---------------- > 4K 62.38 +/- 8.27 33.40 +/- 7.51 54.15 +/- 10.23 43.67 +/- > 10.53 > 16K 58.91 +/- 5.35 35.23 +/- 5.88 41.05 +/- 8.87 56.42 +/- > 9.24 > 32K 64.12 +/- 0.68 31.09 +/- 1.48 44.54 +/- 3.51 52.63 +/- > 3.65 > 64K 54.69 +/- 5.54 39.67 +/- 5.81 35.47 +/- 3.11 61.97 +/- > 3.27 > > RX app sys % drops ~19% from 4K to 64K. > > Throughput: > > niov RX dev Gbps RX flow avg Gbps > ----- ---------------- ----------------- > 4K 300.63 +/- 53.21 75.16 +/- 13.30 > 16K 321.35 +/- 28.20 80.34 +/- 7.05 > 32K 347.63 +/- 2.20 86.91 +/- 0.55 > 64K 332.11 +/- 14.26 83.03 +/- 3.56 > > Throughput seems to increase, but the stdev is pretty wide so could just > be noise. > > kperf support (not yet merged): > https://github.com/facebookexperimental/kperf/commit/8837577f920876bce6986ec18869ac04439ebcd2 > > Signed-off-by: Bobby Eshleman <[email protected]> > Acked-by: Stanislav Fomichev <[email protected]>
I'm pretty happy to see most of this patch being a spot-for-spot replacement of PAGE_SIZE with a variable. FWIW: Reviewed-by: Mina Almasry <[email protected]> > --- > Documentation/netlink/specs/netdev.yaml | 8 +++++ > include/uapi/linux/netdev.h | 1 + > net/core/devmem.c | 55 > +++++++++++++++++++-------------- > net/core/devmem.h | 13 +++++--- > net/core/netdev-genl-gen.c | 5 +-- > net/core/netdev-genl.c | 19 ++++++++++-- > tools/include/uapi/linux/netdev.h | 1 + > 7 files changed, 71 insertions(+), 31 deletions(-) > > diff --git a/Documentation/netlink/specs/netdev.yaml > b/Documentation/netlink/specs/netdev.yaml > index 5f143da7458c..70b902008bd3 100644 > --- a/Documentation/netlink/specs/netdev.yaml > +++ b/Documentation/netlink/specs/netdev.yaml > @@ -598,6 +598,13 @@ attribute-sets: > type: u32 > checks: > min: 1 > + - > + name: rx-buf-size > + doc: | > + Size in bytes of each RX buffer the NIC writes into from the bound > + dmabuf. Must be a power of two and >= PAGE_SIZE; defaults to > + PAGE_SIZE. > + type: u32 > > operations: > list: > @@ -812,6 +819,7 @@ operations: > - ifindex > - fd > - queues > + - rx-buf-size > reply: > attributes: > - id > diff --git a/include/uapi/linux/netdev.h b/include/uapi/linux/netdev.h > index 2f3ab75e8cc0..85e1d20c6268 100644 > --- a/include/uapi/linux/netdev.h > +++ b/include/uapi/linux/netdev.h > @@ -219,6 +219,7 @@ enum { > NETDEV_A_DMABUF_QUEUES, > NETDEV_A_DMABUF_FD, > NETDEV_A_DMABUF_ID, > + NETDEV_A_DMABUF_RX_BUF_SIZE, > > __NETDEV_A_DMABUF_MAX, > NETDEV_A_DMABUF_MAX = (__NETDEV_A_DMABUF_MAX - 1) > diff --git a/net/core/devmem.c b/net/core/devmem.c > index 957d6b96216b..3d6cf35e50f3 100644 > --- a/net/core/devmem.c > +++ b/net/core/devmem.c > @@ -46,7 +46,7 @@ static dma_addr_t net_devmem_get_dma_addr(const struct > net_iov *niov) > > owner = net_devmem_iov_to_chunk_owner(niov); > return owner->base_dma_addr + > - ((dma_addr_t)net_iov_idx(niov) << PAGE_SHIFT); > + ((dma_addr_t)net_iov_idx(niov) << owner->binding->niov_shift); > } > > static void net_devmem_dmabuf_binding_release(struct percpu_ref *ref) > @@ -90,16 +90,17 @@ net_devmem_alloc_dmabuf(struct net_devmem_dmabuf_binding > *binding) > struct dmabuf_genpool_chunk_owner *owner; > unsigned long dma_addr; > struct net_iov *niov; > - ssize_t offset; > - ssize_t index; > + size_t offset; > + size_t index; > nit: I would keep this signed. Some of the most frustrating issues I ran into is some of the underflowing and then passing a > check or something. Although if the LLM is not complaining about this particular case, there is probably no issue with it. I also notice a lot of existing code that deals with indexes and offsets goes for signed. -- Thanks, Mina
