On 5/28/26 21:56, Morten Brørup wrote:
From: Varghese, Vipin [mailto:[email protected]]
Sent: Thursday, 28 May 2026 16.45
Public
Hi @Morten Brørup
<snipped>
+/**
+ * Pre-reserve backing memory.
+ *
+ * Ensures that at least @p size bytes of memzone-backed memory
are
+ * available to the allocator on @p socket_id, reserving
additional
+ * memzones from EAL as needed to reach that total. Subsequent
+ * allocations served from the pre-reserved memory do not incur
+ * memzone-reservation cost.
+ *
+ * The reservation is cumulative: repeated calls to
+ * rte_fastmem_reserve() with the same @p socket_id grow the
+ * reservation monotonically. Reserved memory is never returned to
+ * the system during the allocator's lifetime.
+ *
+ * A typical use is to call rte_fastmem_reserve() once at
+ * application startup, with a size chosen to cover the expected
+ * steady-state working set. Allocations and frees during
+ * steady-state operation then avoid memzone reservations
entirely.
+ *
+ * @param size
+ * The minimum amount of backing memory, in bytes, to make
+ * available on @p socket_id. The allocator may reserve more than
+ * the requested amount due to internal rounding (e.g., to
memzone
+ * or block granularity).
+ *
+ * @param socket_id
+ * The NUMA socket on which to reserve memory, or SOCKET_ID_ANY
+ * to leave the choice to the allocator. With SOCKET_ID_ANY, the
+ * allocator starts on the calling lcore's socket (or the first
+ * configured socket if the caller is not bound to one) and falls
+ * back to other sockets if the preferred socket cannot satisfy
+ * the reservation.
+ *
+ * @return
+ * - 0: Success.
+ * - -ENOMEM: Insufficient huge-page memory to satisfy the
request.
+ * - -EINVAL: Invalid @p socket_id.
+ */
+__rte_experimental
+int
+rte_fastmem_reserve(size_t size, int socket_id);
@Bruce,
I vaguely recall that we discussed something about busses and sockets
a long time
ago, but I cannot remember the details.
Is socket_id the right type (and parameter name) to identify a memory
bus?
@Vipin,
You have been working on topology awareness. Same question to you:
Is socket_id the right type (and parameter name) to identify a memory
bus?
Short answer: socket_id is no longer a precise or sufficient
abstraction to represent a memory bus.
Based on the topology work with libhwloc, we’ve observed the following
across Ampere, Intel, and AMD platforms:
Features like SNC (Sub-NUMA Clustering) on Intel and NPS (NUMA Per
Socket) on AMD change how socket_id maps to hardware.
In these modes:
1) A single physical socket can expose multiple NUMA domains.
2)These NUMA domains align more closely with memory controller
groupings (i.e., memory buses) rather than the full socket.
Depending on the architecture:
a) Memory controllers may be collocated with compute cores or placed on
separate tiles.
b) As a result, socket_id can represent different scopes (full socket
vs. sub-socket domains), making it inconsistent.
Hence practically: In some configurations, socket_id ≈ memory domain.
In others, it is coarser than the actual memory bus topology.
To address this ambiguity, in the topology patches (v5/v6), we are
moving toward clearer separation:
a. Cache domains (L1/L2/L3/L4) for compute locality
b. NUMA domains (memory + IO) as the unit for allocation locality
This direction better reflects real hardware and avoids overloading
socket_id with multiple meanings.
Happy to align this with the topology model we’re introducing so the
abstraction remains consistent going forward.
Thanks,
Vipin
Thank you for the quick and detailed response, Vipin!
I haven't looked deeply into the v5/v6 topology patches yet (it's on my TODO
list).
The rte_fastmem library builds on top of the rte_memzone library.
So, if the rte_memzone library is updated to replace the meaning of its "socket_id" parameter with some NUMA domain
identifier (we better rename the "socket_id" to a new name "numa_domain_id"), then the rte_fastmem library
could remain unaffected, and its "socket_id" parameter would be passed on directly to the rte_memzone library's
"numa_domain_id"?
What is a "domain" here? It's the same as what is usually (in my
experience) referred to as a NUMA node?
I would just call it "node_id".
This is my conclusion: At this point, proper support for allocating memory in
specific NUMA domains is an rte_memzone library issue, and nothing to worry
about for the rte_fastmem library - it will be automatically supported in
rte_fastmem when supported by rte_memzone.
