Hello Amelia, thank you for your analysis, good catch. What do you think of https://gerrit.fd.io/r/c/vpp/+/46182 ?
Best ben ________________________________________ From: [email protected] <[email protected]> on behalf of amelia yin via lists.fd.io <[email protected]> Sent: Friday, June 26, 2026 10:39 To: [email protected] Subject: [vpp-dev] 26.06: Page boundary check in vlib_buffer_pool_create causes severe buffer underutilization with 4KB pages Hi Team, We recently upgraded from VPP 23.02 to 26.06 and observed a significant regression in buffer pool allocation. We'd like to understand the rationale behind the page boundary check introduced in vlib_buffer_pool_create (in src/vlib/buffer.c). The problem: With the same configuration that has worked since 23.02: buffers { buffers-per-numa 128000 default data-size 3072 } On VPP 23.02, buffer allocation works as expected: vpp# show buffers Pool Name Index NUMA Size Data Size Total Avail Cached Used default-numa-0 0 0 3968 3072 128000 127968 32 0 default-numa-1 1 1 3968 3072 128000 128000 0 0 default-numa-2 2 2 3968 3072 128000 128000 0 0 default-numa-3 3 3 3968 3072 128000 128000 0 0 On VPP 26.06, only ~2032 buffers are allocated per NUMA instead of 128000: vpp# show buffers Pool Name Index NUMA Size Data Size Total Avail Cached Used default-numa-0 0 0 3968 3072 2031 460 32 1539 default-numa-1 1 1 3968 3072 2032 2032 0 0 default-numa-2 2 2 3968 3072 2032 2032 0 0 default-numa-3 3 3 3968 3072 2031 2031 0 0 Root cause analysis: The new code in vlib_buffer_pool_create skips any buffer that would span across a page boundary: /* skip if buffer spans across page boundary */ if (((uword) p & page_mask) != ((uword) (p + alloc_size) & page_mask)) continue; With 4KB pages and alloc_size = 3968, only 1 buffer fits per 4KB page (3968 < 4096), but the linear iteration with p += alloc_size causes the buffer start offset to drift across pages. The check then rejects ~94% of candidate slots because most will cross a page boundary when stepping by 3968 through a 4096-aligned address space. The memory is allocated (physmem region is sized for 128000 buffers), but most of it goes unused — an effective utilization rate of only ~1.6%. my questions: 1. Was the impact on non-hugepage deployments (4KB pages) considered? With hugepages (e.g., 2MB), the check is essentially a no-op since many buffers fit within a single page, but with 4KB pages and large buffer sizes (>2048), the utilization drops drastically. 2. Would it be acceptable upstream to use a per-page iteration approach instead — placing buffers at page-aligned offsets to guarantee no cross-page spanning while maintaining full utilization? Thanks & Regards Amelia
-=-=-=-=-=-=-=-=-=-=-=- Links: You receive all messages sent to this group. View/Reply Online (#27094): https://lists.fd.io/g/vpp-dev/message/27094 Mute This Topic: https://lists.fd.io/mt/119987044/21656 Group Owner: [email protected] Unsubscribe: https://lists.fd.io/g/vpp-dev/leave/14379924/21656/631435203/xyzzy [[email protected]] -=-=-=-=-=-=-=-=-=-=-=-
