On 11/1/26 03:13, Gregory Price (Meta) wrote: > Memory hotplug rebuilds zonelists in place while allocations walk them, > and offlining an emptied zone frees pagesets the vmstat worker can still > hold. > > These eleven fixes cover the walkers that crash, fail, or pick the wrong > zone or node as a result, and a twelfth patch removes a helper that > invites the same bug. > > 1. a restarted slowpath loses its retry with reserves > 2. offlining an emptied zone frees pagesets still in use > 3. the reclaim_retry_zone tracepoint rereads a zoneref slot > 4. the zonelist retry is compiled out without MEMORY_HOTREMOVE > 5. zonelist_zone() may load its slot more than once > 6. NULL preferred zone after a shrink rebuild > 7. __next_zones_zonelist() rereads a slot > 8. allocations land outside their zone or nodemask > 9. the hugetlb pool walk takes another node's page > 10. alloc_contig_pages() takes a range outside its filters > 11. mempolicy_slab_node() rereads a slot > 12. zonelist_node_idx() invites the same double read > > Patches 1-11 are for stable. Patch 12 is a cleanup and is not. > > Patches 8-10 need patch 4, without which their retry is compiled > out on HOTREMOVE=n kernels. > > Patch 8 needs patch 1. > > Patches 6, 7 and 11 need patch 5, which makes zonelist_zone() read > the slot once. Patch 12 needs patches 7 and 11, which remove the > other callers of zonelist_node_idx(). > > Patches 2 and 3 are patches from anthropic, pulled in from Vlastimil's > series to keep the backport issues self-contained on the series. > > Testing > ======= > > Each race was reproduced on unmodified source in a QEMU guest with three > NUMA nodes and a 512M pc-dimm on node 1, left offline at boot. > > A test module widens each window with a kprobe that stalls one CPU inside > it, as an interrupt or a vCPU preemption would. > > base this series > patch outcome runs outcome runs > ===== ======================= ===== ======================= ===== > 1 NULL returned 3/3 page from reserves 3/3 > 2 UAF 6/6 no UAF 3/3 > 3 NULL dereference 3/3 allocation succeeds 3/3 > 4 allocation fails 3/3 waits for the rebuild 3/3 > 6 NULL dereference 3/3 ZONE_NORMAL page 3/3 > 7 NULL dereference 3/3 ZONE_NORMAL page 3/3 > 8 wrong zone or node 18/18 right zone and node 18/18 > 9 page from node 1 3/3 page from node 2 3/3 > 10 1G page in ZONE_MOVABLE 3/3 1G page in ZONE_NORMAL 3/3 > 11 NULL dereference 4/4 keeps allocating 3/3 > > > Cost (these tests were run on v4, v5 hasn't substantially changed) > ==== > Kernel instructions per order-0 allocation and free in the QEMU guest, > counted with a task-bound hardware instruction counter around 1M > iterations on a pinned kthread, non-debug kernel, mean of 15 rounds. > > instructions per alloc+free > base this series > alloc_pages() 890.5 900.5 > bulk, nr=1, nodemask 865.5 875.5 > bulk, nr=16, per page 477.2 478.1 > > > Bare metal, two-socket Xeon Gold 6138 (Skylake-SP, 80 CPUs). > > will-it-scale page_fault1-3: each iteration touches one page, so > the unit is millions of faults per second, summed over all tasks. > > Base, millions of faults per second: > > tasks 1 10 40 80 > page_fault1, processes 0.46 4.16 10.51 9.23 > page_fault1, threads 0.46 3.51 4.35 4.19 > page_fault2, processes 0.43 3.66 9.02 8.33 > page_fault2, threads 0.43 3.48 4.37 4.19 > page_fault3, processes 3.89 14.78 8.10 8.00 > page_fault3, threads 3.87 14.29 6.88 7.96 > > Change in faults per second with this series, + is faster: > > tasks 1 10 40 80 > page_fault1, processes -0.3% +0.1% -3.0% -1.7% > page_fault1, threads -0.5% -1.6% -0.9% -0.9% > page_fault2, processes +0.0% +0.9% -1.4% -0.5% > page_fault2, threads +0.3% +1.5% -1.4% -1.0% > page_fault3, processes -1.0% -3.8% -0.7% +1.2% > page_fault3, threads -0.9% -0.7% +0.8% -0.2% > > The boot-to-boot stddev is 1.0-1.8% at 1 task and up to > 15% in 10-80 tasks, so basically this is all within noise. > > Changes in v5 > ============= > - Pulled in patches from Vlastimil's series (patches 2 and 3 here) > - inline functions (Vlastimil) > - Added patch 5: zonelist_zone() reads the slot with READ_ONCE(), as > recommended by sashiko on v4's mempolicy patch. > Patches 6, 7 and 11 call it instead of open-coding the load. > - Added patch 12, removing zonelist_node_idx(). Not for stable. > - Patch 1: reworded the changelog (Vlastimil). > - Collected Reviewed-by tags. > > Link: > https://lore.kernel.org/linux-mm/[email protected]/ > Link: > https://lore.kernel.org/linux-mm/[email protected]/ > Link: > https://lore.kernel.org/linux-mm/[email protected]/ > Link: > https://lore.kernel.org/linux-mm/[email protected]/ > Link: > https://lore.kernel.org/linux-mm/[email protected]/ > > Gregory Price (Meta) (10): > mm/page_alloc: reset can_retry_reserves when the slowpath restarts > mm/page_alloc: detect zonelist rebuilds from memory hot-add > mm/mmzone: read the zoneref slot once in zonelist_zone() > mm/page_alloc: snapshot the preferred zone against shrink rebuilds > mm/mmzone: read each zoneref slot once when filtering by nodemask > mm/page_alloc: retry allocations that raced with a zonelist rebuild > mm/hugetlb: retry the pool walk when the zonelist is rebuilt > mm/page_alloc: retry alloc_contig_pages() when the zonelist is rebuilt > mm/mempolicy: read the zoneref slot once in mempolicy_slab_node() > mm/mmzone: remove zonelist_node_idx() > > Josef Bacik (2): > mm/page_alloc: keep the pagesets of an empty zone > mm, page_alloc: pass zone pointer to trace_reclaim_retry_zone
Thanks, applied to mm/page_alloc [1] page_alloc/for-next-fixes It should result in no conflicts with other trees AFAICS. Following David's MAINTAINERS updates [2] and mm-next call discussions, we'll move ahead with the page_alloc.git tree (integrated to mm-next) now to spread the handling of the endless fixes stream. I'll send the MAINTAINERS update soon. [1] https://git.kernel.org/pub/scm/linux/kernel/git/mm/page_alloc.git [2] https://lore.kernel.org/all/[email protected]/
