Memory hotplug rebuilds zonelists in place while allocations walk them,
and offlining an emptied zone frees pagesets the vmstat worker can still
hold.
These eleven fixes cover the walkers that crash, fail, or pick the wrong
zone or node as a result, and a twelfth patch removes a helper that
invites the same bug.
1. a restarted slowpath loses its retry with reserves
2. offlining an emptied zone frees pagesets still in use
3. the reclaim_retry_zone tracepoint rereads a zoneref slot
4. the zonelist retry is compiled out without MEMORY_HOTREMOVE
5. zonelist_zone() may load its slot more than once
6. NULL preferred zone after a shrink rebuild
7. __next_zones_zonelist() rereads a slot
8. allocations land outside their zone or nodemask
9. the hugetlb pool walk takes another node's page
10. alloc_contig_pages() takes a range outside its filters
11. mempolicy_slab_node() rereads a slot
12. zonelist_node_idx() invites the same double read
Patches 1-11 are for stable. Patch 12 is a cleanup and is not.
Patches 8-10 need patch 4, without which their retry is compiled
out on HOTREMOVE=n kernels.
Patch 8 needs patch 1.
Patches 6, 7 and 11 need patch 5, which makes zonelist_zone() read
the slot once. Patch 12 needs patches 7 and 11, which remove the
other callers of zonelist_node_idx().
Patches 2 and 3 are patches from anthropic, pulled in from Vlastimil's
series to keep the backport issues self-contained on the series.
Testing
=======
Each race was reproduced on unmodified source in a QEMU guest with three
NUMA nodes and a 512M pc-dimm on node 1, left offline at boot.
A test module widens each window with a kprobe that stalls one CPU inside
it, as an interrupt or a vCPU preemption would.
base this series
patch outcome runs outcome runs
===== ======================= ===== ======================= =====
1 NULL returned 3/3 page from reserves 3/3
2 UAF 6/6 no UAF 3/3
3 NULL dereference 3/3 allocation succeeds 3/3
4 allocation fails 3/3 waits for the rebuild 3/3
6 NULL dereference 3/3 ZONE_NORMAL page 3/3
7 NULL dereference 3/3 ZONE_NORMAL page 3/3
8 wrong zone or node 18/18 right zone and node 18/18
9 page from node 1 3/3 page from node 2 3/3
10 1G page in ZONE_MOVABLE 3/3 1G page in ZONE_NORMAL 3/3
11 NULL dereference 4/4 keeps allocating 3/3
Cost (these tests were run on v4, v5 hasn't substantially changed)
====
Kernel instructions per order-0 allocation and free in the QEMU guest,
counted with a task-bound hardware instruction counter around 1M
iterations on a pinned kthread, non-debug kernel, mean of 15 rounds.
instructions per alloc+free
base this series
alloc_pages() 890.5 900.5
bulk, nr=1, nodemask 865.5 875.5
bulk, nr=16, per page 477.2 478.1
Bare metal, two-socket Xeon Gold 6138 (Skylake-SP, 80 CPUs).
will-it-scale page_fault1-3: each iteration touches one page, so
the unit is millions of faults per second, summed over all tasks.
Base, millions of faults per second:
tasks 1 10 40 80
page_fault1, processes 0.46 4.16 10.51 9.23
page_fault1, threads 0.46 3.51 4.35 4.19
page_fault2, processes 0.43 3.66 9.02 8.33
page_fault2, threads 0.43 3.48 4.37 4.19
page_fault3, processes 3.89 14.78 8.10 8.00
page_fault3, threads 3.87 14.29 6.88 7.96
Change in faults per second with this series, + is faster:
tasks 1 10 40 80
page_fault1, processes -0.3% +0.1% -3.0% -1.7%
page_fault1, threads -0.5% -1.6% -0.9% -0.9%
page_fault2, processes +0.0% +0.9% -1.4% -0.5%
page_fault2, threads +0.3% +1.5% -1.4% -1.0%
page_fault3, processes -1.0% -3.8% -0.7% +1.2%
page_fault3, threads -0.9% -0.7% +0.8% -0.2%
The boot-to-boot stddev is 1.0-1.8% at 1 task and up to
15% in 10-80 tasks, so basically this is all within noise.
Changes in v5
=============
- Pulled in patches from Vlastimil's series (patches 2 and 3 here)
- inline functions (Vlastimil)
- Added patch 5: zonelist_zone() reads the slot with READ_ONCE(), as
recommended by sashiko on v4's mempolicy patch.
Patches 6, 7 and 11 call it instead of open-coding the load.
- Added patch 12, removing zonelist_node_idx(). Not for stable.
- Patch 1: reworded the changelog (Vlastimil).
- Collected Reviewed-by tags.
Link:
https://lore.kernel.org/linux-mm/[email protected]/
Link:
https://lore.kernel.org/linux-mm/[email protected]/
Link:
https://lore.kernel.org/linux-mm/[email protected]/
Link:
https://lore.kernel.org/linux-mm/[email protected]/
Link:
https://lore.kernel.org/linux-mm/[email protected]/
Gregory Price (Meta) (10):
mm/page_alloc: reset can_retry_reserves when the slowpath restarts
mm/page_alloc: detect zonelist rebuilds from memory hot-add
mm/mmzone: read the zoneref slot once in zonelist_zone()
mm/page_alloc: snapshot the preferred zone against shrink rebuilds
mm/mmzone: read each zoneref slot once when filtering by nodemask
mm/page_alloc: retry allocations that raced with a zonelist rebuild
mm/hugetlb: retry the pool walk when the zonelist is rebuilt
mm/page_alloc: retry alloc_contig_pages() when the zonelist is rebuilt
mm/mempolicy: read the zoneref slot once in mempolicy_slab_node()
mm/mmzone: remove zonelist_node_idx()
Josef Bacik (2):
mm/page_alloc: keep the pagesets of an empty zone
mm, page_alloc: pass zone pointer to trace_reclaim_retry_zone
include/linux/mmzone.h | 9 +---
include/trace/events/oom.h | 8 +--
mm/hugetlb.c | 13 +++--
mm/mempolicy.c | 6 ++-
mm/mmzone.c | 11 ++--
mm/page_alloc.c | 107 +++++++++++++++++++++++++++----------
mm/page_alloc.h | 19 +++++++
7 files changed, 123 insertions(+), 50 deletions(-)
--
2.56.0
