From: "Kiryl Shutsemau (Meta)" <[email protected]> Anonymous collapse is held back in two ways that its current shape cannot be patched out of.
Functionally, an mTHP is collapsed only when the whole PMD-sized window qualifies: collapse_scan_pmd() reaches mthp_collapse() only on SCAN_SUCCEED. One PTE that disqualifies itself -- uffd-armed, not anonymous, clean lazyfree, off the LRU, pinned -- takes the whole table with it. A 2M range with a single such page yields nothing, even where the half beside it would collapse perfectly well. For scalability, collapse_huge_page() holds mmap_write across a collapse, and anon_vma_lock_write() with it, then does the same again for the next huge page. Every collapse stops every fault in the address space, one huge page at a time. The new engine addresses both. It quiesces its sources the way migration does: migration entries in their PTEs, then a frozen refcount. That is enough to make the copy safe without the exclusive lock, so the engine runs under mmap_read throughout. It also carries a batch of windows through each step together, rather than one window through all of them. Its verdict is per window rather than per table, so a table that cannot become one huge page still yields the largest windows inside it. Lay the design out first and fill it in afterwards. What arrives here is the shape of the engine: the two entry points, and a comment at the top of mm/collapse.c mapping the whole call tree. Every step below them is a stub, filled in before the anonymous path is pointed at the engine. collapse_scan_anon_pmd() finds the table, asks the VMA which orders it allows, scans it, and leaves in the collapse_control what a collapse could use. collapse_anon_pmd() takes that range and cuts it into windows. The line between the two is where the lock goes. A scan only reads a VMA and a page table, so it keeps the mmap_lock it was called under and hands it back. A collapse allocates, copies and flushes, so it is called without the lock and takes it again for each round of its own. Nothing then has to tell a caller whether the lock survived a call, and a caller that finds nothing to collapse never gives the lock up at all. Nothing calls either entry point yet; the anonymous path keeps using the mechanism the engine replaces. Assisted-by: Claude-Code:claude-opus-5 Signed-off-by: Kiryl Shutsemau (Meta) <[email protected]> --- mm/Makefile | 2 +- mm/collapse.c | 165 ++++++++++++++++++++++++++++++++++++++++++++++++ mm/collapse.h | 24 +++++++ mm/khugepaged.c | 4 +- 4 files changed, 192 insertions(+), 3 deletions(-) create mode 100644 mm/collapse.c diff --git a/mm/Makefile b/mm/Makefile index e7245cb88c66..2bef749a5c21 100644 --- a/mm/Makefile +++ b/mm/Makefile @@ -98,7 +98,7 @@ obj-$(CONFIG_MEMTEST) += memtest.o obj-$(CONFIG_MIGRATION) += migrate.o obj-$(CONFIG_NUMA) += memory-tiers.o obj-$(CONFIG_DEVICE_MIGRATION) += migrate_device.o -obj-$(CONFIG_TRANSPARENT_HUGEPAGE) += huge_memory.o khugepaged.o +obj-$(CONFIG_TRANSPARENT_HUGEPAGE) += collapse.o huge_memory.o khugepaged.o obj-$(CONFIG_PAGE_COUNTER) += page_counter.o obj-$(CONFIG_LIVEUPDATE_MEMFD) += memfd_luo.o obj-$(CONFIG_MEMCG_V1) += memcontrol-v1.o diff --git a/mm/collapse.c b/mm/collapse.c new file mode 100644 index 000000000000..0e6c3c68b44c --- /dev/null +++ b/mm/collapse.c @@ -0,0 +1,165 @@ +// SPDX-License-Identifier: GPL-2.0 +#define pr_fmt(fmt) KBUILD_MODNAME ": " fmt + +#include <linux/bitops.h> +#include <linux/highmem.h> +#include <linux/huge_mm.h> +#include <linux/hugetlb.h> /* x86 flush_tlb_range() uses hstate_vma() */ +#include <linux/leafops.h> +#include <linux/memcontrol.h> +#include <linux/mm.h> +#include <linux/mmu_notifier.h> +#include <linux/pagemap.h> +#include <linux/pgalloc.h> +#include <linux/rmap.h> +#include <linux/sched.h> +#include <linux/sizes.h> +#include <linux/slab.h> +#include <linux/swap.h> +#include <linux/userfaultfd_k.h> + +#include <asm/tlb.h> +#include "collapse.h" +#include "internal.h" + +/* + * Anonymous collapse, in rounds. + * + * The folios mapped across a window of PTEs become one folio of that window's + * order, with the sources quiesced by the two barriers migration uses -- + * migration entries in their PTEs, then a frozen refcount -- so the copy itself + * needs no lock. The engine runs under mmap_read throughout. + * + * A round carries a batch of candidate windows through the passes together, + * rather than carrying one window through the whole collapse. [ptl] and + * [pmd lock] mark a pass that takes that lock and drops it again, so no + * page-table lock is ever held across passes; the source folio locks are the + * exception, held from freeze to putback. [rcu] marks a pass that takes no + * page-table lock at all and reads the table racily, which only the scan does. + * + * Allocation happens on both sides of the freeze, and which side comes first + * matters. collapse_provision() tries first, inside the window and after the + * freeze, with reclaim masked out of the gfp: the sources are frozen by then, + * so a faulter on one of them waits for this allocation. A candidate the + * allocator has nothing ready for is not failed -- it goes back to selection, + * and collapse_reserve() allocates for it before the next round freezes + * anything, outside the window, where reclaim costs khugepaged its own + * progress and nobody else's wait. + * + * That second chance needs the policy's gfp to allow reclaim at all. When it + * does not, a retry would miss the same way, so the first miss is the answer. + * + * collapse_scan_anon_pmd() judge one PTE table's worth of a VMA + * `- collapse_scan_table() [rcu] a bit per PTE a collapse can use + * + * collapse_anon_pmd() cut windows from those bits, run them + * |- collapse_next_candidate() the next window worth attempting + * `- collapse_run_batch() run the batch, then classify it + * |- collapse_round() below + * `- collapse_classify_result() carry on / lower / abandon + * `- collapse_push_retry() queue it for a lower order + * + * collapse_round() one batch of candidates + * |- collapse_reserve() second try for what the last round + * | missed, with reclaim; sleeps + * |- collapse_deposit() a page table per PMD-order candidate + * |- collapse_revalidate() check the VMA and the table survived + * |- collapse_faultin() make the sources present and exclusive; + * | sleeps, and may leave the lock dropped + * |- collapse_freeze() raise the barriers [ptl], flush the TLB + * |- collapse_provision() first try for every other destination, + * | without reclaim: the sources are frozen + * |- collapse_copy() copy into the destinations; sleeps + * |- collapse_install() publish them [ptl], or [pmd lock] and a + * | second TLB flush at the PMD order + * |- collapse_putback() lower the barriers, in order + * `- collapse_finish() settle whatever the round reached + * + * Every slot of a candidate is a real source, a hole (pte_none, zero-filled + * and re-verified still-none at install), or the zeropage (cleared at freeze, + * zero-filled). Sources come in "spans" -- consecutive PTEs mapping + * consecutive pages of one folio -- so partially mapped and compound sources + * collapse too: any order below the window's is a source, and a PMD candidate + * takes even a PTE-mapped THP of its own order. + * + * Nothing calls any of this yet: the anon path still uses the mechanism it + * replaces, and is switched over once both halves are complete. + */ + +/* + * Scan the PTEs between @start and @end and record what a collapse could use: a + * bit in cc->eligible_ptes for every PTE that may be a source. Returns + * SCAN_SUCCEED when every PTE in the range qualified, otherwise the reason one + * did not, and narrows cc->select_orders to what is still worth trying here. + */ +static enum scan_result collapse_scan_table(struct vm_area_struct *vma, + pmd_t *pmd, unsigned long start, + unsigned long end, + struct collapse_control *cc) +{ + return SCAN_SUCCEED; +} + +/* Everything a table is judged on starts empty for each table */ +static void collapse_anon_scan_init(struct collapse_control *cc) +{ + bitmap_zero(cc->eligible_ptes, MAX_PTRS_PER_PTE); + memset(cc->node_load, 0, sizeof(cc->node_load)); + nodes_clear(cc->alloc_nmask); + + cc->select_orders = 0; + cc->nr_collapsed = 0; +} + +/* + * Judge one table's worth of @vma, leaving in @cc what a collapse could use: + * which orders are still worth attempting, and why the table was turned down if + * some order was. Holds mmap_lock throughout -- it only reads -- and a caller + * that acts on what it found hands the range to collapse_anon_pmd() afterwards, + * without the lock. + */ +static enum scan_result __maybe_unused +collapse_scan_anon_pmd(struct vm_area_struct *vma, unsigned long start, + unsigned long end, struct collapse_control *cc) +{ + const unsigned long pmd_addr = start & HPAGE_PMD_MASK; + struct mm_struct *mm = vma->vm_mm; + pmd_t *pmd; + + /* One table's worth at most, not empty, and inside the VMA */ + VM_WARN_ON_ONCE(end > pmd_addr + HPAGE_PMD_SIZE || start >= end); + VM_WARN_ON_ONCE(start < vma->vm_start || end > vma->vm_end); + + cc->scan_refusal = find_pmd_or_thp_or_none(mm, pmd_addr, &pmd); + if (cc->scan_refusal != SCAN_SUCCEED) { + cc->progress++; + return cc->scan_refusal; + } + + /* Cleared only once a table has turned out to be there */ + collapse_anon_scan_init(cc); + + cc->select_orders = collapse_possible_orders(vma, vma->vm_flags, + cc->policy.tva_type); + if (!cc->select_orders) { + cc->scan_refusal = SCAN_VMA_CHECK; + return cc->scan_refusal; + } + + /* The scan narrows select_orders to whatever is left worth trying */ + cc->scan_refusal = collapse_scan_table(vma, pmd, start, end, cc); + + return cc->scan_refusal; +} + +/* + * Cut the table into candidate windows and collapse what fits, from the + * largest order downwards. Returns what the table yielded: a collapse, or + * the reason it did not. + */ +static enum scan_result __maybe_unused +collapse_anon_pmd(struct mm_struct *mm, unsigned long start, unsigned long end, + struct collapse_control *cc) +{ + return SCAN_FAIL; +} diff --git a/mm/collapse.h b/mm/collapse.h index 1e969292edcb..e2af4c47cb60 100644 --- a/mm/collapse.h +++ b/mm/collapse.h @@ -105,6 +105,30 @@ struct collapse_control { /* Each bit marks a PTE the scan accepted as a collapse source */ DECLARE_BITMAP(eligible_ptes, MAX_PTRS_PER_PTE); + + /* Orders still worth attempting in the table being scanned */ + unsigned long select_orders; + + /* PTEs collapsed in it so far */ + unsigned int nr_collapsed; + + /* + * Why the scan would not take all of the table, or SCAN_SUCCEED if it + * took every order it was offered. Not the opposite of what the scan + * selected: a table can be worth collapsing at one order and refused at + * another, so a scan that found work still has a reason to report, and + * the collapse reports it when it salvages nothing. + */ + enum scan_result scan_refusal; }; +/* + * Defined in khugepaged.c, which still uses them itself. + * TODO: move each into collapse.c once its last khugepaged.c user is gone. + */ +unsigned long collapse_possible_orders(struct vm_area_struct *vma, + vm_flags_t vm_flags, enum tva_type tva_flags); +enum scan_result find_pmd_or_thp_or_none(struct mm_struct *mm, + unsigned long address, pmd_t **pmd); + #endif /* __MM_COLLAPSE_H */ diff --git a/mm/khugepaged.c b/mm/khugepaged.c index f31689bf75a6..26d25093260b 100644 --- a/mm/khugepaged.c +++ b/mm/khugepaged.c @@ -498,7 +498,7 @@ void __khugepaged_enter(struct mm_struct *mm) * Check what orders are possible based on the vma and collapse type. * This is used to determine if mTHP collapse is a viable option. */ -static unsigned long collapse_possible_orders(struct vm_area_struct *vma, +unsigned long collapse_possible_orders(struct vm_area_struct *vma, vm_flags_t vm_flags, enum tva_type tva_flags) { unsigned long orders; @@ -1090,7 +1090,7 @@ static inline enum scan_result check_pmd_state(pmd_t *pmd) return SCAN_SUCCEED; } -static enum scan_result find_pmd_or_thp_or_none(struct mm_struct *mm, +enum scan_result find_pmd_or_thp_or_none(struct mm_struct *mm, unsigned long address, pmd_t **pmd) { *pmd = mm_find_pmd(mm, address); -- 2.54.0
