From: "Kiryl Shutsemau (Meta)" <[email protected]> khugepaged covers each VMA in whole PTE tables, from its first PMD-aligned address to its last. A VMA smaller than a table is never scanned at all, and in a larger one everything outside its PMD-aligned span is skipped.
That was the right shape when a collapse was always a PMD. It keeps mTHP collapse away from every range that is not PMD-shaped, which is most of what an mTHP is for. The gap is widest where a PMD is largest. On arm64 with 64K base pages a PMD is 512M, so the old walk reached only VMAs big enough and aligned well enough to hold one. A 2M mTHP -- order 5 there -- was unreachable in anything smaller, however many such windows the VMA had room for. Root the coverage at windows of the largest order the VMA allows, and hand the range on one table at a time, clamped to the VMA. The engine already accepts a partial table; this is the first caller that gives it one. The cursor is no longer PMD-aligned, so the assert that said it was goes. The bound beside it goes too: the range is clamped to the VMA where it is computed, leaving nothing for it to catch. For a VMA whose largest allowed order is the PMD order -- every file VMA, and any anonymous VMA with only PMD-order THP enabled -- the walk is exactly what it was. Where smaller orders are enabled, khugepaged now reaches VMAs a table would not fit in, and the edges of VMAs it used to leave out. Assisted-by: Claude-Code:claude-opus-5 Signed-off-by: Kiryl Shutsemau (Meta) <[email protected]> --- mm/khugepaged.c | 36 ++++++++++++++++++++++++------------ 1 file changed, 24 insertions(+), 12 deletions(-) diff --git a/mm/khugepaged.c b/mm/khugepaged.c index d1e031ed3e6f..895183d92fb8 100644 --- a/mm/khugepaged.c +++ b/mm/khugepaged.c @@ -2838,44 +2838,56 @@ static void collapse_scan_mm_slot(unsigned int progress_max, vma_iter_init(&vmi, mm, khugepaged_scan.address); for_each_vma(vmi, vma) { - unsigned long hstart, hend; + unsigned long hstart, hend, window; + unsigned long orders; cond_resched(); if (unlikely(collapse_test_exit_or_disable(mm))) { cc->progress++; break; } - if (!collapse_possible(vma, vma->vm_flags, TVA_KHUGEPAGED)) { + orders = collapse_possible_orders(vma, vma->vm_flags, + TVA_KHUGEPAGED); + if (!orders) { cc->progress++; continue; } - hstart = ALIGN(vma->vm_start, HPAGE_PMD_SIZE); - hend = ALIGN_DOWN(vma->vm_end, HPAGE_PMD_SIZE); + + /* + * Coverage is rooted at windows of the largest order the VMA + * allows: below the PMD order that reaches VMAs a whole table + * would not fit in, and parts of a VMA that a whole table would + * leave out. + */ + window = PAGE_SIZE << __fls(orders); + hstart = ALIGN(vma->vm_start, window); + hend = ALIGN_DOWN(vma->vm_end, window); if (khugepaged_scan.address > hend) { cc->progress++; continue; } if (khugepaged_scan.address < hstart) khugepaged_scan.address = hstart; - VM_BUG_ON(khugepaged_scan.address & ~HPAGE_PMD_MASK); while (khugepaged_scan.address < hend) { + unsigned long pmd_addr, range_end; bool lock_dropped = false; + /* One table's worth at most, and never past the VMA */ + pmd_addr = khugepaged_scan.address & HPAGE_PMD_MASK; + range_end = min(hend, pmd_addr + HPAGE_PMD_SIZE); + cond_resched(); if (unlikely(collapse_test_exit_or_disable(mm))) goto breakouterloop; - VM_WARN_ON_ONCE(khugepaged_scan.address < hstart || - khugepaged_scan.address + HPAGE_PMD_SIZE > - hend); + VM_WARN_ON_ONCE(khugepaged_scan.address < hstart); *result = collapse_single_pmd(khugepaged_scan.address, - khugepaged_scan.address + - HPAGE_PMD_SIZE, - vma, &lock_dropped, cc); + range_end, vma, + &lock_dropped, cc); /* move to next address */ - khugepaged_scan.address += HPAGE_PMD_SIZE; + khugepaged_scan.address = range_end; if (lock_dropped) /* * We released mmap_lock so break loop. Note -- 2.54.0
