Convert vma_alloc_anon_folio_pmd() to pass __GFP_ZERO instead of zeroing at the callsite. post_alloc_hook uses the fault address passed through vma_alloc_folio for cache-friendly zeroing.
Note: before this series, replacing folio_zero_user() with __GFP_ZERO was unsafe on cache-aliasing architectures because __GFP_ZERO uses clear_page() without a dcache flush. With this series, it is safe if the caller passes a valid user address (not USER_ADDR_NONE) to vma_alloc_folio() etc., which delivers it to post_alloc_hook() for the dcache flush via folio_zero_user(). It is only unsafe if USER_ADDR_NONE is passed. Note: with __GFP_ZERO, the folio is zeroed before mem_cgroup_charge(). If the charge fails, the zeroing work is wasted. Previously zeroing was done after a successful charge. This is inherent to moving zeroing into the allocator. Charge failures are rare (only at cgroup limits). Use folio_put_zeroed() on charge failure so the zeroed hint propagates to the buddy allocator, avoiding redundant re-zeroing on the next allocation attempt. Signed-off-by: Michael S. Tsirkin <[email protected]> Reviewed-by: Gregory Price <[email protected]> Assisted-by: Claude:claude-opus-4-6 --- mm/huge_memory.c | 14 +++----------- 1 file changed, 3 insertions(+), 11 deletions(-) diff --git a/mm/huge_memory.c b/mm/huge_memory.c index d689e6491ddb..0dec3c717ff2 100644 --- a/mm/huge_memory.c +++ b/mm/huge_memory.c @@ -1333,7 +1333,7 @@ EXPORT_SYMBOL_GPL(thp_get_unmapped_area); static struct folio *vma_alloc_anon_folio_pmd(struct vm_area_struct *vma, unsigned long addr) { - gfp_t gfp = vma_thp_gfp_mask(vma); + gfp_t gfp = vma_thp_gfp_mask(vma) | __GFP_ZERO; const int order = HPAGE_PMD_ORDER; struct folio *folio; @@ -1347,7 +1347,7 @@ static struct folio *vma_alloc_anon_folio_pmd(struct vm_area_struct *vma, VM_BUG_ON_FOLIO(!folio_test_large(folio), folio); if (mem_cgroup_charge(folio, vma->vm_mm, gfp)) { - folio_put(folio); + folio_put_zeroed(folio); count_vm_event(THP_FAULT_FALLBACK); count_vm_event(THP_FAULT_FALLBACK_CHARGE); count_mthp_stat(order, MTHP_STAT_ANON_FAULT_FALLBACK); @@ -1356,17 +1356,9 @@ static struct folio *vma_alloc_anon_folio_pmd(struct vm_area_struct *vma, } folio_throttle_swaprate(folio, gfp); - /* - * When a folio is not zeroed during allocation (__GFP_ZERO not used) - * or user folios require special handling, folio_zero_user() is used to - * make sure that the page corresponding to the faulting address will be - * hot in the cache after zeroing. - */ - if (user_alloc_needs_zeroing()) - folio_zero_user(folio, addr); /* * The memory barrier inside __folio_mark_uptodate makes sure that - * folio_zero_user writes become visible before the set_pmd_at() + * page zeroing becomes visible before the set_pmd_at() * write. */ __folio_mark_uptodate(folio); -- MST
