For maintainers, Patch 6 requires first 5 patches to work correctly, so it
might be better to get ack from arch maintainers for first 5 patches and
merge the series via mm tree. The patch disabling lazyfree THP discard at
PMD level has been picked up [2] and would take some time to test, so this
patchset probably can be considered after the merge window closes. I am
sending it out for early feedback.
Motivation
===
A bug was found in the PMD-level discard of lazyfree THPs [1]. The
proposed fix keeps the PMD invalidated with pmdp_invalidate() instead of
clearing it, but that does not work on every architecture:
1. riscv, LoongArch and s390 let GUP-fast follow an invalidated PMD;
2. powerpc/book3s64 page table check counts an invalidated PMD as mapped;
3. sparc64's thp_pte_count becomes unbalanced around pmdp_invalidate().
So the PMD-level discard was disabled instead [2]. This series fixes the
arch code and re-enables it.
The discard decides from the folio refcount, so GUP-fast must not take a
reference through an invalidated PMD, and arch accounting must stay
balanced when an invalidated PMD is set again or cleared.
Details
===
Patches 1-5 fix the arch code, patch 6 from Kyle keeps the PMD invalidated
instead of cleared until the discard is committed, and patch 7 enables the
PMD-level discard again:
1. powerpc/book3s64: page table check counts an invalidated PMD as a live
mapping, so setting it again or clearing it decrements the map count a
second time and hits BUG_ON(). This is already reachable via MADV_COLD
and mprotect() with CONFIG_PAGE_TABLE_CHECK.
2. riscv: an invalidated PMD still passes pmd_present() and the generic
pmd_access_permitted(), so GUP-fast follows it. Add a
pmd_access_permitted() that requires _PAGE_PRESENT.
3. LoongArch: same problem and fix as riscv.
4. s390: _SEGMENT_ENTRY_INVALID also marks an old PMD, so GUP-fast follows
an invalidated PMD. Always clear _SEGMENT_ENTRY_READ in
pmdp_invalidate(), which makes it pmd_protnone(), so GUP-fast rejects
it.
5. sparc64: pmdp_invalidate() decrements thp_pte_count assuming a THP
split follows, so setting the PMD again loses a count. Count only
valid huge PMDs in __set_pmd_acct() instead, which keeps invalidate,
restore, clear and split balanced.
6. mm/huge_memory: use pmdp_invalidate() in the discard, so munmap() and
mremap() wait on the PMD lock instead of skipping a none PMD.
7. mm/rmap: discard lazyfree THPs at PMD level again.
Note
===
x86, arm64 and powerpc already make GUP-fast reject an invalidated PMD,
and sparc64 has no GUP-fast. 32-bit MIPS would not reject it, but it
cannot have THP: _PAGE_HUGE makes PFN_PTE_SHIFT exceed PAGE_SHIFT, so the
BUILD_BUG_ON() in arch_mm_preinit() fires, which is why Ingenic dropped
huge pages in commit 69b686ddd579 ("MIPS: ingenic: Remove
CPU_SUPPORTS_HUGEPAGES").
Link: https://lore.kernel.org/all/[email protected]/ [1]
Link:
https://lore.kernel.org/all/[email protected]/
[2]
Signed-off-by: Zi Yan <[email protected]>
---
Kyle Zeng (1):
mm/huge_memory: avoid transient none PMDs during lazyfree reclaim
Zi Yan (6):
powerpc/book3s64/mm: don't count invalidated PMDs in page table check
riscv: mm: don't let GUP-fast follow invalidated THP PMDs
LoongArch: mm: don't let GUP-fast follow invalidated THP PMDs
s390/mm: always clear _SEGMENT_ENTRY_READ in pmdp_invalidate()
sparc64: mm: fix thp_pte_count accounting around pmdp_invalidate()
mm/rmap: re-enable discarding lazyfree THPs at PMD level
arch/loongarch/include/asm/pgtable.h | 6 ++++++
arch/powerpc/include/asm/book3s/64/pgtable.h | 3 ++-
arch/riscv/include/asm/pgtable.h | 6 ++++++
arch/s390/include/asm/pgtable.h | 8 ++++++--
arch/sparc/mm/tlb.c | 26 ++++++++++++--------------
mm/huge_memory.c | 5 +++--
mm/rmap.c | 11 +++++++++++
7 files changed, 46 insertions(+), 19 deletions(-)
---
base-commit: 71a9b024444eba184335b67867897714e4da143c
change-id: 20261007-anon_vma_uaf_fix-e7accd1d2ee1
Best regards,
--
Yan, Zi
