copy_page_range_mt() is the padata-multithreaded copy used by vma_dup()
to preserve a VM_EXEC_KEEP (MADV_DOEXEC) VMA across exec. A MADV_DOEXEC
VMA is private anon, so is_cow_mapping() is true and the copy write
protects the source PTEs - a permission downgrade. Unlike the normal
fork path copy_page_range(), this helper jumped straight into
padata_do_multithreaded() without the COW prologue/epilogue, so it:
- never entered src_mm->write_protect_seq (left it even), which trips
the DEBUG_VM invariant in folio_needs_cow_for_dma() reached from
folio_try_dup_anon_rmap_pte() during the copy:
kernel BUG at include/linux/mm.h:2054!
copy_present_ptes <- copy_pte_range <- copy_page_range_chunk
<- padata_do_multithreaded <- copy_page_range_mt <- vma_dup
<- exec_mmap <- begin_new_exec <- load_elf_binary
- never called mmu_notifier_invalidate_range_start/end(), a real latent
bug: a secondary MMU (KVM, IOMMU SVA, GPU) registered on the old mm
outlives de_thread() and could keep writable mappings of pages whose
PTEs the exec copy just wrprotected and shared into the new mm.
Mirror copy_page_range(): for a COW mapping, invalidate secondary MMUs
and take the write_protect_seq write side around the padata job. The
exec'ing task is single-threaded by now and is the only user of src_mm,
so there is no concurrent seqcount writer (fork needs mmap_write_lock)
or GUP-fast reader; taking the raw write side while holding only
mmap_read_lock is safe, which is why vma_assert_write_locked() from
copy_page_range() is intentionally not carried over here.
Fixes: 11e4b1dc50e3 ("oracle/mm: use padata for copying page ranges in
vma_dup()")
Feature: oracle/mm: MADV_DOEXEC madvise() flag
https://virtuozzo.atlassian.net/browse/VSTOR-137234
Signed-off-by: Konstantin Khorenko <[email protected]>
---
mm/memory.c | 35 ++++++++++++++++++++++++++++++++++-
1 file changed, 34 insertions(+), 1 deletion(-)
diff --git a/mm/memory.c b/mm/memory.c
index 837194e87664..4d04591f8f40 100644
--- a/mm/memory.c
+++ b/mm/memory.c
@@ -1478,6 +1478,10 @@ static int copy_page_range_chunk(unsigned long addr,
int copy_page_range_mt(struct vm_area_struct *dst_vma, struct vm_area_struct
*src_vma)
{
struct copy_page_range_args args = { dst_vma, src_vma };
+ struct mm_struct *src_mm = src_vma->vm_mm;
+ struct mmu_notifier_range range;
+ bool is_cow;
+ int ret;
struct padata_mt_job job = {
.thread_fn = copy_page_range_chunk,
.fn_arg = &args,
@@ -1490,7 +1494,36 @@ int copy_page_range_mt(struct vm_area_struct *dst_vma,
struct vm_area_struct *sr
BUG_ON(!(src_vma->vm_flags & VM_EXEC_KEEP));
- return padata_do_multithreaded(&job);
+ /*
+ * Mirror copy_page_range(): for a COW mapping the copy write
+ * protects the source PTEs, which is a permission downgrade, so
+ * secondary MMUs must be invalidated, and write_protect_seq must be
+ * entered (odd) so GUP-fast cannot race with the wrprotect
+ * (folio_needs_cow_for_dma() asserts this under CONFIG_DEBUG_VM).
+ *
+ * The exec'ing task is single-threaded by now and the only user of
+ * src_mm, so no concurrent write_protect_seq writer (fork, which
+ * needs mmap_write_lock) or GUP-fast reader can exist; taking the
+ * raw write side while holding only mmap_read_lock is safe here, and
+ * the padata helper threads only read the seqcount.
+ */
+ is_cow = is_cow_mapping(src_vma->vm_flags);
+ if (is_cow) {
+ mmu_notifier_range_init(&range, MMU_NOTIFY_PROTECTION_PAGE,
+ 0, src_mm, src_vma->vm_start,
+ src_vma->vm_end);
+ mmu_notifier_invalidate_range_start(&range);
+ raw_write_seqcount_begin(&src_mm->write_protect_seq);
+ }
+
+ ret = padata_do_multithreaded(&job);
+
+ if (is_cow) {
+ raw_write_seqcount_end(&src_mm->write_protect_seq);
+ mmu_notifier_invalidate_range_end(&range);
+ }
+
+ return ret;
}
#endif /* CONFIG_PADATA */
--
2.47.1
_______________________________________________
Devel mailing list
[email protected]
https://lists.openvz.org/mailman/listinfo/devel
