On Fri, 2026-11-13 at 21:43 +0100, David Hildenbrand (Arm) wrote: > On 11/13/26 21:34, Thomas Hellström wrote: > > Hi, David > > > > On Fri, 2026-11-13 at 19:27 +0100, David Hildenbrand (Arm) wrote: > > > On 11/13/26 17:25, Matthew Auld wrote: > > > > When automatic NUMA balancing marks a PTE or PMD as protnone > > > > (PAGE_NONE), hmm_pte_need_fault() requests a fault to restore > > > > access > > > > permissions, including _PAGE_RW when write access is needed. > > > > > > > > Calling handle_mm_fault() from hmm_do_fault() currently > > > > triggers > > > > NUMA > > > > balancing in do_numa_page() and do_huge_pmd_numa_page(). When > > > > hmm_range_fault() runs from a device driver or kworker context > > > > (such as > > > > a GPU SVM or userptr rebind worker), folios get migrated toward > > > > the > > > > CPU > > > > node executing the remote fault. > > > > > > > > Also, each folio migration invokes try_to_migrate_one() -> > > > > mmu_notifier_invalidate_range_start(), which invalidates the > > > > caller's mmu_interval_notifier while holding mmap_lock. That > > > > causes > > > > hmm_range_fault() to abort with -EBUSY after every 2 MiB PMD > > > > and > > > > exhaust > > > > its retry timeout on large buffers, which has been observed > > > > with > > > > large > > > > imported Xe userptr buffers. > > > > > > > > Add FAULT_FLAG_NO_NUMA and set it in hmm_do_fault() so protnone > > > > entries > > > > restore PTE/PMD permissions in-place without performing NUMA > > > > balancing, > > > > matching GUP's default !FOLL_HONOR_NUMA_FAULT behaviour. > > > > > > Ehm, that's not how it works. > > > > > > GUP will not trigger a "special" fault and will try not > > > triggering a > > > fault and > > > not restore permissions. > > > > > > It will try continuing, leaving the protnone pte as it is. > > > > > > See gup_can_follow_protnone(). > > > > > > For read-only access that should always work. > > > > > > For write-access we always trigger an ordinary fault and do NUMA > > > migration etc. > > > > > > > > > So FAULT_FLAG_NO_NUMA is doing something completely different and > > > I > > > don't agree > > > with it. > > > > > > > There are two major problems with the current behaviour. > > > > 1) The work item executing hmm_range_fault() might have no affinity > > at > > all to the GPU about to access the memory. > > > > 2) If migration happens, then hmm_range_fault() will stop and retry > > leading to excessive execution time. > > > > While perhaps Matt's commit message doesn't correctly describe the > > GUP > > behaviour, do you have any alternative suggestion than to avoid > > migration in this case? I think it's a reasonable idea given the > > problems with the current code? > > Adding fault reasons that hack around NUMA hinting such that a actual > CPU > accessing it later will not trigger ... NUMA hinting? > > I might be missing the "reasonable" in here :P
Wouldn't the net effect be that content stays put instead of being migrated? In both cases a later CPU access will not trigger NUMA hinting except that the proposed change doesn't send content to an irrelevant node? > > I recall that John (cc) kept saying "GPUs and NUMA balancing is > inherently > incompatible, you have to disable NUMA balancing". Perhaps we could force hmm_range_fault() to run on the node closest to the GPU. Then at least the migration might be beneficial. > > You could likely implement something similar to GUP, where you would > allow for > bypassing protnone for reading. We could ofc look at that. Might help in some cases. Thanks, Thomas > > That wouldn't give you access to the writing side. There might be > ways to sort > out some of the writing (similar to FOLL_FORCE handling in GUP), but > I'd rather > not go down that path. KVM also doesn't seem to need that.
