On Thu, Feb 26, 2026 at 02:27:24PM +1100, Alistair Popple wrote:
> On 2026-02-25 at 02:17 +1100, Gregory Price <[email protected]> wrote...
> >
> > DEVICE_COHERENT is the odd-man out among ZONE_DEVICE modes. The others
> > use softleaf entries and don't allow direct mappings.
>
> I think you have this around the wrong way - DEVICE_PRIVATE is the odd one
> out as
> it is the one ZONE_DEVICE page type that uses softleaf entries and doesn't
> allow direct mappings. Every other type of ZONE_DEVICE page allows for direct
> mappings.
>
Sorry, you are correct. I have trouble keeping the ZONE_DEVICE modes
straight sometimes, and all the hook sites have different reasons for why
all the different ZONE_DEVICE modes and it mucks with my head :[
Device Coherent is the one that doesn't allow pinning, but still comes
with all the baggage of not being on the lru.
Spoke a bit too bluntly here, apologies.
> Don't you still have to add code to hook every operation you care about for
> your
> private managed nodes?
>
... snip ... below
> > I don't think that's needed if we just recognize ZONE is the wrong
> > abstraction to be operating on.
> >
... snip ... below
> > If your service only allocates movable pages - your ZONE_NORMAL is
> > effectively ZONE_MOVABLE.
>
> This is interesting - it sounds like the conclusion of this is ZONE_* is just
> a
> bad abstraction and should be replaced with something else maybe some like
> this?
>
Yeah i'm not totally married to this being a node, but it makes far more
sense to me than a zone.
ZONE_DEVICE sorta kinda really *wants* to be its own node, but it seems
that "what constitutes a node" was largely informed by ACPI Proximity
Domains. Nothing in the rules say that has to remain the case.
To answer your question above - yea you still need to add code to hook
the operations - but this is essentially already true of ZONE_DEVICE
(except you have to contend with other weird ZONE_DEVICE situations).
Some of the hooks here are an experimentation in what's possible, not
what I think is reasonable (mempolicy is a good example - i don't think
userland should really be doing this anyway... but neat, it works :P)
> And FWIW I'm not tied to the ZONE_DEVICE as being a good abstraction, it's
> just
> what we seem to have today for determing page types. It almost sounds like
> what
> we want is just a bunch of hooks that can be associated with a range of pages,
> and then you just get rid of ZONE_DEVICE and instead install hooks appropriate
> for each page a driver manages. I have to think more about that though, this
> is just what popped into my head when you start saying ZONE_MOVABLE could also
> disappear :-)
Yup! Basically ZONE_MOVABLE and CMA and ZONE_DEVICE/COHERENT all try to
do similar things for different reasons.
Zones manage to somehow be both too-broad AND too-narrow.
In my head, we should just be able to just plop these things "into the
buddy" and provide hooks that say what's allow "for those pages".
That sounds like Non-Uniform Memory Access *cough* :P
Heck, I was even playing with adding these nodes *back into* the
fallback lists for some situations.
NP_OPS_DIRECT / NP_OPS_FALLBACK
don't require __GFP_PRIVATE, but give me the hooks I want :V
> > Where there are new injection sites, it's because ZONE_DEVICE opts
> > out of ever touching that code in some other silently implied way.
>
> Yeah, I hate that aspect of ZONE_DEVICE. There are far too many places where
> we
> "prove" you can't have a ZONE_DEVICE page because of ad-hoc "reasons". Usually
> they take the form of it's not on the LRU, or it's not an anonymous page and
> this isn't DAX, etc.
>
It's kinda the opposite of how operating systems do everything else.
Generally we start from a basis of isolation and then poke deliberate
holes, as opposed to try to patch things up after the fact.
> > If NUMA is the interface we want, then NODE_DATA is the right direction
> > regardless of struct page's future or what zone it lives in.
> >
> > There's no reason to keep per-page pgmap w/ device-to-node mappings.
>
> In reality I suspect that's already the case today. I'm not sure we need
> per-page pgmap.
>
Probably, and maybe there's a good argument for stealing 80-90% of the
common surface here, shunting ZONE_DEVICE to use this instead of pgmap
before we go all the way to private nodes.
cough cough maybe i'll have looked into this by LSFMM cough cough
> > On (1): ZONE_DEVICE NUMA UAPI is harder than it looks from the surface
>
> Ok, I will admit I've only been hovering on the surface so need to give this
> some more thought. Everything you've written below makes sense and is
> definitely
> food for thought. Thanks.
>
cheers! thanks for reading :)
~Gregory
