On 2022-02-15 07:15, David Hildenbrand wrote:
On 11.02.22 17:56, Jason Gunthorpe wrote:
On Fri, Feb 11, 2022 at 05:49:08PM +0100, David Hildenbrand wrote:
On 11.02.22 17:45, Jason Gunthorpe wrote:
On Fri, Feb 11, 2022 at 05:15:25PM +0100, David Hildenbrand wrote:
... I'm pretty sure we cannot FOLL_PIN DEVICE_PRIVATE pages
Currently the only way to get a DEVICE_PRIVATE page out of the page
tables is via hmm_range_fault() and that doesn't manipulate any ref
counts.
Thanks for clarifying Jason! ... and AFAIU, device exclusive entries are
essentially just pointers at ordinary PageAnon() pages. So with DEVICE
COHERENT we'll have the first PageAnon() ZONE_DEVICE pages mapped as
present in the page tables where GUP could FOLL_PIN them.
This is my understanding
Though you probably understand what PageAnon means alot better than I
do.. I wonder if it really makes sense to talk about that together
with ZONE_DEVICE which has alot in common with filesystem originated
pages too.
For me, PageAnon() means that modifications are visible only to the
modifying process. On actual CoW, the underlying page will get replaced
-- in the world of DEVICE_COHERENT that would mean that once you write
to a DEVICE_COHERENT you could suddenly have a !DEVICE_COHERENT page.
PageAnon() pages don't have a mapping, thus they can only be found in
MAP_ANON VMAs or in MAP_SHARED VMAs with MAP_PRIVATE. They can only be
found via a page table, and not looked up via the page cache (excluding
the swap cache).
So if we have PageAnon() pages on ZONE_DEVICE, they generally have the
exact same semantics as !ZONE_DEVICE pages, but the way they "appear" in
the page tables the allocation/freeing path differs -- I guess :)
... and as we want pinning semantics to be different we have to touch GUP.
I'm not sure what AMDs plan is here, is there an expecation that a GPU
driver will somehow stuff these pages into an existing anonymous
memory VMA or do they always come from a driver originated VMA?
My understanding is that a driver can just decide to replace "ordinary"
PageAnon() pages e.g., in a MAP_ANON VMA by these pages. Hopefully AMD
can clarify.
Yes. DEVICE_COHERENT pages are very similar to DEVICE_PRIVATE. They are
in normal anonymous VMAs (e.g. the application called mmap(...,
MAP_ANONYMOUS, ...)). You get DEVICE_PRIVATE/COHERENT pages as a result
of the driver migrating normal anonymous pages into device memory. The
main difference is, that DEVICE_PRIVATE pages aren't host accessible,
while DEVICE_COHERENT pages are host-accessible and can be mapped in the
CPU page table or DMA-mapped by peer devices. That's why GUP needs to
deal with them.
Regards,
Felix
