Re: [PATCH v6 01/10] mm: add zone device coherent type memory support

2022-02-18 Thread Felix Kuehling
Am 2022-02-18 um 14:26 schrieb Jason Gunthorpe: On Fri, Feb 18, 2022 at 02:20:45PM -0500, Felix Kuehling wrote: Am 2022-02-17 um 19:19 schrieb Jason Gunthorpe: On Thu, Feb 17, 2022 at 04:12:20PM -0500, Felix Kuehling wrote: I'm thinking of a more theoretical approach: Instead of auditing all

Re: [PATCH v6 01/10] mm: add zone device coherent type memory support

2022-02-18 Thread Jason Gunthorpe
On Fri, Feb 18, 2022 at 02:20:45PM -0500, Felix Kuehling wrote: > Am 2022-02-17 um 19:19 schrieb Jason Gunthorpe: > > On Thu, Feb 17, 2022 at 04:12:20PM -0500, Felix Kuehling wrote: > > > > > I'm thinking of a more theoretical approach: Instead of auditing all > > > users, > > > I'd ask, what

Re: [PATCH v6 01/10] mm: add zone device coherent type memory support

2022-02-18 Thread Felix Kuehling
Am 2022-02-17 um 19:19 schrieb Jason Gunthorpe: On Thu, Feb 17, 2022 at 04:12:20PM -0500, Felix Kuehling wrote: I'm thinking of a more theoretical approach: Instead of auditing all users, I'd ask, what are the invariants that a vm_normal_page should have. Then check, whether our

Re: [PATCH v6 01/10] mm: add zone device coherent type memory support

2022-02-17 Thread Alistair Popple
Felix Kuehling writes: > Am 2022-02-16 um 07:26 schrieb Jason Gunthorpe: >> The other place that needs careful audit is all the callers using >> vm_normal_page() - they must all be able to accept a ZONE_DEVICE page >> if we don't set pte_devmap. > > How much code are we talking about here? A

Re: [PATCH v6 01/10] mm: add zone device coherent type memory support

2022-02-17 Thread Jason Gunthorpe
On Thu, Feb 17, 2022 at 04:12:20PM -0500, Felix Kuehling wrote: > I'm thinking of a more theoretical approach: Instead of auditing all users, > I'd ask, what are the invariants that a vm_normal_page should have. Then > check, whether our DEVICE_COHERENT pages satisfy them. But maybe the concept >

Re: [PATCH v6 01/10] mm: add zone device coherent type memory support

2022-02-17 Thread Felix Kuehling
Am 2022-02-16 um 07:26 schrieb Jason Gunthorpe: The other place that needs careful audit is all the callers using vm_normal_page() - they must all be able to accept a ZONE_DEVICE page if we don't set pte_devmap. How much code are we talking about here? A quick search finds 26 call-sites in 12

Re: [PATCH v6 01/10] mm: add zone device coherent type memory support

2022-02-16 Thread Alistair Popple
Jason Gunthorpe writes: > On Wed, Feb 16, 2022 at 09:31:03AM +0100, David Hildenbrand wrote: >> On 16.02.22 03:36, Alistair Popple wrote: >> > On Wednesday, 16 February 2022 1:03:57 PM AEDT Jason Gunthorpe wrote: >> >> On Wed, Feb 16, 2022 at 12:23:44PM +1100, Alistair Popple wrote: >> >> >> >>>

Re: [PATCH v6 01/10] mm: add zone device coherent type memory support

2022-02-16 Thread Jason Gunthorpe
On Wed, Feb 16, 2022 at 11:56:51AM -0500, Felix Kuehling wrote: > In the case of DEVICE_COHERENT memory, the pfns correspond to real physical > memory addresses. I don't think they have those PFN_DEV|PFN_MAP bits set. So do DAX pages. The PTE flag does several things. As this would be the first

Re: [PATCH v6 01/10] mm: add zone device coherent type memory support

2022-02-16 Thread Felix Kuehling
Am 2022-02-15 um 21:01 schrieb Jason Gunthorpe: On Tue, Feb 15, 2022 at 05:49:07PM -0500, Felix Kuehling wrote: Userspace does 1) mmap(MAP_PRIVATE) to allocate anon memory 2) something to trigger migration to install a ZONE_DEVICE page 3) munmap() Who decrements the refcout on the

Re: [PATCH v6 01/10] mm: add zone device coherent type memory support

2022-02-16 Thread Jason Gunthorpe
On Wed, Feb 16, 2022 at 09:31:03AM +0100, David Hildenbrand wrote: > On 16.02.22 03:36, Alistair Popple wrote: > > On Wednesday, 16 February 2022 1:03:57 PM AEDT Jason Gunthorpe wrote: > >> On Wed, Feb 16, 2022 at 12:23:44PM +1100, Alistair Popple wrote: > >> > >>> Device private and device

Re: [PATCH v6 01/10] mm: add zone device coherent type memory support

2022-02-16 Thread David Hildenbrand
On 16.02.22 03:36, Alistair Popple wrote: > On Wednesday, 16 February 2022 1:03:57 PM AEDT Jason Gunthorpe wrote: >> On Wed, Feb 16, 2022 at 12:23:44PM +1100, Alistair Popple wrote: >> >>> Device private and device coherent pages are not marked with pte_devmap and >>> they >>> are backed by a

Re: [PATCH v6 01/10] mm: add zone device coherent type memory support

2022-02-15 Thread Alistair Popple
On Wednesday, 16 February 2022 1:03:57 PM AEDT Jason Gunthorpe wrote: > On Wed, Feb 16, 2022 at 12:23:44PM +1100, Alistair Popple wrote: > > > Device private and device coherent pages are not marked with pte_devmap and > > they > > are backed by a struct page. The only way of inserting them is

Re: [PATCH v6 01/10] mm: add zone device coherent type memory support

2022-02-15 Thread Jason Gunthorpe
On Wed, Feb 16, 2022 at 12:23:44PM +1100, Alistair Popple wrote: > Device private and device coherent pages are not marked with pte_devmap and > they > are backed by a struct page. The only way of inserting them is via > migrate_vma. > The refcount is decremented in zap_pte_range() on munmap()

Re: [PATCH v6 01/10] mm: add zone device coherent type memory support

2022-02-15 Thread Jason Gunthorpe
On Tue, Feb 15, 2022 at 05:49:07PM -0500, Felix Kuehling wrote: > > Userspace does > > 1) mmap(MAP_PRIVATE) to allocate anon memory > > 2) something to trigger migration to install a ZONE_DEVICE page > > 3) munmap() > > > > Who decrements the refcout on the munmap? > > > > When a

Re: [PATCH v6 01/10] mm: add zone device coherent type memory support

2022-02-15 Thread Alistair Popple
Jason Gunthorpe writes: > On Tue, Feb 15, 2022 at 04:35:56PM -0500, Felix Kuehling wrote: >> >> On 2022-02-15 14:41, Jason Gunthorpe wrote: >> > On Tue, Feb 15, 2022 at 07:32:09PM +0100, Christoph Hellwig wrote: >> > > On Tue, Feb 15, 2022 at 10:45:24AM -0400, Jason Gunthorpe wrote: >> > > > >

Re: [PATCH v6 01/10] mm: add zone device coherent type memory support

2022-02-15 Thread Felix Kuehling
On 2022-02-15 16:47, Jason Gunthorpe wrote: On Tue, Feb 15, 2022 at 04:35:56PM -0500, Felix Kuehling wrote: On 2022-02-15 14:41, Jason Gunthorpe wrote: On Tue, Feb 15, 2022 at 07:32:09PM +0100, Christoph Hellwig wrote: On Tue, Feb 15, 2022 at 10:45:24AM -0400, Jason Gunthorpe wrote: Do you

Re: [PATCH v6 01/10] mm: add zone device coherent type memory support

2022-02-15 Thread Jason Gunthorpe
On Tue, Feb 15, 2022 at 04:35:56PM -0500, Felix Kuehling wrote: > > On 2022-02-15 14:41, Jason Gunthorpe wrote: > > On Tue, Feb 15, 2022 at 07:32:09PM +0100, Christoph Hellwig wrote: > > > On Tue, Feb 15, 2022 at 10:45:24AM -0400, Jason Gunthorpe wrote: > > > > > Do you know if DEVICE_GENERIC

Re: [PATCH v6 01/10] mm: add zone device coherent type memory support

2022-02-15 Thread Felix Kuehling
On 2022-02-15 14:41, Jason Gunthorpe wrote: On Tue, Feb 15, 2022 at 07:32:09PM +0100, Christoph Hellwig wrote: On Tue, Feb 15, 2022 at 10:45:24AM -0400, Jason Gunthorpe wrote: Do you know if DEVICE_GENERIC pages would end up as PageAnon()? My assumption was that they would be part of a

Re: [PATCH v6 01/10] mm: add zone device coherent type memory support

2022-02-15 Thread Jason Gunthorpe
On Tue, Feb 15, 2022 at 07:32:09PM +0100, Christoph Hellwig wrote: > On Tue, Feb 15, 2022 at 10:45:24AM -0400, Jason Gunthorpe wrote: > > > Do you know if DEVICE_GENERIC pages would end up as PageAnon()? My > > > assumption was that they would be part of a special mapping. > > > > We need to stop

Re: [PATCH v6 01/10] mm: add zone device coherent type memory support

2022-02-15 Thread Felix Kuehling
On 2022-02-15 07:15, David Hildenbrand wrote: On 11.02.22 17:56, Jason Gunthorpe wrote: On Fri, Feb 11, 2022 at 05:49:08PM +0100, David Hildenbrand wrote: On 11.02.22 17:45, Jason Gunthorpe wrote: On Fri, Feb 11, 2022 at 05:15:25PM +0100, David Hildenbrand wrote: ... I'm pretty sure we

Re: [PATCH v6 01/10] mm: add zone device coherent type memory support

2022-02-15 Thread Jason Gunthorpe
On Tue, Feb 15, 2022 at 01:16:43PM +0100, David Hildenbrand wrote: > > fact, the first version of our patches attempted to add migration > > support to DEVICE_GENERIC. But we were convinced to create a new > > ZONE_DEVICE page type for our use case instead. > > Do you know if DEVICE_GENERIC

Re: [PATCH v6 01/10] mm: add zone device coherent type memory support

2022-02-15 Thread David Hildenbrand
On 11.02.22 18:07, Felix Kuehling wrote: > > Am 2022-02-11 um 11:39 schrieb David Hildenbrand: >> On 11.02.22 17:15, David Hildenbrand wrote: >>> On 01.02.22 16:48, Alex Sierra wrote: Device memory that is cache coherent from device and CPU point of view. This is used on platforms that

Re: [PATCH v6 01/10] mm: add zone device coherent type memory support

2022-02-15 Thread David Hildenbrand
On 11.02.22 17:56, Jason Gunthorpe wrote: > On Fri, Feb 11, 2022 at 05:49:08PM +0100, David Hildenbrand wrote: >> On 11.02.22 17:45, Jason Gunthorpe wrote: >>> On Fri, Feb 11, 2022 at 05:15:25PM +0100, David Hildenbrand wrote: >>> ... I'm pretty sure we cannot FOLL_PIN DEVICE_PRIVATE pages

Re: [PATCH v6 01/10] mm: add zone device coherent type memory support

2022-02-13 Thread Alistair Popple
Felix Kuehling writes: > Am 2022-02-11 um 11:15 schrieb David Hildenbrand: >> On 01.02.22 16:48, Alex Sierra wrote: >>> Device memory that is cache coherent from device and CPU point of view. >>> This is used on platforms that have an advanced system bus (like CAPI >>> or CXL). Any page of a

Re: [PATCH v6 01/10] mm: add zone device coherent type memory support

2022-02-11 Thread Felix Kuehling
Am 2022-02-11 um 11:39 schrieb David Hildenbrand: On 11.02.22 17:15, David Hildenbrand wrote: On 01.02.22 16:48, Alex Sierra wrote: Device memory that is cache coherent from device and CPU point of view. This is used on platforms that have an advanced system bus (like CAPI or CXL). Any page

Re: [PATCH v6 01/10] mm: add zone device coherent type memory support

2022-02-11 Thread Felix Kuehling
Am 2022-02-11 um 11:15 schrieb David Hildenbrand: On 01.02.22 16:48, Alex Sierra wrote: Device memory that is cache coherent from device and CPU point of view. This is used on platforms that have an advanced system bus (like CAPI or CXL). Any page of a process can be migrated to such memory.

Re: [PATCH v6 01/10] mm: add zone device coherent type memory support

2022-02-11 Thread Jason Gunthorpe
On Fri, Feb 11, 2022 at 05:49:08PM +0100, David Hildenbrand wrote: > On 11.02.22 17:45, Jason Gunthorpe wrote: > > On Fri, Feb 11, 2022 at 05:15:25PM +0100, David Hildenbrand wrote: > > > >> ... I'm pretty sure we cannot FOLL_PIN DEVICE_PRIVATE pages > > > > Currently the only way to get a

Re: [PATCH v6 01/10] mm: add zone device coherent type memory support

2022-02-11 Thread Sierra Guiza, Alejandro (Alex)
On 2/11/2022 10:39 AM, David Hildenbrand wrote: On 11.02.22 17:15, David Hildenbrand wrote: On 01.02.22 16:48, Alex Sierra wrote: Device memory that is cache coherent from device and CPU point of view. This is used on platforms that have an advanced system bus (like CAPI or CXL). Any page of

Re: [PATCH v6 01/10] mm: add zone device coherent type memory support

2022-02-11 Thread David Hildenbrand
On 11.02.22 17:45, Jason Gunthorpe wrote: > On Fri, Feb 11, 2022 at 05:15:25PM +0100, David Hildenbrand wrote: > >> ... I'm pretty sure we cannot FOLL_PIN DEVICE_PRIVATE pages > > Currently the only way to get a DEVICE_PRIVATE page out of the page > tables is via hmm_range_fault() and that

Re: [PATCH v6 01/10] mm: add zone device coherent type memory support

2022-02-11 Thread Jason Gunthorpe
On Fri, Feb 11, 2022 at 05:15:25PM +0100, David Hildenbrand wrote: > ... I'm pretty sure we cannot FOLL_PIN DEVICE_PRIVATE pages Currently the only way to get a DEVICE_PRIVATE page out of the page tables is via hmm_range_fault() and that doesn't manipulate any ref counts. Jason

Re: [PATCH v6 01/10] mm: add zone device coherent type memory support

2022-02-11 Thread David Hildenbrand
On 11.02.22 17:15, David Hildenbrand wrote: > On 01.02.22 16:48, Alex Sierra wrote: >> Device memory that is cache coherent from device and CPU point of view. >> This is used on platforms that have an advanced system bus (like CAPI >> or CXL). Any page of a process can be migrated to such memory.

Re: [PATCH v6 01/10] mm: add zone device coherent type memory support

2022-02-11 Thread David Hildenbrand
On 01.02.22 16:48, Alex Sierra wrote: > Device memory that is cache coherent from device and CPU point of view. > This is used on platforms that have an advanced system bus (like CAPI > or CXL). Any page of a process can be migrated to such memory. However, > no one should be allowed to pin such

[PATCH v6 01/10] mm: add zone device coherent type memory support

2022-02-01 Thread Alex Sierra
Device memory that is cache coherent from device and CPU point of view. This is used on platforms that have an advanced system bus (like CAPI or CXL). Any page of a process can be migrated to such memory. However, no one should be allowed to pin such memory so that it can always be evicted.