[PATCH v11 26/26] arm64/mm: add speculative page fault

ds, 15% or even more improvement can be got. Signed-off-by: Ganesh Mahendran <opensource.gan...@gmail.com> [handle_speculative_fault() is no more returning the vma pointer] Signed-off-by: Laurent Dufour <lduf...@linux.vnet.ibm.com> --- arch/arm64/mm/fault.c | 12 1 file chan

[PATCH v11 25/26] powerpc/mm: add speculative page fault

for multithreaded process as there is no risk of contention on the mmap_sem otherwise. Signed-off-by: Laurent Dufour --- arch/powerpc/mm/fault.c | 16 1 file changed, 16 insertions(+) diff --git a/arch/powerpc/mm/fault.c b/arch/powerpc/mm/fault.c index ef268d5d9db7..d7b5742ffb2b 100644

[PATCH v11 26/26] arm64/mm: add speculative page fault

be got. Signed-off-by: Ganesh Mahendran [handle_speculative_fault() is no more returning the vma pointer] Signed-off-by: Laurent Dufour --- arch/arm64/mm/fault.c | 12 1 file changed, 12 insertions(+) diff --git a/arch/arm64/mm/fault.c b/arch/arm64/mm/fault.c index 91c53a7d2575

[PATCH v11 19/26] mm: provide speculative fault infrastructure

d a check against mm->mm_users in handle_speculative_fault() defined in mm.h] Signed-off-by: Laurent Dufour <lduf...@linux.vnet.ibm.com> --- include/linux/hugetlb_inline.h | 2 +- include/linux/mm.h | 30 include/linux/pagemap.h| 4 +- mm/internal.

[PATCH v11 19/26] mm: provide speculative fault infrastructure

lative_fault() defined in mm.h] Signed-off-by: Laurent Dufour --- include/linux/hugetlb_inline.h | 2 +- include/linux/mm.h | 30 include/linux/pagemap.h| 4 +- mm/internal.h | 16 +- mm/memory.c| 340

[PATCH v11 17/26] mm: introduce __page_add_new_anon_rmap()

still valid as explained above. Signed-off-by: Laurent Dufour <lduf...@linux.vnet.ibm.com> --- include/linux/rmap.h | 12 ++-- mm/memory.c | 8 mm/rmap.c| 5 ++--- 3 files changed, 16 insertions(+), 9 deletions(-) diff --git a/include/linux/rmap.h b/include/

[PATCH v11 00/26] Speculative page faults

-6-Another-go-at-speculative-page-faults-tt965642.html#none [2] https://patchwork.kernel.org/patch/687/ Laurent Dufour (20): mm: introduce CONFIG_SPECULATIVE_PAGE_FAULT x86/mm: define ARCH_SUPPORTS_SPECULATIVE_PAGE_FAULT powerpc/mm: set ARCH_SUPPORTS_SPECULATIVE_PAGE_FAULT mm: introduce

[PATCH v11 01/26] mm: introduce CONFIG_SPECULATIVE_PAGE_FAULT

is not set. Suggested-by: Thomas Gleixner <t...@linutronix.de> Suggested-by: David Rientjes <rient...@google.com> Signed-off-by: Laurent Dufour <lduf...@linux.vnet.ibm.com> --- mm/Kconfig | 22 ++ 1 file changed, 22 insertions(+) diff --git a/mm/Kconfig

[PATCH v11 17/26] mm: introduce __page_add_new_anon_rmap()

still valid as explained above. Signed-off-by: Laurent Dufour --- include/linux/rmap.h | 12 ++-- mm/memory.c | 8 mm/rmap.c| 5 ++--- 3 files changed, 16 insertions(+), 9 deletions(-) diff --git a/include/linux/rmap.h b/include/linux/rmap.h index 988d176472df..

[PATCH v11 00/26] Speculative page faults

-6-Another-go-at-speculative-page-faults-tt965642.html#none [2] https://patchwork.kernel.org/patch/687/ Laurent Dufour (20): mm: introduce CONFIG_SPECULATIVE_PAGE_FAULT x86/mm: define ARCH_SUPPORTS_SPECULATIVE_PAGE_FAULT powerpc/mm: set ARCH_SUPPORTS_SPECULATIVE_PAGE_FAULT mm: introduce

[PATCH v11 01/26] mm: introduce CONFIG_SPECULATIVE_PAGE_FAULT

is not set. Suggested-by: Thomas Gleixner Suggested-by: David Rientjes Signed-off-by: Laurent Dufour --- mm/Kconfig | 22 ++ 1 file changed, 22 insertions(+) diff --git a/mm/Kconfig b/mm/Kconfig index 1d0888c5b97a..a38796276113 100644 --- a/mm/Kconfig +++ b/mm/Kconfig

[PATCH v11 07/26] mm: make pte_unmap_same compatible with SPF

is then trapped in cow_user_page(). If VM_FAULT_RETRY is returned, it is passed up to the callers to retry the page fault while holding the mmap_sem. Acked-by: David Rientjes <rient...@google.com> Signed-off-by: Laurent Dufour <lduf...@linux.vnet.ibm.com> --- include/linux/mm.h | 4 +++-

[PATCH v11 07/26] mm: make pte_unmap_same compatible with SPF

is then trapped in cow_user_page(). If VM_FAULT_RETRY is returned, it is passed up to the callers to retry the page fault while holding the mmap_sem. Acked-by: David Rientjes Signed-off-by: Laurent Dufour --- include/linux/mm.h | 4 +++- mm/memory.c| 39 --- 2

[PATCH v11 08/26] mm: introduce INIT_VMA()

inline function. Signed-off-by: Laurent Dufour <lduf...@linux.vnet.ibm.com> --- fs/exec.c | 2 +- include/linux/mm.h | 5 + kernel/fork.c | 2 +- mm/mmap.c | 10 +- mm/nommu.c | 2 +- 5 files changed, 13 insertions(+), 8 deletions(-) diff --gi

[PATCH v11 08/26] mm: introduce INIT_VMA()

inline function. Signed-off-by: Laurent Dufour --- fs/exec.c | 2 +- include/linux/mm.h | 5 + kernel/fork.c | 2 +- mm/mmap.c | 10 +- mm/nommu.c | 2 +- 5 files changed, 13 insertions(+), 8 deletions(-) diff --git a/fs/exec.c b/fs/exec.c index

[PATCH v11 04/26] arm64/mm: define ARCH_SUPPORTS_SPECULATIVE_PAGE_FAULT

From: Mahendran Ganesh Set ARCH_SUPPORTS_SPECULATIVE_PAGE_FAULT for arm64. This enables Speculative Page Fault handler. Signed-off-by: Ganesh Mahendran --- arch/arm64/Kconfig | 1 + 1 file changed, 1 insertion(+) diff --git

[PATCH v11 04/26] arm64/mm: define ARCH_SUPPORTS_SPECULATIVE_PAGE_FAULT

From: Mahendran Ganesh Set ARCH_SUPPORTS_SPECULATIVE_PAGE_FAULT for arm64. This enables Speculative Page Fault handler. Signed-off-by: Ganesh Mahendran --- arch/arm64/Kconfig | 1 + 1 file changed, 1 insertion(+) diff --git a/arch/arm64/Kconfig b/arch/arm64/Kconfig index

Re: [PATCH v10 23/25] mm: add speculative page fault vmstats

On 16/05/2018 04:50, Ganesh Mahendran wrote: > 2018-04-17 22:33 GMT+08:00 Laurent Dufour <lduf...@linux.vnet.ibm.com>: >> Add speculative_pgfault vmstat counter to count successful speculative page >> fault handling. >> >> Also fixing a minor typo in include/li

Re: [PATCH v10 23/25] mm: add speculative page fault vmstats

On 16/05/2018 04:50, Ganesh Mahendran wrote: > 2018-04-17 22:33 GMT+08:00 Laurent Dufour : >> Add speculative_pgfault vmstat counter to count successful speculative page >> fault handling. >> >> Also fixing a minor typo in include/linux/vm_event_item.h. >>

Re: [PATCH v10 18/25] mm: provide speculative fault infrastructure

On 15/05/2018 15:09, vinayak menon wrote: > On Tue, Apr 17, 2018 at 8:03 PM, Laurent Dufour > <lduf...@linux.vnet.ibm.com> wrote: >> >> +#ifdef CONFIG_SPECULATIVE_PAGE_FAULT >> + >> +#ifndef __HAVE_ARCH_PTE_SPECIAL >> +/* This is required by vm_normal_pa

Re: [PATCH v10 18/25] mm: provide speculative fault infrastructure

On 15/05/2018 15:09, vinayak menon wrote: > On Tue, Apr 17, 2018 at 8:03 PM, Laurent Dufour > wrote: >> >> +#ifdef CONFIG_SPECULATIVE_PAGE_FAULT >> + >> +#ifndef __HAVE_ARCH_PTE_SPECIAL >> +/* This is required by vm_normal_page() */ >> +#erro

Re: [PATCH v10 09/25] mm: protect VMA modifications using VMA sequence count

On 23/04/2018 09:19, Minchan Kim wrote: > On Tue, Apr 17, 2018 at 04:33:15PM +0200, Laurent Dufour wrote: >> The VMA sequence count has been introduced to allow fast detection of >> VMA modification when running a page fault handler without holding >> the mmap_sem. >

Re: [PATCH v10 09/25] mm: protect VMA modifications using VMA sequence count

On 23/04/2018 09:19, Minchan Kim wrote: > On Tue, Apr 17, 2018 at 04:33:15PM +0200, Laurent Dufour wrote: >> The VMA sequence count has been introduced to allow fast detection of >> VMA modification when running a page fault handler without holding >> the mmap_sem. >

Re: [PATCH v10 06/25] mm: make pte_unmap_same compatible with SPF

On 10/05/2018 18:15, vinayak menon wrote: > On Tue, Apr 17, 2018 at 8:03 PM, Laurent Dufour > <lduf...@linux.vnet.ibm.com> wrote: >> pte_unmap_same() is making the assumption that the page table are still >> around because the mmap_sem is held. >> This is

Re: [PATCH v10 06/25] mm: make pte_unmap_same compatible with SPF

On 10/05/2018 18:15, vinayak menon wrote: > On Tue, Apr 17, 2018 at 8:03 PM, Laurent Dufour > wrote: >> pte_unmap_same() is making the assumption that the page table are still >> around because the mmap_sem is held. >> This is no more the case when running

Re: [PATCH v10 02/25] x86/mm: define ARCH_SUPPORTS_SPECULATIVE_PAGE_FAULT

On 08/05/2018 13:04, Punit Agrawal wrote: > Hi Laurent, > > Laurent Dufour <lduf...@linux.vnet.ibm.com> writes: > >> Set ARCH_SUPPORTS_SPECULATIVE_PAGE_FAULT which turns on the >> Speculative Page Fault handler when building for 64bit. >> >> Cc: Thomas G

Re: [PATCH v10 02/25] x86/mm: define ARCH_SUPPORTS_SPECULATIVE_PAGE_FAULT

On 08/05/2018 13:04, Punit Agrawal wrote: > Hi Laurent, > > Laurent Dufour writes: > >> Set ARCH_SUPPORTS_SPECULATIVE_PAGE_FAULT which turns on the >> Speculative Page Fault handler when building for 64bit. >> >> Cc: Thomas Gleixner >> Signed-off-by: L

Re: [PATCH v10 12/25] mm: cache some VMA fields in the vm_fault structure

On 03/05/2018 17:42, Minchan Kim wrote: > On Thu, May 03, 2018 at 02:25:18PM +0200, Laurent Dufour wrote: >> On 23/04/2018 09:42, Minchan Kim wrote: >>> On Tue, Apr 17, 2018 at 04:33:18PM +0200, Laurent Dufour wrote: >>>> When handling speculative page fault,

Re: [PATCH v10 12/25] mm: cache some VMA fields in the vm_fault structure

On 03/05/2018 17:42, Minchan Kim wrote: > On Thu, May 03, 2018 at 02:25:18PM +0200, Laurent Dufour wrote: >> On 23/04/2018 09:42, Minchan Kim wrote: >>> On Tue, Apr 17, 2018 at 04:33:18PM +0200, Laurent Dufour wrote: >>>> When handling speculative page fault,

Re: [PATCH v10 24/25] x86/mm: add speculative pagefault handling

umping in bit late to the discussion. Hi Punit, Thanks for giving this series a review. I don't have arm64 hardware to play with, but I'll be happy to add arm64 patches to my series and to try to maintain them. > > On Tue, Apr 17, 2018 at 3:33 PM, Laurent Dufour > <lduf...@linux.vnet.ibm.com&g

Re: [PATCH v10 24/25] x86/mm: add speculative pagefault handling

umping in bit late to the discussion. Hi Punit, Thanks for giving this series a review. I don't have arm64 hardware to play with, but I'll be happy to add arm64 patches to my series and to try to maintain them. > > On Tue, Apr 17, 2018 at 3:33 PM, Laurent Dufour > wrote: >> From: Peter Zijlst

Re: [PATCH v10 08/25] mm: VMA sequence count

On 01/05/2018 15:16, Minchan Kim wrote: > On Mon, Apr 30, 2018 at 05:14:27PM +0200, Laurent Dufour wrote: >> >> >> On 23/04/2018 08:42, Minchan Kim wrote: >>> On Tue, Apr 17, 2018 at 04:33:14PM +0200, Laurent Dufour wrote: >>>> From: Peter Zijlstra <

Re: [PATCH v10 08/25] mm: VMA sequence count

On 01/05/2018 15:16, Minchan Kim wrote: > On Mon, Apr 30, 2018 at 05:14:27PM +0200, Laurent Dufour wrote: >> >> >> On 23/04/2018 08:42, Minchan Kim wrote: >>> On Tue, Apr 17, 2018 at 04:33:14PM +0200, Laurent Dufour wrote: >>>> From: Peter Zijlstra >&g

Re: [PATCH v10 12/25] mm: cache some VMA fields in the vm_fault structure

On 23/04/2018 09:42, Minchan Kim wrote: > On Tue, Apr 17, 2018 at 04:33:18PM +0200, Laurent Dufour wrote: >> When handling speculative page fault, the vma->vm_flags and >> vma->vm_page_prot fields are read once the page table lock is released. So >> there is no more

Re: [PATCH v10 12/25] mm: cache some VMA fields in the vm_fault structure

On 23/04/2018 09:42, Minchan Kim wrote: > On Tue, Apr 17, 2018 at 04:33:18PM +0200, Laurent Dufour wrote: >> When handling speculative page fault, the vma->vm_flags and >> vma->vm_page_prot fields are read once the page table lock is released. So >> there is no more

Re: [PATCH v10 00/25] Speculative page faults

On 02/05/2018 16:17, Punit Agrawal wrote: > Hi Laurent, > > One query below - > > Laurent Dufour <lduf...@linux.vnet.ibm.com> writes: > > [...] > >> >> Ebizzy: >> --- >> The test is counting the number of records per second i

Re: [PATCH v10 00/25] Speculative page faults

On 02/05/2018 16:17, Punit Agrawal wrote: > Hi Laurent, > > One query below - > > Laurent Dufour writes: > > [...] > >> >> Ebizzy: >> --- >> The test is counting the number of records per second it can manage, the >> higher

Re: [PATCH 2/2] arm64/mm: add speculative page fault

On 02/05/2018 09:54, Ganesh Mahendran wrote: > This patch enables the speculative page fault on the arm64 > architecture. > > I completed spf porting in 4.9. From the test result, > we can see app launching time improved by about 10% in average. > For the apps which have more than 50 threads, 15%

Re: [PATCH 2/2] arm64/mm: add speculative page fault

On 02/05/2018 09:54, Ganesh Mahendran wrote: > This patch enables the speculative page fault on the arm64 > architecture. > > I completed spf porting in 4.9. From the test result, > we can see app launching time improved by about 10% in average. > For the apps which have more than 50 threads, 15%

Re: [PATCH 1/2] arm64/mm: define ARCH_SUPPORTS_SPECULATIVE_PAGE_FAULT

On 02/05/2018 09:54, Ganesh Mahendran wrote: > Set ARCH_SUPPORTS_SPECULATIVE_PAGE_FAULT for arm64. This > enables Speculative Page Fault handler. > > Signed-off-by: Ganesh Mahendran > --- > This patch is on top of Laurent's v10 spf > --- > arch/arm64/Kconfig | 1 + >

Re: [PATCH 1/2] arm64/mm: define ARCH_SUPPORTS_SPECULATIVE_PAGE_FAULT

On 02/05/2018 09:54, Ganesh Mahendran wrote: > Set ARCH_SUPPORTS_SPECULATIVE_PAGE_FAULT for arm64. This > enables Speculative Page Fault handler. > > Signed-off-by: Ganesh Mahendran > --- > This patch is on top of Laurent's v10 spf > --- > arch/arm64/Kconfig | 1 + > 1 file changed, 1

Re: [PATCH v10 17/25] mm: protect mm_rb tree with a rwlock

Hi Punit, Thanks for reviewing this series. On 30/04/2018 20:47, Punit Agrawal wrote: > Hi Laurent, > > One nitpick below. > > On Tue, Apr 17, 2018 at 3:33 PM, Laurent Dufour > <lduf...@linux.vnet.ibm.com> wrote: >> This change is inspired by the Pe

Re: [PATCH v10 17/25] mm: protect mm_rb tree with a rwlock

Hi Punit, Thanks for reviewing this series. On 30/04/2018 20:47, Punit Agrawal wrote: > Hi Laurent, > > One nitpick below. > > On Tue, Apr 17, 2018 at 3:33 PM, Laurent Dufour > wrote: >> This change is inspired by the Peter's proposal patch [1] which was >>

Re: [PATCH v10 08/25] mm: VMA sequence count

On 23/04/2018 08:42, Minchan Kim wrote: > On Tue, Apr 17, 2018 at 04:33:14PM +0200, Laurent Dufour wrote: >> From: Peter Zijlstra <pet...@infradead.org> >> >> Wrap the VMA modifications (vma_adjust/unmap_page_range) with sequence >> counts such that we ca

Re: [PATCH v10 08/25] mm: VMA sequence count

On 23/04/2018 08:42, Minchan Kim wrote: > On Tue, Apr 17, 2018 at 04:33:14PM +0200, Laurent Dufour wrote: >> From: Peter Zijlstra >> >> Wrap the VMA modifications (vma_adjust/unmap_page_range) with sequence >> counts such that we can easily test if a VMA is

Re: [PATCH v10 06/25] mm: make pte_unmap_same compatible with SPF

On 23/04/2018 08:31, Minchan Kim wrote: > On Tue, Apr 17, 2018 at 04:33:12PM +0200, Laurent Dufour wrote: >> pte_unmap_same() is making the assumption that the page table are still >> around because the mmap_sem is held. >> This is no more the case when running a s

Re: [PATCH v10 06/25] mm: make pte_unmap_same compatible with SPF

On 23/04/2018 08:31, Minchan Kim wrote: > On Tue, Apr 17, 2018 at 04:33:12PM +0200, Laurent Dufour wrote: >> pte_unmap_same() is making the assumption that the page table are still >> around because the mmap_sem is held. >> This is no more the case when running a s

Re: [PATCH v10 01/25] mm: introduce CONFIG_SPECULATIVE_PAGE_FAULT

On 23/04/2018 07:58, Minchan Kim wrote: > Hi Laurent, > > I guess it's good timing to review. Guess LSF/MM goes so might change > a lot since then. :) Anyway, I grap a time to review. Hi, Thanks a lot for reviewing this series. > On Tue, Apr 17, 2018 at 04:33:07PM +0200, Laure

Re: [PATCH v10 01/25] mm: introduce CONFIG_SPECULATIVE_PAGE_FAULT

On 23/04/2018 07:58, Minchan Kim wrote: > Hi Laurent, > > I guess it's good timing to review. Guess LSF/MM goes so might change > a lot since then. :) Anyway, I grap a time to review. Hi, Thanks a lot for reviewing this series. > On Tue, Apr 17, 2018 at 04:33:07PM +0200, Laure

[PATCH v10 00/25] Speculative page faults

abble.com/RFC-PATCH-0-6-Another-go-at-speculative-page-faults-tt965642.html#none [2] https://patchwork.kernel.org/patch/687/ [3] https://lkml.org/lkml/2018/3/21/894 Laurent Dufour (21): mm: introduce CONFIG_SPECULATIVE_PAGE_FAULT x86/mm: define ARCH_SUPPORTS_SPECULATI

[PATCH v10 00/25] Speculative page faults

abble.com/RFC-PATCH-0-6-Another-go-at-speculative-page-faults-tt965642.html#none [2] https://patchwork.kernel.org/patch/687/ [3] https://lkml.org/lkml/2018/3/21/894 Laurent Dufour (21): mm: introduce CONFIG_SPECULATIVE_PAGE_FAULT x86/mm: define ARCH_SUPPORTS_SPECULATI

[PATCH v10 03/25] powerpc/mm: set ARCH_SUPPORTS_SPECULATIVE_PAGE_FAULT

for book3e_hugetlb_preload() called by update_mmu_cache() Cc: Michael Ellerman <m...@ellerman.id.au> Signed-off-by: Laurent Dufour <lduf...@linux.vnet.ibm.com> --- arch/powerpc/Kconfig | 1 + 1 file changed, 1 insertion(+) diff --git a/arch/powerpc/Kconfig b/arch/powerpc/Kconfig index

[PATCH v10 03/25] powerpc/mm: set ARCH_SUPPORTS_SPECULATIVE_PAGE_FAULT

for book3e_hugetlb_preload() called by update_mmu_cache() Cc: Michael Ellerman Signed-off-by: Laurent Dufour --- arch/powerpc/Kconfig | 1 + 1 file changed, 1 insertion(+) diff --git a/arch/powerpc/Kconfig b/arch/powerpc/Kconfig index c32a181a7cbb..21ef887da7a3 100644 --- a/arch/powerpc/Kconfig +++ b/arch

[PATCH v10 01/25] mm: introduce CONFIG_SPECULATIVE_PAGE_FAULT

oogle.com> Signed-off-by: Laurent Dufour <lduf...@linux.vnet.ibm.com> --- mm/Kconfig | 22 ++ 1 file changed, 22 insertions(+) diff --git a/mm/Kconfig b/mm/Kconfig index d5004d82a1d6..5484dca11199 100644 --- a/mm/Kconfig +++ b/mm/Kconfig @@ -752,3 +752,25 @@ co

[PATCH v10 01/25] mm: introduce CONFIG_SPECULATIVE_PAGE_FAULT

This configuration variable will be used to build the code needed to handle speculative page fault. By default it is turned off, and activated depending on architecture support, SMP and MMU. Suggested-by: Thomas Gleixner Suggested-by: David Rientjes Signed-off-by: Laurent Dufour --- mm

[PATCH v10 05/25] mm: introduce pte_spinlock for FAULT_FLAG_SPECULATIVE

When handling page fault without holding the mmap_sem the fetch of the pte lock pointer and the locking will have to be done while ensuring that the VMA is not touched in our back. So move the fetch and locking operations in a dedicated function. Signed-off-by: Laurent Dufour <l

[PATCH v10 04/25] mm: prepare for FAULT_FLAG_SPECULATIVE

ernel] [move pte_map_lock()'s definition upper in the file] [move the define of FAULT_FLAG_SPECULATIVE later in the series] [review error path in do_swap_page(), do_anonymous_page() and wp_page_copy()] Signed-off-by: Laurent Dufour <lduf...@linux.vnet.ibm.com> ---

[PATCH v10 05/25] mm: introduce pte_spinlock for FAULT_FLAG_SPECULATIVE

When handling page fault without holding the mmap_sem the fetch of the pte lock pointer and the locking will have to be done while ensuring that the VMA is not touched in our back. So move the fetch and locking operations in a dedicated function. Signed-off-by: Laurent Dufour --- mm/memory.c

[PATCH v10 04/25] mm: prepare for FAULT_FLAG_SPECULATIVE

] [move the define of FAULT_FLAG_SPECULATIVE later in the series] [review error path in do_swap_page(), do_anonymous_page() and wp_page_copy()] Signed-off-by: Laurent Dufour --- mm/memory.c | 87 - 1 file changed, 58 insertions(+), 29

[PATCH v10 11/25] mm: protect SPF handler against anon_vma changes

sequence counter which is updated in unmap_page_range() before locking the pte, and then in free_pgtables() so when locking the pte the change will be detected. Signed-off-by: Laurent Dufour <lduf...@linux.vnet.ibm.com> --- mm/memory.c | 4 1 file changed, 4 insertions(+) diff --git

[PATCH v10 11/25] mm: protect SPF handler against anon_vma changes

sequence counter which is updated in unmap_page_range() before locking the pte, and then in free_pgtables() so when locking the pte the change will be detected. Signed-off-by: Laurent Dufour --- mm/memory.c | 4 1 file changed, 4 insertions(+) diff --git a/mm/memory.c b/mm/memor

[PATCH v10 09/25] mm: protect VMA modifications using VMA sequence count

to prevent write to be split and intermediate values to be pushed to other CPUs. Signed-off-by: Laurent Dufour <lduf...@linux.vnet.ibm.com> --- fs/proc/task_mmu.c | 5 - fs/userfaultfd.c | 17 + mm/khugepaged.c| 3 +++ mm/madvise.c | 6 +- mm/mempo

[PATCH v10 09/25] mm: protect VMA modifications using VMA sequence count

to prevent write to be split and intermediate values to be pushed to other CPUs. Signed-off-by: Laurent Dufour --- fs/proc/task_mmu.c | 5 - fs/userfaultfd.c | 17 + mm/khugepaged.c| 3 +++ mm/madvise.c | 6 +- mm/mempolicy.c | 51

[PATCH v10 10/25] mm: protect mremap() against SPF hanlder

d by calling vm_raw_write_end() by the callee once the ptes have been moved. Signed-off-by: Laurent Dufour <lduf...@linux.vnet.ibm.com> --- include/linux/mm.h | 24 +++- mm/mmap.c | 53 + mm/mremap.c| 13

[PATCH v10 14/25] mm: introduce __lru_cache_add_active_or_unevictable

pointer. Acked-by: David Rientjes <rient...@google.com> Signed-off-by: Laurent Dufour <lduf...@linux.vnet.ibm.com> --- include/linux/swap.h | 10 -- mm/memory.c | 8 mm/swap.c| 6 +++--- 3 files changed, 15 insertions(+), 9 deletions(-) diff --git a/i

[PATCH v10 10/25] mm: protect mremap() against SPF hanlder

d by calling vm_raw_write_end() by the callee once the ptes have been moved. Signed-off-by: Laurent Dufour --- include/linux/mm.h | 24 +++- mm/mmap.c | 53 + mm/mremap.c| 13 + 3 files changed, 73 inser

[PATCH v10 14/25] mm: introduce __lru_cache_add_active_or_unevictable

pointer. Acked-by: David Rientjes Signed-off-by: Laurent Dufour --- include/linux/swap.h | 10 -- mm/memory.c | 8 mm/swap.c| 6 +++--- 3 files changed, 15 insertions(+), 9 deletions(-) diff --git a/include/linux/swap.h b/include/linux/swap.h index 1985940af479

[PATCH v10 12/25] mm: cache some VMA fields in the vm_fault structure

hanges. This patch also set the fields in hugetlb_no_page() and __collapse_huge_page_swapin even if it is not need for the callee. Signed-off-by: Laurent Dufour <lduf...@linux.vnet.ibm.com> --- include/linux/mm.h | 10 -- mm/huge_memory.c | 6 +++--- mm/hugetlb.c | 2 ++ mm/

[PATCH v10 12/25] mm: cache some VMA fields in the vm_fault structure

hanges. This patch also set the fields in hugetlb_no_page() and __collapse_huge_page_swapin even if it is not need for the callee. Signed-off-by: Laurent Dufour --- include/linux/mm.h | 10 -- mm/huge_memory.c | 6 +++--- mm/hugetlb.c | 2 ++ mm/khugepaged.c| 2 ++ mm/me

[PATCH v10 16/25] mm: introduce __page_add_new_anon_rmap()

still valid as explained above. Signed-off-by: Laurent Dufour <lduf...@linux.vnet.ibm.com> --- include/linux/rmap.h | 12 ++-- mm/memory.c | 8 mm/rmap.c| 5 ++--- 3 files changed, 16 insertions(+), 9 deletions(-) diff --git a/include/linux/rmap.h b/include/

[PATCH v10 16/25] mm: introduce __page_add_new_anon_rmap()

still valid as explained above. Signed-off-by: Laurent Dufour --- include/linux/rmap.h | 12 ++-- mm/memory.c | 8 mm/rmap.c| 5 ++--- 3 files changed, 16 insertions(+), 9 deletions(-) diff --git a/include/linux/rmap.h b/include/linux/rmap.h index 988d176472df..

[PATCH v10 15/25] mm: introduce __vm_normal_page()

value as parameter. Note: The speculative path is turned on for architecture providing support for special PTE flag. So only the first block of vm_normal_page is used during the speculative path. Signed-off-by: Laurent Dufour <lduf...@linux.vnet.ibm.com> --- include/linux/mm.

[PATCH v10 15/25] mm: introduce __vm_normal_page()

value as parameter. Note: The speculative path is turned on for architecture providing support for special PTE flag. So only the first block of vm_normal_page is used during the speculative path. Signed-off-by: Laurent Dufour --- include/linux/mm.h | 18 +++--- mm/memory.c| 25

[PATCH v10 18/25] mm: provide speculative fault infrastructure

d a check against mm->mm_users in handle_speculative_fault() defined in mm.h] Signed-off-by: Laurent Dufour <lduf...@linux.vnet.ibm.com> --- include/linux/hugetlb_inline.h | 2 +- include/linux/mm.h | 30 include/linux/pagemap.h| 4 +- mm/internal.

[PATCH v10 18/25] mm: provide speculative fault infrastructure

lative_fault() defined in mm.h] Signed-off-by: Laurent Dufour --- include/linux/hugetlb_inline.h | 2 +- include/linux/mm.h | 30 include/linux/pagemap.h| 4 +- mm/internal.h | 16 +- mm/memory.c| 340

[PATCH v10 17/25] mm: protect mm_rb tree with a rwlock

ess speculative page fault. [1] https://patchwork.kernel.org/patch/5108281/ Cc: Peter Zijlstra (Intel) <pet...@infradead.org> Cc: Matthew Wilcox <wi...@infradead.org> Signed-off-by: Laurent Dufour <lduf...@linux.vnet.ibm.com> --- include/linux/mm.h | 1 + include/linux/mm_typ

[PATCH v10 17/25] mm: protect mm_rb tree with a rwlock

ess speculative page fault. [1] https://patchwork.kernel.org/patch/5108281/ Cc: Peter Zijlstra (Intel) Cc: Matthew Wilcox Signed-off-by: Laurent Dufour --- include/linux/mm.h | 1 + include/linux/mm_types.h | 4 ++ kernel/fork.c| 3 ++ mm/init-mm.c | 3 ++

[PATCH v10 21/25] perf tools: add support for the SPF perf event

Add support for the new speculative faults event. Acked-by: David Rientjes <rient...@google.com> Signed-off-by: Laurent Dufour <lduf...@linux.vnet.ibm.com> --- tools/include/uapi/linux/perf_event.h | 1 + tools/perf/util/evsel.c | 1 + tools/perf/util/parse-events.

[PATCH v10 21/25] perf tools: add support for the SPF perf event

Add support for the new speculative faults event. Acked-by: David Rientjes Signed-off-by: Laurent Dufour --- tools/include/uapi/linux/perf_event.h | 1 + tools/perf/util/evsel.c | 1 + tools/perf/util/parse-events.c| 4 tools/perf/util/parse-events.l| 1

[PATCH v10 24/25] x86/mm: add speculative pagefault handling

cesses] [Try reuse to the VMA fetch during the speculative path in case of retry] [Call reuse_spf_or_find_vma()] [Handle memory protection key fault] Signed-off-by: Laurent Dufour <lduf...@linux.vnet.ibm.com> --- arch/x86/mm/fault.c | 42 ++ 1 file chan

[PATCH v10 24/25] x86/mm: add speculative pagefault handling

path in case of retry] [Call reuse_spf_or_find_vma()] [Handle memory protection key fault] Signed-off-by: Laurent Dufour --- arch/x86/mm/fault.c | 42 ++ 1 file changed, 38 insertions(+), 4 deletions(-) diff --git a/arch/x86/mm/fault.c b/arch/x86/mm

[PATCH v10 23/25] mm: add speculative page fault vmstats

Add speculative_pgfault vmstat counter to count successful speculative page fault handling. Also fixing a minor typo in include/linux/vm_event_item.h. Signed-off-by: Laurent Dufour <lduf...@linux.vnet.ibm.com> --- include/linux/vm_event_item.h | 3 +++ mm/memory.c | 1

[PATCH v10 23/25] mm: add speculative page fault vmstats

Add speculative_pgfault vmstat counter to count successful speculative page fault handling. Also fixing a minor typo in include/linux/vm_event_item.h. Signed-off-by: Laurent Dufour --- include/linux/vm_event_item.h | 3 +++ mm/memory.c | 1 + mm/vmstat.c | 5

[PATCH v10 25/25] powerpc/mm: add speculative page fault

for multithreaded process as there is no risk of contention on the mmap_sem otherwise. Signed-off-by: Laurent Dufour <lduf...@linux.vnet.ibm.com> --- arch/powerpc/mm/fault.c | 33 +++-- 1 file changed, 31 insertions(+), 2 deletions(-) diff --git a/arch/powerpc/mm/fau

[PATCH v10 25/25] powerpc/mm: add speculative page fault

for multithreaded process as there is no risk of contention on the mmap_sem otherwise. Signed-off-by: Laurent Dufour --- arch/powerpc/mm/fault.c | 33 +++-- 1 file changed, 31 insertions(+), 2 deletions(-) diff --git a/arch/powerpc/mm/fault.c b/arch/powerpc/mm/fault.c index

[PATCH v10 22/25] mm: speculative page fault handler return VMA

() should be made in place of find_vma(), this will handle the check on the spf_vma and if needed call find_vma(). Signed-off-by: Laurent Dufour <lduf...@linux.vnet.ibm.com> --- include/linux/mm.h | 22 +++-- mm/memory.c| 140 --

[PATCH v10 22/25] mm: speculative page fault handler return VMA

() should be made in place of find_vma(), this will handle the check on the spf_vma and if needed call find_vma(). Signed-off-by: Laurent Dufour --- include/linux/mm.h | 22 +++-- mm/memory.c| 140 - 2 files changed, 103 insertions

[PATCH v10 13/25] mm/migrate: Pass vm_fault pointer to migrate_misplaced_page()

igned-off-by: Laurent Dufour <lduf...@linux.vnet.ibm.com> --- include/linux/migrate.h | 4 ++-- mm/memory.c | 2 +- mm/migrate.c| 4 ++-- 3 files changed, 5 insertions(+), 5 deletions(-) diff --git a/include/linux/migrate.h b/include/linux/migrate.h index f2b4abbca55

[PATCH v10 13/25] mm/migrate: Pass vm_fault pointer to migrate_misplaced_page()

migrate_misplaced_page() is only called during the page fault handling so it's better to pass the pointer to the struct vm_fault instead of the vma. This way during the speculative page fault path the saved vma->vm_flags could be used. Acked-by: David Rientjes Signed-off-by: Laurent Duf

[PATCH v10 20/25] perf: add a speculative page fault sw event

Add a new software event to count succeeded speculative page faults. Acked-by: David Rientjes <rient...@google.com> Signed-off-by: Laurent Dufour <lduf...@linux.vnet.ibm.com> --- include/uapi/linux/perf_event.h | 1 + 1 file changed, 1 insertion(+) diff --git a/include/uapi/linux/pe

[PATCH v10 20/25] perf: add a speculative page fault sw event

Add a new software event to count succeeded speculative page faults. Acked-by: David Rientjes Signed-off-by: Laurent Dufour --- include/uapi/linux/perf_event.h | 1 + 1 file changed, 1 insertion(+) diff --git a/include/uapi/linux/perf_event.h b/include/uapi/linux/perf_event.h index

[PATCH v10 19/25] mm: adding speculative page fault failure trace events

This patch a set of new trace events to collect the speculative page fault event failures. Signed-off-by: Laurent Dufour <lduf...@linux.vnet.ibm.com> --- include/trace/events/pagefault.h | 88 mm/memory.c

[PATCH v10 19/25] mm: adding speculative page fault failure trace events

This patch a set of new trace events to collect the speculative page fault event failures. Signed-off-by: Laurent Dufour --- include/trace/events/pagefault.h | 88 mm/memory.c | 62 ++-- 2 files changed, 137

[PATCH v10 07/25] mm: introduce INIT_VMA()

inline function. Signed-off-by: Laurent Dufour <lduf...@linux.vnet.ibm.com> --- fs/exec.c | 2 +- include/linux/mm.h | 5 + kernel/fork.c | 2 +- mm/mmap.c | 10 +- mm/nommu.c | 2 +- 5 files changed, 13 insertions(+), 8 deletions(-) diff --gi

[PATCH v10 07/25] mm: introduce INIT_VMA()

inline function. Signed-off-by: Laurent Dufour --- fs/exec.c | 2 +- include/linux/mm.h | 5 + kernel/fork.c | 2 +- mm/mmap.c | 10 +- mm/nommu.c | 2 +- 5 files changed, 13 insertions(+), 8 deletions(-) diff --git a/fs/exec.c b/fs/exec.c index

[PATCH v10 08/25] mm: VMA sequence count

mmap_region() when entering the error path] [move sequence initialisation INIT_VMA()] Signed-off-by: Laurent Dufour <lduf...@linux.vnet.ibm.com> --- include/linux/mm.h | 44 include/linux/mm_types.h | 3 +++ mm/memory.c | 2 ++ mm/mm

[PATCH v10 08/25] mm: VMA sequence count

quence initialisation INIT_VMA()] Signed-off-by: Laurent Dufour --- include/linux/mm.h | 44 include/linux/mm_types.h | 3 +++ mm/memory.c | 2 ++ mm/mmap.c| 31 +++ 4 files changed, 80 inse

[PATCH v10 06/25] mm: make pte_unmap_same compatible with SPF

is then trapped in cow_user_page(). If VM_FAULT_RETRY is returned, it is passed up to the callers to retry the page fault while holding the mmap_sem. Acked-by: David Rientjes <rient...@google.com> Signed-off-by: Laurent Dufour <lduf...@linux.vnet.ibm.com> --- include/linux/mm.h | 1 + mm/memory.c

[PATCH v10 06/25] mm: make pte_unmap_same compatible with SPF

is then trapped in cow_user_page(). If VM_FAULT_RETRY is returned, it is passed up to the callers to retry the page fault while holding the mmap_sem. Acked-by: David Rientjes Signed-off-by: Laurent Dufour --- include/linux/mm.h | 1 + mm/memory.c| 39 --- 2 files

[PATCH v10 02/25] x86/mm: define ARCH_SUPPORTS_SPECULATIVE_PAGE_FAULT

Set ARCH_SUPPORTS_SPECULATIVE_PAGE_FAULT which turns on the Speculative Page Fault handler when building for 64bit. Cc: Thomas Gleixner <t...@linutronix.de> Signed-off-by: Laurent Dufour <lduf...@linux.vnet.ibm.com> --- arch/x86/Kconfig | 1 + 1 file changed, 1 insertion(+) diff

<    1   2   3   4   5   6   7   8   9   10   >