Re: [RFC v5 PATCH 2/2] mm: mmap: zap pages with read mmap_sem in munmap
+static int vm_munmap_zap_rlock(unsigned long start, size_t len)
+{
+ int ret;
+ struct mm_struct *mm = current->mm;
+ LIST_HEAD(uf);
+
+ ret = do_munmap_zap_rlock(mm, start, len, );
+ userfaultfd_unmap_complete(mm, );
+ return ret;
+}
+
int vm_munmap(unsigned long start, size_t len)
{
int ret;
A stupid question, since the overhead of vm_munmap_zap_rlock() compared to
vm_munmap() is not significant, why not putting that in vm_munmap() instead of
introducing a new vm_munmap_zap_rlock() ?
Since vm_munmap() is called in other paths too, i.e. drm driver, kvm, etc. I'm
not quite sure if those paths are safe enough to this optimization. And, it
looks they are not the main sources of the latency, so here I introduced
vm_munmap_zap_rlock() for munmap() only.
For my information, what could be unsafe for these paths ?
I'm just not sure if they are safe enough nor not, because I'm not
knowledgeable enough to kvm and drm drivers. They might be safe, but I
don't know how to prove that.
So, since they might be not the main sources of latency (I haven't seen
any hung report due to them), so it sounds safe to not touch them for now.
If someone reports or we see they are the sources of latency too, and the
optimization is proved safe to them, we can definitely extend this to all
vm_munmap() calls
Thanks,
Yang
@@ -2855,10 +2939,9 @@ int vm_munmap(unsigned long start, size_t len)
SYSCALL_DEFINE2(munmap, unsigned long, addr, size_t, len)
{
profile_munmap(addr);
- return vm_munmap(addr, len);
+ return vm_munmap_zap_rlock(addr, len);
}
-
/*
* Emulation of deprecated remap_file_pages() syscall.
*/
@@ -3146,7 +3229,7 @@ void exit_mmap(struct mm_struct *mm)
tlb_gather_mmu(, mm, 0, -1);
/* update_hiwater_rss(mm) here? but nobody should be looking */
/* Use -1 here to ensure all VMAs in the mm are unmapped */
- unmap_vmas(, vma, 0, -1);
+ unmap_vmas(, vma, 0, -1, false);
free_pgtables(, vma, FIRST_USER_ADDRESS, USER_PGTABLES_CEILING);
tlb_finish_mmu(, 0, -1);
Re: [RFC v5 PATCH 2/2] mm: mmap: zap pages with read mmap_sem in munmap
+static int vm_munmap_zap_rlock(unsigned long start, size_t len)
+{
+ int ret;
+ struct mm_struct *mm = current->mm;
+ LIST_HEAD(uf);
+
+ ret = do_munmap_zap_rlock(mm, start, len, );
+ userfaultfd_unmap_complete(mm, );
+ return ret;
+}
+
int vm_munmap(unsigned long start, size_t len)
{
int ret;
A stupid question, since the overhead of vm_munmap_zap_rlock() compared to
vm_munmap() is not significant, why not putting that in vm_munmap() instead of
introducing a new vm_munmap_zap_rlock() ?
Since vm_munmap() is called in other paths too, i.e. drm driver, kvm, etc. I'm
not quite sure if those paths are safe enough to this optimization. And, it
looks they are not the main sources of the latency, so here I introduced
vm_munmap_zap_rlock() for munmap() only.
For my information, what could be unsafe for these paths ?
I'm just not sure if they are safe enough nor not, because I'm not
knowledgeable enough to kvm and drm drivers. They might be safe, but I
don't know how to prove that.
So, since they might be not the main sources of latency (I haven't seen
any hung report due to them), so it sounds safe to not touch them for now.
If someone reports or we see they are the sources of latency too, and the
optimization is proved safe to them, we can definitely extend this to all
vm_munmap() calls
Thanks,
Yang
@@ -2855,10 +2939,9 @@ int vm_munmap(unsigned long start, size_t len)
SYSCALL_DEFINE2(munmap, unsigned long, addr, size_t, len)
{
profile_munmap(addr);
- return vm_munmap(addr, len);
+ return vm_munmap_zap_rlock(addr, len);
}
-
/*
* Emulation of deprecated remap_file_pages() syscall.
*/
@@ -3146,7 +3229,7 @@ void exit_mmap(struct mm_struct *mm)
tlb_gather_mmu(, mm, 0, -1);
/* update_hiwater_rss(mm) here? but nobody should be looking */
/* Use -1 here to ensure all VMAs in the mm are unmapped */
- unmap_vmas(, vma, 0, -1);
+ unmap_vmas(, vma, 0, -1, false);
free_pgtables(, vma, FIRST_USER_ADDRESS, USER_PGTABLES_CEILING);
tlb_finish_mmu(, 0, -1);
Re: [RFC v5 PATCH 2/2] mm: mmap: zap pages with read mmap_sem in munmap
On 24/07/2018 19:26, Yang Shi wrote:
>
>
> On 7/24/18 10:18 AM, Laurent Dufour wrote:
>> On 19/07/2018 01:21, Yang Shi wrote:
>>> When running some mmap/munmap scalability tests with large memory (i.e.
300GB), the below hung task issue may happen occasionally.
>>> INFO: task ps:14018 blocked for more than 120 seconds.
>>> Tainted: G E 4.9.79-009.ali3000.alios7.x86_64 #1
>>> "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this
>>> message.
>>> ps D 0 14018 1 0x0004
>>> 885582f84000 885e8682f000 880972943000 885ebf499bc0
>>> 8828ee12 c900349bfca8 817154d0 0040
>>> 00ff812f872a 885ebf499bc0 024000d000948300 880972943000
>>> Call Trace:
>>> [] ? __schedule+0x250/0x730
>>> [] schedule+0x36/0x80
>>> [] rwsem_down_read_failed+0xf0/0x150
>>> [] call_rwsem_down_read_failed+0x18/0x30
>>> [] down_read+0x20/0x40
>>> [] proc_pid_cmdline_read+0xd9/0x4e0
>>> [] ? do_filp_open+0xa5/0x100
>>> [] __vfs_read+0x37/0x150
>>> [] ? security_file_permission+0x9b/0xc0
>>> [] vfs_read+0x96/0x130
>>> [] SyS_read+0x55/0xc0
>>> [] entry_SYSCALL_64_fastpath+0x1a/0xc5
>>>
>>> It is because munmap holds mmap_sem exclusively from very beginning to
>>> all the way down to the end, and doesn't release it in the middle. When
>>> unmapping large mapping, it may take long time (take ~18 seconds to
>>> unmap 320GB mapping with every single page mapped on an idle machine).
>>>
>>> Zapping pages is the most time consuming part, according to the
>>> suggestion from Michal Hocko [1], zapping pages can be done with holding
>>> read mmap_sem, like what MADV_DONTNEED does. Then re-acquire write
>>> mmap_sem to cleanup vmas.
>>>
>>> But, some part may need write mmap_sem, for example, vma splitting. So,
>>> the design is as follows:
>>> acquire write mmap_sem
>>> lookup vmas (find and split vmas)
>>> detach vmas
>>> deal with special mappings
>>> downgrade_write
>>>
>>> zap pages
>>> free page tables
>>> release mmap_sem
>>>
>>> The vm events with read mmap_sem may come in during page zapping, but
>>> since vmas have been detached before, they, i.e. page fault, gup, etc,
>>> will not be able to find valid vma, then just return SIGSEGV or -EFAULT
>>> as expected.
>>>
>>> If the vma has VM_LOCKED | VM_HUGETLB | VM_PFNMAP or uprobe, they are
>>> considered as special mappings. They will be dealt with before zapping
>>> pages with write mmap_sem held. Basically, just update vm_flags.
>>>
>>> And, since they are also manipulated by unmap_single_vma() which is
>>> called by unmap_vma() with read mmap_sem held in this case, to
>>> prevent from updating vm_flags in read critical section, a new
>>> parameter, called "skip_flags" is added to unmap_region(), unmap_vmas()
>>> and unmap_single_vma(). If it is true, then just skip unmap those
>>> special mappings. Currently, the only place which pass true to this
>>> parameter is us.
>>>
>>> With this approach we don't have to re-acquire mmap_sem again to clean
>>> up vmas to avoid race window which might get the address space changed.
>>>
>>> And, since the lock acquire/release cost is managed to the minimum and
>>> almost as same as before, the optimization could be extended to any size
>>> of mapping without incuring significan penalty to small mappings.
>> ^ ^
>> incurring significant
>
> Thanks for catching the typo.
>
>>> For the time being, just do this in munmap syscall path. Other
>>> vm_munmap() or do_munmap() call sites (i.e mmap, mremap, etc) remain
>>> intact for stability reason.
>>>
>>> With the patches, exclusive mmap_sem hold time when munmap a 80GB
>>> address space on a machine with 32 cores of E5-2680 @ 2.70GHz dropped to
>>> us level from second.
>>>
>>> munmap_test-15002 [008] 594.380138: funcgraph_entry: |
>>> vm_munmap_zap_rlock() {
>>> munmap_test-15002 [008] 594.380146: funcgraph_entry: !2485684 us |
>>> unmap_region();
>>> munmap_test-15002 [008] 596.865836: funcgraph_exit: !2485692 us | }
>>>
>>> Here the excution time of unmap_region() is used to evaluate the time of
>>> holding read mmap_sem, then the remaining time is used with holding
>>> exclusive lock.
>>>
>>> [1] https://lwn.net/Articles/753269/
>>>
>>> Suggested-by: Michal Hocko
>>> Suggested-by: Kirill A. Shutemov
>>> Cc: Matthew Wilcox
>>> Cc: Laurent Dufour
>>> Cc: Andrew Morton
>>> Signed-off-by: Yang Shi
>>> ---
>>> include/linux/mm.h | 2 +-
>>> mm/memory.c | 35 +--
>>> mm/mmap.c | 99
>>> +-
>>> 3 files changed, 117 insertions(+), 19 deletions(-)
>>>
>>> diff --git a/include/linux/mm.h b/include/linux/mm.h
>>> index a0fbb9f..95a4e97 100644
>>> --- a/include/linux/mm.h
>>> +++ b/include/linux/mm.h
>>> @@ -1321,7
Re: [RFC v5 PATCH 2/2] mm: mmap: zap pages with read mmap_sem in munmap
On 24/07/2018 19:26, Yang Shi wrote:
>
>
> On 7/24/18 10:18 AM, Laurent Dufour wrote:
>> On 19/07/2018 01:21, Yang Shi wrote:
>>> When running some mmap/munmap scalability tests with large memory (i.e.
300GB), the below hung task issue may happen occasionally.
>>> INFO: task ps:14018 blocked for more than 120 seconds.
>>> Tainted: G E 4.9.79-009.ali3000.alios7.x86_64 #1
>>> "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this
>>> message.
>>> ps D 0 14018 1 0x0004
>>> 885582f84000 885e8682f000 880972943000 885ebf499bc0
>>> 8828ee12 c900349bfca8 817154d0 0040
>>> 00ff812f872a 885ebf499bc0 024000d000948300 880972943000
>>> Call Trace:
>>> [] ? __schedule+0x250/0x730
>>> [] schedule+0x36/0x80
>>> [] rwsem_down_read_failed+0xf0/0x150
>>> [] call_rwsem_down_read_failed+0x18/0x30
>>> [] down_read+0x20/0x40
>>> [] proc_pid_cmdline_read+0xd9/0x4e0
>>> [] ? do_filp_open+0xa5/0x100
>>> [] __vfs_read+0x37/0x150
>>> [] ? security_file_permission+0x9b/0xc0
>>> [] vfs_read+0x96/0x130
>>> [] SyS_read+0x55/0xc0
>>> [] entry_SYSCALL_64_fastpath+0x1a/0xc5
>>>
>>> It is because munmap holds mmap_sem exclusively from very beginning to
>>> all the way down to the end, and doesn't release it in the middle. When
>>> unmapping large mapping, it may take long time (take ~18 seconds to
>>> unmap 320GB mapping with every single page mapped on an idle machine).
>>>
>>> Zapping pages is the most time consuming part, according to the
>>> suggestion from Michal Hocko [1], zapping pages can be done with holding
>>> read mmap_sem, like what MADV_DONTNEED does. Then re-acquire write
>>> mmap_sem to cleanup vmas.
>>>
>>> But, some part may need write mmap_sem, for example, vma splitting. So,
>>> the design is as follows:
>>> acquire write mmap_sem
>>> lookup vmas (find and split vmas)
>>> detach vmas
>>> deal with special mappings
>>> downgrade_write
>>>
>>> zap pages
>>> free page tables
>>> release mmap_sem
>>>
>>> The vm events with read mmap_sem may come in during page zapping, but
>>> since vmas have been detached before, they, i.e. page fault, gup, etc,
>>> will not be able to find valid vma, then just return SIGSEGV or -EFAULT
>>> as expected.
>>>
>>> If the vma has VM_LOCKED | VM_HUGETLB | VM_PFNMAP or uprobe, they are
>>> considered as special mappings. They will be dealt with before zapping
>>> pages with write mmap_sem held. Basically, just update vm_flags.
>>>
>>> And, since they are also manipulated by unmap_single_vma() which is
>>> called by unmap_vma() with read mmap_sem held in this case, to
>>> prevent from updating vm_flags in read critical section, a new
>>> parameter, called "skip_flags" is added to unmap_region(), unmap_vmas()
>>> and unmap_single_vma(). If it is true, then just skip unmap those
>>> special mappings. Currently, the only place which pass true to this
>>> parameter is us.
>>>
>>> With this approach we don't have to re-acquire mmap_sem again to clean
>>> up vmas to avoid race window which might get the address space changed.
>>>
>>> And, since the lock acquire/release cost is managed to the minimum and
>>> almost as same as before, the optimization could be extended to any size
>>> of mapping without incuring significan penalty to small mappings.
>> ^ ^
>> incurring significant
>
> Thanks for catching the typo.
>
>>> For the time being, just do this in munmap syscall path. Other
>>> vm_munmap() or do_munmap() call sites (i.e mmap, mremap, etc) remain
>>> intact for stability reason.
>>>
>>> With the patches, exclusive mmap_sem hold time when munmap a 80GB
>>> address space on a machine with 32 cores of E5-2680 @ 2.70GHz dropped to
>>> us level from second.
>>>
>>> munmap_test-15002 [008] 594.380138: funcgraph_entry: |
>>> vm_munmap_zap_rlock() {
>>> munmap_test-15002 [008] 594.380146: funcgraph_entry: !2485684 us |
>>> unmap_region();
>>> munmap_test-15002 [008] 596.865836: funcgraph_exit: !2485692 us | }
>>>
>>> Here the excution time of unmap_region() is used to evaluate the time of
>>> holding read mmap_sem, then the remaining time is used with holding
>>> exclusive lock.
>>>
>>> [1] https://lwn.net/Articles/753269/
>>>
>>> Suggested-by: Michal Hocko
>>> Suggested-by: Kirill A. Shutemov
>>> Cc: Matthew Wilcox
>>> Cc: Laurent Dufour
>>> Cc: Andrew Morton
>>> Signed-off-by: Yang Shi
>>> ---
>>> include/linux/mm.h | 2 +-
>>> mm/memory.c | 35 +--
>>> mm/mmap.c | 99
>>> +-
>>> 3 files changed, 117 insertions(+), 19 deletions(-)
>>>
>>> diff --git a/include/linux/mm.h b/include/linux/mm.h
>>> index a0fbb9f..95a4e97 100644
>>> --- a/include/linux/mm.h
>>> +++ b/include/linux/mm.h
>>> @@ -1321,7
Re: [RFC v5 PATCH 2/2] mm: mmap: zap pages with read mmap_sem in munmap
On 7/24/18 10:18 AM, Laurent Dufour wrote:
On 19/07/2018 01:21, Yang Shi wrote:
When running some mmap/munmap scalability tests with large memory (i.e.
300GB), the below hung task issue may happen occasionally.
INFO: task ps:14018 blocked for more than 120 seconds.
Tainted: GE 4.9.79-009.ali3000.alios7.x86_64 #1
"echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this
message.
ps D0 14018 1 0x0004
885582f84000 885e8682f000 880972943000 885ebf499bc0
8828ee12 c900349bfca8 817154d0 0040
00ff812f872a 885ebf499bc0 024000d000948300 880972943000
Call Trace:
[] ? __schedule+0x250/0x730
[] schedule+0x36/0x80
[] rwsem_down_read_failed+0xf0/0x150
[] call_rwsem_down_read_failed+0x18/0x30
[] down_read+0x20/0x40
[] proc_pid_cmdline_read+0xd9/0x4e0
[] ? do_filp_open+0xa5/0x100
[] __vfs_read+0x37/0x150
[] ? security_file_permission+0x9b/0xc0
[] vfs_read+0x96/0x130
[] SyS_read+0x55/0xc0
[] entry_SYSCALL_64_fastpath+0x1a/0xc5
It is because munmap holds mmap_sem exclusively from very beginning to
all the way down to the end, and doesn't release it in the middle. When
unmapping large mapping, it may take long time (take ~18 seconds to
unmap 320GB mapping with every single page mapped on an idle machine).
Zapping pages is the most time consuming part, according to the
suggestion from Michal Hocko [1], zapping pages can be done with holding
read mmap_sem, like what MADV_DONTNEED does. Then re-acquire write
mmap_sem to cleanup vmas.
But, some part may need write mmap_sem, for example, vma splitting. So,
the design is as follows:
acquire write mmap_sem
lookup vmas (find and split vmas)
detach vmas
deal with special mappings
downgrade_write
zap pages
free page tables
release mmap_sem
The vm events with read mmap_sem may come in during page zapping, but
since vmas have been detached before, they, i.e. page fault, gup, etc,
will not be able to find valid vma, then just return SIGSEGV or -EFAULT
as expected.
If the vma has VM_LOCKED | VM_HUGETLB | VM_PFNMAP or uprobe, they are
considered as special mappings. They will be dealt with before zapping
pages with write mmap_sem held. Basically, just update vm_flags.
And, since they are also manipulated by unmap_single_vma() which is
called by unmap_vma() with read mmap_sem held in this case, to
prevent from updating vm_flags in read critical section, a new
parameter, called "skip_flags" is added to unmap_region(), unmap_vmas()
and unmap_single_vma(). If it is true, then just skip unmap those
special mappings. Currently, the only place which pass true to this
parameter is us.
With this approach we don't have to re-acquire mmap_sem again to clean
up vmas to avoid race window which might get the address space changed.
And, since the lock acquire/release cost is managed to the minimum and
almost as same as before, the optimization could be extended to any size
of mapping without incuring significan penalty to small mappings.
^ ^
incurring significant
Thanks for catching the typo.
For the time being, just do this in munmap syscall path. Other
vm_munmap() or do_munmap() call sites (i.e mmap, mremap, etc) remain
intact for stability reason.
With the patches, exclusive mmap_sem hold time when munmap a 80GB
address space on a machine with 32 cores of E5-2680 @ 2.70GHz dropped to
us level from second.
munmap_test-15002 [008] 594.380138: funcgraph_entry: | vm_munmap_zap_rlock()
{
munmap_test-15002 [008] 594.380146: funcgraph_entry: !2485684 us |
unmap_region();
munmap_test-15002 [008] 596.865836: funcgraph_exit: !2485692 us | }
Here the excution time of unmap_region() is used to evaluate the time of
holding read mmap_sem, then the remaining time is used with holding
exclusive lock.
[1] https://lwn.net/Articles/753269/
Suggested-by: Michal Hocko
Suggested-by: Kirill A. Shutemov
Cc: Matthew Wilcox
Cc: Laurent Dufour
Cc: Andrew Morton
Signed-off-by: Yang Shi
---
include/linux/mm.h | 2 +-
mm/memory.c| 35 +--
mm/mmap.c | 99 +-
3 files changed, 117 insertions(+), 19 deletions(-)
diff --git a/include/linux/mm.h b/include/linux/mm.h
index a0fbb9f..95a4e97 100644
--- a/include/linux/mm.h
+++ b/include/linux/mm.h
@@ -1321,7 +1321,7 @@ void zap_vma_ptes(struct vm_area_struct *vma, unsigned
long address,
void zap_page_range(struct vm_area_struct *vma, unsigned long address,
unsigned long size);
void unmap_vmas(struct mmu_gather *tlb, struct vm_area_struct *start_vma,
- unsigned long start, unsigned long end);
+ unsigned long start, unsigned long end, bool skip_flags);
/**
* mm_walk - callbacks for walk_page_range
diff
Re: [RFC v5 PATCH 2/2] mm: mmap: zap pages with read mmap_sem in munmap
On 7/24/18 10:18 AM, Laurent Dufour wrote:
On 19/07/2018 01:21, Yang Shi wrote:
When running some mmap/munmap scalability tests with large memory (i.e.
300GB), the below hung task issue may happen occasionally.
INFO: task ps:14018 blocked for more than 120 seconds.
Tainted: GE 4.9.79-009.ali3000.alios7.x86_64 #1
"echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this
message.
ps D0 14018 1 0x0004
885582f84000 885e8682f000 880972943000 885ebf499bc0
8828ee12 c900349bfca8 817154d0 0040
00ff812f872a 885ebf499bc0 024000d000948300 880972943000
Call Trace:
[] ? __schedule+0x250/0x730
[] schedule+0x36/0x80
[] rwsem_down_read_failed+0xf0/0x150
[] call_rwsem_down_read_failed+0x18/0x30
[] down_read+0x20/0x40
[] proc_pid_cmdline_read+0xd9/0x4e0
[] ? do_filp_open+0xa5/0x100
[] __vfs_read+0x37/0x150
[] ? security_file_permission+0x9b/0xc0
[] vfs_read+0x96/0x130
[] SyS_read+0x55/0xc0
[] entry_SYSCALL_64_fastpath+0x1a/0xc5
It is because munmap holds mmap_sem exclusively from very beginning to
all the way down to the end, and doesn't release it in the middle. When
unmapping large mapping, it may take long time (take ~18 seconds to
unmap 320GB mapping with every single page mapped on an idle machine).
Zapping pages is the most time consuming part, according to the
suggestion from Michal Hocko [1], zapping pages can be done with holding
read mmap_sem, like what MADV_DONTNEED does. Then re-acquire write
mmap_sem to cleanup vmas.
But, some part may need write mmap_sem, for example, vma splitting. So,
the design is as follows:
acquire write mmap_sem
lookup vmas (find and split vmas)
detach vmas
deal with special mappings
downgrade_write
zap pages
free page tables
release mmap_sem
The vm events with read mmap_sem may come in during page zapping, but
since vmas have been detached before, they, i.e. page fault, gup, etc,
will not be able to find valid vma, then just return SIGSEGV or -EFAULT
as expected.
If the vma has VM_LOCKED | VM_HUGETLB | VM_PFNMAP or uprobe, they are
considered as special mappings. They will be dealt with before zapping
pages with write mmap_sem held. Basically, just update vm_flags.
And, since they are also manipulated by unmap_single_vma() which is
called by unmap_vma() with read mmap_sem held in this case, to
prevent from updating vm_flags in read critical section, a new
parameter, called "skip_flags" is added to unmap_region(), unmap_vmas()
and unmap_single_vma(). If it is true, then just skip unmap those
special mappings. Currently, the only place which pass true to this
parameter is us.
With this approach we don't have to re-acquire mmap_sem again to clean
up vmas to avoid race window which might get the address space changed.
And, since the lock acquire/release cost is managed to the minimum and
almost as same as before, the optimization could be extended to any size
of mapping without incuring significan penalty to small mappings.
^ ^
incurring significant
Thanks for catching the typo.
For the time being, just do this in munmap syscall path. Other
vm_munmap() or do_munmap() call sites (i.e mmap, mremap, etc) remain
intact for stability reason.
With the patches, exclusive mmap_sem hold time when munmap a 80GB
address space on a machine with 32 cores of E5-2680 @ 2.70GHz dropped to
us level from second.
munmap_test-15002 [008] 594.380138: funcgraph_entry: | vm_munmap_zap_rlock()
{
munmap_test-15002 [008] 594.380146: funcgraph_entry: !2485684 us |
unmap_region();
munmap_test-15002 [008] 596.865836: funcgraph_exit: !2485692 us | }
Here the excution time of unmap_region() is used to evaluate the time of
holding read mmap_sem, then the remaining time is used with holding
exclusive lock.
[1] https://lwn.net/Articles/753269/
Suggested-by: Michal Hocko
Suggested-by: Kirill A. Shutemov
Cc: Matthew Wilcox
Cc: Laurent Dufour
Cc: Andrew Morton
Signed-off-by: Yang Shi
---
include/linux/mm.h | 2 +-
mm/memory.c| 35 +--
mm/mmap.c | 99 +-
3 files changed, 117 insertions(+), 19 deletions(-)
diff --git a/include/linux/mm.h b/include/linux/mm.h
index a0fbb9f..95a4e97 100644
--- a/include/linux/mm.h
+++ b/include/linux/mm.h
@@ -1321,7 +1321,7 @@ void zap_vma_ptes(struct vm_area_struct *vma, unsigned
long address,
void zap_page_range(struct vm_area_struct *vma, unsigned long address,
unsigned long size);
void unmap_vmas(struct mmu_gather *tlb, struct vm_area_struct *start_vma,
- unsigned long start, unsigned long end);
+ unsigned long start, unsigned long end, bool skip_flags);
/**
* mm_walk - callbacks for walk_page_range
diff
Re: [RFC v5 PATCH 2/2] mm: mmap: zap pages with read mmap_sem in munmap
On 19/07/2018 01:21, Yang Shi wrote:
> When running some mmap/munmap scalability tests with large memory (i.e.
>> 300GB), the below hung task issue may happen occasionally.
>
> INFO: task ps:14018 blocked for more than 120 seconds.
>Tainted: GE 4.9.79-009.ali3000.alios7.x86_64 #1
> "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this
> message.
> ps D0 14018 1 0x0004
> 885582f84000 885e8682f000 880972943000 885ebf499bc0
> 8828ee12 c900349bfca8 817154d0 0040
> 00ff812f872a 885ebf499bc0 024000d000948300 880972943000
> Call Trace:
> [] ? __schedule+0x250/0x730
> [] schedule+0x36/0x80
> [] rwsem_down_read_failed+0xf0/0x150
> [] call_rwsem_down_read_failed+0x18/0x30
> [] down_read+0x20/0x40
> [] proc_pid_cmdline_read+0xd9/0x4e0
> [] ? do_filp_open+0xa5/0x100
> [] __vfs_read+0x37/0x150
> [] ? security_file_permission+0x9b/0xc0
> [] vfs_read+0x96/0x130
> [] SyS_read+0x55/0xc0
> [] entry_SYSCALL_64_fastpath+0x1a/0xc5
>
> It is because munmap holds mmap_sem exclusively from very beginning to
> all the way down to the end, and doesn't release it in the middle. When
> unmapping large mapping, it may take long time (take ~18 seconds to
> unmap 320GB mapping with every single page mapped on an idle machine).
>
> Zapping pages is the most time consuming part, according to the
> suggestion from Michal Hocko [1], zapping pages can be done with holding
> read mmap_sem, like what MADV_DONTNEED does. Then re-acquire write
> mmap_sem to cleanup vmas.
>
> But, some part may need write mmap_sem, for example, vma splitting. So,
> the design is as follows:
> acquire write mmap_sem
> lookup vmas (find and split vmas)
> detach vmas
> deal with special mappings
> downgrade_write
>
> zap pages
> free page tables
> release mmap_sem
>
> The vm events with read mmap_sem may come in during page zapping, but
> since vmas have been detached before, they, i.e. page fault, gup, etc,
> will not be able to find valid vma, then just return SIGSEGV or -EFAULT
> as expected.
>
> If the vma has VM_LOCKED | VM_HUGETLB | VM_PFNMAP or uprobe, they are
> considered as special mappings. They will be dealt with before zapping
> pages with write mmap_sem held. Basically, just update vm_flags.
>
> And, since they are also manipulated by unmap_single_vma() which is
> called by unmap_vma() with read mmap_sem held in this case, to
> prevent from updating vm_flags in read critical section, a new
> parameter, called "skip_flags" is added to unmap_region(), unmap_vmas()
> and unmap_single_vma(). If it is true, then just skip unmap those
> special mappings. Currently, the only place which pass true to this
> parameter is us.
>
> With this approach we don't have to re-acquire mmap_sem again to clean
> up vmas to avoid race window which might get the address space changed.
>
> And, since the lock acquire/release cost is managed to the minimum and
> almost as same as before, the optimization could be extended to any size
> of mapping without incuring significan penalty to small mappings.
^ ^
incurring significant
>
> For the time being, just do this in munmap syscall path. Other
> vm_munmap() or do_munmap() call sites (i.e mmap, mremap, etc) remain
> intact for stability reason.
>
> With the patches, exclusive mmap_sem hold time when munmap a 80GB
> address space on a machine with 32 cores of E5-2680 @ 2.70GHz dropped to
> us level from second.
>
> munmap_test-15002 [008] 594.380138: funcgraph_entry: |
> vm_munmap_zap_rlock() {
> munmap_test-15002 [008] 594.380146: funcgraph_entry: !2485684 us |
> unmap_region();
> munmap_test-15002 [008] 596.865836: funcgraph_exit: !2485692 us | }
>
> Here the excution time of unmap_region() is used to evaluate the time of
> holding read mmap_sem, then the remaining time is used with holding
> exclusive lock.
>
> [1] https://lwn.net/Articles/753269/
>
> Suggested-by: Michal Hocko
> Suggested-by: Kirill A. Shutemov
> Cc: Matthew Wilcox
> Cc: Laurent Dufour
> Cc: Andrew Morton
> Signed-off-by: Yang Shi
> ---
> include/linux/mm.h | 2 +-
> mm/memory.c| 35 +--
> mm/mmap.c | 99
> +-
> 3 files changed, 117 insertions(+), 19 deletions(-)
>
> diff --git a/include/linux/mm.h b/include/linux/mm.h
> index a0fbb9f..95a4e97 100644
> --- a/include/linux/mm.h
> +++ b/include/linux/mm.h
> @@ -1321,7 +1321,7 @@ void zap_vma_ptes(struct vm_area_struct *vma, unsigned
> long address,
> void zap_page_range(struct vm_area_struct *vma, unsigned long address,
> unsigned long size);
> void unmap_vmas(struct mmu_gather *tlb, struct vm_area_struct *start_vma,
> - unsigned long start, unsigned long end);
> + unsigned
Re: [RFC v5 PATCH 2/2] mm: mmap: zap pages with read mmap_sem in munmap
On 19/07/2018 01:21, Yang Shi wrote:
> When running some mmap/munmap scalability tests with large memory (i.e.
>> 300GB), the below hung task issue may happen occasionally.
>
> INFO: task ps:14018 blocked for more than 120 seconds.
>Tainted: GE 4.9.79-009.ali3000.alios7.x86_64 #1
> "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this
> message.
> ps D0 14018 1 0x0004
> 885582f84000 885e8682f000 880972943000 885ebf499bc0
> 8828ee12 c900349bfca8 817154d0 0040
> 00ff812f872a 885ebf499bc0 024000d000948300 880972943000
> Call Trace:
> [] ? __schedule+0x250/0x730
> [] schedule+0x36/0x80
> [] rwsem_down_read_failed+0xf0/0x150
> [] call_rwsem_down_read_failed+0x18/0x30
> [] down_read+0x20/0x40
> [] proc_pid_cmdline_read+0xd9/0x4e0
> [] ? do_filp_open+0xa5/0x100
> [] __vfs_read+0x37/0x150
> [] ? security_file_permission+0x9b/0xc0
> [] vfs_read+0x96/0x130
> [] SyS_read+0x55/0xc0
> [] entry_SYSCALL_64_fastpath+0x1a/0xc5
>
> It is because munmap holds mmap_sem exclusively from very beginning to
> all the way down to the end, and doesn't release it in the middle. When
> unmapping large mapping, it may take long time (take ~18 seconds to
> unmap 320GB mapping with every single page mapped on an idle machine).
>
> Zapping pages is the most time consuming part, according to the
> suggestion from Michal Hocko [1], zapping pages can be done with holding
> read mmap_sem, like what MADV_DONTNEED does. Then re-acquire write
> mmap_sem to cleanup vmas.
>
> But, some part may need write mmap_sem, for example, vma splitting. So,
> the design is as follows:
> acquire write mmap_sem
> lookup vmas (find and split vmas)
> detach vmas
> deal with special mappings
> downgrade_write
>
> zap pages
> free page tables
> release mmap_sem
>
> The vm events with read mmap_sem may come in during page zapping, but
> since vmas have been detached before, they, i.e. page fault, gup, etc,
> will not be able to find valid vma, then just return SIGSEGV or -EFAULT
> as expected.
>
> If the vma has VM_LOCKED | VM_HUGETLB | VM_PFNMAP or uprobe, they are
> considered as special mappings. They will be dealt with before zapping
> pages with write mmap_sem held. Basically, just update vm_flags.
>
> And, since they are also manipulated by unmap_single_vma() which is
> called by unmap_vma() with read mmap_sem held in this case, to
> prevent from updating vm_flags in read critical section, a new
> parameter, called "skip_flags" is added to unmap_region(), unmap_vmas()
> and unmap_single_vma(). If it is true, then just skip unmap those
> special mappings. Currently, the only place which pass true to this
> parameter is us.
>
> With this approach we don't have to re-acquire mmap_sem again to clean
> up vmas to avoid race window which might get the address space changed.
>
> And, since the lock acquire/release cost is managed to the minimum and
> almost as same as before, the optimization could be extended to any size
> of mapping without incuring significan penalty to small mappings.
^ ^
incurring significant
>
> For the time being, just do this in munmap syscall path. Other
> vm_munmap() or do_munmap() call sites (i.e mmap, mremap, etc) remain
> intact for stability reason.
>
> With the patches, exclusive mmap_sem hold time when munmap a 80GB
> address space on a machine with 32 cores of E5-2680 @ 2.70GHz dropped to
> us level from second.
>
> munmap_test-15002 [008] 594.380138: funcgraph_entry: |
> vm_munmap_zap_rlock() {
> munmap_test-15002 [008] 594.380146: funcgraph_entry: !2485684 us |
> unmap_region();
> munmap_test-15002 [008] 596.865836: funcgraph_exit: !2485692 us | }
>
> Here the excution time of unmap_region() is used to evaluate the time of
> holding read mmap_sem, then the remaining time is used with holding
> exclusive lock.
>
> [1] https://lwn.net/Articles/753269/
>
> Suggested-by: Michal Hocko
> Suggested-by: Kirill A. Shutemov
> Cc: Matthew Wilcox
> Cc: Laurent Dufour
> Cc: Andrew Morton
> Signed-off-by: Yang Shi
> ---
> include/linux/mm.h | 2 +-
> mm/memory.c| 35 +--
> mm/mmap.c | 99
> +-
> 3 files changed, 117 insertions(+), 19 deletions(-)
>
> diff --git a/include/linux/mm.h b/include/linux/mm.h
> index a0fbb9f..95a4e97 100644
> --- a/include/linux/mm.h
> +++ b/include/linux/mm.h
> @@ -1321,7 +1321,7 @@ void zap_vma_ptes(struct vm_area_struct *vma, unsigned
> long address,
> void zap_page_range(struct vm_area_struct *vma, unsigned long address,
> unsigned long size);
> void unmap_vmas(struct mmu_gather *tlb, struct vm_area_struct *start_vma,
> - unsigned long start, unsigned long end);
> + unsigned
Re: [RFC v5 PATCH 2/2] mm: mmap: zap pages with read mmap_sem in munmap
On 7/24/18 2:26 AM, Kirill A. Shutemov wrote:
On Thu, Jul 19, 2018 at 07:21:41AM +0800, Yang Shi wrote:
When running some mmap/munmap scalability tests with large memory (i.e.
300GB), the below hung task issue may happen occasionally.
INFO: task ps:14018 blocked for more than 120 seconds.
Tainted: GE 4.9.79-009.ali3000.alios7.x86_64 #1
"echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this
message.
ps D0 14018 1 0x0004
885582f84000 885e8682f000 880972943000 885ebf499bc0
8828ee12 c900349bfca8 817154d0 0040
00ff812f872a 885ebf499bc0 024000d000948300 880972943000
Call Trace:
[] ? __schedule+0x250/0x730
[] schedule+0x36/0x80
[] rwsem_down_read_failed+0xf0/0x150
[] call_rwsem_down_read_failed+0x18/0x30
[] down_read+0x20/0x40
[] proc_pid_cmdline_read+0xd9/0x4e0
[] ? do_filp_open+0xa5/0x100
[] __vfs_read+0x37/0x150
[] ? security_file_permission+0x9b/0xc0
[] vfs_read+0x96/0x130
[] SyS_read+0x55/0xc0
[] entry_SYSCALL_64_fastpath+0x1a/0xc5
It is because munmap holds mmap_sem exclusively from very beginning to
all the way down to the end, and doesn't release it in the middle. When
unmapping large mapping, it may take long time (take ~18 seconds to
unmap 320GB mapping with every single page mapped on an idle machine).
Zapping pages is the most time consuming part, according to the
suggestion from Michal Hocko [1], zapping pages can be done with holding
read mmap_sem, like what MADV_DONTNEED does. Then re-acquire write
mmap_sem to cleanup vmas.
But, some part may need write mmap_sem, for example, vma splitting. So,
the design is as follows:
acquire write mmap_sem
lookup vmas (find and split vmas)
detach vmas
deal with special mappings
downgrade_write
zap pages
free page tables
release mmap_sem
The vm events with read mmap_sem may come in during page zapping, but
since vmas have been detached before, they, i.e. page fault, gup, etc,
will not be able to find valid vma, then just return SIGSEGV or -EFAULT
as expected.
If the vma has VM_LOCKED | VM_HUGETLB | VM_PFNMAP or uprobe, they are
considered as special mappings. They will be dealt with before zapping
pages with write mmap_sem held. Basically, just update vm_flags.
And, since they are also manipulated by unmap_single_vma() which is
called by unmap_vma() with read mmap_sem held in this case, to
prevent from updating vm_flags in read critical section, a new
parameter, called "skip_flags" is added to unmap_region(), unmap_vmas()
and unmap_single_vma(). If it is true, then just skip unmap those
special mappings. Currently, the only place which pass true to this
parameter is us.
With this approach we don't have to re-acquire mmap_sem again to clean
up vmas to avoid race window which might get the address space changed.
And, since the lock acquire/release cost is managed to the minimum and
almost as same as before, the optimization could be extended to any size
of mapping without incuring significan penalty to small mappings.
For the time being, just do this in munmap syscall path. Other
vm_munmap() or do_munmap() call sites (i.e mmap, mremap, etc) remain
intact for stability reason.
With the patches, exclusive mmap_sem hold time when munmap a 80GB
address space on a machine with 32 cores of E5-2680 @ 2.70GHz dropped to
us level from second.
munmap_test-15002 [008] 594.380138: funcgraph_entry: | vm_munmap_zap_rlock()
{
munmap_test-15002 [008] 594.380146: funcgraph_entry: !2485684 us |
unmap_region();
munmap_test-15002 [008] 596.865836: funcgraph_exit: !2485692 us | }
Here the excution time of unmap_region() is used to evaluate the time of
holding read mmap_sem, then the remaining time is used with holding
exclusive lock.
[1] https://lwn.net/Articles/753269/
Suggested-by: Michal Hocko
Suggested-by: Kirill A. Shutemov
Cc: Matthew Wilcox
Cc: Laurent Dufour
Cc: Andrew Morton
Signed-off-by: Yang Shi
---
include/linux/mm.h | 2 +-
mm/memory.c| 35 +--
mm/mmap.c | 99 +-
3 files changed, 117 insertions(+), 19 deletions(-)
diff --git a/include/linux/mm.h b/include/linux/mm.h
index a0fbb9f..95a4e97 100644
--- a/include/linux/mm.h
+++ b/include/linux/mm.h
@@ -1321,7 +1321,7 @@ void zap_vma_ptes(struct vm_area_struct *vma, unsigned
long address,
void zap_page_range(struct vm_area_struct *vma, unsigned long address,
unsigned long size);
void unmap_vmas(struct mmu_gather *tlb, struct vm_area_struct *start_vma,
- unsigned long start, unsigned long end);
+ unsigned long start, unsigned long end, bool skip_flags);
skip_flags is not specific enough. Which flags? Maybe skip_vm_flags or
smething.
vm_flags, skip_vm_flags sounds more specific.
/**
*
Re: [RFC v5 PATCH 2/2] mm: mmap: zap pages with read mmap_sem in munmap
On 7/24/18 2:26 AM, Kirill A. Shutemov wrote:
On Thu, Jul 19, 2018 at 07:21:41AM +0800, Yang Shi wrote:
When running some mmap/munmap scalability tests with large memory (i.e.
300GB), the below hung task issue may happen occasionally.
INFO: task ps:14018 blocked for more than 120 seconds.
Tainted: GE 4.9.79-009.ali3000.alios7.x86_64 #1
"echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this
message.
ps D0 14018 1 0x0004
885582f84000 885e8682f000 880972943000 885ebf499bc0
8828ee12 c900349bfca8 817154d0 0040
00ff812f872a 885ebf499bc0 024000d000948300 880972943000
Call Trace:
[] ? __schedule+0x250/0x730
[] schedule+0x36/0x80
[] rwsem_down_read_failed+0xf0/0x150
[] call_rwsem_down_read_failed+0x18/0x30
[] down_read+0x20/0x40
[] proc_pid_cmdline_read+0xd9/0x4e0
[] ? do_filp_open+0xa5/0x100
[] __vfs_read+0x37/0x150
[] ? security_file_permission+0x9b/0xc0
[] vfs_read+0x96/0x130
[] SyS_read+0x55/0xc0
[] entry_SYSCALL_64_fastpath+0x1a/0xc5
It is because munmap holds mmap_sem exclusively from very beginning to
all the way down to the end, and doesn't release it in the middle. When
unmapping large mapping, it may take long time (take ~18 seconds to
unmap 320GB mapping with every single page mapped on an idle machine).
Zapping pages is the most time consuming part, according to the
suggestion from Michal Hocko [1], zapping pages can be done with holding
read mmap_sem, like what MADV_DONTNEED does. Then re-acquire write
mmap_sem to cleanup vmas.
But, some part may need write mmap_sem, for example, vma splitting. So,
the design is as follows:
acquire write mmap_sem
lookup vmas (find and split vmas)
detach vmas
deal with special mappings
downgrade_write
zap pages
free page tables
release mmap_sem
The vm events with read mmap_sem may come in during page zapping, but
since vmas have been detached before, they, i.e. page fault, gup, etc,
will not be able to find valid vma, then just return SIGSEGV or -EFAULT
as expected.
If the vma has VM_LOCKED | VM_HUGETLB | VM_PFNMAP or uprobe, they are
considered as special mappings. They will be dealt with before zapping
pages with write mmap_sem held. Basically, just update vm_flags.
And, since they are also manipulated by unmap_single_vma() which is
called by unmap_vma() with read mmap_sem held in this case, to
prevent from updating vm_flags in read critical section, a new
parameter, called "skip_flags" is added to unmap_region(), unmap_vmas()
and unmap_single_vma(). If it is true, then just skip unmap those
special mappings. Currently, the only place which pass true to this
parameter is us.
With this approach we don't have to re-acquire mmap_sem again to clean
up vmas to avoid race window which might get the address space changed.
And, since the lock acquire/release cost is managed to the minimum and
almost as same as before, the optimization could be extended to any size
of mapping without incuring significan penalty to small mappings.
For the time being, just do this in munmap syscall path. Other
vm_munmap() or do_munmap() call sites (i.e mmap, mremap, etc) remain
intact for stability reason.
With the patches, exclusive mmap_sem hold time when munmap a 80GB
address space on a machine with 32 cores of E5-2680 @ 2.70GHz dropped to
us level from second.
munmap_test-15002 [008] 594.380138: funcgraph_entry: | vm_munmap_zap_rlock()
{
munmap_test-15002 [008] 594.380146: funcgraph_entry: !2485684 us |
unmap_region();
munmap_test-15002 [008] 596.865836: funcgraph_exit: !2485692 us | }
Here the excution time of unmap_region() is used to evaluate the time of
holding read mmap_sem, then the remaining time is used with holding
exclusive lock.
[1] https://lwn.net/Articles/753269/
Suggested-by: Michal Hocko
Suggested-by: Kirill A. Shutemov
Cc: Matthew Wilcox
Cc: Laurent Dufour
Cc: Andrew Morton
Signed-off-by: Yang Shi
---
include/linux/mm.h | 2 +-
mm/memory.c| 35 +--
mm/mmap.c | 99 +-
3 files changed, 117 insertions(+), 19 deletions(-)
diff --git a/include/linux/mm.h b/include/linux/mm.h
index a0fbb9f..95a4e97 100644
--- a/include/linux/mm.h
+++ b/include/linux/mm.h
@@ -1321,7 +1321,7 @@ void zap_vma_ptes(struct vm_area_struct *vma, unsigned
long address,
void zap_page_range(struct vm_area_struct *vma, unsigned long address,
unsigned long size);
void unmap_vmas(struct mmu_gather *tlb, struct vm_area_struct *start_vma,
- unsigned long start, unsigned long end);
+ unsigned long start, unsigned long end, bool skip_flags);
skip_flags is not specific enough. Which flags? Maybe skip_vm_flags or
smething.
vm_flags, skip_vm_flags sounds more specific.
/**
*
Re: [RFC v5 PATCH 2/2] mm: mmap: zap pages with read mmap_sem in munmap
On Thu, Jul 19, 2018 at 07:21:41AM +0800, Yang Shi wrote:
> When running some mmap/munmap scalability tests with large memory (i.e.
> > 300GB), the below hung task issue may happen occasionally.
>
> INFO: task ps:14018 blocked for more than 120 seconds.
>Tainted: GE 4.9.79-009.ali3000.alios7.x86_64 #1
> "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this
> message.
> ps D0 14018 1 0x0004
> 885582f84000 885e8682f000 880972943000 885ebf499bc0
> 8828ee12 c900349bfca8 817154d0 0040
> 00ff812f872a 885ebf499bc0 024000d000948300 880972943000
> Call Trace:
> [] ? __schedule+0x250/0x730
> [] schedule+0x36/0x80
> [] rwsem_down_read_failed+0xf0/0x150
> [] call_rwsem_down_read_failed+0x18/0x30
> [] down_read+0x20/0x40
> [] proc_pid_cmdline_read+0xd9/0x4e0
> [] ? do_filp_open+0xa5/0x100
> [] __vfs_read+0x37/0x150
> [] ? security_file_permission+0x9b/0xc0
> [] vfs_read+0x96/0x130
> [] SyS_read+0x55/0xc0
> [] entry_SYSCALL_64_fastpath+0x1a/0xc5
>
> It is because munmap holds mmap_sem exclusively from very beginning to
> all the way down to the end, and doesn't release it in the middle. When
> unmapping large mapping, it may take long time (take ~18 seconds to
> unmap 320GB mapping with every single page mapped on an idle machine).
>
> Zapping pages is the most time consuming part, according to the
> suggestion from Michal Hocko [1], zapping pages can be done with holding
> read mmap_sem, like what MADV_DONTNEED does. Then re-acquire write
> mmap_sem to cleanup vmas.
>
> But, some part may need write mmap_sem, for example, vma splitting. So,
> the design is as follows:
> acquire write mmap_sem
> lookup vmas (find and split vmas)
> detach vmas
> deal with special mappings
> downgrade_write
>
> zap pages
> free page tables
> release mmap_sem
>
> The vm events with read mmap_sem may come in during page zapping, but
> since vmas have been detached before, they, i.e. page fault, gup, etc,
> will not be able to find valid vma, then just return SIGSEGV or -EFAULT
> as expected.
>
> If the vma has VM_LOCKED | VM_HUGETLB | VM_PFNMAP or uprobe, they are
> considered as special mappings. They will be dealt with before zapping
> pages with write mmap_sem held. Basically, just update vm_flags.
>
> And, since they are also manipulated by unmap_single_vma() which is
> called by unmap_vma() with read mmap_sem held in this case, to
> prevent from updating vm_flags in read critical section, a new
> parameter, called "skip_flags" is added to unmap_region(), unmap_vmas()
> and unmap_single_vma(). If it is true, then just skip unmap those
> special mappings. Currently, the only place which pass true to this
> parameter is us.
>
> With this approach we don't have to re-acquire mmap_sem again to clean
> up vmas to avoid race window which might get the address space changed.
>
> And, since the lock acquire/release cost is managed to the minimum and
> almost as same as before, the optimization could be extended to any size
> of mapping without incuring significan penalty to small mappings.
>
> For the time being, just do this in munmap syscall path. Other
> vm_munmap() or do_munmap() call sites (i.e mmap, mremap, etc) remain
> intact for stability reason.
>
> With the patches, exclusive mmap_sem hold time when munmap a 80GB
> address space on a machine with 32 cores of E5-2680 @ 2.70GHz dropped to
> us level from second.
>
> munmap_test-15002 [008] 594.380138: funcgraph_entry: |
> vm_munmap_zap_rlock() {
> munmap_test-15002 [008] 594.380146: funcgraph_entry: !2485684 us |
> unmap_region();
> munmap_test-15002 [008] 596.865836: funcgraph_exit: !2485692 us | }
>
> Here the excution time of unmap_region() is used to evaluate the time of
> holding read mmap_sem, then the remaining time is used with holding
> exclusive lock.
>
> [1] https://lwn.net/Articles/753269/
>
> Suggested-by: Michal Hocko
> Suggested-by: Kirill A. Shutemov
> Cc: Matthew Wilcox
> Cc: Laurent Dufour
> Cc: Andrew Morton
> Signed-off-by: Yang Shi
> ---
> include/linux/mm.h | 2 +-
> mm/memory.c| 35 +--
> mm/mmap.c | 99
> +-
> 3 files changed, 117 insertions(+), 19 deletions(-)
>
> diff --git a/include/linux/mm.h b/include/linux/mm.h
> index a0fbb9f..95a4e97 100644
> --- a/include/linux/mm.h
> +++ b/include/linux/mm.h
> @@ -1321,7 +1321,7 @@ void zap_vma_ptes(struct vm_area_struct *vma, unsigned
> long address,
> void zap_page_range(struct vm_area_struct *vma, unsigned long address,
> unsigned long size);
> void unmap_vmas(struct mmu_gather *tlb, struct vm_area_struct *start_vma,
> - unsigned long start, unsigned long end);
> + unsigned long start, unsigned long end, bool skip_flags);
Re: [RFC v5 PATCH 2/2] mm: mmap: zap pages with read mmap_sem in munmap
On Thu, Jul 19, 2018 at 07:21:41AM +0800, Yang Shi wrote:
> When running some mmap/munmap scalability tests with large memory (i.e.
> > 300GB), the below hung task issue may happen occasionally.
>
> INFO: task ps:14018 blocked for more than 120 seconds.
>Tainted: GE 4.9.79-009.ali3000.alios7.x86_64 #1
> "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this
> message.
> ps D0 14018 1 0x0004
> 885582f84000 885e8682f000 880972943000 885ebf499bc0
> 8828ee12 c900349bfca8 817154d0 0040
> 00ff812f872a 885ebf499bc0 024000d000948300 880972943000
> Call Trace:
> [] ? __schedule+0x250/0x730
> [] schedule+0x36/0x80
> [] rwsem_down_read_failed+0xf0/0x150
> [] call_rwsem_down_read_failed+0x18/0x30
> [] down_read+0x20/0x40
> [] proc_pid_cmdline_read+0xd9/0x4e0
> [] ? do_filp_open+0xa5/0x100
> [] __vfs_read+0x37/0x150
> [] ? security_file_permission+0x9b/0xc0
> [] vfs_read+0x96/0x130
> [] SyS_read+0x55/0xc0
> [] entry_SYSCALL_64_fastpath+0x1a/0xc5
>
> It is because munmap holds mmap_sem exclusively from very beginning to
> all the way down to the end, and doesn't release it in the middle. When
> unmapping large mapping, it may take long time (take ~18 seconds to
> unmap 320GB mapping with every single page mapped on an idle machine).
>
> Zapping pages is the most time consuming part, according to the
> suggestion from Michal Hocko [1], zapping pages can be done with holding
> read mmap_sem, like what MADV_DONTNEED does. Then re-acquire write
> mmap_sem to cleanup vmas.
>
> But, some part may need write mmap_sem, for example, vma splitting. So,
> the design is as follows:
> acquire write mmap_sem
> lookup vmas (find and split vmas)
> detach vmas
> deal with special mappings
> downgrade_write
>
> zap pages
> free page tables
> release mmap_sem
>
> The vm events with read mmap_sem may come in during page zapping, but
> since vmas have been detached before, they, i.e. page fault, gup, etc,
> will not be able to find valid vma, then just return SIGSEGV or -EFAULT
> as expected.
>
> If the vma has VM_LOCKED | VM_HUGETLB | VM_PFNMAP or uprobe, they are
> considered as special mappings. They will be dealt with before zapping
> pages with write mmap_sem held. Basically, just update vm_flags.
>
> And, since they are also manipulated by unmap_single_vma() which is
> called by unmap_vma() with read mmap_sem held in this case, to
> prevent from updating vm_flags in read critical section, a new
> parameter, called "skip_flags" is added to unmap_region(), unmap_vmas()
> and unmap_single_vma(). If it is true, then just skip unmap those
> special mappings. Currently, the only place which pass true to this
> parameter is us.
>
> With this approach we don't have to re-acquire mmap_sem again to clean
> up vmas to avoid race window which might get the address space changed.
>
> And, since the lock acquire/release cost is managed to the minimum and
> almost as same as before, the optimization could be extended to any size
> of mapping without incuring significan penalty to small mappings.
>
> For the time being, just do this in munmap syscall path. Other
> vm_munmap() or do_munmap() call sites (i.e mmap, mremap, etc) remain
> intact for stability reason.
>
> With the patches, exclusive mmap_sem hold time when munmap a 80GB
> address space on a machine with 32 cores of E5-2680 @ 2.70GHz dropped to
> us level from second.
>
> munmap_test-15002 [008] 594.380138: funcgraph_entry: |
> vm_munmap_zap_rlock() {
> munmap_test-15002 [008] 594.380146: funcgraph_entry: !2485684 us |
> unmap_region();
> munmap_test-15002 [008] 596.865836: funcgraph_exit: !2485692 us | }
>
> Here the excution time of unmap_region() is used to evaluate the time of
> holding read mmap_sem, then the remaining time is used with holding
> exclusive lock.
>
> [1] https://lwn.net/Articles/753269/
>
> Suggested-by: Michal Hocko
> Suggested-by: Kirill A. Shutemov
> Cc: Matthew Wilcox
> Cc: Laurent Dufour
> Cc: Andrew Morton
> Signed-off-by: Yang Shi
> ---
> include/linux/mm.h | 2 +-
> mm/memory.c| 35 +--
> mm/mmap.c | 99
> +-
> 3 files changed, 117 insertions(+), 19 deletions(-)
>
> diff --git a/include/linux/mm.h b/include/linux/mm.h
> index a0fbb9f..95a4e97 100644
> --- a/include/linux/mm.h
> +++ b/include/linux/mm.h
> @@ -1321,7 +1321,7 @@ void zap_vma_ptes(struct vm_area_struct *vma, unsigned
> long address,
> void zap_page_range(struct vm_area_struct *vma, unsigned long address,
> unsigned long size);
> void unmap_vmas(struct mmu_gather *tlb, struct vm_area_struct *start_vma,
> - unsigned long start, unsigned long end);
> + unsigned long start, unsigned long end, bool skip_flags);
[RFC v5 PATCH 2/2] mm: mmap: zap pages with read mmap_sem in munmap
When running some mmap/munmap scalability tests with large memory (i.e.
> 300GB), the below hung task issue may happen occasionally.
INFO: task ps:14018 blocked for more than 120 seconds.
Tainted: GE 4.9.79-009.ali3000.alios7.x86_64 #1
"echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this
message.
ps D0 14018 1 0x0004
885582f84000 885e8682f000 880972943000 885ebf499bc0
8828ee12 c900349bfca8 817154d0 0040
00ff812f872a 885ebf499bc0 024000d000948300 880972943000
Call Trace:
[] ? __schedule+0x250/0x730
[] schedule+0x36/0x80
[] rwsem_down_read_failed+0xf0/0x150
[] call_rwsem_down_read_failed+0x18/0x30
[] down_read+0x20/0x40
[] proc_pid_cmdline_read+0xd9/0x4e0
[] ? do_filp_open+0xa5/0x100
[] __vfs_read+0x37/0x150
[] ? security_file_permission+0x9b/0xc0
[] vfs_read+0x96/0x130
[] SyS_read+0x55/0xc0
[] entry_SYSCALL_64_fastpath+0x1a/0xc5
It is because munmap holds mmap_sem exclusively from very beginning to
all the way down to the end, and doesn't release it in the middle. When
unmapping large mapping, it may take long time (take ~18 seconds to
unmap 320GB mapping with every single page mapped on an idle machine).
Zapping pages is the most time consuming part, according to the
suggestion from Michal Hocko [1], zapping pages can be done with holding
read mmap_sem, like what MADV_DONTNEED does. Then re-acquire write
mmap_sem to cleanup vmas.
But, some part may need write mmap_sem, for example, vma splitting. So,
the design is as follows:
acquire write mmap_sem
lookup vmas (find and split vmas)
detach vmas
deal with special mappings
downgrade_write
zap pages
free page tables
release mmap_sem
The vm events with read mmap_sem may come in during page zapping, but
since vmas have been detached before, they, i.e. page fault, gup, etc,
will not be able to find valid vma, then just return SIGSEGV or -EFAULT
as expected.
If the vma has VM_LOCKED | VM_HUGETLB | VM_PFNMAP or uprobe, they are
considered as special mappings. They will be dealt with before zapping
pages with write mmap_sem held. Basically, just update vm_flags.
And, since they are also manipulated by unmap_single_vma() which is
called by unmap_vma() with read mmap_sem held in this case, to
prevent from updating vm_flags in read critical section, a new
parameter, called "skip_flags" is added to unmap_region(), unmap_vmas()
and unmap_single_vma(). If it is true, then just skip unmap those
special mappings. Currently, the only place which pass true to this
parameter is us.
With this approach we don't have to re-acquire mmap_sem again to clean
up vmas to avoid race window which might get the address space changed.
And, since the lock acquire/release cost is managed to the minimum and
almost as same as before, the optimization could be extended to any size
of mapping without incuring significan penalty to small mappings.
For the time being, just do this in munmap syscall path. Other
vm_munmap() or do_munmap() call sites (i.e mmap, mremap, etc) remain
intact for stability reason.
With the patches, exclusive mmap_sem hold time when munmap a 80GB
address space on a machine with 32 cores of E5-2680 @ 2.70GHz dropped to
us level from second.
munmap_test-15002 [008] 594.380138: funcgraph_entry: | vm_munmap_zap_rlock()
{
munmap_test-15002 [008] 594.380146: funcgraph_entry: !2485684 us |
unmap_region();
munmap_test-15002 [008] 596.865836: funcgraph_exit: !2485692 us | }
Here the excution time of unmap_region() is used to evaluate the time of
holding read mmap_sem, then the remaining time is used with holding
exclusive lock.
[1] https://lwn.net/Articles/753269/
Suggested-by: Michal Hocko
Suggested-by: Kirill A. Shutemov
Cc: Matthew Wilcox
Cc: Laurent Dufour
Cc: Andrew Morton
Signed-off-by: Yang Shi
---
include/linux/mm.h | 2 +-
mm/memory.c| 35 +--
mm/mmap.c | 99 +-
3 files changed, 117 insertions(+), 19 deletions(-)
diff --git a/include/linux/mm.h b/include/linux/mm.h
index a0fbb9f..95a4e97 100644
--- a/include/linux/mm.h
+++ b/include/linux/mm.h
@@ -1321,7 +1321,7 @@ void zap_vma_ptes(struct vm_area_struct *vma, unsigned
long address,
void zap_page_range(struct vm_area_struct *vma, unsigned long address,
unsigned long size);
void unmap_vmas(struct mmu_gather *tlb, struct vm_area_struct *start_vma,
- unsigned long start, unsigned long end);
+ unsigned long start, unsigned long end, bool skip_flags);
/**
* mm_walk - callbacks for walk_page_range
diff --git a/mm/memory.c b/mm/memory.c
index 7206a63..00ecdae 100644
--- a/mm/memory.c
+++ b/mm/memory.c
@@ -1514,7 +1514,7 @@ void unmap_page_range(struct mmu_gather *tlb,
static void unmap_single_vma(struct mmu_gather *tlb,
[RFC v5 PATCH 2/2] mm: mmap: zap pages with read mmap_sem in munmap
When running some mmap/munmap scalability tests with large memory (i.e.
> 300GB), the below hung task issue may happen occasionally.
INFO: task ps:14018 blocked for more than 120 seconds.
Tainted: GE 4.9.79-009.ali3000.alios7.x86_64 #1
"echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this
message.
ps D0 14018 1 0x0004
885582f84000 885e8682f000 880972943000 885ebf499bc0
8828ee12 c900349bfca8 817154d0 0040
00ff812f872a 885ebf499bc0 024000d000948300 880972943000
Call Trace:
[] ? __schedule+0x250/0x730
[] schedule+0x36/0x80
[] rwsem_down_read_failed+0xf0/0x150
[] call_rwsem_down_read_failed+0x18/0x30
[] down_read+0x20/0x40
[] proc_pid_cmdline_read+0xd9/0x4e0
[] ? do_filp_open+0xa5/0x100
[] __vfs_read+0x37/0x150
[] ? security_file_permission+0x9b/0xc0
[] vfs_read+0x96/0x130
[] SyS_read+0x55/0xc0
[] entry_SYSCALL_64_fastpath+0x1a/0xc5
It is because munmap holds mmap_sem exclusively from very beginning to
all the way down to the end, and doesn't release it in the middle. When
unmapping large mapping, it may take long time (take ~18 seconds to
unmap 320GB mapping with every single page mapped on an idle machine).
Zapping pages is the most time consuming part, according to the
suggestion from Michal Hocko [1], zapping pages can be done with holding
read mmap_sem, like what MADV_DONTNEED does. Then re-acquire write
mmap_sem to cleanup vmas.
But, some part may need write mmap_sem, for example, vma splitting. So,
the design is as follows:
acquire write mmap_sem
lookup vmas (find and split vmas)
detach vmas
deal with special mappings
downgrade_write
zap pages
free page tables
release mmap_sem
The vm events with read mmap_sem may come in during page zapping, but
since vmas have been detached before, they, i.e. page fault, gup, etc,
will not be able to find valid vma, then just return SIGSEGV or -EFAULT
as expected.
If the vma has VM_LOCKED | VM_HUGETLB | VM_PFNMAP or uprobe, they are
considered as special mappings. They will be dealt with before zapping
pages with write mmap_sem held. Basically, just update vm_flags.
And, since they are also manipulated by unmap_single_vma() which is
called by unmap_vma() with read mmap_sem held in this case, to
prevent from updating vm_flags in read critical section, a new
parameter, called "skip_flags" is added to unmap_region(), unmap_vmas()
and unmap_single_vma(). If it is true, then just skip unmap those
special mappings. Currently, the only place which pass true to this
parameter is us.
With this approach we don't have to re-acquire mmap_sem again to clean
up vmas to avoid race window which might get the address space changed.
And, since the lock acquire/release cost is managed to the minimum and
almost as same as before, the optimization could be extended to any size
of mapping without incuring significan penalty to small mappings.
For the time being, just do this in munmap syscall path. Other
vm_munmap() or do_munmap() call sites (i.e mmap, mremap, etc) remain
intact for stability reason.
With the patches, exclusive mmap_sem hold time when munmap a 80GB
address space on a machine with 32 cores of E5-2680 @ 2.70GHz dropped to
us level from second.
munmap_test-15002 [008] 594.380138: funcgraph_entry: | vm_munmap_zap_rlock()
{
munmap_test-15002 [008] 594.380146: funcgraph_entry: !2485684 us |
unmap_region();
munmap_test-15002 [008] 596.865836: funcgraph_exit: !2485692 us | }
Here the excution time of unmap_region() is used to evaluate the time of
holding read mmap_sem, then the remaining time is used with holding
exclusive lock.
[1] https://lwn.net/Articles/753269/
Suggested-by: Michal Hocko
Suggested-by: Kirill A. Shutemov
Cc: Matthew Wilcox
Cc: Laurent Dufour
Cc: Andrew Morton
Signed-off-by: Yang Shi
---
include/linux/mm.h | 2 +-
mm/memory.c| 35 +--
mm/mmap.c | 99 +-
3 files changed, 117 insertions(+), 19 deletions(-)
diff --git a/include/linux/mm.h b/include/linux/mm.h
index a0fbb9f..95a4e97 100644
--- a/include/linux/mm.h
+++ b/include/linux/mm.h
@@ -1321,7 +1321,7 @@ void zap_vma_ptes(struct vm_area_struct *vma, unsigned
long address,
void zap_page_range(struct vm_area_struct *vma, unsigned long address,
unsigned long size);
void unmap_vmas(struct mmu_gather *tlb, struct vm_area_struct *start_vma,
- unsigned long start, unsigned long end);
+ unsigned long start, unsigned long end, bool skip_flags);
/**
* mm_walk - callbacks for walk_page_range
diff --git a/mm/memory.c b/mm/memory.c
index 7206a63..00ecdae 100644
--- a/mm/memory.c
+++ b/mm/memory.c
@@ -1514,7 +1514,7 @@ void unmap_page_range(struct mmu_gather *tlb,
static void unmap_single_vma(struct mmu_gather *tlb,
