Re: [RFC PATCH] mm: kasan: suppress soft lockup in slub when !CONFIG_PREEMPT

[Yang Shi]

On 12/8/17 1:16 AM, Andrey Ryabinin wrote:

On 12/08/2017 11:26 AM, Dmitry Vyukov wrote:

On Fri, Dec 8, 2017 at 12:40 AM, Matthew Wilcox  wrote:

On Fri, Dec 08, 2017 at 07:30:07AM +0800, Yang Shi wrote:

When running stress test with KASAN enabled, the below softlockup may
happen occasionally:

NMI watchdog: BUG: soft lockup - CPU#7 stuck for 22s!
hardirqs last  enabled at (0): [<  (null)>]  (null)
hardirqs last disabled at (0): [] copy_process.part.30+0x5c6/0x1f50
softirqs last  enabled at (0): [] copy_process.part.30+0x5c6/0x1f50
softirqs last disabled at (0): [<  (null)>]  (null)



Call Trace:
  [] __slab_free+0x19c/0x270
  [] ___cache_free+0xa6/0xb0
  [] qlist_free_all+0x47/0x80
  [] quarantine_reduce+0x159/0x190
  [] kasan_kmalloc+0xaf/0xc0
  [] kasan_slab_alloc+0x12/0x20
  [] kmem_cache_alloc+0xfa/0x360
  [] ? getname_flags+0x4f/0x1f0
  [] getname_flags+0x4f/0x1f0
  [] getname+0x12/0x20
  [] do_sys_open+0xf9/0x210
  [] SyS_open+0x1e/0x20
  [] entry_SYSCALL_64_fastpath+0x1f/0xc2


This feels like papering over a problem.  KASAN only calls
quarantine_reduce() when it's allowed to block.  Presumably it has
millions of entries on the free list at this point.  I think the right
thing to do is for qlist_free_all() to call cond_resched() after freeing
every N items.



Agree. Adding touch_softlockup_watchdog() to a random low-level
function looks like a wrong thing to do.
quarantine_reduce() already has this logic. Look at
QUARANTINE_BATCHES. It's meant to do exactly this -- limit amount of
work in quarantine_reduce() and in quarantine_remove_cache() to
reasonably-sized batches. We could simply increase number of batches
to make them smaller. But it would be good to understand what exactly
happens in this case. Batches should on a par of ~~1MB. Why freeing
1MB worth of objects (smallest of which is 32b) takes 22 seconds?



I think the problem here is that kernel 4.9.44-003.ali3000.alios7.x86_64.debug
doesn't have 64abdcb24351 ("kasan: eliminate long stalls during quarantine 
reduction").

We probably should ask that commit to be included in stable, but it would be 
good to hear
a confirmation from Yang that it really helps.


Thanks, folks. Yes, my kernel doesn't have this commit. It sounds the 
commit batches the quarantine to smaller group. I will run some tests 
against this commit to see if it could help. Reading the code tells me 
it is likely to help.


Yang

Re: [RFC PATCH] mm: kasan: suppress soft lockup in slub when !CONFIG_PREEMPT

[Yang Shi]

On 12/8/17 1:16 AM, Andrey Ryabinin wrote:

On 12/08/2017 11:26 AM, Dmitry Vyukov wrote:

On Fri, Dec 8, 2017 at 12:40 AM, Matthew Wilcox  wrote:

On Fri, Dec 08, 2017 at 07:30:07AM +0800, Yang Shi wrote:

When running stress test with KASAN enabled, the below softlockup may
happen occasionally:

NMI watchdog: BUG: soft lockup - CPU#7 stuck for 22s!
hardirqs last  enabled at (0): [<  (null)>]  (null)
hardirqs last disabled at (0): [] copy_process.part.30+0x5c6/0x1f50
softirqs last  enabled at (0): [] copy_process.part.30+0x5c6/0x1f50
softirqs last disabled at (0): [<  (null)>]  (null)



Call Trace:
  [] __slab_free+0x19c/0x270
  [] ___cache_free+0xa6/0xb0
  [] qlist_free_all+0x47/0x80
  [] quarantine_reduce+0x159/0x190
  [] kasan_kmalloc+0xaf/0xc0
  [] kasan_slab_alloc+0x12/0x20
  [] kmem_cache_alloc+0xfa/0x360
  [] ? getname_flags+0x4f/0x1f0
  [] getname_flags+0x4f/0x1f0
  [] getname+0x12/0x20
  [] do_sys_open+0xf9/0x210
  [] SyS_open+0x1e/0x20
  [] entry_SYSCALL_64_fastpath+0x1f/0xc2


This feels like papering over a problem.  KASAN only calls
quarantine_reduce() when it's allowed to block.  Presumably it has
millions of entries on the free list at this point.  I think the right
thing to do is for qlist_free_all() to call cond_resched() after freeing
every N items.



Agree. Adding touch_softlockup_watchdog() to a random low-level
function looks like a wrong thing to do.
quarantine_reduce() already has this logic. Look at
QUARANTINE_BATCHES. It's meant to do exactly this -- limit amount of
work in quarantine_reduce() and in quarantine_remove_cache() to
reasonably-sized batches. We could simply increase number of batches
to make them smaller. But it would be good to understand what exactly
happens in this case. Batches should on a par of ~~1MB. Why freeing
1MB worth of objects (smallest of which is 32b) takes 22 seconds?



I think the problem here is that kernel 4.9.44-003.ali3000.alios7.x86_64.debug
doesn't have 64abdcb24351 ("kasan: eliminate long stalls during quarantine 
reduction").

We probably should ask that commit to be included in stable, but it would be 
good to hear
a confirmation from Yang that it really helps.


Thanks, folks. Yes, my kernel doesn't have this commit. It sounds the 
commit batches the quarantine to smaller group. I will run some tests 
against this commit to see if it could help. Reading the code tells me 
it is likely to help.


Yang

Re: [RFC PATCH] mm: kasan: suppress soft lockup in slub when !CONFIG_PREEMPT

[Andrey Ryabinin]

On 12/08/2017 11:26 AM, Dmitry Vyukov wrote:
> On Fri, Dec 8, 2017 at 12:40 AM, Matthew Wilcox  wrote:
>> On Fri, Dec 08, 2017 at 07:30:07AM +0800, Yang Shi wrote:
>>> When running stress test with KASAN enabled, the below softlockup may
>>> happen occasionally:
>>>
>>> NMI watchdog: BUG: soft lockup - CPU#7 stuck for 22s!
>>> hardirqs last  enabled at (0): [<  (null)>]  (null)
>>> hardirqs last disabled at (0): [] copy_process.part.30+0x5c6/0x1f50
>>> softirqs last  enabled at (0): [] copy_process.part.30+0x5c6/0x1f50
>>> softirqs last disabled at (0): [<  (null)>]  (null)
>>
>>> Call Trace:
>>>  [] __slab_free+0x19c/0x270
>>>  [] ___cache_free+0xa6/0xb0
>>>  [] qlist_free_all+0x47/0x80
>>>  [] quarantine_reduce+0x159/0x190
>>>  [] kasan_kmalloc+0xaf/0xc0
>>>  [] kasan_slab_alloc+0x12/0x20
>>>  [] kmem_cache_alloc+0xfa/0x360
>>>  [] ? getname_flags+0x4f/0x1f0
>>>  [] getname_flags+0x4f/0x1f0
>>>  [] getname+0x12/0x20
>>>  [] do_sys_open+0xf9/0x210
>>>  [] SyS_open+0x1e/0x20
>>>  [] entry_SYSCALL_64_fastpath+0x1f/0xc2
>>
>> This feels like papering over a problem.  KASAN only calls
>> quarantine_reduce() when it's allowed to block.  Presumably it has
>> millions of entries on the free list at this point.  I think the right
>> thing to do is for qlist_free_all() to call cond_resched() after freeing
>> every N items.
> 
> 
> Agree. Adding touch_softlockup_watchdog() to a random low-level
> function looks like a wrong thing to do.
> quarantine_reduce() already has this logic. Look at
> QUARANTINE_BATCHES. It's meant to do exactly this -- limit amount of
> work in quarantine_reduce() and in quarantine_remove_cache() to
> reasonably-sized batches. We could simply increase number of batches
> to make them smaller. But it would be good to understand what exactly
> happens in this case. Batches should on a par of ~~1MB. Why freeing
> 1MB worth of objects (smallest of which is 32b) takes 22 seconds?
> 

I think the problem here is that kernel 4.9.44-003.ali3000.alios7.x86_64.debug
doesn't have 64abdcb24351 ("kasan: eliminate long stalls during quarantine 
reduction").

We probably should ask that commit to be included in stable, but it would be 
good to hear
a confirmation from Yang that it really helps.

Re: [RFC PATCH] mm: kasan: suppress soft lockup in slub when !CONFIG_PREEMPT

[Andrey Ryabinin]

On 12/08/2017 11:26 AM, Dmitry Vyukov wrote:
> On Fri, Dec 8, 2017 at 12:40 AM, Matthew Wilcox  wrote:
>> On Fri, Dec 08, 2017 at 07:30:07AM +0800, Yang Shi wrote:
>>> When running stress test with KASAN enabled, the below softlockup may
>>> happen occasionally:
>>>
>>> NMI watchdog: BUG: soft lockup - CPU#7 stuck for 22s!
>>> hardirqs last  enabled at (0): [<  (null)>]  (null)
>>> hardirqs last disabled at (0): [] copy_process.part.30+0x5c6/0x1f50
>>> softirqs last  enabled at (0): [] copy_process.part.30+0x5c6/0x1f50
>>> softirqs last disabled at (0): [<  (null)>]  (null)
>>
>>> Call Trace:
>>>  [] __slab_free+0x19c/0x270
>>>  [] ___cache_free+0xa6/0xb0
>>>  [] qlist_free_all+0x47/0x80
>>>  [] quarantine_reduce+0x159/0x190
>>>  [] kasan_kmalloc+0xaf/0xc0
>>>  [] kasan_slab_alloc+0x12/0x20
>>>  [] kmem_cache_alloc+0xfa/0x360
>>>  [] ? getname_flags+0x4f/0x1f0
>>>  [] getname_flags+0x4f/0x1f0
>>>  [] getname+0x12/0x20
>>>  [] do_sys_open+0xf9/0x210
>>>  [] SyS_open+0x1e/0x20
>>>  [] entry_SYSCALL_64_fastpath+0x1f/0xc2
>>
>> This feels like papering over a problem.  KASAN only calls
>> quarantine_reduce() when it's allowed to block.  Presumably it has
>> millions of entries on the free list at this point.  I think the right
>> thing to do is for qlist_free_all() to call cond_resched() after freeing
>> every N items.
> 
> 
> Agree. Adding touch_softlockup_watchdog() to a random low-level
> function looks like a wrong thing to do.
> quarantine_reduce() already has this logic. Look at
> QUARANTINE_BATCHES. It's meant to do exactly this -- limit amount of
> work in quarantine_reduce() and in quarantine_remove_cache() to
> reasonably-sized batches. We could simply increase number of batches
> to make them smaller. But it would be good to understand what exactly
> happens in this case. Batches should on a par of ~~1MB. Why freeing
> 1MB worth of objects (smallest of which is 32b) takes 22 seconds?
> 

I think the problem here is that kernel 4.9.44-003.ali3000.alios7.x86_64.debug
doesn't have 64abdcb24351 ("kasan: eliminate long stalls during quarantine 
reduction").

We probably should ask that commit to be included in stable, but it would be 
good to hear
a confirmation from Yang that it really helps.

Re: [RFC PATCH] mm: kasan: suppress soft lockup in slub when !CONFIG_PREEMPT

[Dmitry Vyukov]

On Fri, Dec 8, 2017 at 12:40 AM, Matthew Wilcox  wrote:
> On Fri, Dec 08, 2017 at 07:30:07AM +0800, Yang Shi wrote:
>> When running stress test with KASAN enabled, the below softlockup may
>> happen occasionally:
>>
>> NMI watchdog: BUG: soft lockup - CPU#7 stuck for 22s!
>> hardirqs last  enabled at (0): [<  (null)>]  (null)
>> hardirqs last disabled at (0): [] copy_process.part.30+0x5c6/0x1f50
>> softirqs last  enabled at (0): [] copy_process.part.30+0x5c6/0x1f50
>> softirqs last disabled at (0): [<  (null)>]  (null)
>
>> Call Trace:
>>  [] __slab_free+0x19c/0x270
>>  [] ___cache_free+0xa6/0xb0
>>  [] qlist_free_all+0x47/0x80
>>  [] quarantine_reduce+0x159/0x190
>>  [] kasan_kmalloc+0xaf/0xc0
>>  [] kasan_slab_alloc+0x12/0x20
>>  [] kmem_cache_alloc+0xfa/0x360
>>  [] ? getname_flags+0x4f/0x1f0
>>  [] getname_flags+0x4f/0x1f0
>>  [] getname+0x12/0x20
>>  [] do_sys_open+0xf9/0x210
>>  [] SyS_open+0x1e/0x20
>>  [] entry_SYSCALL_64_fastpath+0x1f/0xc2
>
> This feels like papering over a problem.  KASAN only calls
> quarantine_reduce() when it's allowed to block.  Presumably it has
> millions of entries on the free list at this point.  I think the right
> thing to do is for qlist_free_all() to call cond_resched() after freeing
> every N items.


Agree. Adding touch_softlockup_watchdog() to a random low-level
function looks like a wrong thing to do.
quarantine_reduce() already has this logic. Look at
QUARANTINE_BATCHES. It's meant to do exactly this -- limit amount of
work in quarantine_reduce() and in quarantine_remove_cache() to
reasonably-sized batches. We could simply increase number of batches
to make them smaller. But it would be good to understand what exactly
happens in this case. Batches should on a par of ~~1MB. Why freeing
1MB worth of objects (smallest of which is 32b) takes 22 seconds?



>> The code is run in irq disabled or preempt disabled context, so
>> cond_resched() can't be used in this case. Touch softlockup watchdog when
>> KASAN is enabled to suppress the warning.
>>
>> Signed-off-by: Yang Shi 
>> ---
>>  mm/slub.c | 5 +
>>  1 file changed, 5 insertions(+)
>>
>> diff --git a/mm/slub.c b/mm/slub.c
>> index cfd56e5..4ae435e 100644
>> --- a/mm/slub.c
>> +++ b/mm/slub.c
>> @@ -35,6 +35,7 @@
>>  #include 
>>  #include 
>>  #include 
>> +#include 
>>
>>  #include 
>>
>> @@ -2266,6 +2267,10 @@ static void put_cpu_partial(struct kmem_cache *s, 
>> struct page *page, int drain)
>>   page->pobjects = pobjects;
>>   page->next = oldpage;
>>
>> +#ifdef CONFIG_KASAN
>> + touch_softlockup_watchdog();
>> +#endif
>> +
>>   } while (this_cpu_cmpxchg(s->cpu_slab->partial, oldpage, page)
>>   != oldpage);
>>   if (unlikely(!s->cpu_partial)) {
>> --
>> 1.8.3.1
>>
>> --
>> To unsubscribe, send a message with 'unsubscribe linux-mm' in
>> the body to majord...@kvack.org.  For more info on Linux MM,
>> see: http://www.linux-mm.org/ .
>> Don't email: mailto:"d...@kvack.org;> em...@kvack.org 

Re: [RFC PATCH] mm: kasan: suppress soft lockup in slub when !CONFIG_PREEMPT

[Dmitry Vyukov]

On Fri, Dec 8, 2017 at 12:40 AM, Matthew Wilcox  wrote:
> On Fri, Dec 08, 2017 at 07:30:07AM +0800, Yang Shi wrote:
>> When running stress test with KASAN enabled, the below softlockup may
>> happen occasionally:
>>
>> NMI watchdog: BUG: soft lockup - CPU#7 stuck for 22s!
>> hardirqs last  enabled at (0): [<  (null)>]  (null)
>> hardirqs last disabled at (0): [] copy_process.part.30+0x5c6/0x1f50
>> softirqs last  enabled at (0): [] copy_process.part.30+0x5c6/0x1f50
>> softirqs last disabled at (0): [<  (null)>]  (null)
>
>> Call Trace:
>>  [] __slab_free+0x19c/0x270
>>  [] ___cache_free+0xa6/0xb0
>>  [] qlist_free_all+0x47/0x80
>>  [] quarantine_reduce+0x159/0x190
>>  [] kasan_kmalloc+0xaf/0xc0
>>  [] kasan_slab_alloc+0x12/0x20
>>  [] kmem_cache_alloc+0xfa/0x360
>>  [] ? getname_flags+0x4f/0x1f0
>>  [] getname_flags+0x4f/0x1f0
>>  [] getname+0x12/0x20
>>  [] do_sys_open+0xf9/0x210
>>  [] SyS_open+0x1e/0x20
>>  [] entry_SYSCALL_64_fastpath+0x1f/0xc2
>
> This feels like papering over a problem.  KASAN only calls
> quarantine_reduce() when it's allowed to block.  Presumably it has
> millions of entries on the free list at this point.  I think the right
> thing to do is for qlist_free_all() to call cond_resched() after freeing
> every N items.


Agree. Adding touch_softlockup_watchdog() to a random low-level
function looks like a wrong thing to do.
quarantine_reduce() already has this logic. Look at
QUARANTINE_BATCHES. It's meant to do exactly this -- limit amount of
work in quarantine_reduce() and in quarantine_remove_cache() to
reasonably-sized batches. We could simply increase number of batches
to make them smaller. But it would be good to understand what exactly
happens in this case. Batches should on a par of ~~1MB. Why freeing
1MB worth of objects (smallest of which is 32b) takes 22 seconds?



>> The code is run in irq disabled or preempt disabled context, so
>> cond_resched() can't be used in this case. Touch softlockup watchdog when
>> KASAN is enabled to suppress the warning.
>>
>> Signed-off-by: Yang Shi 
>> ---
>>  mm/slub.c | 5 +
>>  1 file changed, 5 insertions(+)
>>
>> diff --git a/mm/slub.c b/mm/slub.c
>> index cfd56e5..4ae435e 100644
>> --- a/mm/slub.c
>> +++ b/mm/slub.c
>> @@ -35,6 +35,7 @@
>>  #include 
>>  #include 
>>  #include 
>> +#include 
>>
>>  #include 
>>
>> @@ -2266,6 +2267,10 @@ static void put_cpu_partial(struct kmem_cache *s, 
>> struct page *page, int drain)
>>   page->pobjects = pobjects;
>>   page->next = oldpage;
>>
>> +#ifdef CONFIG_KASAN
>> + touch_softlockup_watchdog();
>> +#endif
>> +
>>   } while (this_cpu_cmpxchg(s->cpu_slab->partial, oldpage, page)
>>   != oldpage);
>>   if (unlikely(!s->cpu_partial)) {
>> --
>> 1.8.3.1
>>
>> --
>> To unsubscribe, send a message with 'unsubscribe linux-mm' in
>> the body to majord...@kvack.org.  For more info on Linux MM,
>> see: http://www.linux-mm.org/ .
>> Don't email: mailto:"d...@kvack.org;> em...@kvack.org 

Re: [RFC PATCH] mm: kasan: suppress soft lockup in slub when !CONFIG_PREEMPT

[Matthew Wilcox]

On Fri, Dec 08, 2017 at 07:30:07AM +0800, Yang Shi wrote:
> When running stress test with KASAN enabled, the below softlockup may
> happen occasionally:
> 
> NMI watchdog: BUG: soft lockup - CPU#7 stuck for 22s!
> hardirqs last  enabled at (0): [<          (null)>]      (null)
> hardirqs last disabled at (0): [] copy_process.part.30+0x5c6/0x1f50
> softirqs last  enabled at (0): [] copy_process.part.30+0x5c6/0x1f50
> softirqs last disabled at (0): [<          (null)>]      (null)

> Call Trace:
>  [] __slab_free+0x19c/0x270
>  [] ___cache_free+0xa6/0xb0
>  [] qlist_free_all+0x47/0x80
>  [] quarantine_reduce+0x159/0x190
>  [] kasan_kmalloc+0xaf/0xc0
>  [] kasan_slab_alloc+0x12/0x20
>  [] kmem_cache_alloc+0xfa/0x360
>  [] ? getname_flags+0x4f/0x1f0
>  [] getname_flags+0x4f/0x1f0
>  [] getname+0x12/0x20
>  [] do_sys_open+0xf9/0x210
>  [] SyS_open+0x1e/0x20
>  [] entry_SYSCALL_64_fastpath+0x1f/0xc2

This feels like papering over a problem.  KASAN only calls
quarantine_reduce() when it's allowed to block.  Presumably it has
millions of entries on the free list at this point.  I think the right
thing to do is for qlist_free_all() to call cond_resched() after freeing
every N items.

> The code is run in irq disabled or preempt disabled context, so
> cond_resched() can't be used in this case. Touch softlockup watchdog when
> KASAN is enabled to suppress the warning.
> 
> Signed-off-by: Yang Shi 
> ---
>  mm/slub.c | 5 +
>  1 file changed, 5 insertions(+)
> 
> diff --git a/mm/slub.c b/mm/slub.c
> index cfd56e5..4ae435e 100644
> --- a/mm/slub.c
> +++ b/mm/slub.c
> @@ -35,6 +35,7 @@
>  #include 
>  #include 
>  #include 
> +#include 
>  
>  #include 
>  
> @@ -2266,6 +2267,10 @@ static void put_cpu_partial(struct kmem_cache *s, 
> struct page *page, int drain)
>   page->pobjects = pobjects;
>   page->next = oldpage;
>  
> +#ifdef CONFIG_KASAN
> + touch_softlockup_watchdog();
> +#endif
> +
>   } while (this_cpu_cmpxchg(s->cpu_slab->partial, oldpage, page)
>   != oldpage);
>   if (unlikely(!s->cpu_partial)) {
> -- 
> 1.8.3.1
> 
> --
> To unsubscribe, send a message with 'unsubscribe linux-mm' in
> the body to majord...@kvack.org.  For more info on Linux MM,
> see: http://www.linux-mm.org/ .
> Don't email: mailto:"d...@kvack.org;> em...@kvack.org 

Re: [RFC PATCH] mm: kasan: suppress soft lockup in slub when !CONFIG_PREEMPT

[Matthew Wilcox]

On Fri, Dec 08, 2017 at 07:30:07AM +0800, Yang Shi wrote:
> When running stress test with KASAN enabled, the below softlockup may
> happen occasionally:
> 
> NMI watchdog: BUG: soft lockup - CPU#7 stuck for 22s!
> hardirqs last  enabled at (0): [<          (null)>]      (null)
> hardirqs last disabled at (0): [] copy_process.part.30+0x5c6/0x1f50
> softirqs last  enabled at (0): [] copy_process.part.30+0x5c6/0x1f50
> softirqs last disabled at (0): [<          (null)>]      (null)

> Call Trace:
>  [] __slab_free+0x19c/0x270
>  [] ___cache_free+0xa6/0xb0
>  [] qlist_free_all+0x47/0x80
>  [] quarantine_reduce+0x159/0x190
>  [] kasan_kmalloc+0xaf/0xc0
>  [] kasan_slab_alloc+0x12/0x20
>  [] kmem_cache_alloc+0xfa/0x360
>  [] ? getname_flags+0x4f/0x1f0
>  [] getname_flags+0x4f/0x1f0
>  [] getname+0x12/0x20
>  [] do_sys_open+0xf9/0x210
>  [] SyS_open+0x1e/0x20
>  [] entry_SYSCALL_64_fastpath+0x1f/0xc2

This feels like papering over a problem.  KASAN only calls
quarantine_reduce() when it's allowed to block.  Presumably it has
millions of entries on the free list at this point.  I think the right
thing to do is for qlist_free_all() to call cond_resched() after freeing
every N items.

> The code is run in irq disabled or preempt disabled context, so
> cond_resched() can't be used in this case. Touch softlockup watchdog when
> KASAN is enabled to suppress the warning.
> 
> Signed-off-by: Yang Shi 
> ---
>  mm/slub.c | 5 +
>  1 file changed, 5 insertions(+)
> 
> diff --git a/mm/slub.c b/mm/slub.c
> index cfd56e5..4ae435e 100644
> --- a/mm/slub.c
> +++ b/mm/slub.c
> @@ -35,6 +35,7 @@
>  #include 
>  #include 
>  #include 
> +#include 
>  
>  #include 
>  
> @@ -2266,6 +2267,10 @@ static void put_cpu_partial(struct kmem_cache *s, 
> struct page *page, int drain)
>   page->pobjects = pobjects;
>   page->next = oldpage;
>  
> +#ifdef CONFIG_KASAN
> + touch_softlockup_watchdog();
> +#endif
> +
>   } while (this_cpu_cmpxchg(s->cpu_slab->partial, oldpage, page)
>   != oldpage);
>   if (unlikely(!s->cpu_partial)) {
> -- 
> 1.8.3.1
> 
> --
> To unsubscribe, send a message with 'unsubscribe linux-mm' in
> the body to majord...@kvack.org.  For more info on Linux MM,
> see: http://www.linux-mm.org/ .
> Don't email: mailto:"d...@kvack.org;> em...@kvack.org