Re: lockdep warning: console vs. mem hotplug
Hello,
On (07/24/17 14:46), Sebastian Ott wrote:
> [ 347.644660] ==
> [ 347.644660] WARNING: possible circular locking dependency detected
> [ 347.644661] 4.13.0-rc2 #146 Not tainted
> [ 347.644661] --
> [ 347.644662] sh/770 is trying to acquire lock:
> [ 347.644662] (&console_sch_key){-.-...}, at: [<00763d80>]
> raw3215_write+0x58/0x208
> [ 347.644665] but task is already holding lock:
> [ 347.644665] (&(&zone->lock)->rlock){-.-...}, at: [<002b693e>]
> __offline_isolated_pages+0x316/0x388
> [ 347.644668] which lock already depends on the new lock.
> [ 347.644669] the existing dependency chain (in reverse order) is:
> [ 347.644670] -> #4 (&(&zone->lock)->rlock){-.-...}:
> [ 347.644672]validate_chain.isra.10+0xb56/0xd88
> [ 347.644673]__lock_acquire+0x62c/0x850
> [ 347.644673]lock_acquire+0x254/0x2b8
> [ 347.644674]_raw_spin_lock_irqsave+0x70/0xb8
> [ 347.644674]get_page_from_freelist+0x446/0xf30
> [ 347.644675]__alloc_pages_nodemask+0x200/0x1568
> [ 347.644675]allocate_slab+0xf0/0x658
> [ 347.644676]new_slab+0x94/0xa8
> [ 347.644677]___slab_alloc.constprop.23+0x55e/0x580
> [ 347.644677]__slab_alloc.isra.17.constprop.22+0x74/0xa8
> [ 347.644678]kmem_cache_alloc+0x13c/0x4b0
> [ 347.644678]__debug_object_init+0x5c/0x468
> [ 347.644679]hrtimer_init+0x42/0x1d8
> [ 347.644679]init_dl_task_timer+0x3a/0x58
> [ 347.644680]__sched_fork.isra.2+0x82/0xd8
> [ 347.644680]init_idle+0x7a/0x278
> [ 347.644681]fork_idle+0xa4/0xb8
> [ 347.644681]idle_threads_init+0x6a/0xd0
> [ 347.644682]smp_init+0x34/0x110
> [ 347.644682]kernel_init_freeable+0x166/0x2d8
> [ 347.644683]kernel_init+0x2a/0x148
> [ 347.644683]kernel_thread_starter+0x6/0xc
> [ 347.644684]kernel_thread_starter+0x0/0xc
>
> [ 347.644684] -> #3 (&rq->lock){-.-.-.}:
> [ 347.644686]validate_chain.isra.10+0xb56/0xd88
> [ 347.644687]__lock_acquire+0x62c/0x850
> [ 347.644687]lock_acquire+0x254/0x2b8
> [ 347.644688]_raw_spin_lock+0x60/0xa0
> [ 347.644688]task_fork_fair+0x6a/0x160
> [ 347.644689]sched_fork+0x13e/0x2a0
> [ 347.644689]copy_process+0x676/0x1ec0
> [ 347.644690]_do_fork+0xc2/0x6d0
> [ 347.644690]kernel_thread+0x4e/0x60
> [ 347.644691]rest_init+0x48/0x290
> [ 347.644691]start_kernel+0x470/0x480
> [ 347.644692]_stext+0x20/0x80
ok... this part is not exactly clear to me, but we've got
rq->lock->&(&zone->lock)->rlock
dependency.
> [ 347.644693] -> #2 (&p->pi_lock){-.-.-.}:
> [ 347.644695]validate_chain.isra.10+0xb56/0xd88
> [ 347.644695]__lock_acquire+0x62c/0x850
> [ 347.644696]lock_acquire+0x254/0x2b8
> [ 347.644696]_raw_spin_lock_irqsave+0x70/0xb8
> [ 347.644697]try_to_wake_up+0x4a/0x600
> [ 347.644697]autoremove_wake_function+0x2e/0x88
> [ 347.644698]__wake_up_common+0x76/0xc0
> [ 347.644698]__wake_up+0x54/0x68
> [ 347.644699]ccw_device_verify_done+0xae/0x268
> [ 347.644700]ccw_request_handler+0x422/0x560
> [ 347.644700]do_cio_interrupt+0x224/0x2a0
> [ 347.644701]__handle_irq_event_percpu+0x1a6/0x440
> [ 347.644701]handle_irq_event_percpu+0x38/0x88
> [ 347.644702]handle_percpu_irq+0x84/0xb0
> [ 347.644702]generic_handle_irq+0x42/0x60
> [ 347.644703]do_IRQ+0x86/0xc8
> [ 347.644703]io_int_handler+0x104/0x2d4
> [ 347.644704]enabled_wait+0x72/0x140
> [ 347.644704]enabled_wait+0x5a/0x140
> [ 347.644705]arch_cpu_idle+0x32/0x50
> [ 347.644706]default_idle_call+0x52/0x68
> [ 347.644706]do_idle+0x118/0x170
> [ 347.644707]cpu_startup_entry+0x3e/0x48
> [ 347.644707]smp_start_secondary+0x112/0x120
> [ 347.644708]restart_int_handler+0x62/0x78
> [ 347.644708] (null)
>
> [ 347.644709] -> #1 (&priv->wait_q){-.}:
> [ 347.644711]validate_chain.isra.10+0xb56/0xd88
> [ 347.644711]__lock_acquire+0x62c/0x850
> [ 347.644712]lock_acquire+0x254/0x2b8
> [ 347.644712]_raw_spin_lock_irqsave+0x70/0xb8
> [ 347.644713]__wake_up+0x3a/0x68
> [ 347.644713]ccw_device_recog_done+0x28e/0x2c8
> [ 347.644714]snsid_callback+0x324/0x390
> [ 347.644714]ccw_request_handler+0x480/0x560
> [ 347.644715]do_cio_interrupt+0x224/0x2a0
> [ 347.644715]__handle_irq_event_percpu+0x1a6/0x440
> [ 347.644715]handle_irq_event_percpu+0x38/0x88
> [ 347.644716]handle_percpu_irq+0x84/0xb0
> [ 347.644716]generic_handle_irq+0x42/0x60
> [ 347.644717]do_IRQ+0x86/0xc8
> [ 347.644717]io_
Re: lockdep warning: console vs. mem hotplug
Hello Sergey,
On Tue, 21 Mar 2017, Sergey Senozhatsky wrote:
> // I kept only lockdep splats from the original report and cut off .configs
> // full version:
> // lkml.kernel.org/r/alpine.LFD.2.20.1703201736070.1753@schleppi
>
> On (03/20/17 17:43), Sebastian Ott wrote:
> > Since commit f975237b7682 ("printk: use printk_safe buffers in printk")
> > I observe lockdep warnings on s390 when doing memory hotplug:
When using the sclp console I can no longer recreate that lockdep warning
but the other one is still present:
[ 347.644660] ==
[ 347.644660] WARNING: possible circular locking dependency detected
[ 347.644661] 4.13.0-rc2 #146 Not tainted
[ 347.644661] --
[ 347.644662] sh/770 is trying to acquire lock:
[ 347.644662] (&console_sch_key){-.-...}, at: [<00763d80>]
raw3215_write+0x58/0x208
[ 347.644665] but task is already holding lock:
[ 347.644665] (&(&zone->lock)->rlock){-.-...}, at: [<002b693e>]
__offline_isolated_pages+0x316/0x388
[ 347.644668] which lock already depends on the new lock.
[ 347.644669] the existing dependency chain (in reverse order) is:
[ 347.644670] -> #4 (&(&zone->lock)->rlock){-.-...}:
[ 347.644672]validate_chain.isra.10+0xb56/0xd88
[ 347.644673]__lock_acquire+0x62c/0x850
[ 347.644673]lock_acquire+0x254/0x2b8
[ 347.644674]_raw_spin_lock_irqsave+0x70/0xb8
[ 347.644674]get_page_from_freelist+0x446/0xf30
[ 347.644675]__alloc_pages_nodemask+0x200/0x1568
[ 347.644675]allocate_slab+0xf0/0x658
[ 347.644676]new_slab+0x94/0xa8
[ 347.644677]___slab_alloc.constprop.23+0x55e/0x580
[ 347.644677]__slab_alloc.isra.17.constprop.22+0x74/0xa8
[ 347.644678]kmem_cache_alloc+0x13c/0x4b0
[ 347.644678]__debug_object_init+0x5c/0x468
[ 347.644679]hrtimer_init+0x42/0x1d8
[ 347.644679]init_dl_task_timer+0x3a/0x58
[ 347.644680]__sched_fork.isra.2+0x82/0xd8
[ 347.644680]init_idle+0x7a/0x278
[ 347.644681]fork_idle+0xa4/0xb8
[ 347.644681]idle_threads_init+0x6a/0xd0
[ 347.644682]smp_init+0x34/0x110
[ 347.644682]kernel_init_freeable+0x166/0x2d8
[ 347.644683]kernel_init+0x2a/0x148
[ 347.644683]kernel_thread_starter+0x6/0xc
[ 347.644684]kernel_thread_starter+0x0/0xc
[ 347.644684] -> #3 (&rq->lock){-.-.-.}:
[ 347.644686]validate_chain.isra.10+0xb56/0xd88
[ 347.644687]__lock_acquire+0x62c/0x850
[ 347.644687]lock_acquire+0x254/0x2b8
[ 347.644688]_raw_spin_lock+0x60/0xa0
[ 347.644688]task_fork_fair+0x6a/0x160
[ 347.644689]sched_fork+0x13e/0x2a0
[ 347.644689]copy_process+0x676/0x1ec0
[ 347.644690]_do_fork+0xc2/0x6d0
[ 347.644690]kernel_thread+0x4e/0x60
[ 347.644691]rest_init+0x48/0x290
[ 347.644691]start_kernel+0x470/0x480
[ 347.644692]_stext+0x20/0x80
[ 347.644693] -> #2 (&p->pi_lock){-.-.-.}:
[ 347.644695]validate_chain.isra.10+0xb56/0xd88
[ 347.644695]__lock_acquire+0x62c/0x850
[ 347.644696]lock_acquire+0x254/0x2b8
[ 347.644696]_raw_spin_lock_irqsave+0x70/0xb8
[ 347.644697]try_to_wake_up+0x4a/0x600
[ 347.644697]autoremove_wake_function+0x2e/0x88
[ 347.644698]__wake_up_common+0x76/0xc0
[ 347.644698]__wake_up+0x54/0x68
[ 347.644699]ccw_device_verify_done+0xae/0x268
[ 347.644700]ccw_request_handler+0x422/0x560
[ 347.644700]do_cio_interrupt+0x224/0x2a0
[ 347.644701]__handle_irq_event_percpu+0x1a6/0x440
[ 347.644701]handle_irq_event_percpu+0x38/0x88
[ 347.644702]handle_percpu_irq+0x84/0xb0
[ 347.644702]generic_handle_irq+0x42/0x60
[ 347.644703]do_IRQ+0x86/0xc8
[ 347.644703]io_int_handler+0x104/0x2d4
[ 347.644704]enabled_wait+0x72/0x140
[ 347.644704]enabled_wait+0x5a/0x140
[ 347.644705]arch_cpu_idle+0x32/0x50
[ 347.644706]default_idle_call+0x52/0x68
[ 347.644706]do_idle+0x118/0x170
[ 347.644707]cpu_startup_entry+0x3e/0x48
[ 347.644707]smp_start_secondary+0x112/0x120
[ 347.644708]restart_int_handler+0x62/0x78
[ 347.644708] (null)
[ 347.644709] -> #1 (&priv->wait_q){-.}:
[ 347.644711]validate_chain.isra.10+0xb56/0xd88
[ 347.644711]__lock_acquire+0x62c/0x850
[ 347.644712]lock_acquire+0x254/0x2b8
[ 347.644712]_raw_spin_lock_irqsave+0x70/0xb8
[ 347.644713]__wake_up+0x3a/0x68
[ 347.644713]ccw_device_recog_done+0x28e/0x2c8
[ 347.644714]snsid_callback+0x324/0x390
[ 347.644714]ccw_request_handler+0x480/0x560
[ 347.644715]do_cio_interrupt+0x224/0x2a0
[ 347.644715]__handle_irq_event_percpu+0x1a6/0x440
[ 347.644715]handle_irq_event_percpu+0x38/0x8
Re: lockdep warning: console vs. mem hotplug
On Wed 2017-03-29 09:31:47, Michal Hocko wrote: > On Tue 28-03-17 18:00:16, Petr Mladek wrote: > > On Tue 2017-03-28 16:22:27, Michal Hocko wrote: > > > On Sat 25-03-17 09:04:42, Sergey Senozhatsky wrote: > > > > On (03/21/17 13:44), Sergey Senozhatsky wrote: > > > > [..] > > > > > so we probably can > > > > > > > > > > > > > > > 1) move pr_info() out of zone->lock in __offline_isolated_pages(). > > > > >meh... > > > > > > > > > > > > > > > 2) switch to printk_deferred() in __offline_isolated_pages(). > > > > >meh.. there might a bunch of other printks done from under > > > > > zone->lock. > > > > > > > > > > > > > > > 3) move add_timer() out of sclp_con_lock console in > > > > > sclp_console_write(). > > > > >well, there can be other consoles that do something similar. > > > > > > > > > > > > > > > 4) ... something smart. > > > > > > > > > > Regarding the timer code. The problem seems to be with static > > timers. They call debug_object_init() when the timer is used > > for the first time. See the special handling of not-found > > objects in debug_object_activate(). > > Thanks for the clarification! > > > Now, __debug_object_init() calls fill_pool() that allocates > > the memory. A solution would be to either use static > > struct kmem_cache > > I am not sure what do you mean by that. The problem is when the timer is defined using DEFINE_TIMER() and initialized using __TIMER_INITIALIZER(). I had an unclear idea about extending the macro to define also the needed structure that might later be used by debug_object_init() instead of taking it from the pool. I am sorry for the confusion with struct kmem_cache. It is the type of the pointer that is passed when allocating the memory in fill_pool(). It seems we actually need struct debug_obj. But it might be even more complicated. > > for statically defined timers and avoid the allocation. Or we should > > call fill_pool() asynchronously from a safe context. > > I think we should avoid the allocation completely. It is GFP_ATOMIC and > so likely to fail under heavy memory pressure. Async fill will make it > slightly more complicated but still unreliable. > > > What do you think? > > Why cannot we simply embed this debugging data into the timer itself? > It will make the structure larger (I didn't check how much) but this is > an opt in feature so it should be acceptable. This sounds reasonable to me. > A subtle dependecny on the allocator is really bad and > we should get rid of it. Yup. Best Regards, Petr
Re: lockdep warning: console vs. mem hotplug
On (03/28/17 16:22), Michal Hocko wrote:
[..]
> > Sebastian, does this change make lockdep happy?
> >
> > it removes console drivers from the __offline_isolated_pages(). not the
> > best solution I can think of, but the simplest one.
> >
> > ---
> >
> > mm/page_alloc.c | 2 +-
> > 1 file changed, 1 insertion(+), 1 deletion(-)
> >
> > diff --git a/mm/page_alloc.c b/mm/page_alloc.c
> > index f749b7ff7c50..eb61e6ab5f4f 100644
> > --- a/mm/page_alloc.c
> > +++ b/mm/page_alloc.c
> > @@ -7705,7 +7705,7 @@ __offline_isolated_pages(unsigned long start_pfn,
> > unsigned long end_pfn)
> > BUG_ON(!PageBuddy(page));
> > order = page_order(page);
> > #ifdef CONFIG_DEBUG_VM
> > - pr_info("remove from free list %lx %d %lx\n",
> > + printk_deferred(KERN_INFO "remove from free list %lx %d %lx\n",
> > pfn, 1 << order, end_pfn);
> > #endif
> > list_del(&page->lru);
>
> I believe this is not a proper fix.
oh, absolutely. I hate it. didn't really propose it as a fix. mostly
did it just to verify that there are no other lock inversions behind
the one that has been reported (lockdep turns off itself when it
detects the first lock dependency inversion).
> Although this code is ugly and maybe it doesn't really need zone->lock
> because that should be the page allocator internal thing the problem is
> that printk shouldn't impose such a subtle dependency on locks. Why does
> the timer needs to allocate at all?
I believe Petr has answered your questions. sorry for the delay.
-ss
Re: lockdep warning: console vs. mem hotplug
On Tue 28-03-17 18:00:16, Petr Mladek wrote:
> On Tue 2017-03-28 16:22:27, Michal Hocko wrote:
> > On Sat 25-03-17 09:04:42, Sergey Senozhatsky wrote:
> > > On (03/21/17 13:44), Sergey Senozhatsky wrote:
> > > [..]
> > > > so we probably can
> > > >
> > > >
> > > > 1) move pr_info() out of zone->lock in __offline_isolated_pages().
> > > >meh...
> > > >
> > > >
> > > > 2) switch to printk_deferred() in __offline_isolated_pages().
> > > >meh.. there might a bunch of other printks done from under
> > > > zone->lock.
> > > >
> > > >
> > > > 3) move add_timer() out of sclp_con_lock console in
> > > > sclp_console_write().
> > > >well, there can be other consoles that do something similar.
> > > >
> > > >
> > > > 4) ... something smart.
> > >
> > >
> > > Sebastian, does this change make lockdep happy?
> > >
> > > it removes console drivers from the __offline_isolated_pages(). not the
> > > best solution I can think of, but the simplest one.
> > >
> > > ---
> > >
> > > mm/page_alloc.c | 2 +-
> > > 1 file changed, 1 insertion(+), 1 deletion(-)
> > >
> > > diff --git a/mm/page_alloc.c b/mm/page_alloc.c
> > > index f749b7ff7c50..eb61e6ab5f4f 100644
> > > --- a/mm/page_alloc.c
> > > +++ b/mm/page_alloc.c
> > > @@ -7705,7 +7705,7 @@ __offline_isolated_pages(unsigned long start_pfn,
> > > unsigned long end_pfn)
> > > BUG_ON(!PageBuddy(page));
> > > order = page_order(page);
> > > #ifdef CONFIG_DEBUG_VM
> > > - pr_info("remove from free list %lx %d %lx\n",
> > > + printk_deferred(KERN_INFO "remove from free list %lx %d %lx\n",
> > > pfn, 1 << order, end_pfn);
> > > #endif
> > > list_del(&page->lru);
> >
> > I believe this is not a proper fix. Although this code is ugly and maybe
> > it doesn't really need zone->lock because that should be the page
> > allocator internal thing the problem is that printk shouldn't impose
> > such a subtle dependency on locks. Why does the timer needs to allocate
> > at all?
>
> printk/console use timers to postpone flushing of buffers. There are
> often more consequent printks. The code wants to wait a bit and flush
> them together eventually. At the same time, it wants to set a deadline
> for the flushing. It makes sure that they will get flushed in a
> reasonable time even when the buffer is not full. It is questionable
> but it makes some sense.
>
> In each case, the timer code is used also by scheduler and we probably
> need to use scheduler from printk.
>
> Regarding the timer code. The problem seems to be with static
> timers. They call debug_object_init() when the timer is used
> for the first time. See the special handling of not-found
> objects in debug_object_activate().
Thanks for the clarification!
> Now, __debug_object_init() calls fill_pool() that allocates
> the memory. A solution would be to either use static
> struct kmem_cache
I am not sure what do you mean by that.
> for statically defined timers and avoid the allocation. Or we should
> call fill_pool() asynchronously from a safe context.
I think we should avoid the allocation completely. It is GFP_ATOMIC and
so likely to fail under heavy memory pressure. Async fill will make it
slightly more complicated but still unreliable.
> What do you think?
Why cannot we simply embed this debugging data into the timer itself?
It will make the structure larger (I didn't check how much) but this is
an opt in feature so it should be acceptable.
A subtle dependecny on the allocator is really bad and we should get rid
of it.
--
Michal Hocko
SUSE Labs
Re: lockdep warning: console vs. mem hotplug
On Tue 2017-03-28 16:22:27, Michal Hocko wrote:
> On Sat 25-03-17 09:04:42, Sergey Senozhatsky wrote:
> > On (03/21/17 13:44), Sergey Senozhatsky wrote:
> > [..]
> > > so we probably can
> > >
> > >
> > > 1) move pr_info() out of zone->lock in __offline_isolated_pages().
> > >meh...
> > >
> > >
> > > 2) switch to printk_deferred() in __offline_isolated_pages().
> > >meh.. there might a bunch of other printks done from under zone->lock.
> > >
> > >
> > > 3) move add_timer() out of sclp_con_lock console in sclp_console_write().
> > >well, there can be other consoles that do something similar.
> > >
> > >
> > > 4) ... something smart.
> >
> >
> > Sebastian, does this change make lockdep happy?
> >
> > it removes console drivers from the __offline_isolated_pages(). not the
> > best solution I can think of, but the simplest one.
> >
> > ---
> >
> > mm/page_alloc.c | 2 +-
> > 1 file changed, 1 insertion(+), 1 deletion(-)
> >
> > diff --git a/mm/page_alloc.c b/mm/page_alloc.c
> > index f749b7ff7c50..eb61e6ab5f4f 100644
> > --- a/mm/page_alloc.c
> > +++ b/mm/page_alloc.c
> > @@ -7705,7 +7705,7 @@ __offline_isolated_pages(unsigned long start_pfn,
> > unsigned long end_pfn)
> > BUG_ON(!PageBuddy(page));
> > order = page_order(page);
> > #ifdef CONFIG_DEBUG_VM
> > - pr_info("remove from free list %lx %d %lx\n",
> > + printk_deferred(KERN_INFO "remove from free list %lx %d %lx\n",
> > pfn, 1 << order, end_pfn);
> > #endif
> > list_del(&page->lru);
>
> I believe this is not a proper fix. Although this code is ugly and maybe
> it doesn't really need zone->lock because that should be the page
> allocator internal thing the problem is that printk shouldn't impose
> such a subtle dependency on locks. Why does the timer needs to allocate
> at all?
printk/console use timers to postpone flushing of buffers. There are
often more consequent printks. The code wants to wait a bit and flush
them together eventually. At the same time, it wants to set a deadline
for the flushing. It makes sure that they will get flushed in a
reasonable time even when the buffer is not full. It is questionable
but it makes some sense.
In each case, the timer code is used also by scheduler and we probably
need to use scheduler from printk.
Regarding the timer code. The problem seems to be with static
timers. They call debug_object_init() when the timer is used
for the first time. See the special handling of not-found
objects in debug_object_activate().
Now, __debug_object_init() calls fill_pool() that allocates
the memory. A solution would be to either use static
struct kmem_cache for statically defined timers and
avoid the allocation. Or we should call fill_pool()
asynchronously from a safe context.
What do you think?
Best Regards,
Petr
Re: lockdep warning: console vs. mem hotplug
On Sat 25-03-17 09:04:42, Sergey Senozhatsky wrote:
> On (03/21/17 13:44), Sergey Senozhatsky wrote:
> [..]
> > so we probably can
> >
> >
> > 1) move pr_info() out of zone->lock in __offline_isolated_pages().
> >meh...
> >
> >
> > 2) switch to printk_deferred() in __offline_isolated_pages().
> >meh.. there might a bunch of other printks done from under zone->lock.
> >
> >
> > 3) move add_timer() out of sclp_con_lock console in sclp_console_write().
> >well, there can be other consoles that do something similar.
> >
> >
> > 4) ... something smart.
>
>
> Sebastian, does this change make lockdep happy?
>
> it removes console drivers from the __offline_isolated_pages(). not the
> best solution I can think of, but the simplest one.
>
> ---
>
> mm/page_alloc.c | 2 +-
> 1 file changed, 1 insertion(+), 1 deletion(-)
>
> diff --git a/mm/page_alloc.c b/mm/page_alloc.c
> index f749b7ff7c50..eb61e6ab5f4f 100644
> --- a/mm/page_alloc.c
> +++ b/mm/page_alloc.c
> @@ -7705,7 +7705,7 @@ __offline_isolated_pages(unsigned long start_pfn,
> unsigned long end_pfn)
> BUG_ON(!PageBuddy(page));
> order = page_order(page);
> #ifdef CONFIG_DEBUG_VM
> - pr_info("remove from free list %lx %d %lx\n",
> + printk_deferred(KERN_INFO "remove from free list %lx %d %lx\n",
> pfn, 1 << order, end_pfn);
> #endif
> list_del(&page->lru);
I believe this is not a proper fix. Although this code is ugly and maybe
it doesn't really need zone->lock because that should be the page
allocator internal thing the problem is that printk shouldn't impose
such a subtle dependency on locks. Why does the timer needs to allocate
at all?
--
Michal Hocko
SUSE Labs
Re: lockdep warning: console vs. mem hotplug
On Sat, 25 Mar 2017, Sergey Senozhatsky wrote: > On (03/21/17 13:44), Sergey Senozhatsky wrote: > [..] > > so we probably can > > > > > > 1) move pr_info() out of zone->lock in __offline_isolated_pages(). > >meh... > > > > > > 2) switch to printk_deferred() in __offline_isolated_pages(). > >meh.. there might a bunch of other printks done from under zone->lock. > > > > > > 3) move add_timer() out of sclp_con_lock console in sclp_console_write(). > >well, there can be other consoles that do something similar. > > > > > > 4) ... something smart. > > > Sebastian, does this change make lockdep happy? > > it removes console drivers from the __offline_isolated_pages(). not the > best solution I can think of, but the simplest one. It does. No complaint from lockdep.
Re: lockdep warning: console vs. mem hotplug
On (03/24/17 21:08), Steven Rostedt wrote:
> > Sebastian, does this change make lockdep happy?
> >
> > it removes console drivers from the __offline_isolated_pages(). not the
> > best solution I can think of, but the simplest one.
> >
> > ---
> >
> > mm/page_alloc.c | 2 +-
> > 1 file changed, 1 insertion(+), 1 deletion(-)
> >
> > diff --git a/mm/page_alloc.c b/mm/page_alloc.c
> > index f749b7ff7c50..eb61e6ab5f4f 100644
> > --- a/mm/page_alloc.c
> > +++ b/mm/page_alloc.c
> > @@ -7705,7 +7705,7 @@ __offline_isolated_pages(unsigned long start_pfn,
> > unsigned long end_pfn)
> > BUG_ON(!PageBuddy(page));
> > order = page_order(page);
> > #ifdef CONFIG_DEBUG_VM
> > - pr_info("remove from free list %lx %d %lx\n",
> > + printk_deferred(KERN_INFO "remove from free list %lx %d %lx\n",
> > pfn, 1 << order, end_pfn);
> > #endif
> > list_del(&page->lru);
>
>
> My fear is that this will trigger for any printk in page_alloc.c under
> the zone lock.
absolutely true.
I Cc'd debugobjects, mm and sclp_console maintainers because the
real (smart) solution to the problem is somewhere there.
another problem (not reported) is that we have conflicting dependencies
mod_timer -> sclp_console
sclp_console -> mod_timer
which can result in a deadlock:
mod_timer -> debugobjects -> printk -> sclp_console -> mod_timer
this one, I think, can be addressed by switching to a printk_safe in
debugobjects.
// I'm traveling now. there will be delays in replies, sorry.
-ss
Re: lockdep warning: console vs. mem hotplug
On Sat, 25 Mar 2017 09:04:42 +0900
Sergey Senozhatsky wrote:
> On (03/21/17 13:44), Sergey Senozhatsky wrote:
> [..]
> > so we probably can
> >
> >
> > 1) move pr_info() out of zone->lock in __offline_isolated_pages().
> >meh...
> >
> >
> > 2) switch to printk_deferred() in __offline_isolated_pages().
> >meh.. there might a bunch of other printks done from under zone->lock.
> >
> >
> > 3) move add_timer() out of sclp_con_lock console in sclp_console_write().
> >well, there can be other consoles that do something similar.
> >
> >
> > 4) ... something smart.
>
>
> Sebastian, does this change make lockdep happy?
>
> it removes console drivers from the __offline_isolated_pages(). not the
> best solution I can think of, but the simplest one.
>
> ---
>
> mm/page_alloc.c | 2 +-
> 1 file changed, 1 insertion(+), 1 deletion(-)
>
> diff --git a/mm/page_alloc.c b/mm/page_alloc.c
> index f749b7ff7c50..eb61e6ab5f4f 100644
> --- a/mm/page_alloc.c
> +++ b/mm/page_alloc.c
> @@ -7705,7 +7705,7 @@ __offline_isolated_pages(unsigned long start_pfn,
> unsigned long end_pfn)
> BUG_ON(!PageBuddy(page));
> order = page_order(page);
> #ifdef CONFIG_DEBUG_VM
> - pr_info("remove from free list %lx %d %lx\n",
> + printk_deferred(KERN_INFO "remove from free list %lx %d %lx\n",
> pfn, 1 << order, end_pfn);
> #endif
> list_del(&page->lru);
My fear is that this will trigger for any printk in page_alloc.c under
the zone lock.
-- Steve
Re: lockdep warning: console vs. mem hotplug
On Sat, 25 Mar 2017 09:00:05 +0900
Sergey Senozhatsky wrote:
> Hello,
>
> On (03/24/17 12:39), Steven Rostedt wrote:
> [..]
> > Is there a stack trace of where the lockdep dump happened? That is
> > useful too. Otherwise we don't see where the inverse happened.
>
> Steven, isn't it the inversion I describe in [1] (after the first lockdep
> warning)?
>
> [1] lkml.kernel.org/r/20170321044421.GB448@jagdpanzerIV.localdomain
>
Yeah, I believe you are right. I just wanted to make sure. It's the
same backtrace as the "(&(&sclp_con_lock)->rlock){-.-...}:" dump, but I
wanted to make sure.
-- Steve
Re: lockdep warning: console vs. mem hotplug
On (03/21/17 13:44), Sergey Senozhatsky wrote:
[..]
> so we probably can
>
>
> 1) move pr_info() out of zone->lock in __offline_isolated_pages().
>meh...
>
>
> 2) switch to printk_deferred() in __offline_isolated_pages().
>meh.. there might a bunch of other printks done from under zone->lock.
>
>
> 3) move add_timer() out of sclp_con_lock console in sclp_console_write().
>well, there can be other consoles that do something similar.
>
>
> 4) ... something smart.
Sebastian, does this change make lockdep happy?
it removes console drivers from the __offline_isolated_pages(). not the
best solution I can think of, but the simplest one.
---
mm/page_alloc.c | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
diff --git a/mm/page_alloc.c b/mm/page_alloc.c
index f749b7ff7c50..eb61e6ab5f4f 100644
--- a/mm/page_alloc.c
+++ b/mm/page_alloc.c
@@ -7705,7 +7705,7 @@ __offline_isolated_pages(unsigned long start_pfn,
unsigned long end_pfn)
BUG_ON(!PageBuddy(page));
order = page_order(page);
#ifdef CONFIG_DEBUG_VM
- pr_info("remove from free list %lx %d %lx\n",
+ printk_deferred(KERN_INFO "remove from free list %lx %d %lx\n",
pfn, 1 << order, end_pfn);
#endif
list_del(&page->lru);
Re: lockdep warning: console vs. mem hotplug
Hello, On (03/24/17 12:39), Steven Rostedt wrote: [..] > Is there a stack trace of where the lockdep dump happened? That is > useful too. Otherwise we don't see where the inverse happened. Steven, isn't it the inversion I describe in [1] (after the first lockdep warning)? [1] lkml.kernel.org/r/20170321044421.GB448@jagdpanzerIV.localdomain -ss
Re: lockdep warning: console vs. mem hotplug
On Fri, 24 Mar 2017, Steven Rostedt wrote:
> Is there a stack trace of where the lockdep dump happened? That is
> useful too. Otherwise we don't see where the inverse happened.
Sure. Here you go.
[ 132.379220] ==
[ 132.379220] [ INFO: possible circular locking dependency detected ]
[ 132.379221] 4.11.0-rc3 #330 Not tainted
[ 132.379221] ---
[ 132.379222] sh/708 is trying to acquire lock:
[ 132.379222] (&(&sclp_con_lock)->rlock){-.-...}, at: [<0074524a>]
sclp_console_write+0x4a/0x2b8
[ 132.379224]
[ 132.379224] but task is already holding lock:
[ 132.379224] (&(&zone->lock)->rlock){..-...}, at: [<002a65a8>]
__offline_isolated_pages+0xe0/0x340
[ 132.379226]
[ 132.379227] which lock already depends on the new lock.
[ 132.379227]
[ 132.379227]
[ 132.379228] the existing dependency chain (in reverse order) is:
[ 132.379228]
[ 132.379229] -> #2 (&(&zone->lock)->rlock){..-...}:
[ 132.379230]validate_chain.isra.22+0xb36/0xd18
[ 132.379231]__lock_acquire+0x650/0x880
[ 132.379231]lock_acquire+0x24e/0x2b0
[ 132.379232]_raw_spin_lock_irqsave+0x70/0xb8
[ 132.379232]get_page_from_freelist+0x548/0xf30
[ 132.379233]__alloc_pages_nodemask+0x1c8/0x14c0
[ 132.379233]allocate_slab+0xde/0x668
[ 132.379233]new_slab+0x94/0xa8
[ 132.379234]___slab_alloc.constprop.66+0x556/0x578
[ 132.379234]__slab_alloc.isra.62.constprop.65+0x74/0xa8
[ 132.379235]kmem_cache_alloc+0x136/0x4b8
[ 132.379235]__debug_object_init+0xaa/0x448
[ 132.379236]debug_object_activate+0x182/0x238
[ 132.379236]mod_timer+0xe6/0x450
[ 132.379236]prandom_reseed+0x44/0x50
[ 132.379237]do_one_initcall+0xa2/0x150
[ 132.379237]kernel_init_freeable+0x228/0x2d8
[ 132.379238]kernel_init+0x2a/0x138
[ 132.379238]kernel_thread_starter+0x6/0xc
[ 132.379239]kernel_thread_starter+0x0/0xc
[ 132.379239]
[ 132.379239] -> #1 (&(&base->lock)->rlock){-.-.-.}:
[ 132.379241]validate_chain.isra.22+0xb36/0xd18
[ 132.379242]__lock_acquire+0x650/0x880
[ 132.379242]lock_acquire+0x24e/0x2b0
[ 132.379242]_raw_spin_lock_irqsave+0x70/0xb8
[ 132.379243]lock_timer_base+0x78/0xa8
[ 132.379243]mod_timer+0xba/0x450
[ 132.379244]sclp_console_write+0x262/0x2b8
[ 132.379244]console_unlock+0x4a6/0x658
[ 132.379245]register_console+0x31c/0x430
[ 132.379245]sclp_console_init+0x152/0x170
[ 132.379245]console_init+0x3a/0x58
[ 132.379246]start_kernel+0x350/0x480
[ 132.379246]_stext+0x20/0x80
[ 132.379246]
[ 132.379247] -> #0 (&(&sclp_con_lock)->rlock){-.-...}:
[ 132.379249]check_prev_add+0x150/0x648
[ 132.379249]validate_chain.isra.22+0xb36/0xd18
[ 132.379249]__lock_acquire+0x650/0x880
[ 132.379250]lock_acquire+0x24e/0x2b0
[ 132.379250]_raw_spin_lock_irqsave+0x70/0xb8
[ 132.379251]sclp_console_write+0x4a/0x2b8
[ 132.379251]console_unlock+0x4a6/0x658
[ 132.379252]vprintk_emit+0x33e/0x350
[ 132.379252]vprintk_default+0x44/0x58
[ 132.379252]printk+0x4e/0x58
[ 132.379253]__offline_isolated_pages+0x1a0/0x340
[ 132.379253]offline_isolated_pages_cb+0x2e/0x40
[ 132.379254]walk_system_ram_range+0x96/0xe8
[ 132.379254]__offline_pages.constprop.43+0x6da/0x918
[ 132.379255]memory_subsys_offline+0x6a/0xa0
[ 132.379255]device_offline+0x84/0xd8
[ 132.379255]store_mem_state+0xfe/0x120
[ 132.379256]kernfs_fop_write+0x126/0x1d0
[ 132.379256]__vfs_write+0x46/0x140
[ 132.379257]vfs_write+0xb8/0x1a8
[ 132.379257]SyS_write+0x66/0xc0
[ 132.379257]system_call+0xc4/0x240
[ 132.379258]
[ 132.379258] other info that might help us debug this:
[ 132.379258]
[ 132.379259] Chain exists of:
[ 132.379259] &(&sclp_con_lock)->rlock --> &(&base->lock)->rlock -->
&(&zone->lock)->rlock
[ 132.379262]
[ 132.379262] Possible unsafe locking scenario:
[ 132.379262]
[ 132.379263]CPU0CPU1
[ 132.379263]
[ 132.379264] lock(&(&zone->lock)->rlock);
[ 132.379265]lock(&(&base->lock)->rlock);
[ 132.379266]lock(&(&zone->lock)->rlock);
[ 132.379267] lock(&(&sclp_con_lock)->rlock);
[ 132.379268]
[ 132.379268] *** DEADLOCK ***
[ 132.379269]
[ 132.379269] 10 locks held by sh/708:
[ 132.379269] #0: (sb_writers#4){.+.+.+}, at: [<00345d4a>]
vfs_write+0xa2/0x1a8
[ 132.379272] #1: (&of->mutex){+.+.+.}, at: [<003e5e60>]
kernfs_fop_write+0xd8/0x1d0
[ 132.379273] #2: (s_active#180){.+.+.+}, at: [<003e5e6c>]
kernfs_fop_write+0xe4/0x1d0
[ 132.37
Re: lockdep warning: console vs. mem hotplug
On Mon, 20 Mar 2017 17:43:23 +0100 (CET)
Sebastian Ott wrote:
> Since commit f975237b7682 ("printk: use printk_safe buffers in printk")
> I observe lockdep warnings on s390 when doing memory hotplug:
>
> [ 132.379220] ==
> [ 132.379220] [ INFO: possible circular locking dependency detected ]
> [ 132.379221] 4.11.0-rc3 #330 Not tainted
> [ 132.379221] ---
> [ 132.379222] sh/708 is trying to acquire lock:
> [ 132.379222] (&(&sclp_con_lock)->rlock){-.-...}, at: [<0074524a>]
> sclp_console_write+0x4a/0x2b8
> [ 132.379224]
> [ 132.379224] but task is already holding lock:
> [ 132.379224] (&(&zone->lock)->rlock){..-...}, at: [<002a65a8>]
> __offline_isolated_pages+0xe0/0x340
> [ 132.379226]
> [ 132.379227] which lock already depends on the new lock.
> [ 132.379227]
> [ 132.379227]
> [ 132.379228] the existing dependency chain (in reverse order) is:
> [ 132.379228]
> [ 132.379229] -> #2 (&(&zone->lock)->rlock){..-...}:
> [ 132.379230]validate_chain.isra.22+0xb36/0xd18
> [ 132.379231]__lock_acquire+0x650/0x880
> [ 132.379231]lock_acquire+0x24e/0x2b0
> [ 132.379232]_raw_spin_lock_irqsave+0x70/0xb8
> [ 132.379232]get_page_from_freelist+0x548/0xf30
> [ 132.379233]__alloc_pages_nodemask+0x1c8/0x14c0
> [ 132.379233]allocate_slab+0xde/0x668
> [ 132.379233]new_slab+0x94/0xa8
> [ 132.379234]___slab_alloc.constprop.66+0x556/0x578
> [ 132.379234]__slab_alloc.isra.62.constprop.65+0x74/0xa8
> [ 132.379235]kmem_cache_alloc+0x136/0x4b8
> [ 132.379235]__debug_object_init+0xaa/0x448
> [ 132.379236]debug_object_activate+0x182/0x238
> [ 132.379236]mod_timer+0xe6/0x450
> [ 132.379236]prandom_reseed+0x44/0x50
> [ 132.379237]do_one_initcall+0xa2/0x150
> [ 132.379237]kernel_init_freeable+0x228/0x2d8
> [ 132.379238]kernel_init+0x2a/0x138
> [ 132.379238]kernel_thread_starter+0x6/0xc
> [ 132.379239]kernel_thread_starter+0x0/0xc
> [ 132.379239]
> [ 132.379239] -> #1 (&(&base->lock)->rlock){-.-.-.}:
> [ 132.379241]validate_chain.isra.22+0xb36/0xd18
> [ 132.379242]__lock_acquire+0x650/0x880
> [ 132.379242]lock_acquire+0x24e/0x2b0
> [ 132.379242]_raw_spin_lock_irqsave+0x70/0xb8
> [ 132.379243]lock_timer_base+0x78/0xa8
> [ 132.379243]mod_timer+0xba/0x450
> [ 132.379244]sclp_console_write+0x262/0x2b8
> [ 132.379244]console_unlock+0x4a6/0x658
> [ 132.379245]register_console+0x31c/0x430
> [ 132.379245]sclp_console_init+0x152/0x170
> [ 132.379245]console_init+0x3a/0x58
> [ 132.379246]start_kernel+0x350/0x480
> [ 132.379246]_stext+0x20/0x80
> [ 132.379246]
> [ 132.379247] -> #0 (&(&sclp_con_lock)->rlock){-.-...}:
> [ 132.379249]check_prev_add+0x150/0x648
> [ 132.379249]validate_chain.isra.22+0xb36/0xd18
> [ 132.379249]__lock_acquire+0x650/0x880
> [ 132.379250]lock_acquire+0x24e/0x2b0
> [ 132.379250]_raw_spin_lock_irqsave+0x70/0xb8
> [ 132.379251]sclp_console_write+0x4a/0x2b8
> [ 132.379251]console_unlock+0x4a6/0x658
> [ 132.379252]vprintk_emit+0x33e/0x350
> [ 132.379252]vprintk_default+0x44/0x58
> [ 132.379252]printk+0x4e/0x58
> [ 132.379253]__offline_isolated_pages+0x1a0/0x340
> [ 132.379253]offline_isolated_pages_cb+0x2e/0x40
> [ 132.379254]walk_system_ram_range+0x96/0xe8
> [ 132.379254]__offline_pages.constprop.43+0x6da/0x918
> [ 132.379255]memory_subsys_offline+0x6a/0xa0
> [ 132.379255]device_offline+0x84/0xd8
> [ 132.379255]store_mem_state+0xfe/0x120
> [ 132.379256]kernfs_fop_write+0x126/0x1d0
> [ 132.379256]__vfs_write+0x46/0x140
> [ 132.379257]vfs_write+0xb8/0x1a8
> [ 132.379257]SyS_write+0x66/0xc0
> [ 132.379257]system_call+0xc4/0x240
> [ 132.379258]
> [ 132.379258] other info that might help us debug this:
> [ 132.379258]
> [ 132.379259] Chain exists of:
> [ 132.379259] &(&sclp_con_lock)->rlock --> &(&base->lock)->rlock -->
> &(&zone->lock)->rlock
> [ 132.379262]
> [ 132.379262] Possible unsafe locking scenario:
> [ 132.379262]
> [ 132.379263]CPU0CPU1
> [ 132.379263]
> [ 132.379264] lock(&(&zone->lock)->rlock);
> [ 132.379265]lock(&(&base->lock)->rlock);
> [ 132.379266]lock(&(&zone->lock)->rlock);
> [ 132.379267] lock(&(&sclp_con_lock)->rlock);
> [ 132.379268]
> [ 132.379268] *** DEADLOCK ***
> [ 132.379269]
> [ 132.379269] 10 locks held by sh/708:
> [ 132.379269] #0: (sb_writers#4){.+.+.+}, at: [<00345d4a>]
> vfs_write+0xa2/0x1a8
> [ 132.379272
Re: lockdep warning: console vs. mem hotplug
[..] > 3) move add_timer() out of sclp_con_lock console in sclp_console_write(). I take it back. Won't improve anything. -ss
Re: lockdep warning: console vs. mem hotplug
Cc Michal, Linus, Peter, Thomas, Andrew, Martin
// I kept only lockdep splats from the original report and cut off .configs
// full version:
// lkml.kernel.org/r/alpine.LFD.2.20.1703201736070.1753@schleppi
Sebastian, thanks.
On (03/20/17 17:43), Sebastian Ott wrote:
[..]
> Since commit f975237b7682 ("printk: use printk_safe buffers in printk")
> I observe lockdep warnings on s390 when doing memory hotplug:
>
> [ 132.379220] ==
> [ 132.379220] [ INFO: possible circular locking dependency detected ]
> [ 132.379221] 4.11.0-rc3 #330 Not tainted
> [ 132.379221] ---
> [ 132.379222] sh/708 is trying to acquire lock:
> [ 132.379222] (&(&sclp_con_lock)->rlock){-.-...}, at: [<0074524a>]
> sclp_console_write+0x4a/0x2b8
> [ 132.379224]
> [ 132.379224] but task is already holding lock:
> [ 132.379224] (&(&zone->lock)->rlock){..-...}, at: [<002a65a8>]
> __offline_isolated_pages+0xe0/0x340
> [ 132.379226]
> [ 132.379227] which lock already depends on the new lock.
> [ 132.379227]
> [ 132.379227]
> [ 132.379228] the existing dependency chain (in reverse order) is:
> [ 132.379228]
> [ 132.379229] -> #2 (&(&zone->lock)->rlock){..-...}:
> [ 132.379230]validate_chain.isra.22+0xb36/0xd18
> [ 132.379231]__lock_acquire+0x650/0x880
> [ 132.379231]lock_acquire+0x24e/0x2b0
> [ 132.379232]_raw_spin_lock_irqsave+0x70/0xb8
> [ 132.379232]get_page_from_freelist+0x548/0xf30
> [ 132.379233]__alloc_pages_nodemask+0x1c8/0x14c0
> [ 132.379233]allocate_slab+0xde/0x668
> [ 132.379233]new_slab+0x94/0xa8
> [ 132.379234]___slab_alloc.constprop.66+0x556/0x578
> [ 132.379234]__slab_alloc.isra.62.constprop.65+0x74/0xa8
> [ 132.379235]kmem_cache_alloc+0x136/0x4b8
> [ 132.379235]__debug_object_init+0xaa/0x448
> [ 132.379236]debug_object_activate+0x182/0x238
> [ 132.379236]mod_timer+0xe6/0x450
> [ 132.379236]prandom_reseed+0x44/0x50
> [ 132.379237]do_one_initcall+0xa2/0x150
> [ 132.379237]kernel_init_freeable+0x228/0x2d8
> [ 132.379238]kernel_init+0x2a/0x138
> [ 132.379238]kernel_thread_starter+0x6/0xc
> [ 132.379239]kernel_thread_starter+0x0/0xc
> [ 132.379239]
> [ 132.379239] -> #1 (&(&base->lock)->rlock){-.-.-.}:
> [ 132.379241]validate_chain.isra.22+0xb36/0xd18
> [ 132.379242]__lock_acquire+0x650/0x880
> [ 132.379242]lock_acquire+0x24e/0x2b0
> [ 132.379242]_raw_spin_lock_irqsave+0x70/0xb8
> [ 132.379243]lock_timer_base+0x78/0xa8
> [ 132.379243]mod_timer+0xba/0x450
> [ 132.379244]sclp_console_write+0x262/0x2b8
> [ 132.379244]console_unlock+0x4a6/0x658
> [ 132.379245]register_console+0x31c/0x430
> [ 132.379245]sclp_console_init+0x152/0x170
> [ 132.379245]console_init+0x3a/0x58
> [ 132.379246]start_kernel+0x350/0x480
> [ 132.379246]_stext+0x20/0x80
> [ 132.379246]
> [ 132.379247] -> #0 (&(&sclp_con_lock)->rlock){-.-...}:
> [ 132.379249]check_prev_add+0x150/0x648
> [ 132.379249]validate_chain.isra.22+0xb36/0xd18
> [ 132.379249]__lock_acquire+0x650/0x880
> [ 132.379250]lock_acquire+0x24e/0x2b0
> [ 132.379250]_raw_spin_lock_irqsave+0x70/0xb8
> [ 132.379251]sclp_console_write+0x4a/0x2b8
> [ 132.379251]console_unlock+0x4a6/0x658
> [ 132.379252]vprintk_emit+0x33e/0x350
> [ 132.379252]vprintk_default+0x44/0x58
> [ 132.379252]printk+0x4e/0x58
> [ 132.379253]__offline_isolated_pages+0x1a0/0x340
> [ 132.379253]offline_isolated_pages_cb+0x2e/0x40
> [ 132.379254]walk_system_ram_range+0x96/0xe8
> [ 132.379254]__offline_pages.constprop.43+0x6da/0x918
> [ 132.379255]memory_subsys_offline+0x6a/0xa0
> [ 132.379255]device_offline+0x84/0xd8
> [ 132.379255]store_mem_state+0xfe/0x120
> [ 132.379256]kernfs_fop_write+0x126/0x1d0
> [ 132.379256]__vfs_write+0x46/0x140
> [ 132.379257]vfs_write+0xb8/0x1a8
> [ 132.379257]SyS_write+0x66/0xc0
> [ 132.379257]system_call+0xc4/0x240
> [ 132.379258]
> [ 132.379258] other info that might help us debug this:
> [ 132.379258]
> [ 132.379259] Chain exists of:
> [ 132.379259] &(&sclp_con_lock)->rlock --> &(&base->lock)->rlock -->
> &(&zone->lock)->rlock
> [ 132.379262]
> [ 132.379262] Possible unsafe locking scenario:
> [ 132.379262]
> [ 132.379263]CPU0CPU1
> [ 132.379263]
> [ 132.379264] lock(&(&zone->lock)->rlock);
> [ 132.379265]lock(&(&base->lock)->rlock);
> [ 132.379266]lock(&(&zone->lock)->rlock);
> [ 132.379267] lock(&(&sclp_con_lock)->rlock);
after a
