Re: [Query] Preemption (hogging) of the work handler

On (07/29/16 13:42), Viresh Kumar wrote: [..] > I haven't seen any issues with it yet and it works just fine. Please feel free > to add below for the entire series including this hunk. > > Tested-by: Viresh Kumar Thanks a lot! will refresh the series next week. sorry,

Re: [Query] Preemption (hogging) of the work handler

On (07/29/16 13:42), Viresh Kumar wrote: [..] > I haven't seen any issues with it yet and it works just fine. Please feel free > to add below for the entire series including this hunk. > > Tested-by: Viresh Kumar Thanks a lot! will refresh the series next week. sorry, haven't gotten many

Re: [Query] Preemption (hogging) of the work handler

On 13-07-16, 14:45, Sergey Senozhatsky wrote: > something like below, perhaps. will this work for you? > > --- > kernel/printk/printk.c | 12 +++- > 1 file changed, 11 insertions(+), 1 deletion(-) > > diff --git a/kernel/printk/printk.c b/kernel/printk/printk.c > index bbb4180..786690e

Re: [Query] Preemption (hogging) of the work handler

On 13-07-16, 14:45, Sergey Senozhatsky wrote: > something like below, perhaps. will this work for you? > > --- > kernel/printk/printk.c | 12 +++- > 1 file changed, 11 insertions(+), 1 deletion(-) > > diff --git a/kernel/printk/printk.c b/kernel/printk/printk.c > index bbb4180..786690e

Re: [Query] Preemption (hogging) of the work handler

On Monday, July 18, 2016 01:01:34 PM Jan Kara wrote: > On Thu 14-07-16 15:12:51, Viresh Kumar wrote: > > On 14-07-16, 16:12, Jan Kara wrote: > > > Exactly. Calling printk() from certain parts of the kernel (like scheduler > > > code or timer code) has been always unsafe because printk itself uses

Re: [Query] Preemption (hogging) of the work handler

On Monday, July 18, 2016 01:01:34 PM Jan Kara wrote: > On Thu 14-07-16 15:12:51, Viresh Kumar wrote: > > On 14-07-16, 16:12, Jan Kara wrote: > > > Exactly. Calling printk() from certain parts of the kernel (like scheduler > > > code or timer code) has been always unsafe because printk itself uses

Re: [Query] Preemption (hogging) of the work handler

On Thu 14-07-16 15:12:51, Viresh Kumar wrote: > On 14-07-16, 16:12, Jan Kara wrote: > > Exactly. Calling printk() from certain parts of the kernel (like scheduler > > code or timer code) has been always unsafe because printk itself uses these > > parts and so it can lead to deadlocks. That's why

Re: [Query] Preemption (hogging) of the work handler

On Thu 14-07-16 15:12:51, Viresh Kumar wrote: > On 14-07-16, 16:12, Jan Kara wrote: > > Exactly. Calling printk() from certain parts of the kernel (like scheduler > > code or timer code) has been always unsafe because printk itself uses these > > parts and so it can lead to deadlocks. That's why

Re: [Query] Preemption (hogging) of the work handler

On 15-07-16, 22:11, Sergey Senozhatsky wrote: > Hello, > > On (07/14/16 16:52), Viresh Kumar wrote: > > On 12-07-16, 23:03, Sergey Senozhatsky wrote: > > > so, I'm looking at this thing now: > > > > > > : [ 12.874909] sched: RT throttling activated for rt_rq > > > ffc0ac13fcd0 (cpu 0) > >

Re: [Query] Preemption (hogging) of the work handler

On 15-07-16, 22:11, Sergey Senozhatsky wrote: > Hello, > > On (07/14/16 16:52), Viresh Kumar wrote: > > On 12-07-16, 23:03, Sergey Senozhatsky wrote: > > > so, I'm looking at this thing now: > > > > > > : [ 12.874909] sched: RT throttling activated for rt_rq > > > ffc0ac13fcd0 (cpu 0) > >

Re: [Query] Preemption (hogging) of the work handler

Hello, On (07/14/16 16:52), Viresh Kumar wrote: > On 12-07-16, 23:03, Sergey Senozhatsky wrote: > > so, I'm looking at this thing now: > > > > : [ 12.874909] sched: RT throttling activated for rt_rq ffc0ac13fcd0 > > (cpu 0) > > : [ 12.874909] potential CPU hogs: > > : [ 12.874909]

Re: [Query] Preemption (hogging) of the work handler

Hello, On (07/14/16 16:52), Viresh Kumar wrote: > On 12-07-16, 23:03, Sergey Senozhatsky wrote: > > so, I'm looking at this thing now: > > > > : [ 12.874909] sched: RT throttling activated for rt_rq ffc0ac13fcd0 > > (cpu 0) > > : [ 12.874909] potential CPU hogs: > > : [ 12.874909]

Re: [Query] Preemption (hogging) of the work handler

Sorry, but I failed to do any testing on this and answer the questions you raise. But I saw this again today and here are some important points. On 12-07-16, 23:03, Sergey Senozhatsky wrote: > so, I'm looking at this thing now: > > : [ 12.874909] sched: RT throttling activated for rt_rq

Re: [Query] Preemption (hogging) of the work handler

Sorry, but I failed to do any testing on this and answer the questions you raise. But I saw this again today and here are some important points. On 12-07-16, 23:03, Sergey Senozhatsky wrote: > so, I'm looking at this thing now: > > : [ 12.874909] sched: RT throttling activated for rt_rq

Re: [Query] Preemption (hogging) of the work handler

On 14-07-16, 16:55, Jan Kara wrote: > Agree on that - but that seems to be a problem of a particular wakeup > implementation of the 3.10 kernel Viresh is using, not a problem of the > upstream kernel. I think we can get it to trigger on mainline as well. Also to mention that I also don't see it

Re: [Query] Preemption (hogging) of the work handler

On 14-07-16, 16:55, Jan Kara wrote: > Agree on that - but that seems to be a problem of a particular wakeup > implementation of the 3.10 kernel Viresh is using, not a problem of the > upstream kernel. I think we can get it to trigger on mainline as well. Also to mention that I also don't see it

Re: [Query] Preemption (hogging) of the work handler

On 14-07-16, 16:12, Jan Kara wrote: > Exactly. Calling printk() from certain parts of the kernel (like scheduler > code or timer code) has been always unsafe because printk itself uses these > parts and so it can lead to deadlocks. That's why printk_deffered() has > been introduced as you mention

Re: [Query] Preemption (hogging) of the work handler

On 14-07-16, 16:12, Jan Kara wrote: > Exactly. Calling printk() from certain parts of the kernel (like scheduler > code or timer code) has been always unsafe because printk itself uses these > parts and so it can lead to deadlocks. That's why printk_deffered() has > been introduced as you mention

Re: [Query] Preemption (hogging) of the work handler

On 14-07-16, 10:32, Sergey Senozhatsky wrote: > it wouldn't really, this silly question was not directly related to the > deadlock we are discussing here but to Viresh's argument that later stages > of suspending/hibernation seem to printk many messages in sync mode. so I > thought that there

Re: [Query] Preemption (hogging) of the work handler

On 14-07-16, 10:32, Sergey Senozhatsky wrote: > it wouldn't really, this silly question was not directly related to the > deadlock we are discussing here but to Viresh's argument that later stages > of suspending/hibernation seem to printk many messages in sync mode. so I > thought that there

Re: [Query] Preemption (hogging) of the work handler

On 14-07-16, 09:55, Sergey Senozhatsky wrote: > excessive printing is just part of the problem here. if we cab cond_resched() > part of suspend/hibernation is cpu_down(), which lands in > console_cpu_notify(), > that does synchronous printing for every CPU taken down: > > static int

Re: [Query] Preemption (hogging) of the work handler

On 14-07-16, 09:55, Sergey Senozhatsky wrote: > excessive printing is just part of the problem here. if we cab cond_resched() > part of suspend/hibernation is cpu_down(), which lands in > console_cpu_notify(), > that does synchronous printing for every CPU taken down: > > static int

Re: [Query] Preemption (hogging) of the work handler

On Thu 14-07-16 23:34:50, Sergey Senozhatsky wrote: > Hello Jan, > > On (07/14/16 16:12), Jan Kara wrote: > [..] > > > *** a printk() call from here will kill the system. either it will > > > recurse printk(), or spin forever in 'nested' printk() on one of > > > the already taken spin locks. >

Re: [Query] Preemption (hogging) of the work handler

On Thu 14-07-16 23:34:50, Sergey Senozhatsky wrote: > Hello Jan, > > On (07/14/16 16:12), Jan Kara wrote: > [..] > > > *** a printk() call from here will kill the system. either it will > > > recurse printk(), or spin forever in 'nested' printk() on one of > > > the already taken spin locks. >

Re: [Query] Preemption (hogging) of the work handler

On Thu 14-07-16 16:47:11, Rafael J. Wysocki wrote: > On Thursday, July 14, 2016 04:39:39 PM Jan Kara wrote: > > On Thu 14-07-16 16:33:38, Rafael J. Wysocki wrote: > > > On Thursday, July 14, 2016 04:12:16 PM Jan Kara wrote: > > > > On Wed 13-07-16 14:45:07, Sergey Senozhatsky wrote: > > > > > Cc

Re: [Query] Preemption (hogging) of the work handler

On Thu 14-07-16 16:47:11, Rafael J. Wysocki wrote: > On Thursday, July 14, 2016 04:39:39 PM Jan Kara wrote: > > On Thu 14-07-16 16:33:38, Rafael J. Wysocki wrote: > > > On Thursday, July 14, 2016 04:12:16 PM Jan Kara wrote: > > > > On Wed 13-07-16 14:45:07, Sergey Senozhatsky wrote: > > > > > Cc

Re: [Query] Preemption (hogging) of the work handler

On Thursday, July 14, 2016 04:39:39 PM Jan Kara wrote: > On Thu 14-07-16 16:33:38, Rafael J. Wysocki wrote: > > On Thursday, July 14, 2016 04:12:16 PM Jan Kara wrote: > > > On Wed 13-07-16 14:45:07, Sergey Senozhatsky wrote: > > > > Cc Petr Mladek. > > > > > > > > On (07/12/16 16:19), Viresh

Re: [Query] Preemption (hogging) of the work handler

On Thursday, July 14, 2016 04:39:39 PM Jan Kara wrote: > On Thu 14-07-16 16:33:38, Rafael J. Wysocki wrote: > > On Thursday, July 14, 2016 04:12:16 PM Jan Kara wrote: > > > On Wed 13-07-16 14:45:07, Sergey Senozhatsky wrote: > > > > Cc Petr Mladek. > > > > > > > > On (07/12/16 16:19), Viresh

Re: [Query] Preemption (hogging) of the work handler

On Thu 14-07-16 16:33:38, Rafael J. Wysocki wrote: > On Thursday, July 14, 2016 04:12:16 PM Jan Kara wrote: > > On Wed 13-07-16 14:45:07, Sergey Senozhatsky wrote: > > > Cc Petr Mladek. > > > > > > On (07/12/16 16:19), Viresh Kumar wrote: > > > [..] > > > > Okay, we have tracked this BUG and its

Re: [Query] Preemption (hogging) of the work handler

On Thu 14-07-16 16:33:38, Rafael J. Wysocki wrote: > On Thursday, July 14, 2016 04:12:16 PM Jan Kara wrote: > > On Wed 13-07-16 14:45:07, Sergey Senozhatsky wrote: > > > Cc Petr Mladek. > > > > > > On (07/12/16 16:19), Viresh Kumar wrote: > > > [..] > > > > Okay, we have tracked this BUG and its

Re: [Query] Preemption (hogging) of the work handler

Hello Jan, On (07/14/16 16:12), Jan Kara wrote: [..] > > *** a printk() call from here will kill the system. either it will > > recurse printk(), or spin forever in 'nested' printk() on one of > > the already taken spin locks. [..] > And with sync printk the above deadlock doesn't trigger only by

Re: [Query] Preemption (hogging) of the work handler

Hello Jan, On (07/14/16 16:12), Jan Kara wrote: [..] > > *** a printk() call from here will kill the system. either it will > > recurse printk(), or spin forever in 'nested' printk() on one of > > the already taken spin locks. [..] > And with sync printk the above deadlock doesn't trigger only by

Re: [Query] Preemption (hogging) of the work handler

On Thursday, July 14, 2016 04:12:16 PM Jan Kara wrote: > On Wed 13-07-16 14:45:07, Sergey Senozhatsky wrote: > > Cc Petr Mladek. > > > > On (07/12/16 16:19), Viresh Kumar wrote: > > [..] > > > Okay, we have tracked this BUG and its really interesting. > > > > good find! > > > > > I hacked the

Re: [Query] Preemption (hogging) of the work handler

On Thursday, July 14, 2016 04:12:16 PM Jan Kara wrote: > On Wed 13-07-16 14:45:07, Sergey Senozhatsky wrote: > > Cc Petr Mladek. > > > > On (07/12/16 16:19), Viresh Kumar wrote: > > [..] > > > Okay, we have tracked this BUG and its really interesting. > > > > good find! > > > > > I hacked the

Re: [Query] Preemption (hogging) of the work handler

On Wed 13-07-16 14:45:07, Sergey Senozhatsky wrote: > Cc Petr Mladek. > > On (07/12/16 16:19), Viresh Kumar wrote: > [..] > > Okay, we have tracked this BUG and its really interesting. > > good find! > > > I hacked the platform's serial driver to implement a putchar() routine > > that simply

Re: [Query] Preemption (hogging) of the work handler

On Wed 13-07-16 14:45:07, Sergey Senozhatsky wrote: > Cc Petr Mladek. > > On (07/12/16 16:19), Viresh Kumar wrote: > [..] > > Okay, we have tracked this BUG and its really interesting. > > good find! > > > I hacked the platform's serial driver to implement a putchar() routine > > that simply

Re: [Query] Preemption (hogging) of the work handler

On (07/14/16 03:09), Rafael J. Wysocki wrote: > On Thu, Jul 14, 2016 at 2:55 AM, Sergey Senozhatsky > wrote: > > Hello, > > > > On (07/13/16 08:39), Viresh Kumar wrote: > > [..] > >> Maybe not, as this can still lead to the original bug we were all > >> chasing.

Re: [Query] Preemption (hogging) of the work handler

On (07/14/16 03:09), Rafael J. Wysocki wrote: > On Thu, Jul 14, 2016 at 2:55 AM, Sergey Senozhatsky > wrote: > > Hello, > > > > On (07/13/16 08:39), Viresh Kumar wrote: > > [..] > >> Maybe not, as this can still lead to the original bug we were all > >> chasing. This may hog some other CPU if we

Re: [Query] Preemption (hogging) of the work handler

On Thu, Jul 14, 2016 at 2:55 AM, Sergey Senozhatsky wrote: > Hello, > > On (07/13/16 08:39), Viresh Kumar wrote: > [..] >> Maybe not, as this can still lead to the original bug we were all >> chasing. This may hog some other CPU if we are doing excessive >>

Re: [Query] Preemption (hogging) of the work handler

On Thu, Jul 14, 2016 at 2:55 AM, Sergey Senozhatsky wrote: > Hello, > > On (07/13/16 08:39), Viresh Kumar wrote: > [..] >> Maybe not, as this can still lead to the original bug we were all >> chasing. This may hog some other CPU if we are doing excessive >> printing in suspend :( > > excessive

Re: [Query] Preemption (hogging) of the work handler

Hello, On (07/13/16 08:39), Viresh Kumar wrote: [..] > Maybe not, as this can still lead to the original bug we were all > chasing. This may hog some other CPU if we are doing excessive > printing in suspend :( excessive printing is just part of the problem here. if we cab cond_resched() in

Re: [Query] Preemption (hogging) of the work handler

Hello, On (07/13/16 08:39), Viresh Kumar wrote: [..] > Maybe not, as this can still lead to the original bug we were all > chasing. This may hog some other CPU if we are doing excessive > printing in suspend :( excessive printing is just part of the problem here. if we cab cond_resched() in

Re: [Query] Preemption (hogging) of the work handler

On Wed, Jul 13, 2016 at 04:18:58PM -0700, Viresh Kumar wrote: > > Are all of those messages printed actually useful? > > Hmm, maybe not. But that's not the point I was trying to raise, as I > earlier mentioned :) > > We have a problem with asynchronous printing after disabling > interrupts on

Re: [Query] Preemption (hogging) of the work handler

On Wed, Jul 13, 2016 at 04:18:58PM -0700, Viresh Kumar wrote: > > Are all of those messages printed actually useful? > > Hmm, maybe not. But that's not the point I was trying to raise, as I > earlier mentioned :) > > We have a problem with asynchronous printing after disabling > interrupts on

Re: [Query] Preemption (hogging) of the work handler

On 14-07-16, 01:08, Rafael J. Wysocki wrote: > On Wed, Jul 13, 2016 at 5:39 PM, Viresh Kumar wrote: > > Maybe not, as this can still lead to the original bug we were all > > chasing. This may hog some other CPU if we are doing excessive > > printing in suspend :( > > How

Re: [Query] Preemption (hogging) of the work handler

On 14-07-16, 01:08, Rafael J. Wysocki wrote: > On Wed, Jul 13, 2016 at 5:39 PM, Viresh Kumar wrote: > > Maybe not, as this can still lead to the original bug we were all > > chasing. This may hog some other CPU if we are doing excessive > > printing in suspend :( > > How can it hog that CPU,

Re: [Query] Preemption (hogging) of the work handler

On Wed, Jul 13, 2016 at 5:39 PM, Viresh Kumar wrote: > On 13-07-16, 14:45, Sergey Senozhatsky wrote: >> On (07/12/16 16:19), Viresh Kumar wrote: [cut] >> >> something like below, perhaps. will this work for you? > > Maybe not, as this can still lead to the original bug

Re: [Query] Preemption (hogging) of the work handler

On Wed, Jul 13, 2016 at 5:39 PM, Viresh Kumar wrote: > On 13-07-16, 14:45, Sergey Senozhatsky wrote: >> On (07/12/16 16:19), Viresh Kumar wrote: [cut] >> >> something like below, perhaps. will this work for you? > > Maybe not, as this can still lead to the original bug we were all > chasing.

Re: [Query] Preemption (hogging) of the work handler

On 13-07-16, 14:45, Sergey Senozhatsky wrote: > On (07/12/16 16:19), Viresh Kumar wrote: > [..] > > Okay, we have tracked this BUG and its really interesting. > > good find! > > > I hacked the platform's serial driver to implement a putchar() routine > > that simply writes to the FIFO in polling

Re: [Query] Preemption (hogging) of the work handler

On 13-07-16, 14:45, Sergey Senozhatsky wrote: > On (07/12/16 16:19), Viresh Kumar wrote: > [..] > > Okay, we have tracked this BUG and its really interesting. > > good find! > > > I hacked the platform's serial driver to implement a putchar() routine > > that simply writes to the FIFO in polling

Re: [Query] Preemption (hogging) of the work handler

On Wed, Jul 13, 2016 at 2:57 PM, Sergey Senozhatsky wrote: > Hello Rafael, > > On (07/13/16 14:05), Rafael J. Wysocki wrote: > [..] >> > ah, just saw this. OK, very close to what I sent in another thread, so I >> > guess it will work on your side. >> > let me know if

Re: [Query] Preemption (hogging) of the work handler

On Wed, Jul 13, 2016 at 2:57 PM, Sergey Senozhatsky wrote: > Hello Rafael, > > On (07/13/16 14:05), Rafael J. Wysocki wrote: > [..] >> > ah, just saw this. OK, very close to what I sent in another thread, so I >> > guess it will work on your side. >> > let me know if it doesn't, I'll fold it into

Re: [Query] Preemption (hogging) of the work handler

Hello Rafael, On (07/13/16 14:05), Rafael J. Wysocki wrote: [..] > > ah, just saw this. OK, very close to what I sent in another thread, so I > > guess it will work on your side. > > let me know if it doesn't, I'll fold it into 0001 and re-spin the series. > > thanks for your help! > > But you

Re: [Query] Preemption (hogging) of the work handler

Hello Rafael, On (07/13/16 14:05), Rafael J. Wysocki wrote: [..] > > ah, just saw this. OK, very close to what I sent in another thread, so I > > guess it will work on your side. > > let me know if it doesn't, I'll fold it into 0001 and re-spin the series. > > thanks for your help! > > But you

Re: [Query] Preemption (hogging) of the work handler

On Wed, Jul 13, 2016 at 9:00 AM, Sergey Senozhatsky wrote: > On (07/12/16 10:11), Viresh Kumar wrote: >> +extern bool printk_sync_suspended; >> static int suspend_enter(suspend_state_t state, bool *wakeup) >> { >> char

Re: [Query] Preemption (hogging) of the work handler

On Wed, Jul 13, 2016 at 9:00 AM, Sergey Senozhatsky wrote: > On (07/12/16 10:11), Viresh Kumar wrote: >> +extern bool printk_sync_suspended; >> static int suspend_enter(suspend_state_t state, bool *wakeup) >> { >> char suspend_abort[MAX_SUSPEND_ABORT_LEN]; >> @@ -218,6 +219,7 @@ static

Re: [Query] Preemption (hogging) of the work handler

On (07/12/16 10:11), Viresh Kumar wrote: > +extern bool printk_sync_suspended; > static int suspend_enter(suspend_state_t state, bool *wakeup) > { > char suspend_abort[MAX_SUSPEND_ABORT_LEN]; > @@ -218,6 +219,7 @@ static int suspend_enter(suspend_state_t state, bool > *wakeup) >

Re: [Query] Preemption (hogging) of the work handler

On (07/12/16 10:11), Viresh Kumar wrote: > +extern bool printk_sync_suspended; > static int suspend_enter(suspend_state_t state, bool *wakeup) > { > char suspend_abort[MAX_SUSPEND_ABORT_LEN]; > @@ -218,6 +219,7 @@ static int suspend_enter(suspend_state_t state, bool > *wakeup) >

Re: [Query] Preemption (hogging) of the work handler

Cc Petr Mladek. On (07/12/16 16:19), Viresh Kumar wrote: [..] > Okay, we have tracked this BUG and its really interesting. good find! > I hacked the platform's serial driver to implement a putchar() routine > that simply writes to the FIFO in polling mode, that helped us in > tracing on where

Re: [Query] Preemption (hogging) of the work handler

Cc Petr Mladek. On (07/12/16 16:19), Viresh Kumar wrote: [..] > Okay, we have tracked this BUG and its really interesting. good find! > I hacked the platform's serial driver to implement a putchar() routine > that simply writes to the FIFO in polling mode, that helped us in > tracing on where

Re: [Query] Preemption (hogging) of the work handler

On 12-07-16, 16:19, Viresh Kumar wrote: > Okay, we have tracked this BUG and its really interesting. > > I hacked the platform's serial driver to implement a putchar() routine > that simply writes to the FIFO in polling mode, that helped us in > tracing on where we are going wrong. > > The

Re: [Query] Preemption (hogging) of the work handler

On 12-07-16, 16:19, Viresh Kumar wrote: > Okay, we have tracked this BUG and its really interesting. > > I hacked the platform's serial driver to implement a putchar() routine > that simply writes to the FIFO in polling mode, that helped us in > tracing on where we are going wrong. > > The

Re: [Query] Preemption (hogging) of the work handler

On 11-07-16, 15:35, Viresh Kumar wrote: > Sometimes, the platform doesn't come back after suspend. I have tried > enabling no-console-suspend and the last line it prints is: > > Disabling non-boot CPUs > > And nothing after that at all. We have to forcefully reboot the phone > after

Re: [Query] Preemption (hogging) of the work handler

On 11-07-16, 15:35, Viresh Kumar wrote: > Sometimes, the platform doesn't come back after suspend. I have tried > enabling no-console-suspend and the last line it prints is: > > Disabling non-boot CPUs > > And nothing after that at all. We have to forcefully reboot the phone > after

Re: [Query] Preemption (hogging) of the work handler

On 12-07-16, 21:59, Rafael J. Wysocki wrote: > It looks like a new printk() waits for a previous one to make progress > and since progress cannot be made under the suspend conditions, it > waits forever. Thanks. Maybe, but to mention this clearly again, this doesn't happen every time. Sometime

Re: [Query] Preemption (hogging) of the work handler

On 12-07-16, 21:59, Rafael J. Wysocki wrote: > It looks like a new printk() waits for a previous one to make progress > and since progress cannot be made under the suspend conditions, it > waits forever. Thanks. Maybe, but to mention this clearly again, this doesn't happen every time. Sometime

Re: [Query] Preemption (hogging) of the work handler

On Tue, Jul 12, 2016 at 7:11 PM, Viresh Kumar wrote: > On 12-07-16, 06:12, Viresh Kumar wrote: > >> Yeah, so I tried debugging this more and I am able to get printing >> done to just before arch_suspend_disable_irqs() in suspend.c and then >> it stops because of the async

Re: [Query] Preemption (hogging) of the work handler

On Tue, Jul 12, 2016 at 7:11 PM, Viresh Kumar wrote: > On 12-07-16, 06:12, Viresh Kumar wrote: > >> Yeah, so I tried debugging this more and I am able to get printing >> done to just before arch_suspend_disable_irqs() in suspend.c and then >> it stops because of the async nature. >> >> I get to

Re: [Query] Preemption (hogging) of the work handler

On 12-07-16, 06:12, Viresh Kumar wrote: > Yeah, so I tried debugging this more and I am able to get printing > done to just before arch_suspend_disable_irqs() in suspend.c and then > it stops because of the async nature. > > I get to this point for both successful suspend/resume (where system >

Re: [Query] Preemption (hogging) of the work handler

On 12-07-16, 06:12, Viresh Kumar wrote: > Yeah, so I tried debugging this more and I am able to get printing > done to just before arch_suspend_disable_irqs() in suspend.c and then > it stops because of the async nature. > > I get to this point for both successful suspend/resume (where system >

Re: [Query] Preemption (hogging) of the work handler

Hi Sergey, On 12-07-16, 23:03, Sergey Senozhatsky wrote: > Thanks, Petr. > > > so, I'm looking at this thing now: Thanks. Though just to mention, I have seen this only once yet :) > : [ 12.874909] sched: RT throttling activated for rt_rq ffc0ac13fcd0 > (cpu 0) > : [ 12.874909]

Re: [Query] Preemption (hogging) of the work handler

Hi Sergey, On 12-07-16, 23:03, Sergey Senozhatsky wrote: > Thanks, Petr. > > > so, I'm looking at this thing now: Thanks. Though just to mention, I have seen this only once yet :) > : [ 12.874909] sched: RT throttling activated for rt_rq ffc0ac13fcd0 > (cpu 0) > : [ 12.874909]

Re: [Query] Preemption (hogging) of the work handler

On 12-07-16, 15:56, Petr Mladek wrote: > Ah, I have missed that the hang happens only when you use the async > printk patchset. And that also doesn't happen always, but only sometimes. So there is a race somewhere I feel :) > I wonder if it is somehow related to the commit 8d91f8b15361dfb438ab >

Re: [Query] Preemption (hogging) of the work handler

On 12-07-16, 15:56, Petr Mladek wrote: > Ah, I have missed that the hang happens only when you use the async > printk patchset. And that also doesn't happen always, but only sometimes. So there is a race somewhere I feel :) > I wonder if it is somehow related to the commit 8d91f8b15361dfb438ab >

Re: [Query] Preemption (hogging) of the work handler

Hello, On (07/12/16 14:52), Petr Mladek wrote: [..] > > On (07/11/16 15:35), Viresh Kumar wrote: > > [..] > > > Sometimes, the platform doesn't come back after suspend. I have tried > > > enabling no-console-suspend and the last line it prints is: > > > > > > Disabling non-boot CPUs > >

Re: [Query] Preemption (hogging) of the work handler

Hello, On (07/12/16 14:52), Petr Mladek wrote: [..] > > On (07/11/16 15:35), Viresh Kumar wrote: > > [..] > > > Sometimes, the platform doesn't come back after suspend. I have tried > > > enabling no-console-suspend and the last line it prints is: > > > > > > Disabling non-boot CPUs > >

Re: [Query] Preemption (hogging) of the work handler

On Tue 2016-07-12 06:02:31, Viresh Kumar wrote: > On 12-07-16, 14:24, Rafael J. Wysocki wrote: > > On Monday, July 11, 2016 03:46:01 PM Viresh Kumar wrote: > > > Yeah and I am not sure how should I go ahead about this issue now :) > > > > FWIW, I think the reason why the "synchronous printk"

Re: [Query] Preemption (hogging) of the work handler

On Tue 2016-07-12 06:02:31, Viresh Kumar wrote: > On 12-07-16, 14:24, Rafael J. Wysocki wrote: > > On Monday, July 11, 2016 03:46:01 PM Viresh Kumar wrote: > > > Yeah and I am not sure how should I go ahead about this issue now :) > > > > FWIW, I think the reason why the "synchronous printk"

Re: [Query] Preemption (hogging) of the work handler

+Rafael and linux-pm to this thread :) On 12-07-16, 14:52, Petr Mladek wrote: > On Tue 2016-07-12 18:38:05, Sergey Senozhatsky wrote: > > Hello, > > > > On (07/11/16 15:35), Viresh Kumar wrote: > > [..] > > > Sometimes, the platform doesn't come back after suspend. I have tried > > > enabling

Re: [Query] Preemption (hogging) of the work handler

+Rafael and linux-pm to this thread :) On 12-07-16, 14:52, Petr Mladek wrote: > On Tue 2016-07-12 18:38:05, Sergey Senozhatsky wrote: > > Hello, > > > > On (07/11/16 15:35), Viresh Kumar wrote: > > [..] > > > Sometimes, the platform doesn't come back after suspend. I have tried > > > enabling

Re: [Query] Preemption (hogging) of the work handler

On 12-07-16, 14:24, Rafael J. Wysocki wrote: > On Monday, July 11, 2016 03:46:01 PM Viresh Kumar wrote: > > Yeah and I am not sure how should I go ahead about this issue now :) > > FWIW, I think the reason why the "synchronous printk" works is because after > disabling the non-boot CPU, the only

Re: [Query] Preemption (hogging) of the work handler

On 12-07-16, 14:24, Rafael J. Wysocki wrote: > On Monday, July 11, 2016 03:46:01 PM Viresh Kumar wrote: > > Yeah and I am not sure how should I go ahead about this issue now :) > > FWIW, I think the reason why the "synchronous printk" works is because after > disabling the non-boot CPU, the only

Re: [Query] Preemption (hogging) of the work handler

On Tue 2016-07-12 18:38:05, Sergey Senozhatsky wrote: > Hello, > > On (07/11/16 15:35), Viresh Kumar wrote: > [..] > > Sometimes, the platform doesn't come back after suspend. I have tried > > enabling no-console-suspend and the last line it prints is: > > > > Disabling non-boot CPUs I

Re: [Query] Preemption (hogging) of the work handler

On Tue 2016-07-12 18:38:05, Sergey Senozhatsky wrote: > Hello, > > On (07/11/16 15:35), Viresh Kumar wrote: > [..] > > Sometimes, the platform doesn't come back after suspend. I have tried > > enabling no-console-suspend and the last line it prints is: > > > > Disabling non-boot CPUs I

Re: [Query] Preemption (hogging) of the work handler

On Monday, July 11, 2016 03:46:01 PM Viresh Kumar wrote: > On 12-07-16, 00:44, Rafael J. Wysocki wrote: > > On Monday, July 11, 2016 03:35:01 PM Viresh Kumar wrote: > > > Hi Sergey and Jan, > > > > > > On 12-07-16, 00:44, Sergey Senozhatsky wrote: > > > > right. apart from cases when the existing

Re: [Query] Preemption (hogging) of the work handler

On Monday, July 11, 2016 03:46:01 PM Viresh Kumar wrote: > On 12-07-16, 00:44, Rafael J. Wysocki wrote: > > On Monday, July 11, 2016 03:35:01 PM Viresh Kumar wrote: > > > Hi Sergey and Jan, > > > > > > On 12-07-16, 00:44, Sergey Senozhatsky wrote: > > > > right. apart from cases when the existing

Re: [Query] Preemption (hogging) of the work handler

Hello, On (07/11/16 15:35), Viresh Kumar wrote: [..] > Sometimes, the platform doesn't come back after suspend. I have tried > enabling no-console-suspend and the last line it prints is: > > Disabling non-boot CPUs > > And nothing after that at all. We have to forcefully reboot the

Re: [Query] Preemption (hogging) of the work handler

Hello, On (07/11/16 15:35), Viresh Kumar wrote: [..] > Sometimes, the platform doesn't come back after suspend. I have tried > enabling no-console-suspend and the last line it prints is: > > Disabling non-boot CPUs > > And nothing after that at all. We have to forcefully reboot the

Re: [Query] Preemption (hogging) of the work handler

On 12-07-16, 00:44, Rafael J. Wysocki wrote: > On Monday, July 11, 2016 03:35:01 PM Viresh Kumar wrote: > > Hi Sergey and Jan, > > > > On 12-07-16, 00:44, Sergey Senozhatsky wrote: > > > right. apart from cases when the existing console_unlock() behaviour can > > > simply "block" a process to

Re: [Query] Preemption (hogging) of the work handler

On 12-07-16, 00:44, Rafael J. Wysocki wrote: > On Monday, July 11, 2016 03:35:01 PM Viresh Kumar wrote: > > Hi Sergey and Jan, > > > > On 12-07-16, 00:44, Sergey Senozhatsky wrote: > > > right. apart from cases when the existing console_unlock() behaviour can > > > simply "block" a process to

Re: [Query] Preemption (hogging) of the work handler

On Monday, July 11, 2016 03:35:01 PM Viresh Kumar wrote: > Hi Sergey and Jan, > > On 12-07-16, 00:44, Sergey Senozhatsky wrote: > > right. apart from cases when the existing console_unlock() behaviour can > > simply "block" a process to flush the log_buf to slow serial consoles > > (regardless

Re: [Query] Preemption (hogging) of the work handler

On Monday, July 11, 2016 03:35:01 PM Viresh Kumar wrote: > Hi Sergey and Jan, > > On 12-07-16, 00:44, Sergey Senozhatsky wrote: > > right. apart from cases when the existing console_unlock() behaviour can > > simply "block" a process to flush the log_buf to slow serial consoles > > (regardless

Re: [Query] Preemption (hogging) of the work handler

Hi Sergey and Jan, On 12-07-16, 00:44, Sergey Senozhatsky wrote: > right. apart from cases when the existing console_unlock() behaviour can > simply "block" a process to flush the log_buf to slow serial consoles > (regardless the process execution context) and make the system less > responsive,

Re: [Query] Preemption (hogging) of the work handler

Hi Sergey and Jan, On 12-07-16, 00:44, Sergey Senozhatsky wrote: > right. apart from cases when the existing console_unlock() behaviour can > simply "block" a process to flush the log_buf to slow serial consoles > (regardless the process execution context) and make the system less > responsive,

Re: [Query] Preemption (hogging) of the work handler

Hi Jan, On 11-07-16, 12:26, Jan Kara wrote: > Yes. We have similar problems as you observe on machines when they do a lot > of printing (usually due to device discovery or similar reasons). The > problem is not fully solved even upstream as Andrew is reluctant to merge > the patches. Sergey

Re: [Query] Preemption (hogging) of the work handler

Hi Jan, On 11-07-16, 12:26, Jan Kara wrote: > Yes. We have similar problems as you observe on machines when they do a lot > of printing (usually due to device discovery or similar reasons). The > problem is not fully solved even upstream as Andrew is reluctant to merge > the patches. Sergey

Re: [Query] Preemption (hogging) of the work handler

Hello, Thanks for Cc-ing. I'm attending an internal 2-days training now, so I'm a bit slow at answering emails, sorry. On (07/11/16 12:26), Jan Kara wrote: [..] > > These print messages continue from 2994.918 to 2996.268 (1.35 seconds) > > and they hog the work-handler for that long, which

Re: [Query] Preemption (hogging) of the work handler

Hello, Thanks for Cc-ing. I'm attending an internal 2-days training now, so I'm a bit slow at answering emails, sorry. On (07/11/16 12:26), Jan Kara wrote: [..] > > These print messages continue from 2994.918 to 2996.268 (1.35 seconds) > > and they hog the work-handler for that long, which

Re: [Query] Preemption (hogging) of the work handler

On Wed 06-07-16 11:28:42, Viresh Kumar wrote: > On 01-07-16, 12:22, Tejun Heo wrote: > I enabled traces with '-e all' to look at everything happening on the > CPU. > > Following is what starts in the middle of the delayed-work handler: > > kworker/0:1H-40[000] d..1 2994.918766: console:

Re: [Query] Preemption (hogging) of the work handler

On Wed 06-07-16 11:28:42, Viresh Kumar wrote: > On 01-07-16, 12:22, Tejun Heo wrote: > I enabled traces with '-e all' to look at everything happening on the > CPU. > > Following is what starts in the middle of the delayed-work handler: > > kworker/0:1H-40[000] d..1 2994.918766: console:

  1   2   >