Re: [PATCH v5 1/4] printk/nmi: generic solution for safe printk in NMI

2017-04-28 Thread Steven Rostedt
On Fri, 28 Apr 2017 15:58:10 +0200 Peter Zijlstra wrote: > On Fri, Apr 28, 2017 at 03:44:23PM +0200, Petr Mladek wrote: > > On Fri 2017-04-28 11:02:26, Peter Zijlstra wrote: > > > On Thu, Apr 27, 2017 at 03:38:19PM +0200, Petr Mladek wrote: > > > > Also we need to look for alternatives. There

Re: [PATCH v5 1/4] printk/nmi: generic solution for safe printk in NMI

2017-04-28 Thread Steven Rostedt
On Fri, 28 Apr 2017 14:57:25 +0200 Petr Mladek wrote: > Of course, if the problem is reproducible, the easiest solution > is to use bigger main log buffer, for example boot with > log_buf_len=32M. Of course that may not be enough. Especially when I have a machine with 240 CPUs. But it also has

Re: [PATCH v5 1/4] printk/nmi: generic solution for safe printk in NMI

2017-04-28 Thread Peter Zijlstra
On Fri, Apr 28, 2017 at 03:44:23PM +0200, Petr Mladek wrote: > On Fri 2017-04-28 11:02:26, Peter Zijlstra wrote: > > On Thu, Apr 27, 2017 at 03:38:19PM +0200, Petr Mladek wrote: > > > Also we need to look for alternatives. There is a chance > > > to create crashdump and get the ftrace messages from

Re: [PATCH v5 1/4] printk/nmi: generic solution for safe printk in NMI

2017-04-28 Thread Petr Mladek
On Fri 2017-04-28 11:02:26, Peter Zijlstra wrote: > On Thu, Apr 27, 2017 at 03:38:19PM +0200, Petr Mladek wrote: > > Also we need to look for alternatives. There is a chance > > to create crashdump and get the ftrace messages from it. > > Also this might be scenario when we might need to suggest >

Re: [PATCH v5 1/4] printk/nmi: generic solution for safe printk in NMI

2017-04-28 Thread Petr Mladek
On Fri 2017-04-28 10:35:32, Sergey Senozhatsky wrote: > On (04/27/17 12:14), Steven Rostedt wrote: > [..] > > I tried this patch. It's better because I get the end of the trace, but > > I do lose the beginning of it: > > > > ** 196358 printk messages dropped ** [ 102.321182] perf-59810...

Re: [PATCH v5 1/4] printk/nmi: generic solution for safe printk in NMI

2017-04-28 Thread Petr Mladek
On Fri 2017-04-28 10:25:30, Sergey Senozhatsky wrote: > > On (04/20/17 15:11), Petr Mladek wrote: > [..] > > void printk_nmi_enter(void) > > { > > - this_cpu_or(printk_context, PRINTK_NMI_CONTEXT_MASK); > > + /* > > +* The size of the extra per-CPU buffer is limited. Use it > > +* on

Re: [PATCH v5 1/4] printk/nmi: generic solution for safe printk in NMI

2017-04-28 Thread Peter Zijlstra
On Thu, Apr 27, 2017 at 03:38:19PM +0200, Petr Mladek wrote: > Also we need to look for alternatives. There is a chance > to create crashdump and get the ftrace messages from it. > Also this might be scenario when we might need to suggest > the early_printk() patchset from Peter Zijlstra. I'd be h

Re: [PATCH v5 1/4] printk/nmi: generic solution for safe printk in NMI

2017-04-27 Thread Sergey Senozhatsky
On (04/27/17 12:14), Steven Rostedt wrote: [..] > I tried this patch. It's better because I get the end of the trace, but > I do lose the beginning of it: > > ** 196358 printk messages dropped ** [ 102.321182] perf-59810 > 12983650us : d_path <-seq_path many thanks! so we now drop

Re: [PATCH v5 1/4] printk/nmi: generic solution for safe printk in NMI

2017-04-27 Thread Sergey Senozhatsky
On (04/20/17 15:11), Petr Mladek wrote: [..] > void printk_nmi_enter(void) > { > - this_cpu_or(printk_context, PRINTK_NMI_CONTEXT_MASK); > + /* > + * The size of the extra per-CPU buffer is limited. Use it > + * only when really needed. > + */ > + if (this_cpu_read(pri

Re: [PATCH v5 1/4] printk/nmi: generic solution for safe printk in NMI

2017-04-27 Thread Steven Rostedt
On Thu, 20 Apr 2017 15:11:54 +0200 Petr Mladek wrote: > > >From c530d9dee91c74db5e6a198479e2e63b24cb84a2 Mon Sep 17 00:00:00 2001 > From: Petr Mladek > Date: Thu, 20 Apr 2017 10:52:31 +0200 > Subject: [PATCH] printk: Use the main logbuf in NMI when logbuf_lock is > available I tried this

Re: [PATCH v5 1/4] printk/nmi: generic solution for safe printk in NMI

2017-04-27 Thread Steven Rostedt
On Thu, 27 Apr 2017 17:28:07 +0200 Petr Mladek wrote: > > When I get a chance, I'll see if I can insert a trigger to crash the > > kernel from NMI on another box and see if this patch helps. > > I actually tested it here using this hack: > > diff --cc lib/nmi_backtrace.c > index d531f85c0c9b

Re: [PATCH v5 1/4] printk/nmi: generic solution for safe printk in NMI

2017-04-27 Thread Petr Mladek
On Thu 2017-04-27 10:31:18, Steven Rostedt wrote: > On Thu, 27 Apr 2017 15:38:19 +0200 > Petr Mladek wrote: > > > > by the way, > > > does this `nmi_print_seq' bypass even fix anything for Steven? > > > > I think that this is the most important question. > > > > Steven, does the patch from >

Re: [PATCH v5 1/4] printk/nmi: generic solution for safe printk in NMI

2017-04-27 Thread Steven Rostedt
On Thu, 27 Apr 2017 15:38:19 +0200 Petr Mladek wrote: > > by the way, > > does this `nmi_print_seq' bypass even fix anything for Steven? > > I think that this is the most important question. > > Steven, does the patch from > https://lkml.kernel.org/r/20170420131154.gl3...@pathway.suse.cz > he

Re: [PATCH v5 1/4] printk/nmi: generic solution for safe printk in NMI

2017-04-27 Thread Petr Mladek
On Mon 2017-04-24 11:17:47, Sergey Senozhatsky wrote: > On (04/21/17 14:06), Petr Mladek wrote: > [..] > > > I agree that this_cpu_read(printk_context) covers slightly more than > > > logbuf_lock scope, so we may get positive this_cpu_read(printk_context) > > > with unlocked logbuf_lock, but I don'

Re: [PATCH v5 1/4] printk/nmi: generic solution for safe printk in NMI

2017-04-23 Thread Sergey Senozhatsky
On (04/21/17 14:06), Petr Mladek wrote: [..] > > I agree that this_cpu_read(printk_context) covers slightly more than > > logbuf_lock scope, so we may get positive this_cpu_read(printk_context) > > with unlocked logbuf_lock, but I don't tend to think that it's a big > > problem. > > PRINTK_SAFE_CO

Re: [PATCH v5 1/4] printk/nmi: generic solution for safe printk in NMI

2017-04-21 Thread Petr Mladek
On Fri 2017-04-21 10:57:25, Sergey Senozhatsky wrote: > On (04/20/17 15:11), Petr Mladek wrote: > [..] > > void printk_nmi_enter(void) > > { > > - this_cpu_or(printk_context, PRINTK_NMI_CONTEXT_MASK); > > + /* > > +* The size of the extra per-CPU buffer is limited. Use it > > +* only

Re: [PATCH v5 1/4] printk/nmi: generic solution for safe printk in NMI

2017-04-20 Thread Sergey Senozhatsky
Hello, On (04/20/17 15:11), Petr Mladek wrote: [..] > Good analyze. I would summarize it that we need to be careful of: > > + logbug_lock > + PRINTK_SAFE_CONTEXT > + locks used by console drivers > > The first two things are easy to check. Except that a check for logbuf_lock > might produc

Re: [PATCH v5 1/4] printk/nmi: generic solution for safe printk in NMI

2017-04-20 Thread Petr Mladek
On Thu 2017-04-20 12:31:12, Sergey Senozhatsky wrote: > Hello Steven, > > On (04/19/17 13:13), Steven Rostedt wrote: > > > printk() takes some locks and could not be used a safe way in NMI context. > > > > I just found a problem with this solution. It kills ftrace dumps from > > NMI context :-( >

Re: [PATCH v5 1/4] printk/nmi: generic solution for safe printk in NMI

2017-04-19 Thread Sergey Senozhatsky
Hello Steven, On (04/19/17 13:13), Steven Rostedt wrote: > > printk() takes some locks and could not be used a safe way in NMI context. > > I just found a problem with this solution. It kills ftrace dumps from > NMI context :-( > > [ 1295.168495]<...>-67423 10dNh1 38217us : do_raw_spin_

Re: [PATCH v5 1/4] printk/nmi: generic solution for safe printk in NMI

2017-04-19 Thread Peter Zijlstra
On Wed, Apr 19, 2017 at 01:13:41PM -0400, Steven Rostedt wrote: > On Thu, 21 Apr 2016 13:48:42 +0200 > Petr Mladek wrote: > > > printk() takes some locks and could not be used a safe way in NMI context. > > I just found a problem with this solution. It kills ftrace dumps from > NMI context :-( >

Re: [PATCH v5 1/4] printk/nmi: generic solution for safe printk in NMI

2017-04-19 Thread Steven Rostedt
On Thu, 21 Apr 2016 13:48:42 +0200 Petr Mladek wrote: > printk() takes some locks and could not be used a safe way in NMI context. I just found a problem with this solution. It kills ftrace dumps from NMI context :-( [ 1295.168495]<...>-67423 10dNh1 38217us : do_raw_spin_lock <-_raw_s

Re: [PATCH v5 1/4] printk/nmi: generic solution for safe printk in NMI

2016-04-27 Thread Russell King - ARM Linux
On Thu, Apr 21, 2016 at 01:48:42PM +0200, Petr Mladek wrote: > diff --git a/arch/arm/Kconfig b/arch/arm/Kconfig > index cdfa6c2b7626..259543ec6dc9 100644 > --- a/arch/arm/Kconfig > +++ b/arch/arm/Kconfig > @@ -66,6 +66,7 @@ config ARM > select HAVE_KRETPROBES if (HAVE_KPROBES) > select

[PATCH v5 1/4] printk/nmi: generic solution for safe printk in NMI

2016-04-21 Thread Petr Mladek
printk() takes some locks and could not be used a safe way in NMI context. The chance of a deadlock is real especially when printing stacks from all CPUs. This particular problem has been addressed on x86 by the commit a9edc8809328 ("x86/nmi: Perform a safe NMI stack trace on all CPUs"). The pat