Re: kprobes broken since 0d00449c7a28 ("x86: Replace ist_enter() with nmi_enter()")

2021-02-03 Thread Steven Rostedt
On Wed, 3 Feb 2021 22:33:28 +0900 Masami Hiramatsu wrote: > Ah, that is what I worried about. ftrace and kprobes handler usually want to > know "what is the actual status of the system where the probe hits". > > If the new kernel_exception_enter() for ftrace/kprobes or any other kernel >

Re: kprobes broken since 0d00449c7a28 ("x86: Replace ist_enter() with nmi_enter()")

2021-02-03 Thread Masami Hiramatsu
On Tue, 2 Feb 2021 16:05:13 -0500 Steven Rostedt wrote: > On Tue, 2 Feb 2021 19:30:34 +0100 > Peter Zijlstra wrote: > > > That does mean that kprobes are then fundamentally running from > > in_nmi(), which is what started all this. > > I just thought about the fact that tracing records the

Re: kprobes broken since 0d00449c7a28 ("x86: Replace ist_enter() with nmi_enter()")

2021-02-02 Thread Steven Rostedt
On Tue, 2 Feb 2021 19:30:34 +0100 Peter Zijlstra wrote: > That does mean that kprobes are then fundamentally running from > in_nmi(), which is what started all this. I just thought about the fact that tracing records the context of the function it is called in. If you set "in_nmi()" for all

Re: kprobes broken since 0d00449c7a28 ("x86: Replace ist_enter() with nmi_enter()")

2021-02-02 Thread Peter Zijlstra
On Tue, Feb 02, 2021 at 11:56:23AM -0500, Steven Rostedt wrote: > NMIs are special, and they always have been. They shouldn't be doing much > anyway. If they are, then that's a problem. There is a fair amount of NMI level code these days, and it seems to be ever increasing... > My question

Re: kprobes broken since 0d00449c7a28 ("x86: Replace ist_enter() with nmi_enter()")

2021-02-02 Thread Steven Rostedt
On Tue, 2 Feb 2021 17:45:47 +0100 Peter Zijlstra wrote: > On Tue, Feb 02, 2021 at 09:52:49AM -0500, Steven Rostedt wrote: > > > But from a handler, you could do: > > > > if (in_nmi()) > > return; > > local_irq_save(flags); > > /* Now you are safe from being re-entrant.

Re: kprobes broken since 0d00449c7a28 ("x86: Replace ist_enter() with nmi_enter()")

2021-02-02 Thread Peter Zijlstra
On Tue, Feb 02, 2021 at 09:52:49AM -0500, Steven Rostedt wrote: > But from a handler, you could do: > > if (in_nmi()) > return; > local_irq_save(flags); > /* Now you are safe from being re-entrant. */ But that's an utter crap thing to do. That's like saying I

Re: kprobes broken since 0d00449c7a28 ("x86: Replace ist_enter() with nmi_enter()")

2021-02-02 Thread Steven Rostedt
On Tue, 2 Feb 2021 11:45:41 +0100 Peter Zijlstra wrote: > > The stack tracer checks the size of the stack, compares it to the > > largest recorded size, and if it's bigger, it will save the stack. But > > if this happens on two CPUs at the same time, only one can do the > > recording at the same

Re: kprobes broken since 0d00449c7a28 ("x86: Replace ist_enter() with nmi_enter()")

2021-02-02 Thread Peter Zijlstra
On Sat, Jan 30, 2021 at 07:44:10AM -0500, Steven Rostedt wrote: > On Sat, 30 Jan 2021 09:28:32 +0100 > Peter Zijlstra wrote: > > > On Fri, Jan 29, 2021 at 04:24:54PM -0500, Steven Rostedt wrote: > > > Specifically, kprobe and ftrace callbacks may have this: > > > > > > if (in_nmi()) > > >

Re: kprobes broken since 0d00449c7a28 ("x86: Replace ist_enter() with nmi_enter()")

2021-01-30 Thread Steven Rostedt
On Sat, 30 Jan 2021 09:28:32 +0100 Peter Zijlstra wrote: > On Fri, Jan 29, 2021 at 04:24:54PM -0500, Steven Rostedt wrote: > > Specifically, kprobe and ftrace callbacks may have this: > > > > if (in_nmi()) > > return; > > > > raw_spin_lock_irqsave(, flags); > > [..] > >

Re: kprobes broken since 0d00449c7a28 ("x86: Replace ist_enter() with nmi_enter()")

2021-01-30 Thread Masami Hiramatsu
On Fri, 29 Jan 2021 19:08:40 -0800 Alexei Starovoitov wrote: > On Sat, Jan 30, 2021 at 11:02:49AM +0900, Masami Hiramatsu wrote: > > On Fri, 29 Jan 2021 18:59:43 +0100 > > Peter Zijlstra wrote: > > > > > On Fri, Jan 29, 2021 at 09:45:48AM -0800, Alexei Starovoitov wrote: > > > > Same things

Re: kprobes broken since 0d00449c7a28 ("x86: Replace ist_enter() with nmi_enter()")

2021-01-30 Thread Alexei Starovoitov
On Sat, Jan 30, 2021 at 11:02:49AM +0900, Masami Hiramatsu wrote: > On Fri, 29 Jan 2021 18:59:43 +0100 > Peter Zijlstra wrote: > > > On Fri, Jan 29, 2021 at 09:45:48AM -0800, Alexei Starovoitov wrote: > > > Same things apply to bpf side. We can statically prove safety for > > > ftrace and kprobe

Re: kprobes broken since 0d00449c7a28 ("x86: Replace ist_enter() with nmi_enter()")

2021-01-30 Thread Peter Zijlstra
On Fri, Jan 29, 2021 at 04:24:54PM -0500, Steven Rostedt wrote: > Specifically, kprobe and ftrace callbacks may have this: > > if (in_nmi()) > return; > > raw_spin_lock_irqsave(, flags); > [..] > raw_spin_unlock_irqrestore(, flags); > > Which is totally

Re: kprobes broken since 0d00449c7a28 ("x86: Replace ist_enter() with nmi_enter()")

2021-01-29 Thread Masami Hiramatsu
On Fri, 29 Jan 2021 18:59:43 +0100 Peter Zijlstra wrote: > On Fri, Jan 29, 2021 at 09:45:48AM -0800, Alexei Starovoitov wrote: > > Same things apply to bpf side. We can statically prove safety for > > ftrace and kprobe attaching whereas to deal with NMI situation we > > have to use run-time

Re: kprobes broken since 0d00449c7a28 ("x86: Replace ist_enter() with nmi_enter()")

2021-01-29 Thread Masami Hiramatsu
On Fri, 29 Jan 2021 13:05:33 -0800 Alexei Starovoitov wrote: > On Fri, Jan 29, 2021 at 02:01:03PM -0500, Steven Rostedt wrote: > > On Fri, 29 Jan 2021 18:59:43 +0100 > > Peter Zijlstra wrote: > > > > > On Fri, Jan 29, 2021 at 09:45:48AM -0800, Alexei Starovoitov wrote: > > > > Same things

Re: kprobes broken since 0d00449c7a28 ("x86: Replace ist_enter() with nmi_enter()")

2021-01-29 Thread Steven Rostedt
On Fri, 29 Jan 2021 14:01:03 -0500 Steven Rostedt wrote: > On Fri, 29 Jan 2021 18:59:43 +0100 > Peter Zijlstra wrote: > > > On Fri, Jan 29, 2021 at 09:45:48AM -0800, Alexei Starovoitov wrote: > > > Same things apply to bpf side. We can statically prove safety for > > > ftrace and kprobe

Re: kprobes broken since 0d00449c7a28 ("x86: Replace ist_enter() with nmi_enter()")

2021-01-29 Thread Alexei Starovoitov
On Fri, Jan 29, 2021 at 02:01:03PM -0500, Steven Rostedt wrote: > On Fri, 29 Jan 2021 18:59:43 +0100 > Peter Zijlstra wrote: > > > On Fri, Jan 29, 2021 at 09:45:48AM -0800, Alexei Starovoitov wrote: > > > Same things apply to bpf side. We can statically prove safety for > > > ftrace and kprobe

Re: kprobes broken since 0d00449c7a28 ("x86: Replace ist_enter() with nmi_enter()")

2021-01-29 Thread Steven Rostedt
On Fri, 29 Jan 2021 18:59:43 +0100 Peter Zijlstra wrote: > On Fri, Jan 29, 2021 at 09:45:48AM -0800, Alexei Starovoitov wrote: > > Same things apply to bpf side. We can statically prove safety for > > ftrace and kprobe attaching whereas to deal with NMI situation we > > have to use run-time

Re: kprobes broken since 0d00449c7a28 ("x86: Replace ist_enter() with nmi_enter()")

2021-01-29 Thread Peter Zijlstra
On Fri, Jan 29, 2021 at 09:45:48AM -0800, Alexei Starovoitov wrote: > Same things apply to bpf side. We can statically prove safety for > ftrace and kprobe attaching whereas to deal with NMI situation we > have to use run-time checks for recursion prevention, etc. I have no idea what you're

Re: kprobes broken since 0d00449c7a28 ("x86: Replace ist_enter() with nmi_enter()")

2021-01-29 Thread Alexei Starovoitov
On Fri, Jan 29, 2021 at 8:24 AM Peter Zijlstra wrote: > > On Fri, Jan 29, 2021 at 10:59:52AM -0500, Steven Rostedt wrote: > > On Fri, 29 Jan 2021 22:40:11 +0900 > > Masami Hiramatsu wrote: > > > > > > So what, they can all happen with random locks held. Marking them as NMI > > > > enables a

Re: kprobes broken since 0d00449c7a28 ("x86: Replace ist_enter() with nmi_enter()")

2021-01-29 Thread Peter Zijlstra
On Fri, Jan 29, 2021 at 10:59:52AM -0500, Steven Rostedt wrote: > On Fri, 29 Jan 2021 22:40:11 +0900 > Masami Hiramatsu wrote: > > > > So what, they can all happen with random locks held. Marking them as NMI > > > enables a whole bunch of sanity checks that are entirely appropriate. > > > >

Re: kprobes broken since 0d00449c7a28 ("x86: Replace ist_enter() with nmi_enter()")

2021-01-28 Thread Nikolay Borisov
On 29.01.21 г. 3:34 ч., Alexei Starovoitov wrote: > On Thu, Jan 28, 2021 at 07:24:14PM +0100, Peter Zijlstra wrote: >> On Thu, Jan 28, 2021 at 06:45:56PM +0200, Nikolay Borisov wrote: >>> it would be placed on the __fentry__ (and not endbr64) hence it works. >>> So perhaps a workaround outside

Re: kprobes broken since 0d00449c7a28 ("x86: Replace ist_enter() with nmi_enter()")

2021-01-28 Thread Alexei Starovoitov
On Thu, Jan 28, 2021 at 07:24:14PM +0100, Peter Zijlstra wrote: > On Thu, Jan 28, 2021 at 06:45:56PM +0200, Nikolay Borisov wrote: > > it would be placed on the __fentry__ (and not endbr64) hence it works. > > So perhaps a workaround outside of bpf could essentially detect this > > scenario and

Re: kprobes broken since 0d00449c7a28 ("x86: Replace ist_enter() with nmi_enter()")

2021-01-28 Thread Peter Zijlstra
On Thu, Jan 28, 2021 at 06:45:56PM +0200, Nikolay Borisov wrote: > it would be placed on the __fentry__ (and not endbr64) hence it works. > So perhaps a workaround outside of bpf could essentially detect this > scenario and adjust the probe to be on the __fentry__ and not preceding > instruction

Re: kprobes broken since 0d00449c7a28 ("x86: Replace ist_enter() with nmi_enter()")

2021-01-28 Thread Josh Poimboeuf
On Thu, Jan 28, 2021 at 06:45:56PM +0200, Nikolay Borisov wrote: > On 28.01.21 г. 18:12 ч., Nikolay Borisov wrote: > > On 28.01.21 г. 5:38 ч., Masami Hiramatsu wrote: > >> Hi, > > > > > >> > >> Alexei, could you tell me what is the concerning situation for bpf? > > > > Another data point masami

Re: kprobes broken since 0d00449c7a28 ("x86: Replace ist_enter() with nmi_enter()")

2021-01-28 Thread Nikolay Borisov
On 28.01.21 г. 18:12 ч., Nikolay Borisov wrote: > > > On 28.01.21 г. 5:38 ч., Masami Hiramatsu wrote: >> Hi, > > > >> >> Alexei, could you tell me what is the concerning situation for bpf? > > Another data point masami is that this affects bpf kprobes which are > entered via int3,

Re: kprobes broken since 0d00449c7a28 ("x86: Replace ist_enter() with nmi_enter()")

2021-01-28 Thread Nikolay Borisov
On 28.01.21 г. 5:38 ч., Masami Hiramatsu wrote: > Hi, > > Alexei, could you tell me what is the concerning situation for bpf? Another data point masami is that this affects bpf kprobes which are entered via int3, alternatively if the kprobe is entered via kprobe_ftrace_handler it works as

Re: kprobes broken since 0d00449c7a28 ("x86: Replace ist_enter() with nmi_enter()")

2021-01-27 Thread Nikolay Borisov
On 28.01.21 г. 5:38 ч., Masami Hiramatsu wrote: > Hi, > > > Yeah, there is. Nikolay, could you try this tentative patch? I can confirm that with this patch everything is working. I also applied the following diff: diff --git a/kernel/trace/bpf_trace.c b/kernel/trace/bpf_trace.c index

Re: kprobes broken since 0d00449c7a28 ("x86: Replace ist_enter() with nmi_enter()")

2021-01-27 Thread Masami Hiramatsu
Hi, On Thu, 28 Jan 2021 10:34:15 +0900 Masami Hiramatsu wrote: > On Wed, 27 Jan 2021 19:57:56 +0200 > Nikolay Borisov wrote: > > > > > > > On 27.01.21 г. 17:24 ч., Masami Hiramatsu wrote: > > > On Thu, 28 Jan 2021 00:13:53 +0900 > > > Masami Hiramatsu wrote: > > > > > >> Hi Nikolay, > >

Re: kprobes broken since 0d00449c7a28 ("x86: Replace ist_enter() with nmi_enter()")

2021-01-27 Thread Masami Hiramatsu
On Wed, 27 Jan 2021 19:57:56 +0200 Nikolay Borisov wrote: > > > On 27.01.21 г. 17:24 ч., Masami Hiramatsu wrote: > > On Thu, 28 Jan 2021 00:13:53 +0900 > > Masami Hiramatsu wrote: > > > >> Hi Nikolay, > >> > >> On Wed, 27 Jan 2021 15:43:29 +0200 > >> Nikolay Borisov wrote: > >> > >>> Hello,

Re: kprobes broken since 0d00449c7a28 ("x86: Replace ist_enter() with nmi_enter()")

2021-01-27 Thread Nikolay Borisov
On 27.01.21 г. 17:24 ч., Masami Hiramatsu wrote: > On Thu, 28 Jan 2021 00:13:53 +0900 > Masami Hiramatsu wrote: > >> Hi Nikolay, >> >> On Wed, 27 Jan 2021 15:43:29 +0200 >> Nikolay Borisov wrote: >> >>> Hello, >>> >>> I'm currently seeing latest Linus' master being somewhat broken w.r.t >>>

Re: kprobes broken since 0d00449c7a28 ("x86: Replace ist_enter() with nmi_enter()")

2021-01-27 Thread Peter Zijlstra
On Thu, Jan 28, 2021 at 12:13:53AM +0900, Masami Hiramatsu wrote: > Peter, would you have any idea? Without a clear error, I've no clue yet. I'll try and investigate a little more once we have a little more data.

Re: kprobes broken since 0d00449c7a28 ("x86: Replace ist_enter() with nmi_enter()")

2021-01-27 Thread Masami Hiramatsu
On Thu, 28 Jan 2021 00:13:53 +0900 Masami Hiramatsu wrote: > Hi Nikolay, > > On Wed, 27 Jan 2021 15:43:29 +0200 > Nikolay Borisov wrote: > > > Hello, > > > > I'm currently seeing latest Linus' master being somewhat broken w.r.t > > krpobes. In particular I have the following test-case: > >

Re: kprobes broken since 0d00449c7a28 ("x86: Replace ist_enter() with nmi_enter()")

2021-01-27 Thread Masami Hiramatsu
Hi Nikolay, On Wed, 27 Jan 2021 15:43:29 +0200 Nikolay Borisov wrote: > Hello, > > I'm currently seeing latest Linus' master being somewhat broken w.r.t > krpobes. In particular I have the following test-case: > > #!/bin/bash > > mkfs.btrfs -f /dev/vdc &> /dev/null > mount /dev/vdc

kprobes broken since 0d00449c7a28 ("x86: Replace ist_enter() with nmi_enter()")

2021-01-27 Thread Nikolay Borisov
Hello, I'm currently seeing latest Linus' master being somewhat broken w.r.t krpobes. In particular I have the following test-case: #!/bin/bash mkfs.btrfs -f /dev/vdc &> /dev/null mount /dev/vdc /media/scratch/ bpftrace -e 'kprobe:btrfs_sync_file {printf("kprobe: %s\n", kstack());}'