Re: perf_fuzzer compiled for x32 causes reboot

2014-03-04 Thread Borislav Petkov
On Sat, Mar 01, 2014 at 08:50:17AM -0800, H. Peter Anvin wrote: > The bottom line is that if we want hard numbers we probably have to > measure. > > Hoisting the cr2 read is a no-brainer, might even help performance... Btw, I just got word that on AMD, a read from CR2 is 4 cycles on family 0x15 an

Re: perf_fuzzer compiled for x32 causes reboot

2014-03-03 Thread Peter Zijlstra
On Sat, Mar 01, 2014 at 10:16:50AM +0100, Ingo Molnar wrote: > We read CR2 in the page fault hot path, so it's on the top of CPU > architects' minds and it's reasonably optimized. A couple of cycles > IIRC, but would be nice to hear actual numbers. Well, going with what Linus found; it looks lik

Re: perf_fuzzer compiled for x32 causes reboot

2014-03-02 Thread Vince Weaver
On Fri, 28 Feb 2014, Steven Rostedt wrote: > On Fri, 28 Feb 2014 18:34:00 -0500 (EST) > Vince Weaver wrote: > > But perf_event bug finder is a much more prestigious title than > "college professor" ;-) yes, it's something to fall back on if/when I get denied tenure :) I do enjoy tracking down b

Re: perf_fuzzer compiled for x32 causes reboot

2014-03-02 Thread Vince Weaver
On Sat, 1 Mar 2014, Andi Kleen wrote: > Steven Rostedt writes: > > > > BTW, is the perf_fuzzer code posted somewhere? It sounds like it can be > > really useful for us to do our own testing too. > > I believe it's part of trinity. > > http://codemonkey.org.uk/projects/trinity/ > > Perhaps it s

Re: perf_fuzzer compiled for x32 causes reboot

2014-03-01 Thread H. Peter Anvin
The bottom line is that if we want hard numbers we probably have to measure. Hoisting the cr2 read is a no-brainer, might even help performance... On March 1, 2014 1:50:42 AM PST, Borislav Petkov wrote: >On Sat, Mar 01, 2014 at 10:16:50AM +0100, Ingo Molnar wrote: >> >> * Steven Rostedt wrote:

Re: perf_fuzzer compiled for x32 causes reboot

2014-03-01 Thread Andi Kleen
Steven Rostedt writes: > > BTW, is the perf_fuzzer code posted somewhere? It sounds like it can be > really useful for us to do our own testing too. I believe it's part of trinity. http://codemonkey.org.uk/projects/trinity/ Perhaps it should have a "ftracer fuzzer" too? -Andi -- To unsubscribe

Re: perf_fuzzer compiled for x32 causes reboot

2014-03-01 Thread Borislav Petkov
On Sat, Mar 01, 2014 at 10:16:50AM +0100, Ingo Molnar wrote: > > * Steven Rostedt wrote: > > > > Also, this function is called a _LOT_ under certain workloads, I > > > don't know how cheap a CR2 read is, but it had better be really > > > cheap. > > > > That's a HPA question. > > We read CR2

Re: perf_fuzzer compiled for x32 causes reboot

2014-03-01 Thread Ingo Molnar
* Steven Rostedt wrote: > > Also, this function is called a _LOT_ under certain workloads, I > > don't know how cheap a CR2 read is, but it had better be really > > cheap. > > That's a HPA question. We read CR2 in the page fault hot path, so it's on the top of CPU architects' minds and it's

Re: perf_fuzzer compiled for x32 causes reboot

2014-02-28 Thread Steven Rostedt
On Fri, 28 Feb 2014 18:34:00 -0500 (EST) Vince Weaver wrote: > > I was poking fun at you on IRC for this exact reason: > > > > poor Vince, I keep sending him new patches. "No, don't test this > > patch, now test this one. Oh wait, try this one instead" > > * peterz sees Vince thinking: "stop..

Re: perf_fuzzer compiled for x32 causes reboot

2014-02-28 Thread H. Peter Anvin
On 02/28/2014 03:34 PM, Vince Weaver wrote: > > Well while it might appear that I spend all of my days finding perf_event > bugs, I actually am a college professor so I do occasionally have to run > off to teach a class, meet with students, or write papers/grants for other > academics to reject

Re: perf_fuzzer compiled for x32 causes reboot

2014-02-28 Thread Vince Weaver
On Fri, 28 Feb 2014, Steven Rostedt wrote: > On Fri, 28 Feb 2014 16:18:23 -0500 (EST) > Vince Weaver wrote: > > > I was away from the computer this afternoon and of course I have scores of > > e-mails on this topic now with lots of competing patches. Is there one > > in particular I'm suppose

Re: perf_fuzzer compiled for x32 causes reboot

2014-02-28 Thread Paul E. McKenney
On Fri, Feb 28, 2014 at 05:05:53PM -0500, Steven Rostedt wrote: > On Fri, 28 Feb 2014 22:55:11 +0100 > Peter Zijlstra wrote: > > > On Fri, Feb 28, 2014 at 01:51:50PM -0800, Paul E. McKenney wrote: > > > On Fri, Feb 28, 2014 at 10:27:00PM +0100, Peter Zijlstra wrote: > > > > On Fri, Feb 28, 2014 a

Re: perf_fuzzer compiled for x32 causes reboot

2014-02-28 Thread Steven Rostedt
On Fri, 28 Feb 2014 22:55:11 +0100 Peter Zijlstra wrote: > On Fri, Feb 28, 2014 at 01:51:50PM -0800, Paul E. McKenney wrote: > > On Fri, Feb 28, 2014 at 10:27:00PM +0100, Peter Zijlstra wrote: > > > On Fri, Feb 28, 2014 at 01:17:33PM -0800, Paul E. McKenney wrote: > > > > This code isn't running

Re: perf_fuzzer compiled for x32 causes reboot

2014-02-28 Thread Peter Zijlstra
On Fri, Feb 28, 2014 at 01:51:50PM -0800, Paul E. McKenney wrote: > On Fri, Feb 28, 2014 at 10:27:00PM +0100, Peter Zijlstra wrote: > > On Fri, Feb 28, 2014 at 01:17:33PM -0800, Paul E. McKenney wrote: > > > This code isn't running in idle context is it? If so, RCU will happily > > > free out from

Re: perf_fuzzer compiled for x32 causes reboot

2014-02-28 Thread Paul E. McKenney
On Fri, Feb 28, 2014 at 10:27:00PM +0100, Peter Zijlstra wrote: > On Fri, Feb 28, 2014 at 01:17:33PM -0800, Paul E. McKenney wrote: > > This code isn't running in idle context is it? If so, RCU will happily > > free out from under it. CONFIG_PROVE_RCU should detect this sort of thing, > > though.

Re: perf_fuzzer compiled for x32 causes reboot

2014-02-28 Thread Steven Rostedt
On Fri, 28 Feb 2014 16:18:23 -0500 (EST) Vince Weaver wrote: > I was away from the computer this afternoon and of course I have scores of > e-mails on this topic now with lots of competing patches. Is there one > in particular I'm supposed to be testing? I was poking fun at you on IRC for thi

Re: perf_fuzzer compiled for x32 causes reboot

2014-02-28 Thread Peter Zijlstra
On Fri, Feb 28, 2014 at 01:17:33PM -0800, Paul E. McKenney wrote: > This code isn't running in idle context is it? If so, RCU will happily > free out from under it. CONFIG_PROVE_RCU should detect this sort of thing, > though. Well, interrupts/NMIs can happen when idle, but the interrupt/NMI entr

Re: perf_fuzzer compiled for x32 causes reboot

2014-02-28 Thread Paul E. McKenney
On Fri, Feb 28, 2014 at 09:54:09PM +0100, Peter Zijlstra wrote: > On Fri, Feb 28, 2014 at 03:47:16PM -0500, Steven Rostedt wrote: > > > > I'll try your patch momentarily, first I had some other changes I > > > > started > > > > running before I left work (for some reason it recompiled the whole

Re: perf_fuzzer compiled for x32 causes reboot

2014-02-28 Thread Vince Weaver
On Fri, 28 Feb 2014, H. Peter Anvin wrote: > Now we need to figure out if the reboot problem and the segfault problem > are actually the same... I have a nasty feeling they might be different > problems. I'm currently running a script that tries setting EBP to all possible 32-bit pages and run

Re: perf_fuzzer compiled for x32 causes reboot

2014-02-28 Thread Steven Rostedt
On Fri, 28 Feb 2014 21:56:38 +0100 Peter Zijlstra wrote: > Like already said; _trace is an absolutely abysmal name. Also you > _really_ don't want an unconditional CR2 write in there, that's just > stupidly expensive. But a read isn't. Which is why we only do a write if the copy caused a page f

Re: perf_fuzzer compiled for x32 causes reboot

2014-02-28 Thread Peter Zijlstra
On Fri, Feb 28, 2014 at 11:29:46AM -0500, Steven Rostedt wrote: > On Fri, 28 Feb 2014 08:15:11 -0800 > "H. Peter Anvin" wrote: > > > Well, I was talking about the assumption spelled out in the comment > > above copy_from_user_nmi() which pretty much states "cr2 is safe because > > cr2 is saved/re

Re: perf_fuzzer compiled for x32 causes reboot

2014-02-28 Thread Peter Zijlstra
On Fri, Feb 28, 2014 at 08:15:11AM -0800, H. Peter Anvin wrote: > On 02/28/2014 07:40 AM, Peter Zijlstra wrote: > > On Fri, Feb 28, 2014 at 07:13:06AM -0800, H. Peter Anvin wrote: > >> If I'm reading this right we end up going from the page fault > >> tracepoint to copy_from_user_nmi() without goin

Re: perf_fuzzer compiled for x32 causes reboot

2014-02-28 Thread Peter Zijlstra
On Fri, Feb 28, 2014 at 03:47:16PM -0500, Steven Rostedt wrote: > > > I'll try your patch momentarily, first I had some other changes I started > > > running before I left work (for some reason it recompiled the whole > > > kernel). > > > > > > 8: function: perf_output_begin > > > 8:

Re: perf_fuzzer compiled for x32 causes reboot

2014-02-28 Thread Steven Rostedt
On Fri, 28 Feb 2014 12:34:05 -0800 "Paul E. McKenney" wrote: > On Thu, Feb 27, 2014 at 08:00:04PM -0500, Vince Weaver wrote: > > On Thu, 27 Feb 2014, H. Peter Anvin wrote: > > > > > On 02/27/2014 03:30 PM, Steven Rostedt wrote: > > > > On Thu, 27 Feb 2014 14:52:54 -0800 > > > > "H. Peter Anvin"

Re: perf_fuzzer compiled for x32 causes reboot

2014-02-28 Thread Steven Rostedt
On Fri, 28 Feb 2014 12:38:52 -0800 "H. Peter Anvin" wrote: > Now we need to figure out if the reboot problem and the segfault problem are > actually the same... I have a nasty feeling they might be different problems. I wonder if there was any recursion problem. Although, I believe perf has rec

Re: perf_fuzzer compiled for x32 causes reboot

2014-02-28 Thread H. Peter Anvin
Now we need to figure out if the reboot problem and the segfault problem are actually the same... I have a nasty feeling they might be different problems. On February 28, 2014 7:07:29 AM PST, Vince Weaver wrote: >On Fri, 28 Feb 2014, Steven Rostedt wrote: > >> Interesting. Are you doing a perf

Re: perf_fuzzer compiled for x32 causes reboot

2014-02-28 Thread Paul E. McKenney
On Thu, Feb 27, 2014 at 08:00:04PM -0500, Vince Weaver wrote: > On Thu, 27 Feb 2014, H. Peter Anvin wrote: > > > On 02/27/2014 03:30 PM, Steven Rostedt wrote: > > > On Thu, 27 Feb 2014 14:52:54 -0800 > > > "H. Peter Anvin" wrote: > > > > > >> On 02/27/2014 02:31 PM, Steven Rostedt wrote: > > >>>

Re: perf_fuzzer compiled for x32 causes reboot

2014-02-28 Thread Steven Rostedt
On Fri, 28 Feb 2014 08:15:11 -0800 "H. Peter Anvin" wrote: > Well, I was talking about the assumption spelled out in the comment > above copy_from_user_nmi() which pretty much states "cr2 is safe because > cr2 is saved/restored in the NMI wrappers." Yeah, it seems that the name "copy_from_user_n

Re: perf_fuzzer compiled for x32 causes reboot

2014-02-28 Thread H. Peter Anvin
On 02/28/2014 07:40 AM, Peter Zijlstra wrote: > On Fri, Feb 28, 2014 at 07:13:06AM -0800, H. Peter Anvin wrote: >> If I'm reading this right we end up going from the page fault >> tracepoint to copy_from_user_nmi() without going through NMI, and the >> cr2 corruption is obvious. I guess the assump

Re: perf_fuzzer compiled for x32 causes reboot

2014-02-28 Thread Peter Zijlstra
On Fri, Feb 28, 2014 at 07:13:06AM -0800, H. Peter Anvin wrote: > If I'm reading this right we end up going from the page fault > tracepoint to copy_from_user_nmi() without going through NMI, and the > cr2 corruption is obvious. I guess the assumption that only the NMI > path needed to save cr2 is

Re: perf_fuzzer compiled for x32 causes reboot

2014-02-28 Thread Steven Rostedt
On Fri, 28 Feb 2014 10:20:00 -0500 Steven Rostedt wrote: > Below is a patch that should fix this. Please remove all other patches > and try this out. Updated patch, as Peter Zijlstra on IRC asked me if the exception_enter() can be traced. And looking at it, it sure can be. -- Steve diff --git

Re: perf_fuzzer compiled for x32 causes reboot

2014-02-28 Thread Steven Rostedt
On Fri, 28 Feb 2014 10:07:29 -0500 (EST) Vince Weaver wrote: > On Fri, 28 Feb 2014, Steven Rostedt wrote: > 199.900696: function: __module_address > ... > 199.900705: function: __kernel_text_address > 199.900809: kernel_stack: > => perf

Re: perf_fuzzer compiled for x32 causes reboot

2014-02-28 Thread H. Peter Anvin
If I'm reading this right we end up going from the page fault tracepoint to copy_from_user_nmi() without going through NMI, and the cr2 corruption is obvious. I guess the assumption that only the NMI path needed to save cr2 is flawed? On February 28, 2014 7:07:29 AM PST, Vince Weaver wrote:

Re: perf_fuzzer compiled for x32 causes reboot

2014-02-28 Thread Vince Weaver
On Fri, 28 Feb 2014, Steven Rostedt wrote: > Interesting. Are you doing a perf function trace? > > And just in case, can you add this patch and make sure the copy is > called by NMI. 199.900682: function: trace_do_page_fault 199.900683: page_fault_user: address=__per_cpu_end ip=

Re: perf_fuzzer compiled for x32 causes reboot

2014-02-28 Thread Steven Rostedt
On Fri, 28 Feb 2014 09:15:33 -0500 (EST) Vince Weaver wrote: > On Thu, 27 Feb 2014, Steven Rostedt wrote: > > > On Thu, 27 Feb 2014 20:34:34 -0500 (EST) > > Vince Weaver wrote: > > > > > > > > I would actually suggest we do the equivalent on i386 as well. > > > > > > > > Vince, could you try

Re: perf_fuzzer compiled for x32 causes reboot

2014-02-28 Thread Vince Weaver
On Thu, 27 Feb 2014, Steven Rostedt wrote: > On Thu, 27 Feb 2014 20:34:34 -0500 (EST) > Vince Weaver wrote: > > > > > I would actually suggest we do the equivalent on i386 as well. > > > > > > Vince, could you try this patch as an experiment? > > > > OK with your patch applied it does not seg

Re: perf_fuzzer compiled for x32 causes reboot

2014-02-28 Thread Steven Rostedt
On Fri, 28 Feb 2014 12:11:11 +0100 Peter Zijlstra wrote: > On Thu, Feb 27, 2014 at 09:57:26PM -0500, Steven Rostedt wrote: > > @@ -512,8 +508,21 @@ static inline void nmi_nesting_postprocess(void) > > dotraplinkage notrace __kprobes void > > do_nmi(struct pt_regs *regs, long error_code) > > {

Re: perf_fuzzer compiled for x32 causes reboot

2014-02-28 Thread Peter Zijlstra
On Thu, Feb 27, 2014 at 09:57:26PM -0500, Steven Rostedt wrote: > @@ -512,8 +508,21 @@ static inline void nmi_nesting_postprocess(void) > dotraplinkage notrace __kprobes void > do_nmi(struct pt_regs *regs, long error_code) > { > + unsigned long cr2; > + > nmi_nesting_preprocess(re

Re: perf_fuzzer compiled for x32 causes reboot

2014-02-28 Thread Peter Zijlstra
On Thu, Feb 27, 2014 at 05:31:50PM -0500, Steven Rostedt wrote: > Well, the perf ring buffer is vmalloced, right? That can cause a page > fault too. On x86 they're typically not -- although we have a debug CONFIG option to test that code on x86 too. On SPARC/ARM etc.. we have to use vmalloc_user()

Re: perf_fuzzer compiled for x32 causes reboot

2014-02-27 Thread Steven Rostedt
On Thu, 27 Feb 2014 20:34:34 -0500 (EST) Vince Weaver wrote: > > I would actually suggest we do the equivalent on i386 as well. > > > > Vince, could you try this patch as an experiment? > > OK with your patch applied it does not segfault. > Vince, Great! Can you remove Peter's patch, and try

Re: perf_fuzzer compiled for x32 causes reboot

2014-02-27 Thread H. Peter Anvin
Ok... I think we're definitely talking about a cr2 leak. The reboot might be a race condition in the NMI nesting handling maybe? On February 27, 2014 5:34:34 PM PST, Vince Weaver wrote: >On Thu, 27 Feb 2014, H. Peter Anvin wrote: > >> On 02/27/2014 02:31 PM, Steven Rostedt wrote: >> > >> > Ye

Re: perf_fuzzer compiled for x32 causes reboot

2014-02-27 Thread Vince Weaver
On Thu, 27 Feb 2014, H. Peter Anvin wrote: > On 02/27/2014 02:31 PM, Steven Rostedt wrote: > > > > Yeah, something is getting mesed up. > > > > What it *looks* like to me is that we try to nest the cr2 save/restore, > which doesn't nest because it is a percpu variable. > > ... except in the x8

Re: perf_fuzzer compiled for x32 causes reboot

2014-02-27 Thread Vince Weaver
On Thu, 27 Feb 2014, H. Peter Anvin wrote: > On 02/27/2014 03:30 PM, Steven Rostedt wrote: > > On Thu, 27 Feb 2014 14:52:54 -0800 > > "H. Peter Anvin" wrote: > > > >> On 02/27/2014 02:31 PM, Steven Rostedt wrote: > >>> > >>> Yeah, something is getting mesed up. > >>> > >> > >> What it *looks* li

Re: perf_fuzzer compiled for x32 causes reboot

2014-02-27 Thread H. Peter Anvin
On 02/27/2014 03:30 PM, Steven Rostedt wrote: > On Thu, 27 Feb 2014 14:52:54 -0800 > "H. Peter Anvin" wrote: > >> On 02/27/2014 02:31 PM, Steven Rostedt wrote: >>> >>> Yeah, something is getting mesed up. >>> >> >> What it *looks* like to me is that we try to nest the cr2 save/restore, >> which d

Re: perf_fuzzer compiled for x32 causes reboot

2014-02-27 Thread Steven Rostedt
On Thu, 27 Feb 2014 14:52:54 -0800 "H. Peter Anvin" wrote: > On 02/27/2014 02:31 PM, Steven Rostedt wrote: > > > > Yeah, something is getting mesed up. > > > > What it *looks* like to me is that we try to nest the cr2 save/restore, > which doesn't nest because it is a percpu variable. > > ...

Re: perf_fuzzer compiled for x32 causes reboot

2014-02-27 Thread H. Peter Anvin
On 02/27/2014 02:31 PM, Steven Rostedt wrote: > > Yeah, something is getting mesed up. > What it *looks* like to me is that we try to nest the cr2 save/restore, which doesn't nest because it is a percpu variable. ... except in the x86-64 case, we *ALSO* save/restore cr2 inside entry_64.S, which

Re: perf_fuzzer compiled for x32 causes reboot

2014-02-27 Thread Steven Rostedt
On Thu, 27 Feb 2014 17:06:36 -0500 (EST) Vince Weaver wrote: > > I spent some more time on this. > I managed to get a trace that exhibited the bug practically right > away, but still unable to generate a reproducible trace :( > > So instead I'm adding WARN's and trace_printks to see what I c

Re: perf_fuzzer compiled for x32 causes reboot

2014-02-27 Thread Vince Weaver
I spent some more time on this. I managed to get a trace that exhibited the bug practically right away, but still unable to generate a reproducible trace :( So instead I'm adding WARN's and trace_printks to see what I can find out. Here's a summary of what I think is happneing. Please let me

Re: perf_fuzzer compiled for x32 causes reboot

2014-02-26 Thread Vince Weaver
On Tue, 25 Feb 2014, Vince Weaver wrote: > On Tue, 25 Feb 2014, Steven Rostedt wrote: > > > On Tue, 25 Feb 2014 06:34:55 -0800 > > "H. Peter Anvin" wrote: > > > > > #2 is what I really don't understand. > > > > > > I worry something else is going on there > > > > Yeah, me too. > > > > OK, w

Re: perf_fuzzer compiled for x32 causes reboot

2014-02-25 Thread Vince Weaver
On Tue, 25 Feb 2014, Steven Rostedt wrote: > On Tue, 25 Feb 2014 06:34:55 -0800 > "H. Peter Anvin" wrote: > > > #2 is what I really don't understand. > > > > I worry something else is going on there > > Yeah, me too. > OK, well I'll work on isolating that next, I was hoping the segfault issu

Re: perf_fuzzer compiled for x32 causes reboot

2014-02-25 Thread Steven Rostedt
On Tue, 25 Feb 2014 06:34:55 -0800 "H. Peter Anvin" wrote: > #2 is what I really don't understand. > > I worry something else is going on there Yeah, me too. -- Steve > > > > >While the missing cr2 issue made debugging frustrating, I find the > >other > >aspects of the bug more serious: > >

Re: perf_fuzzer compiled for x32 causes reboot

2014-02-25 Thread H. Peter Anvin
#2 is what I really don't understand. I worry something else is going on there On February 25, 2014 6:07:51 AM PST, Vince Weaver wrote: >On Mon, 24 Feb 2014, H. Peter Anvin wrote: > >> On 02/24/2014 11:30 AM, Peter Zijlstra wrote: >> > On Mon, Feb 24, 2014 at 02:13:29PM -0500, Steven Rostedt wr

Re: perf_fuzzer compiled for x32 causes reboot

2014-02-25 Thread Vince Weaver
On Mon, 24 Feb 2014, H. Peter Anvin wrote: > On 02/24/2014 11:30 AM, Peter Zijlstra wrote: > > On Mon, Feb 24, 2014 at 02:13:29PM -0500, Steven Rostedt wrote: > >> Ah, and x86_64 saves off the cr2 register when entering NMI and restores > >> it before returning. But it seems to be missing from the

Re: perf_fuzzer compiled for x32 causes reboot

2014-02-24 Thread H. Peter Anvin
On 02/24/2014 11:30 AM, Peter Zijlstra wrote: > On Mon, Feb 24, 2014 at 02:13:29PM -0500, Steven Rostedt wrote: >> Ah, and x86_64 saves off the cr2 register when entering NMI and restores >> it before returning. But it seems to be missing from the i386 code. > > arch/x86/kernel/nmi.c: > > #define

Re: perf_fuzzer compiled for x32 causes reboot

2014-02-24 Thread Steven Rostedt
On Mon, 24 Feb 2014 20:30:43 +0100 Peter Zijlstra wrote: > On Mon, Feb 24, 2014 at 02:13:29PM -0500, Steven Rostedt wrote: > > Ah, and x86_64 saves off the cr2 register when entering NMI and restores > > it before returning. But it seems to be missing from the i386 code. > > arch/x86/kernel/nmi.

Re: perf_fuzzer compiled for x32 causes reboot

2014-02-24 Thread Peter Zijlstra
On Mon, Feb 24, 2014 at 02:13:29PM -0500, Steven Rostedt wrote: > Ah, and x86_64 saves off the cr2 register when entering NMI and restores > it before returning. But it seems to be missing from the i386 code. arch/x86/kernel/nmi.c: #define nmi_nesting_preprocess(regs)

Re: perf_fuzzer compiled for x32 causes reboot

2014-02-24 Thread H. Peter Anvin
On 02/24/2014 11:13 AM, Steven Rostedt wrote: >> >> Either way, it really seems like we have a case of CR2 leakage out of >> the NMI context. > > Ah, and x86_64 saves off the cr2 register when entering NMI and restores > it before returning. But it seems to be missing from the i386 code. > OK, t

Re: perf_fuzzer compiled for x32 causes reboot

2014-02-24 Thread Steven Rostedt
On Mon, 24 Feb 2014 10:34:13 -0800 "H. Peter Anvin" wrote: > On 02/24/2014 10:07 AM, Vince Weaver wrote: > >> > >> Anyway I've attached the full tail end of the trace if you want to see > >> everything that happens. > > > > and then I note there are *two* kernel page faults. > > > > perf_

Re: perf_fuzzer compiled for x32 causes reboot

2014-02-24 Thread H. Peter Anvin
On 02/24/2014 10:07 AM, Vince Weaver wrote: >> >> Anyway I've attached the full tail end of the trace if you want to see >> everything that happens. > > and then I note there are *two* kernel page faults. > > perf_fuzzer-2979 [000] 161.475924: page_fault_kernel: > address=irq_stack_u

Re: perf_fuzzer compiled for x32 causes reboot

2014-02-24 Thread Vince Weaver
On Mon, 24 Feb 2014, Vince Weaver wrote: > On Mon, 24 Feb 2014, H. Peter Anvin wrote: > > > On 02/24/2014 09:32 AM, Vince Weaver wrote: > > >> > > >> Peter, does x32 have a slightly different ABI/calling convention that > > >> would make any of these patches just slightly 'off'? > > > > > > I do

Re: perf_fuzzer compiled for x32 causes reboot

2014-02-24 Thread H. Peter Anvin
On 02/24/2014 09:25 AM, Peter Zijlstra wrote: >> >> What is likely happening is the user page fault is triggering >> code to do a "perf_callchain" dump, which is calling copy_from_user_nmi() >> which calls copy_user_generic_string() which is somehow getting the user >> RBP in the RDI register some

Re: perf_fuzzer compiled for x32 causes reboot

2014-02-24 Thread H. Peter Anvin
On 02/24/2014 09:41 AM, Vince Weaver wrote: > On Mon, 24 Feb 2014, Vince Weaver wrote: > >> I do note that >> perf_callchain_user(); >> >> Does >> fp = (void __user *)regs->bp; >> >> ... >> >> bytes = copy_from_user_nmi(&frame, fp, sizeof(frame)); >> >> >> And in my part

Re: perf_fuzzer compiled for x32 causes reboot

2014-02-24 Thread Peter Zijlstra
On Mon, Feb 24, 2014 at 12:32:39PM -0500, Vince Weaver wrote: > I do note that > perf_callchain_user(); > > Does > fp = (void __user *)regs->bp; > > ... > > bytes = copy_from_user_nmi(&frame, fp, sizeof(frame)); > > > And in my particular executable RBP has nothi

Re: perf_fuzzer compiled for x32 causes reboot

2014-02-24 Thread H. Peter Anvin
On 02/24/2014 09:32 AM, Vince Weaver wrote: >> >> Peter, does x32 have a slightly different ABI/calling convention that >> would make any of these patches just slightly 'off'? > > I do note that > perf_callchain_user(); > > Does > fp = (void __user *)regs->bp; > > ... >

Re: perf_fuzzer compiled for x32 causes reboot

2014-02-24 Thread Vince Weaver
On Mon, 24 Feb 2014, Vince Weaver wrote: > I do note that > perf_callchain_user(); > > Does > fp = (void __user *)regs->bp; > > ... > > bytes = copy_from_user_nmi(&frame, fp, sizeof(frame)); > > > And in my particular executable RBP has nothing to do with a fram

Re: perf_fuzzer compiled for x32 causes reboot

2014-02-24 Thread Vince Weaver
On Mon, 24 Feb 2014, Peter Zijlstra wrote: > On Mon, Feb 24, 2014 at 12:10:44PM -0500, Vince Weaver wrote: > > On Mon, 24 Feb 2014, H. Peter Anvin wrote: > > > > > On February 24, 2014 8:34:30 AM PST, Vince Weaver > > > wrote: > > > >On Mon, 24 Feb 2014, Vince Weaver wrote: > > > > > > > >> Jus

Re: perf_fuzzer compiled for x32 causes reboot

2014-02-24 Thread Peter Zijlstra
On Mon, Feb 24, 2014 at 12:10:44PM -0500, Vince Weaver wrote: > On Mon, 24 Feb 2014, H. Peter Anvin wrote: > > > On February 24, 2014 8:34:30 AM PST, Vince Weaver > > wrote: > > >On Mon, 24 Feb 2014, Vince Weaver wrote: > > > > > >> Just touching the mmap page with a write of a single byte (it d

Re: perf_fuzzer compiled for x32 causes reboot

2014-02-24 Thread Vince Weaver
On Mon, 24 Feb 2014, H. Peter Anvin wrote: > On February 24, 2014 8:34:30 AM PST, Vince Weaver > wrote: > >On Mon, 24 Feb 2014, Vince Weaver wrote: > > > >> Just touching the mmap page with a write of a single byte (it doesn't > > > >> matter where) is enough to trigger the bug. > > > >OK, inves

Re: perf_fuzzer compiled for x32 causes reboot

2014-02-24 Thread H. Peter Anvin
Ok, so the obvious question is what is at that kernel address? On February 24, 2014 8:34:30 AM PST, Vince Weaver wrote: >On Mon, 24 Feb 2014, Vince Weaver wrote: > >> Just touching the mmap page with a write of a single byte (it doesn't > >> matter where) is enough to trigger the bug. > >OK, inv

Re: perf_fuzzer compiled for x32 causes reboot

2014-02-24 Thread Vince Weaver
On Mon, 24 Feb 2014, Vince Weaver wrote: > Just touching the mmap page with a write of a single byte (it doesn't > matter where) is enough to trigger the bug. OK, investigating this more. perf_fuzzer-2971 [000] 154.944114: page_fault_user: address=0xf7729000 ip=0x41efab error_cod

Re: perf_fuzzer compiled for x32 causes reboot

2014-02-24 Thread Vince Weaver
On Sun, 23 Feb 2014, H. Peter Anvin wrote: > So we do a write to the buffer rather immediately before this happens, > and in particular that will update the head: > > rb->user_page->data_head = head; > > However, that doesn't explain what is going on and in particular the > write to whatev

Re: perf_fuzzer compiled for x32 causes reboot

2014-02-23 Thread H. Peter Anvin
On 02/23/2014 07:02 PM, Vince Weaver wrote: > On Sun, 23 Feb 2014, Vince Weaver wrote: >> >> and as far as I can tell nothing touches rbp again until the segfault. >> Nothing in _memset_sse2 does as far as I can tell. > > I only know enough about ftrace to be dangerous, but here is what I think >

Re: perf_fuzzer compiled for x32 causes reboot

2014-02-23 Thread Vince Weaver
On Sun, 23 Feb 2014, Vince Weaver wrote: > > and as far as I can tell nothing touches rbp again until the segfault. > Nothing in _memset_sse2 does as far as I can tell. I only know enough about ftrace to be dangerous, but here is what I think is the trace of the problem: perf_fuzzer-11492

Re: perf_fuzzer compiled for x32 causes reboot

2014-02-23 Thread Vince Weaver
On Sat, 22 Feb 2014, H. Peter Anvin wrote: > I'd be interested in how rbp gets set, too. It might just be a > coincidence and the value in rbp has some other meaning here. The code in question does this: i=find_random_active_event(); if (i<0) return; if ((event_data[i].

Re: perf_fuzzer compiled for x32 causes reboot

2014-02-22 Thread H. Peter Anvin
I'd be interested in how rbp gets set, too. It might just be a coincidence and the value in rbp has some other meaning here. On February 22, 2014 9:18:17 PM PST, Vince Weaver wrote: >On Fri, 21 Feb 2014, H. Peter Anvin wrote: > >> Error 6 reflects a write in userspace to a not-present page. >>

Re: perf_fuzzer compiled for x32 causes reboot

2014-02-22 Thread H. Peter Anvin
What is the instructions around it, by any chance? On February 22, 2014 9:18:17 PM PST, Vince Weaver wrote: >On Fri, 21 Feb 2014, H. Peter Anvin wrote: > >> Error 6 reflects a write in userspace to a not-present page. >> >> Since your previous trace indicates that the value of the register in >

Re: perf_fuzzer compiled for x32 causes reboot

2014-02-22 Thread Vince Weaver
On Fri, 21 Feb 2014, H. Peter Anvin wrote: > Error 6 reflects a write in userspace to a not-present page. > > Since your previous trace indicates that the value of the register in question > is a different one, I'm guessing that what we have here is PEBS getting > activated. 0x120 is 2*0x90, and

Re: perf_fuzzer compiled for x32 causes reboot

2014-02-21 Thread H. Peter Anvin
On 02/21/2014 08:50 PM, Vince Weaver wrote: So I changed the perf_fuzzer so when it randomly stomps all over the perf_event_mmap_page, it uses a constant value of 0xdeadbeef rather than a random value. The result is below. The segfaults make a bit more sense now, it almost looks like what is h

Re: perf_fuzzer compiled for x32 causes reboot

2014-02-21 Thread H. Peter Anvin
Those are segfaults in user space, though? On February 21, 2014 8:50:38 PM PST, Vince Weaver wrote: > >So I changed the perf_fuzzer so when it randomly stomps all over the >perf_event_mmap_page, it uses a constant value of 0xdeadbeef rather >than a random value. > >The result is below. The segf

Re: perf_fuzzer compiled for x32 causes reboot

2014-02-21 Thread Vince Weaver
So I changed the perf_fuzzer so when it randomly stomps all over the perf_event_mmap_page, it uses a constant value of 0xdeadbeef rather than a random value. The result is below. The segfaults make a bit more sense now, it almost looks like what is happening is we are corrupting an address value

Re: perf_fuzzer compiled for x32 causes reboot

2014-02-21 Thread Vince Weaver
On Fri, 21 Feb 2014, Vince Weaver wrote: > On Fri, 21 Feb 2014, Vince Weaver wrote: > > > So I'm not sure who exactly to report this to. Some perf people CC'd as > > I trigger it while using the perf_fuzzer. > > > > This is with 3.14-rc3 on a core2 machine, although I've had the reboots > > ha

Re: perf_fuzzer compiled for x32 causes reboot

2014-02-21 Thread Vince Weaver
cc'ing x32 people On Fri, 21 Feb 2014, Vince Weaver wrote: > So I'm not sure who exactly to report this to. Some perf people CC'd as > I trigger it while using the perf_fuzzer. > > This is with 3.14-rc3 on a core2 machine, although I've had the reboots > happen throughout at least 3.14-rc* >