Re: rb tree hrtimer lockup bug (found by perf_fuzzer)

2014-04-24 Thread Greg KH
On Thu, Apr 17, 2014 at 09:59:01AM +0200, Thomas Gleixner wrote: > On Wed, 16 Apr 2014, Greg KH wrote: > > On Thu, Apr 17, 2014 at 01:00:53AM +0200, Thomas Gleixner wrote: > > > And cdevs is an array of struct cdev: > > > > > > struct cdev { > > > struct kobject kobj; > > > > Those are not "re

Re: rb tree hrtimer lockup bug (found by perf_fuzzer)

2014-04-17 Thread Thomas Gleixner
On Wed, 16 Apr 2014, Greg KH wrote: > On Thu, Apr 17, 2014 at 01:00:53AM +0200, Thomas Gleixner wrote: > > And cdevs is an array of struct cdev: > > > > struct cdev { > > struct kobject kobj; > > Those are not "real" kobjects, and are never registered with the kobject > core. > > I really n

Re: rb tree hrtimer lockup bug (found by perf_fuzzer)

2014-04-16 Thread Greg KH
On Thu, Apr 17, 2014 at 01:00:53AM +0200, Thomas Gleixner wrote: > On Sat, 5 Apr 2014, Greg KH wrote: > > On Mon, Mar 31, 2014 at 01:18:34PM +0200, Thomas Gleixner wrote: > > > On Thu, 27 Mar 2014, Vince Weaver wrote: > > > > On Wed, 26 Mar 2014, Thomas Gleixner wrote: > > > > > Ok. So we know now

Re: rb tree hrtimer lockup bug (found by perf_fuzzer)

2014-04-16 Thread Thomas Gleixner
On Sat, 5 Apr 2014, Greg KH wrote: > On Mon, Mar 31, 2014 at 01:18:34PM +0200, Thomas Gleixner wrote: > > On Thu, 27 Mar 2014, Vince Weaver wrote: > > > On Wed, 26 Mar 2014, Thomas Gleixner wrote: > > > > Ok. So we know now what we are looking for. > > > > > > > > [1.579996] Serial: 8250/16550

Re: rb tree hrtimer lockup bug (found by perf_fuzzer)

2014-04-05 Thread Greg KH
On Mon, Mar 31, 2014 at 01:18:34PM +0200, Thomas Gleixner wrote: > On Thu, 27 Mar 2014, Vince Weaver wrote: > > On Wed, 26 Mar 2014, Thomas Gleixner wrote: > > > Ok. So we know now what we are looking for. > > > > > > [1.579996] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled > > > ÿ[

Re: rb tree hrtimer lockup bug (found by perf_fuzzer)

2014-03-31 Thread Thomas Gleixner
On Mon, 31 Mar 2014, Vince Weaver wrote: > On Mon, 31 Mar 2014, Thomas Gleixner wrote: > > > > Vince, can you please disable CONFIG_DEBUG_KOBJECT_RELEASE and remove > > all the debug patches to see whether the issue goes away? > > It makes the crash on boot go away, but it is still possible to t

Re: rb tree hrtimer lockup bug (found by perf_fuzzer)

2014-03-31 Thread Vince Weaver
On Mon, 31 Mar 2014, Thomas Gleixner wrote: > > Vince, can you please disable CONFIG_DEBUG_KOBJECT_RELEASE and remove > all the debug patches to see whether the issue goes away? It makes the crash on boot go away, but it is still possible to trigger a similar crash (hrtimer oopsing due to corrupt

Re: rb tree hrtimer lockup bug (found by perf_fuzzer)

2014-03-31 Thread Ingo Molnar
* Thomas Gleixner wrote: > On Thu, 27 Mar 2014, Vince Weaver wrote: > > On Wed, 26 Mar 2014, Thomas Gleixner wrote: > > > Ok. So we know now what we are looking for. > > > > > > [1.579996] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled > > > ÿ[1.607279] 00:09: ttyS0 at I/O 0x3f

Re: rb tree hrtimer lockup bug (found by perf_fuzzer)

2014-03-31 Thread Thomas Gleixner
On Thu, 27 Mar 2014, Vince Weaver wrote: > On Wed, 26 Mar 2014, Thomas Gleixner wrote: > > Ok. So we know now what we are looking for. > > > > [1.579996] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled > > ÿ[1.607279] 00:09: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is > >

Re: rb tree hrtimer lockup bug (found by perf_fuzzer)

2014-03-26 Thread Thomas Gleixner
On Wed, 26 Mar 2014, Vince Weaver wrote: > On Tue, 25 Mar 2014, Thomas Gleixner wrote: > > Do you happen to have the full bootlog up to the crash handy? > > attached is the full bootlog followed by the tracedump. Ok. So we know now what we are looking for. [1.579996] Serial: 8250/16550 drive

Re: rb tree hrtimer lockup bug (found by perf_fuzzer)

2014-03-25 Thread Thomas Gleixner
On Tue, 25 Mar 2014, Vince Weaver wrote: > On Mon, 24 Mar 2014, Thomas Gleixner wrote: > > > But I tooks some time to tweak the debugobjects patch further in the > > hope that it helps. > > I've attached the log from your most recent patch, it's pretty long. > > I'll try without the trace_printk

Re: rb tree hrtimer lockup bug (found by perf_fuzzer)

2014-03-23 Thread Thomas Gleixner
On Sun, 23 Mar 2014, Thomas Gleixner wrote: > On Sat, 22 Mar 2014, Thomas Gleixner wrote: > > On Sat, 22 Mar 2014, Thomas Gleixner wrote: > > > On Fri, 21 Mar 2014, Vince Weaver wrote: > > > > > > > On Fri, 21 Mar 2014, Thomas Gleixner wrote: > > > > > > > > > > I'm a complete idiot. I was starin

Re: rb tree hrtimer lockup bug (found by perf_fuzzer)

2014-03-23 Thread Thomas Gleixner
On Sat, 22 Mar 2014, Thomas Gleixner wrote: > On Sat, 22 Mar 2014, Thomas Gleixner wrote: > > On Fri, 21 Mar 2014, Vince Weaver wrote: > > > > > On Fri, 21 Mar 2014, Thomas Gleixner wrote: > > > > > > > > I'm a complete idiot. I was staring at oaddr and did not notice that > > > > descr->name is

Re: rb tree hrtimer lockup bug (found by perf_fuzzer)

2014-03-22 Thread Thomas Gleixner
On Sat, 22 Mar 2014, Thomas Gleixner wrote: > On Fri, 21 Mar 2014, Vince Weaver wrote: > > > On Fri, 21 Mar 2014, Thomas Gleixner wrote: > > > > > > I'm a complete idiot. I was staring at oaddr and did not notice that > > > descr->name is the real culprit. Sorry. Delta patch below. > > > > OK.

Re: rb tree hrtimer lockup bug (found by perf_fuzzer)

2014-03-22 Thread Thomas Gleixner
On Fri, 21 Mar 2014, Vince Weaver wrote: > On Fri, 21 Mar 2014, Thomas Gleixner wrote: > > > > I'm a complete idiot. I was staring at oaddr and did not notice that > > descr->name is the real culprit. Sorry. Delta patch below. > > OK. The log was much longer this time, attached. Hmmm. [2.

Re: rb tree hrtimer lockup bug (found by perf_fuzzer)

2014-03-21 Thread Thomas Gleixner
On Thu, 20 Mar 2014, Vince Weaver wrote: > On Thu, 20 Mar 2014, Thomas Gleixner wrote: > > > > Yeah, it causes the explosion in the debug object code. This should > > have been: > > > > + trace_printk("Tracking Object free: %s %p %pS\n", > > +

Re: rb tree hrtimer lockup bug (found by perf_fuzzer)

2014-03-20 Thread Vince Weaver
On Thu, 20 Mar 2014, Thomas Gleixner wrote: > > Yeah, it causes the explosion in the debug object code. This should > have been: > > + trace_printk("Tracking Object free: %s %p %pS\n", > +descr->name, obj->object, obj->hint); attached are

Re: rb tree hrtimer lockup bug (found by perf_fuzzer)

2014-03-20 Thread Thomas Gleixner
On Thu, 20 Mar 2014, Vince Weaver wrote: > On Thu, 20 Mar 2014, Thomas Gleixner wrote: > > Find a more brute force method to get this decoded below. Full patch > > this time, please zap the old one. That should spill out the recent > > debug object calls and we can just hope, that we get enough hi

Re: rb tree hrtimer lockup bug (found by perf_fuzzer)

2014-03-20 Thread Thomas Gleixner
On Wed, 19 Mar 2014, Vince Weaver wrote: > On Wed, 19 Mar 2014, Thomas Gleixner wrote: > > > Good, we have at least a point where we can gather useful > > information. The timer belongs to a delayed work. The work itself is > > tracked via debug objects as well. So we can get that as well. Delt

Re: rb tree hrtimer lockup bug (found by perf_fuzzer)

2014-03-19 Thread Vince Weaver
On Wed, 19 Mar 2014, Thomas Gleixner wrote: > Good, we have at least a point where we can gather useful > information. The timer belongs to a delayed work. The work itself is > tracked via debug objects as well. So we can get that as well. Delta > patch below. with that patch applied: [4.314

Re: rb tree hrtimer lockup bug (found by perf_fuzzer)

2014-03-19 Thread Thomas Gleixner
On Wed, 19 Mar 2014, Vince Weaver wrote: > On Wed, 19 Mar 2014, Thomas Gleixner wrote: > > > On Wed, 19 Mar 2014, Vince Weaver wrote: > > > On Tue, 18 Mar 2014, Thomas Gleixner wrote: > > > your patch didn't seem to print anything additional the first time throug. > > > > > > I then tried the tra

Re: rb tree hrtimer lockup bug (found by perf_fuzzer)

2014-03-19 Thread Vince Weaver
On Wed, 19 Mar 2014, Thomas Gleixner wrote: > On Wed, 19 Mar 2014, Vince Weaver wrote: > > On Tue, 18 Mar 2014, Thomas Gleixner wrote: > > your patch didn't seem to print anything additional the first time throug. > > > > I then tried the trace command you suggested, but I'm getting an empty > >

Re: rb tree hrtimer lockup bug (found by perf_fuzzer)

2014-03-19 Thread Thomas Gleixner
On Wed, 19 Mar 2014, Vince Weaver wrote: > On Tue, 18 Mar 2014, Thomas Gleixner wrote: > your patch didn't seem to print anything additional the first time throug. > > I then tried the trace command you suggested, but I'm getting an empty > ftrace buffer which possibly means I don't have enough ft

Re: rb tree hrtimer lockup bug (found by perf_fuzzer)

2014-03-19 Thread Vince Weaver
On Tue, 18 Mar 2014, Thomas Gleixner wrote: > On Tue, 18 Mar 2014, Vince Weaver wrote: > > > On Tue, 18 Mar 2014, Thomas Gleixner wrote: > > > > > > > [3.678040] BUG: unable to handle kernel NULL pointer dereference at > > > > 0018

Re: rb tree hrtimer lockup bug (found by perf_fuzzer)

2014-03-18 Thread Thomas Gleixner
On Tue, 18 Mar 2014, Vince Weaver wrote: > On Tue, 18 Mar 2014, Thomas Gleixner wrote: > > > > > [3.678040] BUG: unable to handle kernel NULL pointer dereference at > > > 0018 > > > > > > [3.686776] IP:

Re: rb tree hrtimer lockup bug (found by perf_fuzzer)

2014-03-18 Thread Vince Weaver
On Tue, 18 Mar 2014, Thomas Gleixner wrote: > > > [3.678040] BUG: unable to handle kernel NULL pointer dereference at > > 0018 > > > > [3.686776] IP: [] > > get_next_timer_interrupt+0x168/0x250 > >

Re: rb tree hrtimer lockup bug (found by perf_fuzzer)

2014-03-18 Thread Thomas Gleixner
On Tue, 18 Mar 2014, Vince Weaver wrote: > On Tue, 18 Mar 2014, Thomas Gleixner wrote: > > > On Tue, 18 Mar 2014, Vince Weaver wrote: > > > > > > > > The perf_fuzzer can quickly cause a machine to lockup with an hrtimer > > > related rb tree related oops. I've had a hard time debugging this in

Re: rb tree hrtimer lockup bug (found by perf_fuzzer)

2014-03-18 Thread Vince Weaver
On Tue, 18 Mar 2014, Thomas Gleixner wrote: > On Tue, 18 Mar 2014, Vince Weaver wrote: > > > > > The perf_fuzzer can quickly cause a machine to lockup with an hrtimer > > related rb tree related oops. I've had a hard time debugging this in any > > useful manner, but I can trigger it on both c

Re: rb tree hrtimer lockup bug (found by perf_fuzzer)

2014-03-18 Thread Thomas Gleixner
On Tue, 18 Mar 2014, Vince Weaver wrote: > > The perf_fuzzer can quickly cause a machine to lockup with an hrtimer > related rb tree related oops. I've had a hard time debugging this in any > useful manner, but I can trigger it on both core2 and haswell test systems > on 3.14-rc7. > > This i

Re: rb tree hrtimer lockup bug (found by perf_fuzzer)

2014-03-18 Thread Vince Weaver
The perf_fuzzer can quickly cause a machine to lockup with an hrtimer related rb tree related oops. I've had a hard time debugging this in any useful manner, but I can trigger it on both core2 and haswell test systems on 3.14-rc7. This involves making a large number of perf_event events of al