Re: 3.4.4-rt13: btrfs + xfstests 006 = BOOM.. and a bonus rt_mutex deadlock report for absolutely free!

2012-07-17 Thread Mike Galbraith
On Tue, 2012-07-17 at 06:18 +0200, Mike Galbraith wrote: > On Mon, 2012-07-16 at 13:03 -0400, Steven Rostedt wrote: > > On Mon, 2012-07-16 at 18:36 +0200, Mike Galbraith wrote: > > > > > > > > Ouch, you just turned the rt_read_lock() into a spin lock. If a higher > > > > > priority process pree

Re: 3.4.4-rt13: btrfs + xfstests 006 = BOOM.. and a bonus rt_mutex deadlock report for absolutely free!

2012-07-16 Thread Mike Galbraith
On Tue, 2012-07-17 at 00:34 -0400, Steven Rostedt wrote: > On Tue, 2012-07-17 at 00:27 -0400, Steven Rostedt wrote: > > > Actually, I was mistaken. I forgot that we defined 'cpu_chill()' as > > msleep(1) on RT, which would keep a deadlock from happening. > > Perhaps cpu_chill() isn't a good name

Re: 3.4.4-rt13: btrfs + xfstests 006 = BOOM.. and a bonus rt_mutex deadlock report for absolutely free!

2012-07-16 Thread Mike Galbraith
On Tue, 2012-07-17 at 00:27 -0400, Steven Rostedt wrote: > On Tue, 2012-07-17 at 06:18 +0200, Mike Galbraith wrote: > > > > > There's that too. But the issue I was talking about is with all trylock > > > loops. As holding an rt-mutex now disables migration, if a high priority > > > process preem

Re: 3.4.4-rt13: btrfs + xfstests 006 = BOOM.. and a bonus rt_mutex deadlock report for absolutely free!

2012-07-16 Thread Steven Rostedt
On Tue, 2012-07-17 at 00:27 -0400, Steven Rostedt wrote: > Actually, I was mistaken. I forgot that we defined 'cpu_chill()' as > msleep(1) on RT, which would keep a deadlock from happening. Perhaps cpu_chill() isn't a good name, as it doesn't really explain what is happening. Perhaps one of the f

Re: 3.4.4-rt13: btrfs + xfstests 006 = BOOM.. and a bonus rt_mutex deadlock report for absolutely free!

2012-07-16 Thread Steven Rostedt
On Tue, 2012-07-17 at 06:18 +0200, Mike Galbraith wrote: > > > There's that too. But the issue I was talking about is with all trylock > > loops. As holding an rt-mutex now disables migration, if a high priority > > process preempts a task that holds the lock, and then the high prio task > > star

Re: 3.4.4-rt13: btrfs + xfstests 006 = BOOM.. and a bonus rt_mutex deadlock report for absolutely free!

2012-07-16 Thread Mike Galbraith
On Mon, 2012-07-16 at 13:03 -0400, Steven Rostedt wrote: > On Mon, 2012-07-16 at 18:36 +0200, Mike Galbraith wrote: > > > > > > Ouch, you just turned the rt_read_lock() into a spin lock. If a higher > > > > priority process preempted a lower priority process that holds the same > > > > lock, it

Re: 3.4.4-rt13: btrfs + xfstests 006 = BOOM.. and a bonus rt_mutex deadlock report for absolutely free!

2012-07-16 Thread Mike Galbraith
On Mon, 2012-07-16 at 11:43 -0400, Chris Mason wrote: > On Mon, Jul 16, 2012 at 04:55:44AM -0600, Mike Galbraith wrote: > > Seems btrfs isn't entirely convinced either. > > > > [ 2292.336229] use_block_rsv: 1810 callbacks suppressed > > [ 2292.336231] [ cut here ] > > [ 2

Re: 3.4.4-rt13: btrfs + xfstests 006 = BOOM.. and a bonus rt_mutex deadlock report for absolutely free!

2012-07-16 Thread Steven Rostedt
On Mon, 2012-07-16 at 18:36 +0200, Mike Galbraith wrote: > > > > Ouch, you just turned the rt_read_lock() into a spin lock. If a higher > > > priority process preempted a lower priority process that holds the same > > > lock, it will deadlock. > > > > Hm, how, it's doing cpu_chill()? > > 'cours

Re: 3.4.4-rt13: btrfs + xfstests 006 = BOOM.. and a bonus rt_mutex deadlock report for absolutely free!

2012-07-16 Thread Mike Galbraith
On Mon, 2012-07-16 at 18:26 +0200, Mike Galbraith wrote: > On Mon, 2012-07-16 at 12:02 -0400, Steven Rostedt wrote: > > On Mon, 2012-07-16 at 04:02 +0200, Mike Galbraith wrote: > > > > > > Great, thanks! I got stuck in bug land on Friday. You mentioned > > > > performance problems earlier on S

Re: 3.4.4-rt13: btrfs + xfstests 006 = BOOM.. and a bonus rt_mutex deadlock report for absolutely free!

2012-07-16 Thread Chris Mason
On Mon, Jul 16, 2012 at 10:26:08AM -0600, Mike Galbraith wrote: > On Mon, 2012-07-16 at 12:02 -0400, Steven Rostedt wrote: > > On Mon, 2012-07-16 at 04:02 +0200, Mike Galbraith wrote: > > > > > > Great, thanks! I got stuck in bug land on Friday. You mentioned > > > > performance problems earlie

Re: 3.4.4-rt13: btrfs + xfstests 006 = BOOM.. and a bonus rt_mutex deadlock report for absolutely free!

2012-07-16 Thread Mike Galbraith
On Mon, 2012-07-16 at 12:02 -0400, Steven Rostedt wrote: > On Mon, 2012-07-16 at 04:02 +0200, Mike Galbraith wrote: > > > > Great, thanks! I got stuck in bug land on Friday. You mentioned > > > performance problems earlier on Saturday, did this improve performance? > > > > Yeah, the read_trylo

Re: 3.4.4-rt13: btrfs + xfstests 006 = BOOM.. and a bonus rt_mutex deadlock report for absolutely free!

2012-07-16 Thread Steven Rostedt
On Mon, 2012-07-16 at 04:02 +0200, Mike Galbraith wrote: > > Great, thanks! I got stuck in bug land on Friday. You mentioned > > performance problems earlier on Saturday, did this improve performance? > > Yeah, the read_trylock() seems to improve throughput. That's not > heavily tested, but it

Re: 3.4.4-rt13: btrfs + xfstests 006 = BOOM.. and a bonus rt_mutex deadlock report for absolutely free!

2012-07-16 Thread Chris Mason
On Mon, Jul 16, 2012 at 04:55:44AM -0600, Mike Galbraith wrote: > On Sat, 2012-07-14 at 12:14 +0200, Mike Galbraith wrote: > > On Fri, 2012-07-13 at 08:50 -0400, Chris Mason wrote: > > > On Wed, Jul 11, 2012 at 11:47:40PM -0600, Mike Galbraith wrote: > > > > Greetings, > > > > > > [ deadlocks wi

Re: 3.4.4-rt13: btrfs + xfstests 006 = BOOM.. and a bonus rt_mutex deadlock report for absolutely free!

2012-07-16 Thread Mike Galbraith
On Sat, 2012-07-14 at 12:14 +0200, Mike Galbraith wrote: > On Fri, 2012-07-13 at 08:50 -0400, Chris Mason wrote: > > On Wed, Jul 11, 2012 at 11:47:40PM -0600, Mike Galbraith wrote: > > > Greetings, > > > > [ deadlocks with btrfs and the recent RT kernels ] > > > > I talked with Thomas about thi

Re: 3.4.4-rt13: btrfs + xfstests 006 = BOOM.. and a bonus rt_mutex deadlock report for absolutely free!

2012-07-15 Thread Mike Galbraith
On Sun, 2012-07-15 at 13:56 -0400, Chris Mason wrote: > On Sat, Jul 14, 2012 at 04:14:43AM -0600, Mike Galbraith wrote: > > On Fri, 2012-07-13 at 08:50 -0400, Chris Mason wrote: > > > On Wed, Jul 11, 2012 at 11:47:40PM -0600, Mike Galbraith wrote: > > > > Greetings, > > > > > > [ deadlocks with

Re: 3.4.4-rt13: btrfs + xfstests 006 = BOOM.. and a bonus rt_mutex deadlock report for absolutely free!

2012-07-15 Thread Chris Mason
On Sat, Jul 14, 2012 at 04:14:43AM -0600, Mike Galbraith wrote: > On Fri, 2012-07-13 at 08:50 -0400, Chris Mason wrote: > > On Wed, Jul 11, 2012 at 11:47:40PM -0600, Mike Galbraith wrote: > > > Greetings, > > > > [ deadlocks with btrfs and the recent RT kernels ] > > > > I talked with Thomas abo

Re: 3.4.4-rt13: btrfs + xfstests 006 = BOOM.. and a bonus rt_mutex deadlock report for absolutely free!

2012-07-14 Thread Mike Galbraith
On Fri, 2012-07-13 at 08:50 -0400, Chris Mason wrote: > There is also a chunk of code in btrfs_clear_path_blocking that makes > sure to strictly honor top down locking order during the conversion. It > only does this when lockdep is enabled because in non-RT kernels we > don't need to worry about

Re: 3.4.4-rt13: btrfs + xfstests 006 = BOOM.. and a bonus rt_mutex deadlock report for absolutely free!

2012-07-14 Thread Mike Galbraith
On Fri, 2012-07-13 at 08:50 -0400, Chris Mason wrote: > On Wed, Jul 11, 2012 at 11:47:40PM -0600, Mike Galbraith wrote: > > Greetings, > > [ deadlocks with btrfs and the recent RT kernels ] > > I talked with Thomas about this and I think the problem is the > single-reader nature of the RW rwlock

Re: 3.4.4-rt13: btrfs + xfstests 006 = BOOM.. and a bonus rt_mutex deadlock report for absolutely free!

2012-07-13 Thread Thomas Gleixner
Chris, On Fri, 13 Jul 2012, Chris Mason wrote: > On Wed, Jul 11, 2012 at 11:47:40PM -0600, Mike Galbraith wrote: > > Greetings, > > [ deadlocks with btrfs and the recent RT kernels ] > > I talked with Thomas about this and I think the problem is the > single-reader nature of the RW rwlocks. The

Re: 3.4.4-rt13: btrfs + xfstests 006 = BOOM.. and a bonus rt_mutex deadlock report for absolutely free!

2012-07-13 Thread Chris Mason
On Wed, Jul 11, 2012 at 11:47:40PM -0600, Mike Galbraith wrote: > Greetings, [ deadlocks with btrfs and the recent RT kernels ] I talked with Thomas about this and I think the problem is the single-reader nature of the RW rwlocks. The lockdep report below mentions that btrfs is calling: > [ 69

Re: 3.4.4-rt13: btrfs + xfstests 006 = BOOM.. and a bonus rt_mutex deadlock report for absolutely free!

2012-07-13 Thread Mike Galbraith
On Fri, 2012-07-13 at 06:47 -0400, Chris Mason wrote: > On Fri, Jul 13, 2012 at 04:26:26AM -0600, Thomas Gleixner wrote: > > On Fri, 13 Jul 2012, Mike Galbraith wrote: > > > On Fri, 2012-07-13 at 11:52 +0200, Thomas Gleixner wrote: > > > > On Fri, 13 Jul 2012, Mike Galbraith wrote: > > > > > On T

Re: 3.4.4-rt13: btrfs + xfstests 006 = BOOM.. and a bonus rt_mutex deadlock report for absolutely free!

2012-07-13 Thread Chris Mason
On Fri, Jul 13, 2012 at 04:26:26AM -0600, Thomas Gleixner wrote: > On Fri, 13 Jul 2012, Mike Galbraith wrote: > > On Fri, 2012-07-13 at 11:52 +0200, Thomas Gleixner wrote: > > > On Fri, 13 Jul 2012, Mike Galbraith wrote: > > > > On Thu, 2012-07-12 at 15:31 +0200, Thomas Gleixner wrote: > > > > >

Re: 3.4.4-rt13: btrfs + xfstests 006 = BOOM.. and a bonus rt_mutex deadlock report for absolutely free!

2012-07-13 Thread Thomas Gleixner
On Fri, 13 Jul 2012, Mike Galbraith wrote: > On Fri, 2012-07-13 at 11:52 +0200, Thomas Gleixner wrote: > > On Fri, 13 Jul 2012, Mike Galbraith wrote: > > > On Thu, 2012-07-12 at 15:31 +0200, Thomas Gleixner wrote: > > > > Bingo, that makes it more likely that this is caused by copying w/o > > > >

Re: 3.4.4-rt13: btrfs + xfstests 006 = BOOM.. and a bonus rt_mutex deadlock report for absolutely free!

2012-07-13 Thread Mike Galbraith
On Fri, 2012-07-13 at 11:52 +0200, Thomas Gleixner wrote: > On Fri, 13 Jul 2012, Mike Galbraith wrote: > > On Thu, 2012-07-12 at 15:31 +0200, Thomas Gleixner wrote: > > > Bingo, that makes it more likely that this is caused by copying w/o > > > initializing the lock and then freeing the original

Re: 3.4.4-rt13: btrfs + xfstests 006 = BOOM.. and a bonus rt_mutex deadlock report for absolutely free!

2012-07-13 Thread Thomas Gleixner
On Thu, 12 Jul 2012, Chris Mason wrote: > On Thu, Jul 12, 2012 at 05:07:58AM -0600, Thomas Gleixner wrote: > > On Thu, 12 Jul 2012, Mike Galbraith wrote: > > > crash> struct rt_mutex 0x8801770601c8 > > > struct rt_mutex { > > > wait_lock = { > > > raw_lock = { > > > slock = 7966 > >

Re: 3.4.4-rt13: btrfs + xfstests 006 = BOOM.. and a bonus rt_mutex deadlock report for absolutely free!

2012-07-13 Thread Thomas Gleixner
On Fri, 13 Jul 2012, Mike Galbraith wrote: > On Thu, 2012-07-12 at 15:31 +0200, Thomas Gleixner wrote: > > Bingo, that makes it more likely that this is caused by copying w/o > > initializing the lock and then freeing the original structure. > > > > A quick check for memcpy finds that __btrfs_clo

Re: 3.4.4-rt13: btrfs + xfstests 006 = BOOM.. and a bonus rt_mutex deadlock report for absolutely free!

2012-07-12 Thread Mike Galbraith
On Thu, 2012-07-12 at 15:31 +0200, Thomas Gleixner wrote: > On Thu, 12 Jul 2012, Mike Galbraith wrote: > > On Thu, 2012-07-12 at 13:43 +0200, Thomas Gleixner wrote: > > > rawlock points to ...968 and the node_list to ...970. > > > > > > struct rt_mutex { > > > raw_spinlock_t wai

Re: 3.4.4-rt13: btrfs + xfstests 006 = BOOM.. and a bonus rt_mutex deadlock report for absolutely free!

2012-07-12 Thread Chris Mason
On Thu, Jul 12, 2012 at 05:07:58AM -0600, Thomas Gleixner wrote: > On Thu, 12 Jul 2012, Mike Galbraith wrote: > > crash> struct rt_mutex 0x8801770601c8 > > struct rt_mutex { > > wait_lock = { > > raw_lock = { > > slock = 7966 > > } > > }, > > wait_list = { > > node_list

Re: 3.4.4-rt13: btrfs + xfstests 006 = BOOM.. and a bonus rt_mutex deadlock report for absolutely free!

2012-07-12 Thread Mike Galbraith
On Thu, 2012-07-12 at 15:48 +0200, Mike Galbraith wrote: > On Thu, 2012-07-12 at 15:43 +0200, Thomas Gleixner wrote: > > On Thu, 12 Jul 2012, Mike Galbraith wrote: > > > On Thu, 2012-07-12 at 15:31 +0200, Thomas Gleixner wrote: > > > > A quick check for memcpy finds that __btrfs_close_devices()

Re: 3.4.4-rt13: btrfs + xfstests 006 = BOOM.. and a bonus rt_mutex deadlock report for absolutely free!

2012-07-12 Thread Mike Galbraith
On Thu, 2012-07-12 at 15:43 +0200, Thomas Gleixner wrote: > On Thu, 12 Jul 2012, Mike Galbraith wrote: > > On Thu, 2012-07-12 at 15:31 +0200, Thomas Gleixner wrote: > > > A quick check for memcpy finds that __btrfs_close_devices() does a > > > memcpy of btrfs_device structs w/o initializing the l

Re: 3.4.4-rt13: btrfs + xfstests 006 = BOOM.. and a bonus rt_mutex deadlock report for absolutely free!

2012-07-12 Thread Thomas Gleixner
On Thu, 12 Jul 2012, Mike Galbraith wrote: > On Thu, 2012-07-12 at 15:31 +0200, Thomas Gleixner wrote: > > A quick check for memcpy finds that __btrfs_close_devices() does a > > memcpy of btrfs_device structs w/o initializing the lock in the new > > copy, but I have no idea whether that's the plac

Re: 3.4.4-rt13: btrfs + xfstests 006 = BOOM.. and a bonus rt_mutex deadlock report for absolutely free!

2012-07-12 Thread Mike Galbraith
On Thu, 2012-07-12 at 15:31 +0200, Thomas Gleixner wrote: > On Thu, 12 Jul 2012, Mike Galbraith wrote: > > On Thu, 2012-07-12 at 13:43 +0200, Thomas Gleixner wrote: > > > rawlock points to ...968 and the node_list to ...970. > > > > > > struct rt_mutex { > > > raw_spinlock_t wai

Re: 3.4.4-rt13: btrfs + xfstests 006 = BOOM.. and a bonus rt_mutex deadlock report for absolutely free!

2012-07-12 Thread Thomas Gleixner
On Thu, 12 Jul 2012, Mike Galbraith wrote: > On Thu, 2012-07-12 at 13:43 +0200, Thomas Gleixner wrote: > > rawlock points to ...968 and the node_list to ...970. > > > > struct rt_mutex { > > raw_spinlock_t wait_lock; > > struct plist_head wait_list; > > > > The raw

Re: 3.4.4-rt13: btrfs + xfstests 006 = BOOM.. and a bonus rt_mutex deadlock report for absolutely free!

2012-07-12 Thread Mike Galbraith
On Thu, 2012-07-12 at 13:43 +0200, Thomas Gleixner wrote: > On Thu, 12 Jul 2012, Mike Galbraith wrote: > > On Thu, 2012-07-12 at 10:44 +0200, Mike Galbraith wrote: > > > On Thu, 2012-07-12 at 07:47 +0200, Mike Galbraith wrote: > > > > Greetings, > > > > > > > > I'm chasing btrfs critters in an

Re: 3.4.4-rt13: btrfs + xfstests 006 = BOOM.. and a bonus rt_mutex deadlock report for absolutely free!

2012-07-12 Thread Thomas Gleixner
On Thu, 12 Jul 2012, Mike Galbraith wrote: > On Thu, 2012-07-12 at 10:44 +0200, Mike Galbraith wrote: > > On Thu, 2012-07-12 at 07:47 +0200, Mike Galbraith wrote: > > > Greetings, > > > > > > I'm chasing btrfs critters in an enterprise 3.0-rt kernel, and just > > > checked to see if they're aliv

Re: 3.4.4-rt13: btrfs + xfstests 006 = BOOM.. and a bonus rt_mutex deadlock report for absolutely free!

2012-07-12 Thread Thomas Gleixner
On Thu, 12 Jul 2012, Mike Galbraith wrote: > crash> struct rt_mutex 0x8801770601c8 > struct rt_mutex { > wait_lock = { > raw_lock = { > slock = 7966 > } > }, > wait_list = { > node_list = { > next = 0x880175eedbe0, > prev = 0x880175eedbe0 > },

Re: 3.4.4-rt13: btrfs + xfstests 006 = BOOM.. and a bonus rt_mutex deadlock report for absolutely free!

2012-07-12 Thread Mike Galbraith
On Thu, 2012-07-12 at 10:44 +0200, Mike Galbraith wrote: > On Thu, 2012-07-12 at 07:47 +0200, Mike Galbraith wrote: > > Greetings, > > > > I'm chasing btrfs critters in an enterprise 3.0-rt kernel, and just > > checked to see if they're alive in virgin latest/greatest rt kernel. > > > > Both

Re: 3.4.4-rt13: btrfs + xfstests 006 = BOOM.. and a bonus rt_mutex deadlock report for absolutely free!

2012-07-12 Thread Mike Galbraith
On Thu, 2012-07-12 at 07:47 +0200, Mike Galbraith wrote: > Greetings, > > I'm chasing btrfs critters in an enterprise 3.0-rt kernel, and just > checked to see if they're alive in virgin latest/greatest rt kernel. > > Both are indeed alive and well, ie I didn't break it, nor did the > zillion p

3.4.4-rt13: btrfs + xfstests 006 = BOOM.. and a bonus rt_mutex deadlock report for absolutely free!

2012-07-11 Thread Mike Galbraith
Greetings, I'm chasing btrfs critters in an enterprise 3.0-rt kernel, and just checked to see if they're alive in virgin latest/greatest rt kernel. Both are indeed alive and well, ie I didn't break it, nor did the zillion patches in enterprise base kernel, so others may have an opportunity to m