Re: docker crashes rcuos in __blkg_release_rcu

2014-06-19 Thread Tejun Heo
Sorry about the late reply. On Wed, Jun 11, 2014 at 12:32:29PM -0400, Vivek Goyal wrote: > Tejun, any thoughts on how to solve this issue. Delaying blkg release > in rcu context and then expecting queue to be still present is causing > this problem. Heh, this is hilarious. If you look at the

Re: docker crashes rcuos in __blkg_release_rcu

2014-06-19 Thread Tejun Heo
Sorry about the late reply. On Wed, Jun 11, 2014 at 12:32:29PM -0400, Vivek Goyal wrote: Tejun, any thoughts on how to solve this issue. Delaying blkg release in rcu context and then expecting queue to be still present is causing this problem. Heh, this is hilarious. If you look at the

Re: docker crashes rcuos in __blkg_release_rcu

2014-06-11 Thread Vivek Goyal
On Tue, Jun 10, 2014 at 02:39:06PM -0400, Joe Lawrence wrote: > > Hi Vivek, > > Thanks for taking a look. For extra debugging, I wrote a quick set of > kprobes that: > > 1 - On blkg_alloc entry, save the request_queue's kobj address in a > list > 2 - On kobject_put entry, dump the

Re: docker crashes rcuos in __blkg_release_rcu

2014-06-11 Thread Vivek Goyal
On Tue, Jun 10, 2014 at 02:39:06PM -0400, Joe Lawrence wrote: Hi Vivek, Thanks for taking a look. For extra debugging, I wrote a quick set of kprobes that: 1 - On blkg_alloc entry, save the request_queue's kobj address in a list 2 - On kobject_put entry, dump the stack if

Re: docker crashes rcuos in __blkg_release_rcu

2014-06-10 Thread Christoph Lameter
On Sun, 8 Jun 2014, Joe Lawrence wrote: > > .tickets is offset 0 from arch_spinlock_t, so RDI should be the > arch_spinlock_t lock: > RDI: 6b6b6b6b6b6b6b6b Slub has overwritten the object when it was freed with 0x6b. So this is an access after free. It works without debug because the object may

Re: docker crashes rcuos in __blkg_release_rcu

2014-06-10 Thread Vivek Goyal
On Tue, Jun 10, 2014 at 02:39:06PM -0400, Joe Lawrence wrote: [..] > > I am wondering if we need to take a reference on the queue > > (blk_get_queue()) in blkg_alloc(), to make sure request queue is > > still around when blkg is being freed. > > I experimented with this and the crash does go

Re: docker crashes rcuos in __blkg_release_rcu

2014-06-10 Thread Joe Lawrence
On Mon, 9 Jun 2014 14:27:29 -0400 Vivek Goyal wrote: > ... snip ... > So the question is why request queue is being freed early. Are there any > reference counting issues. Hi Vivek, Thanks for taking a look. For extra debugging, I wrote a quick set of kprobes that: 1 - On blkg_alloc

Re: docker crashes rcuos in __blkg_release_rcu

2014-06-10 Thread Joe Lawrence
On Mon, 9 Jun 2014 14:27:29 -0400 Vivek Goyal vgo...@redhat.com wrote: ... snip ... So the question is why request queue is being freed early. Are there any reference counting issues. Hi Vivek, Thanks for taking a look. For extra debugging, I wrote a quick set of kprobes that: 1 - On

Re: docker crashes rcuos in __blkg_release_rcu

2014-06-10 Thread Vivek Goyal
On Tue, Jun 10, 2014 at 02:39:06PM -0400, Joe Lawrence wrote: [..] I am wondering if we need to take a reference on the queue (blk_get_queue()) in blkg_alloc(), to make sure request queue is still around when blkg is being freed. I experimented with this and the crash does go away (and

Re: docker crashes rcuos in __blkg_release_rcu

2014-06-10 Thread Christoph Lameter
On Sun, 8 Jun 2014, Joe Lawrence wrote: .tickets is offset 0 from arch_spinlock_t, so RDI should be the arch_spinlock_t lock: RDI: 6b6b6b6b6b6b6b6b Slub has overwritten the object when it was freed with 0x6b. So this is an access after free. It works without debug because the object may

Re: docker crashes rcuos in __blkg_release_rcu

2014-06-09 Thread Vivek Goyal
On Mon, Jun 09, 2014 at 01:47:08PM -0400, Vivek Goyal wrote: > On Sun, Jun 08, 2014 at 06:22:00PM -0400, Joe Lawrence wrote: > > [..] > > Summary thus far: > > > > R12: 88103c17a130 = struct rcu_head *rcu_head > > R13: 88103c17a080 = struct blkcg_gq *blkg > > 88103fc7df90 =

Re: docker crashes rcuos in __blkg_release_rcu

2014-06-09 Thread Vivek Goyal
On Sun, Jun 08, 2014 at 06:22:00PM -0400, Joe Lawrence wrote: [..] > Summary thus far: > > R12: 88103c17a130 = struct rcu_head *rcu_head > R13: 88103c17a080 = struct blkcg_gq *blkg > 88103fc7df90 = struct request_queue *blkg->q (contains 0x6b >

Re: docker crashes rcuos in __blkg_release_rcu

2014-06-09 Thread Vivek Goyal
On Sun, Jun 08, 2014 at 06:22:00PM -0400, Joe Lawrence wrote: [..] Summary thus far: R12: 88103c17a130 = struct rcu_head *rcu_head R13: 88103c17a080 = struct blkcg_gq *blkg 88103fc7df90 = struct request_queue *blkg-q (contains 0x6b

Re: docker crashes rcuos in __blkg_release_rcu

2014-06-09 Thread Vivek Goyal
On Mon, Jun 09, 2014 at 01:47:08PM -0400, Vivek Goyal wrote: On Sun, Jun 08, 2014 at 06:22:00PM -0400, Joe Lawrence wrote: [..] Summary thus far: R12: 88103c17a130 = struct rcu_head *rcu_head R13: 88103c17a080 = struct blkcg_gq *blkg 88103fc7df90 = struct

docker crashes rcuos in __blkg_release_rcu

2014-06-08 Thread Joe Lawrence
Hi Tejun, Vivek, I came across this crash when attempting to run the 'hello world' example from the Getting Started section on the docker.io homepage. Repro kernels: (upstream linus) 3.15.0 (RHEL7 RC-2) 3.10.0-121.el7.x86_64 To reproduce, boot with slub_debug=FZPU and run the example. %

docker crashes rcuos in __blkg_release_rcu

2014-06-08 Thread Joe Lawrence
Hi Tejun, Vivek, I came across this crash when attempting to run the 'hello world' example from the Getting Started section on the docker.io homepage. Repro kernels: (upstream linus) 3.15.0 (RHEL7 RC-2) 3.10.0-121.el7.x86_64 To reproduce, boot with slub_debug=FZPU and run the example. %