Re: [PATCH v2 1/1] blk-mq: fix hang caused by freeze/unfreeze sequence
2016-08-10 13:36 GMT+02:00 Roman Penyaev :
> On Wed, Aug 10, 2016 at 10:42 AM, Roman Penyaev
> wrote:
>> Hi,
>>
>> On Wed, Aug 10, 2016 at 5:55 AM, Tejun Heo wrote:
>>> Hello,
>>>
>>> On Mon, Aug 08, 2016 at 01:39:08PM +0200, Roman Pen wrote:
Long time ago there was a similar fix proposed by Akinobu Mita[1],
but it seems that time everyone decided to fix this subtle race in
percpu-refcount and Tejun Heo[2] did an attempt (as I can see that
patchset was not applied).
>>>
>>> So, I probably forgot about it while waiting for confirmation of fix.
>>> Can you please verify that the patchset fixes the issue? I can apply
>>> the patchset right away.
>>
>> I have not checked your patchset but according to my understanding
>> it should not fix *this* issue.
>
> So, your patchset does not help (but for sure it helps for keeping
> internal percpu-refcount members consistent, but that is not related
> to this issue). That's the backtrace which I observe:
>
> Call Trace:
> [] ? vprintk_default+0x1f/0x30
> [] schedule+0x35/0x80
> [] blk_mq_freeze_queue_wait+0x124/0x1a0
> [] ? wake_atomic_t_function+0x60/0x60
> [] blk_mq_freeze_queue+0x1a/0x20
> [] blk_freeze_queue+0xe/0x10
> [] blk_cleanup_queue+0xe2/0x280
>
> To ease reproduction I do the following:
>
> ---
> static int thread_fn(void *data)
> {
> struct blk_mq_tag_set *tags = data;
> struct request_queue *q;
>
> while (!kthread_should_stop()) {
> q = blk_mq_init_queue(tags);
> BUG_ON(q == NULL);
> /*
> * That is done by blk_register_queue(), but here
> * we are reproducing blk-mq bug and do not require
> * gendisk and friends. Just silently switch to percpu.
> */
> percpu_ref_switch_to_percpu(&q->q_usage_counter);
>
> msleep(prandom_u32_max(10));
> blk_cleanup_queue(q);
> }
>
> return 0;
> }
> ---
>
> o Start 2 threads (exactly 2, we need 2 queues for 1 shared tags)
> o Pass same shared tags pointer for each thread
> o Wait
> o PROFIT
>
> To make immediate reproduction this hunk can be applied:
>
> @@ -129,6 +142,7 @@ void blk_mq_unfreeze_queue(struct request_queue *q)
> freeze_depth = atomic_dec_return(&q->mq_freeze_depth);
> WARN_ON_ONCE(freeze_depth < 0);
> if (!freeze_depth) {
> + msleep(1000);
> percpu_ref_reinit(&q->q_usage_counter);
> wake_up_all(&q->mq_freeze_wq);
> }
>
> --
> Roman
Hi Jens,
I didn't see this patch in you tree, what's the blocker?
Thanks,
Jinpu
Re: [PATCH v2 1/1] blk-mq: fix hang caused by freeze/unfreeze sequence
Hi, On Wed, Aug 10, 2016 at 5:55 AM, Tejun Heo wrote: > Hello, > > On Mon, Aug 08, 2016 at 01:39:08PM +0200, Roman Pen wrote: >> Long time ago there was a similar fix proposed by Akinobu Mita[1], >> but it seems that time everyone decided to fix this subtle race in >> percpu-refcount and Tejun Heo[2] did an attempt (as I can see that >> patchset was not applied). > > So, I probably forgot about it while waiting for confirmation of fix. > Can you please verify that the patchset fixes the issue? I can apply > the patchset right away. I have not checked your patchset but according to my understanding it should not fix *this* issue. What happens here is a wrong order of invocation of percpu_ref_reinit() and percpu_ref_kill(). So what was observed is the following: CPU#0 CPU#1 - percpu_ref_kill() percpu_ref_kill() << atomic reference does percpu_ref_reinit() << not guarantee the order blk_mq_freeze_queue_wait() !! HANG HERE percpu_ref_reinit() blk_mq_freeze_queue_wait() on CPU#1 expects percpu-refcount to be switched to ATOMIC mode (killed), but that does not happen, because CPU#2 was faster and has been switched percpu-refcount to PERCPU mode. This race happens inside blk-mq, because invocation of kill/reinit is controlled by the reference counter, which does not guarantee the order of the following functions calls (kill/reinit). So the fix is the same as originally proposed by Akinobu Mita, but the issue is different. But of course I can run tests on top of your series, just to verify that everything goes smoothly and internally percpu-refcount members are consistent. -- Roman
Re: [PATCH v2 1/1] blk-mq: fix hang caused by freeze/unfreeze sequence
On Wed, Aug 10, 2016 at 10:42 AM, Roman Penyaev
wrote:
> Hi,
>
> On Wed, Aug 10, 2016 at 5:55 AM, Tejun Heo wrote:
>> Hello,
>>
>> On Mon, Aug 08, 2016 at 01:39:08PM +0200, Roman Pen wrote:
>>> Long time ago there was a similar fix proposed by Akinobu Mita[1],
>>> but it seems that time everyone decided to fix this subtle race in
>>> percpu-refcount and Tejun Heo[2] did an attempt (as I can see that
>>> patchset was not applied).
>>
>> So, I probably forgot about it while waiting for confirmation of fix.
>> Can you please verify that the patchset fixes the issue? I can apply
>> the patchset right away.
>
> I have not checked your patchset but according to my understanding
> it should not fix *this* issue.
So, your patchset does not help (but for sure it helps for keeping
internal percpu-refcount members consistent, but that is not related
to this issue). That's the backtrace which I observe:
Call Trace:
[] ? vprintk_default+0x1f/0x30
[] schedule+0x35/0x80
[] blk_mq_freeze_queue_wait+0x124/0x1a0
[] ? wake_atomic_t_function+0x60/0x60
[] blk_mq_freeze_queue+0x1a/0x20
[] blk_freeze_queue+0xe/0x10
[] blk_cleanup_queue+0xe2/0x280
To ease reproduction I do the following:
---
static int thread_fn(void *data)
{
struct blk_mq_tag_set *tags = data;
struct request_queue *q;
while (!kthread_should_stop()) {
q = blk_mq_init_queue(tags);
BUG_ON(q == NULL);
/*
* That is done by blk_register_queue(), but here
* we are reproducing blk-mq bug and do not require
* gendisk and friends. Just silently switch to percpu.
*/
percpu_ref_switch_to_percpu(&q->q_usage_counter);
msleep(prandom_u32_max(10));
blk_cleanup_queue(q);
}
return 0;
}
---
o Start 2 threads (exactly 2, we need 2 queues for 1 shared tags)
o Pass same shared tags pointer for each thread
o Wait
o PROFIT
To make immediate reproduction this hunk can be applied:
@@ -129,6 +142,7 @@ void blk_mq_unfreeze_queue(struct request_queue *q)
freeze_depth = atomic_dec_return(&q->mq_freeze_depth);
WARN_ON_ONCE(freeze_depth < 0);
if (!freeze_depth) {
+ msleep(1000);
percpu_ref_reinit(&q->q_usage_counter);
wake_up_all(&q->mq_freeze_wq);
}
--
Roman
Re: [PATCH v2 1/1] blk-mq: fix hang caused by freeze/unfreeze sequence
On Wed, Aug 10, 2016 at 10:42:09AM +0200, Roman Penyaev wrote: > Hi, > > On Wed, Aug 10, 2016 at 5:55 AM, Tejun Heo wrote: > > Hello, > > > > On Mon, Aug 08, 2016 at 01:39:08PM +0200, Roman Pen wrote: > >> Long time ago there was a similar fix proposed by Akinobu Mita[1], > >> but it seems that time everyone decided to fix this subtle race in > >> percpu-refcount and Tejun Heo[2] did an attempt (as I can see that > >> patchset was not applied). > > > > So, I probably forgot about it while waiting for confirmation of fix. > > Can you please verify that the patchset fixes the issue? I can apply > > the patchset right away. > > I have not checked your patchset but according to my understanding > it should not fix *this* issue. What happens here is a wrong order > of invocation of percpu_ref_reinit() and percpu_ref_kill(). So what > was observed is the following: Ah, understood. Acked-by: Tejun Heo I'll commit the percpu_refcnt patches too. While they don't fix the problem on their own, the changes are generally useful for all mode switching use cases. Thanks. -- tejun
Re: [PATCH v2 1/1] blk-mq: fix hang caused by freeze/unfreeze sequence
Hello, On Mon, Aug 08, 2016 at 01:39:08PM +0200, Roman Pen wrote: > Long time ago there was a similar fix proposed by Akinobu Mita[1], > but it seems that time everyone decided to fix this subtle race in > percpu-refcount and Tejun Heo[2] did an attempt (as I can see that > patchset was not applied). So, I probably forgot about it while waiting for confirmation of fix. Can you please verify that the patchset fixes the issue? I can apply the patchset right away. Thanks. -- tejun
