Re: INFO: task hung in wb_shutdown (2)
Tejun, Jan, Jens,
Can you review this patch? syzbot has hit this bug for nearly 4000 times but
is still unable to find a reproducer. Therefore, the only way to test would be
to apply this patch upstream and test whether the problem is solved.
On 2018/04/24 21:19, Tetsuo Handa wrote:
>>From 39ed6be8a2c12dfe54feaa5abbc2ec46103022bf Mon Sep 17 00:00:00 2001
> From: Tetsuo Handa
> Date: Tue, 24 Apr 2018 11:59:08 +0900
> Subject: [PATCH] bdi: wake up concurrent wb_shutdown() callers.
>
> syzbot is reporting hung tasks at wait_on_bit(WB_shutting_down) in
> wb_shutdown() [1]. This might be because commit 5318ce7d46866e1d ("bdi:
> Shutdown writeback on all cgwbs in cgwb_bdi_destroy()") forgot to call
> wake_up_bit(WB_shutting_down) after clear_bit(WB_shutting_down).
>
> [1]
> https://syzkaller.appspot.com/bug?id=b297474817af98d5796bc544e1bb806fc3da0e5e
>
> Signed-off-by: Tetsuo Handa
> Reported-by: syzbot
> Fixes: 5318ce7d46866e1d ("bdi: Shutdown writeback on all cgwbs in
> cgwb_bdi_destroy()")
> Cc: Tejun Heo
> Cc: Jan Kara
> Cc: Jens Axboe
> ---
> mm/backing-dev.c | 2 ++
> 1 file changed, 2 insertions(+)
>
> diff --git a/mm/backing-dev.c b/mm/backing-dev.c
> index 023190c..dadac99 100644
> --- a/mm/backing-dev.c
> +++ b/mm/backing-dev.c
> @@ -384,6 +384,8 @@ static void wb_shutdown(struct bdi_writeback *wb)
>*/
> smp_wmb();
> clear_bit(WB_shutting_down, &wb->state);
> + smp_mb(); /* advised by wake_up_bit() */
> + wake_up_bit(&wb->state, WB_shutting_down);
> }
>
> static void wb_exit(struct bdi_writeback *wb)
>
Re: INFO: task hung in wb_shutdown (2)
On Tue, May 1, 2018 at 3:27 AM Tetsuo Handa < penguin-ker...@i-love.sakura.ne.jp> wrote: > Can you review this patch? syzbot has hit this bug for nearly 4000 times but > is still unable to find a reproducer. Therefore, the only way to test would be > to apply this patch upstream and test whether the problem is solved. Looks ok to me, except: > > smp_wmb(); > > clear_bit(WB_shutting_down, &wb->state); > > + smp_mb(); /* advised by wake_up_bit() */ > > + wake_up_bit(&wb->state, WB_shutting_down); This whole sequence really should just be a pattern with a helper function. And honestly, the pattern probably *should* be clear_bit_unlock(bit, &mem); smp_mb__after_atomic() wake_up_bit(&mem, bit); which looks like it is a bit cleaner wrt memory ordering rules. Linus
Re: INFO: task hung in wb_shutdown (2)
On 5/1/18 4:27 AM, Tetsuo Handa wrote: > Tejun, Jan, Jens, > > Can you review this patch? syzbot has hit this bug for nearly 4000 times but > is still unable to find a reproducer. Therefore, the only way to test would be > to apply this patch upstream and test whether the problem is solved. I'll review it today. -- Jens Axboe
Re: INFO: task hung in wb_shutdown (2)
On 5/1/18 10:06 AM, Linus Torvalds wrote: > On Tue, May 1, 2018 at 3:27 AM Tetsuo Handa < > penguin-ker...@i-love.sakura.ne.jp> wrote: > >> Can you review this patch? syzbot has hit this bug for nearly 4000 times > but >> is still unable to find a reproducer. Therefore, the only way to test > would be >> to apply this patch upstream and test whether the problem is solved. > > Looks ok to me, except: > >>> smp_wmb(); >>> clear_bit(WB_shutting_down, &wb->state); >>> + smp_mb(); /* advised by wake_up_bit() */ >>> + wake_up_bit(&wb->state, WB_shutting_down); > > This whole sequence really should just be a pattern with a helper function. > > And honestly, the pattern probably *should* be > > clear_bit_unlock(bit, &mem); > smp_mb__after_atomic() > wake_up_bit(&mem, bit); > > which looks like it is a bit cleaner wrt memory ordering rules. Agree, that construct looks saner than introducing a "random" smp_mb(). As a pattern helper, should probably be introduced after the fact. -- Jens Axboe
Re: INFO: task hung in wb_shutdown (2)
>From 1b90d7f71d60e743c69cdff3ba41edd1f9f86f93 Mon Sep 17 00:00:00 2001
From: Tetsuo Handa
Date: Wed, 2 May 2018 07:07:55 +0900
Subject: [PATCH v2] bdi: wake up concurrent wb_shutdown() callers.
syzbot is reporting hung tasks at wait_on_bit(WB_shutting_down) in
wb_shutdown() [1]. This seems to be because commit 5318ce7d46866e1d ("bdi:
Shutdown writeback on all cgwbs in cgwb_bdi_destroy()") forgot to call
wake_up_bit(WB_shutting_down) after clear_bit(WB_shutting_down).
Introduce a helper function clear_and_wake_up_bit() and use it, in order
to avoid similar errors in future.
[1]
https://syzkaller.appspot.com/bug?id=b297474817af98d5796bc544e1bb806fc3da0e5e
Signed-off-by: Tetsuo Handa
Reported-by: syzbot
Fixes: 5318ce7d46866e1d ("bdi: Shutdown writeback on all cgwbs in
cgwb_bdi_destroy()")
Cc: Tejun Heo
Cc: Jan Kara
Cc: Jens Axboe
Suggested-by: Linus Torvalds
---
include/linux/wait_bit.h | 17 +
mm/backing-dev.c | 2 +-
2 files changed, 18 insertions(+), 1 deletion(-)
diff --git a/include/linux/wait_bit.h b/include/linux/wait_bit.h
index 9318b21..2b0072f 100644
--- a/include/linux/wait_bit.h
+++ b/include/linux/wait_bit.h
@@ -305,4 +305,21 @@ struct wait_bit_queue_entry {
__ret; \
})
+/**
+ * clear_and_wake_up_bit - clear a bit and wake up anyone waiting on that bit
+ *
+ * @bit: the bit of the word being waited on
+ * @word: the word being waited on, a kernel virtual address
+ *
+ * You can use this helper if bitflags are manipulated atomically rather than
+ * non-atomically under a lock.
+ */
+static inline void clear_and_wake_up_bit(int bit, void *word)
+{
+ clear_bit_unlock(bit, word);
+ /* See wake_up_bit() for which memory barrier you need to use. */
+ smp_mb__after_atomic();
+ wake_up_bit(word, bit);
+}
+
#endif /* _LINUX_WAIT_BIT_H */
diff --git a/mm/backing-dev.c b/mm/backing-dev.c
index 023190c..fa5e6d7 100644
--- a/mm/backing-dev.c
+++ b/mm/backing-dev.c
@@ -383,7 +383,7 @@ static void wb_shutdown(struct bdi_writeback *wb)
* the barrier provided by test_and_clear_bit() above.
*/
smp_wmb();
- clear_bit(WB_shutting_down, &wb->state);
+ clear_and_wake_up_bit(WB_shutting_down, &wb->state);
}
static void wb_exit(struct bdi_writeback *wb)
--
1.8.3.1
Re: INFO: task hung in wb_shutdown (2)
On Wed 02-05-18 07:14:51, Tetsuo Handa wrote:
> >From 1b90d7f71d60e743c69cdff3ba41edd1f9f86f93 Mon Sep 17 00:00:00 2001
> From: Tetsuo Handa
> Date: Wed, 2 May 2018 07:07:55 +0900
> Subject: [PATCH v2] bdi: wake up concurrent wb_shutdown() callers.
>
> syzbot is reporting hung tasks at wait_on_bit(WB_shutting_down) in
> wb_shutdown() [1]. This seems to be because commit 5318ce7d46866e1d ("bdi:
> Shutdown writeback on all cgwbs in cgwb_bdi_destroy()") forgot to call
> wake_up_bit(WB_shutting_down) after clear_bit(WB_shutting_down).
>
> Introduce a helper function clear_and_wake_up_bit() and use it, in order
> to avoid similar errors in future.
>
> [1]
> https://syzkaller.appspot.com/bug?id=b297474817af98d5796bc544e1bb806fc3da0e5e
>
> Signed-off-by: Tetsuo Handa
> Reported-by: syzbot
> Fixes: 5318ce7d46866e1d ("bdi: Shutdown writeback on all cgwbs in
> cgwb_bdi_destroy()")
> Cc: Tejun Heo
> Cc: Jan Kara
> Cc: Jens Axboe
> Suggested-by: Linus Torvalds
Thanks for debugging this and for the fix Tetsuo! The patch looks good to
me. You can add:
Reviewed-by: Jan Kara
Honza
> ---
> include/linux/wait_bit.h | 17 +
> mm/backing-dev.c | 2 +-
> 2 files changed, 18 insertions(+), 1 deletion(-)
>
> diff --git a/include/linux/wait_bit.h b/include/linux/wait_bit.h
> index 9318b21..2b0072f 100644
> --- a/include/linux/wait_bit.h
> +++ b/include/linux/wait_bit.h
> @@ -305,4 +305,21 @@ struct wait_bit_queue_entry {
> __ret; \
> })
>
> +/**
> + * clear_and_wake_up_bit - clear a bit and wake up anyone waiting on that bit
> + *
> + * @bit: the bit of the word being waited on
> + * @word: the word being waited on, a kernel virtual address
> + *
> + * You can use this helper if bitflags are manipulated atomically rather than
> + * non-atomically under a lock.
> + */
> +static inline void clear_and_wake_up_bit(int bit, void *word)
> +{
> + clear_bit_unlock(bit, word);
> + /* See wake_up_bit() for which memory barrier you need to use. */
> + smp_mb__after_atomic();
> + wake_up_bit(word, bit);
> +}
> +
> #endif /* _LINUX_WAIT_BIT_H */
> diff --git a/mm/backing-dev.c b/mm/backing-dev.c
> index 023190c..fa5e6d7 100644
> --- a/mm/backing-dev.c
> +++ b/mm/backing-dev.c
> @@ -383,7 +383,7 @@ static void wb_shutdown(struct bdi_writeback *wb)
>* the barrier provided by test_and_clear_bit() above.
>*/
> smp_wmb();
> - clear_bit(WB_shutting_down, &wb->state);
> + clear_and_wake_up_bit(WB_shutting_down, &wb->state);
> }
>
> static void wb_exit(struct bdi_writeback *wb)
> --
> 1.8.3.1
--
Jan Kara
SUSE Labs, CR
Re: INFO: task hung in wb_shutdown (2)
On 5/1/18 4:14 PM, Tetsuo Handa wrote:
>>From 1b90d7f71d60e743c69cdff3ba41edd1f9f86f93 Mon Sep 17 00:00:00 2001
> From: Tetsuo Handa
> Date: Wed, 2 May 2018 07:07:55 +0900
> Subject: [PATCH v2] bdi: wake up concurrent wb_shutdown() callers.
>
> syzbot is reporting hung tasks at wait_on_bit(WB_shutting_down) in
> wb_shutdown() [1]. This seems to be because commit 5318ce7d46866e1d ("bdi:
> Shutdown writeback on all cgwbs in cgwb_bdi_destroy()") forgot to call
> wake_up_bit(WB_shutting_down) after clear_bit(WB_shutting_down).
>
> Introduce a helper function clear_and_wake_up_bit() and use it, in order
> to avoid similar errors in future.
Queued up, thanks Tetsuo!
--
Jens Axboe
