Re: block: be more careful about status in __bio_chain_endio

[John Dorminy]

I'm also worried about the other two versions, though:

memory-barriers.txt#1724:

1724 (*) The compiler is within its rights to invent stores to a variable,

i.e. the compiler is free to decide __bio_chain_endio looks like this:

static struct bio *__bio_chain_endio(struct bio *bio)
{
  struct bio *parent = bio->bi_private;
  blk_status_t tmp = parent->bi_status;
  parent->bi_status = bio->bi_status;
  if (!bio->bi_status)
parent->bi_status = tmp;
  bio_put(bio);
  return parent;
}

In which case, the read and later store on the two different threads
may overlap in such a way that bio_endio sometimes sees success, even
if one child had an error.

As a result, I believe the setting of parent->bi_status needs to be a
WRITE_ONCE() and the later reading needs to be a READ_ONCE()
[although, since the later reading happens in many different
functions, perhaps some other barrier to make sure all readers get the
correct value is in order.]

Re: block: be more careful about status in __bio_chain_endio

[Mike Snitzer]

On Fri, Feb 22 2019 at  9:02pm -0500,
John Dorminy  wrote:

> I am perhaps not understanding the intricacies here, or not seeing a
> barrier protecting it, so forgive me if I'm off base. I think reading
> parent->bi_status here is unsafe.
> Consider the following sequence of events on two threads.
> 
> Thread 0 Thread 1
> In __bio_chain_endio:In __bio_chain_endio:
> [A] Child 0 reads parent->bi_status,
> no error.
>  Child bio 1 reads parent, no error 
> seen
>  It sets parent->bi_status to an error
>  It calls bio_put.
> Child bio 0 calls bio_put
> [end __bio_chain_endio]  [end __bio_chain_endio]
>  In bio_chain_endio(), 
> bio_endio(parent)
>  is called, calling 
> bio_remaining_done()
>  which decrements __bi_remaining to 1
>  and returns false, so no further 
> endio
>  stuff is done.
> In bio_chain_endio(), bio_endio(parent)
> is called, calling bio_remaining_done(),
> decrementing parent->__bi_remaining to
>  0, and continuing to finish parent.
> Either for block tracing or for parent's
> bi_end_io(), this thread tries to read
> parent->bi_status again.
> 
> The compiler or the CPU may cache the read from [A], and since there
> are no intervening barriers, parent->bi_status is still believed on
> thread 0 to be success. Thus the bio may still be falsely believed to
> have completed successfully, even though child 1 set an error in it.
> 
> Am I missing a subtlety here?

Either neilb's original or even Jens' suggestion would be fine though.

>   if (!parent->bi_status && bio->bi_status)
>   parent->bi_status = bio->bi_status;

Even if your scenario did play out (which I agree it looks possible)
it'd just degenerate to neilb's version:

>   if (bio->bi_status)
>   parent->bi_status = bio->bi_status;

Which also accomplishes fixing what Neil originally detailed in his
patch header.

Re: block: be more careful about status in __bio_chain_endio

[John Dorminy]

I am perhaps not understanding the intricacies here, or not seeing a
barrier protecting it, so forgive me if I'm off base. I think reading
parent->bi_status here is unsafe.
Consider the following sequence of events on two threads.

Thread 0 Thread 1
In __bio_chain_endio:In __bio_chain_endio:
[A] Child 0 reads parent->bi_status,
no error.
 Child bio 1 reads parent, no error seen
 It sets parent->bi_status to an error
 It calls bio_put.
Child bio 0 calls bio_put
[end __bio_chain_endio]  [end __bio_chain_endio]
 In bio_chain_endio(), bio_endio(parent)
 is called, calling bio_remaining_done()
 which decrements __bi_remaining to 1
 and returns false, so no further endio
 stuff is done.
In bio_chain_endio(), bio_endio(parent)
is called, calling bio_remaining_done(),
decrementing parent->__bi_remaining to
 0, and continuing to finish parent.
Either for block tracing or for parent's
bi_end_io(), this thread tries to read
parent->bi_status again.

The compiler or the CPU may cache the read from [A], and since there
are no intervening barriers, parent->bi_status is still believed on
thread 0 to be success. Thus the bio may still be falsely believed to
have completed successfully, even though child 1 set an error in it.

Am I missing a subtlety here?

Re: block: be more careful about status in __bio_chain_endio

[Mike Snitzer]

On Fri, Feb 22 2019 at  5:46pm -0500,
Jens Axboe  wrote:

> On 2/22/19 2:10 PM, Mike Snitzer wrote:
> > On Thu, Feb 15 2018 at  4:09am -0500,
> > NeilBrown  wrote:
> > 
> >>
> >> If two bios are chained under the one parent (with bio_chain())
> >> it is possible that one will succeed and the other will fail.
> >> __bio_chain_endio must ensure that the failure error status
> >> is reported for the whole, rather than the success.
> >>
> >> It currently tries to be careful, but this test is racy.
> >> If both children finish at the same time, they might both see that
> >> parent->bi_status as zero, and so will assign their own status.
> >> If the assignment to parent->bi_status by the successful bio happens
> >> last, the error status will be lost which can lead to silent data
> >> corruption.
> >>
> >> Instead, __bio_chain_endio should only assign a non-zero status
> >> to parent->bi_status.  There is then no need to test the current
> >> value of parent->bi_status - a test that would be racy anyway.
> >>
> >> Note that this bug hasn't been seen in practice.  It was only discovered
> >> by examination after a similar bug was found in dm.c
> >>
> >> Signed-off-by: NeilBrown 
> >> ---
> >>  block/bio.c | 2 +-
> >>  1 file changed, 1 insertion(+), 1 deletion(-)
> >>
> >> diff --git a/block/bio.c b/block/bio.c
> >> index e1708db48258..ad77140edc6f 100644
> >> --- a/block/bio.c
> >> +++ b/block/bio.c
> >> @@ -312,7 +312,7 @@ static struct bio *__bio_chain_endio(struct bio *bio)
> >>  {
> >>struct bio *parent = bio->bi_private;
> >>  
> >> -  if (!parent->bi_status)
> >> +  if (bio->bi_status)
> >>parent->bi_status = bio->bi_status;
> >>bio_put(bio);
> >>return parent;
> >> -- 
> >> 2.14.0.rc0.dirty
> >>
> > 
> > Reviewed-by: Mike Snitzer 
> > 
> > Jens, this one slipped through the crack just over a year ago.
> > It is available in patchwork here:
> > https://patchwork.kernel.org/patch/10220727/
> 
> Should this be:
> 
>   if (!parent->bi_status && bio->bi_status)
>   parent->bi_status = bio->bi_status;
> 
> perhaps?

Yeap, even better.  Not seeing any reason to have the last error win,
the first in the chain is likely the most important.

Re: block: be more careful about status in __bio_chain_endio

[Jens Axboe]

On 2/22/19 2:10 PM, Mike Snitzer wrote:
> On Thu, Feb 15 2018 at  4:09am -0500,
> NeilBrown  wrote:
> 
>>
>> If two bios are chained under the one parent (with bio_chain())
>> it is possible that one will succeed and the other will fail.
>> __bio_chain_endio must ensure that the failure error status
>> is reported for the whole, rather than the success.
>>
>> It currently tries to be careful, but this test is racy.
>> If both children finish at the same time, they might both see that
>> parent->bi_status as zero, and so will assign their own status.
>> If the assignment to parent->bi_status by the successful bio happens
>> last, the error status will be lost which can lead to silent data
>> corruption.
>>
>> Instead, __bio_chain_endio should only assign a non-zero status
>> to parent->bi_status.  There is then no need to test the current
>> value of parent->bi_status - a test that would be racy anyway.
>>
>> Note that this bug hasn't been seen in practice.  It was only discovered
>> by examination after a similar bug was found in dm.c
>>
>> Signed-off-by: NeilBrown 
>> ---
>>  block/bio.c | 2 +-
>>  1 file changed, 1 insertion(+), 1 deletion(-)
>>
>> diff --git a/block/bio.c b/block/bio.c
>> index e1708db48258..ad77140edc6f 100644
>> --- a/block/bio.c
>> +++ b/block/bio.c
>> @@ -312,7 +312,7 @@ static struct bio *__bio_chain_endio(struct bio *bio)
>>  {
>>  struct bio *parent = bio->bi_private;
>>  
>> -if (!parent->bi_status)
>> +if (bio->bi_status)
>>  parent->bi_status = bio->bi_status;
>>  bio_put(bio);
>>  return parent;
>> -- 
>> 2.14.0.rc0.dirty
>>
> 
> Reviewed-by: Mike Snitzer 
> 
> Jens, this one slipped through the crack just over a year ago.
> It is available in patchwork here:
> https://patchwork.kernel.org/patch/10220727/

Should this be:

if (!parent->bi_status && bio->bi_status)
parent->bi_status = bio->bi_status;

perhaps?

-- 
Jens Axboe

Re: block: be more careful about status in __bio_chain_endio

[Mike Snitzer]

On Thu, Feb 15 2018 at  4:09am -0500,
NeilBrown  wrote:

> 
> If two bios are chained under the one parent (with bio_chain())
> it is possible that one will succeed and the other will fail.
> __bio_chain_endio must ensure that the failure error status
> is reported for the whole, rather than the success.
> 
> It currently tries to be careful, but this test is racy.
> If both children finish at the same time, they might both see that
> parent->bi_status as zero, and so will assign their own status.
> If the assignment to parent->bi_status by the successful bio happens
> last, the error status will be lost which can lead to silent data
> corruption.
> 
> Instead, __bio_chain_endio should only assign a non-zero status
> to parent->bi_status.  There is then no need to test the current
> value of parent->bi_status - a test that would be racy anyway.
> 
> Note that this bug hasn't been seen in practice.  It was only discovered
> by examination after a similar bug was found in dm.c
> 
> Signed-off-by: NeilBrown 
> ---
>  block/bio.c | 2 +-
>  1 file changed, 1 insertion(+), 1 deletion(-)
> 
> diff --git a/block/bio.c b/block/bio.c
> index e1708db48258..ad77140edc6f 100644
> --- a/block/bio.c
> +++ b/block/bio.c
> @@ -312,7 +312,7 @@ static struct bio *__bio_chain_endio(struct bio *bio)
>  {
>   struct bio *parent = bio->bi_private;
>  
> - if (!parent->bi_status)
> + if (bio->bi_status)
>   parent->bi_status = bio->bi_status;
>   bio_put(bio);
>   return parent;
> -- 
> 2.14.0.rc0.dirty
> 

Reviewed-by: Mike Snitzer 

Jens, this one slipped through the crack just over a year ago.
It is available in patchwork here:
https://patchwork.kernel.org/patch/10220727/

[PATCH] block: be more careful about status in __bio_chain_endio

[NeilBrown]

If two bios are chained under the one parent (with bio_chain())
it is possible that one will succeed and the other will fail.
__bio_chain_endio must ensure that the failure error status
is reported for the whole, rather than the success.

It currently tries to be careful, but this test is racy.
If both children finish at the same time, they might both see that
parent->bi_status as zero, and so will assign their own status.
If the assignment to parent->bi_status by the successful bio happens
last, the error status will be lost which can lead to silent data
corruption.

Instead, __bio_chain_endio should only assign a non-zero status
to parent->bi_status.  There is then no need to test the current
value of parent->bi_status - a test that would be racy anyway.

Note that this bug hasn't been seen in practice.  It was only discovered
by examination after a similar bug was found in dm.c

Signed-off-by: NeilBrown 
---
 block/bio.c | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/block/bio.c b/block/bio.c
index e1708db48258..ad77140edc6f 100644
--- a/block/bio.c
+++ b/block/bio.c
@@ -312,7 +312,7 @@ static struct bio *__bio_chain_endio(struct bio *bio)
 {
struct bio *parent = bio->bi_private;
 
-   if (!parent->bi_status)
+   if (bio->bi_status)
parent->bi_status = bio->bi_status;
bio_put(bio);
return parent;
-- 
2.14.0.rc0.dirty



signature.asc
Description: PGP signature

[PATCH] block: be more careful about status in __bio_chain_endio

[NeilBrown]

If two bios are chained under the one parent (with bio_chain())
it is possible that one will succeed and the other will fail.
__bio_chain_endio must ensure that the failure error status
is reported for the whole, rather than the success.

It currently tries to be careful, but this test is racy.
If both children finish at the same time, they might both see that
parent->bi_status as zero, and so will assign their own status.
If the assignment to parent->bi_status by the successful bio happens
last, the error status will be lost which can lead to silent data
corruption.

Instead, __bio_chain_endio should only assign a non-zero status
to parent->bi_status.  There is then no need to test the current
value of parent->bi_status - a test that would be racy anyway.

Note that this bug hasn't been seen in practice.  It was only discovered
by examination after a similar bug was found in dm.c

Signed-off-by: NeilBrown 
---
 block/bio.c | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/block/bio.c b/block/bio.c
index e1708db48258..ad77140edc6f 100644
--- a/block/bio.c
+++ b/block/bio.c
@@ -312,7 +312,7 @@ static struct bio *__bio_chain_endio(struct bio *bio)
 {
struct bio *parent = bio->bi_private;
 
-   if (!parent->bi_status)
+   if (bio->bi_status)
parent->bi_status = bio->bi_status;
bio_put(bio);
return parent;
-- 
2.14.0.rc0.dirty



signature.asc
Description: PGP signature