Re: strange No space left on device issues

[Hans van Kranenburg]

On 07/20/2017 11:53 PM, Christoph Anton Mitterer wrote:
> On Thu, 2017-07-20 at 14:48 -0700, Omar Sandoval wrote:
> [...]
> 
>>> I assume you'll take care to get that patch into stable kernels?
>>> Is this patch alone enough to recommend the Debian maintainers to
>>> include it into their 4.9 long term stable kernels?
>>
>> I'll mark it for stable, assuming Debian tracks the upstream LTS
>> releases it should get in.

+1 \o/

> Okay :-)
> 
> Nevertheless I'll open a bug at their BTS, just to be safe.

The Debian kernel teams tries to minimize the amount of patches on top
of the original kernel code. Every patch added comes with a maintenance
burden.

So, the answer will probably be: Please get it in 4.9 itself.

-- 
Hans van Kranenburg
--
To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: strange No space left on device issues

[Omar Sandoval]

On Thu, Jul 20, 2017 at 11:53:16PM +0200, Christoph Anton Mitterer wrote:
> On Thu, 2017-07-20 at 14:48 -0700, Omar Sandoval wrote:
> > Just to be sure, did you explicitly write 0 to these?
> Nope... that seemed to have been the default value, i.e. I used
> sysctl(8) in read (and not set) mode here.

Okay, all good then.

> > These sysctls are
> > really confusing, see https://www.kernel.org/doc/Documentation/sysctl
> > /vm.txt.
> > Basically, there are two ways to specify these, either as a ratio of
> > system memory (vm.dirty_ratio and vm.dirty_background_ratio) or a
> > static
> > number of bytes (vm.dirty_bytes and vm.dirty_background_bytes). If
> > you
> > set one, the other appears as 0, and the kernel sets the ratios by
> > default. But if you explicitly set them to 0, the kernel is going to
> > flush stuff extremely aggressively.
> I see,... not sure why both are 0 here... at least I didn't change it
> myself - must be something from the distro?

That's normal, the default is to have the ratio set instead:

$ sysctl vm.dirty_{,background_}{bytes,ratio}
vm.dirty_bytes = 0
vm.dirty_ratio = 20
vm.dirty_background_bytes = 0
vm.dirty_background_ratio = 10

> > Awesome, glad to hear it! I hadn't been able to reproduce the issue
> > outside of Facebook. Can I add your tested-by?
> Sure, but better use my other mail address for it, if you don't mind:
> Christoph Anton Mitterer 

No problem. I'll resend the patch with that shortly.

> > > I assume you'll take care to get that patch into stable kernels?
> > > Is this patch alone enough to recommend the Debian maintainers to
> > > include it into their 4.9 long term stable kernels?
> > 
> > I'll mark it for stable, assuming Debian tracks the upstream LTS
> > releases it should get in.
> Okay :-)
> 
> Nevertheless I'll open a bug at their BTS, just to be safe.
> 
> 
> Thanks :)
> 
> Chris.


--
To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: strange No space left on device issues

[Christoph Anton Mitterer]

On Thu, 2017-07-20 at 14:48 -0700, Omar Sandoval wrote:
> Just to be sure, did you explicitly write 0 to these?
Nope... that seemed to have been the default value, i.e. I used
sysctl(8) in read (and not set) mode here.



> These sysctls are
> really confusing, see https://www.kernel.org/doc/Documentation/sysctl
> /vm.txt.
> Basically, there are two ways to specify these, either as a ratio of
> system memory (vm.dirty_ratio and vm.dirty_background_ratio) or a
> static
> number of bytes (vm.dirty_bytes and vm.dirty_background_bytes). If
> you
> set one, the other appears as 0, and the kernel sets the ratios by
> default. But if you explicitly set them to 0, the kernel is going to
> flush stuff extremely aggressively.
I see,... not sure why both are 0 here... at least I didn't change it
myself - must be something from the distro?


> Awesome, glad to hear it! I hadn't been able to reproduce the issue
> outside of Facebook. Can I add your tested-by?
Sure, but better use my other mail address for it, if you don't mind:
Christoph Anton Mitterer 


> > I assume you'll take care to get that patch into stable kernels?
> > Is this patch alone enough to recommend the Debian maintainers to
> > include it into their 4.9 long term stable kernels?
> 
> I'll mark it for stable, assuming Debian tracks the upstream LTS
> releases it should get in.
Okay :-)

Nevertheless I'll open a bug at their BTS, just to be safe.


Thanks :)

Chris.

smime.p7s
Description: S/MIME cryptographic signature

Re: strange No space left on device issues

[Omar Sandoval]

On Thu, Jul 20, 2017 at 11:33:56PM +0200, Christoph Anton Mitterer wrote:
> On Thu, 2017-07-20 at 10:32 -0700, Omar Sandoval wrote:
> > If that doesn't work, could you please also try
> > https://patchwork.kernel.org/patch/9829593/?
> 
> Okay, tried the patch now, applied upon:
> Linux 4.12.0-trunk-amd64 #1 SMP Debian 4.12.2-1~exp1 (2017-07-18) x86_64 
> GNU/Linux
> (that is the Debian source package, with all their further patches and
> their kernel config).
> 
> with the parameters at their defaults:
> # sysctl vm.dirty_bytes
> vm.dirty_bytes = 0
> # sysctl vm.dirty_background_bytes
> vm.dirty_background_bytes = 0

Just to be sure, did you explicitly write 0 to these? These sysctls are
really confusing, see https://www.kernel.org/doc/Documentation/sysctl/vm.txt.
Basically, there are two ways to specify these, either as a ratio of
system memory (vm.dirty_ratio and vm.dirty_background_ratio) or a static
number of bytes (vm.dirty_bytes and vm.dirty_background_bytes). If you
set one, the other appears as 0, and the kernel sets the ratios by
default. But if you explicitly set them to 0, the kernel is going to
flush stuff extremely aggressively.

> Tried copying the whole image three times (before I haven had a single
> copy of the whole image with at least one error, so that should be
> "proof" enough that it fixes the isse) upon the btrfs fs,... no errors
> this time...
> 
> Looks good :-)

Awesome, glad to hear it! I hadn't been able to reproduce the issue
outside of Facebook. Can I add your tested-by?

> I assume you'll take care to get that patch into stable kernels?
> Is this patch alone enough to recommend the Debian maintainers to
> include it into their 4.9 long term stable kernels?

I'll mark it for stable, assuming Debian tracks the upstream LTS
releases it should get in.

> And would you recommend this as an "urgent" fix?

This bug has been around since 4.8, so it's not _that_ urgent, but it
sucks.

Thanks!

> Cheers,
> Chris.


--
To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: strange No space left on device issues

[Christoph Anton Mitterer]

On Thu, 2017-07-20 at 10:32 -0700, Omar Sandoval wrote:
> If that doesn't work, could you please also try
> https://patchwork.kernel.org/patch/9829593/?

Okay, tried the patch now, applied upon:
Linux 4.12.0-trunk-amd64 #1 SMP Debian 4.12.2-1~exp1 (2017-07-18) x86_64 
GNU/Linux
(that is the Debian source package, with all their further patches and
their kernel config).

with the parameters at their defaults:
# sysctl vm.dirty_bytes
vm.dirty_bytes = 0
# sysctl vm.dirty_background_bytes
vm.dirty_background_bytes = 0



Tried copying the whole image three times (before I haven had a single
copy of the whole image with at least one error, so that should be
"proof" enough that it fixes the isse) upon the btrfs fs,... no errors
this time...

Looks good :-)


I assume you'll take care to get that patch into stable kernels?
Is this patch alone enough to recommend the Debian maintainers to
include it into their 4.9 long term stable kernels?

And would you recommend this as an "urgent" fix?


Cheers,
Chris.

smime.p7s
Description: S/MIME cryptographic signature

Re: strange No space left on device issues

[Omar Sandoval]

On Thu, Jul 20, 2017 at 10:28:15PM +0200, Christoph Anton Mitterer wrote:
> On Thu, 2017-07-20 at 11:14 -0700, Omar Sandoval wrote:
> > Yes, that's a safe enough workaround. It's a good idea to change the
> > parameters back after the copy.
> you mean even without having the fix, right?

Yes, even without the fix you should be okay.

> So AFAIU, the bug doesn't really cause FS corruption, but just "false"
> ENOSPC and these happen during having meta-data creating (e.g. during
> operations like mine) only?

Right.
--
To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: strange No space left on device issues

[Christoph Anton Mitterer]

On Thu, 2017-07-20 at 11:14 -0700, Omar Sandoval wrote:
> Yes, that's a safe enough workaround. It's a good idea to change the
> parameters back after the copy.
you mean even without having the fix, right?

So AFAIU, the bug doesn't really cause FS corruption, but just "false"
ENOSPC and these happen during having meta-data creating (e.g. during
operations like mine) only?

smime.p7s
Description: S/MIME cryptographic signature

Re: strange No space left on device issues

[Omar Sandoval]

On Thu, Jul 20, 2017 at 08:06:52PM +0200, Christoph Anton Mitterer wrote:
> On Thu, 2017-07-20 at 10:55 -0700, Omar Sandoval wrote:
> > Against 4.12 would be best, thanks!
> okay,.. but that will take a while to compile...
> 
> 
> in the meantime... do you know whether it's more or less safe to use
> the 4.9 kernel without any fix, when I change the parameters mentioned
> before, during the massive copying?

Yes, that's a safe enough workaround. It's a good idea to change the
parameters back after the copy.

> Cheers,
> Chris.
--
To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: strange No space left on device issues

[Christoph Anton Mitterer]

On Thu, 2017-07-20 at 10:55 -0700, Omar Sandoval wrote:
> Against 4.12 would be best, thanks!
okay,.. but that will take a while to compile...


in the meantime... do you know whether it's more or less safe to use
the 4.9 kernel without any fix, when I change the parameters mentioned
before, during the massive copying?

Cheers,
Chris.

smime.p7s
Description: S/MIME cryptographic signature

Re: strange No space left on device issues

[Omar Sandoval]

On Thu, Jul 20, 2017 at 07:48:24PM +0200, Christoph Anton Mitterer wrote:
> On Thu, 2017-07-20 at 10:32 -0700, Omar Sandoval wrote:
> > Could you try 4.12?
> Linux 4.12.0-trunk-amd64 #1 SMP Debian 4.12.2-1~exp1 (2017-07-18)
> x86_64 GNU/Linux
> from Debian experimental, doesn't fix the issue...

Okay, didn't think it would :)

> >  If that doesn't work, could you please also try
> > https://patchwork.kernel.org/patch/9829593/?
> Against 4.9?

Against 4.12 would be best, thanks!
--
To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: strange No space left on device issues

[Christoph Anton Mitterer]

On Thu, 2017-07-20 at 10:32 -0700, Omar Sandoval wrote:
> Could you try 4.12?
Linux 4.12.0-trunk-amd64 #1 SMP Debian 4.12.2-1~exp1 (2017-07-18)
x86_64 GNU/Linux
from Debian experimental, doesn't fix the issue...


>  If that doesn't work, could you please also try
> https://patchwork.kernel.org/patch/9829593/?
Against 4.9?


Cheers,
Chris.

smime.p7s
Description: S/MIME cryptographic signature

Re: strange No space left on device issues

[Omar Sandoval]

On Thu, Jul 20, 2017 at 05:20:13PM +0200, Christoph Anton Mitterer wrote:
> On Thu, 2017-07-20 at 15:00 +, Martin Raiber wrote:
> > It would be interesting if lowering the dirty ratio is a viable
> > work-around (sysctl vm.dirty_background_bytes=314572800 && sysctl
> > vm.dirty_bytes=1258291200).
> > 
> > Regards,
> > Martin
> 
> I took away a trailing 0 for each of them... and then it goes through
> without error
> 
> sysctl vm.dirty_bytes=125829120
> vm.dirty_bytes = 125829120
> sysctl vm.dirty_background_bytes=31457280
> vm.dirty_background_bytes = 31457280
> 
> 
> But what does that mean now... could there be still any corruptions?
> And do you need to permanently set the value (until this is fixed in
> stable), or is this just necessary when I had this large copying
> operation?
> 
> 
> Cheers,
> Chris.

Could you try 4.12? If that doesn't work, could you please also try
https://patchwork.kernel.org/patch/9829593/?
--
To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: strange No space left on device issues

[Christoph Anton Mitterer]

On Thu, 2017-07-20 at 15:00 +, Martin Raiber wrote:
> there are patches on this list/upstream which could fix this ( e.g.
> "fix
> delalloc accounting leak caused by u32 overflow"/"fix early ENOSPC
> due
> to delalloc").

mhh... it's a bit problematic to test these on that nodes...


> Do you use compression?

nope...


> It would be interesting if lowering the dirty ratio is a viable
> work-around (sysctl vm.dirty_background_bytes=314572800 && sysctl
> vm.dirty_bytes=1258291200).

doesn't seem to change anything.

smime.p7s
Description: S/MIME cryptographic signature

Re: strange No space left on device issues

[Christoph Anton Mitterer]

On Thu, 2017-07-20 at 15:00 +, Martin Raiber wrote:
> It would be interesting if lowering the dirty ratio is a viable
> work-around (sysctl vm.dirty_background_bytes=314572800 && sysctl
> vm.dirty_bytes=1258291200).
> 
> Regards,
> Martin

I took away a trailing 0 for each of them... and then it goes through
without error

sysctl vm.dirty_bytes=125829120
vm.dirty_bytes = 125829120
sysctl vm.dirty_background_bytes=31457280
vm.dirty_background_bytes = 31457280


But what does that mean now... could there be still any corruptions?
And do you need to permanently set the value (until this is fixed in
stable), or is this just necessary when I had this large copying
operation?


Cheers,
Chris.

smime.p7s
Description: S/MIME cryptographic signature

Re: strange No space left on device issues

[Christoph Anton Mitterer]

Oh and I should add:
After such error, cp goes on copying (with other files)...

Same issue occurs when I do something like tar -cf - /media | tar -xf


Cheers,
Chris.

smime.p7s
Description: S/MIME cryptographic signature