Re: strange No space left on device issues

[Hans van Kranenburg]

On 07/20/2017 11:53 PM, Christoph Anton Mitterer wrote:
> On Thu, 2017-07-20 at 14:48 -0700, Omar Sandoval wrote:
> [...]
> 
>>> I assume you'll take care to get that patch into stable kernels?
>>> Is this patch alone enough to recommend the Debian maintainers to
>>> include it into their 4.9 long term stable kernels?
>>
>> I'll mark it for stable, assuming Debian tracks the upstream LTS
>> releases it should get in.

+1 \o/

> Okay :-)
> 
> Nevertheless I'll open a bug at their BTS, just to be safe.

The Debian kernel teams tries to minimize the amount of patches on top
of the original kernel code. Every patch added comes with a maintenance
burden.

So, the answer will probably be: Please get it in 4.9 itself.

-- 
Hans van Kranenburg
--
To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: strange No space left on device issues

[Omar Sandoval]

On Thu, Jul 20, 2017 at 11:53:16PM +0200, Christoph Anton Mitterer wrote:
> On Thu, 2017-07-20 at 14:48 -0700, Omar Sandoval wrote:
> > Just to be sure, did you explicitly write 0 to these?
> Nope... that seemed to have been the default value, i.e. I used
> sysctl(8) in read (and not set) mode here.

Okay, all good then.

> > These sysctls are
> > really confusing, see https://www.kernel.org/doc/Documentation/sysctl
> > /vm.txt.
> > Basically, there are two ways to specify these, either as a ratio of
> > system memory (vm.dirty_ratio and vm.dirty_background_ratio) or a
> > static
> > number of bytes (vm.dirty_bytes and vm.dirty_background_bytes). If
> > you
> > set one, the other appears as 0, and the kernel sets the ratios by
> > default. But if you explicitly set them to 0, the kernel is going to
> > flush stuff extremely aggressively.
> I see,... not sure why both are 0 here... at least I didn't change it
> myself - must be something from the distro?

That's normal, the default is to have the ratio set instead:

$ sysctl vm.dirty_{,background_}{bytes,ratio}
vm.dirty_bytes = 0
vm.dirty_ratio = 20
vm.dirty_background_bytes = 0
vm.dirty_background_ratio = 10

> > Awesome, glad to hear it! I hadn't been able to reproduce the issue
> > outside of Facebook. Can I add your tested-by?
> Sure, but better use my other mail address for it, if you don't mind:
> Christoph Anton Mitterer 

No problem. I'll resend the patch with that shortly.

> > > I assume you'll take care to get that patch into stable kernels?
> > > Is this patch alone enough to recommend the Debian maintainers to
> > > include it into their 4.9 long term stable kernels?
> > 
> > I'll mark it for stable, assuming Debian tracks the upstream LTS
> > releases it should get in.
> Okay :-)
> 
> Nevertheless I'll open a bug at their BTS, just to be safe.
> 
> 
> Thanks :)
> 
> Chris.


--
To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: strange No space left on device issues

[Christoph Anton Mitterer]

On Thu, 2017-07-20 at 14:48 -0700, Omar Sandoval wrote:
> Just to be sure, did you explicitly write 0 to these?
Nope... that seemed to have been the default value, i.e. I used
sysctl(8) in read (and not set) mode here.



> These sysctls are
> really confusing, see https://www.kernel.org/doc/Documentation/sysctl
> /vm.txt.
> Basically, there are two ways to specify these, either as a ratio of
> system memory (vm.dirty_ratio and vm.dirty_background_ratio) or a
> static
> number of bytes (vm.dirty_bytes and vm.dirty_background_bytes). If
> you
> set one, the other appears as 0, and the kernel sets the ratios by
> default. But if you explicitly set them to 0, the kernel is going to
> flush stuff extremely aggressively.
I see,... not sure why both are 0 here... at least I didn't change it
myself - must be something from the distro?


> Awesome, glad to hear it! I hadn't been able to reproduce the issue
> outside of Facebook. Can I add your tested-by?
Sure, but better use my other mail address for it, if you don't mind:
Christoph Anton Mitterer 


> > I assume you'll take care to get that patch into stable kernels?
> > Is this patch alone enough to recommend the Debian maintainers to
> > include it into their 4.9 long term stable kernels?
> 
> I'll mark it for stable, assuming Debian tracks the upstream LTS
> releases it should get in.
Okay :-)

Nevertheless I'll open a bug at their BTS, just to be safe.


Thanks :)

Chris.

smime.p7s
Description: S/MIME cryptographic signature

Re: strange No space left on device issues

[Omar Sandoval]

On Thu, Jul 20, 2017 at 11:33:56PM +0200, Christoph Anton Mitterer wrote:
> On Thu, 2017-07-20 at 10:32 -0700, Omar Sandoval wrote:
> > If that doesn't work, could you please also try
> > https://patchwork.kernel.org/patch/9829593/?
> 
> Okay, tried the patch now, applied upon:
> Linux 4.12.0-trunk-amd64 #1 SMP Debian 4.12.2-1~exp1 (2017-07-18) x86_64 
> GNU/Linux
> (that is the Debian source package, with all their further patches and
> their kernel config).
> 
> with the parameters at their defaults:
> # sysctl vm.dirty_bytes
> vm.dirty_bytes = 0
> # sysctl vm.dirty_background_bytes
> vm.dirty_background_bytes = 0

Just to be sure, did you explicitly write 0 to these? These sysctls are
really confusing, see https://www.kernel.org/doc/Documentation/sysctl/vm.txt.
Basically, there are two ways to specify these, either as a ratio of
system memory (vm.dirty_ratio and vm.dirty_background_ratio) or a static
number of bytes (vm.dirty_bytes and vm.dirty_background_bytes). If you
set one, the other appears as 0, and the kernel sets the ratios by
default. But if you explicitly set them to 0, the kernel is going to
flush stuff extremely aggressively.

> Tried copying the whole image three times (before I haven had a single
> copy of the whole image with at least one error, so that should be
> "proof" enough that it fixes the isse) upon the btrfs fs,... no errors
> this time...
> 
> Looks good :-)

Awesome, glad to hear it! I hadn't been able to reproduce the issue
outside of Facebook. Can I add your tested-by?

> I assume you'll take care to get that patch into stable kernels?
> Is this patch alone enough to recommend the Debian maintainers to
> include it into their 4.9 long term stable kernels?

I'll mark it for stable, assuming Debian tracks the upstream LTS
releases it should get in.

> And would you recommend this as an "urgent" fix?

This bug has been around since 4.8, so it's not _that_ urgent, but it
sucks.

Thanks!

> Cheers,
> Chris.


--
To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: strange No space left on device issues

[Christoph Anton Mitterer]

On Thu, 2017-07-20 at 10:32 -0700, Omar Sandoval wrote:
> If that doesn't work, could you please also try
> https://patchwork.kernel.org/patch/9829593/?

Okay, tried the patch now, applied upon:
Linux 4.12.0-trunk-amd64 #1 SMP Debian 4.12.2-1~exp1 (2017-07-18) x86_64 
GNU/Linux
(that is the Debian source package, with all their further patches and
their kernel config).

with the parameters at their defaults:
# sysctl vm.dirty_bytes
vm.dirty_bytes = 0
# sysctl vm.dirty_background_bytes
vm.dirty_background_bytes = 0



Tried copying the whole image three times (before I haven had a single
copy of the whole image with at least one error, so that should be
"proof" enough that it fixes the isse) upon the btrfs fs,... no errors
this time...

Looks good :-)


I assume you'll take care to get that patch into stable kernels?
Is this patch alone enough to recommend the Debian maintainers to
include it into their 4.9 long term stable kernels?

And would you recommend this as an "urgent" fix?


Cheers,
Chris.

smime.p7s
Description: S/MIME cryptographic signature

Re: strange No space left on device issues

[Omar Sandoval]

On Thu, Jul 20, 2017 at 10:28:15PM +0200, Christoph Anton Mitterer wrote:
> On Thu, 2017-07-20 at 11:14 -0700, Omar Sandoval wrote:
> > Yes, that's a safe enough workaround. It's a good idea to change the
> > parameters back after the copy.
> you mean even without having the fix, right?

Yes, even without the fix you should be okay.

> So AFAIU, the bug doesn't really cause FS corruption, but just "false"
> ENOSPC and these happen during having meta-data creating (e.g. during
> operations like mine) only?

Right.
--
To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: strange No space left on device issues

[Christoph Anton Mitterer]

On Thu, 2017-07-20 at 11:14 -0700, Omar Sandoval wrote:
> Yes, that's a safe enough workaround. It's a good idea to change the
> parameters back after the copy.
you mean even without having the fix, right?

So AFAIU, the bug doesn't really cause FS corruption, but just "false"
ENOSPC and these happen during having meta-data creating (e.g. during
operations like mine) only?

smime.p7s
Description: S/MIME cryptographic signature

Re: strange No space left on device issues

[Omar Sandoval]

On Thu, Jul 20, 2017 at 08:06:52PM +0200, Christoph Anton Mitterer wrote:
> On Thu, 2017-07-20 at 10:55 -0700, Omar Sandoval wrote:
> > Against 4.12 would be best, thanks!
> okay,.. but that will take a while to compile...
> 
> 
> in the meantime... do you know whether it's more or less safe to use
> the 4.9 kernel without any fix, when I change the parameters mentioned
> before, during the massive copying?

Yes, that's a safe enough workaround. It's a good idea to change the
parameters back after the copy.

> Cheers,
> Chris.
--
To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: strange No space left on device issues

[Christoph Anton Mitterer]

On Thu, 2017-07-20 at 10:55 -0700, Omar Sandoval wrote:
> Against 4.12 would be best, thanks!
okay,.. but that will take a while to compile...


in the meantime... do you know whether it's more or less safe to use
the 4.9 kernel without any fix, when I change the parameters mentioned
before, during the massive copying?

Cheers,
Chris.

smime.p7s
Description: S/MIME cryptographic signature

Re: strange No space left on device issues

[Omar Sandoval]

On Thu, Jul 20, 2017 at 07:48:24PM +0200, Christoph Anton Mitterer wrote:
> On Thu, 2017-07-20 at 10:32 -0700, Omar Sandoval wrote:
> > Could you try 4.12?
> Linux 4.12.0-trunk-amd64 #1 SMP Debian 4.12.2-1~exp1 (2017-07-18)
> x86_64 GNU/Linux
> from Debian experimental, doesn't fix the issue...

Okay, didn't think it would :)

> >  If that doesn't work, could you please also try
> > https://patchwork.kernel.org/patch/9829593/?
> Against 4.9?

Against 4.12 would be best, thanks!
--
To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: strange No space left on device issues

[Christoph Anton Mitterer]

On Thu, 2017-07-20 at 10:32 -0700, Omar Sandoval wrote:
> Could you try 4.12?
Linux 4.12.0-trunk-amd64 #1 SMP Debian 4.12.2-1~exp1 (2017-07-18)
x86_64 GNU/Linux
from Debian experimental, doesn't fix the issue...


>  If that doesn't work, could you please also try
> https://patchwork.kernel.org/patch/9829593/?
Against 4.9?


Cheers,
Chris.

smime.p7s
Description: S/MIME cryptographic signature

Re: strange No space left on device issues

[Omar Sandoval]

On Thu, Jul 20, 2017 at 05:20:13PM +0200, Christoph Anton Mitterer wrote:
> On Thu, 2017-07-20 at 15:00 +, Martin Raiber wrote:
> > It would be interesting if lowering the dirty ratio is a viable
> > work-around (sysctl vm.dirty_background_bytes=314572800 && sysctl
> > vm.dirty_bytes=1258291200).
> > 
> > Regards,
> > Martin
> 
> I took away a trailing 0 for each of them... and then it goes through
> without error
> 
> sysctl vm.dirty_bytes=125829120
> vm.dirty_bytes = 125829120
> sysctl vm.dirty_background_bytes=31457280
> vm.dirty_background_bytes = 31457280
> 
> 
> But what does that mean now... could there be still any corruptions?
> And do you need to permanently set the value (until this is fixed in
> stable), or is this just necessary when I had this large copying
> operation?
> 
> 
> Cheers,
> Chris.

Could you try 4.12? If that doesn't work, could you please also try
https://patchwork.kernel.org/patch/9829593/?
--
To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: strange No space left on device issues

[Christoph Anton Mitterer]

On Thu, 2017-07-20 at 15:00 +, Martin Raiber wrote:
> there are patches on this list/upstream which could fix this ( e.g.
> "fix
> delalloc accounting leak caused by u32 overflow"/"fix early ENOSPC
> due
> to delalloc").

mhh... it's a bit problematic to test these on that nodes...


> Do you use compression?

nope...


> It would be interesting if lowering the dirty ratio is a viable
> work-around (sysctl vm.dirty_background_bytes=314572800 && sysctl
> vm.dirty_bytes=1258291200).

doesn't seem to change anything.

smime.p7s
Description: S/MIME cryptographic signature

Re: strange No space left on device issues

[Christoph Anton Mitterer]

On Thu, 2017-07-20 at 15:00 +, Martin Raiber wrote:
> It would be interesting if lowering the dirty ratio is a viable
> work-around (sysctl vm.dirty_background_bytes=314572800 && sysctl
> vm.dirty_bytes=1258291200).
> 
> Regards,
> Martin

I took away a trailing 0 for each of them... and then it goes through
without error

sysctl vm.dirty_bytes=125829120
vm.dirty_bytes = 125829120
sysctl vm.dirty_background_bytes=31457280
vm.dirty_background_bytes = 31457280


But what does that mean now... could there be still any corruptions?
And do you need to permanently set the value (until this is fixed in
stable), or is this just necessary when I had this large copying
operation?


Cheers,
Chris.

smime.p7s
Description: S/MIME cryptographic signature

Re: strange No space left on device issues

[Christoph Anton Mitterer]

Oh and I should add:
After such error, cp goes on copying (with other files)...

Same issue occurs when I do something like tar -cf - /media | tar -xf


Cheers,
Chris.

smime.p7s
Description: S/MIME cryptographic signature

Re: strange "No space left on device"

[Filipe Manana]

On Sat, Nov 7, 2015 at 11:22 PM, Christoph Anton Mitterer
 wrote:
> Hey.
>
> I just repeatedly did the following twice on a ~8GB USB stick, under
> Debian sid (ergo kernel 4.2.0-1-amd64, btrfsprogs 4.2.2-1).
>
> First, created some GPT on the stick:
> Number  Start (sector)End (sector)  Size   Code  Name
>12048 1048575   511.0 MiB   EF02  BIOS boot partition
>2 1048576 3145727   1024.0 MiB  8300  Linux filesystem
>3 3145728 4194303   512.0 MiB   8300  Linux filesystem
>4 4194304 6291455   1024.0 MiB  8300  Linux filesystem
>5 629145615687644   4.5 GiB 8300  Linux filesystem
>
>
> Then filesystems:
> root@heisenberg:~# mkfs.btrfs --nodiscard --label boot-main /dev/sdb2
> btrfs-progs v4.2.2
> See http://btrfs.wiki.kernel.org for more information.
>
> Label:  boot-main
> UUID:   150ee9fb-c650-4b5b-8f64-606e589e429a
> Node size:  16384
> Sector size:4096
> Filesystem size:1.00GiB
> Block group profiles:
>   Data: single8.00MiB
>   Metadata: DUP  59.19MiB
>   System:   DUP  12.00MiB
> SSD detected:   no
> Incompat features:  extref, skinny-metadata
> Number of devices:  1
> Devices:
>IDSIZE  PATH
> 1 1.00GiB  /dev/sdb2
>
> root@heisenberg:~# mkfs.btrfs --nodiscard --label boot-data /dev/sdb3
> btrfs-progs v4.2.2
> See http://btrfs.wiki.kernel.org for more information.
>
> Label:  boot-data
> UUID:   b1c1fc77-c965-4f0c-a2b5-44a301fd8772
> Node size:  16384
> Sector size:4096
> Filesystem size:1.00GiB
> Block group profiles:
>   Data: single8.00MiB
>   Metadata: DUP  59.19MiB
>   System:   DUP  12.00MiB
> SSD detected:   no
> Incompat features:  extref, skinny-metadata
> Number of devices:  1
> Devices:
>IDSIZE  PATH
> 1 1.00GiB  /dev/sdb3
>
> root@heisenberg:~# mkfs.btrfs --nodiscard --label boot-archive
> /dev/sdb4
> btrfs-progs v4.2.2
> See http://btrfs.wiki.kernel.org for more information.
>
> Label:  boot-archive
> UUID:   a063cf3b-0322-4ec7-a8c1-2dabecad9f57
> Node size:  16384
> Sector size:4096
> Filesystem size:1.00GiB
> Block group profiles:
>   Data: single8.00MiB
>   Metadata: DUP  59.19MiB
>   System:   DUP  12.00MiB
> SSD detected:   no
> Incompat features:  extref, skinny-metadata
> Number of devices:  1
> Devices:
>IDSIZE  PATH
> 1 1.00GiB  /dev/sdb4
>
> root@heisenberg:~# mkfs.btrfs --nodiscard --label boot-rescue /dev/sdb5
> btrfs-progs v4.2.2
> See http://btrfs.wiki.kernel.org for more information.
>
> Label:  boot-rescue
> UUID:   104b7bc3-3b8c-4a08-b0a6-9172ed664e68
> Node size:  16384
> Sector size:4096
> Filesystem size:3.98GiB
> Block group profiles:
>   Data: single8.00MiB
>   Metadata: DUP 211.75MiB
>   System:   DUP  12.00MiB
> SSD detected:   no
> Incompat features:  extref, skinny-metadata
> Number of devices:  1
> Devices:
>IDSIZE  PATH
> 1 3.98GiB  /dev/sdb5
>
>
>
> No errors in the kernel log.
>
>
>
> Then I got usage info:
> root@heisenberg:/data/SSS/boot# btrfs filesystem usage data/
> Overall:
> Device size:   1.00GiB
> Device allocated:126.38MiB
> Device unallocated:  897.62MiB
> Device missing:  0.00B
> Used:256.00KiB
> Free (estimated):905.62MiB  (min: 456.81MiB)
> Data ratio:   1.00
> Metadata ratio:   2.00
> Global reserve:   16.00MiB  (used: 0.00B)
>
> Data,single: Size:8.00MiB, Used:0.00B
>/dev/sdb3   8.00MiB
>
> Metadata,DUP: Size:51.19MiB, Used:112.00KiB
>/dev/sdb3 102.38MiB
>
> System,DUP: Size:8.00MiB, Used:16.00KiB
>/dev/sdb3  16.00MiB
>
> Unallocated:
>/dev/sdb3 897.62MiB
> root@heisenberg:/data/SSS/boot# btrfs filesystem usage main/
> Overall:
> Device size:   1.00GiB
> Device allocated:126.38MiB
> Device unallocated:  897.62MiB
> Device missing:  0.00B
> Used:256.00KiB
> Free (estimated):905.62MiB  (min: 456.81MiB)
> Data ratio:   1.00
> Metadata ratio:   2.00
> Global reserve:   16.00MiB  (used: 0.00B)
>
> Data,single: Size:8.00MiB, Used:0.00B
>/dev/sdb2   8.00MiB
>
> Metadata,DUP: Size:51.19MiB, Used:112.00KiB
>/dev/sdb2 102.38MiB
>
> System,DUP: Size:8.00MiB, Used:16.00KiB
>/dev/sdb2  16.00MiB
>
> 

Re: strange "No space left on device"

[Henk Slager]

[...]
>3 3145728 4194303   512.0 MiB   8300  Linux filesystem
[...]
> root@heisenberg:~# mkfs.btrfs --nodiscard --label boot-data /dev/sdb3
> btrfs-progs v4.2.2
> See http://btrfs.wiki.kernel.org for more information.
>
> Label:  boot-data
> UUID:   b1c1fc77-c965-4f0c-a2b5-44a301fd8772
> Node size:  16384
> Sector size:4096
> Filesystem size:1.00GiB
> Block group profiles:
>   Data: single8.00MiB
>   Metadata: DUP  59.19MiB
>   System:   DUP  12.00MiB
> SSD detected:   no
> Incompat features:  extref, skinny-metadata
> Number of devices:  1
> Devices:
>IDSIZE  PATH
> 1 1.00GiB  /dev/sdb3

Somehow your mkfs.btrfs (or some other part of the system) assumed the
wrong size for partition 3. To me, that sounds like a possible cause
of the errors and 'EiB' reported size.
For RAID tests in VMs I have also been using 4-8GB disk sizes (instead
of TB sized like the real hardware) and I encountered many 'No space
left on device' message (with older kernels).
A quick test with kernel 4.3.0 now:

# dd if=/dev/zero of=sd.bin bs=1M count=512

# losetup -f sd.bin

# mkfs.btrfs --nodiscard --label boot-data /dev/loop0
SMALL VOLUME: forcing mixed metadata/data groups
btrfs-progs v4.2.3+20151102
See http://btrfs.wiki.kernel.org for more information.

Label:  boot-data
UUID:   f152781b-b96b-4969-96b8-eede3e3dabc2
Node size:  4096
Sector size:4096
Filesystem size:512.00MiB
Block group profiles:
  Data+Metadata:single8.00MiB
  System:   single4.00MiB
SSD detected:   no
Incompat features:  mixed-bg, extref, skinny-metadata
Number of devices:  1
Devices:
   IDSIZE  PATH
1   512.00MiB  /dev/loop0
--
To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: strange "No space left on device"

[Duncan]

Henk Slager posted on Sun, 08 Nov 2015 19:18:19 +0100 as excerpted:

> [...]
>>3 3145728 4194303   512.0 MiB   8300  Linux filesystem
> [...]
>> root@heisenberg:~# mkfs.btrfs --nodiscard --label boot-data /dev/sdb3
>> btrfs-progs v4.2.2 See http://btrfs.wiki.kernel.org for more
>> information.
>>
>> Label:  boot-data
>> UUID:   b1c1fc77-c965-4f0c-a2b5-44a301fd8772
>> Node size:  16384
>> Sector size:4096
>> Filesystem size:1.00GiB
>> Block group profiles:
>>   Data: single8.00MiB
>>   Metadata: DUP  59.19MiB
>>   System:   DUP  12.00MiB
>> SSD detected:   no
>> Incompat features:  extref, skinny-metadata
>> Number of devices:  1 Devices:
>>IDSIZE  PATH
>> 1 1.00GiB  /dev/sdb3
> 
> Somehow your mkfs.btrfs (or some other part of the system) assumed the
> wrong size for partition 3. To me, that sounds like a possible cause of
> the errors and 'EiB' reported size.

Wow, yes!  Good catch, Henk! =:^)  Hugo obviously didn't catch it, and I 
wouldn't have either, as the bad size detection behavior is so 
unexpected, it just wouldn't occur to me to look!

Something's clearly wrong, and as it could clearly either overwrite the 
next partition(s) if the partitioning detection is at fault and doesn't 
stop it, or throw all sorts of monkey wrenches into btrfs behavior if it 
does, this is rather alarming behavior indeed!  Clearly we need devs to 
take a look and figure out what bugged out, and how to fix it.

That potentially explains why you got separate data/metadata instead of 
mixed mode, and why it worked most of the time but failed occasionally, 
since presumably when it worked the size detection worked as well.

(Apparently, btrfs-progs-4.3 does away with the default to mixed-mode at 
1 GiB or under, tho it is still recommended.  I'm not exactly sure of 
why, tho I think it had to do with being able to use sub-GiB btrfs for 
testing without having to worry about mixed mode.  Still, one would 
think, since it's recommended, that they'd simply provide an option to 
make it split mode, instead of changing the default to something that's 
not recommended.  So from 4.3 userspace you'd have to specifically set 
mixed, but you're on 4.2.2 userspace, so that shouldn't be the case yet.  
And in any case that doesn't explain the very badly detected size and an 
attempt to create a 1 GiB btrfs on a 1/2 GiB partition, which is 
obviously the real problem, regardless of whatever layer it occurred in.)

Meanwhile, btrfs fi usage doesn't understand mixed-mode yet, and returns 
8 EiB for some values if it is used, which is originally what I thought 
was happening.  But the problem is elsewhere, as it wasn't mixed mode, 
but rather, a very /very/ screwed up size detection.

-- 
Duncan - List replies preferred.   No HTML msgs.
"Every nonfree program has a lord, a master --
and if you use the program, he is your master."  Richard Stallman

--
To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: strange "No space left on device"

[Christoph Anton Mitterer]

On Sun, 2015-11-08 at 20:39 +, Duncan wrote:
> Wow, yes!  Good catch, Henk! =:^)  Hugo obviously didn't catch it,
> and I 
> wouldn't have either, as the bad size detection behavior is so 
> unexpected, it just wouldn't occur to me to look!
Hmm... all that *may* be more likely an error of myself when copying
and pasting the terminal output together:

I did actually change the 3rd partition to use 1GiB in later tries at
the expense of the 5th one shrinking, so the part table would have
looked like this in these later tries:
512M
1G
1G
1G
4G

Which would again fit the output of the various mkfs.btrfs.

Sorry if that was the case, apologies for any confusion.


The problem seemed to went away when explicitly using --mixed.


> (Apparently, btrfs-progs-4.3 does away with the default to mixed-
> mode at 1 GiB or under, tho it is still recommended.
Well I still had 4.2 ...

> I'm not exactly sure of
> why, tho I think it had to do with being able to use sub-GiB btrfs
> for 
> testing without having to worry about mixed mode.
Kinda strange... shouldn't it work out of the box for users and not
developers?

To be honest, no one should need to read through the wiki, just to be
able to create a small sized fs.

And even if no mixed D/M block group allocation was used... it
shouldn't just fail out-of-the-box with a few byte large files on a 1
GB fs.





Cheers,
Chris.

smime.p7s
Description: S/MIME cryptographic signature

Re: strange "No space left on device"

[Duncan]

Christoph Anton Mitterer posted on Sun, 08 Nov 2015 23:10:57 +0100 as
excerpted:

> On Sun, 2015-11-08 at 20:39 +, Duncan wrote:
>> Wow, yes!  Good catch, Henk! =:^)  Hugo obviously didn't catch it,
>> and I wouldn't have either, as the bad size detection behavior is so
>> unexpected, it just wouldn't occur to me to look!
> Hmm... all that *may* be more likely an error of myself when copying and
> pasting the terminal output together:
> 
> I did actually change the 3rd partition to use 1GiB in later tries at
> the expense of the 5th one shrinking, so the part table would have
> looked like this in these later tries:
> 512M 1G 1G 1G 4G

Oops! =:^(

> Which would again fit the output of the various mkfs.btrfs.
> 
> Sorry if that was the case, apologies for any confusion.
> 
> 
> The problem seemed to went away when explicitly using --mixed.

Good.  Whatever the cause of it not doing mixed by default (perhaps it's 
an off-by-one error and filesystems of exactly 1 GiB weren't defaulting 
to mixed as expected), the fact that mixed was the default on small 
filesystems would mean separate data/metadata wouldn't have gotten the 
same full testing coverage, so maybe there's a bug...  In which case the 
4.3 change discussed below would certainly tend to trigger it more often, 
hopefully getting it fixed. =:^)


>> (Apparently, btrfs-progs-4.3 does away with the default to mixed- mode
>> at 1 GiB or under, tho it is still recommended.

> Well I still had 4.2 ...

That's why I had that as a parenthetical.  It didn't apply to your 
current case, but could to others, and I've learned to write with both 
the lurkers and the googlers who will happen on my post later, in mind.

>> I'm not exactly sure of why, tho I think it had to do with being able
>> to use sub-GiB btrfs for testing without having to worry about mixed
>> mode.

> Kinda strange... shouldn't it work out of the box for users and not
> developers?

> To be honest, no one should need to read through the wiki, just to be
> able to create a small sized fs.

That's why, tho I didn't press it, I asked for clarification when I saw 
it mentioned on the btrfs-progs 4.3-rc1 announcement, verifying that 
mixed was still recommended.  Chris Murphy (another non-btrfs-dev list 
regular, I think he works or volunteers for one of the distros) 
questioned it as well, on the same announcement.

But not being able to personally show-them-the-code, as long as there's 
at least an option to do it the way I want (which there is, the already 
mentioned --mixed option), I've learned not to press it.[1]

> And even if no mixed D/M block group allocation was used... it shouldn't
> just fail out-of-the-box with a few byte large files on a 1 GB fs.

You are absolutely correct.  There's a bug somewhere.  Earlier I thought 
it was the size detection bug Henk spotted, since even if there's some 
other bug here it'd pale in comparision, but since that bug turned out to 
trace to your copy/paste, that isn't it, and we still have the much more 
ordinary level bug to deal with.

But as I said above, with progs 4.3 defaulting to unmixed even on small 
filesystems now, if indeed it is tied to unmixed on a small filesystem, 
that bug's very likely to get MUCH more common, and thus should be fixed 
relatively quickly, I'd guess by the time 4.4 comes out, given the 
history of other bugs that turn out to be quite common.

---
[1] Tho for my own systems I sometimes patch in the behavior as I think 
it should be, as my patching of existing code skills are dramatically 
better than my new-coding skills, and I run gentoo so already build 
everything from sources.  I finally got tired of the kernel not 
defaulting to noatime, and hacked up a patch to do it, so I could remove 
noatime from all my fstab entries, for instance.  But I already use a 
mkfs.btrfs helper script that adds options I prefer, so in this 
particular case, simply adding --mixed under appropriate conditions in 
the script is a far easier personal solution.

-- 
Duncan - List replies preferred.   No HTML msgs.
"Every nonfree program has a lord, a master --
and if you use the program, he is your master."  Richard Stallman

--
To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: strange "No space left on device"

[Christoph Anton Mitterer]

On Sat, 2015-11-07 at 23:30 +, Hugo Mills wrote:
>    These are all really small.
Well enough for booting =)


>    I would suggest running mkfs with --mixed for all of these
> filesystems and trying again.
I thought btrfs would do that automatically:
https://btrfs.wiki.kernel.org/index.php/FAQ#if_your_device_is_small
"Using mixed block groups is recommended for filesystems of 1GiB or
smaller and mkfs.btrfs will force mixed block groups automatically in
that case."

Anyway I even if that's the reason, than I think something's quite
wrong here... a) it doesn't happen always (I just create 10 times a fs
in the same partition, and no problem) ... b) in those cases where I
could produce the issue, it went away after a balance... so perhaps
they just just balance the fs right after mkfs o.O

Cheers,
Chris.

smime.p7s
Description: S/MIME cryptographic signature

Re: strange "No space left on device"

[Hugo Mills]

On Sun, Nov 08, 2015 at 12:22:42AM +0100, Christoph Anton Mitterer wrote:
> Hey.
> 
> I just repeatedly did the following twice on a ~8GB USB stick, under
> Debian sid (ergo kernel 4.2.0-1-amd64, btrfsprogs 4.2.2-1).
> 
> First, created some GPT on the stick:
> Number  Start (sector)End (sector)  Size   Code  Name
>    12048 1048575   511.0 MiB   EF02  BIOS boot partition
>    2 1048576 3145727   1024.0 MiB  8300  Linux filesystem
>    3 3145728 4194303   512.0 MiB   8300  Linux filesystem
>    4 4194304 6291455   1024.0 MiB  8300  Linux filesystem
>    5 629145615687644   4.5 GiB 8300  Linux filesystem

   These are all really small.

   I would suggest running mkfs with --mixed for all of these
filesystems and trying again.

   Hugo.

> Then filesystems:
> root@heisenberg:~# mkfs.btrfs --nodiscard --label boot-main /dev/sdb2 
> btrfs-progs v4.2.2
> See http://btrfs.wiki.kernel.org for more information.
> 
> Label:  boot-main
> UUID:   150ee9fb-c650-4b5b-8f64-606e589e429a
> Node size:  16384
> Sector size:4096
> Filesystem size:1.00GiB
> Block group profiles:
>   Data: single8.00MiB
>   Metadata: DUP  59.19MiB
>   System:   DUP  12.00MiB
> SSD detected:   no
> Incompat features:  extref, skinny-metadata
> Number of devices:  1
> Devices:
>    IDSIZE  PATH
> 1 1.00GiB  /dev/sdb2
> 
> root@heisenberg:~# mkfs.btrfs --nodiscard --label boot-data /dev/sdb3
> btrfs-progs v4.2.2
> See http://btrfs.wiki.kernel.org for more information.
> 
> Label:  boot-data
> UUID:   b1c1fc77-c965-4f0c-a2b5-44a301fd8772
> Node size:  16384
> Sector size:4096
> Filesystem size:1.00GiB
> Block group profiles:
>   Data: single8.00MiB
>   Metadata: DUP  59.19MiB
>   System:   DUP  12.00MiB
> SSD detected:   no
> Incompat features:  extref, skinny-metadata
> Number of devices:  1
> Devices:
>    IDSIZE  PATH
> 1 1.00GiB  /dev/sdb3
> 
> root@heisenberg:~# mkfs.btrfs --nodiscard --label boot-archive
> /dev/sdb4
> btrfs-progs v4.2.2
> See http://btrfs.wiki.kernel.org for more information.
> 
> Label:  boot-archive
> UUID:   a063cf3b-0322-4ec7-a8c1-2dabecad9f57
> Node size:  16384
> Sector size:4096
> Filesystem size:1.00GiB
> Block group profiles:
>   Data: single8.00MiB
>   Metadata: DUP  59.19MiB
>   System:   DUP  12.00MiB
> SSD detected:   no
> Incompat features:  extref, skinny-metadata
> Number of devices:  1
> Devices:
>    IDSIZE  PATH
> 1 1.00GiB  /dev/sdb4
> 
> root@heisenberg:~# mkfs.btrfs --nodiscard --label boot-rescue /dev/sdb5
> btrfs-progs v4.2.2
> See http://btrfs.wiki.kernel.org for more information.
> 
> Label:  boot-rescue
> UUID:   104b7bc3-3b8c-4a08-b0a6-9172ed664e68
> Node size:  16384
> Sector size:4096
> Filesystem size:3.98GiB
> Block group profiles:
>   Data: single8.00MiB
>   Metadata: DUP 211.75MiB
>   System:   DUP  12.00MiB
> SSD detected:   no
> Incompat features:  extref, skinny-metadata
> Number of devices:  1
> Devices:
>    IDSIZE  PATH
> 1 3.98GiB  /dev/sdb5
> 
> 
> 
> No errors in the kernel log.
> 
> 
> 
> Then I got usage info:
> root@heisenberg:/data/SSS/boot# btrfs filesystem usage data/
> Overall:
> Device size:     1.00GiB
> Device allocated:  126.38MiB
> Device unallocated:    897.62MiB
> Device missing:    0.00B
> Used:  256.00KiB
> Free (estimated):  905.62MiB  (min: 456.81MiB)
> Data ratio:     1.00
> Metadata ratio:     2.00
> Global reserve:     16.00MiB  (used: 0.00B)
> 
> Data,single: Size:8.00MiB, Used:0.00B
>    /dev/sdb3     8.00MiB
> 
> Metadata,DUP: Size:51.19MiB, Used:112.00KiB
>    /dev/sdb3   102.38MiB
> 
> System,DUP: Size:8.00MiB, Used:16.00KiB
>    /dev/sdb3    16.00MiB
> 
> Unallocated:
>    /dev/sdb3   897.62MiB
> root@heisenberg:/data/SSS/boot# btrfs filesystem usage main/
> Overall:
> Device size:     1.00GiB
> Device allocated:  126.38MiB
> Device unallocated:    897.62MiB
> Device missing:    0.00B
> Used:  256.00KiB
> Free (estimated):  905.62MiB  (min: 456.81MiB)
> Data ratio:     1.00
> Metadata ratio:     2.00
> Global reserve:     16.00MiB  (used: 0.00B)
> 
> Data,single: Size:8.00MiB, Used:0.00B
>    /dev/sdb2     8.00MiB
> 
> Metadata,DUP: