Re: btrfs ENOSPC "not the usual problem"

2016-06-04 Thread Omari Stephens 

On 06/03/2016 05:42 PM, Liu Bo wrote:

On Thu, Jun 02, 2016 at 07:45:49PM +, Omari Stephens wrote:

[Note: not on list; please reply-all]

I've read everything I can find about running out of space on btrfs, and it
hasn't helped.  I'm currently dead in the water.

Everything I do seems to make the problem monotonically worse — I tried
adding a loopback device to the fs, and now I can't remove it.  Then I tried
adding a real device (mSATA) to the fs and now I still can't remove the
loopback device (which is making everything super slow), and I also can't
remove the mSATA.  I've removed about 100GB from the filesystem and that
hasn't done anything either.

Is there anything I can to do even figure out how bad things are, what I
need to do to make any kind of forward progress?  This is a laptop, so I
don't want to add an external drive only to find out that I can't remove it
without corrupting my filesystem.

### FILESYSTEM STATE
19:23:14> [root{slobol}@/home/xsdg]
#btrfs fi show /home
Label: none  uuid: 4776be5b-5058-4248-a1b7-7c213757dfbd
Total devices 3 FS bytes used 221.02GiB
devid1 size 418.72GiB used 413.72GiB path /dev/sda3
devid2 size 10.00GiB used 5.00GiB path /dev/loop0
devid3 size 14.91GiB used 3.00GiB path /dev/sdb1


19:23:33> [root{slobol}@/home/xsdg]
#btrfs fi usage /home
Overall:
Device size: 443.63GiB
Device allocated: 421.72GiB
Device unallocated:  21.91GiB
Device missing: 0.00B
Used: 221.68GiB
Free (estimated): 219.24GiB(min: 208.29GiB)
Data ratio:  1.00
Metadata ratio:  2.00
Global reserve: 228.00MiB(used: 36.00KiB)

Data,single: Size:417.69GiB, Used:220.36GiB
   /dev/loop0   5.00GiB
   /dev/sda3 409.69GiB
   /dev/sdb1   3.00GiB

Metadata,single: Size:8.00MiB, Used:0.00B
   /dev/sda3   8.00MiB

Metadata,DUP: Size:2.00GiB, Used:674.45MiB
   /dev/sda3   4.00GiB

System,single: Size:4.00MiB, Used:0.00B
   /dev/sda3   4.00MiB

System,DUP: Size:8.00MiB, Used:56.00KiB
   /dev/sda3  16.00MiB

Unallocated:
   /dev/loop0   5.00GiB
   /dev/sda3   5.00GiB
   /dev/sdb1  11.91GiB


### BALANCE FAILS, EVEN WITH -dusage=0
19:23:02> [root{slobol}@/home/xsdg]
#btrfs balance start -v -dusage=0 .
Dumping filters: flags 0x1, state 0x0, force is off
  DATA (flags 0x2): balancing, usage=0
ERROR: error during balancing '.': No space left on device
There may be more info in syslog - try dmesg | tail


1. Could you please show us your `uname -r`?

2. 
http://git.kernel.org/cgit/linux/kernel/git/kdave/btrfs-progs.git/tree/btrfs-debugfs
We need to know more information about block group in order to take more
fine-grained balance, so there is a tool for developer called
'btrfs-debugfs', you may download it from the above link, it's a python
script, as long as you're able to run it, try btrfs-debugfs -b
/your_partition.

Thanks,

-liubo


So, given some time and today, I backed up all the data from the 
partition and then threw 400GB of external drive at the problem. 
Thankfully, that got me out of the jam and I was able to remove the 
other devices from the filesystem, in addition to rerunning the balances 
(starting with -dusage 0, then -dusage 5, then -dusage 20).  After those 
steps, this is what I've got:


#btrfs fi show /home
Label: none  uuid: 4776be5b-5058-4248-a1b7-7c213757dfbd
Total devices 1 FS bytes used 220.44GiB
devid1 size 418.72GiB used 273.04GiB path /dev/sda3

I've rerun (and attached) the post-balance output of btrfs-debugfs in 
case it's useful for other folks.


Finally, the external drive happened to disappear in the middle of all 
this — my SATA-to-USB converter is a bit flaky — and I'm happy that 
btrfs complained but didn't OOPS or anything like that.  I unplugged it 
and plugged it back in and everything was happy again.



All that said, is there any plan/design to make it more difficult to 
paint yourself into a corner like this?  I still don't understand the 
nature of the problem, or why removing such large portions of the 
filesystem contents didn't seem to help anything.


--xsdg
block group offset 12582912 len 8388608 used 7405568 chunk_objectid 256 flags 1 
usage 0.88
block group offset 1103101952 len 1073741824 used 960143360 chunk_objectid 256 
flags 1 usage 0.89
block group offset 2176843776 len 1073741824 used 1068167168 chunk_objectid 256 
flags 1 usage 0.99
block group offset 3250585600 len 1073741824 used 1037885440 chunk_objectid 256 
flags 1 usage 0.97
block group offset 4324327424 len 1073741824 used 1009328128 chunk_objectid 256 
flags 1 usage 0.94
block group offset 5398069248 len 1073741824 used 850378752 chunk_objectid 256 
flags 1 usage 0.79
block group offset 6471811072 len 1073741824 used 483901440 chunk_objectid 256 
flags 1 usage 0.45
block group offset 7545552896 len 1073741824 used 327196672 chunk_objectid 256 
flags 1 usage 0.30
block

Re: btrfs ENOSPC "not the usual problem"

2016-06-04 Thread Omari Stephens 



On 06/03/2016 05:42 PM, Liu Bo wrote:

On Thu, Jun 02, 2016 at 07:45:49PM +, Omari Stephens wrote:

[Note: not on list; please reply-all]

I've read everything I can find about running out of space on btrfs, and it
hasn't helped.  I'm currently dead in the water.

Everything I do seems to make the problem monotonically worse — I tried
adding a loopback device to the fs, and now I can't remove it.  Then I tried
adding a real device (mSATA) to the fs and now I still can't remove the
loopback device (which is making everything super slow), and I also can't
remove the mSATA.  I've removed about 100GB from the filesystem and that
hasn't done anything either.

Is there anything I can to do even figure out how bad things are, what I
need to do to make any kind of forward progress?  This is a laptop, so I
don't want to add an external drive only to find out that I can't remove it
without corrupting my filesystem.

### FILESYSTEM STATE
19:23:14> [root{slobol}@/home/xsdg]
#btrfs fi show /home
Label: none  uuid: 4776be5b-5058-4248-a1b7-7c213757dfbd
Total devices 3 FS bytes used 221.02GiB
devid1 size 418.72GiB used 413.72GiB path /dev/sda3
devid2 size 10.00GiB used 5.00GiB path /dev/loop0
devid3 size 14.91GiB used 3.00GiB path /dev/sdb1


19:23:33> [root{slobol}@/home/xsdg]
#btrfs fi usage /home
Overall:
Device size: 443.63GiB
Device allocated: 421.72GiB
Device unallocated:  21.91GiB
Device missing: 0.00B
Used: 221.68GiB
Free (estimated): 219.24GiB(min: 208.29GiB)
Data ratio:  1.00
Metadata ratio:  2.00
Global reserve: 228.00MiB(used: 36.00KiB)

Data,single: Size:417.69GiB, Used:220.36GiB
   /dev/loop0   5.00GiB
   /dev/sda3 409.69GiB
   /dev/sdb1   3.00GiB

Metadata,single: Size:8.00MiB, Used:0.00B
   /dev/sda3   8.00MiB

Metadata,DUP: Size:2.00GiB, Used:674.45MiB
   /dev/sda3   4.00GiB

System,single: Size:4.00MiB, Used:0.00B
   /dev/sda3   4.00MiB

System,DUP: Size:8.00MiB, Used:56.00KiB
   /dev/sda3  16.00MiB

Unallocated:
   /dev/loop0   5.00GiB
   /dev/sda3   5.00GiB
   /dev/sdb1  11.91GiB


### BALANCE FAILS, EVEN WITH -dusage=0
19:23:02> [root{slobol}@/home/xsdg]
#btrfs balance start -v -dusage=0 .
Dumping filters: flags 0x1, state 0x0, force is off
  DATA (flags 0x2): balancing, usage=0
ERROR: error during balancing '.': No space left on device
There may be more info in syslog - try dmesg | tail


1. Could you please show us your `uname -r`?

2. 
http://git.kernel.org/cgit/linux/kernel/git/kdave/btrfs-progs.git/tree/btrfs-debugfs
We need to know more information about block group in order to take more
fine-grained balance, so there is a tool for developer called
'btrfs-debugfs', you may download it from the above link, it's a python
script, as long as you're able to run it, try btrfs-debugfs -b
/your_partition.

Thanks,

-liubo


07:41:19> [root{slobol}@/var/tmp]
#uname -a
Linux slobol 4.5.0-2-amd64 #1 SMP Debian 4.5.5-1 (2016-05-29) x86_64 
GNU/Linux


07:42:35> [root{slobol}@/var/tmp]
#btrfs --version
btrfs-progs v4.5.2

btrfs_debug output is attached.

--xsdg
block group offset 12582912 len 8388608 used 7405568 chunk_objectid 256 flags 1 
usage 0.88
block group offset 1103101952 len 1073741824 used 975503360 chunk_objectid 256 
flags 1 usage 0.91
block group offset 2176843776 len 1073741824 used 1068167168 chunk_objectid 256 
flags 1 usage 0.99
block group offset 3250585600 len 1073741824 used 1037885440 chunk_objectid 256 
flags 1 usage 0.97
block group offset 4324327424 len 1073741824 used 772653056 chunk_objectid 256 
flags 1 usage 0.72
block group offset 5398069248 len 1073741824 used 353058816 chunk_objectid 256 
flags 1 usage 0.33
block group offset 6471811072 len 1073741824 used 483901440 chunk_objectid 256 
flags 1 usage 0.45
block group offset 7545552896 len 1073741824 used 327196672 chunk_objectid 256 
flags 1 usage 0.30
block group offset 8619294720 len 1073741824 used 201629696 chunk_objectid 256 
flags 1 usage 0.19
block group offset 9693036544 len 1073741824 used 949104640 chunk_objectid 256 
flags 1 usage 0.88
block group offset 10766778368 len 1073741824 used 1062866944 chunk_objectid 
256 flags 1 usage 0.99
block group offset 11840520192 len 1073741824 used 1064804352 chunk_objectid 
256 flags 1 usage 0.99
block group offset 12914262016 len 1073741824 used 1064931328 chunk_objectid 
256 flags 1 usage 0.99
block group offset 13988003840 len 1073741824 used 1060679680 chunk_objectid 
256 flags 1 usage 0.99
block group offset 15061745664 len 1073741824 used 1065197568 chunk_objectid 
256 flags 1 usage 0.99
block group offset 16135487488 len 1073741824 used 1065959424 chunk_objectid 
256 flags 1 usage 0.99
block group offset 17209229312 len 1073741824 used 1067249664 chunk_objectid 
256 flags 1 usage 0.99
block group offset 18282971136 len 1073741824 used 1063280640 chunk_objectid 
256

Re: btrfs ENOSPC "not the usual problem"

2016-06-03 Thread Liu Bo 
On Thu, Jun 02, 2016 at 07:45:49PM +, Omari Stephens wrote:
> [Note: not on list; please reply-all]
> 
> I've read everything I can find about running out of space on btrfs, and it
> hasn't helped.  I'm currently dead in the water.
> 
> Everything I do seems to make the problem monotonically worse — I tried
> adding a loopback device to the fs, and now I can't remove it.  Then I tried
> adding a real device (mSATA) to the fs and now I still can't remove the
> loopback device (which is making everything super slow), and I also can't
> remove the mSATA.  I've removed about 100GB from the filesystem and that
> hasn't done anything either.
> 
> Is there anything I can to do even figure out how bad things are, what I
> need to do to make any kind of forward progress?  This is a laptop, so I
> don't want to add an external drive only to find out that I can't remove it
> without corrupting my filesystem.
> 
> ### FILESYSTEM STATE
> 19:23:14> [root{slobol}@/home/xsdg]
> #btrfs fi show /home
> Label: none  uuid: 4776be5b-5058-4248-a1b7-7c213757dfbd
> Total devices 3 FS bytes used 221.02GiB
> devid1 size 418.72GiB used 413.72GiB path /dev/sda3
> devid2 size 10.00GiB used 5.00GiB path /dev/loop0
> devid3 size 14.91GiB used 3.00GiB path /dev/sdb1
> 
> 
> 19:23:33> [root{slobol}@/home/xsdg]
> #btrfs fi usage /home
> Overall:
> Device size: 443.63GiB
> Device allocated: 421.72GiB
> Device unallocated:  21.91GiB
> Device missing: 0.00B
> Used: 221.68GiB
> Free (estimated): 219.24GiB(min: 208.29GiB)
> Data ratio:  1.00
> Metadata ratio:  2.00
> Global reserve: 228.00MiB(used: 36.00KiB)
> 
> Data,single: Size:417.69GiB, Used:220.36GiB
>/dev/loop0   5.00GiB
>/dev/sda3 409.69GiB
>/dev/sdb1   3.00GiB
> 
> Metadata,single: Size:8.00MiB, Used:0.00B
>/dev/sda3   8.00MiB
> 
> Metadata,DUP: Size:2.00GiB, Used:674.45MiB
>/dev/sda3   4.00GiB
> 
> System,single: Size:4.00MiB, Used:0.00B
>/dev/sda3   4.00MiB
> 
> System,DUP: Size:8.00MiB, Used:56.00KiB
>/dev/sda3  16.00MiB
> 
> Unallocated:
>/dev/loop0   5.00GiB
>/dev/sda3   5.00GiB
>/dev/sdb1  11.91GiB
> 
> 
> ### BALANCE FAILS, EVEN WITH -dusage=0
> 19:23:02> [root{slobol}@/home/xsdg]
> #btrfs balance start -v -dusage=0 .
> Dumping filters: flags 0x1, state 0x0, force is off
>   DATA (flags 0x2): balancing, usage=0
> ERROR: error during balancing '.': No space left on device
> There may be more info in syslog - try dmesg | tail

1. Could you please show us your `uname -r`?

2. 
http://git.kernel.org/cgit/linux/kernel/git/kdave/btrfs-progs.git/tree/btrfs-debugfs
We need to know more information about block group in order to take more
fine-grained balance, so there is a tool for developer called
'btrfs-debugfs', you may download it from the above link, it's a python
script, as long as you're able to run it, try btrfs-debugfs -b
/your_partition.

Thanks,

-liubo

> 
> 
> ### CAN'T REMOVE DEVICES -> ENOSPC
> #btrfs device remove /dev/loop0 /home
> ERROR: error removing device '/dev/loop0': No space left on device
> 
> --xsdg
> --
> To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in
> the body of a message to majord...@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
--
To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: btrfs ENOSPC "not the usual problem"

2016-06-02 Thread Chris Murphy 
On Thu, Jun 2, 2016 at 1:45 PM, Omari Stephens  wrote:
> [Note: not on list; please reply-all]
>
> I've read everything I can find about running out of space on btrfs, and it
> hasn't helped.  I'm currently dead in the water.
>
> Everything I do seems to make the problem monotonically worse — I tried
> adding a loopback device to the fs, and now I can't remove it.  Then I tried
> adding a real device (mSATA) to the fs and now I still can't remove the
> loopback device (which is making everything super slow), and I also can't
> remove the mSATA.  I've removed about 100GB from the filesystem and that
> hasn't done anything either.
>
> Is there anything I can to do even figure out how bad things are, what I
> need to do to make any kind of forward progress?  This is a laptop, so I
> don't want to add an external drive only to find out that I can't remove it
> without corrupting my filesystem.
>
> ### FILESYSTEM STATE
> 19:23:14> [root{slobol}@/home/xsdg]
> #btrfs fi show /home
> Label: none  uuid: 4776be5b-5058-4248-a1b7-7c213757dfbd
> Total devices 3 FS bytes used 221.02GiB
> devid1 size 418.72GiB used 413.72GiB path /dev/sda3
> devid2 size 10.00GiB used 5.00GiB path /dev/loop0
> devid3 size 14.91GiB used 3.00GiB path /dev/sdb1
>
>
> 19:23:33> [root{slobol}@/home/xsdg]
> #btrfs fi usage /home
> Overall:
> Device size: 443.63GiB
> Device allocated: 421.72GiB
> Device unallocated:  21.91GiB
> Device missing: 0.00B
> Used: 221.68GiB
> Free (estimated): 219.24GiB(min: 208.29GiB)
> Data ratio:  1.00
> Metadata ratio:  2.00
> Global reserve: 228.00MiB(used: 36.00KiB)
>
> Data,single: Size:417.69GiB, Used:220.36GiB
>/dev/loop0   5.00GiB
>/dev/sda3 409.69GiB
>/dev/sdb1   3.00GiB
>
> Metadata,single: Size:8.00MiB, Used:0.00B
>/dev/sda3   8.00MiB
>
> Metadata,DUP: Size:2.00GiB, Used:674.45MiB
>/dev/sda3   4.00GiB
>
> System,single: Size:4.00MiB, Used:0.00B
>/dev/sda3   4.00MiB
>
> System,DUP: Size:8.00MiB, Used:56.00KiB
>/dev/sda3  16.00MiB
>
> Unallocated:
>/dev/loop0   5.00GiB
>/dev/sda3   5.00GiB
>/dev/sdb1  11.91GiB
>
>
> ### BALANCE FAILS, EVEN WITH -dusage=0
> 19:23:02> [root{slobol}@/home/xsdg]
> #btrfs balance start -v -dusage=0 .
> Dumping filters: flags 0x1, state 0x0, force is off
>   DATA (flags 0x2): balancing, usage=0
> ERROR: error during balancing '.': No space left on device
> There may be more info in syslog - try dmesg | tail
>
>
> ### CAN'T REMOVE DEVICES -> ENOSPC
> #btrfs device remove /dev/loop0 /home
> ERROR: error removing device '/dev/loop0': No space left on device


Well the big problem here is that it's a loop device so even if it
were a known/fixed bug you're stuck being unable to boot; well, except
you could add a big enough device, convert to raid1, and reboot with
rootflags=degraded.

I'd use the external to make a backup, and start planning to make a
new fs, at least until someone else with a better idea arrives on
scene.



-- 
Chris Murphy
--
To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

btrfs ENOSPC "not the usual problem"

2016-06-02 Thread Omari Stephens 

[Note: not on list; please reply-all]

I've read everything I can find about running out of space on btrfs, and 
it hasn't helped.  I'm currently dead in the water.


Everything I do seems to make the problem monotonically worse — I tried 
adding a loopback device to the fs, and now I can't remove it.  Then I 
tried adding a real device (mSATA) to the fs and now I still can't 
remove the loopback device (which is making everything super slow), and 
I also can't remove the mSATA.  I've removed about 100GB from the 
filesystem and that hasn't done anything either.


Is there anything I can to do even figure out how bad things are, what I 
need to do to make any kind of forward progress?  This is a laptop, so I 
don't want to add an external drive only to find out that I can't remove 
it without corrupting my filesystem.


### FILESYSTEM STATE
19:23:14> [root{slobol}@/home/xsdg]
#btrfs fi show /home
Label: none  uuid: 4776be5b-5058-4248-a1b7-7c213757dfbd
Total devices 3 FS bytes used 221.02GiB
devid1 size 418.72GiB used 413.72GiB path /dev/sda3
devid2 size 10.00GiB used 5.00GiB path /dev/loop0
devid3 size 14.91GiB used 3.00GiB path /dev/sdb1


19:23:33> [root{slobol}@/home/xsdg]
#btrfs fi usage /home
Overall:
Device size: 443.63GiB
Device allocated: 421.72GiB
Device unallocated:  21.91GiB
Device missing: 0.00B
Used: 221.68GiB
Free (estimated): 219.24GiB(min: 208.29GiB)
Data ratio:  1.00
Metadata ratio:  2.00
Global reserve: 228.00MiB(used: 36.00KiB)

Data,single: Size:417.69GiB, Used:220.36GiB
   /dev/loop0   5.00GiB
   /dev/sda3 409.69GiB
   /dev/sdb1   3.00GiB

Metadata,single: Size:8.00MiB, Used:0.00B
   /dev/sda3   8.00MiB

Metadata,DUP: Size:2.00GiB, Used:674.45MiB
   /dev/sda3   4.00GiB

System,single: Size:4.00MiB, Used:0.00B
   /dev/sda3   4.00MiB

System,DUP: Size:8.00MiB, Used:56.00KiB
   /dev/sda3  16.00MiB

Unallocated:
   /dev/loop0   5.00GiB
   /dev/sda3   5.00GiB
   /dev/sdb1  11.91GiB


### BALANCE FAILS, EVEN WITH -dusage=0
19:23:02> [root{slobol}@/home/xsdg]
#btrfs balance start -v -dusage=0 .
Dumping filters: flags 0x1, state 0x0, force is off
  DATA (flags 0x2): balancing, usage=0
ERROR: error during balancing '.': No space left on device
There may be more info in syslog - try dmesg | tail


### CAN'T REMOVE DEVICES -> ENOSPC
#btrfs device remove /dev/loop0 /home
ERROR: error removing device '/dev/loop0': No space left on device

--xsdg
--
To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html