subject:"ENOSPC errors during balance"

Re: Problem converting data raid0 to raid1: enospc errors during balance

2014-10-27 Thread Chris Murphy

 
  On Oct 26, 2014, at 7:40 PM, Qu Wenruo quwen...@cn.fujitsu.com wrote:
 
 BTW what's the output of 'df' command?

Jasper,
What do you get for the conventional df command when this btrfs volume is 
mounted? Thanks.

Chris Murphy--
To unsubscribe from this list: send the line unsubscribe linux-btrfs in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

RE: Problem converting data raid0 to raid1: enospc errors during balance

2014-10-27 Thread Jasper Verberk

Hej guys!

Thanks for your input on the issue this far.

Too my knowledge raid1 in btrfs means 2 copies of each piece of data 
independent of the amount of disks used.

So 4 x 2,73tb would result in a totaal storage of roughly 5,5tb right?

Shouldn't this be more then enough?

btw, here is the output for df:

http://paste.debian.net/128932/



 Date: Mon, 27 Oct 2014 12:49:15 +0800
 From: quwen...@cn.fujitsu.com
 To: li...@colorremedies.com
 CC: jverb...@hotmail.com; linux-btrfs@vger.kernel.org
 Subject: Re: Problem converting data raid0 to raid1: enospc errors during 
 balance


  Original Message 
 Subject: Re: Problem converting data raid0 to raid1: enospc errors
 during balance
 From: Chris Murphy li...@colorremedies.com
 To: Qu Wenruo quwen...@cn.fujitsu.com
 Date: 2014年10月27日 12:40
 On Oct 26, 2014, at 7:40 PM, Qu Wenruo quwen...@cn.fujitsu.com wrote:

 Hi,

 Although I'm not completely sure, but it seems that, you really ran out of 
 space.

 [1] Your array won't hold raid1 for 1.97T data
 Your array used up 1.97T raid0 data, it takes 1.97T for raid0.
 But if converted to 1.97T, it will occupy 1.97T X2 = 3.94T.
 Your array are only 2.73T, it is too small to contain the data.
 I'm not understanding. The btrfs fi show, shows 4x 2.73TiB devices, so that 
 seems like it's a 10+TiB array.

 There's 2.04TiB raid0 data chunks, so roughly 500GiB per device, yet 1.94TiB 
 is reported used per device by fi show. Confusing.

 Also it's still very confusing: Data, RAID1: total=2.85TiB, used=790.46GiB 
 whether this means 2.85TiB out of 10TiB is allocated, or if it's twice that 
 due to raid1. I can't ever remember this presentation detail, so again the 
 secret decode ring where the UI doesn't expressly tell us what's going on is 
 going to continue to be a source of confusion for users.


 Chris Murphy
 Oh, I misread the output

 That turns strange now
 BTW what's the output of 'df' command?

 Thanks,
 Qu

Re: Problem converting data raid0 to raid1: enospc errors during balance

2014-10-27 Thread Chris Murphy


On Oct 27, 2014, at 9:56 AM, Jasper Verberk jverb...@hotmail.com wrote:

 These are the results to a normal df:
 
 http://paste.debian.net/128932/
 
 The mountpoint is /data.

OK so this is with the new computation in kernel 3.17 (which I think contains a 
bug by counting free space twice); so now it shows available blocks based on 
the loss due to mirroring or parity. So 1k blocks 5860533168 = 5.45TiB. If you 
boot an older kernel my expectation is this shows up as 10.91TiB. In any case, 
df says there's 1.77TiB worth of data, so there should be plenty of space.

Somewhere there's a bug. Either the 'btrfs fi df' is insufficiently 
communicating whether the desired operation can be done, or there's actual 
kernel confusion on how much space is available to do the conversion.

I wonder what happens if you go back to kernel 3.16 and try do do the 
conversion?


Chris Murphy--
To unsubscribe from this list: send the line unsubscribe linux-btrfs in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: Problem converting data raid0 to raid1: enospc errors during balance

2014-10-26 Thread Qu Wenruo


Hi,

Although I'm not completely sure, but it seems that, you really ran out 
of space.


[1] Your array won't hold raid1 for 1.97T data
Your array used up 1.97T raid0 data, it takes 1.97T for raid0.
But if converted to 1.97T, it will occupy 1.97T X2 = 3.94T.
Your array are only 2.73T, it is too small to contain the data.

[2]No unallocated free space
When doing convert, btrfs will *NOT* convert the data/metadata in-place, 
it allocate chunk and move data,
which means, if btrfs wants to convert it needs to allocate some raid1 
data chunk for the convert,

and then copy the raid0 data to raid1 data chunk.

However, your 'btrfs fi show' output shows that there is no unallocated 
free space for raid1 data chunk allocation.

(2.75T space all used up for chunk allocation).
Although your 'btrfs fi df' shows there is still about 800G free space, 
that's just free space inside data/medata chunk,
and chunk allocation needs completely free space, which is not allocated 
for any chunk type.
If you really want to free some space from the unused space, you can try 
balance with -d/-m options,

but it won't really do much help on your case due to [1].

[Conculsion]
So the ENOSPC error is not a false alert.
Add more devices (easiest but expensive) or free up some space and then 
use balance with -d/-m
to free up some free space(balance speed maybe slow) seems to be the 
only 2 ways to finish your RAID1

setup.

Thanks,
Qu


 Original Message 
Subject: Problem converting data raid0 to raid1: enospc errors during 
balance

From: Jasper Verberk jverb...@hotmail.com
To: linux-btrfs@vger.kernel.org linux-btrfs@vger.kernel.org
Date: 2014年10月24日 21:32

Hello,

I'm trying to change my 4 disk btrfs data from raid0 to raid1. The metadata is 
allready in raid1 and now I'm trying to also make the data raid1.

I'm getting the following errors and I got told on irc to go report this as a 
bug.

Hoping there is anybody that can give me a hand here in solving it.


---
Command I issued was:

   btrfs balance start -dconvert=raid1,soft /data
---

---
Error I got:

   ERROR: error during balancing '/data' - No space left on device
   There may be more info in syslog - try dmesg | tail
---

---
Last lines of dmesg:

[60098.584459] BTRFS info (device sdc): relocating block group 18282971136 
flags 9
[60102.422310] BTRFS info (device sdc): relocating block group 5398069248 flags 
9
[60102.776406] BTRFS info (device sdc): relocating block group 1103101952 flags 
9
[60103.168554] BTRFS info (device sdc): relocating block group 4194304 flags 4
[60103.619552] BTRFS info (device sdc): relocating block group 0 flags 2
[60104.040776] BTRFS info (device sdc): 1046 enospc errors during balance
---

---
Further info:

root@BlackMesa:/mnt# btrfs fi show
Label: 'data'  uuid: a1e36575-0843-45ab-850e-618a5d8a4b7e
 Total devices 4 FS bytes used 2.75TiB
 devid1 size 2.73TiB used 1.94TiB path /dev/sdb
 devid2 size 2.73TiB used 1.94TiB path /dev/sdd
 devid3 size 2.73TiB used 1.94TiB path /dev/sdc
 devid4 size 2.73TiB used 1.94TiB path /dev/sde


root@BlackMesa:/mnt# btrfs fi df /data
Data, RAID1: total=2.85TiB, used=790.46GiB
Data, RAID0: total=2.04TiB, used=1.97TiB
System, RAID1: total=8.00MiB, used=612.00KiB
Metadata, RAID1: total=5.00GiB, used=3.39GiB
unknown, single: total=512.00MiB, used=0.00

root@BlackMesa:/mnt# uname -a
Linux BlackMesa 3.17-1-amd64 #1 SMP Debian 3.17-1~exp1 (2014-10-14) x86_64 
GNU/Linux

root@BlackMesa:/mnt# btrfs --version
Btrfs v3.14.1
---

---
And this is what happens when I try to make an image:

root@BlackMesa:/mnt# btrfs-image -c9 -t4 /dev/sdb /home/jasper/dump.image
btrfs-image: disk-io.c:155: readahead_tree_block: Assertion `!(ret)' failed.
Aborted
---   --
To unsubscribe from this list: send the line unsubscribe linux-btrfs in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html


--
To unsubscribe from this list: send the line unsubscribe linux-btrfs in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: Problem converting data raid0 to raid1: enospc errors during balance

2014-10-26 Thread Chris Murphy


On Oct 26, 2014, at 7:40 PM, Qu Wenruo quwen...@cn.fujitsu.com wrote:

 Hi,
 
 Although I'm not completely sure, but it seems that, you really ran out of 
 space.
 
 [1] Your array won't hold raid1 for 1.97T data
 Your array used up 1.97T raid0 data, it takes 1.97T for raid0.
 But if converted to 1.97T, it will occupy 1.97T X2 = 3.94T.
 Your array are only 2.73T, it is too small to contain the data.

I'm not understanding. The btrfs fi show, shows 4x 2.73TiB devices, so that 
seems like it's a 10+TiB array.

There's 2.04TiB raid0 data chunks, so roughly 500GiB per device, yet 1.94TiB is 
reported used per device by fi show. Confusing.

Also it's still very confusing: Data, RAID1: total=2.85TiB, used=790.46GiB 
whether this means 2.85TiB out of 10TiB is allocated, or if it's twice that due 
to raid1. I can't ever remember this presentation detail, so again the secret 
decode ring where the UI doesn't expressly tell us what's going on is going to 
continue to be a source of confusion for users.


Chris Murphy--
To unsubscribe from this list: send the line unsubscribe linux-btrfs in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: Problem converting data raid0 to raid1: enospc errors during balance

2014-10-26 Thread Qu Wenruo

 Original Message 
Subject: Re: Problem converting data raid0 to raid1: enospc errors 
during balance

From: Chris Murphy li...@colorremedies.com
To: Qu Wenruo quwen...@cn.fujitsu.com
Date: 2014年10月27日 12:40

On Oct 26, 2014, at 7:40 PM, Qu Wenruo quwen...@cn.fujitsu.com wrote:

Hi,

Although I'm not completely sure, but it seems that, you really ran out of 
space.

[1] Your array won't hold raid1 for 1.97T data
Your array used up 1.97T raid0 data, it takes 1.97T for raid0.
But if converted to 1.97T, it will occupy 1.97T X2 = 3.94T.
Your array are only 2.73T, it is too small to contain the data.

I'm not understanding. The btrfs fi show, shows 4x 2.73TiB devices, so that 
seems like it's a 10+TiB array.

There's 2.04TiB raid0 data chunks, so roughly 500GiB per device, yet 1.94TiB is 
reported used per device by fi show. Confusing.

Also it's still very confusing: Data, RAID1: total=2.85TiB, used=790.46GiB 
whether this means 2.85TiB out of 10TiB is allocated, or if it's twice that due 
to raid1. I can't ever remember this presentation detail, so again the secret 
decode ring where the UI doesn't expressly tell us what's going on is going to 
continue to be a source of confusion for users.

Chris Murphy

Oh, I misread the output

That turns strange now
BTW what's the output of 'df' command?

Thanks,
Qu

--
To unsubscribe from this list: send the line unsubscribe linux-btrfs in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Problem converting data raid0 to raid1: enospc errors during balance

2014-10-24 Thread Jasper Verberk

Hello,

I'm trying to change my 4 disk btrfs data from raid0 to raid1. The metadata is 
allready in raid1 and now I'm trying to also make the data raid1.

I'm getting the following errors and I got told on irc to go report this as a 
bug.

Hoping there is anybody that can give me a hand here in solving it.


---
Command I issued was:

  btrfs balance start -dconvert=raid1,soft /data
---

---
Error I got:

  ERROR: error during balancing '/data' - No space left on device
  There may be more info in syslog - try dmesg | tail
---

---
Last lines of dmesg:

[60098.584459] BTRFS info (device sdc): relocating block group 18282971136 
flags 9
[60102.422310] BTRFS info (device sdc): relocating block group 5398069248 flags 
9
[60102.776406] BTRFS info (device sdc): relocating block group 1103101952 flags 
9
[60103.168554] BTRFS info (device sdc): relocating block group 4194304 flags 4
[60103.619552] BTRFS info (device sdc): relocating block group 0 flags 2
[60104.040776] BTRFS info (device sdc): 1046 enospc errors during balance
---

---
Further info:

root@BlackMesa:/mnt# btrfs fi show
Label: 'data'  uuid: a1e36575-0843-45ab-850e-618a5d8a4b7e
        Total devices 4 FS bytes used 2.75TiB
        devid    1 size 2.73TiB used 1.94TiB path /dev/sdb
        devid    2 size 2.73TiB used 1.94TiB path /dev/sdd
        devid    3 size 2.73TiB used 1.94TiB path /dev/sdc
        devid    4 size 2.73TiB used 1.94TiB path /dev/sde


root@BlackMesa:/mnt# btrfs fi df /data
Data, RAID1: total=2.85TiB, used=790.46GiB
Data, RAID0: total=2.04TiB, used=1.97TiB
System, RAID1: total=8.00MiB, used=612.00KiB
Metadata, RAID1: total=5.00GiB, used=3.39GiB
unknown, single: total=512.00MiB, used=0.00

root@BlackMesa:/mnt# uname -a
Linux BlackMesa 3.17-1-amd64 #1 SMP Debian 3.17-1~exp1 (2014-10-14) x86_64 
GNU/Linux

root@BlackMesa:/mnt# btrfs --version
Btrfs v3.14.1
---

---
And this is what happens when I try to make an image:

root@BlackMesa:/mnt# btrfs-image -c9 -t4 /dev/sdb /home/jasper/dump.image
btrfs-image: disk-io.c:155: readahead_tree_block: Assertion `!(ret)' failed.
Aborted
---   --
To unsubscribe from this list: send the line unsubscribe linux-btrfs in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: Problem converting data raid0 to raid1: enospc errors during balance

2014-10-24 Thread Chris Murphy


On Oct 24, 2014, at 7:32 AM, Jasper Verberk jverb...@hotmail.com wrote:
 
 [60104.040776] BTRFS info (device sdc): 1046 enospc errors during balance


To get more information, remount with enospc_debug option, and try to convert 
again.

 root@BlackMesa:/mnt# btrfs --version
 Btrfs v3.14.1

There are many fixes since this version, I suggest going to 3.17 which also has 
a lot of new fsck code in case that ends up being needed. You can try 
btrfs-image again. Then see if btrfs check (without --repair) shows anything 
suspicious.


Chris Murphy

--
To unsubscribe from this list: send the line unsubscribe linux-btrfs in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

RE: Problem converting data raid0 to raid1: enospc errors during balance

2014-10-24 Thread Jasper Verberk

Did that, you can find the result in the pastebin.

http://paste.debian.net/128552/

 Subject: Re: Problem converting data raid0 to raid1: enospc errors during 
 balance
 From: li...@colorremedies.com
 Date: Fri, 24 Oct 2014 11:11:21 -0600
 To: linux-btrfs@vger.kernel.org; jverb...@hotmail.com

 On Oct 24, 2014, at 7:32 AM, Jasper Verberk jverb...@hotmail.com wrote:

 [60104.040776] BTRFS info (device sdc): 1046 enospc errors during balance

 To get more information, remount with enospc_debug option, and try to convert 
 again.

 root@BlackMesa:/mnt# btrfs --version
 Btrfs v3.14.1

 There are many fixes since this version, I suggest going to 3.17 which also 
 has a lot of new fsck code in case that ends up being needed. You can try 
 btrfs-image again. Then see if btrfs check (without --repair) shows anything 
 suspicious.

 Chris Murphy

  --
To unsubscribe from this list: send the line unsubscribe linux-btrfs in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: Problem converting data raid0 to raid1: enospc errors during balance

2014-10-24 Thread Chris Murphy


On Oct 24, 2014, at 11:49 AM, Jasper Verberk jverb...@hotmail.com wrote:

 Did that, you can find the result in the pastebin.
 
 http://paste.debian.net/128552/

Maybe Josef or David will have some idea.

I'd still take a btrfs-image and btrfs check with current btrfs-progs.

Chris Murphy

--
To unsubscribe from this list: send the line unsubscribe linux-btrfs in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: ENOSPC errors during balance

2014-07-22 Thread Marc Joliet

Am Tue, 22 Jul 2014 03:26:39 + (UTC)
schrieb Duncan 1i5t5.dun...@cox.net:

 Marc Joliet posted on Tue, 22 Jul 2014 01:30:22 +0200 as excerpted:
 
  And now that the background deletion of the old snapshots is done, the file
  system ended up at:
  
  # btrfs filesystem df /run/media/marcec/MARCEC_BACKUP
  Data, single: total=219.00GiB, used=140.13GiB
  System, DUP: total=32.00MiB, used=36.00KiB
  Metadata, DUP: total=4.50GiB, used=2.40GiB
  unknown, single: total=512.00MiB, used=0.00
  
  I don't know how reliable du is for this, but I used it to estimate how much
  used data I should expect, and I get 138 GiB.  That means that the snapshots
  yield about 2 GiB overhead, which is very reasonable, I think.  Obviously
  I'll be starting a full balance now.
 
[snip total/used discussion]

No, you misunderstand: read my email three steps above yours (from the 21. at
15:22).  I am wondering about why the disk usage ballooned to 200 GiB in the
first place.

-- 
Marc Joliet
--
People who think they know everything really annoy those of us who know we
don't - Bjarne Stroustrup


signature.asc
Description: PGP signature

Re: ENOSPC errors during balance

2014-07-21 Thread Brendan Hide


On 20/07/14 14:59, Duncan wrote:

Marc Joliet posted on Sun, 20 Jul 2014 12:22:33 +0200 as excerpted:


On the other hand, the wiki [0] says that defragmentation (and
balancing) is optional, and the only reason stated for doing either is
because they will have impact on performance.

Yes.  That's what threw off the other guy as well.  He decided to skip it
for the same reason.

If I had a wiki account I'd change it, but for whatever reason I tend to
be far more comfortable writing list replies, sometimes repeatedly, than
writing anything on the web, which I tend to treat as read-only.  So I've
never gotten a wiki account and thus haven't changed it, and apparently
the other guy with the problem and anyone else that knows hasn't changed
it either, so the conversion page still continues to underemphasize the
importance of completing the conversion steps, including the defrag, in
proper order.

I've inserted information specific to this in the wiki. Others with wiki 
accounts, feel free to review:

https://btrfs.wiki.kernel.org/index.php/Conversion_from_Ext3#Before_first_use

--
__
Brendan Hide
http://swiftspirit.co.za/
http://www.webafrica.co.za/?AFF1E97

--
To unsubscribe from this list: send the line unsubscribe linux-btrfs in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: ENOSPC errors during balance

2014-07-21 Thread Marc Joliet

Am Sun, 20 Jul 2014 21:44:40 +0200
schrieb Marc Joliet mar...@gmx.de:

[...]
 What I did:
 
 - delete the single largest file on the file system, a 12 GB VM image, along
   with all subvolumes that contained it
 - rsync it over again
[...]

I want to point out at this point, though, that doing those two steps freed a
disproportionate amount of space.  The image file is only 12 GB, and it hadn't
changed in any of the snapshots (I haven't used this VM since June), so that
subvolume delete -c snapshots returned after a few seconds. Yet deleting it
seems to have freed up twice as much. You can see this from the filesystem df
output: before, used was at 229.04 GiB, and after deleting it and copying it
back (and after a day's worth of backups) went down to 218 GiB.

Does anyone have any idea how this happened?

Actually, now I remember something that is probably related: when I first
moved to my current backup scheme last week, I first copied the data from the
last rsnapshot based backup with cp --reflink to the new backup location, but
forgot to use -a.  I interrupted it and ran cp -a -u --reflink, but it had
already copied a lot, and I was too impatient to start over; after all, the
data hadn't changed.  Then, when rsync (with --inplace) ran for the first time,
all of these files with wrong permissions and different time stamps were copied
over, but for some reason, the space used increased *greatly*; *much* more than
I would expect from changed metadata.

The total size of the file system data should be around 142 GB (+ snapshots),
but, well, it's more than 1.5 times as much.

Perhaps cp --reflink treats hard links differently than expected?  I would have
expected the data pointed to by the hard link to have been referenced, but
maybe something else happened?

-- 
Marc Joliet
--
People who think they know everything really annoy those of us who know we
don't - Bjarne Stroustrup


signature.asc
Description: PGP signature

Re: ENOSPC errors during balance

2014-07-21 Thread Marc Joliet

Am Mon, 21 Jul 2014 15:22:16 +0200
schrieb Marc Joliet mar...@gmx.de:

 Am Sun, 20 Jul 2014 21:44:40 +0200
 schrieb Marc Joliet mar...@gmx.de:
 
 [...]
  What I did:
  
  - delete the single largest file on the file system, a 12 GB VM image, along
with all subvolumes that contained it
  - rsync it over again
 [...]
 
 I want to point out at this point, though, that doing those two steps freed a
 disproportionate amount of space.  The image file is only 12 GB, and it hadn't
 changed in any of the snapshots (I haven't used this VM since June), so that
 subvolume delete -c snapshots returned after a few seconds. Yet deleting 
 it
 seems to have freed up twice as much. You can see this from the filesystem 
 df
 output: before, used was at 229.04 GiB, and after deleting it and copying it
 back (and after a day's worth of backups) went down to 218 GiB.
 
 Does anyone have any idea how this happened?
 
 Actually, now I remember something that is probably related: when I first
 moved to my current backup scheme last week, I first copied the data from the
 last rsnapshot based backup with cp --reflink to the new backup location, 
 but
 forgot to use -a.  I interrupted it and ran cp -a -u --reflink, but it had
 already copied a lot, and I was too impatient to start over; after all, the
 data hadn't changed.  Then, when rsync (with --inplace) ran for the first 
 time,
 all of these files with wrong permissions and different time stamps were 
 copied
 over, but for some reason, the space used increased *greatly*; *much* more 
 than
 I would expect from changed metadata.
 
 The total size of the file system data should be around 142 GB (+ snapshots),
 but, well, it's more than 1.5 times as much.
 
 Perhaps cp --reflink treats hard links differently than expected?  I would 
 have
 expected the data pointed to by the hard link to have been referenced, but
 maybe something else happened?

Hah, OK, apparently when my daily backup removed the oldest daily snapshot, it
freed up whatever was taking up so much space, so as of now the file system
uses only 169.14 GiB (from 218).  Weird.

-- 
Marc Joliet
--
People who think they know everything really annoy those of us who know we
don't - Bjarne Stroustrup


signature.asc
Description: PGP signature

Re: ENOSPC errors during balance

2014-07-21 Thread Marc Joliet

Am Tue, 22 Jul 2014 00:30:57 +0200
schrieb Marc Joliet mar...@gmx.de:

 Am Mon, 21 Jul 2014 15:22:16 +0200
 schrieb Marc Joliet mar...@gmx.de:
 
  Am Sun, 20 Jul 2014 21:44:40 +0200
  schrieb Marc Joliet mar...@gmx.de:
  
  [...]
   What I did:
   
   - delete the single largest file on the file system, a 12 GB VM image, 
   along
 with all subvolumes that contained it
   - rsync it over again
  [...]
  
  I want to point out at this point, though, that doing those two steps freed 
  a
  disproportionate amount of space.  The image file is only 12 GB, and it 
  hadn't
  changed in any of the snapshots (I haven't used this VM since June), so that
  subvolume delete -c snapshots returned after a few seconds. Yet 
  deleting it
  seems to have freed up twice as much. You can see this from the filesystem 
  df
  output: before, used was at 229.04 GiB, and after deleting it and copying 
  it
  back (and after a day's worth of backups) went down to 218 GiB.
  
  Does anyone have any idea how this happened?
  
  Actually, now I remember something that is probably related: when I first
  moved to my current backup scheme last week, I first copied the data from 
  the
  last rsnapshot based backup with cp --reflink to the new backup location, 
  but
  forgot to use -a.  I interrupted it and ran cp -a -u --reflink, but it 
  had
  already copied a lot, and I was too impatient to start over; after all, the
  data hadn't changed.  Then, when rsync (with --inplace) ran for the first 
  time,
  all of these files with wrong permissions and different time stamps were 
  copied
  over, but for some reason, the space used increased *greatly*; *much* more 
  than
  I would expect from changed metadata.
  
  The total size of the file system data should be around 142 GB (+ 
  snapshots),
  but, well, it's more than 1.5 times as much.
  
  Perhaps cp --reflink treats hard links differently than expected?  I would 
  have
  expected the data pointed to by the hard link to have been referenced, but
  maybe something else happened?
 
 Hah, OK, apparently when my daily backup removed the oldest daily snapshot, it
 freed up whatever was taking up so much space, so as of now the file system
 uses only 169.14 GiB (from 218).  Weird.

And now that the background deletion of the old snapshots is done, the file
system ended up at:

# btrfs filesystem df /run/media/marcec/MARCEC_BACKUP
Data, single: total=219.00GiB, used=140.13GiB
System, DUP: total=32.00MiB, used=36.00KiB
Metadata, DUP: total=4.50GiB, used=2.40GiB
unknown, single: total=512.00MiB, used=0.00

I don't know how reliable du is for this, but I used it to estimate how much
used data I should expect, and I get 138 GiB.  That means that the snapshots
yield about 2 GiB overhead, which is very reasonable, I think.  Obviously
I'll be starting a full balance now.

I still think this whole... thing is very odd, hopefully somebody can shed
some light on it for me (maybe it's obvious, but I don't see it).

-- 
Marc Joliet
--
People who think they know everything really annoy those of us who know we
don't - Bjarne Stroustrup


signature.asc
Description: PGP signature

Re: ENOSPC errors during balance

2014-07-21 Thread Duncan

Marc Joliet posted on Tue, 22 Jul 2014 01:30:22 +0200 as excerpted:

 And now that the background deletion of the old snapshots is done, the file
 system ended up at:
 
 # btrfs filesystem df /run/media/marcec/MARCEC_BACKUP
 Data, single: total=219.00GiB, used=140.13GiB
 System, DUP: total=32.00MiB, used=36.00KiB
 Metadata, DUP: total=4.50GiB, used=2.40GiB
 unknown, single: total=512.00MiB, used=0.00
 
 I don't know how reliable du is for this, but I used it to estimate how much
 used data I should expect, and I get 138 GiB.  That means that the snapshots
 yield about 2 GiB overhead, which is very reasonable, I think.  Obviously
 I'll be starting a full balance now.

FWIW, the balance should reduce the data total quite a bit, to 141-ish GiB
(might be 142 or 145, but it should definitely come down from 219 GiB),
because the spread between total and used is relatively high, now, and balance
is what's used to bring that back down.

Metadata total will probably come down a bit as well, to 3.00 GiB or so.

What's going on there is this:  Btrfs allocates and deallocates data and
metadata in two stages.  First it allocates chunks, 1 GiB in size for
data, 256 MiB in size for metadata, but because metadata is dup by default
it allocates two chunks so half a GiB at a time, there.  Then the actual
file data and metadata can be written into the pre-allocated chunks, filling
them up.  As they near full, more chunks will be allocated from the
unallocated pool as necessary.

But on file deletion, btrfs only automatically handles the file
data/metadata level; it doesn't (yet) automatically deallocate the chunks,
nor can it change the allocation from say a data chunk to a metadata chunk.
So when a chunk is allocated, it stays allocated.

That's the spread you see in btrfs filesystem df, between total and used,
for each chunk type.

The way to recover those allocated but unused chunks to the unallocated
pool, so they can be reallocated between data and metadata as necessary,
is with a balance.  That balance, therefore, should reduce the spread
seen in the above between total and used.

Meanwhile, btrfs filesystem df shows the spread between allocated and
used for each type, but what about unallocated?  Simple.  Btrfs
filesystem show lists total filesystem size as well as allocated
usage for each device.  (The total line is something else, I recommend
ignoring it as it's simply confusing.  Only pay attention to the
individual device lines.)

Thus, to get a proper picture of the space usage status on a btrfs
filesystem, you must have both the btrfs filesystem show and
btrfs filesystem df output for that filesystem, show to tell
you how much of the total space is chunk-allocated for each device,
df to tell you what those allocations are, and how much of the
chunk-allocated space is actually used, for each allocation type.

It's wise to keep track of the show output in particular, and
when the spread between used (allocated) and total for each
device gets low, under a few GiB, check btrfs fi df and see
what's using that space unnecessarily and then do a balance
to recover it, if possible.


-- 
Duncan - List replies preferred.   No HTML msgs.
Every nonfree program has a lord, a master --
and if you use the program, he is your master.  Richard Stallman

--
To unsubscribe from this list: send the line unsubscribe linux-btrfs in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: ENOSPC errors during balance

2014-07-20 Thread Marc Joliet

Am Sat, 19 Jul 2014 19:11:00 -0600
schrieb Chris Murphy li...@colorremedies.com:

 I'm seeing this also in the 2nd dmesg:
 
 [  249.893310] BTRFS error (device sdg2): free space inode generation (0) did 
 not match free space cache generation (26286)
 
 
 So you could try umounting the volume. And doing a one time mount with the 
 clear_cache mount option. Give it some time to rebuild the space cache.
 
 After that you could umount again, and mount with enospc_debug and try to 
 reproduce the enospc with another balance and see if dmesg contains more 
 information this time.

OK, I did that, and the new dmesg is attached. Also, some outputs again, first
filesystem df (that total surge at the end sure is consistent):

# btrfs filesystem df /mnt   
Data, single: total=237.00GiB, used=229.67GiB
System, DUP: total=32.00MiB, used=36.00KiB
Metadata, DUP: total=4.50GiB, used=3.49GiB
unknown, single: total=512.00MiB, used=0.00

And here what I described in my initial post, the output of balance status
immediately after the error (turns out my memory was correct):

btrfs filesystem balance status /mnt
Balance on '/mnt' is running
0 out of about 0 chunks balanced (0 considered), -nan% left

(Also, this is with Gentoo kernel 3.15.6 now.)

-- 
Marc Joliet
--
People who think they know everything really annoy those of us who know we
don't - Bjarne Stroustrup


dmesg4.log.xz
Description: application/xz


signature.asc
Description: PGP signature

Re: ENOSPC errors during balance

2014-07-20 Thread Marc Joliet

Am Sat, 19 Jul 2014 18:53:03 -0600
schrieb Chris Murphy li...@colorremedies.com:

 
 On Jul 19, 2014, at 2:58 PM, Marc Joliet mar...@gmx.de wrote:
 
  Am Sat, 19 Jul 2014 22:10:51 +0200
  schrieb Marc Joliet mar...@gmx.de:
  
  [...]
  Another random idea:  the number of errors decreased the second time I ran
  balance (from 4 to 2), I could run another full balance and see if it keeps
  decreasing.
  
  Well, this time there were still 2 ENOSPC errors.  But I can show the df 
  output
  after such an ENOSPC error, to illustrate what I meant with the sudden surge
  in total usage:
  
  # btrfs filesystem df /run/media/marcec/MARCEC_BACKUP 
  Data, single: total=236.00GiB, used=229.04GiB
  System, DUP: total=32.00MiB, used=36.00KiB
  Metadata, DUP: total=4.00GiB, used=3.20GiB
  unknown, single: total=512.00MiB, used=0.00
  
  And then after running a balance and (almost) immediately cancelling:
  
  # btrfs filesystem df /run/media/marcec/MARCEC_BACKUP 
  Data, single: total=230.00GiB, used=229.04GiB
  System, DUP: total=32.00MiB, used=36.00KiB
  Metadata, DUP: total=4.00GiB, used=3.20GiB
  unknown, single: total=512.00MiB, used=0.00
 
 I think it's a bit weird. Two options: a. Keep using the file system, with 
 judicious backups, if a dev wants more info they'll reply to the thread; b. 
 Migrate the data to a new file system, first capture the file system with 
 btrfs-image in case a dev wants more info and you've since blown away the 
 filesystem, and then move it to a new btrfs fs. I'd use send/receive for this 
 to preserve subvolumes and snapshots.

OK, I'll keep that in mind.  I'll keep running the file system for now, just in
case it's a run-time error (i.e., a bug in the balance code, and not a problem
with the file system itself).  If it gets trashed on its own, or I move to a new
file system, I'll be sure to follow the steps you outlined.

 Chris Murphy

Thanks
-- 
Marc Joliet
--
People who think they know everything really annoy those of us who know we
don't - Bjarne Stroustrup


signature.asc
Description: PGP signature

Re: ENOSPC errors during balance

2014-07-20 Thread Marc Joliet

Am Sun, 20 Jul 2014 02:39:27 + (UTC)
schrieb Duncan 1i5t5.dun...@cox.net:

 Chris Murphy posted on Sat, 19 Jul 2014 11:38:08 -0600 as excerpted:
 
  I'm not sure of the reason for the BTRFS info (device sdg2): 2 enospc
  errors during balance but it seems informational rather than either a
  warning or problem. I'd treat ext4-btrfs converted file systems to be
  something of an odd duck, in that it's uncommon, therefore isn't getting
  as much testing and extra caution is a good idea. Make frequent backups.
 
 Expanding on that a bit...
 
 Balance simply rewrites chunks, combining where possible and possibly 
 converting to a different layout (single/dup/raid0/1/10/5/6[1]) in the 
 process.  The most common reason for enospc during balance is of course 
 all space allocated to chunks, with various workarounds for that if it 
 happens, but that doesn't seem to be what was happening to you
 (Mark J./OP).
 
 Based on very similar issues reported by another ext4 - btrfs converter 
 and the discussion on that thread, here's what I think happened:
 
 First a critical question for you as it's a critical piece of this 
 scenario that you didn't mention in your summary.  The wiki page on
 ext4 - btrfs conversion suggests deleting the ext2_saved subvolume and 
 then doing a full defrag and rebalance.  You're attempting a full 
 rebalance, but have you yet deleted ext2_saved and did you do the defrag 
 before attempting the rebalance?
 
 I'm guessing not, as was the case with the other user that reported this 
 issue.  Here's what apparently happened in his case and how we fixed it:

Ah, I actually did, in fact.  I only implicitly said it, though.  Here's what I
wrote:

After converting the backup partition about a week ago, following the wiki
entry on ext4 conversion, I eventually ran a full balance [...]

The wiki says to run a full balance (and defragment before that, but that was
slw, so I didn't do it), *after* deleting the ext4 file system image.
So the full balance was right after doing that :) .

 The problem is that btrfs data chunks are 1 GiB each.  Thus, the maximum 
 size of a btrfs extent is 1 GiB.  But ext4 doesn't have an arbitrary 
 limitation on extent size, and for files over a GiB in size, ext4 extents 
 can /also/ be over a GiB in size.
 
 That results in two potential issues at balance time.  First, btrfs 
 treats the ext2_saved subvolume as a read-only snapshot and won't touch 
 it, thus keeping the ext* data intact in case the user wishes to rollback 
 to ext*.  I don't think btrfs touches that data during a balance either, 
 as it really couldn't do so /safely/ without incorporating all of the 
 ext* code into btrfs.  I'm not sure how it expresses that situation, but 
 it's quite possible that btrfs treats it as enospc.
 
 Second, for files that had ext4 extents greater than a GiB, balance will 
 naturally enospc, because even the biggest possible btrfs extent, a full 
 1 GiB data chunk, is too small to hold the existing file extent.  Of 
 course this only happens on filesystems converted from ext*, because 
 natively btrfs has no way to make an extent larger than a GiB, so it 
 won't run into the problem if it was created natively instead of 
 converted from ext*.
 
 Once the ext2_saved subvolume/snapshot is deleted, defragging should cure 
 the problem as it rewrites those files to btrfs-native chunks, normally 
 defragging but in this case fragging to the 1 GiB btrfs-native data-chunk-
 size extent size.

Hmm, well, I didn't defragment because it would have taken *forever* to go
through all those hardlinks, plus my experience is that ext* doesn't fragment
much at all, so I skipped that step.  But I certainly have files over 1GB in
size.

On the other hand, the wiki [0] says that defragmentation (and balancing) is
optional, and the only reason stated for doing either is because they will have
impact on performance.

 Alternatively, and this is what the other guy did, one can find all the 
 files from the original ext*fs over a GiB in size, and move them off-
 filesystem and back AFAIK he had several gigs of spare RAM and no files 
 larger than that, so he used tmpfs as the temporary storage location, 
 which is memory so the only I/O is that on the btrfs in question.  By 
 doing that he deleted the existing files on btrfs and recreated them, 
 naturally splitting the extents on data-chunk-boundaries as btrfs 
 normally does, in the recreation.
 
 If you had deleted the ext2_saved subvolume/snapshot and done the defrag 
 already, that explanation doesn't work as-is, but I'd still consider it 
 an artifact from the conversion, and try the alternative move-off-
 filesystem-temporarily method.

I'll try this and see, but I think I have more files 1GB than would account
for this error (which comes towards the end of the balance when only a few
chunks are left).  I'll see what find /mnt -type f -size +1G finds :) .

 If you don't have any files over a GiB in size, then I don't know

Re: ENOSPC errors during balance

2014-07-20 Thread Marc Joliet

Am Sun, 20 Jul 2014 12:22:33 +0200
schrieb Marc Joliet mar...@gmx.de:

[...]
 I'll try this and see, but I think I have more files 1GB than would account
 for this error (which comes towards the end of the balance when only a few
 chunks are left).  I'll see what find /mnt -type f -size +1G finds :) .

Now that I think about it, though, it sounds like it could explain the sudden
surge in total data size: for one very big file, several chunks/extents are
created, but the data cannot be copied from the original ext4 extent.

So far, the above find command has only found a few handful of files (plus all
the reflinks in the snapshots), much to my surprise. It still has one subvolume
to go through, though.

And just for completeness, that same find command didn't find any files on /,
which I also converted from ext4, and for which a full balance completed
successfully.  So maybe this is in the right direction, but I'll wait and see
what Chris Murphy (or anyone else) might find in my latest dmesg output.

-- 
Marc Joliet
--
People who think they know everything really annoy those of us who know we
don't - Bjarne Stroustrup


signature.asc
Description: PGP signature

Re: ENOSPC errors during balance

2014-07-20 Thread Duncan

Marc Joliet posted on Sun, 20 Jul 2014 12:22:33 +0200 as excerpted:

 On the other hand, the wiki [0] says that defragmentation (and
 balancing) is optional, and the only reason stated for doing either is
 because they will have impact on performance.

Yes.  That's what threw off the other guy as well.  He decided to skip it 
for the same reason.

If I had a wiki account I'd change it, but for whatever reason I tend to 
be far more comfortable writing list replies, sometimes repeatedly, than 
writing anything on the web, which I tend to treat as read-only.  So I've 
never gotten a wiki account and thus haven't changed it, and apparently 
the other guy with the problem and anyone else that knows hasn't changed 
it either, so the conversion page still continues to underemphasize the 
importance of completing the conversion steps, including the defrag, in 
proper order.

-- 
Duncan - List replies preferred.   No HTML msgs.
Every nonfree program has a lord, a master --
and if you use the program, he is your master.  Richard Stallman

--
To unsubscribe from this list: send the line unsubscribe linux-btrfs in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: ENOSPC errors during balance

2014-07-20 Thread Duncan

Marc Joliet posted on Sun, 20 Jul 2014 21:44:40 +0200 as excerpted:

 Am Sun, 20 Jul 2014 13:40:54 +0200 schrieb Marc Joliet mar...@gmx.de:
 
 Am Sun, 20 Jul 2014 12:22:33 +0200 schrieb Marc Joliet mar...@gmx.de:
 
 [...]
  I'll try this and see, but I think I have more files 1GB than would
  account for this error (which comes towards the end of the balance
  when only a few chunks are left).  I'll see what find /mnt -type f
  -size +1G finds :) .

Note that it's extent's over 1 GiB on the converted former ext4, not 
necessarily files over 1 GiB.  You may have files over a GiB that were 
already broken into extents that are all less than a GiB, and btrfs would 
be able to deal with them fine.  It's only when a single extent ended up 
larger than a GiB on the former ext4 that btrfs can't deal with it.

 Now that I think about it, though, it sounds like it could explain the
 sudden surge in total data size: for one very big file, several
 chunks/extents are created, but the data cannot be copied from the
 original ext4 extent.

I hadn't thought about that effect, but good deductive reasoning. =:^)

 Well, turns out that was it!
 
 What I did:
 
 - delete the single largest file on the file system, a 12 GB VM image,
 along with all subvolumes that contained it
 - rsync it over again - start a full balance
 
 This time, the balance finished successfully :-) .

Good to read!

We're now two for two on this technique working around this problem! =:^)

-- 
Duncan - List replies preferred.   No HTML msgs.
Every nonfree program has a lord, a master --
and if you use the program, he is your master.  Richard Stallman

--
To unsubscribe from this list: send the line unsubscribe linux-btrfs in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: ENOSPC errors during balance

2014-07-19 Thread Chris Murphy

The 2nd dmesg (didn't look at the 1st), has many instances like this;

[96241.882138] ata2.00: exception Emask 0x1 SAct 0x7ffe0fff SErr 0x0 action 0x6 
frozen
[96241.882139] ata2.00: Ata error. fis:0x21
[96241.882142] ata2.00: failed command: READ FPDMA QUEUED
[96241.882148] ata2.00: cmd 60/08:00:68:0a:2d/00:00:18:00:00/40 tag 0 ncq 4096 
in
 res 41/00:58:40:5c:2c/00:00:18:00:00/40 Emask 0x1 (device error)

I'm not sure what this error is, it acts like an unrecoverable read error but 
I'm not seeing UNC reported. It looks like ata 2.00 is sdb, which is a member 
of a btrfs raid10 volume. So this isn't related to your sdg2 and enospc error, 
it's a different problem.


I'm not sure of the reason for the BTRFS info (device sdg2): 2 enospc errors 
during balance but it seems informational rather than either a warning or 
problem. I'd treat ext4-btrfs converted file systems to be something of an odd 
duck, in that it's uncommon, therefore isn't getting as much testing and extra 
caution is a good idea. Make frequent backups.

Chris Murphy--
To unsubscribe from this list: send the line unsubscribe linux-btrfs in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Fw: ENOSPC errors during balance

2014-07-19 Thread Marc Joliet



Start weitergeleitete Nachricht:
Huh, turns out the Reply-To was to Chris Murphy, so here it is again for the
whole list.

Datum: Sat, 19 Jul 2014 20:34:34 +0200
Von: Marc Joliet mar...@gmx.de
An: Chris Murphy li...@colorremedies.com
Betreff: Re: ENOSPC errors during balance


Am Sat, 19 Jul 2014 11:38:08 -0600
schrieb Chris Murphy li...@colorremedies.com:

 The 2nd dmesg (didn't look at the 1st), has many instances like this;
 
 [96241.882138] ata2.00: exception Emask 0x1 SAct 0x7ffe0fff SErr 0x0 action 
 0x6 frozen
 [96241.882139] ata2.00: Ata error. fis:0x21
 [96241.882142] ata2.00: failed command: READ FPDMA QUEUED
 [96241.882148] ata2.00: cmd 60/08:00:68:0a:2d/00:00:18:00:00/40 tag 0 ncq 
 4096 in
  res 41/00:58:40:5c:2c/00:00:18:00:00/40 Emask 0x1 (device error)
 
 I'm not sure what this error is, it acts like an unrecoverable read error but 
 I'm not seeing UNC reported. It looks like ata 2.00 is sdb, which is a member 
 of a btrfs raid10 volume. So this isn't related to your sdg2 and enospc 
 error, it's a different problem.

Yeah, from what I remember reading it's related to nforce2 chipsets, but I
never pursued it, since I never really noticed any consequences (this is an old
computer that I originally build in 2006).  IIRC one workaround is to switch to
1.5gpbs instead of 3gbps (but then, it already is at 1.5 Gbps, but none of the
other ports are?  Might be the hard drive, I *think* it's older than the
others.), another is related to irqbalance (which I forgot about, I've just
switched it off and will see if the messages stop, but then again, my first
dmesg didn't have any of those messages).

Anyway, yes, it's unrelated to my problem :-) .

 I'm not sure of the reason for the BTRFS info (device sdg2): 2 enospc errors 
 during balance but it seems informational rather than either a warning or 
 problem. I'd treat ext4-btrfs converted file systems to be something of an 
 odd duck, in that it's uncommon, therefore isn't getting as much testing and 
 extra caution is a good idea. Make frequent backups.

Well, I *could* just recreate the file system.  Since these are my only backups
(no offsite backup as of yet), I wanted to keep the existing ones.  So
btrfs-convert was a convenient way to upgrade.

But since I ended up deleting those backups anyway, I would only be losing my
hourly and a few daily backups.  But it's not as if the file system is otherwise
misbehaving.

Another random idea:  the number of errors decreased the second time I ran
balance (from 4 to 2), I could run another full balance and see if it keeps
decreasing.

-- 
Marc Joliet
--
People who think they know everything really annoy those of us who know we
don't - Bjarne Stroustrup


-- 
Marc Joliet
--
People who think they know everything really annoy those of us who know we
don't - Bjarne Stroustrup


signature.asc
Description: PGP signature

Re: ENOSPC errors during balance

2014-07-19 Thread Marc Joliet

Am Sat, 19 Jul 2014 22:10:51 +0200
schrieb Marc Joliet mar...@gmx.de:

[...]
 Another random idea:  the number of errors decreased the second time I ran
 balance (from 4 to 2), I could run another full balance and see if it keeps
 decreasing.

Well, this time there were still 2 ENOSPC errors.  But I can show the df output
after such an ENOSPC error, to illustrate what I meant with the sudden surge
in total usage:

# btrfs filesystem df /run/media/marcec/MARCEC_BACKUP 
Data, single: total=236.00GiB, used=229.04GiB
System, DUP: total=32.00MiB, used=36.00KiB
Metadata, DUP: total=4.00GiB, used=3.20GiB
unknown, single: total=512.00MiB, used=0.00

And then after running a balance and (almost) immediately cancelling:

# btrfs filesystem df /run/media/marcec/MARCEC_BACKUP 
Data, single: total=230.00GiB, used=229.04GiB
System, DUP: total=32.00MiB, used=36.00KiB
Metadata, DUP: total=4.00GiB, used=3.20GiB
unknown, single: total=512.00MiB, used=0.00

-- 
Marc Joliet
--
People who think they know everything really annoy those of us who know we
don't - Bjarne Stroustrup


signature.asc
Description: PGP signature

Re: ENOSPC errors during balance

2014-07-19 Thread Piotr Szymaniak

On Sat, Jul 19, 2014 at 11:38:08AM -0600, Chris Murphy wrote:
 [96241.882138] ata2.00: exception Emask 0x1 SAct 0x7ffe0fff SErr 0x0 action 
 0x6 frozen
 [96241.882139] ata2.00: Ata error. fis:0x21
 [96241.882142] ata2.00: failed command: READ FPDMA QUEUED
 [96241.882148] ata2.00: cmd 60/08:00:68:0a:2d/00:00:18:00:00/40 tag 0 ncq 
 4096 in
  res 41/00:58:40:5c:2c/00:00:18:00:00/40 Emask 0x1 (device error)

Afair those are somehow related to NCQ.


Piotr Szymaniak.
-- 
Komnata  audiencyjna  zlotego  Bruna  przerastala  wszystko,  co  dotad
widzialem.  Musial  zatrudnic  dziesiatki  programistow i kreatorow, by
stworzyc tak wyuzdane i wysmakowane  wnetrze.  Dzwieki, barwy, ksztalty
i zapachy wywolywaly erekcje.
  -- Marcin Przybylek, Gamedec: Syndrom Adelheima


signature.asc
Description: Digital signature

Re: ENOSPC errors during balance

2014-07-19 Thread Chris Murphy


On Jul 19, 2014, at 2:58 PM, Marc Joliet mar...@gmx.de wrote:

 Am Sat, 19 Jul 2014 22:10:51 +0200
 schrieb Marc Joliet mar...@gmx.de:
 
 [...]
 Another random idea:  the number of errors decreased the second time I ran
 balance (from 4 to 2), I could run another full balance and see if it keeps
 decreasing.
 
 Well, this time there were still 2 ENOSPC errors.  But I can show the df 
 output
 after such an ENOSPC error, to illustrate what I meant with the sudden surge
 in total usage:
 
 # btrfs filesystem df /run/media/marcec/MARCEC_BACKUP 
 Data, single: total=236.00GiB, used=229.04GiB
 System, DUP: total=32.00MiB, used=36.00KiB
 Metadata, DUP: total=4.00GiB, used=3.20GiB
 unknown, single: total=512.00MiB, used=0.00
 
 And then after running a balance and (almost) immediately cancelling:
 
 # btrfs filesystem df /run/media/marcec/MARCEC_BACKUP 
 Data, single: total=230.00GiB, used=229.04GiB
 System, DUP: total=32.00MiB, used=36.00KiB
 Metadata, DUP: total=4.00GiB, used=3.20GiB
 unknown, single: total=512.00MiB, used=0.00

I think it's a bit weird. Two options: a. Keep using the file system, with 
judicious backups, if a dev wants more info they'll reply to the thread; b. 
Migrate the data to a new file system, first capture the file system with 
btrfs-image in case a dev wants more info and you've since blown away the 
filesystem, and then move it to a new btrfs fs. I'd use send/receive for this 
to preserve subvolumes and snapshots.


Chris Murphy--
To unsubscribe from this list: send the line unsubscribe linux-btrfs in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: ENOSPC errors during balance

2014-07-19 Thread Duncan

Chris Murphy posted on Sat, 19 Jul 2014 11:38:08 -0600 as excerpted:

 I'm not sure of the reason for the BTRFS info (device sdg2): 2 enospc
 errors during balance but it seems informational rather than either a
 warning or problem. I'd treat ext4-btrfs converted file systems to be
 something of an odd duck, in that it's uncommon, therefore isn't getting
 as much testing and extra caution is a good idea. Make frequent backups.

Expanding on that a bit...

Balance simply rewrites chunks, combining where possible and possibly 
converting to a different layout (single/dup/raid0/1/10/5/6[1]) in the 
process.  The most common reason for enospc during balance is of course 
all space allocated to chunks, with various workarounds for that if it 
happens, but that doesn't seem to be what was happening to you
(Mark J./OP).

Based on very similar issues reported by another ext4 - btrfs converter 
and the discussion on that thread, here's what I think happened:

First a critical question for you as it's a critical piece of this 
scenario that you didn't mention in your summary.  The wiki page on
ext4 - btrfs conversion suggests deleting the ext2_saved subvolume and 
then doing a full defrag and rebalance.  You're attempting a full 
rebalance, but have you yet deleted ext2_saved and did you do the defrag 
before attempting the rebalance?

I'm guessing not, as was the case with the other user that reported this 
issue.  Here's what apparently happened in his case and how we fixed it:

The problem is that btrfs data chunks are 1 GiB each.  Thus, the maximum 
size of a btrfs extent is 1 GiB.  But ext4 doesn't have an arbitrary 
limitation on extent size, and for files over a GiB in size, ext4 extents 
can /also/ be over a GiB in size.

That results in two potential issues at balance time.  First, btrfs 
treats the ext2_saved subvolume as a read-only snapshot and won't touch 
it, thus keeping the ext* data intact in case the user wishes to rollback 
to ext*.  I don't think btrfs touches that data during a balance either, 
as it really couldn't do so /safely/ without incorporating all of the 
ext* code into btrfs.  I'm not sure how it expresses that situation, but 
it's quite possible that btrfs treats it as enospc.

Second, for files that had ext4 extents greater than a GiB, balance will 
naturally enospc, because even the biggest possible btrfs extent, a full 
1 GiB data chunk, is too small to hold the existing file extent.  Of 
course this only happens on filesystems converted from ext*, because 
natively btrfs has no way to make an extent larger than a GiB, so it 
won't run into the problem if it was created natively instead of 
converted from ext*.

Once the ext2_saved subvolume/snapshot is deleted, defragging should cure 
the problem as it rewrites those files to btrfs-native chunks, normally 
defragging but in this case fragging to the 1 GiB btrfs-native data-chunk-
size extent size.

Alternatively, and this is what the other guy did, one can find all the 
files from the original ext*fs over a GiB in size, and move them off-
filesystem and back AFAIK he had several gigs of spare RAM and no files 
larger than that, so he used tmpfs as the temporary storage location, 
which is memory so the only I/O is that on the btrfs in question.  By 
doing that he deleted the existing files on btrfs and recreated them, 
naturally splitting the extents on data-chunk-boundaries as btrfs 
normally does, in the recreation.

If you had deleted the ext2_saved subvolume/snapshot and done the defrag 
already, that explanation doesn't work as-is, but I'd still consider it 
an artifact from the conversion, and try the alternative move-off-
filesystem-temporarily method.

If you don't have any files over a GiB in size, then I don't know... 
perhaps it's some other bug.

---
[1] Raid5/6 support not yet complete.  Operational code is there but 
recovery code is still incomplete.

-- 
Duncan - List replies preferred.   No HTML msgs.
Every nonfree program has a lord, a master --
and if you use the program, he is your master.  Richard Stallman

--
To unsubscribe from this list: send the line unsubscribe linux-btrfs in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: convert from RAID5: enospc errors during balance (500 GB free on each device)

2014-04-22 Thread Arjen Nienhuis

 Did you try running another balance soft?

I tried a few. The only thing that happens is that more empty blocks
get allocated. I also tried convert=single,profiles=raid5. I ended up
with a few empty 'single' blocks. (The empty blocks could be discarded
with balance usage=0)

 Meanwhile... kernel v3.13, btrfs-progs v3.12.  You're just a bit behind,
 as kernel 3.14 is out (and 3.15 is rc2), and btrfs-progs is 3.14
 (possibly 3.14.1, I've not updated in a week or so).  When you run into
 problems, consider updating before reporting, as there are still very
 real fixes going into each new version, and they might just fix the
 problem you're reporting.

I just upgraded to Ubuntu 14.04. I can try a kernel ppa but I didn't
find anything in the about this on the wiki.

On Mon, Apr 21, 2014 at 8:32 AM, Arjen Nienhuis a.g.nienh...@gmail.com wrote:
 I experimented with RAID5, but now I want to get rid of it:

 $ sudo btrfs balance start -dconvert=raid1,soft -v /
 Dumping filters: flags 0x1, state 0x0, force is off
   DATA (flags 0x300): converting, target=16, soft is on
 ERROR: error during balancing '/' - No space left on device
 There may be more info in syslog - try dmesg | tail

 This is in dmesg:

 [  166.799139] btrfs: relocating block group 13368320589824 flags 129
 [  248.927524] btrfs: relocating block group 13338255818752 flags 129
 ...
 [  409.184588] btrfs: relocating block group 11607454253056 flags 129
 [  410.528075] btrfs: relocating block group 11605306769408 flags 129
 [  411.957809] btrfs: 62 enospc errors during balance

 Extra info:
 Data, RAID1: total=3.26TiB, used=3.25TiB
 Data, RAID5: total=124.00GiB, used=123.90GiB
 System, RAID1: total=32.00MiB, used=508.00KiB
 System, single: total=4.00MiB, used=0.00
 Metadata, RAID1: total=9.00GiB, used=7.97GiB
 Label: data  uuid: 93be47ab-af7a-4329-8a6c-c6ff220060b0
 Total devices 3 FS bytes used 3.38TiB
 devid1 size 1.82TiB used 1.22TiB path /dev/sdd
 devid2 size 3.64TiB used 2.63TiB path /dev/sdb
 devid6 size 3.64TiB used 2.87TiB path /dev/sdc

 Btrfs v3.12

 Linux maxi 3.13.0-24-generic #46-Ubuntu SMP Thu Apr 10 19:11:08 UTC
 2014 x86_64 x86_64 x86_64 GNU/Linux

 I tried scrub (no errors) and I cleared the free space cache. Nothing worked.

 Now I'm doing full balance (without 'soft') but that will take days.
--
To unsubscribe from this list: send the line unsubscribe linux-btrfs in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: convert from RAID5: enospc errors during balance (500 GB free on each device)

2014-04-22 Thread Duncan

Arjen Nienhuis posted on Tue, 22 Apr 2014 23:52:43 +0200 as excerpted:

 Did you try running another balance soft?
 
 I tried a few. The only thing that happens is that more empty blocks get
 allocated. I also tried convert=single,profiles=raid5. I ended up with a
 few empty 'single' blocks. (The empty blocks could be discarded with
 balance usage=0)

Hmm... So the blocks of the new type get allocated, but nothing gets 
written into them.  That's strange.

I'm not a dev just a btrfs user and list regular, and this is the first 
time I've seen that sort of behavior reported so I can't help much 
further there, but the reason I suggested rerunning the balance is that 
I've had times when I got ENOSPC errors for some chunks on the first 
balance, but it did free enough space from the ones it could rewrite that 
a second pass went just fine, no errors, and since you didn't mention 
retrying in the original post, I thought I would.

The one similar report we did have was with someone that had originally 
done the conversion from ext3/4 following the instructions in the wiki, 
and had removed the ext3/4 subvolume after conversion, but had *NOT* then 
run the defrag and rebalance as recommended.  IIRC he was actually trying 
a later btrfs mode convert (IIRC from single to raid1, after adding a 
second device) as well, when he ran into problems.  Those problems were 
traced to the fact that btrfs data chunks are 1 GiB in size, so that's 
the largest possible extent on native btrfs, while ext3/4 allowed larger 
extents.  The problems thus occurred when the conversion process ran into 
these larger extents, which the original btrfs single mode handled, but 
which the conversion to raid1, copying to the other device in now native 
btrfs mode, couldn't manage.

He ended up having to move a bunch of files off of the btrfs (to tmpfs in 
his case, the files were multi-gig but he had the memory to handle it) 
and then back onto it, in ordered to have them recreated in btrfs-native 
1-GiB data chunks mode.  I don't recall whether they were written back in 
raid1 or single mode, but in any case, once he dealt with the problem 
files that way, the conversion finished properly.

So on the off chance... your filesystem wasn't also originally converted 
from ext3/4 to btrfs, was it?  If so, it might be that issue.  Otherwise 
I don't have a clue.  Maybe the devs can help.

 Meanwhile... kernel v3.13, btrfs-progs v3.12.  You're just a bit
 behind, as kernel 3.14 is out (and 3.15 is rc2), and btrfs-progs is
 3.14 (possibly 3.14.1, I've not updated in a week or so).  When you
 run into problems, consider updating before reporting, as there are
 still very real fixes going into each new version, and they might just
 fix the problem you're reporting.
 
 I just upgraded to Ubuntu 14.04. I can try a kernel ppa but I didn't
 find anything in the about this on the wiki.

When you do a mkfs.btrfs, the output there warns that btrfs is still 
experimental, run current kernels, keep backups, etc.  (I'm not creating 
one ATM so I unlike the wiki I don't have that right in front of me, but 
I just redid my backups with a new mkfs.btrfs a few days ago and the 
experimental warning was still there).  The wiki also strongly encourages 
both.  See the stuff in red right near the top of the getting started 
page:

https://btrfs.wiki.kernel.org/index.php/Getting_started

Further down the page there's both a distros vs shipped versions table, 
and a discussion of building from source when the distro version is 
behind.

https://btrfs.wiki.kernel.org/index.php/
Getting_started#Compiling_Btrfs_from_sources

Also see stability status, which emphasizes running the latest stable if 
not the development version, on the main page:

https://btrfs.wiki.kernel.org/index.php/Main_Page#Stability_status

-- 
Duncan - List replies preferred.   No HTML msgs.
Every nonfree program has a lord, a master --
and if you use the program, he is your master.  Richard Stallman

--
To unsubscribe from this list: send the line unsubscribe linux-btrfs in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

convert from RAID5: enospc errors during balance (500 GB free on each device)

2014-04-21 Thread Arjen Nienhuis

I experimented with RAID5, but now I want to get rid of it:

$ sudo btrfs balance start -dconvert=raid1,soft -v /
Dumping filters: flags 0x1, state 0x0, force is off
  DATA (flags 0x300): converting, target=16, soft is on
ERROR: error during balancing '/' - No space left on device
There may be more info in syslog - try dmesg | tail

This is in dmesg:

[  166.799139] btrfs: relocating block group 13368320589824 flags 129
[  248.927524] btrfs: relocating block group 13338255818752 flags 129
...
[  409.184588] btrfs: relocating block group 11607454253056 flags 129
[  410.528075] btrfs: relocating block group 11605306769408 flags 129
[  411.957809] btrfs: 62 enospc errors during balance

Extra info:
Data, RAID1: total=3.26TiB, used=3.25TiB
Data, RAID5: total=124.00GiB, used=123.90GiB
System, RAID1: total=32.00MiB, used=508.00KiB
System, single: total=4.00MiB, used=0.00
Metadata, RAID1: total=9.00GiB, used=7.97GiB
Label: data  uuid: 93be47ab-af7a-4329-8a6c-c6ff220060b0
Total devices 3 FS bytes used 3.38TiB
devid1 size 1.82TiB used 1.22TiB path /dev/sdd
devid2 size 3.64TiB used 2.63TiB path /dev/sdb
devid6 size 3.64TiB used 2.87TiB path /dev/sdc

Btrfs v3.12

Linux maxi 3.13.0-24-generic #46-Ubuntu SMP Thu Apr 10 19:11:08 UTC
2014 x86_64 x86_64 x86_64 GNU/Linux

I tried scrub (no errors) and I cleared the free space cache. Nothing worked.

Now I'm doing full balance (without 'soft') but that will take days.
--
To unsubscribe from this list: send the line unsubscribe linux-btrfs in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: convert from RAID5: enospc errors during balance (500 GB free on each device)

2014-04-21 Thread Duncan

Arjen Nienhuis posted on Mon, 21 Apr 2014 08:32:56 +0200 as excerpted:

 I experimented with RAID5, but now I want to get rid of it:
 
 $ sudo btrfs balance start -dconvert=raid1,soft -v /
 Dumping filters: flags 0x1, state 0x0, force is off
   DATA (flags 0x300): converting, target=16, soft is on
 ERROR: error during balancing '/' - No space left on device There may be
 more info in syslog - try dmesg | tail
 
 This is in dmesg:
 
 [ 166.799139] btrfs: relocating block group 13368320589824 flags 129
 [ 248.927524] btrfs: relocating block group 13338255818752 flags 129 ...
 [ 409.184588] btrfs: relocating block group 11607454253056 flags 129
 [ 410.528075] btrfs: relocating block group 11605306769408 flags 129
 [ 411.957809] btrfs: 62 enospc errors during balance
 
 Extra info:

 Data, RAID1: total=3.26TiB, used=3.25TiB
 Data, RAID5: total=124.00GiB, used=123.90GiB
 System, RAID1: total=32.00MiB, used=508.00KiB
 System, single: total=4.00MiB, used=0.00
 Metadata, RAID1: total=9.00GiB, used=7.97GiB 

 Label: data  uuid: 93be47ab-af7a-4329-8a6c-c6ff220060b0
 Total devices 3 FS bytes used 3.38TiB
 devid1 size 1.82TiB used 1.22TiB path /dev/sdd
 devid2 size 3.64TiB used 2.63TiB path /dev/sdb
 devid6 size 3.64TiB used 2.87TiB path /dev/sdc
 
 Btrfs v3.12
 
 Linux maxi 3.13.0-24-generic #46-Ubuntu SMP Thu Apr 10 19:11:08 UTC 2014
 x86_64 x86_64 x86_64 GNU/Linux
 
 I tried scrub (no errors) and I cleared the free space cache. Nothing
 worked.
 
 Now I'm doing full balance (without 'soft') but that will take days.

Did you try running another balance soft?

Depending on how fully allocated you were originally, those 62 enospce 
errors during the first balance may have simply been before you cleared 
enough space to rewrite additional raid1 data.  You now have two devices 
with plenty of unallocated room on them (the btrfs fi show output), so 
rerunning the soft conversion to raid1 may well be all you needed to do.

Meanwhile... kernel v3.13, btrfs-progs v3.12.  You're just a bit behind, 
as kernel 3.14 is out (and 3.15 is rc2), and btrfs-progs is 3.14 
(possibly 3.14.1, I've not updated in a week or so).  When you run into 
problems, consider updating before reporting, as there are still very 
real fixes going into each new version, and they might just fix the 
problem you're reporting.

That said, I don't know if updating would help here, but it's worth a 
shot.

-- 
Duncan - List replies preferred.   No HTML msgs.
Every nonfree program has a lord, a master --
and if you use the program, he is your master.  Richard Stallman

--
To unsubscribe from this list: send the line unsubscribe linux-btrfs in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

btrfs: 1 enospc errors during balance when balancing after formerly failed raid1 device re-appeared

2013-11-15 Thread Lutz Vieweg


Hi again,

I just did another test on resilience with btrfs/raid1, this time I tested
the following scenario: One out of two raid1 devices disappears. The filesystem
is written to in degraded mode. The missing device re-appears (think of e.g.
a storage device that temporarily became unavailable due to a cable or 
controller
issue that is later fixed). User issues btrfs filesystem balance.

Alas, this scenario ends in an effor btrfs: 1 enospc errors during balance,
with the raid1 staying degraded.

Here's the test procedure in detail:

Testing was done using vanilla linux-3.12 (x86_64)
plus btrfs-progs at commit 9f0c53f574b242b0d5988db2972c8aac77ef35a9
plus [PATCH] btrfs-progs: for mixed group check opt before default raid profile is 
enforced

Preparing two 100 MB image files:
 # dd if=/dev/zero of=/tmp/img1 bs=1024k count=100
 100+0 records in
 100+0 records out
 104857600 bytes (105 MB) copied, 0.201003 s, 522 MB/s

 # dd if=/dev/zero of=/tmp/img2 bs=1024k count=100
 100+0 records in
 100+0 records out
 104857600 bytes (105 MB) copied, 0.185486 s, 565 MB/s

Preparing two loop devices on those images to act as the underlying
block devices for btrfs:
 # losetup /dev/loop1 /tmp/img1
 # losetup /dev/loop2 /tmp/img2

Mounting / writing to the fs:
 # mount -t btrfs /dev/loop1 /mnt/tmp
 # echo asdfasdfasdfasdf /mnt/tmp/testfile1
 # md5sum /mnt/tmp/testfile1
 f1264d450b9feda62fec5a1e11faba1a  /mnt/tmp/testfile1
 # umount /mnt/tmp

First storage device disappears:
 # losetup -d /dev/loop1

Mounting degraded btrfs:
 # mount -t btrfs -o degraded /dev/loop2 /mnt/tmp

Testing that testfile1 is still readable:
 # md5sum /mnt/tmp/testfile1
f1264d450b9feda62fec5a1e11faba1a  /mnt/tmp/testfile1

Creating testfile2 on the degraded filesystem:
 # echo qwerqwerqwerqwer /mnt/tmp/testfile2
 # md5sum /mnt/tmp/testfile2
 9df26d2f2657462c435d58274cc5bdf0  /mnt/tmp/testfile2
 # umount /mnt/tmp

Now we assume the issue causing the first storage device
to be unavailable to be fixed:

 # losetup /dev/loop1 /tmp/img1
 # mount -t btrfs /dev/loop1 /mnt/tmp

Notice that at this point, I would have expected some kind of warning
in the syslog that the mounted filesystem is not balanced and
thus not redundant.
But there was no such warning.
This may easily lead operators into a situation where they do
not realize that a btrfs is not redundant and losing one storage
device will lose data.

Testing that the two testfiles (one of which is not yet
stored on both devices) are still readable:
 # md5sum /mnt/tmp/testfile1
f1264d450b9feda62fec5a1e11faba1a  /mnt/tmp/testfile1
 # md5sum /mnt/tmp/testfile2
9df26d2f2657462c435d58274cc5bdf0  /mnt/tmp/testfile2

So far, so good.
Now since we know the filesystem is not really redundant,
we start a balance:

 # btrfs filesystem balance /mnt/tmp
 ERROR: error during balancing '/mnt/tmp' - No space left on device
 There may be more info in syslog - try dmesg | tail

Syslog shows:
 kernel: btrfs: relocating block group 20971520 flags 21
 kernel: btrfs: found 3 extents
 kernel: btrfs: relocating block group 4194304 flags 5
 kernel: btrfs: relocating block group 0 flags 2
 kernel: btrfs: 1 enospc errors during balance

So the raid1 remains degraded.

BTW: I wonder why btrfs balance seems to require additional space
for writing data to the re-appeared disk.

I also wonder: Would btrfs try to write _two_ copies of
everything to _one_ remaining device of a degraded two-disk raid1?
(If yes, then this means a raid1 would have to be planned with
twice the capacity just to be sure that one failing disk will
not lead to an out-of-diskspace situation. Not good.)

Regards,

Lutz Vieweg

--
To unsubscribe from this list: send the line unsubscribe linux-btrfs in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: btrfs: 1 enospc errors during balance when balancing after formerly failed raid1 device re-appeared

2013-11-15 Thread Duncan

Hugo Mills posted on Fri, 15 Nov 2013 12:38:41 + as excerpted:

 I also wonder: Would btrfs try to write _two_ copies of everything to
 _one_ remaining device of a degraded two-disk raid1?
 
 No. It would have to degrade from RAID-1 to DUP to do that (and I
 think we prevent DUP data for some reason).

You may be correct about DUP data, but that is unlikely to be the issue 
here, because he's likely using the mixed-mode default due to the 1GB 
filesystem size, and on a multi-device filesystem that should default to 
RAID1 just as metadata by itself does.

However, I noticed that his outlined duplicator SKIPPED the mkfs.btrfs 
command part, and there's no btrfs filesystem show and btrfs filesystem df 
to verify how the kernel's actually treating the filesystem, so...

@ LV:

For further tests, please include these commands and their output:

1) your mkfs.btrfs command

[Then mount, and after mount...]

2) btrfs filesystem show path

3) btrfs filesystem df path

Thanks.  These should make what btrfs is doing far clearer.


Meanwhile, I've been following your efforts with quite some interest as 
they correspond to some of the pre-deployment btrfs raid1 mode testing I 
did.  This was several kernels ago, however, so I had been wondering if 
the behavior had changed, hopefully for the better, and your testing 
looks to be headed toward the same test I did at some point.

Back then, I found a rather counterintuitive result of my own.

Basically, take a two-device raid1 mode (both data and metadata; in my 
case the devices were over a gig so mixed data+metadata wasn't invoked 
and I specified -m raid1 -d raid1 when doing the mkfs.btrfs) btrfs, mount 
it, copy some files to it, unmount it.

Then disconnect one device (I was using actual devices not the loop 
devices you're using) and mount degraded.  Make a change to the degraded 
filesystem.  Unmount.

Then disconnect that device and reconnect the other.  Mount degraded.  
Make a *DIFFERENT* change to the same file.  Unmount.  The two copies 
have now forked in an incompatible manner.

Now reconnect both devices and remount, this time without degraded.

As you, here I expected btrfs to protest, particularly so since the two 
copies were incompatible.  *NO* *PROTEST*!!

OK, so check that file to see which version I got.  I've now forgotten 
which one it was, but it was definitely one of the two forks, not the 
original version.

Now unmount and disconnect the device with the copy it said it had.  
Mount the filesystem degraded with the other device.  Check the file 
again.

!!  I got the other fork! !!

Not only did btrfs not protest when I mounted a raid1 device undegraded 
after making incompatible changes to the file with each of the two 
devices mounted degraded separately, but accessing the file on the 
undegraded filesystem neither protested nor corrected the other copy, 
which remained the incompatibly forked copy as confirmed by remounting 
the filesystem degraded with just that device in ordered to access it.

To my way of thinking, that's downright dangerous, as well as being 
entirely unintuitive.

Unfortunately, I didn't actually do a balance to see what btrfs would do 
with the incompatible versions, I simply blew away that testing 
filesystem with a new mkfs.btrfs (I'm on SSD so mkfs.btrfs automatically 
issues a trim/discard to clear the new filesystem space before mking it), 
and I've been kicking myself for not doing so ever since, because I 
really would like to know balance actually /does/ in such a case!  But I 
was still new enough to btrfs at that time that I didn't really know what 
I was doing, so I didn't realize I'd omitted a critical part of the test 
until it was too late, and I've not been interested /enough/ in the 
outcome to redo the test again, with a newer kernel and tools and a 
balance this time.

What scrub would have done with it would be an interesting testcase as 
well, but I don't know that either, because I never tried it.

At least I hadn't redone the test yet.  But I keep thinking about it, and 
I guess I would have one of these days.  But now it seems like you're 
heading toward doing it for me. =:^)

Meanwhile, the conclusion I took from the test was that if I ever had to 
mount degraded in read/write mode, I should make *VERY* sure I 
CONSISTENTLY used the same device when I did so.  And when I undegraded, 
/hopefully/ a balance would choose the newer version.  Unfortunately I 
never did actually test that, so I figured should I actually need to use 
degraded, even if the degrade was only temporary, the best way to recover 
was probably to trim that entire partition on the lost device and then 
proceed to add it back into the filesystem as if it were a new device and 
do a balance to finally recover raid1 mode.

Meanwhile (2), what btrfs raid1 mode /has/ been providing me with is data 
integrity via the checksumming and scrub features.  And with raid1 
providing a second copy of the data to work with, scrub

Still getting a lot of -28 (ENOSPC?) errors during balance

2013-04-02 Thread Roman Mamedov

Hello,

With kernel 3.7.10 patched with Btrfs: limit the global reserve to 512mb.
(the problem was occuring also without this patch, but seemed to be even worse).

At the start of balance:

Data: total=31.85GB, used=9.96GB
System: total=4.00MB, used=16.00KB
Metadata: total=1.01GB, used=696.17MB

btrfs balance start -musage=5 -dusage=5 is going on for about 50 minutes

Current situation:

Balance on '/mnt/r1/' is running
1 out of about 2 chunks balanced (20 considered),  50% left

Data: total=30.85GB, used=10.04GB
System: total=4.00MB, used=16.00KB
Metadata: total=1.01GB, used=851.69MB

And a constant stream of these in dmesg:

[ 7614.756339] btrfs: block rsv returned -28
[ 7614.756341] [ cut here ]
[ 7614.756370] WARNING: at fs/btrfs/extent-tree.c:6297 
btrfs_alloc_free_block+0x351/0x360 [btrfs]()
[ 7614.756372] Hardware name: GA-880GM-UD2H
[ 7614.756374] Modules linked in: nfsd auth_rpcgss nfs_acl nfs lockd fscache 
sunrpc bridge 8021q garp stp llc aoe it87 hwmon_vid snd_hda_codec_hdmi 
snd_hda_codec_realtek snd_hda_intel snd_hda_codec snd_hwdep snd_pcsp snd_pcm 
kvm_amd kvm snd_page_alloc snd_timer snd soundcore sp5100_tco serio_raw joydev 
k10temp edac_core i2c_piix4 edac_mce_amd mac_hid shpchp wmi btrfs libcrc32c 
zlib_deflate raid1 raid456 async_raid6_recov async_memcpy async_pq async_xor 
xor async_tx raid6_pq nbd sata_nv hid_generic usbhid usb_storage hid microcode 
r8169 sata_mv
[ 7614.756411] Pid: 4708, comm: btrfs Tainted: GW3.7.10-rm3+ #38
[ 7614.756413] Call Trace:
[ 7614.756421]  [8105516f] warn_slowpath_common+0x7f/0xc0
[ 7614.756425]  [810551ca] warn_slowpath_null+0x1a/0x20
[ 7614.756440]  [a00f71c1] btrfs_alloc_free_block+0x351/0x360 [btrfs]
[ 7614.756453]  [a00e3d03] ? __btrfs_cow_block+0x323/0x4f0 [btrfs]
[ 7614.756472]  [a01263ab] ? read_extent_buffer+0xbb/0x110 [btrfs]
[ 7614.756485]  [a00e3b06] __btrfs_cow_block+0x126/0x4f0 [btrfs]
[ 7614.756498]  [a00e4047] btrfs_cow_block+0xf7/0x200 [btrfs]
[ 7614.756515]  [a014abb7] do_relocation+0x467/0x530 [btrfs]
[ 7614.756528]  [a00ecdfa] ? block_rsv_add_bytes+0x5a/0x80 [btrfs]
[ 7614.756545]  [a014e8dd] relocate_tree_blocks+0x61d/0x650 [btrfs]
[ 7614.756562]  [a014f8d6] relocate_block_group+0x446/0x6a0 [btrfs]
[ 7614.756579]  [a014fccd] btrfs_relocate_block_group+0x19d/0x2d0 
[btrfs]
[ 7614.756596]  [a0128685] btrfs_relocate_chunk.isra.53+0x75/0x770 
[btrfs]
[ 7614.756613]  [a0138668] ? btrfs_set_lock_blocking_rw+0xa8/0xf0 
[btrfs]
[ 7614.756630]  [a011ffa1] ? release_extent_buffer.isra.26+0x81/0xf0 
[btrfs]
[ 7614.756647]  [a0125227] ? free_extent_buffer+0x37/0x90 [btrfs]
[ 7614.756663]  [a012cdc7] btrfs_balance+0x877/0xe00 [btrfs]
[ 7614.756680]  [a0132f9c] btrfs_ioctl_balance+0x10c/0x430 [btrfs]
[ 7614.756684]  [8167e38e] ? _raw_spin_lock+0xe/0x20
[ 7614.756702]  [a013739b] btrfs_ioctl+0xf2b/0x1970 [btrfs]
[ 7614.756707]  [8114f7f9] ? handle_mm_fault+0x249/0x310
[ 7614.756711]  [816822d4] ? __do_page_fault+0x254/0x4e0
[ 7614.756714]  [811527d0] ? __vma_link_rb+0x30/0x40
[ 7614.756718]  [8118f9b0] do_vfs_ioctl+0x90/0x570
[ 7614.756723]  [8108793a] ? finish_task_switch+0x4a/0xe0
[ 7614.756726]  [8118ff21] sys_ioctl+0x91/0xb0
[ 7614.756730]  [81686b1d] system_call_fastpath+0x1a/0x1f
[ 7614.756732] ---[ end trace 9fdf5720be6ec4cb ]---
[ 7614.756894] btrfs: block rsv returned -28
[ 7614.756895] [ cut here ]
[ 7614.756910] WARNING: at fs/btrfs/extent-tree.c:6297 
btrfs_alloc_free_block+0x351/0x360 [btrfs]()
[ 7614.756911] Hardware name: GA-880GM-UD2H
[ 7614.756912] Modules linked in: nfsd auth_rpcgss nfs_acl nfs lockd fscache 
sunrpc bridge 8021q garp stp llc aoe it87 hwmon_vid snd_hda_codec_hdmi 
snd_hda_codec_realtek snd_hda_intel snd_hda_codec snd_hwdep snd_pcsp snd_pcm 
kvm_amd kvm snd_page_alloc snd_timer snd soundcore sp5100_tco serio_raw joydev 
k10temp edac_core i2c_piix4 edac_mce_amd mac_hid shpchp wmi btrfs libcrc32c 
zlib_deflate raid1 raid456 async_raid6_recov async_memcpy async_pq async_xor 
xor async_tx raid6_pq nbd sata_nv hid_generic usbhid usb_storage hid microcode 
r8169 sata_mv
[ 7614.756945] Pid: 4708, comm: btrfs Tainted: GW3.7.10-rm3+ #38
[ 7614.756946] Call Trace:
[ 7614.756950]  [8105516f] warn_slowpath_common+0x7f/0xc0
[ 7614.756953]  [810551ca] warn_slowpath_null+0x1a/0x20
[ 7614.756967]  [a00f71c1] btrfs_alloc_free_block+0x351/0x360 [btrfs]
[ 7614.756983]  [a0147ad7] ? add_delayed_tree_ref+0xd7/0x190 [btrfs]
[ 7614.757000]  [a014769e] ? add_delayed_ref_head.isra.7+0xce/0x160 
[btrfs]
[ 7614.757017]  [a01482a5] ? btrfs_add_delayed_tree_ref+0x115/0x1a0 
[btrfs]
[ 7614.757030]  [a00e3b06] __btrfs_cow_block+0x126/0x4f0 [btrfs]
[ 7614.757034]  [8112feba] ?

Re: Still getting a lot of -28 (ENOSPC?) errors during balance

2013-04-02 Thread Roman Mamedov

On Tue, 2 Apr 2013 14:04:52 +0600
Roman Mamedov r...@romanrm.ru wrote:

 With kernel 3.7.10 patched with Btrfs: limit the global reserve to 512mb.
 (the problem was occuring also without this patch, but seemed to be even 
 worse).
 
 At the start of balance:
 
 Data: total=31.85GB, used=9.96GB
 System: total=4.00MB, used=16.00KB
 Metadata: total=1.01GB, used=696.17MB
 
 btrfs balance start -musage=5 -dusage=5 is going on for about 50 minutes
 
 Current situation:
 
 Balance on '/mnt/r1/' is running
 1 out of about 2 chunks balanced (20 considered),  50% left
 
 Data: total=30.85GB, used=10.04GB
 System: total=4.00MB, used=16.00KB
 Metadata: total=1.01GB, used=851.69MB

About 2 hours 10 minutes into the balance, it was still going, with:

Data: total=30.85GB, used=10.06GB
System: total=4.00MB, used=16.00KB
Metadata: total=1.01GB, used=909.16MB

Stream of -28 errors continues non-stop in dmesg;

At ~2hr20min looks like it decided to allocate some more space for metadata:

Data: total=30.85GB, used=10.01GB
System: total=4.00MB, used=16.00KB
Metadata: total=2.01GB, used=748.56MB

And shortly after (~ 2hr25min) it was done. After the balance:

Data: total=29.85GB, used=10.01GB
System: total=4.00MB, used=16.00KB
Metadata: total=2.01GB, used=748.27MB

-- 
With respect,
Roman


signature.asc
Description: PGP signature

Re: Still getting a lot of -28 (ENOSPC?) errors during balance

2013-04-02 Thread Josef Bacik

On Tue, Apr 02, 2013 at 02:04:52AM -0600, Roman Mamedov wrote:
 Hello,
 
 With kernel 3.7.10 patched with Btrfs: limit the global reserve to 512mb.
 (the problem was occuring also without this patch, but seemed to be even 
 worse).
 
 At the start of balance:
 
 Data: total=31.85GB, used=9.96GB
 System: total=4.00MB, used=16.00KB
 Metadata: total=1.01GB, used=696.17MB
 
 btrfs balance start -musage=5 -dusage=5 is going on for about 50 minutes
 
 Current situation:
 
 Balance on '/mnt/r1/' is running
 1 out of about 2 chunks balanced (20 considered),  50% left
 
 Data: total=30.85GB, used=10.04GB
 System: total=4.00MB, used=16.00KB
 Metadata: total=1.01GB, used=851.69MB
 
 And a constant stream of these in dmesg:
 

Can you try this out and see if it helps?  Thanks,

Josef

diff --git a/fs/btrfs/relocation.c b/fs/btrfs/relocation.c
index 0d89ff0..9830e86 100644
--- a/fs/btrfs/relocation.c
+++ b/fs/btrfs/relocation.c
@@ -2548,6 +2548,13 @@ static int do_relocation(struct btrfs_trans_handle 
*trans,
list_for_each_entry(edge, node-upper, list[LOWER]) {
cond_resched();
 
+   ret = btrfs_block_rsv_refill(rc-extent_root, rc-block_rsv,
+rc-extent_root-leafsize,
+BTRFS_RESERVE_FLUSH_ALL);
+   if (ret) {
+   err = ret;
+   break;
+   }
upper = edge-node[UPPER];
root = select_reloc_root(trans, rc, upper, edges, nr);
BUG_ON(!root);
--
To unsubscribe from this list: send the line unsubscribe linux-btrfs in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: Still getting a lot of -28 (ENOSPC?) errors during balance

2013-04-02 Thread Roman Mamedov

On Tue, 2 Apr 2013 09:46:26 -0400
Josef Bacik jba...@fusionio.com wrote:

 On Tue, Apr 02, 2013 at 02:04:52AM -0600, Roman Mamedov wrote:
  Hello,
  
  With kernel 3.7.10 patched with Btrfs: limit the global reserve to 512mb.
  (the problem was occuring also without this patch, but seemed to be even 
  worse).
  
  At the start of balance:
  
  Data: total=31.85GB, used=9.96GB
  System: total=4.00MB, used=16.00KB
  Metadata: total=1.01GB, used=696.17MB
  
  btrfs balance start -musage=5 -dusage=5 is going on for about 50 minutes
  
  Current situation:
  
  Balance on '/mnt/r1/' is running
  1 out of about 2 chunks balanced (20 considered),  50% left
  
  Data: total=30.85GB, used=10.04GB
  System: total=4.00MB, used=16.00KB
  Metadata: total=1.01GB, used=851.69MB
  
  And a constant stream of these in dmesg:
  
 
 Can you try this out and see if it helps?  Thanks,

Hello,

Well that balance has now completed, and unfortunately I don't have a complete
image of the filesystem from before, to apply the patch and check if the same
operation goes better this time.

I'll keep it in mind and will try to test it out if I will get a similar
situation again on some filesystem.

Generally what seems to make me run into various problems with balance, is the
following usage scenario: On an active filesystem (used as /home and root FS),
a snapshot is made every 30 minutes with an unique (timestamped) name; and once
a day snapshots from more than two days ago are purged. And it goes like this
for months.

Another variant of this, a backup partition, where snapshots are made every six
hours, and all snapshots are kept for 1-3 months before getting purged.

I guess this kind of usage causes a lot of internal fragmentation or
something, which makes it difficult for a balance to find enough free space to
work with.

 
 Josef
 
 diff --git a/fs/btrfs/relocation.c b/fs/btrfs/relocation.c
 index 0d89ff0..9830e86 100644
 --- a/fs/btrfs/relocation.c
 +++ b/fs/btrfs/relocation.c
 @@ -2548,6 +2548,13 @@ static int do_relocation(struct btrfs_trans_handle 
 *trans,
   list_for_each_entry(edge, node-upper, list[LOWER]) {
   cond_resched();
  
 + ret = btrfs_block_rsv_refill(rc-extent_root, rc-block_rsv,
 +  rc-extent_root-leafsize,
 +  BTRFS_RESERVE_FLUSH_ALL);
 + if (ret) {
 + err = ret;
 + break;
 + }
   upper = edge-node[UPPER];
   root = select_reloc_root(trans, rc, upper, edges, nr);
   BUG_ON(!root);
 --
 To unsubscribe from this list: send the line unsubscribe linux-btrfs in
 the body of a message to majord...@vger.kernel.org
 More majordomo info at  http://vger.kernel.org/majordomo-info.html


-- 
With respect,
Roman


signature.asc
Description: PGP signature

38 matches

Mail list logo