RE: ENOSPC errors during raid1 rebalance
Alright! After doing: cd /mymedia; find . -type f | while read file; do mv -v "$file" /dev/shm; f2=`basename "$file"`; mv -v "/dev/shm/$f2" "$file"; done I finally moved whatever files out of the "single" allocation and back onto the new RAID1 profile: oot@ossy:~# /usr/src/btrfs-progs/btrfs fi df /mymedia Data, RAID1: total=1.23TiB, used=1000.43GiB Data, single: total=10.00GiB, used=0.00 System, RAID1: total=32.00MiB, used=184.00KiB Metadata, RAID1: total=3.00GiB, used=1.34GiB And then the rebalance finally was able to move those two block groups and I got no errors: root@ossy:~# btrfs balance start -dconvert=raid1,soft /mymedia Done, had to relocate 2 out of 1264 chunks And now I'm totally RAID1: root@ossy:~# /usr/src/btrfs-progs/btrfs fi df /mymedia Data, RAID1: total=1.23TiB, used=1000.43GiB System, RAID1: total=32.00MiB, used=184.00KiB Metadata, RAID1: total=3.00GiB, used=1.34GiB Yay!! If I want to get back the space I can do another defragment but I don't really care right now. Thanks everyone for all your help on this! Hopefully this thread will assist anyone else in the future if this occurs again. Sincerely, Michael Russo, Systems Engineer PaperSolve, Inc. 268 Watchogue Road Staten Island, NY 10314 Randomly generated quote of the last 5 minutes: A good plan today is better than a perfect plan tomorrow. -- Patton -- To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
RE: ENOSPC errors during raid1 rebalance
Thanks Hugo, that makes sense, and maybe leads to a possible way to fix the
issue in future versions of btrfs-convert or a way to handle it in the balance
code.
What I did to find files with extents:
cd /mymedia
find . -type f -print0 | xargs -0 filefrag | grep -v 1\ extent | grep -v 0\
extent | awk -F: '{print $1}' > /tmp/extent
cat /tmp/extent | while read file; do mv -v "$file" /dev/shm; f2=`basename
"$file"`; mv -v "/dev/shm/$f2" "$file"; done
I no longer care that even after doing this I still have a file with multiple
extents, I just don't want them inside those block groups with >1GB extents.
(BTW my terminology may be all off here, I just know that in my syslog btrfs
tries to relocate two particular "block groups" and gets 2 enospc errors for
them.)
But since I'm still getting the errors the files I care about probably aren't
in this list so I'll do it for all the files on the system (since I can't find
out what files are in those block groups).
-Original Message-----
From: Hugo Mills [mailto:h...@carfax.org.uk]
Sent: Friday, March 07, 2014 3:02 AM
To: Mike Russo
Cc: linux-btrfs@vger.kernel.org
Subject: Re: ENOSPC errors during raid1 rebalance
The defrag operation, by its nature, _doesn't_ preserve extents, and thus
can act to break up the large extents, making it possible to balance the chunks
that the offending extents live on.
Hugo.
--
=== Hugo Mills: hugo@... carfax.org.uk | darksatanic.net | lug.org.uk ===
PGP key: 65E74AC0 from wwwkeys.eu.pgp.net or http://www.carfax.org.uk
--- If the first-ever performance is the première, is the ---
last-ever performance the derrière?
--
To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in
the body of a message to majord...@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
RE: ENOSPC errors during raid1 rebalance
Chris Murphy colorremedies.com> writes: > > How can I find out what file is on the block group that > > it's having a problem with? > > I think that's btrfs-debug-tree -b ? But don't hold me to that. > I haven't done enough debugging to find files > from block numbers. Another that might be relevant is > btrfs inspect-internal logical-resolve. > Nah, I don't think they're talking about the same thing. The messages I get in syslog for the 2 blocks are: Mar 4 07:59:23 ossy kernel: [124731.175116] BTRFS info (device sdd1): relocating block group 894460493824 flags 1 Mar 4 07:59:32 ossy kernel: [124739.613467] BTRFS error (device sdd1): allocation failed flags 17, wanted 1368420352 Mar 4 07:59:32 ossy kernel: [124739.613473] BTRFS: space_info 1 has 105551204352 free, is not full Mar 4 07:59:32 ossy kernel: [124739.613476] BTRFS: space_info total=1312112508928, used=1201166741504, pinned=0, reserved=565596160, may_use=1368420352, readonly=4828966912 Mar 4 10:49:33 ossy kernel: [134932.248146] BTRFS info (device sdd1): relocating block group 846142111744 flags 1 Mar 4 10:49:39 ossy kernel: [134938.440347] BTRFS error (device sdd1): allocation failed flags 17, wanted 1219825664 Mar 4 10:49:39 ossy kernel: [134938.440353] BTRFS: space_info 1 has 116232978432 free, is not full Mar 4 10:49:39 ossy kernel: [134938.440356] BTRFS: space_info total=1322849927168, used=1201311391744, pinned=0, reserved=476590080, may_use=1224540160, readonly=4828966912 But if I try to examine either of those block groups I get an error message that makes me think the 2 numbers are not really the same thing. root@ossy:~# /usr/src/btrfs-progs/btrfs-debug-tree -b 894460493824 /dev/sdc1 Check tree block failed, want=894460493824, have=7159859086769936959 Check tree block failed, want=894460493824, have=7159859086769936959 Check tree block failed, want=894460493824, have=7159859086769936959 read block failed check_tree_block Check tree block failed, want=894460493824, have=7159859086769936959 Check tree block failed, want=894460493824, have=7159859086769936959 Check tree block failed, want=894460493824, have=7159859086769936959 read block failed check_tree_block failed to read 894460493824 Sigh. So I'm stuck at 10GB on single, but I don't know which 10GB it is. root@ossy:~# /usr/src/btrfs-progs/btrfs fi df /mymedia Data, RAID1: total=1.21TiB, used=1.09TiB Data, single: total=10.00GiB, used=5.50GiB System, RAID1: total=32.00MiB, used=180.00KiB Metadata, RAID1: total=2.00GiB, used=1.45GiB If anyone knows why that allocation should fail (what is "flags 17"?), or how to force something more to happen, please reply. I'm sure this is due to the ext4 conversion, but that means the utility is making a btrfs filesystem that later can't be converted to another profile for some reason. -- To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
ENOSPC errors during raid1 rebalance
Hi guys - I'm trying to convert a disk from single (/dev/sdc1) to RAID1 (dev/sdd1), and the filesystem was previously ext4 but the conversion seemed to go just fine, and I have no snapshots. System and metadata convert, and almost all my data converts, but there are 70 stubborn GB (14 blocks of 5GB each) that refuse to convert and I get ENOSPC errors when trying to reallocate them. Here's my vital stats: (on kernel 3.14-rc4): root@ossy:/usr/src/btrfs-progs# /usr/src/btrfs-progs/btrfs fi show Label: MyBook uuid: c99cbefb-6a56-41e2-8f99-19f8b5f67884 Total devices 2 FS bytes used 1.09TiB devid1 size 1.82TiB used 1.12TiB path /dev/sdc1 devid2 size 1.82TiB used 1.05TiB path /dev/sdd1 Btrfs v3.12 root@ossy:/usr/src/btrfs-progs# /usr/src/btrfs-progs/btrfs fi df /mymedia Data, RAID1: total=1.04TiB, used=1.03TiB Data, single: total=70.00GiB, used=64.98GiB System, RAID1: total=32.00MiB, used=160.00KiB Metadata, RAID1: total=2.00GiB, used=1.33GiB I've already done a -dusage=0 and -dusage=5 balance and that cleans up anything that gets left around when these 14 blocks fail get moved.When mounting with enospc_debug I get messages like this for each block with the problem: Mar 3 11:58:16 ossy kernel: [52724.423024] BTRFS: block group 935262683136 has 5368709120 bytes, 5321801728 used 0 pinned 0 reserved [readonly] Mar 3 11:58:16 ossy kernel: [52724.423025] BTRFS info (device sdd1): block group has cluster?: no Mar 3 11:58:16 ossy kernel: [52724.423026] BTRFS info (device sdd1): 0 blocks of free space at or bigger than bytes is Mar 3 11:58:16 ossy kernel: [52724.423027] BTRFS: block group 942778875904 has 5368709120 bytes, 5309296640 used 0 pinned 0 reserved [readonly] Mar 3 11:58:16 ossy kernel: [52724.423028] BTRFS info (device sdd1): block group has cluster?: no Mar 3 11:58:16 ossy kernel: [52724.423029] BTRFS info (device sdd1): 0 blocks of free space at or bigger than bytes is I don't have the space to reformat and if this is a genuine bug I'm sure you guys want to fix it too. What else can I do to resolve or help? Sincerely, Michael Russo, Systems Engineer PaperSolve, Inc. 268 Watchogue Road Staten Island, NY 10314 -- To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
