On 2019-06-18 14:26, Stéphane Lesimple wrote:
Hello,
I've been a btrfs user for quite a number of years now, but it seems I
need the wiseness of the btrfs gurus on this one!
I have a 5-hdd btrfs raid1 setup with 4x3T+1x10T drives.
A few days ago, I replaced one of the 3T by a new 10T, running btrfs
replace and then resizing the FS to use all the available space of the
new device.
The filesystem was 90% full before I expanded it so, as expected, most
of the space on the new device wasn't actually allocatable in raid1, as
very few available space was available on the 4 other devs.
Of course the solution is to run a balance, but as the filesystem is now
quite big, I'd like to avoid running a full rebalance. This would be
quite i/o intensive, would be running for several days, and putting and
unecessary stress on the drives. This also seems excessive as in theory
only some Tb would need to be moved: if I'm correct, only one of two
block groups of a sufficient amount of chunks to be moved to the new
device so that the sum of the amount of available space on the 4
preexisting devices would at least equal the available space on the new
device, ~7Tb instead of moving ~22T.
I don't need to have a perfectly balanced FS, I just want all the space
to be allocatable.
I tried using the -ddevid option but it only instructs btrfs to work on
the block groups allocated on said device, as it happens, it tends to
move data between the 4 preexisting devices and doesn't fix my problem.
A full balance with -dlimit=100 did no better.
Is there a way to ask the block group allocator to prefer writing to a
specific device during a balance? Something like -ddestdevid=N? This
would just be a hint to the allocator and the usual constraints would
always apply (and prevail over the hint when needed).
Or is there any obvious solution I'm completely missing?
Based on what you've said, you may actually not have enough free space
that can be allocated to balance things properly.
When a chunk gets balanced, you need to have enough space to create a
new instance of that type of chunk before the old one is removed. As
such, if you can't allocate new chunks at all, you can't balance those
chunks either.
So, that brings up the question of how to deal with your situation.
The first thing I would do is multiple compaction passes using the
`usage` filter. Start with:
btrfs balance -dusage=0 -musage=0 /wherever
That will clear out any empty chunks which haven't been removed (there
shouldn't be any if you're on a recent kernel, but it's good practice
anyway). After that, repeat the same command, but with a value of 10
instead of 0, and then keep repeating in increments of 10 up until 50.
Doing this will clean up chunks that are more than half empty (making
multiple passes like this is a bit more reliable, and in some cases also
more efficient), which should free up enough space for balance to work
with (as well as probably moving most of the block groups it touches to
use the new disk).
