On 23.11.2016 07:09 Duncan wrote: > Yes, you're in a *serious* metadata bind. > Any time global reserve has anything above zero usage, it means the > filesystem is in dire straits, and well over half of your global reserve > is used, a state that is quite rare as btrfs really tries hard not to use > that space at all under normal conditions and under most conditions will > ENOSPC before using the reserve at all. > > And the global reserve comes from metadata but isn't accounted in > metadata usage, so your available metadata is actually negative by the > amount of global reserve used. > > Meanwhile, all available space is allocated to either data or metadata > chunks already -- no unallocated space left to allocate new metadata > chunks to take care of the problem (well, ~1 MiB unallocated, but that's > not enough to allocate a chunk, metadata chunks being nominally 256 MiB > in size and with metadata dup, a pair of metadata chunks must be > allocated together, so 512 MiB would be needed, and of course even if the > 1 MiB could be allocated, it'd be ~1/2 MiB worth of metadata due to > metadata-dup and you're 300+ MiB into global reserve, so it wouldn't even > come close to fixing the problem). > > > Now normally, as mentioned in the ENOSPC discussion in the FAQ on the > wiki, temporarily adding (btrfs device add) another device of some GiB > (32 GiB should do reasonably well, 8 GiB may, a USB thumb drive of > suitable size can be used if necessary) and using the space it makes > available to do a balance (-dusage= incrementing from 0 to perhaps 30 to > 70 percent, higher numbers will take longer and may not work at first) in > ordered to combine partially used chunks and free enough space to then > remove (btrfs device remove) the temporarily added device. > > However, in your case the data usage is 488 of 508 GiB on a 512 GiB > device with space needed for several GiB of metadata as well, so while in > theory you could free up ~20 GiB of space that way and that should get > you out of the immediate bind, the filesystem will still be very close to > full, particularly after clearing out the global reserve usage, with > perhaps 16 GiB unallocated at ideal, ~97% used. And as any veteran > sysadmin or filesystem expert will tell you, filesystems in general like > 10-20% free in ordered to be able to "breath" or work most efficiently, > with btrfs being no exception, so while the above might get you out of > the immediate bind, it's unlikely to work for long. > > Which means once you're out of the immediate bind, you're still going to > need to free some space, one way or another, and that might not be as > simple as the words make it appear.
Yes, adding a temporary disk allowed me to fix it. Though, it wanted to write RAID1 metadata instead of DUP first, which further confused me. File system is being written to by a program that watches disk usage and deletes stuff/stops writing if too much is used. But it could not anticipate the jump from 20GiB to zero free. I have now set "metadata_ratio=8" to prevent that, and will lower it if it still becomes a problem. Perhaps it would be good to somehow show that "global reserve" belongs to metadata and show in btrfs fi usage/df that metadata is full if global reserve>=free metadata, so that future users are not as confused by this situation as I was. Regards, Martin Raiber
smime.p7s
Description: S/MIME Cryptographic Signature
