>> So I was back to a 4-drive raid1, with 3x 6 TB drives and 1x 8 TB drive >> (though that 8 TB drive had very little data on it). Then I tried to >> "remove" (without "-r" this time) the 6 TB drive with the least amount >> of data on it (one had 4.0 TiB, where the other two had 5.45 TiB each). >> This failed after a few minutes because of "no space left on device". >> >> […] >> >> For now I avoided that by removing one of the other two (rather full) 6 >> TB drives at random, and this has been going on for the last 20 hours or >> so. Thanks to running it in a screen I can check the progress this time >> around, and it's doing its thing at ~41 MiB/s, or ~7 hours per TiB, on >> average. > > The ENOSPC errors are likely due to the fact that the raid1 allocator > needs _two_ devices with free space. If your 6T devices get too full, > even if the 8T device is nearly empty, you'll run into ENOSPC, because > you have just one device with unallocated space and the raid1 allocator > needs two.
I see, now this makes total sense. Two of the 6 TB drives were almost completely full, at 5.45 used of 5.46 TiB capacity. Note to self: Maybe I should start using the --si option to make such a condition more obvious when mentally comparing to the advertised capacity of 6 TB (when comparing to to 5.46 TiB would have been correct) "remove"-ing one of these almost-full-drives did finish successfully, and a "replace" of the 3rd 6 TB drive onto a second 8 TB drive is currently in progress (at high speed). > btrfs device usage should help diagnose this condition, with btrfs > filesystem show also showing the individual device space allocation but > not as much other information as usage will. I had mostly been using btrfs filesystem usage, thanks for the reminder about device usage, which is easier to read in this case. > If you run into this, you may just have to do the hardware yank and > replace-missing thing again, yanking a 6T and replacing with an 8T. > Don't forget the resize. That should leave you with two devices with > free space and thus hopefully allow normal raid1 reallocation with a > device remove again. Good to know. For now this doesn't seem necessary, even the other drive that was almost completely full before looks much better now at 4.8/5.46 TiB or 5.27/6.0 TB in --si :), as some data was moved to the first 8 TB drive during the last "remove". So far everything is looking good, thanks very much for the help everyone. -- To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
