On Sun, Jun 23, 2019 at 10:14:34AM -0400, Zygo Blaxell wrote: > On Tue, Apr 23, 2019 at 07:06:51PM -0400, Zygo Blaxell wrote: > > I had a test filesystem that ran out of unallocated space, then ran > > out of metadata space during a snapshot delete, and forced readonly. > > The workload before the failure was a lot of rsync and bees dedupe > > combined with random snapshot creates and deletes. > > Had this happen again on a production filesystem, this time on 5.1.11, > and it happened during orphan inode cleanup instead of snapshot delete: > > [14303.076134][T20882] BTRFS: error (device dm-21) in > add_to_free_space_tree:1037: errno=-28 No space left > [14303.076144][T20882] BTRFS: error (device dm-21) in > __btrfs_free_extent:7196: errno=-28 No space left > [14303.076157][T20882] BTRFS: error (device dm-21) in > btrfs_run_delayed_refs:3008: errno=-28 No space left > [14303.076203][T20882] BTRFS error (device dm-21): Error removing > orphan entry, stopping orphan cleanup > [14303.076210][T20882] BTRFS error (device dm-21): could not do orphan > cleanup -22 > [14303.076281][T20882] BTRFS error (device dm-21): commit super ret -30 > [14303.357337][T20882] BTRFS error (device dm-21): open_ctree failed > > Same fix: I bumped the reserved size limit from 512M to 2G and mounted > normally. (OK, technically, I booted my old 5.0.21 kernel--but my 5.0.21 > kernel has the 2G reserved space patch below in it.)
It turns out that it was the boot with 5.0.21 that fixed this, not the increase reserved limit patch. This filesystem's global reserve is is only 1.71G (it's a 2x150G btrfs raid1), so raising the limit to 2G has no effect on 5.1.x. I was able to mount the filesystem with 5.0.21, but not with 5.1.14--even after successfully mounting with 5.0.21 and finishing orphan inode cleanup, mounting on 5.1.14 or 5.1.15 fails. I tried a few other things: balancing metadata (Size:19.04GiB, Used:7.54GiB) and mounting with 'nossd' (which somehow became 'ssd' after the fact?). Eventually I was able to mount the filesystem with kernel 5.1.15 and the 'nossd' option. I can now wait and see how long 5.1.15 lasts until the next similar event occurs. The full mount options are: rw,noatime,degraded,compress=zstd:3,ssd,flushoncommit,space_cache=v2,skip_balance (filesystem isn't degraded in this case, I just have that option in case a disk fails some day)
signature.asc
Description: PGP signature
