Re: btrfs balance did not progress after 12H, hang on reboot, btrfs check --repair kills the system still
On Mon, Jun 25, 2018 at 01:07:10PM -0400, Austin S. Hemmelgarn wrote: > > - mount -o recovery still hung > > - mount -o ro did not hang though > One tip here specifically, if you had to reboot during a balance and the FS > hangs when it mounts, try mounting with `-o skip_balance`. That should > pause the balance instead of resuming it on mount, at which point you should > also be able to cancel it without it hanging. Very good tip, I have this in all my mountpoints :) #LABEL=dshelf2 /mnt/btrfs_pool2 btrfs defaults,compress=lzo,skip_balance,noatime Marc -- "A mouse is a device used to point at the xterm you want to type in" - A.S.R. Microsoft is to operating systems what McDonalds is to gourmet cooking Home page: http://marc.merlins.org/ | PGP 7F55D5F27AAF9D08 -- To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: btrfs balance did not progress after 12H, hang on reboot, btrfs check --repair kills the system still
On 2018-06-25 12:07, Marc MERLIN wrote: On Tue, Jun 19, 2018 at 12:58:44PM -0400, Austin S. Hemmelgarn wrote: In your situation, I would run "btrfs pause ", wait to hear from a btrfs developer, and not use the volume whatsoever in the meantime. I would say this is probably good advice. I don't really know what's going on here myself actually, though it looks like the balance got stuck (the output hasn't changed for over 36 hours, unless you've got an insanely slow storage array, that's extremely unusual (it should only be moving at most 3GB of data per chunk)). I didn't hear from any developer, so I had to continue. - btrfs scrub cancel did not work (hang) - at reboot mounting the filesystem hung, even with 4.17, which is disappointing (it should not hang) - mount -o recovery still hung - mount -o ro did not hang though One tip here specifically, if you had to reboot during a balance and the FS hangs when it mounts, try mounting with `-o skip_balance`. That should pause the balance instead of resuming it on mount, at which point you should also be able to cancel it without it hanging. Sigh, why is my FS corrupted again? Anyway, back to btrfs check --repair and, it took all my 32GB of RAM on a system I can't add more RAM to, so I'm hosed. I'll note in passing (and it's not ok at all) that check --repair after a 20 to 30mn pause, takes all the kernel RAM more quickly than the system can OOM or log anything, and just deadlocks it. This is repeateable and totally not ok :( I'm now left with btrfs-progs git master, and lowmem which finally does a bit of repair. So far: gargamel:~# btrfs check --mode=lowmem --repair -p /dev/mapper/dshelf2 enabling repair mode WARNING: low-memory mode repair support is only partial Checking filesystem on /dev/mapper/dshelf2 UUID: 0f1a0c9f-4e54-4fa7-8736-fd50818ff73d Fixed 0 roots. ERROR: extent[84302495744, 69632] referencer count mismatch (root: 21872, owner: 374857, offset: 3407872) wanted: 3, have: 4 Created new chunk [18457780224000 1073741824] Delete backref in extent [84302495744 69632] ERROR: extent[84302495744, 69632] referencer count mismatch (root: 22911, owner: 374857, offset: 3407872) wanted: 3, have: 4 Delete backref in extent [84302495744 69632] ERROR: extent[125712527360, 12214272] referencer count mismatch (root: 21872, owner: 374857, offset: 114540544) wanted: 181, have: 240 Delete backref in extent [125712527360 12214272] ERROR: extent[125730848768, 5111808] referencer count mismatch (root: 21872, owner: 374857, offset: 126754816) wanted: 68, have: 115 Delete backref in extent [125730848768 5111808] ERROR: extent[125730848768, 5111808] referencer count mismatch (root: 22911, owner: 374857, offset: 126754816) wanted: 68, have: 115 Delete backref in extent [125730848768 5111808] ERROR: extent[125736914944, 6037504] referencer count mismatch (root: 21872, owner: 374857, offset: 131866624) wanted: 115, have: 143 Delete backref in extent [125736914944 6037504] ERROR: extent[125736914944, 6037504] referencer count mismatch (root: 22911, owner: 374857, offset: 131866624) wanted: 115, have: 143 Delete backref in extent [125736914944 6037504] ERROR: extent[129952120832, 20242432] referencer count mismatch (root: 21872, owner: 374857, offset: 148234240) wanted: 302, have: 431 Delete backref in extent [129952120832 20242432] ERROR: extent[129952120832, 20242432] referencer count mismatch (root: 22911, owner: 374857, offset: 148234240) wanted: 356, have: 433 Delete backref in extent [129952120832 20242432] ERROR: extent[134925357056, 11829248] referencer count mismatch (root: 21872, owner: 374857, offset: 180371456) wanted: 161, have: 240 Delete backref in extent [134925357056 11829248] ERROR: extent[134925357056, 11829248] referencer count mismatch (root: 22911, owner: 374857, offset: 180371456) wanted: 162, have: 240 Delete backref in extent [134925357056 11829248] ERROR: extent[147895111680, 12345344] referencer count mismatch (root: 21872, owner: 374857, offset: 192200704) wanted: 170, have: 249 Delete backref in extent [147895111680 12345344] ERROR: extent[147895111680, 12345344] referencer count mismatch (root: 22911, owner: 374857, offset: 192200704) wanted: 172, have: 251 Delete backref in extent [147895111680 12345344] ERROR: extent[150850146304, 17522688] referencer count mismatch (root: 21872, owner: 374857, offset: 217653248) wanted: 348, have: 418 Delete backref in extent [150850146304 17522688] ERROR: extent[156909494272, 55320576] referencer count mismatch (root: 22911, owner: 374857, offset: 235175936) wanted: 555, have: 1449 Deleted root 2 item[156909494272, 178, 5476627808561673095] ERROR: extent[156909494272, 55320576] referencer count mismatch (root: 21872, owner: 374857, offset: 235175936) wanted: 556, have: 1452 Deleted root 2 item[156909494272, 178, 7338474132555182983] At the rate it's going, it'll probably take days though, it's already been 36H Marc -- To unsubscribe from this list: send the line "unsubscribe
Re: btrfs balance did not progress after 12H, hang on reboot, btrfs check --repair kills the system still
On Mon, Jun 25, 2018 at 06:24:37PM +0200, Hans van Kranenburg wrote: > >> output hasn't changed for over 36 hours, unless you've got an insanely slow > >> storage array, that's extremely unusual (it should only be moving at most > >> 3GB of data per chunk)). > > > > I didn't hear from any developer, so I had to continue. > > - btrfs scrub cancel did not work (hang) > > Did you mean balance cancel? It waits until the current block group is > finished. Yes, I meant that, thanks for correcting me. And you're correct that because it was hung, cancel wasn't going to go anywhere. At least my filesystem was still working at the time (as in IO was going on just fine) > > - at reboot mounting the filesystem hung, even with 4.17, which is > > disappointing (it should not hang) > > - mount -o recovery still hung > > - mount -o ro did not hang though > > > > Sigh, why is my FS corrupted again? > > Again? Do you think balance is corrupting the filesystem? Or have there > been previous btrfs check --repair operations which made smaller > problems bigger in the past? Honestly, I don't fully remember at this point, I keep notes, but not detailled enough and it's been a little while. I know I've had to delete/recreate this filesystem twice already over the last years, but I'm not fully certain I remember when this one was last wiped. Yes, I do run balance along with scrub once a month: btrfs balance start -musage=0 -v $mountpoint 2>&1 | grep -Ev "$FILTER" # After metadata, let's do data: btrfs balance start -dusage=0 -v $mountpoint 2>&1 | grep -Ev "$FILTER" btrfs balance start -dusage=20 -v $mountpoint 2>&1 | grep -Ev "$FILTER" echo btrfs scrub start -Bd $mountpoint ionice -c 3 nice -10 btrfs scrub start -Bd $mountpoint Hard to say if balance has damaged my filesystem over time, but it's definitely possible. > Am I right to interpret the messages below, and see that you have > extents that are referenced hundreds of times? I'm not certain, but it's a backup server with many blocks that are the same, so it could be some COW stuff, even if I didn't run any dedupe commands myself. > Is there heavy snapshotting or deduping going on in this filesystem? If > so, it's not surprising balance will get a hard time moving extents > around, since it has to update all of the metadata for each extent again > in hundreds of places. There is some snapshotting, but maybe around 20 or so per subvolume, not hundreds. > Did you investigate what balance was doing if it takes long? Is is using > cpu all the time, or is it reading from disk slowly (random reads) or is > it writing to disk all the time at full speed? I couldn't see what it was doing, but it's running in the kernel, is it not? (or can you just strace the user space command?) Either way, it's too late for that now, and given that it didn't make progress of a single block in 36H, I'm assuming it was well deadlocked. Thanks for the reply. Marc -- "A mouse is a device used to point at the xterm you want to type in" - A.S.R. Microsoft is to operating systems what McDonalds is to gourmet cooking Home page: http://marc.merlins.org/ | PGP 7F55D5F27AAF9D08 -- To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: btrfs balance did not progress after 12H, hang on reboot, btrfs check --repair kills the system still
On 06/25/2018 06:07 PM, Marc MERLIN wrote: > On Tue, Jun 19, 2018 at 12:58:44PM -0400, Austin S. Hemmelgarn wrote: >>> In your situation, I would run "btrfs pause ", wait to hear from >>> a btrfs developer, and not use the volume whatsoever in the meantime. >> I would say this is probably good advice. I don't really know what's going >> on here myself actually, though it looks like the balance got stuck (the >> output hasn't changed for over 36 hours, unless you've got an insanely slow >> storage array, that's extremely unusual (it should only be moving at most >> 3GB of data per chunk)). > > I didn't hear from any developer, so I had to continue. > - btrfs scrub cancel did not work (hang) Did you mean balance cancel? It waits until the current block group is finished. > - at reboot mounting the filesystem hung, even with 4.17, which is > disappointing (it should not hang) > - mount -o recovery still hung > - mount -o ro did not hang though > > Sigh, why is my FS corrupted again? Again? Do you think balance is corrupting the filesystem? Or have there been previous btrfs check --repair operations which made smaller problems bigger in the past? > Anyway, back to > btrfs check --repair > and, it took all my 32GB of RAM on a system I can't add more RAM to, so > I'm hosed. I'll note in passing (and it's not ok at all) that check > --repair after a 20 to 30mn pause, takes all the kernel RAM more quickly > than the system can OOM or log anything, and just deadlocks it. > This is repeateable and totally not ok :( > > I'm now left with btrfs-progs git master, and lowmem which finally does > a bit of repair. > So far: > gargamel:~# btrfs check --mode=lowmem --repair -p /dev/mapper/dshelf2 > enabling repair mode > WARNING: low-memory mode repair support is only partial > Checking filesystem on /dev/mapper/dshelf2 > UUID: 0f1a0c9f-4e54-4fa7-8736-fd50818ff73d > Fixed 0 roots. Am I right to interpret the messages below, and see that you have extents that are referenced hundreds of times? Is there heavy snapshotting or deduping going on in this filesystem? If so, it's not surprising balance will get a hard time moving extents around, since it has to update all of the metadata for each extent again in hundreds of places. Did you investigate what balance was doing if it takes long? Is is using cpu all the time, or is it reading from disk slowly (random reads) or is it writing to disk all the time at full speed? K > ERROR: extent[84302495744, 69632] referencer count mismatch (root: 21872, > owner: 374857, offset: 3407872) wanted: 3, have: 4 > Created new chunk [18457780224000 1073741824] > Delete backref in extent [84302495744 69632] > ERROR: extent[84302495744, 69632] referencer count mismatch (root: 22911, > owner: 374857, offset: 3407872) wanted: 3, have: 4 > Delete backref in extent [84302495744 69632] > ERROR: extent[125712527360, 12214272] referencer count mismatch (root: 21872, > owner: 374857, offset: 114540544) wanted: 181, have: 240 > Delete backref in extent [125712527360 12214272] > ERROR: extent[125730848768, 5111808] referencer count mismatch (root: 21872, > owner: 374857, offset: 126754816) wanted: 68, have: 115 > Delete backref in extent [125730848768 5111808] > ERROR: extent[125730848768, 5111808] referencer count mismatch (root: 22911, > owner: 374857, offset: 126754816) wanted: 68, have: 115 > Delete backref in extent [125730848768 5111808] > ERROR: extent[125736914944, 6037504] referencer count mismatch (root: 21872, > owner: 374857, offset: 131866624) wanted: 115, have: 143 > Delete backref in extent [125736914944 6037504] > ERROR: extent[125736914944, 6037504] referencer count mismatch (root: 22911, > owner: 374857, offset: 131866624) wanted: 115, have: 143 > Delete backref in extent [125736914944 6037504] > ERROR: extent[129952120832, 20242432] referencer count mismatch (root: 21872, > owner: 374857, offset: 148234240) wanted: 302, have: 431 > Delete backref in extent [129952120832 20242432] > ERROR: extent[129952120832, 20242432] referencer count mismatch (root: 22911, > owner: 374857, offset: 148234240) wanted: 356, have: 433 > Delete backref in extent [129952120832 20242432] > ERROR: extent[134925357056, 11829248] referencer count mismatch (root: 21872, > owner: 374857, offset: 180371456) wanted: 161, have: 240 > Delete backref in extent [134925357056 11829248] > ERROR: extent[134925357056, 11829248] referencer count mismatch (root: 22911, > owner: 374857, offset: 180371456) wanted: 162, have: 240 > Delete backref in extent [134925357056 11829248] > ERROR: extent[147895111680, 12345344] referencer count mismatch (root: 21872, > owner: 374857, offset: 192200704) wanted: 170, have: 249 > Delete backref in extent [147895111680 12345344] > ERROR: extent[147895111680, 12345344] referencer count mismatch (root: 22911, > owner: 374857, offset: 192200704) wanted: 172, have: 251 > Delete backref in extent [147895111680 12345344] > ERROR: extent[150850146304,
Re: btrfs balance did not progress after 12H, hang on reboot, btrfs check --repair kills the system still
On Tue, Jun 19, 2018 at 12:58:44PM -0400, Austin S. Hemmelgarn wrote: > > In your situation, I would run "btrfs pause ", wait to hear from > > a btrfs developer, and not use the volume whatsoever in the meantime. > I would say this is probably good advice. I don't really know what's going > on here myself actually, though it looks like the balance got stuck (the > output hasn't changed for over 36 hours, unless you've got an insanely slow > storage array, that's extremely unusual (it should only be moving at most > 3GB of data per chunk)). I didn't hear from any developer, so I had to continue. - btrfs scrub cancel did not work (hang) - at reboot mounting the filesystem hung, even with 4.17, which is disappointing (it should not hang) - mount -o recovery still hung - mount -o ro did not hang though Sigh, why is my FS corrupted again? Anyway, back to btrfs check --repair and, it took all my 32GB of RAM on a system I can't add more RAM to, so I'm hosed. I'll note in passing (and it's not ok at all) that check --repair after a 20 to 30mn pause, takes all the kernel RAM more quickly than the system can OOM or log anything, and just deadlocks it. This is repeateable and totally not ok :( I'm now left with btrfs-progs git master, and lowmem which finally does a bit of repair. So far: gargamel:~# btrfs check --mode=lowmem --repair -p /dev/mapper/dshelf2 enabling repair mode WARNING: low-memory mode repair support is only partial Checking filesystem on /dev/mapper/dshelf2 UUID: 0f1a0c9f-4e54-4fa7-8736-fd50818ff73d Fixed 0 roots. ERROR: extent[84302495744, 69632] referencer count mismatch (root: 21872, owner: 374857, offset: 3407872) wanted: 3, have: 4 Created new chunk [18457780224000 1073741824] Delete backref in extent [84302495744 69632] ERROR: extent[84302495744, 69632] referencer count mismatch (root: 22911, owner: 374857, offset: 3407872) wanted: 3, have: 4 Delete backref in extent [84302495744 69632] ERROR: extent[125712527360, 12214272] referencer count mismatch (root: 21872, owner: 374857, offset: 114540544) wanted: 181, have: 240 Delete backref in extent [125712527360 12214272] ERROR: extent[125730848768, 5111808] referencer count mismatch (root: 21872, owner: 374857, offset: 126754816) wanted: 68, have: 115 Delete backref in extent [125730848768 5111808] ERROR: extent[125730848768, 5111808] referencer count mismatch (root: 22911, owner: 374857, offset: 126754816) wanted: 68, have: 115 Delete backref in extent [125730848768 5111808] ERROR: extent[125736914944, 6037504] referencer count mismatch (root: 21872, owner: 374857, offset: 131866624) wanted: 115, have: 143 Delete backref in extent [125736914944 6037504] ERROR: extent[125736914944, 6037504] referencer count mismatch (root: 22911, owner: 374857, offset: 131866624) wanted: 115, have: 143 Delete backref in extent [125736914944 6037504] ERROR: extent[129952120832, 20242432] referencer count mismatch (root: 21872, owner: 374857, offset: 148234240) wanted: 302, have: 431 Delete backref in extent [129952120832 20242432] ERROR: extent[129952120832, 20242432] referencer count mismatch (root: 22911, owner: 374857, offset: 148234240) wanted: 356, have: 433 Delete backref in extent [129952120832 20242432] ERROR: extent[134925357056, 11829248] referencer count mismatch (root: 21872, owner: 374857, offset: 180371456) wanted: 161, have: 240 Delete backref in extent [134925357056 11829248] ERROR: extent[134925357056, 11829248] referencer count mismatch (root: 22911, owner: 374857, offset: 180371456) wanted: 162, have: 240 Delete backref in extent [134925357056 11829248] ERROR: extent[147895111680, 12345344] referencer count mismatch (root: 21872, owner: 374857, offset: 192200704) wanted: 170, have: 249 Delete backref in extent [147895111680 12345344] ERROR: extent[147895111680, 12345344] referencer count mismatch (root: 22911, owner: 374857, offset: 192200704) wanted: 172, have: 251 Delete backref in extent [147895111680 12345344] ERROR: extent[150850146304, 17522688] referencer count mismatch (root: 21872, owner: 374857, offset: 217653248) wanted: 348, have: 418 Delete backref in extent [150850146304 17522688] ERROR: extent[156909494272, 55320576] referencer count mismatch (root: 22911, owner: 374857, offset: 235175936) wanted: 555, have: 1449 Deleted root 2 item[156909494272, 178, 5476627808561673095] ERROR: extent[156909494272, 55320576] referencer count mismatch (root: 21872, owner: 374857, offset: 235175936) wanted: 556, have: 1452 Deleted root 2 item[156909494272, 178, 7338474132555182983] At the rate it's going, it'll probably take days though, it's already been 36H Marc -- "A mouse is a device used to point at the xterm you want to type in" - A.S.R. Microsoft is to operating systems what McDonalds is to gourmet cooking Home page: http://marc.merlins.org/ | PGP 7F55D5F27AAF9D08 -- To unsubscribe from this list: send the line "unsubscribe
