On Wed, Mar 3, 2021 at 1:42 PM Josef Bacik <[email protected]> wrote: > > On 2/24/21 10:47 PM, Neal Gompa wrote: > > On Wed, Feb 24, 2021 at 10:44 AM Josef Bacik <[email protected]> wrote: > >> > >> On 2/24/21 9:23 AM, Neal Gompa wrote: > >>> On Tue, Feb 23, 2021 at 10:05 AM Josef Bacik <[email protected]> wrote: > >>>> > >>>> On 2/22/21 11:03 PM, Neal Gompa wrote: > >>>>> On Mon, Feb 22, 2021 at 2:34 PM Josef Bacik <[email protected]> > >>>>> wrote: > >>>>>> > >>>>>> On 2/21/21 1:27 PM, Neal Gompa wrote: > >>>>>>> On Wed, Feb 17, 2021 at 11:44 AM Josef Bacik <[email protected]> > >>>>>>> wrote: > >>>>>>>> > >>>>>>>> On 2/17/21 11:29 AM, Neal Gompa wrote: > >>>>>>>>> On Wed, Feb 17, 2021 at 9:59 AM Josef Bacik <[email protected]> > >>>>>>>>> wrote: > >>>>>>>>>> > >>>>>>>>>> On 2/17/21 9:50 AM, Neal Gompa wrote: > >>>>>>>>>>> On Wed, Feb 17, 2021 at 9:36 AM Josef Bacik > >>>>>>>>>>> <[email protected]> wrote: > >>>>>>>>>>>> > >>>>>>>>>>>> On 2/16/21 9:05 PM, Neal Gompa wrote: > >>>>>>>>>>>>> On Tue, Feb 16, 2021 at 4:24 PM Josef Bacik > >>>>>>>>>>>>> <[email protected]> wrote: > >>>>>>>>>>>>>> > >>>>>>>>>>>>>> On 2/16/21 3:29 PM, Neal Gompa wrote: > >>>>>>>>>>>>>>> On Tue, Feb 16, 2021 at 1:11 PM Josef Bacik > >>>>>>>>>>>>>>> <[email protected]> wrote: > >>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>> On 2/16/21 11:27 AM, Neal Gompa wrote: > >>>>>>>>>>>>>>>>> On Tue, Feb 16, 2021 at 10:19 AM Josef Bacik > >>>>>>>>>>>>>>>>> <[email protected]> wrote: > >>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>> On 2/14/21 3:25 PM, Neal Gompa wrote: > >>>>>>>>>>>>>>>>>>> Hey all, > >>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>> So one of my main computers recently had a disk > >>>>>>>>>>>>>>>>>>> controller failure > >>>>>>>>>>>>>>>>>>> that caused my machine to freeze. After rebooting, Btrfs > >>>>>>>>>>>>>>>>>>> refuses to > >>>>>>>>>>>>>>>>>>> mount. I tried to do a mount and the following errors > >>>>>>>>>>>>>>>>>>> show up in the > >>>>>>>>>>>>>>>>>>> journal: > >>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>> Feb 14 15:20:49 localhost-live kernel: BTRFS info > >>>>>>>>>>>>>>>>>>>> (device sda3): disk space caching is enabled > >>>>>>>>>>>>>>>>>>>> Feb 14 15:20:49 localhost-live kernel: BTRFS info > >>>>>>>>>>>>>>>>>>>> (device sda3): has skinny extents > >>>>>>>>>>>>>>>>>>>> Feb 14 15:20:49 localhost-live kernel: BTRFS critical > >>>>>>>>>>>>>>>>>>>> (device sda3): corrupt leaf: root=401 block=796082176 > >>>>>>>>>>>>>>>>>>>> slot=15 ino=203657, invalid inode transid: has 888896 > >>>>>>>>>>>>>>>>>>>> expect [0, 888895] > >>>>>>>>>>>>>>>>>>>> Feb 14 15:20:49 localhost-live kernel: BTRFS error > >>>>>>>>>>>>>>>>>>>> (device sda3): block=796082176 read time tree block > >>>>>>>>>>>>>>>>>>>> corruption detected > >>>>>>>>>>>>>>>>>>>> Feb 14 15:20:49 localhost-live kernel: BTRFS critical > >>>>>>>>>>>>>>>>>>>> (device sda3): corrupt leaf: root=401 block=796082176 > >>>>>>>>>>>>>>>>>>>> slot=15 ino=203657, invalid inode transid: has 888896 > >>>>>>>>>>>>>>>>>>>> expect [0, 888895] > >>>>>>>>>>>>>>>>>>>> Feb 14 15:20:49 localhost-live kernel: BTRFS error > >>>>>>>>>>>>>>>>>>>> (device sda3): block=796082176 read time tree block > >>>>>>>>>>>>>>>>>>>> corruption detected > >>>>>>>>>>>>>>>>>>>> Feb 14 15:20:49 localhost-live kernel: BTRFS warning > >>>>>>>>>>>>>>>>>>>> (device sda3): couldn't read tree root > >>>>>>>>>>>>>>>>>>>> Feb 14 15:20:49 localhost-live kernel: BTRFS error > >>>>>>>>>>>>>>>>>>>> (device sda3): open_ctree failed > >>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>> I've tried to do -o recovery,ro mount and get the same > >>>>>>>>>>>>>>>>>>> issue. I can't > >>>>>>>>>>>>>>>>>>> seem to find any reasonably good information on how to do > >>>>>>>>>>>>>>>>>>> recovery in > >>>>>>>>>>>>>>>>>>> this scenario, even to just recover enough to copy data > >>>>>>>>>>>>>>>>>>> off. > >>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>> I'm on Fedora 33, the system was on Linux kernel version > >>>>>>>>>>>>>>>>>>> 5.9.16 and > >>>>>>>>>>>>>>>>>>> the Fedora 33 live ISO I'm using has Linux kernel version > >>>>>>>>>>>>>>>>>>> 5.10.14. I'm > >>>>>>>>>>>>>>>>>>> using btrfs-progs v5.10. > >>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>> Can anyone help? > >>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>> Can you try > >>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>> btrfs check --clear-space-cache v1 /dev/whatever > >>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>> That should fix the inode generation thing so it's sane, > >>>>>>>>>>>>>>>>>> and then the tree > >>>>>>>>>>>>>>>>>> checker will allow the fs to be read, hopefully. If not > >>>>>>>>>>>>>>>>>> we can work out some > >>>>>>>>>>>>>>>>>> other magic. Thanks, > >>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>> Josef > >>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>> I got the same error as I did with btrfs-check --readonly... > >>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>> Oh lovely, what does btrfs check --readonly --backup do? > >>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>> > >>>>>>>>>>>>>>> No dice... > >>>>>>>>>>>>>>> > >>>>>>>>>>>>>>> # btrfs check --readonly --backup /dev/sda3 > >>>>>>>>>>>>>>>> Opening filesystem to check... > >>>>>>>>>>>>>>>> parent transid verify failed on 791281664 wanted 888893 > >>>>>>>>>>>>>>>> found 888895 > >>>>>>>>>>>>>>>> parent transid verify failed on 791281664 wanted 888893 > >>>>>>>>>>>>>>>> found 888895 > >>>>>>>>>>>>>>>> parent transid verify failed on 791281664 wanted 888893 > >>>>>>>>>>>>>>>> found 888895 > >>>>>>>>>>>>>> > >>>>>>>>>>>>>> Hey look the block we're looking for, I wrote you some magic, > >>>>>>>>>>>>>> just pull > >>>>>>>>>>>>>> > >>>>>>>>>>>>>> https://github.com/josefbacik/btrfs-progs/tree/for-neal > >>>>>>>>>>>>>> > >>>>>>>>>>>>>> build, and then run > >>>>>>>>>>>>>> > >>>>>>>>>>>>>> btrfs-neal-magic /dev/sda3 791281664 888895 > >>>>>>>>>>>>>> > >>>>>>>>>>>>>> This will force us to point at the old root with (hopefully) > >>>>>>>>>>>>>> the right bytenr > >>>>>>>>>>>>>> and gen, and then hopefully you'll be able to recover from > >>>>>>>>>>>>>> there. This is kind > >>>>>>>>>>>>>> of saucy, so yolo, but I can undo it if it makes things worse. > >>>>>>>>>>>>>> Thanks, > >>>>>>>>>>>>>> > >>>>>>>>>>>>> > >>>>>>>>>>>>> # btrfs check --readonly /dev/sda3 > >>>>>>>>>>>>>> Opening filesystem to check... > >>>>>>>>>>>>>> ERROR: could not setup extent tree > >>>>>>>>>>>>>> ERROR: cannot open file system > >>>>>>>>>>>>> # btrfs check --clear-space-cache v1 /dev/sda3 > >>>>>>>>>>>>>> Opening filesystem to check... > >>>>>>>>>>>>>> ERROR: could not setup extent tree > >>>>>>>>>>>>>> ERROR: cannot open file system > >>>>>>>>>>>>> > >>>>>>>>>>>>> It's better, but still no dice... :( > >>>>>>>>>>>>> > >>>>>>>>>>>>> > >>>>>>>>>>>> > >>>>>>>>>>>> Hmm it's not telling us what's wrong with the extent tree, which > >>>>>>>>>>>> is annoying. > >>>>>>>>>>>> Does mount -o rescue=all,ro work now that the root tree is > >>>>>>>>>>>> normal? Thanks, > >>>>>>>>>>>> > >>>>>>>>>>> > >>>>>>>>>>> Nope, I see this in the journal: > >>>>>>>>>>> > >>>>>>>>>>>> Feb 17 09:49:40 localhost-live kernel: BTRFS info (device sda3): > >>>>>>>>>>>> enabling all of the rescue options > >>>>>>>>>>>> Feb 17 09:49:40 localhost-live kernel: BTRFS info (device sda3): > >>>>>>>>>>>> ignoring data csums > >>>>>>>>>>>> Feb 17 09:49:40 localhost-live kernel: BTRFS info (device sda3): > >>>>>>>>>>>> ignoring bad roots > >>>>>>>>>>>> Feb 17 09:49:40 localhost-live kernel: BTRFS info (device sda3): > >>>>>>>>>>>> disabling log replay at mount time > >>>>>>>>>>>> Feb 17 09:49:40 localhost-live kernel: BTRFS info (device sda3): > >>>>>>>>>>>> disk space caching is enabled > >>>>>>>>>>>> Feb 17 09:49:40 localhost-live kernel: BTRFS info (device sda3): > >>>>>>>>>>>> has skinny extents > >>>>>>>>>>>> Feb 17 09:49:40 localhost-live kernel: BTRFS error (device > >>>>>>>>>>>> sda3): tree level mismatch detected, bytenr=791281664 level > >>>>>>>>>>>> expected=1 has=2 > >>>>>>>>>>>> Feb 17 09:49:40 localhost-live kernel: BTRFS error (device > >>>>>>>>>>>> sda3): tree level mismatch detected, bytenr=791281664 level > >>>>>>>>>>>> expected=1 has=2 > >>>>>>>>>>>> Feb 17 09:49:40 localhost-live kernel: BTRFS warning (device > >>>>>>>>>>>> sda3): couldn't read tree root > >>>>>>>>>>>> Feb 17 09:49:40 localhost-live kernel: BTRFS error (device > >>>>>>>>>>>> sda3): open_ctree failed > >>>>>>>>>>> > >>>>>>>>>>> > >>>>>>>>>> > >>>>>>>>>> Ok git pull for-neal, rebuild, then run > >>>>>>>>>> > >>>>>>>>>> btrfs-neal-magic /dev/sda3 791281664 888895 2 > >>>>>>>>>> > >>>>>>>>>> I thought of this yesterday but in my head was like "naaahhhh, > >>>>>>>>>> whats the chances > >>>>>>>>>> that the level doesn't match??". Thanks, > >>>>>>>>>> > >>>>>>>>> > >>>>>>>>> Tried rescue mount again after running that and got a stack trace in > >>>>>>>>> the kernel, detailed in the following attached log. > >>>>>>>> > >>>>>>>> Huh I wonder how I didn't hit this when testing, I must have only > >>>>>>>> tested with > >>>>>>>> zero'ing the extent root and the csum root. You're going to have to > >>>>>>>> build a > >>>>>>>> kernel with a fix for this > >>>>>>>> > >>>>>>>> https://paste.centos.org/view/7b48aaea > >>>>>>>> > >>>>>>>> and see if that gets you further. Thanks, > >>>>>>>> > >>>>>>> > >>>>>>> I built a kernel build as an RPM with your patch[1] and tried it. > >>>>>>> > >>>>>>> [root@fedora ~]# mount -t btrfs -o rescue=all,ro /dev/sdb3 /mnt > >>>>>>> Killed > >>>>>>> > >>>>>>> The log from the journal is attached. > >>>>>> > >>>>>> > >>>>>> Ahh crud my bad, this should do it > >>>>>> > >>>>>> https://paste.centos.org/view/ac2e61ef > >>>>>> > >>>>> > >>>>> Patch doesn't apply (note it is patch 667 below): > >>>> > >>>> Ah sorry, should have just sent you an iterative patch. You can take > >>>> the above > >>>> patch and just delete the hunk from volumes.c as you already have that > >>>> applied > >>>> and then it'll work. Thanks, > >>>> > >>> > >>> Failed with a weird error...? > >>> > >>> [root@fedora ~]# mount -t btrfs -o rescue=all,ro /dev/sda3 /mnt > >>> mount: /mnt: mount(2) system call failed: No such file or directory. > >>> > >>> Journal log with traceback attached. > >> > >> Last one maybe? > >> > >> https://paste.centos.org/view/80edd6fd > >> > > > > Similar weird failure: > > > > [root@fedora ~]# mount -t btrfs -o rescue=all,ro /dev/sdb3 /mnt > > mount: /mnt: mount(2) system call failed: No such file or directory. > > > > No crash in the journal this time, though: > > > >> Feb 24 22:43:19 fedora kernel: BTRFS info (device sdb3): enabling all of > >> the rescue options > >> Feb 24 22:43:19 fedora kernel: BTRFS info (device sdb3): ignoring data > >> csums > >> Feb 24 22:43:19 fedora kernel: BTRFS info (device sdb3): ignoring bad roots > >> Feb 24 22:43:19 fedora kernel: BTRFS info (device sdb3): disabling log > >> replay at mount time > >> Feb 24 22:43:19 fedora kernel: BTRFS info (device sdb3): disk space > >> caching is enabled > >> Feb 24 22:43:19 fedora kernel: BTRFS info (device sdb3): has skinny extents > >> Feb 24 22:43:19 fedora kernel: BTRFS warning (device sdb3): failed to read > >> fs tree: -2 > >> Feb 24 22:43:19 fedora kernel: BTRFS error (device sdb3): open_ctree failed > > > > > > Sorry Neal, you replied when I was in the middle of something and promptly > forgot about it. I figured the fs root was fine, can you do the following so > I > can figure out from the error messages what might be wrong > > btrfs check --readonly > btrfs restore -D > btrfs restore -l >
It didn't work.. Here's the output: [root@fedora ~]# btrfs check --readonly /dev/sdb3 Opening filesystem to check... ERROR: could not setup extent tree ERROR: cannot open file system [root@fedora ~]# btrfs restore -D /dev/sdb3 /mnt WARNING: could not setup extent tree, skipping it Couldn't setup device tree Could not open root, trying backup super parent transid verify failed on 796082176 wanted 888894 found 888896 parent transid verify failed on 796082176 wanted 888894 found 888896 parent transid verify failed on 796082176 wanted 888894 found 888896 Ignoring transid failure WARNING: could not setup extent tree, skipping it Couldn't setup device tree Could not open root, trying backup super ERROR: superblock bytenr 274877906944 is larger than device size 263132807168 Could not open root, trying backup super [root@fedora ~]# btrfs restore -l /dev/sdb3 /mnt WARNING: could not setup extent tree, skipping it Couldn't setup device tree Could not open root, trying backup super parent transid verify failed on 796082176 wanted 888894 found 888896 parent transid verify failed on 796082176 wanted 888894 found 888896 parent transid verify failed on 796082176 wanted 888894 found 888896 Ignoring transid failure WARNING: could not setup extent tree, skipping it Couldn't setup device tree Could not open root, trying backup super ERROR: superblock bytenr 274877906944 is larger than device size 263132807168 Could not open root, trying backup super -- 真実はいつも一つ！/ Always, there's only one truth!
