Hi Qu, The tool is still running and the log file is now ~300mb. I guess it shouldn't normally take this long.. Is there anything else worth trying?
Kind regards Michael On 2 May 2018 at 06:29, Michael Wade <spikew...@gmail.com> wrote: > Thanks Qu, > > I actually aborted the run with the old btrfs tools once I saw its > output. The new btrfs tools is still running and has produced a log > file of ~85mb filled with that content so far. > > Kind regards > Michael > > On 2 May 2018 at 02:31, Qu Wenruo <quwenruo.bt...@gmx.com> wrote: >> >> >> On 2018年05月01日 23:50, Michael Wade wrote: >>> Hi Qu, >>> >>> Oh dear that is not good news! >>> >>> I have been running the find root command since yesterday but it only >>> seems to be only be outputting the following message: >>> >>> ERROR: tree block bytenr 0 is not aligned to sectorsize 4096 >> >> It's mostly fine, as find-root will go through all tree blocks and try >> to read them as tree blocks. >> Although btrfs-find-root will suppress csum error output, but such basic >> tree validation check is not suppressed, thus you get such message. >> >>> ERROR: tree block bytenr 0 is not aligned to sectorsize 4096 >>> ERROR: tree block bytenr 0 is not aligned to sectorsize 4096 >>> ERROR: tree block bytenr 0 is not aligned to sectorsize 4096 >>> ERROR: tree block bytenr 0 is not aligned to sectorsize 4096 >>> ERROR: tree block bytenr 0 is not aligned to sectorsize 4096 >>> ERROR: tree block bytenr 0 is not aligned to sectorsize 4096 >>> ERROR: tree block bytenr 0 is not aligned to sectorsize 4096 >>> ERROR: tree block bytenr 0 is not aligned to sectorsize 4096 >>> ERROR: tree block bytenr 0 is not aligned to sectorsize 4096 >>> >>> I tried with the latest btrfs tools compiled from source and the ones >>> I have installed with the same result. Is there a CLI utility I could >>> use to determine if the log contains any other content? >> >> Did it report any useful info at the end? >> >> Thanks, >> Qu >> >>> >>> Kind regards >>> Michael >>> >>> >>> On 30 April 2018 at 04:02, Qu Wenruo <quwenruo.bt...@gmx.com> wrote: >>>> >>>> >>>> On 2018年04月29日 22:08, Michael Wade wrote: >>>>> Hi Qu, >>>>> >>>>> Got this error message: >>>>> >>>>> ./btrfs inspect dump-tree -b 20800943685632 /dev/md127 >>>>> btrfs-progs v4.16.1 >>>>> bytenr mismatch, want=20800943685632, have=3118598835113619663 >>>>> ERROR: cannot read chunk root >>>>> ERROR: unable to open /dev/md127 >>>>> >>>>> I have attached the dumps for: >>>>> >>>>> dd if=/dev/md127 of=/tmp/chunk_root.copy1 bs=1 count=32K skip=266325721088 >>>>> dd if=/dev/md127 of=/tmp/chunk_root.copy2 bs=1 count=32K skip=266359275520 >>>> >>>> Unfortunately, both dumps are corrupted and contain mostly garbage. >>>> I think it's the underlying stack (mdraid) has something wrong or failed >>>> to recover its data. >>>> >>>> This means your last chance will be btrfs-find-root. >>>> >>>> Please try: >>>> # btrfs-find-root -o 3 <device> >>>> >>>> And provide all the output. >>>> >>>> But please keep in mind, chunk root is a critical tree, and so far it's >>>> already heavily damaged. >>>> Although I could still continue try to recover, there is pretty low >>>> chance now. >>>> >>>> Thanks, >>>> Qu >>>>> >>>>> Kind regards >>>>> Michael >>>>> >>>>> >>>>> On 29 April 2018 at 10:33, Qu Wenruo <quwenruo.bt...@gmx.com> wrote: >>>>>> >>>>>> >>>>>> On 2018年04月29日 16:59, Michael Wade wrote: >>>>>>> Ok, will it be possible for me to install the new version of the tools >>>>>>> on my current kernel without overriding the existing install? Hesitant >>>>>>> to update kernel/btrfs as it might break the ReadyNAS interface / >>>>>>> future firmware upgrades. >>>>>>> >>>>>>> Perhaps I could grab this: >>>>>>> https://github.com/kdave/btrfs-progs/releases/tag/v4.16.1 and >>>>>>> hopefully build from source and then run the binaries directly? >>>>>> >>>>>> Of course, that's how most of us test btrfs-progs builds. >>>>>> >>>>>> Thanks, >>>>>> Qu >>>>>> >>>>>>> >>>>>>> Kind regards >>>>>>> >>>>>>> On 29 April 2018 at 09:33, Qu Wenruo <quwenruo.bt...@gmx.com> wrote: >>>>>>>> >>>>>>>> >>>>>>>> On 2018年04月29日 16:11, Michael Wade wrote: >>>>>>>>> Thanks Qu, >>>>>>>>> >>>>>>>>> Please find attached the log file for the chunk recover command. >>>>>>>> >>>>>>>> Strangely, btrfs chunk recovery found no extra chunk beyond current >>>>>>>> system chunk range. >>>>>>>> >>>>>>>> Which means, it's chunk tree corrupted. >>>>>>>> >>>>>>>> Please dump the chunk tree with latest btrfs-progs (which provides the >>>>>>>> new --follow option). >>>>>>>> >>>>>>>> # btrfs inspect dump-tree -b 20800943685632 <device> >>>>>>>> >>>>>>>> If it doesn't work, please provide the following binary dump: >>>>>>>> >>>>>>>> # dd if=<dev> of=/tmp/chunk_root.copy1 bs=1 count=32K skip=266325721088 >>>>>>>> # dd if=<dev> of=/tmp/chunk_root.copy2 bs=1 count=32K skip=266359275520 >>>>>>>> (And will need to repeat similar dump for several times according to >>>>>>>> above dump) >>>>>>>> >>>>>>>> Thanks, >>>>>>>> Qu >>>>>>>> >>>>>>>> >>>>>>>>> >>>>>>>>> Kind regards >>>>>>>>> Michael >>>>>>>>> >>>>>>>>> On 28 April 2018 at 12:38, Qu Wenruo <quwenruo.bt...@gmx.com> wrote: >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> On 2018年04月28日 17:37, Michael Wade wrote: >>>>>>>>>>> Hi Qu, >>>>>>>>>>> >>>>>>>>>>> Thanks for your reply. I will investigate upgrading the kernel, >>>>>>>>>>> however I worry that future ReadyNAS firmware upgrades would fail >>>>>>>>>>> on a >>>>>>>>>>> newer kernel version (I don't have much linux experience so maybe my >>>>>>>>>>> concerns are unfounded!?). >>>>>>>>>>> >>>>>>>>>>> I have attached the output of the dump super command. >>>>>>>>>>> >>>>>>>>>>> I did actually run chunk recover before, without the verbose option, >>>>>>>>>>> it took around 24 hours to finish but did not resolve my issue. >>>>>>>>>>> Happy >>>>>>>>>>> to start that again if you need its output. >>>>>>>>>> >>>>>>>>>> The system chunk only contains the following chunks: >>>>>>>>>> [0, 4194304]: Initial temporary chunk, not used at all >>>>>>>>>> [20971520, 29360128]: System chunk created by mkfs, should be full >>>>>>>>>> used up >>>>>>>>>> [20800943685632, 20800977240064]: >>>>>>>>>> The newly created large system chunk. >>>>>>>>>> >>>>>>>>>> The chunk root is still in 2nd chunk thus valid, but some of its >>>>>>>>>> leaf is >>>>>>>>>> out of the range. >>>>>>>>>> >>>>>>>>>> If you can't wait 24h for chunk recovery to run, my advice would be >>>>>>>>>> move >>>>>>>>>> the disk to some other computer, and use latest btrfs-progs to >>>>>>>>>> execute >>>>>>>>>> the following command: >>>>>>>>>> >>>>>>>>>> # btrfs inpsect dump-tree -b 20800943685632 --follow >>>>>>>>>> >>>>>>>>>> If we're lucky enough, we may read out the tree leaf containing the >>>>>>>>>> new >>>>>>>>>> system chunk and save a day. >>>>>>>>>> >>>>>>>>>> Thanks, >>>>>>>>>> Qu >>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> Thanks so much for your help. >>>>>>>>>>> >>>>>>>>>>> Kind regards >>>>>>>>>>> Michael >>>>>>>>>>> >>>>>>>>>>> On 28 April 2018 at 09:45, Qu Wenruo <quwenruo.bt...@gmx.com> wrote: >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> On 2018年04月28日 16:30, Michael Wade wrote: >>>>>>>>>>>>> Hi all, >>>>>>>>>>>>> >>>>>>>>>>>>> I was hoping that someone would be able to help me resolve the >>>>>>>>>>>>> issues >>>>>>>>>>>>> I am having with my ReadyNAS BTRFS volume. Basically my trouble >>>>>>>>>>>>> started after a power cut, subsequently the volume would not >>>>>>>>>>>>> mount. >>>>>>>>>>>>> Here are the details of my setup as it is at the moment: >>>>>>>>>>>>> >>>>>>>>>>>>> uname -a >>>>>>>>>>>>> Linux QAI 4.4.116.alpine.1 #1 SMP Mon Feb 19 21:58:38 PST 2018 >>>>>>>>>>>>> armv7l GNU/Linux >>>>>>>>>>>> >>>>>>>>>>>> The kernel is pretty old for btrfs. >>>>>>>>>>>> Strongly recommended to upgrade. >>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> btrfs --version >>>>>>>>>>>>> btrfs-progs v4.12 >>>>>>>>>>>> >>>>>>>>>>>> So is the user tools. >>>>>>>>>>>> >>>>>>>>>>>> Although I think it won't be a big problem, as needed tool should >>>>>>>>>>>> be there. >>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> btrfs fi show >>>>>>>>>>>>> Label: '11baed92:data' uuid: 20628cda-d98f-4f85-955c-932a367f8821 >>>>>>>>>>>>> Total devices 1 FS bytes used 5.12TiB >>>>>>>>>>>>> devid 1 size 7.27TiB used 6.24TiB path /dev/md127 >>>>>>>>>>>> >>>>>>>>>>>> So, it's btrfs on mdraid. >>>>>>>>>>>> It would normally make things harder to debug, so I could only >>>>>>>>>>>> provide >>>>>>>>>>>> advice from the respect of btrfs. >>>>>>>>>>>> For mdraid part, I can't ensure anything. >>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> Here are the relevant dmesg logs for the current state of the >>>>>>>>>>>>> device: >>>>>>>>>>>>> >>>>>>>>>>>>> [ 19.119391] md: md127 stopped. >>>>>>>>>>>>> [ 19.120841] md: bind<sdb3> >>>>>>>>>>>>> [ 19.121120] md: bind<sdc3> >>>>>>>>>>>>> [ 19.121380] md: bind<sda3> >>>>>>>>>>>>> [ 19.125535] md/raid:md127: device sda3 operational as raid >>>>>>>>>>>>> disk 0 >>>>>>>>>>>>> [ 19.125547] md/raid:md127: device sdc3 operational as raid >>>>>>>>>>>>> disk 2 >>>>>>>>>>>>> [ 19.125554] md/raid:md127: device sdb3 operational as raid >>>>>>>>>>>>> disk 1 >>>>>>>>>>>>> [ 19.126712] md/raid:md127: allocated 3240kB >>>>>>>>>>>>> [ 19.126778] md/raid:md127: raid level 5 active with 3 out of 3 >>>>>>>>>>>>> devices, algorithm 2 >>>>>>>>>>>>> [ 19.126784] RAID conf printout: >>>>>>>>>>>>> [ 19.126789] --- level:5 rd:3 wd:3 >>>>>>>>>>>>> [ 19.126794] disk 0, o:1, dev:sda3 >>>>>>>>>>>>> [ 19.126799] disk 1, o:1, dev:sdb3 >>>>>>>>>>>>> [ 19.126804] disk 2, o:1, dev:sdc3 >>>>>>>>>>>>> [ 19.128118] md127: detected capacity change from 0 to >>>>>>>>>>>>> 7991637573632 >>>>>>>>>>>>> [ 19.395112] Adding 523708k swap on /dev/md1. Priority:-1 >>>>>>>>>>>>> extents:1 >>>>>>>>>>>>> across:523708k >>>>>>>>>>>>> [ 19.434956] BTRFS: device label 11baed92:data devid 1 transid >>>>>>>>>>>>> 151800 /dev/md127 >>>>>>>>>>>>> [ 19.739276] BTRFS info (device md127): setting nodatasum >>>>>>>>>>>>> [ 19.740440] BTRFS critical (device md127): unable to find >>>>>>>>>>>>> logical >>>>>>>>>>>>> 3208757641216 len 4096 >>>>>>>>>>>>> [ 19.740450] BTRFS critical (device md127): unable to find >>>>>>>>>>>>> logical >>>>>>>>>>>>> 3208757641216 len 4096 >>>>>>>>>>>>> [ 19.740498] BTRFS critical (device md127): unable to find >>>>>>>>>>>>> logical >>>>>>>>>>>>> 3208757641216 len 4096 >>>>>>>>>>>>> [ 19.740512] BTRFS critical (device md127): unable to find >>>>>>>>>>>>> logical >>>>>>>>>>>>> 3208757641216 len 4096 >>>>>>>>>>>>> [ 19.740552] BTRFS critical (device md127): unable to find >>>>>>>>>>>>> logical >>>>>>>>>>>>> 3208757641216 len 4096 >>>>>>>>>>>>> [ 19.740560] BTRFS critical (device md127): unable to find >>>>>>>>>>>>> logical >>>>>>>>>>>>> 3208757641216 len 4096 >>>>>>>>>>>>> [ 19.740576] BTRFS error (device md127): failed to read chunk >>>>>>>>>>>>> root >>>>>>>>>>>> >>>>>>>>>>>> This shows it pretty clear, btrfs fails to read chunk root. >>>>>>>>>>>> And according your above "len 4096" it's pretty old fs, as it's >>>>>>>>>>>> still >>>>>>>>>>>> using 4K nodesize other than 16K nodesize. >>>>>>>>>>>> >>>>>>>>>>>> According to above output, it means your superblock by somehow >>>>>>>>>>>> lacks the >>>>>>>>>>>> needed system chunk mapping, which is used to initialize chunk >>>>>>>>>>>> mapping. >>>>>>>>>>>> >>>>>>>>>>>> Please provide the following command output: >>>>>>>>>>>> >>>>>>>>>>>> # btrfs inspect dump-super -fFa /dev/md127 >>>>>>>>>>>> >>>>>>>>>>>> Also, please consider run the following command and dump all its >>>>>>>>>>>> output: >>>>>>>>>>>> >>>>>>>>>>>> # btrfs rescue chunk-recover -v /dev/md127. >>>>>>>>>>>> >>>>>>>>>>>> Please note that, above command can take a long time to finish, >>>>>>>>>>>> and if >>>>>>>>>>>> it works without problem, it may solve your problem. >>>>>>>>>>>> But if it doesn't work, the output could help me to manually craft >>>>>>>>>>>> a fix >>>>>>>>>>>> to your super block. >>>>>>>>>>>> >>>>>>>>>>>> Thanks, >>>>>>>>>>>> Qu >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>>> [ 19.783975] BTRFS error (device md127): open_ctree failed >>>>>>>>>>>>> >>>>>>>>>>>>> In an attempt to recover the volume myself I run a few BTRFS >>>>>>>>>>>>> commands >>>>>>>>>>>>> mostly using advice from here: >>>>>>>>>>>>> https://lists.opensuse.org/opensuse/2017-02/msg00930.html. However >>>>>>>>>>>>> that actually seems to have made things worse as I can no longer >>>>>>>>>>>>> mount >>>>>>>>>>>>> the file system, not even in readonly mode. >>>>>>>>>>>>> >>>>>>>>>>>>> So starting from the beginning here is a list of things I have >>>>>>>>>>>>> done so >>>>>>>>>>>>> far (hopefully I remembered the order in which I ran them!) >>>>>>>>>>>>> >>>>>>>>>>>>> 1. Noticed that my backups to the NAS were not running (didn't get >>>>>>>>>>>>> notified that the volume had basically "died") >>>>>>>>>>>>> 2. ReadyNAS UI indicated that the volume was inactive. >>>>>>>>>>>>> 3. SSHed onto the box and found that the first drive was not >>>>>>>>>>>>> marked as >>>>>>>>>>>>> operational (log showed I/O errors / UNKOWN (0x2003)) so I >>>>>>>>>>>>> replaced >>>>>>>>>>>>> the disk and let the array resync. >>>>>>>>>>>>> 4. After resync the volume still was unaccessible so I looked at >>>>>>>>>>>>> the >>>>>>>>>>>>> logs once more and saw something like the following which seemed >>>>>>>>>>>>> to >>>>>>>>>>>>> indicate that the replay log had been corrupted when the power >>>>>>>>>>>>> went >>>>>>>>>>>>> out: >>>>>>>>>>>>> >>>>>>>>>>>>> BTRFS critical (device md127): corrupt leaf, non-root leaf's >>>>>>>>>>>>> nritems >>>>>>>>>>>>> is 0: block=232292352, root=7, slot=0 >>>>>>>>>>>>> BTRFS critical (device md127): corrupt leaf, non-root leaf's >>>>>>>>>>>>> nritems >>>>>>>>>>>>> is 0: block=232292352, root=7, slot=0 >>>>>>>>>>>>> BTRFS: error (device md127) in btrfs_replay_log:2524: errno=-5 IO >>>>>>>>>>>>> failure (Failed to recover log tree) >>>>>>>>>>>>> BTRFS error (device md127): pending csums is 155648 >>>>>>>>>>>>> BTRFS error (device md127): cleaner transaction attach returned >>>>>>>>>>>>> -30 >>>>>>>>>>>>> BTRFS critical (device md127): corrupt leaf, non-root leaf's >>>>>>>>>>>>> nritems >>>>>>>>>>>>> is 0: block=232292352, root=7, slot=0 >>>>>>>>>>>>> >>>>>>>>>>>>> 5. Then: >>>>>>>>>>>>> >>>>>>>>>>>>> btrfs rescue zero-log >>>>>>>>>>>>> >>>>>>>>>>>>> 6. Was then able to mount the volume in readonly mode. >>>>>>>>>>>>> >>>>>>>>>>>>> btrfs scrub start >>>>>>>>>>>>> >>>>>>>>>>>>> Which fixed some errors but not all: >>>>>>>>>>>>> >>>>>>>>>>>>> scrub status for 20628cda-d98f-4f85-955c-932a367f8821 >>>>>>>>>>>>> >>>>>>>>>>>>> scrub started at Tue Apr 24 17:27:44 2018, running for 04:00:34 >>>>>>>>>>>>> total bytes scrubbed: 224.26GiB with 6 errors >>>>>>>>>>>>> error details: csum=6 >>>>>>>>>>>>> corrected errors: 0, uncorrectable errors: 6, unverified errors: 0 >>>>>>>>>>>>> >>>>>>>>>>>>> scrub status for 20628cda-d98f-4f85-955c-932a367f8821 >>>>>>>>>>>>> scrub started at Tue Apr 24 17:27:44 2018, running for 04:34:43 >>>>>>>>>>>>> total bytes scrubbed: 224.26GiB with 6 errors >>>>>>>>>>>>> error details: csum=6 >>>>>>>>>>>>> corrected errors: 0, uncorrectable errors: 6, unverified errors: 0 >>>>>>>>>>>>> >>>>>>>>>>>>> 6. Seeing this hanging I rebooted the NAS >>>>>>>>>>>>> 7. Think this is when the volume would not mount at all. >>>>>>>>>>>>> 8. Seeing log entries like these: >>>>>>>>>>>>> >>>>>>>>>>>>> BTRFS warning (device md127): checksum error at logical >>>>>>>>>>>>> 20800943685632 >>>>>>>>>>>>> on dev /dev/md127, sector 520167424: metadata node (level 1) in >>>>>>>>>>>>> tree 3 >>>>>>>>>>>>> >>>>>>>>>>>>> I ran >>>>>>>>>>>>> >>>>>>>>>>>>> btrfs check --fix-crc >>>>>>>>>>>>> >>>>>>>>>>>>> And that brings us to where I am now: Some seemly corrupted BTRFS >>>>>>>>>>>>> metadata and unable to mount the drive even with the recovery >>>>>>>>>>>>> option. >>>>>>>>>>>>> >>>>>>>>>>>>> Any help you can give is much appreciated! >>>>>>>>>>>>> >>>>>>>>>>>>> Kind regards >>>>>>>>>>>>> Michael >>>>>>>>>>>>> -- >>>>>>>>>>>>> To unsubscribe from this list: send the line "unsubscribe >>>>>>>>>>>>> linux-btrfs" in >>>>>>>>>>>>> the body of a message to majord...@vger.kernel.org >>>>>>>>>>>>> More majordomo info at http://vger.kernel.org/majordomo-info.html >>>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>> >>>>>>>> >>>>>>> -- >>>>>>> To unsubscribe from this list: send the line "unsubscribe linux-btrfs" >>>>>>> in >>>>>>> the body of a message to majord...@vger.kernel.org >>>>>>> More majordomo info at http://vger.kernel.org/majordomo-info.html >>>>>>> >>>>>> >>>> >> -- To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
