On 10-17-2017 6:24 PM, Qu Wenruo wrote:
>
>
> On 2017-10-18 04:43, Cameron Kelley wrote:
>> Hey btrfs gurus,
>>
>> I have a 4 disk btrfs filesystem that has suddenly stopped mounting
>> after a recent reboot. The data is in an odd configuration due to
>> originally being in a 3 disk RAID1 before adding a 4th disk and running
>> a balance to convert to RAID10. There wasn't enough free space to
>> completely convert, so about half the data is still in RAID1 while the
>> other half is in RAID10. Both metadata and system are RAID10. It has
>> been in this configuration for 6 months or so now since adding the 4th
>> disk. It just holds archived media and hasn't had any data added or
>> modified in quite some time. I feel pretty stupid now for not
>> correcting that sooner though.
>>
>> I have tried mounting with different mount options for recovery, ro,
>> degraded, etc. Log shows errors about "unable to find logical
>> 3746892939264 length 4096"
>>
>> When I do a btrfs check, it doesn't find any issues. Running
>> btrfs-find-root comes up with a message about a block that the
>> generation doesn't match. If I specify that block on the btrfs check, I
>> get transid verify failures.
>>
>> I ran a dry run of a recovery of the entire filesystem which runs
>> through every file with no errors. I would just restore the data and
>> start fresh, but unfortunately I don't have the free space at the
>> moment for the ~4.5TB of data.
>>
>> I also ran full smart self tests on all 4 disks with no errors.
>>
>> root@nas2:~# uname -a
>> Linux nas2 4.13.7-041307-generic #201710141430 SMP Sat Oct 14 14:39:06
>> UTC 2017 i686 i686 i686 GNU/Linux
>
> I don't think i686 kernel will cause any difference, but considering
> most of us are using x86_64 to develop/test, maybe it will be a good
> idea to upgrade to x86_64 kernel?
>
Thanks for the quick response.
This is an old x86 Pentium NAS I inherited, so unfortunately I'm stuck on
a 32-bit kernel. If push comes to shove, I can disassemble another x64
machine to test with.
>>
>> root@nas2:~# btrfs version
>> btrfs-progs v4.13.2
>>
>> root@nas2:~# btrfs fi show
>> Label: none uuid: 827029a4-8625-4a50-a22d-0fd28dbe2d36
>> Total devices 4 FS bytes used 4.60TiB
>> devid 1 size 2.73TiB used 2.33TiB path /dev/sdb1
>> devid 2 size 2.73TiB used 2.33TiB path /dev/sdc
>> devid 3 size 2.73TiB used 2.33TiB path /dev/sdd1
>> devid 4 size 2.73TiB used 2.33TiB path /dev/sde1
>>
>> root@nas2:~# mount /dev/sdb1 /mnt/nas2/
>> mount: wrong fs type, bad option, bad superblock on /dev/sdb1,
>> missing codepage or helper program, or other error
>>
>> In some cases useful info is found in syslog - try
>> dmesg | tail or so.
>>
>> root@nas2:~# dmesg | tail
>> [ 801.332623] BTRFS info (device sdb1): disk space caching is enabled
>> [ 801.332627] BTRFS info (device sdb1): has skinny extents
>> [ 801.333386] BTRFS critical (device sdb1): unable to find logical
>> 3746892939264 length 4096
>> [ 801.333472] BTRFS critical (device sdb1): unable to find logical
>> 3746892939264 length 4096
>> [ 801.333769] BTRFS critical (device sdb1): unable to find logical
>> 3746892939264 length 4096
>> [ 801.333835] BTRFS critical (device sdb1): unable to find logical
>> 3746892939264 length 4096
>> [ 801.333909] BTRFS critical (device sdb1): unable to find logical
>> 3746892939264 length 4096
>> [ 801.333968] BTRFS critical (device sdb1): unable to find logical
>> 3746892939264 length 4096
>> [ 801.334028] BTRFS error (device sdb1): failed to read chunk root
>> [ 801.365452] BTRFS error (device sdb1): open_ctree failed
>
> Some of the chunk tree failed to be read out.
>
> Either chunk tree or system chunk array has some problem.
>
> Would you please dump the chunk tree and superblock by the following
> commands?
>
> # btrfs inspect-internal dump-tree -t chunk /dev/sdb1
> # btrfs inspect-internal dump-super -fa /dev/sdb1
>
# btrfs inspect-internal dump-tree -t chunk /dev/sdb1
http://pastebin.ubuntu.com/25763241/
# btrfs inspect-internal dump-super -fa /dev/sdb1
http://pastebin.ubuntu.com/25763246/
>>
>> root@nas2:~# btrfs check /dev/sdb1
>> Checking filesystem on /dev/sdb1
>> UUID: 827029a4-8625-4a50-a22d-0fd28dbe2d36
>> checking extents
>> checking free space cache
>> cache and super generation don't match, space cache will be invalidated
>> checking fs roots
>> checking csums
>> checking root refs
>> found 5054297628672 bytes used, no error found
>> total csum bytes: 4929567064
>> total tree bytes: 5197856768
>> total fs tree bytes: 15237120
>> total extent tree bytes: 43433984
>> btree space waste bytes: 161510789
>> file data blocks allocated: 5050024812544
>> referenced 5049610178560
>
> Unless we has some bug in btrfs-progs chunk mapping, the result seems
> quite good.
>
> Just in case, would you please also run "btrfs check --mode=lowmem
> /dev/sdb1" to see if it's OK?
>
# btrfs check --mode=lowmem /dev/sdb1
Checking filesystem on /dev/sdb1
UUID: 827029a4-8625-4a50-a22d-0fd28dbe2d36
checking extents
checking free space cache
checking fs roots
checking csums
checking root refs
found 5053072662528 bytes used, no error found
total csum bytes: 4929567064
total tree bytes: 5195988992
total fs tree bytes: 15237120
total extent tree bytes: 43368448
btree space waste bytes: 161593371
file data blocks allocated: 5048801714176
referenced 5048387080192
>>
>> root@nas2:~# btrfs-find-root /dev/sdb1
>> Superblock thinks the generation is 147970
>> Superblock thinks the level is 1
>> Found tree root at 21335861559296 gen 147970 level 1
>> Well block 21335857758208(gen: 147969 level: 1) seems good, but
>> generation/level doesn't match, want gen: 147970 level: 1
>
> Since it's mostly related to chunk tree, would you please try the
> following command?
>
> # btrfs-find-root -o 3 /dev/sdb1
> # btrfs check --chunk-root <the next chunk root bytenr> /dev/sdb1
>
> Thanks,
> Qu
>
# btrfs-find-root -o 3 /dev/sdb1
Superblock thinks the generation is 147728
Superblock thinks the level is 1
Found tree root at 21339078983680 gen 147728 level 1
# btrfs check --chunk-root 21339078983680 /dev/sdb1
Checking filesystem on /dev/sdb1
UUID: 827029a4-8625-4a50-a22d-0fd28dbe2d36
checking extents
checking free space cache
checking fs roots
checking csums
checking root refs
found 5053072662528 bytes used, no error found
total csum bytes: 4929567064
total tree bytes: 5195988992
total fs tree bytes: 15237120
total extent tree bytes: 43368448
btree space waste bytes: 161593371
file data blocks allocated: 5048801714176
referenced 5048387080192
>>
>> root@nas2:~# btrfs check -r 21335857758208 /dev/sdb1
>> parent transid verify failed on 21335857758208 wanted 147970 found 147969
>> parent transid verify failed on 21335857758208 wanted 147970 found 147969
>> parent transid verify failed on 21335857758208 wanted 147970 found 147969
>> parent transid verify failed on 21335857758208 wanted 147970 found 147969
>> Ignoring transid failure
>> Checking filesystem on /dev/sdb1
>> UUID: 827029a4-8625-4a50-a22d-0fd28dbe2d36
>> checking extents
>> checking free space cache
>> cache and super generation don't match, space cache will be invalidated
>> checking fs roots
>> checking csums
>> checking root refs
>> ERROR: transid errors in file system
>> found 5054297628672 bytes used, error(s) found
>> total csum bytes: 4929567064
>> total tree bytes: 5197856768
>> total fs tree bytes: 15237120
>> total extent tree bytes: 43433984
>> btree space waste bytes: 161510789
>> file data blocks allocated: 5050024812544
>> referenced 5049610178560
>> --
>> To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in
>> the body of a message to majord...@vger.kernel.org
>> More majordomo info at http://vger.kernel.org/majordomo-info.html
--
To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in
the body of a message to majord...@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
