Hello,
I will try to provide all information pertinent to the situation i find myself
in.
Yesterday while trying to write some data to a BTRFS filesystem on top of a
mdadm raid5 array encrypted with dmcrypt comprising of 4 1tb HDD my system
became unresponsive and i had no choice but to hard reset. System came back up
no problem and the array in question mounted without a complaint. Once i tried
to write data to it again however the system became unresponsive again and
required another hard reset. Again system came back up and everything mounted
with no complaints.
This time i decided to run some checks. Ran a raid check by issuing 'echo check
> /sys/block/md0/md/sync_action'. This completed without a single error. So i
performed a proper restart just because and once the system came back up i
initiated a scrub on the btrfs filesystem. This greeted me with my first
indication that something is wrong:
btrfs sc stat /media/Storage2
scrub status for e5bd5cf3-c736-48ff-b1c6-c9f678567788
scrub started at Mon Sep 18 06:05:21 2017, running for 07:40:47
total bytes scrubbed: 1.03TiB with 1 errors
error details: super=1
corrected errors: 0, uncorrectable errors: 0, unverified errors: 0
I was concerned but since it was still scrubbing i left it. Now things look
really bleak...
Every few minutes the scrub process goes into a D status as shown by htop it
eventually keeps going and as far as i can see is still scrubbing(slowly). I
decided to check a something else(based on the error above) I ran btrfs
inspect-internal dump-super -a -f /dev/md0 which gave me this:
superblock: bytenr=65536, device=/dev/md0
-
ERROR: bad magic on superblock on /dev/md0 at 65536
superblock: bytenr=67108864, device=/dev/md0
-
ERROR: bad magic on superblock on /dev/md0 at 67108864
superblock: bytenr=274877906944, device=/dev/md0
-
ERROR: bad magic on superblock on /dev/md0 at 274877906944
Now i'm really panicked. Is the FS toast? Can any recovery be attempted?
Here is the output of dump-super with the -F option:
superblock: bytenr=65536, device=/dev/md0
-
csum_type 43668 (INVALID)
csum_size 32
csum
0x76c647b04abf1057f04e40d1dc52522397258064b98a1b8f6aa6934c74c0dd55 [DON'T MATCH]
bytenr 6376050623103086821
flags 0x7edcc412b742c79f
( WRITTEN |
RELOC |
METADUMP |
unknown flag: 0x7edcc410b742c79c )
magic ..l~...q [DON'T MATCH]
fsid2cf827fa-7ab8-e290-b152-1735c2735a37
label
.a.9.@.=4.#.|.D...]..dh=d,..k..n..~.5.i.8...(.._.tl.a.@..2..qidj.>Hy.U..{X5.kG0.)t..;/.2...@.T.|.u.<.`!J*9./8...&.g\.V...*.,/95.uEs..W.i..z..h...n(...VGn^F...H...5.DT..3.A..mK...~..}.1..n.
generation 1769598730239175261
root14863846352370317867
sys_array_size 1744503544
chunk_root_generation 18100024505086712407
root_level 79
chunk_root 10848092274453435018
chunk_root_level156
log_root7514172289378668244
log_root_transid6227239369566282426
log_root_level 18
total_bytes 5481087866519986730
bytes_used 13216280034370888020
sectorsize 4102056786
nodesize1038279258
leafsize276348297
stripesize 2473897044
root_dir12090183195204234845
num_devices 12836127619712721941
compat_flags0xf98ff436fc954bd4
compat_ro_flags 0x3fe8246616164da7
( FREE_SPACE_TREE |
FREE_SPACE_TREE_VALID |
unknown flag: 0x3fe8246616164da4 )
incompat_flags 0x3989a5037330bfd8
( COMPRESS_LZO |
COMPRESS_LZOv2 |
EXTENDED_IREF |
RAID56 |
SKINNY_METADATA |
NO_HOLES |
unknown flag: 0x3989a5037330bc10 )
cache_generation10789185961859482334
uuid_tree_generation14921288820846890813
dev_item.uuid e6e382b3-de66-4c25-7cc9-3cc43cde9c24
dev_item.fsid f8430e37-12ca-adaf-b038-f0ee10ce6327 [DON'T MATCH]
dev_item.type 7909001383421391155
dev_item.total_bytes4839925749276763097
dev_item.bytes_used 14330418354255459170
dev_item.io_align 4136652250
dev_item.io_width 1113335506
dev_item.sector_size1197062542
dev_item.devid 16559830033162408461
dev_item.dev_group 3271056113
