unable to fixup (regular) error

[Alexander Fieroch]

Hi,

My data partition with btrfs RAID 0 (/dev/sdc0 and /dev/sdd0) shows
errors in syslog:

BTRFS error (device sdc): cleaner transaction attach returned -30
BTRFS info (device sdc): disk space caching is enabled
BTRFS info (device sdc): has skinny extents
BTRFS info (device sdc): bdev /dev/sdc errs: wr 0, rd 0, flush 0,
corrupt 3, gen 1
BTRFS info (device sdc): bdev /dev/sdd errs: wr 0, rd 0, flush 0,
corrupt 6, gen 2


BTRFS error (device sdc): scrub: tree block 858803990528 spanning
stripes, ignored. logical=3D858803929088
BTRFS error (device sdc): scrub: tree block 858803990528 spanning
stripes, ignored. logical=3D858803994624
BTRFS warning (device sdc): checksum error at logical 858803961856 on
dev /dev/sdd, physical 385263894528: metadata leaf (level 0) in tree 7
BTRFS warning (device sdc): checksum error at logical 858803961856 on
dev /dev/sdd, physical 385263894528: metadata leaf (level 0) in tree 7
BTRFS error (device sdc): bdev /dev/sdd errs: wr 0, rd 0, flush 0,
corrupt 4, gen 1
BTRFS error (device sdc): scrub: tree block 858820505600 spanning
stripes, ignored. logical=3D858820444160
BTRFS error (device sdc): scrub: tree block 858820505600 spanning
stripes, ignored. logical=3D858820509696
BTRFS error (device sdc): unable to fixup (regular) error at logical
858803961856 on dev /dev/sdd
BTRFS error (device sdc): scrub: tree block 858821292032 spanning
stripes, ignored. logical=3D858821230592
BTRFS error (device sdc): scrub: tree block 858821292032 spanning
stripes, ignored. logical=3D858821296128
BTRFS warning (device sdc): checksum error at logical 858821263360 on
dev /dev/sdd, physical 385281196032: metadata leaf (level 0) in tree 7
BTRFS warning (device sdc): checksum error at logical 858821263360 on
dev /dev/sdd, physical 385281196032: metadata leaf (level 0) in tree 7
BTRFS error (device sdc): bdev /dev/sdd errs: wr 0, rd 0, flush 0,
corrupt 5, gen 1
BTRFS error (device sdc): unable to fixup (regular) error at logical
858821263360 on dev /dev/sdd
BTRFS warning (device sdc): checksum/header error at logical
858820476928 on dev /dev/sdd, physical 385280409600: metadata leaf
(level 0) in tree 7
BTRFS warning (device sdc): checksum/header error at logical
858820476928 on dev /dev/sdd, physical 385280409600: metadata leaf
(level 0) in tree 7
BTRFS error (device sdc): bdev /dev/sdd errs: wr 0, rd 0, flush 0,
corrupt 5, gen 2
BTRFS warning (device sdc): checksum error at logical 858820489216 on
dev /dev/sdd, physical 385280421888: metadata leaf (level 0) in tree 2
BTRFS warning (device sdc): checksum error at logical 858820489216 on
dev /dev/sdd, physical 385280421888: metadata leaf (level 0) in tree 2
BTRFS error (device sdc): bdev /dev/sdd errs: wr 0, rd 0, flush 0,
corrupt 6, gen 2
BTRFS error (device sdc): unable to fixup (regular) error at logical
858820476928 on dev /dev/sdd
BTRFS error (device sdc): unable to fixup (regular) error at logical
858820489216 on dev /dev/sdd0


$ btrfs filesystem show /mnt/data/
Label: none  uuid: 5e6506b0-bf15-4b2e-b5f4-322c44b89db6
          Total devices 2 FS bytes used 10.17TiB
          devid    1 size 5.46TiB used 5.43TiB path /dev/sdc
          devid    2 size 5.46TiB used 5.43TiB path /dev/sdd

$ btrfs --version
btrfs-progs v4.15.1

$ uname -a
Linux gpur1 4.15.0-39-generic #42-Ubuntu SMP Tue Oct 23 15:48:01 UTC
2018 x86_64 x86_64 x86_64 GNU/Linux


$ btrfs dev stats /dev/sdc
[/dev/sdc].write_io_errs    0
[/dev/sdc].read_io_errs     0
[/dev/sdc].flush_io_errs    0
[/dev/sdc].corruption_errs  3
[/dev/sdc].generation_errs  1

$ btrfs dev stats /dev/sdd
[/dev/sdd].write_io_errs    0
[/dev/sdd].read_io_errs     0
[/dev/sdd].flush_io_errs    0
[/dev/sdd].corruption_errs  3
[/dev/sdd].generation_errs  1

$ btrfs fi show
Label: 'system'  uuid: ae121e8e-d483-45f4-8568-2817f5c5d497
        Total devices 1 FS bytes used 194.05GiB
        devid    1 size 228.66GiB used 199.03GiB path /dev/sda3
Label: none  uuid: 5e6506b0-bf15-4b2e-b5f4-322c44b89db6
        Total devices 2 FS bytes used 10.17TiB
        devid    1 size 5.46TiB used 5.43TiB path /dev/sdc
        devid    2 size 5.46TiB used 5.43TiB path /dev/sdd

$ btrfs fi df /mnt/data/
Data, RAID0: total=10.84TiB, used=10.15TiB
System, RAID1: total=8.00MiB, used=896.00KiB
Metadata, RAID1: total=15.00GiB, used=13.28GiB
GlobalReserve, single: total=512.00MiB, used=0.00

$ btrfs scrub start -B /dev/sdc

scrub canceled for 5e6506b0-bf15-4b2e-b5f4-322c44b89db6

         total bytes scrubbed: 1.58TiB with 10 errors
         error details: verify=1 csum=3




I've tried
$ btrfs check /dev/sdc
Checking filesystem on /dev/sdc
UUID: 5e6506b0-bf15-4b2e-b5f4-322c44b89db6
btrfs check --repairchecking extents
  ERROR: add_tree_backref failed (extent items shared block): File exists
ERROR: add_tree_backref failed (extent items tree block): File exists
ERROR: add_tree_backref failed (extent items tree block): File exists
/dev/sdc
ERROR: add_tree_backref failed (non-leaf block): File exists

ERROR: add_tree_backref failed (non-leaf block): File exists
checksum verify failed on 858803961856 found B2C0FAD9 wanted F31F8495
checksum verify failed on 858803961856 found B2C0FAD9 wanted F31F8495
checksum verify failed on 858803961856 found B2C0FAD9 wanted F31F8495
checksum verify failed on 858803961856 found B2C0FAD9 wanted F31F8495
Csum didn't match
checksum verify failed on 858821263360 found 15208BF4 wanted D68B2514
checksum verify failed on 858821263360 found 15208BF4 wanted D68B2514
checksum verify failed on 858821263360 found 15208BF4 wanted D68B2514
checksum verify failed on 858821263360 found 15208BF4 wanted D68B2514
Csum didn't match
ref mismatch on [8631607296 77824] extent item 1, found 0

backpointer mismatch on [8631607296 77824]
owner ref check failed [8631607296 77824]
ref mismatch on [35613634560 77824] extent item 1, found 0

backpointer mismatch on [35613634560 77824]
owner ref check failed [35613634560 77824]
ref mismatch on [36010762240 77824] extent item 1, found 0
[...]
ERROR: errors found in extent allocation tree or chunk allocation
checking free space cache
checking fs roots

btrfs(+0x29d87)[0x55f83fd51d87]
btrfs(+0x2a0b4)[0x55f83fd520b4]
btrfs(alloc_extent_buffer+0x77)[0x55f83fd527af]
btrfs(read_tree_block+0x44)[0x55f83fd45802]
btrfs(btrfs_next_leaf+0x6e)[0x55f83fd43ad9]
btrfs(count_csum_range+0x1e1)[0x55f83fd89fac]
btrfs(+0x14b33)[0x55f83fd3cb33]
btrfs(cmd_check+0x19fb)[0x55f83fd7bfe2]
btrfs(main+0x143)[0x55f83fd3ec87]
/lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0xe7)[0x7ff69e81cb97]
btrfs(_start+0x2a)[0x55f83fd3ecca]
Aborted (core dumped)



$ btrfs check --repair /dev/sdc
enabling repair mode
Checking filesystem on /dev/sdc
UUID: 5e6506b0-bf15-4b2e-b5f4-322c44b89db6
Fixed 0 roots.
checking extents
ERROR: add_tree_backref failed (extent items shared block): File exists
ERROR: add_tree_backref failed (extent items tree block): File exists
ERROR: add_tree_backref failed (extent items tree block): File exists
ERROR: add_tree_backref failed (non-leaf block): File exists
ERROR: add_tree_backref failed (non-leaf block): File exists
checksum verify failed on 858803961856 found B2C0FAD9 wanted F31F8495
checksum verify failed on 858803961856 found B2C0FAD9 wanted F31F8495
checksum verify failed on 858803961856 found B2C0FAD9 wanted F31F8495
checksum verify failed on 858803961856 found B2C0FAD9 wanted F31F8495
Csum didn't match
checksum verify failed on 858821263360 found 15208BF4 wanted D68B2514
checksum verify failed on 858821263360 found 15208BF4 wanted D68B2514
checksum verify failed on 858821263360 found 15208BF4 wanted D68B2514
checksum verify failed on 858821263360 found 15208BF4 wanted D68B2514
Csum didn't match

Aborted (core dumped)




How can I fix the error?
Is there any possibility to see which files are affected?
Please have a look at the full log attached.

Thanks!

Best regards,
Alexander

ERROR: scrubbing /dev/sdc failed for device id 1: ret=-1, errno=5 (Input/output error) scrub started at Thu Nov 22 07:43:45 2018 and was aborted after 02:31:49 corrected errors: 0, uncorrectable errors: 10, unverified errors: 0 incorrect local backref count on 8631607296 parent 858803974144 owner 0 offset 0 found 0 wanted 1 back 0x55f8522a5b10 backref disk bytenr does not match extent record, bytenr=8631607296, ref bytenr=0 incorrect local backref count on 35613634560 parent 858803974144 owner 0 offset 0 found 0 wanted 1 back 0x55f86d87d810 backref disk bytenr does not match extent record, bytenr=35613634560, ref bytenr=0 extent_io.c:605: free_extent_buffer_internal: BUG_ON `eb->refs < 0` triggered, value 1 well this shouldn't happen, extent record overlaps but is metadata? [858803974144, 16384]

btrfs-failure.txt.gz
Description: btrfs-failure.txt.gz

smime.p7s
Description: S/MIME Cryptographic Signature