Re: kernel BUG at fs/btrfs/extent_io.c:1989

2017-09-20 Thread Liu Bo 
On Wed, Sep 20, 2017 at 02:53:57PM +0200, David Sterba wrote:
> On Tue, Sep 19, 2017 at 10:12:39AM -0600, Liu Bo wrote:
> > On Tue, Sep 19, 2017 at 05:07:25PM +0200, David Sterba wrote:
> > > On Tue, Sep 19, 2017 at 11:32:46AM +, Paul Jones wrote:
> > > > > This 'mirror 0' looks fishy, (as mirror comes from 
> > > > > btrfs_io_bio->mirror_num,
> > > > > which should be at least 1 if raid1 setup is in use.)
> > > > > 
> > > > > Not sure if 4.13.2-gentoo made any changes on btrfs, but can you 
> > > > > please
> > > > > verify with the upstream kernel, say, v4.13?
> > > > 
> > > > It's basically a vanilla kernel with a handful of unrelated patches.
> > > > The filesystem fell apart overnight, there were a few thousand
> > > > checksum errors and eventually it went read-only. I tried to remount
> > > > it, but got open_ctree failed. Btrfs check segfaulted, lowmem mode
> > > > completed with so many errors I gave up and will restore from the
> > > > backup.
> > > > 
> > > > I think I know the problem now - the lvm cache was in writeback mode
> > > > (by accident) so during a defrag there would be gigabytes of unwritten
> > > > data in memory only, which was all lost when the system crashed
> > > > (motherboard failure). No wonder the filesystem didn't quite survive.
> > > 
> > > Yeah, the caching layer was my first suspicion, and lack of propagating
> > > of the barriers. Good that you were able to confirm that as the root 
> > > cause.
> > > 
> > > > I must say though, I'm seriously impressed at the data integrity of
> > > > BTRFS - there were near 10,000 checksum errors, 4 which were
> > > > uncorrectable, and from what I could tell nearly all of the data was
> > > > still intact according to rsync checksums.
> > > 
> > > Yay!
> > 
> > But still don't get why mirror_num is 0, do you have an idea on how
> > does writeback cache make that?
> 
> My first idea was that the cached blocks were zeroed, so we'd see the ino
> and mirror as 0. But this is not correct as the blocks would not pass
> the checksum tests, so the blocks must be from some previous generation.
> Ie. the transid verify failure. And all the error reports appear after
> that so I'm slightly suspicious about the way it's actually reported.
> 
> btrfs_print_data_csum_error takes mirror from either io_bio or
> compressed_bio structures, so there might be a case when the structures
> are initialized. If the transid check is ok, then the structures are
> updated. If the check fails we'd see the initial mirror number. All of
> that is just a hypothesis, I haven't checked with the code.
>

Thanks a lot for the input, you're right, mirror_num 0 should come
from compressed read where it doesn't record the bbio->mirror_num but
the mirror passing from the upper layer, and it's not metadata as we
don't yet compress metadata, so this all makes sense.

I think it also disables the ability of read-repair from raid1 for
compressed data, and that's what caused the bug where it hits
BUG_ON(mirror_num == 0) in cleanup_io_failure().

The good news is that I can reproduce it, will send a patch and a
testcase.

> I don't have a theoretical explanation for the ino 0. The inode pointer
> that goes to btrfs_print_data_csum_error should be from a properly
> initialized inode and we print the number using btrfs_ino. That will use
> the vfs i_ino value and we should never get 0 out of that.

ino 0 comes from metadata read-repair, some cleanup may be needed to
make it less confusing.

thanks,
-liubo
--
To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: kernel BUG at fs/btrfs/extent_io.c:1989

2017-09-20 Thread David Sterba 
On Tue, Sep 19, 2017 at 10:12:39AM -0600, Liu Bo wrote:
> On Tue, Sep 19, 2017 at 05:07:25PM +0200, David Sterba wrote:
> > On Tue, Sep 19, 2017 at 11:32:46AM +, Paul Jones wrote:
> > > > This 'mirror 0' looks fishy, (as mirror comes from 
> > > > btrfs_io_bio->mirror_num,
> > > > which should be at least 1 if raid1 setup is in use.)
> > > > 
> > > > Not sure if 4.13.2-gentoo made any changes on btrfs, but can you please
> > > > verify with the upstream kernel, say, v4.13?
> > > 
> > > It's basically a vanilla kernel with a handful of unrelated patches.
> > > The filesystem fell apart overnight, there were a few thousand
> > > checksum errors and eventually it went read-only. I tried to remount
> > > it, but got open_ctree failed. Btrfs check segfaulted, lowmem mode
> > > completed with so many errors I gave up and will restore from the
> > > backup.
> > > 
> > > I think I know the problem now - the lvm cache was in writeback mode
> > > (by accident) so during a defrag there would be gigabytes of unwritten
> > > data in memory only, which was all lost when the system crashed
> > > (motherboard failure). No wonder the filesystem didn't quite survive.
> > 
> > Yeah, the caching layer was my first suspicion, and lack of propagating
> > of the barriers. Good that you were able to confirm that as the root cause.
> > 
> > > I must say though, I'm seriously impressed at the data integrity of
> > > BTRFS - there were near 10,000 checksum errors, 4 which were
> > > uncorrectable, and from what I could tell nearly all of the data was
> > > still intact according to rsync checksums.
> > 
> > Yay!
> 
> But still don't get why mirror_num is 0, do you have an idea on how
> does writeback cache make that?

My first idea was that the cached blocks were zeroed, so we'd see the ino
and mirror as 0. But this is not correct as the blocks would not pass
the checksum tests, so the blocks must be from some previous generation.
Ie. the transid verify failure. And all the error reports appear after
that so I'm slightly suspicious about the way it's actually reported.

btrfs_print_data_csum_error takes mirror from either io_bio or
compressed_bio structures, so there might be a case when the structures
are initialized. If the transid check is ok, then the structures are
updated. If the check fails we'd see the initial mirror number. All of
that is just a hypothesis, I haven't checked with the code.

I don't have a theoretical explanation for the ino 0. The inode pointer
that goes to btrfs_print_data_csum_error should be from a properly
initialized inode and we print the number using btrfs_ino. That will use
the vfs i_ino value and we should never get 0 out of that.
--
To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: kernel BUG at fs/btrfs/extent_io.c:1989

2017-09-19 Thread Liu Bo 
On Tue, Sep 19, 2017 at 05:07:25PM +0200, David Sterba wrote:
> On Tue, Sep 19, 2017 at 11:32:46AM +, Paul Jones wrote:
> > > This 'mirror 0' looks fishy, (as mirror comes from 
> > > btrfs_io_bio->mirror_num,
> > > which should be at least 1 if raid1 setup is in use.)
> > > 
> > > Not sure if 4.13.2-gentoo made any changes on btrfs, but can you please
> > > verify with the upstream kernel, say, v4.13?
> > 
> > It's basically a vanilla kernel with a handful of unrelated patches.
> > The filesystem fell apart overnight, there were a few thousand
> > checksum errors and eventually it went read-only. I tried to remount
> > it, but got open_ctree failed. Btrfs check segfaulted, lowmem mode
> > completed with so many errors I gave up and will restore from the
> > backup.
> > 
> > I think I know the problem now - the lvm cache was in writeback mode
> > (by accident) so during a defrag there would be gigabytes of unwritten
> > data in memory only, which was all lost when the system crashed
> > (motherboard failure). No wonder the filesystem didn't quite survive.
> 
> Yeah, the caching layer was my first suspicion, and lack of propagating
> of the barriers. Good that you were able to confirm that as the root cause.
> 
> > I must say though, I'm seriously impressed at the data integrity of
> > BTRFS - there were near 10,000 checksum errors, 4 which were
> > uncorrectable, and from what I could tell nearly all of the data was
> > still intact according to rsync checksums.
> 
> Yay!

But still don't get why mirror_num is 0, do you have an idea on how
does writeback cache make that?

Thanks,

-liubo
--
To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: kernel BUG at fs/btrfs/extent_io.c:1989

2017-09-19 Thread David Sterba 
On Tue, Sep 19, 2017 at 11:32:46AM +, Paul Jones wrote:
> > This 'mirror 0' looks fishy, (as mirror comes from btrfs_io_bio->mirror_num,
> > which should be at least 1 if raid1 setup is in use.)
> > 
> > Not sure if 4.13.2-gentoo made any changes on btrfs, but can you please
> > verify with the upstream kernel, say, v4.13?
> 
> It's basically a vanilla kernel with a handful of unrelated patches.
> The filesystem fell apart overnight, there were a few thousand
> checksum errors and eventually it went read-only. I tried to remount
> it, but got open_ctree failed. Btrfs check segfaulted, lowmem mode
> completed with so many errors I gave up and will restore from the
> backup.
> 
> I think I know the problem now - the lvm cache was in writeback mode
> (by accident) so during a defrag there would be gigabytes of unwritten
> data in memory only, which was all lost when the system crashed
> (motherboard failure). No wonder the filesystem didn't quite survive.

Yeah, the caching layer was my first suspicion, and lack of propagating
of the barriers. Good that you were able to confirm that as the root cause.

> I must say though, I'm seriously impressed at the data integrity of
> BTRFS - there were near 10,000 checksum errors, 4 which were
> uncorrectable, and from what I could tell nearly all of the data was
> still intact according to rsync checksums.

Yay!
--
To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

RE: kernel BUG at fs/btrfs/extent_io.c:1989

2017-09-19 Thread Paul Jones 
> -Original Message-
> From: Liu Bo [mailto:bo.li@oracle.com]
> Sent: Tuesday, 19 September 2017 3:10 AM
> To: Paul Jones <p...@pauljones.id.au>
> Cc: linux-btrfs@vger.kernel.org
> Subject: Re: kernel BUG at fs/btrfs/extent_io.c:1989
> 
> 
> This 'mirror 0' looks fishy, (as mirror comes from btrfs_io_bio->mirror_num,
> which should be at least 1 if raid1 setup is in use.)
> 
> Not sure if 4.13.2-gentoo made any changes on btrfs, but can you please
> verify with the upstream kernel, say, v4.13?

It's basically a vanilla kernel with a handful of unrelated patches.
The filesystem fell apart overnight, there were a few thousand checksum errors 
and eventually it went read-only. I tried to remount it, but got open_ctree 
failed. Btrfs check segfaulted, lowmem mode completed with so many errors I 
gave up and will restore from the backup.

I think I know the problem now - the lvm cache was in writeback mode (by 
accident) so during a defrag there would be gigabytes of unwritten data in 
memory only, which was all lost when the system crashed (motherboard failure). 
No wonder the filesystem didn't quite survive.

I must say though, I'm seriously impressed at the data integrity of BTRFS - 
there were near 10,000 checksum errors, 4 which were uncorrectable, and from 
what I could tell nearly all of the data was still intact according to rsync 
checksums.

Cheers,
Paul.
--
To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: kernel BUG at fs/btrfs/extent_io.c:1989

2017-09-18 Thread Kai Krakow 
Am Mon, 18 Sep 2017 20:30:41 +0200
schrieb Holger Hoffstätte :

> On 09/18/17 19:09, Liu Bo wrote:
> > This 'mirror 0' looks fishy, (as mirror comes from
> > btrfs_io_bio->mirror_num, which should be at least 1 if raid1 setup
> > is in use.)
> > 
> > Not sure if 4.13.2-gentoo made any changes on btrfs, but can you  
> 
> No, it did not; Gentoo always strives to be as close to mainline as
> possible except for urgent security & low-risk convenience fixes.

According to
https://dev.gentoo.org/~mpagano/genpatches/patches-4.13-2.htm
it's not only security patches.

But as the list shows, there are indeed no btrfs patches. But there's
one that may change btrfs behavior (tho unlikely), that is enabling
native gcc optimizations if you choose so. I don't think that's a
default option in Gentoo.

I'm using native optimizations myself and see no strange mirror issues
in btrfs. OTOH, I've lately switched to gentoo ck patchset to get
better optimizations for gaming and realtime apps. But it's still at
the 4.12 series.

Are you sure the system crashed and wasn't just stuck at reading from
the disks? If the disks have error correction and recovery enabled, the
Linux block layer times out on the requests that the drives eventually
won't fix anyways and resets the link after 30s. The drive timeout is
120s by default.

You can change that on enterprise grade and NAS-ready drives, also a
handful of desktop drives support it. Smartctl is used to set the
values, just google "smartctl scterc". You could also adjust the
timeout of the scsi layer to above the drive timeout, that means more
than 120s if you cannot change scterc. I think it makes most sense to
not reset the link before the drive had its chance to answer the
request.

I think there are pros and cons of changing these values. I always
recommend to increase the scsi timeout above the scterc timeout.
Personally, I lower the scterc timeout to 70 centisecs, and let the
scsi timeout just at its default. RAID setups should use this to get
control of their own error correction methods: The drive returns from
request early and the RAID can do its job of reading from another copy,
i.e. btrfs or mdraid, then repair it by writing back a correct copy
which the drive converts into a sector relocation aka self-repair.

Other people may jump in and recommend their own perspective of why or
why not change which knob to which value.

But well, as long as you saw no scsi errors reported when the "crash"
occurred, these values are not involved in your problem anyways.

What about "btrfs device stats"?


-- 
Regards,
Kai

Replies to list-only preferred.


--
To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: kernel BUG at fs/btrfs/extent_io.c:1989

2017-09-18 Thread Holger Hoffstätte 
On 09/18/17 19:09, Liu Bo wrote:
> This 'mirror 0' looks fishy, (as mirror comes from
> btrfs_io_bio->mirror_num, which should be at least 1 if raid1 setup is
> in use.)
> 
> Not sure if 4.13.2-gentoo made any changes on btrfs, but can you

No, it did not; Gentoo always strives to be as close to mainline as
possible except for urgent security & low-risk convenience fixes.

-h
--
To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: kernel BUG at fs/btrfs/extent_io.c:1989

2017-09-18 Thread Liu Bo 
On Mon, Sep 18, 2017 at 08:55:29AM +, Paul Jones wrote:
> Hi
>  I have a system that crashed during a defrag, upon reboot I got the 
> following trace while resuming the defrag.
> Filesystem is BTRFS Raid1 on lvm+cache, kernel 4.13.2
> Check --repair gives lots of warnings about parent transid verify failed, but 
> otherwise completes without issue.
> 
> Ran scrub which seems to have fixed most of the issues without crashing:
> 
> scrub status for d844164a-239e-4f37-9126-d3b2f3ab72be
> scrub started at Mon Sep 18 15:59:05 2017 and finished after 02:04:00
> total bytes scrubbed: 2.22TiB with 22890 errors
> error details: verify=1078 csum=21812
> corrected errors: 22886, uncorrectable errors: 4, unverified errors: 0
> 
> I'll see how it goes when I use rsync to verify from the other backup.
> 
> Thanks,
> Paul.
> 
> <...>
> [  136.376559] BTRFS info (device dm-15): read error corrected: ino 0 off 
> 6517887115264 (dev /dev/mapper/lvmB-backup--b sector 178527312)
> [  136.376659] BTRFS info (device dm-15): read error corrected: ino 0 off 
> 6517887119360 (dev /dev/mapper/lvmB-backup--b sector 178527320)
> [  174.761517] BTRFS warning (device dm-15): csum failed root 7692 ino 534939 
> off 5639217152 csum 0xdbbb090f expected csum 0x74d6a9b2 mirror 0
> [  174.761800] BTRFS warning (device dm-15): csum failed root 7692 ino 534939 
> off 5639217152 csum 0xdbbb090f expected csum 0x74d6a9b2 mirror 0
> [  174.761838] BTRFS warning (device dm-15): csum failed root 7692 ino 534939 
> off 5639217152 csum 0xdbbb090f expected csum 0x74d6a9b2 mirror 0
> [  174.761880] BTRFS warning (device dm-15): csum failed root 7692 ino 534939 
> off 5639217152 csum 0xdbbb090f expected csum 0x74d6a9b2 mirror 0
> [  174.761924] BTRFS warning (device dm-15): csum failed root 7692 ino 534939 
> off 5639217152 csum 0xdbbb090f expected csum 0x74d6a9b2 mirror 0

This 'mirror 0' looks fishy, (as mirror comes from
btrfs_io_bio->mirror_num, which should be at least 1 if raid1 setup is
in use.)

Not sure if 4.13.2-gentoo made any changes on btrfs, but can you
please verify with the upstream kernel, say, v4.13?


Thanks,

-liubo

> [  174.761986] [ cut here ]
> [  174.761987] kernel BUG at fs/btrfs/extent_io.c:1989!
> [  174.761989] invalid opcode:  [#1] SMP
> [  174.762034] Modules linked in: cls_u32 sch_htb sch_sfq nf_conntrack_pptp 
> nf_conntrack_proto_gre nf_conntrack_sane nf_conntrack_sip ts_kmp 
> nf_conntrack_amanda nf_conntrack_snmp nf_conntrack_h323 
> nf_conntrack_netbios_ns nf_conntrack_broadcast nf_conntrack_tftp 
> nf_conntrack_ftp nf_conntrack_irc xt_NETMAP xt_TCPMSS xt_CHECKSUM 
> ipt_rpfilter xt_DSCP xt_dscp xt_statistic xt_CT xt_AUDIT xt_NFLOG xt_time 
> xt_connlimit xt_realm xt_NFQUEUE xt_tcpmss xt_addrtype xt_pkttype iptable_raw 
> xt_TPROXY nf_defrag_ipv6 xt_CLASSIFY xt_mark xt_hashlimit xt_comment 
> xt_length xt_connmark xt_owner xt_recent xt_iprange xt_physdev xt_policy 
> iptable_mangle xt_nat xt_multiport xt_conntrack ipt_REJECT nf_reject_ipv4 
> ipt_MASQUERADE nf_nat_masquerade_ipv4 ipt_ECN ipt_CLUSTERIP ipt_ah 
> iptable_nat nf_conntrack_ipv4 nf_defrag_ipv4 nf_nat_ipv4
> [  174.762144]  nf_nat
> [  174.762145] --------[ cut here ]----
> [  174.762145] kernel BUG at fs/btrfs/extent_io.c:1989!
> [  174.762267] --------[ cut here ]----
> [  174.762268] kernel BUG at fs/btrfs/extent_io.c:1989!
> [  174.762334] [ cut here ]
> [  174.762335] kernel BUG at fs/btrfs/extent_io.c:1989!
> [  174.762487]  iptable_filter ip_tables nfsd auth_rpcgss oid_registry 
> nfs_acl binfmt_misc dm_cache_smq dm_cache dm_persistent_data dm_bufio 
> dm_bio_prison k10temp intel_powerclamp coretemp pcbc hwmon_vid iTCO_wdt 
> iTCO_vendor_support aesni_intel crypto_simd cryptd glue_helper pcspkr lpc_ich 
> i2c_i801 mfd_core xts aes_x86_64 cbc sha512_generic iscsi_tcp libiscsi_tcp 
> libiscsi scsi_transport_iscsi ixgb macvlan igb dca i2c_algo_bit e1000 atl1c 
> fuse nfs lockd grace sunrpc dm_mirror dm_region_hash dm_log dm_mod 
> hid_sunplus hid_sony hid_samsung hid_pl hid_petalynx hid_gyration 
> xhci_plat_hcd ohci_pci ohci_hcd uhci_hcd usb_storage megaraid_sas 
> megaraid_mbox megaraid_mm megaraid mptsas scsi_transport_sas mptspi 
> scsi_transport_spi mptscsih mptbase sata_inic162x ata_piix sata_nv sata_sil24 
> pata_jmicron pata_amd pata_mpiix
> [  174.762629]  usbhid ahci libahci xhci_pci r8169 ehci_pci xhci_hcd mii 
> ehci_hcd
> [  174.762682] CPU: 5 PID: 6683 Comm: kworker/u16:22 Not tainted 
> 4.13.2-gentoo #5
> [  174.762730] Hardware name: System manufacturer System Product Name/P8Z68-V 
> LE, BIOS 4101 05/09/2013
> [  174.762786] Workqueue: btrfs-endio btrfs_endio_helper
> [  174.76

kernel BUG at fs/btrfs/extent_io.c:1989

2017-09-18 Thread Paul Jones 
 error (device dm-15): parent transid verify failed on 
6522470612992 wanted 1044457 found 1044206
[  127.189398] BTRFS error (device dm-15): parent transid verify failed on 
4594090442752 wanted 1044480 found 1044432
[  127.189899] BTRFS error (device dm-15): parent transid verify failed on 
4594090475520 wanted 1044480 found 1039157
[  127.190503] BTRFS error (device dm-15): parent transid verify failed on 
4594090491904 wanted 1044480 found 1044432
[  131.372119] verify_parent_transid: 80 callbacks suppressed
[  131.372122] BTRFS error (device dm-15): parent transid verify failed on 
6815879430144 wanted 1044473 found 1044169
[  131.373039] repair_io_failure: 350 callbacks suppressed
[  131.373041] BTRFS info (device dm-15): read error corrected: ino 0 off 
6815879430144 (dev /dev/mapper/lvmB-backup--b sector 2327116096)
[  131.373176] BTRFS info (device dm-15): read error corrected: ino 0 off 
6815879434240 (dev /dev/mapper/lvmB-backup--b sector 2327116104)
[  131.373302] BTRFS info (device dm-15): read error corrected: ino 0 off 
6815879438336 (dev /dev/mapper/lvmB-backup--b sector 2327116112)
[  131.373438] BTRFS info (device dm-15): read error corrected: ino 0 off 
6815879442432 (dev /dev/mapper/lvmB-backup--b sector 2327116120)
[  131.378081] BTRFS error (device dm-15): parent transid verify failed on 
4594106400768 wanted 1044480 found 1039472
[  131.378404] BTRFS info (device dm-15): read error corrected: ino 0 off 
4594106400768 (dev /dev/mapper/lvm-backup--a sector 387762752)
[  131.378441] BTRFS info (device dm-15): read error corrected: ino 0 off 
4594106404864 (dev /dev/mapper/lvm-backup--a sector 387762760)
[  131.378577] BTRFS info (device dm-15): read error corrected: ino 0 off 
4594106408960 (dev /dev/mapper/lvm-backup--a sector 387762768)
[  131.378614] BTRFS info (device dm-15): read error corrected: ino 0 off 
4594106413056 (dev /dev/mapper/lvm-backup--a sector 387762776)
[  131.468158] BTRFS error (device dm-15): parent transid verify failed on 
6516948828160 wanted 1044481 found 1044069
[  131.468271] BTRFS error (device dm-15): parent transid verify failed on 
6516948795392 wanted 1044481 found 1039285
[  131.469242] BTRFS info (device dm-15): read error corrected: ino 0 off 
6516948828160 (dev /dev/mapper/lvm-backup--a sector 315129280)
[  131.469260] BTRFS info (device dm-15): read error corrected: ino 0 off 
6516948795392 (dev /dev/mapper/lvm-backup--a sector 315129216)
[  131.469417] BTRFS error (device dm-15): parent transid verify failed on 
6516948811776 wanted 1044481 found 1044069
[  131.510220] BTRFS error (device dm-15): parent transid verify failed on 
6517287829504 wanted 1044482 found 1043368
[  131.510966] BTRFS error (device dm-15): parent transid verify failed on 
6517287862272 wanted 1044482 found 1043368
[  131.756488] BTRFS error (device dm-15): parent transid verify failed on 
4592607641600 wanted 1044479 found 1006875
[  131.801195] BTRFS error (device dm-15): parent transid verify failed on 
4592608034816 wanted 1044479 found 248480
[  131.895363] BTRFS error (device dm-15): parent transid verify failed on 
6523168260096 wanted 1044486 found 1044169
[  136.376338] repair_io_failure: 86 callbacks suppressed
[  136.376343] BTRFS info (device dm-15): read error corrected: ino 0 off 
6517887107072 (dev /dev/mapper/lvmB-backup--b sector 178527296)
[  136.376464] BTRFS info (device dm-15): read error corrected: ino 0 off 
651788768 (dev /dev/mapper/lvmB-backup--b sector 178527304)
[  136.376559] BTRFS info (device dm-15): read error corrected: ino 0 off 
6517887115264 (dev /dev/mapper/lvmB-backup--b sector 178527312)
[  136.376659] BTRFS info (device dm-15): read error corrected: ino 0 off 
6517887119360 (dev /dev/mapper/lvmB-backup--b sector 178527320)
[  174.761517] BTRFS warning (device dm-15): csum failed root 7692 ino 534939 
off 5639217152 csum 0xdbbb090f expected csum 0x74d6a9b2 mirror 0
[  174.761800] BTRFS warning (device dm-15): csum failed root 7692 ino 534939 
off 5639217152 csum 0xdbbb090f expected csum 0x74d6a9b2 mirror 0
[  174.761838] BTRFS warning (device dm-15): csum failed root 7692 ino 534939 
off 5639217152 csum 0xdbbb090f expected csum 0x74d6a9b2 mirror 0
[  174.761880] BTRFS warning (device dm-15): csum failed root 7692 ino 534939 
off 5639217152 csum 0xdbbb090f expected csum 0x74d6a9b2 mirror 0
[  174.761924] BTRFS warning (device dm-15): csum failed root 7692 ino 534939 
off 5639217152 csum 0xdbbb090f expected csum 0x74d6a9b2 mirror 0
[  174.761986] [ cut here ]
[  174.761987] kernel BUG at fs/btrfs/extent_io.c:1989!
[  174.761989] invalid opcode:  [#1] SMP
[  174.762034] Modules linked in: cls_u32 sch_htb sch_sfq nf_conntrack_pptp 
nf_conntrack_proto_gre nf_conntrack_sane nf_conntrack_sip ts_kmp 
nf_conntrack_amanda nf_conntrack_snmp nf_conntrack_h323 nf_conntrack_netbios_ns 
nf_conntrack_broadcast nf_conntrack_tftp nf_conntrack_ftp nf_conntrack_irc 
xt_NETMAP xt_TCPMSS xt_CHECKSUM ipt_rpfilter xt_DSCP xt_dscp