> On 3 May 2017, at 16:21, Christophe de Dinechin <dinec...@redhat.com> wrote: > >> >> On 2 May 2017, at 02:17, Qu Wenruo <quwen...@cn.fujitsu.com> wrote: >> >> >> >> At 04/28/2017 04:47 PM, Christophe de Dinechin wrote: >>>> On 28 Apr 2017, at 02:45, Qu Wenruo <quwen...@cn.fujitsu.com> wrote: >>>> >>>> >>>> >>>> At 04/26/2017 01:50 AM, Christophe de Dinechin wrote: >>>>> Hi, >>>>> I”ve been trying to run btrfs as my primary work filesystem for about 3-4 >>>>> months now on Fedora 25 systems. I ran a few times into filesystem >>>>> corruptions. At least one I attributed to a damaged disk, but the last >>>>> one is with a brand new 3T disk that reports no SMART errors. Worse yet, >>>>> in at least three cases, the filesystem corruption caused btrfsck to >>>>> crash. >>>>> The last filesystem corruption is documented here: >>>>> https://bugzilla.redhat.com/show_bug.cgi?id=1444821. The dmesg log is in >>>>> there. >>>> >>>> According to the bugzilla, the btrfs-progs seems to be too old in btrfs >>>> standard. >>>> What about using the latest btrfs-progs v4.10.2? >>> I tried 4.10.1-1 https://bugzilla.redhat.com/show_bug.cgi?id=1435567#c4. >>> I am currently debugging with a build from the master branch as of Tuesday >>> (commit bd0ab27afbf14370f9f0da1f5f5ecbb0adc654c1), which is 4.10.2 >>> There was no change in behavior. Runs are split about evenly between list >>> crash and abort. >>> I added instrumentation and tried a fix, which brings me a tiny bit >>> further, until I hit a message from delete_duplicate_records: >>> Ok we have overlapping extents that aren't completely covered by each >>> other, this is going to require more careful thought. The extents are >>> [52428800-16384] and [52432896-16384] >> >> Then I think lowmem mode may have better chance to handle it without crash. > > I tried it and got: > > [root@rescue ~]# /usr/local/bin/btrfsck --mode=lowmem --repair /dev/sda4 > enabling repair mode > ERROR: low memory mode doesn't support repair yet > > The problem only occurred in —repair mode anyway.
For what it’s worth, without the --repair option, it gets stuck. I stopped it after 24 hours, it had printed: [root@rescue ~]# /usr/local/bin/btrfsck --mode=lowmem /dev/sda4 Checking filesystem on /dev/sda4 UUID: 26a0c84c-d2ac-4da8-b880-684f2ea48a22 checking extents checksum verify failed on 52428800 found E3ADA767 wanted 7C506C03 checksum verify failed on 52428800 found E3ADA767 wanted 7C506C03 checksum verify failed on 52428800 found E3ADA767 wanted 7C506C03 checksum verify failed on 52428800 found E3ADA767 wanted 7C506C03 Csum didn't match ERROR: extent [52428800 16384] lost referencer (owner: 7, level: 0) checksum verify failed on 52445184 found 8D1BE62F wanted 00000000 checksum verify failed on 52445184 found 8D1BE62F wanted 00000000 checksum verify failed on 52445184 found 8D1BE62F wanted 00000000 checksum verify failed on 52445184 found 8D1BE62F wanted 00000000 bytenr mismatch, want=52445184, have=2199023255552 ERROR: extent [52445184 16384] lost referencer (owner: 2, level: 0) ERROR: extent[52432896 16384] backref lost (owner: 2, level: 0) ERROR: check leaf failed root 2 bytenr 52432896 level 0, force continue check Any tips for further debugging this? Christophe > > >> >>>> Furthermore for v4.10.2, btrfs check provides a new mode called lowmem. >>>> You could try "btrfs check --mode=lowmem" to see if such problem can be >>>> avoided. >>> I will try that, but what makes you think this is a memory-related >>> condition? The machine has 16G of RAM, isn’t that enough for an fsck? >> >> Not for memory usage, but in fact lowmem mode is a completely rework, so I >> just want to see how good or bad the new lowmem mode handles it. > > Is there a prototype with lowmem and repair? > > > Thanks > Christophe > >> >> Thanks, >> Qu >> >>>> >>>> For the kernel bug, it seems to be related to wrongly inserted delayed >>>> ref, but I can totally be wrong. >>> For now, I’m focusing on the “repair” part as much as I can, because I >>> assume the kernel bug is there anyway, so someone else is bound to hit this >>> problem. >>> Thanks >>> Christophe >>>> >>>> Thanks, >>>> Qu >>>>> The btrfsck crash is here: >>>>> https://bugzilla.redhat.com/show_bug.cgi?id=1435567. I have two crash >>>>> modes: either an abort or a SIGSEGV. I checked that both still happens on >>>>> master as of today. >>>>> The cause of the abort is that we call set_extent_dirty from >>>>> check_extent_refs with rec->max_size == 0. I’ve instrumented to try to >>>>> see where we set this to 0 (see >>>>> https://github.com/c3d/btrfs-progs/tree/rhbz1435567), and indeed, we do >>>>> sometimes see max_size set to 0 in a few locations. My instrumentation >>>>> shows this: >>>>> 78655 [1.792241:0x451fe0] MAX_SIZE_ZERO: Add extent rec 0x139eb80 >>>>> max_size 16384 tmpl 0x7fffffffd120 >>>>> 78657 [1.792242:0x451cb8] MAX_SIZE_ZERO: Set max size 0 for rec 0x139ec50 >>>>> from tmpl 0x7fffffffcf80 >>>>> 78660 [1.792244:0x451fe0] MAX_SIZE_ZERO: Add extent rec 0x139ed50 >>>>> max_size 16384 tmpl 0x7fffffffd120 >>>>> I don’t really know what to make of it. >>>>> The cause of the SIGSEGV is that we try to free a list entry that has its >>>>> next set to NULL. >>>>> #0 list_del (entry=0x555555db0420) at >>>>> /usr/src/debug/btrfs-progs-v4.10.1/kernel-lib/list.h:125 >>>>> #1 free_all_extent_backrefs (rec=0x555555db0350) at cmds-check.c:5386 >>>>> #2 maybe_free_extent_rec (extent_cache=0x7fffffffd990, >>>>> rec=0x555555db0350) at cmds-check.c:5417 >>>>> #3 0x00005555555b308f in check_block (flags=<optimized out>, >>>>> buf=0x55557b87cdf0, extent_cache=0x7fffffffd990, root=0x55555587d570) at >>>>> cmds-check.c:5851 >>>>> #4 run_next_block (root=root@entry=0x55555587d570, >>>>> bits=bits@entry=0x5555558841 >>>>> I don’t know if the two problems are related, but they seem to be pretty >>>>> consistent on this specific disk, so I think that we have a good >>>>> opportunity to improve btrfsck to make it more robust to this specific >>>>> form of corruption. But I don’t want to hapazardly modify a code I don’t >>>>> really understand. So if anybody could make a suggestion on what the >>>>> right strategy should be when we have max_size == 0, or how to avoid it >>>>> in the first place. >>>>> I don’t know if this is relevant at all, but all the machines that failed >>>>> that way were used to run VMs with KVM/QEMU. DIsk activity tends to be >>>>> somewhat intense on occasions, since the VMs running there are part of a >>>>> personal Jenkins ring that automatically builds various projects. >>>>> Nominally, there are between three and five guests running (Windows XP, >>>>> WIndows 10, macOS, Fedora25, Ubuntu 16.04). >>>>> Thanks >>>>> Christophe de Dinechin >>>>> -- >>>>> To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in >>>>> the body of a message to majord...@vger.kernel.org >>>>> More majordomo info at http://vger.kernel.org/majordomo-info.html >>>> >>>> >>>> -- >>>> To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in >>>> the body of a message to majord...@vger.kernel.org >>>> More majordomo info at http://vger.kernel.org/majordomo-info.html >> >> >> -- >> To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in >> the body of a message to majord...@vger.kernel.org >> More majordomo info at http://vger.kernel.org/majordomo-info.html > > -- > To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in > the body of a message to majord...@vger.kernel.org > More majordomo info at http://vger.kernel.org/majordomo-info.html -- To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
