[no subject]
I have a failing 2TB disk that is part of a 4 disk RAID 6 system. I have added a new 2TB disk to the computer, and started a BTRFS replace for the old and new disk. The process starts correctly however some hours into the job, there is an error and kernel oops. relevant log below. The disks are configured on top of bcache, in 5 arrays with a small 128GB SSD cache shared. The system in this configuration has worked perfectly for 3 years, until 2 weeks ago csum errors started appearing. I have a crashplan backup of all files on the disk, so I am not concerned about data loss, but I would like to avoid rebuild the system. btrfs dev stats shows [/dev/bcache0].write_io_errs0 [/dev/bcache0].read_io_errs 0 [/dev/bcache0].flush_io_errs0 [/dev/bcache0].corruption_errs 0 [/dev/bcache0].generation_errs 0 [/dev/bcache1].write_io_errs0 [/dev/bcache1].read_io_errs 20 [/dev/bcache1].flush_io_errs0 [/dev/bcache1].corruption_errs 0 [/dev/bcache1].generation_errs 14 [/dev/bcache3].write_io_errs0 [/dev/bcache3].read_io_errs 0 [/dev/bcache3].flush_io_errs0 [/dev/bcache3].corruption_errs 0 [/dev/bcache3].generation_errs 19 [/dev/bcache2].write_io_errs0 [/dev/bcache2].read_io_errs 0 [/dev/bcache2].flush_io_errs0 [/dev/bcache2].corruption_errs 0 [/dev/bcache2].generation_errs 2 and a smart test of the backing disk /dev/bcache1 shows a high read error rate, and lot of reallocated sectors. The disk is 10 years old, and has clearly started to fail. I've tried the latest kernel, and the latest tools, but nothing will allow me to replace, or delete the failed disk. 884.171025] BTRFS info (device bcache0): dev_replace from /dev/bcache1 (devid 2) to /dev/bcache4 started [ 3301.101958] BTRFS error (device bcache0): parent transid verify failed on 8251260944384 wanted 640926 found 640907 [ 3301.241214] BTRFS error (device bcache0): parent transid verify failed on 8251260944384 wanted 640926 found 640907 [ 3301.241398] BTRFS error (device bcache0): parent transid verify failed on 8251260944384 wanted 640926 found 640907 [ 3301.241513] BTRFS error (device bcache0): parent transid verify failed on 8251260944384 wanted 640926 found 640907 [ 3302.381094] BTRFS error (device bcache0): btrfs_scrub_dev(/dev/bcache1, 2, /dev/bcache4) failed -5 [ 3302.394612] WARNING: CPU: 0 PID: 5936 at /build/linux-5s7Xkn/linux-4.15.0/fs/btrfs/dev-replace.c:413 btrfs_dev_replace_start+0x281/0x320 [btrfs] [ 3302.394613] Modules linked in: btrfs zstd_compress xor raid6_pq bcache intel_rapl x86_pkg_temp_thermal intel_powerclamp snd_hda_codec_hdmi coretemp kvm_intel snd_hda_codec_realtek kvm snd_hda_codec_generic snd_hda_intel snd_hda_codec snd_hda_core irqbypass crct10dif_pclmul crc32_pclmul ghash_clmulni_intel snd_hwdep snd_pcm pcbc snd_seq_midi aesni_intel snd_seq_midi_event joydev input_leds aes_x86_64 snd_rawmidi crypto_simd glue_helper snd_seq eeepc_wmi cryptd asus_wmi snd_seq_device snd_timer wmi_bmof sparse_keymap snd intel_cstate intel_rapl_perf soundcore mei_me mei shpchp mac_hid sch_fq_codel acpi_pad parport_pc ppdev lp parport ip_tables x_tables autofs4 overlay nls_iso8859_1 dm_mirror dm_region_hash dm_log hid_generic usbhid hid uas usb_storage i915 i2c_algo_bit drm_kms_helper syscopyarea sysfillrect [ 3302.394640] sysimgblt fb_sys_fops r8169 mxm_wmi mii drm ahci libahci wmi video [ 3302.394646] CPU: 0 PID: 5936 Comm: btrfs Not tainted 4.15.0-20-generic #21-Ubuntu [ 3302.394646] Hardware name: System manufacturer System Product Name/H110M-R, BIOS 3404 10/10/2017 [ 3302.394658] RIP: 0010:btrfs_dev_replace_start+0x281/0x320 [btrfs] [ 3302.394659] RSP: 0018:a8b582b5fd18 EFLAGS: 00010282 [ 3302.394660] RAX: fffb RBX: 927d3afe RCX: [ 3302.394660] RDX: 0001 RSI: 0296 RDI: 927d3afece90 [ 3302.394661] RBP: a8b582b5fd68 R08: R09: a8b582b5fc18 [ 3302.394662] R10: a8b582b5fd10 R11: R12: 927d3afece20 [ 3302.394662] R13: 927d34b59421 R14: 927d34b59020 R15: 0001 [ 3302.394663] FS: 7fba4831c8c0() GS:927df6c0() knlGS: [ 3302.394664] CS: 0010 DS: ES: CR0: 80050033 [ 3302.394664] CR2: 2b9b83db85b8 CR3: 000164d3a002 CR4: 003606f0 [ 3302.394665] Call Trace: [ 3302.394676] btrfs_dev_replace_by_ioctl+0x39/0x60 [btrfs] [ 3302.394686] btrfs_ioctl+0x1988/0x2080 [btrfs] [ 3302.394689] ? iput+0x8d/0x220 [ 3302.394690] ? __blkdev_put+0x199/0x1f0 [ 3302.394692] do_vfs_ioctl+0xa8/0x630 [ 3302.394701] ? btrfs_ioctl_get_supported_features+0x30/0x30 [btrfs] [ 3302.394703] ? do_vfs_ioctl+0xa8/0x630 [ 3302.394704] ? do_sigaction+0xb4/0x1e0 [ 3302.394706] SyS_ioctl+0x79/0x90 [ 3302.394708] do_syscall_64+0x73/0x130 [ 3302.394710] entry_SYSCALL_64_after_hwframe+0x3d/0xa2 [ 3302.394711] RIP: 0033:0x7fba471085d7 [ 3302.394712] RSP: 002b:7ffe5af753b8 EFLAGS: 0246 ORIG_RAX: 0010 [ 3302.3947
Re:
Will capture all of that this evening, and try it with the latest kernel and tools. Thanks for the input on what info is relevant, with gather it asap. On Fri, 23 Nov 2018 at 07:53, Chris Murphy wrote: > > On Thu, Nov 22, 2018 at 11:41 PM Andy Leadbetter > wrote: > > > > I have a failing 2TB disk that is part of a 4 disk RAID 6 system. I > > have added a new 2TB disk to the computer, and started a BTRFS replace > > for the old and new disk. The process starts correctly however some > > hours into the job, there is an error and kernel oops. relevant log > > below. > > The relevant log is the entire dmesg, not a snippet. It's decently > likely there's more than one thing going on here. We also need full > output of 'smartctl -x' for all four drives, and also 'smartctl -l > scterc' for all four drives, and also 'cat > /sys/block/sda/device/timeout' for all four drives. And which bcache > mode you're using. > > The call trace provided is from kernel 4.15 which is sufficiently long > ago I think any dev working on raid56 might want to see where it's > getting tripped up on something a lot newer, and this is why: > > https://git.kernel.org/pub/scm/linux/kernel/git/stable/linux.git/diff/fs/btrfs/raid56.c?id=v4.19.3&id2=v4.15.1 > > That's a lot of changes in just the raid56 code between 4.15 and 4.19. > And then in you call trace, btrfs_dev_replace_start is found in > dev-replace.c which likewise has a lot of changes. But then also, I > think 4.15 might still be in the era where it was not recommended to > use 'btrfs dev replace' for raid56, only non-raid56. I'm not sure if > the problems with device replace were fixed, and if they were fixed > kernel or progs side. Anyway, the latest I recall, it was recommended > on raid56 to 'btrfs dev add' then 'btrfs dev remove'. > > https://git.kernel.org/pub/scm/linux/kernel/git/stable/linux.git/diff/fs/btrfs/dev-replace.c?id=v4.19.3&id2=v4.15.1 > > And that's only a few hundred changes for each. Check out inode.c - > there are over 2000 changes. > > > > The disks are configured on top of bcache, in 5 arrays with a small > > 128GB SSD cache shared. The system in this configuration has worked > > perfectly for 3 years, until 2 weeks ago csum errors started > > appearing. I have a crashplan backup of all files on the disk, so I > > am not concerned about data loss, but I would like to avoid rebuild > > the system. > > btrfs-progs 4.17 still considers raid56 experimental, not for > production use. And three years ago, the current upstream kernel > released was 4.3 so I'm gonna guess the kernel history of this file > system goes back older than that, very close to raid56 code birth. And > then adding bcache to this mix just makes it all the more complicated. > > > > > > > btrfs dev stats shows > > [/dev/bcache0].write_io_errs0 > > [/dev/bcache0].read_io_errs 0 > > [/dev/bcache0].flush_io_errs0 > > [/dev/bcache0].corruption_errs 0 > > [/dev/bcache0].generation_errs 0 > > [/dev/bcache1].write_io_errs0 > > [/dev/bcache1].read_io_errs 20 > > [/dev/bcache1].flush_io_errs0 > > [/dev/bcache1].corruption_errs 0 > > [/dev/bcache1].generation_errs 14 > > [/dev/bcache3].write_io_errs0 > > [/dev/bcache3].read_io_errs 0 > > [/dev/bcache3].flush_io_errs0 > > [/dev/bcache3].corruption_errs 0 > > [/dev/bcache3].generation_errs 19 > > [/dev/bcache2].write_io_errs0 > > [/dev/bcache2].read_io_errs 0 > > [/dev/bcache2].flush_io_errs0 > > [/dev/bcache2].corruption_errs 0 > > [/dev/bcache2].generation_errs 2 > > > 3 of 4 drives have at least one generation error. While there are no > corruptions reported, generation errors can be really tricky to > recover from at all. If only one device had only read errors, this > would be a lot less difficult. > > > > I've tried the latest kernel, and the latest tools, but nothing will > > allow me to replace, or delete the failed disk. > > If the file system is mounted, I would try to make a local backup ASAP > before you lose the whole volume. Whether it's LVM pool of two drives > (linear/concat) with XFS, or if you go with Btrfs -dsingle -mraid1 > (also basically a concat) doesn't really matter, but I'd get whatever > you can off the drive. I expect avoiding a rebuild in some form or > another is very wishful thinking and not very likely. > > The more changes are made to the file system, repair attempts or > otherwise writing to it, decreases the chance of recovery. > > -- > Chris Murphy
Kernel 4.14 RAID5 multi disk array on bcache not mounting
I have a 4 disk array on top of 120GB bcache setup, arranged as follows dev/sda1: UUID="42AE-12E3" TYPE="vfat" PARTLABEL="EFI System" PARTUUID="d337c56a-fb0f-4e87-8d5f-a89122c81167" /dev/sda2: UUID="06e3ce52-f34a-409a-a143-3c04f1d334ff" TYPE="ext4" PARTLABEL="Linux filesystem" PARTUUID="d2d3fa93-eebf-41ab-8162-d81722bf47ec" /dev/sda4: UUID="b729c490-81f0-461f-baa2-977af9a7b6d9" TYPE="bcache" PARTLABEL="Linux filesystem" PARTUUID="84548857-f504-440a-857f-c0838c1eb83d" /dev/sdb1: UUID="6016277c-143d-46b4-ae4e-8565ffc8158f" TYPE="swap" PARTLABEL="Linux swap" PARTUUID="8692bf67-7271-4bf6-a623-b79d74093f2c" /dev/sdb2: UUID="bc93c5e2-705a-4cbe-bcd9-7be1181163b2" TYPE="bcache" PARTLABEL="Linux filesystem" PARTUUID="662a450b-3592-4929-9647-8e8a1dedae69" /dev/sdc1: UUID="9df21d4e-de02-4000-b684-5fb95d4d0492" TYPE="swap" PARTLABEL="Linux swap" PARTUUID="ed9d7b8e-5480-4e70-b983-1a350ecae38a" /dev/sdc2: UUID="7d8feaf6-aa6a-4b13-af49-0ad1bd1efb64" TYPE="bcache" PARTLABEL="Linux filesystem" PARTUUID="d343e23a-39ed-4061-80a2-55b66e20ecc1" /dev/sdd1: UUID="18defba2-594b-402e-b3b2-8e38035c624d" TYPE="swap" PARTLABEL="Linux swap" PARTUUID="fed9ffd6-0480-4496-8e6d-02d263d719b7" /dev/sdd2: UUID="be0f0381-0d7e-46c9-ad04-01415bfc6f61" TYPE="bcache" PARTLABEL="Linux filesystem" PARTUUID="8f56de8a-105f-4d56-b699-59e1215b3c6b" /dev/bcache32: UUID="38d5de43-28fb-40a9-a535-dbf17ff52e75" UUID_SUB="731c31f1-51dd-477a-9bd1-fac73d0e6f69" TYPE="btrfs" /dev/sde: UUID="05514ad3-d90a-4e90-aa11-7c6d34515ca2" TYPE="bcache" /dev/bcache16: UUID="38d5de43-28fb-40a9-a535-dbf17ff52e75" UUID_SUB="79cbcaf1-40b9-4954-a977-537ed3310e76" TYPE="btrfs" /dev/bcache0: UUID="38d5de43-28fb-40a9-a535-dbf17ff52e75" UUID_SUB="42d3a0dd-fbec-4318-9a5b-6d96aa1f6328" TYPE="btrfs" /dev/bcache48: UUID="38d5de43-28fb-40a9-a535-dbf17ff52e75" UUID_SUB="cb7018d6-a27d-493e-b41f-e45c64f6873a" TYPE="btrfs" /dev/sda3: PARTUUID="d9fa3100-5044-4e10-9f2f-f8037786a43f" ubuntu 17.10 with PPA Kernels up to 4.13.x all mount this array perfectly, and the performance of the cache is as expected. Upgraded today to 4.14.1 from their PPA and the running btrfs dev scan finds the btrfs filesystem devices bcache16 and bcache32, bcache0 and bcache48 are not recognised, and thus the file system will not mount. according bcache all devices are present, and attached to the cache device correctly. btrfs fi on Kernel 4.13 gives Label: none uuid: 38d5de43-28fb-40a9-a535-dbf17ff52e75 Total devices 4 FS bytes used 2.03TiB devid1 size 1.82TiB used 1.07TiB path /dev/bcache16 devid2 size 1.82TiB used 1.07TiB path /dev/bcache32 devid3 size 1.82TiB used 1.07TiB path /dev/bcache0 devid4 size 1.82TiB used 1.07TiB path /dev/bcache48 Where do I start in debugging this? btrfs-progs v4.12 btrfs fi df / Data, RAID5: total=3.20TiB, used=2.02TiB System, RAID5: total=192.00MiB, used=288.00KiB Metadata, RAID5: total=6.09GiB, used=3.69GiB GlobalReserve, single: total=512.00MiB, used=0.00B There are no errors in the dmesg that I can see from btrfs scan, simply the two devices are not found. -- To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
