I have a failing 2TB disk that is part of a 4 disk RAID 6 system. I have added a new 2TB disk to the computer, and started a BTRFS replace for the old and new disk. The process starts correctly however some hours into the job, there is an error and kernel oops. relevant log below.
The disks are configured on top of bcache, in 5 arrays with a small 128GB SSD cache shared. The system in this configuration has worked perfectly for 3 years, until 2 weeks ago csum errors started appearing. I have a crashplan backup of all files on the disk, so I am not concerned about data loss, but I would like to avoid rebuild the system. btrfs dev stats shows [/dev/bcache0].write_io_errs 0 [/dev/bcache0].read_io_errs 0 [/dev/bcache0].flush_io_errs 0 [/dev/bcache0].corruption_errs 0 [/dev/bcache0].generation_errs 0 [/dev/bcache1].write_io_errs 0 [/dev/bcache1].read_io_errs 20 [/dev/bcache1].flush_io_errs 0 [/dev/bcache1].corruption_errs 0 [/dev/bcache1].generation_errs 14 [/dev/bcache3].write_io_errs 0 [/dev/bcache3].read_io_errs 0 [/dev/bcache3].flush_io_errs 0 [/dev/bcache3].corruption_errs 0 [/dev/bcache3].generation_errs 19 [/dev/bcache2].write_io_errs 0 [/dev/bcache2].read_io_errs 0 [/dev/bcache2].flush_io_errs 0 [/dev/bcache2].corruption_errs 0 [/dev/bcache2].generation_errs 2 and a smart test of the backing disk /dev/bcache1 shows a high read error rate, and lot of reallocated sectors. The disk is 10 years old, and has clearly started to fail. I've tried the latest kernel, and the latest tools, but nothing will allow me to replace, or delete the failed disk. 884.171025] BTRFS info (device bcache0): dev_replace from /dev/bcache1 (devid 2) to /dev/bcache4 started [ 3301.101958] BTRFS error (device bcache0): parent transid verify failed on 8251260944384 wanted 640926 found 640907 [ 3301.241214] BTRFS error (device bcache0): parent transid verify failed on 8251260944384 wanted 640926 found 640907 [ 3301.241398] BTRFS error (device bcache0): parent transid verify failed on 8251260944384 wanted 640926 found 640907 [ 3301.241513] BTRFS error (device bcache0): parent transid verify failed on 8251260944384 wanted 640926 found 640907 [ 3302.381094] BTRFS error (device bcache0): btrfs_scrub_dev(/dev/bcache1, 2, /dev/bcache4) failed -5 [ 3302.394612] WARNING: CPU: 0 PID: 5936 at /build/linux-5s7Xkn/linux-4.15.0/fs/btrfs/dev-replace.c:413 btrfs_dev_replace_start+0x281/0x320 [btrfs] [ 3302.394613] Modules linked in: btrfs zstd_compress xor raid6_pq bcache intel_rapl x86_pkg_temp_thermal intel_powerclamp snd_hda_codec_hdmi coretemp kvm_intel snd_hda_codec_realtek kvm snd_hda_codec_generic snd_hda_intel snd_hda_codec snd_hda_core irqbypass crct10dif_pclmul crc32_pclmul ghash_clmulni_intel snd_hwdep snd_pcm pcbc snd_seq_midi aesni_intel snd_seq_midi_event joydev input_leds aes_x86_64 snd_rawmidi crypto_simd glue_helper snd_seq eeepc_wmi cryptd asus_wmi snd_seq_device snd_timer wmi_bmof sparse_keymap snd intel_cstate intel_rapl_perf soundcore mei_me mei shpchp mac_hid sch_fq_codel acpi_pad parport_pc ppdev lp parport ip_tables x_tables autofs4 overlay nls_iso8859_1 dm_mirror dm_region_hash dm_log hid_generic usbhid hid uas usb_storage i915 i2c_algo_bit drm_kms_helper syscopyarea sysfillrect [ 3302.394640] sysimgblt fb_sys_fops r8169 mxm_wmi mii drm ahci libahci wmi video [ 3302.394646] CPU: 0 PID: 5936 Comm: btrfs Not tainted 4.15.0-20-generic #21-Ubuntu [ 3302.394646] Hardware name: System manufacturer System Product Name/H110M-R, BIOS 3404 10/10/2017 [ 3302.394658] RIP: 0010:btrfs_dev_replace_start+0x281/0x320 [btrfs] [ 3302.394659] RSP: 0018:ffffa8b582b5fd18 EFLAGS: 00010282 [ 3302.394660] RAX: 00000000fffffffb RBX: ffff927d3afe0000 RCX: 0000000000000000 [ 3302.394660] RDX: 0000000000000001 RSI: 0000000000000296 RDI: ffff927d3afece90 [ 3302.394661] RBP: ffffa8b582b5fd68 R08: 0000000000000000 R09: ffffa8b582b5fc18 [ 3302.394662] R10: ffffa8b582b5fd10 R11: 0000000000000000 R12: ffff927d3afece20 [ 3302.394662] R13: ffff927d34b59421 R14: ffff927d34b59020 R15: 0000000000000001 [ 3302.394663] FS: 00007fba4831c8c0(0000) GS:ffff927df6c00000(0000) knlGS:0000000000000000 [ 3302.394664] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 3302.394664] CR2: 00002b9b83db85b8 CR3: 0000000164d3a002 CR4: 00000000003606f0 [ 3302.394665] Call Trace: [ 3302.394676] btrfs_dev_replace_by_ioctl+0x39/0x60 [btrfs] [ 3302.394686] btrfs_ioctl+0x1988/0x2080 [btrfs] [ 3302.394689] ? iput+0x8d/0x220 [ 3302.394690] ? __blkdev_put+0x199/0x1f0 [ 3302.394692] do_vfs_ioctl+0xa8/0x630 [ 3302.394701] ? btrfs_ioctl_get_supported_features+0x30/0x30 [btrfs] [ 3302.394703] ? do_vfs_ioctl+0xa8/0x630 [ 3302.394704] ? do_sigaction+0xb4/0x1e0 [ 3302.394706] SyS_ioctl+0x79/0x90 [ 3302.394708] do_syscall_64+0x73/0x130 [ 3302.394710] entry_SYSCALL_64_after_hwframe+0x3d/0xa2 [ 3302.394711] RIP: 0033:0x7fba471085d7 [ 3302.394712] RSP: 002b:00007ffe5af753b8 EFLAGS: 00000246 ORIG_RAX: 0000000000000010 [ 3302.394713] RAX: ffffffffffffffda RBX: 000055a8eecfb2a0 RCX: 00007fba471085d7 [ 3302.394713] RDX: 00007ffe5af757f8 RSI: 00000000ca289435 RDI: 0000000000000003 [ 3302.394714] RBP: 00000000ffffffff R08: 0000000000000000 R09: 00007fba47160f00 [ 3302.394715] R10: 0000000000000008 R11: 0000000000000246 R12: 00007ffe5af7880d [ 3302.394715] R13: 0000000000000003 R14: 0000000000000000 R15: 0000000000000004 [ 3302.394716] Code: 88 98 00 00 00 48 8b b0 90 00 00 00 6a 01 e8 67 9b fe ff 48 89 df 89 c6 e8 0d f8 ff ff 83 f8 8d 5a 74 6d 85 c0 0f 84 19 fe ff ff <0f> 0b e9 12 fe ff ff 89 c2 48 c7 c6 81 c1 c1 c0 48 89 df e8 f7 [ 3302.394736] ---[ end trace a5f8501fc7a5d644 ]---e
