Re: BTRFS did it's job nicely (thanks!)
Out of curiosity, what led to you choosing RAID1 for data but RAID10 for metadata? I've flip flipped between these two modes myself after finding out that BTRFS RAID10 doesn't work how I would've expected. Wondering what made you choose your configuration. Thanks! On Fri, Nov 2, 2018 at 3:55 PM waxhead wrote: > > Hi, > > my main computer runs on a 7x SSD BTRFS as rootfs with > data:RAID1 and metadata:RAID10. > > One SSD is probably about to fail, and it seems that BTRFS fixed it > nicely (thanks everyone!) > > I decided to just post the ugly details in case someone just wants to > have a look. Note that I tend to interpret the btrfs de st / output as > if the error was NOT fixed even if (seems clearly that) it was, so I > think the output is a bit misleading... just saying... > > > > -- below are the details for those curious (just for fun) --- > > scrub status for [YOINK!] > scrub started at Fri Nov 2 17:49:45 2018 and finished after > 00:29:26 > total bytes scrubbed: 1.15TiB with 1 errors > error details: csum=1 > corrected errors: 1, uncorrectable errors: 0, unverified errors: 0 > > btrfs fi us -T / > Overall: > Device size: 1.18TiB > Device allocated: 1.17TiB > Device unallocated:9.69GiB > Device missing: 0.00B > Used: 1.17TiB > Free (estimated): 6.30GiB (min: 6.30GiB) > Data ratio: 2.00 > Metadata ratio: 2.00 > Global reserve: 512.00MiB (used: 0.00B) > > Data Metadata System > Id Path RAID1 RAID10RAID10Unallocated > -- - - - - --- > 6 /dev/sda1 236.28GiB 704.00MiB 32.00MiB 485.00MiB > 7 /dev/sdb1 233.72GiB 1.03GiB 32.00MiB 2.69GiB > 2 /dev/sdc1 110.56GiB 352.00MiB - 904.00MiB > 8 /dev/sdd1 234.96GiB 1.03GiB 32.00MiB 1.45GiB > 1 /dev/sde1 164.90GiB 1.03GiB 32.00MiB 1.72GiB > 9 /dev/sdf1 109.00GiB 1.03GiB 32.00MiB 744.00MiB > 10 /dev/sdg1 107.98GiB 1.03GiB 32.00MiB 1.74GiB > -- - - - - --- > Total 598.70GiB 3.09GiB 96.00MiB 9.69GiB > Used 597.25GiB 1.57GiB 128.00KiB > > > > uname -a > Linux main 4.18.0-2-amd64 #1 SMP Debian 4.18.10-2 (2018-10-07) x86_64 > GNU/Linux > > btrfs --version > btrfs-progs v4.17 > > > dmesg | grep -i btrfs > [7.801817] Btrfs loaded, crc32c=crc32c-generic > [8.163288] BTRFS: device label btrfsroot devid 10 transid 669961 > /dev/sdg1 > [8.163433] BTRFS: device label btrfsroot devid 9 transid 669961 > /dev/sdf1 > [8.163591] BTRFS: device label btrfsroot devid 1 transid 669961 > /dev/sde1 > [8.163734] BTRFS: device label btrfsroot devid 8 transid 669961 > /dev/sdd1 > [8.163974] BTRFS: device label btrfsroot devid 2 transid 669961 > /dev/sdc1 > [8.164117] BTRFS: device label btrfsroot devid 7 transid 669961 > /dev/sdb1 > [8.164262] BTRFS: device label btrfsroot devid 6 transid 669961 > /dev/sda1 > [8.206174] BTRFS info (device sde1): disk space caching is enabled > [8.206236] BTRFS info (device sde1): has skinny extents > [8.348610] BTRFS info (device sde1): enabling ssd optimizations > [8.854412] BTRFS info (device sde1): enabling free space tree > [8.854471] BTRFS info (device sde1): using free space tree > [ 68.170580] BTRFS warning (device sde1): csum failed root 3760 ino > 3247424 off 125434560512 csum 0x2e395164 expected csum 0x6514b2c2 mirror 2 > [ 68.185973] BTRFS warning (device sde1): csum failed root 3760 ino > 3247424 off 125434560512 csum 0x2e395164 expected csum 0x6514b2c2 mirror 2 > [ 68.185991] BTRFS warning (device sde1): csum failed root 3760 ino > 3247424 off 125434560512 csum 0x2e395164 expected csum 0x6514b2c2 mirror 2 > [ 68.186003] BTRFS warning (device sde1): csum failed root 3760 ino > 3247424 off 125434560512 csum 0x2e395164 expected csum 0x6514b2c2 mirror 2 > [ 68.186015] BTRFS warning (device sde1): csum failed root 3760 ino > 3247424 off 125434560512 csum 0x2e395164 expected csum 0x6514b2c2 mirror 2 > [ 68.186028] BTRFS warning (device sde1): csum failed root 3760 ino > 3247424 off 125434560512 csum 0x2e395164 expected csum 0x6514b2c2 mirror 2 > [ 68.186041] BTRFS warning (device sde1): csum failed root 3760 ino > 3247424 off 125434560512 csum 0x2e395164 expected csum 0x6514b2c2 mirror 2 > [ 68.186052] BTRFS warning (device sde1): csum failed root 3760 ino > 3247424 off 125434560512 csum 0x2e395164 expected csum 0x6514b2c2 mirror 2 > [ 68.186063] BTRFS warning (device sde1): csum failed root 3760 ino > 3247424 off 125434560512 csum 0x2e395164 expected csum 0x6514b2c2 mirror 2 > [ 68.186075] BTRFS warning (device sde1): csum failed root 3760 ino > 3247424 off 125434560512 csum 0x2e395164 expected csum 0x6514b2c2 mirror 2 > [ 68.199237] BTRF
btrfs raid10 performance
I am running a single btrfs RAID10 volume of eight LUKS devices, each using a 2TB SATA hard drive as a backing store. The SATA drives are a mixture of Seagate and Western Digital drives, some with RPMs ranging from 5400 to 7200. Each seems to individually performance test where I would expect for drives of this caliber. They are all attached to an LSI PCIe SAS controller and configured in JBOD. I have a relatively beefy quad core Xeon CPU that supports AES-NI and don't think LUKS is my bottleneck. Here's some info from the resulting filesystem: btrfs fi df /storage Data, RAID10: total=6.30TiB, used=6.29TiB System, RAID10: total=8.00MiB, used=560.00KiB Metadata, RAID10: total=9.00GiB, used=7.64GiB GlobalReserve, single: total=512.00MiB, used=0.00B In general I see good performance, especially read performance which is enough to regularly saturate my gigabit network when copying files from this host via samba. Reads are definitely taking advantage of the multiple copies of data available and spreading the load among all drives. Writes aren't quite as rosy, however. When writing files using dd like in this example: dd if=/dev/zero of=tempfile bs=1M count=10240 conv=fdatasync,notrun c status=progress And running a command like: iostat -m 1 to monitor disk I/O, writes seem to only focus on one of the eight disks at a time, moving from one drive to the next. This results in a sustained 55-90 MB/sec throughput depending on which disk is being written to (remember, some have faster spindle speed than others). Am I wrong to expect btrfs' RAID10 mode to write to multiple disks simultaneously and to break larger writes into smaller stripes across my four pairs of disks? I had trouble identifying whether btrfs RAID10 is writing (64K?) stripes or (1GB?) blocks to disk in this mode. The latter might make more sense based upon what I'm seeing? Anything else I should be trying to narrow down the bottleneck? Thanks! -- To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
csum failed on raid1 even after clean scrub?
I am using a two disk raid1 btrfs filesystem spanning two external hard drives connected via USB 3.0. While copying ~6TB of data from this filesystem to local disk via rsync I am seeing messages like the following in dmesg output: [ 2213.406267] BTRFS warning (device sdj1): csum failed root 5 ino 830 off 2124197888 csum 0xb5da0cd2 expected csum 0x6e478250 mirror 2 [ 4890.178727] BTRFS warning (device sdj1): csum failed root 5 ino 1058 off 26052067328 csum 0x8ccd1067 expected csum 0x4adb8254 mirror 2 [27463.940218] BTRFS warning (device sdj1): csum failed root 5 ino 5372 off 7954096128 csum 0x9f9b697e expected csum 0xbd61a0e2 mirror 2 [29405.832643] BTRFS warning (device sdj1): csum failed root 5 ino 31374 off 7893983232 csum 0x12fd0ddc expected csum 0xddcd2f8e mirror 2 [31224.279082] BTRFS warning (device sdj1): csum failed root 5 ino 150903 off 183635968 csum 0xea025eb4 expected csum 0x46d64878 mirror 2 [32282.635615] BTRFS warning (device sdj1): csum failed root 5 ino 162774 off 31092424704 csum 0x1ee9b38d expected csum 0x4022e3de mirror 2 [41052.643493] BTRFS warning (device sdj1): csum failed root 5 ino 163742 off 52214816768 csum 0x6723208c expected csum 0x0377e68a mirror 2 [47723.500430] BTRFS warning (device sdj1): csum failed root 5 ino 470775 off 12533760 csum 0x9f50f9a0 expected csum 0x23ddc68e mirror 2 [60060.843425] BTRFS warning (device sdj1): csum failed root 5 ino 786762 off 4178321408 csum 0xcd520ead expected csum 0x46fe6ebc mirror 2 [60900.058745] BTRFS warning (device sdj1): csum failed root 5 ino 786900 off 896303104 csum 0x4c7e26e7 expected csum 0x86554095 mirror 2 [68149.417236] BTRFS warning (device sdj1): csum failed root 5 ino 1058 off 3101224960 csum 0x2b8c363c expected csum 0x8df2991a mirror 1 [69072.272010] BTRFS warning (device sdj1): csum failed root 5 ino 1141 off 2939588608 csum 0xa2969f63 expected csum 0xddf33efd mirror 1 [71342.354453] BTRFS warning (device sdj1): csum failed root 5 ino 1328 off 57047568384 csum 0xd57f5bb7 expected csum 0x421f96e5 mirror 1 Because the device was consistent, it seemed that one of the disks held bad data. I wasn't sure if btrfs was correcting the issue by using the other seemingly good copy on the second disk or if I was copying bad data to the destination filesystem, so I aborted the copy and ran a scrub of the filesystem that includes sdj1 by issuing the following command: btrfs scrub start /external I let the scrub finish and monitored the result using the following command: btrfs scrub status /external Which showed the following output: scrub status for ece518d2-4af0-4ef7-a31d-8c89b13a5ad9 scrub started at Sun Jul 29 11:34:44 2018 and finished after 14:34:58 total bytes scrubbed: 12.80TiB with 0 errors Alright, perhaps btrfs had already fixed the issues upon encountering them. I ran my copy again only to see very similar messages show up in dmesg: [154842.551604] BTRFS warning (device sdj1): csum failed root 5 ino 1284 off 858886144 csum 0x8caf203c expected csum 0x9a3acab6 mirror 2 [159949.727412] BTRFS warning (device sdj1): csum failed root 5 ino 1636 off 4463370240 csum 0x8dfaf00c expected csum 0xa7ab457e mirror 2 [160911.893913] BTRFS warning (device sdj1): csum failed root 5 ino 1729 off 8181428224 csum 0xd57845b5 expected csum 0x6904c54e mirror 2 [165210.245890] BTRFS warning (device sdj1): csum failed root 5 ino 2927 off 1013219328 csum 0xf2d2820d expected csum 0x81bb mirror 2 [169279.620570] BTRFS warning (device sdj1): csum failed root 5 ino 3363 off 900493312 csum 0x6c6a35a2 expected csum 0x2a983a9c mirror 2 [169990.401373] BTRFS warning (device sdj1): csum failed root 5 ino 4277 off 186707968 csum 0xbdd075d5 expected csum 0xf302e9df mirror 2 [171411.085425] BTRFS warning (device sdj1): csum failed root 5 ino 4719 off 593842176 csum 0xcdabc7e6 expected csum 0xc137d47a mirror 2 [173370.025471] BTRFS warning (device sdj1): csum failed root 5 ino 5267 off 2605592576 csum 0xcd2cb8a8 expected csum 0x9de364e9 mirror 2 [180329.942125] BTRFS warning (device sdj1): csum failed root 5 ino 162774 off 22459506688 csum 0xc38e7a53 expected csum 0xad11854c mirror 2 I would have expected the scrub to find these issues or to show some number of corrected errors. Perhaps I misunderstand what scrub does? I also tried tracking down individual files via the referenced inode numbers with the following command: btrfs inspect-internal inode-resolve $INODE /external And ran checksums of the source and destination versions of these files to find them to be identical. So at least the copy on the source and destination appear to match. Maybe I'm experiencing some sort of intermittent USB device / bus issue? Can anyone help explain what might be happening here? Thanks!
Re: csum failed on raid1 even after clean scrub?
Thanks for the speedy reply! Here's my kernel version: 4.17.9-200.fc28.x86_64 dmesg doesn't show any USB related info at all, no signs of errors / warnings. Both drives are identical, Seagate 8TB external drives connected to the following PCIe controller: 03:00.0 USB controller: NEC Corporation uPD720200 USB 3.0 Host Controller (rev 03) lsusb output: Bus 002 Device 002: ID 8087:0024 Intel Corp. Integrated Rate Matching Hub Bus 002 Device 001: ID 1d6b:0002 Linux Foundation 2.0 root hub Bus 004 Device 005: ID 0bc2:ab38 Seagate RSS LLC Backup Plus Hub Bus 004 Device 003: ID 0bc2:ab45 Seagate RSS LLC Bus 004 Device 004: ID 0bc2:ab38 Seagate RSS LLC Backup Plus Hub Bus 004 Device 002: ID 0bc2:ab45 Seagate RSS LLC Bus 004 Device 001: ID 1d6b:0003 Linux Foundation 3.0 root hub Bus 003 Device 003: ID 0bc2:ab44 Seagate RSS LLC Bus 003 Device 002: ID 0bc2:ab44 Seagate RSS LLC Bus 003 Device 001: ID 1d6b:0002 Linux Foundation 2.0 root hub Bus 001 Device 003: ID 0624:0248 Avocent Corp. Virtual Hub Bus 001 Device 002: ID 8087:0024 Intel Corp. Integrated Rate Matching Hub Bus 001 Device 001: ID 1d6b:0002 Linux Foundation 2.0 root hub There's no output in dmesg related to the scrub, only the few csum failed messages I included in my last email. Per your request, I'm starting a scrub on the single device with the following command: btrfs scrub start /dev/sdj1 scrub started on /dev/sdj1, fsid ece518d2-4af0-4ef7-a31d-8c89b13a5ad9 (pid=14684) I'll report back once this is complete. Anything else I can collect that might be helpful in understanding what's happening here? On Mon, Jul 30, 2018 at 8:56 PM Qu Wenruo wrote: > > > On 2018年07月31日 08:43, Sterling Windmill wrote: > > I am using a two disk raid1 btrfs filesystem spanning two external hard > > drives connected via USB 3.0. > > Is there any speed difference between the two device? > And are these 2 devices under the same USB3.0 root hub or different root > hubs? > > lsusb output could help to determine the hierarchy. > > > > > While copying ~6TB of data from this filesystem to local disk via rsync > > I am seeing messages like the following in dmesg output: > > > > [ 2213.406267] BTRFS warning (device sdj1): csum failed root 5 ino 830 > > off 2124197888 csum 0xb5da0cd2 expected csum 0x6e478250 mirror 2 > > Since only one copy shows the problem, the other copy should be good > thus the read should work without problem. > > > [ 4890.178727] BTRFS warning (device sdj1): csum failed root 5 ino 1058 > > off 26052067328 csum 0x8ccd1067 expected csum 0x4adb8254 mirror 2 > > [27463.940218] BTRFS warning (device sdj1): csum failed root 5 ino 5372 > > off 7954096128 csum 0x9f9b697e expected csum 0xbd61a0e2 mirror 2 > > [29405.832643] BTRFS warning (device sdj1): csum failed root 5 ino 31374 > > off 7893983232 csum 0x12fd0ddc expected csum 0xddcd2f8e mirror 2 > > [31224.279082] BTRFS warning (device sdj1): csum failed root 5 ino > > 150903 off 183635968 csum 0xea025eb4 expected csum 0x46d64878 mirror 2 > > [32282.635615] BTRFS warning (device sdj1): csum failed root 5 ino > > 162774 off 31092424704 csum 0x1ee9b38d expected csum 0x4022e3de mirror 2 > > [41052.643493] BTRFS warning (device sdj1): csum failed root 5 ino > > 163742 off 52214816768 csum 0x6723208c expected csum 0x0377e68a mirror 2 > > [47723.500430] BTRFS warning (device sdj1): csum failed root 5 ino > > 470775 off 12533760 csum 0x9f50f9a0 expected csum 0x23ddc68e mirror 2 > > [60060.843425] BTRFS warning (device sdj1): csum failed root 5 ino > > 786762 off 4178321408 csum 0xcd520ead expected csum 0x46fe6ebc mirror 2 > > [60900.058745] BTRFS warning (device sdj1): csum failed root 5 ino > > 786900 off 896303104 csum 0x4c7e26e7 expected csum 0x86554095 mirror 2 > > [68149.417236] BTRFS warning (device sdj1): csum failed root 5 ino 1058 > > off 3101224960 csum 0x2b8c363c expected csum 0x8df2991a mirror 1 > > [69072.272010] BTRFS warning (device sdj1): csum failed root 5 ino 1141 > > off 2939588608 csum 0xa2969f63 expected csum 0xddf33efd mirror 1 > > [71342.354453] BTRFS warning (device sdj1): csum failed root 5 ino 1328 > > off 57047568384 csum 0xd57f5bb7 expected csum 0x421f96e5 mirror 1 > > > > Because the device was consistent, it seemed that one of the disks held > > bad data. I wasn't sure if btrfs was correcting the issue by using the > > other seemingly good copy on the second disk or if I was copying bad > > data to the destination filesystem, so I aborted the copy and ran a > > scrub of the filesystem that includes sdj1 by issuing the following > command: > > > > btrfs scrub start /external > > > > I let the scrub finish and monitored the result using the following &g
