On Mon, Apr 08, 2024 at 10:04:01PM +0200, Marc SCHAEFER wrote: > For off-site long-term offline archiving, no, I am not using RAID.
Now, as I had to think a bit about ONLINE integrity, I found this comparison: https://github.com/t13a/dm-integrity-benchmarks Contenders are btrfs, zfs, and notably ext4+dm-integrity+dm-raid I tend to have a biais favoring UNIX layered solutions against "all-into-one" solutions, and it seems that performance-wise, it's also quite good. I wrote this script to convince myself of auto-correction of the ext4+dm-integrity+dm-raid layered approach. It gives: [ ... ] [ 390.249699] md/raid1:mdX: read error corrected (8 sectors at 21064 on dm-11) [ 390.249701] md/raid1:mdX: redirecting sector 20488 to other mirror: dm-7 [ 390.293807] md/raid1:mdX: dm-11: rescheduling sector 262168 [ 390.293988] md/raid1:mdX: read error corrected (8 sectors at 262320 on dm-11) [ 390.294040] md/raid1:mdX: read error corrected (8 sectors at 262368 on dm-11) [ 390.294125] md/raid1:mdX: read error corrected (8 sectors at 262456 on dm-11) [ 390.294209] md/raid1:mdX: read error corrected (8 sectors at 262544 on dm-11) [ 390.294287] md/raid1:mdX: read error corrected (8 sectors at 262624 on dm-11) [ 390.294586] md/raid1:mdX: read error corrected (8 sectors at 263000 on dm-11) [ 390.294712] md/raid1:mdX: redirecting sector 262168 to other mirror: dm-7 pretty much convicing. So after testing btrfs and being not convinced, after doing some test on a production zfs -- not convinced either -- I am going to ry ext4+dm-integrity+dm-raid. #! /bin/bash set -e function create_lo { local f f=$(losetup -f) losetup $f $1 echo $f } # beware of the rm -r below! tmp_dir=/tmp/$(basename $0) mnt=/mnt mkdir $tmp_dir declare -a pvs for p in pv1 pv2 do truncate -s 250M $tmp_dir/$p l=$(create_lo $tmp_dir/$p) pvcreate $l pvs+=($l) done vg=$(basename $0)-test lv=test vgcreate $vg ${pvs[*]} vgdisplay $vg lvcreate --type raid1 --raidintegrity y -m 1 -L 200M -n $lv $vg lvdisplay $vg # sync/integrity complete? sleep 10 cat /proc/mdstat echo lvs -a -o name,copy_percent,devices $vg echo echo -n Type ENTER read ignore mkfs.ext4 -I 256 /dev/$vg/$lv mount /dev/$vg/$lv $mnt for f in $(seq 1 10) do # ignore errors head -c 20M < /dev/random > $mnt/f_$f || true done (cd $mnt && find . -type f -print0 | xargs -0 md5sum > $tmp_dir/MD5SUMS) # corrupting some data in one PV count=5000 blocks=$(blockdev --getsz ${pvs[1]}) if [ $blocks -lt 32767 ]; then factor=1 else factor=$(( ($blocks - 1) / 32767)) fi p=1 for i in $(seq 1 $count) do offset=$(($RANDOM * $factor)) echo ${pvs[$p]} $offset dd if=/dev/random of=${pvs[$p]} bs=$(blockdev --getpbsz ${pvs[$p]}) seek=$offset count=1 # only doing on 1, not 0, since we have no way to avoid destroying the same sector! #p=$((1 - p)) done dd if=/dev/$vg/$lv of=/dev/null bs=32M dmesg | tail umount $mnt lvremove -y $vg/$lv vgremove -y $vg for p in ${pvs[*]} do pvremove $p losetup -d $p done rm -r $tmp_dir
