Hi, I was using two partitions on same nvme device to compare filesystem performance, and I consistantly observed a strange behavior:
After 10 minutes fio test with bcachefs on one partition, performance degrade significantly for other filesystems on other partition (same device). ext4 150M/s --> 143M/s xfs 150M/s --> 134M/s btrfs 127M/s --> 108M/s Several round tests show the same pattern that bcachefs seems occupy some device resource even when there is no high-level I/O. I monitor /proc/diskstats, and it confirmed that bcachefs do keep writing the device. Following is the time serial samples for "writes_completed" on my bcachefs partition: writes_completed @timestamp 0 @1724748233.712 4 @1724748248.712 <--- mkfs 4 @1724748263.712 65 @1724748278.712 25350 @1724748293.712 63839 @1724748308.712 <--- fio started 352228 @1724748323.712 621350 @1724748338.712 903487 @1724748353.712 ... 12790311 @1724748863.712 13100041 @1724748878.712 13419642 @1724748893.712 13701685 @1724748908.712 <--- fio done (10minutes) 13701769 @1724748923.712 <--- from here, average 5~7writes/second for 2000 seconds 13701852 @1724748938.712 13701953 @1724748953.712 13702032 @1724748968.712 13702133 @1724748983.712 13702213 @1724748998.712 13702265 @1724749013.712 13702357 @1724749028.712 ... 13712984 @1724750858.712 13713076 @1724750873.712 13713196 @1724750888.712 13713299 @1724750903.712 13713386 @1724750918.712 13713463 @1724750933.712 13713501 @1724750948.712 <--- writes stopped here 13713501 @1724750963.712 13713501 @1724750978.712 ... Is this behavior expected? My test script: set -e for fsa in "btrfs" "ext4" "bcachefs" "xfs" do if [ $fsa == 'ext4' ]; then mkfs -t ext4 -F /dev/nvme0n1p1 else mkfs -t $fsa -f /dev/nvme0n1p1 fi mount -t $fsa /dev/nvme0n1p1 /disk02/dir1 for fsb in "ext4" "bcachefs" "xfs" "btrfs" do if [ $fsb == 'ext4' ]; then mkfs -t ext4 -F /dev/nvme0n1p2 else mkfs -t $fsb -f /dev/nvme0n1p2 fi mount -t $fsb /dev/nvme0n1p2 /disk02/dir2 cd /disk02/dir1 && fio --randrepeat=1 --ioengine=libaio --direct=1 --name=test --bs=4k --iodepth=64 --size=1G --readwrite=randrw --runtime=600 --numjobs=8 --time_based=1 --output=/disk02/fio.${fsa}.${fsb}.0 sleep 30 cd /disk02/dir2 && fio --randrepeat=1 --ioengine=libaio --direct=1 --name=test --bs=4k --iodepth=64 --size=1G --readwrite=randrw --runtime=600 --numjobs=8 --time_based=1 --output=/disk02/fio.${fsa}.${fsb}.1 sleep 30 cd /disk02 umount /disk02/dir2 done umount /disk02/dir1 done And here is a report for one round of test matrix: +----------+-----------------------------+-----------------------------+-----------------------------+-----------------------------+ | R|W | ext4 | bcachefs | xfs | btrfs | +----------+-----------------------------+-----------------------------+-----------------------------+-----------------------------+ | ext4 | [ext4]147MB/s|147MB/s | [ext4]146MB/s|146MB/s | [ext4]150MB/s|150MB/s | [ext4]149MB/s|149MB/s | | | [ext4]146MB/s|146MB/s | [bcachefs]72.2MB/s|72.2MB/s | [xfs]149MB/s|149MB/s | [btrfs]132MB/s|132MB/s | | bcachefs | [bcachefs]71.9MB/s|71.9MB/s | [bcachefs]65.1MB/s|65.1MB/s | [bcachefs]69.6MB/s|69.6MB/s | [bcachefs]65.8MB/s|65.8MB/s | | | [ext4]143MB/s|143MB/s | [bcachefs]71.5MB/s|71.5MB/s | [xfs]134MB/s|133MB/s | [btrfs]108MB/s|108MB/s | | xfs | [xfs]148MB/s|148MB/s | [xfs]147MB/s|147MB/s | [xfs]152MB/s|152MB/s | [xfs]151MB/s|151MB/s | | | [ext4]147MB/s|147MB/s | [bcachefs]71.3MB/s|71.3MB/s | [xfs]148MB/s|148MB/s | [btrfs]127MB/s|127MB/s | | btrfs | [btrfs]132MB/s|132MB/s | [btrfs]112MB/s|111MB/s | [btrfs]110MB/s|110MB/s | [btrfs]110MB/s|110MB/s | | | [ext4]147MB/s|146MB/s | [bcachefs]69.7MB/s|69.7MB/s | [xfs]146MB/s|146MB/s | [btrfs]125MB/s|125MB/s | +----------+-----------------------------+-----------------------------+-----------------------------+-----------------------------+ (The rows are for the FS on the first partition, and the cols are on the second partition) The version of bcachefs-tools on my system is 1.9.1. (The impact is worse, ext4 dropped to 80M/s, when I was using bcachefs-tools from debian repos which is too *old*, and known to cause bcachefs problems. And that is the reason that I do this kind of test.) Thanks David