Re: Oddly slow read performance with near-full largish FS

[Austin S Hemmelgarn]

On 2014-12-21 17:53, Charles Cazabon wrote:

Hi, Robert,

My performance issues with btrfs are more-or-less resolved now -- the
performance under btrfs still seems quite variable compared to other
filesystems -- my rsync speed is now varying between 40MB and ~90MB/s, with
occasional intervals where it drops further, down into the 10-20MB/s range.
Still no disk errors or SMART warnings that would indicate that problem is at
the hardware level.

Do make sure you are regularly running the long "offline" test.

Ok, I'll do that.

Otherwise SMART is just going to tell you the disk just died when it dies.

Ya, I'm aware of how limited/useful the SMART diagnostics are.  I'm also
paranoid enough to be using RAID 6...

The thing with "movablecore=" will not lead to an "out of memory"
condition or not, its a question of cache and buffer evictions.

I'm fairly certain memory isn't the issue here.  For what it's worth:

%Cpu(s):  2.1 us, 19.4 sy,  0.0 ni, 78.0 id,  0.2 wa,  0.3 hi,  0.0 si,  0.0 st
KiB Mem:  16469880 total, 16301252 used,   168628 free,      720 buffers
KiB Swap:  7811068 total,        0 used,  7811068 free, 15146580 cached

Swappiness I've left at the default of 60, but I'm not seeing swapping going
on regardless.

Is this remaining difference (25 vs 100+ MB/s) simply due to btrfs not being
tuned for performance yet

I found the cause of this.  Stupidly enough, there was a bwlimit set up in a
shell alias for rsync.

So btrfs is not nearly as slow as I was seeing.  It's still slower than
reading from an ext4 or XFS filesystem on these disks, but the absolute level
of read speed seems reasonable enough given that btrfs has not been under
heavy performance tuning to date.  My only remaining concern would be the
variability I still see in the read speed.

This actually sounds kind of like the issues I have sometimes on my laptop using btrfs on an SSD, I've mostly resolved them by tuning IO scheduler parameters, as the default IO scheduler (the supposedly Completely Fair Queue, which was obviously named by a mathematician who had never actually run the algorithm) has some pretty brain-dead default settings. The other thing I would suggest looking into regarding the variability is tuning the kernel's write-caching settings, with the defaults you're caching ~1.6G worth of writes before it forces write-back, which is a ridiculous amount; I've that the highest value that is actually usable is about 256M, and that's only if you are doing mostly bursty IO and not the throughput focused stuff that rsync does, I'd say try setting /proc/sys/vm/dirty_background_bytes to 67108864 (64M) and see if that helps things some.

smime.p7s
Description: S/MIME Cryptographic Signature