On Mon, 24 Mar 2014, Reginald Beardsley wrote:
My apologies, I thought the references to /dev/zero, /dev/null and bs=65536 made it clear I was using dd.
At least for tests on the server side.
I increased the file size to 65 GB and reran the tests. It turns out the read operation wall time can vary by several minutes. For the 6.5 GB test that makes the thoughput rather low. On the 65 GB tests I got 42-59 MB/s read for 10 trials. This is pretty much what I expect from gigabit using a cheap SOHO switch w/ no other traffic.
The thing about this performance test approach is that if your server only has 4 GB of RAM and the file size is 65 GB then you are preventing the zfs ARC from adding much value on the server side other than to buffer up individual full zfs blocks. You are essentially testing what happens when your server is running out of memory and not testing the normal case.
If the zfs blocksize is 128k, then the 64k dd blocksize is resulting in two reads per zfs block. This can potentially incur more latency, except for the caching (and possibly read-ahead) in the ARC. The NFS client negotiates the maximum block size to use (could be 1 MB in some modern implementations). Hopefully it is using the same block size as 'dd' is requesting rather than a smaller size since then additional latencies would be incurred. Some other NFS clients use much smaller block sizes by default and require tunables to use larger ones.
I suggest that you tell dd to use the same block size as the underlying zfs filesystem. This will usually produce the best performance in such benchmarking.
The largest influence of block size will be when writing on top of an existing huge file. There will be considerably less performance when writing using 64k blocks on top of 128k zfs blocks because of the shortage of memory (whole file is not already cached) and ZFS's COW algorithm (needs to read the underlying zfs block before it can write a new one).
Bob -- Bob Friesenhahn [email protected], http://www.simplesystems.org/users/bfriesen/ GraphicsMagick Maintainer, http://www.GraphicsMagick.org/ _______________________________________________ OpenIndiana-discuss mailing list [email protected] http://openindiana.org/mailman/listinfo/openindiana-discuss
