Bob Friesenhahn wrote:
On Sat, 4 Jul 2009, Phil Harman wrote:
However, it seems that memory mapping is not responsible for the
problem I am seeing here. Memory mapping may make the problem seem
worse, but it is clearly not the cause.
mmap(2) is what brings ZFS files into the page cache. I think you've
shown us that once you've copied files with cp(1) - which does use
mmap(2) - that anything that uses read(2) on the same files is impacted.
The problem is observed with cpio, which does not use mmap. This is
immediately after a reboot or unmount/mount of the filesystem.
Sorry, I didn't get to your other post ...
Ok, here is the scoop on the dire Solaris 10 (Generic_141415-03)
performance bug on my Sun Ultra 40-M2 attached to a StorageTek 2540
with latest firmware. I rebooted the system used cpio to send the
input files to /dev/null, and then immediately used cpio a second time
to send the input files to /dev/null. Note that the amount of file
data (243 GB) is plenty sufficient to purge any file data from the ARC
(which has a cap of 10 GB).
% time cat dpx-files.txt | cpio -o > /dev/null
495713288 blocks
cat dpx-files.txt 0.00s user 0.00s system 0% cpu 1.573 total
cpio -o > /dev/null 78.92s user 360.55s system 43% cpu 16:59.48 total
% time cat dpx-files.txt | cpio -o > /dev/null
495713288 blocks
cat dpx-files.txt 0.00s user 0.00s system 0% cpu 0.198 total
cpio -o > /dev/null 79.92s user 358.75s system 11% cpu 1:01:05.88 total
zpool iostat averaged over 60 seconds reported that the first run
through the files read the data at 251 MB/s and the second run only
achieved 68 MB/s. It seems clear that there is something really bad
about Solaris 10 zfs's file caching code which is causing it to go
into the weeds.
I don't think that the results mean much, but I have attached output
from 'hotkernel' while a subsequent cpio copy is taking place. It
shows that the kernel is mostly sleeping.
This is not a new problem. It seems that I have been banging my head
against this from the time I started using zfs.
I'd like to see mpstat 1 for each case, on an otherwise idle system, but
then there's probably a whole lot of dtrace I'd like to do ... but I'm
just off on vacation for a week, and this will probably have to be my
last post on this thread until I'm back.
Cheers,
Phil
_______________________________________________
zfs-discuss mailing list
zfs-discuss@opensolaris.org
http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
