Hi everyone,
This is my first post to zfs-discuss, so be gentle with me :-)
I've been doing some testing with ZFS - in particular, in checkpointing
the large, proprietary in-memory database which is a key part of the
application I work on. In doing this I've found what seems to be some
fairly unhelpful write throttling behaviour from ZFS.
In summary, the environment is:
* An x4600 with 8 CPUs and 128GBytes of memory
* A 50GByte in-memory database
* A big, fast disk array (a 6140 with a LUN comprised of 4 SATA drives)
* Running Solaris 10 update 4 (problems initially seen on U3 so I got it
patched)
The problems happen when I checkpoint the database, which involves
putting that database on disk as quickly as possible, using the write(2)
system call.
The first time the checkpoint is run, it's quick - about 160MBytes/sec,
even though the disk array is only sustaining 80MBytes/sec. So we're
dirtying stuff in the ARC (and growing the ARC) at a pretty impressive
rate.
After letting the IO subside, running the checkpoint again results in
very different behaviour. It starts running very quickly, again at
160MByte/sec (with the underlying device doing 80MBytes/sec), and after
a while (presumably once the ARC is full) things go badly wrong. In
particular, a write(2) system call hangs for 6-7 minutes, apparently
until all the outstanding IO is done. Any reads from that device also
take a huge amount of time, making the box very unresponsive.
Obviously this isn't good behaviour, but it's particularly unfortunate
given that this checkpoint is stuff that I don't want to retain in any
kind of cache anyway - in fact, preferably I wouldn't pollute the ARC
with it in the first place. But it seems directio(3C) doesn't work with
ZFS (unsurprisingly as I guess this is implemented in segmap), and
madvise(..., MADV_DONTNEED) doesn't drop data from the ARC (again, I
guess, as it's working on segmap/segvn).
Of course, limiting the ARC size to something fairly small makes it
behave much better. But this isn't really the answer.
I also tried using O_DSYNC, which stops the pathological behaviour but
makes things pretty slow - I only get a maximum of about 20MBytes/sec,
which is obviously much less than the hardware can sustain.
It sounds like we could do with different write throttling behaviour to
head this sort of thing off. Of course, the ideal would be to have some
way of telling ZFS not to bother keeping pages in the ARC.
The latter appears to be bug 6429855. But the underlying behaviour
doesn't really seem desirable; are there plans afoot to do any work on
ZFS write throttling to address this kind of thing?
Regards,
--
Philip Beevers
Fidessa Infrastructure Development
mailto:[EMAIL PROTECTED]
phone: +44 1483 206571
This message is intended only for the stated addressee(s) and may be
confidential. Access to this email by anyone else is unauthorised. Any
opinions expressed in this email do not necessarily reflect the opinions of
Fidessa. Any unauthorised disclosure, use or dissemination, either whole or in
part is prohibited. If you are not the intended recipient of this message,
please notify the sender immediately.
Fidessa plc - Registered office:
Dukes Court, Duke Street, Woking, Surrey, GU21 5BH, United Kingdom
Registered in England no. 3781700 VAT registration no. 688 9008 78
Fidessa group plc - Registered Office:
Dukes Court, Duke Street, Woking, Surrey, GU21 5BH, United Kingdom
Registered in England no. 3234176 VAT registration no. 688 9008 78
___
zfs-discuss mailing list
zfs-discuss@opensolaris.org
http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
