On Wed, 24 Jun 2009, Ethan Erchinger wrote:
http://opensolaris.org/jive/thread.jspa?threadID=105702&tstart=0
Yes, this does sound very similar. It looks to me like data from read
files is clogging the ARC so that there is no more room for more
writes when ZFS periodically goes to commit unwritten data.
I'm wondering if changing txg_time to a lower value might help.
There is no doubt that having ZFS sync the written data more often
would help. However, it should not be necessary to tune the OS for
such a common task as batch processing a bunch of files.
A more appropriate solution is for ZFS to notice that more than XXX
megabytes are uncommitted, so maybe it should wake up and go write
some data. It is useful for ZFS to defer data writes in case the same
file is updated many times. In the case where the same file is
updated many times, the total uncommitted data is still limited by the
amount of data which is re-written and so the 30 second cycle is fine.
In my case the amount of uncommitted data is limited by available RAM
and how fast my application is able to produce new data to write.
The problem is very much related to how fast the data is output. If
the new data is created at a slower rate (output files are smaller)
then the problem just goes away.
Bob
--
Bob Friesenhahn
bfrie...@simple.dallas.tx.us, http://www.simplesystems.org/users/bfriesen/
GraphicsMagick Maintainer, http://www.GraphicsMagick.org/
_______________________________________________
zfs-discuss mailing list
zfs-discuss@opensolaris.org
http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
