On Mon, Jun 29, 2009 at 2:48 PM, Bob Friesenhahn<bfrie...@simple.dallas.tx.us> wrote: > On Wed, 24 Jun 2009, Lejun Zhu wrote: > >> There is a bug in the database about reads blocked by writes which may be >> related: >> >> http://bugs.opensolaris.org/view_bug.do?bug_id=6471212 >> >> The symptom is sometimes reducing queue depth makes read perform better. > > I have been banging away at this issue without resolution. Based on Roch > Bourbonnais's blog description of the ZFS write throttle code, it seems that > I am facing a perfect storm. Both the storage write bandwidth (800+ > MB/second) and the memory size of my system (20 GB) result in the algorithm > batching up 2.5 GB of user data to write. Since I am using mirrors, this > results in 5 GB of data being written at full speed to the array on a very > precise schedule since my application is processing fixed-sized files with a > fixed algorithm. The huge writes lead to at least 3 seconds of read > starvation, resulting in a stalled application and a square-wave of system > CPU utilization. I could attempt to modify my application to read ahead by > 3 seconds but that would require gigabytes of memory, lots of complexity, > and would not be efficient. > > Richard Elling thinks that my array is pokey, but based on write speed and > memory size, ZFS is always going to be batching up data to fill the write > channel for 5 seconds so it does not really matter how fast that write > channel is. If I had 32GB of RAM and 2X the write speed, the situation > would be identical. > > Hopefully someone at Sun is indeed working this read starvation issue and it > will be resolved soon. > > Bob > -- > Bob Friesenhahn > bfrie...@simple.dallas.tx.us, http://www.simplesystems.org/users/bfriesen/ > GraphicsMagick Maintainer, http://www.GraphicsMagick.org/ > _______________________________________________ > zfs-discuss mailing list > zfs-discuss@opensolaris.org > http://mail.opensolaris.org/mailman/listinfo/zfs-discuss >
I see similar square-wave performance. However, my load is primarily write-based, when those commits happen, I see all network activity pause while the buffer is commited to disk. I write about 750Mbit/sec over the network to the X4540's during backup windows using primarily iSCSI. When those writes occur to my RaidZ volume, all activity pauses until the writes are fully flushed. One thing to note, on 117, the effects are seemingly reduced and a bit more even performance, but it is still there. -- Brent Jones br...@servuhome.net _______________________________________________ zfs-discuss mailing list zfs-discuss@opensolaris.org http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
