I'm new to ZFS and looking for some assistance with a performance problem:
At the interval of zfs_txg_timeout (I'm using the default of 30), I observe
100-200ms pauses in my application. Based on my application log files, it
appears that the write() system call is blocked. Digging deeper into the
problem with DTrace, I discovered that most of the time/pause is spent in
the function zil_itx_assign. The output below illustrates how several write
system calls are paused in the zil_itx_assign function for approximately
180ms ... then all of them resume simultaneously.
Of interest, this behavior only occurs at the very end of spa_sync.
Timestamp CPU Function
82682408087424 10 write:entry
82682408099903 10 zil_itx_assign:entry
82682408102896 10 zil_itx_assign:return
82682408106429 10 write:return Elapsed Time in micros: 19
82682408174003 10 write:entry
82682408184519 10 zil_itx_assign:entry
## Pause begins here when entering zil_itx_assign
82682408205410 6 write:entry
82682408213613 11 spa_sync:return
82682408215819 6 zil_itx_assign:entry
82682408237359 2 write:entry
82682408252117 2 zil_itx_assign:entry
## Pause ends here
82682588006927 6 zil_itx_assign:return
82682588006967 10 zil_itx_assign:return
82682588007659 2 zil_itx_assign:return
82682588020317 6 write:return Elapsed Time in micros: 179814
82682588020321 10 write:return Elapsed Time in micros: 179846
82682588020326 2 write:return Elapsed Time in micros: 179782
Here is the output of lockstat -cwPW that further implicates
zil_itx_assign:
Adaptive mutex block: 222 events in 5.014 seconds (44 events/sec)
Count indv cuml rcnt nsec Hottest Lock Hottest Caller
-------------------------------------------------------------------------------
1 99% 99% 0.00 *185876716* 0xffffffff82a7e4c0
zil_itx_assign+0x22
86 0% 100% 0.00 5424 0xffffffffa17891f8
cv_timedwait_sig_hires+0x2d4
53 0% 100% 0.00 2328 0xffffffff9f7ef3a0 udp_rcv_enqueue+0xa0
8 0% 100% 0.00 11664 0xffffffffa0016f18 vdev_queue_io_done+0x3a
15 0% 100% 0.00 5350 0xffffffff85962b40 clock_tick_process+0xc4
Looking at the source for zil_itx_assign, it looks like I'm contending for a
mutex on the zilog, but I'm not familiar enough with ZFS to know why.
Any help/suggestions would be appreciated.
Thanks
_______________________________________________
zfs-discuss mailing list
zfs-discuss@opensolaris.org
http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
