On Tue, Mar 27, 2012 at 1:15 AM, Jim Klimov <j...@cos.ru> wrote: > Well, as a further attempt down this road, is it possible for you to rule > out > ZFS from swapping - i.e. if RAM amounts permit, disable the swap at all > (swap -d /dev/zvol/dsk/rpool/swap) or relocate it to dedicated slices of > same or better yet separate disks? >
Thanks Jim for your suggestion! > If you do have lots of swapping activity (that can be seen in "vmstat 1" as > si/so columns) going on in a zvol, you're likely to get much fragmentation > in the pool, and searching for contiguous stretches of space can become > tricky (and time-consuming), or larger writes can get broken down into > many smaller random writes and/or "gang blocks", which is also slower. > At least such waiting on disks can explain the overall large kernel times. I took swapping activity into account, even when the CPU% is 100%, "si" (swap-ins) and "so" (swap-outs) are always ZEROs. > > You can also see the disk wait times ratio in "iostat -xzn 1" column "%w" > and disk busy times ratio in "%b" (second and third from the right). > I dont't remember you posting that. > > If these are accounting in tens, or even close or equal to 100%, then > your disks are the actual bottleneck. Speeding up that subsystem, > including addition of cache (ARC RAM, L2ARC SSD, maybe ZIL > SSD/DDRDrive) and combatting fragmentation by moving swap and > other scratch spaces to dedicated pools or raw slices might help. My storage system is not quite busy, and there are only read operations. ===================================== # iostat -xnz 3 extended device statistics r/s w/s kr/s kw/s wait actv wsvc_t asvc_t %w %b device 112.4 0.0 1691.5 0.0 0.0 0.5 0.0 4.8 0 41 c11t0d0 extended device statistics r/s w/s kr/s kw/s wait actv wsvc_t asvc_t %w %b device 118.7 0.0 1867.0 0.0 0.0 0.5 0.0 4.5 0 42 c11t0d0 extended device statistics r/s w/s kr/s kw/s wait actv wsvc_t asvc_t %w %b device 127.7 0.0 2121.6 0.0 0.0 0.6 0.0 4.7 0 44 c11t0d0 extended device statistics r/s w/s kr/s kw/s wait actv wsvc_t asvc_t %w %b device 141.3 0.0 2158.5 0.0 0.0 0.7 0.0 4.6 0 48 c11t0d0 ============================================== Thanks, -Aubrey _______________________________________________ zfs-discuss mailing list zfs-discuss@opensolaris.org http://mail.opensolaris.org/mailman/listinfo/zfs-discuss