Brian H. Nelson wrote:
Nobody can comment on this?
-Brian
Brian H. Nelson wrote:
I noticed this issue yesterday when I first started playing around with
zfs send/recv. This is on Solaris 10U6.
It seems that a zfs send of a zvol issues 'volblocksize' reads to the
physical devices. This doesn't make any sense to me, as zfs generally
consolidates read/write requests to improve performance. Even the dd
case with the same snapshot does not exhibit this behavior. It seems to
be specific to zfs send.
I checked with 8k, 64k, and 128k volblocksize, and the reads generated
by zfs send always seem to follow that size, while the reads with dd do not.
This is what I would expect, though there seems to be an optimization
opportunity. When doing a send, the things sent are records which are
of the size volblocksize. When doing a dd, the minimum size read will
be a record, but there may also be prefetching involved.
The optimization opportunity would be to enable prefetching when needed.
However, this may not be readily apparent on devices which allow
multiple outstanding I/Os.
The small reads seems to hurt performance of zfs send. I tested with a
mirror, but on another machine with a 7 disk raidz, the performance is
MUCH worse because the 8k reads get broken up into even smaller reads
and spread across the raidz.
Is this a bug, or can someone explain why this is happening?
This is how raidz works, today. A record is split across the columns
of the raidz set.
-- richard
Thanks
-Brian
Using 8k volblocksize:
-bash-3.00# zfs send pool1/vo...@now /dev/null
capacity operationsbandwidth
pool used avail read write read write
--- - - - - - -
pool14.01G 274G 1.88K 0 15.0M 0
mirror 4.01G 274G 1.88K 0 15.0M 0
c0t9d0 - -961 0 7.46M 0
c0t11d0 - -968 0 7.53M 0
--- - - - - - -
== ~8k reads to pool and drives
-bash-3.00# dd if=/dev/zvol/dsk/pool1/vo...@now of=/dev/null bs=8k
capacity operationsbandwidth
pool used avail read write read write
--- - - - - - -
pool14.01G 274G 2.25K 0 17.9M 0
mirror 4.01G 274G 2.25K 0 17.9M 0
c0t9d0 - -108 0 9.00M 0
c0t11d0 - -109 0 8.92M 0
--- - - - - - -
== ~8k reads to pool, ~85k reads to drives
Using volblocksize of 64k:
-bash-3.00# zfs send pool1/vol...@now /dev/null
capacity operationsbandwidth
pool used avail read write read write
--- - - - - - -
pool16.01G 272G378 0 23.5M 0
mirror 6.01G 272G378 0 23.5M 0
c0t9d0 - -189 0 11.8M 0
c0t11d0 - -189 0 11.7M 0
--- - - - - - -
== ~64k reads to pool and drives
-bash-3.00# dd if=/dev/zvol/dsk/pool1/vol...@now of=/dev/null bs=64k
capacity operationsbandwidth
pool used avail read write read write
--- - - - - - -
pool16.01G 272G414 0 25.7M 0
mirror 6.01G 272G414 0 25.7M 0
c0t9d0 - -107 0 12.9M 0
c0t11d0 - -106 0 12.8M 0
--- - - - - - -
== ~64k reads to pool, ~124k reads to drives
Using volblocksize of 128k:
-bash-3.00# zfs send pool1/vol1...@now /dev/null
capacity operationsbandwidth
pool used avail read write read write
--- - - - - - -
pool14.01G 274G188 0 23.3M 0
mirror 4.01G 274G188 0 23.3M 0
c0t9d0 - - 94 0 11.7M 0
c0t11d0 - - 93 0 11.7M 0
--- - - - - - -
== ~128k reads to pool and drives
-bash-3.00# dd if=/dev/zvol/dsk/pool1/vol1...@now of=/dev/null bs=128k
capacity operationsbandwidth
pool used avail read write read write
--- - - - - - -
pool14.01G 274G247 0 30.8M 0
mirror 4.01G 274G247 0 30.8M 0
c0t9d0 - -122 0 15.3M 0
c0t11d0 - -123 0 15.5M 0
--- - - - - - -
== ~128k reads to pool and drives
___
zfs-discuss mailing list
zfs-discuss@opensolaris.org
