Brian H. Nelson wrote: > Nobody can comment on this? > > -Brian > > > Brian H. Nelson wrote: >> I noticed this issue yesterday when I first started playing around with >> zfs send/recv. This is on Solaris 10U6. >> >> It seems that a zfs send of a zvol issues 'volblocksize' reads to the >> physical devices. This doesn't make any sense to me, as zfs generally >> consolidates read/write requests to improve performance. Even the dd >> case with the same snapshot does not exhibit this behavior. It seems to >> be specific to zfs send. >> >> I checked with 8k, 64k, and 128k volblocksize, and the reads generated >> by zfs send always seem to follow that size, while the reads with dd do not.
This is what I would expect, though there seems to be an optimization opportunity. When doing a send, the things sent are records which are of the size volblocksize. When doing a dd, the minimum size read will be a record, but there may also be prefetching involved. The optimization opportunity would be to enable prefetching when needed. However, this may not be readily apparent on devices which allow multiple outstanding I/Os. >> The small reads seems to hurt performance of zfs send. I tested with a >> mirror, but on another machine with a 7 disk raidz, the performance is >> MUCH worse because the 8k reads get broken up into even smaller reads >> and spread across the raidz. >> >> Is this a bug, or can someone explain why this is happening? This is how raidz works, today. A record is split across the columns of the raidz set. -- richard >> Thanks >> -Brian >> >> Using 8k volblocksize: >> >> -bash-3.00# zfs send pool1/vo...@now > /dev/null >> >> capacity operations bandwidth >> pool used avail read write read write >> ----------- ----- ----- ----- ----- ----- ----- >> pool1 4.01G 274G 1.88K 0 15.0M 0 >> mirror 4.01G 274G 1.88K 0 15.0M 0 >> c0t9d0 - - 961 0 7.46M 0 >> c0t11d0 - - 968 0 7.53M 0 >> ----------- ----- ----- ----- ----- ----- ----- >> == ~8k reads to pool and drives >> >> -bash-3.00# dd if=/dev/zvol/dsk/pool1/vo...@now of=/dev/null bs=8k >> >> capacity operations bandwidth >> pool used avail read write read write >> ----------- ----- ----- ----- ----- ----- ----- >> pool1 4.01G 274G 2.25K 0 17.9M 0 >> mirror 4.01G 274G 2.25K 0 17.9M 0 >> c0t9d0 - - 108 0 9.00M 0 >> c0t11d0 - - 109 0 8.92M 0 >> ----------- ----- ----- ----- ----- ----- ----- >> == ~8k reads to pool, ~85k reads to drives >> >> >> Using volblocksize of 64k: >> >> -bash-3.00# zfs send pool1/vol...@now > /dev/null >> >> capacity operations bandwidth >> pool used avail read write read write >> ----------- ----- ----- ----- ----- ----- ----- >> pool1 6.01G 272G 378 0 23.5M 0 >> mirror 6.01G 272G 378 0 23.5M 0 >> c0t9d0 - - 189 0 11.8M 0 >> c0t11d0 - - 189 0 11.7M 0 >> ----------- ----- ----- ----- ----- ----- ----- >> == ~64k reads to pool and drives >> >> -bash-3.00# dd if=/dev/zvol/dsk/pool1/vol...@now of=/dev/null bs=64k >> >> capacity operations bandwidth >> pool used avail read write read write >> ----------- ----- ----- ----- ----- ----- ----- >> pool1 6.01G 272G 414 0 25.7M 0 >> mirror 6.01G 272G 414 0 25.7M 0 >> c0t9d0 - - 107 0 12.9M 0 >> c0t11d0 - - 106 0 12.8M 0 >> ----------- ----- ----- ----- ----- ----- ----- >> == ~64k reads to pool, ~124k reads to drives >> >> >> Using volblocksize of 128k: >> >> -bash-3.00# zfs send pool1/vol1...@now > /dev/null >> >> capacity operations bandwidth >> pool used avail read write read write >> ----------- ----- ----- ----- ----- ----- ----- >> pool1 4.01G 274G 188 0 23.3M 0 >> mirror 4.01G 274G 188 0 23.3M 0 >> c0t9d0 - - 94 0 11.7M 0 >> c0t11d0 - - 93 0 11.7M 0 >> ----------- ----- ----- ----- ----- ----- ----- >> == ~128k reads to pool and drives >> >> -bash-3.00# dd if=/dev/zvol/dsk/pool1/vol1...@now of=/dev/null bs=128k >> >> capacity operations bandwidth >> pool used avail read write read write >> ----------- ----- ----- ----- ----- ----- ----- >> pool1 4.01G 274G 247 0 30.8M 0 >> mirror 4.01G 274G 247 0 30.8M 0 >> c0t9d0 - - 122 0 15.3M 0 >> c0t11d0 - - 123 0 15.5M 0 >> ----------- ----- ----- ----- ----- ----- ----- >> == ~128k reads to pool and drives >> >> > _______________________________________________ zfs-discuss mailing list zfs-discuss@opensolaris.org http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
