Basically for a few seconds at a time I can get very nice speeds through
rsync (saturating a 1 gig link) which is around 112-113 megabytes/sec
which is about as good as I can expect after overhead. The problem is
that every 5 seconds when data is actually written to disks (physically
looking at the disk LEDs I see the activity on the one sending data
stall as the ZFS machine is showing disk activity (writes).
Basically this problem is bringing down the average write speed from
around 112-113 megabytes/sec to around 100 megabytes/sec (sometimes
lower) and thus lowering speeds by 10% (or a bit more). I only really
care as 10% can make a difference when you are copying terrabytes of
data to the machine. Anyway here is what I am seeing on the linux
machine that is sending to the ZFS machine:
Device: rrqm/s wrqm/s r/s w/s rMB/s wMB/s
avgrq-sz avgqu-sz await svctm %util
sdc3 17.00 0.00 496.00 0.00 112.00 0.00
462.45 0.42 0.84 0.16 7.90
avg-cpu: %user %nice %system %iowait %steal %idle
6.34 0.00 3.80 0.00 0.00 89.86
Device: rrqm/s wrqm/s r/s w/s rMB/s wMB/s
avgrq-sz avgqu-sz await svctm %util
sdc3 1.00 0.00 584.00 0.00 111.75 0.00
391.88 0.43 0.74 0.12 7.10
avg-cpu: %user %nice %system %iowait %steal %idle
6.84 0.00 5.44 0.00 0.00 87.72
Device: rrqm/s wrqm/s r/s w/s rMB/s wMB/s
avgrq-sz avgqu-sz await svctm %util
sdc3 4.00 0.00 557.00 0.00 112.00 0.00
411.81 0.43 0.77 0.13 7.10
avg-cpu: %user %nice %system %iowait %steal %idle
4.64 0.00 2.51 0.00 0.00 92.86
Device: rrqm/s wrqm/s r/s w/s rMB/s wMB/s
avgrq-sz avgqu-sz await svctm %util
sdc3 1.98 0.00 104.95 0.00 23.76 0.00
463.70 0.09 0.84 0.17 1.78
avg-cpu: %user %nice %system %iowait %steal %idle
7.11 0.00 5.08 0.00 0.00 87.82
Device: rrqm/s wrqm/s r/s w/s rMB/s wMB/s
avgrq-sz avgqu-sz await svctm %util
sdc3 9.00 0.00 538.00 0.00 112.69 0.00
428.97 0.39 0.72 0.14 7.50
avg-cpu: %user %nice %system %iowait %steal %idle
7.46 0.00 5.56 0.00 0.00 86.98
Device: rrqm/s wrqm/s r/s w/s rMB/s wMB/s
avgrq-sz avgqu-sz await svctm %util
sdc3 8.00 0.00 524.00 0.00 112.00 0.00
437.74 0.38 0.72 0.13 6.90
avg-cpu: %user %nice %system %iowait %steal %idle
7.98 0.00 5.96 0.00 0.00 86.06
Device: rrqm/s wrqm/s r/s w/s rMB/s wMB/s
avgrq-sz avgqu-sz await svctm %util
sdc3 1.00 0.00 493.00 0.00 111.29 0.00
462.31 0.39 0.80 0.16 7.90
This is iostat -xm 1 sdc3
Basically you can see its reading at around full gig speed and then
reads drop down due to writes being stalled on the ZFS machine. These
are 1 second averages.
On the ZFS end with 10 second averages I see approximately 100 MB/sec:
data 25.1G 18.1T 0 834 0 100M
data 26.2G 18.1T 0 833 0 100M
data 27.3G 18.1T 0 833 0 100M
Change this to 1 second and I see:
data 32.7G 18.1T 0 0 0 0
data 32.7G 18.1T 0 2.86K 0 360M
data 33.3G 18.1T 0 264 0 21.8M
data 33.3G 18.1T 0 0 0 0
data 33.3G 18.1T 0 0 0 0
data 33.3G 18.1T 0 0 0 0
data 33.3G 18.1T 0 2.94K 0 369M
data 33.8G 18.1T 0 375 0 35.1M
data 33.8G 18.1T 0 0 0 0
data 33.8G 18.1T 0 0 0 0
data 33.8G 18.1T 0 0 0 0
data 33.8G 18.1T 0 2.90K 0 365M
data 34.4G 18.1T 0 599 0 62.6M
data 34.4G 18.1T 0 0 0 0
data 34.4G 18.1T 0 0 0 0
data 34.4G 18.1T 0 0 0 0
data 34.4G 18.1T 0 2.10K 0 265M
data 34.9G 18.1T 0 1.77K 0 211M
I tried changing the txg sync time from 30 to 1 and that did make things
more smooth but in general lowered speeds (down to 90 megabytes/sec or
so). Actually writing files to the array I see well excess of 112
megabytes/sec so I would think I should be able to get this to go at
full gig speeds without the small stalls:
r...@opensolaris: 11:36 AM :/data# dd bs=1M count=100000 if=/dev/zero
of=./100gb.bin
100000+0 records in
100000+0 records out
104857600000 bytes (105 GB) copied, 233.257 s, 450 MB/s
_______________________________________________
zfs-discuss mailing list
zfs-discuss@opensolaris.org
http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
