On May 30, 2010, at 3:04 PM, Sandon Van Ness wrote: > Basically for a few seconds at a time I can get very nice speeds through > rsync (saturating a 1 gig link) which is around 112-113 megabytes/sec > which is about as good as I can expect after overhead. The problem is > that every 5 seconds when data is actually written to disks (physically > looking at the disk LEDs I see the activity on the one sending data > stall as the ZFS machine is showing disk activity (writes). > > Basically this problem is bringing down the average write speed from > around 112-113 megabytes/sec to around 100 megabytes/sec (sometimes > lower) and thus lowering speeds by 10% (or a bit more). I only really > care as 10% can make a difference when you are copying terrabytes of > data to the machine. Anyway here is what I am seeing on the linux > machine that is sending to the ZFS machine: > > > Device: rrqm/s wrqm/s r/s w/s rMB/s wMB/s > avgrq-sz avgqu-sz await svctm %util > sdc3 17.00 0.00 496.00 0.00 112.00 0.00 > 462.45 0.42 0.84 0.16 7.90 > > avg-cpu: %user %nice %system %iowait %steal %idle > 6.34 0.00 3.80 0.00 0.00 89.86 > > Device: rrqm/s wrqm/s r/s w/s rMB/s wMB/s > avgrq-sz avgqu-sz await svctm %util > sdc3 1.00 0.00 584.00 0.00 111.75 0.00 > 391.88 0.43 0.74 0.12 7.10 > > avg-cpu: %user %nice %system %iowait %steal %idle > 6.84 0.00 5.44 0.00 0.00 87.72 > > Device: rrqm/s wrqm/s r/s w/s rMB/s wMB/s > avgrq-sz avgqu-sz await svctm %util > sdc3 4.00 0.00 557.00 0.00 112.00 0.00 > 411.81 0.43 0.77 0.13 7.10 > > avg-cpu: %user %nice %system %iowait %steal %idle > 4.64 0.00 2.51 0.00 0.00 92.86 > > Device: rrqm/s wrqm/s r/s w/s rMB/s wMB/s > avgrq-sz avgqu-sz await svctm %util > sdc3 1.98 0.00 104.95 0.00 23.76 0.00 > 463.70 0.09 0.84 0.17 1.78 > > avg-cpu: %user %nice %system %iowait %steal %idle > 7.11 0.00 5.08 0.00 0.00 87.82 > > Device: rrqm/s wrqm/s r/s w/s rMB/s wMB/s > avgrq-sz avgqu-sz await svctm %util > sdc3 9.00 0.00 538.00 0.00 112.69 0.00 > 428.97 0.39 0.72 0.14 7.50 > > avg-cpu: %user %nice %system %iowait %steal %idle > 7.46 0.00 5.56 0.00 0.00 86.98 > > Device: rrqm/s wrqm/s r/s w/s rMB/s wMB/s > avgrq-sz avgqu-sz await svctm %util > sdc3 8.00 0.00 524.00 0.00 112.00 0.00 > 437.74 0.38 0.72 0.13 6.90 > > avg-cpu: %user %nice %system %iowait %steal %idle > 7.98 0.00 5.96 0.00 0.00 86.06 > > Device: rrqm/s wrqm/s r/s w/s rMB/s wMB/s > avgrq-sz avgqu-sz await svctm %util > sdc3 1.00 0.00 493.00 0.00 111.29 0.00 > 462.31 0.39 0.80 0.16 7.90 > > > This is iostat -xm 1 sdc3 > > Basically you can see its reading at around full gig speed and then > reads drop down due to writes being stalled on the ZFS machine. These > are 1 second averages. > > On the ZFS end with 10 second averages I see approximately 100 MB/sec: > > > data 25.1G 18.1T 0 834 0 100M > data 26.2G 18.1T 0 833 0 100M > data 27.3G 18.1T 0 833 0 100M > > > Change this to 1 second and I see: > data 32.7G 18.1T 0 0 0 0 > data 32.7G 18.1T 0 2.86K 0 360M > data 33.3G 18.1T 0 264 0 21.8M > data 33.3G 18.1T 0 0 0 0 > data 33.3G 18.1T 0 0 0 0 > data 33.3G 18.1T 0 0 0 0 > data 33.3G 18.1T 0 2.94K 0 369M > data 33.8G 18.1T 0 375 0 35.1M > data 33.8G 18.1T 0 0 0 0 > data 33.8G 18.1T 0 0 0 0 > data 33.8G 18.1T 0 0 0 0 > data 33.8G 18.1T 0 2.90K 0 365M > data 34.4G 18.1T 0 599 0 62.6M > data 34.4G 18.1T 0 0 0 0 > data 34.4G 18.1T 0 0 0 0 > data 34.4G 18.1T 0 0 0 0 > data 34.4G 18.1T 0 2.10K 0 265M > data 34.9G 18.1T 0 1.77K 0 211M > > > I tried changing the txg sync time from 30 to 1 and that did make things > more smooth but in general lowered speeds (down to 90 megabytes/sec or > so). Actually writing files to the array I see well excess of 112 > megabytes/sec so I would think I should be able to get this to go at > full gig speeds without the small stalls:
I have better luck tuning the zfs_txg_synctime_ms from 5000 to 1000 or less. > > r...@opensolaris: 11:36 AM :/data# dd bs=1M count=100000 if=/dev/zero > of=./100gb.bin > 100000+0 records in > 100000+0 records out > 104857600000 bytes (105 GB) copied, 233.257 s, 450 MB/s If you want to decouple the txg commit completely, then you might consider using a buffer of some sort. I use mbuffer for pipes, but that may be tricky to use in an rsync environment. -- richard -- Richard Elling rich...@nexenta.com +1-760-896-4422 ZFS and NexentaStor training, Rotterdam, July 13-15, 2010 http://nexenta-rotterdam.eventbrite.com/ _______________________________________________ zfs-discuss mailing list zfs-discuss@opensolaris.org http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
