Re: zfs send/receive: is this slow?
On Mon, Oct 04, 2010 at 07:39:16PM -0400, Dan Langille wrote: On 10/4/2010 2:10 PM, Jeremy Chadwick wrote: On Mon, Oct 04, 2010 at 01:31:07PM -0400, Dan Langille wrote: On Mon, October 4, 2010 3:27 am, Martin Matuska wrote: Try using zfs receive with the -v flag (gives you some stats at the end): # zfs send storage/bac...@transfer | zfs receive -v storage/compressed/bacula And use the following sysctl (you may set that in /boot/loader.conf, too): # sysctl vfs.zfs.txg.write_limit_override=805306368 I have good results with the 768MB writelimit on systems with at least 8GB RAM. With 4GB ram, you might want to try to set the TXG write limit to a lower threshold (e.g. 256MB): # sysctl vfs.zfs.txg.write_limit_override=268435456 You can experiment with that setting to get the best results on your system. A value of 0 means using calculated default (which is very high). I will experiment with the above. In the meantime: During the operation you can observe what your disks actually do: a) via ZFS pool I/O statistics: # zpool iostat -v 1 b) via GEOM: # gstat -a The following output was produced while the original copy was underway. $ sudo gstat -a -b -I 20s dT: 20.002s w: 20.000s L(q) ops/sr/s kBps ms/rw/s kBps ms/w %busy Name 7452387 248019.5 64 21287.1 79.4 ada0 7452387 248019.5 64 21287.2 79.4 ada0p1 4492427 246556.7 64 21286.6 63.0 ada1 4494428 246916.9 65 21276.6 66.9 ada2 8379313 24798 13.5 65 21277.5 78.6 ada3 5372306 24774 14.2 64 21277.5 77.6 ada4 10355291 24741 15.9 63 21277.4 79.6 ada5 4380316 24807 13.2 64 21287.7 77.0 ada6 7452387 248019.5 64 21287.4 79.7 gpt/disk06-live 4492427 246556.7 64 21286.7 63.1 ada1p1 4494428 246916.9 65 21276.6 66.9 ada2p1 8379313 24798 13.5 65 21277.6 78.6 ada3p1 5372306 24774 14.2 64 21277.6 77.6 ada4p1 10355291 24741 15.9 63 21277.5 79.6 ada5p1 4380316 24807 13.2 64 21287.8 77.0 ada6p1 4492427 246556.8 64 21286.9 63.4 gpt/disk01-live 4494428 246916.9 65 21276.8 67.2 gpt/disk02-live 8379313 24798 13.5 65 21277.7 78.8 gpt/disk03-live 5372306 24774 14.2 64 21277.8 77.8 gpt/disk04-live 10355291 24741 15.9 63 21277.7 79.8 gpt/disk05-live 4380316 24807 13.2 64 21288.0 77.2 gpt/disk07-live $ zpool ol iostat 10 capacity operationsbandwidth pool used avail read write read write -- - - - - - - storage 8.08T 4.60T364161 41.7M 7.94M storage 8.08T 4.60T926133 112M 5.91M storage 8.08T 4.60T738164 89.0M 9.75M storage 8.08T 4.60T 1.18K179 146M 8.10M storage 8.08T 4.60T 1.09K193 135M 9.94M storage 8.08T 4.60T 1010185 122M 8.68M storage 8.08T 4.60T 1.06K184 131M 9.65M storage 8.08T 4.60T867178 105M 11.8M storage 8.08T 4.60T 1.06K198 131M 12.0M storage 8.08T 4.60T 1.06K185 131M 12.4M Yeterday's write bandwidth was more 80-90M. It's down, a lot. I'll look closer this evening. mm Dňa 4. 10. 2010 4:06, Artem Belevich wrote / napÃsal(a): On Sun, Oct 3, 2010 at 6:11 PM, Dan Langilled...@langille.org wrote: I'm rerunning my test after I had a drive go offline[1]. But I'm not getting anything like the previous test: time zfs send storage/bac...@transfer | mbuffer | zfs receive storage/compressed/bacula-buffer $ zpool iostat 10 10 capacity operationsbandwidth pool used avail read write read write -- - - - - - - storage 6.83T 5.86T 8 31 1.00M 2.11M storage 6.83T 5.86T207481 25.7M 17.8M It may be worth checking individual disk activity using gstat -f 'da.$' Some time back I had one drive that was noticeably slower than the rest of the drives in RAID-Z2 vdev and was holding everything back. SMART looked OK, there were no obvious errors and yet performance was much worse than what I'd expect. gstat clearly showed that one drive was almost constantly busy with much lower number of reads and writes per second than its peers. Perhaps previously fast transfer rates were due to caching effects. I.e. if all metadata already made it into ARC, subsequent zfs send commands would avoid a lot of random seeks
Re: zfs send/receive: is this slow?
Try using zfs receive with the -v flag (gives you some stats at the end): # zfs send storage/bac...@transfer | zfs receive -v storage/compressed/bacula And use the following sysctl (you may set that in /boot/loader.conf, too): # sysctl vfs.zfs.txg.write_limit_override=805306368 I have good results with the 768MB writelimit on systems with at least 8GB RAM. With 4GB ram, you might want to try to set the TXG write limit to a lower threshold (e.g. 256MB): # sysctl vfs.zfs.txg.write_limit_override=268435456 You can experiment with that setting to get the best results on your system. A value of 0 means using calculated default (which is very high). During the operation you can observe what your disks actually do: a) via ZFS pool I/O statistics: # zpool iostat -v 1 b) via GEOM: # gstat -a mm Dňa 4. 10. 2010 4:06, Artem Belevich wrote / napísal(a): On Sun, Oct 3, 2010 at 6:11 PM, Dan Langille d...@langille.org wrote: I'm rerunning my test after I had a drive go offline[1]. But I'm not getting anything like the previous test: time zfs send storage/bac...@transfer | mbuffer | zfs receive storage/compressed/bacula-buffer $ zpool iostat 10 10 capacity operationsbandwidth pool used avail read write read write -- - - - - - - storage 6.83T 5.86T 8 31 1.00M 2.11M storage 6.83T 5.86T207481 25.7M 17.8M It may be worth checking individual disk activity using gstat -f 'da.$' Some time back I had one drive that was noticeably slower than the rest of the drives in RAID-Z2 vdev and was holding everything back. SMART looked OK, there were no obvious errors and yet performance was much worse than what I'd expect. gstat clearly showed that one drive was almost constantly busy with much lower number of reads and writes per second than its peers. Perhaps previously fast transfer rates were due to caching effects. I.e. if all metadata already made it into ARC, subsequent zfs send commands would avoid a lot of random seeks and would show much better throughput. --Artem ___ freebsd-stable@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-stable To unsubscribe, send any mail to freebsd-stable-unsubscr...@freebsd.org ___ freebsd-stable@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-stable To unsubscribe, send any mail to freebsd-stable-unsubscr...@freebsd.org
Re: zfs send/receive: is this slow?
On Mon, October 4, 2010 3:27 am, Martin Matuska wrote: Try using zfs receive with the -v flag (gives you some stats at the end): # zfs send storage/bac...@transfer | zfs receive -v storage/compressed/bacula And use the following sysctl (you may set that in /boot/loader.conf, too): # sysctl vfs.zfs.txg.write_limit_override=805306368 I have good results with the 768MB writelimit on systems with at least 8GB RAM. With 4GB ram, you might want to try to set the TXG write limit to a lower threshold (e.g. 256MB): # sysctl vfs.zfs.txg.write_limit_override=268435456 You can experiment with that setting to get the best results on your system. A value of 0 means using calculated default (which is very high). I will experiment with the above. In the meantime: During the operation you can observe what your disks actually do: a) via ZFS pool I/O statistics: # zpool iostat -v 1 b) via GEOM: # gstat -a The following output was produced while the original copy was underway. $ sudo gstat -a -b -I 20s dT: 20.002s w: 20.000s L(q) ops/sr/s kBps ms/rw/s kBps ms/w %busy Name 7452387 248019.5 64 21287.1 79.4 ada0 7452387 248019.5 64 21287.2 79.4 ada0p1 4492427 246556.7 64 21286.6 63.0 ada1 4494428 246916.9 65 21276.6 66.9 ada2 8379313 24798 13.5 65 21277.5 78.6 ada3 5372306 24774 14.2 64 21277.5 77.6 ada4 10355291 24741 15.9 63 21277.4 79.6 ada5 4380316 24807 13.2 64 21287.7 77.0 ada6 7452387 248019.5 64 21287.4 79.7 gpt/disk06-live 4492427 246556.7 64 21286.7 63.1 ada1p1 4494428 246916.9 65 21276.6 66.9 ada2p1 8379313 24798 13.5 65 21277.6 78.6 ada3p1 5372306 24774 14.2 64 21277.6 77.6 ada4p1 10355291 24741 15.9 63 21277.5 79.6 ada5p1 4380316 24807 13.2 64 21287.8 77.0 ada6p1 4492427 246556.8 64 21286.9 63.4 gpt/disk01-live 4494428 246916.9 65 21276.8 67.2 gpt/disk02-live 8379313 24798 13.5 65 21277.7 78.8 gpt/disk03-live 5372306 24774 14.2 64 21277.8 77.8 gpt/disk04-live 10355291 24741 15.9 63 21277.7 79.8 gpt/disk05-live 4380316 24807 13.2 64 21288.0 77.2 gpt/disk07-live $ zpool ol iostat 10 capacity operationsbandwidth pool used avail read write read write -- - - - - - - storage 8.08T 4.60T364161 41.7M 7.94M storage 8.08T 4.60T926133 112M 5.91M storage 8.08T 4.60T738164 89.0M 9.75M storage 8.08T 4.60T 1.18K179 146M 8.10M storage 8.08T 4.60T 1.09K193 135M 9.94M storage 8.08T 4.60T 1010185 122M 8.68M storage 8.08T 4.60T 1.06K184 131M 9.65M storage 8.08T 4.60T867178 105M 11.8M storage 8.08T 4.60T 1.06K198 131M 12.0M storage 8.08T 4.60T 1.06K185 131M 12.4M Yeterday's write bandwidth was more 80-90M. It's down, a lot. I'll look closer this evening. mm DÅa 4. 10. 2010 4:06, Artem Belevich wrote / napÃsal(a): On Sun, Oct 3, 2010 at 6:11 PM, Dan Langille d...@langille.org wrote: I'm rerunning my test after I had a drive go offline[1]. But I'm not getting anything like the previous test: time zfs send storage/bac...@transfer | mbuffer | zfs receive storage/compressed/bacula-buffer $ zpool iostat 10 10 capacity operationsbandwidth pool used avail read write read write -- - - - - - - storage 6.83T 5.86T 8 31 1.00M 2.11M storage 6.83T 5.86T207481 25.7M 17.8M It may be worth checking individual disk activity using gstat -f 'da.$' Some time back I had one drive that was noticeably slower than the rest of the drives in RAID-Z2 vdev and was holding everything back. SMART looked OK, there were no obvious errors and yet performance was much worse than what I'd expect. gstat clearly showed that one drive was almost constantly busy with much lower number of reads and writes per second than its peers. Perhaps previously fast transfer rates were due to caching effects. I.e. if all metadata already made it into ARC, subsequent zfs send commands would avoid a lot of random seeks and would show much better throughput. --Artem ___ freebsd-stable@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-stable To unsubscribe, send any mail to freebsd-stable-unsubscr...@freebsd.org --
Re: zfs send/receive: is this slow?
On Mon, Oct 04, 2010 at 01:31:07PM -0400, Dan Langille wrote: On Mon, October 4, 2010 3:27 am, Martin Matuska wrote: Try using zfs receive with the -v flag (gives you some stats at the end): # zfs send storage/bac...@transfer | zfs receive -v storage/compressed/bacula And use the following sysctl (you may set that in /boot/loader.conf, too): # sysctl vfs.zfs.txg.write_limit_override=805306368 I have good results with the 768MB writelimit on systems with at least 8GB RAM. With 4GB ram, you might want to try to set the TXG write limit to a lower threshold (e.g. 256MB): # sysctl vfs.zfs.txg.write_limit_override=268435456 You can experiment with that setting to get the best results on your system. A value of 0 means using calculated default (which is very high). I will experiment with the above. In the meantime: During the operation you can observe what your disks actually do: a) via ZFS pool I/O statistics: # zpool iostat -v 1 b) via GEOM: # gstat -a The following output was produced while the original copy was underway. $ sudo gstat -a -b -I 20s dT: 20.002s w: 20.000s L(q) ops/sr/s kBps ms/rw/s kBps ms/w %busy Name 7452387 248019.5 64 21287.1 79.4 ada0 7452387 248019.5 64 21287.2 79.4 ada0p1 4492427 246556.7 64 21286.6 63.0 ada1 4494428 246916.9 65 21276.6 66.9 ada2 8379313 24798 13.5 65 21277.5 78.6 ada3 5372306 24774 14.2 64 21277.5 77.6 ada4 10355291 24741 15.9 63 21277.4 79.6 ada5 4380316 24807 13.2 64 21287.7 77.0 ada6 7452387 248019.5 64 21287.4 79.7 gpt/disk06-live 4492427 246556.7 64 21286.7 63.1 ada1p1 4494428 246916.9 65 21276.6 66.9 ada2p1 8379313 24798 13.5 65 21277.6 78.6 ada3p1 5372306 24774 14.2 64 21277.6 77.6 ada4p1 10355291 24741 15.9 63 21277.5 79.6 ada5p1 4380316 24807 13.2 64 21287.8 77.0 ada6p1 4492427 246556.8 64 21286.9 63.4 gpt/disk01-live 4494428 246916.9 65 21276.8 67.2 gpt/disk02-live 8379313 24798 13.5 65 21277.7 78.8 gpt/disk03-live 5372306 24774 14.2 64 21277.8 77.8 gpt/disk04-live 10355291 24741 15.9 63 21277.7 79.8 gpt/disk05-live 4380316 24807 13.2 64 21288.0 77.2 gpt/disk07-live $ zpool ol iostat 10 capacity operationsbandwidth pool used avail read write read write -- - - - - - - storage 8.08T 4.60T364161 41.7M 7.94M storage 8.08T 4.60T926133 112M 5.91M storage 8.08T 4.60T738164 89.0M 9.75M storage 8.08T 4.60T 1.18K179 146M 8.10M storage 8.08T 4.60T 1.09K193 135M 9.94M storage 8.08T 4.60T 1010185 122M 8.68M storage 8.08T 4.60T 1.06K184 131M 9.65M storage 8.08T 4.60T867178 105M 11.8M storage 8.08T 4.60T 1.06K198 131M 12.0M storage 8.08T 4.60T 1.06K185 131M 12.4M Yeterday's write bandwidth was more 80-90M. It's down, a lot. I'll look closer this evening. mm DÅa 4. 10. 2010 4:06, Artem Belevich wrote / napÃsal(a): On Sun, Oct 3, 2010 at 6:11 PM, Dan Langille d...@langille.org wrote: I'm rerunning my test after I had a drive go offline[1]. But I'm not getting anything like the previous test: time zfs send storage/bac...@transfer | mbuffer | zfs receive storage/compressed/bacula-buffer $ zpool iostat 10 10 capacity operationsbandwidth pool used avail read write read write -- - - - - - - storage 6.83T 5.86T 8 31 1.00M 2.11M storage 6.83T 5.86T207481 25.7M 17.8M It may be worth checking individual disk activity using gstat -f 'da.$' Some time back I had one drive that was noticeably slower than the rest of the drives in RAID-Z2 vdev and was holding everything back. SMART looked OK, there were no obvious errors and yet performance was much worse than what I'd expect. gstat clearly showed that one drive was almost constantly busy with much lower number of reads and writes per second than its peers. Perhaps previously fast transfer rates were due to caching effects. I.e. if all metadata already made it into ARC, subsequent zfs send commands would avoid a lot of random seeks and would show much better throughput. --Artem Please read all of the following items before responding
Re: zfs send/receive: is this slow?
On 10/4/2010 2:10 PM, Jeremy Chadwick wrote: On Mon, Oct 04, 2010 at 01:31:07PM -0400, Dan Langille wrote: On Mon, October 4, 2010 3:27 am, Martin Matuska wrote: Try using zfs receive with the -v flag (gives you some stats at the end): # zfs send storage/bac...@transfer | zfs receive -v storage/compressed/bacula And use the following sysctl (you may set that in /boot/loader.conf, too): # sysctl vfs.zfs.txg.write_limit_override=805306368 I have good results with the 768MB writelimit on systems with at least 8GB RAM. With 4GB ram, you might want to try to set the TXG write limit to a lower threshold (e.g. 256MB): # sysctl vfs.zfs.txg.write_limit_override=268435456 You can experiment with that setting to get the best results on your system. A value of 0 means using calculated default (which is very high). I will experiment with the above. In the meantime: During the operation you can observe what your disks actually do: a) via ZFS pool I/O statistics: # zpool iostat -v 1 b) via GEOM: # gstat -a The following output was produced while the original copy was underway. $ sudo gstat -a -b -I 20s dT: 20.002s w: 20.000s L(q) ops/sr/s kBps ms/rw/s kBps ms/w %busy Name 7452387 248019.5 64 21287.1 79.4 ada0 7452387 248019.5 64 21287.2 79.4 ada0p1 4492427 246556.7 64 21286.6 63.0 ada1 4494428 246916.9 65 21276.6 66.9 ada2 8379313 24798 13.5 65 21277.5 78.6 ada3 5372306 24774 14.2 64 21277.5 77.6 ada4 10355291 24741 15.9 63 21277.4 79.6 ada5 4380316 24807 13.2 64 21287.7 77.0 ada6 7452387 248019.5 64 21287.4 79.7 gpt/disk06-live 4492427 246556.7 64 21286.7 63.1 ada1p1 4494428 246916.9 65 21276.6 66.9 ada2p1 8379313 24798 13.5 65 21277.6 78.6 ada3p1 5372306 24774 14.2 64 21277.6 77.6 ada4p1 10355291 24741 15.9 63 21277.5 79.6 ada5p1 4380316 24807 13.2 64 21287.8 77.0 ada6p1 4492427 246556.8 64 21286.9 63.4 gpt/disk01-live 4494428 246916.9 65 21276.8 67.2 gpt/disk02-live 8379313 24798 13.5 65 21277.7 78.8 gpt/disk03-live 5372306 24774 14.2 64 21277.8 77.8 gpt/disk04-live 10355291 24741 15.9 63 21277.7 79.8 gpt/disk05-live 4380316 24807 13.2 64 21288.0 77.2 gpt/disk07-live $ zpool ol iostat 10 capacity operationsbandwidth pool used avail read write read write -- - - - - - - storage 8.08T 4.60T364161 41.7M 7.94M storage 8.08T 4.60T926133 112M 5.91M storage 8.08T 4.60T738164 89.0M 9.75M storage 8.08T 4.60T 1.18K179 146M 8.10M storage 8.08T 4.60T 1.09K193 135M 9.94M storage 8.08T 4.60T 1010185 122M 8.68M storage 8.08T 4.60T 1.06K184 131M 9.65M storage 8.08T 4.60T867178 105M 11.8M storage 8.08T 4.60T 1.06K198 131M 12.0M storage 8.08T 4.60T 1.06K185 131M 12.4M Yeterday's write bandwidth was more 80-90M. It's down, a lot. I'll look closer this evening. mm Dňa 4. 10. 2010 4:06, Artem Belevich wrote / napÃsal(a): On Sun, Oct 3, 2010 at 6:11 PM, Dan Langilled...@langille.org wrote: I'm rerunning my test after I had a drive go offline[1]. But I'm not getting anything like the previous test: time zfs send storage/bac...@transfer | mbuffer | zfs receive storage/compressed/bacula-buffer $ zpool iostat 10 10 capacity operationsbandwidth pool used avail read write read write -- - - - - - - storage 6.83T 5.86T 8 31 1.00M 2.11M storage 6.83T 5.86T207481 25.7M 17.8M It may be worth checking individual disk activity using gstat -f 'da.$' Some time back I had one drive that was noticeably slower than the rest of the drives in RAID-Z2 vdev and was holding everything back. SMART looked OK, there were no obvious errors and yet performance was much worse than what I'd expect. gstat clearly showed that one drive was almost constantly busy with much lower number of reads and writes per second than its peers. Perhaps previously fast transfer rates were due to caching effects. I.e. if all metadata already made it into ARC, subsequent zfs send commands would avoid a lot of random seeks and would show much better throughput. --Artem Please read all of the following items before responding in-line. Some are just informational items for other people reading
Re: zfs send/receive: is this slow?
I've just tested on my box and loopback interface does not seem to be the bottleneck. I can easily push through ~400MB/s through two instances of mbuffer. --Artem On Fri, Oct 1, 2010 at 7:51 PM, Sean s...@gothic.net.au wrote: On 02/10/2010, at 11:43 AM, Artem Belevich wrote: As soon as I opened this email I knew what it would say. # time zfs send storage/bac...@transfer | mbuffer | zfs receive storage/compressed/bacula-mbuffer in @ 197 MB/s, out @ 205 MB/s, 1749 MB total, buffer 0% full .. Big difference. :) I'm glad it helped. Does anyone know why sending/receiving stuff via loopback is so much slower compared to pipe? Up and down the entire network stack, in and out of TCP buffers at both ends... might add some overhead, and other factors in limiting it. Increasing TCP buffers, and disabling delayed acks might help. Nagle might also have to be disabled too. (delayed acks and nagle in combination can interact in odd ways) --Artem ___ freebsd-stable@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-stable To unsubscribe, send any mail to freebsd-stable-unsubscr...@freebsd.org ___ freebsd-stable@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-stable To unsubscribe, send any mail to freebsd-stable-unsubscr...@freebsd.org
Re: zfs send/receive: is this slow?
On 10/1/2010 9:32 PM, Dan Langille wrote: On 10/1/2010 7:00 PM, Artem Belevich wrote: On Fri, Oct 1, 2010 at 3:49 PM, Dan Langilled...@langille.org wrote: FYI: this is all on the same box. In one of the previous emails you've used this command line: # mbuffer -s 128k -m 1G -I 9090 | zfs receive You've used mbuffer in network client mode. I assumed that you did do your transfer over network. If you're running send/receive locally just pipe the data through mbuffer -- zfs send|mbuffer|zfs receive As soon as I opened this email I knew what it would say. # time zfs send storage/bac...@transfer | mbuffer | zfs receive storage/compressed/bacula-mbuffer in @ 197 MB/s, out @ 205 MB/s, 1749 MB total, buffer 0% full $ zpool iostat 10 10 capacity operations bandwidth pool used avail read write read write -- - - - - - - storage 9.78T 2.91T 1.11K 336 92.0M 17.3M storage 9.78T 2.91T 769 436 95.5M 30.5M storage 9.78T 2.91T 797 853 98.9M 78.5M storage 9.78T 2.91T 865 962 107M 78.0M storage 9.78T 2.91T 828 881 103M 82.6M storage 9.78T 2.90T 1023 1.12K 127M 91.0M storage 9.78T 2.90T 1.01K 1.01K 128M 89.3M storage 9.79T 2.90T 962 1.08K 119M 89.1M storage 9.79T 2.90T 1.09K 1.25K 139M 67.8M Big difference. :) I'm rerunning my test after I had a drive go offline[1]. But I'm not getting anything like the previous test: time zfs send storage/bac...@transfer | mbuffer | zfs receive storage/compressed/bacula-buffer $ zpool iostat 10 10 capacity operationsbandwidth pool used avail read write read write -- - - - - - - storage 6.83T 5.86T 8 31 1.00M 2.11M storage 6.83T 5.86T207481 25.7M 17.8M storage 6.83T 5.86T220516 27.4M 17.2M storage 6.83T 5.86T221523 27.5M 21.0M storage 6.83T 5.86T198430 24.5M 20.4M storage 6.83T 5.86T248528 30.8M 26.7M storage 6.83T 5.86T273508 33.9M 22.6M storage 6.83T 5.86T331499 41.1M 22.7M storage 6.83T 5.86T424662 52.6M 34.7M storage 6.83T 5.86T413605 51.3M 36.7M [1] - http://docs.freebsd.org/cgi/mid.cgi?4CA73702.5080203 -- Dan Langille - http://langille.org/ ___ freebsd-stable@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-stable To unsubscribe, send any mail to freebsd-stable-unsubscr...@freebsd.org
Re: zfs send/receive: is this slow?
On Sun, Oct 3, 2010 at 6:11 PM, Dan Langille d...@langille.org wrote: I'm rerunning my test after I had a drive go offline[1]. But I'm not getting anything like the previous test: time zfs send storage/bac...@transfer | mbuffer | zfs receive storage/compressed/bacula-buffer $ zpool iostat 10 10 capacity operations bandwidth pool used avail read write read write -- - - - - - - storage 6.83T 5.86T 8 31 1.00M 2.11M storage 6.83T 5.86T 207 481 25.7M 17.8M It may be worth checking individual disk activity using gstat -f 'da.$' Some time back I had one drive that was noticeably slower than the rest of the drives in RAID-Z2 vdev and was holding everything back. SMART looked OK, there were no obvious errors and yet performance was much worse than what I'd expect. gstat clearly showed that one drive was almost constantly busy with much lower number of reads and writes per second than its peers. Perhaps previously fast transfer rates were due to caching effects. I.e. if all metadata already made it into ARC, subsequent zfs send commands would avoid a lot of random seeks and would show much better throughput. --Artem ___ freebsd-stable@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-stable To unsubscribe, send any mail to freebsd-stable-unsubscr...@freebsd.org
Re: zfs send/receive: is this slow?
On Wed, September 29, 2010 3:57 pm, Artem Belevich wrote: On Wed, Sep 29, 2010 at 11:04 AM, Dan Langille d...@langille.org wrote: It's taken about 15 hours to copy 800GB. I'm sure there's some tuning I can do. The system is now running: # zfs send storage/bac...@transfer | zfs receive storage/compressed/bacula Try piping zfs data through mbuffer (misc/mbuffer in ports). I've found that it does help a lot to smooth out data flow and increase send/receive throughput even when send/receive happens on the same host. Run it with a buffer large enough to accommodate few seconds worth of write throughput for your target disks. Here's an example: http://blogs.everycity.co.uk/alasdair/2010/07/using-mbuffer-to-speed-up-slow-zfs-send-zfs-receive/ I'm failing. In one session: # mbuffer -s 128k -m 1G -I 9090 | zfs receive storage/compressed/bacula-mbuffer Assertion failed: ((err == 0) (bsize == sizeof(rcvsize))), function openNetworkInput, file mbuffer.c, line 1358. cannot receive: failed to read from stream In the other session: # time zfs send storage/bac...@transfer | mbuffer -s 128k -m 1G -O 10.55.0.44:9090 Assertion failed: ((err == 0) (bsize == sizeof(sndsize))), function openNetworkOutput, file mbuffer.c, line 897. warning: cannot send 'storage/bac...@transfer': Broken pipe Abort trap: 6 (core dumped) real0m17.709s user0m0.000s sys 0m2.502s -- Dan Langille -- http://langille.org/ ___ freebsd-stable@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-stable To unsubscribe, send any mail to freebsd-stable-unsubscr...@freebsd.org
Re: zfs send/receive: is this slow?
On Fri, October 1, 2010 11:45 am, Dan Langille wrote: On Wed, September 29, 2010 3:57 pm, Artem Belevich wrote: On Wed, Sep 29, 2010 at 11:04 AM, Dan Langille d...@langille.org wrote: It's taken about 15 hours to copy 800GB. I'm sure there's some tuning I can do. The system is now running: # zfs send storage/bac...@transfer | zfs receive storage/compressed/bacula Try piping zfs data through mbuffer (misc/mbuffer in ports). I've found that it does help a lot to smooth out data flow and increase send/receive throughput even when send/receive happens on the same host. Run it with a buffer large enough to accommodate few seconds worth of write throughput for your target disks. Here's an example: http://blogs.everycity.co.uk/alasdair/2010/07/using-mbuffer-to-speed-up-slow-zfs-send-zfs-receive/ I'm failing. In one session: # mbuffer -s 128k -m 1G -I 9090 | zfs receive storage/compressed/bacula-mbuffer Assertion failed: ((err == 0) (bsize == sizeof(rcvsize))), function openNetworkInput, file mbuffer.c, line 1358. cannot receive: failed to read from stream In the other session: # time zfs send storage/bac...@transfer | mbuffer -s 128k -m 1G -O 10.55.0.44:9090 Assertion failed: ((err == 0) (bsize == sizeof(sndsize))), function openNetworkOutput, file mbuffer.c, line 897. warning: cannot send 'storage/bac...@transfer': Broken pipe Abort trap: 6 (core dumped) real0m17.709s user0m0.000s sys 0m2.502s My installed mbuffer was out of date. After an upgrade: # mbuffer -s 128k -m 1G -I 9090 | zfs receive storage/compressed/bacula-mbuffer mbuffer: warning: unable to set socket buffer size: No buffer space available in @ 0.0 kB/s, out @ 0.0 kB/s, 1897 MB total, buffer 100% full # time zfs send storage/bac...@transfer | mbuffer -s 128k -m 1G -O ::1:9090 mbuffer: warning: unable to set socket buffer size: No buffer space available in @ 4343 kB/s, out @ 2299 kB/s, 3104 MB total, buffer 85% full -- Dan Langille -- http://langille.org/ ___ freebsd-stable@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-stable To unsubscribe, send any mail to freebsd-stable-unsubscr...@freebsd.org
Re: zfs send/receive: is this slow?
On Wed, September 29, 2010 2:04 pm, Dan Langille wrote: $ zpool iostat 10 capacity operationsbandwidth pool used avail read write read write -- - - - - - - storage 7.67T 5.02T358 38 43.1M 1.96M storage 7.67T 5.02T317475 39.4M 30.9M storage 7.67T 5.02T357533 44.3M 34.4M storage 7.67T 5.02T371556 46.0M 35.8M storage 7.67T 5.02T313521 38.9M 28.7M storage 7.67T 5.02T309457 38.4M 30.4M storage 7.67T 5.02T388589 48.2M 37.8M storage 7.67T 5.02T377581 46.8M 36.5M storage 7.67T 5.02T310559 38.4M 30.4M storage 7.67T 5.02T430611 53.4M 41.3M Now that I'm using mbuffer: $ zpool iostat 10 capacity operationsbandwidth pool used avail read write read write -- - - - - - - storage 9.96T 2.73T 2.01K131 151M 6.72M storage 9.96T 2.73T615515 76.3M 33.5M storage 9.96T 2.73T360492 44.7M 33.7M storage 9.96T 2.73T388554 48.3M 38.4M storage 9.96T 2.73T403562 50.1M 39.6M storage 9.96T 2.73T313468 38.9M 28.0M storage 9.96T 2.73T462677 57.3M 22.4M storage 9.96T 2.73T383581 47.5M 21.6M storage 9.96T 2.72T142571 17.7M 15.4M storage 9.96T 2.72T 80598 10.0M 18.8M storage 9.96T 2.72T718503 89.1M 13.6M storage 9.96T 2.72T594517 73.8M 14.1M storage 9.96T 2.72T367528 45.6M 15.1M storage 9.96T 2.72T338520 41.9M 16.4M storage 9.96T 2.72T348499 43.3M 21.5M storage 9.96T 2.72T398553 49.4M 14.4M storage 9.96T 2.72T346481 43.0M 6.78M If anything, it's slower. The above was without -s 128. The following used that setting: $ zpool iostat 10 capacity operationsbandwidth pool used avail read write read write -- - - - - - - storage 9.78T 2.91T 1.98K137 149M 6.92M storage 9.78T 2.91T761577 94.4M 42.6M storage 9.78T 2.91T462411 57.4M 24.6M storage 9.78T 2.91T492497 61.1M 27.6M storage 9.78T 2.91T632446 78.5M 22.5M storage 9.78T 2.91T554414 68.7M 21.8M storage 9.78T 2.91T459434 57.0M 31.4M storage 9.78T 2.91T398570 49.4M 32.7M storage 9.78T 2.91T338495 41.9M 26.5M storage 9.78T 2.91T358526 44.5M 33.3M storage 9.78T 2.91T385555 47.8M 39.8M storage 9.78T 2.91T271453 33.6M 23.3M storage 9.78T 2.91T270456 33.5M 28.8M ___ freebsd-stable@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-stable To unsubscribe, send any mail to freebsd-stable-unsubscr...@freebsd.org
Re: zfs send/receive: is this slow?
On Fri, Oct 01, 2010 at 02:51:12PM -0400, Dan Langille wrote: On Wed, September 29, 2010 2:04 pm, Dan Langille wrote: $ zpool iostat 10 capacity operationsbandwidth pool used avail read write read write -- - - - - - - storage 7.67T 5.02T358 38 43.1M 1.96M storage 7.67T 5.02T317475 39.4M 30.9M storage 7.67T 5.02T357533 44.3M 34.4M storage 7.67T 5.02T371556 46.0M 35.8M storage 7.67T 5.02T313521 38.9M 28.7M storage 7.67T 5.02T309457 38.4M 30.4M storage 7.67T 5.02T388589 48.2M 37.8M storage 7.67T 5.02T377581 46.8M 36.5M storage 7.67T 5.02T310559 38.4M 30.4M storage 7.67T 5.02T430611 53.4M 41.3M Now that I'm using mbuffer: $ zpool iostat 10 capacity operationsbandwidth pool used avail read write read write -- - - - - - - storage 9.96T 2.73T 2.01K131 151M 6.72M storage 9.96T 2.73T615515 76.3M 33.5M storage 9.96T 2.73T360492 44.7M 33.7M storage 9.96T 2.73T388554 48.3M 38.4M storage 9.96T 2.73T403562 50.1M 39.6M storage 9.96T 2.73T313468 38.9M 28.0M storage 9.96T 2.73T462677 57.3M 22.4M storage 9.96T 2.73T383581 47.5M 21.6M storage 9.96T 2.72T142571 17.7M 15.4M storage 9.96T 2.72T 80598 10.0M 18.8M storage 9.96T 2.72T718503 89.1M 13.6M storage 9.96T 2.72T594517 73.8M 14.1M storage 9.96T 2.72T367528 45.6M 15.1M storage 9.96T 2.72T338520 41.9M 16.4M storage 9.96T 2.72T348499 43.3M 21.5M storage 9.96T 2.72T398553 49.4M 14.4M storage 9.96T 2.72T346481 43.0M 6.78M If anything, it's slower. The above was without -s 128. The following used that setting: $ zpool iostat 10 capacity operationsbandwidth pool used avail read write read write -- - - - - - - storage 9.78T 2.91T 1.98K137 149M 6.92M storage 9.78T 2.91T761577 94.4M 42.6M storage 9.78T 2.91T462411 57.4M 24.6M storage 9.78T 2.91T492497 61.1M 27.6M storage 9.78T 2.91T632446 78.5M 22.5M storage 9.78T 2.91T554414 68.7M 21.8M storage 9.78T 2.91T459434 57.0M 31.4M storage 9.78T 2.91T398570 49.4M 32.7M storage 9.78T 2.91T338495 41.9M 26.5M storage 9.78T 2.91T358526 44.5M 33.3M storage 9.78T 2.91T385555 47.8M 39.8M storage 9.78T 2.91T271453 33.6M 23.3M storage 9.78T 2.91T270456 33.5M 28.8M For what it's worth, this mimics the behaviour I saw long ago when using flexbackup[1] (which used SSH) to back up numerous machines on our local gigE network. flexbackup strongly advocates use of mbuffer or afio to attempt to buffer I/O between source and destination. What I witnessed was I/O rates that were either identical or worse (most of the time, worse) when mbuffer was used (regardless of what I chose for -s and -m). I switched to rsnapshot (which uses rsync via SSH) for a lot of reasons which are outside of the scope of this topic. I don't care to get into a discussion about the I/O bottlenecks stock OpenSSH has (vs. one patched with the high-performance patches) either; the point is that mbuffer did absolutely nothing or made things worse. This[2] didn't impress me either. [1]: http://www.edwinh.org/flexbackup/ [2]: http://www.edwinh.org/flexbackup/faq.html#Common%20problems4 -- | Jeremy Chadwick j...@parodius.com | | Parodius Networking http://www.parodius.com/ | | UNIX Systems Administrator Mountain View, CA, USA | | Making life hard for others since 1977. PGP: 4BD6C0CB | ___ freebsd-stable@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-stable To unsubscribe, send any mail to freebsd-stable-unsubscr...@freebsd.org
Re: zfs send/receive: is this slow?
Hmm. It did help me a lot when I was replicating ~2TB worth of data over GigE. Without mbuffer things were roughly in the ballpark of your numbers. With mbuffer I've got around 100MB/s. Assuming that you have two boxes connected via ethernet, it would be good to check that nobody generates PAUSE frames. Some time back I've discovered that el-cheapo switch I've been using for some reason could not keep up with traffic bursts and generated tons of PAUSE frames that severely limited throughput. If you're using Intel adapters, check xon/xoff counters in sysctl dev.em.0.mac_stats. If you see them increasing, that may explain slow speed. If you have a switch between your boxes, try bypassing it and connect boxes directly. --Artem On Fri, Oct 1, 2010 at 11:51 AM, Dan Langille d...@langille.org wrote: On Wed, September 29, 2010 2:04 pm, Dan Langille wrote: $ zpool iostat 10 capacity operations bandwidth pool used avail read write read write -- - - - - - - storage 7.67T 5.02T 358 38 43.1M 1.96M storage 7.67T 5.02T 317 475 39.4M 30.9M storage 7.67T 5.02T 357 533 44.3M 34.4M storage 7.67T 5.02T 371 556 46.0M 35.8M storage 7.67T 5.02T 313 521 38.9M 28.7M storage 7.67T 5.02T 309 457 38.4M 30.4M storage 7.67T 5.02T 388 589 48.2M 37.8M storage 7.67T 5.02T 377 581 46.8M 36.5M storage 7.67T 5.02T 310 559 38.4M 30.4M storage 7.67T 5.02T 430 611 53.4M 41.3M Now that I'm using mbuffer: $ zpool iostat 10 capacity operations bandwidth pool used avail read write read write -- - - - - - - storage 9.96T 2.73T 2.01K 131 151M 6.72M storage 9.96T 2.73T 615 515 76.3M 33.5M storage 9.96T 2.73T 360 492 44.7M 33.7M storage 9.96T 2.73T 388 554 48.3M 38.4M storage 9.96T 2.73T 403 562 50.1M 39.6M storage 9.96T 2.73T 313 468 38.9M 28.0M storage 9.96T 2.73T 462 677 57.3M 22.4M storage 9.96T 2.73T 383 581 47.5M 21.6M storage 9.96T 2.72T 142 571 17.7M 15.4M storage 9.96T 2.72T 80 598 10.0M 18.8M storage 9.96T 2.72T 718 503 89.1M 13.6M storage 9.96T 2.72T 594 517 73.8M 14.1M storage 9.96T 2.72T 367 528 45.6M 15.1M storage 9.96T 2.72T 338 520 41.9M 16.4M storage 9.96T 2.72T 348 499 43.3M 21.5M storage 9.96T 2.72T 398 553 49.4M 14.4M storage 9.96T 2.72T 346 481 43.0M 6.78M If anything, it's slower. The above was without -s 128. The following used that setting: $ zpool iostat 10 capacity operations bandwidth pool used avail read write read write -- - - - - - - storage 9.78T 2.91T 1.98K 137 149M 6.92M storage 9.78T 2.91T 761 577 94.4M 42.6M storage 9.78T 2.91T 462 411 57.4M 24.6M storage 9.78T 2.91T 492 497 61.1M 27.6M storage 9.78T 2.91T 632 446 78.5M 22.5M storage 9.78T 2.91T 554 414 68.7M 21.8M storage 9.78T 2.91T 459 434 57.0M 31.4M storage 9.78T 2.91T 398 570 49.4M 32.7M storage 9.78T 2.91T 338 495 41.9M 26.5M storage 9.78T 2.91T 358 526 44.5M 33.3M storage 9.78T 2.91T 385 555 47.8M 39.8M storage 9.78T 2.91T 271 453 33.6M 23.3M storage 9.78T 2.91T 270 456 33.5M 28.8M ___ freebsd-stable@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-stable To unsubscribe, send any mail to freebsd-stable-unsubscr...@freebsd.org ___ freebsd-stable@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-stable To unsubscribe, send any mail to freebsd-stable-unsubscr...@freebsd.org
Re: zfs send/receive: is this slow?
On Fri, Oct 1, 2010 at 3:49 PM, Dan Langille d...@langille.org wrote: FYI: this is all on the same box. In one of the previous emails you've used this command line: # mbuffer -s 128k -m 1G -I 9090 | zfs receive You've used mbuffer in network client mode. I assumed that you did do your transfer over network. If you're running send/receive locally just pipe the data through mbuffer -- zfs send|mbuffer|zfs receive --Artem -- Dan Langille http://langille.org/ On Oct 1, 2010, at 5:56 PM, Artem Belevich fbsdl...@src.cx wrote: Hmm. It did help me a lot when I was replicating ~2TB worth of data over GigE. Without mbuffer things were roughly in the ballpark of your numbers. With mbuffer I've got around 100MB/s. Assuming that you have two boxes connected via ethernet, it would be good to check that nobody generates PAUSE frames. Some time back I've discovered that el-cheapo switch I've been using for some reason could not keep up with traffic bursts and generated tons of PAUSE frames that severely limited throughput. If you're using Intel adapters, check xon/xoff counters in sysctl dev.em.0.mac_stats. If you see them increasing, that may explain slow speed. If you have a switch between your boxes, try bypassing it and connect boxes directly. --Artem On Fri, Oct 1, 2010 at 11:51 AM, Dan Langille d...@langille.org wrote: On Wed, September 29, 2010 2:04 pm, Dan Langille wrote: $ zpool iostat 10 capacity operations bandwidth pool used avail read write read write -- - - - - - - storage 7.67T 5.02T 358 38 43.1M 1.96M storage 7.67T 5.02T 317 475 39.4M 30.9M storage 7.67T 5.02T 357 533 44.3M 34.4M storage 7.67T 5.02T 371 556 46.0M 35.8M storage 7.67T 5.02T 313 521 38.9M 28.7M storage 7.67T 5.02T 309 457 38.4M 30.4M storage 7.67T 5.02T 388 589 48.2M 37.8M storage 7.67T 5.02T 377 581 46.8M 36.5M storage 7.67T 5.02T 310 559 38.4M 30.4M storage 7.67T 5.02T 430 611 53.4M 41.3M Now that I'm using mbuffer: $ zpool iostat 10 capacity operations bandwidth pool used avail read write read write -- - - - - - - storage 9.96T 2.73T 2.01K 131 151M 6.72M storage 9.96T 2.73T 615 515 76.3M 33.5M storage 9.96T 2.73T 360 492 44.7M 33.7M storage 9.96T 2.73T 388 554 48.3M 38.4M storage 9.96T 2.73T 403 562 50.1M 39.6M storage 9.96T 2.73T 313 468 38.9M 28.0M storage 9.96T 2.73T 462 677 57.3M 22.4M storage 9.96T 2.73T 383 581 47.5M 21.6M storage 9.96T 2.72T 142 571 17.7M 15.4M storage 9.96T 2.72T 80 598 10.0M 18.8M storage 9.96T 2.72T 718 503 89.1M 13.6M storage 9.96T 2.72T 594 517 73.8M 14.1M storage 9.96T 2.72T 367 528 45.6M 15.1M storage 9.96T 2.72T 338 520 41.9M 16.4M storage 9.96T 2.72T 348 499 43.3M 21.5M storage 9.96T 2.72T 398 553 49.4M 14.4M storage 9.96T 2.72T 346 481 43.0M 6.78M If anything, it's slower. The above was without -s 128. The following used that setting: $ zpool iostat 10 capacity operations bandwidth pool used avail read write read write -- - - - - - - storage 9.78T 2.91T 1.98K 137 149M 6.92M storage 9.78T 2.91T 761 577 94.4M 42.6M storage 9.78T 2.91T 462 411 57.4M 24.6M storage 9.78T 2.91T 492 497 61.1M 27.6M storage 9.78T 2.91T 632 446 78.5M 22.5M storage 9.78T 2.91T 554 414 68.7M 21.8M storage 9.78T 2.91T 459 434 57.0M 31.4M storage 9.78T 2.91T 398 570 49.4M 32.7M storage 9.78T 2.91T 338 495 41.9M 26.5M storage 9.78T 2.91T 358 526 44.5M 33.3M storage 9.78T 2.91T 385 555 47.8M 39.8M storage 9.78T 2.91T 271 453 33.6M 23.3M storage 9.78T 2.91T 270 456 33.5M 28.8M ___ freebsd-stable@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-stable To unsubscribe, send any mail to freebsd-stable-unsubscr...@freebsd.org ___ freebsd-stable@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-stable To unsubscribe, send any mail to freebsd-stable-unsubscr...@freebsd.org
Re: zfs send/receive: is this slow?
FYI: this is all on the same box. -- Dan Langille http://langille.org/ On Oct 1, 2010, at 5:56 PM, Artem Belevich fbsdl...@src.cx wrote: Hmm. It did help me a lot when I was replicating ~2TB worth of data over GigE. Without mbuffer things were roughly in the ballpark of your numbers. With mbuffer I've got around 100MB/s. Assuming that you have two boxes connected via ethernet, it would be good to check that nobody generates PAUSE frames. Some time back I've discovered that el-cheapo switch I've been using for some reason could not keep up with traffic bursts and generated tons of PAUSE frames that severely limited throughput. If you're using Intel adapters, check xon/xoff counters in sysctl dev.em.0.mac_stats. If you see them increasing, that may explain slow speed. If you have a switch between your boxes, try bypassing it and connect boxes directly. --Artem On Fri, Oct 1, 2010 at 11:51 AM, Dan Langille d...@langille.org wrote: On Wed, September 29, 2010 2:04 pm, Dan Langille wrote: $ zpool iostat 10 capacity operationsbandwidth pool used avail read write read write -- - - - - - - storage 7.67T 5.02T358 38 43.1M 1.96M storage 7.67T 5.02T317475 39.4M 30.9M storage 7.67T 5.02T357533 44.3M 34.4M storage 7.67T 5.02T371556 46.0M 35.8M storage 7.67T 5.02T313521 38.9M 28.7M storage 7.67T 5.02T309457 38.4M 30.4M storage 7.67T 5.02T388589 48.2M 37.8M storage 7.67T 5.02T377581 46.8M 36.5M storage 7.67T 5.02T310559 38.4M 30.4M storage 7.67T 5.02T430611 53.4M 41.3M Now that I'm using mbuffer: $ zpool iostat 10 capacity operationsbandwidth pool used avail read write read write -- - - - - - - storage 9.96T 2.73T 2.01K131 151M 6.72M storage 9.96T 2.73T615515 76.3M 33.5M storage 9.96T 2.73T360492 44.7M 33.7M storage 9.96T 2.73T388554 48.3M 38.4M storage 9.96T 2.73T403562 50.1M 39.6M storage 9.96T 2.73T313468 38.9M 28.0M storage 9.96T 2.73T462677 57.3M 22.4M storage 9.96T 2.73T383581 47.5M 21.6M storage 9.96T 2.72T142571 17.7M 15.4M storage 9.96T 2.72T 80598 10.0M 18.8M storage 9.96T 2.72T718503 89.1M 13.6M storage 9.96T 2.72T594517 73.8M 14.1M storage 9.96T 2.72T367528 45.6M 15.1M storage 9.96T 2.72T338520 41.9M 16.4M storage 9.96T 2.72T348499 43.3M 21.5M storage 9.96T 2.72T398553 49.4M 14.4M storage 9.96T 2.72T346481 43.0M 6.78M If anything, it's slower. The above was without -s 128. The following used that setting: $ zpool iostat 10 capacity operationsbandwidth pool used avail read write read write -- - - - - - - storage 9.78T 2.91T 1.98K137 149M 6.92M storage 9.78T 2.91T761577 94.4M 42.6M storage 9.78T 2.91T462411 57.4M 24.6M storage 9.78T 2.91T492497 61.1M 27.6M storage 9.78T 2.91T632446 78.5M 22.5M storage 9.78T 2.91T554414 68.7M 21.8M storage 9.78T 2.91T459434 57.0M 31.4M storage 9.78T 2.91T398570 49.4M 32.7M storage 9.78T 2.91T338495 41.9M 26.5M storage 9.78T 2.91T358526 44.5M 33.3M storage 9.78T 2.91T385555 47.8M 39.8M storage 9.78T 2.91T271453 33.6M 23.3M storage 9.78T 2.91T270456 33.5M 28.8M ___ freebsd-stable@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-stable To unsubscribe, send any mail to freebsd-stable-unsubscr...@freebsd.org ___ freebsd-stable@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-stable To unsubscribe, send any mail to freebsd-stable-unsubscr...@freebsd.org
Re: zfs send/receive: is this slow?
On 10/1/2010 7:00 PM, Artem Belevich wrote: On Fri, Oct 1, 2010 at 3:49 PM, Dan Langilled...@langille.org wrote: FYI: this is all on the same box. In one of the previous emails you've used this command line: # mbuffer -s 128k -m 1G -I 9090 | zfs receive You've used mbuffer in network client mode. I assumed that you did do your transfer over network. If you're running send/receive locally just pipe the data through mbuffer -- zfs send|mbuffer|zfs receive As soon as I opened this email I knew what it would say. # time zfs send storage/bac...@transfer | mbuffer | zfs receive storage/compressed/bacula-mbuffer in @ 197 MB/s, out @ 205 MB/s, 1749 MB total, buffer 0% full $ zpool iostat 10 10 capacity operationsbandwidth pool used avail read write read write -- - - - - - - storage 9.78T 2.91T 1.11K336 92.0M 17.3M storage 9.78T 2.91T769436 95.5M 30.5M storage 9.78T 2.91T797853 98.9M 78.5M storage 9.78T 2.91T865962 107M 78.0M storage 9.78T 2.91T828881 103M 82.6M storage 9.78T 2.90T 1023 1.12K 127M 91.0M storage 9.78T 2.90T 1.01K 1.01K 128M 89.3M storage 9.79T 2.90T962 1.08K 119M 89.1M storage 9.79T 2.90T 1.09K 1.25K 139M 67.8M Big difference. :) --Artem -- Dan Langille http://langille.org/ On Oct 1, 2010, at 5:56 PM, Artem Belevichfbsdl...@src.cx wrote: Hmm. It did help me a lot when I was replicating ~2TB worth of data over GigE. Without mbuffer things were roughly in the ballpark of your numbers. With mbuffer I've got around 100MB/s. Assuming that you have two boxes connected via ethernet, it would be good to check that nobody generates PAUSE frames. Some time back I've discovered that el-cheapo switch I've been using for some reason could not keep up with traffic bursts and generated tons of PAUSE frames that severely limited throughput. If you're using Intel adapters, check xon/xoff counters in sysctl dev.em.0.mac_stats. If you see them increasing, that may explain slow speed. If you have a switch between your boxes, try bypassing it and connect boxes directly. --Artem On Fri, Oct 1, 2010 at 11:51 AM, Dan Langilled...@langille.org wrote: On Wed, September 29, 2010 2:04 pm, Dan Langille wrote: $ zpool iostat 10 capacity operationsbandwidth pool used avail read write read write -- - - - - - - storage 7.67T 5.02T358 38 43.1M 1.96M storage 7.67T 5.02T317475 39.4M 30.9M storage 7.67T 5.02T357533 44.3M 34.4M storage 7.67T 5.02T371556 46.0M 35.8M storage 7.67T 5.02T313521 38.9M 28.7M storage 7.67T 5.02T309457 38.4M 30.4M storage 7.67T 5.02T388589 48.2M 37.8M storage 7.67T 5.02T377581 46.8M 36.5M storage 7.67T 5.02T310559 38.4M 30.4M storage 7.67T 5.02T430611 53.4M 41.3M Now that I'm using mbuffer: $ zpool iostat 10 capacity operationsbandwidth pool used avail read write read write -- - - - - - - storage 9.96T 2.73T 2.01K131 151M 6.72M storage 9.96T 2.73T615515 76.3M 33.5M storage 9.96T 2.73T360492 44.7M 33.7M storage 9.96T 2.73T388554 48.3M 38.4M storage 9.96T 2.73T403562 50.1M 39.6M storage 9.96T 2.73T313468 38.9M 28.0M storage 9.96T 2.73T462677 57.3M 22.4M storage 9.96T 2.73T383581 47.5M 21.6M storage 9.96T 2.72T142571 17.7M 15.4M storage 9.96T 2.72T 80598 10.0M 18.8M storage 9.96T 2.72T718503 89.1M 13.6M storage 9.96T 2.72T594517 73.8M 14.1M storage 9.96T 2.72T367528 45.6M 15.1M storage 9.96T 2.72T338520 41.9M 16.4M storage 9.96T 2.72T348499 43.3M 21.5M storage 9.96T 2.72T398553 49.4M 14.4M storage 9.96T 2.72T346481 43.0M 6.78M If anything, it's slower. The above was without -s 128. The following used that setting: $ zpool iostat 10 capacity operationsbandwidth pool used avail read write read write -- - - - - - - storage 9.78T 2.91T 1.98K137 149M 6.92M storage 9.78T 2.91T761577 94.4M 42.6M storage 9.78T 2.91T462411 57.4M 24.6M storage 9.78T 2.91T492497 61.1M 27.6M storage 9.78T 2.91T632446 78.5M 22.5M storage 9.78T 2.91T554414 68.7M 21.8M storage 9.78T 2.91T459434 57.0M 31.4M storage 9.78T 2.91T398570 49.4M 32.7M storage 9.78T 2.91T338495 41.9M 26.5M storage 9.78T 2.91T358526 44.5M 33.3M storage
Re: zfs send/receive: is this slow?
As soon as I opened this email I knew what it would say. # time zfs send storage/bac...@transfer | mbuffer | zfs receive storage/compressed/bacula-mbuffer in @ 197 MB/s, out @ 205 MB/s, 1749 MB total, buffer 0% full ... Big difference. :) I'm glad it helped. Does anyone know why sending/receiving stuff via loopback is so much slower compared to pipe? --Artem ___ freebsd-stable@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-stable To unsubscribe, send any mail to freebsd-stable-unsubscr...@freebsd.org
Re: zfs send/receive: is this slow?
On Fri, Oct 1, 2010 at 8:43 PM, Artem Belevich fbsdl...@src.cx wrote: As soon as I opened this email I knew what it would say. # time zfs send storage/bac...@transfer | mbuffer | zfs receive storage/compressed/bacula-mbuffer in @ 197 MB/s, out @ 205 MB/s, 1749 MB total, buffer 0% full ... Big difference. :) I'm glad it helped. Does anyone know why sending/receiving stuff via loopback is so much slower compared to pipe? This may shed some light on that topic: http://lists.freebsd.org/pipermail/freebsd-current/2010-September/019877.html ___ freebsd-stable@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-stable To unsubscribe, send any mail to freebsd-stable-unsubscr...@freebsd.org
Re: zfs send/receive: is this slow?
On 02/10/2010, at 11:43 AM, Artem Belevich wrote: As soon as I opened this email I knew what it would say. # time zfs send storage/bac...@transfer | mbuffer | zfs receive storage/compressed/bacula-mbuffer in @ 197 MB/s, out @ 205 MB/s, 1749 MB total, buffer 0% full .. Big difference. :) I'm glad it helped. Does anyone know why sending/receiving stuff via loopback is so much slower compared to pipe? Up and down the entire network stack, in and out of TCP buffers at both ends... might add some overhead, and other factors in limiting it. Increasing TCP buffers, and disabling delayed acks might help. Nagle might also have to be disabled too. (delayed acks and nagle in combination can interact in odd ways) --Artem ___ freebsd-stable@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-stable To unsubscribe, send any mail to freebsd-stable-unsubscr...@freebsd.org ___ freebsd-stable@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-stable To unsubscribe, send any mail to freebsd-stable-unsubscr...@freebsd.org
zfs send/receive: is this slow?
It's taken about 15 hours to copy 800GB. I'm sure there's some tuning I can do. The system is now running: # zfs send storage/bac...@transfer | zfs receive storage/compressed/bacula All the drives are Hitachi HDS722020ALA330 JKAOA28A ATA-8 SATA 2.x device from systat: 1 usersLoad 0.36 0.58 0.57 Sep 29 13:47 Mem:KBREALVIRTUAL VN PAGER SWAP PAGER Tot Share TotShareFree in out in out Act 420127584 54404411028 204492 count All 9623568736 1074363k18220 pages Proc:Interrupts r p d s w Csw Trp Sys Int Sof Flt141 cow9951 total 42 23k 668 3094 1951 2166 657288 zfod ohci0 ohci ozfod ohci2 ohci 13.6%Sys 0.8%Intr 0.2%User 0.0%Nice 85.5%Idle%ozfod ahc0 irq20 ||||||||||| daefr ahci0 22 === 366 prcfr 2000 cpu0: time 26 dtbuf47129 totfr 3 em0 irq256 Namei Name-cache Dir-cache10 desvn react 892 siis0 257 Callshits %hits % 87983 numvn pdwak 1056 siis1 259 46084608 100 24981 frevn pdpgs 2000 cpu3: time intrn 2000 cpu1: time Disks ada0 ada1 ada2 ada3 ada4 ada5 ada6 1355484 wire 2000 cpu2: time KB/t 35.95 37.00 36.75 41.44 40.05 40.86 41.11 25936 act tps 306 299 301 267 276 271 269 2452756 inact MB/s 10.75 10.82 10.79 10.79 10.80 10.80 10.81 76664 cache %busy27502537272727127828 free 427728 buf $ zpool iostat 10 capacity operationsbandwidth pool used avail read write read write -- - - - - - - storage 7.67T 5.02T358 38 43.1M 1.96M storage 7.67T 5.02T317475 39.4M 30.9M storage 7.67T 5.02T357533 44.3M 34.4M storage 7.67T 5.02T371556 46.0M 35.8M storage 7.67T 5.02T313521 38.9M 28.7M storage 7.67T 5.02T309457 38.4M 30.4M storage 7.67T 5.02T388589 48.2M 37.8M storage 7.67T 5.02T377581 46.8M 36.5M storage 7.67T 5.02T310559 38.4M 30.4M storage 7.67T 5.02T430611 53.4M 41.3M $ zfs get all storage/compressed NAMEPROPERTY VALUE SOURCE storage/compressed type filesystem - storage/compressed creation Tue Sep 28 20:35 2010 - storage/compressed used 856G - storage/compressed available 3.38T - storage/compressed referenced44.8K - storage/compressed compressratio 1.60x - storage/compressed mounted yes- storage/compressed quota none default storage/compressed reservation none default storage/compressed recordsize128K default storage/compressed mountpoint/storage/compresseddefault storage/compressed sharenfs offdefault storage/compressed checksum on default storage/compressed compression on local storage/compressed atime on default storage/compressed devices on default storage/compressed exec on default storage/compressed setuidon default storage/compressed readonly offdefault storage/compressed jailedoffdefault storage/compressed snapdir hidden default storage/compressed aclmode groupmask default storage/compressed aclinheritrestricted default storage/compressed canmount on default storage/compressed shareiscsioffdefault storage/compressed xattr offtemporary storage/compressed copies1 default storage/compressed version 4 - storage/compressed utf8only off- storage/compressed normalization none - storage/compressed casesensitivity sensitive - storage/compressed vscan
Re: zfs send/receive: is this slow?
On Wed, Sep 29, 2010 at 11:04 AM, Dan Langille d...@langille.org wrote: It's taken about 15 hours to copy 800GB. I'm sure there's some tuning I can do. The system is now running: # zfs send storage/bac...@transfer | zfs receive storage/compressed/bacula Try piping zfs data through mbuffer (misc/mbuffer in ports). I've found that it does help a lot to smooth out data flow and increase send/receive throughput even when send/receive happens on the same host. Run it with a buffer large enough to accommodate few seconds worth of write throughput for your target disks. Here's an example: http://blogs.everycity.co.uk/alasdair/2010/07/using-mbuffer-to-speed-up-slow-zfs-send-zfs-receive/ --Artem ___ freebsd-stable@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-stable To unsubscribe, send any mail to freebsd-stable-unsubscr...@freebsd.org
Re: zfs send/receive: is this slow?
On 9/29/2010 3:57 PM, Artem Belevich wrote: On Wed, Sep 29, 2010 at 11:04 AM, Dan Langilled...@langille.org wrote: It's taken about 15 hours to copy 800GB. I'm sure there's some tuning I can do. The system is now running: # zfs send storage/bac...@transfer | zfs receive storage/compressed/bacula Try piping zfs data through mbuffer (misc/mbuffer in ports). I've found that it does help a lot to smooth out data flow and increase send/receive throughput even when send/receive happens on the same host. Run it with a buffer large enough to accommodate few seconds worth of write throughput for your target disks. Thanks. I just installed it. I'll use it next time. I don't want to interrupt this one. I'd like to see how long it takes. Then compare. Here's an example: http://blogs.everycity.co.uk/alasdair/2010/07/using-mbuffer-to-speed-up-slow-zfs-send-zfs-receive/ That looks really good. Thank you. -- Dan Langille - http://langille.org/ ___ freebsd-stable@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-stable To unsubscribe, send any mail to freebsd-stable-unsubscr...@freebsd.org
