Re: [zfs-discuss] any more efficient way to transfer snapshot between two hosts than ssh tunnel?
> >Even with infinite wire speed, you're bound by the ability of the source server >to generate the snapshot stream and the ability of the destination server to >write the snapshots to the media. > >Our little servers in-house using ZFS don't read/write that fast when pulling >snapshot contents off the disks, since they're essentially random access on a >server that's been creating/deleting snapshots for a long time. > >--eric > That is true. This discussion is assumed under ideal condition. We want to minimize the overhead in transportation layer only. Thanks. Fred ___ zfs-discuss mailing list zfs-discuss@opensolaris.org http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
Re: [zfs-discuss] any more efficient way to transfer snapshot between two hosts than ssh tunnel?
On Fri, Dec 14 at 9:29, Fred Liu wrote: We have found mbuffer to be the fastest solution. Our rates for large transfers on 10GbE are: 280MB/smbuffer 220MB/srsh 180MB/sHPN-ssh unencrypted 60MB/s standard ssh The tradeoff mbuffer is a little more complicated to script; rsh is, well, you know; and hpn-ssh requires rebuilding ssh and (probably) maintaining a second copy of it. -- Trey Palmer In 10GbE env, even 280MB/s is not a so decent result. Maybe the alternative could be a two-step way. Putting snapshots via NFS/iSCSI and receiving them locally. But that is not perfect. Even with infinite wire speed, you're bound by the ability of the source server to generate the snapshot stream and the ability of the destination server to write the snapshots to the media. Our little servers in-house using ZFS don't read/write that fast when pulling snapshot contents off the disks, since they're essentially random access on a server that's been creating/deleting snapshots for a long time. --eric -- Eric D. Mudama edmud...@bounceswoosh.org ___ zfs-discuss mailing list zfs-discuss@opensolaris.org http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
Re: [zfs-discuss] any more efficient way to transfer snapshot between two hosts than ssh tunnel?
In my own experiments with my own equivalent of mbuffer, it's well worth giving the receiving side a buffer which is sized to hold the amount of data in a transaction commit, which allows ZFS to be banging out one tx group to disk, whilst the network is bringing the next one across for it. This will be roughly the link speed in bytes/second x 5, plus a bit more for good measure, say 250-300Mbytes for a gigabit link. It seems to be most important when the disks and the network link have similar max theoretical bandwidths (100Mbytes/sec is what you might expect from both gigabit ethernet and reasonable disks), and it becomes less important as the difference in max performance between them increases. Without the buffer, you tend to see the network run flat out for 5 seconds, and then the receiving disks run flat out for 5 seconds, alternating back and forth, whereas with the buffer, both continue streaming at full gigabit speed without a break. I have not seen any benefit of buffering on the sending side, although I'd still be inclined to include a small one. YMMV... Palmer, Trey wrote: We have found mbuffer to be the fastest solution. Our rates for large transfers on 10GbE are: 280MB/smbuffer 220MB/srsh 180MB/sHPN-ssh unencrypted 60MB/s standard ssh The tradeoff mbuffer is a little more complicated to script; rsh is, well, you know; and hpn-ssh requires rebuilding ssh and (probably) maintaining a second copy of it. -- Andrew Gabriel ___ zfs-discuss mailing list zfs-discuss@opensolaris.org http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
Re: [zfs-discuss] any more efficient way to transfer snapshot between two hosts than ssh tunnel?
> > We have found mbuffer to be the fastest solution. Our rates for large > transfers on 10GbE are: > > 280MB/smbuffer > 220MB/srsh > 180MB/sHPN-ssh unencrypted > 60MB/s standard ssh > > The tradeoff mbuffer is a little more complicated to script; rsh is, > well, you know; and hpn-ssh requires rebuilding ssh and (probably) > maintaining a second copy of it. > > -- Trey Palmer > In 10GbE env, even 280MB/s is not a so decent result. Maybe the alternative could be a two-step way. Putting snapshots via NFS/iSCSI and receiving them locally. But that is not perfect. ___ zfs-discuss mailing list zfs-discuss@opensolaris.org http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
Re: [zfs-discuss] any more efficient way to transfer snapshot between two hosts than ssh tunnel?
Post in the list. > -Original Message- > From: Fred Liu > Sent: 星期五, 十二月 14, 2012 23:41 > To: 'real-men-dont-cl...@gmx.net' > Subject: RE: [zfs-discuss] any more efficient way to transfer snapshot > between two hosts than ssh tunnel? > > > > > > > Hi Fred, > > > > I played with zfs send/reveive some time ago. One important thing I > > learned was that netcat is not the first choice to use. > > There is a tool called mbuffer out there. mbuffer works similar to > > netcat but allows a specific buffer size and block size. > > From various resources I found out that the best buffer and block > sizes > > for zfs send/receive seem to be 1GB for the buffer with a block size > of > > 131073. > > Replacing netcat by mubuffer dramatically increases the throughput. > > > > > > The resulting commands are like: > > > > ssh -f $REMOTESRV "/opt/csw/bin/mbuffer -q -I $PORT -m 1G -s 131072 | > > zfs receive -vFd $REMOTEPOOL" > > > > zfs send $CURRENTLOCAL | /opt/csw/bin/mbuffer -q -O $REMOTESRV:$PORT > -m > > 1G -s 131072 > /dev/null > > > > > > cu > > > > Carsten, > > Thank you so much for the sharing and I will try it. > > Fred ___ zfs-discuss mailing list zfs-discuss@opensolaris.org http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
Re: [zfs-discuss] any more efficient way to transfer snapshot between two hosts than ssh tunnel?
> > I've heard you could, but I've never done it. Sorry I'm not much help, > except as a cheer leader. You can do it! I think you can! Don't give > up! heheheheh > Please post back whatever you find, or if you have to figure it out for > yourself, then blog about it and post that. Aha! Gotcha! I will give it a try. ___ zfs-discuss mailing list zfs-discuss@opensolaris.org http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
Re: [zfs-discuss] any more efficient way to transfer snapshot between two hosts than ssh tunnel?
We have found mbuffer to be the fastest solution. Our rates for large transfers on 10GbE are: 280MB/smbuffer 220MB/srsh 180MB/sHPN-ssh unencrypted 60MB/s standard ssh The tradeoff mbuffer is a little more complicated to script; rsh is, well, you know; and hpn-ssh requires rebuilding ssh and (probably) maintaining a second copy of it. -- Trey Palmer From: zfs-discuss-boun...@opensolaris.org [zfs-discuss-boun...@opensolaris.org] on behalf of Fred Liu [fred_...@issi.com] Sent: Thursday, December 13, 2012 11:23 PM To: Freddie Cash Cc: zfs-discuss Subject: Re: [zfs-discuss] any more efficient way to transfer snapshot between two hosts than ssh tunnel? >Add the HPN patches to OpenSSH and enable the NONE cipher. We can saturate a >gigabits link (980 mbps) between two FreeBSD hosts using that. >Without it, we were only able to hit ~480 mbps on a good day. >If you want 0 overhead, there's always netcat. :) 980mbps is awesome! I am thinking running two ssh services -- one normal and one with HPN patches only for backup job. But now sure they can work before I try them. I will also try netcat. Many thanks. Fred ___ zfs-discuss mailing list zfs-discuss@opensolaris.org http://mail.opensolaris.org/mailman/listinfo/zfs-discuss ___ zfs-discuss mailing list zfs-discuss@opensolaris.org http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
Re: [zfs-discuss] any more efficient way to transfer snapshot between two hosts than ssh tunnel?
> From: zfs-discuss-boun...@opensolaris.org [mailto:zfs-discuss- > boun...@opensolaris.org] On Behalf Of Fred Liu > > BTW, anyone played NDMP in solaris? Or is it feasible to transfer snapshot via > NDMP protocol? I've heard you could, but I've never done it. Sorry I'm not much help, except as a cheer leader. You can do it! I think you can! Don't give up! heheheheh Please post back whatever you find, or if you have to figure it out for yourself, then blog about it and post that. ___ zfs-discuss mailing list zfs-discuss@opensolaris.org http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
Re: [zfs-discuss] any more efficient way to transfer snapshot between two hosts than ssh tunnel?
>Add the HPN patches to OpenSSH and enable the NONE cipher. We can saturate a >gigabits link (980 mbps) between two FreeBSD hosts using that. >Without it, we were only able to hit ~480 mbps on a good day. >If you want 0 overhead, there's always netcat. :) 980mbps is awesome! I am thinking running two ssh services -- one normal and one with HPN patches only for backup job. But now sure they can work before I try them. I will also try netcat. Many thanks. Fred ___ zfs-discuss mailing list zfs-discuss@opensolaris.org http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
Re: [zfs-discuss] any more efficient way to transfer snapshot between two hosts than ssh tunnel?
On Dec 13, 2012 8:02 PM, "Fred Liu" wrote: > > Assuming in a secure and trusted env, we want to get the maximum transfer speed without the overhead from ssh. Add the HPN patches to OpenSSH and enable the NONE cipher. We can saturate a gigabits link (980 mbps) between two FreeBSD hosts using that. Without it, we were only able to hit ~480 mbps on a good day. If you want 0 overhead, there's always netcat. :) ___ zfs-discuss mailing list zfs-discuss@opensolaris.org http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
Re: [zfs-discuss] any more efficient way to transfer snapshot between two hosts than ssh tunnel?
Adrian, That is cool! Thank you so much! BTW, anyone played NDMP in solaris? Or is it feasible to transfer snapshot via NDMP protocol? Before the acquisition, SUN advocated the NDMP backup feature in the openstorage/fishwork. I am sorry if it is the wrong place to ask this question. Thanks. Fred From: Adrian Smith [mailto:adrian.sm...@rmit.edu.au] Sent: 星期五, 十二月 14, 2012 12:08 To: Fred Liu Cc: zfs-discuss@opensolaris.org Subject: Re: [zfs-discuss] any more efficient way to transfer snapshot between two hosts than ssh tunnel? Hi Fred, Try mbuffer (http://www.maier-komor.de/mbuffer.html) On 14 December 2012 15:01, Fred Liu mailto:fred_...@issi.com>> wrote: Assuming in a secure and trusted env, we want to get the maximum transfer speed without the overhead from ssh. Thanks. Fred ___ zfs-discuss mailing list zfs-discuss@opensolaris.org<mailto:zfs-discuss@opensolaris.org> http://mail.opensolaris.org/mailman/listinfo/zfs-discuss -- Adrian Smith (ISUnix), Ext: 55070 ___ zfs-discuss mailing list zfs-discuss@opensolaris.org http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
Re: [zfs-discuss] any more efficient way to transfer snapshot between two hosts than ssh tunnel?
Hi Fred, Try mbuffer (http://www.maier-komor.de/mbuffer.html) On 14 December 2012 15:01, Fred Liu wrote: > Assuming in a secure and trusted env, we want to get the maximum > transfer speed without the overhead from ssh. > > ** ** > > Thanks. > > ** ** > > Fred > > ___ > zfs-discuss mailing list > zfs-discuss@opensolaris.org > http://mail.opensolaris.org/mailman/listinfo/zfs-discuss > > -- Adrian Smith (ISUnix), Ext: 55070 ___ zfs-discuss mailing list zfs-discuss@opensolaris.org http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
