Re: Copying files without scp
A. Wright wrote: > On Tue, 7 Apr 2009, Steve Bertrand wrote: > >> Can someone recommend a *known good* production quality copy mechanism >> that will act like scp, but without the overhead? rsh? nc? > > If you are happy with rsh authentication, then have you looked at > plain old rcp? I reviewed carefully all feedback I received, and since then, I loosely benchmarked my options. After all was said and done, using SSH, I found: - across the 100Mbps infrastructure, I could copy at 89Mbps - across the Gi infrastructure, I could copy at ~770Mbps My concern (I found) was coming from my Windows workstation. I was using a Windows binary version of SCP that is clearly lacking somewhere in the stack. FBSD to FBSD produced the above results. The 100Mb infrastructure hops one router and two switches, and the gig makes a hop across one edge router, a core router, and three switches. When I can get 75-90% line rate encrypted, I'll stay with that. I do appreciate all the feedback, as always ;) Steve ___ freebsd-questions@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-questions To unsubscribe, send any mail to "freebsd-questions-unsubscr...@freebsd.org"
Re: Copying files without scp
To copy data from one server, I normally (always) use scp. man rcp ___ freebsd-questions@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-questions To unsubscribe, send any mail to "freebsd-questions-unsubscr...@freebsd.org"
Re: Copying files without scp
Steve Bertrand wrote: > To copy data from one server, I normally (always) use scp. > > I'm looking for a method to perform this copy task without the overhead > of encryption for infrequent, high-volume transfers (hundreds to > thousands of GB). > > The data will be transferred server-to-server within a private datacentre. There are quite a lot of ways to do that. You could NFS-export then files and then use a tool to copy them on the other box locally (tar, cpio, cpdup, whatever). You could run an FTP server and then use one of the various FTP mirror tools to copy the files (e.g. ports/ftp/omi). You could use plain old rcp. You could apply this (trivial) patch that adds support for cipher "none" in ssh and scp: http://www.secnetix.de/olli/FreeBSD/patches/openssh-cipher-none The advantage of using scp (with "-c none") is that you can use all of the ssh features, such as key authentication, server aliases (via ~/.ssh/config) etc. You can also use other file copy tools (such as cpdup) that can be tunneled through ssh. Best regards Oliver -- Oliver Fromme, secnetix GmbH & Co. KG, Marktplatz 29, 85567 Grafing b. M. Handelsregister: Registergericht Muenchen, HRA 74606, Geschäftsfuehrung: secnetix Verwaltungsgesellsch. mbH, Handelsregister: Registergericht Mün- chen, HRB 125758, Geschäftsführer: Maik Bachmann, Olaf Erb, Ralf Gebhart FreeBSD-Dienstleistungen, -Produkte und mehr: http://www.secnetix.de/bsd "IRIX is about as stable as a one-legged drunk with hypothermia in a four-hundred mile per hour wind, balancing on a banana peel on a greased cookie sheet -- when someone throws him an elephant with bad breath and a worse temper." -- Ralf Hildebrandt ___ freebsd-questions@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-questions To unsubscribe, send any mail to "freebsd-questions-unsubscr...@freebsd.org"
Re: Copying files without scp
Steve Bertrand writes: > To copy data from one server, I normally (always) use scp. > > I'm looking for a method to perform this copy task without the overhead > of encryption for infrequent, high-volume transfers (hundreds to > thousands of GB). > > The data will be transferred server-to-server within a private datacentre. > > Can someone recommend a *known good* production quality copy mechanism > that will act like scp, but without the overhead? rsh? nc? > > I recall a thread not too long ago regarding this, but I'd like to have > a simple working example if possible, without getting into detail why > one shouldn't transfer data unencrypted. I haven't hit a case in years where the encryption overhead was actually measurable as a significant issue. Still, anything you can do over ssh can be done just as well over rsh. There's always rcp if you want the same syntax as scp, but if the data consists of a lot of different files, using tar on both ends of a pipe will probably be much faster. -- Lowell Gilbert, embedded/networking software engineer, Boston area http://be-well.ilk.org/~lowell/ ___ freebsd-questions@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-questions To unsubscribe, send any mail to "freebsd-questions-unsubscr...@freebsd.org"
Re: Copying files without scp
On Tue, 7 Apr 2009, Steve Bertrand wrote: Can someone recommend a *known good* production quality copy mechanism that will act like scp, but without the overhead? rsh? nc? If you are happy with rsh authentication, then have you looked at plain old rcp? A. ___ freebsd-questions@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-questions To unsubscribe, send any mail to "freebsd-questions-unsubscr...@freebsd.org"
Re: Copying files without scp
On Wednesday 08 April 2009 01:31:18 Steve Bertrand wrote: > Doug Hardie wrote: > > On Apr 7, 2009, at 16:13, Steve Bertrand wrote: > >> Hi all, > >> > >> To copy data from one server, I normally (always) use scp. > >> > >> I'm looking for a method to perform this copy task without the overhead > >> of encryption for infrequent, high-volume transfers (hundreds to > >> thousands of GB). > >> > >> The data will be transferred server-to-server within a private > >> datacentre. > >> > >> Can someone recommend a *known good* production quality copy mechanism > >> that will act like scp, but without the overhead? rsh? nc? > > > > In that environment you can use ftp just fine. Make sure to restrict it > > to the local IP addresses. > > Thanks, but I've never found a way to copy complete directories with FTP. > > I'll need to copy entire multi-nested directory structures. > > Do you have an example how to do this via FTP? (CLI-only). ftp/ncftp3: ncftpget -R ftp://servername/path/to/start/ -- Mel ___ freebsd-questions@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-questions To unsubscribe, send any mail to "freebsd-questions-unsubscr...@freebsd.org"
Re: Copying files without scp
Steve Bertrand wrote: Hi all, To copy data from one server, I normally (always) use scp. I'm looking for a method to perform this copy task without the overhead of encryption for infrequent, high-volume transfers (hundreds to thousands of GB). The data will be transferred server-to-server within a private datacentre. Can someone recommend a *known good* production quality copy mechanism that will act like scp, but without the overhead? rsh? nc? On the listening end: cd / ; nc -l 12345 | tar xpvf - On the sending end: cd / ; tar cf - /path/file | nc 12345 Replace 'x' by 't' on the listening end to verify that it's going to do what you would want/expect. -- Frederique ___ freebsd-questions@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-questions To unsubscribe, send any mail to "freebsd-questions-unsubscr...@freebsd.org"
Re: Copying files without scp
Hi, > Can someone recommend a *known good* production quality copy mechanism > that will act like scp, but without the overhead? rsh? nc? I sometime use tar+rsh. Tar because I want to be sure to preserve all ownership and modes of the files and directories. Bests, olivier ___ freebsd-questions@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-questions To unsubscribe, send any mail to "freebsd-questions-unsubscr...@freebsd.org"
Re: Copying files without scp
Chuck Swiger wrote: > On Apr 7, 2009, at 4:13 PM, Steve Bertrand wrote: >> I'm looking for a method to perform this copy task without the overhead >> of encryption for infrequent, high-volume transfers (hundreds to >> thousands of GB). >> >> The data will be transferred server-to-server within a private >> datacentre. >> >> Can someone recommend a *known good* production quality copy mechanism >> that will act like scp, but without the overhead? rsh? nc? > > Install /usr/ports/security/openssh-portable, and set the "Enable > HPN-SSH patch" option. You should then be able to use "scp -c none" > option, which is documented more fully here: > > http://www.psc.edu/networking/projects/hpn-ssh/none.php > > You could also use rsync + rsyncd Thanks Chuck, I think I'll just go the rsync route. I'm very familiar with it. I don't particularly want to install it on the boxes I'm concerned with momentarily, but I know how it works. Your other option seems intriguing, but I'd rather not install more software on these boxes if possible. I was hoping for a magical, don't have to install anything-type solution :) Perhaps I left out an important piece... even though I'm copying directory structures, in many cases the bulk of the data will be contained within massively large individual files. (Hence why rsync wasn't my original choice). Steve ___ freebsd-questions@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-questions To unsubscribe, send any mail to "freebsd-questions-unsubscr...@freebsd.org"
Re: Copying files without scp
On Apr 7, 2009, at 4:13 PM, Steve Bertrand wrote: I'm looking for a method to perform this copy task without the overhead of encryption for infrequent, high-volume transfers (hundreds to thousands of GB). The data will be transferred server-to-server within a private datacentre. Can someone recommend a *known good* production quality copy mechanism that will act like scp, but without the overhead? rsh? nc? Install /usr/ports/security/openssh-portable, and set the "Enable HPN- SSH patch" option. You should then be able to use "scp -c none" option, which is documented more fully here: http://www.psc.edu/networking/projects/hpn-ssh/none.php You could also use rsync + rsyncd Regards, -- -Chuck ___ freebsd-questions@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-questions To unsubscribe, send any mail to "freebsd-questions-unsubscr...@freebsd.org"
Re: Copying files without scp
Doug Hardie wrote: > > On Apr 7, 2009, at 16:13, Steve Bertrand wrote: > >> Hi all, >> >> To copy data from one server, I normally (always) use scp. >> >> I'm looking for a method to perform this copy task without the overhead >> of encryption for infrequent, high-volume transfers (hundreds to >> thousands of GB). >> >> The data will be transferred server-to-server within a private >> datacentre. >> >> Can someone recommend a *known good* production quality copy mechanism >> that will act like scp, but without the overhead? rsh? nc? > > In that environment you can use ftp just fine. Make sure to restrict it > to the local IP addresses. Thanks, but I've never found a way to copy complete directories with FTP. I'll need to copy entire multi-nested directory structures. Do you have an example how to do this via FTP? (CLI-only). Steve ___ freebsd-questions@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-questions To unsubscribe, send any mail to "freebsd-questions-unsubscr...@freebsd.org"
Re: Copying files without scp
On Apr 7, 2009, at 16:13, Steve Bertrand wrote: Hi all, To copy data from one server, I normally (always) use scp. I'm looking for a method to perform this copy task without the overhead of encryption for infrequent, high-volume transfers (hundreds to thousands of GB). The data will be transferred server-to-server within a private datacentre. Can someone recommend a *known good* production quality copy mechanism that will act like scp, but without the overhead? rsh? nc? In that environment you can use ftp just fine. Make sure to restrict it to the local IP addresses. ___ freebsd-questions@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-questions To unsubscribe, send any mail to "freebsd-questions-unsubscr...@freebsd.org"
Re: Copying files without scp
Steve Bertrand wrote: Hi all, To copy data from one server, I normally (always) use scp. I'm looking for a method to perform this copy task without the overhead of encryption for infrequent, high-volume transfers (hundreds to thousands of GB). The data will be transferred server-to-server within a private datacentre. Can someone recommend a *known good* production quality copy mechanism that will act like scp, but without the overhead? rsh? nc? I recall a thread not too long ago regarding this, but I'd like to have a simple working example if possible, without getting into detail why one shouldn't transfer data unencrypted. Steve ___ freebsd-questions@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-questions To unsubscribe, send any mail to "freebsd-questions-unsubscr...@freebsd.org" dump should work, it uses rsh. something like cpio could be piped into rsh obviously. You may want to benchmark it as I've had better transfer rates using scp/ssh but have not done it against rsh method. -- Adam Vandemore Systems Administrator IMED Mobility (605) 498-1610 ___ freebsd-questions@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-questions To unsubscribe, send any mail to "freebsd-questions-unsubscr...@freebsd.org"
