On Monday 20 June 2011, Paul Hartman wrote: > On Mon, Jun 20, 2011 at 10:25 AM, Mark Knecht <markkne...@gmail.com> wrote: > > Hi, > > Is split an appropriate program to use to break a single 10GB > > file into 100 100MB files to transfer over the net using rsync, > > and then use cat to reassemble? > > I think it should work just fine. I've split huge files into huge > chunks and never had any issues. > > > Is there some better way to do this? > > I wonder if splitting is even necessary; rsync will analyze the file > and only transmit the differences, right?. So I'd think that even if > the transfer fails, a retry would pick up where it left off (assuming > rsync keeps the failed copy). > > Also check out net-misc/unison. It seems to be designed for just this > sort of thing.
Unison is wonderful for more complex tasks but is very inefficient with large files. As a matter of fact it uses rsync algorithm in order to get good performance, still isn't the best choice in this scenario, see: http://www.cis.upenn.edu/~bcpierce/unison/download/releases/stable/unison- manual.html#speeding I keep in sync the user home across two computer, and I use both of them daily. It would be impossible without unison, but the 20GB virtual machine is excluded from the sync. Cheers Francesco -- Linux Version 2.6.39-gentoo-r1, Compiled #1 SMP PREEMPT Thu Jun 9 11:20:57 CEST 2011 Two 1GHz AMD Athlon 64 X2 Processors, 4GB RAM, 4021.84 Bogomips Total aemaeth
