On 24/09/10 19:02, Paul Eggert wrote: > OK, OK, so I shouldn't waste time reading The Register, but I do, > so I can't resist sharing a pointer to this story: > > Pott T. How do you copy 60m files? The Register (2010-09-24) > <http://www.theregister.co.uk/2010/09/24/sysadmin_file_tools/> > > Basically, Pott's problem was that he had to copy 60 million > files from one Windows file server to another. He tried all > sorts of Windowish tools, from Microsoft and from other suppliers. > > Then the light bulb hit: he booted CentOS on a Linux virtual > machine, mounted the source and destinations as SMB shares, > and used coreutils' "cp". Voilà. His conclusion: > > "So the best way to move 60 million files from one Windows server > to another turns out to be: use Linux."
Cool, though he was stretching a bit to say cp was better than richcopy given the latency of cp doing a read then write over the network. For 60m files, ouch! This would have been better, with 2 processes reading and writing asynchronously, with a little buffering in between. (cd /server1 && tar -c) | (cd /server2 && tar -xp) One could bump up the buffering by placing `pv` in the pipe between the 2, and get a progress bar also. That reminds me of this post: http://lists.gnu.org/archive/html/bug-coreutils/2005-11/msg00190.html cheers, Pádraig.
