When It comes with million of files to copy from one place to another,there are two considerations: 1,performance 2,repeatable(fault tolerance),when it fails in-middle,it can continue with uncopied files.`cp -R` will not be ok for this because it begains from the very start, and `rsync` is good at this.
2009-09-29 xufengnju 发件人： Steve Bertrand 发送时间： 2009-09-29 14:46:50 收件人： Uri Guttman 抄送： xufengnju; beginners 主题： Re: how to do a `cp` on millions of files Uri Guttman wrote: >>>>>> "SB" == Steve Bertrand <[email protected]> writes: > > SB> my $results > SB> = timethese( 10, { > SB> 'rsync' => sub { `rsync -arc $directory $backup` }, > SB> 'perl-cp' => sub { dircopy( $directory, $backup ) }, > > to make this a proper benchmark, you need to remove all the copied files > beforehand. rsync isn't supposed to copy files unless they have changed. > basically rsync isn't doing any copies after the first time. > > this would be better to compare cp -r and dircopy as they are similar in > function. then you wouldn't need to remove the files as you would always > overwrite them with each copy pass. Heh. I knew I shouldn't have written the file creation function into the program. I figured that if the rsync test ran first, results would be ok. However, I see how the results may have been skewed. ...I think I've got the bench, and the dir struct fixed ( nothing in u* or p* -backup). On a normal day, I'd rather be in bed with my wife about now, but I wanted to give this a crack...I'm certain that I'll regret Perl for this... %ll -rwxr-xr-x 1 steve steve 348 Sep 29 02:13 bench.pl drwxr-xr-x 2 steve steve 1002 Sep 29 00:32 files drwxr-xr-x 2 steve steve 2 Sep 29 02:15 p-backup drwxr-xr-x 2 steve steve 2 Sep 29 02:15 u-backup #!/usr/bin/perl use warnings; use strict; use File::Copy::Recursive qw ( dircopy ); use Benchmark qw( :all ); my $directory = './files'; my $unix_bk = './u-backup'; my $perl_bk = './p-backup'; my $results = timethese( 5, { 'unix-cp' => sub { `cp -r $directory $unix_bk` }, 'perl-cp' => sub { dircopy( $directory, $perl_bk ) }, } ); cmpthese $results; __END__ Benchmark: timing 5 iterations of perl-cp, unix-cp... perl-cp: 74 wallclock secs ( 1.60 usr + 11.84 sys = 13.45 CPU) @ 0.37/s (n=5) unix-cp: 49 wallclock secs ( 0.00 usr 0.00 sys + 0.04 cusr 8.91 csys = 8.95 CPU) s/iter perl-cp unix-cp perl-cp 2.69 -- -100% unix-cp 2.00e-16 1344531250000000256% -- Steve ps. let me know if this is the test you wanted to see.
