I checked the tree -- there are about 1.5 million files/dirs in it.
Eric Whiting wrote:
>
> I'm mirroring a 25G tree to about a dozen offsite servers. I run it
> with --delete and I use ssh as the transport.
>
> I have a toplevel dir /home/apps. I tried in the past to mirror the
> whole /home/apps with a single rsync, but it takes too much memory --
> there are too many dirs/files in the tree. There are about 50 subdirs
> in the /home/apps dir. The rsync process on some of those subdirs
> grows to almost 100M -- If I did all the subdirs it runs my
> destination boxes out of memory. (source box is 2G, destinations are
> 128M) I went to an approach that does a single dir at a time:
>
> /bin/bash version (actually I'm using perl now instead)
> --------------------------------------
> #!/tools/local/bin/bash
>
> for host in $1 $2 $3 $4 $5 $6 $7 $8 $9
> do
> for dir in `ls /home/apps`
> do
> echo Working on $dir
> /tools/local/bin/rsync --delete -e "/usr/local/bin/ssh "
> -v --timeout 600 --compress --archive
> --exclude 'apps/.snapshot/*' /home/apps/$dir ${host}:/home/apps
> done
> done
>
> Problem with this approach is it won't fire a delete on a toplevel dir
> if one is removed from the source tree. It will leave the deleted dir
> on the destination. Anyone know of a solution for this using rysnc --
> and without the rsync -r option on the top? I'm thinking of going to a
> simple "ssh ls perlmagic rm -rf" to clean up the toplevel stuff. I
> wish I could somehow use rsync for this instead.
>
> Anyone found an approach that works better for large trees? Is there a
> way to run rsync that doesn't sink so much memory? I love rsync. Is
> there some way to better deal with these large trees?
>
--
__________________________________________________________________
Eric T. Whiting AMI Semiconductors
(208) 234-6717 2300 Buckskin Road
(208) 234-6659 (fax) Pocatello,ID 83201
[EMAIL PROTECTED]
