I'm mirroring a 25G tree to about a dozen offsite servers. I run it
with --delete and I use ssh as the transport.
I have a toplevel dir /home/apps. I tried in the past to mirror the
whole /home/apps with a single rsync, but it takes too much memory --
there are too many dirs/files in the tree. There are about 50 subdirs
in the /home/apps dir. The rsync process on some of those subdirs
grows to almost 100M -- If I did all the subdirs it runs my
destination boxes out of memory. (source box is 2G, destinations are
128M) I went to an approach that does a single dir at a time:
/bin/bash version (actually I'm using perl now instead)
--------------------------------------
#!/tools/local/bin/bash
for host in $1 $2 $3 $4 $5 $6 $7 $8 $9
do
for dir in `ls /home/apps`
do
echo Working on $dir
/tools/local/bin/rsync --delete -e "/usr/local/bin/ssh "
-v --timeout 600 --compress --archive
--exclude 'apps/.snapshot/*' /home/apps/$dir ${host}:/home/apps
done
done
Problem with this approach is it won't fire a delete on a toplevel dir
if one is removed from the source tree. It will leave the deleted dir
on the destination. Anyone know of a solution for this using rysnc --
and without the rsync -r option on the top? I'm thinking of going to a
simple "ssh ls perlmagic rm -rf" to clean up the toplevel stuff. I
wish I could somehow use rsync for this instead.
Anyone found an approach that works better for large trees? Is there a
way to run rsync that doesn't sink so much memory? I love rsync. Is
there some way to better deal with these large trees?
Thanks,
eric
--
__________________________________________________________________
Eric T. Whiting AMI Semiconductors
(208) 234-6717 2300 Buckskin Road
(208) 234-6659 (fax) Pocatello,ID 83201
[EMAIL PROTECTED]
