Actually, I personally use the following "2 part" copy technique to copy files to a cluster of boxes:
tar cf - myfile | dsh -f host-list-file -i -c -M tar xCfv /tmp - The first tar packages myfile into a tar file. dsh runs a tar that unpacks the tar (in the above case all boxes listed in host-list-file would have a /tmp/myfile after the command). Tar options that are relevant include C (chdir) and v (verbose, can be given twice) so you see what got copied. dsh options that are relevant: -i copy stdin to all ssh processes, requires -c -c do the ssh calls concurrently. -M prefix the out from the ssh with the hostname. While this is not rsync, it has the benefit of being processed concurrently, and quite flexible. Andreas Am Donnerstag, den 20.03.2008, 19:57 +0200 schrieb Andrey Pankov: > Hi, > > Did you see hadoop-0.16.0/src/contrib/ec2/bin/start-hadoop script? It > already contains such part: > > echo "Copying private key to slaves" > for slave in `cat slaves`; do > scp $SSH_OPTS $PRIVATE_KEY_PATH "[EMAIL PROTECTED]:/root/.ssh/id_rsa" > ssh $SSH_OPTS "[EMAIL PROTECTED]" "chmod 600 /root/.ssh/id_rsa" > sleep 1 > done > > Anyway, did you tried hadoop-ec2 script? It works well for task you > described. > > > Prasan Ary wrote: > > Hi All, > > I have been trying to configure Hadoop on EC2 for large number of > > clusters ( 100 plus). It seems that I have to copy EC2 private key to all > > the machines in the cluster so that they can have SSH connections. > > For now it seems I have to run a script to copy the key file to each of > > the EC2 instances. I wanted to know if there is a better way to accomplish > > this. > > > > Thanks, > > PA > > > > > > --------------------------------- > > Never miss a thing. Make Yahoo your homepage. > > --- > Andrey Pankov
signature.asc
Description: Dies ist ein digital signierter Nachrichtenteil
