[
https://issues.apache.org/jira/browse/HADOOP-2410?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Tom White updated HADOOP-2410:
------------------------------
Resolution: Fixed
Fix Version/s: 0.17.0
Assignee: Chris K Wensel
Release Note: The command "hadoop-ec2 run" has been replaced by
"hadoop-ec2 launch-cluster <group> <number of instances>", and "hadoop-ec2
start-hadoop" has been removed since Hadoop is started on instance start up.
See http://wiki.apache.org/hadoop/AmazonEC2 for details.
Hadoop Flags: [Incompatible change, Reviewed]
Status: Resolved (was: Patch Available)
I've just committed this. Thanks Chris!
I tried out the new scripts and they worked fine. I changed the version of
Hadoop in the env file to be 0.17.0 so that it picks up the new AMI when it is
created (after 0.17.0 is released). I also changed the version of Java to
1.6.0_05.
Chris, could you update the documentation on the wiki page with the changes
please? It would be worth keeping the instructions for the older scripts around
on the same page.
> Make EC2 cluster nodes more independent of each other
> -----------------------------------------------------
>
> Key: HADOOP-2410
> URL: https://issues.apache.org/jira/browse/HADOOP-2410
> Project: Hadoop Core
> Issue Type: Improvement
> Components: contrib/ec2
> Affects Versions: 0.16.1
> Reporter: Tom White
> Assignee: Chris K Wensel
> Fix For: 0.17.0
>
> Attachments: concurrent-clusters-2.patch,
> concurrent-clusters-3.patch, concurrent-clusters.patch, ec2.tgz
>
>
> The cluster start up scripts currently wait for each node to start up before
> appointing a master (to run the namenode and jobtracker on), and copying
> private keys to all the nodes, and writing the private IP address of the
> master to the hadoop-site.xml file (which is then copied to the slaves via
> rsync). Only once this is all done is hadoop started on the cluster (from the
> master). This can fail if any of the nodes fails to come up, which can happen
> as EC2 doesn't guarantee that you get a cluster of the size you ask for (I've
> seen this happen).
> The process would be more robust if each node was told the address of the
> master as user metadata and then started its own daemons. This is complicated
> by the fact that the public DNS alias of the master resolves to a public IP
> address so cannot be used by EC2 nodes (see
> http://docs.amazonwebservices.com/AWSEC2/2007-08-29/DeveloperGuide/instance-addressing.html).
> Instead we need to use a trick
> (http://developer.amazonwebservices.com/connect/message.jspa?messageID=71126#71126)
> to find the private IP, and what's more we need to attempt to resolve the
> private IP in a loop until it is available since the DNS will only be set up
> after the master has started.
> This change will also mean the private key doesn't need to be copied to each
> node, which can be slow and has dubious security. Configuration can be
> handled using the mechanism described in HADOOP-2409.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
