Ok sounds good. Is there a command to add additional nodes to the same cluster?
Seeing as that's broken perhaps the easiest work around is to launch the cluster, then add nodes to it to run the job? On Thu, Jan 5, 2012 at 11:25 AM, Andrei Savu <[email protected]> wrote: > David I will give it a try later today. > > -- Andrei > > On Wed, Jan 4, 2012 at 10:50 PM, David Dundas <[email protected]> wrote: > >> Actually it seems that this command returns the same error >> >> whirr.instance-templates=1 hadoop-namenode+hadoop-jobtracker, 2 >> hadoop-datanode+hadoop-tasktracker >> >> but this one doesn't >> >> whirr.instance-templates=1 >> hadoop-namenode+hadoop-jobtracker+hadoop-datanode+hadoop-tasktracker >> >> It looks like it's breaking when there are multiple groupings of >> instances. >> >> >> >> On Wed, Jan 4, 2012 at 3:41 PM, Andrei Savu <[email protected]>wrote: >> >>> Let me know if a similar recipe for Hadoop works with Whirr 0.7.0 - it >>> seems like the trunk is a bit broken now. >>> >>> -- Andrei Savu >>> >>> >>> On Wed, Jan 4, 2012 at 10:30 PM, David Dundas <[email protected]>wrote: >>> >>>> Ok that recipe works. It also looks like something in my original >>>> whir.instance-templates >>>> >>>> May have been causing an error as well. >>>> >>>> Thanks! >>>> >>>> On Wed, Jan 4, 2012 at 10:48 AM, Andrei Savu <[email protected]>wrote: >>>> >>>>> I have done some testing and I have been able to start a fully >>>>> functional ZooKeeper cluster from a VM also running inside the Amazon >>>>> Cloud >>>>> using a recipe like this: >>>>> >>>>> whirr.cluster-name=zookeeper-asavu-from-ec2 >>>>> whirr.instance-templates=3 zookeeper >>>>> whirr.cluster-user=asavu >>>>> whirr.provider=aws-ec2 >>>>> whirr.identity=${env:AWS_ACCESS_KEY_ID} >>>>> whirr.credential=${env:AWS_SECRET_ACCESS_KEY} >>>>> >>>>> Hadoop fails with the following exception but this is unrelated to >>>>> this discussion: >>>>> >>>>> Authorizing firewall ingress to [us-east-1/i-e2187780] on ports [8020, >>>>> 8021] for [184.72.197.142/32] >>>>> Unable to start the cluster. Terminating all nodes. >>>>> java.lang.NullPointerException >>>>> at >>>>> org.apache.whirr.service.hadoop.HadoopConfigurationBuilder.buildMapReduceConfiguration(HadoopConfigurationBuilder.java:113) >>>>> at >>>>> org.apache.whirr.service.hadoop.HadoopConfigurationBuilder.buildMapReduce(HadoopConfigurationBuilder.java:71) >>>>> at >>>>> org.apache.whirr.service.hadoop.HadoopClusterActionHandler.createHadoopConfigFiles(HadoopClusterActionHandler.java:102) >>>>> at >>>>> org.apache.whirr.service.hadoop.HadoopClusterActionHandler.beforeConfigure(HadoopClusterActionHandler.java:88) >>>>> at >>>>> org.apache.whirr.service.ClusterActionHandlerSupport.beforeAction(ClusterActionHandlerSupport.java:53) >>>>> at >>>>> org.apache.whirr.actions.ScriptBasedClusterAction.execute(ScriptBasedClusterAction.java:100) >>>>> at >>>>> org.apache.whirr.ClusterController.configureServices(ClusterController.java:148) >>>>> at >>>>> org.apache.whirr.ClusterController.launchCluster(ClusterController.java:110) >>>>> at >>>>> org.apache.whirr.cli.command.LaunchClusterCommand.run(LaunchClusterCommand.java:63) >>>>> at org.apache.whirr.cli.Main.run(Main.java:64) >>>>> at org.apache.whirr.cli.Main.main(Main.java:97) >>>>> >>>>> I am now running the integration tests on my machine to make sure that >>>>> everything still works fine. >>>>> >>>>> -- Andrei Savu / andreisavu.ro >>>>> >>>>> On Wed, Jan 4, 2012 at 12:23 PM, Andrei Savu <[email protected]>wrote: >>>>> >>>>>> I will give it a try now and get back to you with more feedback. >>>>>> >>>>>> >>>>>> On Wed, Jan 4, 2012 at 2:22 AM, David Dundas <[email protected]>wrote: >>>>>> >>>>>>> Also just added whir as the cluster-user, and it looks like I'm >>>>>>> still getting the same errors. >>>>>>> >>>>>>> >>>>>>> On Tue, Jan 3, 2012 at 7:21 PM, David Dundas <[email protected]>wrote: >>>>>>> >>>>>>>> Oh I understand now. I definitely misread your post. Ok then what >>>>>>>> is the difference between >>>>>>>> whirr.cluster-user and whirr.login-user? >>>>>>>> >>>>>>>> Do I need to create ssh key for the cluster user as well? >>>>>>>> >>>>>>>> Should I run the command as the cluster user or the command user? >>>>>>>> >>>>>>>> Can the login-user and cluster-user be the same? >>>>>>>> >>>>>>>> Thanks! >>>>>>>> >>>>>>>> On Tue, Jan 3, 2012 at 7:04 PM, Andrei Savu >>>>>>>> <[email protected]>wrote: >>>>>>>> >>>>>>>>> >>>>>>>>> On Wed, Jan 4, 2012 at 1:57 AM, David Dundas >>>>>>>>> <[email protected]>wrote: >>>>>>>>> >>>>>>>>>> * have you added something like whirr.cluster-user=whirr to >>>>>>>>>> hadoop.properties? >>>>>>>>>> >>> No I havent - should I be? >>>>>>>>>> >>>>>>>>> >>>>>>>>> Yes!! Whirr is trying to create the ec2-user on the remote machine >>>>>>>>> and it fails because that user already exists. >>>>>>>>> >>>>>>>>> I have created the following two related JIRA issues: >>>>>>>>> * https://issues.apache.org/jira/browse/WHIRR-463 >>>>>>>>> * https://issues.apache.org/jira/browse/WHIRR-464 >>>>>>>>> >>>>>>>>> As I said already in a comment my advice is to add: >>>>>>>>> >>>>>>>>> whirr.cluster-user=whirr (to hadoop.properties) >>>>>>>>> >>>>>>>>> and you should be able to login using something like: >>>>>>>>> >>>>>>>>> ssh -i ~/.ssh/id_rsa_whirr whirr@remote-host-ip >>>>>>>>> >>>>>>>>> -- Andrei >>>>>>>>> >>>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> >>>>>> >>>>> >>>> >>>> >>> >> >> >
