Actually it seems that this command returns the same error whirr.instance-templates=1 hadoop-namenode+hadoop-jobtracker, 2 hadoop-datanode+hadoop-tasktracker
but this one doesn't whirr.instance-templates=1 hadoop-namenode+hadoop-jobtracker+hadoop-datanode+hadoop-tasktracker It looks like it's breaking when there are multiple groupings of instances. On Wed, Jan 4, 2012 at 3:41 PM, Andrei Savu <[email protected]> wrote: > Let me know if a similar recipe for Hadoop works with Whirr 0.7.0 - it > seems like the trunk is a bit broken now. > > -- Andrei Savu > > > On Wed, Jan 4, 2012 at 10:30 PM, David Dundas <[email protected]> wrote: > >> Ok that recipe works. It also looks like something in my original >> whir.instance-templates >> >> May have been causing an error as well. >> >> Thanks! >> >> On Wed, Jan 4, 2012 at 10:48 AM, Andrei Savu <[email protected]>wrote: >> >>> I have done some testing and I have been able to start a fully >>> functional ZooKeeper cluster from a VM also running inside the Amazon Cloud >>> using a recipe like this: >>> >>> whirr.cluster-name=zookeeper-asavu-from-ec2 >>> whirr.instance-templates=3 zookeeper >>> whirr.cluster-user=asavu >>> whirr.provider=aws-ec2 >>> whirr.identity=${env:AWS_ACCESS_KEY_ID} >>> whirr.credential=${env:AWS_SECRET_ACCESS_KEY} >>> >>> Hadoop fails with the following exception but this is unrelated to this >>> discussion: >>> >>> Authorizing firewall ingress to [us-east-1/i-e2187780] on ports [8020, >>> 8021] for [184.72.197.142/32] >>> Unable to start the cluster. Terminating all nodes. >>> java.lang.NullPointerException >>> at >>> org.apache.whirr.service.hadoop.HadoopConfigurationBuilder.buildMapReduceConfiguration(HadoopConfigurationBuilder.java:113) >>> at >>> org.apache.whirr.service.hadoop.HadoopConfigurationBuilder.buildMapReduce(HadoopConfigurationBuilder.java:71) >>> at >>> org.apache.whirr.service.hadoop.HadoopClusterActionHandler.createHadoopConfigFiles(HadoopClusterActionHandler.java:102) >>> at >>> org.apache.whirr.service.hadoop.HadoopClusterActionHandler.beforeConfigure(HadoopClusterActionHandler.java:88) >>> at >>> org.apache.whirr.service.ClusterActionHandlerSupport.beforeAction(ClusterActionHandlerSupport.java:53) >>> at >>> org.apache.whirr.actions.ScriptBasedClusterAction.execute(ScriptBasedClusterAction.java:100) >>> at >>> org.apache.whirr.ClusterController.configureServices(ClusterController.java:148) >>> at >>> org.apache.whirr.ClusterController.launchCluster(ClusterController.java:110) >>> at >>> org.apache.whirr.cli.command.LaunchClusterCommand.run(LaunchClusterCommand.java:63) >>> at org.apache.whirr.cli.Main.run(Main.java:64) >>> at org.apache.whirr.cli.Main.main(Main.java:97) >>> >>> I am now running the integration tests on my machine to make sure that >>> everything still works fine. >>> >>> -- Andrei Savu / andreisavu.ro >>> >>> On Wed, Jan 4, 2012 at 12:23 PM, Andrei Savu <[email protected]>wrote: >>> >>>> I will give it a try now and get back to you with more feedback. >>>> >>>> >>>> On Wed, Jan 4, 2012 at 2:22 AM, David Dundas <[email protected]>wrote: >>>> >>>>> Also just added whir as the cluster-user, and it looks like I'm still >>>>> getting the same errors. >>>>> >>>>> >>>>> On Tue, Jan 3, 2012 at 7:21 PM, David Dundas <[email protected]>wrote: >>>>> >>>>>> Oh I understand now. I definitely misread your post. Ok then what is >>>>>> the difference between >>>>>> whirr.cluster-user and whirr.login-user? >>>>>> >>>>>> Do I need to create ssh key for the cluster user as well? >>>>>> >>>>>> Should I run the command as the cluster user or the command user? >>>>>> >>>>>> Can the login-user and cluster-user be the same? >>>>>> >>>>>> Thanks! >>>>>> >>>>>> On Tue, Jan 3, 2012 at 7:04 PM, Andrei Savu <[email protected]>wrote: >>>>>> >>>>>>> >>>>>>> On Wed, Jan 4, 2012 at 1:57 AM, David Dundas <[email protected]>wrote: >>>>>>> >>>>>>>> * have you added something like whirr.cluster-user=whirr to >>>>>>>> hadoop.properties? >>>>>>>> >>> No I havent - should I be? >>>>>>>> >>>>>>> >>>>>>> Yes!! Whirr is trying to create the ec2-user on the remote machine >>>>>>> and it fails because that user already exists. >>>>>>> >>>>>>> I have created the following two related JIRA issues: >>>>>>> * https://issues.apache.org/jira/browse/WHIRR-463 >>>>>>> * https://issues.apache.org/jira/browse/WHIRR-464 >>>>>>> >>>>>>> As I said already in a comment my advice is to add: >>>>>>> >>>>>>> whirr.cluster-user=whirr (to hadoop.properties) >>>>>>> >>>>>>> and you should be able to login using something like: >>>>>>> >>>>>>> ssh -i ~/.ssh/id_rsa_whirr whirr@remote-host-ip >>>>>>> >>>>>>> -- Andrei >>>>>>> >>>>>>> >>>>>> >>>>>> >>>>>> >>>> >>> >> >> >
