Hi Max, Not sure if you have already, but you might also want to look into Apache Ambari [1] for provisioning, managing, and monitoring Hadoop clusters. Many have successfully deployed Hadoop clusters on EC2 using Ambari.
[1] http://ambari.apache.org/ Yusaku On Fri, Mar 28, 2014 at 7:07 PM, Max Zhao <gz123forhad...@gmail.com> wrote: > Hi Everybody, > > I am trying to get my first hadoop cluster started using the Amazon EC2. I > tried quite a few times and searched the web for the solutions, yet I still > cannot get it up. I hope somebody can help out here. > > Here is what I did based on the Apache Whirr Quick Guide > (http://whirr.apache.org/docs/0.8.1/quick-start-guide.html): > > 1) I downloaded a Whirr tar ball and installed it. > bin/whirr version shows the following: Apache Whirr > 0.8.2 & jclouds 1.5.8 > 2) I created the ./whirr directory and edit the credential file with my > Amazon "PROVIDER", "IDENTITY" and "CREDENTIAL" > "IDENTITY=AAAAAASSSSSSSSS", with no extra quotes or curly > quotes around the actual key_id > 3) I used the following command to creat the key pair for whirr and stored > it at the folder .ssh > ssh-keygen -t rsa -P '' -f ~/.ssh/id_rsa_whirr > 4) I think I am ready to use one of the properties file provided with whirr > in the recipes folder. Here is the command I ran: > bin/whirr launch-cluster --config > recipes/hadoop-yarn-ec2.properties --private-key-file ~/.ssh/id_rsa_whi > The command ran into the error and did not bring up the hadoop. My questin > is: Do we need to change anything the default properties provided in the > recipes folder in the "whirr-0.8.2" folder, such as the > "hadoop-yarn-ec2.properties " I used? > > Here are the error messages: > > --------------------------------------------------- > [ec2-user@ip-172-31-20-120 whirr-0.8.2]$ bin/whirr launch-cluster --config > recipes/hadoop-yarn-ec2.properties --private-key-file ~/.ssh/id_rsa_whirr > Running on provider aws-ec2 using identity AKIAJLFVRARQ3IZE3KGF > Unable to start the cluster. Terminating all nodes. > com.google.common.util.concurrent.UncheckedExecutionException: > com.google.inject.CreationException: Guice creation errors: > 1) org.jclouds.rest.RestContext<org.jclouds.aws.ec2.AWSEC2Client, A> cannot > be used as a key; It is not fully specified. > 1 error > at > com.google.common.cache.LocalCache$Segment.get(LocalCache.java:2258) > at com.google.common.cache.LocalCache.get(LocalCache.java:3990) > at > com.google.common.cache.LocalCache.getOrLoad(LocalCache.java:3994) > at > com.google.common.cache.LocalCache$LocalLoadingCache.get(LocalCache.java:4878) > at > com.google.common.cache.LocalCache$LocalLoadingCache.getUnchecked(LocalCache.java:4884) > at org.apache.whirr.service.ComputeCache.apply(ComputeCache.java:88) > at org.apache.whirr.service.ComputeCache.apply(ComputeCache.java:80) > at > org.apache.whirr.actions.ScriptBasedClusterAction.execute(ScriptBasedClusterAction.java:110) > at > org.apache.whirr.ClusterController.bootstrapCluster(ClusterController.java:137) > at > org.apache.whirr.ClusterController.launchCluster(ClusterController.java:113) > at > org.apache.whirr.cli.command.LaunchClusterCommand.run(LaunchClusterCommand.java:69) > at > org.apache.whirr.cli.command.LaunchClusterCommand.run(LaunchClusterCommand.java:59) > at org.apache.whirr.cli.Main.run(Main.java:69) > at org.apache.whirr.cli.Main.main(Main.java:102) > Caused by: com.google.inject.CreationException: Guice creation errors: > 1) org.jclouds.rest.RestContext<org.jclouds.aws.ec2.AWSEC2Client, A> cannot > be used as a key; It is not fully specified. > 1 error > at > com.google.inject.internal.Errors.throwCreationExceptionIfErrorsExist(Errors.java:435) > at > com.google.inject.internal.InternalInjectorCreator.initializeStatically(InternalInjectorCreator.java:154) > at > com.google.inject.internal.InternalInjectorCreator.build(InternalInjectorCreator.java:106) > at com.google.inject.Guice.createInjector(Guice.java:95) > at org.jclouds.ContextBuilder.buildInjector(ContextBuilder.java:401) > at org.jclouds.ContextBuilder.buildInjector(ContextBuilder.java:325) > at org.jclouds.ContextBuilder.buildView(ContextBuilder.java:600) > at org.jclouds.ContextBuilder.buildView(ContextBuilder.java:580) > at > org.apache.whirr.service.ComputeCache$1.load(ComputeCache.java:119) > at > org.apache.whirr.service.ComputeCache$1.load(ComputeCache.java:98) > at > com.google.common.cache.LocalCache$LoadingValueReference.loadFuture(LocalCache.java:3589) > at > com.google.common.cache.LocalCache$Segment.loadSync(LocalCache.java:2374) > at > com.google.common.cache.LocalCache$Segment.lockedGetOrLoad(LocalCache.java:2337) > at > com.google.common.cache.LocalCache$Segment.get(LocalCache.java:2252) > ... 13 more > Unable to load cluster state, assuming it has no running nodes. > java.io.FileNotFoundException: /home/ec2-user/.whirr/hadoop-yarn/instances > (No such file or directory) > at java.io.FileInputStream.open(Native Method) > at java.io.FileInputStream.<init>(FileInputStream.java:140) > at com.google.common.io.Files$1.getInput(Files.java:109) > at com.google.common.io.Files$1.getInput(Files.java:106) > at com.google.common.io.CharStreams$2.getInput(CharStreams.java:93) > at com.google.common.io.CharStreams$2.getInput(CharStreams.java:90) > at com.google.common.io.CharStreams.readLines(CharStreams.java:307) > at com.google.common.io.Files.readLines(Files.java:543) > at > org.apache.whirr.state.FileClusterStateStore.load(FileClusterStateStore.java:55) > at > org.apache.whirr.state.ClusterStateStore.tryLoadOrEmpty(ClusterStateStore.java:59) > at > org.apache.whirr.ClusterController.destroyCluster(ClusterController.java:226) > at > org.apache.whirr.ClusterController.launchCluster(ClusterController.java:121) > at > org.apache.whirr.cli.command.LaunchClusterCommand.run(LaunchClusterCommand.java:69) > at > org.apache.whirr.cli.command.LaunchClusterCommand.run(LaunchClusterCommand.java:59) > at org.apache.whirr.cli.Main.run(Main.java:69) > at org.apache.whirr.cli.Main.main(Main.java:102) > Exception in thread "main" > com.google.common.util.concurrent.UncheckedExecutionException: > com.google.inject.CreationException: Guice creation errors: > 1) org.jclouds.rest.RestContext<org.jclouds.aws.ec2.AWSEC2Client, A> cannot > be used as a key; It is not fully specified. > 1 error > at > com.google.common.cache.LocalCache$Segment.get(LocalCache.java:2258) > at com.google.common.cache.LocalCache.get(LocalCache.java:3990) > at > com.google.common.cache.LocalCache.getOrLoad(LocalCache.java:3994) > at > com.google.common.cache.LocalCache$LocalLoadingCache.get(LocalCache.java:4878) > at > com.google.common.cache.LocalCache$LocalLoadingCache.getUnchecked(LocalCache.java:4884) > at org.apache.whirr.service.ComputeCache.apply(ComputeCache.java:88) > at org.apache.whirr.service.ComputeCache.apply(ComputeCache.java:80) > at > org.apache.whirr.actions.ScriptBasedClusterAction.execute(ScriptBasedClusterAction.java:110) > at > org.apache.whirr.ClusterController.destroyCluster(ClusterController.java:233) > at > org.apache.whirr.ClusterController.destroyCluster(ClusterController.java:226) > at > org.apache.whirr.ClusterController.launchCluster(ClusterController.java:121) > at > org.apache.whirr.cli.command.LaunchClusterCommand.run(LaunchClusterCommand.java:69) > at > org.apache.whirr.cli.command.LaunchClusterCommand.run(LaunchClusterCommand.java:59) > at org.apache.whirr.cli.Main.run(Main.java:69) > at org.apache.whirr.cli.Main.main(Main.java:102) > Caused by: com.google.inject.CreationException: Guice creation errors: > 1) org.jclouds.rest.RestContext<org.jclouds.aws.ec2.AWSEC2Client, A> cannot > be used as a key; It is not fully specified. > 1 error > at > com.google.inject.internal.Errors.throwCreationExceptionIfErrorsExist(Errors.java:435) > at > com.google.inject.internal.InternalInjectorCreator.initializeStatically(InternalInjectorCreator.java:154) > at > com.google.inject.internal.InternalInjectorCreator.build(InternalInjectorCreator.java:106) > at com.google.inject.Guice.createInjector(Guice.java:95) > at org.jclouds.ContextBuilder.buildInjector(ContextBuilder.java:401) > at org.jclouds.ContextBuilder.buildInjector(ContextBuilder.java:325) > at org.jclouds.ContextBuilder.buildView(ContextBuilder.java:600) > at org.jclouds.ContextBuilder.buildView(ContextBuilder.java:580) > at > org.apache.whirr.service.ComputeCache$1.load(ComputeCache.java:119) > at > org.apache.whirr.service.ComputeCache$1.load(ComputeCache.java:98) > at > com.google.common.cache.LocalCache$LoadingValueReference.loadFuture(LocalCache.java:3589) > at > com.google.common.cache.LocalCache$Segment.loadSync(LocalCache.java:2374) > at > com.google.common.cache.LocalCache$Segment.lockedGetOrLoad(LocalCache.java:2337) > at > com.google.common.cache.LocalCache$Segment.get(LocalCache.java:2252) > I would really appreciate if somebody can help pinpoint my problem here. > > Thanks > Max -- CONFIDENTIALITY NOTICE NOTICE: This message is intended for the use of the individual or entity to which it is addressed and may contain information that is confidential, privileged and exempt from disclosure under applicable law. If the reader of this message is not the intended recipient, you are hereby notified that any printing, copying, dissemination, distribution, disclosure or forwarding of this communication is strictly prohibited. If you have received this communication in error, please contact the sender immediately and delete it from your system. Thank You.
