Hey Phil, I have been having the exact same problems as you (I am also setting up Giraph on EC2), but this solution did not work for me.
Do you recall what error you saw in resourcemanager logs? I am also looking at these logs, but nothing is standing out to me. In fact, it almost seems like the application should have successfully finished. The log stops updating and I see a lot of "COMPLETED", "RESULT=SUCCESS", "FINISHED" at the end of the log. Though, it does look like one of the containers is not transitioning to these states. Thanks, Steve On Wed, Mar 11, 2015 at 11:54 PM, Phillip Rhodes <motley.crue....@gmail.com> wrote: > OK, this was easy enough to fix, once I understood what > was actually happening. Since I'm running on EC2 nodes on > AWS, it is not the case that any give node can talk to any other > node on any port (at least not by default). I had tried to > cherry-pick which ports to whitelist in the security group, > but I missed one or more that YARN needed for internal > communication. I discovered this when examining the > resourcemanager logs. > > > For now, instead of trying to enumerate exactly which ports > to allow, I added a rule to allow "all traffic" for address 10.0.0.0/24 > and that solved this. > > > Cheers, > > > Phil > > > On Wed, Mar 11, 2015 at 1:39 PM, Phillip Rhodes > <motley.crue....@gmail.com> wrote: > > Interesting... It totally did not work for me when built using the > > hadoop_2 profile, but with the hadoop_yarn profile everything at least > > starts up. I'm pretty baffled right now... my cluster is essentially > > working, and I can run, for example, the WordCount example just fine. > > And the Giraph job starts and shows no apparent errors, but I get no > > output and it seems to run forever. > > > > It's probably some really small detail of my Hadoop configuration, or > > some environmental issue. The problem is, I don't even know where to > > start looking right now. :-( > > > > > > Phil > > This message optimized for indexing by NSA PRISM > > > > > > On Wed, Mar 11, 2015 at 3:16 AM, Martin Junghanns > > <martin.jungha...@gmx.net> wrote: > >> Hi Phillip, > >> > >> I am using Hadoop 2.5.2 with Giraph 1.1.0 and it runs fine with > >> -Phadoop2 (from scratch) and -Phadoop_yarn (after removing > >> STATIC_SASL_SYMBOL from munge.symbols in pom.xml). > >> > >> Maybe you can also try the stable Giraph > >> version and report your problem as an issue? > >> > >> Cheers, > >> Martin > >> > >> On 11.03.2015 04:03, Phillip Rhodes wrote: > >>> Giraph crew: > >>> > >>> I'm trying to run the SimpleShortestPathsComputation example using > >>> the latest Giraph code and Hadoop 2.5.2. My command line looks > >>> like this: > >>> > >>> hadoop jar > >>> > /home/prhodes/giraph/giraph-examples/target/giraph-examples-1.2.0-SNAPSHOT-for-hadoop-2.5.2-jar-with-dependencies.jar > >>> > >>> > >> org.apache.giraph.GiraphRunner > >>> org.apache.giraph.examples.SimpleShortestPathsComputation -vif > >>> org.apache.giraph.io.formats.JsonLongDoubleFloatDoubleVertexInputFormat > >>> > >>> > >> -vip /user/prhodes/input/tiny_graph.txt -vof > >>> org.apache.giraph.io.formats.IdWithValueTextOutputFormat -op > >>> /user/prhodes/giraph_output/shortestpaths -w 4 > >>> > >>> > >>> and the job appears to start OK. But then it starts outputing > >>> these kinds of messages, and this just continues (seemingly) > >>> forever until you ctrl+c it. > >>> > >>> 15/03/11 02:54:31 INFO yarn.GiraphYarnClient: Giraph: > >>> org.apache.giraph.examples.SimpleShortestPathsComputation, > >>> Elapsed: 305.43 secs 15/03/11 02:54:31 INFO yarn.GiraphYarnClient: > >>> appattempt_1426041786848_0002_000001, State: ACCEPTED, Containers > >>> used: 1 15/03/11 02:54:35 INFO yarn.GiraphYarnClient: Giraph: > >>> org.apache.giraph.examples.SimpleShortestPathsComputation, > >>> Elapsed: 309.44 secs 15/03/11 02:54:35 INFO yarn.GiraphYarnClient: > >>> appattempt_1426041786848_0002_000001, State: ACCEPTED, Containers > >>> used: 1 15/03/11 02:54:39 INFO yarn.GiraphYarnClient: Giraph: > >>> org.apache.giraph.examples.SimpleShortestPathsComputation, > >>> Elapsed: 313.45 secs 15/03/11 02:54:39 INFO yarn.GiraphYarnClient: > >>> appattempt_1426041786848_0002_000001, State: ACCEPTED, Containers > >>> used: 1 15/03/11 02:54:43 INFO yarn.GiraphYarnClient: Giraph: > >>> org.apache.giraph.examples.SimpleShortestPathsComputation, > >>> Elapsed: 317.45 secs 15/03/11 02:54:43 INFO yarn.GiraphYarnClient: > >>> appattempt_1426041786848_0002_000001, State: ACCEPTED, Containers > >>> used: 1 ^C15/03/11 02:54:47 INFO yarn.GiraphYarnClient: Giraph: > >>> org.apache.giraph.examples.SimpleShortestPathsComputation, > >>> Elapsed: 321.46 secs 15/03/11 02:54:47 INFO yarn.GiraphYarnClient: > >>> appattempt_1426041786848_0002_000001, State: ACCEPTED, Containers > >>> used: 1 > >>> > >>> Any idea what is going on here? > >>> > >>> > >>> Thanks, > >>> > >>> > >>> Phil --- > >>> > >>> > >>> This message optimized for indexing by NSA PRISM > >>> >
