Hi Yorn,
I figured this out and detailed the solution in my post earlier this
morning (6/23 2:46). The key is the following: -ca
mapred.job.tracker=localhost:5431. Without this, you'll see the exception
you detailed above.
--John
On Thu, Jun 5, 2014 at 5:36 AM, Yørn de Jong y...@uninett.no wrote:
Hi group
I have set up Giraph on a YARN cluster. I have no trouble running the
shortest paths example as described in [1], but when I try to run my own
algorithm, the program stops with:
Exception in thread main Java.lang.IllegalArgumentException:
checkLocalJobRunnerConfiguration: When using LocalJobRunner, must have only
one worker since only 1 task at a time!
When I change the command to -w 1, it stops with
Exception in thread main java.lang.IllegalArgumentException:
checkLocalJobRunnerConfiguration: When using LocalJobRunner, you cannot run
in split master / worker mode since there is only 1 task at a time!
The command I try to run is
hadoop \
jar
giraph-rank-1.1.0-SNAPSHOT-for-hadoop-2.3.0-cdh5.0.1-jar-with-dependencies.jar
\
org.apache.giraph.GiraphRunner
no.uninett.yorn.giraph.computation.DOSRank \
-eif no.uninett.yorn.giraph.format.io.NetflowCSVEdgeInputFormat \
-eip /user/hdfs/trd_gw1_12_01_normalized.csv \
-vof org.apache.giraph.io.formats.IdWithValueTextOutputFormat \
-op /user/yarn/output \
-wc org.apache.giraph.worker.DefaultWorkerContext \
-w 5 \
-yj
giraph-rank-1.1.0-SNAPSHOT-for-hadoop-2.3.0-cdh5.0.1-jar-with-dependencies.jar
Where, naturally, no.uninett.yorn.giraph.computation.DOSRank is my own
algorithm, which is contained in
giraph-rank-1.1.0-SNAPSHOT-for-hadoop-2.3.0-cdh5.0.1-jar-with-dependencies.jar.
giraph-rank is built using the same command I used to build the
giraph-examples project, and the pom.xml for giraph-rank is made by copying
from giraph-examples, and replacing «examples» with «rank». The command
used to build both projects is:
mvn -Phadoop_yarn -Dhadoop.version=2.3.0-cdh5.0.1 clean package
-DskipTests
Interestingly enough, when I change the input format to something else, I
get error messages having to do with type mismatches. This seems to suggest
that my EdgeInputFormat does start, but that the problem occurs while it
runs or after it runs. I don’t know how to debug this.
Am I missing something? How can I get my algorithm to run?
The source code is available on [2].
[1] https://giraph.apache.org/quick_start.html#qs_section_5
[2] https://scm.uninett.no/yorn/giraph
