Here's more details regarding my attempts are running Shortest Path. Any
help would be greatly appreciated as the root cause for the Giraph job
failing is not obvious to me.
Thanks
---John
Command Line:
$ hadoop jar
giraph-examples-1.1.0-SNAPSHOT-for-hadoop-2.2.0-jar-with-dependencies.jar
org.apache.giraph.GiraphRunner
org.apache.giraph.examples.SimpleShortestPathsComputation -vif
org.apache.giraph.io.formats.JsonLongDoubleFloatDoubleVertexInputFormat
-vip /user/hadoop/tiny.txt -vof
org.apache.giraph.io.formats.IdWithValueTextOutputFormat -op
/user/hadoop/shortestpaths -w 1 -yj
giraph-examples-1.1.0-SNAPSHOT-for-hadoop-2.2.0-jar-with-dependencies.jar
Console Output:
14/06/20 21:49:02 WARN util.NativeCodeLoader: Unable to load native-hadoop
library for your platform... using builtin-java classes where applicable
14/06/20 21:49:03 INFO utils.ConfigurationUtils: No edge input format
specified. Ensure your InputFormat does not require one.
14/06/20 21:49:03 INFO utils.ConfigurationUtils: No edge output format
specified. Ensure your OutputFormat does not require one.
14/06/20 21:49:03 INFO yarn.GiraphYarnClient: Final output path is:
hdfs://localhost.localdomain:8020/user/hadoop/shortestpaths
14/06/20 21:49:03 INFO yarn.GiraphYarnClient: Running Client
14/06/20 21:49:03 INFO client.RMProxy: Connecting to ResourceManager at /
0.0.0.0:8032
14/06/20 21:49:03 INFO yarn.GiraphYarnClient: Got node report from ASM for,
nodeId=localhost.localdomain:36056, nodeAddress localhost.localdomain:8042,
nodeRackName /default-rack, nodeNumContainers 0
14/06/20 21:49:03 INFO yarn.GiraphYarnClient: Defaulting per-task heap size
to 1024MB.
14/06/20 21:49:03 INFO yarn.GiraphYarnClient: Obtained new Application ID:
application_1402926902901_0001
14/06/20 21:49:03 INFO Configuration.deprecation: mapred.job.id is
deprecated. Instead, use mapreduce.job.id
14/06/20 21:49:03 INFO yarn.GiraphYarnClient: Set the environment for the
application master
14/06/20 21:49:03 INFO yarn.GiraphYarnClient: Environment for AM
:{CLASSPATH=${CLASSPATH}:./*:$HADOOP_CONF_DIR:$HADOOP_COMMON_HOME/share/hadoop/common/*:$HADOOP_COMMON_HOME/share/hadoop/common/lib/*:$HADOOP_HDFS_HOME/share/hadoop/hdfs/*:$HADOOP_HDFS_HOME/share/hadoop/hdfs/lib/*:$HADOOP_YARN_HOME/share/hadoop/yarn/*:$HADOOP_YARN_HOME/share/hadoop/yarn/lib/*:$HADOOP_MAPRED_HOME/share/hadoop/mapreduce/*:$HADOOP_MAPRED_HOME/share/hadoop/mapreduce/lib/*}
14/06/20 21:49:03 INFO yarn.GiraphYarnClient: buildLocalResourceMap ....
14/06/20 21:49:03 INFO Configuration.deprecation: mapred.output.dir is
deprecated. Instead, use mapreduce.output.fileoutputformat.outputdir
14/06/20 21:49:04 INFO yarn.YarnUtils: Registered file in LocalResources ::
hdfs://localhost.localdomain:8020/user/hadoop/giraph_yarn_jar_cache/application_1402926902901_0001/giraph-conf.xml
14/06/20 21:49:04 INFO yarn.GiraphYarnClient: LIB JARS
:giraph-examples-1.1.0-SNAPSHOT-for-hadoop-2.2.0-jar-with-dependencies.jar
14/06/20 21:49:04 INFO yarn.YarnUtils: Class path name .
14/06/20 21:49:04 INFO yarn.YarnUtils: base path checking .
14/06/20 21:49:04 INFO yarn.GiraphYarnClient: Made local resource for
:/home/hadoop/Downloads/giraph/giraph-1200915/giraph-examples/target/giraph-examples-1.1.0-SNAPSHOT-for-hadoop-2.2.0-jar-with-dependencies.jar
to
hdfs://localhost.localdomain:8020/user/hadoop/giraph_yarn_jar_cache/application_1402926902901_0001/giraph-examples-1.1.0-SNAPSHOT-for-hadoop-2.2.0-jar-with-dependencies.jar
14/06/20 21:49:04 INFO yarn.YarnUtils: Registered file in LocalResources ::
hdfs://localhost.localdomain:8020/user/hadoop/giraph_yarn_jar_cache/application_1402926902901_0001/giraph-examples-1.1.0-SNAPSHOT-for-hadoop-2.2.0-jar-with-dependencies.jar
14/06/20 21:49:04 INFO yarn.GiraphYarnClient: ApplicationSumbissionContext
for GiraphApplicationMaster launch container is populated.
14/06/20 21:49:04 INFO yarn.GiraphYarnClient: Submitting application to ASM
14/06/20 21:49:04 INFO impl.YarnClientImpl: Submitted application
application_1402926902901_0001 to ResourceManager at /0.0.0.0:8032
14/06/20 21:49:04 INFO yarn.GiraphYarnClient: Got new appId after
submission :application_1402926902901_0001
14/06/20 21:49:04 INFO yarn.GiraphYarnClient: GiraphApplicationMaster
container request was submitted to ResourceManager for job: Giraph:
org.apache.giraph.examples.SimpleShortestPathsComputation
14/06/20 21:49:05 INFO yarn.GiraphYarnClient: Giraph:
org.apache.giraph.examples.SimpleShortestPathsComputation, Elapsed: 0.88
secs
14/06/20 21:49:05 INFO yarn.GiraphYarnClient:
appattempt_1402926902901_0001_000001, State: ACCEPTED, Containers used: 1
14/06/20 21:49:09 INFO yarn.GiraphYarnClient: Giraph:
org.apache.giraph.examples.SimpleShortestPathsComputation, Elapsed: 4.90
secs
14/06/20 21:49:09 INFO yarn.GiraphYarnClient:
appattempt_1402926902901_0001_000001, State: RUNNING, Containers used: 1
14/06/20 21:49:13 INFO yarn.GiraphYarnClient: Cleaning up HDFS distributed
cache directory for Giraph job.
14/06/20 21:49:13 INFO yarn.GiraphYarnClient: Completed Giraph:
org.apache.giraph.examples.SimpleShortestPathsComputation: FAILED, total
running time: 0 minutes, 8 seconds.
Log:
2014-06-20 21:49:07,193 INFO [main] yarn.GiraphApplicationMaster
(GiraphApplicationMaster.java:main(421)) - Starting GitaphAM
2014-06-20 21:49:07,979 WARN [main] util.NativeCodeLoader
(NativeCodeLoader.java:<clinit>(62)) - Unable to load native-hadoop
library for your platform... using builtin-java classes where
applicable
2014-06-20 21:49:08,871 INFO [main] yarn.GiraphApplicationMaster
(GiraphApplicationMaster.java:<init>(168)) - GiraphAM for ContainerId
container_1402926902901_0001_01_000001 ApplicationAttemptId
appattempt_1402926902901_0001_000001
2014-06-20 21:49:08,945 INFO [main] client.RMProxy
(RMProxy.java:createRMProxy(56)) - Connecting to ResourceManager at
/0.0.0.0:8030
2014-06-20 21:49:08,952 INFO [main] impl.NMClientAsyncImpl
(NMClientAsyncImpl.java:serviceInit(107)) - Upper bound of the thread
pool size is 500
2014-06-20 21:49:08,953 INFO [main]
impl.ContainerManagementProtocolProxy
(ContainerManagementProtocolProxy.java:<init>(71)) -
yarn.client.max-nodemanagers-proxies : 500
2014-06-20 21:49:09,187 INFO [main] yarn.GiraphApplicationMaster
(GiraphApplicationMaster.java:setupContainerAskForRM(279)) - Requested
container ask: Capability[<memory:1024, vCores:0>]Priority[10]
2014-06-20 21:49:09,192 INFO [main] yarn.GiraphApplicationMaster
(GiraphApplicationMaster.java:setupContainerAskForRM(279)) - Requested
container ask: Capability[<memory:1024, vCores:0>]Priority[10]
2014-06-20 21:49:09,193 INFO [main] yarn.GiraphApplicationMaster
(GiraphApplicationMaster.java:run(185)) - Wait to finish ..
2014-06-20 21:49:11,264 INFO [AMRM Callback Handler Thread]
yarn.GiraphApplicationMaster
(GiraphApplicationMaster.java:onContainersAllocated(605)) - Got
response from RM for container ask, allocatedCnt=1
2014-06-20 21:49:11,264 INFO [AMRM Callback Handler Thread]
yarn.GiraphApplicationMaster
(GiraphApplicationMaster.java:onContainersAllocated(608)) - Total
allocated # of container so far : 1 allocated out of 2 required.
2014-06-20 21:49:11,265 INFO [AMRM Callback Handler Thread]
yarn.GiraphApplicationMaster
(GiraphApplicationMaster.java:startContainerLaunchingThreads(359)) -
Launching command on a new container.,
containerId=container_1402926902901_0001_01_000002,
containerNode=localhost.localdomain:36056,
containerNodeURI=localhost.localdomain:8042,
containerResourceMemory=1024
2014-06-20 21:49:11,267 INFO [pool-2-thread-1]
yarn.GiraphApplicationMaster
(GiraphApplicationMaster.java:buildContainerLaunchContext(492)) -
Setting up container launch container for
containerid=container_1402926902901_0001_01_000002
2014-06-20 21:49:11,291 INFO [pool-2-thread-1]
yarn.GiraphApplicationMaster
(GiraphApplicationMaster.java:buildContainerLaunchContext(498)) -
Conatain launch Commands :java -Xmx1024M -Xms1024M -cp .:${CLASSPATH}
org.apache.giraph.yarn.GiraphYarnTask 1402926902901 1 2 1
1><LOG_DIR>/task-2-stdout.log 2><LOG_DIR>/task-2-stderr.log
2014-06-20 21:49:11,292 INFO [pool-2-thread-1]
yarn.GiraphApplicationMaster
(GiraphApplicationMaster.java:buildContainerLaunchContext(518)) -
Setting username in ContainerLaunchContext to: hadoop
2014-06-20 21:49:11,685 INFO [pool-2-thread-1] yarn.YarnUtils
(YarnUtils.java:addFsResourcesToMap(72)) - Adding
giraph-examples-1.1.0-SNAPSHOT-for-hadoop-2.2.0-jar-with-dependencies.jar
to LocalResources for export.to
hdfs://localhost.localdomain:8020/user/hadoop/giraph_yarn_jar_cache/application_1402926902901_0001/giraph-examples-1.1.0-SNAPSHOT-for-hadoop-2.2.0-jar-with-dependencies.jar
2014-06-20 21:49:11,725 INFO [pool-2-thread-1] yarn.YarnUtils
(YarnUtils.java:addFileToResourceMap(160)) - Registered file in
LocalResources ::
hdfs://localhost.localdomain:8020/user/hadoop/giraph_yarn_jar_cache/application_1402926902901_0001/giraph-examples-1.1.0-SNAPSHOT-for-hadoop-2.2.0-jar-with-dependencies.jar
2014-06-20 21:49:11,725 WARN [pool-2-thread-1] yarn.YarnUtils
(YarnUtils.java:addFsResourcesToMap(81)) - Job jars (-yj option)
didn't include giraph-core.
2014-06-20 21:49:11,726 INFO [pool-2-thread-1] yarn.YarnUtils
(YarnUtils.java:addFileToResourceMap(160)) - Registered file in
LocalResources ::
hdfs://localhost.localdomain:8020/user/hadoop/giraph_yarn_jar_cache/application_1402926902901_0001/giraph-conf.xml
2014-06-20 21:49:11,739 INFO
[org.apache.hadoop.yarn.client.api.async.impl.NMClientAsyncImpl #0]
impl.NMClientAsyncImpl (NMClientAsyncImpl.java:run(531)) - Processing
Event EventType: START_CONTAINER for Container
container_1402926902901_0001_01_000002
2014-06-20 21:49:11,740 INFO
[org.apache.hadoop.yarn.client.api.async.impl.NMClientAsyncImpl #0]
impl.ContainerManagementProtocolProxy
(ContainerManagementProtocolProxy.java:newProxy(205)) - Opening proxy
: localhost.localdomain:36056
2014-06-20 21:49:11,789 INFO
[org.apache.hadoop.yarn.client.api.async.impl.NMClientAsyncImpl #1]
impl.NMClientAsyncImpl (NMClientAsyncImpl.java:run(531)) - Processing
Event EventType: QUERY_CONTAINER for Container
container_1402926902901_0001_01_000002
2014-06-20 21:49:12,266 INFO [AMRM Callback Handler Thread]
yarn.GiraphApplicationMaster
(GiraphApplicationMaster.java:onContainersAllocated(605)) - Got
response from RM for container ask, allocatedCnt=1
2014-06-20 21:49:12,267 INFO [AMRM Callback Handler Thread]
yarn.GiraphApplicationMaster
(GiraphApplicationMaster.java:onContainersAllocated(608)) - Total
allocated # of container so far : 2 allocated out of 2 required.
2014-06-20 21:49:12,267 INFO [AMRM Callback Handler Thread]
yarn.GiraphApplicationMaster
(GiraphApplicationMaster.java:startContainerLaunchingThreads(359)) -
Launching command on a new container.,
containerId=container_1402926902901_0001_01_000003,
containerNode=localhost.localdomain:36056,
containerNodeURI=localhost.localdomain:8042,
containerResourceMemory=1024
2014-06-20 21:49:12,268 INFO [pool-2-thread-2]
yarn.GiraphApplicationMaster
(GiraphApplicationMaster.java:buildContainerLaunchContext(492)) -
Setting up container launch container for
containerid=container_1402926902901_0001_01_000003
2014-06-20 21:49:12,269 INFO [pool-2-thread-2]
yarn.GiraphApplicationMaster
(GiraphApplicationMaster.java:buildContainerLaunchContext(498)) -
Conatain launch Commands :java -Xmx1024M -Xms1024M -cp .:${CLASSPATH}
org.apache.giraph.yarn.GiraphYarnTask 1402926902901 1 3 1
1><LOG_DIR>/task-3-stdout.log 2><LOG_DIR>/task-3-stderr.log
2014-06-20 21:49:12,269 INFO [pool-2-thread-2]
yarn.GiraphApplicationMaster
(GiraphApplicationMaster.java:buildContainerLaunchContext(518)) -
Setting username in ContainerLaunchContext to: hadoop
2014-06-20 21:49:12,271 INFO
[org.apache.hadoop.yarn.client.api.async.impl.NMClientAsyncImpl #2]
impl.NMClientAsyncImpl (NMClientAsyncImpl.java:run(531)) - Processing
Event EventType: START_CONTAINER for Container
container_1402926902901_0001_01_000003
2014-06-20 21:49:12,284 INFO
[org.apache.hadoop.yarn.client.api.async.impl.NMClientAsyncImpl #3]
impl.NMClientAsyncImpl (NMClientAsyncImpl.java:run(531)) - Processing
Event EventType: QUERY_CONTAINER for Container
container_1402926902901_0001_01_000003
2014-06-20 21:49:13,272 INFO [AMRM Callback Handler Thread]
yarn.GiraphApplicationMaster
(GiraphApplicationMaster.java:onContainersCompleted(571)) - Got
response from RM for container ask, completedCnt=2
2014-06-20 21:49:13,273 INFO [AMRM Callback Handler Thread]
yarn.GiraphApplicationMaster
(GiraphApplicationMaster.java:onContainersCompleted(574)) - Got
container status for
containerID=container_1402926902901_0001_01_000002, state=COMPLETE,
exitStatus=1, diagnostics=Exception from container-launch:
org.apache.hadoop.util.Shell$ExitCodeException:
at org.apache.hadoop.util.Shell.runCommand(Shell.java:464)
at org.apache.hadoop.util.Shell.run(Shell.java:379)
at
org.apache.hadoop.util.Shell$ShellCommandExecutor.execute(Shell.java:589)
at
org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor.launchContainer(DefaultContainerExecutor.java:195)
at
org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:283)
at
org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:79)
at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334)
at java.util.concurrent.FutureTask.run(FutureTask.java:166)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
at java.lang.Thread.run(Thread.java:722)
2014-06-20 21:49:13,273 INFO [AMRM Callback Handler Thread]
yarn.GiraphApplicationMaster
(GiraphApplicationMaster.java:onContainersCompleted(574)) - Got
container status for
containerID=container_1402926902901_0001_01_000003, state=COMPLETE,
exitStatus=1, diagnostics=Exception from container-launch:
org.apache.hadoop.util.Shell$ExitCodeException:
at org.apache.hadoop.util.Shell.runCommand(Shell.java:464)
at org.apache.hadoop.util.Shell.run(Shell.java:379)
at
org.apache.hadoop.util.Shell$ShellCommandExecutor.execute(Shell.java:589)
at
org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor.launchContainer(DefaultContainerExecutor.java:195)
at
org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:283)
at
org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:79)
at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334)
at java.util.concurrent.FutureTask.run(FutureTask.java:166)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
at java.lang.Thread.run(Thread.java:722)
2014-06-20 21:49:13,274 INFO [AMRM Callback Handler Thread]
yarn.GiraphApplicationMaster
(GiraphApplicationMaster.java:onContainersCompleted(594)) - All
container compeleted. done = true
2014-06-20 21:49:13,403 INFO [main] yarn.GiraphApplicationMaster
(GiraphApplicationMaster.java:run(194)) - Done true
2014-06-20 21:49:13,403 INFO [main] yarn.GiraphApplicationMaster
(GiraphApplicationMaster.java:run(198)) - Forcefully terminating
executors with done =:true
2014-06-20 21:49:13,403 INFO [main] yarn.GiraphApplicationMaster
(GiraphApplicationMaster.java:finish(212)) - Application completed.
Stopping running containers
2014-06-20 21:49:13,427 INFO [main]
impl.ContainerManagementProtocolProxy
(ContainerManagementProtocolProxy.java:mayBeCloseProxy(138)) - Closing
proxy : localhost.localdomain:36056
2014-06-20 21:49:13,427 INFO [main] yarn.GiraphApplicationMaster
(GiraphApplicationMaster.java:finish(217)) - Application completed.
Signalling finish to RM
2014-06-20 21:49:13,440 INFO [main] impl.AMRMClientImpl
(AMRMClientImpl.java:unregisterApplicationMaster(316)) - Waiting for
application to be successfully unregistered.
2014-06-20 21:49:13,543 INFO [AMRM Heartbeater thread]
impl.AMRMClientAsyncImpl (AMRMClientAsyncImpl.java:run(251)) -
Heartbeater interrupted
java.lang.InterruptedException: sleep interrupted
at java.lang.Thread.sleep(Native Method)
at
org.apache.hadoop.yarn.client.api.async.impl.AMRMClientAsyncImpl$HeartbeatThread.run(AMRMClientAsyncImpl.java:249)
2014-06-20 21:49:13,544 INFO [AMRM Callback Handler Thread]
impl.AMRMClientAsyncImpl (AMRMClientAsyncImpl.java:run(277)) -
Interrupted while waiting for queue
java.lang.InterruptedException
at
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.reportInterruptAfterWait(AbstractQueuedSynchronizer.java:2017)
at
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2052)
at
java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
at
org.apache.hadoop.yarn.client.api.async.impl.AMRMClientAsyncImpl$CallbackHandlerThread.run(AMRMClientAsyncImpl.java:275)
2014-06-20 21:49:13,545 INFO [main] yarn.GiraphApplicationMaster
(GiraphApplicationMaster.java:main(445)) - Giraph Application Master
failed. exiting
