Sorry, in the previous email, I meant to say: when I run with only
-yj giraph-examples-1.1.0-hadoop-2.5.1-jar-with-dependencies.jar
the container log complains that it did not contain giraph-core and those
were the resulting errors.
On Sun, Nov 23, 2014 at 9:28 AM, D Adams <dadam...@gmail.com> wrote:
> Well, maybe my giraph build did not build correctly because when I run
>
> 2014-11-19 02:04:21,764 INFO [main] yarn.GiraphApplicationMaster
> (GiraphApplicationMaster.java:main(421)) - Starting GitaphAM
> 2014-11-19 02:04:23,137 WARN [main] util.NativeCodeLoader
> (NativeCodeLoader.java:<clinit>(62)) - Unable to load native-hadoop library
> for your platform... using builtin-java classes where applicable
> 2014-11-19 02:04:25,548 INFO [main] yarn.GiraphApplicationMaster
> (GiraphApplicationMaster.java:<init>(168)) - GiraphAM for ContainerId
> container_1416359131664_0002_01_000001 ApplicationAttemptId
> appattempt_1416359131664_0002_000001
> 2014-11-19 02:04:25,890 INFO [main] client.RMProxy
> (RMProxy.java:createRMProxy(92)) - Connecting to ResourceManager at
> localhost/127.0.0.1:8030
> 2014-11-19 02:04:25,924 INFO [main] impl.NMClientAsyncImpl
> (NMClientAsyncImpl.java:serviceInit(107)) - Upper bound of the thread pool
> size is 500
> 2014-11-19 02:04:25,925 INFO [main] impl.ContainerManagementProtocolProxy
> (ContainerManagementProtocolProxy.java:<init>(78)) -
> yarn.client.max-nodemanagers-proxies : 500
> 2014-11-19 02:04:26,309 INFO [main] yarn.GiraphApplicationMaster
> (GiraphApplicationMaster.java:setupContainerAskForRM(279)) - Requested
> container ask: Capability[<memory:3000, vCores:0>]Priority[10]
> 2014-11-19 02:04:26,577 INFO [main] yarn.GiraphApplicationMaster
> (GiraphApplicationMaster.java:setupContainerAskForRM(279)) - Requested
> container ask: Capability[<memory:3000, vCores:0>]Priority[10]
> 2014-11-19 02:04:26,577 INFO [main] yarn.GiraphApplicationMaster
> (GiraphApplicationMaster.java:run(185)) - Wait to finish ..
> 2014-11-19 02:04:28,612 INFO [AMRM Heartbeater thread] impl.AMRMClientImpl
> (AMRMClientImpl.java:populateNMTokens(299)) - Received new token for :
> roosevelt:58195
> 2014-11-19 02:04:28,614 INFO [AMRM Callback Handler Thread]
> yarn.GiraphApplicationMaster
> (GiraphApplicationMaster.java:onContainersAllocated(605)) - Got response from
> RM for container ask, allocatedCnt=1
> 2014-11-19 02:04:28,614 INFO [AMRM Callback Handler Thread]
> yarn.GiraphApplicationMaster
> (GiraphApplicationMaster.java:onContainersAllocated(608)) - Total allocated #
> of container so far : 1 allocated out of 2 required.
> 2014-11-19 02:04:28,614 INFO [AMRM Callback Handler Thread]
> yarn.GiraphApplicationMaster
> (GiraphApplicationMaster.java:startContainerLaunchingThreads(359)) -
> Launching command on a new container.,
> containerId=container_1416359131664_0002_01_000002,
> containerNode=roosevelt:58195, containerNodeURI=roosevelt:8042,
> containerResourceMemory=3072
> 2014-11-19 02:04:28,615 INFO [pool-4-thread-1] yarn.GiraphApplicationMaster
> (GiraphApplicationMaster.java:buildContainerLaunchContext(492)) - Setting up
> container launch container for
> containerid=container_1416359131664_0002_01_000002
> 2014-11-19 02:04:28,629 INFO [pool-4-thread-1] yarn.GiraphApplicationMaster
> (GiraphApplicationMaster.java:buildContainerLaunchContext(498)) - Conatain
> launch Commands :java -Xmx3000M -Xms3000M -cp .:${CLASSPATH}
> org.apache.giraph.yarn.GiraphYarnTask 1416359131664 2 2 1
> 1><LOG_DIR>/task-2-stdout.log 2><LOG_DIR>/task-2-stderr.log
> 2014-11-19 02:04:28,630 INFO [pool-4-thread-1] yarn.GiraphApplicationMaster
> (GiraphApplicationMaster.java:buildContainerLaunchContext(518)) - Setting
> username in ContainerLaunchContext to: hduser
> 2014-11-19 02:04:29,149 INFO [pool-4-thread-1] yarn.YarnUtils
> (YarnUtils.java:addFsResourcesToMap(72)) - Adding
> giraph-examples-1.1.0-hadoop-2.5.1-jar-with-dependencies.jar to
> LocalResources for export.to
> hdfs://localhost:9000/user/hduser/giraph_yarn_jar_cache/application_1416359131664_0002/giraph-examples-1.1.0-hadoop-2.5.1-jar-with-dependencies.jar
> 2014-11-19 02:04:29,188 INFO [pool-4-thread-1] yarn.YarnUtils
> (YarnUtils.java:addFileToResourceMap(160)) - Registered file in
> LocalResources ::
> hdfs://localhost:9000/user/hduser/giraph_yarn_jar_cache/application_1416359131664_0002/giraph-examples-1.1.0-hadoop-2.5.1-jar-with-dependencies.jar*2014-11-19
> 02:04:29,188 WARN [pool-4-thread-1] yarn.YarnUtils
> (YarnUtils.java:addFsResourcesToMap(81)) - Job jars (-yj option) didn't
> include giraph-core.*
> 2014-11-19 02:04:29,189 INFO [pool-4-thread-1] yarn.YarnUtils
> (YarnUtils.java:addFileToResourceMap(160)) - Registered file in
> LocalResources ::
> hdfs://localhost:9000/user/hduser/giraph_yarn_jar_cache/application_1416359131664_0002/giraph-conf.xml
> 2014-11-19 02:04:29,201 INFO
> [org.apache.hadoop.yarn.client.api.async.impl.NMClientAsyncImpl #0]
> impl.NMClientAsyncImpl (NMClientAsyncImpl.java:run(531)) - Processing Event
> EventType: START_CONTAINER for Container
> container_1416359131664_0002_01_000002
> 2014-11-19 02:04:29,202 INFO
> [org.apache.hadoop.yarn.client.api.async.impl.NMClientAsyncImpl #0]
> impl.ContainerManagementProtocolProxy
> (ContainerManagementProtocolProxy.java:newProxy(212)) - Opening proxy :
> roosevelt:58195
> 2014-11-19 02:04:29,257 INFO
> [org.apache.hadoop.yarn.client.api.async.impl.NMClientAsyncImpl #1]
> impl.NMClientAsyncImpl (NMClientAsyncImpl.java:run(531)) - Processing Event
> EventType: QUERY_CONTAINER for Container
> container_1416359131664_0002_01_000002
> 2014-11-19 02:04:29,618 INFO [AMRM Callback Handler Thread]
> yarn.GiraphApplicationMaster
> (GiraphApplicationMaster.java:onContainersAllocated(605)) - Got response from
> RM for container ask, allocatedCnt=1
> 2014-11-19 02:04:29,618 INFO [AMRM Callback Handler Thread]
> yarn.GiraphApplicationMaster
> (GiraphApplicationMaster.java:onContainersAllocated(608)) - Total allocated #
> of container so far : 2 allocated out of 2 required.
> 2014-11-19 02:04:29,618 INFO [AMRM Callback Handler Thread]
> yarn.GiraphApplicationMaster
> (GiraphApplicationMaster.java:startContainerLaunchingThreads(359)) -
> Launching command on a new container.,
> containerId=container_1416359131664_0002_01_000003,
> containerNode=roosevelt:58195, containerNodeURI=roosevelt:8042,
> containerResourceMemory=3072
> 2014-11-19 02:04:29,619 INFO [pool-4-thread-2] yarn.GiraphApplicationMaster
> (GiraphApplicationMaster.java:buildContainerLaunchContext(492)) - Setting up
> container launch container for
> containerid=container_1416359131664_0002_01_000003
> 2014-11-19 02:04:29,619 INFO [pool-4-thread-2] yarn.GiraphApplicationMaster
> (GiraphApplicationMaster.java:buildContainerLaunchContext(498)) - Conatain
> launch Commands :java -Xmx3000M -Xms3000M -cp .:${CLASSPATH}
> org.apache.giraph.yarn.GiraphYarnTask 1416359131664 2 3 1
> 1><LOG_DIR>/task-3-stdout.log 2><LOG_DIR>/task-3-stderr.log
> 2014-11-19 02:04:29,619 INFO [pool-4-thread-2] yarn.GiraphApplicationMaster
> (GiraphApplicationMaster.java:buildContainerLaunchContext(518)) - Setting
> username in ContainerLaunchContext to: hduser
> 2014-11-19 02:04:29,620 INFO
> [org.apache.hadoop.yarn.client.api.async.impl.NMClientAsyncImpl #2]
> impl.NMClientAsyncImpl (NMClientAsyncImpl.java:run(531)) - Processing Event
> EventType: START_CONTAINER for Container
> container_1416359131664_0002_01_000003
> 2014-11-19 02:04:29,623 INFO
> [org.apache.hadoop.yarn.client.api.async.impl.NMClientAsyncImpl #3]
> impl.NMClientAsyncImpl (NMClientAsyncImpl.java:run(531)) - Processing Event
> EventType: QUERY_CONTAINER for Container
> container_1416359131664_0002_01_000003
> 2014-11-19 02:04:30,620 INFO [AMRM Callback Handler Thread]
> yarn.GiraphApplicationMaster
> (GiraphApplicationMaster.java:onContainersCompleted(571)) - Got response from
> RM for container ask, completedCnt=2
> 2014-11-19 02:04:30,620 INFO [AMRM Callback Handler Thread]
> yarn.GiraphApplicationMaster
> (GiraphApplicationMaster.java:onContainersCompleted(574)) - Got container
> status for containerID=container_1416359131664_0002_01_000002,
> state=COMPLETE, exitStatus=1, diagnostics=Exception from container-launch:
> ExitCodeException exitCode=1:
> ExitCodeException exitCode=1:
> at org.apache.hadoop.util.Shell.runCommand(Shell.java:538)
> at org.apache.hadoop.util.Shell.run(Shell.java:455)
> at
> org.apache.hadoop.util.Shell$ShellCommandExecutor.execute(Shell.java:702)
> at
> org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor.launchContainer(DefaultContainerExecutor.java:195)
> at
> org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:300)
> at
> org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:81)
> at java.util.concurrent.FutureTask.run(FutureTask.java:262)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
> at java.lang.Thread.run(Thread.java:745)
>
>
> Container exited with a non-zero exit code 1
>
> 2014-11-19 02:04:30,620 INFO [AMRM Callback Handler Thread]
> yarn.GiraphApplicationMaster
> (GiraphApplicationMaster.java:onContainersCompleted(574)) - Got container
> status for containerID=container_1416359131664_0002_01_000003,
> state=COMPLETE, exitStatus=1, diagnostics=Exception from container-launch:
> ExitCodeException exitCode=1:
> ExitCodeException exitCode=1:
> at org.apache.hadoop.util.Shell.runCommand(Shell.java:538)
> at org.apache.hadoop.util.Shell.run(Shell.java:455)
> at
> org.apache.hadoop.util.Shell$ShellCommandExecutor.execute(Shell.java:702)
> at
> org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor.launchContainer(DefaultContainerExecutor.java:195)
> at
> org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:300)
> at
> org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:81)
> at java.util.concurrent.FutureTask.run(FutureTask.java:262)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
> at java.lang.Thread.run(Thread.java:745)
>
>
> Container exited with a non-zero exit code 1
>
> 2014-11-19 02:04:30,620 INFO [AMRM Callback Handler Thread]
> yarn.GiraphApplicationMaster
> (GiraphApplicationMaster.java:onContainersCompleted(594)) - All container
> compeleted. done = true
> 2014-11-19 02:04:30,791 INFO [main] yarn.GiraphApplicationMaster
> (GiraphApplicationMaster.java:run(194)) - Done true
> 2014-11-19 02:04:30,791 INFO [main] yarn.GiraphApplicationMaster
> (GiraphApplicationMaster.java:run(198)) - Forcefully terminating executors
> with done =:true
> 2014-11-19 02:04:30,791 INFO [main] yarn.GiraphApplicationMaster
> (GiraphApplicationMaster.java:finish(212)) - Application completed. Stopping
> running containers
> 2014-11-19 02:04:30,803 INFO [main] impl.ContainerManagementProtocolProxy
> (ContainerManagementProtocolProxy.java:mayBeCloseProxy(145)) - Closing proxy
> : roosevelt:58195
> 2014-11-19 02:04:30,803 INFO [main] yarn.GiraphApplicationMaster
> (GiraphApplicationMaster.java:finish(217)) - Application completed.
> Signalling finish to RM
> 2014-11-19 02:04:30,815 INFO [main] impl.AMRMClientImpl
> (AMRMClientImpl.java:unregisterApplicationMaster(321)) - Waiting for
> application to be successfully unregistered.
> 2014-11-19 02:04:30,923 INFO [AMRM Callback Handler Thread]
> impl.AMRMClientAsyncImpl (AMRMClientAsyncImpl.java:run(277)) - Interrupted
> while waiting for queue
> java.lang.InterruptedException
> at
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.reportInterruptAfterWait(AbstractQueuedSynchronizer.java:2017)
> at
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2052)
> at
> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
> at
> org.apache.hadoop.yarn.client.api.async.impl.AMRMClientAsyncImpl$CallbackHandlerThread.run(AMRMClientAsyncImpl.java:275)
> 2014-11-19 02:04:30,944 INFO [main] yarn.GiraphApplicationMaster
> (GiraphApplicationMaster.java:main(445)) - Giraph Application Master failed.
> exiting
>
> In a moment, I will paste all the logs I can find.
>
> On Sun, Nov 23, 2014 at 1:11 AM, Tripti Singh <tri...@yahoo-inc.com>
> wrote:
>
>> Sorry, the e-mail was sent without me being able to complete it.
>> The issue I was facing was due to a Giraph’s hard coded dependency on a
>> task with id 0. This task is the designated Master responsible for creating
>> a zookeeperServerList.
>> These task ids are assigned using ContainerId as input and with the
>> assumption that we will be assigned containers starting with id of 2 (1 for
>> Application Master)
>> Following is the comment available in Giraph Yarn class
>>
>> /**
>>
>> * Utility to create a TaskAttemptId we can feed to our fake
>> Mapper#Context.
>>
>> *
>>
>> * NOTE: ContainerId will serve as MR TaskID for Giraph tasks.
>>
>> * YARN container 1 is always AppMaster, so the least container id we
>> will
>>
>> * ever get from YARN for a Giraph task is container id 2. Giraph on
>> MapReduce
>>
>> * tasks must start at index 0. So we SUBTRACT TWO from each container
>> id.
>>
>> *
>>
>> * @param args the command line args, fed to us by
>> GiraphApplicationMaster.
>>
>> * @return the TaskAttemptId object, populated with YARN job data.
>>
>> */
>>
>>
>> Now, my problem was that in our cluster, this guarantee was not met and
>> at times, I was not assigned the Container with id of 2 (And container ids
>> are also not assigned in increasing order).
>>
>> Due to this, the Zookeeper was not starting in many workflow runs.
>>
>> The fix I did, was to get rid of this issue.
>>
>>
>> Thanks,
>> Tripti
>> From: Tripti Singh <tri...@yahoo-inc.com>
>> Reply-To: "user@giraph.apache.org" <user@giraph.apache.org>
>> Date: Sunday, November 23, 2014 at 12:23 PM
>>
>> To: "user@giraph.apache.org" <user@giraph.apache.org>
>> Subject: Re: Help with Giraph on Yarn
>>
>> Hi Das,
>> Could u try what Alessandro is suggesting. Giraph-example jar has all the
>> classes required..
>> Also, you have shared only AM logs. Can u also add the worker logs?
>> From a quick look at the AM, I’m not sure if this is the same problem
>> that I faced.
>>
>> Thanks,
>> Tripti
>> From: Alessandro Negro <alenegr...@yahoo.it>
>> Reply-To: "user@giraph.apache.org" <user@giraph.apache.org>
>> Date: Sunday, November 23, 2014 at 5:10 AM
>> To: "user@giraph.apache.org" <user@giraph.apache.org>
>> Subject: Re: Help with Giraph on Yarn
>>
>> HI Das,
>> what about the hadoop logs? Have a look there and let me know.
>> In the meantime you can remove redundant jars from yarn jars (exampe
>> contains all jars needed):
>>
>> yarnjars=giraph-examples-1.1.0-hadoop-2.5.1-jar-with-dependencies.jar
>>
>>
>> Should be enough
>>
>> Alessandro.
>>
>> Il giorno 23/nov/2014, alle ore 00:16, D Adams <dadam...@gmail.com> ha
>> scritto:
>>
>> Hi Alessandro,
>> Do you run tyarn that way because your hadoop or giraph file
>> permissions are chowned by a user named yarn?
>>
>> I have made changes as suggested on
>> http://mail-archives.apache.org/mod_mbox/giraph-user/201408.mbox/%3c53f4c689.5060...@web.de%3E
>>
>> And although things do look more hopeful, I still get an infinite loop.
>> I run with as script which includes:
>>
>> user_dir=/user/hduser
>> jar=giraph-examples-1.1.0-hadoop-2.5.1-jar-with-dependencies.jar
>> runner=org.apache.giraph.GiraphRunner
>> computation=org.apache.giraph.examples.SimpleShortestPathsComputation
>>
>> informat=org.apache.giraph.io.formats.JsonLongDoubleFloatDoubleVertexInputFormat
>> outformat=org.apache.giraph.io.formats.IdWithValueTextOutputFormat
>>
>> yarnjars=giraph-examples-1.1.0-hadoop-2.5.1-jar-with-dependencies.jar,giraph-core-1.1.0-SNAPSHOT.jar,giraph-1.1.0-hadoop-2.5.1.jar
>>
>> bin/hdfs dfs -rm -r $user_dir/output/shortestpaths
>> bin/yarn jar $GIRAPH_HOME/$jar $runner -Dgiraph.yarn.task.heap.mb=3000
>> $computation -vif $informat -vip $user_dir/input/tiny_graph.txt -vof
>> $outformat -op $user_dir/output/shortestpaths -w 1 -yj $yarnjars
>>
>> And the resulting stdout:
>> hduser@Roosevelt:/usr/local/hadoop$ ./hduser_jobs.sh
>> Running hduser script ...
>> 14/11/18 19:05:47 WARN util.NativeCodeLoader: Unable to load
>> native-hadoop library for your platform... using builtin-java classes where
>> applicable
>> rm: `/user/hduser/output/shortestpaths': No such file or directory
>> 14/11/18 19:06:13 WARN util.NativeCodeLoader: Unable to load
>> native-hadoop library for your platform... using builtin-java classes where
>> applicable
>> 14/11/18 19:06:15 INFO utils.ConfigurationUtils: No edge input format
>> specified. Ensure your InputFormat does not require one.
>> 14/11/18 19:06:15 INFO utils.ConfigurationUtils: No edge output format
>> specified. Ensure your OutputFormat does not require one.
>> 14/11/18 19:06:16 INFO yarn.GiraphYarnClient: Final output path is:
>> hdfs://localhost:9000/user/hduser/output/shortestpaths
>> 14/11/18 19:06:16 INFO yarn.GiraphYarnClient: Running Client
>> 14/11/18 19:06:16 INFO client.RMProxy: Connecting to ResourceManager at
>> localhost/127.0.0.1:8050
>> 14/11/18 19:06:17 INFO yarn.GiraphYarnClient: Got node report from ASM
>> for, nodeId=roosevelt:58195, nodeAddress roosevelt:8042, nodeRackName
>> /default-rack, nodeNumContainers 0
>> 14/11/18 19:06:17 INFO yarn.GiraphYarnClient: Obtained new Application
>> ID: application_1416359131664_0001
>> 14/11/18 19:06:17 INFO Configuration.deprecation: mapred.job.id is
>> deprecated. Instead, use mapreduce.job.id
>> 14/11/18 19:06:17 INFO yarn.GiraphYarnClient: Set the environment for the
>> application master
>> 14/11/18 19:06:17 INFO yarn.GiraphYarnClient: Environment for AM
>> :{CLASSPATH=${CLASSPATH}:./*:$HADOOP_CONF_DIR:$HADOOP_COMMON_HOME/share/hadoop/common/*:$HADOOP_COMMON_HOME/share/hadoop/common/lib/*:$HADOOP_HDFS_HOME/share/hadoop/hdfs/*:$HADOOP_HDFS_HOME/share/hadoop/hdfs/lib/*:$HADOOP_YARN_HOME/share/hadoop/yarn/*:$HADOOP_YARN_HOME/share/hadoop/yarn/lib/*:$HADOOP_MAPRED_HOME/share/hadoop/mapreduce/*:$HADOOP_MAPRED_HOME/share/hadoop/mapreduce/lib/*}
>> 14/11/18 19:06:17 INFO yarn.GiraphYarnClient: buildLocalResourceMap ....
>> 14/11/18 19:06:19 INFO yarn.YarnUtils: Registered file in LocalResources
>> ::
>> hdfs://localhost:9000/user/hduser/giraph_yarn_jar_cache/application_1416359131664_0001/giraph-conf.xml
>> 14/11/18 19:06:19 INFO yarn.GiraphYarnClient: LIB JARS
>> :giraph-examples-1.1.0-hadoop-2.5.1-jar-with-dependencies.jar,giraph-core-1.1.0-SNAPSHOT.jar,giraph-1.1.0-hadoop-2.5.1.jar
>> 14/11/18 19:06:19 INFO yarn.YarnUtils: Class path name .
>> 14/11/18 19:06:19 INFO yarn.YarnUtils: base path checking .
>> 14/11/18 19:06:19 INFO yarn.YarnUtils: Class path name /usr/local/hadoop
>> 14/11/18 19:06:19 INFO yarn.YarnUtils: base path checking
>> /usr/local/hadoop
>> 14/11/18 19:06:19 INFO yarn.YarnUtils: Class path name null
>> 14/11/18 19:06:19 INFO yarn.YarnUtils: base path checking null
>> 14/11/18 19:06:19 INFO yarn.GiraphYarnClient: Made local resource for
>> :/usr/local/hadoop/share/myLib/giraph-examples-1.1.0-hadoop-2.5.1-jar-with-dependencies.jar
>> to
>> hdfs://localhost:9000/user/hduser/giraph_yarn_jar_cache/application_1416359131664_0001/giraph-examples-1.1.0-hadoop-2.5.1-jar-with-dependencies.jar
>> 14/11/18 19:06:23 INFO yarn.YarnUtils: Registered file in LocalResources
>> ::
>> hdfs://localhost:9000/user/hduser/giraph_yarn_jar_cache/application_1416359131664_0001/giraph-examples-1.1.0-hadoop-2.5.1-jar-with-dependencies.jar
>> 14/11/18 19:06:23 INFO yarn.GiraphYarnClient:
>> ApplicationSumbissionContext for GiraphApplicationMaster launch container
>> is populated.
>> 14/11/18 19:06:23 INFO yarn.GiraphYarnClient: Submitting application to
>> ASM
>> 14/11/18 19:06:23 INFO impl.YarnClientImpl: Submitted application
>> application_1416359131664_0001
>> 14/11/18 19:06:23 INFO yarn.GiraphYarnClient: Got new appId after
>> submission :application_1416359131664_0001
>> 14/11/18 19:06:23 INFO yarn.GiraphYarnClient: GiraphApplicationMaster
>> container request was submitted to ResourceManager for job: Giraph:
>> org.apache.giraph.examples.SimpleShortestPathsComputation
>> 14/11/18 19:06:24 INFO yarn.GiraphYarnClient: Giraph:
>> org.apache.giraph.examples.SimpleShortestPathsComputation, Elapsed: 0.90
>> secs
>> 14/11/18 19:06:24 INFO yarn.GiraphYarnClient:
>> appattempt_1416359131664_0001_000001, State: ACCEPTED, Containers used: 0
>> 14/11/18 19:06:28 INFO yarn.GiraphYarnClient: Giraph:
>> org.apache.giraph.examples.SimpleShortestPathsComputation, Elapsed: 5.05
>> secs
>> 14/11/18 19:06:28 INFO yarn.GiraphYarnClient:
>> appattempt_1416359131664_0001_000001, State: ACCEPTED, Containers used: 1
>> 14/11/18 19:06:33 INFO yarn.GiraphYarnClient: Giraph:
>> org.apache.giraph.examples.SimpleShortestPathsComputation, Elapsed: 9.89
>> secs
>> 14/11/18 19:06:33 INFO yarn.GiraphYarnClient:
>> appattempt_1416359131664_0001_000001, State: ACCEPTED, Containers used: 1
>> 14/11/18 19:06:37 INFO yarn.GiraphYarnClient: Giraph:
>> org.apache.giraph.examples.SimpleShortestPathsComputation, Elapsed: 14.37
>> secs
>> 14/11/18 19:06:37 INFO yarn.GiraphYarnClient:
>> appattempt_1416359131664_0001_000001, State: ACCEPTED, Containers used: 1
>> 14/11/18 19:06:42 INFO yarn.GiraphYarnClient: Giraph:
>> org.apache.giraph.examples.SimpleShortestPathsComputation, Elapsed: 18.81
>> secs
>> 14/11/18 19:06:42 INFO yarn.GiraphYarnClient:
>> appattempt_1416359131664_0001_000001, State: ACCEPTED, Containers used: 1
>> 14/11/18 19:06:46 INFO yarn.GiraphYarnClient: Giraph:
>> org.apache.giraph.examples.SimpleShortestPathsComputation, Elapsed: 22.82
>> secs
>> 14/11/18 19:06:46 INFO yarn.GiraphYarnClient:
>> appattempt_1416359131664_0001_000001, State: RUNNING, Containers used: 2
>> 14/11/18 19:06:50 INFO yarn.GiraphYarnClient: Giraph:
>> org.apache.giraph.examples.SimpleShortestPathsComputation, Elapsed: 26.83
>> secs
>> 14/11/18 19:06:50 INFO yarn.GiraphYarnClient:
>> appattempt_1416359131664_0001_000001, State: RUNNING, Containers used: 2
>> 14/11/18 19:06:54 INFO yarn.GiraphYarnClient: Giraph:
>> org.apache.giraph.examples.SimpleShortestPathsComputation, Elapsed: 30.83
>> secs
>> 14/11/18 19:06:54 INFO yarn.GiraphYarnClient:
>> appattempt_1416359131664_0001_000001, State: RUNNING, Containers used: 2
>> 14/11/18 19:06:58 INFO yarn.GiraphYarnClient: Giraph:
>> org.apache.giraph.examples.SimpleShortestPathsComputation, Elapsed: 34.85
>> secs
>> 14/11/18 19:06:58 INFO yarn.GiraphYarnClient:
>> appattempt_1416359131664_0001_000001, State: RUNNING, Containers used: 2
>> 14/11/18 19:07:02 INFO yarn.GiraphYarnClient: Giraph:
>> org.apache.giraph.examples.SimpleShortestPathsComputation, Elapsed: 38.85
>> secs
>> 14/11/18 19:07:02 INFO yarn.GiraphYarnClient:
>> appattempt_1416359131664_0001_000001, State: RUNNING, Containers used: 2
>> 14/11/18 19:07:06 INFO yarn.GiraphYarnClient: Giraph:
>> org.apache.giraph.examples.SimpleShortestPathsComputation, Elapsed: 42.86
>> secs
>> 14/11/18 19:07:06 INFO yarn.GiraphYarnClient:
>> appattempt_1416359131664_0001_000001, State: RUNNING, Containers used: 2
>> 14/11/18 19:07:10 INFO yarn.GiraphYarnClient: Giraph:
>> org.apache.giraph.examples.SimpleShortestPathsComputation, Elapsed: 46.88
>> secs
>> 14/11/18 19:07:10 INFO yarn.GiraphYarnClient:
>> appattempt_1416359131664_0001_000001, State: RUNNING, Containers used: 2
>> 14/11/18 19:07:14 INFO yarn.GiraphYarnClient: Giraph:
>> org.apache.giraph.examples.SimpleShortestPathsComputation, Elapsed: 50.89
>> secs
>> 14/11/18 19:07:14 INFO yarn.GiraphYarnClient:
>> appattempt_1416359131664_0001_000001, State: RUNNING, Containers used: 2
>> 14/11/18 19:07:18 INFO yarn.GiraphYarnClient: Giraph:
>> org.apache.giraph.examples.SimpleShortestPathsComputation, Elapsed: 54.90
>> secs
>>
>> Once yarn hits the RUNNING state, it enters an endless loop. It seems it
>> is caused by the following:
>>
>>
>> 2014-11-18 19:06:37,382 INFO [main] yarn.GiraphApplicationMaster
>> (GiraphApplicationMaster.java:main(421)) - Starting GitaphAM
>> 2014-11-18 19:06:39,214 WARN [main] util.NativeCodeLoader
>> (NativeCodeLoader.java:<clinit>(62)) - Unable to load native-hadoop library
>> for your platform... using builtin-java classes where applicable
>> 2014-11-18 19:06:41,438 INFO [main] yarn.GiraphApplicationMaster
>> (GiraphApplicationMaster.java:<init>(168)) - GiraphAM for ContainerId
>> container_1416359131664_0001_01_000001 ApplicationAttemptId
>> appattempt_1416359131664_0001_000001
>> 2014-11-18 19:06:41,497 INFO [main] client.RMProxy
>> (RMProxy.java:createRMProxy(92)) - Connecting to ResourceManager at
>> localhost/127.0.0.1:8030
>> 2014-11-18 19:06:41,581 INFO [main] impl.NMClientAsyncImpl
>> (NMClientAsyncImpl.java:serviceInit(107)) - Upper bound of the thread pool
>> size is 500
>> 2014-11-18 19:06:41,583 INFO [main] impl.ContainerManagementProtocolProxy
>> (ContainerManagementProtocolProxy.java:<init>(78)) -
>> yarn.client.max-nodemanagers-proxies : 500
>> 2014-11-18 19:06:42,355 INFO [main] yarn.GiraphApplicationMaster
>> (GiraphApplicationMaster.java:setupContainerAskForRM(279)) - Requested
>> container ask: Capability[<memory:3000, vCores:0>]Priority[10]
>> 2014-11-18 19:06:42,370 INFO [main] yarn.GiraphApplicationMaster
>> (GiraphApplicationMaster.java:setupContainerAskForRM(279)) - Requested
>> container ask: Capability[<memory:3000, vCores:0>]Priority[10]
>> 2014-11-18 19:06:42,618 INFO [main] yarn.GiraphApplicationMaster
>> (GiraphApplicationMaster.java:run(185)) - Wait to finish ..
>> 2014-11-18 19:06:44,673 INFO [AMRM Heartbeater thread] impl.AMRMClientImpl
>> (AMRMClientImpl.java:populateNMTokens(299)) - Received new token for :
>> roosevelt:58195
>> 2014-11-18 19:06:44,677 INFO [AMRM Callback Handler Thread]
>> yarn.GiraphApplicationMaster
>> (GiraphApplicationMaster.java:onContainersAllocated(605)) - Got response
>> from RM for container ask, allocatedCnt=1
>> 2014-11-18 19:06:44,677 INFO [AMRM Callback Handler Thread]
>> yarn.GiraphApplicationMaster
>> (GiraphApplicationMaster.java:onContainersAllocated(608)) - Total allocated
>> # of container so far : 1 allocated out of 2 required.
>> 2014-11-18 19:06:44,678 INFO [AMRM Callback Handler Thread]
>> yarn.GiraphApplicationMaster
>> (GiraphApplicationMaster.java:startContainerLaunchingThreads(359)) -
>> Launching command on a new container.,
>> containerId=container_1416359131664_0001_01_000002,
>> containerNode=roosevelt:58195, containerNodeURI=roosevelt:8042,
>> containerResourceMemory=3072
>> 2014-11-18 19:06:44,679 INFO [pool-4-thread-1] yarn.GiraphApplicationMaster
>> (GiraphApplicationMaster.java:buildContainerLaunchContext(492)) - Setting up
>> container launch container for
>> containerid=container_1416359131664_0001_01_000002
>> 2014-11-18 19:06:44,694 INFO [pool-4-thread-1] yarn.GiraphApplicationMaster
>> (GiraphApplicationMaster.java:buildContainerLaunchContext(498)) - Conatain
>> launch Commands :java -Xmx3000M -Xms3000M -cp .:${CLASSPATH}
>> org.apache.giraph.yarn.GiraphYarnTask 1416359131664 1 2 1
>> 1><LOG_DIR>/task-2-stdout.log 2><LOG_DIR>/task-2-stderr.log
>> 2014-11-18 19:06:44,694 INFO [pool-4-thread-1] yarn.GiraphApplicationMaster
>> (GiraphApplicationMaster.java:buildContainerLaunchContext(518)) - Setting
>> username in ContainerLaunchContext to: hduser
>> 2014-11-18 19:06:45,191 INFO [pool-4-thread-1] yarn.YarnUtils
>> (YarnUtils.java:addFsResourcesToMap(72)) - Adding
>> giraph-examples-1.1.0-hadoop-2.5.1-jar-with-dependencies.jar to
>> LocalResources for export.to
>> hdfs://localhost:9000/user/hduser/giraph_yarn_jar_cache/application_1416359131664_0001/giraph-examples-1.1.0-hadoop-2.5.1-jar-with-dependencies.jar
>> 2014-11-18 19:06:45,228 INFO [pool-4-thread-1] yarn.YarnUtils
>> (YarnUtils.java:addFileToResourceMap(160)) - Registered file in
>> LocalResources ::
>> hdfs://localhost:9000/user/hduser/giraph_yarn_jar_cache/application_1416359131664_0001/giraph-examples-1.1.0-hadoop-2.5.1-jar-with-dependencies.jar
>> 2014-11-18 19:06:45,228 INFO [pool-4-thread-1] yarn.YarnUtils
>> (YarnUtils.java:addFsResourcesToMap(72)) - Adding
>> giraph-core-1.1.0-SNAPSHOT.jar to LocalResources for export.to
>> hdfs://localhost:9000/user/hduser/giraph_yarn_jar_cache/application_1416359131664_0001/giraph-core-1.1.0-SNAPSHOT.jar
>> 2014-11-18 19:06:45,683 INFO [AMRM Callback Handler Thread]
>> yarn.GiraphApplicationMaster
>> (GiraphApplicationMaster.java:onContainersAllocated(605)) - Got response
>> from RM for container ask, allocatedCnt=1
>> 2014-11-18 19:06:45,683 INFO [AMRM Callback Handler Thread]
>> yarn.GiraphApplicationMaster
>> (GiraphApplicationMaster.java:onContainersAllocated(608)) - Total allocated
>> # of container so far : 2 allocated out of 2 required.
>> 2014-11-18 19:06:45,683 INFO [AMRM Callback Handler Thread]
>> yarn.GiraphApplicationMaster
>> (GiraphApplicationMaster.java:startContainerLaunchingThreads(359)) -
>> Launching command on a new container.,
>> containerId=container_1416359131664_0001_01_000003,
>> containerNode=roosevelt:58195, containerNodeURI=roosevelt:8042,
>> containerResourceMemory=3072
>> 2014-11-18 19:06:45,684 INFO [pool-4-thread-3] yarn.GiraphApplicationMaster
>> (GiraphApplicationMaster.java:buildContainerLaunchContext(492)) - Setting up
>> container launch container for
>> containerid=container_1416359131664_0001_01_000003
>> 2014-11-18 19:06:45,684 INFO [pool-4-thread-3] yarn.GiraphApplicationMaster
>> (GiraphApplicationMaster.java:buildContainerLaunchContext(498)) - Conatain
>> launch Commands :java -Xmx3000M -Xms3000M -cp .:${CLASSPATH}
>> org.apache.giraph.yarn.GiraphYarnTask 1416359131664 1 3 1
>> 1><LOG_DIR>/task-3-stdout.log 2><LOG_DIR>/task-3-stderr.log
>> 2014-11-18 19:06:45,684 INFO [pool-4-thread-3] yarn.GiraphApplicationMaster
>> (GiraphApplicationMaster.java:buildContainerLaunchContext(518)) - Setting
>> username in ContainerLaunchContext to: hduser
>> 2014-11-18 19:06:45,697 INFO
>> [org.apache.hadoop.yarn.client.api.async.impl.NMClientAsyncImpl #0]
>> impl.NMClientAsyncImpl (NMClientAsyncImpl.java:run(531)) - Processing Event
>> EventType: START_CONTAINER for Container
>> container_1416359131664_0001_01_000003
>> 2014-11-18 19:06:45,699 INFO
>> [org.apache.hadoop.yarn.client.api.async.impl.NMClientAsyncImpl #0]
>> impl.ContainerManagementProtocolProxy
>> (ContainerManagementProtocolProxy.java:newProxy(212)) - Opening proxy :
>> roosevelt:58195
>> 2014-11-18 19:06:45,761 INFO
>> [org.apache.hadoop.yarn.client.api.async.impl.NMClientAsyncImpl #1]
>> impl.NMClientAsyncImpl (NMClientAsyncImpl.java:run(531)) - Processing Event
>> EventType: QUERY_CONTAINER for Container
>> container_1416359131664_0001_01_000003
>> 2014-11-18 19:06:46,687 INFO [AMRM Callback Handler Thread]
>> yarn.GiraphApplicationMaster
>> (GiraphApplicationMaster.java:onContainersCompleted(571)) - Got response
>> from RM for container ask, completedCnt=1
>> 2014-11-18 19:06:46,688 INFO [AMRM Callback Handler Thread]
>> yarn.GiraphApplicationMaster
>> (GiraphApplicationMaster.java:onContainersCompleted(574)) - Got container
>> status for containerID=container_1416359131664_0001_01_000003,
>> state=COMPLETE, exitStatus=1, diagnostics=Exception from container-launch:
>> ExitCodeException exitCode=1:
>> ExitCodeException exitCode=1:
>> at org.apache.hadoop.util.Shell.runCommand(Shell.java:538)
>> at org.apache.hadoop.util.Shell.run(Shell.java:455)
>> at
>> org.apache.hadoop.util.Shell$ShellCommandExecutor.execute(Shell.java:702)
>> at
>> org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor.launchContainer(DefaultContainerExecutor.java:195)
>> at
>> org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:300)
>> at
>> org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:81)
>> at java.util.concurrent.FutureTask.run(FutureTask.java:262)
>> at
>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
>> at
>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
>> at java.lang.Thread.run(Thread.java:745)
>>
>>
>> Container exited with a non-zero exit code 1
>>
>> 2014-11-18 19:06:46,688 INFO [AMRM Callback Handler Thread]
>> yarn.GiraphApplicationMaster
>> (GiraphApplicationMaster.java:onContainersCompleted(596)) - After completion
>> of one conatiner. current status is: completedCount :1 containersToLaunch :2
>> successfulCount :0 failedCount :1
>>
>>
>> On Sat, Nov 22, 2014 at 11:42 AM, D Adams <dadam...@gmail.com> wrote:
>>
>>> I'm sorry, please excuse my ignorance, but, I have no idea where to
>>> find my configuration. That is, I have no idea where these lines came from
>>> (or what file):
>>>
>>> yarn 1029 0,1 1,2 3499572 209560 s001 S 10:29am
>>> 0:06.04 /Library/Java/
>>> JavaVirtualMachines/jdk1.7.0_45.jdk/Contents/Home/bin/java
>>> -Dproc_resourcemanager -Xmx500m
>>> -Dhadoop.log.dir=/opt/yarn/hadoop-2.5.1/logs
>>> -Dyarn.log.dir=/opt/yarn/hadoop-2.5.1/logs
>>> ...
>>> As far as configuration goes, I'm only really aware of the
>>> etc/hadoop/*-env.sh and .xml files and main pom.xml file. Or is the above
>>> part of the command you use to run giraph jobs?
>>> I will make changes to hadoop-env.sh file as you suggest and let you
>>> know how it works out.
>>>
>>> On Fri, Nov 21, 2014 at 6:05 AM, Alessandro Negro <alenegr...@yahoo.it>
>>> wrote:
>>>
>>>> Hi,
>>>> I forgot to say that I added also this line:
>>>>
>>>> export HADOOP_CLASSPATH=$HADOOP_CLASSPATH:$MYLIB
>>>>
>>>>
>>>> Il giorno 21/nov/2014, alle ore 10:34, Alessandro Negro <
>>>> alenegr...@yahoo.it> ha scritto:
>>>>
>>>> Hi Das,
>>>> this is my configuration of hadoop:
>>>>
>>>> yarn 1029 0,1 1,2 3499572 209560 s001 S 10:29am
>>>> 0:06.04
>>>> /Library/Java/JavaVirtualMachines/jdk1.7.0_45.jdk/Contents/Home/bin/java
>>>> -Dproc_resourcemanager -Xmx500m
>>>> -Dhadoop.log.dir=/opt/yarn/hadoop-2.5.1/logs
>>>> -Dyarn.log.dir=/opt/yarn/hadoop-2.5.1/logs
>>>> -Dhadoop.log.file=yarn-yarn-resourcemanager-MacBook-Pro-di-Alessandro.local.log
>>>> -Dyarn.log.file=yarn-yarn-resourcemanager-MacBook-Pro-di-Alessandro.local.log
>>>> -Dyarn.home.dir= -Dyarn.id.str=yarn -Dhadoop.root.logger=INFO,RFA
>>>> -Dyarn.root.logger=INFO,RFA
>>>> -Djava.library.path=/opt/yarn/hadoop-2.5.1/lib/native
>>>> -Dyarn.policy.file=hadoop-policy.xml
>>>> -Dhadoop.log.dir=/opt/yarn/hadoop-2.5.1/logs
>>>> -Dyarn.log.dir=/opt/yarn/hadoop-2.5.1/logs
>>>> -Dhadoop.log.file=yarn-yarn-resourcemanager-MacBook-Pro-di-Alessandro.local.log
>>>> -Dyarn.log.file=yarn-yarn-resourcemanager-MacBook-Pro-di-Alessandro.local.log
>>>> -Dyarn.home.dir=/opt/yarn/hadoop-2.5.1
>>>> -Dhadoop.home.dir=/opt/yarn/hadoop-2.5.1 -Dhadoop.root.logger=INFO,RFA
>>>> -Dyarn.root.logger=INFO,RFA
>>>> -Djava.library.path=/opt/yarn/hadoop-2.5.1/lib/native -classpath
>>>> /opt/yarn/hadoop-2.5.1/etc/hadoop:/opt/yarn/hadoop-2.5.1/etc/hadoop:/opt/yarn/hadoop-2.5.1/etc/hadoop:/opt/yarn/hadoop-2.5.1/share/hadoop/common/lib/*:/opt/yarn/hadoop-2.5.1/share/hadoop/common/*:/opt/yarn/hadoop-2.5.1/share/hadoop/hdfs:/opt/yarn/hadoop-2.5.1/share/hadoop/hdfs/lib/*:/opt/yarn/hadoop-2.5.1/share/hadoop/hdfs/*:/opt/yarn/hadoop-2.5.1/share/hadoop/yarn/lib/*:/opt/yarn/hadoop-2.5.1/share/hadoop/yarn/*:/opt/yarn/hadoop-2.5.1/share/hadoop/mapreduce/lib/*:/opt/yarn/hadoop-2.5.1/share/hadoop/mapreduce/*::/opt/yarn/hadoop-2.5.1/share/myLib/*.jar:/Users/ale/extprj/giraph/myGiraph/*.jar:/Users/ale/extprj/giraph/myGiraph/lib/*.jar:/contrib/capacity-scheduler/*.jar:/opt/yarn/hadoop-2.5.1/share/myLib/*.jar:/Users/ale/extprj/giraph/myGiraph/*.jar:/Users/ale/extprj/giraph/myGiraph/lib/*.jar:/contrib/capacity-scheduler/*.jar:/opt/yarn/hadoop-2.5.1/share/hadoop/yarn/*:/opt/yarn/hadoop-2.5.1/share/hadoop/yarn/lib/*:/opt/yarn/hadoop-2.5.1/etc/hadoop/rm-config/log4j.properties
>>>> org.apache.hadoop.yarn.server.resourcemanager.ResourceManager
>>>> yarn 1073 0,1 1,0 3347132 174644 s001 S 10:29am
>>>> 0:04.77
>>>> /Library/Java/JavaVirtualMachines/jdk1.7.0_45.jdk/Contents/Home/bin/java
>>>> -Dproc_nodemanager -Xmx500m -Dhadoop.log.dir=/opt/yarn/hadoop-2.5.1/logs
>>>> -Dyarn.log.dir=/opt/yarn/hadoop-2.5.1/logs
>>>> -Dhadoop.log.file=yarn-yarn-nodemanager-MacBook-Pro-di-Alessandro.local.log
>>>> -Dyarn.log.file=yarn-yarn-nodemanager-MacBook-Pro-di-Alessandro.local.log
>>>> -Dyarn.home.dir= -Dyarn.id.str=yarn -Dhadoop.root.logger=INFO,RFA
>>>> -Dyarn.root.logger=INFO,RFA
>>>> -Djava.library.path=/opt/yarn/hadoop-2.5.1/lib/native
>>>> -Dyarn.policy.file=hadoop-policy.xml -server
>>>> -Dhadoop.log.dir=/opt/yarn/hadoop-2.5.1/logs
>>>> -Dyarn.log.dir=/opt/yarn/hadoop-2.5.1/logs
>>>> -Dhadoop.log.file=yarn-yarn-nodemanager-MacBook-Pro-di-Alessandro.local.log
>>>> -Dyarn.log.file=yarn-yarn-nodemanager-MacBook-Pro-di-Alessandro.local.log
>>>> -Dyarn.home.dir=/opt/yarn/hadoop-2.5.1
>>>> -Dhadoop.home.dir=/opt/yarn/hadoop-2.5.1 -Dhadoop.root.logger=INFO,RFA
>>>> -Dyarn.root.logger=INFO,RFA
>>>> -Djava.library.path=/opt/yarn/hadoop-2.5.1/lib/native -classpath
>>>> /opt/yarn/hadoop-2.5.1/etc/hadoop:/opt/yarn/hadoop-2.5.1/etc/hadoop:/opt/yarn/hadoop-2.5.1/etc/hadoop:/opt/yarn/hadoop-2.5.1/share/hadoop/common/lib/*:/opt/yarn/hadoop-2.5.1/share/hadoop/common/*:/opt/yarn/hadoop-2.5.1/share/hadoop/hdfs:/opt/yarn/hadoop-2.5.1/share/hadoop/hdfs/lib/*:/opt/yarn/hadoop-2.5.1/share/hadoop/hdfs/*:/opt/yarn/hadoop-2.5.1/share/hadoop/yarn/lib/*:/opt/yarn/hadoop-2.5.1/share/hadoop/yarn/*:/opt/yarn/hadoop-2.5.1/share/hadoop/mapreduce/lib/*:/opt/yarn/hadoop-2.5.1/share/hadoop/mapreduce/*::/opt/yarn/hadoop-2.5.1/share/myLib/*.jar:/Users/ale/extprj/giraph/myGiraph/*.jar:/Users/ale/extprj/giraph/myGiraph/lib/*.jar:/contrib/capacity-scheduler/*.jar:/opt/yarn/hadoop-2.5.1/share/myLib/*.jar:/Users/ale/extprj/giraph/myGiraph/*.jar:/Users/ale/extprj/giraph/myGiraph/lib/*.jar:/contrib/capacity-scheduler/*.jar:/opt/yarn/hadoop-2.5.1/share/hadoop/yarn/*:/opt/yarn/hadoop-2.5.1/share/hadoop/yarn/lib/*:/opt/yarn/hadoop-2.5.1/etc/hadoop/nm-config/log4j.properties
>>>> org.apache.hadoop.yarn.server.nodemanager.NodeManager
>>>>
>>>> So I run it as yarn user in this way:
>>>>
>>>> sudo -u yarn ….
>>>>
>>>> Moreover I have just notice that I added the following lines into the
>>>> hadoop-env.sh
>>>>
>>>> hadoop-env.sh:export
>>>> MYLIB=/opt/yarn/hadoop-2.5.1/share/myLib/*.jar:/Users/ale/extprj/giraph/myGiraph/*.jar:/Users/ale/extprj/giraph/myGiraph/lib/*.jar
>>>>
>>>> Accordingly to this other thread:
>>>>
>>>>
>>>> http://mail-archives.apache.org/mod_mbox/giraph-user/201408.mbox/%3c53f4c689.5060...@web.de%3E
>>>>
>>>> Let me know,
>>>> Alessandro
>>>>
>>>>
>>>> Il giorno 20/nov/2014, alle ore 23:51, D Adams <dadam...@gmail.com>
>>>> ha scritto:
>>>>
>>>> --f46d044283ecf837440508522939
>>>> Content-Type: text/plain; charset=UTF-8
>>>> Content-Transfer-Encoding: quoted-printable
>>>>
>>>> Ok, in the following, I use bin/yarn. Doesn't seem like much has
>>>> changed:
>>>>
>>>> hduser@Roosevelt:/usr/local/hadoop$ ./hduser_jobs.sh
>>>> Running hduser script ...
>>>> bin/yarn jar
>>>>
>>>> /usr/local/giraph/giraph-examples/target/giraph-examples-1.1.0-SNAPSHOT-for=
>>>> -hadoop-2.5.1-jar-with-dependencies.jar
>>>> org.apache.giraph.GiraphRunner
>>>> org.apache.giraph.examples.SimpleShortestPathsComputation -vif
>>>> org.apache.giraph.io.formats.JsonLongDoubleFloatDoubleVertexInputFormat
>>>> -vip /user/hduser/input/tiny_graph.txt -vof
>>>> org.apache.giraph.io.formats.IdWithValueTextOutputFormat -op
>>>> /user/hduser/output/shortestpaths -w 1 -yj
>>>>
>>>> giraph-examples-1.1.0-SNAPSHOT-for-hadoop-2.5.1-jar-with-dependencies.jar
>>>> 14/11/18 05:36:05 WARN util.NativeCodeLoader: Unable to load
>>>> native-hadoop
>>>> library for your platform... using builtin-java classes where applicable
>>>> 14/11/18 05:36:08 INFO utils.ConfigurationUtils: No edge input format
>>>> specified. Ensure your InputFormat does not require one.
>>>> 14/11/18 05:36:08 INFO utils.ConfigurationUtils: No edge output format
>>>> specified. Ensure your OutputFormat does not require one.
>>>> 14/11/18 05:36:08 INFO yarn.GiraphYarnClient: Final output path is:
>>>> hdfs://localhost:9000/user/hduser/output/shortestpaths
>>>> 14/11/18 05:36:08 INFO yarn.GiraphYarnClient: Running Client
>>>> 14/11/18 05:36:08 INFO client.RMProxy: Connecting to ResourceManager at
>>>> localhost/127.0.0.1:8050
>>>> 14/11/18 05:36:09 INFO yarn.GiraphYarnClient: Got node report from ASM
>>>> for,
>>>> nodeId=3Droosevelt:60486, nodeAddress roosevelt:8042, nodeRackName
>>>> /default-rack, nodeNumContainers 0
>>>> 14/11/18 05:36:09 INFO yarn.GiraphYarnClient: Defaulting per-task heap
>>>> size
>>>> to 1024MB.
>>>> 14/11/18 05:36:09 INFO yarn.GiraphYarnClient: Obtained new Application
>>>> ID:
>>>> application_1416310496070_0001
>>>> 14/11/18 05:36:09 INFO Configuration.deprecation: mapred.job.id is
>>>> deprecated. Instead, use mapreduce.job.id
>>>> 14/11/18 05:36:09 INFO yarn.GiraphYarnClient: Set the environment for
>>>> the
>>>> application master
>>>> 14/11/18 05:36:09 INFO yarn.GiraphYarnClient: Environment for AM
>>>>
>>>> :{CLASSPATH=3D${CLASSPATH}:./*:$HADOOP_CONF_DIR:$HADOOP_COMMON_HOME/share/h=
>>>>
>>>> adoop/common/*:$HADOOP_COMMON_HOME/share/hadoop/common/lib/*:$HADOOP_HDFS_H=
>>>>
>>>> OME/share/hadoop/hdfs/*:$HADOOP_HDFS_HOME/share/hadoop/hdfs/lib/*:$HADOOP_Y=
>>>>
>>>> ARN_HOME/share/hadoop/yarn/*:$HADOOP_YARN_HOME/share/hadoop/yarn/lib/*:$HAD=
>>>>
>>>> OOP_MAPRED_HOME/share/hadoop/mapreduce/*:$HADOOP_MAPRED_HOME/share/hadoop/m=
>>>> apreduce/lib/*}
>>>> 14/11/18 05:36:09 INFO yarn.GiraphYarnClient: buildLocalResourceMap ....
>>>> 14/11/18 05:36:12 INFO yarn.YarnUtils: Registered file in
>>>> LocalResources ::
>>>>
>>>> hdfs://localhost:9000/user/hduser/giraph_yarn_jar_cache/application_1416310=
>>>> 496070_0001/giraph-conf.xml
>>>> 14/11/18 05:36:12 INFO yarn.GiraphYarnClient: LIB JARS
>>>>
>>>> :giraph-examples-1.1.0-SNAPSHOT-for-hadoop-2.5.1-jar-with-dependencies.jar
>>>> 14/11/18 05:36:12 INFO yarn.YarnUtils: Class path name .
>>>> 14/11/18 05:36:12 INFO yarn.YarnUtils: base path checking .
>>>> 14/11/18 05:36:12 INFO yarn.GiraphYarnClient: Made local resource for
>>>>
>>>> :/usr/local/hadoop/share/giraph-examples-1.1.0-SNAPSHOT-for-hadoop-2.5.1-ja=
>>>> r-with-dependencies.jar
>>>> to
>>>>
>>>> hdfs://localhost:9000/user/hduser/giraph_yarn_jar_cache/application_1416310=
>>>>
>>>> 496070_0001/giraph-examples-1.1.0-SNAPSHOT-for-hadoop-2.5.1-jar-with-depend=
>>>> encies.jar
>>>> 14/11/18 05:36:16 INFO yarn.YarnUtils: Registered file in
>>>> LocalResources ::
>>>>
>>>> hdfs://localhost:9000/user/hduser/giraph_yarn_jar_cache/application_1416310=
>>>>
>>>> 496070_0001/giraph-examples-1.1.0-SNAPSHOT-for-hadoop-2.5.1-jar-with-depend=
>>>> encies.jar
>>>> 14/11/18 05:36:16 INFO yarn.GiraphYarnClient:
>>>> ApplicationSumbissionContext
>>>> for GiraphApplicationMaster launch container is populated.
>>>> 14/11/18 05:36:16 INFO yarn.GiraphYarnClient: Submitting application to
>>>> ASM
>>>> 14/11/18 05:36:16 INFO impl.YarnClientImpl: Submitted application
>>>> application_1416310496070_0001
>>>> 14/11/18 05:36:16 INFO yarn.GiraphYarnClient: Got new appId after
>>>> submission :application_1416310496070_0001
>>>> 14/11/18 05:36:16 INFO yarn.GiraphYarnClient: GiraphApplicationMaster
>>>> container request was submitted to ResourceManager for job: Giraph:
>>>> org.apache.giraph.examples.SimpleShortestPathsComputation
>>>> 14/11/18 05:36:17 INFO yarn.GiraphYarnClient: Giraph:
>>>> org.apache.giraph.examples.SimpleShortestPathsComputation, Elapsed: 1.01
>>>> secs
>>>> 14/11/18 05:36:17 INFO yarn.GiraphYarnClient:
>>>> appattempt_1416310496070_0001_000001, State: ACCEPTED, Containers used:
>>>> 1
>>>> 14/11/18 05:36:22 INFO yarn.GiraphYarnClient: Giraph:
>>>> org.apache.giraph.examples.SimpleShortestPathsComputation, Elapsed: 5.86
>>>> secs
>>>> 14/11/18 05:36:22 INFO yarn.GiraphYarnClient:
>>>> appattempt_1416310496070_0001_000001, State: ACCEPTED, Containers used:
>>>> 1
>>>> 14/11/18 05:36:26 INFO yarn.GiraphYarnClient: Giraph:
>>>> org.apache.giraph.examples.SimpleShortestPathsComputation, Elapsed:
>>>> 10.02
>>>> secs
>>>> 14/11/18 05:36:26 INFO yarn.GiraphYarnClient:
>>>> appattempt_1416310496070_0001_000001, State: ACCEPTED, Containers used:
>>>> 1
>>>> 14/11/18 05:36:30 INFO yarn.GiraphYarnClient: Giraph:
>>>> org.apache.giraph.examples.SimpleShortestPathsComputation, Elapsed:
>>>> 14.39
>>>> secs
>>>> 14/11/18 05:36:30 INFO yarn.GiraphYarnClient:
>>>> appattempt_1416310496070_0001_000001, State: ACCEPTED, Containers used:
>>>> 1
>>>> 14/11/18 05:36:34 INFO yarn.GiraphYarnClient: Giraph:
>>>> org.apache.giraph.examples.SimpleShortestPathsComputation, Elapsed:
>>>> 18.57
>>>> secs
>>>> 14/11/18 05:36:34 INFO yarn.GiraphYarnClient:
>>>> appattempt_1416310496070_0001_000001, State: RUNNING, Containers used: 2
>>>>
>>>> gam_stderr.log
>>>>
>>>> LF4J: Class path contains multiple SLF4J bindings.
>>>> SLF4J: Found binding in
>>>>
>>>> [jar:file:/app/hadoop/tmp/nm-local-dir/usercache/hduser/appcache/applicatio=
>>>>
>>>> n_1416310496070_0001/filecache/10/giraph-examples-1.1.0-SNAPSHOT-for-hadoop=
>>>>
>>>> -2.5.1-jar-with-dependencies.jar!/org/slf4j/impl/StaticLoggerBinder.class]
>>>> SLF4J: Found binding in
>>>>
>>>> [jar:file:/usr/local/hadoop/share/hadoop/common/lib/slf4j-log4j12-1.7.5.jar=
>>>> !/org/slf4j/impl/StaticLoggerBinder.class]
>>>> SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an
>>>> explana=
>>>> tion.
>>>> SLF4J: Actual binding is of type [org.slf4j.impl.Log4jLoggerFactory]
>>>>
>>>> gam_stdout.log
>>>>
>>>> 2014-11-18 05:36:28,143 INFO [main] yarn.GiraphApplicationMaster
>>>> (GiraphApplicationMaster.java:main(421)) - Starting GitaphAM
>>>> 2014-11-18 05:36:30,404 WARN [main] util.NativeCodeLoader
>>>> (NativeCodeLoader.java:<clinit>(62)) - Unable to load native-hadoop
>>>> library for your platform... using builtin-java classes where
>>>> applicable
>>>> 2014-11-18 05:36:32,649 INFO [main] yarn.GiraphApplicationMaster
>>>> (GiraphApplicationMaster.java:<init>(168)) - GiraphAM for ContainerId
>>>> container_1416310496070_0001_01_000001 ApplicationAttemptId
>>>> appattempt_1416310496070_0001_000001
>>>> 2014-11-18 05:36:32,852 INFO [main] client.RMProxy
>>>> (RMProxy.java:createRMProxy(92)) - Connecting to ResourceManager at
>>>> localhost/127.0.0.1:8030
>>>> 2014-11-18 05:36:32,888 INFO [main] impl.NMClientAsyncImpl
>>>> (NMClientAsyncImpl.java:serviceInit(107)) - Upper bound of the thread
>>>> pool size is 500
>>>> 2014-11-18 05:36:32,889 INFO [main]
>>>> impl.ContainerManagementProtocolProxy
>>>> (ContainerManagementProtocolProxy.java:<init>(78)) -
>>>> yarn.client.max-nodemanagers-proxies : 500
>>>> 2014-11-18 05:36:33,589 INFO [main] yarn.GiraphApplicationMaster
>>>> (GiraphApplicationMaster.java:setupContainerAskForRM(279)) - Requested
>>>> container ask: Capability[<memory:1024, vCores:0>]Priority[10]
>>>> 2014-11-18 05:36:33,617 INFO [main] yarn.GiraphApplicationMaster
>>>> (GiraphApplicationMaster.java:setupContainerAskForRM(279)) - Requested
>>>> container ask: Capability[<memory:1024, vCores:0>]Priority[10]
>>>> 2014-11-18 05:36:33,617 INFO [main] yarn.GiraphApplicationMaster
>>>> (GiraphApplicationMaster.java:run(185)) - Wait to finish ..
>>>> 2014-11-18 05:36:35,674 INFO [AMRM Heartbeater thread]
>>>> impl.AMRMClientImpl (AMRMClientImpl.java:populateNMTokens(299)) -
>>>> Received new token for : roosevelt:60486
>>>> 2014-11-18 05:36:35,681 INFO [AMRM Callback Handler Thread]
>>>> yarn.GiraphApplicationMaster
>>>> (GiraphApplicationMaster.java:onContainersAllocated(605)) - Got
>>>> response from RM for container ask, allocatedCnt=3D1
>>>> 2014-11-18 05:36:35,681 INFO [AMRM Callback Handler Thread]
>>>> yarn.GiraphApplicationMaster
>>>> (GiraphApplicationMaster.java:onContainersAllocated(608)) - Total
>>>> allocated # of container so far : 1 allocated out of 2 required.
>>>> 2014-11-18 05:36:35,681 INFO [AMRM Callback Handler Thread]
>>>> yarn.GiraphApplicationMaster
>>>> (GiraphApplicationMaster.java:startContainerLaunchingThreads(359)) -
>>>> Launching command on a new container.,
>>>> containerId=3Dcontainer_1416310496070_0001_01_000002,
>>>> containerNode=3Droosevelt:60486, containerNodeURI=3Droosevelt:8042,
>>>> containerResourceMemory=3D1024
>>>> 2014-11-18 05:36:35,688 INFO [pool-4-thread-1]
>>>> yarn.GiraphApplicationMaster
>>>> (GiraphApplicationMaster.java:buildContainerLaunchContext(492)) -
>>>> Setting up container launch container for
>>>> containerid=3Dcontainer_1416310496070_0001_01_000002
>>>> 2014-11-18 05:36:35,720 INFO [pool-4-thread-1]
>>>> yarn.GiraphApplicationMaster
>>>> (GiraphApplicationMaster.java:buildContainerLaunchContext(498)) -
>>>> Conatain launch Commands :java -Xmx1024M -Xms1024M -cp .:${CLASSPATH}
>>>> org.apache.giraph.yarn.GiraphYarnTask 1416310496070 1 2 1
>>>> 1><LOG_DIR>/task-2-stdout.log 2><LOG_DIR>/task-2-stderr.log
>>>> 2014-11-18 05:36:35,721 INFO [pool-4-thread-1]
>>>> yarn.GiraphApplicationMaster
>>>> (GiraphApplicationMaster.java:buildContainerLaunchContext(518)) -
>>>> Setting username in ContainerLaunchContext to: hduser
>>>> 2014-11-18 05:36:36,789 INFO [AMRM Callback Handler Thread]
>>>> yarn.GiraphApplicationMaster
>>>> (GiraphApplicationMaster.java:onContainersAllocated(605)) - Got
>>>> response from RM for container ask, allocatedCnt=3D1
>>>> 2014-11-18 05:36:36,789 INFO [AMRM Callback Handler Thread]
>>>> yarn.GiraphApplicationMaster
>>>> (GiraphApplicationMaster.java:onContainersAllocated(608)) - Total
>>>> allocated # of container so far : 2 allocated out of 2 required.
>>>> 2014-11-18 05:36:36,790 INFO [AMRM Callback Handler Thread]
>>>> yarn.GiraphApplicationMaster
>>>> (GiraphApplicationMaster.java:startContainerLaunchingThreads(359)) -
>>>> Launching command on a new container.,
>>>> containerId=3Dcontainer_1416310496070_0001_01_000003,
>>>> containerNode=3Droosevelt:60486, containerNodeURI=3Droosevelt:8042,
>>>> containerResourceMemory=3D1024
>>>> 2014-11-18 05:36:37,227 INFO [pool-4-thread-2]
>>>> yarn.GiraphApplicationMaster
>>>> (GiraphApplicationMaster.java:buildContainerLaunchContext(492)) -
>>>> Setting up container launch container for
>>>> containerid=3Dcontainer_1416310496070_0001_01_000003
>>>> 2014-11-18 05:36:37,227 INFO [pool-4-thread-2]
>>>> yarn.GiraphApplicationMaster
>>>> (GiraphApplicationMaster.java:buildContainerLaunchContext(498)) -
>>>> Conatain launch Commands :java -Xmx1024M -Xms1024M -cp .:${CLASSPATH}
>>>> org.apache.giraph.yarn.GiraphYarnTask 1416310496070 1 3 1
>>>> 1><LOG_DIR>/task-3-stdout.log 2><LOG_DIR>/task-3-stderr.log
>>>> 2014-11-18 05:36:37,227 INFO [pool-4-thread-2]
>>>> yarn.GiraphApplicationMaster
>>>> (GiraphApplicationMaster.java:buildContainerLaunchContext(518)) -
>>>> Setting username in ContainerLaunchContext to: hduser
>>>> 2014-11-18 05:36:37,463 INFO [pool-4-thread-1] yarn.YarnUtils
>>>> (YarnUtils.java:addFsResourcesToMap(72)) - Adding
>>>>
>>>> giraph-examples-1.1.0-SNAPSHOT-for-hadoop-2.5.1-jar-with-dependencies.jar
>>>> to LocalResources for export.to
>>>>
>>>> hdfs://localhost:9000/user/hduser/giraph_yarn_jar_cache/application_1416310=
>>>>
>>>> 496070_0001/giraph-examples-1.1.0-SNAPSHOT-for-hadoop-2.5.1-jar-with-depend=
>>>> encies.jar
>>>> 2014-11-18 05:36:37,678 INFO [pool-4-thread-1] yarn.YarnUtils
>>>> (YarnUtils.java:addFileToResourceMap(160)) - Registered file in
>>>> LocalResources ::
>>>>
>>>> hdfs://localhost:9000/user/hduser/giraph_yarn_jar_cache/application_1416310=
>>>>
>>>> 496070_0001/giraph-examples-1.1.0-SNAPSHOT-for-hadoop-2.5.1-jar-with-depend=
>>>> encies.jar
>>>> 2014-11-18 05:36:37,678 WARN [pool-4-thread-1] yarn.YarnUtils
>>>> (YarnUtils.java:addFsResourcesToMap(81)) - Job jars (-yj option)
>>>> didn't include giraph-core.
>>>> 2014-11-18 05:36:37,680 INFO [pool-4-thread-1] yarn.YarnUtils
>>>> (YarnUtils.java:addFileToResourceMap(160)) - Registered file in
>>>> LocalResources ::
>>>>
>>>> hdfs://localhost:9000/user/hduser/giraph_yarn_jar_cache/application_1416310=
>>>> 496070_0001/giraph-conf.xml
>>>> 2014-11-18 05:36:37,694 INFO
>>>> [org.apache.hadoop.yarn.client.api.async.impl.NMClientAsyncImpl #0]
>>>> impl.NMClientAsyncImpl (NMClientAsyncImpl.java:run(531)) - Processing
>>>> Event EventType: START_CONTAINER for Container
>>>> container_1416310496070_0001_01_000002
>>>> 2014-11-18 05:36:37,699 INFO
>>>> [org.apache.hadoop.yarn.client.api.async.impl.NMClientAsyncImpl #1]
>>>> impl.NMClientAsyncImpl (NMClientAsyncImpl.java:run(531)) - Processing
>>>> Event EventType: START_CONTAINER for Container
>>>> container_1416310496070_0001_01_000003
>>>> 2014-11-18 05:36:37,698 INFO
>>>> [org.apache.hadoop.yarn.client.api.async.impl.NMClientAsyncImpl #0]
>>>> impl.ContainerManagementProtocolProxy
>>>> (ContainerManagementProtocolProxy.java:newProxy(212)) - Opening proxy
>>>> : roosevelt:60486
>>>> 2014-11-18 05:36:37,761 INFO
>>>> [org.apache.hadoop.yarn.client.api.async.impl.NMClientAsyncImpl #2]
>>>> impl.NMClientAsyncImpl (NMClientAsyncImpl.java:run(531)) - Processing
>>>> Event EventType: QUERY_CONTAINER for Container
>>>> container_1416310496070_0001_01_000003
>>>> 2014-11-18 05:36:37,771 INFO
>>>> [org.apache.hadoop.yarn.client.api.async.impl.NMClientAsyncImpl #3]
>>>> impl.NMClientAsyncImpl (NMClientAsyncImpl.java:run(531)) - Processing
>>>> Event EventType: QUERY_CONTAINER for Container
>>>> container_1416310496070_0001_01_000002
>>>>
>>>>
>>>>
>>>> On Thu, Nov 20, 2014 at 1:32 PM, Alessandro Negro <alenegr...@yahoo.it>
>>>> wrote:
>>>>
>>>> Hi Das,
>>>> I mean as user you run Hadoop yarn resource manager.
>>>>
>>>> Alessandro
>>>>
>>>> Il giorno 20/nov/2014, alle ore 19:22, D Adams <dadam...@gmail.com> ha
>>>> scritto:
>>>>
>>>> Alessandro,
>>>> I'm not sure what you mean, should I create a new user on my system
>>>> named 'yarn'? I'm new to both hadoop and giraph, so I'm not sure.
>>>>
>>>> Tripti,
>>>> I'll get those logs as soon as I can.
>>>>
>>>> Thank you both,
>>>>
>>>> V/r
>>>> Das
>>>> On Nov 20, 2014 3:13 AM, "Alessandro Negro" <alenegr...@yahoo.it>
>>>> wrote:
>>>>
>>>> Hi Tripti,
>>>> I agree that a more detailed error log could be useful.
>>>>
>>>> Thanks,
>>>> Alessandro
>>>>
>>>>
>>>>
>>>>
>>>> Il giorno 19/nov/2014, alle ore 15:54, Tripti Singh <
>>>> tri...@yahoo-inc.co=
>>>>
>>>> m>
>>>>
>>>> ha scritto:
>>>>
>>>> --_000_D092AB9C1CB42triptiyahooinccom_
>>>> Content-Type: text/plain; charset=3D"Windows-1252"
>>>> Content-Transfer-Encoding: quoted-printable
>>>>
>>>> Hi Alessandro,
>>>>
>>>> I guess u r talking about the documentation on Giraph webpage which
>>>>
>>>> mention=3D
>>>>
>>>> s addition of this new option which is more or less mandatory for yarn
>>>>
>>>> base=3D
>>>>
>>>> d profiles.
>>>> When I first ran the Giraph on yarn without the proper =3D96yj option,
>>>>
>>>> there =3D
>>>>
>>>> was no way I could figure out from the Application/Container logs that
>>>>
>>>> this=3D
>>>>
>>>> was the issue.
>>>> I think, it=3D92ll be a good idea to have this message in the logs for
>>>>
>>>> easy d=3D
>>>>
>>>> ebugging.
>>>>
>>>> Thanks,
>>>> Tripti.
>>>>
>>>>
>>>> From: Alessandro Negro <alenegr...@yahoo.it<mailto:alenegr...@yahoo.it
>>>> <alenegr...@yahoo.it>
>>>>
>>>>
>>>> Reply-To: "user@giraph.apache.org<mailto:user@giraph.apache.org
>>>> <user@giraph.apache.org>>"
>>>>
>>>> <user@gir=3D
>>>>
>>>> aph.apache.org<mailto:user@giraph.apache.org <user@giraph.apache.org>>>
>>>> Date: Tuesday, November 18, 2014 at 4:31 PM
>>>> To: "user@giraph.apache.org<mailto:user@giraph.apache.org
>>>> <user@giraph.apache.org>>" <
>>>>
>>>> u...@giraph.ap=3D
>>>>
>>>> ache.org<mailto:user@giraph.apache.org <user@giraph.apache.org>>>
>>>> Subject: Re: Help with Giraph on Yarn
>>>>
>>>> Hi Eli,
>>>> no I think that the help message is clear enough, generally when I rea=
>>>>
>>>> d
>>>>
>>>> =3D93=3D
>>>>
>>>> jar=3D94 I always mean absolute path for jar, but this is a my persona=
>>>>
>>>> l
>>>>
>>>> misun=3D
>>>>
>>>> derstanding.
>>>>
>>>> Thanks,
>>>> Alessandro
>>>>
>>>>
>>>> Il giorno 16/nov/2014, alle ore 21:39, Eli Reisman <
>>>>
>>>> apache.mail...@gmail.co=3D
>>>>
>>>> m<mailto:apache.mail...@gmail.com <apache.mail...@gmail.com>>> ha
>>>> scritto:
>>>>
>>>> Sounds like you got it figured out since last time I checked this list=
>>>>
>>>> ,
>>>>
>>>> sor=3D
>>>>
>>>> ry it was a pain. Feel free to drop a JIRA ticket if we can make help
>>>>
>>>> messa=3D
>>>>
>>>> ge etc. for -yj option more clear, there's lots to improve there.
>>>>
>>>> On Sat, Nov 8, 2014 at 7:26 AM, Alessandro Negro <alenegr...@yahoo.it
>>>>
>>>> <mailt=3D
>>>>
>>>> o:alenegr...@yahoo.it>> wrote:
>>>> Hi Tripti,
>>>> finally I was able to run the test with success. It was an issue of
>>>>
>>>> permiss=3D
>>>>
>>>> ion since I was running as ale not as yarn.
>>>> Let me say that now I=3D92m able to run graph examples on Yarn 2.5.1.
>>>>
>>>> This is=3D
>>>>
>>>> the final result:
>>>>
>>>> 14/11/08 16:24:00 INFO yarn.GiraphYarnClient: Completed Giraph:
>>>>
>>>> org.apache.=3D
>>>>
>>>> giraph.examples.SimpleShortestPathsComputation: SUCCEEDED, total
>>>>
>>>> running ti=3D
>>>>
>>>> me: 0 minutes, 21 seconds.
>>>>
>>>>
>>>> Many thanks for your support,
>>>> Alessandro
>>>>
>>>> Il giorno 06/nov/2014, alle ore 15:16, Tripti Singh <
>>>>
>>>> tri...@yahoo-inc.com<m=3D
>>>>
>>>> ailto:tri...@yahoo-inc.com>> ha scritto:
>>>>
>>>> I don't know if u have access to this node. But if u do, u could check
>>>>
>>>> if t=3D
>>>>
>>>> he file is indeed there and u have access to it.
>>>>
>>>> Sent from my iPhone
>>>>
>>>> On 06-Nov-2014, at 6:12 pm, "Alessandro Negro" <alenegr...@yahoo.it
>>>>
>>>> <mailto:=3D <=3D>
>>>>
>>>> alenegr...@yahoo.it>> wrote:
>>>>
>>>> You are right it works, but now I receive the following error:
>>>>
>>>> SLF4J: Class path contains multiple SLF4J bindings.
>>>> SLF4J: Found binding in
>>>>
>>>> [jar:file:/private/tmp/hadoop-yarn/nm-local-dir/use=3D
>>>>
>>>>
>>>>
>>>> rcache/ale/appcache/application_1415264041937_0009/filecache/10/giraph-e=
>>>>
>>>> xam=3D
>>>>
>>>>
>>>>
>>>> ples-1.1.0-SNAPSHOT-for-hadoop-2.5.1-jar-with-dependencies.jar!/org/slf4=
>>>>
>>>> j/i=3D
>>>>
>>>> mpl/StaticLoggerBinder.class]
>>>> SLF4J: Found binding in
>>>>
>>>> [jar:file:/opt/yarn/hadoop-2.5.1/share/hadoop/commo=3D
>>>>
>>>> n/lib/slf4j-log4j12-1.7.5.jar!/org/slf4j/impl/StaticLoggerBinder.class=
>>>>
>>>> ]
>>>>
>>>> SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an
>>>>
>>>> explana=3D
>>>>
>>>> tion.
>>>> SLF4J: Actual binding is of type [org.slf4j.impl.Log4jLoggerFactory]
>>>> 2014-11-06 13:15:37.120 java[10158:1803] Unable to load realm info fro=
>>>>
>>>> m
>>>>
>>>> SCD=3D
>>>>
>>>> ynamicStore
>>>> Exception in thread "pool-4-thread-1" java.lang.IllegalStateException:
>>>>
>>>> Coul=3D
>>>>
>>>> d not configure the containerlaunch context for GiraphYarnTasks.
>>>> at
>>>>
>>>>
>>>> org.apache.giraph.yarn.GiraphApplicationMaster.getTaskResourceMap(Giraph=
>>>>
>>>> =3D
>>>>
>>>> ApplicationMaster.java:391)
>>>> at
>>>>
>>>>
>>>> org.apache.giraph.yarn.GiraphApplicationMaster.access$500(GiraphApplicat=
>>>>
>>>> =3D
>>>>
>>>> ionMaster.java:78)
>>>> at
>>>>
>>>>
>>>> org.apache.giraph.yarn.GiraphApplicationMaster$LaunchContainerRunnable.b=
>>>>
>>>> =3D
>>>>
>>>> uildContainerLaunchContext(GiraphApplicationMaster.java:522)
>>>> at
>>>>
>>>>
>>>> org.apache.giraph.yarn.GiraphApplicationMaster$LaunchContainerRunnable.r=
>>>>
>>>> =3D
>>>>
>>>> un(GiraphApplicationMaster.java:479)
>>>> at
>>>>
>>>>
>>>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.jav=
>>>>
>>>> =3D
>>>>
>>>> a:1145)
>>>> at
>>>>
>>>>
>>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.ja=
>>>>
>>>> =3D
>>>>
>>>> va:615)
>>>> at java.lang.Thread.run(Thread.java:744)
>>>> Caused by: java.io.FileNotFoundException: File does not exist:
>>>>
>>>> hdfs://hadoo=3D
>>>>
>>>>
>>>>
>>>> p-master:9000/user/yarn/giraph_yarn_jar_cache/application_1415264041937_=
>>>>
>>>> 000=3D
>>>>
>>>>
>>>>
>>>> 9/giraph-examples-1.1.0-SNAPSHOT-for-hadoop-2.5.1-jar-with-dependencies.=
>>>>
>>>> jar
>>>>
>>>> at
>>>>
>>>>
>>>> org.apache.hadoop.hdfs.DistributedFileSystem$17.doCall(DistributedFileSy=
>>>>
>>>> =3D
>>>>
>>>> stem.java:1072)
>>>> at
>>>>
>>>>
>>>> org.apache.hadoop.hdfs.DistributedFileSystem$17.doCall(DistributedFileSy=
>>>>
>>>> =3D
>>>>
>>>> stem.java:1064)
>>>> at
>>>>
>>>>
>>>> org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolv=
>>>>
>>>> =3D
>>>>
>>>> er.java:81)
>>>> at
>>>>
>>>>
>>>> org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFi=
>>>>
>>>> =3D
>>>>
>>>> leSystem.java:1064)
>>>> at
>>>>
>>>>
>>>> org.apache.giraph.yarn.YarnUtils.addFileToResourceMap(YarnUtils.java:153=
>>>>
>>>> =3D
>>>>
>>>> )
>>>> at
>>>>
>>>> org.apache.giraph.yarn.YarnUtils.addFsResourcesToMap(YarnUtils.java:77)
>>>>
>>>> at
>>>>
>>>>
>>>> org.apache.giraph.yarn.GiraphApplicationMaster.getTaskResourceMap(Giraph=
>>>>
>>>> =3D
>>>>
>>>> ApplicationMaster.java:387)
>>>> ... 6 more
>>>>
>>>>
>>>> That justify the other error I receive in the task:
>>>> Could not find or load main class org.apache.giraph.yarn.GiraphYarnTas=
>>>>
>>>> k
>>>>
>>>>
>>>> Thanks,
>>>>
>>>> Il giorno 06/nov/2014, alle ore 13:07, Tripti Singh <
>>>>
>>>> tri...@yahoo-inc.com<m=3D
>>>>
>>>> ailto:tri...@yahoo-inc.com>> ha scritto:
>>>>
>>>> Why r u adding two jars? Example jar ideally contains core library so
>>>>
>>>> every=3D
>>>>
>>>> thing should be available with just one example jar included
>>>>
>>>> Sent from my iPhone
>>>>
>>>> On 06-Nov-2014, at 4:33 pm, "Alessandro Negro" <alenegr...@yahoo.it
>>>>
>>>> <mailto:=3D <=3D>
>>>>
>>>> alenegr...@yahoo.it>> wrote:
>>>>
>>>> Hi,
>>>> now it seems better, I need to add:
>>>>
>>>>
>>>>
>>>> giraph-1.1.0-SNAPSHOT-for-hadoop-2.5.1-jar-with-dependencies.jar,giraph-=
>>>>
>>>> exa=3D
>>>>
>>>> mples-1.1.0-SNAPSHOT-for-hadoop-2.5.1-jar-with-dependencies.jar
>>>>
>>>>
>>>> Now it seems that after a lot of cycle it fail with this error:
>>>>
>>>> Could not find or load main class org.apache.giraph.yarn.GiraphYarnTas=
>>>>
>>>> k
>>>>
>>>>
>>>> But in this case the error appear in task-3-stderr.log not in
>>>>
>>>> gam-stderr.l=3D
>>>>
>>>> og where there is the following error:
>>>>
>>>> LF4J: Class path contains multiple SLF4J bindings.
>>>> SLF4J: Found binding in
>>>>
>>>> [jar:file:/private/tmp/hadoop-yarn/nm-local-dir/use=3D
>>>>
>>>>
>>>>
>>>> rcache/ale/appcache/application_1415264041937_0006/filecache/12/giraph-1=
>>>>
>>>> .1.=3D
>>>>
>>>>
>>>>
>>>> 0-SNAPSHOT-for-hadoop-2.5.1-jar-with-dependencies.jar!/org/slf4j/impl/St=
>>>>
>>>> ati=3D
>>>>
>>>> cLoggerBinder.class]
>>>> SLF4J: Found binding in
>>>>
>>>> [jar:file:/private/tmp/hadoop-yarn/nm-local-dir/use=3D
>>>>
>>>>
>>>>
>>>> rcache/ale/appcache/application_1415264041937_0006/filecache/10/giraph-e=
>>>>
>>>> xam=3D
>>>>
>>>>
>>>>
>>>> ples-1.1.0-SNAPSHOT-for-hadoop-2.5.1-jar-with-dependencies.jar!/org/slf4=
>>>>
>>>> j/i=3D
>>>>
>>>> mpl/StaticLoggerBinder.class]
>>>> SLF4J: Found binding in
>>>>
>>>> [jar:file:/opt/yarn/hadoop-2.5.1/share/hadoop/commo=3D
>>>>
>>>> n/lib/slf4j-log4j12-1.7.5.jar!/org/slf4j/impl/StaticLoggerBinder.class=
>>>>
>>>> ]
>>>>
>>>> SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an
>>>>
>>>>
>>>>
>>>>
>>>
>>
>>
>
