HI Das, what about the hadoop logs? Have a look there and let me know. In the meantime you can remove redundant jars from yarn jars (exampe contains all jars needed):
> yarnjars=giraph-examples-1.1.0-hadoop-2.5.1-jar-with-dependencies.jar Should be enough Alessandro. Il giorno 23/nov/2014, alle ore 00:16, D Adams <dadam...@gmail.com> ha scritto: > Hi Alessandro, > Do you run tyarn that way because your hadoop or giraph file permissions > are chowned by a user named yarn? > > I have made changes as suggested on > http://mail-archives.apache.org/mod_mbox/giraph-user/201408.mbox/%3c53f4c689.5060...@web.de%3E > > And although things do look more hopeful, I still get an infinite loop. I run > with as script which includes: > > user_dir=/user/hduser > jar=giraph-examples-1.1.0-hadoop-2.5.1-jar-with-dependencies.jar > runner=org.apache.giraph.GiraphRunner > computation=org.apache.giraph.examples.SimpleShortestPathsComputation > informat=org.apache.giraph.io.formats.JsonLongDoubleFloatDoubleVertexInputFormat > outformat=org.apache.giraph.io.formats.IdWithValueTextOutputFormat > yarnjars=giraph-examples-1.1.0-hadoop-2.5.1-jar-with-dependencies.jar,giraph-core-1.1.0-SNAPSHOT.jar,giraph-1.1.0-hadoop-2.5.1.jar > > bin/hdfs dfs -rm -r $user_dir/output/shortestpaths > bin/yarn jar $GIRAPH_HOME/$jar $runner -Dgiraph.yarn.task.heap.mb=3000 > $computation -vif $informat -vip $user_dir/input/tiny_graph.txt -vof > $outformat -op $user_dir/output/shortestpaths -w 1 -yj $yarnjars > > And the resulting stdout: > hduser@Roosevelt:/usr/local/hadoop$ ./hduser_jobs.sh > Running hduser script ... > 14/11/18 19:05:47 WARN util.NativeCodeLoader: Unable to load native-hadoop > library for your platform... using builtin-java classes where applicable > rm: `/user/hduser/output/shortestpaths': No such file or directory > 14/11/18 19:06:13 WARN util.NativeCodeLoader: Unable to load native-hadoop > library for your platform... using builtin-java classes where applicable > 14/11/18 19:06:15 INFO utils.ConfigurationUtils: No edge input format > specified. Ensure your InputFormat does not require one. > 14/11/18 19:06:15 INFO utils.ConfigurationUtils: No edge output format > specified. Ensure your OutputFormat does not require one. > 14/11/18 19:06:16 INFO yarn.GiraphYarnClient: Final output path is: > hdfs://localhost:9000/user/hduser/output/shortestpaths > 14/11/18 19:06:16 INFO yarn.GiraphYarnClient: Running Client > 14/11/18 19:06:16 INFO client.RMProxy: Connecting to ResourceManager at > localhost/127.0.0.1:8050 > 14/11/18 19:06:17 INFO yarn.GiraphYarnClient: Got node report from ASM for, > nodeId=roosevelt:58195, nodeAddress roosevelt:8042, nodeRackName > /default-rack, nodeNumContainers 0 > 14/11/18 19:06:17 INFO yarn.GiraphYarnClient: Obtained new Application ID: > application_1416359131664_0001 > 14/11/18 19:06:17 INFO Configuration.deprecation: mapred.job.id is > deprecated. Instead, use mapreduce.job.id > 14/11/18 19:06:17 INFO yarn.GiraphYarnClient: Set the environment for the > application master > 14/11/18 19:06:17 INFO yarn.GiraphYarnClient: Environment for AM > :{CLASSPATH=${CLASSPATH}:./*:$HADOOP_CONF_DIR:$HADOOP_COMMON_HOME/share/hadoop/common/*:$HADOOP_COMMON_HOME/share/hadoop/common/lib/*:$HADOOP_HDFS_HOME/share/hadoop/hdfs/*:$HADOOP_HDFS_HOME/share/hadoop/hdfs/lib/*:$HADOOP_YARN_HOME/share/hadoop/yarn/*:$HADOOP_YARN_HOME/share/hadoop/yarn/lib/*:$HADOOP_MAPRED_HOME/share/hadoop/mapreduce/*:$HADOOP_MAPRED_HOME/share/hadoop/mapreduce/lib/*} > 14/11/18 19:06:17 INFO yarn.GiraphYarnClient: buildLocalResourceMap .... > 14/11/18 19:06:19 INFO yarn.YarnUtils: Registered file in LocalResources :: > hdfs://localhost:9000/user/hduser/giraph_yarn_jar_cache/application_1416359131664_0001/giraph-conf.xml > 14/11/18 19:06:19 INFO yarn.GiraphYarnClient: LIB JARS > :giraph-examples-1.1.0-hadoop-2.5.1-jar-with-dependencies.jar,giraph-core-1.1.0-SNAPSHOT.jar,giraph-1.1.0-hadoop-2.5.1.jar > 14/11/18 19:06:19 INFO yarn.YarnUtils: Class path name . > 14/11/18 19:06:19 INFO yarn.YarnUtils: base path checking . > 14/11/18 19:06:19 INFO yarn.YarnUtils: Class path name /usr/local/hadoop > 14/11/18 19:06:19 INFO yarn.YarnUtils: base path checking /usr/local/hadoop > 14/11/18 19:06:19 INFO yarn.YarnUtils: Class path name null > 14/11/18 19:06:19 INFO yarn.YarnUtils: base path checking null > 14/11/18 19:06:19 INFO yarn.GiraphYarnClient: Made local resource for > :/usr/local/hadoop/share/myLib/giraph-examples-1.1.0-hadoop-2.5.1-jar-with-dependencies.jar > to > hdfs://localhost:9000/user/hduser/giraph_yarn_jar_cache/application_1416359131664_0001/giraph-examples-1.1.0-hadoop-2.5.1-jar-with-dependencies.jar > 14/11/18 19:06:23 INFO yarn.YarnUtils: Registered file in LocalResources :: > hdfs://localhost:9000/user/hduser/giraph_yarn_jar_cache/application_1416359131664_0001/giraph-examples-1.1.0-hadoop-2.5.1-jar-with-dependencies.jar > 14/11/18 19:06:23 INFO yarn.GiraphYarnClient: ApplicationSumbissionContext > for GiraphApplicationMaster launch container is populated. > 14/11/18 19:06:23 INFO yarn.GiraphYarnClient: Submitting application to ASM > 14/11/18 19:06:23 INFO impl.YarnClientImpl: Submitted application > application_1416359131664_0001 > 14/11/18 19:06:23 INFO yarn.GiraphYarnClient: Got new appId after submission > :application_1416359131664_0001 > 14/11/18 19:06:23 INFO yarn.GiraphYarnClient: GiraphApplicationMaster > container request was submitted to ResourceManager for job: Giraph: > org.apache.giraph.examples.SimpleShortestPathsComputation > 14/11/18 19:06:24 INFO yarn.GiraphYarnClient: Giraph: > org.apache.giraph.examples.SimpleShortestPathsComputation, Elapsed: 0.90 secs > 14/11/18 19:06:24 INFO yarn.GiraphYarnClient: > appattempt_1416359131664_0001_000001, State: ACCEPTED, Containers used: 0 > 14/11/18 19:06:28 INFO yarn.GiraphYarnClient: Giraph: > org.apache.giraph.examples.SimpleShortestPathsComputation, Elapsed: 5.05 secs > 14/11/18 19:06:28 INFO yarn.GiraphYarnClient: > appattempt_1416359131664_0001_000001, State: ACCEPTED, Containers used: 1 > 14/11/18 19:06:33 INFO yarn.GiraphYarnClient: Giraph: > org.apache.giraph.examples.SimpleShortestPathsComputation, Elapsed: 9.89 secs > 14/11/18 19:06:33 INFO yarn.GiraphYarnClient: > appattempt_1416359131664_0001_000001, State: ACCEPTED, Containers used: 1 > 14/11/18 19:06:37 INFO yarn.GiraphYarnClient: Giraph: > org.apache.giraph.examples.SimpleShortestPathsComputation, Elapsed: 14.37 secs > 14/11/18 19:06:37 INFO yarn.GiraphYarnClient: > appattempt_1416359131664_0001_000001, State: ACCEPTED, Containers used: 1 > 14/11/18 19:06:42 INFO yarn.GiraphYarnClient: Giraph: > org.apache.giraph.examples.SimpleShortestPathsComputation, Elapsed: 18.81 secs > 14/11/18 19:06:42 INFO yarn.GiraphYarnClient: > appattempt_1416359131664_0001_000001, State: ACCEPTED, Containers used: 1 > 14/11/18 19:06:46 INFO yarn.GiraphYarnClient: Giraph: > org.apache.giraph.examples.SimpleShortestPathsComputation, Elapsed: 22.82 secs > 14/11/18 19:06:46 INFO yarn.GiraphYarnClient: > appattempt_1416359131664_0001_000001, State: RUNNING, Containers used: 2 > 14/11/18 19:06:50 INFO yarn.GiraphYarnClient: Giraph: > org.apache.giraph.examples.SimpleShortestPathsComputation, Elapsed: 26.83 secs > 14/11/18 19:06:50 INFO yarn.GiraphYarnClient: > appattempt_1416359131664_0001_000001, State: RUNNING, Containers used: 2 > 14/11/18 19:06:54 INFO yarn.GiraphYarnClient: Giraph: > org.apache.giraph.examples.SimpleShortestPathsComputation, Elapsed: 30.83 secs > 14/11/18 19:06:54 INFO yarn.GiraphYarnClient: > appattempt_1416359131664_0001_000001, State: RUNNING, Containers used: 2 > 14/11/18 19:06:58 INFO yarn.GiraphYarnClient: Giraph: > org.apache.giraph.examples.SimpleShortestPathsComputation, Elapsed: 34.85 secs > 14/11/18 19:06:58 INFO yarn.GiraphYarnClient: > appattempt_1416359131664_0001_000001, State: RUNNING, Containers used: 2 > 14/11/18 19:07:02 INFO yarn.GiraphYarnClient: Giraph: > org.apache.giraph.examples.SimpleShortestPathsComputation, Elapsed: 38.85 secs > 14/11/18 19:07:02 INFO yarn.GiraphYarnClient: > appattempt_1416359131664_0001_000001, State: RUNNING, Containers used: 2 > 14/11/18 19:07:06 INFO yarn.GiraphYarnClient: Giraph: > org.apache.giraph.examples.SimpleShortestPathsComputation, Elapsed: 42.86 secs > 14/11/18 19:07:06 INFO yarn.GiraphYarnClient: > appattempt_1416359131664_0001_000001, State: RUNNING, Containers used: 2 > 14/11/18 19:07:10 INFO yarn.GiraphYarnClient: Giraph: > org.apache.giraph.examples.SimpleShortestPathsComputation, Elapsed: 46.88 secs > 14/11/18 19:07:10 INFO yarn.GiraphYarnClient: > appattempt_1416359131664_0001_000001, State: RUNNING, Containers used: 2 > 14/11/18 19:07:14 INFO yarn.GiraphYarnClient: Giraph: > org.apache.giraph.examples.SimpleShortestPathsComputation, Elapsed: 50.89 secs > 14/11/18 19:07:14 INFO yarn.GiraphYarnClient: > appattempt_1416359131664_0001_000001, State: RUNNING, Containers used: 2 > 14/11/18 19:07:18 INFO yarn.GiraphYarnClient: Giraph: > org.apache.giraph.examples.SimpleShortestPathsComputation, Elapsed: 54.90 secs > > Once yarn hits the RUNNING state, it enters an endless loop. It seems it is > caused by the following: > > > 2014-11-18 19:06:37,382 INFO [main] yarn.GiraphApplicationMaster > (GiraphApplicationMaster.java:main(421)) - Starting GitaphAM > 2014-11-18 19:06:39,214 WARN [main] util.NativeCodeLoader > (NativeCodeLoader.java:<clinit>(62)) - Unable to load native-hadoop library > for your platform... using builtin-java classes where applicable > 2014-11-18 19:06:41,438 INFO [main] yarn.GiraphApplicationMaster > (GiraphApplicationMaster.java:<init>(168)) - GiraphAM for ContainerId > container_1416359131664_0001_01_000001 ApplicationAttemptId > appattempt_1416359131664_0001_000001 > 2014-11-18 19:06:41,497 INFO [main] client.RMProxy > (RMProxy.java:createRMProxy(92)) - Connecting to ResourceManager at > localhost/127.0.0.1:8030 > 2014-11-18 19:06:41,581 INFO [main] impl.NMClientAsyncImpl > (NMClientAsyncImpl.java:serviceInit(107)) - Upper bound of the thread pool > size is 500 > 2014-11-18 19:06:41,583 INFO [main] impl.ContainerManagementProtocolProxy > (ContainerManagementProtocolProxy.java:<init>(78)) - > yarn.client.max-nodemanagers-proxies : 500 > 2014-11-18 19:06:42,355 INFO [main] yarn.GiraphApplicationMaster > (GiraphApplicationMaster.java:setupContainerAskForRM(279)) - Requested > container ask: Capability[<memory:3000, vCores:0>]Priority[10] > 2014-11-18 19:06:42,370 INFO [main] yarn.GiraphApplicationMaster > (GiraphApplicationMaster.java:setupContainerAskForRM(279)) - Requested > container ask: Capability[<memory:3000, vCores:0>]Priority[10] > 2014-11-18 19:06:42,618 INFO [main] yarn.GiraphApplicationMaster > (GiraphApplicationMaster.java:run(185)) - Wait to finish .. > 2014-11-18 19:06:44,673 INFO [AMRM Heartbeater thread] impl.AMRMClientImpl > (AMRMClientImpl.java:populateNMTokens(299)) - Received new token for : > roosevelt:58195 > 2014-11-18 19:06:44,677 INFO [AMRM Callback Handler Thread] > yarn.GiraphApplicationMaster > (GiraphApplicationMaster.java:onContainersAllocated(605)) - Got response from > RM for container ask, allocatedCnt=1 > 2014-11-18 19:06:44,677 INFO [AMRM Callback Handler Thread] > yarn.GiraphApplicationMaster > (GiraphApplicationMaster.java:onContainersAllocated(608)) - Total allocated # > of container so far : 1 allocated out of 2 required. > 2014-11-18 19:06:44,678 INFO [AMRM Callback Handler Thread] > yarn.GiraphApplicationMaster > (GiraphApplicationMaster.java:startContainerLaunchingThreads(359)) - > Launching command on a new container., > containerId=container_1416359131664_0001_01_000002, > containerNode=roosevelt:58195, containerNodeURI=roosevelt:8042, > containerResourceMemory=3072 > 2014-11-18 19:06:44,679 INFO [pool-4-thread-1] yarn.GiraphApplicationMaster > (GiraphApplicationMaster.java:buildContainerLaunchContext(492)) - Setting up > container launch container for > containerid=container_1416359131664_0001_01_000002 > 2014-11-18 19:06:44,694 INFO [pool-4-thread-1] yarn.GiraphApplicationMaster > (GiraphApplicationMaster.java:buildContainerLaunchContext(498)) - Conatain > launch Commands :java -Xmx3000M -Xms3000M -cp .:${CLASSPATH} > org.apache.giraph.yarn.GiraphYarnTask 1416359131664 1 2 1 > 1><LOG_DIR>/task-2-stdout.log 2><LOG_DIR>/task-2-stderr.log > 2014-11-18 19:06:44,694 INFO [pool-4-thread-1] yarn.GiraphApplicationMaster > (GiraphApplicationMaster.java:buildContainerLaunchContext(518)) - Setting > username in ContainerLaunchContext to: hduser > 2014-11-18 19:06:45,191 INFO [pool-4-thread-1] yarn.YarnUtils > (YarnUtils.java:addFsResourcesToMap(72)) - Adding > giraph-examples-1.1.0-hadoop-2.5.1-jar-with-dependencies.jar to > LocalResources for export.to > hdfs://localhost:9000/user/hduser/giraph_yarn_jar_cache/application_1416359131664_0001/giraph-examples-1.1.0-hadoop-2.5.1-jar-with-dependencies.jar > 2014-11-18 19:06:45,228 INFO [pool-4-thread-1] yarn.YarnUtils > (YarnUtils.java:addFileToResourceMap(160)) - Registered file in > LocalResources :: > hdfs://localhost:9000/user/hduser/giraph_yarn_jar_cache/application_1416359131664_0001/giraph-examples-1.1.0-hadoop-2.5.1-jar-with-dependencies.jar > 2014-11-18 19:06:45,228 INFO [pool-4-thread-1] yarn.YarnUtils > (YarnUtils.java:addFsResourcesToMap(72)) - Adding > giraph-core-1.1.0-SNAPSHOT.jar to LocalResources for export.to > hdfs://localhost:9000/user/hduser/giraph_yarn_jar_cache/application_1416359131664_0001/giraph-core-1.1.0-SNAPSHOT.jar > 2014-11-18 19:06:45,683 INFO [AMRM Callback Handler Thread] > yarn.GiraphApplicationMaster > (GiraphApplicationMaster.java:onContainersAllocated(605)) - Got response from > RM for container ask, allocatedCnt=1 > 2014-11-18 19:06:45,683 INFO [AMRM Callback Handler Thread] > yarn.GiraphApplicationMaster > (GiraphApplicationMaster.java:onContainersAllocated(608)) - Total allocated # > of container so far : 2 allocated out of 2 required. > 2014-11-18 19:06:45,683 INFO [AMRM Callback Handler Thread] > yarn.GiraphApplicationMaster > (GiraphApplicationMaster.java:startContainerLaunchingThreads(359)) - > Launching command on a new container., > containerId=container_1416359131664_0001_01_000003, > containerNode=roosevelt:58195, containerNodeURI=roosevelt:8042, > containerResourceMemory=3072 > 2014-11-18 19:06:45,684 INFO [pool-4-thread-3] yarn.GiraphApplicationMaster > (GiraphApplicationMaster.java:buildContainerLaunchContext(492)) - Setting up > container launch container for > containerid=container_1416359131664_0001_01_000003 > 2014-11-18 19:06:45,684 INFO [pool-4-thread-3] yarn.GiraphApplicationMaster > (GiraphApplicationMaster.java:buildContainerLaunchContext(498)) - Conatain > launch Commands :java -Xmx3000M -Xms3000M -cp .:${CLASSPATH} > org.apache.giraph.yarn.GiraphYarnTask 1416359131664 1 3 1 > 1><LOG_DIR>/task-3-stdout.log 2><LOG_DIR>/task-3-stderr.log > 2014-11-18 19:06:45,684 INFO [pool-4-thread-3] yarn.GiraphApplicationMaster > (GiraphApplicationMaster.java:buildContainerLaunchContext(518)) - Setting > username in ContainerLaunchContext to: hduser > 2014-11-18 19:06:45,697 INFO > [org.apache.hadoop.yarn.client.api.async.impl.NMClientAsyncImpl #0] > impl.NMClientAsyncImpl (NMClientAsyncImpl.java:run(531)) - Processing Event > EventType: START_CONTAINER for Container > container_1416359131664_0001_01_000003 > 2014-11-18 19:06:45,699 INFO > [org.apache.hadoop.yarn.client.api.async.impl.NMClientAsyncImpl #0] > impl.ContainerManagementProtocolProxy > (ContainerManagementProtocolProxy.java:newProxy(212)) - Opening proxy : > roosevelt:58195 > 2014-11-18 19:06:45,761 INFO > [org.apache.hadoop.yarn.client.api.async.impl.NMClientAsyncImpl #1] > impl.NMClientAsyncImpl (NMClientAsyncImpl.java:run(531)) - Processing Event > EventType: QUERY_CONTAINER for Container > container_1416359131664_0001_01_000003 > 2014-11-18 19:06:46,687 INFO [AMRM Callback Handler Thread] > yarn.GiraphApplicationMaster > (GiraphApplicationMaster.java:onContainersCompleted(571)) - Got response from > RM for container ask, completedCnt=1 > 2014-11-18 19:06:46,688 INFO [AMRM Callback Handler Thread] > yarn.GiraphApplicationMaster > (GiraphApplicationMaster.java:onContainersCompleted(574)) - Got container > status for containerID=container_1416359131664_0001_01_000003, > state=COMPLETE, exitStatus=1, diagnostics=Exception from container-launch: > ExitCodeException exitCode=1: > ExitCodeException exitCode=1: > at org.apache.hadoop.util.Shell.runCommand(Shell.java:538) > at org.apache.hadoop.util.Shell.run(Shell.java:455) > at > org.apache.hadoop.util.Shell$ShellCommandExecutor.execute(Shell.java:702) > at > org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor.launchContainer(DefaultContainerExecutor.java:195) > at > org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:300) > at > org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:81) > at java.util.concurrent.FutureTask.run(FutureTask.java:262) > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) > at java.lang.Thread.run(Thread.java:745) > > > Container exited with a non-zero exit code 1 > > 2014-11-18 19:06:46,688 INFO [AMRM Callback Handler Thread] > yarn.GiraphApplicationMaster > (GiraphApplicationMaster.java:onContainersCompleted(596)) - After completion > of one conatiner. current status is: completedCount :1 containersToLaunch :2 > successfulCount :0 failedCount :1 > > On Sat, Nov 22, 2014 at 11:42 AM, D Adams <dadam...@gmail.com> wrote: > I'm sorry, please excuse my ignorance, but, I have no idea where to find my > configuration. That is, I have no idea where these lines came from (or what > file): > > yarn 1029 0,1 1,2 3499572 209560 s001 S 10:29am 0:06.04 > /Library/Java/ > JavaVirtualMachines/jdk1.7.0_45.jdk/Contents/Home/bin/java > -Dproc_resourcemanager -Xmx500m -Dhadoop.log.dir=/opt/yarn/hadoop-2.5.1/logs > -Dyarn.log.dir=/opt/yarn/hadoop-2.5.1/logs > ... > As far as configuration goes, I'm only really aware of the > etc/hadoop/*-env.sh and .xml files and main pom.xml file. Or is the above > part of the command you use to run giraph jobs? > I will make changes to hadoop-env.sh file as you suggest and let you know how > it works out. > > On Fri, Nov 21, 2014 at 6:05 AM, Alessandro Negro <alenegr...@yahoo.it> wrote: > Hi, > I forgot to say that I added also this line: > > export HADOOP_CLASSPATH=$HADOOP_CLASSPATH:$MYLIB > > > Il giorno 21/nov/2014, alle ore 10:34, Alessandro Negro <alenegr...@yahoo.it> > ha scritto: > >> Hi Das, >> this is my configuration of hadoop: >> >> yarn 1029 0,1 1,2 3499572 209560 s001 S 10:29am >> 0:06.04 >> /Library/Java/JavaVirtualMachines/jdk1.7.0_45.jdk/Contents/Home/bin/java >> -Dproc_resourcemanager -Xmx500m -Dhadoop.log.dir=/opt/yarn/hadoop-2.5.1/logs >> -Dyarn.log.dir=/opt/yarn/hadoop-2.5.1/logs >> -Dhadoop.log.file=yarn-yarn-resourcemanager-MacBook-Pro-di-Alessandro.local.log >> >> -Dyarn.log.file=yarn-yarn-resourcemanager-MacBook-Pro-di-Alessandro.local.log >> -Dyarn.home.dir= -Dyarn.id.str=yarn -Dhadoop.root.logger=INFO,RFA >> -Dyarn.root.logger=INFO,RFA >> -Djava.library.path=/opt/yarn/hadoop-2.5.1/lib/native >> -Dyarn.policy.file=hadoop-policy.xml >> -Dhadoop.log.dir=/opt/yarn/hadoop-2.5.1/logs >> -Dyarn.log.dir=/opt/yarn/hadoop-2.5.1/logs >> -Dhadoop.log.file=yarn-yarn-resourcemanager-MacBook-Pro-di-Alessandro.local.log >> >> -Dyarn.log.file=yarn-yarn-resourcemanager-MacBook-Pro-di-Alessandro.local.log >> -Dyarn.home.dir=/opt/yarn/hadoop-2.5.1 >> -Dhadoop.home.dir=/opt/yarn/hadoop-2.5.1 -Dhadoop.root.logger=INFO,RFA >> -Dyarn.root.logger=INFO,RFA >> -Djava.library.path=/opt/yarn/hadoop-2.5.1/lib/native -classpath >> /opt/yarn/hadoop-2.5.1/etc/hadoop:/opt/yarn/hadoop-2.5.1/etc/hadoop:/opt/yarn/hadoop-2.5.1/etc/hadoop:/opt/yarn/hadoop-2.5.1/share/hadoop/common/lib/*:/opt/yarn/hadoop-2.5.1/share/hadoop/common/*:/opt/yarn/hadoop-2.5.1/share/hadoop/hdfs:/opt/yarn/hadoop-2.5.1/share/hadoop/hdfs/lib/*:/opt/yarn/hadoop-2.5.1/share/hadoop/hdfs/*:/opt/yarn/hadoop-2.5.1/share/hadoop/yarn/lib/*:/opt/yarn/hadoop-2.5.1/share/hadoop/yarn/*:/opt/yarn/hadoop-2.5.1/share/hadoop/mapreduce/lib/*:/opt/yarn/hadoop-2.5.1/share/hadoop/mapreduce/*::/opt/yarn/hadoop-2.5.1/share/myLib/*.jar:/Users/ale/extprj/giraph/myGiraph/*.jar:/Users/ale/extprj/giraph/myGiraph/lib/*.jar:/contrib/capacity-scheduler/*.jar:/opt/yarn/hadoop-2.5.1/share/myLib/*.jar:/Users/ale/extprj/giraph/myGiraph/*.jar:/Users/ale/extprj/giraph/myGiraph/lib/*.jar:/contrib/capacity-scheduler/*.jar:/opt/yarn/hadoop-2.5.1/share/hadoop/yarn/*:/opt/yarn/hadoop-2.5.1/share/hadoop/yarn/lib/*:/opt/yarn/hadoop-2.5.1/etc/hadoop/rm-config/log4j.properties >> org.apache.hadoop.yarn.server.resourcemanager.ResourceManager >> yarn 1073 0,1 1,0 3347132 174644 s001 S 10:29am >> 0:04.77 >> /Library/Java/JavaVirtualMachines/jdk1.7.0_45.jdk/Contents/Home/bin/java >> -Dproc_nodemanager -Xmx500m -Dhadoop.log.dir=/opt/yarn/hadoop-2.5.1/logs >> -Dyarn.log.dir=/opt/yarn/hadoop-2.5.1/logs >> -Dhadoop.log.file=yarn-yarn-nodemanager-MacBook-Pro-di-Alessandro.local.log >> -Dyarn.log.file=yarn-yarn-nodemanager-MacBook-Pro-di-Alessandro.local.log >> -Dyarn.home.dir= -Dyarn.id.str=yarn -Dhadoop.root.logger=INFO,RFA >> -Dyarn.root.logger=INFO,RFA >> -Djava.library.path=/opt/yarn/hadoop-2.5.1/lib/native >> -Dyarn.policy.file=hadoop-policy.xml -server >> -Dhadoop.log.dir=/opt/yarn/hadoop-2.5.1/logs >> -Dyarn.log.dir=/opt/yarn/hadoop-2.5.1/logs >> -Dhadoop.log.file=yarn-yarn-nodemanager-MacBook-Pro-di-Alessandro.local.log >> -Dyarn.log.file=yarn-yarn-nodemanager-MacBook-Pro-di-Alessandro.local.log >> -Dyarn.home.dir=/opt/yarn/hadoop-2.5.1 >> -Dhadoop.home.dir=/opt/yarn/hadoop-2.5.1 -Dhadoop.root.logger=INFO,RFA >> -Dyarn.root.logger=INFO,RFA >> -Djava.library.path=/opt/yarn/hadoop-2.5.1/lib/native -classpath >> /opt/yarn/hadoop-2.5.1/etc/hadoop:/opt/yarn/hadoop-2.5.1/etc/hadoop:/opt/yarn/hadoop-2.5.1/etc/hadoop:/opt/yarn/hadoop-2.5.1/share/hadoop/common/lib/*:/opt/yarn/hadoop-2.5.1/share/hadoop/common/*:/opt/yarn/hadoop-2.5.1/share/hadoop/hdfs:/opt/yarn/hadoop-2.5.1/share/hadoop/hdfs/lib/*:/opt/yarn/hadoop-2.5.1/share/hadoop/hdfs/*:/opt/yarn/hadoop-2.5.1/share/hadoop/yarn/lib/*:/opt/yarn/hadoop-2.5.1/share/hadoop/yarn/*:/opt/yarn/hadoop-2.5.1/share/hadoop/mapreduce/lib/*:/opt/yarn/hadoop-2.5.1/share/hadoop/mapreduce/*::/opt/yarn/hadoop-2.5.1/share/myLib/*.jar:/Users/ale/extprj/giraph/myGiraph/*.jar:/Users/ale/extprj/giraph/myGiraph/lib/*.jar:/contrib/capacity-scheduler/*.jar:/opt/yarn/hadoop-2.5.1/share/myLib/*.jar:/Users/ale/extprj/giraph/myGiraph/*.jar:/Users/ale/extprj/giraph/myGiraph/lib/*.jar:/contrib/capacity-scheduler/*.jar:/opt/yarn/hadoop-2.5.1/share/hadoop/yarn/*:/opt/yarn/hadoop-2.5.1/share/hadoop/yarn/lib/*:/opt/yarn/hadoop-2.5.1/etc/hadoop/nm-config/log4j.properties >> org.apache.hadoop.yarn.server.nodemanager.NodeManager >> >> So I run it as yarn user in this way: >> >> sudo -u yarn …. >> >> Moreover I have just notice that I added the following lines into the >> hadoop-env.sh >> >> hadoop-env.sh:export >> MYLIB=/opt/yarn/hadoop-2.5.1/share/myLib/*.jar:/Users/ale/extprj/giraph/myGiraph/*.jar:/Users/ale/extprj/giraph/myGiraph/lib/*.jar >> >> Accordingly to this other thread: >> >> http://mail-archives.apache.org/mod_mbox/giraph-user/201408.mbox/%3c53f4c689.5060...@web.de%3E >> >> Let me know, >> Alessandro >> >> >> Il giorno 20/nov/2014, alle ore 23:51, D Adams <dadam...@gmail.com> ha >> scritto: >> >>> --f46d044283ecf837440508522939 >>> Content-Type: text/plain; charset=UTF-8 >>> Content-Transfer-Encoding: quoted-printable >>> >>> Ok, in the following, I use bin/yarn. Doesn't seem like much has changed: >>> >>> hduser@Roosevelt:/usr/local/hadoop$ ./hduser_jobs.sh >>> Running hduser script ... >>> bin/yarn jar >>> /usr/local/giraph/giraph-examples/target/giraph-examples-1.1.0-SNAPSHOT-for= >>> -hadoop-2.5.1-jar-with-dependencies.jar >>> org.apache.giraph.GiraphRunner >>> org.apache.giraph.examples.SimpleShortestPathsComputation -vif >>> org.apache.giraph.io.formats.JsonLongDoubleFloatDoubleVertexInputFormat >>> -vip /user/hduser/input/tiny_graph.txt -vof >>> org.apache.giraph.io.formats.IdWithValueTextOutputFormat -op >>> /user/hduser/output/shortestpaths -w 1 -yj >>> giraph-examples-1.1.0-SNAPSHOT-for-hadoop-2.5.1-jar-with-dependencies.jar >>> 14/11/18 05:36:05 WARN util.NativeCodeLoader: Unable to load native-hadoop >>> library for your platform... using builtin-java classes where applicable >>> 14/11/18 05:36:08 INFO utils.ConfigurationUtils: No edge input format >>> specified. Ensure your InputFormat does not require one. >>> 14/11/18 05:36:08 INFO utils.ConfigurationUtils: No edge output format >>> specified. Ensure your OutputFormat does not require one. >>> 14/11/18 05:36:08 INFO yarn.GiraphYarnClient: Final output path is: >>> hdfs://localhost:9000/user/hduser/output/shortestpaths >>> 14/11/18 05:36:08 INFO yarn.GiraphYarnClient: Running Client >>> 14/11/18 05:36:08 INFO client.RMProxy: Connecting to ResourceManager at >>> localhost/127.0.0.1:8050 >>> 14/11/18 05:36:09 INFO yarn.GiraphYarnClient: Got node report from ASM for, >>> nodeId=3Droosevelt:60486, nodeAddress roosevelt:8042, nodeRackName >>> /default-rack, nodeNumContainers 0 >>> 14/11/18 05:36:09 INFO yarn.GiraphYarnClient: Defaulting per-task heap size >>> to 1024MB. >>> 14/11/18 05:36:09 INFO yarn.GiraphYarnClient: Obtained new Application ID: >>> application_1416310496070_0001 >>> 14/11/18 05:36:09 INFO Configuration.deprecation: mapred.job.id is >>> deprecated. Instead, use mapreduce.job.id >>> 14/11/18 05:36:09 INFO yarn.GiraphYarnClient: Set the environment for the >>> application master >>> 14/11/18 05:36:09 INFO yarn.GiraphYarnClient: Environment for AM >>> :{CLASSPATH=3D${CLASSPATH}:./*:$HADOOP_CONF_DIR:$HADOOP_COMMON_HOME/share/h= >>> adoop/common/*:$HADOOP_COMMON_HOME/share/hadoop/common/lib/*:$HADOOP_HDFS_H= >>> OME/share/hadoop/hdfs/*:$HADOOP_HDFS_HOME/share/hadoop/hdfs/lib/*:$HADOOP_Y= >>> ARN_HOME/share/hadoop/yarn/*:$HADOOP_YARN_HOME/share/hadoop/yarn/lib/*:$HAD= >>> OOP_MAPRED_HOME/share/hadoop/mapreduce/*:$HADOOP_MAPRED_HOME/share/hadoop/m= >>> apreduce/lib/*} >>> 14/11/18 05:36:09 INFO yarn.GiraphYarnClient: buildLocalResourceMap .... >>> 14/11/18 05:36:12 INFO yarn.YarnUtils: Registered file in LocalResources :: >>> hdfs://localhost:9000/user/hduser/giraph_yarn_jar_cache/application_1416310= >>> 496070_0001/giraph-conf.xml >>> 14/11/18 05:36:12 INFO yarn.GiraphYarnClient: LIB JARS >>> :giraph-examples-1.1.0-SNAPSHOT-for-hadoop-2.5.1-jar-with-dependencies.jar >>> 14/11/18 05:36:12 INFO yarn.YarnUtils: Class path name . >>> 14/11/18 05:36:12 INFO yarn.YarnUtils: base path checking . >>> 14/11/18 05:36:12 INFO yarn.GiraphYarnClient: Made local resource for >>> :/usr/local/hadoop/share/giraph-examples-1.1.0-SNAPSHOT-for-hadoop-2.5.1-ja= >>> r-with-dependencies.jar >>> to >>> hdfs://localhost:9000/user/hduser/giraph_yarn_jar_cache/application_1416310= >>> 496070_0001/giraph-examples-1.1.0-SNAPSHOT-for-hadoop-2.5.1-jar-with-depend= >>> encies.jar >>> 14/11/18 05:36:16 INFO yarn.YarnUtils: Registered file in LocalResources :: >>> hdfs://localhost:9000/user/hduser/giraph_yarn_jar_cache/application_1416310= >>> 496070_0001/giraph-examples-1.1.0-SNAPSHOT-for-hadoop-2.5.1-jar-with-depend= >>> encies.jar >>> 14/11/18 05:36:16 INFO yarn.GiraphYarnClient: ApplicationSumbissionContext >>> for GiraphApplicationMaster launch container is populated. >>> 14/11/18 05:36:16 INFO yarn.GiraphYarnClient: Submitting application to ASM >>> 14/11/18 05:36:16 INFO impl.YarnClientImpl: Submitted application >>> application_1416310496070_0001 >>> 14/11/18 05:36:16 INFO yarn.GiraphYarnClient: Got new appId after >>> submission :application_1416310496070_0001 >>> 14/11/18 05:36:16 INFO yarn.GiraphYarnClient: GiraphApplicationMaster >>> container request was submitted to ResourceManager for job: Giraph: >>> org.apache.giraph.examples.SimpleShortestPathsComputation >>> 14/11/18 05:36:17 INFO yarn.GiraphYarnClient: Giraph: >>> org.apache.giraph.examples.SimpleShortestPathsComputation, Elapsed: 1.01 >>> secs >>> 14/11/18 05:36:17 INFO yarn.GiraphYarnClient: >>> appattempt_1416310496070_0001_000001, State: ACCEPTED, Containers used: 1 >>> 14/11/18 05:36:22 INFO yarn.GiraphYarnClient: Giraph: >>> org.apache.giraph.examples.SimpleShortestPathsComputation, Elapsed: 5.86 >>> secs >>> 14/11/18 05:36:22 INFO yarn.GiraphYarnClient: >>> appattempt_1416310496070_0001_000001, State: ACCEPTED, Containers used: 1 >>> 14/11/18 05:36:26 INFO yarn.GiraphYarnClient: Giraph: >>> org.apache.giraph.examples.SimpleShortestPathsComputation, Elapsed: 10.02 >>> secs >>> 14/11/18 05:36:26 INFO yarn.GiraphYarnClient: >>> appattempt_1416310496070_0001_000001, State: ACCEPTED, Containers used: 1 >>> 14/11/18 05:36:30 INFO yarn.GiraphYarnClient: Giraph: >>> org.apache.giraph.examples.SimpleShortestPathsComputation, Elapsed: 14.39 >>> secs >>> 14/11/18 05:36:30 INFO yarn.GiraphYarnClient: >>> appattempt_1416310496070_0001_000001, State: ACCEPTED, Containers used: 1 >>> 14/11/18 05:36:34 INFO yarn.GiraphYarnClient: Giraph: >>> org.apache.giraph.examples.SimpleShortestPathsComputation, Elapsed: 18.57 >>> secs >>> 14/11/18 05:36:34 INFO yarn.GiraphYarnClient: >>> appattempt_1416310496070_0001_000001, State: RUNNING, Containers used: 2 >>> >>> gam_stderr.log >>> >>> LF4J: Class path contains multiple SLF4J bindings. >>> SLF4J: Found binding in >>> [jar:file:/app/hadoop/tmp/nm-local-dir/usercache/hduser/appcache/applicatio= >>> n_1416310496070_0001/filecache/10/giraph-examples-1.1.0-SNAPSHOT-for-hadoop= >>> -2.5.1-jar-with-dependencies.jar!/org/slf4j/impl/StaticLoggerBinder.class] >>> SLF4J: Found binding in >>> [jar:file:/usr/local/hadoop/share/hadoop/common/lib/slf4j-log4j12-1.7.5.jar= >>> !/org/slf4j/impl/StaticLoggerBinder.class] >>> SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explana= >>> tion. >>> SLF4J: Actual binding is of type [org.slf4j.impl.Log4jLoggerFactory] >>> >>> gam_stdout.log >>> >>> 2014-11-18 05:36:28,143 INFO [main] yarn.GiraphApplicationMaster >>> (GiraphApplicationMaster.java:main(421)) - Starting GitaphAM >>> 2014-11-18 05:36:30,404 WARN [main] util.NativeCodeLoader >>> (NativeCodeLoader.java:<clinit>(62)) - Unable to load native-hadoop >>> library for your platform... using builtin-java classes where >>> applicable >>> 2014-11-18 05:36:32,649 INFO [main] yarn.GiraphApplicationMaster >>> (GiraphApplicationMaster.java:<init>(168)) - GiraphAM for ContainerId >>> container_1416310496070_0001_01_000001 ApplicationAttemptId >>> appattempt_1416310496070_0001_000001 >>> 2014-11-18 05:36:32,852 INFO [main] client.RMProxy >>> (RMProxy.java:createRMProxy(92)) - Connecting to ResourceManager at >>> localhost/127.0.0.1:8030 >>> 2014-11-18 05:36:32,888 INFO [main] impl.NMClientAsyncImpl >>> (NMClientAsyncImpl.java:serviceInit(107)) - Upper bound of the thread >>> pool size is 500 >>> 2014-11-18 05:36:32,889 INFO [main] >>> impl.ContainerManagementProtocolProxy >>> (ContainerManagementProtocolProxy.java:<init>(78)) - >>> yarn.client.max-nodemanagers-proxies : 500 >>> 2014-11-18 05:36:33,589 INFO [main] yarn.GiraphApplicationMaster >>> (GiraphApplicationMaster.java:setupContainerAskForRM(279)) - Requested >>> container ask: Capability[<memory:1024, vCores:0>]Priority[10] >>> 2014-11-18 05:36:33,617 INFO [main] yarn.GiraphApplicationMaster >>> (GiraphApplicationMaster.java:setupContainerAskForRM(279)) - Requested >>> container ask: Capability[<memory:1024, vCores:0>]Priority[10] >>> 2014-11-18 05:36:33,617 INFO [main] yarn.GiraphApplicationMaster >>> (GiraphApplicationMaster.java:run(185)) - Wait to finish .. >>> 2014-11-18 05:36:35,674 INFO [AMRM Heartbeater thread] >>> impl.AMRMClientImpl (AMRMClientImpl.java:populateNMTokens(299)) - >>> Received new token for : roosevelt:60486 >>> 2014-11-18 05:36:35,681 INFO [AMRM Callback Handler Thread] >>> yarn.GiraphApplicationMaster >>> (GiraphApplicationMaster.java:onContainersAllocated(605)) - Got >>> response from RM for container ask, allocatedCnt=3D1 >>> 2014-11-18 05:36:35,681 INFO [AMRM Callback Handler Thread] >>> yarn.GiraphApplicationMaster >>> (GiraphApplicationMaster.java:onContainersAllocated(608)) - Total >>> allocated # of container so far : 1 allocated out of 2 required. >>> 2014-11-18 05:36:35,681 INFO [AMRM Callback Handler Thread] >>> yarn.GiraphApplicationMaster >>> (GiraphApplicationMaster.java:startContainerLaunchingThreads(359)) - >>> Launching command on a new container., >>> containerId=3Dcontainer_1416310496070_0001_01_000002, >>> containerNode=3Droosevelt:60486, containerNodeURI=3Droosevelt:8042, >>> containerResourceMemory=3D1024 >>> 2014-11-18 05:36:35,688 INFO [pool-4-thread-1] >>> yarn.GiraphApplicationMaster >>> (GiraphApplicationMaster.java:buildContainerLaunchContext(492)) - >>> Setting up container launch container for >>> containerid=3Dcontainer_1416310496070_0001_01_000002 >>> 2014-11-18 05:36:35,720 INFO [pool-4-thread-1] >>> yarn.GiraphApplicationMaster >>> (GiraphApplicationMaster.java:buildContainerLaunchContext(498)) - >>> Conatain launch Commands :java -Xmx1024M -Xms1024M -cp .:${CLASSPATH} >>> org.apache.giraph.yarn.GiraphYarnTask 1416310496070 1 2 1 >>> 1><LOG_DIR>/task-2-stdout.log 2><LOG_DIR>/task-2-stderr.log >>> 2014-11-18 05:36:35,721 INFO [pool-4-thread-1] >>> yarn.GiraphApplicationMaster >>> (GiraphApplicationMaster.java:buildContainerLaunchContext(518)) - >>> Setting username in ContainerLaunchContext to: hduser >>> 2014-11-18 05:36:36,789 INFO [AMRM Callback Handler Thread] >>> yarn.GiraphApplicationMaster >>> (GiraphApplicationMaster.java:onContainersAllocated(605)) - Got >>> response from RM for container ask, allocatedCnt=3D1 >>> 2014-11-18 05:36:36,789 INFO [AMRM Callback Handler Thread] >>> yarn.GiraphApplicationMaster >>> (GiraphApplicationMaster.java:onContainersAllocated(608)) - Total >>> allocated # of container so far : 2 allocated out of 2 required. >>> 2014-11-18 05:36:36,790 INFO [AMRM Callback Handler Thread] >>> yarn.GiraphApplicationMaster >>> (GiraphApplicationMaster.java:startContainerLaunchingThreads(359)) - >>> Launching command on a new container., >>> containerId=3Dcontainer_1416310496070_0001_01_000003, >>> containerNode=3Droosevelt:60486, containerNodeURI=3Droosevelt:8042, >>> containerResourceMemory=3D1024 >>> 2014-11-18 05:36:37,227 INFO [pool-4-thread-2] >>> yarn.GiraphApplicationMaster >>> (GiraphApplicationMaster.java:buildContainerLaunchContext(492)) - >>> Setting up container launch container for >>> containerid=3Dcontainer_1416310496070_0001_01_000003 >>> 2014-11-18 05:36:37,227 INFO [pool-4-thread-2] >>> yarn.GiraphApplicationMaster >>> (GiraphApplicationMaster.java:buildContainerLaunchContext(498)) - >>> Conatain launch Commands :java -Xmx1024M -Xms1024M -cp .:${CLASSPATH} >>> org.apache.giraph.yarn.GiraphYarnTask 1416310496070 1 3 1 >>> 1><LOG_DIR>/task-3-stdout.log 2><LOG_DIR>/task-3-stderr.log >>> 2014-11-18 05:36:37,227 INFO [pool-4-thread-2] >>> yarn.GiraphApplicationMaster >>> (GiraphApplicationMaster.java:buildContainerLaunchContext(518)) - >>> Setting username in ContainerLaunchContext to: hduser >>> 2014-11-18 05:36:37,463 INFO [pool-4-thread-1] yarn.YarnUtils >>> (YarnUtils.java:addFsResourcesToMap(72)) - Adding >>> giraph-examples-1.1.0-SNAPSHOT-for-hadoop-2.5.1-jar-with-dependencies.jar >>> to LocalResources for export.to >>> hdfs://localhost:9000/user/hduser/giraph_yarn_jar_cache/application_1416310= >>> 496070_0001/giraph-examples-1.1.0-SNAPSHOT-for-hadoop-2.5.1-jar-with-depend= >>> encies.jar >>> 2014-11-18 05:36:37,678 INFO [pool-4-thread-1] yarn.YarnUtils >>> (YarnUtils.java:addFileToResourceMap(160)) - Registered file in >>> LocalResources :: >>> hdfs://localhost:9000/user/hduser/giraph_yarn_jar_cache/application_1416310= >>> 496070_0001/giraph-examples-1.1.0-SNAPSHOT-for-hadoop-2.5.1-jar-with-depend= >>> encies.jar >>> 2014-11-18 05:36:37,678 WARN [pool-4-thread-1] yarn.YarnUtils >>> (YarnUtils.java:addFsResourcesToMap(81)) - Job jars (-yj option) >>> didn't include giraph-core. >>> 2014-11-18 05:36:37,680 INFO [pool-4-thread-1] yarn.YarnUtils >>> (YarnUtils.java:addFileToResourceMap(160)) - Registered file in >>> LocalResources :: >>> hdfs://localhost:9000/user/hduser/giraph_yarn_jar_cache/application_1416310= >>> 496070_0001/giraph-conf.xml >>> 2014-11-18 05:36:37,694 INFO >>> [org.apache.hadoop.yarn.client.api.async.impl.NMClientAsyncImpl #0] >>> impl.NMClientAsyncImpl (NMClientAsyncImpl.java:run(531)) - Processing >>> Event EventType: START_CONTAINER for Container >>> container_1416310496070_0001_01_000002 >>> 2014-11-18 05:36:37,699 INFO >>> [org.apache.hadoop.yarn.client.api.async.impl.NMClientAsyncImpl #1] >>> impl.NMClientAsyncImpl (NMClientAsyncImpl.java:run(531)) - Processing >>> Event EventType: START_CONTAINER for Container >>> container_1416310496070_0001_01_000003 >>> 2014-11-18 05:36:37,698 INFO >>> [org.apache.hadoop.yarn.client.api.async.impl.NMClientAsyncImpl #0] >>> impl.ContainerManagementProtocolProxy >>> (ContainerManagementProtocolProxy.java:newProxy(212)) - Opening proxy >>> : roosevelt:60486 >>> 2014-11-18 05:36:37,761 INFO >>> [org.apache.hadoop.yarn.client.api.async.impl.NMClientAsyncImpl #2] >>> impl.NMClientAsyncImpl (NMClientAsyncImpl.java:run(531)) - Processing >>> Event EventType: QUERY_CONTAINER for Container >>> container_1416310496070_0001_01_000003 >>> 2014-11-18 05:36:37,771 INFO >>> [org.apache.hadoop.yarn.client.api.async.impl.NMClientAsyncImpl #3] >>> impl.NMClientAsyncImpl (NMClientAsyncImpl.java:run(531)) - Processing >>> Event EventType: QUERY_CONTAINER for Container >>> container_1416310496070_0001_01_000002 >>> >>> >>> >>> On Thu, Nov 20, 2014 at 1:32 PM, Alessandro Negro <alenegr...@yahoo.it> >>> wrote: >>> >>>> Hi Das, >>>> I mean as user you run Hadoop yarn resource manager. >>>> >>>> Alessandro >>>> >>>> Il giorno 20/nov/2014, alle ore 19:22, D Adams <dadam...@gmail.com> ha >>>> scritto: >>>> >>>> Alessandro, >>>> I'm not sure what you mean, should I create a new user on my system >>>> named 'yarn'? I'm new to both hadoop and giraph, so I'm not sure. >>>> >>>> Tripti, >>>> I'll get those logs as soon as I can. >>>> >>>> Thank you both, >>>> >>>> V/r >>>> Das >>>> On Nov 20, 2014 3:13 AM, "Alessandro Negro" <alenegr...@yahoo.it> wrote: >>>> >>>>> Hi Tripti, >>>>> I agree that a more detailed error log could be useful. >>>>> >>>>> Thanks, >>>>> Alessandro >>>>> >>>>> >>>>> >>>>> >>>>> Il giorno 19/nov/2014, alle ore 15:54, Tripti Singh <tri...@yahoo-inc.co= >>> m> >>>>> ha scritto: >>>>> >>>>>> --_000_D092AB9C1CB42triptiyahooinccom_ >>>>>> Content-Type: text/plain; charset=3D"Windows-1252" >>>>>> Content-Transfer-Encoding: quoted-printable >>>>>> >>>>>> Hi Alessandro, >>>>>> >>>>>> I guess u r talking about the documentation on Giraph webpage which >>>>> mention=3D >>>>>> s addition of this new option which is more or less mandatory for yarn >>>>> base=3D >>>>>> d profiles. >>>>>> When I first ran the Giraph on yarn without the proper =3D96yj option, >>>>> there =3D >>>>>> was no way I could figure out from the Application/Container logs that >>>>> this=3D >>>>>> was the issue. >>>>>> I think, it=3D92ll be a good idea to have this message in the logs for >>>>> easy d=3D >>>>>> ebugging. >>>>>> >>>>>> Thanks, >>>>>> Tripti. >>>>>> >>>>>> >>>>>> From: Alessandro Negro <alenegr...@yahoo.it<mailto:alenegr...@yahoo.it >>>>>>> >>>>>> Reply-To: "user@giraph.apache.org<mailto:user@giraph.apache.org>" >>>>> <user@gir=3D >>>>>> aph.apache.org<mailto:user@giraph.apache.org>> >>>>>> Date: Tuesday, November 18, 2014 at 4:31 PM >>>>>> To: "user@giraph.apache.org<mailto:user@giraph.apache.org>" < >>>>> u...@giraph.ap=3D >>>>>> ache.org<mailto:user@giraph.apache.org>> >>>>>> Subject: Re: Help with Giraph on Yarn >>>>>> >>>>>> Hi Eli, >>>>>> no I think that the help message is clear enough, generally when I rea= >>> d >>>>> =3D93=3D >>>>>> jar=3D94 I always mean absolute path for jar, but this is a my persona= >>> l >>>>> misun=3D >>>>>> derstanding. >>>>>> >>>>>> Thanks, >>>>>> Alessandro >>>>>> >>>>>> >>>>>> Il giorno 16/nov/2014, alle ore 21:39, Eli Reisman < >>>>> apache.mail...@gmail.co=3D >>>>>> m<mailto:apache.mail...@gmail.com>> ha scritto: >>>>>> >>>>>> Sounds like you got it figured out since last time I checked this list= >>> , >>>>> sor=3D >>>>>> ry it was a pain. Feel free to drop a JIRA ticket if we can make help >>>>> messa=3D >>>>>> ge etc. for -yj option more clear, there's lots to improve there. >>>>>> >>>>>> On Sat, Nov 8, 2014 at 7:26 AM, Alessandro Negro <alenegr...@yahoo.it >>>>> <mailt=3D >>>>>> o:alenegr...@yahoo.it>> wrote: >>>>>> Hi Tripti, >>>>>> finally I was able to run the test with success. It was an issue of >>>>> permiss=3D >>>>>> ion since I was running as ale not as yarn. >>>>>> Let me say that now I=3D92m able to run graph examples on Yarn 2.5.1. >>>>> This is=3D >>>>>> the final result: >>>>>> >>>>>> 14/11/08 16:24:00 INFO yarn.GiraphYarnClient: Completed Giraph: >>>>> org.apache.=3D >>>>>> giraph.examples.SimpleShortestPathsComputation: SUCCEEDED, total >>>>> running ti=3D >>>>>> me: 0 minutes, 21 seconds. >>>>>> >>>>>> >>>>>> Many thanks for your support, >>>>>> Alessandro >>>>>> >>>>>> Il giorno 06/nov/2014, alle ore 15:16, Tripti Singh < >>>>> tri...@yahoo-inc.com<m=3D >>>>>> ailto:tri...@yahoo-inc.com>> ha scritto: >>>>>> >>>>>> I don't know if u have access to this node. But if u do, u could check >>>>> if t=3D >>>>>> he file is indeed there and u have access to it. >>>>>> >>>>>> Sent from my iPhone >>>>>> >>>>>> On 06-Nov-2014, at 6:12 pm, "Alessandro Negro" <alenegr...@yahoo.it >>>>> <mailto:=3D >>>>>> alenegr...@yahoo.it>> wrote: >>>>>> >>>>>> You are right it works, but now I receive the following error: >>>>>> >>>>>> SLF4J: Class path contains multiple SLF4J bindings. >>>>>> SLF4J: Found binding in >>>>> [jar:file:/private/tmp/hadoop-yarn/nm-local-dir/use=3D >>>>>> >>>>> rcache/ale/appcache/application_1415264041937_0009/filecache/10/giraph-e= >>> xam=3D >>>>>> >>>>> ples-1.1.0-SNAPSHOT-for-hadoop-2.5.1-jar-with-dependencies.jar!/org/slf4= >>> j/i=3D >>>>>> mpl/StaticLoggerBinder.class] >>>>>> SLF4J: Found binding in >>>>> [jar:file:/opt/yarn/hadoop-2.5.1/share/hadoop/commo=3D >>>>>> n/lib/slf4j-log4j12-1.7.5.jar!/org/slf4j/impl/StaticLoggerBinder.class= >>> ] >>>>>> SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an >>>>> explana=3D >>>>>> tion. >>>>>> SLF4J: Actual binding is of type [org.slf4j.impl.Log4jLoggerFactory] >>>>>> 2014-11-06 13:15:37.120 java[10158:1803] Unable to load realm info fro= >>> m >>>>> SCD=3D >>>>>> ynamicStore >>>>>> Exception in thread "pool-4-thread-1" java.lang.IllegalStateException: >>>>> Coul=3D >>>>>> d not configure the containerlaunch context for GiraphYarnTasks. >>>>>> at >>>>> org.apache.giraph.yarn.GiraphApplicationMaster.getTaskResourceMap(Giraph= >>> =3D >>>>>> ApplicationMaster.java:391) >>>>>> at >>>>> org.apache.giraph.yarn.GiraphApplicationMaster.access$500(GiraphApplicat= >>> =3D >>>>>> ionMaster.java:78) >>>>>> at >>>>> org.apache.giraph.yarn.GiraphApplicationMaster$LaunchContainerRunnable.b= >>> =3D >>>>>> uildContainerLaunchContext(GiraphApplicationMaster.java:522) >>>>>> at >>>>> org.apache.giraph.yarn.GiraphApplicationMaster$LaunchContainerRunnable.r= >>> =3D >>>>>> un(GiraphApplicationMaster.java:479) >>>>>> at >>>>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.jav= >>> =3D >>>>>> a:1145) >>>>>> at >>>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.ja= >>> =3D >>>>>> va:615) >>>>>> at java.lang.Thread.run(Thread.java:744) >>>>>> Caused by: java.io.FileNotFoundException: File does not exist: >>>>> hdfs://hadoo=3D >>>>>> >>>>> p-master:9000/user/yarn/giraph_yarn_jar_cache/application_1415264041937_= >>> 000=3D >>>>>> >>>>> 9/giraph-examples-1.1.0-SNAPSHOT-for-hadoop-2.5.1-jar-with-dependencies.= >>> jar >>>>>> at >>>>> org.apache.hadoop.hdfs.DistributedFileSystem$17.doCall(DistributedFileSy= >>> =3D >>>>>> stem.java:1072) >>>>>> at >>>>> org.apache.hadoop.hdfs.DistributedFileSystem$17.doCall(DistributedFileSy= >>> =3D >>>>>> stem.java:1064) >>>>>> at >>>>> org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolv= >>> =3D >>>>>> er.java:81) >>>>>> at >>>>> org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFi= >>> =3D >>>>>> leSystem.java:1064) >>>>>> at >>>>> org.apache.giraph.yarn.YarnUtils.addFileToResourceMap(YarnUtils.java:153= >>> =3D >>>>>> ) >>>>>> at >>>>> org.apache.giraph.yarn.YarnUtils.addFsResourcesToMap(YarnUtils.java:77) >>>>>> at >>>>> org.apache.giraph.yarn.GiraphApplicationMaster.getTaskResourceMap(Giraph= >>> =3D >>>>>> ApplicationMaster.java:387) >>>>>> ... 6 more >>>>>> >>>>>> >>>>>> That justify the other error I receive in the task: >>>>>> Could not find or load main class org.apache.giraph.yarn.GiraphYarnTas= >>> k >>>>>> >>>>>> Thanks, >>>>>> >>>>>> Il giorno 06/nov/2014, alle ore 13:07, Tripti Singh < >>>>> tri...@yahoo-inc.com<m=3D >>>>>> ailto:tri...@yahoo-inc.com>> ha scritto: >>>>>> >>>>>> Why r u adding two jars? Example jar ideally contains core library so >>>>> every=3D >>>>>> thing should be available with just one example jar included >>>>>> >>>>>> Sent from my iPhone >>>>>> >>>>>> On 06-Nov-2014, at 4:33 pm, "Alessandro Negro" <alenegr...@yahoo.it >>>>> <mailto:=3D >>>>>> alenegr...@yahoo.it>> wrote: >>>>>> >>>>>> Hi, >>>>>> now it seems better, I need to add: >>>>>> >>>>>> >>>>> giraph-1.1.0-SNAPSHOT-for-hadoop-2.5.1-jar-with-dependencies.jar,giraph-= >>> exa=3D >>>>>> mples-1.1.0-SNAPSHOT-for-hadoop-2.5.1-jar-with-dependencies.jar >>>>>> >>>>>> >>>>>> Now it seems that after a lot of cycle it fail with this error: >>>>>> >>>>>> Could not find or load main class org.apache.giraph.yarn.GiraphYarnTas= >>> k >>>>>> >>>>>> But in this case the error appear in task-3-stderr.log not in >>>>> gam-stderr.l=3D >>>>>> og where there is the following error: >>>>>> >>>>>> LF4J: Class path contains multiple SLF4J bindings. >>>>>> SLF4J: Found binding in >>>>> [jar:file:/private/tmp/hadoop-yarn/nm-local-dir/use=3D >>>>>> >>>>> rcache/ale/appcache/application_1415264041937_0006/filecache/12/giraph-1= >>> .1.=3D >>>>>> >>>>> 0-SNAPSHOT-for-hadoop-2.5.1-jar-with-dependencies.jar!/org/slf4j/impl/St= >>> ati=3D >>>>>> cLoggerBinder.class] >>>>>> SLF4J: Found binding in >>>>> [jar:file:/private/tmp/hadoop-yarn/nm-local-dir/use=3D >>>>>> >>>>> rcache/ale/appcache/application_1415264041937_0006/filecache/10/giraph-e= >>> xam=3D >>>>>> >>>>> ples-1.1.0-SNAPSHOT-for-hadoop-2.5.1-jar-with-dependencies.jar!/org/slf4= >>> j/i=3D >>>>>> mpl/StaticLoggerBinder.class] >>>>>> SLF4J: Found binding in >>>>> [jar:file:/opt/yarn/hadoop-2.5.1/share/hadoop/commo=3D >>>>>> n/lib/slf4j-log4j12-1.7.5.jar!/org/slf4j/impl/StaticLoggerBinder.class= >>> ] >>>>>> SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an >> > > >
