I'm sorry, please excuse my ignorance, but, I have no idea where to find my configuration. That is, I have no idea where these lines came from (or what file):
yarn 1029 0,1 1,2 3499572 209560 s001 S 10:29am 0:06.04 /Library/Java/ JavaVirtualMachines/jdk1.7.0_45.jdk/Contents/Home/bin/java -Dproc_resourcemanager -Xmx500m -Dhadoop.log.dir=/opt/yarn/hadoop-2.5.1/logs -Dyarn.log.dir=/opt/yarn/hadoop-2.5.1/logs ... As far as configuration goes, I'm only really aware of the etc/hadoop/*-env.sh and .xml files and main pom.xml file. Or is the above part of the command you use to run giraph jobs? I will make changes to hadoop-env.sh file as you suggest and let you know how it works out. On Fri, Nov 21, 2014 at 6:05 AM, Alessandro Negro <alenegr...@yahoo.it> wrote: > Hi, > I forgot to say that I added also this line: > > export HADOOP_CLASSPATH=$HADOOP_CLASSPATH:$MYLIB > > > Il giorno 21/nov/2014, alle ore 10:34, Alessandro Negro < > alenegr...@yahoo.it> ha scritto: > > Hi Das, > this is my configuration of hadoop: > > yarn 1029 0,1 1,2 3499572 209560 s001 S 10:29am > 0:06.04 > /Library/Java/JavaVirtualMachines/jdk1.7.0_45.jdk/Contents/Home/bin/java > -Dproc_resourcemanager -Xmx500m > -Dhadoop.log.dir=/opt/yarn/hadoop-2.5.1/logs > -Dyarn.log.dir=/opt/yarn/hadoop-2.5.1/logs > -Dhadoop.log.file=yarn-yarn-resourcemanager-MacBook-Pro-di-Alessandro.local.log > -Dyarn.log.file=yarn-yarn-resourcemanager-MacBook-Pro-di-Alessandro.local.log > -Dyarn.home.dir= -Dyarn.id.str=yarn -Dhadoop.root.logger=INFO,RFA > -Dyarn.root.logger=INFO,RFA > -Djava.library.path=/opt/yarn/hadoop-2.5.1/lib/native > -Dyarn.policy.file=hadoop-policy.xml > -Dhadoop.log.dir=/opt/yarn/hadoop-2.5.1/logs > -Dyarn.log.dir=/opt/yarn/hadoop-2.5.1/logs > -Dhadoop.log.file=yarn-yarn-resourcemanager-MacBook-Pro-di-Alessandro.local.log > -Dyarn.log.file=yarn-yarn-resourcemanager-MacBook-Pro-di-Alessandro.local.log > -Dyarn.home.dir=/opt/yarn/hadoop-2.5.1 > -Dhadoop.home.dir=/opt/yarn/hadoop-2.5.1 -Dhadoop.root.logger=INFO,RFA > -Dyarn.root.logger=INFO,RFA > -Djava.library.path=/opt/yarn/hadoop-2.5.1/lib/native -classpath > /opt/yarn/hadoop-2.5.1/etc/hadoop:/opt/yarn/hadoop-2.5.1/etc/hadoop:/opt/yarn/hadoop-2.5.1/etc/hadoop:/opt/yarn/hadoop-2.5.1/share/hadoop/common/lib/*:/opt/yarn/hadoop-2.5.1/share/hadoop/common/*:/opt/yarn/hadoop-2.5.1/share/hadoop/hdfs:/opt/yarn/hadoop-2.5.1/share/hadoop/hdfs/lib/*:/opt/yarn/hadoop-2.5.1/share/hadoop/hdfs/*:/opt/yarn/hadoop-2.5.1/share/hadoop/yarn/lib/*:/opt/yarn/hadoop-2.5.1/share/hadoop/yarn/*:/opt/yarn/hadoop-2.5.1/share/hadoop/mapreduce/lib/*:/opt/yarn/hadoop-2.5.1/share/hadoop/mapreduce/*::/opt/yarn/hadoop-2.5.1/share/myLib/*.jar:/Users/ale/extprj/giraph/myGiraph/*.jar:/Users/ale/extprj/giraph/myGiraph/lib/*.jar:/contrib/capacity-scheduler/*.jar:/opt/yarn/hadoop-2.5.1/share/myLib/*.jar:/Users/ale/extprj/giraph/myGiraph/*.jar:/Users/ale/extprj/giraph/myGiraph/lib/*.jar:/contrib/capacity-scheduler/*.jar:/opt/yarn/hadoop-2.5.1/share/hadoop/yarn/*:/opt/yarn/hadoop-2.5.1/share/hadoop/yarn/lib/*:/opt/yarn/hadoop-2.5.1/etc/hadoop/rm-config/log4j.properties > org.apache.hadoop.yarn.server.resourcemanager.ResourceManager > yarn 1073 0,1 1,0 3347132 174644 s001 S 10:29am > 0:04.77 > /Library/Java/JavaVirtualMachines/jdk1.7.0_45.jdk/Contents/Home/bin/java > -Dproc_nodemanager -Xmx500m -Dhadoop.log.dir=/opt/yarn/hadoop-2.5.1/logs > -Dyarn.log.dir=/opt/yarn/hadoop-2.5.1/logs > -Dhadoop.log.file=yarn-yarn-nodemanager-MacBook-Pro-di-Alessandro.local.log > -Dyarn.log.file=yarn-yarn-nodemanager-MacBook-Pro-di-Alessandro.local.log > -Dyarn.home.dir= -Dyarn.id.str=yarn -Dhadoop.root.logger=INFO,RFA > -Dyarn.root.logger=INFO,RFA > -Djava.library.path=/opt/yarn/hadoop-2.5.1/lib/native > -Dyarn.policy.file=hadoop-policy.xml -server > -Dhadoop.log.dir=/opt/yarn/hadoop-2.5.1/logs > -Dyarn.log.dir=/opt/yarn/hadoop-2.5.1/logs > -Dhadoop.log.file=yarn-yarn-nodemanager-MacBook-Pro-di-Alessandro.local.log > -Dyarn.log.file=yarn-yarn-nodemanager-MacBook-Pro-di-Alessandro.local.log > -Dyarn.home.dir=/opt/yarn/hadoop-2.5.1 > -Dhadoop.home.dir=/opt/yarn/hadoop-2.5.1 -Dhadoop.root.logger=INFO,RFA > -Dyarn.root.logger=INFO,RFA > -Djava.library.path=/opt/yarn/hadoop-2.5.1/lib/native -classpath > /opt/yarn/hadoop-2.5.1/etc/hadoop:/opt/yarn/hadoop-2.5.1/etc/hadoop:/opt/yarn/hadoop-2.5.1/etc/hadoop:/opt/yarn/hadoop-2.5.1/share/hadoop/common/lib/*:/opt/yarn/hadoop-2.5.1/share/hadoop/common/*:/opt/yarn/hadoop-2.5.1/share/hadoop/hdfs:/opt/yarn/hadoop-2.5.1/share/hadoop/hdfs/lib/*:/opt/yarn/hadoop-2.5.1/share/hadoop/hdfs/*:/opt/yarn/hadoop-2.5.1/share/hadoop/yarn/lib/*:/opt/yarn/hadoop-2.5.1/share/hadoop/yarn/*:/opt/yarn/hadoop-2.5.1/share/hadoop/mapreduce/lib/*:/opt/yarn/hadoop-2.5.1/share/hadoop/mapreduce/*::/opt/yarn/hadoop-2.5.1/share/myLib/*.jar:/Users/ale/extprj/giraph/myGiraph/*.jar:/Users/ale/extprj/giraph/myGiraph/lib/*.jar:/contrib/capacity-scheduler/*.jar:/opt/yarn/hadoop-2.5.1/share/myLib/*.jar:/Users/ale/extprj/giraph/myGiraph/*.jar:/Users/ale/extprj/giraph/myGiraph/lib/*.jar:/contrib/capacity-scheduler/*.jar:/opt/yarn/hadoop-2.5.1/share/hadoop/yarn/*:/opt/yarn/hadoop-2.5.1/share/hadoop/yarn/lib/*:/opt/yarn/hadoop-2.5.1/etc/hadoop/nm-config/log4j.properties > org.apache.hadoop.yarn.server.nodemanager.NodeManager > > So I run it as yarn user in this way: > > sudo -u yarn …. > > Moreover I have just notice that I added the following lines into the > hadoop-env.sh > > hadoop-env.sh:export > MYLIB=/opt/yarn/hadoop-2.5.1/share/myLib/*.jar:/Users/ale/extprj/giraph/myGiraph/*.jar:/Users/ale/extprj/giraph/myGiraph/lib/*.jar > > Accordingly to this other thread: > > > http://mail-archives.apache.org/mod_mbox/giraph-user/201408.mbox/%3c53f4c689.5060...@web.de%3E > > Let me know, > Alessandro > > > Il giorno 20/nov/2014, alle ore 23:51, D Adams <dadam...@gmail.com> ha > scritto: > > --f46d044283ecf837440508522939 > Content-Type: text/plain; charset=UTF-8 > Content-Transfer-Encoding: quoted-printable > > Ok, in the following, I use bin/yarn. Doesn't seem like much has changed: > > hduser@Roosevelt:/usr/local/hadoop$ ./hduser_jobs.sh > Running hduser script ... > bin/yarn jar > > /usr/local/giraph/giraph-examples/target/giraph-examples-1.1.0-SNAPSHOT-for= > -hadoop-2.5.1-jar-with-dependencies.jar > org.apache.giraph.GiraphRunner > org.apache.giraph.examples.SimpleShortestPathsComputation -vif > org.apache.giraph.io.formats.JsonLongDoubleFloatDoubleVertexInputFormat > -vip /user/hduser/input/tiny_graph.txt -vof > org.apache.giraph.io.formats.IdWithValueTextOutputFormat -op > /user/hduser/output/shortestpaths -w 1 -yj > giraph-examples-1.1.0-SNAPSHOT-for-hadoop-2.5.1-jar-with-dependencies.jar > 14/11/18 05:36:05 WARN util.NativeCodeLoader: Unable to load native-hadoop > library for your platform... using builtin-java classes where applicable > 14/11/18 05:36:08 INFO utils.ConfigurationUtils: No edge input format > specified. Ensure your InputFormat does not require one. > 14/11/18 05:36:08 INFO utils.ConfigurationUtils: No edge output format > specified. Ensure your OutputFormat does not require one. > 14/11/18 05:36:08 INFO yarn.GiraphYarnClient: Final output path is: > hdfs://localhost:9000/user/hduser/output/shortestpaths > 14/11/18 05:36:08 INFO yarn.GiraphYarnClient: Running Client > 14/11/18 05:36:08 INFO client.RMProxy: Connecting to ResourceManager at > localhost/127.0.0.1:8050 > 14/11/18 05:36:09 INFO yarn.GiraphYarnClient: Got node report from ASM for, > nodeId=3Droosevelt:60486, nodeAddress roosevelt:8042, nodeRackName > /default-rack, nodeNumContainers 0 > 14/11/18 05:36:09 INFO yarn.GiraphYarnClient: Defaulting per-task heap size > to 1024MB. > 14/11/18 05:36:09 INFO yarn.GiraphYarnClient: Obtained new Application ID: > application_1416310496070_0001 > 14/11/18 05:36:09 INFO Configuration.deprecation: mapred.job.id is > deprecated. Instead, use mapreduce.job.id > 14/11/18 05:36:09 INFO yarn.GiraphYarnClient: Set the environment for the > application master > 14/11/18 05:36:09 INFO yarn.GiraphYarnClient: Environment for AM > > :{CLASSPATH=3D${CLASSPATH}:./*:$HADOOP_CONF_DIR:$HADOOP_COMMON_HOME/share/h= > > adoop/common/*:$HADOOP_COMMON_HOME/share/hadoop/common/lib/*:$HADOOP_HDFS_H= > > OME/share/hadoop/hdfs/*:$HADOOP_HDFS_HOME/share/hadoop/hdfs/lib/*:$HADOOP_Y= > > ARN_HOME/share/hadoop/yarn/*:$HADOOP_YARN_HOME/share/hadoop/yarn/lib/*:$HAD= > > OOP_MAPRED_HOME/share/hadoop/mapreduce/*:$HADOOP_MAPRED_HOME/share/hadoop/m= > apreduce/lib/*} > 14/11/18 05:36:09 INFO yarn.GiraphYarnClient: buildLocalResourceMap .... > 14/11/18 05:36:12 INFO yarn.YarnUtils: Registered file in LocalResources :: > > hdfs://localhost:9000/user/hduser/giraph_yarn_jar_cache/application_1416310= > 496070_0001/giraph-conf.xml > 14/11/18 05:36:12 INFO yarn.GiraphYarnClient: LIB JARS > :giraph-examples-1.1.0-SNAPSHOT-for-hadoop-2.5.1-jar-with-dependencies.jar > 14/11/18 05:36:12 INFO yarn.YarnUtils: Class path name . > 14/11/18 05:36:12 INFO yarn.YarnUtils: base path checking . > 14/11/18 05:36:12 INFO yarn.GiraphYarnClient: Made local resource for > > :/usr/local/hadoop/share/giraph-examples-1.1.0-SNAPSHOT-for-hadoop-2.5.1-ja= > r-with-dependencies.jar > to > > hdfs://localhost:9000/user/hduser/giraph_yarn_jar_cache/application_1416310= > > 496070_0001/giraph-examples-1.1.0-SNAPSHOT-for-hadoop-2.5.1-jar-with-depend= > encies.jar > 14/11/18 05:36:16 INFO yarn.YarnUtils: Registered file in LocalResources :: > > hdfs://localhost:9000/user/hduser/giraph_yarn_jar_cache/application_1416310= > > 496070_0001/giraph-examples-1.1.0-SNAPSHOT-for-hadoop-2.5.1-jar-with-depend= > encies.jar > 14/11/18 05:36:16 INFO yarn.GiraphYarnClient: ApplicationSumbissionContext > for GiraphApplicationMaster launch container is populated. > 14/11/18 05:36:16 INFO yarn.GiraphYarnClient: Submitting application to ASM > 14/11/18 05:36:16 INFO impl.YarnClientImpl: Submitted application > application_1416310496070_0001 > 14/11/18 05:36:16 INFO yarn.GiraphYarnClient: Got new appId after > submission :application_1416310496070_0001 > 14/11/18 05:36:16 INFO yarn.GiraphYarnClient: GiraphApplicationMaster > container request was submitted to ResourceManager for job: Giraph: > org.apache.giraph.examples.SimpleShortestPathsComputation > 14/11/18 05:36:17 INFO yarn.GiraphYarnClient: Giraph: > org.apache.giraph.examples.SimpleShortestPathsComputation, Elapsed: 1.01 > secs > 14/11/18 05:36:17 INFO yarn.GiraphYarnClient: > appattempt_1416310496070_0001_000001, State: ACCEPTED, Containers used: 1 > 14/11/18 05:36:22 INFO yarn.GiraphYarnClient: Giraph: > org.apache.giraph.examples.SimpleShortestPathsComputation, Elapsed: 5.86 > secs > 14/11/18 05:36:22 INFO yarn.GiraphYarnClient: > appattempt_1416310496070_0001_000001, State: ACCEPTED, Containers used: 1 > 14/11/18 05:36:26 INFO yarn.GiraphYarnClient: Giraph: > org.apache.giraph.examples.SimpleShortestPathsComputation, Elapsed: 10.02 > secs > 14/11/18 05:36:26 INFO yarn.GiraphYarnClient: > appattempt_1416310496070_0001_000001, State: ACCEPTED, Containers used: 1 > 14/11/18 05:36:30 INFO yarn.GiraphYarnClient: Giraph: > org.apache.giraph.examples.SimpleShortestPathsComputation, Elapsed: 14.39 > secs > 14/11/18 05:36:30 INFO yarn.GiraphYarnClient: > appattempt_1416310496070_0001_000001, State: ACCEPTED, Containers used: 1 > 14/11/18 05:36:34 INFO yarn.GiraphYarnClient: Giraph: > org.apache.giraph.examples.SimpleShortestPathsComputation, Elapsed: 18.57 > secs > 14/11/18 05:36:34 INFO yarn.GiraphYarnClient: > appattempt_1416310496070_0001_000001, State: RUNNING, Containers used: 2 > > gam_stderr.log > > LF4J: Class path contains multiple SLF4J bindings. > SLF4J: Found binding in > > [jar:file:/app/hadoop/tmp/nm-local-dir/usercache/hduser/appcache/applicatio= > > n_1416310496070_0001/filecache/10/giraph-examples-1.1.0-SNAPSHOT-for-hadoop= > -2.5.1-jar-with-dependencies.jar!/org/slf4j/impl/StaticLoggerBinder.class] > SLF4J: Found binding in > > [jar:file:/usr/local/hadoop/share/hadoop/common/lib/slf4j-log4j12-1.7.5.jar= > !/org/slf4j/impl/StaticLoggerBinder.class] > SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an > explana= > tion. > SLF4J: Actual binding is of type [org.slf4j.impl.Log4jLoggerFactory] > > gam_stdout.log > > 2014-11-18 05:36:28,143 INFO [main] yarn.GiraphApplicationMaster > (GiraphApplicationMaster.java:main(421)) - Starting GitaphAM > 2014-11-18 05:36:30,404 WARN [main] util.NativeCodeLoader > (NativeCodeLoader.java:<clinit>(62)) - Unable to load native-hadoop > library for your platform... using builtin-java classes where > applicable > 2014-11-18 05:36:32,649 INFO [main] yarn.GiraphApplicationMaster > (GiraphApplicationMaster.java:<init>(168)) - GiraphAM for ContainerId > container_1416310496070_0001_01_000001 ApplicationAttemptId > appattempt_1416310496070_0001_000001 > 2014-11-18 05:36:32,852 INFO [main] client.RMProxy > (RMProxy.java:createRMProxy(92)) - Connecting to ResourceManager at > localhost/127.0.0.1:8030 > 2014-11-18 05:36:32,888 INFO [main] impl.NMClientAsyncImpl > (NMClientAsyncImpl.java:serviceInit(107)) - Upper bound of the thread > pool size is 500 > 2014-11-18 05:36:32,889 INFO [main] > impl.ContainerManagementProtocolProxy > (ContainerManagementProtocolProxy.java:<init>(78)) - > yarn.client.max-nodemanagers-proxies : 500 > 2014-11-18 05:36:33,589 INFO [main] yarn.GiraphApplicationMaster > (GiraphApplicationMaster.java:setupContainerAskForRM(279)) - Requested > container ask: Capability[<memory:1024, vCores:0>]Priority[10] > 2014-11-18 05:36:33,617 INFO [main] yarn.GiraphApplicationMaster > (GiraphApplicationMaster.java:setupContainerAskForRM(279)) - Requested > container ask: Capability[<memory:1024, vCores:0>]Priority[10] > 2014-11-18 05:36:33,617 INFO [main] yarn.GiraphApplicationMaster > (GiraphApplicationMaster.java:run(185)) - Wait to finish .. > 2014-11-18 05:36:35,674 INFO [AMRM Heartbeater thread] > impl.AMRMClientImpl (AMRMClientImpl.java:populateNMTokens(299)) - > Received new token for : roosevelt:60486 > 2014-11-18 05:36:35,681 INFO [AMRM Callback Handler Thread] > yarn.GiraphApplicationMaster > (GiraphApplicationMaster.java:onContainersAllocated(605)) - Got > response from RM for container ask, allocatedCnt=3D1 > 2014-11-18 05:36:35,681 INFO [AMRM Callback Handler Thread] > yarn.GiraphApplicationMaster > (GiraphApplicationMaster.java:onContainersAllocated(608)) - Total > allocated # of container so far : 1 allocated out of 2 required. > 2014-11-18 05:36:35,681 INFO [AMRM Callback Handler Thread] > yarn.GiraphApplicationMaster > (GiraphApplicationMaster.java:startContainerLaunchingThreads(359)) - > Launching command on a new container., > containerId=3Dcontainer_1416310496070_0001_01_000002, > containerNode=3Droosevelt:60486, containerNodeURI=3Droosevelt:8042, > containerResourceMemory=3D1024 > 2014-11-18 05:36:35,688 INFO [pool-4-thread-1] > yarn.GiraphApplicationMaster > (GiraphApplicationMaster.java:buildContainerLaunchContext(492)) - > Setting up container launch container for > containerid=3Dcontainer_1416310496070_0001_01_000002 > 2014-11-18 05:36:35,720 INFO [pool-4-thread-1] > yarn.GiraphApplicationMaster > (GiraphApplicationMaster.java:buildContainerLaunchContext(498)) - > Conatain launch Commands :java -Xmx1024M -Xms1024M -cp .:${CLASSPATH} > org.apache.giraph.yarn.GiraphYarnTask 1416310496070 1 2 1 > 1><LOG_DIR>/task-2-stdout.log 2><LOG_DIR>/task-2-stderr.log > 2014-11-18 05:36:35,721 INFO [pool-4-thread-1] > yarn.GiraphApplicationMaster > (GiraphApplicationMaster.java:buildContainerLaunchContext(518)) - > Setting username in ContainerLaunchContext to: hduser > 2014-11-18 05:36:36,789 INFO [AMRM Callback Handler Thread] > yarn.GiraphApplicationMaster > (GiraphApplicationMaster.java:onContainersAllocated(605)) - Got > response from RM for container ask, allocatedCnt=3D1 > 2014-11-18 05:36:36,789 INFO [AMRM Callback Handler Thread] > yarn.GiraphApplicationMaster > (GiraphApplicationMaster.java:onContainersAllocated(608)) - Total > allocated # of container so far : 2 allocated out of 2 required. > 2014-11-18 05:36:36,790 INFO [AMRM Callback Handler Thread] > yarn.GiraphApplicationMaster > (GiraphApplicationMaster.java:startContainerLaunchingThreads(359)) - > Launching command on a new container., > containerId=3Dcontainer_1416310496070_0001_01_000003, > containerNode=3Droosevelt:60486, containerNodeURI=3Droosevelt:8042, > containerResourceMemory=3D1024 > 2014-11-18 05:36:37,227 INFO [pool-4-thread-2] > yarn.GiraphApplicationMaster > (GiraphApplicationMaster.java:buildContainerLaunchContext(492)) - > Setting up container launch container for > containerid=3Dcontainer_1416310496070_0001_01_000003 > 2014-11-18 05:36:37,227 INFO [pool-4-thread-2] > yarn.GiraphApplicationMaster > (GiraphApplicationMaster.java:buildContainerLaunchContext(498)) - > Conatain launch Commands :java -Xmx1024M -Xms1024M -cp .:${CLASSPATH} > org.apache.giraph.yarn.GiraphYarnTask 1416310496070 1 3 1 > 1><LOG_DIR>/task-3-stdout.log 2><LOG_DIR>/task-3-stderr.log > 2014-11-18 05:36:37,227 INFO [pool-4-thread-2] > yarn.GiraphApplicationMaster > (GiraphApplicationMaster.java:buildContainerLaunchContext(518)) - > Setting username in ContainerLaunchContext to: hduser > 2014-11-18 05:36:37,463 INFO [pool-4-thread-1] yarn.YarnUtils > (YarnUtils.java:addFsResourcesToMap(72)) - Adding > giraph-examples-1.1.0-SNAPSHOT-for-hadoop-2.5.1-jar-with-dependencies.jar > to LocalResources for export.to > > hdfs://localhost:9000/user/hduser/giraph_yarn_jar_cache/application_1416310= > > 496070_0001/giraph-examples-1.1.0-SNAPSHOT-for-hadoop-2.5.1-jar-with-depend= > encies.jar > 2014-11-18 05:36:37,678 INFO [pool-4-thread-1] yarn.YarnUtils > (YarnUtils.java:addFileToResourceMap(160)) - Registered file in > LocalResources :: > > hdfs://localhost:9000/user/hduser/giraph_yarn_jar_cache/application_1416310= > > 496070_0001/giraph-examples-1.1.0-SNAPSHOT-for-hadoop-2.5.1-jar-with-depend= > encies.jar > 2014-11-18 05:36:37,678 WARN [pool-4-thread-1] yarn.YarnUtils > (YarnUtils.java:addFsResourcesToMap(81)) - Job jars (-yj option) > didn't include giraph-core. > 2014-11-18 05:36:37,680 INFO [pool-4-thread-1] yarn.YarnUtils > (YarnUtils.java:addFileToResourceMap(160)) - Registered file in > LocalResources :: > > hdfs://localhost:9000/user/hduser/giraph_yarn_jar_cache/application_1416310= > 496070_0001/giraph-conf.xml > 2014-11-18 05:36:37,694 INFO > [org.apache.hadoop.yarn.client.api.async.impl.NMClientAsyncImpl #0] > impl.NMClientAsyncImpl (NMClientAsyncImpl.java:run(531)) - Processing > Event EventType: START_CONTAINER for Container > container_1416310496070_0001_01_000002 > 2014-11-18 05:36:37,699 INFO > [org.apache.hadoop.yarn.client.api.async.impl.NMClientAsyncImpl #1] > impl.NMClientAsyncImpl (NMClientAsyncImpl.java:run(531)) - Processing > Event EventType: START_CONTAINER for Container > container_1416310496070_0001_01_000003 > 2014-11-18 05:36:37,698 INFO > [org.apache.hadoop.yarn.client.api.async.impl.NMClientAsyncImpl #0] > impl.ContainerManagementProtocolProxy > (ContainerManagementProtocolProxy.java:newProxy(212)) - Opening proxy > : roosevelt:60486 > 2014-11-18 05:36:37,761 INFO > [org.apache.hadoop.yarn.client.api.async.impl.NMClientAsyncImpl #2] > impl.NMClientAsyncImpl (NMClientAsyncImpl.java:run(531)) - Processing > Event EventType: QUERY_CONTAINER for Container > container_1416310496070_0001_01_000003 > 2014-11-18 05:36:37,771 INFO > [org.apache.hadoop.yarn.client.api.async.impl.NMClientAsyncImpl #3] > impl.NMClientAsyncImpl (NMClientAsyncImpl.java:run(531)) - Processing > Event EventType: QUERY_CONTAINER for Container > container_1416310496070_0001_01_000002 > > > > On Thu, Nov 20, 2014 at 1:32 PM, Alessandro Negro <alenegr...@yahoo.it> > wrote: > > Hi Das, > I mean as user you run Hadoop yarn resource manager. > > Alessandro > > Il giorno 20/nov/2014, alle ore 19:22, D Adams <dadam...@gmail.com> ha > scritto: > > Alessandro, > I'm not sure what you mean, should I create a new user on my system > named 'yarn'? I'm new to both hadoop and giraph, so I'm not sure. > > Tripti, > I'll get those logs as soon as I can. > > Thank you both, > > V/r > Das > On Nov 20, 2014 3:13 AM, "Alessandro Negro" <alenegr...@yahoo.it> wrote: > > Hi Tripti, > I agree that a more detailed error log could be useful. > > Thanks, > Alessandro > > > > > Il giorno 19/nov/2014, alle ore 15:54, Tripti Singh <tri...@yahoo-inc.co= > > m> > > ha scritto: > > --_000_D092AB9C1CB42triptiyahooinccom_ > Content-Type: text/plain; charset=3D"Windows-1252" > Content-Transfer-Encoding: quoted-printable > > Hi Alessandro, > > I guess u r talking about the documentation on Giraph webpage which > > mention=3D > > s addition of this new option which is more or less mandatory for yarn > > base=3D > > d profiles. > When I first ran the Giraph on yarn without the proper =3D96yj option, > > there =3D > > was no way I could figure out from the Application/Container logs that > > this=3D > > was the issue. > I think, it=3D92ll be a good idea to have this message in the logs for > > easy d=3D > > ebugging. > > Thanks, > Tripti. > > > From: Alessandro Negro <alenegr...@yahoo.it<mailto:alenegr...@yahoo.it > <alenegr...@yahoo.it> > > > Reply-To: "user@giraph.apache.org<mailto:user@giraph.apache.org > <user@giraph.apache.org>>" > > <user@gir=3D > > aph.apache.org<mailto:user@giraph.apache.org <user@giraph.apache.org>>> > Date: Tuesday, November 18, 2014 at 4:31 PM > To: "user@giraph.apache.org<mailto:user@giraph.apache.org > <user@giraph.apache.org>>" < > > u...@giraph.ap=3D > > ache.org<mailto:user@giraph.apache.org <user@giraph.apache.org>>> > Subject: Re: Help with Giraph on Yarn > > Hi Eli, > no I think that the help message is clear enough, generally when I rea= > > d > > =3D93=3D > > jar=3D94 I always mean absolute path for jar, but this is a my persona= > > l > > misun=3D > > derstanding. > > Thanks, > Alessandro > > > Il giorno 16/nov/2014, alle ore 21:39, Eli Reisman < > > apache.mail...@gmail.co=3D > > m<mailto:apache.mail...@gmail.com <apache.mail...@gmail.com>>> ha scritto: > > Sounds like you got it figured out since last time I checked this list= > > , > > sor=3D > > ry it was a pain. Feel free to drop a JIRA ticket if we can make help > > messa=3D > > ge etc. for -yj option more clear, there's lots to improve there. > > On Sat, Nov 8, 2014 at 7:26 AM, Alessandro Negro <alenegr...@yahoo.it > > <mailt=3D > > o:alenegr...@yahoo.it>> wrote: > Hi Tripti, > finally I was able to run the test with success. It was an issue of > > permiss=3D > > ion since I was running as ale not as yarn. > Let me say that now I=3D92m able to run graph examples on Yarn 2.5.1. > > This is=3D > > the final result: > > 14/11/08 16:24:00 INFO yarn.GiraphYarnClient: Completed Giraph: > > org.apache.=3D > > giraph.examples.SimpleShortestPathsComputation: SUCCEEDED, total > > running ti=3D > > me: 0 minutes, 21 seconds. > > > Many thanks for your support, > Alessandro > > Il giorno 06/nov/2014, alle ore 15:16, Tripti Singh < > > tri...@yahoo-inc.com<m=3D > > ailto:tri...@yahoo-inc.com>> ha scritto: > > I don't know if u have access to this node. But if u do, u could check > > if t=3D > > he file is indeed there and u have access to it. > > Sent from my iPhone > > On 06-Nov-2014, at 6:12 pm, "Alessandro Negro" <alenegr...@yahoo.it > > <mailto:=3D > > alenegr...@yahoo.it>> wrote: > > You are right it works, but now I receive the following error: > > SLF4J: Class path contains multiple SLF4J bindings. > SLF4J: Found binding in > > [jar:file:/private/tmp/hadoop-yarn/nm-local-dir/use=3D > > > rcache/ale/appcache/application_1415264041937_0009/filecache/10/giraph-e= > > xam=3D > > > ples-1.1.0-SNAPSHOT-for-hadoop-2.5.1-jar-with-dependencies.jar!/org/slf4= > > j/i=3D > > mpl/StaticLoggerBinder.class] > SLF4J: Found binding in > > [jar:file:/opt/yarn/hadoop-2.5.1/share/hadoop/commo=3D > > n/lib/slf4j-log4j12-1.7.5.jar!/org/slf4j/impl/StaticLoggerBinder.class= > > ] > > SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an > > explana=3D > > tion. > SLF4J: Actual binding is of type [org.slf4j.impl.Log4jLoggerFactory] > 2014-11-06 13:15:37.120 java[10158:1803] Unable to load realm info fro= > > m > > SCD=3D > > ynamicStore > Exception in thread "pool-4-thread-1" java.lang.IllegalStateException: > > Coul=3D > > d not configure the containerlaunch context for GiraphYarnTasks. > at > > org.apache.giraph.yarn.GiraphApplicationMaster.getTaskResourceMap(Giraph= > > =3D > > ApplicationMaster.java:391) > at > > org.apache.giraph.yarn.GiraphApplicationMaster.access$500(GiraphApplicat= > > =3D > > ionMaster.java:78) > at > > org.apache.giraph.yarn.GiraphApplicationMaster$LaunchContainerRunnable.b= > > =3D > > uildContainerLaunchContext(GiraphApplicationMaster.java:522) > at > > org.apache.giraph.yarn.GiraphApplicationMaster$LaunchContainerRunnable.r= > > =3D > > un(GiraphApplicationMaster.java:479) > at > > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.jav= > > =3D > > a:1145) > at > > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.ja= > > =3D > > va:615) > at java.lang.Thread.run(Thread.java:744) > Caused by: java.io.FileNotFoundException: File does not exist: > > hdfs://hadoo=3D > > > p-master:9000/user/yarn/giraph_yarn_jar_cache/application_1415264041937_= > > 000=3D > > > 9/giraph-examples-1.1.0-SNAPSHOT-for-hadoop-2.5.1-jar-with-dependencies.= > > jar > > at > > org.apache.hadoop.hdfs.DistributedFileSystem$17.doCall(DistributedFileSy= > > =3D > > stem.java:1072) > at > > org.apache.hadoop.hdfs.DistributedFileSystem$17.doCall(DistributedFileSy= > > =3D > > stem.java:1064) > at > > org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolv= > > =3D > > er.java:81) > at > > org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFi= > > =3D > > leSystem.java:1064) > at > > org.apache.giraph.yarn.YarnUtils.addFileToResourceMap(YarnUtils.java:153= > > =3D > > ) > at > > org.apache.giraph.yarn.YarnUtils.addFsResourcesToMap(YarnUtils.java:77) > > at > > org.apache.giraph.yarn.GiraphApplicationMaster.getTaskResourceMap(Giraph= > > =3D > > ApplicationMaster.java:387) > ... 6 more > > > That justify the other error I receive in the task: > Could not find or load main class org.apache.giraph.yarn.GiraphYarnTas= > > k > > > Thanks, > > Il giorno 06/nov/2014, alle ore 13:07, Tripti Singh < > > tri...@yahoo-inc.com<m=3D > > ailto:tri...@yahoo-inc.com>> ha scritto: > > Why r u adding two jars? Example jar ideally contains core library so > > every=3D > > thing should be available with just one example jar included > > Sent from my iPhone > > On 06-Nov-2014, at 4:33 pm, "Alessandro Negro" <alenegr...@yahoo.it > > <mailto:=3D > > alenegr...@yahoo.it>> wrote: > > Hi, > now it seems better, I need to add: > > > giraph-1.1.0-SNAPSHOT-for-hadoop-2.5.1-jar-with-dependencies.jar,giraph-= > > exa=3D > > mples-1.1.0-SNAPSHOT-for-hadoop-2.5.1-jar-with-dependencies.jar > > > Now it seems that after a lot of cycle it fail with this error: > > Could not find or load main class org.apache.giraph.yarn.GiraphYarnTas= > > k > > > But in this case the error appear in task-3-stderr.log not in > > gam-stderr.l=3D > > og where there is the following error: > > LF4J: Class path contains multiple SLF4J bindings. > SLF4J: Found binding in > > [jar:file:/private/tmp/hadoop-yarn/nm-local-dir/use=3D > > > rcache/ale/appcache/application_1415264041937_0006/filecache/12/giraph-1= > > .1.=3D > > > 0-SNAPSHOT-for-hadoop-2.5.1-jar-with-dependencies.jar!/org/slf4j/impl/St= > > ati=3D > > cLoggerBinder.class] > SLF4J: Found binding in > > [jar:file:/private/tmp/hadoop-yarn/nm-local-dir/use=3D > > > rcache/ale/appcache/application_1415264041937_0006/filecache/10/giraph-e= > > xam=3D > > > ples-1.1.0-SNAPSHOT-for-hadoop-2.5.1-jar-with-dependencies.jar!/org/slf4= > > j/i=3D > > mpl/StaticLoggerBinder.class] > SLF4J: Found binding in > > [jar:file:/opt/yarn/hadoop-2.5.1/share/hadoop/commo=3D > > n/lib/slf4j-log4j12-1.7.5.jar!/org/slf4j/impl/StaticLoggerBinder.class= > > ] > > SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an > > > >
