Hi Das, this is my configuration of hadoop: yarn 1029 0,1 1,2 3499572 209560 s001 S 10:29am 0:06.04 /Library/Java/JavaVirtualMachines/jdk1.7.0_45.jdk/Contents/Home/bin/java -Dproc_resourcemanager -Xmx500m -Dhadoop.log.dir=/opt/yarn/hadoop-2.5.1/logs -Dyarn.log.dir=/opt/yarn/hadoop-2.5.1/logs -Dhadoop.log.file=yarn-yarn-resourcemanager-MacBook-Pro-di-Alessandro.local.log -Dyarn.log.file=yarn-yarn-resourcemanager-MacBook-Pro-di-Alessandro.local.log -Dyarn.home.dir= -Dyarn.id.str=yarn -Dhadoop.root.logger=INFO,RFA -Dyarn.root.logger=INFO,RFA -Djava.library.path=/opt/yarn/hadoop-2.5.1/lib/native -Dyarn.policy.file=hadoop-policy.xml -Dhadoop.log.dir=/opt/yarn/hadoop-2.5.1/logs -Dyarn.log.dir=/opt/yarn/hadoop-2.5.1/logs -Dhadoop.log.file=yarn-yarn-resourcemanager-MacBook-Pro-di-Alessandro.local.log -Dyarn.log.file=yarn-yarn-resourcemanager-MacBook-Pro-di-Alessandro.local.log -Dyarn.home.dir=/opt/yarn/hadoop-2.5.1 -Dhadoop.home.dir=/opt/yarn/hadoop-2.5.1 -Dhadoop.root.logger=INFO,RFA -Dyarn.root.logger=INFO,RFA -Djava.library.path=/opt/yarn/hadoop-2.5.1/lib/native -classpath /opt/yarn/hadoop-2.5.1/etc/hadoop:/opt/yarn/hadoop-2.5.1/etc/hadoop:/opt/yarn/hadoop-2.5.1/etc/hadoop:/opt/yarn/hadoop-2.5.1/share/hadoop/common/lib/*:/opt/yarn/hadoop-2.5.1/share/hadoop/common/*:/opt/yarn/hadoop-2.5.1/share/hadoop/hdfs:/opt/yarn/hadoop-2.5.1/share/hadoop/hdfs/lib/*:/opt/yarn/hadoop-2.5.1/share/hadoop/hdfs/*:/opt/yarn/hadoop-2.5.1/share/hadoop/yarn/lib/*:/opt/yarn/hadoop-2.5.1/share/hadoop/yarn/*:/opt/yarn/hadoop-2.5.1/share/hadoop/mapreduce/lib/*:/opt/yarn/hadoop-2.5.1/share/hadoop/mapreduce/*::/opt/yarn/hadoop-2.5.1/share/myLib/*.jar:/Users/ale/extprj/giraph/myGiraph/*.jar:/Users/ale/extprj/giraph/myGiraph/lib/*.jar:/contrib/capacity-scheduler/*.jar:/opt/yarn/hadoop-2.5.1/share/myLib/*.jar:/Users/ale/extprj/giraph/myGiraph/*.jar:/Users/ale/extprj/giraph/myGiraph/lib/*.jar:/contrib/capacity-scheduler/*.jar:/opt/yarn/hadoop-2.5.1/share/hadoop/yarn/*:/opt/yarn/hadoop-2.5.1/share/hadoop/yarn/lib/*:/opt/yarn/hadoop-2.5.1/etc/hadoop/rm-config/log4j.properties org.apache.hadoop.yarn.server.resourcemanager.ResourceManager yarn 1073 0,1 1,0 3347132 174644 s001 S 10:29am 0:04.77 /Library/Java/JavaVirtualMachines/jdk1.7.0_45.jdk/Contents/Home/bin/java -Dproc_nodemanager -Xmx500m -Dhadoop.log.dir=/opt/yarn/hadoop-2.5.1/logs -Dyarn.log.dir=/opt/yarn/hadoop-2.5.1/logs -Dhadoop.log.file=yarn-yarn-nodemanager-MacBook-Pro-di-Alessandro.local.log -Dyarn.log.file=yarn-yarn-nodemanager-MacBook-Pro-di-Alessandro.local.log -Dyarn.home.dir= -Dyarn.id.str=yarn -Dhadoop.root.logger=INFO,RFA -Dyarn.root.logger=INFO,RFA -Djava.library.path=/opt/yarn/hadoop-2.5.1/lib/native -Dyarn.policy.file=hadoop-policy.xml -server -Dhadoop.log.dir=/opt/yarn/hadoop-2.5.1/logs -Dyarn.log.dir=/opt/yarn/hadoop-2.5.1/logs -Dhadoop.log.file=yarn-yarn-nodemanager-MacBook-Pro-di-Alessandro.local.log -Dyarn.log.file=yarn-yarn-nodemanager-MacBook-Pro-di-Alessandro.local.log -Dyarn.home.dir=/opt/yarn/hadoop-2.5.1 -Dhadoop.home.dir=/opt/yarn/hadoop-2.5.1 -Dhadoop.root.logger=INFO,RFA -Dyarn.root.logger=INFO,RFA -Djava.library.path=/opt/yarn/hadoop-2.5.1/lib/native -classpath /opt/yarn/hadoop-2.5.1/etc/hadoop:/opt/yarn/hadoop-2.5.1/etc/hadoop:/opt/yarn/hadoop-2.5.1/etc/hadoop:/opt/yarn/hadoop-2.5.1/share/hadoop/common/lib/*:/opt/yarn/hadoop-2.5.1/share/hadoop/common/*:/opt/yarn/hadoop-2.5.1/share/hadoop/hdfs:/opt/yarn/hadoop-2.5.1/share/hadoop/hdfs/lib/*:/opt/yarn/hadoop-2.5.1/share/hadoop/hdfs/*:/opt/yarn/hadoop-2.5.1/share/hadoop/yarn/lib/*:/opt/yarn/hadoop-2.5.1/share/hadoop/yarn/*:/opt/yarn/hadoop-2.5.1/share/hadoop/mapreduce/lib/*:/opt/yarn/hadoop-2.5.1/share/hadoop/mapreduce/*::/opt/yarn/hadoop-2.5.1/share/myLib/*.jar:/Users/ale/extprj/giraph/myGiraph/*.jar:/Users/ale/extprj/giraph/myGiraph/lib/*.jar:/contrib/capacity-scheduler/*.jar:/opt/yarn/hadoop-2.5.1/share/myLib/*.jar:/Users/ale/extprj/giraph/myGiraph/*.jar:/Users/ale/extprj/giraph/myGiraph/lib/*.jar:/contrib/capacity-scheduler/*.jar:/opt/yarn/hadoop-2.5.1/share/hadoop/yarn/*:/opt/yarn/hadoop-2.5.1/share/hadoop/yarn/lib/*:/opt/yarn/hadoop-2.5.1/etc/hadoop/nm-config/log4j.properties org.apache.hadoop.yarn.server.nodemanager.NodeManager
So I run it as yarn user in this way: sudo -u yarn …. Moreover I have just notice that I added the following lines into the hadoop-env.sh hadoop-env.sh:export MYLIB=/opt/yarn/hadoop-2.5.1/share/myLib/*.jar:/Users/ale/extprj/giraph/myGiraph/*.jar:/Users/ale/extprj/giraph/myGiraph/lib/*.jar Accordingly to this other thread: http://mail-archives.apache.org/mod_mbox/giraph-user/201408.mbox/%3c53f4c689.5060...@web.de%3E Let me know, Alessandro Il giorno 20/nov/2014, alle ore 23:51, D Adams <dadam...@gmail.com> ha scritto: > --f46d044283ecf837440508522939 > Content-Type: text/plain; charset=UTF-8 > Content-Transfer-Encoding: quoted-printable > > Ok, in the following, I use bin/yarn. Doesn't seem like much has changed: > > hduser@Roosevelt:/usr/local/hadoop$ ./hduser_jobs.sh > Running hduser script ... > bin/yarn jar > /usr/local/giraph/giraph-examples/target/giraph-examples-1.1.0-SNAPSHOT-for= > -hadoop-2.5.1-jar-with-dependencies.jar > org.apache.giraph.GiraphRunner > org.apache.giraph.examples.SimpleShortestPathsComputation -vif > org.apache.giraph.io.formats.JsonLongDoubleFloatDoubleVertexInputFormat > -vip /user/hduser/input/tiny_graph.txt -vof > org.apache.giraph.io.formats.IdWithValueTextOutputFormat -op > /user/hduser/output/shortestpaths -w 1 -yj > giraph-examples-1.1.0-SNAPSHOT-for-hadoop-2.5.1-jar-with-dependencies.jar > 14/11/18 05:36:05 WARN util.NativeCodeLoader: Unable to load native-hadoop > library for your platform... using builtin-java classes where applicable > 14/11/18 05:36:08 INFO utils.ConfigurationUtils: No edge input format > specified. Ensure your InputFormat does not require one. > 14/11/18 05:36:08 INFO utils.ConfigurationUtils: No edge output format > specified. Ensure your OutputFormat does not require one. > 14/11/18 05:36:08 INFO yarn.GiraphYarnClient: Final output path is: > hdfs://localhost:9000/user/hduser/output/shortestpaths > 14/11/18 05:36:08 INFO yarn.GiraphYarnClient: Running Client > 14/11/18 05:36:08 INFO client.RMProxy: Connecting to ResourceManager at > localhost/127.0.0.1:8050 > 14/11/18 05:36:09 INFO yarn.GiraphYarnClient: Got node report from ASM for, > nodeId=3Droosevelt:60486, nodeAddress roosevelt:8042, nodeRackName > /default-rack, nodeNumContainers 0 > 14/11/18 05:36:09 INFO yarn.GiraphYarnClient: Defaulting per-task heap size > to 1024MB. > 14/11/18 05:36:09 INFO yarn.GiraphYarnClient: Obtained new Application ID: > application_1416310496070_0001 > 14/11/18 05:36:09 INFO Configuration.deprecation: mapred.job.id is > deprecated. Instead, use mapreduce.job.id > 14/11/18 05:36:09 INFO yarn.GiraphYarnClient: Set the environment for the > application master > 14/11/18 05:36:09 INFO yarn.GiraphYarnClient: Environment for AM > :{CLASSPATH=3D${CLASSPATH}:./*:$HADOOP_CONF_DIR:$HADOOP_COMMON_HOME/share/h= > adoop/common/*:$HADOOP_COMMON_HOME/share/hadoop/common/lib/*:$HADOOP_HDFS_H= > OME/share/hadoop/hdfs/*:$HADOOP_HDFS_HOME/share/hadoop/hdfs/lib/*:$HADOOP_Y= > ARN_HOME/share/hadoop/yarn/*:$HADOOP_YARN_HOME/share/hadoop/yarn/lib/*:$HAD= > OOP_MAPRED_HOME/share/hadoop/mapreduce/*:$HADOOP_MAPRED_HOME/share/hadoop/m= > apreduce/lib/*} > 14/11/18 05:36:09 INFO yarn.GiraphYarnClient: buildLocalResourceMap .... > 14/11/18 05:36:12 INFO yarn.YarnUtils: Registered file in LocalResources :: > hdfs://localhost:9000/user/hduser/giraph_yarn_jar_cache/application_1416310= > 496070_0001/giraph-conf.xml > 14/11/18 05:36:12 INFO yarn.GiraphYarnClient: LIB JARS > :giraph-examples-1.1.0-SNAPSHOT-for-hadoop-2.5.1-jar-with-dependencies.jar > 14/11/18 05:36:12 INFO yarn.YarnUtils: Class path name . > 14/11/18 05:36:12 INFO yarn.YarnUtils: base path checking . > 14/11/18 05:36:12 INFO yarn.GiraphYarnClient: Made local resource for > :/usr/local/hadoop/share/giraph-examples-1.1.0-SNAPSHOT-for-hadoop-2.5.1-ja= > r-with-dependencies.jar > to > hdfs://localhost:9000/user/hduser/giraph_yarn_jar_cache/application_1416310= > 496070_0001/giraph-examples-1.1.0-SNAPSHOT-for-hadoop-2.5.1-jar-with-depend= > encies.jar > 14/11/18 05:36:16 INFO yarn.YarnUtils: Registered file in LocalResources :: > hdfs://localhost:9000/user/hduser/giraph_yarn_jar_cache/application_1416310= > 496070_0001/giraph-examples-1.1.0-SNAPSHOT-for-hadoop-2.5.1-jar-with-depend= > encies.jar > 14/11/18 05:36:16 INFO yarn.GiraphYarnClient: ApplicationSumbissionContext > for GiraphApplicationMaster launch container is populated. > 14/11/18 05:36:16 INFO yarn.GiraphYarnClient: Submitting application to ASM > 14/11/18 05:36:16 INFO impl.YarnClientImpl: Submitted application > application_1416310496070_0001 > 14/11/18 05:36:16 INFO yarn.GiraphYarnClient: Got new appId after > submission :application_1416310496070_0001 > 14/11/18 05:36:16 INFO yarn.GiraphYarnClient: GiraphApplicationMaster > container request was submitted to ResourceManager for job: Giraph: > org.apache.giraph.examples.SimpleShortestPathsComputation > 14/11/18 05:36:17 INFO yarn.GiraphYarnClient: Giraph: > org.apache.giraph.examples.SimpleShortestPathsComputation, Elapsed: 1.01 > secs > 14/11/18 05:36:17 INFO yarn.GiraphYarnClient: > appattempt_1416310496070_0001_000001, State: ACCEPTED, Containers used: 1 > 14/11/18 05:36:22 INFO yarn.GiraphYarnClient: Giraph: > org.apache.giraph.examples.SimpleShortestPathsComputation, Elapsed: 5.86 > secs > 14/11/18 05:36:22 INFO yarn.GiraphYarnClient: > appattempt_1416310496070_0001_000001, State: ACCEPTED, Containers used: 1 > 14/11/18 05:36:26 INFO yarn.GiraphYarnClient: Giraph: > org.apache.giraph.examples.SimpleShortestPathsComputation, Elapsed: 10.02 > secs > 14/11/18 05:36:26 INFO yarn.GiraphYarnClient: > appattempt_1416310496070_0001_000001, State: ACCEPTED, Containers used: 1 > 14/11/18 05:36:30 INFO yarn.GiraphYarnClient: Giraph: > org.apache.giraph.examples.SimpleShortestPathsComputation, Elapsed: 14.39 > secs > 14/11/18 05:36:30 INFO yarn.GiraphYarnClient: > appattempt_1416310496070_0001_000001, State: ACCEPTED, Containers used: 1 > 14/11/18 05:36:34 INFO yarn.GiraphYarnClient: Giraph: > org.apache.giraph.examples.SimpleShortestPathsComputation, Elapsed: 18.57 > secs > 14/11/18 05:36:34 INFO yarn.GiraphYarnClient: > appattempt_1416310496070_0001_000001, State: RUNNING, Containers used: 2 > > gam_stderr.log > > LF4J: Class path contains multiple SLF4J bindings. > SLF4J: Found binding in > [jar:file:/app/hadoop/tmp/nm-local-dir/usercache/hduser/appcache/applicatio= > n_1416310496070_0001/filecache/10/giraph-examples-1.1.0-SNAPSHOT-for-hadoop= > -2.5.1-jar-with-dependencies.jar!/org/slf4j/impl/StaticLoggerBinder.class] > SLF4J: Found binding in > [jar:file:/usr/local/hadoop/share/hadoop/common/lib/slf4j-log4j12-1.7.5.jar= > !/org/slf4j/impl/StaticLoggerBinder.class] > SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explana= > tion. > SLF4J: Actual binding is of type [org.slf4j.impl.Log4jLoggerFactory] > > gam_stdout.log > > 2014-11-18 05:36:28,143 INFO [main] yarn.GiraphApplicationMaster > (GiraphApplicationMaster.java:main(421)) - Starting GitaphAM > 2014-11-18 05:36:30,404 WARN [main] util.NativeCodeLoader > (NativeCodeLoader.java:<clinit>(62)) - Unable to load native-hadoop > library for your platform... using builtin-java classes where > applicable > 2014-11-18 05:36:32,649 INFO [main] yarn.GiraphApplicationMaster > (GiraphApplicationMaster.java:<init>(168)) - GiraphAM for ContainerId > container_1416310496070_0001_01_000001 ApplicationAttemptId > appattempt_1416310496070_0001_000001 > 2014-11-18 05:36:32,852 INFO [main] client.RMProxy > (RMProxy.java:createRMProxy(92)) - Connecting to ResourceManager at > localhost/127.0.0.1:8030 > 2014-11-18 05:36:32,888 INFO [main] impl.NMClientAsyncImpl > (NMClientAsyncImpl.java:serviceInit(107)) - Upper bound of the thread > pool size is 500 > 2014-11-18 05:36:32,889 INFO [main] > impl.ContainerManagementProtocolProxy > (ContainerManagementProtocolProxy.java:<init>(78)) - > yarn.client.max-nodemanagers-proxies : 500 > 2014-11-18 05:36:33,589 INFO [main] yarn.GiraphApplicationMaster > (GiraphApplicationMaster.java:setupContainerAskForRM(279)) - Requested > container ask: Capability[<memory:1024, vCores:0>]Priority[10] > 2014-11-18 05:36:33,617 INFO [main] yarn.GiraphApplicationMaster > (GiraphApplicationMaster.java:setupContainerAskForRM(279)) - Requested > container ask: Capability[<memory:1024, vCores:0>]Priority[10] > 2014-11-18 05:36:33,617 INFO [main] yarn.GiraphApplicationMaster > (GiraphApplicationMaster.java:run(185)) - Wait to finish .. > 2014-11-18 05:36:35,674 INFO [AMRM Heartbeater thread] > impl.AMRMClientImpl (AMRMClientImpl.java:populateNMTokens(299)) - > Received new token for : roosevelt:60486 > 2014-11-18 05:36:35,681 INFO [AMRM Callback Handler Thread] > yarn.GiraphApplicationMaster > (GiraphApplicationMaster.java:onContainersAllocated(605)) - Got > response from RM for container ask, allocatedCnt=3D1 > 2014-11-18 05:36:35,681 INFO [AMRM Callback Handler Thread] > yarn.GiraphApplicationMaster > (GiraphApplicationMaster.java:onContainersAllocated(608)) - Total > allocated # of container so far : 1 allocated out of 2 required. > 2014-11-18 05:36:35,681 INFO [AMRM Callback Handler Thread] > yarn.GiraphApplicationMaster > (GiraphApplicationMaster.java:startContainerLaunchingThreads(359)) - > Launching command on a new container., > containerId=3Dcontainer_1416310496070_0001_01_000002, > containerNode=3Droosevelt:60486, containerNodeURI=3Droosevelt:8042, > containerResourceMemory=3D1024 > 2014-11-18 05:36:35,688 INFO [pool-4-thread-1] > yarn.GiraphApplicationMaster > (GiraphApplicationMaster.java:buildContainerLaunchContext(492)) - > Setting up container launch container for > containerid=3Dcontainer_1416310496070_0001_01_000002 > 2014-11-18 05:36:35,720 INFO [pool-4-thread-1] > yarn.GiraphApplicationMaster > (GiraphApplicationMaster.java:buildContainerLaunchContext(498)) - > Conatain launch Commands :java -Xmx1024M -Xms1024M -cp .:${CLASSPATH} > org.apache.giraph.yarn.GiraphYarnTask 1416310496070 1 2 1 > 1><LOG_DIR>/task-2-stdout.log 2><LOG_DIR>/task-2-stderr.log > 2014-11-18 05:36:35,721 INFO [pool-4-thread-1] > yarn.GiraphApplicationMaster > (GiraphApplicationMaster.java:buildContainerLaunchContext(518)) - > Setting username in ContainerLaunchContext to: hduser > 2014-11-18 05:36:36,789 INFO [AMRM Callback Handler Thread] > yarn.GiraphApplicationMaster > (GiraphApplicationMaster.java:onContainersAllocated(605)) - Got > response from RM for container ask, allocatedCnt=3D1 > 2014-11-18 05:36:36,789 INFO [AMRM Callback Handler Thread] > yarn.GiraphApplicationMaster > (GiraphApplicationMaster.java:onContainersAllocated(608)) - Total > allocated # of container so far : 2 allocated out of 2 required. > 2014-11-18 05:36:36,790 INFO [AMRM Callback Handler Thread] > yarn.GiraphApplicationMaster > (GiraphApplicationMaster.java:startContainerLaunchingThreads(359)) - > Launching command on a new container., > containerId=3Dcontainer_1416310496070_0001_01_000003, > containerNode=3Droosevelt:60486, containerNodeURI=3Droosevelt:8042, > containerResourceMemory=3D1024 > 2014-11-18 05:36:37,227 INFO [pool-4-thread-2] > yarn.GiraphApplicationMaster > (GiraphApplicationMaster.java:buildContainerLaunchContext(492)) - > Setting up container launch container for > containerid=3Dcontainer_1416310496070_0001_01_000003 > 2014-11-18 05:36:37,227 INFO [pool-4-thread-2] > yarn.GiraphApplicationMaster > (GiraphApplicationMaster.java:buildContainerLaunchContext(498)) - > Conatain launch Commands :java -Xmx1024M -Xms1024M -cp .:${CLASSPATH} > org.apache.giraph.yarn.GiraphYarnTask 1416310496070 1 3 1 > 1><LOG_DIR>/task-3-stdout.log 2><LOG_DIR>/task-3-stderr.log > 2014-11-18 05:36:37,227 INFO [pool-4-thread-2] > yarn.GiraphApplicationMaster > (GiraphApplicationMaster.java:buildContainerLaunchContext(518)) - > Setting username in ContainerLaunchContext to: hduser > 2014-11-18 05:36:37,463 INFO [pool-4-thread-1] yarn.YarnUtils > (YarnUtils.java:addFsResourcesToMap(72)) - Adding > giraph-examples-1.1.0-SNAPSHOT-for-hadoop-2.5.1-jar-with-dependencies.jar > to LocalResources for export.to > hdfs://localhost:9000/user/hduser/giraph_yarn_jar_cache/application_1416310= > 496070_0001/giraph-examples-1.1.0-SNAPSHOT-for-hadoop-2.5.1-jar-with-depend= > encies.jar > 2014-11-18 05:36:37,678 INFO [pool-4-thread-1] yarn.YarnUtils > (YarnUtils.java:addFileToResourceMap(160)) - Registered file in > LocalResources :: > hdfs://localhost:9000/user/hduser/giraph_yarn_jar_cache/application_1416310= > 496070_0001/giraph-examples-1.1.0-SNAPSHOT-for-hadoop-2.5.1-jar-with-depend= > encies.jar > 2014-11-18 05:36:37,678 WARN [pool-4-thread-1] yarn.YarnUtils > (YarnUtils.java:addFsResourcesToMap(81)) - Job jars (-yj option) > didn't include giraph-core. > 2014-11-18 05:36:37,680 INFO [pool-4-thread-1] yarn.YarnUtils > (YarnUtils.java:addFileToResourceMap(160)) - Registered file in > LocalResources :: > hdfs://localhost:9000/user/hduser/giraph_yarn_jar_cache/application_1416310= > 496070_0001/giraph-conf.xml > 2014-11-18 05:36:37,694 INFO > [org.apache.hadoop.yarn.client.api.async.impl.NMClientAsyncImpl #0] > impl.NMClientAsyncImpl (NMClientAsyncImpl.java:run(531)) - Processing > Event EventType: START_CONTAINER for Container > container_1416310496070_0001_01_000002 > 2014-11-18 05:36:37,699 INFO > [org.apache.hadoop.yarn.client.api.async.impl.NMClientAsyncImpl #1] > impl.NMClientAsyncImpl (NMClientAsyncImpl.java:run(531)) - Processing > Event EventType: START_CONTAINER for Container > container_1416310496070_0001_01_000003 > 2014-11-18 05:36:37,698 INFO > [org.apache.hadoop.yarn.client.api.async.impl.NMClientAsyncImpl #0] > impl.ContainerManagementProtocolProxy > (ContainerManagementProtocolProxy.java:newProxy(212)) - Opening proxy > : roosevelt:60486 > 2014-11-18 05:36:37,761 INFO > [org.apache.hadoop.yarn.client.api.async.impl.NMClientAsyncImpl #2] > impl.NMClientAsyncImpl (NMClientAsyncImpl.java:run(531)) - Processing > Event EventType: QUERY_CONTAINER for Container > container_1416310496070_0001_01_000003 > 2014-11-18 05:36:37,771 INFO > [org.apache.hadoop.yarn.client.api.async.impl.NMClientAsyncImpl #3] > impl.NMClientAsyncImpl (NMClientAsyncImpl.java:run(531)) - Processing > Event EventType: QUERY_CONTAINER for Container > container_1416310496070_0001_01_000002 > > > > On Thu, Nov 20, 2014 at 1:32 PM, Alessandro Negro <alenegr...@yahoo.it> > wrote: > >> Hi Das, >> I mean as user you run Hadoop yarn resource manager. >> >> Alessandro >> >> Il giorno 20/nov/2014, alle ore 19:22, D Adams <dadam...@gmail.com> ha >> scritto: >> >> Alessandro, >> I'm not sure what you mean, should I create a new user on my system >> named 'yarn'? I'm new to both hadoop and giraph, so I'm not sure. >> >> Tripti, >> I'll get those logs as soon as I can. >> >> Thank you both, >> >> V/r >> Das >> On Nov 20, 2014 3:13 AM, "Alessandro Negro" <alenegr...@yahoo.it> wrote: >> >>> Hi Tripti, >>> I agree that a more detailed error log could be useful. >>> >>> Thanks, >>> Alessandro >>> >>> >>> >>> >>> Il giorno 19/nov/2014, alle ore 15:54, Tripti Singh <tri...@yahoo-inc.co= > m> >>> ha scritto: >>> >>>> --_000_D092AB9C1CB42triptiyahooinccom_ >>>> Content-Type: text/plain; charset=3D"Windows-1252" >>>> Content-Transfer-Encoding: quoted-printable >>>> >>>> Hi Alessandro, >>>> >>>> I guess u r talking about the documentation on Giraph webpage which >>> mention=3D >>>> s addition of this new option which is more or less mandatory for yarn >>> base=3D >>>> d profiles. >>>> When I first ran the Giraph on yarn without the proper =3D96yj option, >>> there =3D >>>> was no way I could figure out from the Application/Container logs that >>> this=3D >>>> was the issue. >>>> I think, it=3D92ll be a good idea to have this message in the logs for >>> easy d=3D >>>> ebugging. >>>> >>>> Thanks, >>>> Tripti. >>>> >>>> >>>> From: Alessandro Negro <alenegr...@yahoo.it<mailto:alenegr...@yahoo.it >>>>> >>>> Reply-To: "user@giraph.apache.org<mailto:user@giraph.apache.org>" >>> <user@gir=3D >>>> aph.apache.org<mailto:user@giraph.apache.org>> >>>> Date: Tuesday, November 18, 2014 at 4:31 PM >>>> To: "user@giraph.apache.org<mailto:user@giraph.apache.org>" < >>> u...@giraph.ap=3D >>>> ache.org<mailto:user@giraph.apache.org>> >>>> Subject: Re: Help with Giraph on Yarn >>>> >>>> Hi Eli, >>>> no I think that the help message is clear enough, generally when I rea= > d >>> =3D93=3D >>>> jar=3D94 I always mean absolute path for jar, but this is a my persona= > l >>> misun=3D >>>> derstanding. >>>> >>>> Thanks, >>>> Alessandro >>>> >>>> >>>> Il giorno 16/nov/2014, alle ore 21:39, Eli Reisman < >>> apache.mail...@gmail.co=3D >>>> m<mailto:apache.mail...@gmail.com>> ha scritto: >>>> >>>> Sounds like you got it figured out since last time I checked this list= > , >>> sor=3D >>>> ry it was a pain. Feel free to drop a JIRA ticket if we can make help >>> messa=3D >>>> ge etc. for -yj option more clear, there's lots to improve there. >>>> >>>> On Sat, Nov 8, 2014 at 7:26 AM, Alessandro Negro <alenegr...@yahoo.it >>> <mailt=3D >>>> o:alenegr...@yahoo.it>> wrote: >>>> Hi Tripti, >>>> finally I was able to run the test with success. It was an issue of >>> permiss=3D >>>> ion since I was running as ale not as yarn. >>>> Let me say that now I=3D92m able to run graph examples on Yarn 2.5.1. >>> This is=3D >>>> the final result: >>>> >>>> 14/11/08 16:24:00 INFO yarn.GiraphYarnClient: Completed Giraph: >>> org.apache.=3D >>>> giraph.examples.SimpleShortestPathsComputation: SUCCEEDED, total >>> running ti=3D >>>> me: 0 minutes, 21 seconds. >>>> >>>> >>>> Many thanks for your support, >>>> Alessandro >>>> >>>> Il giorno 06/nov/2014, alle ore 15:16, Tripti Singh < >>> tri...@yahoo-inc.com<m=3D >>>> ailto:tri...@yahoo-inc.com>> ha scritto: >>>> >>>> I don't know if u have access to this node. But if u do, u could check >>> if t=3D >>>> he file is indeed there and u have access to it. >>>> >>>> Sent from my iPhone >>>> >>>> On 06-Nov-2014, at 6:12 pm, "Alessandro Negro" <alenegr...@yahoo.it >>> <mailto:=3D >>>> alenegr...@yahoo.it>> wrote: >>>> >>>> You are right it works, but now I receive the following error: >>>> >>>> SLF4J: Class path contains multiple SLF4J bindings. >>>> SLF4J: Found binding in >>> [jar:file:/private/tmp/hadoop-yarn/nm-local-dir/use=3D >>>> >>> rcache/ale/appcache/application_1415264041937_0009/filecache/10/giraph-e= > xam=3D >>>> >>> ples-1.1.0-SNAPSHOT-for-hadoop-2.5.1-jar-with-dependencies.jar!/org/slf4= > j/i=3D >>>> mpl/StaticLoggerBinder.class] >>>> SLF4J: Found binding in >>> [jar:file:/opt/yarn/hadoop-2.5.1/share/hadoop/commo=3D >>>> n/lib/slf4j-log4j12-1.7.5.jar!/org/slf4j/impl/StaticLoggerBinder.class= > ] >>>> SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an >>> explana=3D >>>> tion. >>>> SLF4J: Actual binding is of type [org.slf4j.impl.Log4jLoggerFactory] >>>> 2014-11-06 13:15:37.120 java[10158:1803] Unable to load realm info fro= > m >>> SCD=3D >>>> ynamicStore >>>> Exception in thread "pool-4-thread-1" java.lang.IllegalStateException: >>> Coul=3D >>>> d not configure the containerlaunch context for GiraphYarnTasks. >>>> at >>> org.apache.giraph.yarn.GiraphApplicationMaster.getTaskResourceMap(Giraph= > =3D >>>> ApplicationMaster.java:391) >>>> at >>> org.apache.giraph.yarn.GiraphApplicationMaster.access$500(GiraphApplicat= > =3D >>>> ionMaster.java:78) >>>> at >>> org.apache.giraph.yarn.GiraphApplicationMaster$LaunchContainerRunnable.b= > =3D >>>> uildContainerLaunchContext(GiraphApplicationMaster.java:522) >>>> at >>> org.apache.giraph.yarn.GiraphApplicationMaster$LaunchContainerRunnable.r= > =3D >>>> un(GiraphApplicationMaster.java:479) >>>> at >>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.jav= > =3D >>>> a:1145) >>>> at >>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.ja= > =3D >>>> va:615) >>>> at java.lang.Thread.run(Thread.java:744) >>>> Caused by: java.io.FileNotFoundException: File does not exist: >>> hdfs://hadoo=3D >>>> >>> p-master:9000/user/yarn/giraph_yarn_jar_cache/application_1415264041937_= > 000=3D >>>> >>> 9/giraph-examples-1.1.0-SNAPSHOT-for-hadoop-2.5.1-jar-with-dependencies.= > jar >>>> at >>> org.apache.hadoop.hdfs.DistributedFileSystem$17.doCall(DistributedFileSy= > =3D >>>> stem.java:1072) >>>> at >>> org.apache.hadoop.hdfs.DistributedFileSystem$17.doCall(DistributedFileSy= > =3D >>>> stem.java:1064) >>>> at >>> org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolv= > =3D >>>> er.java:81) >>>> at >>> org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFi= > =3D >>>> leSystem.java:1064) >>>> at >>> org.apache.giraph.yarn.YarnUtils.addFileToResourceMap(YarnUtils.java:153= > =3D >>>> ) >>>> at >>> org.apache.giraph.yarn.YarnUtils.addFsResourcesToMap(YarnUtils.java:77) >>>> at >>> org.apache.giraph.yarn.GiraphApplicationMaster.getTaskResourceMap(Giraph= > =3D >>>> ApplicationMaster.java:387) >>>> ... 6 more >>>> >>>> >>>> That justify the other error I receive in the task: >>>> Could not find or load main class org.apache.giraph.yarn.GiraphYarnTas= > k >>>> >>>> Thanks, >>>> >>>> Il giorno 06/nov/2014, alle ore 13:07, Tripti Singh < >>> tri...@yahoo-inc.com<m=3D >>>> ailto:tri...@yahoo-inc.com>> ha scritto: >>>> >>>> Why r u adding two jars? Example jar ideally contains core library so >>> every=3D >>>> thing should be available with just one example jar included >>>> >>>> Sent from my iPhone >>>> >>>> On 06-Nov-2014, at 4:33 pm, "Alessandro Negro" <alenegr...@yahoo.it >>> <mailto:=3D >>>> alenegr...@yahoo.it>> wrote: >>>> >>>> Hi, >>>> now it seems better, I need to add: >>>> >>>> >>> giraph-1.1.0-SNAPSHOT-for-hadoop-2.5.1-jar-with-dependencies.jar,giraph-= > exa=3D >>>> mples-1.1.0-SNAPSHOT-for-hadoop-2.5.1-jar-with-dependencies.jar >>>> >>>> >>>> Now it seems that after a lot of cycle it fail with this error: >>>> >>>> Could not find or load main class org.apache.giraph.yarn.GiraphYarnTas= > k >>>> >>>> But in this case the error appear in task-3-stderr.log not in >>> gam-stderr.l=3D >>>> og where there is the following error: >>>> >>>> LF4J: Class path contains multiple SLF4J bindings. >>>> SLF4J: Found binding in >>> [jar:file:/private/tmp/hadoop-yarn/nm-local-dir/use=3D >>>> >>> rcache/ale/appcache/application_1415264041937_0006/filecache/12/giraph-1= > .1.=3D >>>> >>> 0-SNAPSHOT-for-hadoop-2.5.1-jar-with-dependencies.jar!/org/slf4j/impl/St= > ati=3D >>>> cLoggerBinder.class] >>>> SLF4J: Found binding in >>> [jar:file:/private/tmp/hadoop-yarn/nm-local-dir/use=3D >>>> >>> rcache/ale/appcache/application_1415264041937_0006/filecache/10/giraph-e= > xam=3D >>>> >>> ples-1.1.0-SNAPSHOT-for-hadoop-2.5.1-jar-with-dependencies.jar!/org/slf4= > j/i=3D >>>> mpl/StaticLoggerBinder.class] >>>> SLF4J: Found binding in >>> [jar:file:/opt/yarn/hadoop-2.5.1/share/hadoop/commo=3D >>>> n/lib/slf4j-log4j12-1.7.5.jar!/org/slf4j/impl/StaticLoggerBinder.class= > ] >>>> SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an
