Re: Run giraph on Hadoop yarn 2.6.0
Hi Flavian, do you try follow all this thread: http://mail-archives.apache.org/mod_mbox/giraph-user/201411.mbox/%3cca+bectmbb2x3mxsn_e0g_4iuwtwlhvwyyteu9hdvt+h7vqu...@mail.gmail.com%3e In any case I’ll test it as soon as I can against the new version. Cheers, Alessandro Il giorno 02/dic/2014, alle ore 11:09, Flavian Jacquot ha scritto: > Hello, I'm trying to use Giraph on hadoop 2.6.0 with yarn. > > I've managed to build it by removing STATIC_SASL_SYMBOL in in > the yarn profile. with the command : > sudo mvn -Phadoop_yarn -Dhadoop.version=2.6.0 -DskipTests package > > Then i've setup a single node cluster with hadoop 2.6.0 and I'm able to > perform the basic wordcount example. > I've copied the jar built into the directory share/hadoop/mapreduce to remove > the GiraphMaster not found execption. > > Now I'm trying to run the SimpleShortestPaths example with the command : > > $HADOOP_PREFIX/bin/hadoop jar > /usr/local/giraph/giraph-examples/target/giraph-examples-1.2.0-SNAPSHOT-for-hadoop-2.6.0-jar-with-dependencies.jar > org.apache.giraph.GiraphRunner > org.apache.giraph.examples.SimpleShortestPathsComputation -vif > org.apache.giraph.io.formats.JsonLongDoubleFloatDoubleVertexInputFormat -vip > input_tiny2.txt -vof org.apache.giraph.io.formats.IdWithValueTextOutputFormat > -op /user/hduser/output/shortestpaths445 -w 1 -ca > giraph.SplitMasterWorker=false -yj > giraph-examples-1.2.0-SNAPSHOT-for-hadoop-2.6.0-jar-with-dependencies.jar > > The data used is the tiny graph on the giraph quick start page and the job > should takes minutes to complete but it doesn't. > > The full log is at http://pastebin.com/CW1FnWz2 > > It's possible to make giraph work on hadoop 2.6.0 or do I have to use an > earlier version ? If so which one would you suggest ?
Re: Help with Giraph on Yarn
inerResourceMemory=3072
> 2014-11-18 19:06:45,684 INFO [pool-4-thread-3] yarn.GiraphApplicationMaster
> (GiraphApplicationMaster.java:buildContainerLaunchContext(492)) - Setting up
> container launch container for
> containerid=container_1416359131664_0001_01_03
> 2014-11-18 19:06:45,684 INFO [pool-4-thread-3] yarn.GiraphApplicationMaster
> (GiraphApplicationMaster.java:buildContainerLaunchContext(498)) - Conatain
> launch Commands :java -Xmx3000M -Xms3000M -cp .:${CLASSPATH}
> org.apache.giraph.yarn.GiraphYarnTask 1416359131664 1 3 1
> 1>/task-3-stdout.log 2>/task-3-stderr.log
> 2014-11-18 19:06:45,684 INFO [pool-4-thread-3] yarn.GiraphApplicationMaster
> (GiraphApplicationMaster.java:buildContainerLaunchContext(518)) - Setting
> username in ContainerLaunchContext to: hduser
> 2014-11-18 19:06:45,697 INFO
> [org.apache.hadoop.yarn.client.api.async.impl.NMClientAsyncImpl #0]
> impl.NMClientAsyncImpl (NMClientAsyncImpl.java:run(531)) - Processing Event
> EventType: START_CONTAINER for Container
> container_1416359131664_0001_01_03
> 2014-11-18 19:06:45,699 INFO
> [org.apache.hadoop.yarn.client.api.async.impl.NMClientAsyncImpl #0]
> impl.ContainerManagementProtocolProxy
> (ContainerManagementProtocolProxy.java:newProxy(212)) - Opening proxy :
> roosevelt:58195
> 2014-11-18 19:06:45,761 INFO
> [org.apache.hadoop.yarn.client.api.async.impl.NMClientAsyncImpl #1]
> impl.NMClientAsyncImpl (NMClientAsyncImpl.java:run(531)) - Processing Event
> EventType: QUERY_CONTAINER for Container
> container_1416359131664_0001_01_03
> 2014-11-18 19:06:46,687 INFO [AMRM Callback Handler Thread]
> yarn.GiraphApplicationMaster
> (GiraphApplicationMaster.java:onContainersCompleted(571)) - Got response from
> RM for container ask, completedCnt=1
> 2014-11-18 19:06:46,688 INFO [AMRM Callback Handler Thread]
> yarn.GiraphApplicationMaster
> (GiraphApplicationMaster.java:onContainersCompleted(574)) - Got container
> status for containerID=container_1416359131664_0001_01_03,
> state=COMPLETE, exitStatus=1, diagnostics=Exception from container-launch:
> ExitCodeException exitCode=1:
> ExitCodeException exitCode=1:
> at org.apache.hadoop.util.Shell.runCommand(Shell.java:538)
> at org.apache.hadoop.util.Shell.run(Shell.java:455)
> at
> org.apache.hadoop.util.Shell$ShellCommandExecutor.execute(Shell.java:702)
> at
> org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor.launchContainer(DefaultContainerExecutor.java:195)
> at
> org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:300)
> at
> org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:81)
> at java.util.concurrent.FutureTask.run(FutureTask.java:262)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
> at java.lang.Thread.run(Thread.java:745)
>
>
> Container exited with a non-zero exit code 1
>
> 2014-11-18 19:06:46,688 INFO [AMRM Callback Handler Thread]
> yarn.GiraphApplicationMaster
> (GiraphApplicationMaster.java:onContainersCompleted(596)) - After completion
> of one conatiner. current status is: completedCount :1 containersToLaunch :2
> successfulCount :0 failedCount :1
>
> On Sat, Nov 22, 2014 at 11:42 AM, D Adams wrote:
> I'm sorry, please excuse my ignorance, but, I have no idea where to find my
> configuration. That is, I have no idea where these lines came from (or what
> file):
>
> yarn 1029 0,1 1,2 3499572 209560 s001 S10:29am 0:06.04
> /Library/Java/
> JavaVirtualMachines/jdk1.7.0_45.jdk/Contents/Home/bin/java
> -Dproc_resourcemanager -Xmx500m -Dhadoop.log.dir=/opt/yarn/hadoop-2.5.1/logs
> -Dyarn.log.dir=/opt/yarn/hadoop-2.5.1/logs
> ...
> As far as configuration goes, I'm only really aware of the
> etc/hadoop/*-env.sh and .xml files and main pom.xml file. Or is the above
> part of the command you use to run giraph jobs?
> I will make changes to hadoop-env.sh file as you suggest and let you know how
> it works out.
>
> On Fri, Nov 21, 2014 at 6:05 AM, Alessandro Negro wrote:
> Hi,
> I forgot to say that I added also this line:
>
> export HADOOP_CLASSPATH=$HADOOP_CLASSPATH:$MYLIB
>
>
> Il giorno 21/nov/2014, alle ore 10:34, Alessandro Negro
> ha scritto:
>
>> Hi Das,
>> this is my configuration of hadoop:
>>
>> yarn 1029 0,1 1,2 3499572 209560 s001 S10:29am
>> 0:06.04
>> /Library/Java/JavaVirtualMachines/jdk1.7.0_45.jdk/Contents/Home/bin/java
>> -D
Re: Help with Giraph on Yarn
Hi, I forgot to say that I added also this line: export HADOOP_CLASSPATH=$HADOOP_CLASSPATH:$MYLIB Il giorno 21/nov/2014, alle ore 10:34, Alessandro Negro ha scritto: > Hi Das, > this is my configuration of hadoop: > > yarn 1029 0,1 1,2 3499572 209560 s001 S10:29am 0:06.04 > /Library/Java/JavaVirtualMachines/jdk1.7.0_45.jdk/Contents/Home/bin/java > -Dproc_resourcemanager -Xmx500m -Dhadoop.log.dir=/opt/yarn/hadoop-2.5.1/logs > -Dyarn.log.dir=/opt/yarn/hadoop-2.5.1/logs > -Dhadoop.log.file=yarn-yarn-resourcemanager-MacBook-Pro-di-Alessandro.local.log > > -Dyarn.log.file=yarn-yarn-resourcemanager-MacBook-Pro-di-Alessandro.local.log > -Dyarn.home.dir= -Dyarn.id.str=yarn -Dhadoop.root.logger=INFO,RFA > -Dyarn.root.logger=INFO,RFA > -Djava.library.path=/opt/yarn/hadoop-2.5.1/lib/native > -Dyarn.policy.file=hadoop-policy.xml > -Dhadoop.log.dir=/opt/yarn/hadoop-2.5.1/logs > -Dyarn.log.dir=/opt/yarn/hadoop-2.5.1/logs > -Dhadoop.log.file=yarn-yarn-resourcemanager-MacBook-Pro-di-Alessandro.local.log > > -Dyarn.log.file=yarn-yarn-resourcemanager-MacBook-Pro-di-Alessandro.local.log > -Dyarn.home.dir=/opt/yarn/hadoop-2.5.1 > -Dhadoop.home.dir=/opt/yarn/hadoop-2.5.1 -Dhadoop.root.logger=INFO,RFA > -Dyarn.root.logger=INFO,RFA > -Djava.library.path=/opt/yarn/hadoop-2.5.1/lib/native -classpath > /opt/yarn/hadoop-2.5.1/etc/hadoop:/opt/yarn/hadoop-2.5.1/etc/hadoop:/opt/yarn/hadoop-2.5.1/etc/hadoop:/opt/yarn/hadoop-2.5.1/share/hadoop/common/lib/*:/opt/yarn/hadoop-2.5.1/share/hadoop/common/*:/opt/yarn/hadoop-2.5.1/share/hadoop/hdfs:/opt/yarn/hadoop-2.5.1/share/hadoop/hdfs/lib/*:/opt/yarn/hadoop-2.5.1/share/hadoop/hdfs/*:/opt/yarn/hadoop-2.5.1/share/hadoop/yarn/lib/*:/opt/yarn/hadoop-2.5.1/share/hadoop/yarn/*:/opt/yarn/hadoop-2.5.1/share/hadoop/mapreduce/lib/*:/opt/yarn/hadoop-2.5.1/share/hadoop/mapreduce/*::/opt/yarn/hadoop-2.5.1/share/myLib/*.jar:/Users/ale/extprj/giraph/myGiraph/*.jar:/Users/ale/extprj/giraph/myGiraph/lib/*.jar:/contrib/capacity-scheduler/*.jar:/opt/yarn/hadoop-2.5.1/share/myLib/*.jar:/Users/ale/extprj/giraph/myGiraph/*.jar:/Users/ale/extprj/giraph/myGiraph/lib/*.jar:/contrib/capacity-scheduler/*.jar:/opt/yarn/hadoop-2.5.1/share/hadoop/yarn/*:/opt/yarn/hadoop-2.5.1/share/hadoop/yarn/lib/*:/opt/yarn/hadoop-2.5.1/etc/hadoop/rm-config/log4j.properties > org.apache.hadoop.yarn.server.resourcemanager.ResourceManager > yarn 1073 0,1 1,0 3347132 174644 s001 S10:29am 0:04.77 > /Library/Java/JavaVirtualMachines/jdk1.7.0_45.jdk/Contents/Home/bin/java > -Dproc_nodemanager -Xmx500m -Dhadoop.log.dir=/opt/yarn/hadoop-2.5.1/logs > -Dyarn.log.dir=/opt/yarn/hadoop-2.5.1/logs > -Dhadoop.log.file=yarn-yarn-nodemanager-MacBook-Pro-di-Alessandro.local.log > -Dyarn.log.file=yarn-yarn-nodemanager-MacBook-Pro-di-Alessandro.local.log > -Dyarn.home.dir= -Dyarn.id.str=yarn -Dhadoop.root.logger=INFO,RFA > -Dyarn.root.logger=INFO,RFA > -Djava.library.path=/opt/yarn/hadoop-2.5.1/lib/native > -Dyarn.policy.file=hadoop-policy.xml -server > -Dhadoop.log.dir=/opt/yarn/hadoop-2.5.1/logs > -Dyarn.log.dir=/opt/yarn/hadoop-2.5.1/logs > -Dhadoop.log.file=yarn-yarn-nodemanager-MacBook-Pro-di-Alessandro.local.log > -Dyarn.log.file=yarn-yarn-nodemanager-MacBook-Pro-di-Alessandro.local.log > -Dyarn.home.dir=/opt/yarn/hadoop-2.5.1 > -Dhadoop.home.dir=/opt/yarn/hadoop-2.5.1 -Dhadoop.root.logger=INFO,RFA > -Dyarn.root.logger=INFO,RFA > -Djava.library.path=/opt/yarn/hadoop-2.5.1/lib/native -classpath > /opt/yarn/hadoop-2.5.1/etc/hadoop:/opt/yarn/hadoop-2.5.1/etc/hadoop:/opt/yarn/hadoop-2.5.1/etc/hadoop:/opt/yarn/hadoop-2.5.1/share/hadoop/common/lib/*:/opt/yarn/hadoop-2.5.1/share/hadoop/common/*:/opt/yarn/hadoop-2.5.1/share/hadoop/hdfs:/opt/yarn/hadoop-2.5.1/share/hadoop/hdfs/lib/*:/opt/yarn/hadoop-2.5.1/share/hadoop/hdfs/*:/opt/yarn/hadoop-2.5.1/share/hadoop/yarn/lib/*:/opt/yarn/hadoop-2.5.1/share/hadoop/yarn/*:/opt/yarn/hadoop-2.5.1/share/hadoop/mapreduce/lib/*:/opt/yarn/hadoop-2.5.1/share/hadoop/mapreduce/*::/opt/yarn/hadoop-2.5.1/share/myLib/*.jar:/Users/ale/extprj/giraph/myGiraph/*.jar:/Users/ale/extprj/giraph/myGiraph/lib/*.jar:/contrib/capacity-scheduler/*.jar:/opt/yarn/hadoop-2.5.1/share/myLib/*.jar:/Users/ale/extprj/giraph/myGiraph/*.jar:/Users/ale/extprj/giraph/myGiraph/lib/*.jar:/contrib/capacity-scheduler/*.jar:/opt/yarn/hadoop-2.5.1/share/hadoop/yarn/*:/opt/yarn/hadoop-2.5.1/share/hadoop/yarn/lib/*:/opt/yarn/hadoop-2.5.1/etc/hadoop/nm-config/log4j.properties > org.apache.hadoop.yarn.server.nodemanager.NodeManager > > So I run it as yarn user in this way: > > sudo -u yarn …. > > Moreover I have just notice that I added the following lines into the > hadoop-env.sh > > hadoop-env.sh:export > MYLIB=/opt/yarn/hadoop-2.5.1/share/myLib/*.jar:/Users/ale/extprj/giraph/myGiraph/*.jar:/Use
Re: Help with Giraph on Yarn
to: hduser
> 2014-11-18 05:36:36,789 INFO [AMRM Callback Handler Thread]
> yarn.GiraphApplicationMaster
> (GiraphApplicationMaster.java:onContainersAllocated(605)) - Got
> response from RM for container ask, allocatedCnt=3D1
> 2014-11-18 05:36:36,789 INFO [AMRM Callback Handler Thread]
> yarn.GiraphApplicationMaster
> (GiraphApplicationMaster.java:onContainersAllocated(608)) - Total
> allocated # of container so far : 2 allocated out of 2 required.
> 2014-11-18 05:36:36,790 INFO [AMRM Callback Handler Thread]
> yarn.GiraphApplicationMaster
> (GiraphApplicationMaster.java:startContainerLaunchingThreads(359)) -
> Launching command on a new container.,
> containerId=3Dcontainer_1416310496070_0001_01_03,
> containerNode=3Droosevelt:60486, containerNodeURI=3Droosevelt:8042,
> containerResourceMemory=3D1024
> 2014-11-18 05:36:37,227 INFO [pool-4-thread-2]
> yarn.GiraphApplicationMaster
> (GiraphApplicationMaster.java:buildContainerLaunchContext(492)) -
> Setting up container launch container for
> containerid=3Dcontainer_1416310496070_0001_01_03
> 2014-11-18 05:36:37,227 INFO [pool-4-thread-2]
> yarn.GiraphApplicationMaster
> (GiraphApplicationMaster.java:buildContainerLaunchContext(498)) -
> Conatain launch Commands :java -Xmx1024M -Xms1024M -cp .:${CLASSPATH}
> org.apache.giraph.yarn.GiraphYarnTask 1416310496070 1 3 1
> 1>/task-3-stdout.log 2>/task-3-stderr.log
> 2014-11-18 05:36:37,227 INFO [pool-4-thread-2]
> yarn.GiraphApplicationMaster
> (GiraphApplicationMaster.java:buildContainerLaunchContext(518)) -
> Setting username in ContainerLaunchContext to: hduser
> 2014-11-18 05:36:37,463 INFO [pool-4-thread-1] yarn.YarnUtils
> (YarnUtils.java:addFsResourcesToMap(72)) - Adding
> giraph-examples-1.1.0-SNAPSHOT-for-hadoop-2.5.1-jar-with-dependencies.jar
> to LocalResources for export.to
> hdfs://localhost:9000/user/hduser/giraph_yarn_jar_cache/application_1416310=
> 496070_0001/giraph-examples-1.1.0-SNAPSHOT-for-hadoop-2.5.1-jar-with-depend=
> encies.jar
> 2014-11-18 05:36:37,678 INFO [pool-4-thread-1] yarn.YarnUtils
> (YarnUtils.java:addFileToResourceMap(160)) - Registered file in
> LocalResources ::
> hdfs://localhost:9000/user/hduser/giraph_yarn_jar_cache/application_1416310=
> 496070_0001/giraph-examples-1.1.0-SNAPSHOT-for-hadoop-2.5.1-jar-with-depend=
> encies.jar
> 2014-11-18 05:36:37,678 WARN [pool-4-thread-1] yarn.YarnUtils
> (YarnUtils.java:addFsResourcesToMap(81)) - Job jars (-yj option)
> didn't include giraph-core.
> 2014-11-18 05:36:37,680 INFO [pool-4-thread-1] yarn.YarnUtils
> (YarnUtils.java:addFileToResourceMap(160)) - Registered file in
> LocalResources ::
> hdfs://localhost:9000/user/hduser/giraph_yarn_jar_cache/application_1416310=
> 496070_0001/giraph-conf.xml
> 2014-11-18 05:36:37,694 INFO
> [org.apache.hadoop.yarn.client.api.async.impl.NMClientAsyncImpl #0]
> impl.NMClientAsyncImpl (NMClientAsyncImpl.java:run(531)) - Processing
> Event EventType: START_CONTAINER for Container
> container_1416310496070_0001_01_02
> 2014-11-18 05:36:37,699 INFO
> [org.apache.hadoop.yarn.client.api.async.impl.NMClientAsyncImpl #1]
> impl.NMClientAsyncImpl (NMClientAsyncImpl.java:run(531)) - Processing
> Event EventType: START_CONTAINER for Container
> container_1416310496070_0001_01_03
> 2014-11-18 05:36:37,698 INFO
> [org.apache.hadoop.yarn.client.api.async.impl.NMClientAsyncImpl #0]
> impl.ContainerManagementProtocolProxy
> (ContainerManagementProtocolProxy.java:newProxy(212)) - Opening proxy
> : roosevelt:60486
> 2014-11-18 05:36:37,761 INFO
> [org.apache.hadoop.yarn.client.api.async.impl.NMClientAsyncImpl #2]
> impl.NMClientAsyncImpl (NMClientAsyncImpl.java:run(531)) - Processing
> Event EventType: QUERY_CONTAINER for Container
> container_1416310496070_0001_01_03
> 2014-11-18 05:36:37,771 INFO
> [org.apache.hadoop.yarn.client.api.async.impl.NMClientAsyncImpl #3]
> impl.NMClientAsyncImpl (NMClientAsyncImpl.java:run(531)) - Processing
> Event EventType: QUERY_CONTAINER for Container
> container_1416310496070_0001_01_02
>
>
>
> On Thu, Nov 20, 2014 at 1:32 PM, Alessandro Negro
> wrote:
>
>> Hi Das,
>> I mean as user you run Hadoop yarn resource manager.
>>
>> Alessandro
>>
>> Il giorno 20/nov/2014, alle ore 19:22, D Adams ha
>> scritto:
>>
>> Alessandro,
>> I'm not sure what you mean, should I create a new user on my system
>> named 'yarn'? I'm new to both hadoop and giraph, so I'm not sure.
>>
>> Tripti,
>> I'll get those logs as soon as I can.
>>
>> Thank you both,
>>
>> V/r
>> Das
>> On Nov 20, 2014 3:13 AM, "Alessandro Negro" wrote:
>>
>>>
Re: Help with Giraph on Yarn
Hi Das, I mean as user you run Hadoop yarn resource manager. Alessandro > Il giorno 20/nov/2014, alle ore 19:22, D Adams ha > scritto: > > Alessandro, > I'm not sure what you mean, should I create a new user on my system > named 'yarn'? I'm new to both hadoop and giraph, so I'm not sure. > > Tripti, > I'll get those logs as soon as I can. > > Thank you both, > > V/r > Das > >> On Nov 20, 2014 3:13 AM, "Alessandro Negro" wrote: >> Hi Tripti, >> I agree that a more detailed error log could be useful. >> >> Thanks, >> Alessandro >> >> >> >> >> Il giorno 19/nov/2014, alle ore 15:54, Tripti Singh >> ha scritto: >> >> > --_000_D092AB9C1CB42triptiyahooinccom_ >> > Content-Type: text/plain; charset="Windows-1252" >> > Content-Transfer-Encoding: quoted-printable >> > >> > Hi Alessandro, >> > >> > I guess u r talking about the documentation on Giraph webpage which >> > mention= >> > s addition of this new option which is more or less mandatory for yarn >> > base= >> > d profiles. >> > When I first ran the Giraph on yarn without the proper =96yj option, there >> > = >> > was no way I could figure out from the Application/Container logs that >> > this= >> > was the issue. >> > I think, it=92ll be a good idea to have this message in the logs for easy >> > d= >> > ebugging. >> > >> > Thanks, >> > Tripti. >> > >> > >> > From: Alessandro Negro mailto:alenegr...@yahoo.it>> >> > Reply-To: "user@giraph.apache.org<mailto:user@giraph.apache.org>" >> > > > aph.apache.org<mailto:user@giraph.apache.org>> >> > Date: Tuesday, November 18, 2014 at 4:31 PM >> > To: "user@giraph.apache.org<mailto:user@giraph.apache.org>" >> > > > ache.org<mailto:user@giraph.apache.org>> >> > Subject: Re: Help with Giraph on Yarn >> > >> > Hi Eli, >> > no I think that the help message is clear enough, generally when I read >> > =93= >> > jar=94 I always mean absolute path for jar, but this is a my personal >> > misun= >> > derstanding. >> > >> > Thanks, >> > Alessandro >> > >> > >> > Il giorno 16/nov/2014, alle ore 21:39, Eli Reisman >> > > > m<mailto:apache.mail...@gmail.com>> ha scritto: >> > >> > Sounds like you got it figured out since last time I checked this list, >> > sor= >> > ry it was a pain. Feel free to drop a JIRA ticket if we can make help >> > messa= >> > ge etc. for -yj option more clear, there's lots to improve there. >> > >> > On Sat, Nov 8, 2014 at 7:26 AM, Alessandro Negro >> > > > o:alenegr...@yahoo.it>> wrote: >> > Hi Tripti, >> > finally I was able to run the test with success. It was an issue of >> > permiss= >> > ion since I was running as ale not as yarn. >> > Let me say that now I=92m able to run graph examples on Yarn 2.5.1. This >> > is= >> > the final result: >> > >> > 14/11/08 16:24:00 INFO yarn.GiraphYarnClient: Completed Giraph: >> > org.apache.= >> > giraph.examples.SimpleShortestPathsComputation: SUCCEEDED, total running >> > ti= >> > me: 0 minutes, 21 seconds. >> > >> > >> > Many thanks for your support, >> > Alessandro >> > >> > Il giorno 06/nov/2014, alle ore 15:16, Tripti Singh >> > > > ailto:tri...@yahoo-inc.com>> ha scritto: >> > >> > I don't know if u have access to this node. But if u do, u could check if >> > t= >> > he file is indeed there and u have access to it. >> > >> > Sent from my iPhone >> > >> > On 06-Nov-2014, at 6:12 pm, "Alessandro Negro" >> > > > alenegr...@yahoo.it>> wrote: >> > >> > You are right it works, but now I receive the following error: >> > >> > SLF4J: Class path contains multiple SLF4J bindings. >> > SLF4J: Found binding in >> > [jar:file:/private/tmp/hadoop-yarn/nm-local-dir/use= >> > rcache/ale/appcache/application_1415264041937_0009/filecache/10/giraph-exam= >> > ples-1.1.0-SNAPSHOT-for-hadoop-2.5.1-jar-with-dependencies.jar!/org/slf4j/i= >> > mpl/StaticLoggerBinder.class] >> > SLF4J: Found binding in &g
Re: Help with Giraph on Yarn
Hi Tripti, I agree that a more detailed error log could be useful. Thanks, Alessandro Il giorno 19/nov/2014, alle ore 15:54, Tripti Singh ha scritto: > --_000_D092AB9C1CB42triptiyahooinccom_ > Content-Type: text/plain; charset="Windows-1252" > Content-Transfer-Encoding: quoted-printable > > Hi Alessandro, > > I guess u r talking about the documentation on Giraph webpage which mention= > s addition of this new option which is more or less mandatory for yarn base= > d profiles. > When I first ran the Giraph on yarn without the proper =96yj option, there = > was no way I could figure out from the Application/Container logs that this= > was the issue. > I think, it=92ll be a good idea to have this message in the logs for easy d= > ebugging. > > Thanks, > Tripti. > > > From: Alessandro Negro mailto:alenegr...@yahoo.it>> > Reply-To: "user@giraph.apache.org<mailto:user@giraph.apache.org>" aph.apache.org<mailto:user@giraph.apache.org>> > Date: Tuesday, November 18, 2014 at 4:31 PM > To: "user@giraph.apache.org<mailto:user@giraph.apache.org>" ache.org<mailto:user@giraph.apache.org>> > Subject: Re: Help with Giraph on Yarn > > Hi Eli, > no I think that the help message is clear enough, generally when I read =93= > jar=94 I always mean absolute path for jar, but this is a my personal misun= > derstanding. > > Thanks, > Alessandro > > > Il giorno 16/nov/2014, alle ore 21:39, Eli Reisman m<mailto:apache.mail...@gmail.com>> ha scritto: > > Sounds like you got it figured out since last time I checked this list, sor= > ry it was a pain. Feel free to drop a JIRA ticket if we can make help messa= > ge etc. for -yj option more clear, there's lots to improve there. > > On Sat, Nov 8, 2014 at 7:26 AM, Alessandro Negro o:alenegr...@yahoo.it>> wrote: > Hi Tripti, > finally I was able to run the test with success. It was an issue of permiss= > ion since I was running as ale not as yarn. > Let me say that now I=92m able to run graph examples on Yarn 2.5.1. This is= > the final result: > > 14/11/08 16:24:00 INFO yarn.GiraphYarnClient: Completed Giraph: org.apache.= > giraph.examples.SimpleShortestPathsComputation: SUCCEEDED, total running ti= > me: 0 minutes, 21 seconds. > > > Many thanks for your support, > Alessandro > > Il giorno 06/nov/2014, alle ore 15:16, Tripti Singh ailto:tri...@yahoo-inc.com>> ha scritto: > > I don't know if u have access to this node. But if u do, u could check if t= > he file is indeed there and u have access to it. > > Sent from my iPhone > > On 06-Nov-2014, at 6:12 pm, "Alessandro Negro" alenegr...@yahoo.it>> wrote: > > You are right it works, but now I receive the following error: > > SLF4J: Class path contains multiple SLF4J bindings. > SLF4J: Found binding in [jar:file:/private/tmp/hadoop-yarn/nm-local-dir/use= > rcache/ale/appcache/application_1415264041937_0009/filecache/10/giraph-exam= > ples-1.1.0-SNAPSHOT-for-hadoop-2.5.1-jar-with-dependencies.jar!/org/slf4j/i= > mpl/StaticLoggerBinder.class] > SLF4J: Found binding in [jar:file:/opt/yarn/hadoop-2.5.1/share/hadoop/commo= > n/lib/slf4j-log4j12-1.7.5.jar!/org/slf4j/impl/StaticLoggerBinder.class] > SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explana= > tion. > SLF4J: Actual binding is of type [org.slf4j.impl.Log4jLoggerFactory] > 2014-11-06 13:15:37.120 java[10158:1803] Unable to load realm info from SCD= > ynamicStore > Exception in thread "pool-4-thread-1" java.lang.IllegalStateException: Coul= > d not configure the containerlaunch context for GiraphYarnTasks. > at org.apache.giraph.yarn.GiraphApplicationMaster.getTaskResourceMap(Giraph= > ApplicationMaster.java:391) > at org.apache.giraph.yarn.GiraphApplicationMaster.access$500(GiraphApplicat= > ionMaster.java:78) > at org.apache.giraph.yarn.GiraphApplicationMaster$LaunchContainerRunnable.b= > uildContainerLaunchContext(GiraphApplicationMaster.java:522) > at org.apache.giraph.yarn.GiraphApplicationMaster$LaunchContainerRunnable.r= > un(GiraphApplicationMaster.java:479) > at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.jav= > a:1145) > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.ja= > va:615) > at java.lang.Thread.run(Thread.java:744) > Caused by: java.io.FileNotFoundException: File does not exist: hdfs://hadoo= > p-master:9000/user/yarn/giraph_yarn_jar_cache/application_1415264041937_000= > 9/giraph-examples-1.1.0-SNAPSHOT-for-hadoop-2.5.1-jar-with-dependencies.jar > at org.apache.hadoop.hdfs.DistributedFileSystem$17.doCall(DistributedFileSy= > stem.java:1072) > at org.ap
Re: Help with Giraph on Yarn
Hi Das, if you are referring to the following error: >> Caused by: java.io.FileNotFoundException: File does not exist: >> hdfs://hadoop-master:9000/user/yarn/giraph_yarn_jar_cache/application_1415264041937_0009/giraph-examples-1.1.0-SNAPSHOT-for-hadoop-2.5.1-jar-with-dependencies.jar > I just run the script as yarn user, since as you can see from the error, it try to access to the graph cache on the yarn user. What error are you receiving? Regards, Alessandro Il giorno 17/nov/2014, alle ore 17:10, D Adams ha scritto: > Greetings, > Alessandro, I keep getting the same errors you were getting but from > reading the thread, I still cannot figure out how to remedy it. Could you > please break down how you solved this issue? Thank you > > V/r > Das > > On Nov 16, 2014 3:40 PM, "Eli Reisman" wrote: > Sounds like you got it figured out since last time I checked this list, sorry > it was a pain. Feel free to drop a JIRA ticket if we can make help message > etc. for -yj option more clear, there's lots to improve there. > > On Sat, Nov 8, 2014 at 7:26 AM, Alessandro Negro wrote: > Hi Tripti, > finally I was able to run the test with success. It was an issue of > permission since I was running as ale not as yarn. > Let me say that now I’m able to run graph examples on Yarn 2.5.1. This is the > final result: > > 14/11/08 16:24:00 INFO yarn.GiraphYarnClient: Completed Giraph: > org.apache.giraph.examples.SimpleShortestPathsComputation: SUCCEEDED, total > running time: 0 minutes, 21 seconds. > > > Many thanks for your support, > Alessandro > > Il giorno 06/nov/2014, alle ore 15:16, Tripti Singh ha > scritto: > >> I don't know if u have access to this node. But if u do, u could check if >> the file is indeed there and u have access to it. >> >> Sent from my iPhone >> >> On 06-Nov-2014, at 6:12 pm, "Alessandro Negro" wrote: >> >>> You are right it works, but now I receive the following error: >>> >>> SLF4J: Class path contains multiple SLF4J bindings. >>> SLF4J: Found binding in >>> [jar:file:/private/tmp/hadoop-yarn/nm-local-dir/usercache/ale/appcache/application_1415264041937_0009/filecache/10/giraph-examples-1.1.0-SNAPSHOT-for-hadoop-2.5.1-jar-with-dependencies.jar!/org/slf4j/impl/StaticLoggerBinder.class] >>> SLF4J: Found binding in >>> [jar:file:/opt/yarn/hadoop-2.5.1/share/hadoop/common/lib/slf4j-log4j12-1.7.5.jar!/org/slf4j/impl/StaticLoggerBinder.class] >>> SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an >>> explanation. >>> SLF4J: Actual binding is of type [org.slf4j.impl.Log4jLoggerFactory] >>> 2014-11-06 13:15:37.120 java[10158:1803] Unable to load realm info from >>> SCDynamicStore >>> Exception in thread "pool-4-thread-1" java.lang.IllegalStateException: >>> Could not configure the containerlaunch context for GiraphYarnTasks. >>> at >>> org.apache.giraph.yarn.GiraphApplicationMaster.getTaskResourceMap(GiraphApplicationMaster.java:391) >>> at >>> org.apache.giraph.yarn.GiraphApplicationMaster.access$500(GiraphApplicationMaster.java:78) >>> at >>> org.apache.giraph.yarn.GiraphApplicationMaster$LaunchContainerRunnable.buildContainerLaunchContext(GiraphApplicationMaster.java:522) >>> at >>> org.apache.giraph.yarn.GiraphApplicationMaster$LaunchContainerRunnable.run(GiraphApplicationMaster.java:479) >>> at >>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) >>> at >>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) >>> at java.lang.Thread.run(Thread.java:744) >>> Caused by: java.io.FileNotFoundException: File does not exist: >>> hdfs://hadoop-master:9000/user/yarn/giraph_yarn_jar_cache/application_1415264041937_0009/giraph-examples-1.1.0-SNAPSHOT-for-hadoop-2.5.1-jar-with-dependencies.jar >>> at >>> org.apache.hadoop.hdfs.DistributedFileSystem$17.doCall(DistributedFileSystem.java:1072) >>> at >>> org.apache.hadoop.hdfs.DistributedFileSystem$17.doCall(DistributedFileSystem.java:1064) >>> at >>> org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) >>> at >>> org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1064) >>> at org.apache.giraph.yarn.YarnUtils.addFileToResourceMap(YarnUtils.java:153) >>> at org.apache.giraph.yarn.YarnUtils.addFsResourcesToMap(YarnUtils.java:77) >>> at >>> org.apache.giraph.yarn.GiraphApplicationMaster.getTaskResourceMap(GiraphApplic
Re: Help with Giraph on Yarn
Hi Eli, no I think that the help message is clear enough, generally when I read “jar” I always mean absolute path for jar, but this is a my personal misunderstanding. Thanks, Alessandro Il giorno 16/nov/2014, alle ore 21:39, Eli Reisman ha scritto: > Sounds like you got it figured out since last time I checked this list, sorry > it was a pain. Feel free to drop a JIRA ticket if we can make help message > etc. for -yj option more clear, there's lots to improve there. > > On Sat, Nov 8, 2014 at 7:26 AM, Alessandro Negro wrote: > Hi Tripti, > finally I was able to run the test with success. It was an issue of > permission since I was running as ale not as yarn. > Let me say that now I’m able to run graph examples on Yarn 2.5.1. This is the > final result: > > 14/11/08 16:24:00 INFO yarn.GiraphYarnClient: Completed Giraph: > org.apache.giraph.examples.SimpleShortestPathsComputation: SUCCEEDED, total > running time: 0 minutes, 21 seconds. > > > Many thanks for your support, > Alessandro > > Il giorno 06/nov/2014, alle ore 15:16, Tripti Singh ha > scritto: > >> I don't know if u have access to this node. But if u do, u could check if >> the file is indeed there and u have access to it. >> >> Sent from my iPhone >> >> On 06-Nov-2014, at 6:12 pm, "Alessandro Negro" wrote: >> >>> You are right it works, but now I receive the following error: >>> >>> SLF4J: Class path contains multiple SLF4J bindings. >>> SLF4J: Found binding in >>> [jar:file:/private/tmp/hadoop-yarn/nm-local-dir/usercache/ale/appcache/application_1415264041937_0009/filecache/10/giraph-examples-1.1.0-SNAPSHOT-for-hadoop-2.5.1-jar-with-dependencies.jar!/org/slf4j/impl/StaticLoggerBinder.class] >>> SLF4J: Found binding in >>> [jar:file:/opt/yarn/hadoop-2.5.1/share/hadoop/common/lib/slf4j-log4j12-1.7.5.jar!/org/slf4j/impl/StaticLoggerBinder.class] >>> SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an >>> explanation. >>> SLF4J: Actual binding is of type [org.slf4j.impl.Log4jLoggerFactory] >>> 2014-11-06 13:15:37.120 java[10158:1803] Unable to load realm info from >>> SCDynamicStore >>> Exception in thread "pool-4-thread-1" java.lang.IllegalStateException: >>> Could not configure the containerlaunch context for GiraphYarnTasks. >>> at >>> org.apache.giraph.yarn.GiraphApplicationMaster.getTaskResourceMap(GiraphApplicationMaster.java:391) >>> at >>> org.apache.giraph.yarn.GiraphApplicationMaster.access$500(GiraphApplicationMaster.java:78) >>> at >>> org.apache.giraph.yarn.GiraphApplicationMaster$LaunchContainerRunnable.buildContainerLaunchContext(GiraphApplicationMaster.java:522) >>> at >>> org.apache.giraph.yarn.GiraphApplicationMaster$LaunchContainerRunnable.run(GiraphApplicationMaster.java:479) >>> at >>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) >>> at >>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) >>> at java.lang.Thread.run(Thread.java:744) >>> Caused by: java.io.FileNotFoundException: File does not exist: >>> hdfs://hadoop-master:9000/user/yarn/giraph_yarn_jar_cache/application_1415264041937_0009/giraph-examples-1.1.0-SNAPSHOT-for-hadoop-2.5.1-jar-with-dependencies.jar >>> at >>> org.apache.hadoop.hdfs.DistributedFileSystem$17.doCall(DistributedFileSystem.java:1072) >>> at >>> org.apache.hadoop.hdfs.DistributedFileSystem$17.doCall(DistributedFileSystem.java:1064) >>> at >>> org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) >>> at >>> org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1064) >>> at org.apache.giraph.yarn.YarnUtils.addFileToResourceMap(YarnUtils.java:153) >>> at org.apache.giraph.yarn.YarnUtils.addFsResourcesToMap(YarnUtils.java:77) >>> at >>> org.apache.giraph.yarn.GiraphApplicationMaster.getTaskResourceMap(GiraphApplicationMaster.java:387) >>> ... 6 more >>> >>> >>> That justify the other error I receive in the task: >>>>> Could not find or load main class org.apache.giraph.yarn.GiraphYarnTask >>> >>> >>> Thanks, >>> >>> Il giorno 06/nov/2014, alle ore 13:07, Tripti Singh >>> ha scritto: >>> >>>> Why r u adding two jars? Example jar ideally contains core library so >>>> everything should be available with just one example jar included >>>> >>>> Sent from my
Re: Help with Giraph on Yarn
Hi Tripti, finally I was able to run the test with success. It was an issue of permission since I was running as ale not as yarn. Let me say that now I’m able to run graph examples on Yarn 2.5.1. This is the final result: 14/11/08 16:24:00 INFO yarn.GiraphYarnClient: Completed Giraph: org.apache.giraph.examples.SimpleShortestPathsComputation: SUCCEEDED, total running time: 0 minutes, 21 seconds. Many thanks for your support, Alessandro Il giorno 06/nov/2014, alle ore 15:16, Tripti Singh ha scritto: > I don't know if u have access to this node. But if u do, u could check if the > file is indeed there and u have access to it. > > Sent from my iPhone > > On 06-Nov-2014, at 6:12 pm, "Alessandro Negro" wrote: > >> You are right it works, but now I receive the following error: >> >> SLF4J: Class path contains multiple SLF4J bindings. >> SLF4J: Found binding in >> [jar:file:/private/tmp/hadoop-yarn/nm-local-dir/usercache/ale/appcache/application_1415264041937_0009/filecache/10/giraph-examples-1.1.0-SNAPSHOT-for-hadoop-2.5.1-jar-with-dependencies.jar!/org/slf4j/impl/StaticLoggerBinder.class] >> SLF4J: Found binding in >> [jar:file:/opt/yarn/hadoop-2.5.1/share/hadoop/common/lib/slf4j-log4j12-1.7.5.jar!/org/slf4j/impl/StaticLoggerBinder.class] >> SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an >> explanation. >> SLF4J: Actual binding is of type [org.slf4j.impl.Log4jLoggerFactory] >> 2014-11-06 13:15:37.120 java[10158:1803] Unable to load realm info from >> SCDynamicStore >> Exception in thread "pool-4-thread-1" java.lang.IllegalStateException: Could >> not configure the containerlaunch context for GiraphYarnTasks. >> at >> org.apache.giraph.yarn.GiraphApplicationMaster.getTaskResourceMap(GiraphApplicationMaster.java:391) >> at >> org.apache.giraph.yarn.GiraphApplicationMaster.access$500(GiraphApplicationMaster.java:78) >> at >> org.apache.giraph.yarn.GiraphApplicationMaster$LaunchContainerRunnable.buildContainerLaunchContext(GiraphApplicationMaster.java:522) >> at >> org.apache.giraph.yarn.GiraphApplicationMaster$LaunchContainerRunnable.run(GiraphApplicationMaster.java:479) >> at >> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) >> at >> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) >> at java.lang.Thread.run(Thread.java:744) >> Caused by: java.io.FileNotFoundException: File does not exist: >> hdfs://hadoop-master:9000/user/yarn/giraph_yarn_jar_cache/application_1415264041937_0009/giraph-examples-1.1.0-SNAPSHOT-for-hadoop-2.5.1-jar-with-dependencies.jar >> at >> org.apache.hadoop.hdfs.DistributedFileSystem$17.doCall(DistributedFileSystem.java:1072) >> at >> org.apache.hadoop.hdfs.DistributedFileSystem$17.doCall(DistributedFileSystem.java:1064) >> at >> org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) >> at >> org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1064) >> at org.apache.giraph.yarn.YarnUtils.addFileToResourceMap(YarnUtils.java:153) >> at org.apache.giraph.yarn.YarnUtils.addFsResourcesToMap(YarnUtils.java:77) >> at >> org.apache.giraph.yarn.GiraphApplicationMaster.getTaskResourceMap(GiraphApplicationMaster.java:387) >> ... 6 more >> >> >> That justify the other error I receive in the task: >>>> Could not find or load main class org.apache.giraph.yarn.GiraphYarnTask >> >> >> Thanks, >> >> Il giorno 06/nov/2014, alle ore 13:07, Tripti Singh >> ha scritto: >> >>> Why r u adding two jars? Example jar ideally contains core library so >>> everything should be available with just one example jar included >>> >>> Sent from my iPhone >>> >>> On 06-Nov-2014, at 4:33 pm, "Alessandro Negro" wrote: >>> >>>> Hi, >>>> now it seems better, I need to add: >>>> >>>> giraph-1.1.0-SNAPSHOT-for-hadoop-2.5.1-jar-with-dependencies.jar,giraph-examples-1.1.0-SNAPSHOT-for-hadoop-2.5.1-jar-with-dependencies.jar >>>> >>>> >>>> Now it seems that after a lot of cycle it fail with this error: >>>> >>>> Could not find or load main class org.apache.giraph.yarn.GiraphYarnTask >>>> >>>> But in this case the error appear in task-3-stderr.log not in >>>> gam-stderr.log where there is the following error: >>>> >>>> LF4J: Class path contains multiple SLF4J bindings. >>>> SLF4J: Found binding in >>>> [jar:file
Re: Help with Giraph on Yarn
You are right it works, but now I receive the following error: SLF4J: Class path contains multiple SLF4J bindings. SLF4J: Found binding in [jar:file:/private/tmp/hadoop-yarn/nm-local-dir/usercache/ale/appcache/application_1415264041937_0009/filecache/10/giraph-examples-1.1.0-SNAPSHOT-for-hadoop-2.5.1-jar-with-dependencies.jar!/org/slf4j/impl/StaticLoggerBinder.class] SLF4J: Found binding in [jar:file:/opt/yarn/hadoop-2.5.1/share/hadoop/common/lib/slf4j-log4j12-1.7.5.jar!/org/slf4j/impl/StaticLoggerBinder.class] SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation. SLF4J: Actual binding is of type [org.slf4j.impl.Log4jLoggerFactory] 2014-11-06 13:15:37.120 java[10158:1803] Unable to load realm info from SCDynamicStore Exception in thread "pool-4-thread-1" java.lang.IllegalStateException: Could not configure the containerlaunch context for GiraphYarnTasks. at org.apache.giraph.yarn.GiraphApplicationMaster.getTaskResourceMap(GiraphApplicationMaster.java:391) at org.apache.giraph.yarn.GiraphApplicationMaster.access$500(GiraphApplicationMaster.java:78) at org.apache.giraph.yarn.GiraphApplicationMaster$LaunchContainerRunnable.buildContainerLaunchContext(GiraphApplicationMaster.java:522) at org.apache.giraph.yarn.GiraphApplicationMaster$LaunchContainerRunnable.run(GiraphApplicationMaster.java:479) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) at java.lang.Thread.run(Thread.java:744) Caused by: java.io.FileNotFoundException: File does not exist: hdfs://hadoop-master:9000/user/yarn/giraph_yarn_jar_cache/application_1415264041937_0009/giraph-examples-1.1.0-SNAPSHOT-for-hadoop-2.5.1-jar-with-dependencies.jar at org.apache.hadoop.hdfs.DistributedFileSystem$17.doCall(DistributedFileSystem.java:1072) at org.apache.hadoop.hdfs.DistributedFileSystem$17.doCall(DistributedFileSystem.java:1064) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1064) at org.apache.giraph.yarn.YarnUtils.addFileToResourceMap(YarnUtils.java:153) at org.apache.giraph.yarn.YarnUtils.addFsResourcesToMap(YarnUtils.java:77) at org.apache.giraph.yarn.GiraphApplicationMaster.getTaskResourceMap(GiraphApplicationMaster.java:387) ... 6 more That justify the other error I receive in the task: >> Could not find or load main class org.apache.giraph.yarn.GiraphYarnTask Thanks, Il giorno 06/nov/2014, alle ore 13:07, Tripti Singh ha scritto: > Why r u adding two jars? Example jar ideally contains core library so > everything should be available with just one example jar included > > Sent from my iPhone > > On 06-Nov-2014, at 4:33 pm, "Alessandro Negro" wrote: > >> Hi, >> now it seems better, I need to add: >> >> giraph-1.1.0-SNAPSHOT-for-hadoop-2.5.1-jar-with-dependencies.jar,giraph-examples-1.1.0-SNAPSHOT-for-hadoop-2.5.1-jar-with-dependencies.jar >> >> >> Now it seems that after a lot of cycle it fail with this error: >> >> Could not find or load main class org.apache.giraph.yarn.GiraphYarnTask >> >> But in this case the error appear in task-3-stderr.log not in >> gam-stderr.log where there is the following error: >> >> LF4J: Class path contains multiple SLF4J bindings. >> SLF4J: Found binding in >> [jar:file:/private/tmp/hadoop-yarn/nm-local-dir/usercache/ale/appcache/application_1415264041937_0006/filecache/12/giraph-1.1.0-SNAPSHOT-for-hadoop-2.5.1-jar-with-dependencies.jar!/org/slf4j/impl/StaticLoggerBinder.class] >> SLF4J: Found binding in >> [jar:file:/private/tmp/hadoop-yarn/nm-local-dir/usercache/ale/appcache/application_1415264041937_0006/filecache/10/giraph-examples-1.1.0-SNAPSHOT-for-hadoop-2.5.1-jar-with-dependencies.jar!/org/slf4j/impl/StaticLoggerBinder.class] >> SLF4J: Found binding in >> [jar:file:/opt/yarn/hadoop-2.5.1/share/hadoop/common/lib/slf4j-log4j12-1.7.5.jar!/org/slf4j/impl/StaticLoggerBinder.class] >> SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an >> explanation. >> SLF4J: Actual binding is of type [org.slf4j.impl.Log4jLoggerFactory] >> 2014-11-06 11:31:25.186 java[8162:1803] Unable to load realm info from >> SCDynamicStore >> Exception in thread "pool-4-thread-1" java.lang.IllegalStateException: Could >> not configure the containerlaunch context for GiraphYarnTasks. >> at >> org.apache.giraph.yarn.GiraphApplicationMaster.getTaskResourceMap(GiraphApplicationMaster.java:391) >> at >> org.apache.giraph.yarn.GiraphApplicationMaster.
Re: Help with Giraph on Yarn
Hi, now it seems better, I need to add: giraph-1.1.0-SNAPSHOT-for-hadoop-2.5.1-jar-with-dependencies.jar,giraph-examples-1.1.0-SNAPSHOT-for-hadoop-2.5.1-jar-with-dependencies.jar Now it seems that after a lot of cycle it fail with this error: Could not find or load main class org.apache.giraph.yarn.GiraphYarnTask But in this case the error appear in task-3-stderr.log not in gam-stderr.log where there is the following error: LF4J: Class path contains multiple SLF4J bindings. SLF4J: Found binding in [jar:file:/private/tmp/hadoop-yarn/nm-local-dir/usercache/ale/appcache/application_1415264041937_0006/filecache/12/giraph-1.1.0-SNAPSHOT-for-hadoop-2.5.1-jar-with-dependencies.jar!/org/slf4j/impl/StaticLoggerBinder.class] SLF4J: Found binding in [jar:file:/private/tmp/hadoop-yarn/nm-local-dir/usercache/ale/appcache/application_1415264041937_0006/filecache/10/giraph-examples-1.1.0-SNAPSHOT-for-hadoop-2.5.1-jar-with-dependencies.jar!/org/slf4j/impl/StaticLoggerBinder.class] SLF4J: Found binding in [jar:file:/opt/yarn/hadoop-2.5.1/share/hadoop/common/lib/slf4j-log4j12-1.7.5.jar!/org/slf4j/impl/StaticLoggerBinder.class] SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation. SLF4J: Actual binding is of type [org.slf4j.impl.Log4jLoggerFactory] 2014-11-06 11:31:25.186 java[8162:1803] Unable to load realm info from SCDynamicStore Exception in thread "pool-4-thread-1" java.lang.IllegalStateException: Could not configure the containerlaunch context for GiraphYarnTasks. at org.apache.giraph.yarn.GiraphApplicationMaster.getTaskResourceMap(GiraphApplicationMaster.java:391) at org.apache.giraph.yarn.GiraphApplicationMaster.access$500(GiraphApplicationMaster.java:78) at org.apache.giraph.yarn.GiraphApplicationMaster$LaunchContainerRunnable.buildContainerLaunchContext(GiraphApplicationMaster.java:522) at org.apache.giraph.yarn.GiraphApplicationMaster$LaunchContainerRunnable.run(GiraphApplicationMaster.java:479) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) at java.lang.Thread.run(Thread.java:744) Caused by: java.io.FileNotFoundException: File does not exist: hdfs://hadoop-master:9000/user/yarn/giraph_yarn_jar_cache/application_1415264041937_0006/giraph-1.1.0-SNAPSHOT-for-hadoop-2.5.1-jar-with-dependencies.jar at org.apache.hadoop.hdfs.DistributedFileSystem$17.doCall(DistributedFileSystem.java:1072) at org.apache.hadoop.hdfs.DistributedFileSystem$17.doCall(DistributedFileSystem.java:1064) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1064) at org.apache.giraph.yarn.YarnUtils.addFileToResourceMap(YarnUtils.java:153) at org.apache.giraph.yarn.YarnUtils.addFsResourcesToMap(YarnUtils.java:77) at org.apache.giraph.yarn.GiraphApplicationMaster.getTaskResourceMap(GiraphApplicationMaster.java:387) ... 6 more Il giorno 06/nov/2014, alle ore 11:32, Tripti Singh ha scritto: > Hi, > I remember seeing this thread, but since i was already deep into fixing this > issue in production, I didn't really wait for the patch, not even the munge > symbol as I was not even able to compile the yarn profile( sorry for being so > ruthless to giraph :-( ) > I did later cross check with the patch and it was quite similar to my change > but sure it wouldn't work for all profiles(in fact some profiles may even > break due to Hadoop class incompatibility ) I can share those changes with u > once I get back to my computer but I don't think that could have caused this > problem. > Just try the small changed I mentioned in my previous reply and let me know > if that helped. > > Thanks, > Tripti > Sent from my iPhone > > On 06-Nov-2014, at 3:52 pm, "Alessandro Negro" wrote: > >> Hi Tripti, >> let me add one info, As you can see from this thread of email: >> >> http://mail-archives.apache.org/mod_mbox/giraph-user/201410.mbox/%3cc3f75a98-bde8-4205-971f-1f7241e6e...@yahoo.it%3E >> >> I had some issue while compiling for Hadoop 2.5.1 could be that into the >> process I loose something? >> >> Thanks again, >> Alessandro >> >> >> >> Il giorno 06/nov/2014, alle ore 10:47, Alessandro Negro >> ha scritto: >> >>> Hi Tripti, >>> yes I run in this way: >>> >>> >>> $HADOOP_INSTALL/bin/hadoop jar >>> /Users/ale/extprj/giraph/giraph-examples/target/giraph-examples-1.1.0-SNAPSHOT-for-hadoop-2.5.1-jar-with-dependencies.jar >>> org.apache.giraph.GiraphRunner >>> org.a
Re: Help with Giraph on Yarn
Hi Tripti,
let me add one info, As you can see from this thread of email:
http://mail-archives.apache.org/mod_mbox/giraph-user/201410.mbox/%3cc3f75a98-bde8-4205-971f-1f7241e6e...@yahoo.it%3E
I had some issue while compiling for Hadoop 2.5.1 could be that into the
process I loose something?
Thanks again,
Alessandro
Il giorno 06/nov/2014, alle ore 10:47, Alessandro Negro
ha scritto:
> Hi Tripti,
> yes I run in this way:
>
>
> $HADOOP_INSTALL/bin/hadoop jar
> /Users/ale/extprj/giraph/giraph-examples/target/giraph-examples-1.1.0-SNAPSHOT-for-hadoop-2.5.1-jar-with-dependencies.jar
> org.apache.giraph.GiraphRunner
> org.apache.giraph.examples.SimpleShortestPathsComputation -vif
> org.apache.giraph.io.formats.JsonLongDoubleFloatDoubleVertexInputFormat -vip
> /tiny_graph.txt -vof org.apache.giraph.io.formats.IdWithValueTextOutputFormat
> -op /shortestpaths -w 1 -yj
> /Users/ale/extprj/giraph/giraph-core/target/giraph-1.1.0-SNAPSHOT-for-hadoop-2.5.1-jar-with-dependencies.jar,/Users/ale/extprj/giraph/giraph-examples/target/giraph-examples-1.1.0-SNAPSHOT-for-hadoop-2.5.1-jar-with-dependencies.jar
>
> And this is the complete trace:
>
> 2014-11-06 10:06:05.638 java[7778:1903] Unable to load realm info from
> SCDynamicStore
> 14/11/06 10:06:05 WARN util.NativeCodeLoader: Unable to load native-hadoop
> library for your platform... using builtin-java classes where applicable
> 14/11/06 10:06:06 INFO utils.ConfigurationUtils: No edge input format
> specified. Ensure your InputFormat does not require one.
> 14/11/06 10:06:06 INFO utils.ConfigurationUtils: No edge output format
> specified. Ensure your OutputFormat does not require one.
> 14/11/06 10:06:06 INFO yarn.GiraphYarnClient: Final output path is:
> hdfs://hadoop-master:9000/shortestpaths
> 14/11/06 10:06:06 INFO yarn.GiraphYarnClient: Running Client
> 14/11/06 10:06:06 INFO client.RMProxy: Connecting to ResourceManager at
> /0.0.0.0:8032
> 14/11/06 10:06:06 INFO yarn.GiraphYarnClient: Got node report from ASM for,
> nodeId=10.61.137.28:59677, nodeAddress 10.61.137.28:8042, nodeRackName
> /default-rack, nodeNumContainers 0
> 14/11/06 10:06:06 INFO yarn.GiraphYarnClient: Defaulting per-task heap size
> to 1024MB.
> 14/11/06 10:06:06 INFO yarn.GiraphYarnClient: Obtained new Application ID:
> application_1415264041937_0005
> 14/11/06 10:06:06 INFO Configuration.deprecation: mapred.job.id is
> deprecated. Instead, use mapreduce.job.id
> 14/11/06 10:06:06 INFO yarn.GiraphYarnClient: Set the environment for the
> application master
> 14/11/06 10:06:06 INFO yarn.GiraphYarnClient: Environment for AM
> :{CLASSPATH=${CLASSPATH}:./*:$HADOOP_CONF_DIR:$HADOOP_COMMON_HOME/share/hadoop/common/*:$HADOOP_COMMON_HOME/share/hadoop/common/lib/*:$HADOOP_HDFS_HOME/share/hadoop/hdfs/*:$HADOOP_HDFS_HOME/share/hadoop/hdfs/lib/*:$HADOOP_YARN_HOME/share/hadoop/yarn/*:$HADOOP_YARN_HOME/share/hadoop/yarn/lib/*:$HADOOP_MAPRED_HOME/share/hadoop/mapreduce/*:$HADOOP_MAPRED_HOME/share/hadoop/mapreduce/lib/*}
> 14/11/06 10:06:06 INFO yarn.GiraphYarnClient: buildLocalResourceMap
> 14/11/06 10:06:06 INFO yarn.YarnUtils: Registered file in LocalResources ::
> hdfs://hadoop-master:9000/user/ale/giraph_yarn_jar_cache/application_1415264041937_0005/giraph-conf.xml
> 14/11/06 10:06:06 INFO yarn.GiraphYarnClient: LIB JARS
> :/Users/ale/extprj/giraph/giraph-core/target/giraph-1.1.0-SNAPSHOT-for-hadoop-2.5.1-jar-with-dependencies.jar,/Users/ale/extprj/giraph/giraph-examples/target/giraph-examples-1.1.0-SNAPSHOT-for-hadoop-2.5.1-jar-with-dependencies.jar
> 14/11/06 10:06:06 INFO yarn.YarnUtils: Class path name .
> 14/11/06 10:06:06 INFO yarn.YarnUtils: base path checking .
> 14/11/06 10:06:06 INFO yarn.YarnUtils: Class path name null
> 14/11/06 10:06:06 INFO yarn.YarnUtils: base path checking null
> 14/11/06 10:06:06 INFO yarn.YarnUtils: Class path name
> /opt/yarn/hadoop-2.5.1/etc/hadoop
> 14/11/06 10:06:06 INFO yarn.YarnUtils: base path checking
> /opt/yarn/hadoop-2.5.1/etc/hadoop
> 14/11/06 10:06:06 INFO yarn.YarnUtils: Class path name
> /opt/yarn/hadoop-2.5.1/share/hadoop/common/lib/*
> 14/11/06 10:06:06 INFO yarn.YarnUtils: base path checking
> /opt/yarn/hadoop-2.5.1/share/hadoop/common/lib
> 14/11/06 10:06:06 INFO yarn.YarnUtils: Class path name
> /opt/yarn/hadoop-2.5.1/share/hadoop/common/*
> 14/11/06 10:06:06 INFO yarn.YarnUtils: base path checking
> /opt/yarn/hadoop-2.5.1/share/hadoop/common
> 14/11/06 10:06:06 INFO yarn.YarnUtils: Class path name
> /opt/yarn/hadoop-2.5.1/share/hadoop/hdfs
> 14/11/06 10:06:06 INFO yarn.YarnUtils: base path checking
> /opt/yarn/hadoop-2.5.1/share/hadoop/hdfs
> 14/11/06 10:06:06 INFO yarn.YarnUtils: Class path name
> /opt/yarn/hadoop-2.5.1/share/hadoop/hdfs/lib/*
> 14/11/06 10:06:06 INFO yarn.YarnUtils: base path che
Re: Help with Giraph on Yarn
/share/hadoop/mapreduce/lib/*
14/11/06 10:06:06 INFO yarn.YarnUtils: base path checking
/opt/yarn/hadoop-2.5.1/share/hadoop/mapreduce/lib
14/11/06 10:06:06 INFO yarn.YarnUtils: Class path name
/opt/yarn/hadoop-2.5.1/share/hadoop/mapreduce/*
14/11/06 10:06:06 INFO yarn.YarnUtils: base path checking
/opt/yarn/hadoop-2.5.1/share/hadoop/mapreduce
14/11/06 10:06:06 INFO yarn.YarnUtils: Class path name
14/11/06 10:06:06 INFO yarn.YarnUtils: Class path name
/opt/yarn/hadoop-2.5.1/share/myLib/*.jar
14/11/06 10:06:06 INFO yarn.YarnUtils: base path checking
/opt/yarn/hadoop-2.5.1/share/myLib
14/11/06 10:06:06 INFO yarn.YarnUtils: Class path name
/Users/ale/extprj/giraph/myGiraph/*.jar
14/11/06 10:06:06 INFO yarn.YarnUtils: base path checking
/Users/ale/extprj/giraph/myGiraph
14/11/06 10:06:06 INFO yarn.YarnUtils: Class path name
/Users/ale/extprj/giraph/myGiraph/lib/*.jar
14/11/06 10:06:06 INFO yarn.YarnUtils: base path checking
/Users/ale/extprj/giraph/myGiraph/lib
14/11/06 10:06:06 INFO yarn.YarnUtils: Class path name
/contrib/capacity-scheduler/*.jar
14/11/06 10:06:06 INFO yarn.YarnUtils: base path checking
/contrib/capacity-scheduler
14/11/06 10:06:06 INFO yarn.GiraphYarnClient: ApplicationSumbissionContext for
GiraphApplicationMaster launch container is populated.
14/11/06 10:06:06 INFO yarn.GiraphYarnClient: Submitting application to ASM
14/11/06 10:06:06 INFO impl.YarnClientImpl: Submitted application
application_1415264041937_0005
14/11/06 10:06:06 INFO yarn.GiraphYarnClient: Got new appId after submission
:application_1415264041937_0005
14/11/06 10:06:06 INFO yarn.GiraphYarnClient: GiraphApplicationMaster container
request was submitted to ResourceManager for job: Giraph:
org.apache.giraph.examples.SimpleShortestPathsComputation
14/11/06 10:06:07 INFO yarn.GiraphYarnClient: Giraph:
org.apache.giraph.examples.SimpleShortestPathsComputation, Elapsed: 0,83 secs
14/11/06 10:06:07 INFO yarn.GiraphYarnClient:
appattempt_1415264041937_0005_01, State: ACCEPTED, Containers used: 1
14/11/06 10:06:10 ERROR yarn.GiraphYarnClient: Giraph:
org.apache.giraph.examples.SimpleShortestPathsComputation reports FAILED state,
diagnostics show: Application application_1415264041937_0005 failed 2 times due
to AM Container for appattempt_1415264041937_0005_02 exited with exitCode:
1 due to: Exception from container-launch: ExitCodeException exitCode=1:
ExitCodeException exitCode=1:
at org.apache.hadoop.util.Shell.runCommand(Shell.java:538)
at org.apache.hadoop.util.Shell.run(Shell.java:455)
at
org.apache.hadoop.util.Shell$ShellCommandExecutor.execute(Shell.java:702)
at
org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor.launchContainer(DefaultContainerExecutor.java:195)
at
org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:300)
at
org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:81)
at java.util.concurrent.FutureTask.run(FutureTask.java:262)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:744)
Container exited with a non-zero exit code 1
.Failing this attempt.. Failing the application.
14/11/06 10:06:10 INFO yarn.GiraphYarnClient: Cleaning up HDFS distributed
cache directory for Giraph job.
14/11/06 10:06:10 INFO yarn.GiraphYarnClient: Completed Giraph:
org.apache.giraph.examples.SimpleShortestPathsComputation: FAILED, total
running time: 0 minutes, 3 seconds.
ale:~/extprj/giraph/myGiraph />yarn logs -applicationId
application_1415264041937_0005
As you can see there is this piece:
14/11/06 10:06:06 INFO yarn.GiraphYarnClient: LIB JARS
:/Users/ale/extprj/giraph/giraph-core/target/giraph-1.1.0-SNAPSHOT-for-hadoop-2.5.1-jar-with-dependencies.jar,/Users/ale/extprj/giraph/giraph-examples/target/giraph-examples-1.1.0-SNAPSHOT-for-hadoop-2.5.1-jar-with-dependencies.jar
But I continue to receive the error:
Thanks,
Alessandro
Il giorno 06/nov/2014, alle ore 10:25, Tripti Singh ha
scritto:
> For GiraphApplicationMaster not found, the simple thing that worked for me
> was adding a -yj parameter to my command.
> (hadoop jar {jar-file-path} {path to giraph runner} {computation-class}
> {bunch of parameters} -yj {jar-file-name})
> If u look at job logs,
> Do u see a log statement defining "LIB JARS"?
>
> Sent from my iPhone
>
> On 06-Nov-2014, at 2:45 pm, "Alessandro Negro" wrote:
>
>> Hi,
>> yea my logs shows that error. I follow this example:
>>
>> http://mail-archives.apache.org/mod_mbox/giraph-user/201408.mbox/%3c53f4c689.5060...@web.de%3E
>>
>> without success.
>> Then I try to add all jars I can from giraph adding the foll
Re: Help with Giraph on Yarn
Hi, yea my logs shows that error. I follow this example: http://mail-archives.apache.org/mod_mbox/giraph-user/201408.mbox/%3c53f4c689.5060...@web.de%3E without success. Then I try to add all jars I can from giraph adding the following lines into the hadoop-env.sh: export MYLIB=/opt/yarn/hadoop-2.5.1/share/myLib/*.jar:/Users/ale/extprj/giraph/myGiraph/*.jar:/Users/ale/extprj/giraph/myGiraph/lib/*.jar export HADOOP_CLASSPATH=$HADOOP_CLASSPATH:$MYLIB when I run the application now I see this: 14/11/06 10:06:06 INFO yarn.YarnUtils: base path checking /opt/yarn/hadoop-2.5.1/share/hadoop/mapreduce 14/11/06 10:06:06 INFO yarn.YarnUtils: Class path name 14/11/06 10:06:06 INFO yarn.YarnUtils: Class path name /opt/yarn/hadoop-2.5.1/share/myLib/*.jar 14/11/06 10:06:06 INFO yarn.YarnUtils: base path checking /opt/yarn/hadoop-2.5.1/share/myLib 14/11/06 10:06:06 INFO yarn.YarnUtils: Class path name /Users/ale/extprj/giraph/myGiraph/*.jar 14/11/06 10:06:06 INFO yarn.YarnUtils: base path checking /Users/ale/extprj/giraph/myGiraph 14/11/06 10:06:06 INFO yarn.YarnUtils: Class path name /Users/ale/extprj/giraph/myGiraph/lib/*.jar 14/11/06 10:06:06 INFO yarn.YarnUtils: base path checking /Users/ale/extprj/giraph/myGiraph/lib But again the same error. What I think is that there is no problem in the library but while is starting up. I saw this piece in the logs: 14/11/06 10:06:06 INFO yarn.GiraphYarnClient: ApplicationSumbissionContext for GiraphApplicationMaster launch container is populated. 14/11/06 10:06:06 INFO yarn.GiraphYarnClient: Submitting application to ASM 14/11/06 10:06:06 INFO impl.YarnClientImpl: Submitted application application_1415264041937_0005 14/11/06 10:06:06 INFO yarn.GiraphYarnClient: Got new appId after submission :application_1415264041937_0005 14/11/06 10:06:06 INFO yarn.GiraphYarnClient: GiraphApplicationMaster container request was submitted to ResourceManager for job: Giraph: org.apache.giraph.examples.SimpleShortestPathsComputation 14/11/06 10:06:07 INFO yarn.GiraphYarnClient: Giraph: org.apache.giraph.examples.SimpleShortestPathsComputation, Elapsed: 0,83 secs That show that it is able to load GiraphApplicationMaster but fail during the startup. Where can I find more logs? Thanks, Alessandro Il giorno 06/nov/2014, alle ore 06:56, Tripti Singh ha scritto: > Can u run the following command to find container logs and post it here: > > yarn logs -applicationId application_1415203231170_0002 > > If u find this error: > Could not find or load main class > org.apache.giraph.yarn.GiraphApplicationMaster > It may just be due to non-availablity of the yarn jars > Thanks, > Tripti > Tech Yahoo > > From: Jing Fan > Reply-To: "user@giraph.apache.org" > Date: Thursday, November 6, 2014 at 1:56 AM > To: "user@giraph.apache.org" > Subject: Re: Help with Giraph on Yarn > > Then do you know to solve this error? > > 14/11/05 10:25:29 ERROR yarn.GiraphYarnClient: Giraph: > org.apache.giraph.examples.SimpleShortestPathsComputation reports FAILED > state, diagnostics show: Application application_1415203231170_0002 failed 2 > times due to AM Container for appattempt_1415203231170_0002_02 exited > with exitCode: 1 due to: Exception from container-launch. > Container id: container_1415203231170_0002_02_01 > Exit code: 1 > Stack trace: ExitCodeException exitCode=1: > at org.apache.hadoop.util.Shell.runCommand(Shell.java:538) > at org.apache.hadoop.util.Shell.run(Shell.java:455) > at org.apache.hadoop.util.Shell$ShellCommandExecutor.execute(Shell.java:702) > at > org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor.launchContainer(DefaultContainerExecutor.java:196) > at > org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:299) > at > org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:81) > at java.util.concurrent.FutureTask.run(FutureTask.java:262) > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) > at java.lang.Thread.run(Thread.java:745) > > > On Wed, Nov 5, 2014 at 12:10 PM, Alessandro Negro wrote: >> Hi, >> in the mean time I found the reason why it hangs for long time here: >> >> base path checking . >> >> >> It scan all the directory tree starting from current directory, if you run >> it into your home (and you have a lot of files and directories) it takes >> ages to complete. >> >> I continue to got the same error. >> >> Regards, >> Alessandro >> >> Il giorno 05/nov/2014, alle ore 18:33, Jing Fan ha >> scr
Re: Help with Giraph on Yarn
Hi,
in the mean time I found the reason why it hangs for long time here:
base path checking .
It scan all the directory tree starting from current directory, if you run it
into your home (and you have a lot of files and directories) it takes ages to
complete.
I continue to got the same error.
Regards,
Alessandro
Il giorno 05/nov/2014, alle ore 18:33, Jing Fan ha
scritto:
> I run with:
>
> hadoop jar
> giraph-examples/target/giraph-examples-1.1.0-SNAPSHOT-for-hadoop-2.5.0-cdh5.2.0-jar-with-dependencies.jar
> org.apache.giraph.GiraphRunner
> org.apache.giraph.examples.SimpleShortestPathsComputation -vif
> org.apache.giraph.io.formats.JsonLongDoubleFloatDoubleVertexInputFormat -vip
> /user/jing/tiny_graph.txt -vof
> org.apache.giraph.io.formats.IdWithValueTextOutputFormat -op
> /user/jing/shortestpaths -w 1 -yj
> giraph-core/target/giraph-1.1.0-SNAPSHOT-for-hadoop-2.5.0-cdh5.2.0-jar-with-dependencies.jar
>
> On Wed, Nov 5, 2014 at 11:25 AM, Alessandro Negro wrote:
> This is my complete logs with the command I run:
>
>
> ale:~ />$HADOOP_INSTALL/bin/hadoop jar
> /Users/ale/extprj/giraph/giraph-examples/target/giraph-examples-1.1.0-SNAPSHOT-for-hadoop-2.5.1-jar-with-dependencies.jar
> org.apache.giraph.GiraphRunner
> org.apache.giraph.examples.SimpleShortestPathsComputation -vif
> org.apache.giraph.io.formats.JsonLongDoubleFloatDoubleVertexInputFormat -vip
> /tiny_graph.txt -vof org.apache.giraph.io.formats.IdWithValueTextOutputFormat
> -op /shortestpaths -w 1
> 2014-11-05 13:39:30.235 java[3147:1903] Unable to load realm info from
> SCDynamicStore
> 14/11/05 13:39:30 WARN util.NativeCodeLoader: Unable to load native-hadoop
> library for your platform... using builtin-java classes where applicable
> 14/11/05 13:39:30 INFO utils.ConfigurationUtils: No edge input format
> specified. Ensure your InputFormat does not require one.
> 14/11/05 13:39:30 INFO utils.ConfigurationUtils: No edge output format
> specified. Ensure your OutputFormat does not require one.
> 14/11/05 13:39:31 INFO yarn.GiraphYarnClient: Final output path is:
> hdfs://hadoop-master:9000/shortestpaths
> 14/11/05 13:39:31 INFO yarn.GiraphYarnClient: Running Client
> 14/11/05 13:39:31 INFO client.RMProxy: Connecting to ResourceManager at
> /0.0.0.0:8032
> 14/11/05 13:39:31 INFO yarn.GiraphYarnClient: Got node report from ASM for,
> nodeId=192.168.0.105:50658, nodeAddress 192.168.0.105:8042, nodeRackName
> /default-rack, nodeNumContainers 0
> 14/11/05 13:39:31 INFO yarn.GiraphYarnClient: Defaulting per-task heap size
> to 1024MB.
> 14/11/05 13:39:31 INFO yarn.GiraphYarnClient: Obtained new Application ID:
> application_1415189069846_0004
> 14/11/05 13:39:31 INFO Configuration.deprecation: mapred.job.id is
> deprecated. Instead, use mapreduce.job.id
> 14/11/05 13:39:31 INFO yarn.GiraphYarnClient: Set the environment for the
> application master
> 14/11/05 13:39:31 INFO yarn.GiraphYarnClient: Environment for AM
> :{CLASSPATH=${CLASSPATH}:./*:$HADOOP_CONF_DIR:$HADOOP_COMMON_HOME/share/hadoop/common/*:$HADOOP_COMMON_HOME/share/hadoop/common/lib/*:$HADOOP_HDFS_HOME/share/hadoop/hdfs/*:$HADOOP_HDFS_HOME/share/hadoop/hdfs/lib/*:$HADOOP_YARN_HOME/share/hadoop/yarn/*:$HADOOP_YARN_HOME/share/hadoop/yarn/lib/*:$HADOOP_MAPRED_HOME/share/hadoop/mapreduce/*:$HADOOP_MAPRED_HOME/share/hadoop/mapreduce/lib/*}
> 14/11/05 13:39:31 INFO yarn.GiraphYarnClient: buildLocalResourceMap
> 14/11/05 13:39:31 INFO yarn.YarnUtils: Registered file in LocalResources ::
> hdfs://hadoop-master:9000/user/ale/giraph_yarn_jar_cache/application_1415189069846_0004/giraph-conf.xml
> 14/11/05 13:39:31 INFO yarn.GiraphYarnClient: LIB JARS :
> 14/11/05 13:39:31 INFO yarn.YarnUtils: Class path name .
> 14/11/05 13:39:31 INFO yarn.YarnUtils: base path checking .
> 14/11/05 14:42:56 INFO yarn.YarnUtils: Class path name null
> 14/11/05 14:42:56 INFO yarn.YarnUtils: base path checking null
> 14/11/05 14:42:56 INFO yarn.YarnUtils: Class path name
> /opt/yarn/hadoop-2.5.1/etc/hadoop
> 14/11/05 14:42:56 INFO yarn.YarnUtils: base path checking
> /opt/yarn/hadoop-2.5.1/etc/hadoop
> 14/11/05 14:42:56 INFO yarn.YarnUtils: Class path name
> /opt/yarn/hadoop-2.5.1/share/hadoop/common/lib/*
> 14/11/05 14:42:56 INFO yarn.YarnUtils: base path checking
> /opt/yarn/hadoop-2.5.1/share/hadoop/common/lib
> 14/11/05 14:42:56 INFO yarn.YarnUtils: Class path name
> /opt/yarn/hadoop-2.5.1/share/hadoop/common/*
> 14/11/05 14:42:56 INFO yarn.YarnUtils: base path checking
> /opt/yarn/hadoop-2.5.1/share/hadoop/common
> 14/11/05 14:42:56 INFO yarn.YarnUtils: Class path name
> /opt/yarn/hadoop-2.5.1/share/hadoop/hdfs
> 14/11/05 14:42:56 INFO yarn.YarnUtils: base path checking
> /opt/yarn/hadoop-2.5.1/share/hadoop/hdfs
> 14/11/05 14:
Re: Help with Giraph on Yarn
ase path checking /contrib/capacity-scheduler 14/11/05 14:42:56 INFO yarn.GiraphYarnClient: ApplicationSumbissionContext for GiraphApplicationMaster launch container is populated. 14/11/05 14:42:56 INFO yarn.GiraphYarnClient: Submitting application to ASM 14/11/05 14:42:57 INFO impl.YarnClientImpl: Submitted application application_1415189069846_0004 14/11/05 14:42:57 INFO yarn.GiraphYarnClient: Got new appId after submission :application_1415189069846_0004 14/11/05 14:42:57 INFO yarn.GiraphYarnClient: GiraphApplicationMaster container request was submitted to ResourceManager for job: Giraph: org.apache.giraph.examples.SimpleShortestPathsComputation 14/11/05 14:42:57 INFO yarn.GiraphYarnClient: Giraph: org.apache.giraph.examples.SimpleShortestPathsComputation, Elapsed: 0,82 secs 14/11/05 14:42:57 INFO yarn.GiraphYarnClient: appattempt_1415189069846_0004_01, State: ACCEPTED, Containers used: 1 14/11/05 14:43:01 ERROR yarn.GiraphYarnClient: Giraph: org.apache.giraph.examples.SimpleShortestPathsComputation reports FAILED state, diagnostics show: Application application_1415189069846_0004 failed 2 times due to AM Container for appattempt_1415189069846_0004_02 exited with exitCode: 1 due to: Exception from container-launch: ExitCodeException exitCode=1: ExitCodeException exitCode=1: at org.apache.hadoop.util.Shell.runCommand(Shell.java:538) at org.apache.hadoop.util.Shell.run(Shell.java:455) at org.apache.hadoop.util.Shell$ShellCommandExecutor.execute(Shell.java:702) at org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor.launchContainer(DefaultContainerExecutor.java:195) at org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:300) at org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:81) at java.util.concurrent.FutureTask.run(FutureTask.java:262) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) at java.lang.Thread.run(Thread.java:744) Container exited with a non-zero exit code 1 .Failing this attempt.. Failing the application. 14/11/05 14:43:01 INFO yarn.GiraphYarnClient: Cleaning up HDFS distributed cache directory for Giraph job. 14/11/05 14:43:01 INFO yarn.GiraphYarnClient: Completed Giraph: org.apache.giraph.examples.SimpleShortestPathsComputation: FAILED, total running time: 0 minutes, 3 seconds. Il giorno 05/nov/2014, alle ore 18:17, Tripti Singh ha scritto: > I faced the similar problem where the workflow just hangs. > Can u see in ur logs if there is a task with id 0. > I made a local fix and I could give u the patch if u see a similar problem > and ur workflow is unable to find the zookeeper manager file and repeats the > following message: > For task xyz, got file 'null' > > Thanks, > Tripti > Sent from my iPhone > > On 05-Nov-2014, at 10:40 pm, "Alessandro Negro" wrote: > >> Hi, >> have you seen the hadoop logs? >> Could you write here how are you running the command? >> >> Thanks, >> Alessandro >> >> Il giorno 05/nov/2014, alle ore 18:04, Jing Fan ha >> scritto: >> >>> Yeah, I add -yj option. The result is the same. Still error here. >>> >>> On Wed, Nov 5, 2014 at 10:52 AM, Alessandro Negro >>> wrote: >>> Hi Jing, >>> I’m facing the same problem. Do you check the hadoop logs? In my case I >>> got: >>> Could not find or load main class >>> org.apache.giraph.yarn.GiraphApplicationMaster >>> Moreover I don’t know why my application hang for a long time here: >>> >>> 14/11/05 17:11:17 INFO yarn.YarnUtils: base path checking . >>> >>> And then continue failing with your error into the output and the other >>> into the hadoop logs. >>> >>> Did you test adding this? >>> >>> -yj >>> /Users/ale/extprj/giraph/giraph-core/target/giraph-1.1.0-SNAPSHOT-for-hadoop-2.5.1-jar-with-dependencies.jar >>> >>> Thanks, >>> Alessandro >>> >>> >>> Il giorno 05/nov/2014, alle ore 17:32, Jing Fan ha >>> scritto: >>> >>>> Hi, >>>> >>>> I install the Giraph with Yarn. >>>> But when I run the program with following command: >>>> >>>> hadoop jar >>>> giraph-examples/target/giraph-examples-1.1.0-SNAPSHOT-for-hadoop-2.5.0-cdh5.2.0-jar-with-dependencies.jar >>>> org.apache.giraph.GiraphRunner >>>> org.apache.giraph.examples.SimpleShortestPat
Re: Help with Giraph on Yarn
Hi, have you seen the hadoop logs? Could you write here how are you running the command? Thanks, Alessandro Il giorno 05/nov/2014, alle ore 18:04, Jing Fan ha scritto: > Yeah, I add -yj option. The result is the same. Still error here. > > On Wed, Nov 5, 2014 at 10:52 AM, Alessandro Negro wrote: > Hi Jing, > I’m facing the same problem. Do you check the hadoop logs? In my case I got: > Could not find or load main class > org.apache.giraph.yarn.GiraphApplicationMaster > Moreover I don’t know why my application hang for a long time here: > > 14/11/05 17:11:17 INFO yarn.YarnUtils: base path checking . > > And then continue failing with your error into the output and the other into > the hadoop logs. > > Did you test adding this? > > -yj > /Users/ale/extprj/giraph/giraph-core/target/giraph-1.1.0-SNAPSHOT-for-hadoop-2.5.1-jar-with-dependencies.jar > > Thanks, > Alessandro > > > Il giorno 05/nov/2014, alle ore 17:32, Jing Fan ha > scritto: > >> Hi, >> >> I install the Giraph with Yarn. >> But when I run the program with following command: >> >> hadoop jar >> giraph-examples/target/giraph-examples-1.1.0-SNAPSHOT-for-hadoop-2.5.0-cdh5.2.0-jar-with-dependencies.jar >> org.apache.giraph.GiraphRunner >> org.apache.giraph.examples.SimpleShortestPathsComputation -vif >> org.apache.giraph.io.formats.JsonLongDoubleFloatDoubleVertexInputFormat -vip >> /user/j/tiny_graph.txt -vof >> org.apache.giraph.io.formats.IdWithValueTextOutputFormat -op >> /user/j/shortestpaths -w 1 >> >> The program exited with error: >> >> 14/11/05 10:25:14 INFO yarn.GiraphYarnClient: Giraph: >> org.apache.giraph.examples.SimpleShortestPathsComputation, Elapsed: 0.83 secs >> 14/11/05 10:25:14 INFO yarn.GiraphYarnClient: >> appattempt_1415203231170_0002_01, State: ACCEPTED, Containers used: 0 >> 14/11/05 10:25:18 INFO yarn.GiraphYarnClient: Giraph: >> org.apache.giraph.examples.SimpleShortestPathsComputation, Elapsed: 4.84 secs >> 14/11/05 10:25:18 INFO yarn.GiraphYarnClient: >> appattempt_1415203231170_0002_01, State: ACCEPTED, Containers used: 0 >> 14/11/05 10:25:22 INFO yarn.GiraphYarnClient: Giraph: >> org.apache.giraph.examples.SimpleShortestPathsComputation, Elapsed: 8.85 secs >> 14/11/05 10:25:22 INFO yarn.GiraphYarnClient: >> appattempt_1415203231170_0002_02, State: ACCEPTED, Containers used: 0 >> 14/11/05 10:25:26 INFO yarn.GiraphYarnClient: Giraph: >> org.apache.giraph.examples.SimpleShortestPathsComputation, Elapsed: 12.86 >> secs >> 14/11/05 10:25:26 INFO yarn.GiraphYarnClient: >> appattempt_1415203231170_0002_02, State: ACCEPTED, Containers used: 0 >> 14/11/05 10:25:29 ERROR yarn.GiraphYarnClient: Giraph: >> org.apache.giraph.examples.SimpleShortestPathsComputation reports FAILED >> state, diagnostics show: Application application_1415203231170_0002 failed 2 >> times due to AM Container for appattempt_1415203231170_0002_02 exited >> with exitCode: 1 due to: Exception from container-launch. >> Container id: container_1415203231170_0002_02_01 >> Exit code: 1 >> Stack trace: ExitCodeException exitCode=1: >> at org.apache.hadoop.util.Shell.runCommand(Shell.java:538) >> at org.apache.hadoop.util.Shell.run(Shell.java:455) >> at >> org.apache.hadoop.util.Shell$ShellCommandExecutor.execute(Shell.java:702) >> at >> org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor.launchContainer(DefaultContainerExecutor.java:196) >> at >> org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:299) >> at >> org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:81) >> at java.util.concurrent.FutureTask.run(FutureTask.java:262) >> at >> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) >> at >> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) >> at java.lang.Thread.run(Thread.java:745) >> >> >> Anyone can help? >> > >
Re: Help with Giraph on Yarn
Hi Jing, I’m facing the same problem. Do you check the hadoop logs? In my case I got: Could not find or load main class org.apache.giraph.yarn.GiraphApplicationMaster Moreover I don’t know why my application hang for a long time here: 14/11/05 17:11:17 INFO yarn.YarnUtils: base path checking . And then continue failing with your error into the output and the other into the hadoop logs. Did you test adding this? -yj /Users/ale/extprj/giraph/giraph-core/target/giraph-1.1.0-SNAPSHOT-for-hadoop-2.5.1-jar-with-dependencies.jar Thanks, Alessandro Il giorno 05/nov/2014, alle ore 17:32, Jing Fan ha scritto: > Hi, > > I install the Giraph with Yarn. > But when I run the program with following command: > > hadoop jar > giraph-examples/target/giraph-examples-1.1.0-SNAPSHOT-for-hadoop-2.5.0-cdh5.2.0-jar-with-dependencies.jar > org.apache.giraph.GiraphRunner > org.apache.giraph.examples.SimpleShortestPathsComputation -vif > org.apache.giraph.io.formats.JsonLongDoubleFloatDoubleVertexInputFormat -vip > /user/j/tiny_graph.txt -vof > org.apache.giraph.io.formats.IdWithValueTextOutputFormat -op > /user/j/shortestpaths -w 1 > > The program exited with error: > > 14/11/05 10:25:14 INFO yarn.GiraphYarnClient: Giraph: > org.apache.giraph.examples.SimpleShortestPathsComputation, Elapsed: 0.83 secs > 14/11/05 10:25:14 INFO yarn.GiraphYarnClient: > appattempt_1415203231170_0002_01, State: ACCEPTED, Containers used: 0 > 14/11/05 10:25:18 INFO yarn.GiraphYarnClient: Giraph: > org.apache.giraph.examples.SimpleShortestPathsComputation, Elapsed: 4.84 secs > 14/11/05 10:25:18 INFO yarn.GiraphYarnClient: > appattempt_1415203231170_0002_01, State: ACCEPTED, Containers used: 0 > 14/11/05 10:25:22 INFO yarn.GiraphYarnClient: Giraph: > org.apache.giraph.examples.SimpleShortestPathsComputation, Elapsed: 8.85 secs > 14/11/05 10:25:22 INFO yarn.GiraphYarnClient: > appattempt_1415203231170_0002_02, State: ACCEPTED, Containers used: 0 > 14/11/05 10:25:26 INFO yarn.GiraphYarnClient: Giraph: > org.apache.giraph.examples.SimpleShortestPathsComputation, Elapsed: 12.86 secs > 14/11/05 10:25:26 INFO yarn.GiraphYarnClient: > appattempt_1415203231170_0002_02, State: ACCEPTED, Containers used: 0 > 14/11/05 10:25:29 ERROR yarn.GiraphYarnClient: Giraph: > org.apache.giraph.examples.SimpleShortestPathsComputation reports FAILED > state, diagnostics show: Application application_1415203231170_0002 failed 2 > times due to AM Container for appattempt_1415203231170_0002_02 exited > with exitCode: 1 due to: Exception from container-launch. > Container id: container_1415203231170_0002_02_01 > Exit code: 1 > Stack trace: ExitCodeException exitCode=1: > at org.apache.hadoop.util.Shell.runCommand(Shell.java:538) > at org.apache.hadoop.util.Shell.run(Shell.java:455) > at > org.apache.hadoop.util.Shell$ShellCommandExecutor.execute(Shell.java:702) > at > org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor.launchContainer(DefaultContainerExecutor.java:196) > at > org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:299) > at > org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:81) > at java.util.concurrent.FutureTask.run(FutureTask.java:262) > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) > at java.lang.Thread.run(Thread.java:745) > > > Anyone can help? >
Re: Compiling Giraph with Hadoop 2.5.1 & YARN
Hi Nicola,
how are you running the test? Could you help me to test it on yarn?
I would like to see the error.
Thanks,
Alessandro
Il giorno 27/ott/2014, alle ore 17:34, Nicola Vitucci
ha scritto:
> Hi Alessandro,
>
> I can now complete the compiling process and get the full .jar. I can't
> get it working right away, but I think it's more an Hadoop problem than
> a Giraph problem (can't get past "superstep -1" in the PageRank
> example). I'll keep you updated.
>
> Thanks again,
>
> Nicola
>
> Il 27/10/2014 17:15, Alessandro Negro ha scritto:
>> Hi Nicola,
>> I have just fixed this other issue, but I’m not sure that it will work
>> completely.
>> After I checked the code I notice that the issue is related to the munge
>> symbol STATIC_SASL_SYMBOL.
>> Try removing STATIC_SASL_SYMBOL from the munge symbols list in the
>> hadoop_yarn profile.
>>
>> My profile was:
>>
>> PURE_YARN,STATIC_SASL_SYMBOL
>>
>> now it is:
>>
>> PURE_YARN
>>
>>
>> Now I’m able to continue. As I said I’m not sure that this will produce
>> a working jar.
>> I checked into the code and it seems so. If you can please test and let
>> me know.
>>
>> Cheers,
>> Alessandro
>>
>>
>> Il giorno 27/ott/2014, alle ore 15:52, Nicola Vitucci
>> mailto:nicola.vitu...@gmail.com>> ha scritto:
>>
>>> Hi Alessandro,
>>>
>>> I can confirm that after your addition I'm able to build Giraph for
>>> Hadoop 2.4.0, so this works:
>>>
>>> mvn package -DskipTests -Dhadoop.version=2.4.0 -Phadoop_yarn clean install
>>>
>>> Do you know whether there are any plans to support Hadoop 2.5.x too?
>>>
>>> Thank you very much,
>>>
>>> Nicola
>>>
>>> Il 27/10/2014 15:38, Alessandro Negro ha scritto:
>>>> Hi Nicola,
>>>> it’s true I got the same error but also before changing the plugin
>>>> configuration.
>>>> I was able to compile only for version 2.4.0. I tested with 2.5.0 as
>>>> well.
>>>>
>>>> I tested everything on the main trunk.
>>>>
>>>> Cheers,
>>>> Alessandro
>>>>
>>>>
>>>>
>>>> Il giorno 27/ott/2014, alle ore 15:00, Nicola Vitucci
>>>> mailto:nicola.vitu...@gmail.com>> ha scritto:
>>>>
>>>>> Hi Alessandro,
>>>>>
>>>>> thanks for the reply. In order to start fresh, before adding the line
>>>>> you suggested I re-cloned the repository and tried to compile from trunk
>>>>> using the same command:
>>>>>
>>>>> mvn package -DskipTests -Dhadoop.version=2.5.1 -Phadoop_yarn clean
>>>>> install
>>>>>
>>>>> I get an error (on the variable SASL_PROPS) that I could get rid of
>>>>> before by applying the patch GIRAPH-947, but now the patch is not
>>>>> applicable any longer, so I can't even get to the stage where the
>>>>> 'munged' directory was the problem. Can you tell me which branch have
>>>>> you used and what is the exact sequence of changes that you applied to
>>>>> get Giraph compiled?
>>>>>
>>>>> Thanks,
>>>>>
>>>>> Nicola
>>>>>
>>>>> Il 26/10/2014 23:34, Alessandro Negro ha scritto:
>>>>>> Hi Nicola,
>>>>>> I was able to fix this issue adding the following line into the
>>>>>> munge maven plugin:
>>>>>>
>>>>>> ${project.build.directory}/munged
>>>>>>
>>>>>> Now it is:
>>>>>>
>>>>>>
>>>>>>org.sonatype.plugins
>>>>>>munge-maven-plugin
>>>>>>1.0
>>>>>>
>>>>>>
>>>>>>munge
>>>>>>generate-sources
>>>>>>
>>>>>> munge
>>>>>>
>>>>>>
>>>>>>
>>>>>> ${project.build.directory}/munged
>>>>>> ${munge.symbols}
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>> I hope that this will help you.
>>>>>>
>>>>>> Regards,
>>>>>> Alessandro
Re: Compiling Giraph with Hadoop 2.5.1 & YARN
Hi Nicola,
I have just fixed this other issue, but I’m not sure that it will work
completely.
After I checked the code I notice that the issue is related to the munge symbol
STATIC_SASL_SYMBOL.
Try removing STATIC_SASL_SYMBOL from the munge symbols list in the hadoop_yarn
profile.
My profile was:
PURE_YARN,STATIC_SASL_SYMBOL
now it is:
PURE_YARN
Now I’m able to continue. As I said I’m not sure that this will produce a
working jar.
I checked into the code and it seems so. If you can please test and let me know.
Cheers,
Alessandro
Il giorno 27/ott/2014, alle ore 15:52, Nicola Vitucci
ha scritto:
> Hi Alessandro,
>
> I can confirm that after your addition I'm able to build Giraph for
> Hadoop 2.4.0, so this works:
>
> mvn package -DskipTests -Dhadoop.version=2.4.0 -Phadoop_yarn clean install
>
> Do you know whether there are any plans to support Hadoop 2.5.x too?
>
> Thank you very much,
>
> Nicola
>
> Il 27/10/2014 15:38, Alessandro Negro ha scritto:
>> Hi Nicola,
>> it’s true I got the same error but also before changing the plugin
>> configuration.
>> I was able to compile only for version 2.4.0. I tested with 2.5.0 as well.
>>
>> I tested everything on the main trunk.
>>
>> Cheers,
>> Alessandro
>>
>>
>>
>> Il giorno 27/ott/2014, alle ore 15:00, Nicola Vitucci
>> ha scritto:
>>
>>> Hi Alessandro,
>>>
>>> thanks for the reply. In order to start fresh, before adding the line
>>> you suggested I re-cloned the repository and tried to compile from trunk
>>> using the same command:
>>>
>>> mvn package -DskipTests -Dhadoop.version=2.5.1 -Phadoop_yarn clean install
>>>
>>> I get an error (on the variable SASL_PROPS) that I could get rid of
>>> before by applying the patch GIRAPH-947, but now the patch is not
>>> applicable any longer, so I can't even get to the stage where the
>>> 'munged' directory was the problem. Can you tell me which branch have
>>> you used and what is the exact sequence of changes that you applied to
>>> get Giraph compiled?
>>>
>>> Thanks,
>>>
>>> Nicola
>>>
>>> Il 26/10/2014 23:34, Alessandro Negro ha scritto:
>>>> Hi Nicola,
>>>> I was able to fix this issue adding the following line into the munge
>>>> maven plugin:
>>>>
>>>> ${project.build.directory}/munged
>>>>
>>>> Now it is:
>>>>
>>>>
>>>> org.sonatype.plugins
>>>> munge-maven-plugin
>>>> 1.0
>>>>
>>>>
>>>> munge
>>>> generate-sources
>>>>
>>>> munge
>>>>
>>>>
>>>>
>>>> ${project.build.directory}/munged
>>>> ${munge.symbols}
>>>>
>>>>
>>>>
>>>>
>>>>
>>>> I hope that this will help you.
>>>>
>>>> Regards,
>>>> Alessandro
>>>>
>>
>>
Re: Compiling Giraph with Hadoop 2.5.1 & YARN
Hi Nicola,
it’s true I got the same error but also before changing the plugin
configuration.
I was able to compile only for version 2.4.0. I tested with 2.5.0 as well.
I tested everything on the main trunk.
Cheers,
Alessandro
Il giorno 27/ott/2014, alle ore 15:00, Nicola Vitucci
ha scritto:
> Hi Alessandro,
>
> thanks for the reply. In order to start fresh, before adding the line
> you suggested I re-cloned the repository and tried to compile from trunk
> using the same command:
>
> mvn package -DskipTests -Dhadoop.version=2.5.1 -Phadoop_yarn clean install
>
> I get an error (on the variable SASL_PROPS) that I could get rid of
> before by applying the patch GIRAPH-947, but now the patch is not
> applicable any longer, so I can't even get to the stage where the
> 'munged' directory was the problem. Can you tell me which branch have
> you used and what is the exact sequence of changes that you applied to
> get Giraph compiled?
>
> Thanks,
>
> Nicola
>
> Il 26/10/2014 23:34, Alessandro Negro ha scritto:
>> Hi Nicola,
>> I was able to fix this issue adding the following line into the munge maven
>> plugin:
>>
>> ${project.build.directory}/munged
>>
>> Now it is:
>>
>>
>> org.sonatype.plugins
>> munge-maven-plugin
>> 1.0
>>
>>
>> munge
>> generate-sources
>>
>>munge
>>
>>
>>
>> ${project.build.directory}/munged
>>${munge.symbols}
>>
>>
>>
>>
>>
>> I hope that this will help you.
>>
>> Regards,
>> Alessandro
>>
Re: Compiling Giraph with Hadoop 2.5.1 & YARN
Hi Nicola,
I was able to fix this issue adding the following line into the munge maven
plugin:
${project.build.directory}/munged
Now it is:
org.sonatype.plugins
munge-maven-plugin
1.0
munge
generate-sources
munge
${project.build.directory}/munged
${munge.symbols}
I hope that this will help you.
Regards,
Alessandro
