Re: ShortestPath Code execution on Hadoop 2.4.0 Inbox x
Hi Vikalp, finally I got to run Giraph successfully :-) Following steps: #Preparation * su * rm -rf /usr/local/giraph; mkdir /usr/local/giraph * chown hadoop:hadoop /usr/local/giraph * mkdir ~/setup; cd ~/setup #Sourcecode (Version 1.1.0) * git clone https://git-wip-us.apache.org/repos/asf/giraph.git * mv giraph giraph_src; cd giraph_src * mvn clean * mvn -Phadoop_yarn -Dhadoop.version=2.4.0 -DskipTests package #Installation (little ugly) * rm -rf myGiraph; mkdir -p myGiraph/lib; mkdir myGiraph/source * find . -name \*.jar -exec cp {} myGiraph/lib \; * cp -r conf myGiraph * cp -r bin myGiraph * cp CH* CO* RE* LI* NO* myGiraph * cd myGiraph * mv lib/*source*.jar source * mv lib/giraph*.jar . * rename 's/-SNAPSHOT-for//' *.jar * rm -rf ../myGiraph; cd ..; mv myGiraph .. * cd ../myGiraph; rm -rf /usr/local/giraph/*; cp -r * /usr/local/giraph Now you have to put a variable into hadoop-env.sh (at the end is ok): o /MY_LIBS=$HADOOP_HOME/share/myLib/*.jar/ o /export HADOOP_TASKTRACKER_OPTS=-classpath ${MY_LIBS} ${HADOOP_TASKTRACKER_OPTS}/ The folder myLib you have to create and put the Giraph-Jar: * mkdir /usr/local/hadoop/share/myLib * cp /usr/local/giraph/giraph-examples-1.1.0-hadoop-2.4.0-jar-with-dependencies.jar /usr/local/hadoop/share/myLib #Configuration Put property namemapred.map.tasks/name value4/value /property into mapred-site.xml * echo export PATH=\$PATH:/usr/local/giraph/bin ~/.bashrc * echo export GIRAPH_HOME=/usr/local/giraph ~/.bashrc * source ~/.bashrc Now you can test: #creating testdata * rm -rf ~/giraph; mkdir ~/giraph; cd ~/giraph * cat EOF tiny_graph.txt * [0,0,[[1,1],[3,3]]] [1,0,[[0,1],[2,2],[3,1]]] [2,0,[[1,2],[4,4]]] [3,0,[[0,3],[1,1],[4,4]]] [4,0,[[3,4],[2,4]]] * EOF #Run example-Jar * jar=giraph-examples-1.1.0-hadoop-2.4.0-jar-with-dependencies.jar * runner=org.apache.giraph.GiraphRunner * computation=org.apache.giraph.examples.SimpleShortestPathsComputation * informat=org.apache.giraph.io.formats.JsonLongDoubleFloatDoubleVertexInputFormat * outformat=org.apache.giraph.io.formats.IdWithValueTextOutputFormat * hdfs dfs -rm -r shortestpaths * yarn jar $GIRAPH_HOME/$jar $runner -Dgiraph.yarn.task.heap.mb=3000 $computation -vif $informat -vip tiny_graph.txt -vof $outformat -op shortestpaths -w 1 -yj $jar * hdfs dfs -cat shortestpaths/* Finished!! Hope it functions on your site. Best regards, Alexander On 12.08.2014 15:52, Vikalp Handa wrote: Hi Alexander, I looked into my gam-stderr.log file and found *Error: Could not find or load main class org.apache.giraph.yarn.GiraphApplicationMaster* As you have already mentioned in your reply about that for loop in *hadoop-env.sh. * So can you please tell me what is *mylib *and***share/myLib/giraph** ? *Because I am having only doc and hadoop directories inside $HADOOP_HOME/share/ Also how to get *giraph-1.1.0-hadoop-2.4.0.jar *and*giraph-examples-1.1.0-hadoop-*2.4.0.jar (without the dependencies) as I am having only giraph-examples-1.1.0-SNAPSHOT-for-hadoop-2.4.0-jar-with-dependencies.jar inside giraph-examples/target/munged/.
Re: ShortestPath Code execution on Hadoop 2.4.0 Inbox x
available cluster heap of 0MB. Aborting Job. at org.apache.giraph.yarn.GiraphYarnClient.checkPerNodeResourcesAvailable(GiraphYarnClient.java:230) at org.apache.giraph.yarn.GiraphYarnClient.run(GiraphYarnClient.java:124) at org.apache.giraph.GiraphRunner.run(GiraphRunner.java:96) at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70) at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:84) at org.apache.giraph.GiraphRunner.main(GiraphRunner.java:126) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:483) at org.apache.hadoop.util.RunJar.main(RunJar.java:212) Regards, Vikalp Handa On Tue, Aug 12, 2014 at 11:18 AM, 이성일 si503@samsung.com mailto:si503@samsung.com wrote: You just run 'mvn clean' before that command :) Best Regards, Seong-il --- Original Message --- Sender : Vikalp Handahanda.vik...@gmail.com mailto:handa.vik...@gmail.com Date : 2014-08-12 14:35 (GMT+09:00) Title : Re: ShortestPath Code execution on Hadoop 2.4.0 Inbox x Thanks for your replies. Do I need to uninstall anything before running the above mvn command ? Regards, Vikalp Handa On Tue, Aug 12, 2014 at 10:26 AM, 이성일 si503@samsung.com mailto:si503@samsung.com wrote: Hi, Vikalp I think you complied as wrong hadoop version. Try below command If you use yarn mvn -Phadoop_yarn -Dhadoop.version=2.4.0 -DskipTests package or if you use just hadoop2 mvn -Phadoop_2 -Dhadoop.version=2.4.0 -DskipTests package Best Regards, Seong-il --- Original Message --- Sender : Vikalp Handahanda.vik...@gmail.com mailto:handa.vik...@gmail.com Date : 2014-08-12 13:40 (GMT+09:00) Title : ShortestPath Code execution on Hadoop 2.4.0 Inbox x Hi everyone, I am new to Apache Giraph and would like execute Shortestpath and PageRank example code on Hadoop 2.4.0 single node cluster (my machine) having Centos 6.5. I have successfully build Giraph on my machine but unable to execute ShortestPath code. Please let me know if there are any dependencies to be resolved before code execution. P.S. : Command used: hadoop jar /usr/local/giraph/giraph-examples/target/giraph-examples-1.1.0-SNAPSHOT-for-hadoop-1.2.1-jar-with-dependencies.jar org.apache.giraph.GiraphRunner org.apache.giraph.examples.SimpleShortestPathsComputation -vif org.apache.giraph.io.formats.JsonLongDoubleFloatDoubleVertexInputFormat -vip /user/hadoop/input/tiny_graph.txt -vof org.apache.giraph.io.formats.IdWithValueTextOutputFormat -op /user/hduser/output/shortestpaths -w 1 -ca giraph.SplitMasterWorker=false Execution Result: 14/08/11 18:48:37 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable 14/08/11 18:48:40 INFO utils.ConfigurationUtils: No edge input format specified. Ensure your InputFormat does not require one. 14/08/11 18:48:40 INFO utils.ConfigurationUtils: No edge output format specified. Ensure your OutputFormat does not require one. 14/08/11 18:48:40 INFO utils.ConfigurationUtils: Setting custom argument [giraph.SplitMasterWorker] to [false] in GiraphConfiguration 14/08/11 18:48:40 INFO Configuration.deprecation: mapreduce.job.counters.limit is deprecated. Instead, use mapreduce.job.counters.max 14/08/11 18:48:40 INFO Configuration.deprecation: mapred.job.map.memory.mb is deprecated. Instead, use mapreduce.map.memory.mb 14/08/11 18:48:40 INFO Configuration.deprecation: mapred.job.reduce.memory.mb is deprecated. Instead, use mapreduce.reduce.memory.mb 14/08/11 18:48:40 INFO Configuration.deprecation: mapred.map.tasks.speculative.execution is deprecated. Instead, use mapreduce.map.speculative 14/08/11 18:48:40 INFO Configuration.deprecation: mapreduce.user.classpath.first is deprecated. Instead, use mapreduce.job.user.classpath.first 14/08/11 18:48:40 INFO Configuration.deprecation: mapred.map.max.attempts is deprecated. Instead, use mapreduce.map.maxattempts 14/08/11 18:48:40 INFO job.GiraphJob: run: Since checkpointing is disabled (default), do not allow any task retries (setting mapred.map.max.attempts = 0, old value = 4) 14/08/11 18:48:40 INFO Configuration.deprecation: mapred.job.tracker is deprecated. Instead, use mapreduce.jobtracker.address 14/08/11 18:48:41 INFO Configuration.deprecation: session.id http://session.id is deprecated. Instead, use dfs.metrics.session-id 14/08/11 18:48:41 INFO jvm.JvmMetrics: Initializing JVM
Re: ShortestPath Code execution on Hadoop 2.4.0 Inbox x
: Vikalp Handahanda.vik...@gmail.com Date : 2014-08-12 14:35 (GMT+09:00) Title : Re: ShortestPath Code execution on Hadoop 2.4.0 Inbox x Thanks for your replies. Do I need to uninstall anything before running the above mvn command ? Regards, Vikalp Handa On Tue, Aug 12, 2014 at 10:26 AM, 이성일 si503@samsung.com wrote: Hi, Vikalp I think you complied as wrong hadoop version. Try below command If you use yarn mvn -Phadoop_yarn -Dhadoop.version=2.4.0 -DskipTests package or if you use just hadoop2 mvn -Phadoop_2 -Dhadoop.version=2.4.0 -DskipTests package Best Regards, Seong-il --- Original Message --- Sender : Vikalp Handahanda.vik...@gmail.com Date : 2014-08-12 13:40 (GMT+09:00) Title : ShortestPath Code execution on Hadoop 2.4.0 Inbox x Hi everyone, I am new to Apache Giraph and would like execute Shortestpath and PageRank example code on Hadoop 2.4.0 single node cluster (my machine) having Centos 6.5. I have successfully build Giraph on my machine but unable to execute ShortestPath code. Please let me know if there are any dependencies to be resolved before code execution. P.S. : Command used: hadoop jar /usr/local/giraph/giraph-examples/target/giraph-examples-1.1.0-SNAPSHOT-for-hadoop-1.2.1-jar-with-dependencies.jar org.apache.giraph.GiraphRunner org.apache.giraph.examples.SimpleShortestPathsComputation -vif org.apache.giraph.io.formats.JsonLongDoubleFloatDoubleVertexInputFormat -vip /user/hadoop/input/tiny_graph.txt -vof org.apache.giraph.io.formats.IdWithValueTextOutputFormat -op /user/hduser/output/shortestpaths -w 1 -ca giraph.SplitMasterWorker=false Execution Result: 14/08/11 18:48:37 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable 14/08/11 18:48:40 INFO utils.ConfigurationUtils: No edge input format specified. Ensure your InputFormat does not require one. 14/08/11 18:48:40 INFO utils.ConfigurationUtils: No edge output format specified. Ensure your OutputFormat does not require one. 14/08/11 18:48:40 INFO utils.ConfigurationUtils: Setting custom argument [giraph.SplitMasterWorker] to [false] in GiraphConfiguration 14/08/11 18:48:40 INFO Configuration.deprecation: mapreduce.job.counters.limit is deprecated. Instead, use mapreduce.job.counters.max 14/08/11 18:48:40 INFO Configuration.deprecation: mapred.job.map.memory.mb is deprecated. Instead, use mapreduce.map.memory.mb 14/08/11 18:48:40 INFO Configuration.deprecation: mapred.job.reduce.memory.mb is deprecated. Instead, use mapreduce.reduce.memory.mb 14/08/11 18:48:40 INFO Configuration.deprecation: mapred.map.tasks.speculative.execution is deprecated. Instead, use mapreduce.map.speculative 14/08/11 18:48:40 INFO Configuration.deprecation: mapreduce.user.classpath.first is deprecated. Instead, use mapreduce.job.user.classpath.first 14/08/11 18:48:40 INFO Configuration.deprecation: mapred.map.max.attempts is deprecated. Instead, use mapreduce.map.maxattempts 14/08/11 18:48:40 INFO job.GiraphJob: run: Since checkpointing is disabled (default), do not allow any task retries (setting mapred.map.max.attempts = 0, old value = 4) 14/08/11 18:48:40 INFO Configuration.deprecation: mapred.job.tracker is deprecated. Instead, use mapreduce.jobtracker.address 14/08/11 18:48:41 INFO Configuration.deprecation: session.id is deprecated. Instead, use dfs.metrics.session-id 14/08/11 18:48:41 INFO jvm.JvmMetrics: Initializing JVM Metrics with processName=JobTracker, sessionId= Exception in thread main java.lang.IncompatibleClassChangeError: Found interface org.apache.hadoop.mapreduce.JobContext, but class was expected at org.apache.giraph.bsp.BspOutputFormat.checkOutputSpecs(BspOutputFormat.java:44) at org.apache.hadoop.mapreduce.JobSubmitter.checkSpecs(JobSubmitter.java:458) at org.apache.hadoop.mapreduce.JobSubmitter.submitJobInternal(JobSubmitter.java:343) at org.apache.hadoop.mapreduce.Job$10.run(Job.java:1285) at org.apache.hadoop.mapreduce.Job$10.run(Job.java:1282) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:422) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1548) at org.apache.hadoop.mapreduce.Job.submit(Job.java:1282) at org.apache.giraph.job.GiraphJob.run(GiraphJob.java:250) at org.apache.giraph.GiraphRunner.run(GiraphRunner.java:94) at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70) at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:84) at org.apache.giraph.GiraphRunner.main(GiraphRunner.java:124) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke
Re: ShortestPath Code execution on Hadoop 2.4.0 Inbox x
Hello Vikalp, you are showing the output of the client. The errors I ve showed were coming from the ResourceManager, the GiraphApplicationMaster (gam-stderr.log) and the GiraphYarnTask (TaskRunner). Checkout these logfiles on your systems. In case you get an error-message class GiraphApplicationMaster not found: I solved it by adding a for-loop in hadoop-env.sh: for f in `find $HADOOP_HOME/share/myLib/ -name \*.jar`; do if [ $HADOOP_CLASSPATH ]; then export HADOOP_CLASSPATH=$HADOOP_CLASSPATH:$f else export HADOOP_CLASSPATH=$f fi done In share/myLib/giraph I have put giraph-1.1.0-hadoop-2.4.0.jar and giraph-examples-1.1.0-hadoop-2.4.0.jar (without the dependencies). Additionally I copied every jar from /usr/local/giraph/lib also to this folder. Then you can restart the cluster and every instance should know giraph. After this it should function, if you have enough memory. I get then a message, that the GiraphApplicationMaster dont have enough memory and then get killed. But before this, it has started the TaskRunner, which keeps the whole job endlessly alive. Sincerely, On 12.08.2014 11:55, Vikalp Handa wrote: @Alexander Sirotin : Thanks for your reply. I am really sorry I haven't faced this problem after I executed it. Rather I am now having a different issue with Containers : * * *Result : * 14/08/12 15:17:51 INFO yarn.GiraphYarnClient: ApplicationSumbissionContext for GiraphApplicationMaster launch container is populated. 14/08/12 15:17:51 INFO yarn.GiraphYarnClient: Submitting application to ASM 14/08/12 15:17:52 INFO impl.YarnClientImpl: Submitted application application_1407836750214_0001 14/08/12 15:17:52 INFO yarn.GiraphYarnClient: Got new appId after submission :application_1407836750214_0001 14/08/12 15:17:52 INFO yarn.GiraphYarnClient: GiraphApplicationMaster container request was submitted to ResourceManager for job: Giraph: org.apache.giraph.examples.SimpleShortestPathsComputation 14/08/12 15:17:52 INFO yarn.GiraphYarnClient: Giraph: org.apache.giraph.examples.SimpleShortestPathsComputation, Elapsed: 0.99 secs 14/08/12 15:17:52 INFO yarn.GiraphYarnClient: appattempt_1407836750214_0001_01, State: ACCEPTED, Containers used: 1 14/08/12 15:17:56 INFO yarn.GiraphYarnClient: Giraph: org.apache.giraph.examples.SimpleShortestPathsComputation, Elapsed: 5.01 secs 14/08/12 15:17:56 INFO yarn.GiraphYarnClient: appattempt_1407836750214_0001_02, State: ACCEPTED, Containers used: 0 14/08/12 15:18:00*ERROR yarn.GiraphYarnClient: Giraph: org.apache.giraph.examples.SimpleShortestPathsComputation reports FAILED state, diagnostics show: Application application_1407836750214_0001 failed 2 times due to AM Container for appattempt_1407836750214_0001_02 exited with exitCode: 1 due to: Exception from container-launch: org.apache.hadoop.util.Shell$ExitCodeException: * *org.apache.hadoop.util.Shell$ExitCodeException:* at org.apache.hadoop.util.Shell.runCommand(Shell.java:505) at org.apache.hadoop.util.Shell.run(Shell.java:418) at org.apache.hadoop.util.Shell$ShellCommandExecutor.execute(Shell.java:650) at org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor.launchContainer(DefaultContainerExecutor.java:195) at org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:300) at org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:81) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) Container exited with a non-zero exit code 1 .Failing this attempt.. Failing the application. 14/08/12 15:18:00 INFO yarn.GiraphYarnClient: Cleaning up HDFS distributed cache directory for Giraph job. 14/08/12 15:18:00 INFO yarn.GiraphYarnClient: Completed Giraph: org.apache.giraph.examples.SimpleShortestPathsComputation: FAILED, total running time: 0 minutes, 7 seconds. I have also checked my *yarn-site.xml* file and updated with the following property - value pairs inside configuration: |property||| |name||yarn.nodemanager.aux-services.mapreduce.shuffle.class/||name||| |value||org.apache.hadoop.mapred.ShuffleHandler/||value||| |||/||property||| |property||| |name||yarn.application.classpath/||name||| |value||| |||%HADOOP_HOME%\etc\hadoop,| |||%HADOOP_HOME%\share\hadoop\common\*,| |||%HADOOP_HOME%\share\hadoop\common\lib\*,| |||%HADOOP_HOME%\share\hadoop\hdfs\*,| |||%HADOOP_HOME%\share\hadoop\hdfs\lib\*,| |||%HADOOP_HOME%\share\hadoop\mapreduce\*,| |||%HADOOP_HOME%\share\hadoop\mapreduce\lib\*,| |||%HADOOP_HOME%\share\hadoop\yarn\*,| |||%HADOOP_HOME%\share\hadoop\yarn\lib\*| |||/||value||| |||/||property||| *| |* Regards, Vikalp Handa
Re: ShortestPath Code execution on Hadoop 2.4.0 Inbox x
Hi Alexander, I looked into my gam-stderr.log file and found *Error: Could not find or load main class org.apache.giraph.yarn.GiraphApplicationMaster* As you have already mentioned in your reply about that for loop in *hadoop-env.sh. * So can you please tell me what is *mylib *and *share/myLib/giraph** ? *Because I am having only doc and hadoop directories inside $HADOOP_HOME/share/ Also how to get *giraph-1.1.0-hadoop-2.4.0.jar *and* giraph-examples-1.1.0-hadoop-*2.4.0.jar (without the dependencies) as I am having only giraph-examples-1.1.0-SNAPSHOT-for-hadoop-2.4.0-jar-with-dependencies.jar inside giraph-examples/target/munged/. Regards, Vikalp Handa On Tue, Aug 12, 2014 at 4:25 PM, Alexander Sirotin sirot...@web.de wrote: Hello Vikalp, you are showing the output of the client. The errors I ve showed were coming from the ResourceManager, the GiraphApplicationMaster (gam-stderr.log) and the GiraphYarnTask (TaskRunner). Checkout these logfiles on your systems. In case you get an error-message class GiraphApplicationMaster not found: I solved it by adding a for-loop in hadoop-env.sh: for f in `find $HADOOP_HOME/share/myLib/ -name \*.jar`; do if [ $HADOOP_CLASSPATH ]; then export HADOOP_CLASSPATH=$HADOOP_CLASSPATH:$f else export HADOOP_CLASSPATH=$f fi done In share/myLib/giraph I have put giraph-1.1.0-hadoop-2.4.0.jar and giraph-examples-1.1.0-hadoop-2.4.0.jar (without the dependencies). Additionally I copied every jar from /usr/local/giraph/lib also to this folder. Then you can restart the cluster and every instance should know giraph. After this it should function, if you have enough memory. I get then a message, that the GiraphApplicationMaster dont have enough memory and then get killed. But before this, it has started the TaskRunner, which keeps the whole job endlessly alive. Sincerely, On 12.08.2014 11:55, Vikalp Handa wrote: @Alexander Sirotin : Thanks for your reply. I am really sorry I haven't faced this problem after I executed it. Rather I am now having a different issue with Containers : *Result : * 14/08/12 15:17:51 INFO yarn.GiraphYarnClient: ApplicationSumbissionContext for GiraphApplicationMaster launch container is populated. 14/08/12 15:17:51 INFO yarn.GiraphYarnClient: Submitting application to ASM 14/08/12 15:17:52 INFO impl.YarnClientImpl: Submitted application application_1407836750214_0001 14/08/12 15:17:52 INFO yarn.GiraphYarnClient: Got new appId after submission :application_1407836750214_0001 14/08/12 15:17:52 INFO yarn.GiraphYarnClient: GiraphApplicationMaster container request was submitted to ResourceManager for job: Giraph: org.apache.giraph.examples.SimpleShortestPathsComputation 14/08/12 15:17:52 INFO yarn.GiraphYarnClient: Giraph: org.apache.giraph.examples.SimpleShortestPathsComputation, Elapsed: 0.99 secs 14/08/12 15:17:52 INFO yarn.GiraphYarnClient: appattempt_1407836750214_0001_01, State: ACCEPTED, Containers used: 1 14/08/12 15:17:56 INFO yarn.GiraphYarnClient: Giraph: org.apache.giraph.examples.SimpleShortestPathsComputation, Elapsed: 5.01 secs 14/08/12 15:17:56 INFO yarn.GiraphYarnClient: appattempt_1407836750214_0001_02, State: ACCEPTED, Containers used: 0 14/08/12 15:18:00* ERROR yarn.GiraphYarnClient: Giraph: org.apache.giraph.examples.SimpleShortestPathsComputation reports FAILED state, diagnostics show: Application application_1407836750214_0001 failed 2 times due to AM Container for appattempt_1407836750214_0001_02 exited with exitCode: 1 due to: Exception from container-launch: org.apache.hadoop.util.Shell$ExitCodeException: * *org.apache.hadoop.util.Shell$ExitCodeException:* at org.apache.hadoop.util.Shell.runCommand(Shell.java:505) at org.apache.hadoop.util.Shell.run(Shell.java:418) at org.apache.hadoop.util.Shell$ShellCommandExecutor.execute(Shell.java:650) at org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor.launchContainer(DefaultContainerExecutor.java:195) at org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:300) at org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:81) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) Container exited with a non-zero exit code 1 .Failing this attempt.. Failing the application. 14/08/12 15:18:00 INFO yarn.GiraphYarnClient: Cleaning up HDFS distributed cache directory for Giraph job. 14/08/12 15:18:00 INFO yarn.GiraphYarnClient: Completed Giraph: org.apache.giraph.examples.SimpleShortestPathsComputation: FAILED, total running time: 0 minutes, 7 seconds. I have also checked my *yarn-site.xml* file and updated with the following
Re: ShortestPath Code execution on Hadoop 2.4.0 Inbox x
Hi Vikalp, in $HADOOP_HOME/share there you will find Jars. But also Hadoop itself uses this place. Important is also there the subdirectory hadoop/mapreduce, where Jars will be used for TaskRunner and ApplicationMasters. If you would put your giraph-Jar there, the ApplicationMaster should find it, but the bad thing is, it is not your directory ;-) and 2nd in the Jar with dependencies, there are Classes included, which has Hadoop already and this caused errors on my systems. Therefore I created myLib by my own and put only additional Jars into it. In hadoop-env.sh you can search for a loop-command 'for f in ' This loop I just copied and modified for my own usage. I am sorry I am not an expert, but I do not want to waste so much time on it and made this workaround :-P If I start to run my system successfully ( I also using Hadoop 2.4 with Giraph 1.1.0), then I will let you know how. Sincerely On 12.08.2014 15:52, Vikalp Handa wrote: Hi Alexander, I looked into my gam-stderr.log file and found *Error: Could not find or load main class org.apache.giraph.yarn.GiraphApplicationMaster* As you have already mentioned in your reply about that for loop in *hadoop-env.sh. * So can you please tell me what is *mylib *and***share/myLib/giraph** ? *Because I am having only doc and hadoop directories inside $HADOOP_HOME/share/ Also how to get *giraph-1.1.0-hadoop-2.4.0.jar *and*giraph-examples-1.1.0-hadoop-*2.4.0.jar (without the dependencies) as I am having only giraph-examples-1.1.0-SNAPSHOT-for-hadoop-2.4.0-jar-with-dependencies.jar inside giraph-examples/target/munged/. Regards, Vikalp Handa On Tue, Aug 12, 2014 at 4:25 PM, Alexander Sirotin sirot...@web.de mailto:sirot...@web.de wrote: Hello Vikalp, you are showing the output of the client. The errors I ve showed were coming from the ResourceManager, the GiraphApplicationMaster (gam-stderr.log) and the GiraphYarnTask (TaskRunner). Checkout these logfiles on your systems. In case you get an error-message class GiraphApplicationMaster not found: I solved it by adding a for-loop in hadoop-env.sh: for f in `find $HADOOP_HOME/share/myLib/ -name \*.jar`; do if [ $HADOOP_CLASSPATH ]; then export HADOOP_CLASSPATH=$HADOOP_CLASSPATH:$f else export HADOOP_CLASSPATH=$f fi done In share/myLib/giraph I have put giraph-1.1.0-hadoop-2.4.0.jar and giraph-examples-1.1.0-hadoop-2.4.0.jar (without the dependencies). Additionally I copied every jar from /usr/local/giraph/lib also to this folder. Then you can restart the cluster and every instance should know giraph. After this it should function, if you have enough memory. I get then a message, that the GiraphApplicationMaster dont have enough memory and then get killed. But before this, it has started the TaskRunner, which keeps the whole job endlessly alive. Sincerely, On 12.08.2014 11:55, Vikalp Handa wrote: @Alexander Sirotin : Thanks for your reply. I am really sorry I haven't faced this problem after I executed it. Rather I am now having a different issue with Containers : * * *Result : * 14/08/12 15:17:51 INFO yarn.GiraphYarnClient: ApplicationSumbissionContext for GiraphApplicationMaster launch container is populated. 14/08/12 15:17:51 INFO yarn.GiraphYarnClient: Submitting application to ASM 14/08/12 15:17:52 INFO impl.YarnClientImpl: Submitted application application_1407836750214_0001 14/08/12 15:17:52 INFO yarn.GiraphYarnClient: Got new appId after submission :application_1407836750214_0001 14/08/12 15:17:52 INFO yarn.GiraphYarnClient: GiraphApplicationMaster container request was submitted to ResourceManager for job: Giraph: org.apache.giraph.examples.SimpleShortestPathsComputation 14/08/12 15:17:52 INFO yarn.GiraphYarnClient: Giraph: org.apache.giraph.examples.SimpleShortestPathsComputation, Elapsed: 0.99 secs 14/08/12 15:17:52 INFO yarn.GiraphYarnClient: appattempt_1407836750214_0001_01, State: ACCEPTED, Containers used: 1 14/08/12 15:17:56 INFO yarn.GiraphYarnClient: Giraph: org.apache.giraph.examples.SimpleShortestPathsComputation, Elapsed: 5.01 secs 14/08/12 15:17:56 INFO yarn.GiraphYarnClient: appattempt_1407836750214_0001_02, State: ACCEPTED, Containers used: 0 14/08/12 15:18:00*ERROR yarn.GiraphYarnClient: Giraph: org.apache.giraph.examples.SimpleShortestPathsComputation reports FAILED state, diagnostics show: Application application_1407836750214_0001 failed 2 times due to AM Container for appattempt_1407836750214_0001_02 exited with exitCode: 1 due to: Exception from container-launch: org.apache.hadoop.util.Shell$ExitCodeException: * *org.apache.hadoop.util.Shell$ExitCodeException:* at org.apache.hadoop.util.Shell.runCommand(Shell.java:505) at
Re: ShortestPath Code execution on Hadoop 2.4.0 Inbox x
Hi, You built Giraph for the Hadoop version 1.2.1 which is evident from your command line *hadoop jar /usr/local/giraph/giraph-examples/target/giraph-examples-1.1.0-SNAPSHOT-for-hadoop-1.2.1-jar-with-dependencies.jar * You have the build Giraph against the Hadoop version you'll be using. If you are using Hadoop 2.4.0, the command to build would be *mvn -Phadoop_2 -Dhadoop.version=2.4.0 -DskipTests ...* or *mvn -Phadoop_yarn -Dhadoop.version=2.4.0 -DskipTests ...* You can find this in the *README* file from Giraph trunk Hope this helps. On Tue, Aug 12, 2014 at 10:10 AM, Vikalp Handa handa.vik...@gmail.com wrote: Hi everyone, I am new to Apache Giraph and would like execute Shortestpath and PageRank example code on *Hadoop 2.4.0 single node cluster* (my machine) having Centos 6.5. I have successfully build Giraph on my machine but unable to execute ShortestPath code. Please let me know if there are any dependencies to be resolved before code execution. *P.S. :* *Command used:* hadoop jar /usr/local/giraph/giraph-examples/target/giraph-examples-1.1.0-SNAPSHOT-for-hadoop-1.2.1-jar-with-dependencies.jar org.apache.giraph.GiraphRunner org.apache.giraph.examples.SimpleShortestPathsComputation -vif org.apache.giraph.io.formats.JsonLongDoubleFloatDoubleVertexInputFormat -vip /user/hadoop/input/tiny_graph.txt -vof org.apache.giraph.io.formats.IdWithValueTextOutputFormat -op /user/hduser/output/shortestpaths -w 1 -ca giraph.SplitMasterWorker=false *Execution Result:* 14/08/11 18:48:37 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable 14/08/11 18:48:40 INFO utils.ConfigurationUtils: No edge input format specified. Ensure your InputFormat does not require one. 14/08/11 18:48:40 INFO utils.ConfigurationUtils: No edge output format specified. Ensure your OutputFormat does not require one. 14/08/11 18:48:40 INFO utils.ConfigurationUtils: Setting custom argument [giraph.SplitMasterWorker] to [false] in GiraphConfiguration 14/08/11 18:48:40 INFO Configuration.deprecation: mapreduce.job.counters.limit is deprecated. Instead, use mapreduce.job.counters.max 14/08/11 18:48:40 INFO Configuration.deprecation: mapred.job.map.memory.mb is deprecated. Instead, use mapreduce.map.memory.mb 14/08/11 18:48:40 INFO Configuration.deprecation: mapred.job.reduce.memory.mb is deprecated. Instead, use mapreduce.reduce.memory.mb 14/08/11 18:48:40 INFO Configuration.deprecation: mapred.map.tasks.speculative.execution is deprecated. Instead, use mapreduce.map.speculative 14/08/11 18:48:40 INFO Configuration.deprecation: mapreduce.user.classpath.first is deprecated. Instead, use mapreduce.job.user.classpath.first 14/08/11 18:48:40 INFO Configuration.deprecation: mapred.map.max.attempts is deprecated. Instead, use mapreduce.map.maxattempts 14/08/11 18:48:40 INFO job.GiraphJob: run: Since checkpointing is disabled (default), do not allow any task retries (setting mapred.map.max.attempts = 0, old value = 4) 14/08/11 18:48:40 INFO Configuration.deprecation: mapred.job.tracker is deprecated. Instead, use mapreduce.jobtracker.address 14/08/11 18:48:41 INFO Configuration.deprecation: session.id is deprecated. Instead, use dfs.metrics.session-id 14/08/11 18:48:41 INFO jvm.JvmMetrics: Initializing JVM Metrics with processName=JobTracker, sessionId= Exception in thread main java.lang.IncompatibleClassChangeError: Found interface org.apache.hadoop.mapreduce.JobContext, but class was expected at org.apache.giraph.bsp.BspOutputFormat.checkOutputSpecs(BspOutputFormat.java:44) at org.apache.hadoop.mapreduce.JobSubmitter.checkSpecs(JobSubmitter.java:458) at org.apache.hadoop.mapreduce.JobSubmitter.submitJobInternal(JobSubmitter.java:343) at org.apache.hadoop.mapreduce.Job$10.run(Job.java:1285) at org.apache.hadoop.mapreduce.Job$10.run(Job.java:1282) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:422) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1548) at org.apache.hadoop.mapreduce.Job.submit(Job.java:1282) at org.apache.giraph.job.GiraphJob.run(GiraphJob.java:250) at org.apache.giraph.GiraphRunner.run(GiraphRunner.java:94) at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70) at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:84) at org.apache.giraph.GiraphRunner.main(GiraphRunner.java:124) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:483) at org.apache.hadoop.util.RunJar.main(RunJar.java:212) Regards, Vikalp Handa
Re: Re: ShortestPath Code execution on Hadoop 2.4.0 Inbox x
You just run 'mvn clean' before that command :) Best Regards, Seong-il --- Original Message --- Sender : Vikalp Handahanda.vik...@gmail.com Date : 2014-08-12 14:35 (GMT+09:00) Title : Re: ShortestPath Code execution on Hadoop 2.4.0 Inbox x Thanks for your replies. Do I need to uninstall anything before running the above mvn command ? Regards, Vikalp Handa On Tue, Aug 12, 2014 at 10:26 AM, 이성일 si503@samsung.com wrote: Hi, Vikalp I think you complied as wrong hadoop version. Try below command If you use yarn mvn -Phadoop_yarn -Dhadoop.version=2.4.0 -DskipTests package or if you use just hadoop2 mvn -Phadoop_2 -Dhadoop.version=2.4.0 -DskipTests package Best Regards, Seong-il --- Original Message --- Sender : Vikalp Handahanda.vik...@gmail.com Date : 2014-08-12 13:40 (GMT+09:00) Title : ShortestPath Code execution on Hadoop 2.4.0 Inbox x Hi everyone, I am new to Apache Giraph and would like execute Shortestpath and PageRank example code on Hadoop 2.4.0 single node cluster (my machine) having Centos 6.5. I have successfully build Giraph on my machine but unable to execute ShortestPath code. Please let me know if there are any dependencies to be resolved before code execution. P.S. : Command used: hadoop jar /usr/local/giraph/giraph-examples/target/giraph-examples-1.1.0-SNAPSHOT-for-hadoop-1.2.1-jar-with-dependencies.jar org.apache.giraph.GiraphRunner org.apache.giraph.examples.SimpleShortestPathsComputation -vif org.apache.giraph.io.formats.JsonLongDoubleFloatDoubleVertexInputFormat -vip /user/hadoop/input/tiny_graph.txt -vof org.apache.giraph.io.formats.IdWithValueTextOutputFormat -op /user/hduser/output/shortestpaths -w 1 -ca giraph.SplitMasterWorker=false Execution Result: 14/08/11 18:48:37 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable 14/08/11 18:48:40 INFO utils.ConfigurationUtils: No edge input format specified. Ensure your InputFormat does not require one. 14/08/11 18:48:40 INFO utils.ConfigurationUtils: No edge output format specified. Ensure your OutputFormat does not require one. 14/08/11 18:48:40 INFO utils.ConfigurationUtils: Setting custom argument [giraph.SplitMasterWorker] to [false] in GiraphConfiguration 14/08/11 18:48:40 INFO Configuration.deprecation: mapreduce.job.counters.limit is deprecated. Instead, use mapreduce.job.counters.max 14/08/11 18:48:40 INFO Configuration.deprecation: mapred.job.map.memory.mb is deprecated. Instead, use mapreduce.map.memory.mb 14/08/11 18:48:40 INFO Configuration.deprecation: mapred.job.reduce.memory.mb is deprecated. Instead, use mapreduce.reduce.memory.mb 14/08/11 18:48:40 INFO Configuration.deprecation: mapred.map.tasks.speculative.execution is deprecated. Instead, use mapreduce.map.speculative 14/08/11 18:48:40 INFO Configuration.deprecation: mapreduce.user.classpath.first is deprecated. Instead, use mapreduce.job.user.classpath.first 14/08/11 18:48:40 INFO Configuration.deprecation: mapred.map.max.attempts is deprecated. Instead, use mapreduce.map.maxattempts 14/08/11 18:48:40 INFO job.GiraphJob: run: Since checkpointing is disabled (default), do not allow any task retries (setting mapred.map.max.attempts = 0, old value = 4) 14/08/11 18:48:40 INFO Configuration.deprecation: mapred.job.tracker is deprecated. Instead, use mapreduce.jobtracker.address 14/08/11 18:48:41 INFO Configuration.deprecation: session.id is deprecated. Instead, use dfs.metrics.session-id 14/08/11 18:48:41 INFO jvm.JvmMetrics: Initializing JVM Metrics with processName=JobTracker, sessionId= Exception in thread main java.lang.IncompatibleClassChangeError: Found interface org.apache.hadoop.mapreduce.JobContext, but class was expected at org.apache.giraph.bsp.BspOutputFormat.checkOutputSpecs(BspOutputFormat.java:44) at org.apache.hadoop.mapreduce.JobSubmitter.checkSpecs(JobSubmitter.java:458) at org.apache.hadoop.mapreduce.JobSubmitter.submitJobInternal(JobSubmitter.java:343) at org.apache.hadoop.mapreduce.Job$10.run(Job.java:1285) at org.apache.hadoop.mapreduce.Job$10.run(Job.java:1282) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:422) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1548) at org.apache.hadoop.mapreduce.Job.submit(Job.java:1282) at org.apache.giraph.job.GiraphJob.run(GiraphJob.java:250) at org.apache.giraph.GiraphRunner.run(GiraphRunner.java:94) at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70) at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:84) at org.apache.giraph.GiraphRunner.main(GiraphRunner.java:124) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62