Giraph Use Case
Hi All, Although I have installed and ran Giraph example on my Hadoop Cluster referring to example below https://giraph.apache.org/quick_start.html its working great but I wanted to know what could be the other possible use case scenario/implementation of Giraph. Experts advice would be highly appreciated! Thanks!
ShortestPath Code execution on Hadoop 2.4.0 Inbox x
Hi everyone, I am new to Apache Giraph and would like execute Shortestpath and PageRank example code on *Hadoop 2.4.0 single node cluster* (my machine) having Centos 6.5. I have successfully build Giraph on my machine but unable to execute ShortestPath code. Please let me know if there are any dependencies to be resolved before code execution. *P.S. :* *Command used:* hadoop jar /usr/local/giraph/giraph-examples/target/giraph-examples-1.1.0-SNAPSHOT-for-hadoop-1.2.1-jar-with-dependencies.jar org.apache.giraph.GiraphRunner org.apache.giraph.examples.SimpleShortestPathsComputation -vif org.apache.giraph.io.formats.JsonLongDoubleFloatDoubleVertexInputFormat -vip /user/hadoop/input/tiny_graph.txt -vof org.apache.giraph.io.formats.IdWithValueTextOutputFormat -op /user/hduser/output/shortestpaths -w 1 -ca giraph.SplitMasterWorker=false *Execution Result:* 14/08/11 18:48:37 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable 14/08/11 18:48:40 INFO utils.ConfigurationUtils: No edge input format specified. Ensure your InputFormat does not require one. 14/08/11 18:48:40 INFO utils.ConfigurationUtils: No edge output format specified. Ensure your OutputFormat does not require one. 14/08/11 18:48:40 INFO utils.ConfigurationUtils: Setting custom argument [giraph.SplitMasterWorker] to [false] in GiraphConfiguration 14/08/11 18:48:40 INFO Configuration.deprecation: mapreduce.job.counters.limit is deprecated. Instead, use mapreduce.job.counters.max 14/08/11 18:48:40 INFO Configuration.deprecation: mapred.job.map.memory.mb is deprecated. Instead, use mapreduce.map.memory.mb 14/08/11 18:48:40 INFO Configuration.deprecation: mapred.job.reduce.memory.mb is deprecated. Instead, use mapreduce.reduce.memory.mb 14/08/11 18:48:40 INFO Configuration.deprecation: mapred.map.tasks.speculative.execution is deprecated. Instead, use mapreduce.map.speculative 14/08/11 18:48:40 INFO Configuration.deprecation: mapreduce.user.classpath.first is deprecated. Instead, use mapreduce.job.user.classpath.first 14/08/11 18:48:40 INFO Configuration.deprecation: mapred.map.max.attempts is deprecated. Instead, use mapreduce.map.maxattempts 14/08/11 18:48:40 INFO job.GiraphJob: run: Since checkpointing is disabled (default), do not allow any task retries (setting mapred.map.max.attempts = 0, old value = 4) 14/08/11 18:48:40 INFO Configuration.deprecation: mapred.job.tracker is deprecated. Instead, use mapreduce.jobtracker.address 14/08/11 18:48:41 INFO Configuration.deprecation: session.id is deprecated. Instead, use dfs.metrics.session-id 14/08/11 18:48:41 INFO jvm.JvmMetrics: Initializing JVM Metrics with processName=JobTracker, sessionId= Exception in thread main java.lang.IncompatibleClassChangeError: Found interface org.apache.hadoop.mapreduce.JobContext, but class was expected at org.apache.giraph.bsp.BspOutputFormat.checkOutputSpecs(BspOutputFormat.java:44) at org.apache.hadoop.mapreduce.JobSubmitter.checkSpecs(JobSubmitter.java:458) at org.apache.hadoop.mapreduce.JobSubmitter.submitJobInternal(JobSubmitter.java:343) at org.apache.hadoop.mapreduce.Job$10.run(Job.java:1285) at org.apache.hadoop.mapreduce.Job$10.run(Job.java:1282) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:422) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1548) at org.apache.hadoop.mapreduce.Job.submit(Job.java:1282) at org.apache.giraph.job.GiraphJob.run(GiraphJob.java:250) at org.apache.giraph.GiraphRunner.run(GiraphRunner.java:94) at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70) at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:84) at org.apache.giraph.GiraphRunner.main(GiraphRunner.java:124) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:483) at org.apache.hadoop.util.RunJar.main(RunJar.java:212) Regards, Vikalp Handa
Re: ShortestPath Code execution on Hadoop 2.4.0 Inbox x
Hi, You built Giraph for the Hadoop version 1.2.1 which is evident from your command line *hadoop jar /usr/local/giraph/giraph-examples/target/giraph-examples-1.1.0-SNAPSHOT-for-hadoop-1.2.1-jar-with-dependencies.jar * You have the build Giraph against the Hadoop version you'll be using. If you are using Hadoop 2.4.0, the command to build would be *mvn -Phadoop_2 -Dhadoop.version=2.4.0 -DskipTests ...* or *mvn -Phadoop_yarn -Dhadoop.version=2.4.0 -DskipTests ...* You can find this in the *README* file from Giraph trunk Hope this helps. On Tue, Aug 12, 2014 at 10:10 AM, Vikalp Handa handa.vik...@gmail.com wrote: Hi everyone, I am new to Apache Giraph and would like execute Shortestpath and PageRank example code on *Hadoop 2.4.0 single node cluster* (my machine) having Centos 6.5. I have successfully build Giraph on my machine but unable to execute ShortestPath code. Please let me know if there are any dependencies to be resolved before code execution. *P.S. :* *Command used:* hadoop jar /usr/local/giraph/giraph-examples/target/giraph-examples-1.1.0-SNAPSHOT-for-hadoop-1.2.1-jar-with-dependencies.jar org.apache.giraph.GiraphRunner org.apache.giraph.examples.SimpleShortestPathsComputation -vif org.apache.giraph.io.formats.JsonLongDoubleFloatDoubleVertexInputFormat -vip /user/hadoop/input/tiny_graph.txt -vof org.apache.giraph.io.formats.IdWithValueTextOutputFormat -op /user/hduser/output/shortestpaths -w 1 -ca giraph.SplitMasterWorker=false *Execution Result:* 14/08/11 18:48:37 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable 14/08/11 18:48:40 INFO utils.ConfigurationUtils: No edge input format specified. Ensure your InputFormat does not require one. 14/08/11 18:48:40 INFO utils.ConfigurationUtils: No edge output format specified. Ensure your OutputFormat does not require one. 14/08/11 18:48:40 INFO utils.ConfigurationUtils: Setting custom argument [giraph.SplitMasterWorker] to [false] in GiraphConfiguration 14/08/11 18:48:40 INFO Configuration.deprecation: mapreduce.job.counters.limit is deprecated. Instead, use mapreduce.job.counters.max 14/08/11 18:48:40 INFO Configuration.deprecation: mapred.job.map.memory.mb is deprecated. Instead, use mapreduce.map.memory.mb 14/08/11 18:48:40 INFO Configuration.deprecation: mapred.job.reduce.memory.mb is deprecated. Instead, use mapreduce.reduce.memory.mb 14/08/11 18:48:40 INFO Configuration.deprecation: mapred.map.tasks.speculative.execution is deprecated. Instead, use mapreduce.map.speculative 14/08/11 18:48:40 INFO Configuration.deprecation: mapreduce.user.classpath.first is deprecated. Instead, use mapreduce.job.user.classpath.first 14/08/11 18:48:40 INFO Configuration.deprecation: mapred.map.max.attempts is deprecated. Instead, use mapreduce.map.maxattempts 14/08/11 18:48:40 INFO job.GiraphJob: run: Since checkpointing is disabled (default), do not allow any task retries (setting mapred.map.max.attempts = 0, old value = 4) 14/08/11 18:48:40 INFO Configuration.deprecation: mapred.job.tracker is deprecated. Instead, use mapreduce.jobtracker.address 14/08/11 18:48:41 INFO Configuration.deprecation: session.id is deprecated. Instead, use dfs.metrics.session-id 14/08/11 18:48:41 INFO jvm.JvmMetrics: Initializing JVM Metrics with processName=JobTracker, sessionId= Exception in thread main java.lang.IncompatibleClassChangeError: Found interface org.apache.hadoop.mapreduce.JobContext, but class was expected at org.apache.giraph.bsp.BspOutputFormat.checkOutputSpecs(BspOutputFormat.java:44) at org.apache.hadoop.mapreduce.JobSubmitter.checkSpecs(JobSubmitter.java:458) at org.apache.hadoop.mapreduce.JobSubmitter.submitJobInternal(JobSubmitter.java:343) at org.apache.hadoop.mapreduce.Job$10.run(Job.java:1285) at org.apache.hadoop.mapreduce.Job$10.run(Job.java:1282) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:422) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1548) at org.apache.hadoop.mapreduce.Job.submit(Job.java:1282) at org.apache.giraph.job.GiraphJob.run(GiraphJob.java:250) at org.apache.giraph.GiraphRunner.run(GiraphRunner.java:94) at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70) at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:84) at org.apache.giraph.GiraphRunner.main(GiraphRunner.java:124) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:483) at org.apache.hadoop.util.RunJar.main(RunJar.java:212) Regards, Vikalp Handa
Re: Re: ShortestPath Code execution on Hadoop 2.4.0 Inbox x
You just run 'mvn clean' before that command :) Best Regards, Seong-il --- Original Message --- Sender : Vikalp Handahanda.vik...@gmail.com Date : 2014-08-12 14:35 (GMT+09:00) Title : Re: ShortestPath Code execution on Hadoop 2.4.0 Inbox x Thanks for your replies. Do I need to uninstall anything before running the above mvn command ? Regards, Vikalp Handa On Tue, Aug 12, 2014 at 10:26 AM, 이성일 si503@samsung.com wrote: Hi, Vikalp I think you complied as wrong hadoop version. Try below command If you use yarn mvn -Phadoop_yarn -Dhadoop.version=2.4.0 -DskipTests package or if you use just hadoop2 mvn -Phadoop_2 -Dhadoop.version=2.4.0 -DskipTests package Best Regards, Seong-il --- Original Message --- Sender : Vikalp Handahanda.vik...@gmail.com Date : 2014-08-12 13:40 (GMT+09:00) Title : ShortestPath Code execution on Hadoop 2.4.0 Inbox x Hi everyone, I am new to Apache Giraph and would like execute Shortestpath and PageRank example code on Hadoop 2.4.0 single node cluster (my machine) having Centos 6.5. I have successfully build Giraph on my machine but unable to execute ShortestPath code. Please let me know if there are any dependencies to be resolved before code execution. P.S. : Command used: hadoop jar /usr/local/giraph/giraph-examples/target/giraph-examples-1.1.0-SNAPSHOT-for-hadoop-1.2.1-jar-with-dependencies.jar org.apache.giraph.GiraphRunner org.apache.giraph.examples.SimpleShortestPathsComputation -vif org.apache.giraph.io.formats.JsonLongDoubleFloatDoubleVertexInputFormat -vip /user/hadoop/input/tiny_graph.txt -vof org.apache.giraph.io.formats.IdWithValueTextOutputFormat -op /user/hduser/output/shortestpaths -w 1 -ca giraph.SplitMasterWorker=false Execution Result: 14/08/11 18:48:37 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable 14/08/11 18:48:40 INFO utils.ConfigurationUtils: No edge input format specified. Ensure your InputFormat does not require one. 14/08/11 18:48:40 INFO utils.ConfigurationUtils: No edge output format specified. Ensure your OutputFormat does not require one. 14/08/11 18:48:40 INFO utils.ConfigurationUtils: Setting custom argument [giraph.SplitMasterWorker] to [false] in GiraphConfiguration 14/08/11 18:48:40 INFO Configuration.deprecation: mapreduce.job.counters.limit is deprecated. Instead, use mapreduce.job.counters.max 14/08/11 18:48:40 INFO Configuration.deprecation: mapred.job.map.memory.mb is deprecated. Instead, use mapreduce.map.memory.mb 14/08/11 18:48:40 INFO Configuration.deprecation: mapred.job.reduce.memory.mb is deprecated. Instead, use mapreduce.reduce.memory.mb 14/08/11 18:48:40 INFO Configuration.deprecation: mapred.map.tasks.speculative.execution is deprecated. Instead, use mapreduce.map.speculative 14/08/11 18:48:40 INFO Configuration.deprecation: mapreduce.user.classpath.first is deprecated. Instead, use mapreduce.job.user.classpath.first 14/08/11 18:48:40 INFO Configuration.deprecation: mapred.map.max.attempts is deprecated. Instead, use mapreduce.map.maxattempts 14/08/11 18:48:40 INFO job.GiraphJob: run: Since checkpointing is disabled (default), do not allow any task retries (setting mapred.map.max.attempts = 0, old value = 4) 14/08/11 18:48:40 INFO Configuration.deprecation: mapred.job.tracker is deprecated. Instead, use mapreduce.jobtracker.address 14/08/11 18:48:41 INFO Configuration.deprecation: session.id is deprecated. Instead, use dfs.metrics.session-id 14/08/11 18:48:41 INFO jvm.JvmMetrics: Initializing JVM Metrics with processName=JobTracker, sessionId= Exception in thread main java.lang.IncompatibleClassChangeError: Found interface org.apache.hadoop.mapreduce.JobContext, but class was expected at org.apache.giraph.bsp.BspOutputFormat.checkOutputSpecs(BspOutputFormat.java:44) at org.apache.hadoop.mapreduce.JobSubmitter.checkSpecs(JobSubmitter.java:458) at org.apache.hadoop.mapreduce.JobSubmitter.submitJobInternal(JobSubmitter.java:343) at org.apache.hadoop.mapreduce.Job$10.run(Job.java:1285) at org.apache.hadoop.mapreduce.Job$10.run(Job.java:1282) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:422) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1548) at org.apache.hadoop.mapreduce.Job.submit(Job.java:1282) at org.apache.giraph.job.GiraphJob.run(GiraphJob.java:250) at org.apache.giraph.GiraphRunner.run(GiraphRunner.java:94) at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70) at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:84) at org.apache.giraph.GiraphRunner.main(GiraphRunner.java:124) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at
