Giraph job doesn't work on Oozie workflow
Hi, dears
Now, I'm trying to run giraph job on Apache Oozie workflow.
But, it's not finish and running eternally.
Is there any one like me?
I'm attaching my oozie files, hadoop log message and s/w version
Thanks,
Seong-il
- Hadoop : 2.2.0
- Giraph : release-1.1.0-RC0 (compiled as mvn -Phadoop_2
-Dhadoop.version=2.2.0 -DskipTests package)
[[job.properties]]
nameNode=hdfs://localhost:9000
jobTracker=localhost:8032
queueName=default
appName=giraph-shortestpath
appPath=${nameNode}/user/${user.name}/${appName}
oozie.wf.application.path=${appPath}
oozie.libpath=${appPath}/lib
[[workflow.xml]]
workflow-app name=giraph xmlns=uri:oozie:workflow:0.4
start to=shortestpath/
action name=shortestpath cred=
java
job-tracker${jobTracker}/job-tracker
name-node${nameNode}/name-node
main-classorg.apache.giraph.GiraphRunner/main-class
argcom.example.ShortestPath/arg
arg-vif/arg
argorg.apache.giraph.io.formats.JsonLongDoubleFloatDoubleVertexInputFormat/arg
arg-vip/arg
arginput/tiny_graph.txt/arg
arg-vof/arg
argorg.apache.giraph.io.formats.IdWithValueTextOutputFormat/arg
arg-op/arg
argoutput/shortestpath/arg
arg-w/arg
arg1/arg
/java
ok to=end/
error to=kill/
/action
kill name=kill
messageAction failed, error message/message
/kill
end name=end/
/workflow-app
[[hadoop log message]]
2014-10-23 16:36:34,426 INFO [main]
org.apache.hadoop.mapreduce.v2.app.MRAppMaster: Created MRAppMaster for
application appattempt_1414046872824_0007_01
2014-10-23 16:36:34,656 WARN [main] org.apache.hadoop.conf.Configuration:
job.xml:an attempt to override final parameter:
mapreduce.jobtracker.system.dir; Ignoring.
2014-10-23 16:36:34,658 WARN [main] org.apache.hadoop.conf.Configuration:
job.xml:an attempt to override final parameter: mapreduce.cluster.local.dir;
Ignoring.
2014-10-23 16:36:34,660 WARN [main] org.apache.hadoop.conf.Configuration:
job.xml:an attempt to override final parameter: mapreduce.cluster.local.dir;
Ignoring.
2014-10-23 16:36:34,663 WARN [main] org.apache.hadoop.conf.Configuration:
job.xml:an attempt to override final parameter:
mapreduce.job.end-notification.max.retry.interval; Ignoring.
2014-10-23 16:36:34,663 WARN [main] org.apache.hadoop.conf.Configuration:
job.xml:an attempt to override final parameter:
mapreduce.job.end-notification.max.attempts; Ignoring.
2014-10-23 16:36:34,669 WARN [main] org.apache.hadoop.conf.Configuration:
job.xml:an attempt to override final parameter: fs.defaultFS; Ignoring.
2014-10-23 16:36:34,672 WARN [main] org.apache.hadoop.conf.Configuration:
job.xml:an attempt to override final parameter: hadoop.tmp.dir; Ignoring.
2014-10-23 16:36:34,672 WARN [main] org.apache.hadoop.conf.Configuration:
job.xml:an attempt to override final parameter: fs.defaultFS; Ignoring.
2014-10-23 16:36:34,672 WARN [main] org.apache.hadoop.conf.Configuration:
job.xml:an attempt to override final parameter:
mapreduce.jobtracker.system.dir; Ignoring.
2014-10-23 16:36:34,792 INFO [main]
org.apache.hadoop.mapreduce.v2.app.MRAppMaster: Executing with tokens:
2014-10-23 16:36:34,792 INFO [main]
org.apache.hadoop.mapreduce.v2.app.MRAppMaster: Kind: YARN_AM_RM_TOKEN,
Service: , Ident: (org.apache.hadoop.yarn.security.AMRMTokenIdentifier@701c57)
2014-10-23 16:36:34,805 INFO [main]
org.apache.hadoop.mapreduce.v2.app.MRAppMaster: Kind: RM_DELEGATION_TOKEN,
Service: localhost:8032, Ident: (owner=si503lee, renewer=oozie mr token,
realUser=si503lee, issueDate=1414049792851, maxDate=1414654592851,
sequenceNumber=6, masterKeyId=2)
2014-10-23 16:36:34,812 INFO [main]
org.apache.hadoop.mapreduce.v2.app.MRAppMaster: The specific max attempts: 2
for application: 7. Attempt num: 1 is last retry: false
2014-10-23 16:36:34,912 WARN [main] org.apache.hadoop.conf.Configuration:
job.xml:an attempt to override final parameter:
mapreduce.jobtracker.system.dir; Ignoring.
2014-10-23 16:36:34,913 WARN [main] org.apache.hadoop.conf.Configuration:
job.xml:an attempt to override final parameter: mapreduce.cluster.local.dir;
Ignoring.
2014-10-23 16:36:34,914 WARN [main] org.apache.hadoop.conf.Configuration:
job.xml:an attempt to override final parameter: dfs.datanode.data.dir;
Ignoring.
2014-10-23 16:36:34,914 WARN [main] org.apache.hadoop.conf.Configuration:
job.xml:an attempt to override final parameter: dfs.namenode.name.dir;
Ignoring.
2014-10-23 16:36:34,915 WARN [main] org.apache.hadoop.conf.Configuration:
job.xml:an attempt to override final parameter: mapreduce.cluster.local.dir;
Ignoring.
2014-10-23 16:36:34,916 WARN [main] org.apache.hadoop.conf.Configuration:
job.xml:an attempt to override final parameter:
mapreduce.job.end-notification.max.retry.interval; Ignoring.
2014-10-23 16:36:34,916 WARN [main] org.apache.hadoop.conf.Configuration:
job.xml:an
Re: Re: ShortestPath Code execution on Hadoop 2.4.0 Inbox x
You just run 'mvn clean' before that command :) Best Regards, Seong-il --- Original Message --- Sender : Vikalp Handahanda.vik...@gmail.com Date : 2014-08-12 14:35 (GMT+09:00) Title : Re: ShortestPath Code execution on Hadoop 2.4.0 Inbox x Thanks for your replies. Do I need to uninstall anything before running the above mvn command ? Regards, Vikalp Handa On Tue, Aug 12, 2014 at 10:26 AM, 이성일 si503@samsung.com wrote: Hi, Vikalp I think you complied as wrong hadoop version. Try below command If you use yarn mvn -Phadoop_yarn -Dhadoop.version=2.4.0 -DskipTests package or if you use just hadoop2 mvn -Phadoop_2 -Dhadoop.version=2.4.0 -DskipTests package Best Regards, Seong-il --- Original Message --- Sender : Vikalp Handahanda.vik...@gmail.com Date : 2014-08-12 13:40 (GMT+09:00) Title : ShortestPath Code execution on Hadoop 2.4.0 Inbox x Hi everyone, I am new to Apache Giraph and would like execute Shortestpath and PageRank example code on Hadoop 2.4.0 single node cluster (my machine) having Centos 6.5. I have successfully build Giraph on my machine but unable to execute ShortestPath code. Please let me know if there are any dependencies to be resolved before code execution. P.S. : Command used: hadoop jar /usr/local/giraph/giraph-examples/target/giraph-examples-1.1.0-SNAPSHOT-for-hadoop-1.2.1-jar-with-dependencies.jar org.apache.giraph.GiraphRunner org.apache.giraph.examples.SimpleShortestPathsComputation -vif org.apache.giraph.io.formats.JsonLongDoubleFloatDoubleVertexInputFormat -vip /user/hadoop/input/tiny_graph.txt -vof org.apache.giraph.io.formats.IdWithValueTextOutputFormat -op /user/hduser/output/shortestpaths -w 1 -ca giraph.SplitMasterWorker=false Execution Result: 14/08/11 18:48:37 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable 14/08/11 18:48:40 INFO utils.ConfigurationUtils: No edge input format specified. Ensure your InputFormat does not require one. 14/08/11 18:48:40 INFO utils.ConfigurationUtils: No edge output format specified. Ensure your OutputFormat does not require one. 14/08/11 18:48:40 INFO utils.ConfigurationUtils: Setting custom argument [giraph.SplitMasterWorker] to [false] in GiraphConfiguration 14/08/11 18:48:40 INFO Configuration.deprecation: mapreduce.job.counters.limit is deprecated. Instead, use mapreduce.job.counters.max 14/08/11 18:48:40 INFO Configuration.deprecation: mapred.job.map.memory.mb is deprecated. Instead, use mapreduce.map.memory.mb 14/08/11 18:48:40 INFO Configuration.deprecation: mapred.job.reduce.memory.mb is deprecated. Instead, use mapreduce.reduce.memory.mb 14/08/11 18:48:40 INFO Configuration.deprecation: mapred.map.tasks.speculative.execution is deprecated. Instead, use mapreduce.map.speculative 14/08/11 18:48:40 INFO Configuration.deprecation: mapreduce.user.classpath.first is deprecated. Instead, use mapreduce.job.user.classpath.first 14/08/11 18:48:40 INFO Configuration.deprecation: mapred.map.max.attempts is deprecated. Instead, use mapreduce.map.maxattempts 14/08/11 18:48:40 INFO job.GiraphJob: run: Since checkpointing is disabled (default), do not allow any task retries (setting mapred.map.max.attempts = 0, old value = 4) 14/08/11 18:48:40 INFO Configuration.deprecation: mapred.job.tracker is deprecated. Instead, use mapreduce.jobtracker.address 14/08/11 18:48:41 INFO Configuration.deprecation: session.id is deprecated. Instead, use dfs.metrics.session-id 14/08/11 18:48:41 INFO jvm.JvmMetrics: Initializing JVM Metrics with processName=JobTracker, sessionId= Exception in thread main java.lang.IncompatibleClassChangeError: Found interface org.apache.hadoop.mapreduce.JobContext, but class was expected at org.apache.giraph.bsp.BspOutputFormat.checkOutputSpecs(BspOutputFormat.java:44) at org.apache.hadoop.mapreduce.JobSubmitter.checkSpecs(JobSubmitter.java:458) at org.apache.hadoop.mapreduce.JobSubmitter.submitJobInternal(JobSubmitter.java:343) at org.apache.hadoop.mapreduce.Job$10.run(Job.java:1285) at org.apache.hadoop.mapreduce.Job$10.run(Job.java:1282) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:422) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1548) at org.apache.hadoop.mapreduce.Job.submit(Job.java:1282) at org.apache.giraph.job.GiraphJob.run(GiraphJob.java:250) at org.apache.giraph.GiraphRunner.run(GiraphRunner.java:94) at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70) at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:84) at org.apache.giraph.GiraphRunner.main(GiraphRunner.java:124) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62
