[
https://issues.apache.org/jira/browse/OOZIE-2071?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14277871#comment-14277871
]
Robert Kanter commented on OOZIE-2071:
--------------------------------------
I tried running it but got a {{ClassNotFoundException}} because it couldn't
find {{SparkMain}}. That's easily fixed by adding
{{oozie.use.system.libpath=true}} to the job.properties. However, I then get
this exception:
{noformat}
Failing Oozie Launcher, Main class [org.apache.oozie.action.hadoop.SparkMain],
main() threw exception,
org.slf4j.impl.StaticLoggerBinder.getSingleton()Lorg/slf4j/impl/StaticLoggerBinder;
java.lang.NoSuchMethodError:
org.slf4j.impl.StaticLoggerBinder.getSingleton()Lorg/slf4j/impl/StaticLoggerBinder;
at org.apache.spark.Logging$class.initializeLogging(Logging.scala:116)
at
org.apache.spark.Logging$class.initializeIfNecessary(Logging.scala:107)
at org.apache.spark.Logging$class.log(Logging.scala:51)
at org.apache.spark.SecurityManager.log(SecurityManager.scala:143)
at org.apache.spark.Logging$class.logInfo(Logging.scala:59)
at org.apache.spark.SecurityManager.logInfo(SecurityManager.scala:143)
at
org.apache.spark.SecurityManager.setViewAcls(SecurityManager.scala:208)
at org.apache.spark.SecurityManager.<init>(SecurityManager.scala:167)
at org.apache.spark.SparkEnv$.create(SparkEnv.scala:151)
at org.apache.spark.SparkContext.<init>(SparkContext.scala:203)
at
org.apache.spark.api.java.JavaSparkContext.<init>(JavaSparkContext.scala:53)
at org.apache.spark.examples.JavaWordCount.main(JavaWordCount.java:44)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at org.apache.spark.deploy.SparkSubmit$.launch(SparkSubmit.scala:328)
at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:75)
at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)
at org.apache.oozie.action.hadoop.SparkMain.runSpark(SparkMain.java:105)
at org.apache.oozie.action.hadoop.SparkMain.run(SparkMain.java:96)
at org.apache.oozie.action.hadoop.LauncherMain.run(LauncherMain.java:44)
at org.apache.oozie.action.hadoop.SparkMain.main(SparkMain.java:40)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at
org.apache.oozie.action.hadoop.LauncherMapper.map(LauncherMapper.java:228)
at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:50)
at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:430)
at org.apache.hadoop.mapred.MapTask.run(MapTask.java:366)
at org.apache.hadoop.mapred.Child$4.run(Child.java:255)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:415)
at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1190)
at org.apache.hadoop.mapred.Child.main(Child.java:249)
{noformat}
Can you:
- Add the sharelib property mentioned above
- Look into and fix the {{NoSuchMethodError}} exception
I also had some minor comments:
- It should use the latest workflow schema version, 0.5
- Can you rename the spark node in the example workflow.xml from "sparkexample
to "spark-node" to be more consistent with the rest of the examples?
- In the examples pom.xml, the artifactId for the added Spark dependency can be
the {{spark.examples.artifact}} variable you created in the root pom.xml
> Add a Spark example
> -------------------
>
> Key: OOZIE-2071
> URL: https://issues.apache.org/jira/browse/OOZIE-2071
> Project: Oozie
> Issue Type: Task
> Components: examples
> Affects Versions: trunk
> Reporter: Robert Kanter
> Assignee: pavan kumar kolamuri
> Fix For: trunk
>
> Attachments: OOZIE-2071.patch
>
>
> With the new Spark action, we should have a Spark example workflow that runs
> some Spark job.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
