Hi
We are trying to change our existing oozie workflows to use SparkAction
instead of ShellAction.
We are passing spark configuration in spark-opts with --conf, but these
values are not accessible in Spark and it is throwing error.
Please note we are able to use SparkAction successfully in yarn-cluster
mode if we are not using the spark configurations. I have attached oozie
workflow.xml, oozie log and yarncontainer-spark log files.
Thanks
Raja
Log Type: stderr
Log Upload Time: Fri Dec 04 10:26:17 -0800 2015
Log Length: 6035
15/12/04 10:26:04 INFO yarn.ApplicationMaster: Registered signal handlers for
[TERM, HUP, INT]
15/12/04 10:26:05 INFO yarn.ApplicationMaster: ApplicationAttemptId:
appattempt_1447700095990_2804_000002
15/12/04 10:26:06 WARN util.NativeCodeLoader: Unable to load native-hadoop
library for your platform... using builtin-java classes where applicable
15/12/04 10:26:06 INFO spark.SecurityManager: Changing view acls to: dev
15/12/04 10:26:06 INFO spark.SecurityManager: Changing modify acls to: dev
15/12/04 10:26:06 INFO spark.SecurityManager: SecurityManager: authentication
disabled; ui acls disabled; users with view permissions: Set(dev); users with
modify permissions: Set(dev)
15/12/04 10:26:06 INFO yarn.ApplicationMaster: Starting the user application in
a separate Thread
15/12/04 10:26:06 INFO yarn.ApplicationMaster: Waiting for spark context
initialization
15/12/04 10:26:06 INFO yarn.ApplicationMaster: Waiting for spark context
initialization ...
15/12/04 10:26:06 ERROR yarn.ApplicationMaster: User class threw exception: null
java.lang.ExceptionInInitializerError
at com.toyota.genericmodule.info.Info$.<init>(Info.scala:20)
at com.toyota.genericmodule.info.Info$.<clinit>(Info.scala)
at com.toyota.genericmodule.info.Info.main(Info.scala)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at
org.apache.spark.deploy.yarn.ApplicationMaster$$anon$2.run(ApplicationMaster.scala:480)
Caused by: com.typesafe.config.ConfigException$UnresolvedSubstitution:
application.conf: 53: Could not resolve substitution to a value: ${inputdb}
at
com.typesafe.config.impl.ConfigReference.resolveSubstitutions(ConfigReference.java:84)
at
com.typesafe.config.impl.ResolveSource.resolveCheckingReplacement(ResolveSource.java:110)
at
com.typesafe.config.impl.ResolveContext.resolve(ResolveContext.java:114)
at
com.typesafe.config.impl.ConfigConcatenation.resolveSubstitutions(ConfigConcatenation.java:178)
at
com.typesafe.config.impl.ResolveSource.resolveCheckingReplacement(ResolveSource.java:110)
at
com.typesafe.config.impl.ResolveContext.resolve(ResolveContext.java:114)
at
com.typesafe.config.impl.ConfigDelayedMerge.resolveSubstitutions(ConfigDelayedMerge.java:96)
at
com.typesafe.config.impl.ConfigDelayedMerge.resolveSubstitutions(ConfigDelayedMerge.java:59)
at
com.typesafe.config.impl.ResolveSource.resolveCheckingReplacement(ResolveSource.java:110)
at
com.typesafe.config.impl.ResolveContext.resolve(ResolveContext.java:114)
at
com.typesafe.config.impl.ResolveSource.lookupSubst(ResolveSource.java:62)
at
com.typesafe.config.impl.ConfigReference.resolveSubstitutions(ConfigReference.java:73)
at
com.typesafe.config.impl.ResolveSource.resolveCheckingReplacement(ResolveSource.java:110)
at
com.typesafe.config.impl.ResolveContext.resolve(ResolveContext.java:114)
at
com.typesafe.config.impl.ConfigConcatenation.resolveSubstitutions(ConfigConcatenation.java:178)
at
com.typesafe.config.impl.ResolveSource.resolveCheckingReplacement(ResolveSource.java:110)
at
com.typesafe.config.impl.ResolveContext.resolve(ResolveContext.java:114)
at
com.typesafe.config.impl.ResolveSource.lookupSubst(ResolveSource.java:62)
at
com.typesafe.config.impl.ConfigReference.resolveSubstitutions(ConfigReference.java:73)
at
com.typesafe.config.impl.ResolveSource.resolveCheckingReplacement(ResolveSource.java:110)
at
com.typesafe.config.impl.ResolveContext.resolve(ResolveContext.java:114)
at
com.typesafe.config.impl.SimpleConfigObject$1.modifyChildMayThrow(SimpleConfigObject.java:340)
at
com.typesafe.config.impl.SimpleConfigObject.modifyMayThrow(SimpleConfigObject.java:279)
at
com.typesafe.config.impl.SimpleConfigObject.resolveSubstitutions(SimpleConfigObject.java:320)
at
com.typesafe.config.impl.SimpleConfigObject.resolveSubstitutions(SimpleConfigObject.java:24)
at
com.typesafe.config.impl.ResolveSource.resolveCheckingReplacement(ResolveSource.java:110)
at
com.typesafe.config.impl.ResolveContext.resolve(ResolveContext.java:114)
at
com.typesafe.config.impl.SimpleConfigObject$1.modifyChildMayThrow(SimpleConfigObject.java:340)
at
com.typesafe.config.impl.SimpleConfigObject.modifyMayThrow(SimpleConfigObject.java:279)
at
com.typesafe.config.impl.SimpleConfigObject.resolveSubstitutions(SimpleConfigObject.java:320)
at
com.typesafe.config.impl.SimpleConfigObject.resolveSubstitutions(SimpleConfigObject.java:24)
at
com.typesafe.config.impl.ResolveSource.resolveCheckingReplacement(ResolveSource.java:110)
at
com.typesafe.config.impl.ResolveContext.resolve(ResolveContext.java:114)
at
com.typesafe.config.impl.ResolveContext.resolve(ResolveContext.java:147)
at com.typesafe.config.impl.SimpleConfig.resolve(SimpleConfig.java:60)
at com.typesafe.config.impl.SimpleConfig.resolve(SimpleConfig.java:33)
at com.typesafe.config.ConfigFactory.load(ConfigFactory.java:184)
at com.typesafe.config.ConfigFactory.load(ConfigFactory.java:113)
at com.typesafe.config.ConfigFactory.load(ConfigFactory.java:74)
at com.toyota.util.Config$.<init>(Config.scala:12)
at com.toyota.util.Config$.<clinit>(Config.scala)
... 8 more
15/12/04 10:26:06 INFO yarn.ApplicationMaster: Final app status: FAILED,
exitCode: 15, (reason: User class threw exception: null)
15/12/04 10:26:16 ERROR yarn.ApplicationMaster: SparkContext did not initialize
after waiting for 100000 ms. Please check earlier log output for errors.
Failing the application.
15/12/04 10:26:16 INFO yarn.ApplicationMaster: Unregistering ApplicationMaster
with FAILED (diag message: User class threw exception: null)
15/12/04 10:26:16 INFO yarn.ApplicationMaster: Deleting staging directory
.sparkStaging/application_1447700095990_2804Checking Oozie logs we can see that the spark configuration values are being
sent to SparkSubmit as below
=================================================================
>>> Invoking Main class now >>>
Spark Action Main class : org.apache.spark.deploy.SparkSubmit
=================================================================
>>> Invoking Main class now >>>
Spark Action Main class : org.apache.spark.deploy.SparkSubmit
Oozie Spark action configuration
=================================================================
--master
yarn-cluster
--name
MySpark
--class
com.toyota.genericmodule.info.Info
--conf
inputdb="dev.db"
--conf
outputdb="dev.db"
--verbose
hdfs://nameservice1/user/dev/spark/conf/TrySparkAction.jar
=================================================================
>>> Invoking Spark class now >>>
Heart beat
<<< Invocation of Main class completed <<<
Failing Oozie Launcher, Main class
[org.apache.oozie.action.hadoop.SparkMain], main() threw exception, Application
finished with failed status
org.apache.spark.SparkException: Application finished with failed status
at org.apache.spark.deploy.yarn.Client.run(Client.scala:656)
at org.apache.spark.deploy.yarn.Client$.main(Client.scala:681)
at org.apache.spark.deploy.yarn.Client.main(Client.scala)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at
org.apache.spark.deploy.SparkSubmit$.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:569)
at org.apache.spark.deploy.SparkSubmit$.doRunMain$1(SparkSubmit.scala:166)
at org.apache.spark.deploy.SparkSubmit$.submit(SparkSubmit.scala:189)
at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:110)
at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)
at org.apache.oozie.action.hadoop.SparkMain.runSpark(SparkMain.java:105)
at org.apache.oozie.action.hadoop.SparkMain.run(SparkMain.java:96)
at org.apache.oozie.action.hadoop.LauncherMain.run(LauncherMain.java:46)
at org.apache.oozie.action.hadoop.SparkMain.main(SparkMain.java:40)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at org.apache.oozie.action.hadoop.LauncherMapper.map(LauncherMapper.java:228)
at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:54)
at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:453)
at org.apache.hadoop.mapred.MapTask.run(MapTask.java:343)
at
org.apache.hadoop.mapred.LocalContainerLauncher$EventHandler.runSubtask(LocalContainerLauncher.java:370)
at
org.apache.hadoop.mapred.LocalContainerLauncher$EventHandler.runTask(LocalContainerLauncher.java:295)
at
org.apache.hadoop.mapred.LocalContainerLauncher$EventHandler.access$200(LocalContainerLauncher.java:181)
at
org.apache.hadoop.mapred.LocalContainerLauncher$EventHandler$1.run(LocalContainerLauncher.java:224)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
at java.util.concurrent.FutureTask.run(FutureTask.java:262)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)
Oozie Launcher failed, finishing Hadoop job gracefully
Oozie Launcher, uploading action data to HDFS sequence file:
hdfs://nameservice1/user/c360prodev/oozie-oozi/0000026-151201161646171-oozie-oozi-W/spark-03a1--spark/action-data.seq
Oozie Launcher ends
<workflow-app name="TrySparkAction" xmlns="uri:oozie:workflow:0.5">
<start to="spark-03a1"/>
<kill name="Kill">
<message>Action failed, error message[${wf:errorMessage(wf:lastErrorNode())}]</message>
</kill>
<action name="spark-03a1">
<spark xmlns="uri:oozie:spark-action:0.1">
<job-tracker>${jobTracker}</job-tracker>
<name-node>${nameNode}</name-node>
<master>yarn-cluster</master>
<mode></mode>
<name>MySpark</name>
<class>com.toyota.genericmodule.info.Info</class>
<jar>hdfs://nameservice1/user/dev/spark/conf/TrySparkAction.jar</jar>
<spark-opts>--conf inputdb="dev.db" --conf outputdb="dev.db"</spark-opts>
</spark>
<ok to="End"/>
<error to="Kill"/>
</action>
<end name="End"/>
</workflow-app>
---------------------------------------------------------------------
To unsubscribe, e-mail: user-unsubscr...@spark.apache.org
For additional commands, e-mail: user-h...@spark.apache.org
