[
https://issues.apache.org/jira/browse/OOZIE-2391?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Praveen Bathala updated OOZIE-2391:
-----------------------------------
Description:
When providing spark-opts in workflow.xml for oozie, the "--conf" property
values is not parsed properly.
Example sparkopts
<spark-opts>--conf spark.executor.extraJavaOptions=-Dconfig.url=hdfs:///my.conf
-Dmy.env=${env}</spark-opts>
In this case the value for property "spark.executor.extraJavaOptions" is
"-Dconfig.url=hdfs:///my.conf -Dmy.env=${env}", but in the oozie code here
https://github.com/apache/oozie/blob/38e33811bc3f9d3b1a1a024dcab6af760734b696/sharelib/spark/src/main/java/org/apache/oozie/action/hadoop/SparkMain.java#L110
the string is just split by space, which make the "-Dmy.env=${env}" and spark
fails job as that argument is not valid.
This doesn't work with enclosing the whole value in quotes too.
Error from stderr:
Error: Unrecognized option '-Dmy.env=development"'.
Run with --help for usage help or --verbose for debug output
Intercepting System.exit(1)
Failing Oozie Launcher, Main class [org.apache.oozie.action.hadoop.SparkMain],
exit code [1]
was:
When providing spark-opts in workflow.xml for oozie, the "--conf" property
values is not parsed properly.
Example sparkopts
<spark-opts>--conf spark.executor.extraJavaOptions=-Dconfig.url=hdfs:///my.conf
-Dmy.env=${env}</spark-opts>
In this case the value for property "spark.executor.extraJavaOptions" is
"-Dconfig.url=hdfs:///my.conf -Dmy.env=${env}", but in the oozie code here
https://github.com/apache/oozie/blob/38e33811bc3f9d3b1a1a024dcab6af760734b696/sharelib/spark/src/main/java/org/apache/oozie/action/hadoop/SparkMain.java#L110
the string is just split by space, which make the "-Dmy.env=${env}" and spark
fails job as that argument is not valid.
This doesn't work with enclosing the whole value in quotes too.
> spark-opts value in workflow.xml is not parsed properly
> -------------------------------------------------------
>
> Key: OOZIE-2391
> URL: https://issues.apache.org/jira/browse/OOZIE-2391
> Project: Oozie
> Issue Type: Bug
> Reporter: Praveen Bathala
>
> When providing spark-opts in workflow.xml for oozie, the "--conf" property
> values is not parsed properly.
> Example sparkopts
> <spark-opts>--conf
> spark.executor.extraJavaOptions=-Dconfig.url=hdfs:///my.conf
> -Dmy.env=${env}</spark-opts>
> In this case the value for property "spark.executor.extraJavaOptions" is
> "-Dconfig.url=hdfs:///my.conf -Dmy.env=${env}", but in the oozie code here
> https://github.com/apache/oozie/blob/38e33811bc3f9d3b1a1a024dcab6af760734b696/sharelib/spark/src/main/java/org/apache/oozie/action/hadoop/SparkMain.java#L110
> the string is just split by space, which make the "-Dmy.env=${env}" and
> spark fails job as that argument is not valid.
> This doesn't work with enclosing the whole value in quotes too.
> Error from stderr:
> Error: Unrecognized option '-Dmy.env=development"'.
> Run with --help for usage help or --verbose for debug output
> Intercepting System.exit(1)
> Failing Oozie Launcher, Main class
> [org.apache.oozie.action.hadoop.SparkMain], exit code [1]
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
