[ https://issues.apache.org/jira/browse/OOZIE-2923?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Satish Subhashrao Saley updated OOZIE-2923: ------------------------------------------- Cherry-picked from master to branch-4.3. > Improve Spark options parsing > ----------------------------- > > Key: OOZIE-2923 > URL: https://issues.apache.org/jira/browse/OOZIE-2923 > Project: Oozie > Issue Type: Improvement > Components: action > Affects Versions: 4.3.0 > Reporter: Andras Piros > Assignee: Andras Piros > Fix For: 5.0.0b1, 4.3.1 > > Attachments: OOZIE-2923.001.patch, OOZIE-2923.002.patch, > OOZIE-2923.003.patch, OOZIE-2923.004.patch, OOZIE-2923.005.patch, > OOZIE-2923.006.patch > > > There are two issues w/ Spark action's argument parsing within {{SparkMain}} > h5. Driver and executor extra classpaths: equals sign used > When the user specifies {{\-\-conf spark.executor.extraClassPath=XYZ}} or > {{\-\-conf spark.driver.extraClassPath=ABC}}, the option {{\-\-conf}} will be > added to {{sparkArgs}}. Then when the code tries to evaluate > {{spark.executor.extraClassPath=XYZ}}, it uses special logic and set > {{addToSparkArgs = false}}. As a result there will be a extra {{\-\-conf}} in > the {{sparkArgs}} eventually. > For example: {{\-\-conf spark.executor.extraClassPath=XYZ \-\-conf > otherProperty=ABC}} will become {{\-\-conf \-\-conf otherProperty=ABC}}, > which will cause spark job submit failure later. > We might need to remove one prior {{\-\-conf}} in {{sparkArgs}} if the > current evaluated {{opt}} is {{EXECUTOR_CLASSPATH}} or {{DRIVER_CLASSPATH}}. > h5. User provided files and archives: equals sign used > For the following workflow XML snippet: > {code:xml} > <spark-opts>--files=${nameNode}/home/share/hive-site.xml --num-executors 4 > --executor-memory 7g --driver-memory 7g</spark-opts> > {code} > the {{\-\-files=$\{nameNode\}/home/share/hive-site.xml}} {{opt}} will be > placed into {{sparkArgs}} in previous Oozie version without any modification, > because we don't have special handling for {{\-\-files}} {{opt}}. > If the user specifies {{\-\-files=$\{nameNode\}/home/share/hive-site.xml > --num-executor 4}}, then {{SparkMain}} code treats {{\-\-num-executor}} as a > file path / name. That caused the issue as I described in my previous > comment. We might need to change the handling logic for {{FILES_OPTION}} and > {{ARCHIVES_OPTION}} to be the same to {{DRIVER_CLASSPATH_OPTION}}. -- This message was sent by Atlassian JIRA (v6.4.14#64029)