[
https://issues.apache.org/jira/browse/OOZIE-2923?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16028596#comment-16028596
]
Hadoop QA commented on OOZIE-2923:
----------------------------------
Testing JIRA OOZIE-2923
Cleaning local git workspace
----------------------------
{color:green}+1 PATCH_APPLIES{color}
{color:green}+1 CLEAN{color}
{color:green}+1 RAW_PATCH_ANALYSIS{color}
. {color:green}+1{color} the patch does not introduce any @author tags
. {color:green}+1{color} the patch does not introduce any tabs
. {color:green}+1{color} the patch does not introduce any trailing spaces
. {color:green}+1{color} the patch does not introduce any line longer than
132
. {color:green}+1{color} the patch does adds/modifies 3 testcase(s)
{color:green}+1 RAT{color}
. {color:green}+1{color} the patch does not seem to introduce new RAT
warnings
{color:green}+1 JAVADOC{color}
. {color:green}+1{color} the patch does not seem to introduce new Javadoc
warnings
. {color:red}WARNING{color}: the current HEAD has 6 Javadoc warning(s)
{color:green}+1 COMPILE{color}
. {color:green}+1{color} HEAD compiles
. {color:green}+1{color} patch compiles
. {color:green}+1{color} the patch does not seem to introduce new javac
warnings
{color:red}-1{color} There are [1] new bugs found below threshold in total that
must be fixed.
. {color:green}+1{color} There are no new bugs found in [sharelib/hive].
. {color:green}+1{color} There are no new bugs found in [sharelib/hcatalog].
. {color:green}+1{color} There are no new bugs found in [sharelib/sqoop].
. {color:green}+1{color} There are no new bugs found in [sharelib/pig].
. {color:green}+1{color} There are no new bugs found in [sharelib/streaming].
. {color:red}-1{color} There are [1] new bugs found below threshold in
[sharelib/spark] that must be fixed.
. You can find the FindBugs diff here (look for the red and orange ones):
sharelib/spark/findbugs-new.html
. The most important FindBugs errors are:
. Dereferenced at SparkMain.java:[line 165]: Possible null pointer dereference
in org.apache.oozie.action.hadoop.SparkMain.getMatchingFile(Pattern) due to
return value of called method
. Known null at SparkMain.java:[line 165]
. {color:green}+1{color} There are no new bugs found in [sharelib/oozie].
. {color:green}+1{color} There are no new bugs found in [sharelib/distcp].
. {color:green}+1{color} There are no new bugs found in [sharelib/hive2].
. {color:green}+1{color} There are no new bugs found in [server].
. {color:green}+1{color} There are no new bugs found in [core].
. {color:green}+1{color} There are no new bugs found in [tools].
. {color:green}+1{color} There are no new bugs found in [docs].
. {color:green}+1{color} There are no new bugs found in [client].
. {color:green}+1{color} There are no new bugs found in [examples].
{color:green}+1 BACKWARDS_COMPATIBILITY{color}
. {color:green}+1{color} the patch does not change any JPA
Entity/Colum/Basic/Lob/Transient annotations
. {color:green}+1{color} the patch does not modify JPA files
{color:green}+1 TESTS{color}
. Tests run: 1961
. Tests rerun: 39
. Tests failed at first run:
org.apache.oozie.action.hadoop.TestLauncherAM,org.apache.oozie.action.hadoop.TestHdfsOperations,org.apache.oozie.event.TestEventGeneration,org.apache.oozie.command.coord.TestAbandonedCoordChecker,
{color:green}+1 DISTRO{color}
. {color:green}+1{color} distro tarball builds with the patch
----------------------------
{color:red}*-1 Overall result, please check the reported -1(s)*{color}
{color:red}. There is at least one warning, please check{color}
The full output of the test-patch run is available at
. https://builds.apache.org/job/oozie-trunk-precommit-build/3839/
> Improve Spark options parsing
> -----------------------------
>
> Key: OOZIE-2923
> URL: https://issues.apache.org/jira/browse/OOZIE-2923
> Project: Oozie
> Issue Type: Improvement
> Components: action
> Affects Versions: 4.3.0
> Reporter: Andras Piros
> Assignee: Andras Piros
> Fix For: 5.0.0
>
> Attachments: OOZIE-2923.001.patch
>
>
> There are two issues w/ Spark action's argument parsing within {{SparkMain}}
> h5. Driver and executor extra classpaths: equals sign used
> When the user specifies {{\-\-conf spark.executor.extraClassPath=XYZ}} or
> {{\-\-conf spark.driver.extraClassPath=ABC}}, the option {{\-\-conf}} will be
> added to {{sparkArgs}}. Then when the code tries to evaluate
> {{spark.executor.extraClassPath=XYZ}}, it uses special logic and set
> {{addToSparkArgs = false}}. As a result there will be a extra {{\-\-conf}} in
> the {{sparkArgs}} eventually.
> For example: {{\-\-conf spark.executor.extraClassPath=XYZ \-\-conf
> otherProperty=ABC}} will become {{\-\-conf \-\-conf otherProperty=ABC}},
> which will cause spark job submit failure later.
> We might need to remove one prior {{\-\-conf}} in {{sparkArgs}} if the
> current evaluated {{opt}} is {{EXECUTOR_CLASSPATH}} or {{DRIVER_CLASSPATH}}.
> h5. User provided files and archives: equals sign used
> For the following workflow XML snippet:
> {code:xml}
> <spark-opts>--files=${nameNode}/home/share/hive-site.xml --num-executors 4
> --executor-memory 7g --driver-memory 7g</spark-opts>
> {code}
> the {{\-\-files=$\{nameNode\}/home/share/hive-site.xml}} {{opt}} will be
> placed into {{sparkArgs}} in previous Oozie version without any modification,
> because we don't have special handling for {{\-\-files}} {{opt}}.
> If the user specifies {{\-\-files=$\{nameNode\}/home/share/hive-site.xml
> --num-executor 4}}, then {{SparkMain}} code treats {{\-\-num-executor}} as a
> file path / name. That caused the issue as I described in my previous
> comment. We might need to change the handling logic for {{FILES_OPTION}} and
> {{ARCHIVES_OPTION}} to be the same to {{DRIVER_CLASSPATH_OPTION}}.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
