[jira] [Commented] (OOZIE-2812) SparkConfigurationService should support loading configurations from multiple Spark versions
[
https://issues.apache.org/jira/browse/OOZIE-2812?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16681437#comment-16681437
]
Hadoop QA commented on OOZIE-2812:
--
Testing JIRA OOZIE-2812
Cleaning local git workspace
{color:red}-1{color} Patch failed to apply to head of branch
> SparkConfigurationService should support loading configurations from multiple
> Spark versions
>
>
> Key: OOZIE-2812
> URL: https://issues.apache.org/jira/browse/OOZIE-2812
> Project: Oozie
> Issue Type: Improvement
>Reporter: Peter Cseh
>Priority: Major
> Attachments: OOZIE-2812.001.patch, OOZIE-2812.002.patch,
> OOZIE-2812.003.patch, OOZIE-2812.004.patch
>
>
> Right now SparkConfigruationService serves one Spark configuration set by
> {{oozie.service.SparkConfigurationService.spark.configurations}}
> We cloud improve this to support more versions depending on the name of the
> sharelib.
> E.g. the property could change to
> oozie.service.SparkConfigurationService..configurations
> This would be backward compatible as the name for the default Spark sharelib
> is spark while it would be possible to add a sharelib named spark2 or
> spark2.1 and define itheir configuration via
> oozie.service.SparkConfigurationService.spark2.configurations and
> oozie.service.SparkConfigurationService.spark2.1.configurations.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (OOZIE-2812) SparkConfigurationService should support loading configurations from multiple Spark versions
[
https://issues.apache.org/jira/browse/OOZIE-2812?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16681427#comment-16681427
]
Hadoop QA commented on OOZIE-2812:
--
PreCommit-OOZIE-Build started
> SparkConfigurationService should support loading configurations from multiple
> Spark versions
>
>
> Key: OOZIE-2812
> URL: https://issues.apache.org/jira/browse/OOZIE-2812
> Project: Oozie
> Issue Type: Improvement
>Reporter: Peter Cseh
>Priority: Major
> Attachments: OOZIE-2812.001.patch, OOZIE-2812.002.patch,
> OOZIE-2812.003.patch, OOZIE-2812.004.patch
>
>
> Right now SparkConfigruationService serves one Spark configuration set by
> {{oozie.service.SparkConfigurationService.spark.configurations}}
> We cloud improve this to support more versions depending on the name of the
> sharelib.
> E.g. the property could change to
> oozie.service.SparkConfigurationService..configurations
> This would be backward compatible as the name for the default Spark sharelib
> is spark while it would be possible to add a sharelib named spark2 or
> spark2.1 and define itheir configuration via
> oozie.service.SparkConfigurationService.spark2.configurations and
> oozie.service.SparkConfigurationService.spark2.1.configurations.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (OOZIE-2812) SparkConfigurationService should support loading configurations from multiple Spark versions
[
https://issues.apache.org/jira/browse/OOZIE-2812?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16162688#comment-16162688
]
Hadoop QA commented on OOZIE-2812:
--
Testing JIRA OOZIE-2812
Cleaning local git workspace
{color:red}-1{color} Patch failed to apply to head of branch
> SparkConfigurationService should support loading configurations from multiple
> Spark versions
>
>
> Key: OOZIE-2812
> URL: https://issues.apache.org/jira/browse/OOZIE-2812
> Project: Oozie
> Issue Type: Improvement
>Reporter: Peter Cseh
>Assignee: Peter Cseh
> Attachments: OOZIE-2812.001.patch, OOZIE-2812.002.patch,
> OOZIE-2812.003.patch, OOZIE-2812.004.patch
>
>
> Right now SparkConfigruationService serves one Spark configuration set by
> {{oozie.service.SparkConfigurationService.spark.configurations}}
> We cloud improve this to support more versions depending on the name of the
> sharelib.
> E.g. the property could change to
> oozie.service.SparkConfigurationService..configurations
> This would be backward compatible as the name for the default Spark sharelib
> is spark while it would be possible to add a sharelib named spark2 or
> spark2.1 and define itheir configuration via
> oozie.service.SparkConfigurationService.spark2.configurations and
> oozie.service.SparkConfigurationService.spark2.1.configurations.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (OOZIE-2812) SparkConfigurationService should support loading configurations from multiple Spark versions
[
https://issues.apache.org/jira/browse/OOZIE-2812?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16162682#comment-16162682
]
Sergey Zhemzhitsky commented on OOZIE-2812:
---
Hello guys,
I'm wondering whether
*oozie.service.SparkConfigurationService.spark.configurations* configuration
option is really necessary?
Here are just my two cents regarding this not so obvious option
# Spark jobs are just yarn applications as well as flink applications, as well
as java applications that can run on yarn, etc.
There are may be multiple Spark jobs which use completely different spark
versions (for example custom patched ones), so it is not necessary to create
indirect reference between yarn resource manager and spark configuration,
although for map-reduce the similar
*oozie.service.HadoopAccessorService.hadoop.configurations* option makes sense,
because currently it's hardly possible to run multiple map-reduce
implementations on top of single yarn resource manager.
# *oozie.service.SparkConfigurationService.spark.configurations* option reads
*spark-defaults.properties* from the default location (i.e. */etc/spark/conf*)
into java.util.Properties and then just appends these ones to the *spark-opts*
as if these options were specified by means of *--conf* command line options.
But, usually, when using *spark\-submit.sh*, *\-\-conf* command line options
have precedence over properties specified in *spark\-defaults.properties* and
there is a chance that options provided by the user by means of *--conf* spark
option will be overriden by properties provided in
*oozie.service.SparkConfigurationService.spark.configurations*.
# Latest implementation of spark action already supports ** element
and *spark\-defaults.properties* in current working directory, and these two
possibilities give more flexibility than
*oozie.service.SparkConfigurationService.spark.configurations*, because user can
## add *spark-default.properties* to the workflow application and oozie will
pick it up
## add
{code}
local:/etc/spark/conf/spark-defaults.properties
{code}
and oozie's spark action will automatically pick up this spark-defaults by
means of *\-\-properties\-file* option preserving semantics of precedence of
properties
** provided by means of *spark\-defaults.properties* file,
** provided by means of *\-\-properties\-file* command line option
** provided by means of *\-\-conf* command line options
> SparkConfigurationService should support loading configurations from multiple
> Spark versions
>
>
> Key: OOZIE-2812
> URL: https://issues.apache.org/jira/browse/OOZIE-2812
> Project: Oozie
> Issue Type: Improvement
>Reporter: Peter Cseh
>Assignee: Peter Cseh
> Attachments: OOZIE-2812.001.patch, OOZIE-2812.002.patch,
> OOZIE-2812.003.patch, OOZIE-2812.004.patch
>
>
> Right now SparkConfigruationService serves one Spark configuration set by
> {{oozie.service.SparkConfigurationService.spark.configurations}}
> We cloud improve this to support more versions depending on the name of the
> sharelib.
> E.g. the property could change to
> oozie.service.SparkConfigurationService..configurations
> This would be backward compatible as the name for the default Spark sharelib
> is spark while it would be possible to add a sharelib named spark2 or
> spark2.1 and define itheir configuration via
> oozie.service.SparkConfigurationService.spark2.configurations and
> oozie.service.SparkConfigurationService.spark2.1.configurations.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (OOZIE-2812) SparkConfigurationService should support loading configurations from multiple Spark versions
[
https://issues.apache.org/jira/browse/OOZIE-2812?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15937447#comment-15937447
]
Hadoop QA commented on OOZIE-2812:
--
Testing JIRA OOZIE-2812
Cleaning local git workspace
{color:green}+1 PATCH_APPLIES{color}
{color:green}+1 CLEAN{color}
{color:red}-1 RAW_PATCH_ANALYSIS{color}
.{color:green}+1{color} the patch does not introduce any @author tags
.{color:green}+1{color} the patch does not introduce any tabs
.{color:green}+1{color} the patch does not introduce any trailing spaces
.{color:red}-1{color} the patch contains 1 line(s) longer than 132
characters
.{color:green}+1{color} the patch does adds/modifies 2 testcase(s)
{color:green}+1 RAT{color}
.{color:green}+1{color} the patch does not seem to introduce new RAT
warnings
{color:green}+1 JAVADOC{color}
.{color:green}+1{color} the patch does not seem to introduce new Javadoc
warnings
{color:green}+1 COMPILE{color}
.{color:green}+1{color} HEAD compiles
.{color:green}+1{color} patch compiles
.{color:green}+1{color} the patch does not seem to introduce new javac
warnings
{color:red}-1{color} There are [444] new bugs found below threshold in total
that must be fixed.
.{color:red}-1{color} There are [98] new bugs found below threshold in
[server] that must be fixed.
.You can find the FindBugs diff here (look for the red and orange ones):
server/findbugs-new.html
.The most important FindBugs errors are:
.At CLIParser.java:[line 169]: Found reliance on default encoding in
org.apache.oozie.cli.CLIParser.showHelp(CommandLine): new
java.io.PrintWriter(OutputStream)
.At CLIParser.java:[line 57]: new org.apache.oozie.cli.CLIParser(String,
String[]) may expose internal representation by storing an externally mutable
object into CLIParser.cliHelp
.At CLIParser.java:[lines 80-104]: Should
org.apache.oozie.cli.CLIParser$Command be a _static_ inner class?
.At OozieCLI.java:[line 845]: Found reliance on default encoding in
org.apache.oozie.cli.OozieCLI.getConfiguration(OozieClient, CommandLine): new
java.io.FileReader(File)
.At OozieCLI.java:[line 2175]: Found reliance on default encoding in
org.apache.oozie.cli.OozieCLI.validateCommandV41(CommandLine): new
java.io.FileReader(File)
.Obligation to clean up resource created at OozieCLI.java:[line 848] is not
discharged: org.apache.oozie.cli.OozieCLI.getConfiguration(OozieClient,
CommandLine) may fail to clean up java.io.InputStream
.Path continues at OozieCLI.java:[line 858]: Path continues at
OozieCLI.java:[line 854]
.Obligation to clean up resource created at OozieCLI.java:[line 845] is not
discharged: org.apache.oozie.cli.OozieCLI.getConfiguration(OozieClient,
CommandLine) may fail to clean up java.io.Reader
.Path continues at OozieCLI.java:[line 858]: Path continues at
OozieCLI.java:[line 854]
.At OozieCLI.java:[line 845]:
org.apache.oozie.cli.OozieCLI.getConfiguration(OozieClient, CommandLine) may
fail to close stream
.At OozieCLI.java:[line 2178]: Exception is caught when Exception is not
thrown in org.apache.oozie.cli.OozieCLI.validateCommandV41(CommandLine)
.At OozieCLI.java:[line 1407]: Format-string method String.format(String,
Object[]) called with format string "%-43s%-10s%-37s%-10s%-21s%-21s" wants 6
arguments but is given 7 in
org.apache.oozie.cli.OozieCLI.printCoordJob(CoordinatorJob, String, boolean)
.At AuthOozieClient.java:[line 218]: Found reliance on default encoding in
org.apache.oozie.client.AuthOozieClient.readAuthToken(): new
java.io.FileReader(File)
.At AuthOozieClient.java:[line 248]: Found reliance on default encoding in
org.apache.oozie.client.AuthOozieClient.writeAuthToken(AuthenticatedURL$Token):
new java.io.FileWriter(File)
.At AuthOozieClient.java:[line 285]: Load of known null value in
org.apache.oozie.client.AuthOozieClient.getAuthenticator()
.At AuthOozieClient.java:[line 326]: Load of known null value in
org.apache.oozie.client.AuthOozieClient.getAuthenticator()
.Obligation to clean up resource created at AuthOozieClient.java:[line 248]
is not discharged:
org.apache.oozie.client.AuthOozieClient.writeAuthToken(AuthenticatedURL$Token)
may fail to clean up java.io.Writer on checked exception
.Path continues at AuthOozieClient.java:[line 257]: Path continues at
AuthOozieClient.java:[line 249]
.Path continues at AuthOozieClient.java:[line 264]: Path continues at
AuthOozieClient.java:[line 261]
.At AuthOozieClient.java:[line 129]: Exceptional return value of
java.io.File.delete() ignored in
org.apache.oozie.client.AuthOozieClient.createConnection(URL, String)
.Another occurrence at AuthOozieClient.java:[line 159]: Another occurrence
at AuthOozieClient.java:[line 143]
.Exceptional return value of java.io.File.delete() ignored in
[jira] [Commented] (OOZIE-2812) SparkConfigurationService should support loading configurations from multiple Spark versions
[
https://issues.apache.org/jira/browse/OOZIE-2812?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15935508#comment-15935508
]
Robert Kanter commented on OOZIE-2812:
--
That seems cleaner. Good idea.
I've tried to leave some comments on RB, but it seems to be having problems.
I'll try again later, but if not, I managed to take a screenshot and will post
that here.
> SparkConfigurationService should support loading configurations from multiple
> Spark versions
>
>
> Key: OOZIE-2812
> URL: https://issues.apache.org/jira/browse/OOZIE-2812
> Project: Oozie
> Issue Type: Improvement
>Reporter: Peter Cseh
>Assignee: Peter Cseh
> Attachments: OOZIE-2812.001.patch, OOZIE-2812.002.patch,
> OOZIE-2812.003.patch
>
>
> Right now SparkConfigruationService serves one Spark configuration set by
> {{oozie.service.SparkConfigurationService.spark.configurations}}
> We cloud improve this to support more versions depending on the name of the
> sharelib.
> E.g. the property could change to
> oozie.service.SparkConfigurationService..configurations
> This would be backward compatible as the name for the default Spark sharelib
> is spark while it would be possible to add a sharelib named spark2 or
> spark2.1 and define itheir configuration via
> oozie.service.SparkConfigurationService.spark2.configurations and
> oozie.service.SparkConfigurationService.spark2.1.configurations.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Commented] (OOZIE-2812) SparkConfigurationService should support loading configurations from multiple Spark versions
[
https://issues.apache.org/jira/browse/OOZIE-2812?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15934840#comment-15934840
]
Hadoop QA commented on OOZIE-2812:
--
Testing JIRA OOZIE-2812
Cleaning local git workspace
{color:green}+1 PATCH_APPLIES{color}
{color:green}+1 CLEAN{color}
{color:red}-1 RAW_PATCH_ANALYSIS{color}
.{color:green}+1{color} the patch does not introduce any @author tags
.{color:red}-1{color} the patch contains 2 line(s) with tabs
.{color:green}+1{color} the patch does not introduce any trailing spaces
.{color:red}-1{color} the patch contains 2 line(s) longer than 132
characters
.{color:green}+1{color} the patch does adds/modifies 2 testcase(s)
{color:green}+1 RAT{color}
.{color:green}+1{color} the patch does not seem to introduce new RAT
warnings
{color:green}+1 JAVADOC{color}
.{color:green}+1{color} the patch does not seem to introduce new Javadoc
warnings
{color:green}+1 COMPILE{color}
.{color:green}+1{color} HEAD compiles
.{color:green}+1{color} patch compiles
.{color:green}+1{color} the patch does not seem to introduce new javac
warnings
{color:orange}0{color} There are [1] new bugs found in total that would be nice
to have fixed.
.{color:orange}0{color} There are [1] new bugs found in [core] that would
be nice to have fixed.
.You can find the FindBugs diff here: core/findbugs-new.html
.{color:green}+1{color} There are no new bugs found in [client].
.{color:green}+1{color} There are no new bugs found in [sharelib/distcp].
.{color:green}+1{color} There are no new bugs found in [sharelib/hcatalog].
.{color:green}+1{color} There are no new bugs found in [sharelib/oozie].
.{color:green}+1{color} There are no new bugs found in
[hadooplibs/hadoop-utils-2].
{color:green}+1 BACKWARDS_COMPATIBILITY{color}
.{color:green}+1{color} the patch does not change any JPA
Entity/Colum/Basic/Lob/Transient annotations
.{color:green}+1{color} the patch does not modify JPA files
{color:green}+1 TESTS{color}
.Tests run: 1895
{color:green}+1 DISTRO{color}
.{color:green}+1{color} distro tarball builds with the patch
{color:red}*-1 Overall result, please check the reported -1(s)*{color}
The full output of the test-patch run is available at
. https://builds.apache.org/job/oozie-trunk-precommit-build/3713/
> SparkConfigurationService should support loading configurations from multiple
> Spark versions
>
>
> Key: OOZIE-2812
> URL: https://issues.apache.org/jira/browse/OOZIE-2812
> Project: Oozie
> Issue Type: Improvement
>Reporter: Peter Cseh
>Assignee: Peter Cseh
> Attachments: OOZIE-2812.001.patch, OOZIE-2812.002.patch,
> OOZIE-2812.003.patch
>
>
> Right now SparkConfigruationService serves one Spark configuration set by
> {{oozie.service.SparkConfigurationService.spark.configurations}}
> We cloud improve this to support more versions depending on the name of the
> sharelib.
> E.g. the property could change to
> oozie.service.SparkConfigurationService..configurations
> This would be backward compatible as the name for the default Spark sharelib
> is spark while it would be possible to add a sharelib named spark2 or
> spark2.1 and define itheir configuration via
> oozie.service.SparkConfigurationService.spark2.configurations and
> oozie.service.SparkConfigurationService.spark2.1.configurations.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Commented] (OOZIE-2812) SparkConfigurationService should support loading configurations from multiple Spark versions
[
https://issues.apache.org/jira/browse/OOZIE-2812?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15925349#comment-15925349
]
Hadoop QA commented on OOZIE-2812:
--
Testing JIRA OOZIE-2812
Cleaning local git workspace
{color:green}+1 PATCH_APPLIES{color}
{color:green}+1 CLEAN{color}
{color:green}+1 RAW_PATCH_ANALYSIS{color}
.{color:green}+1{color} the patch does not introduce any @author tags
.{color:green}+1{color} the patch does not introduce any tabs
.{color:green}+1{color} the patch does not introduce any trailing spaces
.{color:green}+1{color} the patch does not introduce any line longer than
132
.{color:green}+1{color} the patch does adds/modifies 1 testcase(s)
{color:green}+1 RAT{color}
.{color:green}+1{color} the patch does not seem to introduce new RAT
warnings
{color:green}+1 JAVADOC{color}
.{color:green}+1{color} the patch does not seem to introduce new Javadoc
warnings
{color:green}+1 COMPILE{color}
.{color:green}+1{color} HEAD compiles
.{color:green}+1{color} patch compiles
.{color:green}+1{color} the patch does not seem to introduce new javac
warnings
{color:orange}0{color} There are [1] new bugs found in total that would be nice
to have fixed.
.{color:green}+1{color} There are no new bugs found in [client].
.{color:green}+1{color} There are no new bugs found in [sharelib/hcatalog].
.{color:green}+1{color} There are no new bugs found in [sharelib/distcp].
.{color:green}+1{color} There are no new bugs found in [sharelib/oozie].
.{color:green}+1{color} There are no new bugs found in
[hadooplibs/hadoop-utils-2].
.{color:orange}0{color} There are [1] new bugs found in [core] that would
be nice to have fixed.
.You can find the FindBugs diff here: core/findbugs-new.html
{color:green}+1 BACKWARDS_COMPATIBILITY{color}
.{color:green}+1{color} the patch does not change any JPA
Entity/Colum/Basic/Lob/Transient annotations
.{color:green}+1{color} the patch does not modify JPA files
{color:red}-1 TESTS{color}
.Tests run: 1892
.Tests failed: 1
.Tests errors: 0
.The patch failed the following testcases:
. testConnectionDrop(org.apache.oozie.jms.TestJMSJobEventListener)
.Tests failing with errors:
.
{color:green}+1 DISTRO{color}
.{color:green}+1{color} distro tarball builds with the patch
{color:red}*-1 Overall result, please check the reported -1(s)*{color}
The full output of the test-patch run is available at
. https://builds.apache.org/job/oozie-trunk-precommit-build/3691/
> SparkConfigurationService should support loading configurations from multiple
> Spark versions
>
>
> Key: OOZIE-2812
> URL: https://issues.apache.org/jira/browse/OOZIE-2812
> Project: Oozie
> Issue Type: Improvement
>Reporter: Peter Cseh
>Assignee: Peter Cseh
> Attachments: OOZIE-2812.001.patch, OOZIE-2812.002.patch
>
>
> Right now SparkConfigruationService serves one Spark configuration set by
> {{oozie.service.SparkConfigurationService.spark.configurations}}
> We cloud improve this to support more versions depending on the name of the
> sharelib.
> E.g. the property could change to
> oozie.service.SparkConfigurationService..configurations
> This would be backward compatible as the name for the default Spark sharelib
> is spark while it would be possible to add a sharelib named spark2 or
> spark2.1 and define itheir configuration via
> oozie.service.SparkConfigurationService.spark2.configurations and
> oozie.service.SparkConfigurationService.spark2.1.configurations.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
