[jira] Subscription: Oozie Patch Available
Issue Subscription Filter: Oozie Patch Available (102 issues) Subscriber: ooziedaily Key Summary OOZIE-3135 Configure log4j2 in SqoopMain https://issues.apache.org/jira/browse/OOZIE-3135 OOZIE-3121 bump all maven plugins to latest versions https://issues.apache.org/jira/browse/OOZIE-3121 OOZIE-3113 Retry for ZK lock release https://issues.apache.org/jira/browse/OOZIE-3113 OOZIE-3105 testJMXInstrumentation from the org.apache.oozie.util.TestMetricsInstrumentation class is flaky https://issues.apache.org/jira/browse/OOZIE-3105 OOZIE-3094 fix for grammar mistake https://issues.apache.org/jira/browse/OOZIE-3094 OOZIE-3091 Oozie Sqoop Avro Import fails with "java.lang.NoClassDefFoundError: org/apache/avro/mapred/AvroWrapper" https://issues.apache.org/jira/browse/OOZIE-3091 OOZIE-3083 Make improved version Info backward compatible https://issues.apache.org/jira/browse/OOZIE-3083 OOZIE-3071 Oozie 4.3 Spark sharelib ueses a different version of commons-lang3 than Spark 2.2.0 https://issues.apache.org/jira/browse/OOZIE-3071 OOZIE-3063 Sanitizing variables that are part of openjpa.ConnectionProperties https://issues.apache.org/jira/browse/OOZIE-3063 OOZIE-3062 Set HADOOP_CONF_DIR for spark action https://issues.apache.org/jira/browse/OOZIE-3062 OOZIE-3002 address findbugs errors in client lib https://issues.apache.org/jira/browse/OOZIE-3002 OOZIE-2975 code clean up in pig sharelib, replace Exception with more explicit, add try with resources, StringBuilder instead of StringBuffer https://issues.apache.org/jira/browse/OOZIE-2975 OOZIE-2973 Make sure Oozie works with Hadoop 3 https://issues.apache.org/jira/browse/OOZIE-2973 OOZIE-2957 Documentation states that starting a coordinator is possible https://issues.apache.org/jira/browse/OOZIE-2957 OOZIE-2956 Fix Findbugs warnings related to reliance on default encoding in oozie-core https://issues.apache.org/jira/browse/OOZIE-2956 OOZIE-2955 Fix Findbugs warnings related to reliance on default encoding in oozie-client https://issues.apache.org/jira/browse/OOZIE-2955 OOZIE-2954 Fix Checkstyle issues in oozie-client https://issues.apache.org/jira/browse/OOZIE-2954 OOZIE-2953 Fix Checkstyle issues in oozie-tools https://issues.apache.org/jira/browse/OOZIE-2953 OOZIE-2952 Fix Findbugs warnings in oozie-sharelib-oozie https://issues.apache.org/jira/browse/OOZIE-2952 OOZIE-2949 Escape quotes whitespaces in Sqoop field https://issues.apache.org/jira/browse/OOZIE-2949 OOZIE-2942 Fix Findbugs warnings in oozie-examples https://issues.apache.org/jira/browse/OOZIE-2942 OOZIE-2937 Remove redundant groupId from the child pom's https://issues.apache.org/jira/browse/OOZIE-2937 OOZIE-2934 Fix "Exceptional return value of java.io.File.mkdirs() ignored" Findbugs error in oozie-sharelib-spark https://issues.apache.org/jira/browse/OOZIE-2934 OOZIE-2927 Append new line character for Hive2 query using query tag https://issues.apache.org/jira/browse/OOZIE-2927 OOZIE-2914 Consolidate Trim https://issues.apache.org/jira/browse/OOZIE-2914 OOZIE-2883 OOZIE throw the error "Missing [oozie.service.ProxyUserService.proxyuser.oozie.service.ProxyUserService.proxyuser.mr.groups] property" https://issues.apache.org/jira/browse/OOZIE-2883 OOZIE-2877 Oozie Git Action https://issues.apache.org/jira/browse/OOZIE-2877 OOZIE-2867 Timezone handling for Coordinators: emphasize "Continent/City" format https://issues.apache.org/jira/browse/OOZIE-2867 OOZIE-2834 ParameterVerifier logging non-useful warning for workflow definition https://issues.apache.org/jira/browse/OOZIE-2834 OOZIE-2833 when using uber mode the regex pattern used in the extractHeapSizeMB method does not allow heap sizes specified in bytes. https://issues.apache.org/jira/browse/OOZIE-2833 OOZIE-2829 Improve sharelib upload to accept multiple source folders https://issues.apache.org/jira/browse/OOZIE-2829 OOZIE-2826 Falcon feed fails to aws s3; Oozie joda time version does not meet required jar version 2.2 or later https://issues.apache.org/jira/browse/OOZIE-2826 OOZIE-2812 SparkConfigurationService should support loading configurations from multiple Spark versions https://issues.apache.org/jira/browse/OOZIE-2812 OOZIE-2795 Create lib directory or symlink for Oozie CLI during packaging https://issues.apache.org/jira/browse/OOZIE-2795 OOZIE-2791 ShareLib installation may fail on busy Hadoop clusters https://issues.apache.org/jira/browse/OOZIE-2791 OOZIE-2784 Include WEEK as a parameter in the Coordinator Expression Language Evaulator
[jira] Subscription: Oozie Patch Available
Issue Subscription Filter: Oozie Patch Available (111 issues) Subscriber: ooziedaily Key Summary OOZIE-3129 Fix test TestConfigurationService.testOozieConfig https://issues-test.apache.org/jira/browse/OOZIE-3129 OOZIE-3127 Remove redundant check for user https://issues-test.apache.org/jira/browse/OOZIE-3127 OOZIE-3126 Add option to allow list of users to access system config https://issues-test.apache.org/jira/browse/OOZIE-3126 OOZIE-3121 bump all maven plugins to latest versions https://issues-test.apache.org/jira/browse/OOZIE-3121 OOZIE-3118 fix for error: self-closing element not allowed https://issues-test.apache.org/jira/browse/OOZIE-3118 OOZIE-3117 fix for warning: no @throws for https://issues-test.apache.org/jira/browse/OOZIE-3117 OOZIE-3116 fix for warning: no description for @throws https://issues-test.apache.org/jira/browse/OOZIE-3116 OOZIE-3115 fix for javadoc warning: empty tag https://issues-test.apache.org/jira/browse/OOZIE-3115 OOZIE-3113 Retry for ZK lock release https://issues-test.apache.org/jira/browse/OOZIE-3113 OOZIE-3112 SparkConfigrationService overwrites properties provided via --properties-file option in SparkAction https://issues-test.apache.org/jira/browse/OOZIE-3112 OOZIE-3105 testJMXInstrumentation from the org.apache.oozie.util.TestMetricsInstrumentation class is flaky https://issues-test.apache.org/jira/browse/OOZIE-3105 OOZIE-3094 fix for grammar mistake https://issues-test.apache.org/jira/browse/OOZIE-3094 OOZIE-3091 Oozie Sqoop Avro Import fails with "java.lang.NoClassDefFoundError: org/apache/avro/mapred/AvroWrapper" https://issues-test.apache.org/jira/browse/OOZIE-3091 OOZIE-3083 Make improved version Info backward compatible https://issues-test.apache.org/jira/browse/OOZIE-3083 OOZIE-3071 Oozie 4.3 Spark sharelib ueses a different version of commons-lang3 than Spark 2.2.0 https://issues-test.apache.org/jira/browse/OOZIE-3071 OOZIE-3063 Sanitizing variables that are part of openjpa.ConnectionProperties https://issues-test.apache.org/jira/browse/OOZIE-3063 OOZIE-3062 Set HADOOP_CONF_DIR for spark action https://issues-test.apache.org/jira/browse/OOZIE-3062 OOZIE-3002 address findbugs errors in client lib https://issues-test.apache.org/jira/browse/OOZIE-3002 OOZIE-2975 code clean up in pig sharelib, replace Exception with more explicit, add try with resources, StringBuilder instead of StringBuffer https://issues-test.apache.org/jira/browse/OOZIE-2975 OOZIE-2973 Make sure Oozie works with Hadoop 3 https://issues-test.apache.org/jira/browse/OOZIE-2973 OOZIE-2969 Drop support for Java 1.7 https://issues-test.apache.org/jira/browse/OOZIE-2969 OOZIE-2957 Documentation states that starting a coordinator is possible https://issues-test.apache.org/jira/browse/OOZIE-2957 OOZIE-2956 Fix Findbugs warnings related to reliance on default encoding in oozie-core https://issues-test.apache.org/jira/browse/OOZIE-2956 OOZIE-2955 Fix Findbugs warnings related to reliance on default encoding in oozie-client https://issues-test.apache.org/jira/browse/OOZIE-2955 OOZIE-2954 Fix Checkstyle issues in oozie-client https://issues-test.apache.org/jira/browse/OOZIE-2954 OOZIE-2953 Fix Checkstyle issues in oozie-tools https://issues-test.apache.org/jira/browse/OOZIE-2953 OOZIE-2952 Fix Findbugs warnings in oozie-sharelib-oozie https://issues-test.apache.org/jira/browse/OOZIE-2952 OOZIE-2949 Escape quotes whitespaces in Sqoop field https://issues-test.apache.org/jira/browse/OOZIE-2949 OOZIE-2942 Fix Findbugs warnings in oozie-examples https://issues-test.apache.org/jira/browse/OOZIE-2942 OOZIE-2937 Remove redundant groupId from the child pom's https://issues-test.apache.org/jira/browse/OOZIE-2937 OOZIE-2934 Fix "Exceptional return value of java.io.File.mkdirs() ignored" Findbugs error in oozie-sharelib-spark https://issues-test.apache.org/jira/browse/OOZIE-2934 OOZIE-2927 Append new line character for Hive2 query using query tag https://issues-test.apache.org/jira/browse/OOZIE-2927 OOZIE-2914 Consolidate Trim https://issues-test.apache.org/jira/browse/OOZIE-2914 OOZIE-2883 OOZIE throw the error "Missing [oozie.service.ProxyUserService.proxyuser.oozie.service.ProxyUserService.proxyuser.mr.groups] property" https://issues-test.apache.org/jira/browse/OOZIE-2883 OOZIE-2877 Oozie Git Action https://issues-test.apache.org/jira/browse/OOZIE-2877 OOZIE-2867 Timezone handling for Coordinators: emphasize "Continent/City" format https://issues-test.apache.org/jira/browse/OOZIE-2867 OOZIE-2834 ParameterVerifier logging
[jira] [Commented] (OOZIE-2872) Address backward compatibility issue introduced by OOZIE-2748
[
https://issues.apache.org/jira/browse/OOZIE-2872?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16284537#comment-16284537
]
Satish Subhashrao Saley commented on OOZIE-2872:
Cherry-picked from master to branch-4.3.
> Address backward compatibility issue introduced by OOZIE-2748
> -
>
> Key: OOZIE-2872
> URL: https://issues.apache.org/jira/browse/OOZIE-2872
> Project: Oozie
> Issue Type: Bug
>Reporter: Peter Bacsko
>Assignee: Peter Bacsko
> Fix For: 5.0.0b1, 4.3.1
>
> Attachments: OOZIE-2872-001.patch, OOZIE-2872-002.patch
>
>
> OOZIE-2748 introduced a change which eliminates {{null}} arguments in certain
> actions. For example if the argument list in a Java action contains
> {{}} then Oozie used to pass {{null}} in the args[] array to the
> {{main()}} method.
> After OOZIE-2748, {{null}} elements are removed. However, there are Oozie
> users who actually rely on this behaviour and now their workflows are broken.
> We should add a property which tells Oozie whether nulls should be inserted
> or not.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Updated] (OOZIE-2872) Address backward compatibility issue introduced by OOZIE-2748
[
https://issues.apache.org/jira/browse/OOZIE-2872?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Satish Subhashrao Saley updated OOZIE-2872:
---
Fix Version/s: 4.3.1
> Address backward compatibility issue introduced by OOZIE-2748
> -
>
> Key: OOZIE-2872
> URL: https://issues.apache.org/jira/browse/OOZIE-2872
> Project: Oozie
> Issue Type: Bug
>Reporter: Peter Bacsko
>Assignee: Peter Bacsko
> Fix For: 5.0.0b1, 4.3.1
>
> Attachments: OOZIE-2872-001.patch, OOZIE-2872-002.patch
>
>
> OOZIE-2748 introduced a change which eliminates {{null}} arguments in certain
> actions. For example if the argument list in a Java action contains
> {{}} then Oozie used to pass {{null}} in the args[] array to the
> {{main()}} method.
> After OOZIE-2748, {{null}} elements are removed. However, there are Oozie
> users who actually rely on this behaviour and now their workflows are broken.
> We should add a property which tells Oozie whether nulls should be inserted
> or not.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Updated] (OOZIE-3028) Oozie Pig Action fails with no python dependencies
[
https://issues.apache.org/jira/browse/OOZIE-3028?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Satish Subhashrao Saley updated OOZIE-3028:
---
Cherry-picked from master to branch-4.3.
> Oozie Pig Action fails with no python dependencies
> --
>
> Key: OOZIE-3028
> URL: https://issues.apache.org/jira/browse/OOZIE-3028
> Project: Oozie
> Issue Type: Bug
> Components: build
>Affects Versions: 4.3.0
>Reporter: Prabhu Joseph
>Assignee: Artem Ervits
> Labels: pig
> Fix For: 5.0.0b1, 4.3.1
>
> Attachments: OOZIE-3028-0.patch, OOZIE-3028-1.patch
>
>
> Oozie Pig Action with Python script fails with below as the pig sharelib
> jython-2.5.3.jar does not have python dependencies. We should include
> jython-standalone jar into pig sharelib which has the right dependencies.
> {code}
> Caused by: java.lang.NoSuchFieldError: __code__
> at
> org.apache.pig.scripting.jython.JythonFunction.(JythonFunction.java:57)
> ... 51 more
> {code}
> # /usr/hdp/current/oozie-server/bin/oozie admin -shareliblist pig |grep
> jython
>
> hdfs://mycluster/user/oozie/share/lib/lib_20170106054154/pig/jython-2.5.3.jar
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Updated] (OOZIE-2777) Config-default.xml longer than 64k results in java.io.UTFDataFormatException
[ https://issues.apache.org/jira/browse/OOZIE-2777?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Satish Subhashrao Saley updated OOZIE-2777: --- Cherry-picked from master to branch-4.3. > Config-default.xml longer than 64k results in java.io.UTFDataFormatException > - > > Key: OOZIE-2777 > URL: https://issues.apache.org/jira/browse/OOZIE-2777 > Project: Oozie > Issue Type: Bug >Reporter: Peter Cseh >Assignee: Peter Cseh > Fix For: trunk, 4.3.1 > > Attachments: OOZIE-2777-1.patch, OOZIE-2777-10.patch, > OOZIE-2777-11.patch, OOZIE-2777-12.patch, OOZIE-2777-13.patch, > OOZIE-2777-14.patch, OOZIE-2777-2.patch, OOZIE-2777-3.patch, > OOZIE-2777-4.patch, OOZIE-2777-5.patch, OOZIE-2777-7.patch, > OOZIE-2777-8.patch, OOZIE-2777-9.patch > > > OOZIE-1642 introduced a workaround for DataOutputStream's 64k limit for > Strings. > We've encountered the same error when a config-default.xml is larger than the > limit. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (OOZIE-2863) SLACalculatorMemory.loadOnRestart causing delay in server start
[ https://issues.apache.org/jira/browse/OOZIE-2863?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Satish Subhashrao Saley updated OOZIE-2863: --- Cherry-picked from master to branch-4.3. > SLACalculatorMemory.loadOnRestart causing delay in server start > --- > > Key: OOZIE-2863 > URL: https://issues.apache.org/jira/browse/OOZIE-2863 > Project: Oozie > Issue Type: Bug >Reporter: Purshotam Shah >Assignee: Purshotam Shah > Fix For: 4.3.1 > > Attachments: OOZIE-2863-V2.patch > > > LoadOnRestart load all job and tries to calculate SLA status. Based on job > and SLA status, it moves job into historyMap or slaMap. > Mostly, the slowness is caused by calculating SLA status (since it needs to > acquire ZK lock and make DB calls for each SLA job modified in last seven > days). > LoadOnRestart should only populate slaMap entry, and periodic SLA check > should take care of calculating SLA status. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (OOZIE-2799) Setting log location for spark sql on hive
[
https://issues.apache.org/jira/browse/OOZIE-2799?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Satish Subhashrao Saley updated OOZIE-2799:
---
Cherry-picked from master to branch-4.3.
> Setting log location for spark sql on hive
> --
>
> Key: OOZIE-2799
> URL: https://issues.apache.org/jira/browse/OOZIE-2799
> Project: Oozie
> Issue Type: Bug
>Reporter: Satish Subhashrao Saley
>Assignee: Satish Subhashrao Saley
> Fix For: 4.3.1
>
> Attachments: OOZIE-2799-1.patch
>
>
> For HiveAction, Oozie added {{hive.querylog.location}} and
> {{hive.exec.local.scratchdir}} inside hive-site.xml as part of OOZIE-2380. We
> need to do the same for Spark action.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Updated] (OOZIE-2790) log4j configuration is not passed to spark executors
[
https://issues.apache.org/jira/browse/OOZIE-2790?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Satish Subhashrao Saley updated OOZIE-2790:
---
Cherry-picked from master to branch-4.3.
> log4j configuration is not passed to spark executors
>
>
> Key: OOZIE-2790
> URL: https://issues.apache.org/jira/browse/OOZIE-2790
> Project: Oozie
> Issue Type: Bug
>Reporter: Satish Subhashrao Saley
>Assignee: Satish Subhashrao Saley
> Fix For: 4.3.1
>
> Attachments: OOZIE-2790-1.patch
>
>
> We prepare the {{spark-log4j.properties}} properties file but do not pass it
> over the spark executors in --files option. Therefore the log settings do get
> applied to the executor logs.
> Solution- pass the {{spark-log4j.properties}} file with --files option.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Updated] (OOZIE-2748) NPE in LauncherMapper.printArgs()
[
https://issues.apache.org/jira/browse/OOZIE-2748?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Satish Subhashrao Saley updated OOZIE-2748:
---
Cherry-picked from master to branch-4.3.
> NPE in LauncherMapper.printArgs()
> -
>
> Key: OOZIE-2748
> URL: https://issues.apache.org/jira/browse/OOZIE-2748
> Project: Oozie
> Issue Type: Bug
> Components: core
>Affects Versions: 4.2.0
>Reporter: Peter Bacsko
>Assignee: Peter Bacsko
> Fix For: 5.0.0b1, 4.3.1
>
> Attachments: OOZIE-2748-001.patch, OOZIE-2748-002.patch,
> OOZIE-2748-003.patch, OOZIE-2748-004.patch, OOZIE-2748-005.patch,
> OOZIE-2748-006.patch
>
>
> If we define argument for the Spark action like this
> {code}
>
> ${jobTracker}
> ${nameNode}
>
>
> oozie.use.system.libpath
> true
>
>
>
>
>
>
> yarn-cluster
> cluster
> Something
> package.of.my.Class
> ${nameNode}/myjar.jar
> <--- cause of the NPE
>
> {code}
> then we get a NullPointerException in LauncherMapper:
> {code}
> Failing Oozie Launcher, Main class
> [org.apache.oozie.action.hadoop.SparkMain], exception invoking main(), null
> java.lang.NullPointerException
> at
> org.apache.oozie.action.hadoop.LauncherMapper.printArgs(LauncherMapper.java:627)
> at
> org.apache.oozie.action.hadoop.LauncherMapper.map(LauncherMapper.java:212)
> at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:54)
> ...
> {code}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Updated] (OOZIE-2886) Ensure consistent versioning of hadoop jars in sharelibs
[
https://issues.apache.org/jira/browse/OOZIE-2886?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Satish Subhashrao Saley updated OOZIE-2886:
---
Cherry-picked from master to branch-4.3.
> Ensure consistent versioning of hadoop jars in sharelibs
>
>
> Key: OOZIE-2886
> URL: https://issues.apache.org/jira/browse/OOZIE-2886
> Project: Oozie
> Issue Type: Bug
> Components: build
>Affects Versions: 5.0.0
>Reporter: Robert Kanter
>Assignee: Artem Ervits
> Fix For: 5.0.0b1, 4.3.1
>
> Attachments: OOZIE-2886-0.patch, OOZIE-2886-0.patch,
> OOZIE-2886-1.patch, OOZIE-2886-2.patch, OOZIE-2886-3.patch,
> OOZIE-2886-4.patch, OOZIE-2886-5.patch, OOZIE-2886-6.patch
>
>
> The hive and hive2 sharelibs are pulling in Hadoop jars - they shouldn't be.
> Also, we should ensure that if you specify {{-Dhadoop.version=foo}} (e.g.
> 2.7.0), that all Hadoop jars end up as that version.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Updated] (OOZIE-3036) Spark 2.2.0 support: tell Spark not to get any delegation tokens
[
https://issues.apache.org/jira/browse/OOZIE-3036?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Satish Subhashrao Saley updated OOZIE-3036:
---
Cherry-picked from master to branch-4.3.
> Spark 2.2.0 support: tell Spark not to get any delegation tokens
>
>
> Key: OOZIE-3036
> URL: https://issues.apache.org/jira/browse/OOZIE-3036
> Project: Oozie
> Issue Type: Bug
>Affects Versions: 4.3.0
> Environment: * [*Spark
> 2.2.0*|https://github.com/apache/spark/tree/branch-2.2]
> * [*Hadoop 3 alpha
> 4*|https://github.com/apache/hadoop/tree/branch-3.0.0-alpha4]
>Reporter: Andras Piros
>Assignee: Andras Piros
> Fix For: 5.0.0b1, 4.3.1
>
> Attachments: OOZIE-3036.001.patch
>
>
> For Spark 2.2.0 along w/ Hadoop 3 alpha 4, Spark uses another set of
> arguments that ensure Spark won't get any delegation tokens:
> * {{spark.yarn.security.credentials.hadoopfs.enabled}}
> * {{spark.yarn.security.credentials.hive.enabled}}
> * {{spark.yarn.security.credentials.hbase.enabled}}
> For Spark 1.6.0 along w/ Hadoop 3 alpha 4, we need following argument to set
> extra:
> * {{spark.yarn.security.tokens.hadoopfs.enabled}}
> As we can't tell runtime which Spark / Hadoop versions is the Oozie Spark
> action running on, it's best to use the union of those arguments.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Updated] (OOZIE-2825) Custom Authentication doc page is not well formatted
[ https://issues.apache.org/jira/browse/OOZIE-2825?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Satish Subhashrao Saley updated OOZIE-2825: --- Cherry-picked from master to branch-4.3. > Custom Authentication doc page is not well formatted > > > Key: OOZIE-2825 > URL: https://issues.apache.org/jira/browse/OOZIE-2825 > Project: Oozie > Issue Type: Sub-task >Reporter: Laszlo Zeke >Assignee: Jan Hentschel >Priority: Minor > Fix For: 5.0.0b1, 4.3.1 > > Attachments: OOZIE-2825-1.patch, OOZIE-2825-2.patch, > OOZIE-2825-3.patch, OOZIE-2825-4.patch > > > At line 30 there's a verbatim tag staring but don't end. > Also at line 45 a paragraph is staring but it does not make a lot of sense > with a code sample seems to be cut in half. > All in ENG_Custom_Authentication.twiki -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (OOZIE-2803) Mask passwords when printing out configs/args in MapReduceMain and SparkMain
[
https://issues.apache.org/jira/browse/OOZIE-2803?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Satish Subhashrao Saley updated OOZIE-2803:
---
Cherry-picked from master to branch-4.3.
> Mask passwords when printing out configs/args in MapReduceMain and SparkMain
>
>
> Key: OOZIE-2803
> URL: https://issues.apache.org/jira/browse/OOZIE-2803
> Project: Oozie
> Issue Type: Bug
> Components: action
>Reporter: Peter Bacsko
>Assignee: Peter Bacsko
>Priority: Critical
> Fix For: 5.0.0b1, 4.3.1
>
> Attachments: OOZIE-2803-001.patch, OOZIE-2803-002.patch,
> OOZIE-2803-003.patch, OOZIE-2803-004.patch, OOZIE-2803-005.patch
>
>
> Sometimes passwords are displayed in both MapReduce and Spark action.
> *MapReduce*: when using {{HADOOP_CREDSTORE_PASSWORD}}, it must be passed to
> some Hadoop-specific config values, like {{mapred.child.env}}. This is easy
> to fix because we already have a method {{logMasking()}} where you can define
> a {{maskSet}} which contains a list of property keys to be masked.
> Note that this is not necessarily the perfect solution, since you can pass
> multiple env. vars separated by a colon, and only the password specific parts
> should be masked. But we need a working solution relatively quickly - later
> we can enhance this, eg. we can re-use {{PasswordMasker}} in some way (right
> now it only works with {{Map}}).
> *Spark*: for Spark, we have to pass passwords like this:
> {{--conf spark.executorEnv.HADOOP_CREDSTORE_PASSWORD= password>}}
> The Spark arguments are printed in {{SparkMain.run()}}. There is already a
> code in {{LauncherMapper.printArgs()}} which deals with situations like this,
> but it's not perfect because it only works if the args look something like
> {{--password pwd123}}. So if a single arg contains a password, it doesn't
> work, therefore we need a different approach here.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Updated] (OOZIE-2811) Add support for filtering out properties from SparkConfigurationService
[
https://issues.apache.org/jira/browse/OOZIE-2811?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Satish Subhashrao Saley updated OOZIE-2811:
---
Cherry-picked from master to branch-4.3.
> Add support for filtering out properties from SparkConfigurationService
> ---
>
> Key: OOZIE-2811
> URL: https://issues.apache.org/jira/browse/OOZIE-2811
> Project: Oozie
> Issue Type: Bug
>Reporter: Peter Cseh
>Assignee: Peter Cseh
> Fix For: 5.0.0b1, 4.3.1
>
> Attachments: OOZIE-2811-001.patch, OOZIE-2811-002.patch,
> OOZIE-2811-003.patch, OOZIE-2811-004.patch, OOZIE-2811-005.patch,
> OOZIE-2811-amend.patch
>
>
> There is support for filtering out
> [spark.yarn.jar|https://github.com/apache/oozie/blob/master/core/src/main/java/org/apache/oozie/service/SparkConfigurationService.java#L89],
> but it's changed to spark.yarn.jars in Spark2.
> We should add a general blacklist of properties instead of the current
> boolean of {{spark.configurations.ignore.spark.yarn.jar}}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Updated] (OOZIE-2802) Spark action failure on Spark 2.1.0 due to duplicate sharelibs
[
https://issues.apache.org/jira/browse/OOZIE-2802?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Satish Subhashrao Saley updated OOZIE-2802:
---
Cherry-picked from master to branch-4.3.
> Spark action failure on Spark 2.1.0 due to duplicate sharelibs
> --
>
> Key: OOZIE-2802
> URL: https://issues.apache.org/jira/browse/OOZIE-2802
> Project: Oozie
> Issue Type: Bug
>Reporter: Satish Subhashrao Saley
>Assignee: Peter Cseh
> Fix For: 5.0.0b1, 4.3.1
>
> Attachments: OOZIE-2802-001.patch, OOZIE-2802-002.patch,
> OOZIE-2802-003.patch, spark-2.1-failure-launcherlib.txt
>
>
> There are some same jars in launcher lib and sharelib. These jars are causing
> failures in spark action for spark 2.1.0. SPARK-18099 added an exception to
> be thrown if same file gets added multiple times to distributed cache.
> {code}
> $ hdfs dfs -ls /user/saley/share/lib/launcher_20170216150752/oozie
> 17/02/16 15:14:22 WARN util.NativeCodeLoader: Unable to load native-hadoop
> library for your platform... using builtin-java classes where applicable
> Found 3 items
> -rwxr-xr-x 1 saley supergroup 11871 2017-02-16 15:07
> /user/saley/share/lib/launcher_20170216150752/oozie/oozie-hadoop-utils-hadoop-2-4.4.0-SNAPSHOT.jar
> -rwxr-xr-x 1 saley supergroup 17514 2017-02-16 15:07
> /user/saley/share/lib/launcher_20170216150752/oozie/oozie-sharelib-hcatalog-4.4.0-SNAPSHOT.jar
> -rwxr-xr-x 1 saley supergroup 55092 2017-02-16 15:07
> /user/saley/share/lib/launcher_20170216150752/oozie/oozie-sharelib-oozie-4.4.0-SNAPSHOT.jar
> $ hdfs dfs -ls /user/saley/share/lib/lib_20170216150706/oozie
> 17/02/16 15:14:39 WARN util.NativeCodeLoader: Unable to load native-hadoop
> library for your platform... using builtin-java classes where applicable
> Found 3 items
> -rw-r--r-- 1 saley supergroup 16046 2017-02-16 15:07
> /user/saley/share/lib/lib_20170216150706/oozie/json-simple-1.1.jar
> -rw-r--r-- 1 saley supergroup 11871 2017-02-16 15:07
> /user/saley/share/lib/lib_20170216150706/oozie/oozie-hadoop-utils-hadoop-2-4.4.0-SNAPSHOT.jar
> -rw-r--r-- 1 saley supergroup 55092 2017-02-16 15:07
> /user/saley/share/lib/lib_20170216150706/oozie/oozie-sharelib-oozie-4.4.0-SNAPSHOT.jar
> {code}
> The file with same name can't be in twice in any of *--files , --archives,
> --jars* (ie one file in both --files and --jars) and can't be in --files
> twice and can't be in --archives twice. It is allowed to be in --jars twice.
> Possible solutions:
> 1. While specifying the --files options, filter out the duplicate jars
> 2. Separate out jars and files. Use --jars options to distribute jars and
> --files for files. Also remember to remove duplicates from --files, and make
> sure that there is nothing common in --jars and --files.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Updated] (OOZIE-2690) OOZIE NPE while executing kill()
[ https://issues.apache.org/jira/browse/OOZIE-2690?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Satish Subhashrao Saley updated OOZIE-2690: --- Cherry-picked from master to branch-4.3. > OOZIE NPE while executing kill() > > > Key: OOZIE-2690 > URL: https://issues.apache.org/jira/browse/OOZIE-2690 > Project: Oozie > Issue Type: Bug >Reporter: Cheng Xu >Assignee: Abhishek Bafna > Fix For: 5.0.0b1, 4.3.1 > > Attachments: OOZIE-2690-00.patch > > > ION[027-160917110703953-oozie-oozi-W@java2] Exception while executing > kill(). Error Code [NullPointerException], Message[NullPointerException: null] > org.apache.oozie.action.ActionExecutorException: NullPointerException: null > at > org.apache.oozie.action.ActionExecutor.convertException(ActionExecutor.java:446) > at > org.apache.oozie.action.hadoop.JavaActionExecutor.kill(JavaActionExecutor.java:1632) > at > org.apache.oozie.command.wf.ActionKillXCommand.execute(ActionKillXCommand.java:146) > at > org.apache.oozie.command.wf.ActionKillXCommand.execute(ActionKillXCommand.java:62) > at org.apache.oozie.command.XCommand.call(XCommand.java:287) > at > org.apache.oozie.service.CallableQueueService$CompositeCallable.call(CallableQueueService.java:331) > at > org.apache.oozie.service.CallableQueueService$CompositeCallable.call(CallableQueueService.java:260) > at java.util.concurrent.FutureTask.run(FutureTask.java:266) > at > org.apache.oozie.service.CallableQueueService$CallableWrapper.run(CallableQueueService.java:178) > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) > at java.lang.Thread.run(Thread.java:745) > Caused by: java.lang.NullPointerException > at org.apache.hadoop.mapreduce.TypeConverter.toYarn(TypeConverter.java:79) > at > org.apache.hadoop.mapred.ClientServiceDelegate.(ClientServiceDelegate.java:120) > at org.apache.hadoop.mapred.ClientCache.getClient(ClientCache.java:68) > at org.apache.hadoop.mapred.YARNRunner.getJobStatus(YARNRunner.java:573) > at org.apache.hadoop.mapreduce.Cluster.getJob(Cluster.java:184) > at org.apache.hadoop.mapred.JobClient$2.run(JobClient.java:601) > at org.apache.hadoop.mapred.JobClient$2.run(JobClient.java:599) > at java.security.AccessController.doPrivileged(Native Method) > at javax.security.auth.Subject.doAs(Subject.java:422) > at > org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1724) > at > org.apache.hadoop.mapred.JobClient.getJobUsingCluster(JobClient.java:599) > at org.apache.hadoop.mapred.JobClient.getJobInner(JobClient.java:609) > at org.apache.hadoop.mapred.JobClient.getJob(JobClient.java:639) > at > org.apache.oozie.action.hadoop.JavaActionExecutor.getJobClientWithRetries(JavaActionExecutor.java:1432) > at > org.apache.oozie.action.hadoop.JavaActionExecutor.getRunningJob(JavaActionExecutor.java:1420) > at > org.apache.oozie.action.hadoop.JavaActionExecutor.kill(JavaActionExecutor.java:1623) > ... 10 more -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (OOZIE-2787) Oozie distributes application jar twice making the spark job fail
[
https://issues.apache.org/jira/browse/OOZIE-2787?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Satish Subhashrao Saley updated OOZIE-2787:
---
Cherry-picked from master to branch-4.3.
> Oozie distributes application jar twice making the spark job fail
> -
>
> Key: OOZIE-2787
> URL: https://issues.apache.org/jira/browse/OOZIE-2787
> Project: Oozie
> Issue Type: Bug
>Reporter: Satish Subhashrao Saley
>Assignee: Satish Subhashrao Saley
> Fix For: 4.3.1
>
> Attachments: OOZIE-2787-1.patch, OOZIE-2787-2.patch,
> OOZIE-2787-3.patch, OOZIE-2787-4.patch, OOZIE-2787-5.patch,
> OOZIE-2787-amend-1.patch, OOZIE-2787-amend-2.patch, OOZIE-2787-amend-3.patch,
> OOZIE-2787-amend-4.patch, OOZIE-2787-amend-5.patch
>
>
> Oozie adds the application jar to the list of files to be uploaded to
> distributed cache. Since this gets added twice, the job fails. This is
> observed from spark 2.1.0 which introduces a check for same file and fails
> the job.
> {code}
> --master
> yarn
> --deploy-mode
> cluster
> --name
> oozieSparkStarter
> --class
> ScalaWordCount
> --queue
> default
> --conf
> spark.executor.extraClassPath=$PWD/*
> --conf
> spark.driver.extraClassPath=$PWD/*
> --conf
> spark.executor.extraJavaOptions=-Dlog4j.configuration=spark-log4j.properties
> --conf
> spark.driver.extraJavaOptions=-Dlog4j.configuration=spark-log4j.properties
> --conf
> spark.yarn.security.tokens.hive.enabled=false
> --conf
> spark.yarn.security.tokens.hbase.enabled=false
> --files
> hdfs://mycluster.com/user/saley/oozie/apps/sparkapp/lib/spark-example.jar
> --properties-file
> spark-defaults.conf
> --verbose
> spark-example.jar
> samplefile.txt
> output
> {code}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Updated] (OOZIE-2654) Zookeeper dependent services should not depend on Connectionstate to be valid before cleaning up
[
https://issues.apache.org/jira/browse/OOZIE-2654?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Satish Subhashrao Saley updated OOZIE-2654:
---
Cherry-picked from master to branch-4.3.
> Zookeeper dependent services should not depend on Connectionstate to be valid
> before cleaning up
>
>
> Key: OOZIE-2654
> URL: https://issues.apache.org/jira/browse/OOZIE-2654
> Project: Oozie
> Issue Type: Bug
> Components: HA
>Affects Versions: 4.2.0
>Reporter: Venkat Ranganathan
>Assignee: Venkat Ranganathan
> Fix For: 5.0.0b1, 4.3.1
>
> Attachments: OOZIE-2654.diff
>
>
> Currently in ZKUtils, ZKLocks and ZKJobsConcurrency services, we don't
> properly teardown the zookeeper connections when the callback was not
> received from zookeeper to change the connection state.
> We can get into this situation if the ZK session for example was closed by ZK
> before any callback was received to update the connection state. This can
> cause the oozie server in a HA mode to not terminate with one or more
> sockets in close_wait state.
> Here is an instance of this issue
> From the network connections, we have one connection still on close_wait with
> indefinite wait.
> {quote} tcp6 143 0 x.x.x.1:46710 x.x.x.2:2181 CLOSE_WAIT 4688/java off
> (0.00/0/0)
> {quote}
> From the zookeeper logs,
> {quote}
> 016-08-18 20:45:29,921 - INFO
> NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:ZooKeeperServer@868 - Client
> attempting to establish new session at /x.x.x.1:46710 2016-08-18 20:45:29,926
> - INFO CommitProcessor:1:ZooKeeperServer@617 - Established session
> 0x1569f576843000e with negotiated timeout 4 for client /x.x.x.1:46710
> {quote}
> and later
> {quote}
> 2016-08-18 20:46:34,008 - INFO CommitProcessor:1:NIOServerCnxn@1007 - Closed
> socket connection for client /x.x.x.1:46710 which had sessionid
> 0x1569f576843000e
> {quote}
> The fix is to not check for the connectionstate during service destroy and
> teardown the zk connections.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Updated] (OOZIE-2862) Coord change command doesn't change job to running if job was killed without creating any actions
[
https://issues.apache.org/jira/browse/OOZIE-2862?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Satish Subhashrao Saley updated OOZIE-2862:
---
Cherry-picked from master to branch-4.3.
> Coord change command doesn't change job to running if job was killed without
> creating any actions
> -
>
> Key: OOZIE-2862
> URL: https://issues.apache.org/jira/browse/OOZIE-2862
> Project: Oozie
> Issue Type: Bug
>Reporter: Purshotam Shah
>Assignee: Purshotam Shah
> Fix For: 5.0.0b1, 4.3.1
>
> Attachments: OOZIE-2862-V1.patch, OOZIE-2862-V2.patch
>
>
> {code:title=CoordChangeXCommand.java}
> if (jobStatus != null) {
> coordJob.setStatus(jobStatus);
> LOG.info("Coord status is changed to " + jobStatus + " from " +
> prevStatus);
> if (jobStatus.equals(CoordinatorJob.Status.RUNNING)) {
> coordJob.setPending();
> if (coordJob.getNextMaterializedTime() != null
> &&
> coordJob.getEndTime().after(coordJob.getNextMaterializedTime())) {
> coordJob.resetDoneMaterialization();
> }
> } else if (jobStatus.equals(CoordinatorJob.Status.IGNORED)) {
> coordJob.resetPending();
> coordJob.setDoneMaterialization();
> }
> }
> {code}
> If the job was killed without creating any action, then
> {{coordJob.getNextMaterializedTime()}} will be null and change command will
> not reset done materialization.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Updated] (OOZIE-2786) Pass Oozie workflow ID and settings to Spark application configuration
[ https://issues.apache.org/jira/browse/OOZIE-2786?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Satish Subhashrao Saley updated OOZIE-2786: --- Cherry-picked from master to branch-4.3. > Pass Oozie workflow ID and settings to Spark application configuration > -- > > Key: OOZIE-2786 > URL: https://issues.apache.org/jira/browse/OOZIE-2786 > Project: Oozie > Issue Type: Improvement > Components: action >Reporter: Xiaobin Zheng >Assignee: Xiaobin Zheng >Priority: Critical > Fix For: 5.0.0b1, 4.3.1 > > Attachments: OOZIE-2786-1.patch, OOZIE-2786-2.patch, > OOZIE-2786-3.patch > > > When using Oozie to launch Spark applications, the Oozie work flow ID and etc > do not show up in Spark Application's Environment settings. There is no Spark > application ID exposed from Oozie side as well about the Spark applications > Oozie launches. > When looking at applications in Spark History Server, it is hard to figure > out which Oozie workflow launched it. This makes debugging the workflow hard. > We want to be able to let Spark History server display the Oozie workflow ID > in the application's 'Environment' settings. > For reference, oozie properties that can be passed to Spark are: > oozie.job.id > oozie.HadoopAccessorService.created > oozie.action.spark.setup.hadoop.conf.dir > oozie.child.mapreduce.job.tags > oozie.action.id > oozie.action.rootlogger.log.level -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (OOZIE-2792) Hive2 action is not parsing Spark application ID from log file properly when Hive is on Spark
[ https://issues.apache.org/jira/browse/OOZIE-2792?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Satish Subhashrao Saley updated OOZIE-2792: --- Cherry-picked from master to branch-4.3. > Hive2 action is not parsing Spark application ID from log file properly when > Hive is on Spark > - > > Key: OOZIE-2792 > URL: https://issues.apache.org/jira/browse/OOZIE-2792 > Project: Oozie > Issue Type: Bug > Components: action >Reporter: Xiaobin Zheng >Assignee: Xiaobin Zheng >Priority: Minor > Fix For: 5.0.0b1, 4.3.1 > > Attachments: OOZIE-2792-1.patch, OOZIE-2792-2.patch, > OOZIE-2792-3.patch, OOZIE-2792-4-amendment.patch, OOZIE-2792-4.patch > > > When Hive2 is on Spark, hive action is not able to parse Spark application ID > from log file as 'externalChildID' like Spark/MR actions. This makes it hard > to tell which job hive launches from Oozie server for a particular workflow. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (OOZIE-2747) README.txt is out of date
[ https://issues.apache.org/jira/browse/OOZIE-2747?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Satish Subhashrao Saley updated OOZIE-2747: --- Cherry-picked from master to branch-4.3. > README.txt is out of date > - > > Key: OOZIE-2747 > URL: https://issues.apache.org/jira/browse/OOZIE-2747 > Project: Oozie > Issue Type: Sub-task > Components: docs >Affects Versions: 4.0.0 >Reporter: Robert Kanter >Assignee: Jan Hentschel > Fix For: 5.0.0b1, 4.3.1 > > Attachments: OOZIE-2747-1.patch > > > The README.txt file is out of date. The main text may require some updating > as it hasn't been touched in a while, but the really obvious problems here > are that it refers to > "http://oozie.apache.org/docs/3.2.0-incubating/DG_QuickStart.html; > (3.2.0-incubating version) and says "This version of Oozie was primarily > tested against Hadoop 0.20.205.x. This will not work on earlier versions of > Hadoop such as 0.20.x. and 0.21." which way off. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (OOZIE-2984) Parse spark-defaults.conf values with spaces without needing the quotes
[
https://issues.apache.org/jira/browse/OOZIE-2984?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Satish Subhashrao Saley updated OOZIE-2984:
---
Cherry-picked from master to branch-4.3.
> Parse spark-defaults.conf values with spaces without needing the quotes
> ---
>
> Key: OOZIE-2984
> URL: https://issues.apache.org/jira/browse/OOZIE-2984
> Project: Oozie
> Issue Type: Bug
>Affects Versions: 4.3.0
>Reporter: Fahd Siddiqui
>Assignee: Andras Piros
> Fix For: 5.0.0b1, 4.3.1
>
> Attachments: OOZIE-2984.001.patch, OOZIE-2984.002.patch,
> OOZIE-2984.003.patch, OOZIE-2984.004.patch, OOZIE-2984.005.patch
>
>
> Oozie requires multiple java opts to be enclosed in quotes. For example:
> {code}
> spark.driver.extraJavaOptions="-Xmn2703m -XX:SurvivorRatio=2
> -XX:ParallelGCThreads=20"
> {code}
> However the above breaks spark-shell.
> {code}
> $ ./spark2-shell
> Invalid initial young generation size: -Xmn2212m -XX:SurvivorRatio=2
> -XX:ParallelGCThreads=20
> Error: Could not create the Java Virtual Machine.
> Error: A fatal exception has occurred. Program will exit.
> {code}
> To fix spark-shell, we have to remove the quotes, which errors out Oozie:
> {code}
> Error: Unrecognized option '-XX:SurvivorRatio=2'
> ...
> --conf
> spark.executor.extraJavaOptions=-Xmn2703m
> -Dlog4j.configuration=spark-log4j.properties
> -XX:SurvivorRatio=2
> {code}
> Oozie should be able to parse spark-defaults.conf values with spaces without
> needing the quotes.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Updated] (OOZIE-2923) Improve Spark options parsing
[
https://issues.apache.org/jira/browse/OOZIE-2923?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Satish Subhashrao Saley updated OOZIE-2923:
---
Cherry-picked from master to branch-4.3.
> Improve Spark options parsing
> -
>
> Key: OOZIE-2923
> URL: https://issues.apache.org/jira/browse/OOZIE-2923
> Project: Oozie
> Issue Type: Improvement
> Components: action
>Affects Versions: 4.3.0
>Reporter: Andras Piros
>Assignee: Andras Piros
> Fix For: 5.0.0b1, 4.3.1
>
> Attachments: OOZIE-2923.001.patch, OOZIE-2923.002.patch,
> OOZIE-2923.003.patch, OOZIE-2923.004.patch, OOZIE-2923.005.patch,
> OOZIE-2923.006.patch
>
>
> There are two issues w/ Spark action's argument parsing within {{SparkMain}}
> h5. Driver and executor extra classpaths: equals sign used
> When the user specifies {{\-\-conf spark.executor.extraClassPath=XYZ}} or
> {{\-\-conf spark.driver.extraClassPath=ABC}}, the option {{\-\-conf}} will be
> added to {{sparkArgs}}. Then when the code tries to evaluate
> {{spark.executor.extraClassPath=XYZ}}, it uses special logic and set
> {{addToSparkArgs = false}}. As a result there will be a extra {{\-\-conf}} in
> the {{sparkArgs}} eventually.
> For example: {{\-\-conf spark.executor.extraClassPath=XYZ \-\-conf
> otherProperty=ABC}} will become {{\-\-conf \-\-conf otherProperty=ABC}},
> which will cause spark job submit failure later.
> We might need to remove one prior {{\-\-conf}} in {{sparkArgs}} if the
> current evaluated {{opt}} is {{EXECUTOR_CLASSPATH}} or {{DRIVER_CLASSPATH}}.
> h5. User provided files and archives: equals sign used
> For the following workflow XML snippet:
> {code:xml}
> --files=${nameNode}/home/share/hive-site.xml --num-executors 4
> --executor-memory 7g --driver-memory 7g
> {code}
> the {{\-\-files=$\{nameNode\}/home/share/hive-site.xml}} {{opt}} will be
> placed into {{sparkArgs}} in previous Oozie version without any modification,
> because we don't have special handling for {{\-\-files}} {{opt}}.
> If the user specifies {{\-\-files=$\{nameNode\}/home/share/hive-site.xml
> --num-executor 4}}, then {{SparkMain}} code treats {{\-\-num-executor}} as a
> file path / name. That caused the issue as I described in my previous
> comment. We might need to change the handling logic for {{FILES_OPTION}} and
> {{ARCHIVES_OPTION}} to be the same to {{DRIVER_CLASSPATH_OPTION}}.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Updated] (OOZIE-2807) Oozie gets RM delegation token even for checking job status
[ https://issues.apache.org/jira/browse/OOZIE-2807?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Satish Subhashrao Saley updated OOZIE-2807: --- Cherry-picked from master to branch-4.3. > Oozie gets RM delegation token even for checking job status > --- > > Key: OOZIE-2807 > URL: https://issues.apache.org/jira/browse/OOZIE-2807 > Project: Oozie > Issue Type: Bug >Reporter: Rohini Palaniswamy >Assignee: Satish Subhashrao Saley > Fix For: 5.0.0b1, 4.3.1 > > Attachments: OOZIE-2807-1.patch, OOZIE-2807-2.patch, > OOZIE-2807-3.patch, OOZIE-2807-4.patch > > > We had one user submitting way too many workflows with single hive query - > ~3600 workflows running concurrently. Surprisingly Oozie held up well without > issues. > But [~daryn] from our hadoop team saw that the amount of delegation tokens > fetched by Oozie was very high compared to actual number of jobs submitted > and was stressing RM with the calls and also pushing it close to its memory > limits. This is because we are fetching the delegation token every time we > create a JobClient instead of only during job submission. > https://github.com/apache/oozie/blob/master/core/src/main/java/org/apache/oozie/service/HadoopAccessorService.java#L503-L519 > So for one job we fetch > 1) 1 token during submission > 2) 1 token every 5 minutes when we check status of job > 3) 1 token after the job ends to retrieve status. > 4) 1 token if we are killing the job. > So for a job running for 11 minutes, we would have fetched the token 4 times. > May be more in other cases like mapreduce where we check for end of launcher > and child job. > Only 1 out of the token (used in the job submission) will be cancelled after > job completes. Other tokens are kind of leaked and will only be cleaned up by > RM after the expiry period (24 hrs is default). This can make RM go out of > memory. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (OOZIE-2771) Allow retrieving keystore and truststore passwords from Hadoop Credential Provider
[
https://issues.apache.org/jira/browse/OOZIE-2771?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Satish Subhashrao Saley updated OOZIE-2771:
---
Cherry-picked from master to branch-4.3.
> Allow retrieving keystore and truststore passwords from Hadoop Credential
> Provider
> --
>
> Key: OOZIE-2771
> URL: https://issues.apache.org/jira/browse/OOZIE-2771
> Project: Oozie
> Issue Type: Improvement
>Reporter: Attila Sasvari
>Assignee: Attila Sasvari
> Fix For: 5.0.0b1, 4.3.1
>
> Attachments: OOZIE-2771-01.patch
>
>
> Right now passwords for keystore and truststore
> ({{oozie.https.keystore.pass}}, {{oozie.https.truststore.pass}}) are stored
> in {{oozie-site.xml}} as cleartext.
> However, Oozie could take advantage of the Hadoop Credential Provider for
> storing and retrieving that passwords similarly how the JDBC password
> ({{oozie.service.JPAService.jdbc.password}}) is handled today (see
> https://issues.apache.org/jira/browse/OOZIE-2272).
> This way keystore and truststore passwords could be masked in oozie-site.
> Note: {{ConfigurationService.getPassword}} is worth to look at.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Assigned] (OOZIE-2551) Feature request: epoch timestamp generation
[
https://issues.apache.org/jira/browse/OOZIE-2551?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Daniel Takamori reassigned OOZIE-2551:
--
Assignee: Jacob Tolar (was: Jacob Tolar)
> Feature request: epoch timestamp generation
> ---
>
> Key: OOZIE-2551
> URL: https://issues.apache.org/jira/browse/OOZIE-2551
> Project: Oozie
> Issue Type: Improvement
> Components: core
>Reporter: Jacob Tolar
>Assignee: Jacob Tolar
>Priority: Minor
> Fix For: 4.3.0
>
> Attachments: OOZIE-2551-1.patch, OOZIE-2551-2.patch,
> OOZIE-2551-3.patch
>
>
> Currently, Oozie has a coord:formatTime function for formatting time strings.
> Unfortunately, as this is backed by SimpleDateFormat, there is no way to
> produce the time in Unix-style epoch. (I.e. seconds or milliseconds since the
> epoch.)
> Please add a coord:unixTime (or similarly named) function that, given a time
> string, converts it to either seconds since the epoch or milliseconds since
> the epoch. Something like:
> {code}
> coord:unixTime(String ts, Boolean millis)
> {code}
> Requested by [~jmartell7]
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Assigned] (OOZIE-2188) Fix typos in twiki documentation
[ https://issues.apache.org/jira/browse/OOZIE-2188?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Daniel Takamori reassigned OOZIE-2188: -- Assignee: Jacob Tolar (was: Jacob Tolar) > Fix typos in twiki documentation > > > Key: OOZIE-2188 > URL: https://issues.apache.org/jira/browse/OOZIE-2188 > Project: Oozie > Issue Type: Bug > Components: docs >Reporter: Jacob Tolar >Assignee: Jacob Tolar >Priority: Trivial > Labels: documentation > Fix For: 4.2.0 > > Attachments: typos.patch > > > Fixes some minor typos in the documentation. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Comment Edited] (OOZIE-3127) Remove redundant check for user
[ https://issues.apache.org/jira/browse/OOZIE-3127?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16284501#comment-16284501 ] Satish Subhashrao Saley edited comment on OOZIE-3127 at 12/9/17 1:25 AM: - Thank you Rohini for review. Committed to master. was (Author: satishsaley): Thank you Rohini for review. Committed to trunk. > Remove redundant check for user > --- > > Key: OOZIE-3127 > URL: https://issues.apache.org/jira/browse/OOZIE-3127 > Project: Oozie > Issue Type: Bug >Reporter: Satish Subhashrao Saley >Assignee: Satish Subhashrao Saley > Fix For: 4.3.1 > > Attachments: OOZIE-3127-1.patch > > > [validateJobConfiguration(conf);|https://github.com/apache/oozie/blob/master/core/src/main/java/org/apache/oozie/servlet/BaseJobsServlet.java#L95] > is redundant because we are checking it anyways [inside > here|https://github.com/apache/oozie/blob/master/core/src/main/java/org/apache/oozie/servlet/BaseJobsServlet.java#L100] -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (OOZIE-3127) Remove redundant check for user
[ https://issues.apache.org/jira/browse/OOZIE-3127?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16284501#comment-16284501 ] Satish Subhashrao Saley commented on OOZIE-3127: Thank you Rohini for review. Committed to trunk. > Remove redundant check for user > --- > > Key: OOZIE-3127 > URL: https://issues.apache.org/jira/browse/OOZIE-3127 > Project: Oozie > Issue Type: Bug >Reporter: Satish Subhashrao Saley >Assignee: Satish Subhashrao Saley > Fix For: 4.3.1 > > Attachments: OOZIE-3127-1.patch > > > [validateJobConfiguration(conf);|https://github.com/apache/oozie/blob/master/core/src/main/java/org/apache/oozie/servlet/BaseJobsServlet.java#L95] > is redundant because we are checking it anyways [inside > here|https://github.com/apache/oozie/blob/master/core/src/main/java/org/apache/oozie/servlet/BaseJobsServlet.java#L100] -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Assigned] (OOZIE-2150) Shell launcher should print shell script
[ https://issues.apache.org/jira/browse/OOZIE-2150?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Purshotam Shah reassigned OOZIE-2150: - Assignee: Purshotam Shah (was: Jacob Tolar) > Shell launcher should print shell script > > > Key: OOZIE-2150 > URL: https://issues.apache.org/jira/browse/OOZIE-2150 > Project: Oozie > Issue Type: Bug >Reporter: Purshotam Shah >Assignee: Purshotam Shah >Priority: Minor > > Pig and Hive does this. It will be helpful in debugging. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (OOZIE-3139) Oozie validates workflow incorrectly
[
https://issues.apache.org/jira/browse/OOZIE-3139?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16283434#comment-16283434
]
Peter Bacsko commented on OOZIE-3139:
-
+1
thanks for the contribution [~dionusos], committed to master!
> Oozie validates workflow incorrectly
>
>
> Key: OOZIE-3139
> URL: https://issues.apache.org/jira/browse/OOZIE-3139
> Project: Oozie
> Issue Type: Bug
> Components: core
>Affects Versions: 4.3.0
>Reporter: Denes Bodo
>Assignee: Denes Bodo
>Priority: Critical
> Fix For: 5.0.0b1
>
> Attachments: OOZIE-3139.001.patch, OOZIE-3139.002.patch,
> OOZIE-3139.003.patch, OOZIE-3139.004.patch, OOZIE-3139.005.patch,
> exception.log, workflow.pdf
>
>
> We found that after OOZIE-1978 the attached workflow becomes invalid. Aa we
> found out the following line brings the problem:
> https://github.com/apache/oozie/commit/8e9b9042b3270dc5ff975c44a5c977fcc41250e4#diff-a38ae78886b7c3afb711d32cfb4dbbbdR267
> . The parameter "okPath" changed from okPath to true.
> The following workflow is problematic:
> {noformat}
>
>
>
>
>
>
>
>
> ${jobTracker}
> ${nameNode}
> ${myscript}
> ${myscriptPath}
>
>
>
>
>
>
>
> ${jobTracker}
> ${nameNode}
> ${myscript}
> ${myscriptPath}
>
>
>
>
>
>
>
>
> ${jobTracker}
> ${nameNode}
> ${myscript}
> ${myscriptPath}
>
>
>
>
>
>
>
> ${jobTracker}
> ${nameNode}
> ${myscript}
> ${myscriptPath}
>
>
>
>
>
>
>
> ${jobTracker}
> ${nameNode}
> ${myscript}
> ${myscriptPath}
>
>
>
>
>
>
>
> Shell action failed, error
> message[${wf:errorMessage(wf:lastErrorNode())}]
>
>
>
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (OOZIE-3139) Oozie validates workflow incorrectly
[
https://issues.apache.org/jira/browse/OOZIE-3139?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16283418#comment-16283418
]
Peter Bacsko commented on OOZIE-3139:
-
[~dionusos] I'll check the latest patch then +1 and commit it if it looks good.
> Oozie validates workflow incorrectly
>
>
> Key: OOZIE-3139
> URL: https://issues.apache.org/jira/browse/OOZIE-3139
> Project: Oozie
> Issue Type: Bug
> Components: core
>Affects Versions: 4.3.0
>Reporter: Denes Bodo
>Assignee: Denes Bodo
>Priority: Critical
> Attachments: OOZIE-3139.001.patch, OOZIE-3139.002.patch,
> OOZIE-3139.003.patch, OOZIE-3139.004.patch, OOZIE-3139.005.patch,
> exception.log, workflow.pdf
>
>
> We found that after OOZIE-1978 the attached workflow becomes invalid. Aa we
> found out the following line brings the problem:
> https://github.com/apache/oozie/commit/8e9b9042b3270dc5ff975c44a5c977fcc41250e4#diff-a38ae78886b7c3afb711d32cfb4dbbbdR267
> . The parameter "okPath" changed from okPath to true.
> The following workflow is problematic:
> {noformat}
>
>
>
>
>
>
>
>
> ${jobTracker}
> ${nameNode}
> ${myscript}
> ${myscriptPath}
>
>
>
>
>
>
>
> ${jobTracker}
> ${nameNode}
> ${myscript}
> ${myscriptPath}
>
>
>
>
>
>
>
>
> ${jobTracker}
> ${nameNode}
> ${myscript}
> ${myscriptPath}
>
>
>
>
>
>
>
> ${jobTracker}
> ${nameNode}
> ${myscript}
> ${myscriptPath}
>
>
>
>
>
>
>
> ${jobTracker}
> ${nameNode}
> ${myscript}
> ${myscriptPath}
>
>
>
>
>
>
>
> Shell action failed, error
> message[${wf:errorMessage(wf:lastErrorNode())}]
>
>
>
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (OOZIE-3139) Oozie validates workflow incorrectly
[
https://issues.apache.org/jira/browse/OOZIE-3139?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16283416#comment-16283416
]
Denes Bodo commented on OOZIE-3139:
---
Does anybody have any comments? Is it okay to merge? Change is on review board,
also.
> Oozie validates workflow incorrectly
>
>
> Key: OOZIE-3139
> URL: https://issues.apache.org/jira/browse/OOZIE-3139
> Project: Oozie
> Issue Type: Bug
> Components: core
>Affects Versions: 4.3.0
>Reporter: Denes Bodo
>Assignee: Denes Bodo
>Priority: Critical
> Attachments: OOZIE-3139.001.patch, OOZIE-3139.002.patch,
> OOZIE-3139.003.patch, OOZIE-3139.004.patch, OOZIE-3139.005.patch,
> exception.log, workflow.pdf
>
>
> We found that after OOZIE-1978 the attached workflow becomes invalid. Aa we
> found out the following line brings the problem:
> https://github.com/apache/oozie/commit/8e9b9042b3270dc5ff975c44a5c977fcc41250e4#diff-a38ae78886b7c3afb711d32cfb4dbbbdR267
> . The parameter "okPath" changed from okPath to true.
> The following workflow is problematic:
> {noformat}
>
>
>
>
>
>
>
>
> ${jobTracker}
> ${nameNode}
> ${myscript}
> ${myscriptPath}
>
>
>
>
>
>
>
> ${jobTracker}
> ${nameNode}
> ${myscript}
> ${myscriptPath}
>
>
>
>
>
>
>
>
> ${jobTracker}
> ${nameNode}
> ${myscript}
> ${myscriptPath}
>
>
>
>
>
>
>
> ${jobTracker}
> ${nameNode}
> ${myscript}
> ${myscriptPath}
>
>
>
>
>
>
>
> ${jobTracker}
> ${nameNode}
> ${myscript}
> ${myscriptPath}
>
>
>
>
>
>
>
> Shell action failed, error
> message[${wf:errorMessage(wf:lastErrorNode())}]
>
>
>
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Created] (OOZIE-3142) Integer Overflows in Purge retentionTime
Prabhu Joseph created OOZIE-3142: Summary: Integer Overflows in Purge retentionTime Key: OOZIE-3142 URL: https://issues.apache.org/jira/browse/OOZIE-3142 Project: Oozie Issue Type: Bug Components: core Reporter: Prabhu Joseph Assignee: Prabhu Joseph Oozie Purge ShareLibService retentionTime overflows when oozie.service.ShareLibService.temp.sharelib.retention.days is set to 25 and becomes negative value. Value has to be in long. final long retentionTime = 1000 * 60 * 60 * 24 * ConfigurationService.getInt(LAUNCHERJAR_LIB_RETENTION); -- This message was sent by Atlassian JIRA (v6.4.14#64029)
