[jira] Subscription: Oozie Patch Available
Issue Subscription Filter: Oozie Patch Available (91 issues) Subscriber: ooziedaily Key Summary OOZIE-3596 When the SSH action is killed, it must be changed to the kill command that can terminate the related subprocess. https://issues.apache.org/jira/browse/OOZIE-3596 OOZIE-3568 Have large amount of log information “WARN messages [main] openjpa.MetaData” in jetty.log need to clean https://issues.apache.org/jira/browse/OOZIE-3568 OOZIE-3567 Oozie ShellAction should support absolute bash file path https://issues.apache.org/jira/browse/OOZIE-3567 OOZIE-3560 IDEA shows have some error in index.jsp https://issues.apache.org/jira/browse/OOZIE-3560 OOZIE-3482 Fix bug in CoordSubmitXCommand#validateCoordinatorJob https://issues.apache.org/jira/browse/OOZIE-3482 OOZIE-3480 Add windowactionstatus metrics in DBLiteWorkflowStoreService https://issues.apache.org/jira/browse/OOZIE-3480 OOZIE-3461 CoordMaterializeTriggerService code cleanup https://issues.apache.org/jira/browse/OOZIE-3461 OOZIE-3449 Make spark-2 as the default profile https://issues.apache.org/jira/browse/OOZIE-3449 OOZIE-3447 Run test case in local : It shows oozie-hsqldb-orm.xml exception https://issues.apache.org/jira/browse/OOZIE-3447 OOZIE-3434 Filtering for invalid jobtype should give error message https://issues.apache.org/jira/browse/OOZIE-3434 OOZIE-3418 Upgrade to Guava 27 https://issues.apache.org/jira/browse/OOZIE-3418 OOZIE-3404 The env variable of SPARK_HOME needs to be set when running pySpark https://issues.apache.org/jira/browse/OOZIE-3404 OOZIE-3375 Can't use empty in coordinator https://issues.apache.org/jira/browse/OOZIE-3375 OOZIE-3367 Using && in EL expressions in oozie bundle.xml files generates parse errors https://issues.apache.org/jira/browse/OOZIE-3367 OOZIE-3366 Update workflow status and subworkflow status on suspend command https://issues.apache.org/jira/browse/OOZIE-3366 OOZIE-3364 Rerunning Oozie bundle jobs starts the coordinators in indeterminate order https://issues.apache.org/jira/browse/OOZIE-3364 OOZIE-3362 When killed, SSH action should kill the spawned processes on target host https://issues.apache.org/jira/browse/OOZIE-3362 OOZIE-3335 Cleanup parseFilter methods https://issues.apache.org/jira/browse/OOZIE-3335 OOZIE-3328 Create Hive compatibility action executor to run hive actions using beeline https://issues.apache.org/jira/browse/OOZIE-3328 OOZIE-3319 Log SSH action callback error output https://issues.apache.org/jira/browse/OOZIE-3319 OOZIE-3301 Update NOTICE file https://issues.apache.org/jira/browse/OOZIE-3301 OOZIE-3274 Remove slf4j https://issues.apache.org/jira/browse/OOZIE-3274 OOZIE-3266 Coord action rerun support RERUN_SKIP_NODES option https://issues.apache.org/jira/browse/OOZIE-3266 OOZIE-3256 refactor OozieCLI class https://issues.apache.org/jira/browse/OOZIE-3256 OOZIE-3254 [coordinator] LAST_ONLY and NONE execution modes: possible OutOfMemoryError when there are too many coordinator actions to materialize https://issues.apache.org/jira/browse/OOZIE-3254 OOZIE-3196 Authorization: restrict world readability by user https://issues.apache.org/jira/browse/OOZIE-3196 OOZIE-3170 Oozie Diagnostic Bundle tool fails with NPE due to missing service class https://issues.apache.org/jira/browse/OOZIE-3170 OOZIE-3137 Add support for log4j2 in HiveMain https://issues.apache.org/jira/browse/OOZIE-3137 OOZIE-3135 Configure log4j2 in SqoopMain https://issues.apache.org/jira/browse/OOZIE-3135 OOZIE-3071 Oozie 4.3 Spark sharelib ueses a different version of commons-lang3 than Spark 2.2.0 https://issues.apache.org/jira/browse/OOZIE-3071 OOZIE-3063 Sanitizing variables that are part of openjpa.ConnectionProperties https://issues.apache.org/jira/browse/OOZIE-3063 OOZIE-3062 Set HADOOP_CONF_DIR for spark action https://issues.apache.org/jira/browse/OOZIE-3062 OOZIE-2952 Fix Findbugs warnings in oozie-sharelib-oozie https://issues.apache.org/jira/browse/OOZIE-2952 OOZIE-2834 ParameterVerifier logging non-useful warning for workflow definition https://issues.apache.org/jira/browse/OOZIE-2834 OOZIE-2812 SparkConfigurationService should support loading configurations from multiple Spark versions https://issues.apache.org/jira/browse/OOZIE-2812 OOZIE-2795 Create lib directory or symlink for Oozie CLI during packaging https://issues.apache.org/jira/browse/OOZIE-2795 OOZIE-2784 Include WEEK as a parameter in the Coordinator Expression Language Evaulator https://issues.apache.org/jira/browse/OOZIE-2784 OOZIE-2779 Mask Hive2 action Bee
[jira] [Updated] (OOZIE-3575) Add credential support for cloud file systems
[
https://issues.apache.org/jira/browse/OOZIE-3575?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Mate Juhasz updated OOZIE-3575:
---
Description:
Oozie by default gathers delegation tokens for the nodes defined in
_mapreduce.job.hdfs-servers_ (or _oozie.launcher.mapreduce.job.hdfs-servers_ in
case of distcp actions) and for the workflow path.
Though this implementation is good for hdfs, we dont support occasions where
the job related resources, which we want to access in runtime are present on
different file systems/buckets etc...
The HDFSCredentials class should be revised to handle getting tokens for
different cloud storages.
*The following scenarios should be addressed:*
Oozie should obtain delegation token in case
* the defaultFs is cloud
* the workload.xml is in cloud
* input/output/auxiliary files referred from workflow are in cloud
* (newly introduced feature) user could define filesystem credentials for the
workflow (as its done with hive/hcat etc..) -> this would allow the user to
handle the situation where Oozie could not decide which tokens are needed at
launch time by default and could also get tokens for different cloud storages
and buckets as well
Example for credentials addition:
{noformat}
filesystem.path
s3a://qe-s3-bucket-mst
{noformat}
was:
Oozie by default gathers delegation tokens for the nodes defined in
_mapreduce.job.hdfs-servers_ (or _oozie.launcher.mapreduce.job.hdfs-servers_ in
case of distcp actions) and for the workflow path.
Though this implementation is good for hdfs, we dont support occasions where
the job related resources, which we want to access in runtime are present on
different file systems/buckets etc...
The HDFSCredentials class should be revised to handle getting tokens for
different cloud storages.
*The following scenarios should be addressed:*
Oozie should obtain delegation token in case
* the defaultFs is cloud
* the workload.xml is in cloud
* input/output/auxiliary files referred from workflow are in cloud
* (newly introduced feature) user could define filesystem credentials for the
workflow (as its done with hive/hcat etc..) -> this would allow the user to
handle the situation where Oozie could not decide which tokens are needed at
launch time by default and could also get tokens for different cloud storages
and buckets as well
Example for credentials addition:
{noformat}
filesystem
s3a://qe-s3-bucket-mst
{noformat}
> Add credential support for cloud file systems
> -
>
> Key: OOZIE-3575
> URL: https://issues.apache.org/jira/browse/OOZIE-3575
> Project: Oozie
> Issue Type: Sub-task
> Components: core
>Affects Versions: 5.2.0
>Reporter: Mate Juhasz
>Assignee: Mate Juhasz
>Priority: Major
> Fix For: 5.3.0
>
> Attachments: OOZIE-3575-v2.patch, OOZIE-3575-v3.patch,
> OOZIE-3575.patch
>
>
> Oozie by default gathers delegation tokens for the nodes defined in
> _mapreduce.job.hdfs-servers_ (or _oozie.launcher.mapreduce.job.hdfs-servers_
> in case of distcp actions) and for the workflow path.
> Though this implementation is good for hdfs, we dont support occasions where
> the job related resources, which we want to access in runtime are present on
> different file systems/buckets etc...
> The HDFSCredentials class should be revised to handle getting tokens for
> different cloud storages.
> *The following scenarios should be addressed:*
> Oozie should obtain delegation token in case
> * the defaultFs is cloud
> * the workload.xml is in cloud
> * input/output/auxiliary files referred from workflow are in cloud
> * (newly introduced feature) user could define filesystem credentials for the
> workflow (as its done with hive/hcat etc..) -> this would allow the user to
> handle the situation where Oozie could not decide which tokens are needed at
> launch time by default and could also get tokens for different cloud storages
> and buckets as well
> Example for credentials addition:
> {noformat}
>
>
> filesystem.path
> s3a://qe-s3-bucket-mst
>
>
> {noformat}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
