[jira] [Commented] (OOZIE-3601) Upgrade quartz to 2.3.2
[ https://issues.apache.org/jira/browse/OOZIE-3601?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17154424#comment-17154424 ] Mate Juhasz commented on OOZIE-3601: Thanks [~dengliming] for the contribution, committed to master! > Upgrade quartz to 2.3.2 > --- > > Key: OOZIE-3601 > URL: https://issues.apache.org/jira/browse/OOZIE-3601 > Project: Oozie > Issue Type: Bug >Affects Versions: 5.2.0 >Reporter: Andras Salamon >Assignee: dengliming >Priority: Major > Attachments: OOZIE-3601-001.patch > > > We should upgrade quartz to 2.3.2: > https://nvd.nist.gov/vuln/detail/CVE-2019-13990 > Most likely it's just a simple version bump. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (OOZIE-3599) Upgrade Jetty to 9.4.28
Mate Juhasz created OOZIE-3599: -- Summary: Upgrade Jetty to 9.4.28 Key: OOZIE-3599 URL: https://issues.apache.org/jira/browse/OOZIE-3599 Project: Oozie Issue Type: Bug Affects Versions: 5.2.0 Reporter: Mate Juhasz Fix For: trunk Jetty could be upgraded to the latest version, which is now 9.4.28.v20200408, though it has to be checked before if its feasible. In case we upgrade we have to be aware of wildcard certs with SAN as well and should change the code to use SslContextFactory.server instead of SsslContextFactory in org.apache.oozie.server.SSLServerConnectorFactory class. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (OOZIE-3575) Add credential support for cloud file systems
[
https://issues.apache.org/jira/browse/OOZIE-3575?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Mate Juhasz updated OOZIE-3575:
---
Description:
Oozie by default gathers delegation tokens for the nodes defined in
_mapreduce.job.hdfs-servers_ (or _oozie.launcher.mapreduce.job.hdfs-servers_ in
case of distcp actions) and for the workflow path.
Though this implementation is good for hdfs, we dont support occasions where
the job related resources, which we want to access in runtime are present on
different file systems/buckets etc...
The HDFSCredentials class should be revised to handle getting tokens for
different cloud storages.
*The following scenarios should be addressed:*
Oozie should obtain delegation token in case
* the defaultFs is cloud
* the workload.xml is in cloud
* input/output/auxiliary files referred from workflow are in cloud
* (newly introduced feature) user could define filesystem credentials for the
workflow (as its done with hive/hcat etc..) -> this would allow the user to
handle the situation where Oozie could not decide which tokens are needed at
launch time by default and could also get tokens for different cloud storages
and buckets as well
Example for credentials addition:
{noformat}
filesystem.path
s3a://qe-s3-bucket-mst
{noformat}
was:
Oozie by default gathers delegation tokens for the nodes defined in
_mapreduce.job.hdfs-servers_ (or _oozie.launcher.mapreduce.job.hdfs-servers_ in
case of distcp actions) and for the workflow path.
Though this implementation is good for hdfs, we dont support occasions where
the job related resources, which we want to access in runtime are present on
different file systems/buckets etc...
The HDFSCredentials class should be revised to handle getting tokens for
different cloud storages.
*The following scenarios should be addressed:*
Oozie should obtain delegation token in case
* the defaultFs is cloud
* the workload.xml is in cloud
* input/output/auxiliary files referred from workflow are in cloud
* (newly introduced feature) user could define filesystem credentials for the
workflow (as its done with hive/hcat etc..) -> this would allow the user to
handle the situation where Oozie could not decide which tokens are needed at
launch time by default and could also get tokens for different cloud storages
and buckets as well
Example for credentials addition:
{noformat}
filesystem
s3a://qe-s3-bucket-mst
{noformat}
> Add credential support for cloud file systems
> -
>
> Key: OOZIE-3575
> URL: https://issues.apache.org/jira/browse/OOZIE-3575
> Project: Oozie
> Issue Type: Sub-task
> Components: core
>Affects Versions: 5.2.0
>Reporter: Mate Juhasz
>Assignee: Mate Juhasz
>Priority: Major
> Fix For: 5.3.0
>
> Attachments: OOZIE-3575-v2.patch, OOZIE-3575-v3.patch,
> OOZIE-3575.patch
>
>
> Oozie by default gathers delegation tokens for the nodes defined in
> _mapreduce.job.hdfs-servers_ (or _oozie.launcher.mapreduce.job.hdfs-servers_
> in case of distcp actions) and for the workflow path.
> Though this implementation is good for hdfs, we dont support occasions where
> the job related resources, which we want to access in runtime are present on
> different file systems/buckets etc...
> The HDFSCredentials class should be revised to handle getting tokens for
> different cloud storages.
> *The following scenarios should be addressed:*
> Oozie should obtain delegation token in case
> * the defaultFs is cloud
> * the workload.xml is in cloud
> * input/output/auxiliary files referred from workflow are in cloud
> * (newly introduced feature) user could define filesystem credentials for the
> workflow (as its done with hive/hcat etc..) -> this would allow the user to
> handle the situation where Oozie could not decide which tokens are needed at
> launch time by default and could also get tokens for different cloud storages
> and buckets as well
> Example for credentials addition:
> {noformat}
>
>
> filesystem.path
> s3a://qe-s3-bucket-mst
>
>
> {noformat}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (OOZIE-3592) Do not print misleading System.exit() exception for successful jobs
[
https://issues.apache.org/jira/browse/OOZIE-3592?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Mate Juhasz updated OOZIE-3592:
---
Attachment: OOZIE-3592-v3.patch
> Do not print misleading System.exit() exception for successful jobs
> ---
>
> Key: OOZIE-3592
> URL: https://issues.apache.org/jira/browse/OOZIE-3592
> Project: Oozie
> Issue Type: Bug
>Affects Versions: 5.2.0
>Reporter: Mate Juhasz
>Assignee: Mate Juhasz
>Priority: Major
> Fix For: trunk
>
> Attachments: OOZIE-3592-v1.patch, OOZIE-3592-v2.patch,
> OOZIE-3592-v3.patch
>
>
> The following exception is logged by Oozie due to System.exit() calls in
> application JVM-s even after successful workflow actions:
> {noformat}
> Hadoop Job IDs executed by Sqoop: job_1583161376812_0018
> java.lang.reflect.InvocationTargetException
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at
> org.apache.oozie.action.hadoop.LauncherAM.runActionMain(LauncherAM.java:413)
> at
> org.apache.oozie.action.hadoop.LauncherAM.access$400(LauncherAM.java:55)
> at org.apache.oozie.action.hadoop.LauncherAM$2.run(LauncherAM.java:226)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:422)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1876)
> at org.apache.oozie.action.hadoop.LauncherAM.run(LauncherAM.java:220)
> at org.apache.oozie.action.hadoop.LauncherAM$1.run(LauncherAM.java:156)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:422)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1876)
> at org.apache.oozie.action.hadoop.LauncherAM.main(LauncherAM.java:144)
> Caused by: java.lang.SecurityException: Intercepted System.exit(0)
> at
> org.apache.oozie.action.hadoop.security.LauncherSecurityManager.checkExit(LauncherSecurityManager.java:57)
> at java.lang.Runtime.exit(Runtime.java:107)
> at java.lang.System.exit(System.java:971)
> at org.apache.sqoop.Sqoop.main(Sqoop.java:260)
> at
> org.apache.oozie.action.hadoop.SqoopMain.runSqoopJob(SqoopMain.java:221)
> at org.apache.oozie.action.hadoop.SqoopMain.run(SqoopMain.java:206)
> at
> org.apache.oozie.action.hadoop.LauncherMain.run(LauncherMain.java:107)
> at org.apache.oozie.action.hadoop.SqoopMain.main(SqoopMain.java:52)
> ... 16 more
> Intercepting System.exit(0)
> {noformat}
> Typically happens in Sqoop actions. This can be very misleading for users as
> they may suspect that some error has occurred even if their yarn application
> and Oozie workflow has been marked as succeeded.
> The "error" is already ignored by Oozie when the System.exit code is 0, but
> we should not print either this exception in such cases.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (OOZIE-3592) Do not print misleading System.exit() exception for successful jobs
[
https://issues.apache.org/jira/browse/OOZIE-3592?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Mate Juhasz updated OOZIE-3592:
---
Attachment: (was: OOZIE-3592-v2.patch)
> Do not print misleading System.exit() exception for successful jobs
> ---
>
> Key: OOZIE-3592
> URL: https://issues.apache.org/jira/browse/OOZIE-3592
> Project: Oozie
> Issue Type: Bug
>Affects Versions: 5.2.0
>Reporter: Mate Juhasz
>Assignee: Mate Juhasz
>Priority: Major
> Fix For: trunk
>
> Attachments: OOZIE-3592-v1.patch, OOZIE-3592-v2.patch
>
>
> The following exception is logged by Oozie due to System.exit() calls in
> application JVM-s even after successful workflow actions:
> {noformat}
> Hadoop Job IDs executed by Sqoop: job_1583161376812_0018
> java.lang.reflect.InvocationTargetException
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at
> org.apache.oozie.action.hadoop.LauncherAM.runActionMain(LauncherAM.java:413)
> at
> org.apache.oozie.action.hadoop.LauncherAM.access$400(LauncherAM.java:55)
> at org.apache.oozie.action.hadoop.LauncherAM$2.run(LauncherAM.java:226)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:422)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1876)
> at org.apache.oozie.action.hadoop.LauncherAM.run(LauncherAM.java:220)
> at org.apache.oozie.action.hadoop.LauncherAM$1.run(LauncherAM.java:156)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:422)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1876)
> at org.apache.oozie.action.hadoop.LauncherAM.main(LauncherAM.java:144)
> Caused by: java.lang.SecurityException: Intercepted System.exit(0)
> at
> org.apache.oozie.action.hadoop.security.LauncherSecurityManager.checkExit(LauncherSecurityManager.java:57)
> at java.lang.Runtime.exit(Runtime.java:107)
> at java.lang.System.exit(System.java:971)
> at org.apache.sqoop.Sqoop.main(Sqoop.java:260)
> at
> org.apache.oozie.action.hadoop.SqoopMain.runSqoopJob(SqoopMain.java:221)
> at org.apache.oozie.action.hadoop.SqoopMain.run(SqoopMain.java:206)
> at
> org.apache.oozie.action.hadoop.LauncherMain.run(LauncherMain.java:107)
> at org.apache.oozie.action.hadoop.SqoopMain.main(SqoopMain.java:52)
> ... 16 more
> Intercepting System.exit(0)
> {noformat}
> Typically happens in Sqoop actions. This can be very misleading for users as
> they may suspect that some error has occurred even if their yarn application
> and Oozie workflow has been marked as succeeded.
> The "error" is already ignored by Oozie when the System.exit code is 0, but
> we should not print either this exception in such cases.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (OOZIE-3592) Do not print misleading System.exit() exception for successful jobs
[
https://issues.apache.org/jira/browse/OOZIE-3592?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Mate Juhasz updated OOZIE-3592:
---
Attachment: OOZIE-3592-v2.patch
> Do not print misleading System.exit() exception for successful jobs
> ---
>
> Key: OOZIE-3592
> URL: https://issues.apache.org/jira/browse/OOZIE-3592
> Project: Oozie
> Issue Type: Bug
>Affects Versions: 5.2.0
>Reporter: Mate Juhasz
>Assignee: Mate Juhasz
>Priority: Major
> Fix For: trunk
>
> Attachments: OOZIE-3592-v1.patch, OOZIE-3592-v2.patch
>
>
> The following exception is logged by Oozie due to System.exit() calls in
> application JVM-s even after successful workflow actions:
> {noformat}
> Hadoop Job IDs executed by Sqoop: job_1583161376812_0018
> java.lang.reflect.InvocationTargetException
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at
> org.apache.oozie.action.hadoop.LauncherAM.runActionMain(LauncherAM.java:413)
> at
> org.apache.oozie.action.hadoop.LauncherAM.access$400(LauncherAM.java:55)
> at org.apache.oozie.action.hadoop.LauncherAM$2.run(LauncherAM.java:226)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:422)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1876)
> at org.apache.oozie.action.hadoop.LauncherAM.run(LauncherAM.java:220)
> at org.apache.oozie.action.hadoop.LauncherAM$1.run(LauncherAM.java:156)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:422)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1876)
> at org.apache.oozie.action.hadoop.LauncherAM.main(LauncherAM.java:144)
> Caused by: java.lang.SecurityException: Intercepted System.exit(0)
> at
> org.apache.oozie.action.hadoop.security.LauncherSecurityManager.checkExit(LauncherSecurityManager.java:57)
> at java.lang.Runtime.exit(Runtime.java:107)
> at java.lang.System.exit(System.java:971)
> at org.apache.sqoop.Sqoop.main(Sqoop.java:260)
> at
> org.apache.oozie.action.hadoop.SqoopMain.runSqoopJob(SqoopMain.java:221)
> at org.apache.oozie.action.hadoop.SqoopMain.run(SqoopMain.java:206)
> at
> org.apache.oozie.action.hadoop.LauncherMain.run(LauncherMain.java:107)
> at org.apache.oozie.action.hadoop.SqoopMain.main(SqoopMain.java:52)
> ... 16 more
> Intercepting System.exit(0)
> {noformat}
> Typically happens in Sqoop actions. This can be very misleading for users as
> they may suspect that some error has occurred even if their yarn application
> and Oozie workflow has been marked as succeeded.
> The "error" is already ignored by Oozie when the System.exit code is 0, but
> we should not print either this exception in such cases.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (OOZIE-3592) Do not print misleading System.exit() exception for successful jobs
[
https://issues.apache.org/jira/browse/OOZIE-3592?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Mate Juhasz updated OOZIE-3592:
---
Attachment: OOZIE-3592-v2.patch
> Do not print misleading System.exit() exception for successful jobs
> ---
>
> Key: OOZIE-3592
> URL: https://issues.apache.org/jira/browse/OOZIE-3592
> Project: Oozie
> Issue Type: Bug
>Affects Versions: 5.2.0
>Reporter: Mate Juhasz
>Assignee: Mate Juhasz
>Priority: Major
> Fix For: trunk
>
> Attachments: OOZIE-3592-v1.patch, OOZIE-3592-v2.patch
>
>
> The following exception is logged by Oozie due to System.exit() calls in
> application JVM-s even after successful workflow actions:
> {noformat}
> Hadoop Job IDs executed by Sqoop: job_1583161376812_0018
> java.lang.reflect.InvocationTargetException
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at
> org.apache.oozie.action.hadoop.LauncherAM.runActionMain(LauncherAM.java:413)
> at
> org.apache.oozie.action.hadoop.LauncherAM.access$400(LauncherAM.java:55)
> at org.apache.oozie.action.hadoop.LauncherAM$2.run(LauncherAM.java:226)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:422)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1876)
> at org.apache.oozie.action.hadoop.LauncherAM.run(LauncherAM.java:220)
> at org.apache.oozie.action.hadoop.LauncherAM$1.run(LauncherAM.java:156)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:422)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1876)
> at org.apache.oozie.action.hadoop.LauncherAM.main(LauncherAM.java:144)
> Caused by: java.lang.SecurityException: Intercepted System.exit(0)
> at
> org.apache.oozie.action.hadoop.security.LauncherSecurityManager.checkExit(LauncherSecurityManager.java:57)
> at java.lang.Runtime.exit(Runtime.java:107)
> at java.lang.System.exit(System.java:971)
> at org.apache.sqoop.Sqoop.main(Sqoop.java:260)
> at
> org.apache.oozie.action.hadoop.SqoopMain.runSqoopJob(SqoopMain.java:221)
> at org.apache.oozie.action.hadoop.SqoopMain.run(SqoopMain.java:206)
> at
> org.apache.oozie.action.hadoop.LauncherMain.run(LauncherMain.java:107)
> at org.apache.oozie.action.hadoop.SqoopMain.main(SqoopMain.java:52)
> ... 16 more
> Intercepting System.exit(0)
> {noformat}
> Typically happens in Sqoop actions. This can be very misleading for users as
> they may suspect that some error has occurred even if their yarn application
> and Oozie workflow has been marked as succeeded.
> The "error" is already ignored by Oozie when the System.exit code is 0, but
> we should not print either this exception in such cases.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (OOZIE-3592) Do not print misleading System.exit() exception for successful jobs
[
https://issues.apache.org/jira/browse/OOZIE-3592?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17053394#comment-17053394
]
Mate Juhasz commented on OOZIE-3592:
Added an initial patch, but I still owe with unit tests
> Do not print misleading System.exit() exception for successful jobs
> ---
>
> Key: OOZIE-3592
> URL: https://issues.apache.org/jira/browse/OOZIE-3592
> Project: Oozie
> Issue Type: Bug
>Affects Versions: 5.2.0
>Reporter: Mate Juhasz
>Assignee: Mate Juhasz
>Priority: Major
> Fix For: trunk
>
> Attachments: OOZIE-3592-v1.patch
>
>
> The following exception is logged by Oozie due to System.exit() calls in
> application JVM-s even after successful workflow actions:
> {noformat}
> Hadoop Job IDs executed by Sqoop: job_1583161376812_0018
> java.lang.reflect.InvocationTargetException
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at
> org.apache.oozie.action.hadoop.LauncherAM.runActionMain(LauncherAM.java:413)
> at
> org.apache.oozie.action.hadoop.LauncherAM.access$400(LauncherAM.java:55)
> at org.apache.oozie.action.hadoop.LauncherAM$2.run(LauncherAM.java:226)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:422)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1876)
> at org.apache.oozie.action.hadoop.LauncherAM.run(LauncherAM.java:220)
> at org.apache.oozie.action.hadoop.LauncherAM$1.run(LauncherAM.java:156)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:422)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1876)
> at org.apache.oozie.action.hadoop.LauncherAM.main(LauncherAM.java:144)
> Caused by: java.lang.SecurityException: Intercepted System.exit(0)
> at
> org.apache.oozie.action.hadoop.security.LauncherSecurityManager.checkExit(LauncherSecurityManager.java:57)
> at java.lang.Runtime.exit(Runtime.java:107)
> at java.lang.System.exit(System.java:971)
> at org.apache.sqoop.Sqoop.main(Sqoop.java:260)
> at
> org.apache.oozie.action.hadoop.SqoopMain.runSqoopJob(SqoopMain.java:221)
> at org.apache.oozie.action.hadoop.SqoopMain.run(SqoopMain.java:206)
> at
> org.apache.oozie.action.hadoop.LauncherMain.run(LauncherMain.java:107)
> at org.apache.oozie.action.hadoop.SqoopMain.main(SqoopMain.java:52)
> ... 16 more
> Intercepting System.exit(0)
> {noformat}
> Typically happens in Sqoop actions. This can be very misleading for users as
> they may suspect that some error has occurred even if their yarn application
> and Oozie workflow has been marked as succeeded.
> The "error" is already ignored by Oozie when the System.exit code is 0, but
> we should not print either this exception in such cases.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (OOZIE-3592) Do not print misleading System.exit() exception for successful jobs
[
https://issues.apache.org/jira/browse/OOZIE-3592?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Mate Juhasz updated OOZIE-3592:
---
Attachment: OOZIE-3592-v1.patch
> Do not print misleading System.exit() exception for successful jobs
> ---
>
> Key: OOZIE-3592
> URL: https://issues.apache.org/jira/browse/OOZIE-3592
> Project: Oozie
> Issue Type: Bug
>Affects Versions: 5.2.0
>Reporter: Mate Juhasz
>Assignee: Mate Juhasz
>Priority: Major
> Fix For: trunk
>
> Attachments: OOZIE-3592-v1.patch
>
>
> The following exception is logged by Oozie due to System.exit() calls in
> application JVM-s even after successful workflow actions:
> {noformat}
> Hadoop Job IDs executed by Sqoop: job_1583161376812_0018
> java.lang.reflect.InvocationTargetException
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at
> org.apache.oozie.action.hadoop.LauncherAM.runActionMain(LauncherAM.java:413)
> at
> org.apache.oozie.action.hadoop.LauncherAM.access$400(LauncherAM.java:55)
> at org.apache.oozie.action.hadoop.LauncherAM$2.run(LauncherAM.java:226)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:422)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1876)
> at org.apache.oozie.action.hadoop.LauncherAM.run(LauncherAM.java:220)
> at org.apache.oozie.action.hadoop.LauncherAM$1.run(LauncherAM.java:156)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:422)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1876)
> at org.apache.oozie.action.hadoop.LauncherAM.main(LauncherAM.java:144)
> Caused by: java.lang.SecurityException: Intercepted System.exit(0)
> at
> org.apache.oozie.action.hadoop.security.LauncherSecurityManager.checkExit(LauncherSecurityManager.java:57)
> at java.lang.Runtime.exit(Runtime.java:107)
> at java.lang.System.exit(System.java:971)
> at org.apache.sqoop.Sqoop.main(Sqoop.java:260)
> at
> org.apache.oozie.action.hadoop.SqoopMain.runSqoopJob(SqoopMain.java:221)
> at org.apache.oozie.action.hadoop.SqoopMain.run(SqoopMain.java:206)
> at
> org.apache.oozie.action.hadoop.LauncherMain.run(LauncherMain.java:107)
> at org.apache.oozie.action.hadoop.SqoopMain.main(SqoopMain.java:52)
> ... 16 more
> Intercepting System.exit(0)
> {noformat}
> Typically happens in Sqoop actions. This can be very misleading for users as
> they may suspect that some error has occurred even if their yarn application
> and Oozie workflow has been marked as succeeded.
> The "error" is already ignored by Oozie when the System.exit code is 0, but
> we should not print either this exception in such cases.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (OOZIE-3592) Do not print misleading System.exit() exception for successful jobs
[
https://issues.apache.org/jira/browse/OOZIE-3592?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Mate Juhasz updated OOZIE-3592:
---
Summary: Do not print misleading System.exit() exception for successful
jobs (was: Do not print misleading security exception for successful jobs)
> Do not print misleading System.exit() exception for successful jobs
> ---
>
> Key: OOZIE-3592
> URL: https://issues.apache.org/jira/browse/OOZIE-3592
> Project: Oozie
> Issue Type: Bug
>Affects Versions: 5.2.0
>Reporter: Mate Juhasz
>Assignee: Mate Juhasz
>Priority: Major
> Fix For: trunk
>
>
> The following exception is logged by Oozie due to System.exit() calls in
> application JVM-s even after successful workflow actions:
> {noformat}
> Hadoop Job IDs executed by Sqoop: job_1583161376812_0018
> java.lang.reflect.InvocationTargetException
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at
> org.apache.oozie.action.hadoop.LauncherAM.runActionMain(LauncherAM.java:413)
> at
> org.apache.oozie.action.hadoop.LauncherAM.access$400(LauncherAM.java:55)
> at org.apache.oozie.action.hadoop.LauncherAM$2.run(LauncherAM.java:226)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:422)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1876)
> at org.apache.oozie.action.hadoop.LauncherAM.run(LauncherAM.java:220)
> at org.apache.oozie.action.hadoop.LauncherAM$1.run(LauncherAM.java:156)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:422)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1876)
> at org.apache.oozie.action.hadoop.LauncherAM.main(LauncherAM.java:144)
> Caused by: java.lang.SecurityException: Intercepted System.exit(1)
> at
> org.apache.oozie.action.hadoop.security.LauncherSecurityManager.checkExit(LauncherSecurityManager.java:57)
> at java.lang.Runtime.exit(Runtime.java:107)
> at java.lang.System.exit(System.java:971)
> at org.apache.sqoop.Sqoop.main(Sqoop.java:260)
> at
> org.apache.oozie.action.hadoop.SqoopMain.runSqoopJob(SqoopMain.java:221)
> at org.apache.oozie.action.hadoop.SqoopMain.run(SqoopMain.java:206)
> at
> org.apache.oozie.action.hadoop.LauncherMain.run(LauncherMain.java:107)
> at org.apache.oozie.action.hadoop.SqoopMain.main(SqoopMain.java:52)
> ... 16 more
> Intercepting System.exit(0)
> {noformat}
> Typically happens in Sqoop actions. This can be very misleading for users as
> they may suspect that some error has occurred even if their yarn application
> and Oozie workflow has been marked as succeeded.
> The "error" is already ignored by Oozie when the System.exit code is 0, but
> we should not print either this exception in such cases.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Created] (OOZIE-3592) Do not print misleading security exception for successful jobs
Mate Juhasz created OOZIE-3592:
--
Summary: Do not print misleading security exception for successful
jobs
Key: OOZIE-3592
URL: https://issues.apache.org/jira/browse/OOZIE-3592
Project: Oozie
Issue Type: Bug
Affects Versions: 5.2.0
Reporter: Mate Juhasz
Assignee: Mate Juhasz
Fix For: trunk
The following exception is logged by Oozie due to System.exit() calls in
application JVM-s even after successful workflow actions:
{noformat}
Hadoop Job IDs executed by Sqoop: job_1583161376812_0018
java.lang.reflect.InvocationTargetException
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at
org.apache.oozie.action.hadoop.LauncherAM.runActionMain(LauncherAM.java:413)
at
org.apache.oozie.action.hadoop.LauncherAM.access$400(LauncherAM.java:55)
at org.apache.oozie.action.hadoop.LauncherAM$2.run(LauncherAM.java:226)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:422)
at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1876)
at org.apache.oozie.action.hadoop.LauncherAM.run(LauncherAM.java:220)
at org.apache.oozie.action.hadoop.LauncherAM$1.run(LauncherAM.java:156)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:422)
at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1876)
at org.apache.oozie.action.hadoop.LauncherAM.main(LauncherAM.java:144)
Caused by: java.lang.SecurityException: Intercepted System.exit(1)
at
org.apache.oozie.action.hadoop.security.LauncherSecurityManager.checkExit(LauncherSecurityManager.java:57)
at java.lang.Runtime.exit(Runtime.java:107)
at java.lang.System.exit(System.java:971)
at org.apache.sqoop.Sqoop.main(Sqoop.java:260)
at
org.apache.oozie.action.hadoop.SqoopMain.runSqoopJob(SqoopMain.java:221)
at org.apache.oozie.action.hadoop.SqoopMain.run(SqoopMain.java:206)
at
org.apache.oozie.action.hadoop.LauncherMain.run(LauncherMain.java:107)
at org.apache.oozie.action.hadoop.SqoopMain.main(SqoopMain.java:52)
... 16 more
Intercepting System.exit(0)
{noformat}
Typically happens in Sqoop actions. This can be very misleading for users as
they may suspect that some error has occurred even if their yarn application
and Oozie workflow has been marked as succeeded.
The "error" is already ignored by Oozie when the System.exit code is 0, but we
should not print either this exception in such cases.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (OOZIE-3584) Fork-join action issue when action param cannot be resolved
[ https://issues.apache.org/jira/browse/OOZIE-3584?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mate Juhasz updated OOZIE-3584: --- Fix Version/s: trunk > Fork-join action issue when action param cannot be resolved > --- > > Key: OOZIE-3584 > URL: https://issues.apache.org/jira/browse/OOZIE-3584 > Project: Oozie > Issue Type: Bug > Components: core >Affects Versions: 5.1.0 >Reporter: Janos Makai >Assignee: Janos Makai >Priority: Major > Fix For: trunk > > Attachments: OOZIE-3584-001.patch, forkjoin_actionparam_issue.log > > > _Current bug_ > *=* > There is a sub workflow run in independent mode that runs a fork action which > contains two (or more) actions. > These actions inside the fork action run in parallel mode, and they have some > seconds delay in between them. > If a parameter is passed to one of these actions, that cannot be resolved, > then it changes its status to FAILED, and also the workflow’s state to > FAILED. The other action’s state which are not started yet will stuck in PREP > state forever. The correct behaviour would be to KILL the remaining actions > as well as the workflow. > Note: this bug only occurs when it is run in independent mode. If it has a > parent workflow, then the parent workflow will kill this workflow after 10 > minutes because of the callback process. > > _Log_ > *===* > 2020-02-14 11:59:26,698 ERROR org.apache.oozie.command.wf.SignalXCommand: > SERVER[quasar-nqrrjp-4.quasar-nqrrjp.root.hwx.site] USER[admin] GROUP[-] > TOKEN[] APP[Sub flow fork join] JOB[005-200214101441478-oozie-oozi-W] > ACTION[005-200214101441478-oozie-oozi-W@fork-4a1c] Error running forked > jobs parallely > org.apache.oozie.command.CommandException: E0718: Workflow already completed > at > org.apache.oozie.command.wf.ActionXCommand.failJob(ActionXCommand.java:213) > at > org.apache.oozie.command.wf.ActionXCommand.failJob(ActionXCommand.java:185) > at > org.apache.oozie.command.wf.SignalXCommand.startForkedActions(SignalXCommand.java:498) > at > org.apache.oozie.command.wf.SignalXCommand.execute(SignalXCommand.java:462) > at org.apache.oozie.command.wf.SignalXCommand.execute(SignalXCommand.java:82) > at org.apache.oozie.command.XCommand.call(XCommand.java:291) > at > org.apache.oozie.command.wf.ActionEndXCommand.execute(ActionEndXCommand.java:283) > at > org.apache.oozie.command.wf.ActionEndXCommand.execute(ActionEndXCommand.java:62) > at org.apache.oozie.command.XCommand.call(XCommand.java:291) > at > org.apache.oozie.command.wf.ActionStartXCommand.callActionEnd(ActionStartXCommand.java:352) > at > org.apache.oozie.command.wf.ActionStartXCommand.execute(ActionStartXCommand.java:338) > at > org.apache.oozie.command.wf.ActionStartXCommand.execute(ActionStartXCommand.java:68) > at org.apache.oozie.command.XCommand.call(XCommand.java:291) > at > org.apache.oozie.service.CallableQueueService$CompositeCallable.call(CallableQueueService.java:363) > at > org.apache.oozie.service.CallableQueueService$CompositeCallable.call(CallableQueueService.java:292) > at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) > at > org.apache.oozie.service.CallableQueueService$CallableWrapper.run(CallableQueueService.java:210) > at > java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) > at > java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) > at java.base/java.lang.Thread.run(Thread.java:834) > Caused by: org.apache.oozie.workflow.WorkflowException: E0718: Workflow > already completed > at > org.apache.oozie.workflow.lite.LiteWorkflowInstance.fail(LiteWorkflowInstance.java:337) > at > org.apache.oozie.command.wf.ActionXCommand.failJob(ActionXCommand.java:201) > ... 19 more > Full log added as attachment. > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (OOZIE-3071) Oozie 4.3 Spark sharelib ueses a different version of commons-lang3 than Spark 2.2.0
[
https://issues.apache.org/jira/browse/OOZIE-3071?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17031477#comment-17031477
]
Mate Juhasz commented on OOZIE-3071:
Bumping up the topic again... commons-lang3 3.3.2 does not support JDK 11, the
below error can be observed when running spark actions on Java 11.
{noformat}
Caused by: java.lang.NullPointerException
at
org.apache.commons.lang3.SystemUtils.isJavaVersionAtLeast(SystemUtils.java:1654)
{noformat}
commons-lang3:3.8.1 fixes it, but we should use the same 3.9 as in SPARK-30035,
which is the first release to support JDK9+.
> Oozie 4.3 Spark sharelib ueses a different version of commons-lang3 than
> Spark 2.2.0
>
>
> Key: OOZIE-3071
> URL: https://issues.apache.org/jira/browse/OOZIE-3071
> Project: Oozie
> Issue Type: Bug
> Components: core
>Affects Versions: 4.3.0
>Reporter: Ran Gu
>Assignee: Artem Ervits
>Priority: Major
> Labels: patch
> Attachments: OOZIE-3071-1.patch, OOZIE-3071.patch
>
>
> Currently Oozie 4.3.0 uses commons-lang3 version 3.3.2 in Spark sharelib.
> Spark 2.2.0 uses commons-lang3 version 3.5.
> This causes Oozie(/Spark) job failures on EMR-5.8.0 clusters.
> Error message:
> 17/08/22 00:22:43 ERROR ApplicationMaster: User class threw exception:
> java.lang.IllegalArgumentException: Illegal pattern component: XXX
> java.lang.IllegalArgumentException: Illegal pattern component: XXX
> at
> org.apache.commons.lang3.time.FastDatePrinter.parsePattern(FastDatePrinter.java:282)
> at
> org.apache.commons.lang3.time.FastDatePrinter.init(FastDatePrinter.java:149)
> at
> org.apache.commons.lang3.time.FastDatePrinter.(FastDatePrinter.java:142)
> at
> org.apache.commons.lang3.time.FastDateFormat.(FastDateFormat.java:384)
> at
> org.apache.commons.lang3.time.FastDateFormat.(FastDateFormat.java:369)
> at
> org.apache.commons.lang3.time.FastDateFormat$1.createInstance(FastDateFormat.java:91)
> at
> org.apache.commons.lang3.time.FastDateFormat$1.createInstance(FastDateFormat.java:88)
> at org.apache.commons.lang3.time.FormatCache.getInstance(FormatCache.java:82)
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (OOZIE-3580) setLibFilesArchives is called twice unnecessarily
[
https://issues.apache.org/jira/browse/OOZIE-3580?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17017024#comment-17017024
]
Mate Juhasz commented on OOZIE-3580:
Ok, so its really not safe to drop any of these method calls. There is some
magic done with configuration files, I need to check it further
> setLibFilesArchives is called twice unnecessarily
> -
>
> Key: OOZIE-3580
> URL: https://issues.apache.org/jira/browse/OOZIE-3580
> Project: Oozie
> Issue Type: Bug
> Components: core
>Affects Versions: 5.2.0
>Reporter: Mate Juhasz
>Assignee: Mate Juhasz
>Priority: Major
> Fix For: trunk
>
>
> The JavaActionExecutor#setLibFilesArchives method can be costly as its
> iterating through all the files, archives, lib files and share lib files
> added for an action. Its invoked twice per workflow at the moment, but the
> second call does make any difference.
> {code:java}
> public void setLibFilesArchives(Context context, Element actionXml, Path
> appPath, Configuration conf)
> throws ActionExecutorException {
> addWfApplicationLibs(appPath, conf, context.getProtoActionConf());
> addActionXmlFilesAndArchives(actionXml, appPath, conf);
> initShareLibExcluder(conf, context);
> addActionLibs(appPath, conf);
> addAllShareLibs(appPath, conf, context, actionXml);
> }{code}
> First setLibFilesArchives is invoked in JavaActionExecutor#submitLauncher and
> then submitLauncher calls createLauncherConf, where its invoked again, but
> one of them is unnecessary.
> I tested around a bit and its probably safe to remove the setLibFilesArchives
> call for example from submitLauncher.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Created] (OOZIE-3580) setLibFilesArchives is called twice unnecessarily
Mate Juhasz created OOZIE-3580:
--
Summary: setLibFilesArchives is called twice unnecessarily
Key: OOZIE-3580
URL: https://issues.apache.org/jira/browse/OOZIE-3580
Project: Oozie
Issue Type: Bug
Components: core
Affects Versions: 5.2.0
Reporter: Mate Juhasz
Assignee: Mate Juhasz
Fix For: trunk
The JavaActionExecutor#setLibFilesArchives method can be costly as its
iterating through all the files, archives, lib files and share lib files added
for an action. Its invoked twice per workflow at the moment, but the second
call does make any difference.
{code:java}
public void setLibFilesArchives(Context context, Element actionXml, Path
appPath, Configuration conf)
throws ActionExecutorException {
addWfApplicationLibs(appPath, conf, context.getProtoActionConf());
addActionXmlFilesAndArchives(actionXml, appPath, conf);
initShareLibExcluder(conf, context);
addActionLibs(appPath, conf);
addAllShareLibs(appPath, conf, context, actionXml);
}{code}
First setLibFilesArchives is invoked in JavaActionExecutor#submitLauncher and
then submitLauncher calls createLauncherConf, where its invoked again, but one
of them is unnecessary.
I tested around a bit and its probably safe to remove the setLibFilesArchives
call for example from submitLauncher.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (OOZIE-3574) JavaAction create incorrect fileSystem instance in addActionLibs method
[
https://issues.apache.org/jira/browse/OOZIE-3574?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17016113#comment-17016113
]
Mate Juhasz commented on OOZIE-3574:
pinging [~zuston] :)
> JavaAction create incorrect fileSystem instance in addActionLibs method
> ---
>
> Key: OOZIE-3574
> URL: https://issues.apache.org/jira/browse/OOZIE-3574
> Project: Oozie
> Issue Type: Sub-task
>Reporter: Junfan Zhang
>Assignee: Junfan Zhang
>Priority: Major
> Attachments: OOZIE-3574-v1.patch
>
>
> Code is
> [here|https://github.com/apache/oozie/blob/9c288fe5cea6f2fbbae76f720b9e215acdd07709/core/src/main/java/org/apache/oozie/action/hadoop/JavaActionExecutor.java#L734].
> If actionlibPath scheme is different from appPath (like actionLibPath's
> scheme is s3a, but the appPath is hdfs), this will fail to execute
> {{fs.exist(actionLibsPath)}}. So i think Oozie should create fileSystem by
> actionLibsPath.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (OOZIE-3575) Add credential support for cloud file systems
[
https://issues.apache.org/jira/browse/OOZIE-3575?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17016038#comment-17016038
]
Mate Juhasz commented on OOZIE-3575:
Prerequisite oozie-site config changes:
{code:java}
'oozie.service.HadoopAccessorService.supported.filesystems': '*'
'oozie.service.HadoopAccessorService.nameNode.whitelist': ''
{code}
Added s3a filesystem credentials to a simple distcp workflow:
{code:xml}
filesystem.path
${my_s3a_bucket}
${resourceManager}
${nameNode}
-update
-skipcrccheck
${basePath}/${inputDir}/data.txt
${basePath}/${outputDir}/data.txt
{code}
The following scenarios were tested:
* workflow.xml: s3a, basePath: s3a
* workflow.xml: s3a, basePath: HDFS
* workflow.xml: HDFS, basePath: s3a - extra filesystem credentials only needed
in this case
> Add credential support for cloud file systems
> -
>
> Key: OOZIE-3575
> URL: https://issues.apache.org/jira/browse/OOZIE-3575
> Project: Oozie
> Issue Type: Sub-task
> Components: core
>Affects Versions: 5.2.0
>Reporter: Mate Juhasz
>Assignee: Mate Juhasz
>Priority: Major
> Fix For: trunk
>
> Attachments: OOZIE-3575-v2.patch, OOZIE-3575-v3.patch,
> OOZIE-3575.patch
>
>
> Oozie by default gathers delegation tokens for the nodes defined in
> _mapreduce.job.hdfs-servers_ (or _oozie.launcher.mapreduce.job.hdfs-servers_
> in case of distcp actions) and for the workflow path.
> Though this implementation is good for hdfs, we dont support occasions where
> the job related resources, which we want to access in runtime are present on
> different file systems/buckets etc...
> The HDFSCredentials class should be revised to handle getting tokens for
> different cloud storages.
> *The following scenarios should be addressed:*
> Oozie should obtain delegation token in case
> * the defaultFs is cloud
> * the workload.xml is in cloud
> * input/output/auxiliary files referred from workflow are in cloud
> * (newly introduced feature) user could define filesystem credentials for the
> workflow (as its done with hive/hcat etc..) -> this would allow the user to
> handle the situation where Oozie could not decide which tokens are needed at
> launch time by default and could also get tokens for different cloud storages
> and buckets as well
> Example for credentials addition:
> {noformat}
>
>
> filesystem
> s3a://qe-s3-bucket-mst
>
>
> {noformat}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (OOZIE-3575) Add credential support for cloud file systems
[
https://issues.apache.org/jira/browse/OOZIE-3575?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17014301#comment-17014301
]
Mate Juhasz commented on OOZIE-3575:
Thanks [~asalamon74] and [~gezapeti]! I dont like this test failure either,
trying to address it with a new patch. Still running some tests for each
scenarios, I will update with the results.
> Add credential support for cloud file systems
> -
>
> Key: OOZIE-3575
> URL: https://issues.apache.org/jira/browse/OOZIE-3575
> Project: Oozie
> Issue Type: Sub-task
> Components: core
>Affects Versions: 5.2.0
>Reporter: Mate Juhasz
>Assignee: Mate Juhasz
>Priority: Major
> Fix For: trunk
>
> Attachments: OOZIE-3575-v2.patch, OOZIE-3575-v3.patch,
> OOZIE-3575.patch
>
>
> Oozie by default gathers delegation tokens for the nodes defined in
> _mapreduce.job.hdfs-servers_ (or _oozie.launcher.mapreduce.job.hdfs-servers_
> in case of distcp actions) and for the workflow path.
> Though this implementation is good for hdfs, we dont support occasions where
> the job related resources, which we want to access in runtime are present on
> different file systems/buckets etc...
> The HDFSCredentials class should be revised to handle getting tokens for
> different cloud storages.
> *The following scenarios should be addressed:*
> Oozie should obtain delegation token in case
> * the defaultFs is cloud
> * the workload.xml is in cloud
> * input/output/auxiliary files referred from workflow are in cloud
> * (newly introduced feature) user could define filesystem credentials for the
> workflow (as its done with hive/hcat etc..) -> this would allow the user to
> handle the situation where Oozie could not decide which tokens are needed at
> launch time by default and could also get tokens for different cloud storages
> and buckets as well
> Example for credentials addition:
> {noformat}
>
>
> filesystem
> s3a://qe-s3-bucket-mst
>
>
> {noformat}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (OOZIE-3575) Add credential support for cloud file systems
[
https://issues.apache.org/jira/browse/OOZIE-3575?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Mate Juhasz updated OOZIE-3575:
---
Attachment: OOZIE-3575-v3.patch
> Add credential support for cloud file systems
> -
>
> Key: OOZIE-3575
> URL: https://issues.apache.org/jira/browse/OOZIE-3575
> Project: Oozie
> Issue Type: Sub-task
> Components: core
>Affects Versions: 5.2.0
>Reporter: Mate Juhasz
>Assignee: Mate Juhasz
>Priority: Major
> Fix For: trunk
>
> Attachments: OOZIE-3575-v2.patch, OOZIE-3575-v3.patch,
> OOZIE-3575.patch
>
>
> Oozie by default gathers delegation tokens for the nodes defined in
> _mapreduce.job.hdfs-servers_ (or _oozie.launcher.mapreduce.job.hdfs-servers_
> in case of distcp actions) and for the workflow path.
> Though this implementation is good for hdfs, we dont support occasions where
> the job related resources, which we want to access in runtime are present on
> different file systems/buckets etc...
> The HDFSCredentials class should be revised to handle getting tokens for
> different cloud storages.
> *The following scenarios should be addressed:*
> Oozie should obtain delegation token in case
> * the defaultFs is cloud
> * the workload.xml is in cloud
> * input/output/auxiliary files referred from workflow are in cloud
> * (newly introduced feature) user could define filesystem credentials for the
> workflow (as its done with hive/hcat etc..) -> this would allow the user to
> handle the situation where Oozie could not decide which tokens are needed at
> launch time by default and could also get tokens for different cloud storages
> and buckets as well
> Example for credentials addition:
> {noformat}
>
>
> filesystem
> s3a://qe-s3-bucket-mst
>
>
> {noformat}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (OOZIE-3575) Add credential support for cloud file systems
[
https://issues.apache.org/jira/browse/OOZIE-3575?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Mate Juhasz updated OOZIE-3575:
---
Attachment: OOZIE-3575-v2.patch
> Add credential support for cloud file systems
> -
>
> Key: OOZIE-3575
> URL: https://issues.apache.org/jira/browse/OOZIE-3575
> Project: Oozie
> Issue Type: Sub-task
> Components: core
>Affects Versions: 5.2.0
>Reporter: Mate Juhasz
>Assignee: Mate Juhasz
>Priority: Major
> Fix For: trunk
>
> Attachments: OOZIE-3575-v2.patch, OOZIE-3575.patch
>
>
> Oozie by default gathers delegation tokens for the nodes defined in
> _mapreduce.job.hdfs-servers_ (or _oozie.launcher.mapreduce.job.hdfs-servers_
> in case of distcp actions) and for the workflow path.
> Though this implementation is good for hdfs, we dont support occasions where
> the job related resources, which we want to access in runtime are present on
> different file systems/buckets etc...
> The HDFSCredentials class should be revised to handle getting tokens for
> different cloud storages.
> *The following scenarios should be addressed:*
> Oozie should obtain delegation token in case
> * the defaultFs is cloud
> * the workload.xml is in cloud
> * input/output/auxiliary files referred from workflow are in cloud
> * (newly introduced feature) user could define filesystem credentials for the
> workflow (as its done with hive/hcat etc..) -> this would allow the user to
> handle the situation where Oozie could not decide which tokens are needed at
> launch time by default and could also get tokens for different cloud storages
> and buckets as well
> Example for credentials addition:
> {noformat}
>
>
> filesystem
> s3a://qe-s3-bucket-mst
>
>
> {noformat}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (OOZIE-3575) Add credential support for cloud file systems
[
https://issues.apache.org/jira/browse/OOZIE-3575?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Mate Juhasz updated OOZIE-3575:
---
Attachment: OOZIE-3575.patch
> Add credential support for cloud file systems
> -
>
> Key: OOZIE-3575
> URL: https://issues.apache.org/jira/browse/OOZIE-3575
> Project: Oozie
> Issue Type: Sub-task
> Components: core
>Affects Versions: 5.2.0
>Reporter: Mate Juhasz
>Assignee: Mate Juhasz
>Priority: Major
> Fix For: trunk
>
> Attachments: OOZIE-3575.patch
>
>
> Oozie by default gathers delegation tokens for the nodes defined in
> _mapreduce.job.hdfs-servers_ (or _oozie.launcher.mapreduce.job.hdfs-servers_
> in case of distcp actions) and for the workflow path.
> Though this implementation is good for hdfs, we dont support occasions where
> the job related resources, which we want to access in runtime are present on
> different file systems/buckets etc...
> The HDFSCredentials class should be revised to handle getting tokens for
> different cloud storages.
> *The following scenarios should be addressed:*
> Oozie should obtain delegation token in case
> * the defaultFs is cloud
> * the workload.xml is in cloud
> * input/output/auxiliary files referred from workflow are in cloud
> * (newly introduced feature) user could define filesystem credentials for the
> workflow (as its done with hive/hcat etc..) -> this would allow the user to
> handle the situation where Oozie could not decide which tokens are needed at
> launch time by default and could also get tokens for different cloud storages
> and buckets as well
> Example for credentials addition:
> {noformat}
>
>
> filesystem
> s3a://qe-s3-bucket-mst
>
>
> {noformat}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (OOZIE-3575) Add credential support for cloud file systems
[
https://issues.apache.org/jira/browse/OOZIE-3575?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17000820#comment-17000820
]
Mate Juhasz commented on OOZIE-3575:
The recently opened OOZIE-3574 is actually a bug, but its strongly connecting
to OOZIE-3575
> Add credential support for cloud file systems
> -
>
> Key: OOZIE-3575
> URL: https://issues.apache.org/jira/browse/OOZIE-3575
> Project: Oozie
> Issue Type: Improvement
> Components: core
>Affects Versions: 5.2.0
>Reporter: Mate Juhasz
>Assignee: Mate Juhasz
>Priority: Major
> Fix For: trunk
>
>
> Oozie by default gathers delegation tokens for the nodes defined in
> _mapreduce.job.hdfs-servers_ (or _oozie.launcher.mapreduce.job.hdfs-servers_
> in case of distcp actions) and for the workflow path.
> Though this implementation is good for hdfs, we dont support occasions where
> the job related resources, which we want to access in runtime are present on
> different file systems/buckets etc...
> The HDFSCredentials class should be revised to handle getting tokens for
> different cloud storages.
> *The following scenarios should be addressed:*
> Oozie should obtain delegation token in case
> * the defaultFs is cloud
> * the workload.xml is in cloud
> * input/output/auxiliary files referred from workflow are in cloud
> * (newly introduced feature) user could define filesystem credentials for the
> workflow (as its done with hive/hcat etc..) -> this would allow the user to
> handle the situation where Oozie could not decide which tokens are needed at
> launch time by default and could also get tokens for different cloud storages
> and buckets as well
> Example for credentials addition:
> {noformat}
>
>
> filesystem
> s3a://qe-s3-bucket-mst
>
>
> {noformat}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (OOZIE-3575) Add credential support for cloud file systems
[
https://issues.apache.org/jira/browse/OOZIE-3575?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Mate Juhasz updated OOZIE-3575:
---
Issue Type: Improvement (was: Bug)
> Add credential support for cloud file systems
> -
>
> Key: OOZIE-3575
> URL: https://issues.apache.org/jira/browse/OOZIE-3575
> Project: Oozie
> Issue Type: Improvement
> Components: core
>Affects Versions: 5.2.0
>Reporter: Mate Juhasz
>Assignee: Mate Juhasz
>Priority: Major
> Fix For: trunk
>
>
> Oozie by default gathers delegation tokens for the nodes defined in
> _mapreduce.job.hdfs-servers_ (or _oozie.launcher.mapreduce.job.hdfs-servers_
> in case of distcp actions) and for the workflow path.
> Though this implementation is good for hdfs, we dont support occasions where
> the job related resources, which we want to access in runtime are present on
> different file systems/buckets etc...
> The HDFSCredentials class should be revised to handle getting tokens for
> different cloud storages.
> *The following scenarios should be addressed:*
> Oozie should obtain delegation token in case
> * the defaultFs is cloud
> * the workload.xml is in cloud
> * input/output/auxiliary files referred from workflow are in cloud
> * (newly introduced feature) user could define filesystem credentials for the
> workflow (as its done with hive/hcat etc..) -> this would allow the user to
> handle the situation where Oozie could not decide which tokens are needed at
> launch time by default and could also get tokens for different cloud storages
> and buckets as well
> Example for credentials addition:
> {noformat}
>
>
> filesystem
> s3a://qe-s3-bucket-mst
>
>
> {noformat}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (OOZIE-3575) Add credential support for cloud file systems
[
https://issues.apache.org/jira/browse/OOZIE-3575?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17000817#comment-17000817
]
Mate Juhasz commented on OOZIE-3575:
cc: [~gezapeti] [~asalamon74] [~dionusos]
> Add credential support for cloud file systems
> -
>
> Key: OOZIE-3575
> URL: https://issues.apache.org/jira/browse/OOZIE-3575
> Project: Oozie
> Issue Type: Bug
> Components: core
>Affects Versions: 5.2.0
>Reporter: Mate Juhasz
>Assignee: Mate Juhasz
>Priority: Major
> Fix For: trunk
>
>
> Oozie by default gathers delegation tokens for the nodes defined in
> _mapreduce.job.hdfs-servers_ (or _oozie.launcher.mapreduce.job.hdfs-servers_
> in case of distcp actions) and for the workflow path.
> Though this implementation is good for hdfs, we dont support occasions where
> the job related resources, which we want to access in runtime are present on
> different file systems/buckets etc...
> The HDFSCredentials class should be revised to handle getting tokens for
> different cloud storages.
> *The following scenarios should be addressed:*
> Oozie should obtain delegation token in case
> * the defaultFs is cloud
> * the workload.xml is in cloud
> * input/output/auxiliary files referred from workflow are in cloud
> * (newly introduced feature) user could define filesystem credentials for the
> workflow (as its done with hive/hcat etc..) -> this would allow the user to
> handle the situation where Oozie could not decide which tokens are needed at
> launch time by default and could also get tokens for different cloud storages
> and buckets as well
> Example for credentials addition:
> {noformat}
>
>
> filesystem
> s3a://qe-s3-bucket-mst
>
>
> {noformat}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (OOZIE-3575) Add credential support for cloud file systems
[
https://issues.apache.org/jira/browse/OOZIE-3575?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Mate Juhasz updated OOZIE-3575:
---
Summary: Add credential support for cloud file systems (was: Add
credential support for cloud file storages)
> Add credential support for cloud file systems
> -
>
> Key: OOZIE-3575
> URL: https://issues.apache.org/jira/browse/OOZIE-3575
> Project: Oozie
> Issue Type: Bug
> Components: core
>Affects Versions: 5.2.0
>Reporter: Mate Juhasz
>Assignee: Mate Juhasz
>Priority: Major
> Fix For: trunk
>
>
> Oozie by default gathers delegation tokens for the nodes defined in
> _mapreduce.job.hdfs-servers_ (or _oozie.launcher.mapreduce.job.hdfs-servers_
> in case of distcp actions) and for the workflow path.
> Though this implementation is good for hdfs, we dont support occasions where
> the job related resources, which we want to access in runtime are present on
> different file systems/buckets etc...
> The HDFSCredentials class should be revised to handle getting tokens for
> different cloud storages.
> *The following scenarios should be addressed:*
> Oozie should obtain delegation token in case
> * the defaultFs is cloud
> * the workload.xml is in cloud
> * input/output/auxiliary files referred from workflow are in cloud
> * (newly introduced feature) user could define filesystem credentials for the
> workflow (as its done with hive/hcat etc..) -> this would allow the user to
> handle the situation where Oozie could not decide which tokens are needed at
> launch time by default and could also get tokens for different cloud storages
> and buckets as well
> Example for credentials addition:
> {noformat}
>
>
> filesystem
> s3a://qe-s3-bucket-mst
>
>
> {noformat}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Created] (OOZIE-3575) Add credential support for cloud file storages
Mate Juhasz created OOZIE-3575:
--
Summary: Add credential support for cloud file storages
Key: OOZIE-3575
URL: https://issues.apache.org/jira/browse/OOZIE-3575
Project: Oozie
Issue Type: Bug
Components: core
Affects Versions: 5.2.0
Reporter: Mate Juhasz
Assignee: Mate Juhasz
Fix For: trunk
Oozie by default gathers delegation tokens for the nodes defined in
_mapreduce.job.hdfs-servers_ (or _oozie.launcher.mapreduce.job.hdfs-servers_ in
case of distcp actions) and for the workflow path.
Though this implementation is good for hdfs, we dont support occasions where
the job related resources, which we want to access in runtime are present on
different file systems/buckets etc...
The HDFSCredentials class should be revised to handle getting tokens for
different cloud storages.
*The following scenarios should be addressed:*
Oozie should obtain delegation token in case
* the defaultFs is cloud
* the workload.xml is in cloud
* input/output/auxiliary files referred from workflow are in cloud
* (newly introduced feature) user could define filesystem credentials for the
workflow (as its done with hive/hcat etc..) -> this would allow the user to
handle the situation where Oozie could not decide which tokens are needed at
launch time by default and could also get tokens for different cloud storages
and buckets as well
Example for credentials addition:
{noformat}
filesystem
s3a://qe-s3-bucket-mst
{noformat}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (OOZIE-3574) JavaAction create uncorrect fileSystem instance in addActionLibs method
[
https://issues.apache.org/jira/browse/OOZIE-3574?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17000791#comment-17000791
]
Mate Juhasz commented on OOZIE-3574:
Thats a great finding [~zuston]! That FileSystem instance should be created
definitely with actionLibsPath instead of appPath.
Its probably a minor change in the below code part, but I did not check in
depths.
{code:java}
FileSystem fs =
Services.get().get(HadoopAccessorService.class).createFileSystem(user,
appPath.toUri(), conf);
{code}
This could be a further enhancement for making Oozie more cloud file system
friendly.
> JavaAction create uncorrect fileSystem instance in addActionLibs method
> ---
>
> Key: OOZIE-3574
> URL: https://issues.apache.org/jira/browse/OOZIE-3574
> Project: Oozie
> Issue Type: Bug
>Reporter: Junfan Zhang
>Assignee: Junfan Zhang
>Priority: Major
>
> Code is
> [here|https://github.com/apache/oozie/blob/9c288fe5cea6f2fbbae76f720b9e215acdd07709/core/src/main/java/org/apache/oozie/action/hadoop/JavaActionExecutor.java#L734].
> If actionlibPath scheme is different from appPath (like actionLibPath's
> scheme is s3a, but the appPath is hdfs), this will fail to execute
> {{fs.exist(actionLibsPath)}}. So i think Oozie should create fileSystem by
> actionLibsPath.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (OOZIE-2966) OozieCLI and TestOozieCLI test class are in different packages
[ https://issues.apache.org/jira/browse/OOZIE-2966?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16991422#comment-16991422 ] Mate Juhasz commented on OOZIE-2966: The idea is good and valid I think, but the solution is not that trivial unfortunately. If we move the test classes to the client project we would need to reference for example to core and fluent-job sub-projects and that would mean circular dependencies among core and client. The only solution for resolving the circularity is to restructure a lot of code, but this way I dont think it worth the time. > OozieCLI and TestOozieCLI test class are in different packages > -- > > Key: OOZIE-2966 > URL: https://issues.apache.org/jira/browse/OOZIE-2966 > Project: Oozie > Issue Type: Task > Components: tests >Affects Versions: 4.3.0 >Reporter: Artem Ervits >Assignee: Artem Ervits >Priority: Trivial > > OozieCLI class is in client/org/apache/oozie/cli whereas the associated test > class resides in core/org/apache/oozie/client shouldn't they belong to same > project? -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (OOZIE-3562) Fix Hive example
[
https://issues.apache.org/jira/browse/OOZIE-3562?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16984312#comment-16984312
]
Mate Juhasz commented on OOZIE-3562:
[~kmarton] could you share some info about the env where it is failing? I could
see this with Java 11, so I suspected Java 8+ is not yet supported by the Hive
version we use upstream
> Fix Hive example
>
>
> Key: OOZIE-3562
> URL: https://issues.apache.org/jira/browse/OOZIE-3562
> Project: Oozie
> Issue Type: Bug
> Components: examples
>Affects Versions: 5.1.0
>Reporter: Kinga Marton
>Priority: Major
> Attachments: hive1StachTrace.txt
>
>
> Hive example in pseudo distributed mode is failing with the following error
> message:
> {code:java}
> ACTION[008-191121145732587-oozie-mart-W@hive-node] Launcher exception:
> java.lang.RuntimeException: Unable to instantiate
> org.apache.hadoop.hive.ql.metadata.SessionHiveMetaStoreClient
> {code}
> See the full stacktrace attached.
>
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (OOZIE-3549) Add back support for truststore passwords
[
https://issues.apache.org/jira/browse/OOZIE-3549?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Mate Juhasz updated OOZIE-3549:
---
Attachment: OOZIE-3549-v4.patch
> Add back support for truststore passwords
> -
>
> Key: OOZIE-3549
> URL: https://issues.apache.org/jira/browse/OOZIE-3549
> Project: Oozie
> Issue Type: Improvement
>Affects Versions: trunk
>Reporter: Andras Salamon
>Assignee: Mate Juhasz
>Priority: Major
> Attachments: OOZIE-3549-v2.patch, OOZIE-3549-v3.patch,
> OOZIE-3549-v4.patch, OOZIE-3549.patch
>
>
> OOZIE-3157 removed {{oozie.https.truststore.pass}} property, because we
> (Oozie + Jetty) don't write the truststore and the password is not required
> for reading.
> This is no longer true, Java 11's keytool now defaults to creating PKCS12
> keystores instead of JKS, and according to
> [this|https://bugs.launchpad.net/ubuntu/+source/ca-certificates-java/+bug/1771363]
> bug description "A JKS keystore can be read without supplying a password (or
> by supplying an empty one) while a PKCS12 keystore requires a password to be
> set."
> We should reintroduce this property and allow the it again to specify this
> password and pass it to jetty.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (OOZIE-3549) Add back support for truststore passwords
[
https://issues.apache.org/jira/browse/OOZIE-3549?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16983340#comment-16983340
]
Mate Juhasz commented on OOZIE-3549:
Uploaded new patch, where I modified the teardown of the failing unit tests
where we wanted to verify the call of setKeyManagerPassword method
> Add back support for truststore passwords
> -
>
> Key: OOZIE-3549
> URL: https://issues.apache.org/jira/browse/OOZIE-3549
> Project: Oozie
> Issue Type: Improvement
>Affects Versions: trunk
>Reporter: Andras Salamon
>Assignee: Mate Juhasz
>Priority: Major
> Attachments: OOZIE-3549-v2.patch, OOZIE-3549-v3.patch,
> OOZIE-3549.patch
>
>
> OOZIE-3157 removed {{oozie.https.truststore.pass}} property, because we
> (Oozie + Jetty) don't write the truststore and the password is not required
> for reading.
> This is no longer true, Java 11's keytool now defaults to creating PKCS12
> keystores instead of JKS, and according to
> [this|https://bugs.launchpad.net/ubuntu/+source/ca-certificates-java/+bug/1771363]
> bug description "A JKS keystore can be read without supplying a password (or
> by supplying an empty one) while a PKCS12 keystore requires a password to be
> set."
> We should reintroduce this property and allow the it again to specify this
> password and pass it to jetty.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (OOZIE-3549) Add back support for truststore passwords
[
https://issues.apache.org/jira/browse/OOZIE-3549?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Mate Juhasz updated OOZIE-3549:
---
Attachment: OOZIE-3549-v3.patch
> Add back support for truststore passwords
> -
>
> Key: OOZIE-3549
> URL: https://issues.apache.org/jira/browse/OOZIE-3549
> Project: Oozie
> Issue Type: Improvement
>Affects Versions: trunk
>Reporter: Andras Salamon
>Assignee: Mate Juhasz
>Priority: Major
> Attachments: OOZIE-3549-v2.patch, OOZIE-3549-v3.patch,
> OOZIE-3549.patch
>
>
> OOZIE-3157 removed {{oozie.https.truststore.pass}} property, because we
> (Oozie + Jetty) don't write the truststore and the password is not required
> for reading.
> This is no longer true, Java 11's keytool now defaults to creating PKCS12
> keystores instead of JKS, and according to
> [this|https://bugs.launchpad.net/ubuntu/+source/ca-certificates-java/+bug/1771363]
> bug description "A JKS keystore can be read without supplying a password (or
> by supplying an empty one) while a PKCS12 keystore requires a password to be
> set."
> We should reintroduce this property and allow the it again to specify this
> password and pass it to jetty.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (OOZIE-3549) Add back support for truststore passwords
[
https://issues.apache.org/jira/browse/OOZIE-3549?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Mate Juhasz updated OOZIE-3549:
---
Attachment: OOZIE-3549-v2.patch
> Add back support for truststore passwords
> -
>
> Key: OOZIE-3549
> URL: https://issues.apache.org/jira/browse/OOZIE-3549
> Project: Oozie
> Issue Type: Improvement
>Affects Versions: trunk
>Reporter: Andras Salamon
>Assignee: Mate Juhasz
>Priority: Major
> Attachments: OOZIE-3549-v2.patch, OOZIE-3549.patch
>
>
> OOZIE-3157 removed {{oozie.https.truststore.pass}} property, because we
> (Oozie + Jetty) don't write the truststore and the password is not required
> for reading.
> This is no longer true, Java 11's keytool now defaults to creating PKCS12
> keystores instead of JKS, and according to
> [this|https://bugs.launchpad.net/ubuntu/+source/ca-certificates-java/+bug/1771363]
> bug description "A JKS keystore can be read without supplying a password (or
> by supplying an empty one) while a PKCS12 keystore requires a password to be
> set."
> We should reintroduce this property and allow the it again to specify this
> password and pass it to jetty.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (OOZIE-3549) Add back support for truststore passwords
[
https://issues.apache.org/jira/browse/OOZIE-3549?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Mate Juhasz updated OOZIE-3549:
---
Attachment: (was: OOZIE-3549-v2.patch)
> Add back support for truststore passwords
> -
>
> Key: OOZIE-3549
> URL: https://issues.apache.org/jira/browse/OOZIE-3549
> Project: Oozie
> Issue Type: Improvement
>Affects Versions: trunk
>Reporter: Andras Salamon
>Assignee: Mate Juhasz
>Priority: Major
> Attachments: OOZIE-3549-v2.patch, OOZIE-3549.patch
>
>
> OOZIE-3157 removed {{oozie.https.truststore.pass}} property, because we
> (Oozie + Jetty) don't write the truststore and the password is not required
> for reading.
> This is no longer true, Java 11's keytool now defaults to creating PKCS12
> keystores instead of JKS, and according to
> [this|https://bugs.launchpad.net/ubuntu/+source/ca-certificates-java/+bug/1771363]
> bug description "A JKS keystore can be read without supplying a password (or
> by supplying an empty one) while a PKCS12 keystore requires a password to be
> set."
> We should reintroduce this property and allow the it again to specify this
> password and pass it to jetty.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (OOZIE-3549) Add back support for truststore passwords
[
https://issues.apache.org/jira/browse/OOZIE-3549?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Mate Juhasz updated OOZIE-3549:
---
Attachment: OOZIE-3549-v2.patch
> Add back support for truststore passwords
> -
>
> Key: OOZIE-3549
> URL: https://issues.apache.org/jira/browse/OOZIE-3549
> Project: Oozie
> Issue Type: Improvement
>Affects Versions: trunk
>Reporter: Andras Salamon
>Assignee: Mate Juhasz
>Priority: Major
> Attachments: OOZIE-3549-v2.patch, OOZIE-3549.patch
>
>
> OOZIE-3157 removed {{oozie.https.truststore.pass}} property, because we
> (Oozie + Jetty) don't write the truststore and the password is not required
> for reading.
> This is no longer true, Java 11's keytool now defaults to creating PKCS12
> keystores instead of JKS, and according to
> [this|https://bugs.launchpad.net/ubuntu/+source/ca-certificates-java/+bug/1771363]
> bug description "A JKS keystore can be read without supplying a password (or
> by supplying an empty one) while a PKCS12 keystore requires a password to be
> set."
> We should reintroduce this property and allow the it again to specify this
> password and pass it to jetty.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (OOZIE-3549) Add back support for truststore passwords
[
https://issues.apache.org/jira/browse/OOZIE-3549?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16982579#comment-16982579
]
Mate Juhasz commented on OOZIE-3549:
Uploading a new patch, tested in a deployment. Please see the diff in
SSLServerConnectorFactory#setKeystorePass.
Some explanation on the relation of KeyManagerPassword and KeyStorePassword
from the eclipse docs:
https://www.eclipse.org/jetty/documentation/9.3.27.v20190418/configuring-ssl.html#configuring-sslcontextfactory
I would say that using a PKCS12 keystore requires the
sslContextFactory.KeyStorePassword to be set instead (or besides) of
KeyManagerPassword. If jetty's KeyManagerPassword is not set the
KeyStorePassword will be used for that as well.
> Add back support for truststore passwords
> -
>
> Key: OOZIE-3549
> URL: https://issues.apache.org/jira/browse/OOZIE-3549
> Project: Oozie
> Issue Type: Improvement
>Affects Versions: trunk
>Reporter: Andras Salamon
>Assignee: Mate Juhasz
>Priority: Major
> Attachments: OOZIE-3549.patch
>
>
> OOZIE-3157 removed {{oozie.https.truststore.pass}} property, because we
> (Oozie + Jetty) don't write the truststore and the password is not required
> for reading.
> This is no longer true, Java 11's keytool now defaults to creating PKCS12
> keystores instead of JKS, and according to
> [this|https://bugs.launchpad.net/ubuntu/+source/ca-certificates-java/+bug/1771363]
> bug description "A JKS keystore can be read without supplying a password (or
> by supplying an empty one) while a PKCS12 keystore requires a password to be
> set."
> We should reintroduce this property and allow the it again to specify this
> password and pass it to jetty.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (OOZIE-3491) Confusing System ID error message
[
https://issues.apache.org/jira/browse/OOZIE-3491?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16982497#comment-16982497
]
Mate Juhasz commented on OOZIE-3491:
Oops, fixed :) thanks
> Confusing System ID error message
> -
>
> Key: OOZIE-3491
> URL: https://issues.apache.org/jira/browse/OOZIE-3491
> Project: Oozie
> Issue Type: Bug
>Affects Versions: trunk
>Reporter: Andras Salamon
>Assignee: Mate Juhasz
>Priority: Minor
> Labels: newbie
> Attachments: OOZIE-3491-V2.patch, OOZIE-3491-V3.patch,
> OOZIE-3491.patch
>
>
> If the System ID is longer than 10 characters Oozie trims it and prints out
> an error message:
> {noformat}
> System ID [oozie-oozi] exceeds maximum length [10], trimming
> {noformat}
> Which is confusing, because "oozie-oozi" does not exceeds 10 characters. This
> is not the original long system ID but the trimmed short one.
> I think we should fix the error message and print out both the original and
> the new value.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (OOZIE-3491) Confusing System ID error message
[
https://issues.apache.org/jira/browse/OOZIE-3491?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Mate Juhasz updated OOZIE-3491:
---
Attachment: OOZIE-3491-V3.patch
> Confusing System ID error message
> -
>
> Key: OOZIE-3491
> URL: https://issues.apache.org/jira/browse/OOZIE-3491
> Project: Oozie
> Issue Type: Bug
>Affects Versions: trunk
>Reporter: Andras Salamon
>Assignee: Mate Juhasz
>Priority: Minor
> Labels: newbie
> Attachments: OOZIE-3491-V2.patch, OOZIE-3491-V3.patch,
> OOZIE-3491.patch
>
>
> If the System ID is longer than 10 characters Oozie trims it and prints out
> an error message:
> {noformat}
> System ID [oozie-oozi] exceeds maximum length [10], trimming
> {noformat}
> Which is confusing, because "oozie-oozi" does not exceeds 10 characters. This
> is not the original long system ID but the trimmed short one.
> I think we should fix the error message and print out both the original and
> the new value.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (OOZIE-3491) Confusing System ID error message
[
https://issues.apache.org/jira/browse/OOZIE-3491?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Mate Juhasz updated OOZIE-3491:
---
Attachment: OOZIE-3491-V2.patch
> Confusing System ID error message
> -
>
> Key: OOZIE-3491
> URL: https://issues.apache.org/jira/browse/OOZIE-3491
> Project: Oozie
> Issue Type: Bug
>Affects Versions: trunk
>Reporter: Andras Salamon
>Assignee: Mate Juhasz
>Priority: Minor
> Labels: newbie
> Attachments: OOZIE-3491-V2.patch, OOZIE-3491.patch
>
>
> If the System ID is longer than 10 characters Oozie trims it and prints out
> an error message:
> {noformat}
> System ID [oozie-oozi] exceeds maximum length [10], trimming
> {noformat}
> Which is confusing, because "oozie-oozi" does not exceeds 10 characters. This
> is not the original long system ID but the trimmed short one.
> I think we should fix the error message and print out both the original and
> the new value.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (OOZIE-3549) Add back support for truststore passwords
[
https://issues.apache.org/jira/browse/OOZIE-3549?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16959011#comment-16959011
]
Mate Juhasz commented on OOZIE-3549:
Thanks [~asalamon74] for the summary. However it turned out, that the issue
might be somewhere else... there is this function
SSLServerConnectorFactory#setKeystorePass where we call
sslContextFactory.KeyManagerPassword(keystorePass) instead of
setKeyStorePassword.
The patch is working if we change this part, which I could not understand well,
so I would like to test it out better in a deployment.
> Add back support for truststore passwords
> -
>
> Key: OOZIE-3549
> URL: https://issues.apache.org/jira/browse/OOZIE-3549
> Project: Oozie
> Issue Type: Improvement
>Affects Versions: trunk
>Reporter: Andras Salamon
>Assignee: Mate Juhasz
>Priority: Major
> Attachments: OOZIE-3549.patch
>
>
> OOZIE-3157 removed {{oozie.https.truststore.pass}} property, because we
> (Oozie + Jetty) don't write the truststore and the password is not required
> for reading.
> This is no longer true, Java 11's keytool now defaults to creating PKCS12
> keystores instead of JKS, and according to
> [this|https://bugs.launchpad.net/ubuntu/+source/ca-certificates-java/+bug/1771363]
> bug description "A JKS keystore can be read without supplying a password (or
> by supplying an empty one) while a PKCS12 keystore requires a password to be
> set."
> We should reintroduce this property and allow the it again to specify this
> password and pass it to jetty.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (OOZIE-3487) Confusing E0820 error message
[
https://issues.apache.org/jira/browse/OOZIE-3487?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Mate Juhasz updated OOZIE-3487:
---
Attachment: OOZIE-3487.patch
> Confusing E0820 error message
> -
>
> Key: OOZIE-3487
> URL: https://issues.apache.org/jira/browse/OOZIE-3487
> Project: Oozie
> Issue Type: Bug
>Affects Versions: trunk
>Reporter: Andras Salamon
>Assignee: Mate Juhasz
>Priority: Minor
> Attachments: OOZIE-3487.patch
>
>
> E0820 error messages are confusing:
> {noformat}
> Action user retry max [3] is over system defined max [3], re-assign to use
> system max.{noformat}
> It's not true that 3 is over 3.
>
> It's a bug in
> [LiteWorkflowStoreService|https://github.com/apache/oozie/blob/master/core/src/main/java/org/apache/oozie/service/LiteWorkflowStoreService.java]
> {noformat}
> if (ret > max) {
> ret = max;
> log.warn(ErrorCode.E0820.getTemplate(), ret, max);
> }
> {noformat}
> We overwrite the max value before we print out the error messge.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Assigned] (OOZIE-3487) Confusing E0820 error message
[
https://issues.apache.org/jira/browse/OOZIE-3487?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Mate Juhasz reassigned OOZIE-3487:
--
Assignee: Mate Juhasz
> Confusing E0820 error message
> -
>
> Key: OOZIE-3487
> URL: https://issues.apache.org/jira/browse/OOZIE-3487
> Project: Oozie
> Issue Type: Bug
>Affects Versions: trunk
>Reporter: Andras Salamon
>Assignee: Mate Juhasz
>Priority: Minor
> Attachments: OOZIE-3487.patch
>
>
> E0820 error messages are confusing:
> {noformat}
> Action user retry max [3] is over system defined max [3], re-assign to use
> system max.{noformat}
> It's not true that 3 is over 3.
>
> It's a bug in
> [LiteWorkflowStoreService|https://github.com/apache/oozie/blob/master/core/src/main/java/org/apache/oozie/service/LiteWorkflowStoreService.java]
> {noformat}
> if (ret > max) {
> ret = max;
> log.warn(ErrorCode.E0820.getTemplate(), ret, max);
> }
> {noformat}
> We overwrite the max value before we print out the error messge.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (OOZIE-3491) Confusing System ID error message
[
https://issues.apache.org/jira/browse/OOZIE-3491?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Mate Juhasz updated OOZIE-3491:
---
Attachment: OOZIE-3491.patch
> Confusing System ID error message
> -
>
> Key: OOZIE-3491
> URL: https://issues.apache.org/jira/browse/OOZIE-3491
> Project: Oozie
> Issue Type: Bug
>Affects Versions: trunk
>Reporter: Andras Salamon
>Assignee: Mate Juhasz
>Priority: Minor
> Labels: newbie
> Attachments: OOZIE-3491.patch
>
>
> If the System ID is longer than 10 characters Oozie trims it and prints out
> an error message:
> {noformat}
> System ID [oozie-oozi] exceeds maximum length [10], trimming
> {noformat}
> Which is confusing, because "oozie-oozi" does not exceeds 10 characters. This
> is not the original long system ID but the trimmed short one.
> I think we should fix the error message and print out both the original and
> the new value.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Assigned] (OOZIE-3491) Confusing System ID error message
[
https://issues.apache.org/jira/browse/OOZIE-3491?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Mate Juhasz reassigned OOZIE-3491:
--
Assignee: Mate Juhasz
> Confusing System ID error message
> -
>
> Key: OOZIE-3491
> URL: https://issues.apache.org/jira/browse/OOZIE-3491
> Project: Oozie
> Issue Type: Bug
>Affects Versions: trunk
>Reporter: Andras Salamon
>Assignee: Mate Juhasz
>Priority: Minor
> Labels: newbie
>
> If the System ID is longer than 10 characters Oozie trims it and prints out
> an error message:
> {noformat}
> System ID [oozie-oozi] exceeds maximum length [10], trimming
> {noformat}
> Which is confusing, because "oozie-oozi" does not exceeds 10 characters. This
> is not the original long system ID but the trimmed short one.
> I think we should fix the error message and print out both the original and
> the new value.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (OOZIE-3549) Add back support for truststore passwords
[
https://issues.apache.org/jira/browse/OOZIE-3549?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16952869#comment-16952869
]
Mate Juhasz commented on OOZIE-3549:
Uploaded a patch, but it is still to be tested on a real deployment
> Add back support for truststore passwords
> -
>
> Key: OOZIE-3549
> URL: https://issues.apache.org/jira/browse/OOZIE-3549
> Project: Oozie
> Issue Type: Improvement
>Affects Versions: trunk
>Reporter: Andras Salamon
>Assignee: Mate Juhasz
>Priority: Major
> Attachments: OOZIE-3549.patch
>
>
> OOZIE-3157 removed {{oozie.https.truststore.pass}} property, because we
> (Oozie + Jetty) don't write the truststore and the password is not required
> for reading.
> This is no longer true, Java 11's keytool now defaults to creating PKCS12
> keystores instead of JKS, and according to
> [this|https://bugs.launchpad.net/ubuntu/+source/ca-certificates-java/+bug/1771363]
> bug description "A JKS keystore can be read without supplying a password (or
> by supplying an empty one) while a PKCS12 keystore requires a password to be
> set."
> We should reintroduce this property and allow the it again to specify this
> password and pass it to jetty.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (OOZIE-3549) Add back support for truststore passwords
[
https://issues.apache.org/jira/browse/OOZIE-3549?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Mate Juhasz updated OOZIE-3549:
---
Attachment: OOZIE-3549.patch
> Add back support for truststore passwords
> -
>
> Key: OOZIE-3549
> URL: https://issues.apache.org/jira/browse/OOZIE-3549
> Project: Oozie
> Issue Type: Improvement
>Affects Versions: trunk
>Reporter: Andras Salamon
>Assignee: Mate Juhasz
>Priority: Major
> Attachments: OOZIE-3549.patch
>
>
> OOZIE-3157 removed {{oozie.https.truststore.pass}} property, because we
> (Oozie + Jetty) don't write the truststore and the password is not required
> for reading.
> This is no longer true, Java 11's keytool now defaults to creating PKCS12
> keystores instead of JKS, and according to
> [this|https://bugs.launchpad.net/ubuntu/+source/ca-certificates-java/+bug/1771363]
> bug description "A JKS keystore can be read without supplying a password (or
> by supplying an empty one) while a PKCS12 keystore requires a password to be
> set."
> We should reintroduce this property and allow the it again to specify this
> password and pass it to jetty.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Assigned] (OOZIE-3549) Add back support for truststore passwords
[
https://issues.apache.org/jira/browse/OOZIE-3549?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Mate Juhasz reassigned OOZIE-3549:
--
Assignee: Mate Juhasz
> Add back support for truststore passwords
> -
>
> Key: OOZIE-3549
> URL: https://issues.apache.org/jira/browse/OOZIE-3549
> Project: Oozie
> Issue Type: Improvement
>Affects Versions: trunk
>Reporter: Andras Salamon
>Assignee: Mate Juhasz
>Priority: Major
>
> OOZIE-3157 removed {{oozie.https.truststore.pass}} property, because we
> (Oozie + Jetty) don't write the truststore and the password is not required
> for reading.
> This is no longer true, Java 11's keytool now defaults to creating PKCS12
> keystores instead of JKS, and according to
> [this|https://bugs.launchpad.net/ubuntu/+source/ca-certificates-java/+bug/1771363]
> bug description "A JKS keystore can be read without supplying a password (or
> by supplying an empty one) while a PKCS12 keystore requires a password to be
> set."
> We should reintroduce this property and allow the it again to specify this
> password and pass it to jetty.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (OOZIE-1974) SSH Action doesn't handle compound commands eg: cmd1 && cmd2 and stuck in [PREP] stage
[
https://issues.apache.org/jira/browse/OOZIE-1974?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16948605#comment-16948605
]
Mate Juhasz commented on OOZIE-1974:
Uploaded a patch with initial ideas... not sure about how much business value
is added by this "feature" though. Ssh unit tests are still not working, we
should fix them before, so no new tests this time
> SSH Action doesn't handle compound commands eg: cmd1 && cmd2 and stuck in
> [PREP] stage
> --
>
> Key: OOZIE-1974
> URL: https://issues.apache.org/jira/browse/OOZIE-1974
> Project: Oozie
> Issue Type: Bug
> Components: action
>Affects Versions: trunk
>Reporter: Michalis Kongtongk
>Assignee: Mate Juhasz
>Priority: Major
> Attachments: OOZIE-1974.patch
>
>
> example WF that will fail:
> {code}
>
>
>
>
> oozie-u...@somedomain.com
> kinit
> oozie-u...@somedomain.com
> -k
> -t
> /home/oozie-user/oozie.keytab
>
> hdfs
> dfs
> -put
> /tmp/random-file.txt
> /tmp/random-file.txt
>
>
>
>
>
> Action failed, error
> message[${wf:errorMessage(wf:lastErrorNode())}]
>
>
>
> {code}
> Workaround is to execute the compound command in subshell eg: $(cmd1 && cmd2)
> {code}
>
>
>
>
> oozie-u...@somedomain.com
> $(kinit
> oozie-u...@somedomain.com
> -k
> -t
> /home/oozie-user/oozie.keytab
>
> hdfs
> dfs
> -put
> /tmp/random-file.txt
> /tmp/random-file.txt
> )
>
>
>
>
>
> Action failed, error
> message[${wf:errorMessage(wf:lastErrorNode())}]
>
>
>
> {code}
> Stack trace "org.apache.oozie.command.CommandException: E0800: Action it is
> not running its in [PREP] state,"
> {code}
> 2014-08-05 23:29:49,721 INFO org.apache.oozie.action.ssh.SshActionExecutor:
> SERVER[192-168-88-213.lunix.lan] USER[mko] GROUP[-] TOKEN[] APP[Ssh-copy]
> JOB[008-140805224842389-oozie-oozi-W]
> ACTION[008-140805224842389-oozie-oozi-W@Ssh] start() begins
> 2014-08-05 23:29:49,723 INFO org.apache.oozie.action.ssh.SshActionExecutor:
> SERVER[192-168-88-213.lunix.lan] USER[mko] GROUP[-] TOKEN[] APP[Ssh-copy]
> JOB[008-140805224842389-oozie-oozi-W]
> ACTION[008-140805224842389-oozie-oozi-W@Ssh] Attempting to copy ssh base
> scripts to remote host [m...@192-168-88-213.lunix.lan]
> 2014-08-05 23:29:52,691 INFO org.apache.oozie.servlet.CallbackServlet:
> SERVER[192-168-88-213.lunix.lan] USER[-] GROUP[-] TOKEN[-] APP[-]
> JOB[008-140805224842389-oozie-oozi-W]
> ACTION[008-140805224842389-oozie-oozi-W@Ssh] callback for action
> [008-140805224842389-oozie-oozi-W@Ssh]
> 2014-08-05 23:29:52,714 ERROR
> org.apache.oozie.command.wf.CompletedActionXCommand:
> SERVER[192-168-88-213.lunix.lan] USER[-] GROUP[-] TOKEN[] APP[-]
> JOB[008-140805224842389-oozie-oozi-W]
> ACTION[008-140805224842389-oozie-oozi-W@Ssh] XException,
> org.apache.oozie.command.CommandException: E0800: Action it is not running
> its in [PREP] state, action [008-140805224842389-oozie-oozi-W@Ssh]
> at
> org.apache.oozie.command.wf.CompletedActionXCommand.eagerVerifyPrecondition(CompletedActionXCommand.java:77)
>
> at org.apache.oozie.command.XCommand.call(XCommand.java:251)
> at
> org.apache.oozie.service.CallableQueueService$CallableWrapper.run(CallableQueueService.java:174)
>
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>
> at java.lang.Thread.run(Thread.java:662)
> 2014-08-05 23:29:52,714 WARN
> org.apache.oozie.service.CallableQueueService$CallableWrapper:
> SERVER[192-168-88-213.lunix.lan] USER[-] GROUP[-] TOKEN[-] APP[-] JOB[-]
> ACTION[-] exception callable [callback], E0800: Action it is not running its
> in [PREP] state, action [008-140805224842389-oozie-oozi-W@Ssh]
> org.apache.oozie.command.CommandException: E0800: Action it is not running
> its in [PREP] state, action [008-140805224842389-oozie-oozi-W@Ssh]
> at
> org.apache.oozie.command.wf.CompletedActionXCommand.eagerVerifyPrecondition(CompletedActionXCommand.java:77)
>
> at org.apache.oozie.command.XCommand.call(XCommand.java:251)
> at
> org.apache.oozie.service.CallableQueueService$CallableWrapper.run(CallableQueueService.java:174)
>
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>
> at java.lang.Thread.run(Thread.java:662)
> 2014-08-05 23:29:57,262 INFO org.apache.oozie.action.ssh.SshActionExecutor:
> SERVER[192-168-88-213.lunix.lan] USER[mko] GROUP[-] TOKEN[] APP[Ssh-copy]
> JOB[008-140805224842389-oozie-oozi-W]
>
[jira] [Updated] (OOZIE-1974) SSH Action doesn't handle compound commands eg: cmd1 && cmd2 and stuck in [PREP] stage
[
https://issues.apache.org/jira/browse/OOZIE-1974?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Mate Juhasz updated OOZIE-1974:
---
Attachment: OOZIE-1974.patch
> SSH Action doesn't handle compound commands eg: cmd1 && cmd2 and stuck in
> [PREP] stage
> --
>
> Key: OOZIE-1974
> URL: https://issues.apache.org/jira/browse/OOZIE-1974
> Project: Oozie
> Issue Type: Bug
> Components: action
>Affects Versions: trunk
>Reporter: Michalis Kongtongk
>Assignee: Mate Juhasz
>Priority: Major
> Attachments: OOZIE-1974.patch
>
>
> example WF that will fail:
> {code}
>
>
>
>
> oozie-u...@somedomain.com
> kinit
> oozie-u...@somedomain.com
> -k
> -t
> /home/oozie-user/oozie.keytab
>
> hdfs
> dfs
> -put
> /tmp/random-file.txt
> /tmp/random-file.txt
>
>
>
>
>
> Action failed, error
> message[${wf:errorMessage(wf:lastErrorNode())}]
>
>
>
> {code}
> Workaround is to execute the compound command in subshell eg: $(cmd1 && cmd2)
> {code}
>
>
>
>
> oozie-u...@somedomain.com
> $(kinit
> oozie-u...@somedomain.com
> -k
> -t
> /home/oozie-user/oozie.keytab
>
> hdfs
> dfs
> -put
> /tmp/random-file.txt
> /tmp/random-file.txt
> )
>
>
>
>
>
> Action failed, error
> message[${wf:errorMessage(wf:lastErrorNode())}]
>
>
>
> {code}
> Stack trace "org.apache.oozie.command.CommandException: E0800: Action it is
> not running its in [PREP] state,"
> {code}
> 2014-08-05 23:29:49,721 INFO org.apache.oozie.action.ssh.SshActionExecutor:
> SERVER[192-168-88-213.lunix.lan] USER[mko] GROUP[-] TOKEN[] APP[Ssh-copy]
> JOB[008-140805224842389-oozie-oozi-W]
> ACTION[008-140805224842389-oozie-oozi-W@Ssh] start() begins
> 2014-08-05 23:29:49,723 INFO org.apache.oozie.action.ssh.SshActionExecutor:
> SERVER[192-168-88-213.lunix.lan] USER[mko] GROUP[-] TOKEN[] APP[Ssh-copy]
> JOB[008-140805224842389-oozie-oozi-W]
> ACTION[008-140805224842389-oozie-oozi-W@Ssh] Attempting to copy ssh base
> scripts to remote host [m...@192-168-88-213.lunix.lan]
> 2014-08-05 23:29:52,691 INFO org.apache.oozie.servlet.CallbackServlet:
> SERVER[192-168-88-213.lunix.lan] USER[-] GROUP[-] TOKEN[-] APP[-]
> JOB[008-140805224842389-oozie-oozi-W]
> ACTION[008-140805224842389-oozie-oozi-W@Ssh] callback for action
> [008-140805224842389-oozie-oozi-W@Ssh]
> 2014-08-05 23:29:52,714 ERROR
> org.apache.oozie.command.wf.CompletedActionXCommand:
> SERVER[192-168-88-213.lunix.lan] USER[-] GROUP[-] TOKEN[] APP[-]
> JOB[008-140805224842389-oozie-oozi-W]
> ACTION[008-140805224842389-oozie-oozi-W@Ssh] XException,
> org.apache.oozie.command.CommandException: E0800: Action it is not running
> its in [PREP] state, action [008-140805224842389-oozie-oozi-W@Ssh]
> at
> org.apache.oozie.command.wf.CompletedActionXCommand.eagerVerifyPrecondition(CompletedActionXCommand.java:77)
>
> at org.apache.oozie.command.XCommand.call(XCommand.java:251)
> at
> org.apache.oozie.service.CallableQueueService$CallableWrapper.run(CallableQueueService.java:174)
>
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>
> at java.lang.Thread.run(Thread.java:662)
> 2014-08-05 23:29:52,714 WARN
> org.apache.oozie.service.CallableQueueService$CallableWrapper:
> SERVER[192-168-88-213.lunix.lan] USER[-] GROUP[-] TOKEN[-] APP[-] JOB[-]
> ACTION[-] exception callable [callback], E0800: Action it is not running its
> in [PREP] state, action [008-140805224842389-oozie-oozi-W@Ssh]
> org.apache.oozie.command.CommandException: E0800: Action it is not running
> its in [PREP] state, action [008-140805224842389-oozie-oozi-W@Ssh]
> at
> org.apache.oozie.command.wf.CompletedActionXCommand.eagerVerifyPrecondition(CompletedActionXCommand.java:77)
>
> at org.apache.oozie.command.XCommand.call(XCommand.java:251)
> at
> org.apache.oozie.service.CallableQueueService$CallableWrapper.run(CallableQueueService.java:174)
>
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>
> at java.lang.Thread.run(Thread.java:662)
> 2014-08-05 23:29:57,262 INFO org.apache.oozie.action.ssh.SshActionExecutor:
> SERVER[192-168-88-213.lunix.lan] USER[mko] GROUP[-] TOKEN[] APP[Ssh-copy]
> JOB[008-140805224842389-oozie-oozi-W]
> ACTION[008-140805224842389-oozie-oozi-W@Ssh] start() ends
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Reopened] (OOZIE-3363) Hadoop's cleanup of local directory in uber mode causing failures for sqoop actions
[
https://issues.apache.org/jira/browse/OOZIE-3363?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Mate Juhasz reopened OOZIE-3363:
Reopening, I closed it by mistake... :)
> Hadoop's cleanup of local directory in uber mode causing failures for sqoop
> actions
> ---
>
> Key: OOZIE-3363
> URL: https://issues.apache.org/jira/browse/OOZIE-3363
> Project: Oozie
> Issue Type: Bug
>Affects Versions: 4.3.0, 5.0.0
>Reporter: Clemens Valiente
>Priority: Blocker
>
> See related task OOZIE-2536 and discussion
> https://issues.apache.org/jira/browse/OOZIE-2536?focusedCommentId=16617206=com.atlassian.jira.plugin.system.issuetabpanels%3Acomment-tabpanel#comment-16617206
> the same issue cleaning up local directories also affects sqoop jobs that
> create the sqoop.xml file inbetween.
> {code:java}
> 2018-09-15 00:31:09,416 [AsyncDispatcher event handler] INFO
> org.apache.hadoop.mapreduce.v2.app.job.impl.JobImpl - Num completed Tasks: 1
> 2018-09-15 00:31:09,417 [AsyncDispatcher event handler] INFO
> org.apache.hadoop.mapreduce.v2.app.job.impl.JobImpl -
> job_1535972033593_259806Job Transitioned from RUNNING to COMMITTING
> 2018-09-15 00:31:09,419 [CommitterEvent Processor #1] INFO
> org.apache.hadoop.mapreduce.v2.app.commit.CommitterEventHandler - Processing
> the event EventType: JOB_COMMIT
> 2018-09-15 00:31:09,455 [uber-SubtaskRunner] WARN
> org.apache.hadoop.mapred.LocalContainerLauncher - Unable to delete
> unexpected local file/dir .action.xml.crc: insufficient permissions?
> 2018-09-15 00:31:09,455 [uber-SubtaskRunner] WARN
> org.apache.hadoop.mapred.LocalContainerLauncher - Unable to delete
> unexpected local file/dir .action.xml.crc: insufficient permissions?
> 2018-09-15 00:31:09,459 [CommitterEvent Processor #1] FATAL
> org.apache.hadoop.conf.Configuration - error parsing conf sqoop-site.xml
> java.io.FileNotFoundException:
> /appdata/hdfs/v7/yarn/nm/usercache/SEM/appcache/application_1535972033593_259806/container_e100_1535972033593_259806_01_01/sqoop-site.xml
> (No such file or directory)
> at java.io.FileInputStream.open(Native Method)
> at java.io.FileInputStream.(FileInputStream.java:146)
> at java.io.FileInputStream.(FileInputStream.java:101)
> at
> sun.net.www.protocol.file.FileURLConnection.connect(FileURLConnection.java:90)
> at
> sun.net.www.protocol.file.FileURLConnection.getInputStream(FileURLConnection.java:188)
> at org.apache.hadoop.conf.Configuration.parse(Configuration.java:2483)
> at
> org.apache.hadoop.conf.Configuration.loadResource(Configuration.java:2554)
> at
> org.apache.hadoop.conf.Configuration.loadResources(Configuration.java:2507)
> at
> org.apache.hadoop.conf.Configuration.getProps(Configuration.java:2413)
> at org.apache.hadoop.conf.Configuration.get(Configuration.java:984)
> at
> org.apache.hadoop.conf.Configuration.getTrimmed(Configuration.java:1034)
> at org.apache.hadoop.conf.Configuration.getInt(Configuration.java:1254)
> at org.apache.hadoop.fs.FileSystem.create(FileSystem.java:804)
> at
> org.apache.hadoop.mapreduce.v2.app.commit.CommitterEventHandler$EventProcessor.touchz(CommitterEventHandler.java:268)
> at
> org.apache.hadoop.mapreduce.v2.app.commit.CommitterEventHandler$EventProcessor.handleJobCommit(CommitterEventHandler.java:282)
> at
> org.apache.hadoop.mapreduce.v2.app.commit.CommitterEventHandler$EventProcessor.run(CommitterEventHandler.java:237)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
> at java.lang.Thread.run(Thread.java:745){code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (OOZIE-1974) SSH Action doesn't handle compound commands eg: cmd1 && cmd2 and stuck in [PREP] stage
[
https://issues.apache.org/jira/browse/OOZIE-1974?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16938413#comment-16938413
]
Mate Juhasz commented on OOZIE-1974:
Not related, the ssh-wrapper.sh code is "designed" in a way that it does not
support compound commands
> SSH Action doesn't handle compound commands eg: cmd1 && cmd2 and stuck in
> [PREP] stage
> --
>
> Key: OOZIE-1974
> URL: https://issues.apache.org/jira/browse/OOZIE-1974
> Project: Oozie
> Issue Type: Bug
> Components: action
>Affects Versions: trunk
>Reporter: Michalis Kongtongk
>Assignee: Mate Juhasz
>Priority: Major
>
> example WF that will fail:
> {code}
>
>
>
>
> oozie-u...@somedomain.com
> kinit
> oozie-u...@somedomain.com
> -k
> -t
> /home/oozie-user/oozie.keytab
>
> hdfs
> dfs
> -put
> /tmp/random-file.txt
> /tmp/random-file.txt
>
>
>
>
>
> Action failed, error
> message[${wf:errorMessage(wf:lastErrorNode())}]
>
>
>
> {code}
> Workaround is to execute the compound command in subshell eg: $(cmd1 && cmd2)
> {code}
>
>
>
>
> oozie-u...@somedomain.com
> $(kinit
> oozie-u...@somedomain.com
> -k
> -t
> /home/oozie-user/oozie.keytab
>
> hdfs
> dfs
> -put
> /tmp/random-file.txt
> /tmp/random-file.txt
> )
>
>
>
>
>
> Action failed, error
> message[${wf:errorMessage(wf:lastErrorNode())}]
>
>
>
> {code}
> Stack trace "org.apache.oozie.command.CommandException: E0800: Action it is
> not running its in [PREP] state,"
> {code}
> 2014-08-05 23:29:49,721 INFO org.apache.oozie.action.ssh.SshActionExecutor:
> SERVER[192-168-88-213.lunix.lan] USER[mko] GROUP[-] TOKEN[] APP[Ssh-copy]
> JOB[008-140805224842389-oozie-oozi-W]
> ACTION[008-140805224842389-oozie-oozi-W@Ssh] start() begins
> 2014-08-05 23:29:49,723 INFO org.apache.oozie.action.ssh.SshActionExecutor:
> SERVER[192-168-88-213.lunix.lan] USER[mko] GROUP[-] TOKEN[] APP[Ssh-copy]
> JOB[008-140805224842389-oozie-oozi-W]
> ACTION[008-140805224842389-oozie-oozi-W@Ssh] Attempting to copy ssh base
> scripts to remote host [m...@192-168-88-213.lunix.lan]
> 2014-08-05 23:29:52,691 INFO org.apache.oozie.servlet.CallbackServlet:
> SERVER[192-168-88-213.lunix.lan] USER[-] GROUP[-] TOKEN[-] APP[-]
> JOB[008-140805224842389-oozie-oozi-W]
> ACTION[008-140805224842389-oozie-oozi-W@Ssh] callback for action
> [008-140805224842389-oozie-oozi-W@Ssh]
> 2014-08-05 23:29:52,714 ERROR
> org.apache.oozie.command.wf.CompletedActionXCommand:
> SERVER[192-168-88-213.lunix.lan] USER[-] GROUP[-] TOKEN[] APP[-]
> JOB[008-140805224842389-oozie-oozi-W]
> ACTION[008-140805224842389-oozie-oozi-W@Ssh] XException,
> org.apache.oozie.command.CommandException: E0800: Action it is not running
> its in [PREP] state, action [008-140805224842389-oozie-oozi-W@Ssh]
> at
> org.apache.oozie.command.wf.CompletedActionXCommand.eagerVerifyPrecondition(CompletedActionXCommand.java:77)
>
> at org.apache.oozie.command.XCommand.call(XCommand.java:251)
> at
> org.apache.oozie.service.CallableQueueService$CallableWrapper.run(CallableQueueService.java:174)
>
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>
> at java.lang.Thread.run(Thread.java:662)
> 2014-08-05 23:29:52,714 WARN
> org.apache.oozie.service.CallableQueueService$CallableWrapper:
> SERVER[192-168-88-213.lunix.lan] USER[-] GROUP[-] TOKEN[-] APP[-] JOB[-]
> ACTION[-] exception callable [callback], E0800: Action it is not running its
> in [PREP] state, action [008-140805224842389-oozie-oozi-W@Ssh]
> org.apache.oozie.command.CommandException: E0800: Action it is not running
> its in [PREP] state, action [008-140805224842389-oozie-oozi-W@Ssh]
> at
> org.apache.oozie.command.wf.CompletedActionXCommand.eagerVerifyPrecondition(CompletedActionXCommand.java:77)
>
> at org.apache.oozie.command.XCommand.call(XCommand.java:251)
> at
> org.apache.oozie.service.CallableQueueService$CallableWrapper.run(CallableQueueService.java:174)
>
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>
> at java.lang.Thread.run(Thread.java:662)
> 2014-08-05 23:29:57,262 INFO org.apache.oozie.action.ssh.SshActionExecutor:
> SERVER[192-168-88-213.lunix.lan] USER[mko] GROUP[-] TOKEN[] APP[Ssh-copy]
> JOB[008-140805224842389-oozie-oozi-W]
> ACTION[008-140805224842389-oozie-oozi-W@Ssh] start() ends
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (OOZIE-3544) Upgrade commons-beanutils to 1.9.4
[
https://issues.apache.org/jira/browse/OOZIE-3544?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Mate Juhasz updated OOZIE-3544:
---
Attachment: OOZIE-3544.patch
> Upgrade commons-beanutils to 1.9.4
> --
>
> Key: OOZIE-3544
> URL: https://issues.apache.org/jira/browse/OOZIE-3544
> Project: Oozie
> Issue Type: Bug
>Affects Versions: trunk
>Reporter: Andras Salamon
>Assignee: Mate Juhasz
>Priority: Major
> Fix For: 5.2.0
>
> Attachments: OOZIE-3544.patch
>
>
> Oozie does not define the {{commons-beanutils}} version, so we end up having
> multiple versions. In my last compilation we have 1.7.0, 1.9.1, and 1.9.3 at
> the same time.
> Let's consolidate it and use the latest 1.9.4 everywhere.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Assigned] (OOZIE-3544) Upgrade commons-beanutils to 1.9.4
[
https://issues.apache.org/jira/browse/OOZIE-3544?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Mate Juhasz reassigned OOZIE-3544:
--
Assignee: Mate Juhasz
> Upgrade commons-beanutils to 1.9.4
> --
>
> Key: OOZIE-3544
> URL: https://issues.apache.org/jira/browse/OOZIE-3544
> Project: Oozie
> Issue Type: Bug
>Affects Versions: trunk
>Reporter: Andras Salamon
>Assignee: Mate Juhasz
>Priority: Major
> Fix For: 5.2.0
>
>
> Oozie does not define the {{commons-beanutils}} version, so we end up having
> multiple versions. In my last compilation we have 1.7.0, 1.9.1, and 1.9.3 at
> the same time.
> Let's consolidate it and use the latest 1.9.4 everywhere.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Assigned] (OOZIE-1974) SSH Action doesn't handle compound commands eg: cmd1 && cmd2 and stuck in [PREP] stage
[
https://issues.apache.org/jira/browse/OOZIE-1974?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Mate Juhasz reassigned OOZIE-1974:
--
Assignee: Mate Juhasz
> SSH Action doesn't handle compound commands eg: cmd1 && cmd2 and stuck in
> [PREP] stage
> --
>
> Key: OOZIE-1974
> URL: https://issues.apache.org/jira/browse/OOZIE-1974
> Project: Oozie
> Issue Type: Bug
> Components: action
>Affects Versions: trunk
>Reporter: Michalis Kongtongk
>Assignee: Mate Juhasz
>Priority: Major
>
> example WF that will fail:
> {code}
>
>
>
>
> oozie-u...@somedomain.com
> kinit
> oozie-u...@somedomain.com
> -k
> -t
> /home/oozie-user/oozie.keytab
>
> hdfs
> dfs
> -put
> /tmp/random-file.txt
> /tmp/random-file.txt
>
>
>
>
>
> Action failed, error
> message[${wf:errorMessage(wf:lastErrorNode())}]
>
>
>
> {code}
> Workaround is to execute the compound command in subshell eg: $(cmd1 && cmd2)
> {code}
>
>
>
>
> oozie-u...@somedomain.com
> $(kinit
> oozie-u...@somedomain.com
> -k
> -t
> /home/oozie-user/oozie.keytab
>
> hdfs
> dfs
> -put
> /tmp/random-file.txt
> /tmp/random-file.txt
> )
>
>
>
>
>
> Action failed, error
> message[${wf:errorMessage(wf:lastErrorNode())}]
>
>
>
> {code}
> Stack trace "org.apache.oozie.command.CommandException: E0800: Action it is
> not running its in [PREP] state,"
> {code}
> 2014-08-05 23:29:49,721 INFO org.apache.oozie.action.ssh.SshActionExecutor:
> SERVER[192-168-88-213.lunix.lan] USER[mko] GROUP[-] TOKEN[] APP[Ssh-copy]
> JOB[008-140805224842389-oozie-oozi-W]
> ACTION[008-140805224842389-oozie-oozi-W@Ssh] start() begins
> 2014-08-05 23:29:49,723 INFO org.apache.oozie.action.ssh.SshActionExecutor:
> SERVER[192-168-88-213.lunix.lan] USER[mko] GROUP[-] TOKEN[] APP[Ssh-copy]
> JOB[008-140805224842389-oozie-oozi-W]
> ACTION[008-140805224842389-oozie-oozi-W@Ssh] Attempting to copy ssh base
> scripts to remote host [m...@192-168-88-213.lunix.lan]
> 2014-08-05 23:29:52,691 INFO org.apache.oozie.servlet.CallbackServlet:
> SERVER[192-168-88-213.lunix.lan] USER[-] GROUP[-] TOKEN[-] APP[-]
> JOB[008-140805224842389-oozie-oozi-W]
> ACTION[008-140805224842389-oozie-oozi-W@Ssh] callback for action
> [008-140805224842389-oozie-oozi-W@Ssh]
> 2014-08-05 23:29:52,714 ERROR
> org.apache.oozie.command.wf.CompletedActionXCommand:
> SERVER[192-168-88-213.lunix.lan] USER[-] GROUP[-] TOKEN[] APP[-]
> JOB[008-140805224842389-oozie-oozi-W]
> ACTION[008-140805224842389-oozie-oozi-W@Ssh] XException,
> org.apache.oozie.command.CommandException: E0800: Action it is not running
> its in [PREP] state, action [008-140805224842389-oozie-oozi-W@Ssh]
> at
> org.apache.oozie.command.wf.CompletedActionXCommand.eagerVerifyPrecondition(CompletedActionXCommand.java:77)
>
> at org.apache.oozie.command.XCommand.call(XCommand.java:251)
> at
> org.apache.oozie.service.CallableQueueService$CallableWrapper.run(CallableQueueService.java:174)
>
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>
> at java.lang.Thread.run(Thread.java:662)
> 2014-08-05 23:29:52,714 WARN
> org.apache.oozie.service.CallableQueueService$CallableWrapper:
> SERVER[192-168-88-213.lunix.lan] USER[-] GROUP[-] TOKEN[-] APP[-] JOB[-]
> ACTION[-] exception callable [callback], E0800: Action it is not running its
> in [PREP] state, action [008-140805224842389-oozie-oozi-W@Ssh]
> org.apache.oozie.command.CommandException: E0800: Action it is not running
> its in [PREP] state, action [008-140805224842389-oozie-oozi-W@Ssh]
> at
> org.apache.oozie.command.wf.CompletedActionXCommand.eagerVerifyPrecondition(CompletedActionXCommand.java:77)
>
> at org.apache.oozie.command.XCommand.call(XCommand.java:251)
> at
> org.apache.oozie.service.CallableQueueService$CallableWrapper.run(CallableQueueService.java:174)
>
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>
> at java.lang.Thread.run(Thread.java:662)
> 2014-08-05 23:29:57,262 INFO org.apache.oozie.action.ssh.SshActionExecutor:
> SERVER[192-168-88-213.lunix.lan] USER[mko] GROUP[-] TOKEN[] APP[Ssh-copy]
> JOB[008-140805224842389-oozie-oozi-W]
> ACTION[008-140805224842389-oozie-oozi-W@Ssh] start() ends
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Assigned] (OOZIE-3392) Refactor ShellMain to use BufferDrainer class
[
https://issues.apache.org/jira/browse/OOZIE-3392?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Mate Juhasz reassigned OOZIE-3392:
--
Assignee: Mate Juhasz
> Refactor ShellMain to use BufferDrainer class
> -
>
> Key: OOZIE-3392
> URL: https://issues.apache.org/jira/browse/OOZIE-3392
> Project: Oozie
> Issue Type: Improvement
>Affects Versions: trunk
>Reporter: Andras Salamon
>Assignee: Mate Juhasz
>Priority: Major
> Fix For: 5.3.0
>
>
> OOZIE-3382 introduced a new {{BufferDrainer}} class to drain the standard
> output and standard error stream of an ssh process. {{ShellMain}} should also
> use this class for reading the standard output and standard error of a shell
> process to avoid code duplication.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (OOZIE-3465) Migrate from commons-codec
[
https://issues.apache.org/jira/browse/OOZIE-3465?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16937669#comment-16937669
]
Mate Juhasz commented on OOZIE-3465:
You are right [~asalamon74], thanks for pointing out! I bumped up the version
to 1.10, which is the latest among the transitive libs
> Migrate from commons-codec
> --
>
> Key: OOZIE-3465
> URL: https://issues.apache.org/jira/browse/OOZIE-3465
> Project: Oozie
> Issue Type: Improvement
>Affects Versions: trunk
>Reporter: Andras Salamon
>Assignee: Mate Juhasz
>Priority: Major
> Attachments: OOZIE-3465-V2.patch, OOZIE-3465.patch
>
>
> Ooize uses {{commons-codec}} 1.4 which is almost [10
> years|https://commons.apache.org/proper/commons-codec/changes-report.html]
> old. We hardly use this library I've only found a few references to the
> {{Base64}} class in Oozie. Java8 defines a
> [Base64|https://docs.oracle.com/javase/8/docs/api/java/util/Base64.html]
> class which we could use instead.
> We should be careful to put this jar into the sharelibs if and only if some
> other library requires it.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (OOZIE-3465) Migrate from commons-codec
[
https://issues.apache.org/jira/browse/OOZIE-3465?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Mate Juhasz updated OOZIE-3465:
---
Attachment: OOZIE-3465-V2.patch
> Migrate from commons-codec
> --
>
> Key: OOZIE-3465
> URL: https://issues.apache.org/jira/browse/OOZIE-3465
> Project: Oozie
> Issue Type: Improvement
>Affects Versions: trunk
>Reporter: Andras Salamon
>Assignee: Mate Juhasz
>Priority: Major
> Attachments: OOZIE-3465-V2.patch, OOZIE-3465.patch
>
>
> Ooize uses {{commons-codec}} 1.4 which is almost [10
> years|https://commons.apache.org/proper/commons-codec/changes-report.html]
> old. We hardly use this library I've only found a few references to the
> {{Base64}} class in Oozie. Java8 defines a
> [Base64|https://docs.oracle.com/javase/8/docs/api/java/util/Base64.html]
> class which we could use instead.
> We should be careful to put this jar into the sharelibs if and only if some
> other library requires it.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (OOZIE-3465) Migrate from commons-codec
[
https://issues.apache.org/jira/browse/OOZIE-3465?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16937572#comment-16937572
]
Mate Juhasz commented on OOZIE-3465:
Andras I uploaded an initial patch in which we dont use commons-codec directly
anymore, however it comes in as a transitive dependecy from hadoop-yarn-common,
hadoop-common, hadoop-auth, hadoop-hdfs, httpclient, spark-hive, hive-exec and
pig... so its a mess :) I added exclusions where we include oozie-core, because
it brings the transitive commons-codec from direct dependencies with itself. I
tried to minimise the presence of commons-codec on the sharelib this way.
What do you think?
> Migrate from commons-codec
> --
>
> Key: OOZIE-3465
> URL: https://issues.apache.org/jira/browse/OOZIE-3465
> Project: Oozie
> Issue Type: Improvement
>Affects Versions: trunk
>Reporter: Andras Salamon
>Assignee: Mate Juhasz
>Priority: Major
> Attachments: OOZIE-3465.patch
>
>
> Ooize uses {{commons-codec}} 1.4 which is almost [10
> years|https://commons.apache.org/proper/commons-codec/changes-report.html]
> old. We hardly use this library I've only found a few references to the
> {{Base64}} class in Oozie. Java8 defines a
> [Base64|https://docs.oracle.com/javase/8/docs/api/java/util/Base64.html]
> class which we could use instead.
> We should be careful to put this jar into the sharelibs if and only if some
> other library requires it.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (OOZIE-3465) Migrate from commons-codec
[
https://issues.apache.org/jira/browse/OOZIE-3465?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Mate Juhasz updated OOZIE-3465:
---
Attachment: OOZIE-3465.patch
> Migrate from commons-codec
> --
>
> Key: OOZIE-3465
> URL: https://issues.apache.org/jira/browse/OOZIE-3465
> Project: Oozie
> Issue Type: Improvement
>Affects Versions: trunk
>Reporter: Andras Salamon
>Assignee: Mate Juhasz
>Priority: Major
> Attachments: OOZIE-3465.patch
>
>
> Ooize uses {{commons-codec}} 1.4 which is almost [10
> years|https://commons.apache.org/proper/commons-codec/changes-report.html]
> old. We hardly use this library I've only found a few references to the
> {{Base64}} class in Oozie. Java8 defines a
> [Base64|https://docs.oracle.com/javase/8/docs/api/java/util/Base64.html]
> class which we could use instead.
> We should be careful to put this jar into the sharelibs if and only if some
> other library requires it.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Assigned] (OOZIE-3465) Migrate from commons-codec
[
https://issues.apache.org/jira/browse/OOZIE-3465?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Mate Juhasz reassigned OOZIE-3465:
--
Assignee: Mate Juhasz
> Migrate from commons-codec
> --
>
> Key: OOZIE-3465
> URL: https://issues.apache.org/jira/browse/OOZIE-3465
> Project: Oozie
> Issue Type: Improvement
>Affects Versions: trunk
>Reporter: Andras Salamon
>Assignee: Mate Juhasz
>Priority: Major
>
> Ooize uses {{commons-codec}} 1.4 which is almost [10
> years|https://commons.apache.org/proper/commons-codec/changes-report.html]
> old. We hardly use this library I've only found a few references to the
> {{Base64}} class in Oozie. Java8 defines a
> [Base64|https://docs.oracle.com/javase/8/docs/api/java/util/Base64.html]
> class which we could use instead.
> We should be careful to put this jar into the sharelibs if and only if some
> other library requires it.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Resolved] (OOZIE-3363) Hadoop's cleanup of local directory in uber mode causing failures for sqoop actions
[
https://issues.apache.org/jira/browse/OOZIE-3363?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Mate Juhasz resolved OOZIE-3363.
Resolution: Duplicate
> Hadoop's cleanup of local directory in uber mode causing failures for sqoop
> actions
> ---
>
> Key: OOZIE-3363
> URL: https://issues.apache.org/jira/browse/OOZIE-3363
> Project: Oozie
> Issue Type: Bug
>Affects Versions: 4.3.0, 5.0.0
>Reporter: Clemens Valiente
>Priority: Blocker
>
> See related task OOZIE-2536 and discussion
> https://issues.apache.org/jira/browse/OOZIE-2536?focusedCommentId=16617206=com.atlassian.jira.plugin.system.issuetabpanels%3Acomment-tabpanel#comment-16617206
> the same issue cleaning up local directories also affects sqoop jobs that
> create the sqoop.xml file inbetween.
> {code:java}
> 2018-09-15 00:31:09,416 [AsyncDispatcher event handler] INFO
> org.apache.hadoop.mapreduce.v2.app.job.impl.JobImpl - Num completed Tasks: 1
> 2018-09-15 00:31:09,417 [AsyncDispatcher event handler] INFO
> org.apache.hadoop.mapreduce.v2.app.job.impl.JobImpl -
> job_1535972033593_259806Job Transitioned from RUNNING to COMMITTING
> 2018-09-15 00:31:09,419 [CommitterEvent Processor #1] INFO
> org.apache.hadoop.mapreduce.v2.app.commit.CommitterEventHandler - Processing
> the event EventType: JOB_COMMIT
> 2018-09-15 00:31:09,455 [uber-SubtaskRunner] WARN
> org.apache.hadoop.mapred.LocalContainerLauncher - Unable to delete
> unexpected local file/dir .action.xml.crc: insufficient permissions?
> 2018-09-15 00:31:09,455 [uber-SubtaskRunner] WARN
> org.apache.hadoop.mapred.LocalContainerLauncher - Unable to delete
> unexpected local file/dir .action.xml.crc: insufficient permissions?
> 2018-09-15 00:31:09,459 [CommitterEvent Processor #1] FATAL
> org.apache.hadoop.conf.Configuration - error parsing conf sqoop-site.xml
> java.io.FileNotFoundException:
> /appdata/hdfs/v7/yarn/nm/usercache/SEM/appcache/application_1535972033593_259806/container_e100_1535972033593_259806_01_01/sqoop-site.xml
> (No such file or directory)
> at java.io.FileInputStream.open(Native Method)
> at java.io.FileInputStream.(FileInputStream.java:146)
> at java.io.FileInputStream.(FileInputStream.java:101)
> at
> sun.net.www.protocol.file.FileURLConnection.connect(FileURLConnection.java:90)
> at
> sun.net.www.protocol.file.FileURLConnection.getInputStream(FileURLConnection.java:188)
> at org.apache.hadoop.conf.Configuration.parse(Configuration.java:2483)
> at
> org.apache.hadoop.conf.Configuration.loadResource(Configuration.java:2554)
> at
> org.apache.hadoop.conf.Configuration.loadResources(Configuration.java:2507)
> at
> org.apache.hadoop.conf.Configuration.getProps(Configuration.java:2413)
> at org.apache.hadoop.conf.Configuration.get(Configuration.java:984)
> at
> org.apache.hadoop.conf.Configuration.getTrimmed(Configuration.java:1034)
> at org.apache.hadoop.conf.Configuration.getInt(Configuration.java:1254)
> at org.apache.hadoop.fs.FileSystem.create(FileSystem.java:804)
> at
> org.apache.hadoop.mapreduce.v2.app.commit.CommitterEventHandler$EventProcessor.touchz(CommitterEventHandler.java:268)
> at
> org.apache.hadoop.mapreduce.v2.app.commit.CommitterEventHandler$EventProcessor.handleJobCommit(CommitterEventHandler.java:282)
> at
> org.apache.hadoop.mapreduce.v2.app.commit.CommitterEventHandler$EventProcessor.run(CommitterEventHandler.java:237)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
> at java.lang.Thread.run(Thread.java:745){code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (OOZIE-2576) Oozie ssh action Cannot run program "scp"
[
https://issues.apache.org/jira/browse/OOZIE-2576?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16936818#comment-16936818
]
Mate Juhasz commented on OOZIE-2576:
Its a bit late to answer, but I think the error is because adding multiple
commands are not supported in case of ssh actions.
It should work if you specify the command like below:
{noformat}
/tmp/test.sh
{noformat}
Or:
{noformat}
sh
/tmp/test.sh
{noformat}
> Oozie ssh action Cannot run program "scp"
> -
>
> Key: OOZIE-2576
> URL: https://issues.apache.org/jira/browse/OOZIE-2576
> Project: Oozie
> Issue Type: Bug
> Components: action
>Affects Versions: 4.0.0
> Environment: Oozie server build version: 4.0.0-cdh5.2.0
>Reporter: ywheel
>Priority: Blocker
>
> We used oozie ssh action in a production environment, the following xml codes
> is the example:
> {noformat}
>
>
>
>
> root@192.168.1.154
> sh /tmp/test.sh
>
>
>
>
>
>
> Action failed, error
> message[${wf:errorMessage(wf:lastErrorNode())}]
>
>
>
> {noformat}
> However, we meet the errors below:
> {noformat}
> 2016-06-12 22:30:54,713 INFO org.apache.oozie.action.ssh.SshActionExecutor:
> SERVER[Master] USER[hdfs] GROUP[-] TOKEN[] APP[TestSsh]
> JOB[201-160113124428061-oozie-oozi-W]
> ACTION[201-160113124428061-oozie-oozi-W@ShellAction] Attempting to copy
> ssh base scripts to remote host [root@192.168.1.154]
> 2016-06-12 22:30:54,869 WARN org.apache.oozie.action.ssh.SshActionExecutor:
> SERVER[Master] USER[hdfs] GROUP[-] TOKEN[] APP[TestSsh]
> JOB[201-160113124428061-oozie-oozi-W]
> ACTION[201-160113124428061-oozie-oozi-W@ShellAction] Error while
> executing ssh EXECUTION
> 2016-06-12 22:30:54,870 WARN org.apache.oozie.command.wf.ActionStartXCommand:
> SERVER[Master] USER[hdfs] GROUP[-] TOKEN[] APP[TestSsh]
> JOB[201-160113124428061-oozie-oozi-W]
> ACTION[201-160113124428061-oozie-oozi-W@ShellAction] Error starting
> action [ShellAction]. ErrorType [ERROR], ErrorCode [UNKOWN_ERROR], Message
> [UNKOWN_ERROR: Cannot run program "scp": error=2, No such file or directory]
> org.apache.oozie.action.ActionExecutorException: UNKOWN_ERROR: Cannot run
> program "scp": error=2, No such file or directory
> at
> org.apache.oozie.action.ssh.SshActionExecutor.execute(SshActionExecutor.java:599)
> at
> org.apache.oozie.action.ssh.SshActionExecutor.start(SshActionExecutor.java:204)
> at
> org.apache.oozie.command.wf.ActionStartXCommand.execute(ActionStartXCommand.java:228)
> at
> org.apache.oozie.command.wf.ActionStartXCommand.execute(ActionStartXCommand.java:63)
> at org.apache.oozie.command.XCommand.call(XCommand.java:281)
> at
> org.apache.oozie.service.CallableQueueService$CompositeCallable.call(CallableQueueService.java:323)
> at
> org.apache.oozie.service.CallableQueueService$CompositeCallable.call(CallableQueueService.java:252)
> at
> org.apache.oozie.service.CallableQueueService$CallableWrapper.run(CallableQueueService.java:174)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
> at java.lang.Thread.run(Thread.java:744)
> Caused by: java.io.IOException: Cannot run program "scp": error=2, No such
> file or directory
> at java.lang.ProcessBuilder.start(ProcessBuilder.java:1041)
> at java.lang.Runtime.exec(Runtime.java:617)
> at java.lang.Runtime.exec(Runtime.java:485)
> at
> org.apache.oozie.action.ssh.SshActionExecutor.executeCommand(SshActionExecutor.java:332)
> at
> org.apache.oozie.action.ssh.SshActionExecutor.setupRemote(SshActionExecutor.java:376)
> at
> org.apache.oozie.action.ssh.SshActionExecutor$1.call(SshActionExecutor.java:206)
> at
> org.apache.oozie.action.ssh.SshActionExecutor$1.call(SshActionExecutor.java:204)
> at
> org.apache.oozie.action.ssh.SshActionExecutor.execute(SshActionExecutor.java:548)
> ... 10 more
> Caused by: java.io.IOException: error=2, No such file or directory
> at java.lang.UNIXProcess.forkAndExec(Native Method)
> at java.lang.UNIXProcess.(UNIXProcess.java:135)
> at java.lang.ProcessImpl.start(ProcessImpl.java:130)
> at java.lang.ProcessBuilder.start(ProcessBuilder.java:1022)
> ... 17 more
> {noformat}
> I checked the codes in {{org.apache.oozie.action.ssh.SshActionExecutor}}
> class, and I found oozie would execute 'scp' command to copy two
> files(ssh-base.sh,ssh-wrapper.sh) to the workspace folder on remote host. But
> the logs could not show any information about which file is not found.
> The workspace folder on remote host was created so the 'ssh'+ 'mkdir'
[jira] [Updated] (OOZIE-3543) Upgrade quartz to 2.3.1
[ https://issues.apache.org/jira/browse/OOZIE-3543?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mate Juhasz updated OOZIE-3543: --- Attachment: OOZIE-3543.patch > Upgrade quartz to 2.3.1 > --- > > Key: OOZIE-3543 > URL: https://issues.apache.org/jira/browse/OOZIE-3543 > Project: Oozie > Issue Type: Bug >Reporter: Andras Salamon >Assignee: Mate Juhasz >Priority: Blocker > Fix For: 5.2.0 > > Attachments: OOZIE-3543.patch > > > Oozie uses quartz 2.1.7, which is six years old and depends on c3p0:c3p0 > 0.9.1.1 which has LGPL license. We should upgrade to quartz 2.3.1 which > depends on com.mchange:c3p0. This newer version of c3p0 has dual license, EPL > is good for us. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Assigned] (OOZIE-3543) Upgrade quartz to 2.3.1
[ https://issues.apache.org/jira/browse/OOZIE-3543?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mate Juhasz reassigned OOZIE-3543: -- Assignee: Mate Juhasz > Upgrade quartz to 2.3.1 > --- > > Key: OOZIE-3543 > URL: https://issues.apache.org/jira/browse/OOZIE-3543 > Project: Oozie > Issue Type: Bug >Reporter: Andras Salamon >Assignee: Mate Juhasz >Priority: Blocker > Fix For: 5.2.0 > > > Oozie uses quartz 2.1.7, which is six years old and depends on c3p0:c3p0 > 0.9.1.1 which has LGPL license. We should upgrade to quartz 2.3.1 which > depends on com.mchange:c3p0. This newer version of c3p0 has dual license, EPL > is good for us. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (OOZIE-3405) SSH action shows empty error Message and Error code
[
https://issues.apache.org/jira/browse/OOZIE-3405?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Mate Juhasz updated OOZIE-3405:
---
Attachment: OOZIE-3405-V2.patch
> SSH action shows empty error Message and Error code
> ---
>
> Key: OOZIE-3405
> URL: https://issues.apache.org/jira/browse/OOZIE-3405
> Project: Oozie
> Issue Type: Improvement
>Affects Versions: 5.1.0
>Reporter: Peter Orova
>Assignee: Mate Juhasz
>Priority: Minor
> Fix For: trunk
>
> Attachments: OOZIE-3405-V1.patch, OOZIE-3405-V2.patch
>
>
> Currently, when an SSH action fails the only message that is returned is the
> Status. Neither the {{error Message}} nor {{Error code}} fields are filled.
> This makes reporting on the causes of SSH Action failures via Oozie highly
> impractical: the only meaningful bit of information there is on a failed SSH
> Action is the Status.
> The Status is filled based on the presence (or lack of) the {{.error file}}
> that is produced in case the user submitted script returns with any other
> value than 0.
> {noformat}
> SshActionExecutor#getActionStatus
> ...
> String outFile = getRemoteFileName(context, action, "error", false, true);
> String checkErrorCmd = SSH_COMMAND_BASE + action.getTrackerUri() + " ls " +
> outFile;
> int retVal = getReturnValue(checkErrorCmd);
> ...
> {noformat}
>
> User requirement is to provide some more detailed information on the
> success/failure of the user-submitted script. That could be at a minimum the
> return value, optionally the last ~1K of the stderr that is drained. This
> information could then be communicated via {{errorMessage}} and {{ErrorCode}}
--
This message was sent by Atlassian Jira
(v8.3.2#803003)
[jira] [Updated] (OOZIE-3405) SSH action shows empty error Message and Error code
[
https://issues.apache.org/jira/browse/OOZIE-3405?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Mate Juhasz updated OOZIE-3405:
---
Attachment: OOZIE-3405-V1.patch
> SSH action shows empty error Message and Error code
> ---
>
> Key: OOZIE-3405
> URL: https://issues.apache.org/jira/browse/OOZIE-3405
> Project: Oozie
> Issue Type: Improvement
>Reporter: Peter Orova
>Assignee: Mate Juhasz
>Priority: Minor
> Attachments: OOZIE-3405-V1.patch
>
>
> Currently, when an SSH action fails the only message that is returned is the
> Status. Neither the {{error Message}} nor {{Error code}} fields are filled.
> This makes reporting on the causes of SSH Action failures via Oozie highly
> impractical: the only meaningful bit of information there is on a failed SSH
> Action is the Status.
> The Status is filled based on the presence (or lack of) the {{.error file}}
> that is produced in case the user submitted script returns with any other
> value than 0.
> {noformat}
> SshActionExecutor#getActionStatus
> ...
> String outFile = getRemoteFileName(context, action, "error", false, true);
> String checkErrorCmd = SSH_COMMAND_BASE + action.getTrackerUri() + " ls " +
> outFile;
> int retVal = getReturnValue(checkErrorCmd);
> ...
> {noformat}
>
> User requirement is to provide some more detailed information on the
> success/failure of the user-submitted script. That could be at a minimum the
> return value, optionally the last ~1K of the stderr that is drained. This
> information could then be communicated via {{errorMessage}} and {{ErrorCode}}
--
This message was sent by Atlassian Jira
(v8.3.2#803003)
[jira] [Commented] (OOZIE-3405) SSH action shows empty error Message and Error code
[
https://issues.apache.org/jira/browse/OOZIE-3405?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16921406#comment-16921406
]
Mate Juhasz commented on OOZIE-3405:
Taking this over [~orova] if you dont mind :)
> SSH action shows empty error Message and Error code
> ---
>
> Key: OOZIE-3405
> URL: https://issues.apache.org/jira/browse/OOZIE-3405
> Project: Oozie
> Issue Type: Improvement
>Reporter: Peter Orova
>Assignee: Mate Juhasz
>Priority: Minor
>
> Currently, when an SSH action fails the only message that is returned is the
> Status. Neither the {{error Message}} nor {{Error code}} fields are filled.
> This makes reporting on the causes of SSH Action failures via Oozie highly
> impractical: the only meaningful bit of information there is on a failed SSH
> Action is the Status.
> The Status is filled based on the presence (or lack of) the {{.error file}}
> that is produced in case the user submitted script returns with any other
> value than 0.
> {noformat}
> SshActionExecutor#getActionStatus
> ...
> String outFile = getRemoteFileName(context, action, "error", false, true);
> String checkErrorCmd = SSH_COMMAND_BASE + action.getTrackerUri() + " ls " +
> outFile;
> int retVal = getReturnValue(checkErrorCmd);
> ...
> {noformat}
>
> User requirement is to provide some more detailed information on the
> success/failure of the user-submitted script. That could be at a minimum the
> return value, optionally the last ~1K of the stderr that is drained. This
> information could then be communicated via {{errorMessage}} and {{ErrorCode}}
--
This message was sent by Atlassian Jira
(v8.3.2#803003)
[jira] [Assigned] (OOZIE-3405) SSH action shows empty error Message and Error code
[
https://issues.apache.org/jira/browse/OOZIE-3405?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Mate Juhasz reassigned OOZIE-3405:
--
Assignee: Mate Juhasz (was: Peter Orova)
> SSH action shows empty error Message and Error code
> ---
>
> Key: OOZIE-3405
> URL: https://issues.apache.org/jira/browse/OOZIE-3405
> Project: Oozie
> Issue Type: Improvement
>Reporter: Peter Orova
>Assignee: Mate Juhasz
>Priority: Minor
>
> Currently, when an SSH action fails the only message that is returned is the
> Status. Neither the {{error Message}} nor {{Error code}} fields are filled.
> This makes reporting on the causes of SSH Action failures via Oozie highly
> impractical: the only meaningful bit of information there is on a failed SSH
> Action is the Status.
> The Status is filled based on the presence (or lack of) the {{.error file}}
> that is produced in case the user submitted script returns with any other
> value than 0.
> {noformat}
> SshActionExecutor#getActionStatus
> ...
> String outFile = getRemoteFileName(context, action, "error", false, true);
> String checkErrorCmd = SSH_COMMAND_BASE + action.getTrackerUri() + " ls " +
> outFile;
> int retVal = getReturnValue(checkErrorCmd);
> ...
> {noformat}
>
> User requirement is to provide some more detailed information on the
> success/failure of the user-submitted script. That could be at a minimum the
> return value, optionally the last ~1K of the stderr that is drained. This
> information could then be communicated via {{errorMessage}} and {{ErrorCode}}
--
This message was sent by Atlassian Jira
(v8.3.2#803003)
[jira] [Comment Edited] (OOZIE-3024) email action ignores value of content_type attribute when attachment attribute is set
[
https://issues.apache.org/jira/browse/OOZIE-3024?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16887091#comment-16887091
]
Mate Juhasz edited comment on OOZIE-3024 at 7/17/19 2:20 PM:
-
[~asalamon74], I added a test for emails with attachments and verified on a
cluster as well that the fix is working.
was (Author: matijhs):
[~asalamon74], I added a test for emails with attachments and verified that the
fix is working.
> email action ignores value of content_type attribute when attachment
> attribute is set
> -
>
> Key: OOZIE-3024
> URL: https://issues.apache.org/jira/browse/OOZIE-3024
> Project: Oozie
> Issue Type: Bug
> Components: core
>Affects Versions: 4.2.0
>Reporter: Jim Hopper
>Assignee: Mate Juhasz
>Priority: Major
> Attachments: OOZIE-3024-V2.patch, OOZIE-3024.patch
>
>
> The email action ignores value of content_type attribute when attachment
> attribute is set.
> This results in an email with a html body sent as a plaintext.
> https://github.com/apache/oozie/blob/master/core/src/main/java/org/apache/oozie/action/email/EmailActionExecutor.java#L251
> {code}
> if(contentType.equals("text/html")) {
> MimeBodyPart bodyHtmlPart = new MimeBodyPart();
> bodyHtmlPart.setContent(body, contentType);
> multipart.addBodyPart(bodyHtmlPart);
> }
> else{
> MimeBodyPart bodyTextPart = new MimeBodyPart();
> bodyTextPart.setText(body);
> multipart.addBodyPart(bodyTextPart);
> }
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.14#76016)
[jira] [Commented] (OOZIE-3024) email action ignores value of content_type attribute when attachment attribute is set
[
https://issues.apache.org/jira/browse/OOZIE-3024?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16887091#comment-16887091
]
Mate Juhasz commented on OOZIE-3024:
[~asalamon74], I added a test for emails with attachments and verified that the
fix is working.
> email action ignores value of content_type attribute when attachment
> attribute is set
> -
>
> Key: OOZIE-3024
> URL: https://issues.apache.org/jira/browse/OOZIE-3024
> Project: Oozie
> Issue Type: Bug
> Components: core
>Affects Versions: 4.2.0
>Reporter: Jim Hopper
>Assignee: Mate Juhasz
>Priority: Major
> Attachments: OOZIE-3024-V2.patch, OOZIE-3024.patch
>
>
> The email action ignores value of content_type attribute when attachment
> attribute is set.
> This results in an email with a html body sent as a plaintext.
> https://github.com/apache/oozie/blob/master/core/src/main/java/org/apache/oozie/action/email/EmailActionExecutor.java#L251
> {code}
> if(contentType.equals("text/html")) {
> MimeBodyPart bodyHtmlPart = new MimeBodyPart();
> bodyHtmlPart.setContent(body, contentType);
> multipart.addBodyPart(bodyHtmlPart);
> }
> else{
> MimeBodyPart bodyTextPart = new MimeBodyPart();
> bodyTextPart.setText(body);
> multipart.addBodyPart(bodyTextPart);
> }
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.14#76016)
[jira] [Updated] (OOZIE-3024) email action ignores value of content_type attribute when attachment attribute is set
[
https://issues.apache.org/jira/browse/OOZIE-3024?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Mate Juhasz updated OOZIE-3024:
---
Attachment: OOZIE-3024-V2.patch
> email action ignores value of content_type attribute when attachment
> attribute is set
> -
>
> Key: OOZIE-3024
> URL: https://issues.apache.org/jira/browse/OOZIE-3024
> Project: Oozie
> Issue Type: Bug
> Components: core
>Affects Versions: 4.2.0
>Reporter: Jim Hopper
>Assignee: Mate Juhasz
>Priority: Major
> Attachments: OOZIE-3024-V2.patch, OOZIE-3024.patch
>
>
> The email action ignores value of content_type attribute when attachment
> attribute is set.
> This results in an email with a html body sent as a plaintext.
> https://github.com/apache/oozie/blob/master/core/src/main/java/org/apache/oozie/action/email/EmailActionExecutor.java#L251
> {code}
> if(contentType.equals("text/html")) {
> MimeBodyPart bodyHtmlPart = new MimeBodyPart();
> bodyHtmlPart.setContent(body, contentType);
> multipart.addBodyPart(bodyHtmlPart);
> }
> else{
> MimeBodyPart bodyTextPart = new MimeBodyPart();
> bodyTextPart.setText(body);
> multipart.addBodyPart(bodyTextPart);
> }
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.14#76016)
[jira] [Commented] (OOZIE-3024) email action ignores value of content_type attribute when attachment attribute is set
[
https://issues.apache.org/jira/browse/OOZIE-3024?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16885210#comment-16885210
]
Mate Juhasz commented on OOZIE-3024:
This issue has popped up recently, so I picked it up quickly. [~asalamon74]
could you take a look?
> email action ignores value of content_type attribute when attachment
> attribute is set
> -
>
> Key: OOZIE-3024
> URL: https://issues.apache.org/jira/browse/OOZIE-3024
> Project: Oozie
> Issue Type: Bug
> Components: core
>Affects Versions: 4.2.0
>Reporter: Jim Hopper
>Assignee: Mate Juhasz
>Priority: Major
> Attachments: OOZIE-3024.patch
>
>
> The email action ignores value of content_type attribute when attachment
> attribute is set.
> This results in an email with a html body sent as a plaintext.
> https://github.com/apache/oozie/blob/master/core/src/main/java/org/apache/oozie/action/email/EmailActionExecutor.java#L251
> {code}
> if(contentType.equals("text/html")) {
> MimeBodyPart bodyHtmlPart = new MimeBodyPart();
> bodyHtmlPart.setContent(body, contentType);
> multipart.addBodyPart(bodyHtmlPart);
> }
> else{
> MimeBodyPart bodyTextPart = new MimeBodyPart();
> bodyTextPart.setText(body);
> multipart.addBodyPart(bodyTextPart);
> }
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.14#76016)
[jira] [Updated] (OOZIE-3024) email action ignores value of content_type attribute when attachment attribute is set
[
https://issues.apache.org/jira/browse/OOZIE-3024?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Mate Juhasz updated OOZIE-3024:
---
Attachment: OOZIE-3024.patch
> email action ignores value of content_type attribute when attachment
> attribute is set
> -
>
> Key: OOZIE-3024
> URL: https://issues.apache.org/jira/browse/OOZIE-3024
> Project: Oozie
> Issue Type: Bug
> Components: core
>Affects Versions: 4.2.0
>Reporter: Jim Hopper
>Assignee: Mate Juhasz
>Priority: Major
> Attachments: OOZIE-3024.patch
>
>
> The email action ignores value of content_type attribute when attachment
> attribute is set.
> This results in an email with a html body sent as a plaintext.
> https://github.com/apache/oozie/blob/master/core/src/main/java/org/apache/oozie/action/email/EmailActionExecutor.java#L251
> {code}
> if(contentType.equals("text/html")) {
> MimeBodyPart bodyHtmlPart = new MimeBodyPart();
> bodyHtmlPart.setContent(body, contentType);
> multipart.addBodyPart(bodyHtmlPart);
> }
> else{
> MimeBodyPart bodyTextPart = new MimeBodyPart();
> bodyTextPart.setText(body);
> multipart.addBodyPart(bodyTextPart);
> }
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.14#76016)
[jira] [Assigned] (OOZIE-3024) email action ignores value of content_type attribute when attachment attribute is set
[
https://issues.apache.org/jira/browse/OOZIE-3024?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Mate Juhasz reassigned OOZIE-3024:
--
Assignee: Mate Juhasz
> email action ignores value of content_type attribute when attachment
> attribute is set
> -
>
> Key: OOZIE-3024
> URL: https://issues.apache.org/jira/browse/OOZIE-3024
> Project: Oozie
> Issue Type: Bug
> Components: core
>Affects Versions: 4.2.0
>Reporter: Jim Hopper
>Assignee: Mate Juhasz
>Priority: Major
>
> The email action ignores value of content_type attribute when attachment
> attribute is set.
> This results in an email with a html body sent as a plaintext.
> https://github.com/apache/oozie/blob/master/core/src/main/java/org/apache/oozie/action/email/EmailActionExecutor.java#L251
> {code}
> if(contentType.equals("text/html")) {
> MimeBodyPart bodyHtmlPart = new MimeBodyPart();
> bodyHtmlPart.setContent(body, contentType);
> multipart.addBodyPart(bodyHtmlPart);
> }
> else{
> MimeBodyPart bodyTextPart = new MimeBodyPart();
> bodyTextPart.setText(body);
> multipart.addBodyPart(bodyTextPart);
> }
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.14#76016)
[jira] [Commented] (OOZIE-3196) Authorization: restrict world readability by user
[ https://issues.apache.org/jira/browse/OOZIE-3196?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16812246#comment-16812246 ] Mate Juhasz commented on OOZIE-3196: [~orova] thank you for letting me to take over the remaining tasks. I would like to get introduced with the related code parts first, then I will update the jira, hopefully with a POC. > Authorization: restrict world readability by user > - > > Key: OOZIE-3196 > URL: https://issues.apache.org/jira/browse/OOZIE-3196 > Project: Oozie > Issue Type: New Feature > Components: bundle, coordinator, workflow >Affects Versions: 5.0.0b1, 5.0.0 >Reporter: Andras Piros >Assignee: Mate Juhasz >Priority: Major > Fix For: 5.2.0 > > Attachments: OOZIE-3196.001.patch > > > The [*current authorization > model*|https://issues.apache.org/jira/browse/OOZIE-228] does not fit the > enterprise requirements as everything is readable and writable by everyone by > default. > Write access can be restricted using authorization but restricting read > rights is only possible via Yarn ACLs and HDFS rights which still does not > prevent accessing the workflow, coordinator or bundle job’s configurations > for everyone. > Improve authorization so it’s possible to configure read/write access for > workflows, coordinators, and bundles in a more granular way. Could involve > Sentry during implementation or create and design a new system that fits the > needs. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Assigned] (OOZIE-3196) Authorization: restrict world readability by user
[ https://issues.apache.org/jira/browse/OOZIE-3196?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mate Juhasz reassigned OOZIE-3196: -- Assignee: Mate Juhasz (was: Peter Orova) > Authorization: restrict world readability by user > - > > Key: OOZIE-3196 > URL: https://issues.apache.org/jira/browse/OOZIE-3196 > Project: Oozie > Issue Type: New Feature > Components: bundle, coordinator, workflow >Affects Versions: 5.0.0b1, 5.0.0 >Reporter: Andras Piros >Assignee: Mate Juhasz >Priority: Major > Fix For: 5.2.0 > > Attachments: OOZIE-3196.001.patch > > > The [*current authorization > model*|https://issues.apache.org/jira/browse/OOZIE-228] does not fit the > enterprise requirements as everything is readable and writable by everyone by > default. > Write access can be restricted using authorization but restricting read > rights is only possible via Yarn ACLs and HDFS rights which still does not > prevent accessing the workflow, coordinator or bundle job’s configurations > for everyone. > Improve authorization so it’s possible to configure read/write access for > workflows, coordinators, and bundles in a more granular way. Could involve > Sentry during implementation or create and design a new system that fits the > needs. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (OOZIE-3450) Investigate and clean git sharelib
[ https://issues.apache.org/jira/browse/OOZIE-3450?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16804899#comment-16804899 ] Mate Juhasz commented on OOZIE-3450: Thanks [~asalamon74] for the testing. Could you please give the new patch a try as well? Interesting that each of the sharelib projects are using the oozie-core as a provided dependency, the only difference is that while most of them just reffering to static constants in the ActionExecutor-s, sharelib-git's GitMain tries to call a method in GitActionExecutor$ActionConfVerifier. This method is the ActionConfVerifier#checkAndGetTrimmed, which only returns an action conf value, not connecting strongly anywhere, so I moved it to the GitMain class. > Investigate and clean git sharelib > -- > > Key: OOZIE-3450 > URL: https://issues.apache.org/jira/browse/OOZIE-3450 > Project: Oozie > Issue Type: Improvement >Affects Versions: trunk >Reporter: Andras Salamon >Assignee: Mate Juhasz >Priority: Major > Attachments: OOZIE-3450-v2.patch, OOZIE-3540-v1.patch > > > I've checked the number of jars in the Oozie sharelibs and realized that git > sharelib contains the highest number of jars (203), it's much more than the > hive (85), pig (67). Not to mention that we have really small sharelibs like > distcp (3). > I don't really understand the reason for this, we need to check if we really > need all the jars here. The huge number of jars make it slower and it's more > likely that we get strange errors because of jar conflicts. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (OOZIE-3443) Migrate from joda time to java.time
[ https://issues.apache.org/jira/browse/OOZIE-3443?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mate Juhasz updated OOZIE-3443: --- Attachment: OOZIE-3443-V4.patch > Migrate from joda time to java.time > --- > > Key: OOZIE-3443 > URL: https://issues.apache.org/jira/browse/OOZIE-3443 > Project: Oozie > Issue Type: Improvement >Affects Versions: trunk >Reporter: Andras Salamon >Assignee: Mate Juhasz >Priority: Minor > Attachments: OOZIE-3443-V1.patch, OOZIE-3443-V2.patch, > OOZIE-3443-V3.patch, OOZIE-3443-V4.patch > > > From Java SE 8 onwards, users are asked to migrate to from [joda > time|https://www.joda.org/joda-time/] to java.time (JSR-310). > It seems to me, that we only use it directly in one place: > [TestWorkflowActionRetryInfoXCommand.java|https://github.com/apache/oozie/blob/master/core/src/test/java/org/apache/oozie/command/wf/TestWorkflowActionRetryInfoXCommand.java] > so it would be not too difficult to migrate. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (OOZIE-3443) Migrate from joda time to java.time
[ https://issues.apache.org/jira/browse/OOZIE-3443?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16803043#comment-16803043 ] Mate Juhasz commented on OOZIE-3443: I agree [~asalamon74], but in this case I would exclude the transitive joda-time from oozie-core, so we will only package them in the necessary sharelib projects. > Migrate from joda time to java.time > --- > > Key: OOZIE-3443 > URL: https://issues.apache.org/jira/browse/OOZIE-3443 > Project: Oozie > Issue Type: Improvement >Affects Versions: trunk >Reporter: Andras Salamon >Assignee: Mate Juhasz >Priority: Minor > Attachments: OOZIE-3443-V1.patch, OOZIE-3443-V2.patch, > OOZIE-3443-V3.patch, OOZIE-3443-V4.patch > > > From Java SE 8 onwards, users are asked to migrate to from [joda > time|https://www.joda.org/joda-time/] to java.time (JSR-310). > It seems to me, that we only use it directly in one place: > [TestWorkflowActionRetryInfoXCommand.java|https://github.com/apache/oozie/blob/master/core/src/test/java/org/apache/oozie/command/wf/TestWorkflowActionRetryInfoXCommand.java] > so it would be not too difficult to migrate. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (OOZIE-3450) Investigate and clean git sharelib
[
https://issues.apache.org/jira/browse/OOZIE-3450?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16802711#comment-16802711
]
Mate Juhasz commented on OOZIE-3450:
I have managed to reduce to number of jars on the git sharelib (9) with better
chosen dependency scopes, but unfortunately I could not find an environment to
run the examples code. [~asalamon74] could you please help to check what
happens if we keep the below jars only? I upload a patch as well, but these are
really minor changes in code.
{noformat}
find share/lib/git -name "*.jar" | sort
share/lib/git/JavaEWAH-1.1.6.jar
share/lib/git/commons-lang3-3.3.2.jar
share/lib/git/httpclient-4.3.6.jar
share/lib/git/httpcore-4.3.3.jar
share/lib/git/jsch-0.1.54.jar
share/lib/git/jzlib-1.1.1.jar
share/lib/git/oozie-sharelib-git-5.2.0-SNAPSHOT.jar
share/lib/git/org.eclipse.jgit-5.0.1.201806211838-r.jar
share/lib/git/slf4j-api-1.6.6.jar
{noformat}
> Investigate and clean git sharelib
> --
>
> Key: OOZIE-3450
> URL: https://issues.apache.org/jira/browse/OOZIE-3450
> Project: Oozie
> Issue Type: Improvement
>Affects Versions: trunk
>Reporter: Andras Salamon
>Priority: Major
> Attachments: OOZIE-3540-v1.patch
>
>
> I've checked the number of jars in the Oozie sharelibs and realized that git
> sharelib contains the highest number of jars (203), it's much more than the
> hive (85), pig (67). Not to mention that we have really small sharelibs like
> distcp (3).
> I don't really understand the reason for this, we need to check if we really
> need all the jars here. The huge number of jars make it slower and it's more
> likely that we get strange errors because of jar conflicts.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (OOZIE-3450) Investigate and clean git sharelib
[ https://issues.apache.org/jira/browse/OOZIE-3450?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mate Juhasz updated OOZIE-3450: --- Attachment: OOZIE-3540-v1.patch > Investigate and clean git sharelib > -- > > Key: OOZIE-3450 > URL: https://issues.apache.org/jira/browse/OOZIE-3450 > Project: Oozie > Issue Type: Improvement >Affects Versions: trunk >Reporter: Andras Salamon >Priority: Major > Attachments: OOZIE-3540-v1.patch > > > I've checked the number of jars in the Oozie sharelibs and realized that git > sharelib contains the highest number of jars (203), it's much more than the > hive (85), pig (67). Not to mention that we have really small sharelibs like > distcp (3). > I don't really understand the reason for this, we need to check if we really > need all the jars here. The huge number of jars make it slower and it's more > likely that we get strange errors because of jar conflicts. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (OOZIE-3443) Migrate from joda time to java.time
[
https://issues.apache.org/jira/browse/OOZIE-3443?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16801857#comment-16801857
]
Mate Juhasz commented on OOZIE-3443:
I agree :) I put back the dependency to the root pom.xml to force the versions
to 2.9.9, thanks [~asalamon74]!
Now the distro will look like this:
{noformat}
./embedded-oozie-server/webapp/WEB-INF/lib/joda-time-2.9.9.jar
./share/lib/pig/joda-time-2.9.9.jar
./share/lib/hcatalog/joda-time-2.9.9.jar
./share/lib/hive2/joda-time-2.9.9.jar
./share/lib/hive/joda-time-2.9.9.jar
./share/lib/spark/joda-time-2.9.9.jar
{noformat}
> Migrate from joda time to java.time
> ---
>
> Key: OOZIE-3443
> URL: https://issues.apache.org/jira/browse/OOZIE-3443
> Project: Oozie
> Issue Type: Improvement
>Affects Versions: trunk
>Reporter: Andras Salamon
>Assignee: Mate Juhasz
>Priority: Minor
> Attachments: OOZIE-3443-V1.patch, OOZIE-3443-V2.patch,
> OOZIE-3443-V3.patch
>
>
> From Java SE 8 onwards, users are asked to migrate to from [joda
> time|https://www.joda.org/joda-time/] to java.time (JSR-310).
> It seems to me, that we only use it directly in one place:
> [TestWorkflowActionRetryInfoXCommand.java|https://github.com/apache/oozie/blob/master/core/src/test/java/org/apache/oozie/command/wf/TestWorkflowActionRetryInfoXCommand.java]
> so it would be not too difficult to migrate.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (OOZIE-3443) Migrate from joda time to java.time
[ https://issues.apache.org/jira/browse/OOZIE-3443?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mate Juhasz updated OOZIE-3443: --- Attachment: OOZIE-3443-V3.patch > Migrate from joda time to java.time > --- > > Key: OOZIE-3443 > URL: https://issues.apache.org/jira/browse/OOZIE-3443 > Project: Oozie > Issue Type: Improvement >Affects Versions: trunk >Reporter: Andras Salamon >Assignee: Mate Juhasz >Priority: Minor > Attachments: OOZIE-3443-V1.patch, OOZIE-3443-V2.patch, > OOZIE-3443-V3.patch > > > From Java SE 8 onwards, users are asked to migrate to from [joda > time|https://www.joda.org/joda-time/] to java.time (JSR-310). > It seems to me, that we only use it directly in one place: > [TestWorkflowActionRetryInfoXCommand.java|https://github.com/apache/oozie/blob/master/core/src/test/java/org/apache/oozie/command/wf/TestWorkflowActionRetryInfoXCommand.java] > so it would be not too difficult to migrate. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (OOZIE-3443) Migrate from joda time to java.time
[ https://issues.apache.org/jira/browse/OOZIE-3443?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mate Juhasz updated OOZIE-3443: --- Attachment: OOZIE-3443-V2.patch > Migrate from joda time to java.time > --- > > Key: OOZIE-3443 > URL: https://issues.apache.org/jira/browse/OOZIE-3443 > Project: Oozie > Issue Type: Improvement >Affects Versions: trunk >Reporter: Andras Salamon >Assignee: Mate Juhasz >Priority: Minor > Attachments: OOZIE-3443-V1.patch, OOZIE-3443-V2.patch > > > From Java SE 8 onwards, users are asked to migrate to from [joda > time|https://www.joda.org/joda-time/] to java.time (JSR-310). > It seems to me, that we only use it directly in one place: > [TestWorkflowActionRetryInfoXCommand.java|https://github.com/apache/oozie/blob/master/core/src/test/java/org/apache/oozie/command/wf/TestWorkflowActionRetryInfoXCommand.java] > so it would be not too difficult to migrate. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (OOZIE-3443) Migrate from joda time to java.time
[ https://issues.apache.org/jira/browse/OOZIE-3443?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mate Juhasz updated OOZIE-3443: --- Attachment: OOZIE-3443-V1.patch > Migrate from joda time to java.time > --- > > Key: OOZIE-3443 > URL: https://issues.apache.org/jira/browse/OOZIE-3443 > Project: Oozie > Issue Type: Improvement >Affects Versions: trunk >Reporter: Andras Salamon >Assignee: Mate Juhasz >Priority: Minor > Attachments: OOZIE-3443-V1.patch > > > From Java SE 8 onwards, users are asked to migrate to from [joda > time|https://www.joda.org/joda-time/] to java.time (JSR-310). > It seems to me, that we only use it directly in one place: > [TestWorkflowActionRetryInfoXCommand.java|https://github.com/apache/oozie/blob/master/core/src/test/java/org/apache/oozie/command/wf/TestWorkflowActionRetryInfoXCommand.java] > so it would be not too difficult to migrate. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (OOZIE-3443) Migrate from joda time to java.time
[ https://issues.apache.org/jira/browse/OOZIE-3443?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mate Juhasz updated OOZIE-3443: --- Attachment: (was: OOZIE-3443-V1.patch) > Migrate from joda time to java.time > --- > > Key: OOZIE-3443 > URL: https://issues.apache.org/jira/browse/OOZIE-3443 > Project: Oozie > Issue Type: Improvement >Affects Versions: trunk >Reporter: Andras Salamon >Assignee: Mate Juhasz >Priority: Minor > Attachments: OOZIE-3443-V1.patch > > > From Java SE 8 onwards, users are asked to migrate to from [joda > time|https://www.joda.org/joda-time/] to java.time (JSR-310). > It seems to me, that we only use it directly in one place: > [TestWorkflowActionRetryInfoXCommand.java|https://github.com/apache/oozie/blob/master/core/src/test/java/org/apache/oozie/command/wf/TestWorkflowActionRetryInfoXCommand.java] > so it would be not too difficult to migrate. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (OOZIE-3443) Migrate from joda time to java.time
[ https://issues.apache.org/jira/browse/OOZIE-3443?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mate Juhasz updated OOZIE-3443: --- Attachment: OOZIE-3443-V1.patch > Migrate from joda time to java.time > --- > > Key: OOZIE-3443 > URL: https://issues.apache.org/jira/browse/OOZIE-3443 > Project: Oozie > Issue Type: Improvement >Affects Versions: trunk >Reporter: Andras Salamon >Assignee: Mate Juhasz >Priority: Minor > Attachments: OOZIE-3443-V1.patch > > > From Java SE 8 onwards, users are asked to migrate to from [joda > time|https://www.joda.org/joda-time/] to java.time (JSR-310). > It seems to me, that we only use it directly in one place: > [TestWorkflowActionRetryInfoXCommand.java|https://github.com/apache/oozie/blob/master/core/src/test/java/org/apache/oozie/command/wf/TestWorkflowActionRetryInfoXCommand.java] > so it would be not too difficult to migrate. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Assigned] (OOZIE-3443) Migrate from joda time to java.time
[ https://issues.apache.org/jira/browse/OOZIE-3443?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mate Juhasz reassigned OOZIE-3443: -- Assignee: Mate Juhasz > Migrate from joda time to java.time > --- > > Key: OOZIE-3443 > URL: https://issues.apache.org/jira/browse/OOZIE-3443 > Project: Oozie > Issue Type: Improvement >Affects Versions: trunk >Reporter: Andras Salamon >Assignee: Mate Juhasz >Priority: Minor > Attachments: OOZIE-3443-V1.patch > > > From Java SE 8 onwards, users are asked to migrate to from [joda > time|https://www.joda.org/joda-time/] to java.time (JSR-310). > It seems to me, that we only use it directly in one place: > [TestWorkflowActionRetryInfoXCommand.java|https://github.com/apache/oozie/blob/master/core/src/test/java/org/apache/oozie/command/wf/TestWorkflowActionRetryInfoXCommand.java] > so it would be not too difficult to migrate. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (OOZIE-1624) Exclusion pattern for sharelib JARs
[ https://issues.apache.org/jira/browse/OOZIE-1624?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mate Juhasz updated OOZIE-1624: --- Attachment: OOZIE-1624-V10.patch > Exclusion pattern for sharelib JARs > --- > > Key: OOZIE-1624 > URL: https://issues.apache.org/jira/browse/OOZIE-1624 > Project: Oozie > Issue Type: Sub-task >Affects Versions: 4.3.1 >Reporter: Purshotam Shah >Assignee: Mate Juhasz >Priority: Major > Attachments: OOZIE-1624-V10.patch, OOZIE-1624-V2.patch, > OOZIE-1624-V3.patch, OOZIE-1624-V4.patch, OOZIE-1624-V5.patch, > OOZIE-1624-v1.patch > > > Sharelib may bring some jar which might conflict with user jars. > Ex. Sharelib hive has json-2..jar, where as some of the user use-case > need higher version of json jar. > He should be able to exclude sharelib json jar and bring his own version. > > oozie.action.sharelib.for.hive.exclusion > json-\*.jar|abc-*.jar > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (OOZIE-1624) Exclusion pattern for sharelib JARs
[ https://issues.apache.org/jira/browse/OOZIE-1624?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mate Juhasz updated OOZIE-1624: --- Attachment: OOZIE-1624-V10.patch > Exclusion pattern for sharelib JARs > --- > > Key: OOZIE-1624 > URL: https://issues.apache.org/jira/browse/OOZIE-1624 > Project: Oozie > Issue Type: Sub-task >Affects Versions: 4.3.1 >Reporter: Purshotam Shah >Assignee: Mate Juhasz >Priority: Major > Attachments: OOZIE-1624-V10.patch, OOZIE-1624-V2.patch, > OOZIE-1624-V3.patch, OOZIE-1624-V4.patch, OOZIE-1624-V5.patch, > OOZIE-1624-v1.patch > > > Sharelib may bring some jar which might conflict with user jars. > Ex. Sharelib hive has json-2..jar, where as some of the user use-case > need higher version of json jar. > He should be able to exclude sharelib json jar and bring his own version. > > oozie.action.sharelib.for.hive.exclusion > json-\*.jar|abc-*.jar > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (OOZIE-1624) Exclusion pattern for sharelib JARs
[ https://issues.apache.org/jira/browse/OOZIE-1624?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16786844#comment-16786844 ] Mate Juhasz commented on OOZIE-1624: Fixed the typos and line breaks and also added the warning message for using exclusions :) > Exclusion pattern for sharelib JARs > --- > > Key: OOZIE-1624 > URL: https://issues.apache.org/jira/browse/OOZIE-1624 > Project: Oozie > Issue Type: Sub-task >Affects Versions: 4.3.1 >Reporter: Purshotam Shah >Assignee: Mate Juhasz >Priority: Major > Attachments: OOZIE-1624-V2.patch, OOZIE-1624-V3.patch, > OOZIE-1624-V4.patch, OOZIE-1624-V5.patch, OOZIE-1624-v1.patch > > > Sharelib may bring some jar which might conflict with user jars. > Ex. Sharelib hive has json-2..jar, where as some of the user use-case > need higher version of json jar. > He should be able to exclude sharelib json jar and bring his own version. > > oozie.action.sharelib.for.hive.exclusion > json-\*.jar|abc-*.jar > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (OOZIE-1624) Exclusion pattern for sharelib JARs
[ https://issues.apache.org/jira/browse/OOZIE-1624?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16786802#comment-16786802 ] Mate Juhasz commented on OOZIE-1624: Thanks [~asalamon74], I forgot to add it. Attached to the review, hope it's clear and understandable. > Exclusion pattern for sharelib JARs > --- > > Key: OOZIE-1624 > URL: https://issues.apache.org/jira/browse/OOZIE-1624 > Project: Oozie > Issue Type: Sub-task >Affects Versions: 4.3.1 >Reporter: Purshotam Shah >Assignee: Mate Juhasz >Priority: Major > Attachments: OOZIE-1624-V2.patch, OOZIE-1624-V3.patch, > OOZIE-1624-V4.patch, OOZIE-1624-V5.patch, OOZIE-1624-v1.patch > > > Sharelib may bring some jar which might conflict with user jars. > Ex. Sharelib hive has json-2..jar, where as some of the user use-case > need higher version of json jar. > He should be able to exclude sharelib json jar and bring his own version. > > oozie.action.sharelib.for.hive.exclusion > json-\*.jar|abc-*.jar > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (OOZIE-1624) Exclusion pattern for sharelib JARs
[ https://issues.apache.org/jira/browse/OOZIE-1624?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16781760#comment-16781760 ] Mate Juhasz commented on OOZIE-1624: Thanks [~asalamon74], uploaded to the review board as well. I agree with you, the ActionMarshaller could go in separately. Waiting for your feedback, thanks! > Exclusion pattern for sharelib JARs > --- > > Key: OOZIE-1624 > URL: https://issues.apache.org/jira/browse/OOZIE-1624 > Project: Oozie > Issue Type: Sub-task >Affects Versions: 4.3.1 >Reporter: Purshotam Shah >Assignee: Mate Juhasz >Priority: Major > Attachments: OOZIE-1624-V2.patch, OOZIE-1624-V3.patch, > OOZIE-1624-V4.patch, OOZIE-1624-V5.patch, OOZIE-1624-v1.patch > > > Sharelib may bring some jar which might conflict with user jars. > Ex. Sharelib hive has json-2..jar, where as some of the user use-case > need higher version of json jar. > He should be able to exclude sharelib json jar and bring his own version. > > oozie.action.sharelib.for.hive.exclusion > json-\*.jar|abc-*.jar > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (OOZIE-1624) Exclusion pattern for sharelib JARs
[ https://issues.apache.org/jira/browse/OOZIE-1624?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16781688#comment-16781688 ] Mate Juhasz commented on OOZIE-1624: Uploaded new patch based on the current master > Exclusion pattern for sharelib JARs > --- > > Key: OOZIE-1624 > URL: https://issues.apache.org/jira/browse/OOZIE-1624 > Project: Oozie > Issue Type: Sub-task >Affects Versions: 4.3.1 >Reporter: Purshotam Shah >Assignee: Mate Juhasz >Priority: Major > Attachments: OOZIE-1624-V2.patch, OOZIE-1624-V3.patch, > OOZIE-1624-V4.patch, OOZIE-1624-V5.patch, OOZIE-1624-v1.patch > > > Sharelib may bring some jar which might conflict with user jars. > Ex. Sharelib hive has json-2..jar, where as some of the user use-case > need higher version of json jar. > He should be able to exclude sharelib json jar and bring his own version. > > oozie.action.sharelib.for.hive.exclusion > json-\*.jar|abc-*.jar > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (OOZIE-1624) Exclusion pattern for sharelib JARs
[ https://issues.apache.org/jira/browse/OOZIE-1624?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mate Juhasz updated OOZIE-1624: --- Attachment: OOZIE-1624-V5.patch > Exclusion pattern for sharelib JARs > --- > > Key: OOZIE-1624 > URL: https://issues.apache.org/jira/browse/OOZIE-1624 > Project: Oozie > Issue Type: Sub-task >Affects Versions: 4.3.1 >Reporter: Purshotam Shah >Assignee: Mate Juhasz >Priority: Major > Attachments: OOZIE-1624-V2.patch, OOZIE-1624-V3.patch, > OOZIE-1624-V4.patch, OOZIE-1624-V5.patch, OOZIE-1624-v1.patch > > > Sharelib may bring some jar which might conflict with user jars. > Ex. Sharelib hive has json-2..jar, where as some of the user use-case > need higher version of json jar. > He should be able to exclude sharelib json jar and bring his own version. > > oozie.action.sharelib.for.hive.exclusion > json-\*.jar|abc-*.jar > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (OOZIE-3282) Prototyping: create Angular 6 workflow list page
[ https://issues.apache.org/jira/browse/OOZIE-3282?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16776773#comment-16776773 ] Mate Juhasz commented on OOZIE-3282: [~kmarton] I didnt have time to work on it and I found the react protoype really great, maybe we should go with that one > Prototyping: create Angular 6 workflow list page > > > Key: OOZIE-3282 > URL: https://issues.apache.org/jira/browse/OOZIE-3282 > Project: Oozie > Issue Type: Sub-task >Reporter: Andras Piros >Assignee: Mate Juhasz >Priority: Major > > Create an [*Angular 6*|https://angular.io/docs] prototype of the workflow > list page. Don't have to handle authentication / authorization for now. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (OOZIE-3355) Regex based search option for searching workflows in the WFM-View of Ambari
[
https://issues.apache.org/jira/browse/OOZIE-3355?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16647712#comment-16647712
]
Mate Juhasz commented on OOZIE-3355:
[~andras.piros], these are the available filter parameters from the ambari wfm:
* name
* id (oozie job id)
* user (job submitter user)
* status (SUCCEEDED, RUNNING, FAILED, KILLED, SUSPENDED)
* jobtype (wf, coord, bundles)
* startCreatedTime
* endCreatedTime
The offset and len parameters are used for paginating the results.
* len - always 20
* offset - always (page number - 1) * len
The filter parameter is filled with each of the user defined conditions, but
these seem to be in OR relation instead of AND:
_filter=status=SUCCEEDED;name=mr-wf_
Another observation about time filtering, that the wfm shows the local time,
but the filtering has to be done in GMT as oozie works.
> Regex based search option for searching workflows in the WFM-View of Ambari
> ---
>
> Key: OOZIE-3355
> URL: https://issues.apache.org/jira/browse/OOZIE-3355
> Project: Oozie
> Issue Type: New Feature
> Components: bundle, coordinator, core, tools, ui, workflow
>Affects Versions: 4.2.0
>Reporter: Krishnadevan Purushothaman
>Priority: Critical
>
> *Challenge faced :*
> _{color:#d04437}In the WFM view of ambari, there is no Filter option
> available to search for the Workflows. In order to search for the desired
> workflow, one has to type the full name of workflow,coordinator,bundles else
> it does not return anything which becomes a time-consuming job.{color}_
> *Feature description:*
> _{color:#14892c}There is a need for Regex based filter option in order to
> search the workflows without the need of entering the complete name of the
> workflow,coordinator,bundles rather by just typing the first three letters of
> the work post which it should populate suggestions based on the first three
> letters of the workflow,coordinator,bundles through which we can refine and
> optimize the searching mechanism.{color}_
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (OOZIE-3355) Regex based search option for searching workflows in the WFM-View of Ambari
[
https://issues.apache.org/jira/browse/OOZIE-3355?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16639866#comment-16639866
]
Mate Juhasz commented on OOZIE-3355:
As I see, the wfm executes a request like this for getting the jobs in
_org/apache/oozie/ambari/view/OozieDelegate.java#readFromOozie_:
{code:java}
Proxy request for url: [GET]
http://ctr-e138-1518143905142-496852-01-03.hwx.site:11000/oozie/v2/jobs?filter=name=sparkcompare=1=1=wf
{code}
Then Oozie handles it in _org/apache/oozie/servlet/V0JobsServlet.java#getJobs,_
and the actual query is done in
_org/apache/oozie/executor/jpa/WorkflowsJobGetJPAExecutor.java#execute_
WorkflowsJobGetJPAExecutor is a bit heavy, but if we can agree on a solution I
am happy to do it.
> Regex based search option for searching workflows in the WFM-View of Ambari
> ---
>
> Key: OOZIE-3355
> URL: https://issues.apache.org/jira/browse/OOZIE-3355
> Project: Oozie
> Issue Type: New Feature
> Components: bundle, coordinator, core, tools, ui, workflow
>Affects Versions: 4.2.0
>Reporter: Krishnadevan Purushothaman
>Priority: Critical
>
> *Challenge faced :*
> _{color:#d04437}In the WFM view of ambari, there is no Filter option
> available to search for the Workflows. In order to search for the desired
> workflow, one has to type the full name of workflow,coordinator,bundles else
> it does not return anything which becomes a time-consuming job.{color}_
> *Feature description:*
> _{color:#14892c}There is a need for Regex based filter option in order to
> search the workflows without the need of entering the complete name of the
> workflow,coordinator,bundles rather by just typing the first three letters of
> the work post which it should populate suggestions based on the first three
> letters of the workflow,coordinator,bundles through which we can refine and
> optimize the searching mechanism.{color}_
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (OOZIE-3061) Kill only those child jobs which are not already killed
[ https://issues.apache.org/jira/browse/OOZIE-3061?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16602159#comment-16602159 ] Mate Juhasz commented on OOZIE-3061: Thanks [~andras.piros], I uploaded a patch according to your comments > Kill only those child jobs which are not already killed > --- > > Key: OOZIE-3061 > URL: https://issues.apache.org/jira/browse/OOZIE-3061 > Project: Oozie > Issue Type: Bug >Affects Versions: 4.3.0 >Reporter: Satish Subhashrao Saley >Assignee: Mate Juhasz >Priority: Trivial > Labels: newbie, newbiee > Fix For: 5.1.0 > > Attachments: OOZIE-3061-001.patch, OOZIE-3061-002.patch, > OOZIE-3061-003.patch, OOZIE-3061-004.patch, OOZIE-3061-005.patch, > OOZIE-3061-006.patch > > > Here we kill all child jobs. > https://github.com/apache/oozie/blob/master/sharelib/oozie/src/main/java/org/apache/oozie/action/hadoop/LauncherMain.java#L265 > We should check before killing for already killed application. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (OOZIE-3061) Kill only those child jobs which are not already killed
[ https://issues.apache.org/jira/browse/OOZIE-3061?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mate Juhasz updated OOZIE-3061: --- Attachment: OOZIE-3061-006.patch > Kill only those child jobs which are not already killed > --- > > Key: OOZIE-3061 > URL: https://issues.apache.org/jira/browse/OOZIE-3061 > Project: Oozie > Issue Type: Bug >Affects Versions: 4.3.0 >Reporter: Satish Subhashrao Saley >Assignee: Mate Juhasz >Priority: Trivial > Labels: newbie, newbiee > Fix For: 5.1.0 > > Attachments: OOZIE-3061-001.patch, OOZIE-3061-002.patch, > OOZIE-3061-003.patch, OOZIE-3061-004.patch, OOZIE-3061-005.patch, > OOZIE-3061-006.patch > > > Here we kill all child jobs. > https://github.com/apache/oozie/blob/master/sharelib/oozie/src/main/java/org/apache/oozie/action/hadoop/LauncherMain.java#L265 > We should check before killing for already killed application. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
