[jira] [Commented] (OOZIE-3370) Property filtering is not consistent across job submission
[ https://issues.apache.org/jira/browse/OOZIE-3370?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16656818#comment-16656818 ] Andras Piros commented on OOZIE-3370: - [~gezapeti] can you please review patch 002 (amendment patch)? Thanks! > Property filtering is not consistent across job submission > -- > > Key: OOZIE-3370 > URL: https://issues.apache.org/jira/browse/OOZIE-3370 > Project: Oozie > Issue Type: Bug >Affects Versions: 5.0.0 >Reporter: Peter Cseh >Assignee: Andras Piros >Priority: Major > Fix For: 5.1.0 > > Attachments: OOZIE-3370.001.patch, OOZIE-3370.002.patch > > > There are some properties which are filtered out in JavaActionExecutor, but > not in other places. We should make this filtering consistent. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (OOZIE-3371) TestSubWorkflowActionExecutor#testSubWorkflowRerun() is flaky
[
https://issues.apache.org/jira/browse/OOZIE-3371?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16657755#comment-16657755
]
Andras Piros commented on OOZIE-3371:
-
[~gezapeti] [~pbacsko] can you please review? Thanks!
> TestSubWorkflowActionExecutor#testSubWorkflowRerun() is flaky
> -
>
> Key: OOZIE-3371
> URL: https://issues.apache.org/jira/browse/OOZIE-3371
> Project: Oozie
> Issue Type: Sub-task
> Components: core, tests
>Affects Versions: 5.0.0
>Reporter: Andras Piros
>Assignee: Andras Piros
>Priority: Major
> Attachments: OOZIE-3371.001.patch
>
>
> {noformat}
> 2018-10-19 09:43:39 [INFO]
> ---
> 2018-10-19 09:43:39 [INFO] T E S T S
> 2018-10-19 09:43:39 [INFO]
> ---
> 2018-10-19 09:46:54 [INFO] Running
> org.apache.oozie.action.oozie.TestSubWorkflowActionExecutor
> 2018-10-19 09:46:54 [ERROR] Tests run: 15, Failures: 1, Errors: 0, Skipped:
> 0, Time elapsed: 194.156 s <<< FAILURE! - in
> org.apache.oozie.action.oozie.TestSubWorkflowActionExecutor
> 2018-10-19 09:46:54 [ERROR]
> testSubWorkflowRerun(org.apache.oozie.action.oozie.TestSubWorkflowActionExecutor)
> Time elapsed: 103.736 s <<< FAILURE!
> 2018-10-19 09:46:54 junit.framework.AssertionFailedError:
> expected: but was:
> 2018-10-19 09:46:54 at junit.framework.Assert.fail(Assert.java:57)
> 2018-10-19 09:46:54 at junit.framework.Assert.failNotEquals(Assert.java:329)
> 2018-10-19 09:46:54 at junit.framework.Assert.assertEquals(Assert.java:78)
> 2018-10-19 09:46:54 at junit.framework.Assert.assertEquals(Assert.java:86)
> 2018-10-19 09:46:54 at
> junit.framework.TestCase.assertEquals(TestCase.java:253)
> 2018-10-19 09:46:54 at
> org.apache.oozie.action.oozie.TestSubWorkflowActionExecutor.testSubWorkflowRerun(TestSubWorkflowActionExecutor.java:580)
> ...
> 2018-10-19 09:46:54 [ERROR]
> TestSubWorkflowActionExecutor.testSubWorkflowRerun:580 expected:
> but was:
> {noformat}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (OOZIE-3373) Logging of lock information inconsistent
[
https://issues.apache.org/jira/browse/OOZIE-3373?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16661880#comment-16661880
]
Andras Piros commented on OOZIE-3373:
-
Thanks [~Prabhu Joseph] for creating this JIRA. Do you have a patch?
> Logging of lock information inconsistent
>
>
> Key: OOZIE-3373
> URL: https://issues.apache.org/jira/browse/OOZIE-3373
> Project: Oozie
> Issue Type: Bug
> Components: core
>Affects Versions: 4.3.0
>Reporter: Prabhu Joseph
>Assignee: Prabhu Joseph
>Priority: Minor
>
> The lock resource name getEntityKey() is printed when lock is acquired
> whereas getName() + "_" + actionId is printed when Could not get lock. This
> makes difficult to correlate the logs.
> {code}
> LOG.debug("Acquired lock for [{0}] in [{1}]", getEntityKey(), getName());
> LOG.debug("Could not get lock [{0}], timed out [{1}]ms, and requeue itself
> [{2}]", this.toString(),
> getLockTimeOut(), getName());
> {code}
> Example:
> {code}
> 2018-10-10 12:00:05,698 pool-6-thread-5 DEBUG ActionCheckXCommand:526 -
> SERVER[dn3-george] USER[-] GROUP[-] TOKEN[-] APP[-] JOB[-]
> ACTION[134-181002155022081-oozie-oozi-W@shell1] Could not get lock
> [action.check_134-181002155022081-oozie-oozi-W@shell1], timed out
> [5,000]ms, and requeue itself [action.check]
> {code}
> The actual lock it waits is for 134-181002155022081-oozie-oozi-W which is
> not printed.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (OOZIE-3372) Adding credentials to multiple workflows after enabling Kerberos
[
https://issues.apache.org/jira/browse/OOZIE-3372?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16661958#comment-16661958
]
Andras Piros commented on OOZIE-3372:
-
Interesting idea [~kuldeepkulkarn...@gmail.com] having a default set of
{{CredentialProvider}} classes that would be always given to each workflow.
What can be done today is by using the [Fluent Job
API|https://github.com/apache/oozie/blob/master/docs/src/site/markdown/DG_FluentJobAPI.md]
to define a common {{Credential}} object w/ a {{CredentialBuilder}} and use it
multiple times [as in the
example|https://github.com/apache/oozie/blob/master/examples/src/main/java/org/apache/oozie/example/fluentjob/CredentialsRetrying.java].
This gives the user a possibility to reuse the same Fluent Job {{Credential}}
across different {{Workflow}} instances. It doesn't have an effect for all the
workflows, though - only for the ones you'll be injecting.
I see following caveats for having one single place adding a {{Credential}} to
all the {{Workflow}} instances of an Oozie service:
* all the {{Workflow}} instances would be affected with no exception. This can
be problematic if some workflows don't need e.g. {{HbaseCredential}} instances
(because these don't need to talk to HBase), for performance reasons
* {{HCatCredentials}} and {{Hive2Credentials}} use parameters that are
specific to the HCatalog or HiveServer2 instances these workflow actions will
be connecting. There are scenarios where there are multiple HiveServer2
instances w/ different {{Hive2Credentials}} parameters, needed for two
different workflow actions for the same workflow job. In case we'd have a
default {{Hive2Credentials}} created and supplied for both actions (but
necessary only for the first one), and a special one (necessary for only the
other action), I can imagine the second action wouldn't be able to authenticate
correctly, based on the incorrect {{Hive2Credentials}} parameters supplied by
the default one
To summarize, even if we'd implement a way to inject special
{{CredentialsProvider}} instances into each workflow action by default, we
still would want some kind of intelligent blacklisting, based on e.g. action
type / default providers mapping (e.g. {{Hive2Credentials}} wouldn't be
injected to Sqoop actions but for Hive2 and Spark actions), and explicit
definition (the default {{Hive2Credentials}} wouldn't be injected for Hive2
actions that have an explicit setting). [~kmarton] [~asalamon74] what do you
think?
> Adding credentials to multiple workflows after enabling Kerberos
>
>
> Key: OOZIE-3372
> URL: https://issues.apache.org/jira/browse/OOZIE-3372
> Project: Oozie
> Issue Type: Improvement
> Components: workflow
>Reporter: Kuldeep Kulkarni
>Priority: Minor
>
> If I have thousands of running workflows in a cluster(unsecured) and I
> configure Kerberos authentication, I will have to edit each workflow to add a
> credential section which can be difficult and time consuming. Is there any
> option to avoid manual edit? If not, can we add a feature using which we can
> configure credentials at one place (maybe in Oozie configs) and can be
> referred by respective action in the workflow?
> I know it can be complex to do this however just wanted to check if there is
> any better way or this can be considered as a feature request.
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Created] (OOZIE-3376) [tests] TestGraphGenerator should assume JDK8 minor version at least 1.8.0_u40
Andras Piros created OOZIE-3376:
---
Summary: [tests] TestGraphGenerator should assume JDK8 minor
version at least 1.8.0_u40
Key: OOZIE-3376
URL: https://issues.apache.org/jira/browse/OOZIE-3376
Project: Oozie
Issue Type: Bug
Components: docs, tests
Affects Versions: 5.1.0
Reporter: Andras Piros
Assignee: Andras Piros
{{GraphGenerator}} uses {{guru.nidi:graphviz-java:0.7.0}} that assumes JDK8
minor version [at least
1.8.0_u40|https://github.com/nidi3/graphviz-java/commit/b7cf5761f97f1491d3bdc65367ec00e38d66291d].
Goal is to make the same assumption inside {{TestGraphGenerator}} and document
behavior within {{WebServicesAPI.md}}.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (OOZIE-3376) [tests] TestGraphGenerator should assume JDK8 minor version at least 1.8.0_u40
[
https://issues.apache.org/jira/browse/OOZIE-3376?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Andras Piros updated OOZIE-3376:
Attachment: OOZIE-3376.001.patch
> [tests] TestGraphGenerator should assume JDK8 minor version at least 1.8.0_u40
> --
>
> Key: OOZIE-3376
> URL: https://issues.apache.org/jira/browse/OOZIE-3376
> Project: Oozie
> Issue Type: Bug
> Components: docs, tests
>Affects Versions: 5.1.0
>Reporter: Andras Piros
>Assignee: Andras Piros
>Priority: Major
> Attachments: OOZIE-3376.001.patch
>
>
> {{GraphGenerator}} uses {{guru.nidi:graphviz-java:0.7.0}} that assumes JDK8
> minor version [at least
> 1.8.0_u40|https://github.com/nidi3/graphviz-java/commit/b7cf5761f97f1491d3bdc65367ec00e38d66291d].
> Goal is to make the same assumption inside {{TestGraphGenerator}} and
> document behavior within {{WebServicesAPI.md}}.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Created] (OOZIE-3377) [docs] Remaining 5.1.0 documentation changes
Andras Piros created OOZIE-3377: --- Summary: [docs] Remaining 5.1.0 documentation changes Key: OOZIE-3377 URL: https://issues.apache.org/jira/browse/OOZIE-3377 Project: Oozie Issue Type: Task Components: docs Affects Versions: 5.1.0 Reporter: Andras Piros Assignee: Andras Piros Following documentation changes needed for 5.1.0: * link DG_FluentJobAPI.md from index.md * extract docs on Git action from WorkflowFunctionalSpecification.md to its own action extension docs: DG_GitActionExtension.md -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (OOZIE-3373) Logging of lock information inconsistent
[
https://issues.apache.org/jira/browse/OOZIE-3373?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16674823#comment-16674823
]
Andras Piros commented on OOZIE-3373:
-
Thanks for the contribution [~Prabhu Joseph]! +1
> Logging of lock information inconsistent
>
>
> Key: OOZIE-3373
> URL: https://issues.apache.org/jira/browse/OOZIE-3373
> Project: Oozie
> Issue Type: Bug
> Components: core
>Affects Versions: 4.3.0
>Reporter: Prabhu Joseph
>Assignee: Prabhu Joseph
>Priority: Minor
> Attachments: OOZIE-3373-1.patch
>
>
> The lock resource name getEntityKey() is printed when lock is acquired
> whereas getName() + "_" + actionId is printed when Could not get lock. This
> makes difficult to correlate the logs.
> {code}
> LOG.debug("Acquired lock for [{0}] in [{1}]", getEntityKey(), getName());
> LOG.debug("Could not get lock [{0}], timed out [{1}]ms, and requeue itself
> [{2}]", this.toString(),
> getLockTimeOut(), getName());
> {code}
> Example:
> {code}
> 2018-10-10 12:00:05,698 pool-6-thread-5 DEBUG ActionCheckXCommand:526 -
> SERVER[dn3-george] USER[-] GROUP[-] TOKEN[-] APP[-] JOB[-]
> ACTION[134-181002155022081-oozie-oozi-W@shell1] Could not get lock
> [action.check_134-181002155022081-oozie-oozi-W@shell1], timed out
> [5,000]ms, and requeue itself [action.check]
> {code}
> The actual lock it waits is for 134-181002155022081-oozie-oozi-W which is
> not printed.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (OOZIE-3373) Logging of lock information is inconsistent
[
https://issues.apache.org/jira/browse/OOZIE-3373?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Andras Piros updated OOZIE-3373:
Summary: Logging of lock information is inconsistent (was: Logging of lock
information inconsistent)
> Logging of lock information is inconsistent
> ---
>
> Key: OOZIE-3373
> URL: https://issues.apache.org/jira/browse/OOZIE-3373
> Project: Oozie
> Issue Type: Bug
> Components: core
>Affects Versions: 4.3.0
>Reporter: Prabhu Joseph
>Assignee: Prabhu Joseph
>Priority: Minor
> Attachments: OOZIE-3373-1.patch
>
>
> The lock resource name getEntityKey() is printed when lock is acquired
> whereas getName() + "_" + actionId is printed when Could not get lock. This
> makes difficult to correlate the logs.
> {code}
> LOG.debug("Acquired lock for [{0}] in [{1}]", getEntityKey(), getName());
> LOG.debug("Could not get lock [{0}], timed out [{1}]ms, and requeue itself
> [{2}]", this.toString(),
> getLockTimeOut(), getName());
> {code}
> Example:
> {code}
> 2018-10-10 12:00:05,698 pool-6-thread-5 DEBUG ActionCheckXCommand:526 -
> SERVER[dn3-george] USER[-] GROUP[-] TOKEN[-] APP[-] JOB[-]
> ACTION[134-181002155022081-oozie-oozi-W@shell1] Could not get lock
> [action.check_134-181002155022081-oozie-oozi-W@shell1], timed out
> [5,000]ms, and requeue itself [action.check]
> {code}
> The actual lock it waits is for 134-181002155022081-oozie-oozi-W which is
> not printed.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (OOZIE-3373) [core] Logging of lock information is inconsistent
[
https://issues.apache.org/jira/browse/OOZIE-3373?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Andras Piros updated OOZIE-3373:
Summary: [core] Logging of lock information is inconsistent (was: Logging
of lock information is inconsistent)
> [core] Logging of lock information is inconsistent
> --
>
> Key: OOZIE-3373
> URL: https://issues.apache.org/jira/browse/OOZIE-3373
> Project: Oozie
> Issue Type: Bug
> Components: core
>Affects Versions: 4.3.0
>Reporter: Prabhu Joseph
>Assignee: Prabhu Joseph
>Priority: Minor
> Attachments: OOZIE-3373-1.patch
>
>
> The lock resource name getEntityKey() is printed when lock is acquired
> whereas getName() + "_" + actionId is printed when Could not get lock. This
> makes difficult to correlate the logs.
> {code}
> LOG.debug("Acquired lock for [{0}] in [{1}]", getEntityKey(), getName());
> LOG.debug("Could not get lock [{0}], timed out [{1}]ms, and requeue itself
> [{2}]", this.toString(),
> getLockTimeOut(), getName());
> {code}
> Example:
> {code}
> 2018-10-10 12:00:05,698 pool-6-thread-5 DEBUG ActionCheckXCommand:526 -
> SERVER[dn3-george] USER[-] GROUP[-] TOKEN[-] APP[-] JOB[-]
> ACTION[134-181002155022081-oozie-oozi-W@shell1] Could not get lock
> [action.check_134-181002155022081-oozie-oozi-W@shell1], timed out
> [5,000]ms, and requeue itself [action.check]
> {code}
> The actual lock it waits is for 134-181002155022081-oozie-oozi-W which is
> not printed.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (OOZIE-3376) [tests] TestGraphGenerator should assume JDK8 minor version at least 1.8.0_u40
[
https://issues.apache.org/jira/browse/OOZIE-3376?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16674860#comment-16674860
]
Andras Piros commented on OOZIE-3376:
-
[~pbacsko] [~gezapeti] can you please review? Thanks!
> [tests] TestGraphGenerator should assume JDK8 minor version at least 1.8.0_u40
> --
>
> Key: OOZIE-3376
> URL: https://issues.apache.org/jira/browse/OOZIE-3376
> Project: Oozie
> Issue Type: Bug
> Components: docs, tests
>Affects Versions: 5.1.0
>Reporter: Andras Piros
>Assignee: Andras Piros
>Priority: Major
> Attachments: OOZIE-3376.001.patch
>
>
> {{GraphGenerator}} uses {{guru.nidi:graphviz-java:0.7.0}} that assumes JDK8
> minor version [at least
> 1.8.0_u40|https://github.com/nidi3/graphviz-java/commit/b7cf5761f97f1491d3bdc65367ec00e38d66291d].
> Goal is to make the same assumption inside {{TestGraphGenerator}} and
> document behavior within {{WebServicesAPI.md}}.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (OOZIE-3377) [docs] Remaining 5.1.0 documentation changes
[
https://issues.apache.org/jira/browse/OOZIE-3377?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Andras Piros updated OOZIE-3377:
Description:
Following documentation changes needed for 5.1.0:
* link {{DG_FluentJobAPI.md}} from {{index.md}}
* extract docs on Git action from {{WorkflowFunctionalSpecification.md}} to
its own action extension docs: {{DG_GitActionExtension.md}}
was:
Following documentation changes needed for 5.1.0:
* link DG_FluentJobAPI.md from index.md
* extract docs on Git action from WorkflowFunctionalSpecification.md to its
own action extension docs: DG_GitActionExtension.md
> [docs] Remaining 5.1.0 documentation changes
>
>
> Key: OOZIE-3377
> URL: https://issues.apache.org/jira/browse/OOZIE-3377
> Project: Oozie
> Issue Type: Task
> Components: docs
>Affects Versions: 5.1.0
>Reporter: Andras Piros
>Assignee: Andras Piros
>Priority: Major
>
> Following documentation changes needed for 5.1.0:
> * link {{DG_FluentJobAPI.md}} from {{index.md}}
> * extract docs on Git action from {{WorkflowFunctionalSpecification.md}} to
> its own action extension docs: {{DG_GitActionExtension.md}}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Assigned] (OOZIE-3375) Can't use empty in coordinator
[
https://issues.apache.org/jira/browse/OOZIE-3375?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Andras Piros reassigned OOZIE-3375:
---
Assignee: Jacob Tolar
> Can't use empty in coordinator
> ---
>
> Key: OOZIE-3375
> URL: https://issues.apache.org/jira/browse/OOZIE-3375
> Project: Oozie
> Issue Type: Bug
>Reporter: Jacob Tolar
>Assignee: Jacob Tolar
>Priority: Major
> Attachments: OOZIE-3375-001.patch
>
>
> If I set a property to empty string in the {{}} block of my
> coordinator and later use it in the {{}} block Oozie throws an error.
> That is, this code fails:
> {code:java}
> end="2018-10-01T00:04Z" timezone="UTC" xmlns="uri:oozie:coordinator:0.5">
>
>
> test_param
>
>
>
>
> 5
> 1
>
>
>
> /workflow.xml
>
>
> renamed_param
> ${test_param}
>
>
>
>
>
> {code}
> The error is like this:
> {code:java}
> org.apache.oozie.command.CommandException: E1021: Coord Action Input Check
> Error: E1004: Expression language evaluation error, Unable to evaluate
> :${test_param}:
> ...
> Caused by: javax.servlet.jsp.el.ELException: variable [test_param] cannot be
> resolved
> {code}
> What happens: The coordinator submits successfully. When the first action
> materializes in
> [CoordMaterializeTransitionXCommand|https://github.com/apache/oozie/blob/65936460e263f9076bb552190be85396c2cc6d33/core/src/main/java/org/apache/oozie/command/coord/CoordMaterializeTransitionXCommand.java#L373-L379],
> the coordinator conf is parsed into an Oozie {{XConfiguration}} object.
> In
> [XConfiguration.processNodes|https://github.com/apache/oozie/blob/65936460e263f9076bb552190be85396c2cc6d33/core/src/main/java/org/apache/oozie/util/XConfiguration.java#L313-L354],
> present-but-empty values are not added to the configuration. Specifically,
> this condition:
> {code:java}
> if ("value".equals(field.getLocalName()) &&
> field.hasChildNodes()) {
> value = ((Text) field.getFirstChild()).getData();
> }
> {code}
> fails – {{field.hasChildNodes()}} returns {{false}} if the input is
> {{}} or {{}}. This prevents the configuration setting
> from being added to the {{XConfiguration}}.
> I suggest a fix like this:
> {code:java}
> if ("value".equals(field.getLocalName())) {
> if (field.hasChildNodes()) {
> value = ((Text) field.getFirstChild()).getData();
> }
> if (value == null) {
> value = "";
> }
> }
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (OOZIE-3377) [docs] Remaining 5.1.0 documentation changes
[
https://issues.apache.org/jira/browse/OOZIE-3377?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Andras Piros updated OOZIE-3377:
Attachment: OOZIE-3377.001.patch
> [docs] Remaining 5.1.0 documentation changes
>
>
> Key: OOZIE-3377
> URL: https://issues.apache.org/jira/browse/OOZIE-3377
> Project: Oozie
> Issue Type: Task
> Components: docs
>Affects Versions: 5.1.0
>Reporter: Andras Piros
>Assignee: Andras Piros
>Priority: Major
> Attachments: OOZIE-3377.001.patch
>
>
> Following documentation changes needed for 5.1.0:
> * link {{DG_FluentJobAPI.md}} from {{index.md}}
> * extract docs on Git action from {{WorkflowFunctionalSpecification.md}} to
> its own action extension docs: {{DG_GitActionExtension.md}}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (OOZIE-3376) [tests] TestGraphGenerator should assume JDK8 minor version at least 1.8.0_u40
[
https://issues.apache.org/jira/browse/OOZIE-3376?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16675102#comment-16675102
]
Andras Piros commented on OOZIE-3376:
-
[~gezapeti] we can't use JUnit4's {{Assume.assume}} in {{@Before}} or
{{@BeforeClass}} effectively since we're using {{JUnit38ClassRunner}} that
throws an {{AssumptionViolatedException}} instead of passing by. That's why we
need 5x {{try-catch}} :(
Tested w/ various Oracle JDK8 builds ({{1.8.0_u31}}, {{1.8.0_u102}}) so far.
Will test w/ OpenJDK 8 builds, as well.
> [tests] TestGraphGenerator should assume JDK8 minor version at least 1.8.0_u40
> --
>
> Key: OOZIE-3376
> URL: https://issues.apache.org/jira/browse/OOZIE-3376
> Project: Oozie
> Issue Type: Bug
> Components: docs, tests
>Affects Versions: 5.1.0
>Reporter: Andras Piros
>Assignee: Andras Piros
>Priority: Major
> Attachments: OOZIE-3376.001.patch
>
>
> {{GraphGenerator}} uses {{guru.nidi:graphviz-java:0.7.0}} that assumes JDK8
> minor version [at least
> 1.8.0_u40|https://github.com/nidi3/graphviz-java/commit/b7cf5761f97f1491d3bdc65367ec00e38d66291d].
> Goal is to make the same assumption inside {{TestGraphGenerator}} and
> document behavior within {{WebServicesAPI.md}}.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (OOZIE-3376) [tests] TestGraphGenerator should assume JDK8 minor version at least 1.8.0_u40
[
https://issues.apache.org/jira/browse/OOZIE-3376?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16675157#comment-16675157
]
Andras Piros commented on OOZIE-3376:
-
Unfortunately, no. Even in that case an {{AssumptionViolatedException}} is
thrown instead of passing by.
> [tests] TestGraphGenerator should assume JDK8 minor version at least 1.8.0_u40
> --
>
> Key: OOZIE-3376
> URL: https://issues.apache.org/jira/browse/OOZIE-3376
> Project: Oozie
> Issue Type: Bug
> Components: docs, tests
>Affects Versions: 5.1.0
>Reporter: Andras Piros
>Assignee: Andras Piros
>Priority: Major
> Attachments: OOZIE-3376.001.patch
>
>
> {{GraphGenerator}} uses {{guru.nidi:graphviz-java:0.7.0}} that assumes JDK8
> minor version [at least
> 1.8.0_u40|https://github.com/nidi3/graphviz-java/commit/b7cf5761f97f1491d3bdc65367ec00e38d66291d].
> Goal is to make the same assumption inside {{TestGraphGenerator}} and
> document behavior within {{WebServicesAPI.md}}.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (OOZIE-3376) [tests] TestGraphGenerator should assume JDK8 minor version at least 1.8.0_u40
[
https://issues.apache.org/jira/browse/OOZIE-3376?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16675331#comment-16675331
]
Andras Piros commented on OOZIE-3376:
-
Thanks for the review [~gezapeti]!
Using current OpenJDK version {{1.8.0_u191}} {{TestGraphGenerator}} passes:
{noformat}
$ java -version
openjdk version "1.8.0_191"
OpenJDK Runtime Environment (build 1.8.0_191-b12)
OpenJDK 64-Bit Server VM (build 25.191-b12, mixed mode)
$ mvn test -Dtest=TestGraphGenerator
...
[INFO] ---
[INFO] T E S T S
[INFO] ---
[INFO] Running org.apache.oozie.util.graph.TestGraphGenerator
[INFO] Tests run: 5, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 31.445 s
- in org.apache.oozie.util.graph.TestGraphGenerator
[INFO]
[INFO] Results:
[INFO]
[INFO] Tests run: 5, Failures: 0, Errors: 0, Skipped: 0
...{noformat}
> [tests] TestGraphGenerator should assume JDK8 minor version at least 1.8.0_u40
> --
>
> Key: OOZIE-3376
> URL: https://issues.apache.org/jira/browse/OOZIE-3376
> Project: Oozie
> Issue Type: Bug
> Components: docs, tests
>Affects Versions: 5.1.0
>Reporter: Andras Piros
>Assignee: Andras Piros
>Priority: Major
> Attachments: OOZIE-3376.001.patch
>
>
> {{GraphGenerator}} uses {{guru.nidi:graphviz-java:0.7.0}} that assumes JDK8
> minor version [at least
> 1.8.0_u40|https://github.com/nidi3/graphviz-java/commit/b7cf5761f97f1491d3bdc65367ec00e38d66291d].
> Goal is to make the same assumption inside {{TestGraphGenerator}} and
> document behavior within {{WebServicesAPI.md}}.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Comment Edited] (OOZIE-3376) [tests] TestGraphGenerator should assume JDK8 minor version at least 1.8.0_u40
[
https://issues.apache.org/jira/browse/OOZIE-3376?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16675331#comment-16675331
]
Andras Piros edited comment on OOZIE-3376 at 11/5/18 3:57 PM:
--
Thanks for the review [~gezapeti]!
Using current OpenJDK version {{1.8.0_191-b12}} {{TestGraphGenerator}} passes:
{noformat}
$ java -version
openjdk version "1.8.0_191"
OpenJDK Runtime Environment (build 1.8.0_191-b12)
OpenJDK 64-Bit Server VM (build 25.191-b12, mixed mode)
$ mvn test -Dtest=TestGraphGenerator
...
[INFO] ---
[INFO] T E S T S
[INFO] ---
[INFO] Running org.apache.oozie.util.graph.TestGraphGenerator
[INFO] Tests run: 5, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 31.445 s
- in org.apache.oozie.util.graph.TestGraphGenerator
[INFO]
[INFO] Results:
[INFO]
[INFO] Tests run: 5, Failures: 0, Errors: 0, Skipped: 0
...{noformat}
was (Author: andras.piros):
Thanks for the review [~gezapeti]!
Using current OpenJDK version {{1.8.0_u191}} {{TestGraphGenerator}} passes:
{noformat}
$ java -version
openjdk version "1.8.0_191"
OpenJDK Runtime Environment (build 1.8.0_191-b12)
OpenJDK 64-Bit Server VM (build 25.191-b12, mixed mode)
$ mvn test -Dtest=TestGraphGenerator
...
[INFO] ---
[INFO] T E S T S
[INFO] ---
[INFO] Running org.apache.oozie.util.graph.TestGraphGenerator
[INFO] Tests run: 5, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 31.445 s
- in org.apache.oozie.util.graph.TestGraphGenerator
[INFO]
[INFO] Results:
[INFO]
[INFO] Tests run: 5, Failures: 0, Errors: 0, Skipped: 0
...{noformat}
> [tests] TestGraphGenerator should assume JDK8 minor version at least 1.8.0_u40
> --
>
> Key: OOZIE-3376
> URL: https://issues.apache.org/jira/browse/OOZIE-3376
> Project: Oozie
> Issue Type: Bug
> Components: docs, tests
>Affects Versions: 5.1.0
>Reporter: Andras Piros
>Assignee: Andras Piros
>Priority: Major
> Attachments: OOZIE-3376.001.patch
>
>
> {{GraphGenerator}} uses {{guru.nidi:graphviz-java:0.7.0}} that assumes JDK8
> minor version [at least
> 1.8.0_u40|https://github.com/nidi3/graphviz-java/commit/b7cf5761f97f1491d3bdc65367ec00e38d66291d].
> Goal is to make the same assumption inside {{TestGraphGenerator}} and
> document behavior within {{WebServicesAPI.md}}.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (OOZIE-3376) [tests] TestGraphGenerator should assume JDK8 minor version at least 1.8.0_u40
[
https://issues.apache.org/jira/browse/OOZIE-3376?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16675355#comment-16675355
]
Andras Piros commented on OOZIE-3376:
-
Committed to {{master}} and cherry-picked to {{branch-5.1}}.
> [tests] TestGraphGenerator should assume JDK8 minor version at least 1.8.0_u40
> --
>
> Key: OOZIE-3376
> URL: https://issues.apache.org/jira/browse/OOZIE-3376
> Project: Oozie
> Issue Type: Bug
> Components: docs, tests
>Affects Versions: 5.1.0
>Reporter: Andras Piros
>Assignee: Andras Piros
>Priority: Major
> Fix For: 5.1.0
>
> Attachments: OOZIE-3376.001.patch
>
>
> {{GraphGenerator}} uses {{guru.nidi:graphviz-java:0.7.0}} that assumes JDK8
> minor version [at least
> 1.8.0_u40|https://github.com/nidi3/graphviz-java/commit/b7cf5761f97f1491d3bdc65367ec00e38d66291d].
> Goal is to make the same assumption inside {{TestGraphGenerator}} and
> document behavior within {{WebServicesAPI.md}}.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (OOZIE-3375) Can't use empty in coordinator
[
https://issues.apache.org/jira/browse/OOZIE-3375?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16675449#comment-16675449
]
Andras Piros commented on OOZIE-3375:
-
Thanks for the contribution [~jtolar]! The test failures in general show that
the change would break existing functionality. Here, having an empty string in
the {{oozie-site.xml}} means attribute name / value pair should not be filled.
This behavior should not be changed as being depended on in existing code.
In particular:
# {{TestSubmitXCommand.testProtoConfStorage}}: the test case has to remain the
same, but
{{[WorkflowAppService.createProtoActionConf|https://github.com/apache/oozie/blob/master/core/src/main/java/org/apache/oozie/service/WorkflowAppService.java#L232]}}
has to be extended in order only set {{APP_LIB_PATH_LIST}} of non-empty
elements of {{filePaths}}
# {{TestConfigurationService.testOozieConfig}}: the test case has to remain
the same, and I don't see any reasons why we should change configuration parsing
# {{TestSLAAlertXCommand}}: test cases have to remain the same
I'd suggest to modify the coordinator definition to not use a parameter like
{{$\{test_param}}} but the empty string under
{{//coordinator-app/action/workflow/configuration/property/value}}.
> Can't use empty in coordinator
> ---
>
> Key: OOZIE-3375
> URL: https://issues.apache.org/jira/browse/OOZIE-3375
> Project: Oozie
> Issue Type: Bug
>Reporter: Jacob Tolar
>Assignee: Jacob Tolar
>Priority: Major
> Attachments: OOZIE-3375-001.patch
>
>
> If I set a property to empty string in the {{}} block of my
> coordinator and later use it in the {{}} block Oozie throws an error.
> That is, this code fails:
> {code:java}
> end="2018-10-01T00:04Z" timezone="UTC" xmlns="uri:oozie:coordinator:0.5">
>
>
> test_param
>
>
>
>
> 5
> 1
>
>
>
> /workflow.xml
>
>
> renamed_param
> ${test_param}
>
>
>
>
>
> {code}
> The error is like this:
> {code:java}
> org.apache.oozie.command.CommandException: E1021: Coord Action Input Check
> Error: E1004: Expression language evaluation error, Unable to evaluate
> :${test_param}:
> ...
> Caused by: javax.servlet.jsp.el.ELException: variable [test_param] cannot be
> resolved
> {code}
> What happens: The coordinator submits successfully. When the first action
> materializes in
> [CoordMaterializeTransitionXCommand|https://github.com/apache/oozie/blob/65936460e263f9076bb552190be85396c2cc6d33/core/src/main/java/org/apache/oozie/command/coord/CoordMaterializeTransitionXCommand.java#L373-L379],
> the coordinator conf is parsed into an Oozie {{XConfiguration}} object.
> In
> [XConfiguration.processNodes|https://github.com/apache/oozie/blob/65936460e263f9076bb552190be85396c2cc6d33/core/src/main/java/org/apache/oozie/util/XConfiguration.java#L313-L354],
> present-but-empty values are not added to the configuration. Specifically,
> this condition:
> {code:java}
> if ("value".equals(field.getLocalName()) &&
> field.hasChildNodes()) {
> value = ((Text) field.getFirstChild()).getData();
> }
> {code}
> fails – {{field.hasChildNodes()}} returns {{false}} if the input is
> {{}} or {{}}. This prevents the configuration setting
> from being added to the {{XConfiguration}}.
> I suggest a fix like this:
> {code:java}
> if ("value".equals(field.getLocalName())) {
> if (field.hasChildNodes()) {
> value = ((Text) field.getFirstChild()).getData();
> }
> if (value == null) {
> value = "";
> }
> }
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (OOZIE-3375) Can't use empty in coordinator
[
https://issues.apache.org/jira/browse/OOZIE-3375?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16675630#comment-16675630
]
Andras Piros commented on OOZIE-3375:
-
I see your point [~jtolar]. What about an additive change: defining an EL
expression that would mean populating the parameter name defined as part of
{{}} level but at the {{}} level substitute it with an
empty string? Actually, did you try using [the existing EL function {{$\{trim('
')}}}|https://oozie.apache.org/docs/5.0.0/WorkflowFunctionalSpec.html#a4.2.2_Basic_EL_Functions]?
> Can't use empty in coordinator
> ---
>
> Key: OOZIE-3375
> URL: https://issues.apache.org/jira/browse/OOZIE-3375
> Project: Oozie
> Issue Type: Bug
>Reporter: Jacob Tolar
>Assignee: Jacob Tolar
>Priority: Major
> Attachments: OOZIE-3375-001.patch
>
>
> If I set a property to empty string in the {{}} block of my
> coordinator and later use it in the {{}} block Oozie throws an error.
> That is, this code fails:
> {code:java}
> end="2018-10-01T00:04Z" timezone="UTC" xmlns="uri:oozie:coordinator:0.5">
>
>
> test_param
>
>
>
>
> 5
> 1
>
>
>
> /workflow.xml
>
>
> renamed_param
> ${test_param}
>
>
>
>
>
> {code}
> The error is like this:
> {code:java}
> org.apache.oozie.command.CommandException: E1021: Coord Action Input Check
> Error: E1004: Expression language evaluation error, Unable to evaluate
> :${test_param}:
> ...
> Caused by: javax.servlet.jsp.el.ELException: variable [test_param] cannot be
> resolved
> {code}
> What happens: The coordinator submits successfully. When the first action
> materializes in
> [CoordMaterializeTransitionXCommand|https://github.com/apache/oozie/blob/65936460e263f9076bb552190be85396c2cc6d33/core/src/main/java/org/apache/oozie/command/coord/CoordMaterializeTransitionXCommand.java#L373-L379],
> the coordinator conf is parsed into an Oozie {{XConfiguration}} object.
> In
> [XConfiguration.processNodes|https://github.com/apache/oozie/blob/65936460e263f9076bb552190be85396c2cc6d33/core/src/main/java/org/apache/oozie/util/XConfiguration.java#L313-L354],
> present-but-empty values are not added to the configuration. Specifically,
> this condition:
> {code:java}
> if ("value".equals(field.getLocalName()) &&
> field.hasChildNodes()) {
> value = ((Text) field.getFirstChild()).getData();
> }
> {code}
> fails – {{field.hasChildNodes()}} returns {{false}} if the input is
> {{}} or {{}}. This prevents the configuration setting
> from being added to the {{XConfiguration}}.
> I suggest a fix like this:
> {code:java}
> if ("value".equals(field.getLocalName())) {
> if (field.hasChildNodes()) {
> value = ((Text) field.getFirstChild()).getData();
> }
> if (value == null) {
> value = "";
> }
> }
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (OOZIE-3378) Coordinator action's status is SUBMITTED after E1003 error
[
https://issues.apache.org/jira/browse/OOZIE-3378?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16676394#comment-16676394
]
Andras Piros commented on OOZIE-3378:
-
Thanks for the contribution [~asalamon74]! +1 (pending Jenkins for patch 002)
> Coordinator action's status is SUBMITTED after E1003 error
> --
>
> Key: OOZIE-3378
> URL: https://issues.apache.org/jira/browse/OOZIE-3378
> Project: Oozie
> Issue Type: Improvement
>Affects Versions: trunk
>Reporter: Andras Salamon
>Assignee: Andras Salamon
>Priority: Major
> Fix For: 5.2.0
>
> Attachments: OOZIE-3378-01.patch, OOZIE-3378-02.patch
>
>
> If I try to run a coordinator job which gives an {{E1003}} error code, the
> coordinator's status is not changed to {{FAILED}}.
> I was using the following {{coordinator.xml}}
> {noformat}
> end="${end}" timezone="UTC"
> xmlns="uri:oozie:coordinator:0.2">
>
>
> ${workflowAppUri}
>
>
> resourceManager
> ${resourceManager}
>
>
> nameNode
> ${nameNode}
>
>
> queueName
> ${queueName}
>
>
> user.name
> admin
>
>
>
>
>
> {noformat}
> The status of the coordinator job is {{RUNNING}}
> {noformat}
> $ oozie job -oozie http://localhost:11000/oozie -info
> 000-181105104843399-oozie-andr-C
> Job ID : 000-181105104843399-oozie-andr-C
>
> Job Name: cron-coord
> App Path:
> hdfs://localhost:9000/user/andrassalamon/examples/apps/cron-schedule
> Status : RUNNING
> Start Time : 2010-01-01 00:00 GMT
> End Time: 2010-01-01 01:00 GMT
> Pause Time : -
> Concurrency : 1
>
> ID StatusExt ID
> Err Code Created Nominal Time
> 000-181105104843399-oozie-andr-C@1 SUBMITTED -
> - 2018-11-05 09:54 GMT 2010-01-01 00:00 GMT
>
> 000-181105104843399-oozie-andr-C@2 READY -
> - 2018-11-05 09:54 GMT 2010-01-01 00:10 GMT
>
> 000-181105104843399-oozie-andr-C@3 READY -
> - 2018-11-05 09:54 GMT 2010-01-01 00:20 GMT
>
> 000-181105104843399-oozie-andr-C@4 READY -
> - 2018-11-05 09:54 GMT 2010-01-01 00:30 GMT
>
> 000-181105104843399-oozie-andr-C@5 READY -
> - 2018-11-05 09:54 GMT 2010-01-01 00:40 GMT
>
> 000-181105104843399-oozie-andr-C@6 READY -
> - 2018-11-05 09:54 GMT 2010-01-01 00:50 GMT
>
> {noformat}
> The status of the first coordinator action is {{SUBMITTED}}
> {noformat}
> $ ./distro/target/oozie-5.2.0-SNAPSHOT-distro/oozie-5.2.0-SNAPSHOT/bin/oozie
> job -oozie http://localhost:11000/oozie -info
> 000-181105104843399-oozie-andr-C@1
> ID : 000-181105104843399-oozie-andr-C@1
>
> Action Number: 1
> Console URL : -
> Error Code : -
> Error Message: -
> External ID : -
> External Status : -
> Job ID : 000-181105104843399-oozie-andr-C
> Tracker URI : -
> Created : 2018-11-05 09:54 GMT
[jira] [Assigned] (OOZIE-3379) Auth token cache file name should include Oozie URL
[ https://issues.apache.org/jira/browse/OOZIE-3379?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andras Piros reassigned OOZIE-3379: --- Assignee: ZhangJunfan > Auth token cache file name should include Oozie URL > --- > > Key: OOZIE-3379 > URL: https://issues.apache.org/jira/browse/OOZIE-3379 > Project: Oozie > Issue Type: Bug > Components: client >Affects Versions: 5.0.0 >Reporter: ZhangJunfan >Assignee: ZhangJunfan >Priority: Major > Attachments: oozie-3379-1.patch > > > We have a program that uses the oozie client, but when the client connects to > multiple clusters, > the authOozieClient class frequently requests the kdc server because the > authentication token cache file is invalid. > This will cause subsequent requests in our program to be blocked, resulting > in unstable services. > So, oozie client's auth token cache file name should include Oozie URL. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (OOZIE-3379) Auth token cache file name should include Oozie URL
[
https://issues.apache.org/jira/browse/OOZIE-3379?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16676463#comment-16676463
]
Andras Piros commented on OOZIE-3379:
-
Thanks for the contribution [~zuston]!
A couple of comments:
# file length should be trimmed to 255 chars
# if there is already a {{File}} with the same name, at least {{LOG.warn()}}
# new unit test cases covering the new functionality should be created,
explicitly testing for multiple {{AuthOozieClient}} instances in one JVM
# instead of {{public static File}}, the auth token cache should be
{{@VisibleForTesting final File}} that could be asserted in a unit test. In a
JVM we can have multiple {{AuthOozieClient}} instances w/ multiple Oozie URLs,
that's also why we should not use {{static}} here
> Auth token cache file name should include Oozie URL
> ---
>
> Key: OOZIE-3379
> URL: https://issues.apache.org/jira/browse/OOZIE-3379
> Project: Oozie
> Issue Type: Bug
> Components: client
>Affects Versions: 5.0.0
>Reporter: ZhangJunfan
>Assignee: ZhangJunfan
>Priority: Major
> Attachments: oozie-3379-1.patch
>
>
> We have a program that uses the oozie client, but when the client connects to
> multiple clusters,
> the authOozieClient class frequently requests the kdc server because the
> authentication token cache file is invalid.
> This will cause subsequent requests in our program to be blocked, resulting
> in unstable services.
> So, oozie client's auth token cache file name should include Oozie URL.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (OOZIE-3378) [core] Coordinator action's status is SUBMITTED after E1003 error
[
https://issues.apache.org/jira/browse/OOZIE-3378?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Andras Piros updated OOZIE-3378:
Summary: [core] Coordinator action's status is SUBMITTED after E1003 error
(was: Coordinator action's status is SUBMITTED after E1003 error)
> [core] Coordinator action's status is SUBMITTED after E1003 error
> -
>
> Key: OOZIE-3378
> URL: https://issues.apache.org/jira/browse/OOZIE-3378
> Project: Oozie
> Issue Type: Improvement
> Components: core
>Affects Versions: trunk
>Reporter: Andras Salamon
>Assignee: Andras Salamon
>Priority: Major
> Fix For: 5.2.0
>
> Attachments: OOZIE-3378-01.patch, OOZIE-3378-02.patch
>
>
> If I try to run a coordinator job which gives an {{E1003}} error code, the
> coordinator's status is not changed to {{FAILED}}.
> I was using the following {{coordinator.xml}}
> {noformat}
> end="${end}" timezone="UTC"
> xmlns="uri:oozie:coordinator:0.2">
>
>
> ${workflowAppUri}
>
>
> resourceManager
> ${resourceManager}
>
>
> nameNode
> ${nameNode}
>
>
> queueName
> ${queueName}
>
>
> user.name
> admin
>
>
>
>
>
> {noformat}
> The status of the coordinator job is {{RUNNING}}
> {noformat}
> $ oozie job -oozie http://localhost:11000/oozie -info
> 000-181105104843399-oozie-andr-C
> Job ID : 000-181105104843399-oozie-andr-C
>
> Job Name: cron-coord
> App Path:
> hdfs://localhost:9000/user/andrassalamon/examples/apps/cron-schedule
> Status : RUNNING
> Start Time : 2010-01-01 00:00 GMT
> End Time: 2010-01-01 01:00 GMT
> Pause Time : -
> Concurrency : 1
>
> ID StatusExt ID
> Err Code Created Nominal Time
> 000-181105104843399-oozie-andr-C@1 SUBMITTED -
> - 2018-11-05 09:54 GMT 2010-01-01 00:00 GMT
>
> 000-181105104843399-oozie-andr-C@2 READY -
> - 2018-11-05 09:54 GMT 2010-01-01 00:10 GMT
>
> 000-181105104843399-oozie-andr-C@3 READY -
> - 2018-11-05 09:54 GMT 2010-01-01 00:20 GMT
>
> 000-181105104843399-oozie-andr-C@4 READY -
> - 2018-11-05 09:54 GMT 2010-01-01 00:30 GMT
>
> 000-181105104843399-oozie-andr-C@5 READY -
> - 2018-11-05 09:54 GMT 2010-01-01 00:40 GMT
>
> 000-181105104843399-oozie-andr-C@6 READY -
> - 2018-11-05 09:54 GMT 2010-01-01 00:50 GMT
>
> {noformat}
> The status of the first coordinator action is {{SUBMITTED}}
> {noformat}
> $ ./distro/target/oozie-5.2.0-SNAPSHOT-distro/oozie-5.2.0-SNAPSHOT/bin/oozie
> job -oozie http://localhost:11000/oozie -info
> 000-181105104843399-oozie-andr-C@1
> ID : 000-181105104843399-oozie-andr-C@1
>
> Action Number: 1
> Console URL : -
> Error Code : -
> Error Message: -
> External ID : -
> External Status : -
> Job ID : 000-181105104843399-oozie-andr-C
> Tracker URI
[jira] [Updated] (OOZIE-3378) Coordinator action's status is SUBMITTED after E1003 error
[
https://issues.apache.org/jira/browse/OOZIE-3378?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Andras Piros updated OOZIE-3378:
Component/s: core
> Coordinator action's status is SUBMITTED after E1003 error
> --
>
> Key: OOZIE-3378
> URL: https://issues.apache.org/jira/browse/OOZIE-3378
> Project: Oozie
> Issue Type: Improvement
> Components: core
>Affects Versions: trunk
>Reporter: Andras Salamon
>Assignee: Andras Salamon
>Priority: Major
> Fix For: 5.2.0
>
> Attachments: OOZIE-3378-01.patch, OOZIE-3378-02.patch
>
>
> If I try to run a coordinator job which gives an {{E1003}} error code, the
> coordinator's status is not changed to {{FAILED}}.
> I was using the following {{coordinator.xml}}
> {noformat}
> end="${end}" timezone="UTC"
> xmlns="uri:oozie:coordinator:0.2">
>
>
> ${workflowAppUri}
>
>
> resourceManager
> ${resourceManager}
>
>
> nameNode
> ${nameNode}
>
>
> queueName
> ${queueName}
>
>
> user.name
> admin
>
>
>
>
>
> {noformat}
> The status of the coordinator job is {{RUNNING}}
> {noformat}
> $ oozie job -oozie http://localhost:11000/oozie -info
> 000-181105104843399-oozie-andr-C
> Job ID : 000-181105104843399-oozie-andr-C
>
> Job Name: cron-coord
> App Path:
> hdfs://localhost:9000/user/andrassalamon/examples/apps/cron-schedule
> Status : RUNNING
> Start Time : 2010-01-01 00:00 GMT
> End Time: 2010-01-01 01:00 GMT
> Pause Time : -
> Concurrency : 1
>
> ID StatusExt ID
> Err Code Created Nominal Time
> 000-181105104843399-oozie-andr-C@1 SUBMITTED -
> - 2018-11-05 09:54 GMT 2010-01-01 00:00 GMT
>
> 000-181105104843399-oozie-andr-C@2 READY -
> - 2018-11-05 09:54 GMT 2010-01-01 00:10 GMT
>
> 000-181105104843399-oozie-andr-C@3 READY -
> - 2018-11-05 09:54 GMT 2010-01-01 00:20 GMT
>
> 000-181105104843399-oozie-andr-C@4 READY -
> - 2018-11-05 09:54 GMT 2010-01-01 00:30 GMT
>
> 000-181105104843399-oozie-andr-C@5 READY -
> - 2018-11-05 09:54 GMT 2010-01-01 00:40 GMT
>
> 000-181105104843399-oozie-andr-C@6 READY -
> - 2018-11-05 09:54 GMT 2010-01-01 00:50 GMT
>
> {noformat}
> The status of the first coordinator action is {{SUBMITTED}}
> {noformat}
> $ ./distro/target/oozie-5.2.0-SNAPSHOT-distro/oozie-5.2.0-SNAPSHOT/bin/oozie
> job -oozie http://localhost:11000/oozie -info
> 000-181105104843399-oozie-andr-C@1
> ID : 000-181105104843399-oozie-andr-C@1
>
> Action Number: 1
> Console URL : -
> Error Code : -
> Error Message: -
> External ID : -
> External Status : -
> Job ID : 000-181105104843399-oozie-andr-C
> Tracker URI : -
> Created : 2018-11-05 09:54 GMT
> Nominal Time : 2010-01-01 00:00 GMT
> Status : SUBMITTED
> Las
[jira] [Commented] (OOZIE-3378) [core] Coordinator action's status is SUBMITTED after E1003 error
[
https://issues.apache.org/jira/browse/OOZIE-3378?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16676657#comment-16676657
]
Andras Piros commented on OOZIE-3378:
-
Committed to {{master}}.
> [core] Coordinator action's status is SUBMITTED after E1003 error
> -
>
> Key: OOZIE-3378
> URL: https://issues.apache.org/jira/browse/OOZIE-3378
> Project: Oozie
> Issue Type: Improvement
> Components: core
>Affects Versions: trunk
>Reporter: Andras Salamon
>Assignee: Andras Salamon
>Priority: Major
> Fix For: trunk, 5.2.0
>
> Attachments: OOZIE-3378-01.patch, OOZIE-3378-02.patch
>
>
> If I try to run a coordinator job which gives an {{E1003}} error code, the
> coordinator's status is not changed to {{FAILED}}.
> I was using the following {{coordinator.xml}}
> {noformat}
> end="${end}" timezone="UTC"
> xmlns="uri:oozie:coordinator:0.2">
>
>
> ${workflowAppUri}
>
>
> resourceManager
> ${resourceManager}
>
>
> nameNode
> ${nameNode}
>
>
> queueName
> ${queueName}
>
>
> user.name
> admin
>
>
>
>
>
> {noformat}
> The status of the coordinator job is {{RUNNING}}
> {noformat}
> $ oozie job -oozie http://localhost:11000/oozie -info
> 000-181105104843399-oozie-andr-C
> Job ID : 000-181105104843399-oozie-andr-C
>
> Job Name: cron-coord
> App Path:
> hdfs://localhost:9000/user/andrassalamon/examples/apps/cron-schedule
> Status : RUNNING
> Start Time : 2010-01-01 00:00 GMT
> End Time: 2010-01-01 01:00 GMT
> Pause Time : -
> Concurrency : 1
>
> ID StatusExt ID
> Err Code Created Nominal Time
> 000-181105104843399-oozie-andr-C@1 SUBMITTED -
> - 2018-11-05 09:54 GMT 2010-01-01 00:00 GMT
>
> 000-181105104843399-oozie-andr-C@2 READY -
> - 2018-11-05 09:54 GMT 2010-01-01 00:10 GMT
>
> 000-181105104843399-oozie-andr-C@3 READY -
> - 2018-11-05 09:54 GMT 2010-01-01 00:20 GMT
>
> 000-181105104843399-oozie-andr-C@4 READY -
> - 2018-11-05 09:54 GMT 2010-01-01 00:30 GMT
>
> 000-181105104843399-oozie-andr-C@5 READY -
> - 2018-11-05 09:54 GMT 2010-01-01 00:40 GMT
>
> 000-181105104843399-oozie-andr-C@6 READY -
> - 2018-11-05 09:54 GMT 2010-01-01 00:50 GMT
>
> {noformat}
> The status of the first coordinator action is {{SUBMITTED}}
> {noformat}
> $ ./distro/target/oozie-5.2.0-SNAPSHOT-distro/oozie-5.2.0-SNAPSHOT/bin/oozie
> job -oozie http://localhost:11000/oozie -info
> 000-181105104843399-oozie-andr-C@1
> ID : 000-181105104843399-oozie-andr-C@1
>
> Action Number: 1
> Console URL : -
> Error Code : -
> Error Message: -
> External ID : -
> External Status : -
> Job ID : 000-181105104843399-oozie-andr-C
> Tracker URI : -
> Created : 2018-11-05 09:54 GMT
> No
[jira] [Commented] (OOZIE-3379) Auth token cache file name should include Oozie URL
[
https://issues.apache.org/jira/browse/OOZIE-3379?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16679577#comment-16679577
]
Andras Piros commented on OOZIE-3379:
-
Thanks for the new patch [~zuston]!
Can you please take care of the pre-commit errors:
# address FindBugs issues
# address failing test
[*{{TestAuthFilterAuthOozieClient.testClientAuthTokenCache}}*|https://builds.apache.org/job/PreCommit-OOZIE-Build/908/testReport/org.apache.oozie.servlet/TestAuthFilterAuthOozieClient/testClientAuthTokenCache/]
> Auth token cache file name should include Oozie URL
> ---
>
> Key: OOZIE-3379
> URL: https://issues.apache.org/jira/browse/OOZIE-3379
> Project: Oozie
> Issue Type: Bug
> Components: client
>Affects Versions: 5.0.0
>Reporter: ZhangJunfan
>Assignee: ZhangJunfan
>Priority: Major
> Attachments: oozie-3379-1.patch, oozie-3379-2.patch,
> oozie-3379-3.patch
>
>
> We have a program that uses the oozie client, but when the client connects to
> multiple clusters,
> the authOozieClient class frequently requests the kdc server because the
> authentication token cache file is invalid.
> This will cause subsequent requests in our program to be blocked, resulting
> in unstable services.
> So, oozie client's auth token cache file name should include Oozie URL.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (OOZIE-3381) [coordinator] Enhance logging of CoordElFunctions.coord_latestRange_sync
[
https://issues.apache.org/jira/browse/OOZIE-3381?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Andras Piros updated OOZIE-3381:
Description:
When using [{{$\{coord:latest\(n\)}}} coordinator EL
function|https://oozie.apache.org/docs/5.0.0/CoordinatorFunctionalSpec.html#a6.6.6._coord:latestint_n_EL_Function_for_Synchronous_Datasets]
inside an input dataset dependency, it's often the case that more information
is needed how many HDFS URIs are being checked for each {{}}.
Right now we don't have this information. While debugging and fine tuning
parameters like {{dataset frequency}}, {{initial-instance}}, and {{data-in}}
{{instance}}, it would be very useful to know how many HDFS roundtrips are
issues by the current settings {{CoordELFunctions#coord_latestRange_sync()}}
having called {{DFSClient#exists()}}. We need appropriate logging there.
was:
When using [{{$\{coord:latest(n)}}} coordinator EL
function|https://oozie.apache.org/docs/5.0.0/CoordinatorFunctionalSpec.html#a6.6.6._coord:latestint_n_EL_Function_for_Synchronous_Datasets]
inside an input dataset dependency, it's often the case that more information
is needed how many HDFS URIs are being checked for each {{}}.
Right now we don't have this information. While debugging and fine tuning
parameters like {{dataset frequency}}, {{initial-instance}}, and {{data-in}}
{{instance}}, it would be very useful to know how many HDFS roundtrips are
issues by the current settings {{CoordELFunctions#coord_latestRange_sync()}}
having called {{DFSClient#exists()}}. We need appropriate logging there.
> [coordinator] Enhance logging of CoordElFunctions.coord_latestRange_sync
>
>
> Key: OOZIE-3381
> URL: https://issues.apache.org/jira/browse/OOZIE-3381
> Project: Oozie
> Issue Type: Task
> Components: coordinator
>Affects Versions: 5.1.0
>Reporter: Andras Piros
>Assignee: Andras Piros
>Priority: Major
>
> When using [{{$\{coord:latest\(n\)}}} coordinator EL
> function|https://oozie.apache.org/docs/5.0.0/CoordinatorFunctionalSpec.html#a6.6.6._coord:latestint_n_EL_Function_for_Synchronous_Datasets]
> inside an input dataset dependency, it's often the case that more
> information is needed how many HDFS URIs are being checked for each
> {{}}.
> Right now we don't have this information. While debugging and fine tuning
> parameters like {{dataset frequency}}, {{initial-instance}}, and {{data-in}}
> {{instance}}, it would be very useful to know how many HDFS roundtrips are
> issues by the current settings {{CoordELFunctions#coord_latestRange_sync()}}
> having called {{DFSClient#exists()}}. We need appropriate logging there.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Created] (OOZIE-3381) [coordinator] Enhance logging of CoordElFunctions.coord_latestRange_sync
Andras Piros created OOZIE-3381:
---
Summary: [coordinator] Enhance logging of
CoordElFunctions.coord_latestRange_sync
Key: OOZIE-3381
URL: https://issues.apache.org/jira/browse/OOZIE-3381
Project: Oozie
Issue Type: Task
Components: coordinator
Affects Versions: 5.1.0
Reporter: Andras Piros
Assignee: Andras Piros
When using [{{$\{coord:latest(n)}}} coordinator EL
function|https://oozie.apache.org/docs/5.0.0/CoordinatorFunctionalSpec.html#a6.6.6._coord:latestint_n_EL_Function_for_Synchronous_Datasets]
inside an input dataset dependency, it's often the case that more information
is needed how many HDFS URIs are being checked for each {{}}.
Right now we don't have this information. While debugging and fine tuning
parameters like {{dataset frequency}}, {{initial-instance}}, and {{data-in}}
{{instance}}, it would be very useful to know how many HDFS roundtrips are
issues by the current settings {{CoordELFunctions#coord_latestRange_sync()}}
having called {{DFSClient#exists()}}. We need appropriate logging there.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (OOZIE-3381) [coordinator] Enhance logging of CoordElFunctions
[
https://issues.apache.org/jira/browse/OOZIE-3381?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Andras Piros updated OOZIE-3381:
Description:
When using [{{$\{coord:latest\(n\)}}} coordinator EL
function|https://oozie.apache.org/docs/5.0.0/CoordinatorFunctionalSpec.html#a6.6.6._coord:latestint_n_EL_Function_for_Synchronous_Datasets]
inside an input dataset dependency, it's often the case that more information
is needed how many HDFS URIs are being checked for each {{}}.
Right now we don't have this information. While debugging and fine tuning
parameters like {{dataset frequency}}, {{initial-instance}}, and {{data-in}}
{{instance}}, it would be very useful to know how many HDFS roundtrips are
issues by the current settings {{CoordELFunctions#coord_latestRange_sync()}}
and {{CoordELFunctions#coord_futureRange_sync()}} having called
{{DFSClient#exists()}}. We need appropriate logging there.
was:
When using [{{$\{coord:latest\(n\)}}} coordinator EL
function|https://oozie.apache.org/docs/5.0.0/CoordinatorFunctionalSpec.html#a6.6.6._coord:latestint_n_EL_Function_for_Synchronous_Datasets]
inside an input dataset dependency, it's often the case that more information
is needed how many HDFS URIs are being checked for each {{}}.
Right now we don't have this information. While debugging and fine tuning
parameters like {{dataset frequency}}, {{initial-instance}}, and {{data-in}}
{{instance}}, it would be very useful to know how many HDFS roundtrips are
issues by the current settings {{CoordELFunctions#coord_latestRange_sync()}}
having called {{DFSClient#exists()}}. We need appropriate logging there.
> [coordinator] Enhance logging of CoordElFunctions
> -
>
> Key: OOZIE-3381
> URL: https://issues.apache.org/jira/browse/OOZIE-3381
> Project: Oozie
> Issue Type: Task
> Components: coordinator
>Affects Versions: 5.1.0
>Reporter: Andras Piros
>Assignee: Andras Piros
>Priority: Major
>
> When using [{{$\{coord:latest\(n\)}}} coordinator EL
> function|https://oozie.apache.org/docs/5.0.0/CoordinatorFunctionalSpec.html#a6.6.6._coord:latestint_n_EL_Function_for_Synchronous_Datasets]
> inside an input dataset dependency, it's often the case that more
> information is needed how many HDFS URIs are being checked for each
> {{}}.
> Right now we don't have this information. While debugging and fine tuning
> parameters like {{dataset frequency}}, {{initial-instance}}, and {{data-in}}
> {{instance}}, it would be very useful to know how many HDFS roundtrips are
> issues by the current settings {{CoordELFunctions#coord_latestRange_sync()}}
> and {{CoordELFunctions#coord_futureRange_sync()}} having called
> {{DFSClient#exists()}}. We need appropriate logging there.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (OOZIE-3381) [coordinator] Enhance logging of CoordElFunctions
[
https://issues.apache.org/jira/browse/OOZIE-3381?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Andras Piros updated OOZIE-3381:
Summary: [coordinator] Enhance logging of CoordElFunctions (was:
[coordinator] Enhance logging of CoordElFunctions.coord_latestRange_sync)
> [coordinator] Enhance logging of CoordElFunctions
> -
>
> Key: OOZIE-3381
> URL: https://issues.apache.org/jira/browse/OOZIE-3381
> Project: Oozie
> Issue Type: Task
> Components: coordinator
>Affects Versions: 5.1.0
>Reporter: Andras Piros
>Assignee: Andras Piros
>Priority: Major
>
> When using [{{$\{coord:latest\(n\)}}} coordinator EL
> function|https://oozie.apache.org/docs/5.0.0/CoordinatorFunctionalSpec.html#a6.6.6._coord:latestint_n_EL_Function_for_Synchronous_Datasets]
> inside an input dataset dependency, it's often the case that more
> information is needed how many HDFS URIs are being checked for each
> {{}}.
> Right now we don't have this information. While debugging and fine tuning
> parameters like {{dataset frequency}}, {{initial-instance}}, and {{data-in}}
> {{instance}}, it would be very useful to know how many HDFS roundtrips are
> issues by the current settings {{CoordELFunctions#coord_latestRange_sync()}}
> having called {{DFSClient#exists()}}. We need appropriate logging there.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (OOZIE-3338) Remove SVN references
[
https://issues.apache.org/jira/browse/OOZIE-3338?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16683449#comment-16683449
]
Andras Piros commented on OOZIE-3338:
-
Thanks for the contribution [~asalamon74]! +1
> Remove SVN references
> -
>
> Key: OOZIE-3338
> URL: https://issues.apache.org/jira/browse/OOZIE-3338
> Project: Oozie
> Issue Type: Improvement
> Components: build, docs
>Affects Versions: trunk
>Reporter: Andras Salamon
>Assignee: Andras Salamon
>Priority: Minor
> Fix For: 5.2.0
>
> Attachments: OOZIE-3338-01.patch, OOZIE-3338-02.patch
>
>
> Oozie
> [documentation|https://github.com/apache/oozie/blob/master/docs/src/site/twiki/ENG_MiniOozie.twiki#L20-L26]
> still refers to the SVN server:
> {noformat}
> https://svn.apache.org/repos/asf/incubator/oozie/trunk
> {noformat}
> which is no longer available.
> The current address is the following:
> {noformat}
> https://svn.apache.org/repos/asf/oozie/
> {noformat}
> It only contains a README file and forwards to the git server.
> We should remove the SVN references from the file.
>
> It would also be useful to simplify
> [mkdistro.sh|https://github.com/apache/oozie/blob/master/bin/mkdistro.sh] and
> remove the SVN support.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (OOZIE-3338) [build] Remove SVN references
[
https://issues.apache.org/jira/browse/OOZIE-3338?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Andras Piros updated OOZIE-3338:
Summary: [build] Remove SVN references (was: Remove SVN references)
> [build] Remove SVN references
> -
>
> Key: OOZIE-3338
> URL: https://issues.apache.org/jira/browse/OOZIE-3338
> Project: Oozie
> Issue Type: Improvement
> Components: build, docs
>Affects Versions: trunk
>Reporter: Andras Salamon
>Assignee: Andras Salamon
>Priority: Minor
> Fix For: 5.2.0
>
> Attachments: OOZIE-3338-01.patch, OOZIE-3338-02.patch
>
>
> Oozie
> [documentation|https://github.com/apache/oozie/blob/master/docs/src/site/twiki/ENG_MiniOozie.twiki#L20-L26]
> still refers to the SVN server:
> {noformat}
> https://svn.apache.org/repos/asf/incubator/oozie/trunk
> {noformat}
> which is no longer available.
> The current address is the following:
> {noformat}
> https://svn.apache.org/repos/asf/oozie/
> {noformat}
> It only contains a README file and forwards to the git server.
> We should remove the SVN references from the file.
>
> It would also be useful to simplify
> [mkdistro.sh|https://github.com/apache/oozie/blob/master/bin/mkdistro.sh] and
> remove the SVN support.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (OOZIE-3381) [coordinator] Enhance logging of CoordElFunctions
[
https://issues.apache.org/jira/browse/OOZIE-3381?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Andras Piros updated OOZIE-3381:
Attachment: OOZIE-3381.001.patch
> [coordinator] Enhance logging of CoordElFunctions
> -
>
> Key: OOZIE-3381
> URL: https://issues.apache.org/jira/browse/OOZIE-3381
> Project: Oozie
> Issue Type: Task
> Components: coordinator
>Affects Versions: 5.1.0
>Reporter: Andras Piros
>Assignee: Andras Piros
>Priority: Major
> Attachments: OOZIE-3381.001.patch
>
>
> When using [{{$\{coord:latest\(n\)}}} coordinator EL
> function|https://oozie.apache.org/docs/5.0.0/CoordinatorFunctionalSpec.html#a6.6.6._coord:latestint_n_EL_Function_for_Synchronous_Datasets]
> inside an input dataset dependency, it's often the case that more
> information is needed how many HDFS URIs are being checked for each
> {{}}.
> Right now we don't have this information. While debugging and fine tuning
> parameters like {{dataset frequency}}, {{initial-instance}}, and {{data-in}}
> {{instance}}, it would be very useful to know how many HDFS roundtrips are
> issues by the current settings {{CoordELFunctions#coord_latestRange_sync()}}
> and {{CoordELFunctions#coord_futureRange_sync()}} having called
> {{DFSClient#exists()}}. We need appropriate logging there.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (OOZIE-3384) [tests] TestWorkflowActionRetryInfoXCommand#testRetryConsoleUrlForked() is flaky
[
https://issues.apache.org/jira/browse/OOZIE-3384?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Andras Piros updated OOZIE-3384:
Component/s: tests
> [tests] TestWorkflowActionRetryInfoXCommand#testRetryConsoleUrlForked() is
> flaky
>
>
> Key: OOZIE-3384
> URL: https://issues.apache.org/jira/browse/OOZIE-3384
> Project: Oozie
> Issue Type: Sub-task
> Components: tests
>Reporter: Andras Piros
>Priority: Major
>
> {code:java}
> junit.framework.AssertionFailedError: Expected :2 Actual :1 difference> at junit.framework.Assert.fail(Assert.java:57) at
> ...
> org.apache.oozie.command.wf.TestWorkflowActionRetryInfoXCommand.validateRetryConsoleUrl(TestWorkflowActionRetryInfoXCommand.java:172)
> at
> org.apache.oozie.command.wf.TestWorkflowActionRetryInfoXCommand.testRetryConsoleUrlForked(TestWorkflowActionRetryInfoXCommand.java:125)
> at
> ...
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Created] (OOZIE-3384) [tests] TestWorkflowActionRetryInfoXCommand#testRetryConsoleUrlForked() is flaky
Andras Piros created OOZIE-3384:
---
Summary: [tests]
TestWorkflowActionRetryInfoXCommand#testRetryConsoleUrlForked() is flaky
Key: OOZIE-3384
URL: https://issues.apache.org/jira/browse/OOZIE-3384
Project: Oozie
Issue Type: Sub-task
Reporter: Andras Piros
{code:java}
junit.framework.AssertionFailedError: Expected :2 Actual :1 at junit.framework.Assert.fail(Assert.java:57) at
...
org.apache.oozie.command.wf.TestWorkflowActionRetryInfoXCommand.validateRetryConsoleUrl(TestWorkflowActionRetryInfoXCommand.java:172)
at
org.apache.oozie.command.wf.TestWorkflowActionRetryInfoXCommand.testRetryConsoleUrlForked(TestWorkflowActionRetryInfoXCommand.java:125)
at
...
{code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (OOZIE-3381) [coordinator] Enhance logging of CoordElFunctions
[
https://issues.apache.org/jira/browse/OOZIE-3381?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16686507#comment-16686507
]
Andras Piros commented on OOZIE-3381:
-
Test failure is unrelated and caused by OOZIE-3384. [~kmarton] can you please
review the patch? Thanks!
> [coordinator] Enhance logging of CoordElFunctions
> -
>
> Key: OOZIE-3381
> URL: https://issues.apache.org/jira/browse/OOZIE-3381
> Project: Oozie
> Issue Type: Task
> Components: coordinator
>Affects Versions: 5.1.0
>Reporter: Andras Piros
>Assignee: Andras Piros
>Priority: Major
> Attachments: OOZIE-3381.001.patch
>
>
> When using [{{$\{coord:latest\(n\)}}} coordinator EL
> function|https://oozie.apache.org/docs/5.0.0/CoordinatorFunctionalSpec.html#a6.6.6._coord:latestint_n_EL_Function_for_Synchronous_Datasets]
> inside an input dataset dependency, it's often the case that more
> information is needed how many HDFS URIs are being checked for each
> {{}}.
> Right now we don't have this information. While debugging and fine tuning
> parameters like {{dataset frequency}}, {{initial-instance}}, and {{data-in}}
> {{instance}}, it would be very useful to know how many HDFS roundtrips are
> issues by the current settings {{CoordELFunctions#coord_latestRange_sync()}}
> and {{CoordELFunctions#coord_futureRange_sync()}} having called
> {{DFSClient#exists()}}. We need appropriate logging there.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (OOZIE-3381) [coordinator] Enhance logging of CoordElFunctions
[
https://issues.apache.org/jira/browse/OOZIE-3381?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16687910#comment-16687910
]
Andras Piros commented on OOZIE-3381:
-
[~kmarton] sure thing, [here u go|https://reviews.apache.org/r/69348/].
> [coordinator] Enhance logging of CoordElFunctions
> -
>
> Key: OOZIE-3381
> URL: https://issues.apache.org/jira/browse/OOZIE-3381
> Project: Oozie
> Issue Type: Task
> Components: coordinator
>Affects Versions: 5.1.0
>Reporter: Andras Piros
>Assignee: Andras Piros
>Priority: Major
> Attachments: OOZIE-3381.001.patch
>
>
> When using [{{$\{coord:latest\(n\)}}} coordinator EL
> function|https://oozie.apache.org/docs/5.0.0/CoordinatorFunctionalSpec.html#a6.6.6._coord:latestint_n_EL_Function_for_Synchronous_Datasets]
> inside an input dataset dependency, it's often the case that more
> information is needed how many HDFS URIs are being checked for each
> {{}}.
> Right now we don't have this information. While debugging and fine tuning
> parameters like {{dataset frequency}}, {{initial-instance}}, and {{data-in}}
> {{instance}}, it would be very useful to know how many HDFS roundtrips are
> issues by the current settings {{CoordELFunctions#coord_latestRange_sync()}}
> and {{CoordELFunctions#coord_futureRange_sync()}} having called
> {{DFSClient#exists()}}. We need appropriate logging there.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (OOZIE-3385) The situation multi user submit workflows , occasionally, occur the HDFS visitor user become another one
[
https://issues.apache.org/jira/browse/OOZIE-3385?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16691555#comment-16691555
]
Andras Piros commented on OOZIE-3385:
-
[~luguangming] the submitting user [is defined by the {{user.name}}
property|https://oozie.apache.org/docs/5.0.0/WorkflowFunctionalSpec.html#a6_User_Propagation]
of the submitted {{workflow.xml}} / {{job.properties}}. While submitting a
workflow / coordinator / bundle job, setting the HTTP parameter
{{oozie.user.name}} would override the one given in the workflow configuration.
There are other possibilities of [user
authentication|https://oozie.apache.org/docs/5.0.0/AG_Install.html#Oozie_User_Authentication_Configuration],
though. I'd recommend to do {{klist}} just before job submission - you might
have been authenticated as another user. Checking the exact CLI command might
also be helpful.
> The situation multi user submit workflows , occasionally, occur the HDFS
> visitor user become another one
> -
>
> Key: OOZIE-3385
> URL: https://issues.apache.org/jira/browse/OOZIE-3385
> Project: Oozie
> Issue Type: Bug
> Components: core
>Affects Versions: 4.3.1
>Reporter: LuGuangMing
>Priority: Blocker
> Attachments: oozie-server-error.log,
> part_source_HadoopAccessorService.txt, part_source_WorkflowAppService.txt
>
>
> The situation multi user submit workflows , occasionally, occur the HDFS
> visitor user become another one . for example, I need submit a workflow by
> proxy user "{color:#ff}platform{color}" via user oozie (kerberos) , an
> error occur in oozie source code WorkflowAppService.readDefinition read
> workflow.xml.
> *2018-11-14 00:00:00,497 ERROR
> [CallableQueue-42]org.apache.oozie.command.wf.SubmitXCommand(517)
> {color:#ff}USER[platform]{color} GROUP[-] TOKEN[]
> APP[myBulkload-Scheduler-CS_TTL-1539689446]
> JOB[0002497-180928143722290-oozie-root-C]
> ACTION[0002497-180928143722290-oozie-root-C@1354] XException,
> org.apache.oozie.command.CommandException: E0710: Could not read the workflow
> definition, Permission denied: user={color:#ff}dbzq04{color},
> access=READ,
> inode="/phoebus/_fileservice/users/nsplatform/platform/workflows/DataLoadWF-1427-1129/workflow.xml":{color:#ff}platform{color}:supergroup:-rw---*
> note: user "{color:#ff}dbzq04{color}" also submit some workflow at
> before, but current submit the workflow of user is user
> {color:#ff}platform. In order to prove current user is platform , I
> insert some logs at oozie source code {color}
>
>
> {code:java}
> /** org.apache.oozie.service.HadoopAccessorService */
> public FileSystem createFileSystem(String user, final URI uri, final
> Configuration conf) throws HadoopAccessorException {
> //.omit..
> try {
>UserGroupInformation ugi = getUGI(user);
>LOG.info("current user="+ugi); //-- my insert log, to print proxy ugi
> info
>return ugi.doAs(new PrivilegedExceptionAction() {
>public FileSystem run() throws Exception {
> FileSystem fs = FileSystem.get(uri, conf);
> //-- my insert log, to print fs inner ugi info
> if(fs instanceof DistributedFileSystem){
> LOG.info("hdfs client user,
> "+((DistributedFileSystem)fs).getClient().toString());
> }
> return fs;
>}
> });
> }catch (InterruptedException ex) {
>throw new HadoopAccessorException(ErrorCode.E0902, ex.getMessage(), ex);
> }catch (IOException ex) {
>throw new HadoopAccessorException(ErrorCode.E0902, ex.getMessage(), ex);
> }
> }{code}
> *my log print result follows:*
>
> 2018-11-14 00:00:00,492 INFO
> [CallableQueue-42]org.apache.oozie.service.HadoopAccessorService(520)
> USER[platform] GROUP[-] TOKEN[] APP[myBulkload-Scheduler-CS_TTL-1539689446]
> JOB[0002497-180928143722290-oozie-root-C]
> ACTION[0002497-180928143722290-oozie-root-C@1354] *current user=platform
> (auth:PROXY) via oozie/nsplatfor...@dc1.fh.com (auth:KERBEROS)*
> 2018-11-14 00:00:00,493 INFO
> [CallableQueue-42]org.apache.oozie.service.HadoopAccessorService(520)
> USER[platform] GROUP[-] TOKEN[] APP[myBulkload-Scheduler-CS_TTL-1539689446]
> JOB[0002497-180928143722290-oozie-root-C]
> ACTION[0002497-180928143722290-oozie-root-C@1354] *hdfs client user,
> DFSClient[clientName=DFSClient_NONMAPREDUCE_-515910437_325, ugi=platform
> (auth:PROXY) via oozie/nsplatfor...@dc1.fh.com (auth:KERBEROS)]*
>
> *over above the proves Indicate at visited HDFS path of user has been
> altered, where did user "**{color:#ff}dbzq04" come from? please help me
> check this problem--{color}***
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (OOZIE-3381) [coordinator] Enhance logging of CoordElFunctions
[
https://issues.apache.org/jira/browse/OOZIE-3381?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Andras Piros updated OOZIE-3381:
Attachment: OOZIE-3381.002.patch
> [coordinator] Enhance logging of CoordElFunctions
> -
>
> Key: OOZIE-3381
> URL: https://issues.apache.org/jira/browse/OOZIE-3381
> Project: Oozie
> Issue Type: Task
> Components: coordinator
>Affects Versions: 5.1.0
>Reporter: Andras Piros
>Assignee: Andras Piros
>Priority: Major
> Attachments: OOZIE-3381.001.patch, OOZIE-3381.002.patch
>
>
> When using [{{$\{coord:latest\(n\)}}} coordinator EL
> function|https://oozie.apache.org/docs/5.0.0/CoordinatorFunctionalSpec.html#a6.6.6._coord:latestint_n_EL_Function_for_Synchronous_Datasets]
> inside an input dataset dependency, it's often the case that more
> information is needed how many HDFS URIs are being checked for each
> {{}}.
> Right now we don't have this information. While debugging and fine tuning
> parameters like {{dataset frequency}}, {{initial-instance}}, and {{data-in}}
> {{instance}}, it would be very useful to know how many HDFS roundtrips are
> issues by the current settings {{CoordELFunctions#coord_latestRange_sync()}}
> and {{CoordELFunctions#coord_futureRange_sync()}} having called
> {{DFSClient#exists()}}. We need appropriate logging there.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (OOZIE-3381) [coordinator] Enhance logging of CoordElFunctions
[
https://issues.apache.org/jira/browse/OOZIE-3381?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16693044#comment-16693044
]
Andras Piros commented on OOZIE-3381:
-
Thanks for the review [~kmarton]! Updated patch based on review comments.
> [coordinator] Enhance logging of CoordElFunctions
> -
>
> Key: OOZIE-3381
> URL: https://issues.apache.org/jira/browse/OOZIE-3381
> Project: Oozie
> Issue Type: Task
> Components: coordinator
>Affects Versions: 5.1.0
>Reporter: Andras Piros
>Assignee: Andras Piros
>Priority: Major
> Attachments: OOZIE-3381.001.patch, OOZIE-3381.002.patch
>
>
> When using [{{$\{coord:latest\(n\)}}} coordinator EL
> function|https://oozie.apache.org/docs/5.0.0/CoordinatorFunctionalSpec.html#a6.6.6._coord:latestint_n_EL_Function_for_Synchronous_Datasets]
> inside an input dataset dependency, it's often the case that more
> information is needed how many HDFS URIs are being checked for each
> {{}}.
> Right now we don't have this information. While debugging and fine tuning
> parameters like {{dataset frequency}}, {{initial-instance}}, and {{data-in}}
> {{instance}}, it would be very useful to know how many HDFS roundtrips are
> issues by the current settings {{CoordELFunctions#coord_latestRange_sync()}}
> and {{CoordELFunctions#coord_futureRange_sync()}} having called
> {{DFSClient#exists()}}. We need appropriate logging there.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (OOZIE-3379) Auth token cache file name should include Oozie URL
[ https://issues.apache.org/jira/browse/OOZIE-3379?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16693113#comment-16693113 ] Andras Piros commented on OOZIE-3379: - Thanks for the new patch [~zuston]! Left comments on ReviewBoard. > Auth token cache file name should include Oozie URL > --- > > Key: OOZIE-3379 > URL: https://issues.apache.org/jira/browse/OOZIE-3379 > Project: Oozie > Issue Type: Bug > Components: client >Affects Versions: 5.0.0 >Reporter: Junfan Zhang >Assignee: Junfan Zhang >Priority: Major > Attachments: oozie-3379-1.patch, oozie-3379-2.patch, > oozie-3379-3.patch, oozie-3379-4.patch, oozie-3379-5.patch > > > We have a program that uses the oozie client, but when the client connects to > multiple clusters, > the authOozieClient class frequently requests the kdc server because the > authentication token cache file is invalid. > This will cause subsequent requests in our program to be blocked, resulting > in unstable services. > So, oozie client's auth token cache file name should include Oozie URL. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (OOZIE-3387) Optimize coordinator data input dependency search
[
https://issues.apache.org/jira/browse/OOZIE-3387?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Andras Piros updated OOZIE-3387:
Affects Version/s: 5.1.0
> Optimize coordinator data input dependency search
> -
>
> Key: OOZIE-3387
> URL: https://issues.apache.org/jira/browse/OOZIE-3387
> Project: Oozie
> Issue Type: Improvement
>Affects Versions: 5.1.0
>Reporter: Andras Salamon
>Priority: Major
>
> During data input dependency check Oozie evaluates EL functions like {{
> coord:latest}} using a non-optimal way which may result more than necessary
> HDFS URI checks.
> 1. If the {{dataset}} frequency does not match the {{uri-template}} it checks
> the same HDFS URI multiple times. For instance in the following definition:
> {noformat}
> initial-instance="2017-01-01T08:15Z" timezone="UTC">
>
> ${nameNode}/${rootDir}/${YEAR}-${MONTH}-${DAY}
> _SUCCESS
>
> ...
>
> ${coord:latest(0)}
>
> {noformat}
> oozie check the same {{.../2018-11-20/_SUCCESS}} file 24*60=1440 times. It
> would be enough to check the file only once and skip the other 1439 tests.
> 2. If the frequency is 1 day and {{uri-template}} is definied in the
> following way:
> {noformat}
> ${nameNode}/${rootDir}/${YEAR}/${MONTH}/${DAY}
> {noformat}
> oozie will check the following directories one by one even if the some of the
> parent directories are missing:
> {noformat}
> 2018/11/20
> 2018/11/19
> 2018/11/18
> ...
> {noformat}
> If there is no {{2018/11}} directory then it is not necessary to check all
> the {{2018/11/xx}} directories. It would be possible to reduce the number of
> HDFS URI checks.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (OOZIE-3387) Optimize coordinator data input dependency search
[
https://issues.apache.org/jira/browse/OOZIE-3387?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Andras Piros updated OOZIE-3387:
Description:
During data input dependency check Oozie evaluates EL functions like
{{coord:latest()}} using a non-optimal way which may result more than necessary
HDFS URI checks.
1. If the {{dataset}} frequency does not match the {{uri-template}} it checks
the same HDFS URI multiple times. For instance in the following definition:
{noformat}
${nameNode}/${rootDir}/${YEAR}-${MONTH}-${DAY}
_SUCCESS
...
${coord:latest(0)}
{noformat}
oozie check the same {{.../2018-11-20/_SUCCESS}} file 24*60=1440 times. It
would be enough to check the file only once and skip the other 1439 tests.
2. If the frequency is 1 day and {{uri-template}} is definied in the following
way:
{noformat}
${nameNode}/${rootDir}/${YEAR}/${MONTH}/${DAY}
{noformat}
oozie will check the following directories one by one even if the some of the
parent directories are missing:
{noformat}
2018/11/20
2018/11/19
2018/11/18
...
{noformat}
If there is no {{2018/11}} directory then it is not necessary to check all the
{{2018/11/xx}} directories. It would be possible to reduce the number of HDFS
URI checks.
was:
During data input dependency check Oozie evaluates EL functions like {{
coord:latest}} using a non-optimal way which may result more than necessary
HDFS URI checks.
1. If the {{dataset}} frequency does not match the {{uri-template}} it checks
the same HDFS URI multiple times. For instance in the following definition:
{noformat}
${nameNode}/${rootDir}/${YEAR}-${MONTH}-${DAY}
_SUCCESS
...
${coord:latest(0)}
{noformat}
oozie check the same {{.../2018-11-20/_SUCCESS}} file 24*60=1440 times. It
would be enough to check the file only once and skip the other 1439 tests.
2. If the frequency is 1 day and {{uri-template}} is definied in the following
way:
{noformat}
${nameNode}/${rootDir}/${YEAR}/${MONTH}/${DAY}
{noformat}
oozie will check the following directories one by one even if the some of the
parent directories are missing:
{noformat}
2018/11/20
2018/11/19
2018/11/18
...
{noformat}
If there is no {{2018/11}} directory then it is not necessary to check all the
{{2018/11/xx}} directories. It would be possible to reduce the number of HDFS
URI checks.
> Optimize coordinator data input dependency search
> -
>
> Key: OOZIE-3387
> URL: https://issues.apache.org/jira/browse/OOZIE-3387
> Project: Oozie
> Issue Type: Improvement
>Affects Versions: 5.1.0
>Reporter: Andras Salamon
>Priority: Major
>
> During data input dependency check Oozie evaluates EL functions like
> {{coord:latest()}} using a non-optimal way which may result more than
> necessary HDFS URI checks.
> 1. If the {{dataset}} frequency does not match the {{uri-template}} it checks
> the same HDFS URI multiple times. For instance in the following definition:
> {noformat}
> initial-instance="2017-01-01T08:15Z" timezone="UTC">
>
> ${nameNode}/${rootDir}/${YEAR}-${MONTH}-${DAY}
> _SUCCESS
>
> ...
>
> ${coord:latest(0)}
>
> {noformat}
> oozie check the same {{.../2018-11-20/_SUCCESS}} file 24*60=1440 times. It
> would be enough to check the file only once and skip the other 1439 tests.
> 2. If the frequency is 1 day and {{uri-template}} is definied in the
> following way:
> {noformat}
> ${nameNode}/${rootDir}/${YEAR}/${MONTH}/${DAY}
> {noformat}
> oozie will check the following directories one by one even if the some of the
> parent directories are missing:
> {noformat}
> 2018/11/20
> 2018/11/19
> 2018/11/18
> ...
> {noformat}
> If there is no {{2018/11}} directory then it is not necessary to check all
> the {{2018/11/xx}} directories. It would be possible to reduce the number of
> HDFS URI checks.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (OOZIE-3388) When use oozie5.0.0 submit a Action on Yarn,Launcher filed
[
https://issues.apache.org/jira/browse/OOZIE-3388?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16695803#comment-16695803
]
Andras Piros commented on OOZIE-3388:
-
Thanks for reporting this [~nobigo]! Can you please check whether this issue is
the same as OOZIE-3307?
> When use oozie5.0.0 submit a Action on Yarn,Launcher filed
> ---
>
> Key: OOZIE-3388
> URL: https://issues.apache.org/jira/browse/OOZIE-3388
> Project: Oozie
> Issue Type: Bug
> Components: core
>Affects Versions: 5.0.0
> Environment: Spark 2.1.0
> Yarn 2.6.0
> Oozie 5.0.0
>Reporter: duan xiong
>Priority: Major
>
> When use oozie5.0.0 submit a Action on Yarn,Launcher filed
> {color:#FF}Application application_1542858285831_0002 failed 1 times due
> to AM Container for appattempt_1542858285831_0002_01 exited with
> exitCode: -103
> For more detailed output, check application tracking
> page:http://ude242:18088/proxy/application_1542858285831_0002/AThen, click on
> links to logs of each attempt.
> Diagnostics: Container
> [pid=1383,containerID=container_1542858285831_0002_01_01] is running
> beyond virtual memory limits. Current usage: 479.9 MB of 5 GB physical memory
> used; 33.0 GB of 25 GB virtual memory used. Killing container.
> {color}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Assigned] (OOZIE-3389) NPE after OOZIE-3370
[
https://issues.apache.org/jira/browse/OOZIE-3389?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Andras Piros reassigned OOZIE-3389:
---
Assignee: Andras Piros
> NPE after OOZIE-3370
> --
>
> Key: OOZIE-3389
> URL: https://issues.apache.org/jira/browse/OOZIE-3389
> Project: Oozie
> Issue Type: Bug
>Affects Versions: 5.1.0
>Reporter: Attila Sasvari
>Assignee: Andras Piros
>Priority: Blocker
>
> I was testing 5.1 rc1. I submitted the cron-schedule example coordinator,
> clicked on one of the coordinator jobs on the Oozie Web UI, but the details
> of the job were not displayed. I noticed the following NPE in the server log:
> {code}
> 2018-11-21 15:45:00,019 ERROR CoordActionMissingDependenciesXCommand:517 -
> SERVER[Budapests-MacBook-Pro-10.local] USER[-] GROUP[-] TOKEN[-] APP[-]
> JOB[-] ACTION[-] XException,
> org.apache.oozie.command.CommandException: E1028: Coord input logic error.
> null
> at
> org.apache.oozie.command.coord.CoordActionMissingDependenciesXCommand.execute(CoordActionMissingDependenciesXCommand.java:126)
> at
> org.apache.oozie.command.coord.CoordActionMissingDependenciesXCommand.execute(CoordActionMissingDependenciesXCommand.java:38)
> at org.apache.oozie.command.XCommand.call(XCommand.java:291)
> at
> org.apache.oozie.CoordinatorEngine.getCoordActionMissingDependencies(CoordinatorEngine.java:910)
> at
> org.apache.oozie.servlet.V2JobServlet.getCoordActionMissingDependencies(V2JobServlet.java:336)
> at org.apache.oozie.servlet.BaseJobServlet.doGet(BaseJobServlet.java:370)
> at javax.servlet.http.HttpServlet.service(HttpServlet.java:687)
> at
> org.apache.oozie.servlet.JsonRestServlet.service(JsonRestServlet.java:305)
> at javax.servlet.http.HttpServlet.service(HttpServlet.java:790)
> at org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:848)
> at
> org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1772)
> at org.apache.oozie.servlet.AuthFilter$2.doFilter(AuthFilter.java:171)
> at
> org.apache.hadoop.security.authentication.server.AuthenticationFilter.doFilter(AuthenticationFilter.java:594)
> at
> org.apache.hadoop.security.authentication.server.AuthenticationFilter.doFilter(AuthenticationFilter.java:553)
> at org.apache.oozie.servlet.AuthFilter.doFilter(AuthFilter.java:176)
> at
> org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1759)
> at
> org.apache.oozie.servlet.HostnameFilter.doFilter(HostnameFilter.java:86)
> at
> org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1759)
> at
> org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:582)
> at
> org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:143)
> at
> org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:548)
> at
> org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:226)
> at
> org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1180)
> at
> org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:512)
> at
> org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:185)
> at
> org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1112)
> at
> org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141)
> at
> org.eclipse.jetty.server.handler.HandlerCollection.handle(HandlerCollection.java:119)
> at
> org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:134)
> at org.eclipse.jetty.server.Server.handle(Server.java:534)
> at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:320)
> at
> org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:251)
> at
> org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:283)
> at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:108)
> at
> org.eclipse.jetty.io.SelectChannelEndPoint$2.run(SelectChannelEndPoint.java:93)
> at
> org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.executeProduceConsume(ExecuteProduceConsume.java:303)
> at
> org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.produceConsume(ExecuteProduceConsume.java:148)
> at
> org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.run(ExecuteProduceConsume.java:136)
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Assigned] (OOZIE-3390) shell action's stderr contains a bogus error message (since 5.0.0)
[
https://issues.apache.org/jira/browse/OOZIE-3390?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Andras Piros reassigned OOZIE-3390:
---
Assignee: Andras Piros
> shell action's stderr contains a bogus error message (since 5.0.0)
> ---
>
> Key: OOZIE-3390
> URL: https://issues.apache.org/jira/browse/OOZIE-3390
> Project: Oozie
> Issue Type: Bug
>Affects Versions: 5.1.0
>Reporter: Attila Sasvari
>Assignee: Andras Piros
>Priority: Blocker
>
> Shell action's stderr contains an error message about path.
> I executed shell action example to test Oozie 5.1 rc1, the job finished with
> success, but stderr of the job contained "Path echo doesn't appear to exist".
> https://github.com/apache/oozie/blob/27e4bf1688a6a7750b9c8454de5021337696fd61/sharelib/oozie/src/main/java/org/apache/oozie/action/hadoop/ShellContentWriter.java#L91
> {code}
> writeLine(errorStream, "Path " + filename + " doesn't appear
> to exist");
> {code}
> The problem is that we ignore the PATH of the process.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (OOZIE-3382) Optimize SshActionExecutor's drainBuffers method
[
https://issues.apache.org/jira/browse/OOZIE-3382?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16698887#comment-16698887
]
Andras Piros commented on OOZIE-3382:
-
[~asalamon74] thanks for the works, it's a very neat improvement to SSH action.
In general, a nice approach and pretty good test coverage. Most of my comments
on RB point towards better readability / maintainability.
I'd extend the test coverage by adding:
* slow processes that drain up till several seconds
* processes that don't drain on a linear scale, but after a while streams get
paused and then resumed w/ random pause times
* drain a couple of processes at a time in different threads, and see whether
all of those finish correctly
* randomly failing while draining some parallel processes
> Optimize SshActionExecutor's drainBuffers method
>
>
> Key: OOZIE-3382
> URL: https://issues.apache.org/jira/browse/OOZIE-3382
> Project: Oozie
> Issue Type: Improvement
>Reporter: Andras Salamon
>Assignee: Andras Salamon
>Priority: Major
> Attachments: OOZIE-3382-01.patch, OOZIE-3382-02.patch,
> OOZIE-3382-03.patch
>
>
> OOZIE-3354 improved {{SshActionExecutor}} to avoid {{Process#waitFor()}}
> blocks and modified the {{drainBuffers}} method to keep draining the standard
> output (and standard error) continuously.
> Right now the speed of the drain is hardwired. As long as the process is
> running the method only reads 1024 bytes in each cycle (half a second) which
> can take very long time if we want to drain several megabytes (for instance
> {{oozie.servlet.CallbackServlet.max.data.len}} is increased).
> Let's optimize the draining.
> We can either read 1024 bytes multiple times in each cycle (as long as there
> are data in the buffer), or we can increase the value of the buffer size
> (1024).
> In the latter case the default of the buffer size could be half of the
> {{oozie.servlet.CallbackServlet.max.data.len}} value, but we also need an
> additional property to specify the buffer size (to avoid memory problems
> because of using a very big buffer). We can keep 1024 as a minimum buffer
> size.
> It would be also useful to refactor the code and put the buffer draining into
> a separate class and create unit tests for the class. Using this class in
> {{ShellMain}} to avoid code duplication would also be very useful, but we
> have to fix OOZIE-3359 first.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (OOZIE-3386) Misleading error message when workflow application does not exist
[
https://issues.apache.org/jira/browse/OOZIE-3386?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16699403#comment-16699403
]
Andras Piros commented on OOZIE-3386:
-
Thanks for the contribution [~kmarton]! For backwards compatibility, do you
mind adding a prefix like that to the error message:
App directory [...] does not exist and it cannot be created because of missing
configuration value [...]
> Misleading error message when workflow application does not exist
> -
>
> Key: OOZIE-3386
> URL: https://issues.apache.org/jira/browse/OOZIE-3386
> Project: Oozie
> Issue Type: Bug
>Affects Versions: 5.1.0
>Reporter: Attila Sasvari
>Assignee: Julia Kinga Marton
>Priority: Major
> Fix For: 5.1.0
>
> Attachments: OOZIE-3386-001.patch
>
>
> Using 5.1.0 rc1, I tried to run an example workflow. Because of user error
> {{oozie.wf.application.path}} in job.properties pointed to a directory in
> HDFS that did not exist. Upon submitting the workflow, the following was
> returned
> {code}
> bin/oozie job -oozie http://localhost:11000/oozie -config
> examples/apps/demo/job.properties -Dmode=client -Dmaster=yarn -run
> -DnameNode=hdfs://localhost:9000
> log4j:WARN No appenders could be found for logger
> (org.apache.hadoop.security.authentication.client.KerberosAuthenticator).
> log4j:WARN Please initialize the log4j system properly.
> log4j:WARN See http://logging.apache.org/log4j/1.2/faq.html#noconfig for more
> info.
> Error: E0307 : E0307: Runtime error [Configuration entry
> oozie.jobs.api.generated.xml not present]
> {code}
> The server.log contained the following:
> {code}
> 2018-11-19 15:07:13,244 WARN V1JobsServlet:523 -
> SERVER[Budapests-MacBook-Pro-10.local] USER[-] GROUP[-] TOKEN[-] APP[-]
> JOB[-] ACTION[-] URL[POST
> http://localhost:11000/oozie/v2/jobs?action=start&user=asasvari]
> error[E0307], E0307: Runtime error [Configuration entry
> oozie.jobs.api.generated.xml not present]
> org.apache.oozie.servlet.XServletException: E0307: Runtime error
> [Configuration entry oozie.jobs.api.generated.xml not present]
>at
> org.apache.oozie.servlet.V1JobsServlet.checkAndWriteApplicationXMLToHDFS(V1JobsServlet.java:172)
>at
> org.apache.oozie.servlet.BaseJobsServlet.doPost(BaseJobsServlet.java:111)
>at javax.servlet.http.HttpServlet.service(HttpServlet.java:707)
>at
> org.apache.oozie.servlet.JsonRestServlet.service(JsonRestServlet.java:305)
>at javax.servlet.http.HttpServlet.service(HttpServlet.java:790)
>at
> org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:848)
>at
> org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1772)
>at org.apache.oozie.servlet.AuthFilter$2.doFilter(AuthFilter.java:171)
>at
> org.apache.hadoop.security.authentication.server.AuthenticationFilter.doFilter(AuthenticationFilter.java:572)
>at
> org.apache.hadoop.security.authentication.server.AuthenticationFilter.doFilter(AuthenticationFilter.java:542)
>at org.apache.oozie.servlet.AuthFilter.doFilter(AuthFilter.java:176)
>at
> org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1759)
>at
> org.apache.oozie.servlet.HostnameFilter.doFilter(HostnameFilter.java:86)
>at
> org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1759)
>at
> org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:582)
>at
> org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:143)
>at
> org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:548)
>at
> org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:226)
>at
> org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1180)
>at
> org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:512)
>at
> org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:185)
>at
> org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1112)
>at
> org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141)
>at
> org.eclipse.jetty.server.handler.HandlerCollection.handle(HandlerCollection.java:119)
>at
> org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:134)
>at org.eclipse.jetty.server.Server.handle(Server.java:534)
>at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:320)
>at
> org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:251)
>at
> org.eclipse.jetty.io.AbstractConnection$ReadCallback
[jira] [Comment Edited] (OOZIE-3386) Misleading error message when workflow application does not exist
[
https://issues.apache.org/jira/browse/OOZIE-3386?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16699403#comment-16699403
]
Andras Piros edited comment on OOZIE-3386 at 11/26/18 6:28 PM:
---
Thanks for the contribution [~kmarton]! For backwards compatibility, do you
mind adding a prefix like that to the error message:
{{App directory [...] does not exist and it cannot be created because of
missing configuration value [...].}}
was (Author: andras.piros):
Thanks for the contribution [~kmarton]! For backwards compatibility, do you
mind adding a prefix like that to the error message:
App directory [...] does not exist and it cannot be created because of missing
configuration value [...]
> Misleading error message when workflow application does not exist
> -
>
> Key: OOZIE-3386
> URL: https://issues.apache.org/jira/browse/OOZIE-3386
> Project: Oozie
> Issue Type: Bug
>Affects Versions: 5.1.0
>Reporter: Attila Sasvari
>Assignee: Julia Kinga Marton
>Priority: Major
> Fix For: 5.1.0
>
> Attachments: OOZIE-3386-001.patch
>
>
> Using 5.1.0 rc1, I tried to run an example workflow. Because of user error
> {{oozie.wf.application.path}} in job.properties pointed to a directory in
> HDFS that did not exist. Upon submitting the workflow, the following was
> returned
> {code}
> bin/oozie job -oozie http://localhost:11000/oozie -config
> examples/apps/demo/job.properties -Dmode=client -Dmaster=yarn -run
> -DnameNode=hdfs://localhost:9000
> log4j:WARN No appenders could be found for logger
> (org.apache.hadoop.security.authentication.client.KerberosAuthenticator).
> log4j:WARN Please initialize the log4j system properly.
> log4j:WARN See http://logging.apache.org/log4j/1.2/faq.html#noconfig for more
> info.
> Error: E0307 : E0307: Runtime error [Configuration entry
> oozie.jobs.api.generated.xml not present]
> {code}
> The server.log contained the following:
> {code}
> 2018-11-19 15:07:13,244 WARN V1JobsServlet:523 -
> SERVER[Budapests-MacBook-Pro-10.local] USER[-] GROUP[-] TOKEN[-] APP[-]
> JOB[-] ACTION[-] URL[POST
> http://localhost:11000/oozie/v2/jobs?action=start&user=asasvari]
> error[E0307], E0307: Runtime error [Configuration entry
> oozie.jobs.api.generated.xml not present]
> org.apache.oozie.servlet.XServletException: E0307: Runtime error
> [Configuration entry oozie.jobs.api.generated.xml not present]
>at
> org.apache.oozie.servlet.V1JobsServlet.checkAndWriteApplicationXMLToHDFS(V1JobsServlet.java:172)
>at
> org.apache.oozie.servlet.BaseJobsServlet.doPost(BaseJobsServlet.java:111)
>at javax.servlet.http.HttpServlet.service(HttpServlet.java:707)
>at
> org.apache.oozie.servlet.JsonRestServlet.service(JsonRestServlet.java:305)
>at javax.servlet.http.HttpServlet.service(HttpServlet.java:790)
>at
> org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:848)
>at
> org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1772)
>at org.apache.oozie.servlet.AuthFilter$2.doFilter(AuthFilter.java:171)
>at
> org.apache.hadoop.security.authentication.server.AuthenticationFilter.doFilter(AuthenticationFilter.java:572)
>at
> org.apache.hadoop.security.authentication.server.AuthenticationFilter.doFilter(AuthenticationFilter.java:542)
>at org.apache.oozie.servlet.AuthFilter.doFilter(AuthFilter.java:176)
>at
> org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1759)
>at
> org.apache.oozie.servlet.HostnameFilter.doFilter(HostnameFilter.java:86)
>at
> org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1759)
>at
> org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:582)
>at
> org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:143)
>at
> org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:548)
>at
> org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:226)
>at
> org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1180)
>at
> org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:512)
>at
> org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:185)
>at
> org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1112)
>at
> org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141)
>at
> org.eclipse.jetty.server.handler.HandlerCollection.handle(HandlerCollection.java:119)
>at
> org.eclipse.jetty.server.handler.HandlerWrapper.handle(Handle
[jira] [Commented] (OOZIE-3386) Misleading error message when workflow application does not exist
[
https://issues.apache.org/jira/browse/OOZIE-3386?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16700406#comment-16700406
]
Andras Piros commented on OOZIE-3386:
-
Thanks for the new patch [~kmarton]! +1 (pending Jenkins)
> Misleading error message when workflow application does not exist
> -
>
> Key: OOZIE-3386
> URL: https://issues.apache.org/jira/browse/OOZIE-3386
> Project: Oozie
> Issue Type: Bug
>Affects Versions: 5.1.0
>Reporter: Attila Sasvari
>Assignee: Julia Kinga Marton
>Priority: Major
> Fix For: 5.1.0
>
> Attachments: OOZIE-3386-001.patch, OOZIE-3386-002.patch
>
>
> Using 5.1.0 rc1, I tried to run an example workflow. Because of user error
> {{oozie.wf.application.path}} in job.properties pointed to a directory in
> HDFS that did not exist. Upon submitting the workflow, the following was
> returned
> {code}
> bin/oozie job -oozie http://localhost:11000/oozie -config
> examples/apps/demo/job.properties -Dmode=client -Dmaster=yarn -run
> -DnameNode=hdfs://localhost:9000
> log4j:WARN No appenders could be found for logger
> (org.apache.hadoop.security.authentication.client.KerberosAuthenticator).
> log4j:WARN Please initialize the log4j system properly.
> log4j:WARN See http://logging.apache.org/log4j/1.2/faq.html#noconfig for more
> info.
> Error: E0307 : E0307: Runtime error [Configuration entry
> oozie.jobs.api.generated.xml not present]
> {code}
> The server.log contained the following:
> {code}
> 2018-11-19 15:07:13,244 WARN V1JobsServlet:523 -
> SERVER[Budapests-MacBook-Pro-10.local] USER[-] GROUP[-] TOKEN[-] APP[-]
> JOB[-] ACTION[-] URL[POST
> http://localhost:11000/oozie/v2/jobs?action=start&user=asasvari]
> error[E0307], E0307: Runtime error [Configuration entry
> oozie.jobs.api.generated.xml not present]
> org.apache.oozie.servlet.XServletException: E0307: Runtime error
> [Configuration entry oozie.jobs.api.generated.xml not present]
>at
> org.apache.oozie.servlet.V1JobsServlet.checkAndWriteApplicationXMLToHDFS(V1JobsServlet.java:172)
>at
> org.apache.oozie.servlet.BaseJobsServlet.doPost(BaseJobsServlet.java:111)
>at javax.servlet.http.HttpServlet.service(HttpServlet.java:707)
>at
> org.apache.oozie.servlet.JsonRestServlet.service(JsonRestServlet.java:305)
>at javax.servlet.http.HttpServlet.service(HttpServlet.java:790)
>at
> org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:848)
>at
> org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1772)
>at org.apache.oozie.servlet.AuthFilter$2.doFilter(AuthFilter.java:171)
>at
> org.apache.hadoop.security.authentication.server.AuthenticationFilter.doFilter(AuthenticationFilter.java:572)
>at
> org.apache.hadoop.security.authentication.server.AuthenticationFilter.doFilter(AuthenticationFilter.java:542)
>at org.apache.oozie.servlet.AuthFilter.doFilter(AuthFilter.java:176)
>at
> org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1759)
>at
> org.apache.oozie.servlet.HostnameFilter.doFilter(HostnameFilter.java:86)
>at
> org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1759)
>at
> org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:582)
>at
> org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:143)
>at
> org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:548)
>at
> org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:226)
>at
> org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1180)
>at
> org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:512)
>at
> org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:185)
>at
> org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1112)
>at
> org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141)
>at
> org.eclipse.jetty.server.handler.HandlerCollection.handle(HandlerCollection.java:119)
>at
> org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:134)
>at org.eclipse.jetty.server.Server.handle(Server.java:534)
>at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:320)
>at
> org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:251)
>at
> org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:283)
>at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:108)
>at
> org.eclipse.jetty.io.Se
[jira] [Commented] (OOZIE-3386) Misleading error message when workflow application does not exist
[
https://issues.apache.org/jira/browse/OOZIE-3386?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16700504#comment-16700504
]
Andras Piros commented on OOZIE-3386:
-
The [pre-commit Jenkins
job|https://builds.apache.org/job/PreCommit-OOZIE-Build/928/consoleFull] failed:
{noformat}
14:52:40 [INFO]
14:52:40 [INFO] Results:
14:52:40 [INFO]
14:52:40 [INFO] Tests run: 589, Failures: 0, Errors: 0, Skipped: 0
14:52:40 [INFO]
14:52:40 [ERROR] There are test failures.
14:52:40
14:52:40 Please refer to
/home/jenkins/jenkins-slave/workspace/PreCommit-OOZIE-Build/core/target/surefire-reports
for the individual test results.
14:52:40 Please refer to dump files (if any exist) [date]-jvmRun[N].dump,
[date].dumpstream and [date]-jvmRun[N].dumpstream.
14:52:40 The forked VM terminated without properly saying goodbye. VM crash or
System.exit called?
14:52:40 Command was /bin/sh -c cd
/home/jenkins/jenkins-slave/workspace/PreCommit-OOZIE-Build/core &&
/usr/local/asfpackages/java/jdk1.8.0_191/jre/bin/java -Xmx2048m -da
-XX:MetaspaceSize=512M -XX:MaxMetaspaceSize=1024M -XX:+CMSClassUnloadingEnabled
-XX:+UseConcMarkSweepGC -jar
/home/jenkins/jenkins-slave/workspace/PreCommit-OOZIE-Build/core/target/surefire/surefirebooter4402261180471750437.jar
/home/jenkins/jenkins-slave/workspace/PreCommit-OOZIE-Build/core/target/surefire
2018-11-27T13-37-04_331-jvmRun1 surefire8557612660307744506tmp
surefire_41344644475305313520tmp
14:52:40 Error occurred in starting fork, check output in log
14:52:40 Process Exit Code: 255
{noformat}
[~kmarton] can you please have a look?
> Misleading error message when workflow application does not exist
> -
>
> Key: OOZIE-3386
> URL: https://issues.apache.org/jira/browse/OOZIE-3386
> Project: Oozie
> Issue Type: Bug
>Affects Versions: 5.1.0
>Reporter: Attila Sasvari
>Assignee: Julia Kinga Marton
>Priority: Major
> Fix For: 5.1.0
>
> Attachments: OOZIE-3386-001.patch, OOZIE-3386-002.patch
>
>
> Using 5.1.0 rc1, I tried to run an example workflow. Because of user error
> {{oozie.wf.application.path}} in job.properties pointed to a directory in
> HDFS that did not exist. Upon submitting the workflow, the following was
> returned
> {code}
> bin/oozie job -oozie http://localhost:11000/oozie -config
> examples/apps/demo/job.properties -Dmode=client -Dmaster=yarn -run
> -DnameNode=hdfs://localhost:9000
> log4j:WARN No appenders could be found for logger
> (org.apache.hadoop.security.authentication.client.KerberosAuthenticator).
> log4j:WARN Please initialize the log4j system properly.
> log4j:WARN See http://logging.apache.org/log4j/1.2/faq.html#noconfig for more
> info.
> Error: E0307 : E0307: Runtime error [Configuration entry
> oozie.jobs.api.generated.xml not present]
> {code}
> The server.log contained the following:
> {code}
> 2018-11-19 15:07:13,244 WARN V1JobsServlet:523 -
> SERVER[Budapests-MacBook-Pro-10.local] USER[-] GROUP[-] TOKEN[-] APP[-]
> JOB[-] ACTION[-] URL[POST
> http://localhost:11000/oozie/v2/jobs?action=start&user=asasvari]
> error[E0307], E0307: Runtime error [Configuration entry
> oozie.jobs.api.generated.xml not present]
> org.apache.oozie.servlet.XServletException: E0307: Runtime error
> [Configuration entry oozie.jobs.api.generated.xml not present]
>at
> org.apache.oozie.servlet.V1JobsServlet.checkAndWriteApplicationXMLToHDFS(V1JobsServlet.java:172)
>at
> org.apache.oozie.servlet.BaseJobsServlet.doPost(BaseJobsServlet.java:111)
>at javax.servlet.http.HttpServlet.service(HttpServlet.java:707)
>at
> org.apache.oozie.servlet.JsonRestServlet.service(JsonRestServlet.java:305)
>at javax.servlet.http.HttpServlet.service(HttpServlet.java:790)
>at
> org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:848)
>at
> org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1772)
>at org.apache.oozie.servlet.AuthFilter$2.doFilter(AuthFilter.java:171)
>at
> org.apache.hadoop.security.authentication.server.AuthenticationFilter.doFilter(AuthenticationFilter.java:572)
>at
> org.apache.hadoop.security.authentication.server.AuthenticationFilter.doFilter(AuthenticationFilter.java:542)
>at org.apache.oozie.servlet.AuthFilter.doFilter(AuthFilter.java:176)
>at
> org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1759)
>at
> org.apache.oozie.servlet.HostnameFilter.doFilter(HostnameFilter.java:86)
>at
> org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1759)
>at
> org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:582)
>at
> org.eclipse.jetty.server.handler.Scop
[jira] [Commented] (OOZIE-3393) Add Oozie instrumentation delayed metric in CoordMaterializeTriggerService
[ https://issues.apache.org/jira/browse/OOZIE-3393?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16701763#comment-16701763 ] Andras Piros commented on OOZIE-3393: - Thanks for the contribution [~zuston]! Can you please upload the patch to ReviewBoard? Thanks! > Add Oozie instrumentation delayed metric in CoordMaterializeTriggerService > -- > > Key: OOZIE-3393 > URL: https://issues.apache.org/jira/browse/OOZIE-3393 > Project: Oozie > Issue Type: Improvement >Reporter: Junfan Zhang >Assignee: Junfan Zhang >Priority: Major > Attachments: oozie-3393-1.patch > > > We have multiple Oozie scheduled clusters. The problem that is currently > encountered is that some materialize queues are too late to process during > the peak hours of the scheduled tasks. We need to get the metrics of the > delay queue for alarm monitoring and adjust the configuration to optimize. > Currently we have added metrics for queue size and latency in Oozie > CoordMaterializeTriggerService class, in order to reflect the general trend > of current scheduling. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (OOZIE-3379) Auth token cache file name should include Oozie URL
[ https://issues.apache.org/jira/browse/OOZIE-3379?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16701781#comment-16701781 ] Andras Piros commented on OOZIE-3379: - Thanks [~zuston] for the new patch 006! Can you please also upload to ReviewBoard via [Update diff|https://www.reviewboard.org/docs/manual/1.5/users/review-requests/updating/]? > Auth token cache file name should include Oozie URL > --- > > Key: OOZIE-3379 > URL: https://issues.apache.org/jira/browse/OOZIE-3379 > Project: Oozie > Issue Type: Bug > Components: client >Affects Versions: 5.0.0 >Reporter: Junfan Zhang >Assignee: Junfan Zhang >Priority: Major > Attachments: oozie-3379-1.patch, oozie-3379-2.patch, > oozie-3379-3.patch, oozie-3379-4.patch, oozie-3379-5.patch, oozie-3379-6.patch > > > We have a program that uses the oozie client, but when the client connects to > multiple clusters, > the authOozieClient class frequently requests the kdc server because the > authentication token cache file is invalid. > This will cause subsequent requests in our program to be blocked, resulting > in unstable services. > So, oozie client's auth token cache file name should include Oozie URL. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (OOZIE-3379) Auth token cache file name should include Oozie URL
[ https://issues.apache.org/jira/browse/OOZIE-3379?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16701952#comment-16701952 ] Andras Piros commented on OOZIE-3379: - Thanks [~zuston]. Left some more comments over on RB. Can you please fix those, and close old comments? Thanks! > Auth token cache file name should include Oozie URL > --- > > Key: OOZIE-3379 > URL: https://issues.apache.org/jira/browse/OOZIE-3379 > Project: Oozie > Issue Type: Bug > Components: client >Affects Versions: 5.0.0 >Reporter: Junfan Zhang >Assignee: Junfan Zhang >Priority: Major > Attachments: oozie-3379-1.patch, oozie-3379-2.patch, > oozie-3379-3.patch, oozie-3379-4.patch, oozie-3379-5.patch, oozie-3379-6.patch > > > We have a program that uses the oozie client, but when the client connects to > multiple clusters, > the authOozieClient class frequently requests the kdc server because the > authentication token cache file is invalid. > This will cause subsequent requests in our program to be blocked, resulting > in unstable services. > So, oozie client's auth token cache file name should include Oozie URL. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (OOZIE-3393) Add Oozie instrumentation delayed metric in CoordMaterializeTriggerService
[
https://issues.apache.org/jira/browse/OOZIE-3393?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16702003#comment-16702003
]
Andras Piros commented on OOZIE-3393:
-
Thanks [~zuston]. Overall direction is OK. Please add some unit tests, and
sample {{oozie-instrumentation.log}} snippets.
> Add Oozie instrumentation delayed metric in CoordMaterializeTriggerService
> --
>
> Key: OOZIE-3393
> URL: https://issues.apache.org/jira/browse/OOZIE-3393
> Project: Oozie
> Issue Type: Improvement
>Reporter: Junfan Zhang
>Assignee: Junfan Zhang
>Priority: Major
> Attachments: oozie-3393-1.patch
>
>
> We have multiple Oozie scheduled clusters. The problem that is currently
> encountered is that some materialize queues are too late to process during
> the peak hours of the scheduled tasks. We need to get the metrics of the
> delay queue for alarm monitoring and adjust the configuration to optimize.
> Currently we have added metrics for queue size and latency in Oozie
> CoordMaterializeTriggerService class, in order to reflect the general trend
> of current scheduling.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Comment Edited] (OOZIE-3350) Forkjoin validation does not fail if a node is reachable from two different forks
[
https://issues.apache.org/jira/browse/OOZIE-3350?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16702935#comment-16702935
]
Andras Piros edited comment on OOZIE-3350 at 11/29/18 9:41 AM:
---
[~pbacsko] the current {{LiteWorkflowValidator#validateForkJoin()}} works well
for either nested {{}} nodes and their implicitly joining
{{}} nodes, or nested {{}} and {{}} instances, but not
for cases when those scenarios are mixed. Here the main problem is the state is
stored in multiple inout parameters like {{nodeAndDecisionParents}},
{{forkJoins}}, {{path}}, and {{topDecisionParent}} - those pieces are
independent from each other and not considered as a whole, hence those errors.
[~kmarton] the first step would be to have decent amount of unit testing
covering the bulk of {{LiteWorkflowValidator}}, introducing unit tests also for
such mixed cases. Next thing would be to reimplement {{validateForkJoin()}}
with a state object where fields are interconnected and all possible state
transition dependencies are handled correctly. I suggest to use the State
pattern for that purpose.
was (Author: andras.piros):
[~pbacsko] the current {{LiteWorkflowValidator#validateForkJoin()}} works well
for either nested {{}} nodes and their implicitly joining
{{}} nodes, or nested {{}}s and {{}}s, but not for cases
when those scenarios are mixed. Here the main problem is the state is stored in
multiple inout parameters like {{nodeAndDecisionParents}}, {{forkJoins}},
{{path}}, and {{topDecisionParent}} - those pieces are independent from each
other and not considered as a whole, hence those errors.
[~kmarton] the first step would be to have decent amount of unit testing
covering the bulk of {{LiteWorkflowValidator}}, introducing unit tests also for
such mixed cases. Next thing would be to reimplement {{validateForkJoin()}}
with a state object where fields are interconnected and all possible state
transition dependencies are handled correctly. I suggest to use the State
pattern for that purpose.
> Forkjoin validation does not fail if a node is reachable from two different
> forks
> -
>
> Key: OOZIE-3350
> URL: https://issues.apache.org/jira/browse/OOZIE-3350
> Project: Oozie
> Issue Type: Bug
> Components: core
>Affects Versions: 4.3.1
>Reporter: wang jinyin
>Assignee: Julia Kinga Marton
>Priority: Major
> Fix For: trunk
>
> Original Estimate: 120h
> Remaining Estimate: 120h
>
> when "multiple ok to same node" under decision node, forkjoin validation
> error.
>
> such as below example, 'action_C' and 'action_D' both transition to
> 'action_E'.
> But, because they are under same topDecisionParent 'decision_A', validator
> will not throw any exception.
>
> {quote}
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
> {quote}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (OOZIE-3350) Forkjoin validation does not fail if a node is reachable from two different forks
[
https://issues.apache.org/jira/browse/OOZIE-3350?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16702935#comment-16702935
]
Andras Piros commented on OOZIE-3350:
-
[~pbacsko] the current {{LiteWorkflowValidator#validateForkJoin()}} works well
for either nested {{}} nodes and their implicitly joining
{{}} nodes, or nested {{}}s and {{}}s, but not for cases
when those scenarios are mixed. Here the main problem is the state is stored in
multiple inout parameters like {{nodeAndDecisionParents}}, {{forkJoins}},
{{path}}, and {{topDecisionParent}} - those pieces are independent from each
other and not considered as a whole, hence those errors.
[~kmarton] the first step would be to have decent amount of unit testing
covering the bulk of {{LiteWorkflowValidator}}, introducing unit tests also for
such mixed cases. Next thing would be to reimplement {{validateForkJoin()}}
with a state object where fields are interconnected and all possible state
transition dependencies are handled correctly. I suggest to use the State
pattern for that purpose.
> Forkjoin validation does not fail if a node is reachable from two different
> forks
> -
>
> Key: OOZIE-3350
> URL: https://issues.apache.org/jira/browse/OOZIE-3350
> Project: Oozie
> Issue Type: Bug
> Components: core
>Affects Versions: 4.3.1
>Reporter: wang jinyin
>Assignee: Julia Kinga Marton
>Priority: Major
> Fix For: trunk
>
> Original Estimate: 120h
> Remaining Estimate: 120h
>
> when "multiple ok to same node" under decision node, forkjoin validation
> error.
>
> such as below example, 'action_C' and 'action_D' both transition to
> 'action_E'.
> But, because they are under same topDecisionParent 'decision_A', validator
> will not throw any exception.
>
> {quote}
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
> {quote}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (OOZIE-3350) Forkjoin validation does not fail if a node is reachable from two different forks
[
https://issues.apache.org/jira/browse/OOZIE-3350?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16703144#comment-16703144
]
Andras Piros commented on OOZIE-3350:
-
[~pbacsko] [~kmarton] can you post the proposed solution summary here? Thanks!
> Forkjoin validation does not fail if a node is reachable from two different
> forks
> -
>
> Key: OOZIE-3350
> URL: https://issues.apache.org/jira/browse/OOZIE-3350
> Project: Oozie
> Issue Type: Bug
> Components: core
>Affects Versions: 4.3.1
>Reporter: wang jinyin
>Assignee: Julia Kinga Marton
>Priority: Major
> Fix For: trunk
>
> Original Estimate: 120h
> Remaining Estimate: 120h
>
> when "multiple ok to same node" under decision node, forkjoin validation
> error.
>
> such as below example, 'action_C' and 'action_D' both transition to
> 'action_E'.
> But, because they are under same topDecisionParent 'decision_A', validator
> will not throw any exception.
>
> {quote}
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
> {quote}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (OOZIE-3389) Getting input dependency list on the UI throws NPE
[
https://issues.apache.org/jira/browse/OOZIE-3389?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Andras Piros updated OOZIE-3389:
Summary: Getting input dependency list on the UI throws NPE (was: NPE
after OOZIE-3370 )
> Getting input dependency list on the UI throws NPE
> --
>
> Key: OOZIE-3389
> URL: https://issues.apache.org/jira/browse/OOZIE-3389
> Project: Oozie
> Issue Type: Bug
>Affects Versions: 5.1.0
>Reporter: Attila Sasvari
>Assignee: Andras Piros
>Priority: Blocker
> Fix For: 5.1.0
>
>
> I was testing 5.1 rc1. I submitted the cron-schedule example coordinator,
> clicked on one of the coordinator jobs on the Oozie Web UI, but the details
> of the job were not displayed. I noticed the following NPE in the server log:
> {code}
> 2018-11-21 15:45:00,019 ERROR CoordActionMissingDependenciesXCommand:517 -
> SERVER[Budapests-MacBook-Pro-10.local] USER[-] GROUP[-] TOKEN[-] APP[-]
> JOB[-] ACTION[-] XException,
> org.apache.oozie.command.CommandException: E1028: Coord input logic error.
> null
> at
> org.apache.oozie.command.coord.CoordActionMissingDependenciesXCommand.execute(CoordActionMissingDependenciesXCommand.java:126)
> at
> org.apache.oozie.command.coord.CoordActionMissingDependenciesXCommand.execute(CoordActionMissingDependenciesXCommand.java:38)
> at org.apache.oozie.command.XCommand.call(XCommand.java:291)
> at
> org.apache.oozie.CoordinatorEngine.getCoordActionMissingDependencies(CoordinatorEngine.java:910)
> at
> org.apache.oozie.servlet.V2JobServlet.getCoordActionMissingDependencies(V2JobServlet.java:336)
> at org.apache.oozie.servlet.BaseJobServlet.doGet(BaseJobServlet.java:370)
> at javax.servlet.http.HttpServlet.service(HttpServlet.java:687)
> at
> org.apache.oozie.servlet.JsonRestServlet.service(JsonRestServlet.java:305)
> at javax.servlet.http.HttpServlet.service(HttpServlet.java:790)
> at org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:848)
> at
> org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1772)
> at org.apache.oozie.servlet.AuthFilter$2.doFilter(AuthFilter.java:171)
> at
> org.apache.hadoop.security.authentication.server.AuthenticationFilter.doFilter(AuthenticationFilter.java:594)
> at
> org.apache.hadoop.security.authentication.server.AuthenticationFilter.doFilter(AuthenticationFilter.java:553)
> at org.apache.oozie.servlet.AuthFilter.doFilter(AuthFilter.java:176)
> at
> org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1759)
> at
> org.apache.oozie.servlet.HostnameFilter.doFilter(HostnameFilter.java:86)
> at
> org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1759)
> at
> org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:582)
> at
> org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:143)
> at
> org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:548)
> at
> org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:226)
> at
> org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1180)
> at
> org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:512)
> at
> org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:185)
> at
> org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1112)
> at
> org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141)
> at
> org.eclipse.jetty.server.handler.HandlerCollection.handle(HandlerCollection.java:119)
> at
> org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:134)
> at org.eclipse.jetty.server.Server.handle(Server.java:534)
> at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:320)
> at
> org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:251)
> at
> org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:283)
> at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:108)
> at
> org.eclipse.jetty.io.SelectChannelEndPoint$2.run(SelectChannelEndPoint.java:93)
> at
> org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.executeProduceConsume(ExecuteProduceConsume.java:303)
> at
> org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.produceConsume(ExecuteProduceConsume.java:148)
> at
> org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.run(ExecuteProduceConsume.java:136)
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (OOZIE-3389) Getting input dependency list on the UI throws NPE
[
https://issues.apache.org/jira/browse/OOZIE-3389?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Andras Piros updated OOZIE-3389:
Attachment: OOZIE-3389.001.patch
> Getting input dependency list on the UI throws NPE
> --
>
> Key: OOZIE-3389
> URL: https://issues.apache.org/jira/browse/OOZIE-3389
> Project: Oozie
> Issue Type: Bug
>Affects Versions: 5.1.0
>Reporter: Attila Sasvari
>Assignee: Andras Piros
>Priority: Blocker
> Fix For: 5.1.0
>
> Attachments: OOZIE-3389.001.patch
>
>
> I was testing 5.1 rc1. I submitted the cron-schedule example coordinator,
> clicked on one of the coordinator jobs on the Oozie Web UI, but the details
> of the job were not displayed. I noticed the following NPE in the server log:
> {code}
> 2018-11-21 15:45:00,019 ERROR CoordActionMissingDependenciesXCommand:517 -
> SERVER[Budapests-MacBook-Pro-10.local] USER[-] GROUP[-] TOKEN[-] APP[-]
> JOB[-] ACTION[-] XException,
> org.apache.oozie.command.CommandException: E1028: Coord input logic error.
> null
> at
> org.apache.oozie.command.coord.CoordActionMissingDependenciesXCommand.execute(CoordActionMissingDependenciesXCommand.java:126)
> at
> org.apache.oozie.command.coord.CoordActionMissingDependenciesXCommand.execute(CoordActionMissingDependenciesXCommand.java:38)
> at org.apache.oozie.command.XCommand.call(XCommand.java:291)
> at
> org.apache.oozie.CoordinatorEngine.getCoordActionMissingDependencies(CoordinatorEngine.java:910)
> at
> org.apache.oozie.servlet.V2JobServlet.getCoordActionMissingDependencies(V2JobServlet.java:336)
> at org.apache.oozie.servlet.BaseJobServlet.doGet(BaseJobServlet.java:370)
> at javax.servlet.http.HttpServlet.service(HttpServlet.java:687)
> at
> org.apache.oozie.servlet.JsonRestServlet.service(JsonRestServlet.java:305)
> at javax.servlet.http.HttpServlet.service(HttpServlet.java:790)
> at org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:848)
> at
> org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1772)
> at org.apache.oozie.servlet.AuthFilter$2.doFilter(AuthFilter.java:171)
> at
> org.apache.hadoop.security.authentication.server.AuthenticationFilter.doFilter(AuthenticationFilter.java:594)
> at
> org.apache.hadoop.security.authentication.server.AuthenticationFilter.doFilter(AuthenticationFilter.java:553)
> at org.apache.oozie.servlet.AuthFilter.doFilter(AuthFilter.java:176)
> at
> org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1759)
> at
> org.apache.oozie.servlet.HostnameFilter.doFilter(HostnameFilter.java:86)
> at
> org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1759)
> at
> org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:582)
> at
> org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:143)
> at
> org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:548)
> at
> org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:226)
> at
> org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1180)
> at
> org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:512)
> at
> org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:185)
> at
> org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1112)
> at
> org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141)
> at
> org.eclipse.jetty.server.handler.HandlerCollection.handle(HandlerCollection.java:119)
> at
> org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:134)
> at org.eclipse.jetty.server.Server.handle(Server.java:534)
> at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:320)
> at
> org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:251)
> at
> org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:283)
> at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:108)
> at
> org.eclipse.jetty.io.SelectChannelEndPoint$2.run(SelectChannelEndPoint.java:93)
> at
> org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.executeProduceConsume(ExecuteProduceConsume.java:303)
> at
> org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.produceConsume(ExecuteProduceConsume.java:148)
> at
> org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.run(ExecuteProduceConsume.java:136)
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (OOZIE-3389) Getting input dependency list on the UI throws NPE
[
https://issues.apache.org/jira/browse/OOZIE-3389?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16704565#comment-16704565
]
Andras Piros commented on OOZIE-3389:
-
[~asasvari] thanks for reporting this. Unfortunately it has been there for
quite a while, isn't related to OOZIE-3370 or to any {{5.1.0}} code changes.
Fixing that, nevertheless.
> Getting input dependency list on the UI throws NPE
> --
>
> Key: OOZIE-3389
> URL: https://issues.apache.org/jira/browse/OOZIE-3389
> Project: Oozie
> Issue Type: Bug
>Affects Versions: 5.1.0
>Reporter: Attila Sasvari
>Assignee: Andras Piros
>Priority: Blocker
> Fix For: 5.1.0, 5.2.0
>
> Attachments: OOZIE-3389.001.patch
>
>
> I was testing 5.1 rc1. I submitted the cron-schedule example coordinator,
> clicked on one of the coordinator jobs on the Oozie Web UI, but the details
> of the job were not displayed. I noticed the following NPE in the server log:
> {code}
> 2018-11-21 15:45:00,019 ERROR CoordActionMissingDependenciesXCommand:517 -
> SERVER[Budapests-MacBook-Pro-10.local] USER[-] GROUP[-] TOKEN[-] APP[-]
> JOB[-] ACTION[-] XException,
> org.apache.oozie.command.CommandException: E1028: Coord input logic error.
> null
> at
> org.apache.oozie.command.coord.CoordActionMissingDependenciesXCommand.execute(CoordActionMissingDependenciesXCommand.java:126)
> at
> org.apache.oozie.command.coord.CoordActionMissingDependenciesXCommand.execute(CoordActionMissingDependenciesXCommand.java:38)
> at org.apache.oozie.command.XCommand.call(XCommand.java:291)
> at
> org.apache.oozie.CoordinatorEngine.getCoordActionMissingDependencies(CoordinatorEngine.java:910)
> at
> org.apache.oozie.servlet.V2JobServlet.getCoordActionMissingDependencies(V2JobServlet.java:336)
> at org.apache.oozie.servlet.BaseJobServlet.doGet(BaseJobServlet.java:370)
> at javax.servlet.http.HttpServlet.service(HttpServlet.java:687)
> at
> org.apache.oozie.servlet.JsonRestServlet.service(JsonRestServlet.java:305)
> at javax.servlet.http.HttpServlet.service(HttpServlet.java:790)
> at org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:848)
> at
> org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1772)
> at org.apache.oozie.servlet.AuthFilter$2.doFilter(AuthFilter.java:171)
> at
> org.apache.hadoop.security.authentication.server.AuthenticationFilter.doFilter(AuthenticationFilter.java:594)
> at
> org.apache.hadoop.security.authentication.server.AuthenticationFilter.doFilter(AuthenticationFilter.java:553)
> at org.apache.oozie.servlet.AuthFilter.doFilter(AuthFilter.java:176)
> at
> org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1759)
> at
> org.apache.oozie.servlet.HostnameFilter.doFilter(HostnameFilter.java:86)
> at
> org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1759)
> at
> org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:582)
> at
> org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:143)
> at
> org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:548)
> at
> org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:226)
> at
> org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1180)
> at
> org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:512)
> at
> org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:185)
> at
> org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1112)
> at
> org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141)
> at
> org.eclipse.jetty.server.handler.HandlerCollection.handle(HandlerCollection.java:119)
> at
> org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:134)
> at org.eclipse.jetty.server.Server.handle(Server.java:534)
> at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:320)
> at
> org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:251)
> at
> org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:283)
> at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:108)
> at
> org.eclipse.jetty.io.SelectChannelEndPoint$2.run(SelectChannelEndPoint.java:93)
> at
> org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.executeProduceConsume(ExecuteProduceConsume.java:303)
> at
> org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.produceConsume(ExecuteProduceConsume.java:148)
> at
> org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.run(E
[jira] [Commented] (OOZIE-3395) Findbugs is no longer maintained
[
https://issues.apache.org/jira/browse/OOZIE-3395?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16704713#comment-16704713
]
Andras Piros commented on OOZIE-3395:
-
[~asasvari] actually we're using [FindSecBugs and
Spotbugs|https://github.com/apache/oozie/blob/master/pom.xml#L1829-L1866]
behind the curtain. Might be interesting checking whether there is a valid
migration path between the FindBugs and Spotbugs Maven plugins, indeed.
> Findbugs is no longer maintained
>
>
> Key: OOZIE-3395
> URL: https://issues.apache.org/jira/browse/OOZIE-3395
> Project: Oozie
> Issue Type: Task
>Reporter: Attila Sasvari
>Priority: Minor
>
> https://gleclaire.github.io/findbugs-maven-plugin/
> {quote}
> Status: Since Findbugs is no longer maintained, please use Spotbugs which has
> a Maven plugin.
> {quote}
> The plugin author recommends to migrate to Spotbugs:
> https://spotbugs.github.io/
> It might worth to investigate this plugin.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (OOZIE-3389) Getting input dependency list on the UI throws NPE
[
https://issues.apache.org/jira/browse/OOZIE-3389?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16704683#comment-16704683
]
Andras Piros commented on OOZIE-3389:
-
[~kmarton] can you please review? Thanks!
> Getting input dependency list on the UI throws NPE
> --
>
> Key: OOZIE-3389
> URL: https://issues.apache.org/jira/browse/OOZIE-3389
> Project: Oozie
> Issue Type: Bug
>Affects Versions: 5.1.0
>Reporter: Attila Sasvari
>Assignee: Andras Piros
>Priority: Blocker
> Fix For: 5.1.0, 5.2.0
>
> Attachments: OOZIE-3389.001.patch
>
>
> I was testing 5.1 rc1. I submitted the cron-schedule example coordinator,
> clicked on one of the coordinator jobs on the Oozie Web UI, but the details
> of the job were not displayed. I noticed the following NPE in the server log:
> {code}
> 2018-11-21 15:45:00,019 ERROR CoordActionMissingDependenciesXCommand:517 -
> SERVER[Budapests-MacBook-Pro-10.local] USER[-] GROUP[-] TOKEN[-] APP[-]
> JOB[-] ACTION[-] XException,
> org.apache.oozie.command.CommandException: E1028: Coord input logic error.
> null
> at
> org.apache.oozie.command.coord.CoordActionMissingDependenciesXCommand.execute(CoordActionMissingDependenciesXCommand.java:126)
> at
> org.apache.oozie.command.coord.CoordActionMissingDependenciesXCommand.execute(CoordActionMissingDependenciesXCommand.java:38)
> at org.apache.oozie.command.XCommand.call(XCommand.java:291)
> at
> org.apache.oozie.CoordinatorEngine.getCoordActionMissingDependencies(CoordinatorEngine.java:910)
> at
> org.apache.oozie.servlet.V2JobServlet.getCoordActionMissingDependencies(V2JobServlet.java:336)
> at org.apache.oozie.servlet.BaseJobServlet.doGet(BaseJobServlet.java:370)
> at javax.servlet.http.HttpServlet.service(HttpServlet.java:687)
> at
> org.apache.oozie.servlet.JsonRestServlet.service(JsonRestServlet.java:305)
> at javax.servlet.http.HttpServlet.service(HttpServlet.java:790)
> at org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:848)
> at
> org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1772)
> at org.apache.oozie.servlet.AuthFilter$2.doFilter(AuthFilter.java:171)
> at
> org.apache.hadoop.security.authentication.server.AuthenticationFilter.doFilter(AuthenticationFilter.java:594)
> at
> org.apache.hadoop.security.authentication.server.AuthenticationFilter.doFilter(AuthenticationFilter.java:553)
> at org.apache.oozie.servlet.AuthFilter.doFilter(AuthFilter.java:176)
> at
> org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1759)
> at
> org.apache.oozie.servlet.HostnameFilter.doFilter(HostnameFilter.java:86)
> at
> org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1759)
> at
> org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:582)
> at
> org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:143)
> at
> org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:548)
> at
> org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:226)
> at
> org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1180)
> at
> org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:512)
> at
> org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:185)
> at
> org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1112)
> at
> org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141)
> at
> org.eclipse.jetty.server.handler.HandlerCollection.handle(HandlerCollection.java:119)
> at
> org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:134)
> at org.eclipse.jetty.server.Server.handle(Server.java:534)
> at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:320)
> at
> org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:251)
> at
> org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:283)
> at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:108)
> at
> org.eclipse.jetty.io.SelectChannelEndPoint$2.run(SelectChannelEndPoint.java:93)
> at
> org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.executeProduceConsume(ExecuteProduceConsume.java:303)
> at
> org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.produceConsume(ExecuteProduceConsume.java:148)
> at
> org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.run(ExecuteProduceConsume.java:136)
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (OOZIE-3382) Optimize SshActionExecutor's drainBuffers method
[
https://issues.apache.org/jira/browse/OOZIE-3382?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16705796#comment-16705796
]
Andras Piros commented on OOZIE-3382:
-
Thanks for the new patch [~asalamon74]! +1
> Optimize SshActionExecutor's drainBuffers method
>
>
> Key: OOZIE-3382
> URL: https://issues.apache.org/jira/browse/OOZIE-3382
> Project: Oozie
> Issue Type: Improvement
>Reporter: Andras Salamon
>Assignee: Andras Salamon
>Priority: Major
> Attachments: OOZIE-3382-01.patch, OOZIE-3382-02.patch,
> OOZIE-3382-03.patch, OOZIE-3382-04.patch
>
>
> OOZIE-3354 improved {{SshActionExecutor}} to avoid {{Process#waitFor()}}
> blocks and modified the {{drainBuffers}} method to keep draining the standard
> output (and standard error) continuously.
> Right now the speed of the drain is hardwired. As long as the process is
> running the method only reads 1024 bytes in each cycle (half a second) which
> can take very long time if we want to drain several megabytes (for instance
> {{oozie.servlet.CallbackServlet.max.data.len}} is increased).
> Let's optimize the draining.
> We can either read 1024 bytes multiple times in each cycle (as long as there
> are data in the buffer), or we can increase the value of the buffer size
> (1024).
> In the latter case the default of the buffer size could be half of the
> {{oozie.servlet.CallbackServlet.max.data.len}} value, but we also need an
> additional property to specify the buffer size (to avoid memory problems
> because of using a very big buffer). We can keep 1024 as a minimum buffer
> size.
> It would be also useful to refactor the code and put the buffer draining into
> a separate class and create unit tests for the class. Using this class in
> {{ShellMain}} to avoid code duplication would also be very useful, but we
> have to fix OOZIE-3359 first.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (OOZIE-3382) [SSH action] Optimize process streams draining
[
https://issues.apache.org/jira/browse/OOZIE-3382?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Andras Piros updated OOZIE-3382:
Summary: [SSH action] Optimize process streams draining (was: Optimize
SshActionExecutor's drainBuffers method)
> [SSH action] Optimize process streams draining
> --
>
> Key: OOZIE-3382
> URL: https://issues.apache.org/jira/browse/OOZIE-3382
> Project: Oozie
> Issue Type: Improvement
>Reporter: Andras Salamon
>Assignee: Andras Salamon
>Priority: Major
> Attachments: OOZIE-3382-01.patch, OOZIE-3382-02.patch,
> OOZIE-3382-03.patch, OOZIE-3382-04.patch
>
>
> OOZIE-3354 improved {{SshActionExecutor}} to avoid {{Process#waitFor()}}
> blocks and modified the {{drainBuffers}} method to keep draining the standard
> output (and standard error) continuously.
> Right now the speed of the drain is hardwired. As long as the process is
> running the method only reads 1024 bytes in each cycle (half a second) which
> can take very long time if we want to drain several megabytes (for instance
> {{oozie.servlet.CallbackServlet.max.data.len}} is increased).
> Let's optimize the draining.
> We can either read 1024 bytes multiple times in each cycle (as long as there
> are data in the buffer), or we can increase the value of the buffer size
> (1024).
> In the latter case the default of the buffer size could be half of the
> {{oozie.servlet.CallbackServlet.max.data.len}} value, but we also need an
> additional property to specify the buffer size (to avoid memory problems
> because of using a very big buffer). We can keep 1024 as a minimum buffer
> size.
> It would be also useful to refactor the code and put the buffer draining into
> a separate class and create unit tests for the class. Using this class in
> {{ShellMain}} to avoid code duplication would also be very useful, but we
> have to fix OOZIE-3359 first.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (OOZIE-3382) [SSH action] Optimize process streams draining
[
https://issues.apache.org/jira/browse/OOZIE-3382?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Andras Piros updated OOZIE-3382:
Affects Version/s: 5.1.0
> [SSH action] Optimize process streams draining
> --
>
> Key: OOZIE-3382
> URL: https://issues.apache.org/jira/browse/OOZIE-3382
> Project: Oozie
> Issue Type: Improvement
>Affects Versions: 5.1.0
>Reporter: Andras Salamon
>Assignee: Andras Salamon
>Priority: Major
> Fix For: 5.2.0
>
> Attachments: OOZIE-3382-01.patch, OOZIE-3382-02.patch,
> OOZIE-3382-03.patch, OOZIE-3382-04.patch
>
>
> OOZIE-3354 improved {{SshActionExecutor}} to avoid {{Process#waitFor()}}
> blocks and modified the {{drainBuffers}} method to keep draining the standard
> output (and standard error) continuously.
> Right now the speed of the drain is hardwired. As long as the process is
> running the method only reads 1024 bytes in each cycle (half a second) which
> can take very long time if we want to drain several megabytes (for instance
> {{oozie.servlet.CallbackServlet.max.data.len}} is increased).
> Let's optimize the draining.
> We can either read 1024 bytes multiple times in each cycle (as long as there
> are data in the buffer), or we can increase the value of the buffer size
> (1024).
> In the latter case the default of the buffer size could be half of the
> {{oozie.servlet.CallbackServlet.max.data.len}} value, but we also need an
> additional property to specify the buffer size (to avoid memory problems
> because of using a very big buffer). We can keep 1024 as a minimum buffer
> size.
> It would be also useful to refactor the code and put the buffer draining into
> a separate class and create unit tests for the class. Using this class in
> {{ShellMain}} to avoid code duplication would also be very useful, but we
> have to fix OOZIE-3359 first.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (OOZIE-3389) Getting input dependency list on the UI throws NPE
[
https://issues.apache.org/jira/browse/OOZIE-3389?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16705843#comment-16705843
]
Andras Piros commented on OOZIE-3389:
-
Addressing review comments w/ patch 002. [~kmarton] can you please review?
Thanks!
> Getting input dependency list on the UI throws NPE
> --
>
> Key: OOZIE-3389
> URL: https://issues.apache.org/jira/browse/OOZIE-3389
> Project: Oozie
> Issue Type: Bug
>Affects Versions: 5.1.0
>Reporter: Attila Sasvari
>Assignee: Andras Piros
>Priority: Blocker
> Fix For: 5.1.0, 5.2.0
>
> Attachments: OOZIE-3389.001.patch, OOZIE-3389.002.patch
>
>
> I was testing 5.1 rc1. I submitted the cron-schedule example coordinator,
> clicked on one of the coordinator jobs on the Oozie Web UI, but the details
> of the job were not displayed. I noticed the following NPE in the server log:
> {code}
> 2018-11-21 15:45:00,019 ERROR CoordActionMissingDependenciesXCommand:517 -
> SERVER[Budapests-MacBook-Pro-10.local] USER[-] GROUP[-] TOKEN[-] APP[-]
> JOB[-] ACTION[-] XException,
> org.apache.oozie.command.CommandException: E1028: Coord input logic error.
> null
> at
> org.apache.oozie.command.coord.CoordActionMissingDependenciesXCommand.execute(CoordActionMissingDependenciesXCommand.java:126)
> at
> org.apache.oozie.command.coord.CoordActionMissingDependenciesXCommand.execute(CoordActionMissingDependenciesXCommand.java:38)
> at org.apache.oozie.command.XCommand.call(XCommand.java:291)
> at
> org.apache.oozie.CoordinatorEngine.getCoordActionMissingDependencies(CoordinatorEngine.java:910)
> at
> org.apache.oozie.servlet.V2JobServlet.getCoordActionMissingDependencies(V2JobServlet.java:336)
> at org.apache.oozie.servlet.BaseJobServlet.doGet(BaseJobServlet.java:370)
> at javax.servlet.http.HttpServlet.service(HttpServlet.java:687)
> at
> org.apache.oozie.servlet.JsonRestServlet.service(JsonRestServlet.java:305)
> at javax.servlet.http.HttpServlet.service(HttpServlet.java:790)
> at org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:848)
> at
> org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1772)
> at org.apache.oozie.servlet.AuthFilter$2.doFilter(AuthFilter.java:171)
> at
> org.apache.hadoop.security.authentication.server.AuthenticationFilter.doFilter(AuthenticationFilter.java:594)
> at
> org.apache.hadoop.security.authentication.server.AuthenticationFilter.doFilter(AuthenticationFilter.java:553)
> at org.apache.oozie.servlet.AuthFilter.doFilter(AuthFilter.java:176)
> at
> org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1759)
> at
> org.apache.oozie.servlet.HostnameFilter.doFilter(HostnameFilter.java:86)
> at
> org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1759)
> at
> org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:582)
> at
> org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:143)
> at
> org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:548)
> at
> org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:226)
> at
> org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1180)
> at
> org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:512)
> at
> org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:185)
> at
> org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1112)
> at
> org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141)
> at
> org.eclipse.jetty.server.handler.HandlerCollection.handle(HandlerCollection.java:119)
> at
> org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:134)
> at org.eclipse.jetty.server.Server.handle(Server.java:534)
> at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:320)
> at
> org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:251)
> at
> org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:283)
> at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:108)
> at
> org.eclipse.jetty.io.SelectChannelEndPoint$2.run(SelectChannelEndPoint.java:93)
> at
> org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.executeProduceConsume(ExecuteProduceConsume.java:303)
> at
> org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.produceConsume(ExecuteProduceConsume.java:148)
> at
> org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.run(ExecuteProduceConsume.java:136)
> {code}
--
This message was sent by Atla
[jira] [Updated] (OOZIE-3389) Getting input dependency list on the UI throws NPE
[
https://issues.apache.org/jira/browse/OOZIE-3389?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Andras Piros updated OOZIE-3389:
Attachment: OOZIE-3389.002.patch
> Getting input dependency list on the UI throws NPE
> --
>
> Key: OOZIE-3389
> URL: https://issues.apache.org/jira/browse/OOZIE-3389
> Project: Oozie
> Issue Type: Bug
>Affects Versions: 5.1.0
>Reporter: Attila Sasvari
>Assignee: Andras Piros
>Priority: Blocker
> Fix For: 5.1.0, 5.2.0
>
> Attachments: OOZIE-3389.001.patch, OOZIE-3389.002.patch
>
>
> I was testing 5.1 rc1. I submitted the cron-schedule example coordinator,
> clicked on one of the coordinator jobs on the Oozie Web UI, but the details
> of the job were not displayed. I noticed the following NPE in the server log:
> {code}
> 2018-11-21 15:45:00,019 ERROR CoordActionMissingDependenciesXCommand:517 -
> SERVER[Budapests-MacBook-Pro-10.local] USER[-] GROUP[-] TOKEN[-] APP[-]
> JOB[-] ACTION[-] XException,
> org.apache.oozie.command.CommandException: E1028: Coord input logic error.
> null
> at
> org.apache.oozie.command.coord.CoordActionMissingDependenciesXCommand.execute(CoordActionMissingDependenciesXCommand.java:126)
> at
> org.apache.oozie.command.coord.CoordActionMissingDependenciesXCommand.execute(CoordActionMissingDependenciesXCommand.java:38)
> at org.apache.oozie.command.XCommand.call(XCommand.java:291)
> at
> org.apache.oozie.CoordinatorEngine.getCoordActionMissingDependencies(CoordinatorEngine.java:910)
> at
> org.apache.oozie.servlet.V2JobServlet.getCoordActionMissingDependencies(V2JobServlet.java:336)
> at org.apache.oozie.servlet.BaseJobServlet.doGet(BaseJobServlet.java:370)
> at javax.servlet.http.HttpServlet.service(HttpServlet.java:687)
> at
> org.apache.oozie.servlet.JsonRestServlet.service(JsonRestServlet.java:305)
> at javax.servlet.http.HttpServlet.service(HttpServlet.java:790)
> at org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:848)
> at
> org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1772)
> at org.apache.oozie.servlet.AuthFilter$2.doFilter(AuthFilter.java:171)
> at
> org.apache.hadoop.security.authentication.server.AuthenticationFilter.doFilter(AuthenticationFilter.java:594)
> at
> org.apache.hadoop.security.authentication.server.AuthenticationFilter.doFilter(AuthenticationFilter.java:553)
> at org.apache.oozie.servlet.AuthFilter.doFilter(AuthFilter.java:176)
> at
> org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1759)
> at
> org.apache.oozie.servlet.HostnameFilter.doFilter(HostnameFilter.java:86)
> at
> org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1759)
> at
> org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:582)
> at
> org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:143)
> at
> org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:548)
> at
> org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:226)
> at
> org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1180)
> at
> org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:512)
> at
> org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:185)
> at
> org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1112)
> at
> org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141)
> at
> org.eclipse.jetty.server.handler.HandlerCollection.handle(HandlerCollection.java:119)
> at
> org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:134)
> at org.eclipse.jetty.server.Server.handle(Server.java:534)
> at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:320)
> at
> org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:251)
> at
> org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:283)
> at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:108)
> at
> org.eclipse.jetty.io.SelectChannelEndPoint$2.run(SelectChannelEndPoint.java:93)
> at
> org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.executeProduceConsume(ExecuteProduceConsume.java:303)
> at
> org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.produceConsume(ExecuteProduceConsume.java:148)
> at
> org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.run(ExecuteProduceConsume.java:136)
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (OOZIE-3390) shell action's stderr contains a bogus error message (since 5.0.0)
[
https://issues.apache.org/jira/browse/OOZIE-3390?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Andras Piros updated OOZIE-3390:
Attachment: OOZIE-3390.003.patch
> shell action's stderr contains a bogus error message (since 5.0.0)
> ---
>
> Key: OOZIE-3390
> URL: https://issues.apache.org/jira/browse/OOZIE-3390
> Project: Oozie
> Issue Type: Bug
>Affects Versions: 5.1.0
>Reporter: Attila Sasvari
>Assignee: Julia Kinga Marton
>Priority: Blocker
> Fix For: 5.1.0
>
> Attachments: OOZIE-3390-001.patch, OOZIE-3390-002.patch,
> OOZIE-3390.003.patch
>
>
> Shell action's stderr contains an error message about path.
> I executed shell action example to test Oozie 5.1 rc1, the job finished with
> success, but stderr of the job contained "Path echo doesn't appear to exist".
> https://github.com/apache/oozie/blob/27e4bf1688a6a7750b9c8454de5021337696fd61/sharelib/oozie/src/main/java/org/apache/oozie/action/hadoop/ShellContentWriter.java#L91
> {code}
> writeLine(errorStream, "Path " + filename + " doesn't appear
> to exist");
> {code}
> The problem is that we ignore the PATH of the process.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (OOZIE-3390) shell action's stderr contains a bogus error message (since 5.0.0)
[
https://issues.apache.org/jira/browse/OOZIE-3390?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16705940#comment-16705940
]
Andras Piros commented on OOZIE-3390:
-
[~asalamon74] unfortunately, {{command -v}} isn't available on every tested
platform, most notably, on GCE slaves' OSes that are used by {{dist_test}}. So
we're stuck with using {{which}} that's indeed everywhere available.
[~kmarton] [~asalamon74] please review patch 003.
> shell action's stderr contains a bogus error message (since 5.0.0)
> ---
>
> Key: OOZIE-3390
> URL: https://issues.apache.org/jira/browse/OOZIE-3390
> Project: Oozie
> Issue Type: Bug
>Affects Versions: 5.1.0
>Reporter: Attila Sasvari
>Assignee: Julia Kinga Marton
>Priority: Blocker
> Fix For: 5.1.0
>
> Attachments: OOZIE-3390-001.patch, OOZIE-3390-002.patch,
> OOZIE-3390.003.patch
>
>
> Shell action's stderr contains an error message about path.
> I executed shell action example to test Oozie 5.1 rc1, the job finished with
> success, but stderr of the job contained "Path echo doesn't appear to exist".
> https://github.com/apache/oozie/blob/27e4bf1688a6a7750b9c8454de5021337696fd61/sharelib/oozie/src/main/java/org/apache/oozie/action/hadoop/ShellContentWriter.java#L91
> {code}
> writeLine(errorStream, "Path " + filename + " doesn't appear
> to exist");
> {code}
> The problem is that we ignore the PATH of the process.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (OOZIE-3390) shell action's stderr contains a bogus error message (since 5.0.0)
[
https://issues.apache.org/jira/browse/OOZIE-3390?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16706948#comment-16706948
]
Andras Piros commented on OOZIE-3390:
-
[~asalamon74] thanks for the review!
Now checking also the return value of {{Process#waitFor(long, TimeUnit)}} to
know whether the process is still running. If it is, returning {{false}}. If it
has already been finished, returning {{exitValue}}. Also returning {{false}} in
case of any and all {{IllegalThreadStateException}} instances happening during
method calls.
> shell action's stderr contains a bogus error message (since 5.0.0)
> ---
>
> Key: OOZIE-3390
> URL: https://issues.apache.org/jira/browse/OOZIE-3390
> Project: Oozie
> Issue Type: Bug
>Affects Versions: 5.1.0
>Reporter: Attila Sasvari
>Assignee: Julia Kinga Marton
>Priority: Blocker
> Fix For: 5.1.0
>
> Attachments: OOZIE-3390-001.patch, OOZIE-3390-002.patch,
> OOZIE-3390.003.patch, OOZIE-3390.004.patch
>
>
> Shell action's stderr contains an error message about path.
> I executed shell action example to test Oozie 5.1 rc1, the job finished with
> success, but stderr of the job contained "Path echo doesn't appear to exist".
> https://github.com/apache/oozie/blob/27e4bf1688a6a7750b9c8454de5021337696fd61/sharelib/oozie/src/main/java/org/apache/oozie/action/hadoop/ShellContentWriter.java#L91
> {code}
> writeLine(errorStream, "Path " + filename + " doesn't appear
> to exist");
> {code}
> The problem is that we ignore the PATH of the process.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (OOZIE-3390) shell action's stderr contains a bogus error message (since 5.0.0)
[
https://issues.apache.org/jira/browse/OOZIE-3390?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Andras Piros updated OOZIE-3390:
Attachment: OOZIE-3390.004.patch
> shell action's stderr contains a bogus error message (since 5.0.0)
> ---
>
> Key: OOZIE-3390
> URL: https://issues.apache.org/jira/browse/OOZIE-3390
> Project: Oozie
> Issue Type: Bug
>Affects Versions: 5.1.0
>Reporter: Attila Sasvari
>Assignee: Julia Kinga Marton
>Priority: Blocker
> Fix For: 5.1.0
>
> Attachments: OOZIE-3390-001.patch, OOZIE-3390-002.patch,
> OOZIE-3390.003.patch, OOZIE-3390.004.patch
>
>
> Shell action's stderr contains an error message about path.
> I executed shell action example to test Oozie 5.1 rc1, the job finished with
> success, but stderr of the job contained "Path echo doesn't appear to exist".
> https://github.com/apache/oozie/blob/27e4bf1688a6a7750b9c8454de5021337696fd61/sharelib/oozie/src/main/java/org/apache/oozie/action/hadoop/ShellContentWriter.java#L91
> {code}
> writeLine(errorStream, "Path " + filename + " doesn't appear
> to exist");
> {code}
> The problem is that we ignore the PATH of the process.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (OOZIE-3396) AuthOozieClient leak memeory
[
https://issues.apache.org/jira/browse/OOZIE-3396?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16706999#comment-16706999
]
Andras Piros commented on OOZIE-3396:
-
Thanks for the ideas and the accurate root cause analysis [~zuston]! Now I see
the use case behind OOZIE-3379: you're using {{AuthOozieClient}} from inside a
server JVM, where it's practically never stopped / restarted. For that cases,
usage of {{File#deleteOnExit()}} is not recommended.
I suggest following points while delivering the needed functionality while
keeping commits / Jiras atomic:
# finish patch for OOZIE-3379 while using
{{currentToken.toString().equals(readToken.toString())}} instead of
{{currentToken.equals(readToken)}}. Create also unit tests that cover the
functionality change: when authentication URL is the same, auth token file
shouldn't be recreated or rewritten within its TTL
# in this Jira OOZIE-3396 create a mechanism that periodically purges auth
token cache files from file system and from the heap. Usage of Guava's
{{LoadingCache#removalListener()}} is [much
recommended|https://github.com/google/guava/wiki/CachesExplained]. I'd also
create unit / integration tests there, e.g. checking different combinations of
short- and long-running {{AuthOozieClient}} instances opening all variations of
same / different authentication URLs
> AuthOozieClient leak memeory
>
>
> Key: OOZIE-3396
> URL: https://issues.apache.org/jira/browse/OOZIE-3396
> Project: Oozie
> Issue Type: Bug
>Affects Versions: 5.0.0
>Reporter: Junfan Zhang
>Assignee: Junfan Zhang
>Priority: Major
>
> Phenomenon:
> Recently we upgraded the AuthOozieClient and added the OOZIE-2447 (Illegal
> character 0x0 oozie client) patch. When we were running online, the server
> memory usage continued to increase. Through the jmap and jhat command
> analysis, it was found that the deleteOnExist method in the writeAuthToken
> method under the AuthOozieClient class occupies a large amount of memory,
> which is caused by
> oozie-2447(https://issues.apache.org/jira/browse/OOZIE-2447).
> Also, because I have applied the oozie-3379 patch in the production
> environment. I still find that the .oozie-auth-token- file update time
> has changed, but the file content has not changed.
> After reviewing the code, I found that currentToken.equals(readToken) in the
> AuthOozieClient class is always false because the AuthenticatedURL.token does
> not have an equals method.
> The reason is because currentToken.equals(readToken) is always false, causing
> the tmp file to be created continuously, but there is a memory leak due to
> the deleteOnExist method
> (https://stackoverflow.com/questions/40119188/memory-leak-on-deleteonexithook)
>
> Solution:
> 1. In the AuthOozieClient, change to
> currentToken.toString().equals(readToken.toString())
> 2. In addition, the deleteOnExist method should be changed to the delete() in
> AuthOozieClient class.
> Doubt:
> Why delete the equals method in the AuthenticatedURL.Token class in Hadoop?
> See: https://issues.apache.org/jira/browse/HADOOP-10771
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (OOZIE-3390) [Shell action] STDERR contains a bogus error message
[
https://issues.apache.org/jira/browse/OOZIE-3390?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Andras Piros updated OOZIE-3390:
Summary: [Shell action] STDERR contains a bogus error message (was: shell
action's stderr contains a bogus error message (since 5.0.0))
> [Shell action] STDERR contains a bogus error message
>
>
> Key: OOZIE-3390
> URL: https://issues.apache.org/jira/browse/OOZIE-3390
> Project: Oozie
> Issue Type: Bug
>Affects Versions: 5.1.0
>Reporter: Attila Sasvari
>Assignee: Julia Kinga Marton
>Priority: Blocker
> Fix For: 5.1.0
>
> Attachments: OOZIE-3390-001.patch, OOZIE-3390-002.patch,
> OOZIE-3390.003.patch, OOZIE-3390.004.patch
>
>
> Shell action's stderr contains an error message about path.
> I executed shell action example to test Oozie 5.1 rc1, the job finished with
> success, but stderr of the job contained "Path echo doesn't appear to exist".
> https://github.com/apache/oozie/blob/27e4bf1688a6a7750b9c8454de5021337696fd61/sharelib/oozie/src/main/java/org/apache/oozie/action/hadoop/ShellContentWriter.java#L91
> {code}
> writeLine(errorStream, "Path " + filename + " doesn't appear
> to exist");
> {code}
> The problem is that we ignore the PATH of the process.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (OOZIE-3286) [spark-action] Launch Spark application from Oozie server
[
https://issues.apache.org/jira/browse/OOZIE-3286?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16707011#comment-16707011
]
Andras Piros commented on OOZIE-3286:
-
An interesting idea to support also [Spark on
Kubernetes|https://stackoverflow.com/questions/50311831/ozzie-with-spark-2-3-on-kubernetes]
using the very same mechanism as Spark on YARN.
> [spark-action] Launch Spark application from Oozie server
> -
>
> Key: OOZIE-3286
> URL: https://issues.apache.org/jira/browse/OOZIE-3286
> Project: Oozie
> Issue Type: New Feature
> Components: action, core
>Affects Versions: 5.0.0
>Reporter: Andras Piros
>Assignee: Andras Piros
>Priority: Major
>
> This is a major refactor / rework of the Spark action.
> Today Oozie Spark actions run as follows:
> # {{SparkActionExecutor extends JavaActionExecutor}} (running as part of
> Oozie server code) launches a {{SparkMain}} (running as YARN application) the
> usual way on YARN using {{LauncherAM}}
> # {{SparkMain}} runs on a YARN NodeManager container and calls from the same
> JVM
> [*{{SparkSubmit}}*|https://github.com/apache/spark/blob/master/core/src/main/scala/org/apache/spark/deploy/SparkSubmit.scala]
> # {{SparkSubmit}} fires up a Spark driver:
> ** in {{local}} or {{yarn-client}} mode in the same YARN NodeManager JVM
> where {{SparkMain}} runs. In {{yarn-client}} mode Spark's
> [*{{ApplicationMaster}}*|https://github.com/apache/spark/blob/master/resource-managers/yarn/src/main/scala/org/apache/spark/deploy/yarn/ApplicationMaster.scala]
> runs inside the same JVM where the driver runs
> ** [*in {{yarn-cluster}}
> mode*|https://spark.apache.org/docs/latest/running-on-yarn.html#launching-spark-on-yarn]
> Spark driver runs in a different JVM than Spark's ApplicationMaster, maybe
> on a different host. It can go away after the YARN application has been
> submitted
> Problems with this approach are:
> * too many levels of indirection cause lots of latency, and a whole new world
> of communication errors can happen
> * since {{SparkSubmit}} is launched from the same JVM where {{SparkMain}}
> runs, the Spark application will share all the environment variables,
> classpath, sharelib dependencies etc. with Oozie's Spark launcher code,
> causing hard-to-nail-down environment and classpath issues
> The future of Oozie Spark action launching looks like this:
> # {{SparkActionExecutor}} (running as part of Oozie server code) launches an
> [*{{InProcessLauncher}}*|https://github.com/apache/spark/blob/master/launcher/src/main/java/org/apache/spark/launcher/InProcessLauncher.java]
> (available since Spark 2.3) always in {{yarn-cluster}} mode
> # {{InProcessLauncher}} calls either {{InProcessSparkSubmit}} or
> {{SparkSubmit}}. We translate any Spark application modes to
> {{yarn-cluster}}, so that Spark driver will run in a JVM different than the
> Spark driver
> # this allows for much better resource usage, since we have one YARN
> ApplicationMaster (Oozie's {{LauncherAM}}) less
> # since the Spark driver and executor are always launched in different JVMs,
> we don't have any interference of environment variables, driver, or executor
> classpath
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (OOZIE-3390) [Shell action] STDERR contains a bogus error message
[
https://issues.apache.org/jira/browse/OOZIE-3390?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Andras Piros updated OOZIE-3390:
Component/s: action
> [Shell action] STDERR contains a bogus error message
>
>
> Key: OOZIE-3390
> URL: https://issues.apache.org/jira/browse/OOZIE-3390
> Project: Oozie
> Issue Type: Bug
> Components: action
>Affects Versions: 5.1.0
>Reporter: Attila Sasvari
>Assignee: Julia Kinga Marton
>Priority: Blocker
> Fix For: 5.1.0
>
> Attachments: OOZIE-3390-001.patch, OOZIE-3390-002.patch,
> OOZIE-3390.003.patch, OOZIE-3390.004.patch
>
>
> Shell action's stderr contains an error message about path.
> I executed shell action example to test Oozie 5.1 rc1, the job finished with
> success, but stderr of the job contained "Path echo doesn't appear to exist".
> https://github.com/apache/oozie/blob/27e4bf1688a6a7750b9c8454de5021337696fd61/sharelib/oozie/src/main/java/org/apache/oozie/action/hadoop/ShellContentWriter.java#L91
> {code}
> writeLine(errorStream, "Path " + filename + " doesn't appear
> to exist");
> {code}
> The problem is that we ignore the PATH of the process.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (OOZIE-3379) Auth token cache file name should include Oozie URL
[ https://issues.apache.org/jira/browse/OOZIE-3379?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16709912#comment-16709912 ] Andras Piros commented on OOZIE-3379: - Thanks [~zuston] for the new patch on ReviewBoard. +1 (pending pre-commit Jenkins run). Can you please also upload it here so that the pre-commit Jenkins job will be triggered again? Thanks! As having arranged on different caching / file removal options, these will be part of a separate Jira OOZIE-3396. > Auth token cache file name should include Oozie URL > --- > > Key: OOZIE-3379 > URL: https://issues.apache.org/jira/browse/OOZIE-3379 > Project: Oozie > Issue Type: Bug > Components: client >Affects Versions: 5.0.0 >Reporter: Junfan Zhang >Assignee: Junfan Zhang >Priority: Major > Attachments: oozie-3379-1.patch, oozie-3379-2.patch, > oozie-3379-3.patch, oozie-3379-4.patch, oozie-3379-5.patch, oozie-3379-6.patch > > > We have a program that uses the oozie client, but when the client connects to > multiple clusters, > the authOozieClient class frequently requests the kdc server because the > authentication token cache file is invalid. > This will cause subsequent requests in our program to be blocked, resulting > in unstable services. > So, oozie client's auth token cache file name should include Oozie URL. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Comment Edited] (OOZIE-3397) Improve logging in NotificationXCommand
[
https://issues.apache.org/jira/browse/OOZIE-3397?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16710066#comment-16710066
]
Andras Piros edited comment on OOZIE-3397 at 12/5/18 1:30 PM:
--
Thanks for the contribution [~kmarton]! Can you please also add a {{DEBUG}}
level message to know about the state or retries? Like {{Trying for the \{0}
time out of total {1}}}.
was (Author: andras.piros):
Thanks for the contribution [~kmarton]! Can you please also add a {{DEBUG}}
level message to know about the state or replies? Like \{{Trying for the \{0\}
time out of total \{1\}}}.
> Improve logging in NotificationXCommand
> ---
>
> Key: OOZIE-3397
> URL: https://issues.apache.org/jira/browse/OOZIE-3397
> Project: Oozie
> Issue Type: Improvement
>Affects Versions: trunk
>Reporter: Julia Kinga Marton
>Assignee: Julia Kinga Marton
>Priority: Major
> Attachments: OOZIE-3397.001.patch
>
>
> Around the notification sending (NotificationXCommand) there is not so much
> logging. For example if the HTTP call fails, the error is suppressed.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (OOZIE-3397) Improve logging in NotificationXCommand
[
https://issues.apache.org/jira/browse/OOZIE-3397?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16710066#comment-16710066
]
Andras Piros commented on OOZIE-3397:
-
Thanks for the contribution [~kmarton]! Can you please also add a {{DEBUG}}
level message to know about the state or replies? Like \{{Trying for the \{0\}
time out of total \{1\}}}.
> Improve logging in NotificationXCommand
> ---
>
> Key: OOZIE-3397
> URL: https://issues.apache.org/jira/browse/OOZIE-3397
> Project: Oozie
> Issue Type: Improvement
>Affects Versions: trunk
>Reporter: Julia Kinga Marton
>Assignee: Julia Kinga Marton
>Priority: Major
> Attachments: OOZIE-3397.001.patch
>
>
> Around the notification sending (NotificationXCommand) there is not so much
> logging. For example if the HTTP call fails, the error is suppressed.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Comment Edited] (OOZIE-3397) Improve logging in NotificationXCommand
[
https://issues.apache.org/jira/browse/OOZIE-3397?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16710066#comment-16710066
]
Andras Piros edited comment on OOZIE-3397 at 12/5/18 1:31 PM:
--
Thanks for the contribution [~kmarton]! Can you please also add a {{DEBUG}}
level message to know about the state or retries? Like {{Trying for the \{0\}
time out of total \{1\}}}.
was (Author: andras.piros):
Thanks for the contribution [~kmarton]! Can you please also add a {{DEBUG}}
level message to know about the state or retries? Like {{Trying for the \{0}
time out of total {1}}}.
> Improve logging in NotificationXCommand
> ---
>
> Key: OOZIE-3397
> URL: https://issues.apache.org/jira/browse/OOZIE-3397
> Project: Oozie
> Issue Type: Improvement
>Affects Versions: trunk
>Reporter: Julia Kinga Marton
>Assignee: Julia Kinga Marton
>Priority: Major
> Attachments: OOZIE-3397.001.patch
>
>
> Around the notification sending (NotificationXCommand) there is not so much
> logging. For example if the HTTP call fails, the error is suppressed.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (OOZIE-3379) Auth token cache file name should include Oozie URL
[
https://issues.apache.org/jira/browse/OOZIE-3379?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16710117#comment-16710117
]
Andras Piros commented on OOZIE-3379:
-
[~zuston] to address the one remaining FindBugs warning, can you please put
following annotation on the actual method (constructor) level
({{AuthOozieClient.java#L93-100}}):
{code:java}
@SuppressFBWarnings(value = "PATH_TRAVERSAL_IN", justification = "FilenameUtils
is used to filter user input. JDK8+ is used.")
{code}
Can you please also add following {{FilenameUtils#getName()}} call when
creating a new {{File}} given user input like this:
{code:java}
FilenameUtils.getFullPath(System.getProperty("user.home")) +
FilenameUtils.getName(filename)
{code}
> Auth token cache file name should include Oozie URL
> ---
>
> Key: OOZIE-3379
> URL: https://issues.apache.org/jira/browse/OOZIE-3379
> Project: Oozie
> Issue Type: Bug
> Components: client
>Affects Versions: 5.0.0
>Reporter: Junfan Zhang
>Assignee: Junfan Zhang
>Priority: Major
> Attachments: oozie-3379-1.patch, oozie-3379-2.patch,
> oozie-3379-3.patch, oozie-3379-4.patch, oozie-3379-5.patch,
> oozie-3379-6.patch, oozie-3379-7.patch
>
>
> We have a program that connects to multiple Oozie clusters with multiple
> {{AuthOozieClient}} instances, which frequently request the KDC server
> because the authentication token cache is invalid.
> After some investigation we found that the auth token cache file is
> incorrectly shared by all {{AuthOozieClient}} instances. Therefore, we
> propose that the auth token cache file name include Oozie URL as postfix.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (OOZIE-3397) Improve logging in NotificationXCommand
[ https://issues.apache.org/jira/browse/OOZIE-3397?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16710246#comment-16710246 ] Andras Piros commented on OOZIE-3397: - Thanks for the contribution [~kmarton]! +1 > Improve logging in NotificationXCommand > --- > > Key: OOZIE-3397 > URL: https://issues.apache.org/jira/browse/OOZIE-3397 > Project: Oozie > Issue Type: Improvement >Affects Versions: trunk >Reporter: Julia Kinga Marton >Assignee: Julia Kinga Marton >Priority: Major > Attachments: OOZIE-3397.001.patch, OOZIE-3397.002.patch > > > Around the notification sending (NotificationXCommand) there is not so much > logging. For example if the HTTP call fails, the error is suppressed. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (OOZIE-3397) Improve logging in NotificationXCommand
[ https://issues.apache.org/jira/browse/OOZIE-3397?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andras Piros updated OOZIE-3397: Component/s: core > Improve logging in NotificationXCommand > --- > > Key: OOZIE-3397 > URL: https://issues.apache.org/jira/browse/OOZIE-3397 > Project: Oozie > Issue Type: Improvement > Components: core >Affects Versions: trunk >Reporter: Julia Kinga Marton >Assignee: Julia Kinga Marton >Priority: Major > Attachments: OOZIE-3397.001.patch, OOZIE-3397.002.patch > > > Around the notification sending (NotificationXCommand) there is not so much > logging. For example if the HTTP call fails, the error is suppressed. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (OOZIE-3379) Auth token cache file name should include Oozie URL
[
https://issues.apache.org/jira/browse/OOZIE-3379?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16711343#comment-16711343
]
Andras Piros commented on OOZIE-3379:
-
Thanks for the contribution [~zuston]! +1
> Auth token cache file name should include Oozie URL
> ---
>
> Key: OOZIE-3379
> URL: https://issues.apache.org/jira/browse/OOZIE-3379
> Project: Oozie
> Issue Type: Bug
> Components: client
>Affects Versions: 5.0.0
>Reporter: Junfan Zhang
>Assignee: Junfan Zhang
>Priority: Major
> Attachments: oozie-3379-1.patch, oozie-3379-2.patch,
> oozie-3379-3.patch, oozie-3379-4.patch, oozie-3379-5.patch,
> oozie-3379-6.patch, oozie-3379-7.patch, oozie-3379-8.patch
>
>
> We have a program that connects to multiple Oozie clusters with multiple
> {{AuthOozieClient}} instances, which frequently request the KDC server
> because the authentication token cache is invalid.
> After some investigation we found that the auth token cache file is
> incorrectly shared by all {{AuthOozieClient}} instances. Therefore, we
> propose that the auth token cache file name include Oozie URL as postfix.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (OOZIE-3379) [client] Auth token cache file name should include OOZIE_URL
[
https://issues.apache.org/jira/browse/OOZIE-3379?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Andras Piros updated OOZIE-3379:
Summary: [client] Auth token cache file name should include OOZIE_URL
(was: Auth token cache file name should include Oozie URL)
> [client] Auth token cache file name should include OOZIE_URL
>
>
> Key: OOZIE-3379
> URL: https://issues.apache.org/jira/browse/OOZIE-3379
> Project: Oozie
> Issue Type: Bug
> Components: client
>Affects Versions: 5.0.0
>Reporter: Junfan Zhang
>Assignee: Junfan Zhang
>Priority: Major
> Attachments: oozie-3379-1.patch, oozie-3379-2.patch,
> oozie-3379-3.patch, oozie-3379-4.patch, oozie-3379-5.patch,
> oozie-3379-6.patch, oozie-3379-7.patch, oozie-3379-8.patch
>
>
> We have a program that connects to multiple Oozie clusters with multiple
> {{AuthOozieClient}} instances, which frequently request the KDC server
> because the authentication token cache is invalid.
> After some investigation we found that the auth token cache file is
> incorrectly shared by all {{AuthOozieClient}} instances. Therefore, we
> propose that the auth token cache file name include Oozie URL as postfix.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (OOZIE-3393) Add Oozie instrumentation delayed metric in CoordMaterializeTriggerService
[ https://issues.apache.org/jira/browse/OOZIE-3393?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andras Piros updated OOZIE-3393: Component/s: coordinator > Add Oozie instrumentation delayed metric in CoordMaterializeTriggerService > -- > > Key: OOZIE-3393 > URL: https://issues.apache.org/jira/browse/OOZIE-3393 > Project: Oozie > Issue Type: Improvement > Components: coordinator >Reporter: Junfan Zhang >Assignee: Junfan Zhang >Priority: Major > Attachments: oozie-3393-1.patch, oozie-3393-2.patch, > oozie-instrumentation-1.log > > > We have multiple Oozie scheduled clusters. The problem that is currently > encountered is that some materialize queues are too late to process during > the peak hours of the scheduled tasks. We need to get the metrics of the > delay queue for alarm monitoring and adjust the configuration to optimize. > Currently we have added metrics for queue size and latency in Oozie > CoordMaterializeTriggerService class, in order to reflect the general trend > of current scheduling. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (OOZIE-3393) Add Oozie instrumentation delayed metric in CoordMaterializeTriggerService
[
https://issues.apache.org/jira/browse/OOZIE-3393?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16711540#comment-16711540
]
Andras Piros commented on OOZIE-3393:
-
[~zuston] you're welcome. Patch 002 looks good to me thus far. One remaining
point being unit testing.
I'd create new test cases in {{TestCoordMaterializeTriggerService}} with
following semantics:
* setting things up and waiting till a task gets into
{{CoordinatorJob.Status.RUNNING}} [like
this|https://github.com/apache/oozie/blob/master/core/src/test/java/org/apache/oozie/service/TestCoordMaterializeTriggerService.java#L77-L84]
* asserting that the service was instrumented by accessing
{{Services.get().get(InstrumentationService).get().getVariables()}}
> Add Oozie instrumentation delayed metric in CoordMaterializeTriggerService
> --
>
> Key: OOZIE-3393
> URL: https://issues.apache.org/jira/browse/OOZIE-3393
> Project: Oozie
> Issue Type: Improvement
> Components: coordinator
>Reporter: Junfan Zhang
>Assignee: Junfan Zhang
>Priority: Major
> Attachments: oozie-3393-1.patch, oozie-3393-2.patch,
> oozie-instrumentation-1.log
>
>
> We have multiple Oozie scheduled clusters. The problem that is currently
> encountered is that some materialize queues are too late to process during
> the peak hours of the scheduled tasks. We need to get the metrics of the
> delay queue for alarm monitoring and adjust the configuration to optimize.
> Currently we have added metrics for queue size and latency in Oozie
> CoordMaterializeTriggerService class, in order to reflect the general trend
> of current scheduling.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (OOZIE-3301) Update NOTICE file
[
https://issues.apache.org/jira/browse/OOZIE-3301?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16711555#comment-16711555
]
Andras Piros commented on OOZIE-3301:
-
[~kmarton] best would be:
* using an automatic method, e.g. {{maven-license-plugin}}
* generating {{NOTICE.txt}} or, even better, {{NOTICE.md}}
* as one accumulated file for all the dependencies
* each dependency merged additively to the main {{NOTICE.md}} file
I think regenerating the {{NOTICE.md}} on every build is a burden, regenerating
it once every release as per {{bin/create-release-artifact}} would be OK.
[~asasvari] what do you think?
> Update NOTICE file
> --
>
> Key: OOZIE-3301
> URL: https://issues.apache.org/jira/browse/OOZIE-3301
> Project: Oozie
> Issue Type: Bug
>Affects Versions: 5.0.0
>Reporter: Attila Sasvari
>Assignee: Julia Kinga Marton
>Priority: Major
> Attachments: OOZIE-3301-001.patch, THIRD-PARTY.txt
>
>
> Oozie NOTICE file is not up to date. Please overview and update it as per
> http://www.apache.org/licenses/LICENSE-2.0.html#redistribution
> For example: Oozie uses embedded jetty, however Eclipse license
> (http://www.eclipse.org/jetty/documentation/9.3.x/embedding-jetty.html) is
> not listed in
> [NOTICE.txt|https://github.com/apache/oozie/blob/a299d4a6d435a2c92cd1d0ffce7f35a2ef8d639b/NOTICE.txt#L9].
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (OOZIE-3399) [tests] Eliminate nested class in TestV1JobsServletBundleEngine and TestV1JobServletBundleEngine
[
https://issues.apache.org/jira/browse/OOZIE-3399?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Andras Piros updated OOZIE-3399:
Summary: [tests] Eliminate nested class in TestV1JobsServletBundleEngine
and TestV1JobServletBundleEngine (was: Eliminate nested class in
TestV1JobsServletBundleEngine and TestV1JobServletBundleEngine)
> [tests] Eliminate nested class in TestV1JobsServletBundleEngine and
> TestV1JobServletBundleEngine
>
>
> Key: OOZIE-3399
> URL: https://issues.apache.org/jira/browse/OOZIE-3399
> Project: Oozie
> Issue Type: Improvement
> Components: tests
>Affects Versions: trunk
>Reporter: Andras Salamon
>Assignee: Andras Salamon
>Priority: Major
> Fix For: 5.2.0
>
> Attachments: OOZIE-3399-01.patch
>
>
> There is a useless nested class in {{TestV1JobsServletBundleEngine}} and
> {{TestV1JobServletBundleEngine}}:
> {noformat}
> /**
> * This class is needed in order to reuse some methods of class {@link
> XDataTestCase}. We cannot directly extend it there as
> * we extend {@link DagServletTestCase}. Anonymous inner class is also
> not an option since we cannot assign it an annotation.
> * The @Ignore annotation is needed to prevent JUnit from recognizing
> this inner class as a test.
> */
> @Ignore
> private static class XDataTestCase1 extends XDataTestCase {
> }
> {noformat}
> The comment is no longer relevant, the classes extend {{DagServletTestCase}}
> which extends {{XDataTestCase}}, no need for the nested class.
> I'm not able to reproduce it, but I find more than suspicious that different
> tests fail in the precommit unit tests right after the unit test runs (and
> ignores) the {{XDataTestCase1}} nested class. Maybe the double initialization
> of {{XDataTestCase}} causes some problems:
> [https://builds.apache.org/job/PreCommit-OOZIE-Build/938/consoleFull]
> [https://builds.apache.org/job/PreCommit-OOZIE-Build/940/consoleFull]
> [https://builds.apache.org/job/PreCommit-OOZIE-Build/941/consoleFull]
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (OOZIE-3399) [tests] Eliminate nested class in TestV1JobsServletBundleEngine and TestV1JobServletBundleEngine
[
https://issues.apache.org/jira/browse/OOZIE-3399?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16711826#comment-16711826
]
Andras Piros commented on OOZIE-3399:
-
Thanks for the contribution [~asalamon74]! +1
> [tests] Eliminate nested class in TestV1JobsServletBundleEngine and
> TestV1JobServletBundleEngine
>
>
> Key: OOZIE-3399
> URL: https://issues.apache.org/jira/browse/OOZIE-3399
> Project: Oozie
> Issue Type: Improvement
> Components: tests
>Affects Versions: trunk
>Reporter: Andras Salamon
>Assignee: Andras Salamon
>Priority: Major
> Fix For: 5.2.0
>
> Attachments: OOZIE-3399-01.patch
>
>
> There is a useless nested class in {{TestV1JobsServletBundleEngine}} and
> {{TestV1JobServletBundleEngine}}:
> {noformat}
> /**
> * This class is needed in order to reuse some methods of class {@link
> XDataTestCase}. We cannot directly extend it there as
> * we extend {@link DagServletTestCase}. Anonymous inner class is also
> not an option since we cannot assign it an annotation.
> * The @Ignore annotation is needed to prevent JUnit from recognizing
> this inner class as a test.
> */
> @Ignore
> private static class XDataTestCase1 extends XDataTestCase {
> }
> {noformat}
> The comment is no longer relevant, the classes extend {{DagServletTestCase}}
> which extends {{XDataTestCase}}, no need for the nested class.
> I'm not able to reproduce it, but I find more than suspicious that different
> tests fail in the precommit unit tests right after the unit test runs (and
> ignores) the {{XDataTestCase1}} nested class. Maybe the double initialization
> of {{XDataTestCase}} causes some problems:
> [https://builds.apache.org/job/PreCommit-OOZIE-Build/938/consoleFull]
> [https://builds.apache.org/job/PreCommit-OOZIE-Build/940/consoleFull]
> [https://builds.apache.org/job/PreCommit-OOZIE-Build/941/consoleFull]
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (OOZIE-3398) [docs] Fix oozie sub-workflow documentation
[
https://issues.apache.org/jira/browse/OOZIE-3398?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16712567#comment-16712567
]
Andras Piros commented on OOZIE-3398:
-
Thanks for the contribution [~asalamon74]! It's good to see you spot also
subtle documentation anomalies like that :) +1
> [docs] Fix oozie sub-workflow documentation
> ---
>
> Key: OOZIE-3398
> URL: https://issues.apache.org/jira/browse/OOZIE-3398
> Project: Oozie
> Issue Type: Improvement
> Components: docs
>Affects Versions: trunk
>Reporter: Andras Salamon
>Assignee: Andras Salamon
>Priority: Minor
> Fix For: 5.2.0
>
> Attachments: OOZIE-3398-01.patch
>
>
> Workflow functional spec
> [documentation|https://github.com/apache/oozie/blob/master/docs/src/site/markdown/WorkflowFunctionalSpec.md]
> has inconsistent information about the sub workflows which should be fixed:
> {noformat}
> ...the child workflow job can be in the same Oozie system or in another Oozie
> system
> {noformat}
> {noformat}
> The child workflow job runs in the same Oozie system instance where the
> parent workflow job is running.
> {noformat}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Created] (OOZIE-3402) [client] Error message for missing workflow definition contains error code duplication
Andras Piros created OOZIE-3402:
---
Summary: [client] Error message for missing workflow definition
contains error code duplication
Key: OOZIE-3402
URL: https://issues.apache.org/jira/browse/OOZIE-3402
Project: Oozie
Issue Type: Bug
Components: client
Affects Versions: 5.1.0
Reporter: Andras Piros
When a workflow definition is not present on HDFS, and the property
{{oozie.jobs.api.generated.xml}} is not present in the HTTP request sent to
{{V2JobServlet}} by {{OozieCLI}}, following error message is displayed:
{noformat}
Error: E0307 : E0307: Runtime error [App directory
[hdfs://localhost:9000/user/root/examples/apps/subwf] does not exist and app
definition cannot be created because of missing config value
[oozie.jobs.api.generated.xml]
{noformat}
The error code {{E0307}} is displayed two times, need to remove one.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Created] (OOZIE-3403) [fluent-job] Workflow definition is stored in an insecure place on client host
Andras Piros created OOZIE-3403:
---
Summary: [fluent-job] Workflow definition is stored in an insecure
place on client host
Key: OOZIE-3403
URL: https://issues.apache.org/jira/browse/OOZIE-3403
Project: Oozie
Issue Type: Bug
Components: fluent-job
Affects Versions: 5.1.0
Reporter: Andras Piros
When {{OozieCLI}} is called with {{job -validatejar}} and {{–-verbose}}
options, the resulting {{workflow.xml}} is stored in an insecure place:
{{/tmp}} on the host computer.
To reduce world readability, it's required that the resulting {{workflow.xml}}
be stored in the currend working directory with rights only readable to the
caller where {{OozieCLI}} has just been called.
Since this information is also available via normal [{{OozieCLI}} call {{job
-definition}}|https://oozie.apache.org/docs/5.0.0/DG_CommandLineTool.html#Checking_the_xml_definition_of_a_Workflow_Coordinator_or_Bundle_Job],
it's considered a minor issue.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (OOZIE-3401) TestPySpark failure
[
https://issues.apache.org/jira/browse/OOZIE-3401?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16722074#comment-16722074
]
Andras Piros commented on OOZIE-3401:
-
[~kmarton] we need another patch based on the fact [~asalamon74] mentioned,
right?
> TestPySpark failure
> ---
>
> Key: OOZIE-3401
> URL: https://issues.apache.org/jira/browse/OOZIE-3401
> Project: Oozie
> Issue Type: Bug
> Components: tests
>Affects Versions: trunk
>Reporter: Andras Salamon
>Assignee: Andras Salamon
>Priority: Major
> Fix For: 5.2.0
>
> Attachments: OOZIE-3401-01.patch
>
>
> [TestPySpark|https://github.com/apache/oozie/blob/master/sharelib/spark/src/test/java/org/apache/oozie/action/hadoop/TestPyspark.java]
> fails with the following error:
> {noformat}[INFO] Running org.apache.oozie.action.hadoop.TestPyspark
> [ERROR] Tests run: 1, Failures: 1, Errors: 0, Skipped: 0, Time elapsed:
> 131.97 s <<< FAILURE! - in org.apache.oozie.action.hadoop.TestPyspark
> [ERROR] testPyspark(org.apache.oozie.action.hadoop.TestPyspark) Time
> elapsed: 131.97 s <<< FAILURE!
> junit.framework.ComparisonFailure: expected:<[SUCCEED]ED> but
> was:<[FAILED/KILL]ED>
> at junit.framework.Assert.assertEquals(Assert.java:100)
> at junit.framework.Assert.assertEquals(Assert.java:107)
>
> at junit.framework.TestCase.assertEquals(TestCase.java:269)
> at
> org.apache.oozie.action.hadoop.TestPyspark.testPysparkHelper(TestPyspark.java:107)
> at
> org.apache.oozie.action.hadoop.TestPyspark.testPyspark(TestPyspark.java:85)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
>
> at junit.framework.TestCase.runTest(TestCase.java:176)
>
> at junit.framework.TestCase.runBare(TestCase.java:141)
>
> at junit.framework.TestResult$1.protect(TestResult.java:122)
>
> at junit.framework.TestResult.runProtected(TestResult.java:142)
>
> at junit.framework.TestResult.run(TestResult.java:125)
>
> at junit.framework.TestCase.run(TestCase.java:129)
>
> at junit.framework.TestSuite.runTest(TestSuite.java:255)
>
> at junit.framework.TestSuite.run(TestSuite.java:250)
>
> at
> org.junit.internal.runners.JUnit38ClassRunner.run(JUnit38ClassRunner.java:84)
> at org.junit.runners.Suite.runChild(Suite.java:127)
>
> at org.junit.runners.Suite.runChild(Suite.java:26)
>
> at org.junit.runners.ParentRunner$3.run(ParentRunner.java:238)
> at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:63)
> at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:236)
> at org.junit.runners.ParentRunner.access$000(ParentRunner.java:53)
> at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:229)
> at org.junit.runners.ParentRunner.run(ParentRunner.java:309)
> at org.junit.runners.Suite.runChild(Suite.java:127)
> at org.junit.runners.Suite.runChild(Suite.java:26)
> at org.junit.runners.ParentRunner$3.run(ParentRunner.java:238)
> at
> org.apache.maven.surefire.junitcore.pc.Scheduler$1.run(Scheduler.java:410)
> at
> org.apache.maven.surefire.junitcore.pc.InvokerStrategy.schedule(InvokerStrategy.java:54)
> at
> org.apache.maven.surefire.junitcore.pc.Scheduler.schedule(Scheduler.java:367)
> at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:236)
> at org.junit.runners.ParentRunner.access$000(ParentRunner.java:53)
> at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:229)
>
> at org.junit.runners.ParentRunner.run(ParentRunner.java:309)
>
>
> at
> org.apache.maven.surefire.junitcore.pc.ParallelComputerBuilder$PC$1.run(ParallelComputerBuilder.java:593)
> at
> org.apache.maven.surefire.junitcore.JUnitCore.run(JUnitCore.java:55)
>
> at
> org.apache.maven.surefire.junitcore.JUnitCoreWrapper.createRequestAndRun(JUnitCoreWrapper.java:137)
>
> at
> org.apache.maven.surefire.junitcore.JUnitCoreWrapper.executeEager(JUnitCoreWrapper.java:107)
> at
> org.apache.maven.sure
[jira] [Commented] (OOZIE-3397) Improve logging in NotificationXCommand
[ https://issues.apache.org/jira/browse/OOZIE-3397?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16722185#comment-16722185 ] Andras Piros commented on OOZIE-3397: - Thanks [~asalamon74] for the amendment patch 001 and the test cases! Wherever it makes sense, can you please cover existing manual test cases w/ unit tests as well? Can you please test manually a few more scenarios (I think it's not applicable to test those automatically): * HTTPS connection, server certificate has been expired * HTTPS connection, server certificate is not trusted * HTTPS connection, self-signed server certificate that isn't part > Improve logging in NotificationXCommand > --- > > Key: OOZIE-3397 > URL: https://issues.apache.org/jira/browse/OOZIE-3397 > Project: Oozie > Issue Type: Improvement > Components: core >Affects Versions: trunk >Reporter: Julia Kinga Marton >Assignee: Andras Salamon >Priority: Major > Fix For: 5.2.0 > > Attachments: OOZIE-3397-amend-01-01.patch, OOZIE-3397.001.patch, > OOZIE-3397.002.patch > > > Around the notification sending (NotificationXCommand) there is not so much > logging. For example if the HTTP call fails, the error is suppressed. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Comment Edited] (OOZIE-3397) Improve logging in NotificationXCommand
[ https://issues.apache.org/jira/browse/OOZIE-3397?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16722185#comment-16722185 ] Andras Piros edited comment on OOZIE-3397 at 12/15/18 3:13 PM: --- Thanks [~asalamon74] for the amendment patch 001 and the test cases! Wherever it makes sense, can you please cover existing manual test cases w/ unit tests as well? Can you please test manually a few more scenarios (I think it's not applicable to test those automatically): * HTTPS connection, server certificate has been expired * HTTPS connection, server certificate is not trusted by Oozie * HTTPS connection, server and Oozie don't have any cipher suites in common was (Author: andras.piros): Thanks [~asalamon74] for the amendment patch 001 and the test cases! Wherever it makes sense, can you please cover existing manual test cases w/ unit tests as well? Can you please test manually a few more scenarios (I think it's not applicable to test those automatically): * HTTPS connection, server certificate has been expired * HTTPS connection, server certificate is not trusted * HTTPS connection, self-signed server certificate that isn't part > Improve logging in NotificationXCommand > --- > > Key: OOZIE-3397 > URL: https://issues.apache.org/jira/browse/OOZIE-3397 > Project: Oozie > Issue Type: Improvement > Components: core >Affects Versions: trunk >Reporter: Julia Kinga Marton >Assignee: Andras Salamon >Priority: Major > Fix For: 5.2.0 > > Attachments: OOZIE-3397-amend-01-01.patch, OOZIE-3397.001.patch, > OOZIE-3397.002.patch > > > Around the notification sending (NotificationXCommand) there is not so much > logging. For example if the HTTP call fails, the error is suppressed. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (OOZIE-3397) Improve logging in NotificationXCommand
[ https://issues.apache.org/jira/browse/OOZIE-3397?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16722238#comment-16722238 ] Andras Piros commented on OOZIE-3397: - [~asalamon74] thanks for testing also those extended HTTPS scenarios manually! Do you think whether it makes sense to cover existing manual test cases w/ unit tests as well? > Improve logging in NotificationXCommand > --- > > Key: OOZIE-3397 > URL: https://issues.apache.org/jira/browse/OOZIE-3397 > Project: Oozie > Issue Type: Improvement > Components: core >Affects Versions: trunk >Reporter: Julia Kinga Marton >Assignee: Andras Salamon >Priority: Major > Fix For: 5.2.0 > > Attachments: OOZIE-3397-amend-01-01.patch, OOZIE-3397.001.patch, > OOZIE-3397.002.patch > > > Around the notification sending (NotificationXCommand) there is not so much > logging. For example if the HTTP call fails, the error is suppressed. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (OOZIE-3397) Improve logging in NotificationXCommand
[ https://issues.apache.org/jira/browse/OOZIE-3397?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16723083#comment-16723083 ] Andras Piros commented on OOZIE-3397: - I'm convinced [~asalamon74] we already have the necessary amount of automatic testing on that part of functionality. +1 (pending Jenkins job for amendment patch 01) > Improve logging in NotificationXCommand > --- > > Key: OOZIE-3397 > URL: https://issues.apache.org/jira/browse/OOZIE-3397 > Project: Oozie > Issue Type: Improvement > Components: core >Affects Versions: trunk >Reporter: Julia Kinga Marton >Assignee: Andras Salamon >Priority: Major > Fix For: 5.2.0 > > Attachments: OOZIE-3397-amend-01-01.patch, OOZIE-3397.001.patch, > OOZIE-3397.002.patch > > > Around the notification sending (NotificationXCommand) there is not so much > logging. For example if the HTTP call fails, the error is suppressed. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
