[jira] [Commented] (OOZIE-1581) Workflow performance optimizations
[
https://issues.apache.org/jira/browse/OOZIE-1581?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14119480#comment-14119480
]
Shwetha G S commented on OOZIE-1581:
[~puru], thanks for checking this
{quote}
And i think that is a correct behavior, rerunning of WF haven't triggered
rerunning of coord action. Running for coord action does lot of other things
beside submitting new WF like input dependency check, coord definition update (
if needed), delete output ( if needed). So it may not be correct to say coord
action has rerun if we rerun WF.
{quote}
Isn't workflow re-run same as coord action re-run with no refresh, no
cleanup(which are anyway options in coord action re-run)? Its just that it
doesn't handle concurrency check, but nothing else changes.
Assuming that the workflow and coord action are killed initially, and the user
re-runs the workflow, workflow moves to running state, but coord action remains
in killed state. Finally, if the workflow succeeds, we change the coord action
to succeeded(yes, we do this already). So, the coord action changes from killed
to succeeded. So, my question is why not update the coord action status when
workflow is running as well?
Workflow performance optimizations
--
Key: OOZIE-1581
URL: https://issues.apache.org/jira/browse/OOZIE-1581
Project: Oozie
Issue Type: Improvement
Components: workflow
Affects Versions: trunk
Reporter: Mona Chitnis
Assignee: Mona Chitnis
Fix For: 4.1.0
Attachments: OOZIE-1581-2.patch, OOZIE-1581-final.patch,
OOZIE-1581-r6.patch, OOZIE-1581-r6.patch, OOZIE-1581-r7.patch,
OOZIE-1581.patch
Creating a combo JIRA for small performance optimizations.
1. changing from asynchronous action start to a synchronous one, to overcome
the undue delay in transitioning from ::start:: control node to the actual
first node, owing to a loaded queue. This delay has been observed to be close
to 30 min at times in stress conditions.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (OOZIE-1581) Workflow performance optimizations
[ https://issues.apache.org/jira/browse/OOZIE-1581?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14118916#comment-14118916 ] Purshotam Shah commented on OOZIE-1581: --- This is how the whole framework is. If we rerun a coord from a completed bundle, bundle status doesn't change. To change bundle status we have to rerun bundle with coord as param, then only bundle status will change. Same as with WF and coord action. If we rerun a WF, coord action status shouldn't change. And i think that is a correct behavior, rerunning of WF haven't triggered rerunning of coord action. Running for coord action does lot of other things beside submitting new WF like input dependency check, coord definition update ( if needed), delete output ( if needed). So it may not be correct to say coord action has rerun if we rerun WF. Workflow performance optimizations -- Key: OOZIE-1581 URL: https://issues.apache.org/jira/browse/OOZIE-1581 Project: Oozie Issue Type: Improvement Components: workflow Affects Versions: trunk Reporter: Mona Chitnis Assignee: Mona Chitnis Fix For: 4.1.0 Attachments: OOZIE-1581-2.patch, OOZIE-1581-final.patch, OOZIE-1581-r6.patch, OOZIE-1581-r6.patch, OOZIE-1581-r7.patch, OOZIE-1581.patch Creating a combo JIRA for small performance optimizations. 1. changing from asynchronous action start to a synchronous one, to overcome the undue delay in transitioning from ::start:: control node to the actual first node, owing to a loaded queue. This delay has been observed to be close to 30 min at times in stress conditions. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (OOZIE-1581) Workflow performance optimizations
[
https://issues.apache.org/jira/browse/OOZIE-1581?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14108996#comment-14108996
]
Shwetha G S commented on OOZIE-1581:
In SignalXCommand:
{code}
// Changing to synchronous call from asynchronous queuing to prevent
// undue delay from between end of previous and start of next action
if (wfJob.getStatus() != WorkflowJob.Status.RUNNING
wfJob.getStatus() != WorkflowJob.Status.SUSPENDED) {
// only for asynchronous actions, parent coord action's external id
will
// persisted and following update will succeed.
updateParentIfNecessary(wfJob);
new WfEndXCommand(wfJob).call(); // To delete the WF temp dir
}
{code}
[~chitnis], why is there check on status while updating parent?
For the usecase where coord action and the workflow is killed, a re-run of the
workflow moves the status to running, but coord action remains in killed state
because of this?
Workflow performance optimizations
--
Key: OOZIE-1581
URL: https://issues.apache.org/jira/browse/OOZIE-1581
Project: Oozie
Issue Type: Improvement
Components: workflow
Affects Versions: trunk
Reporter: Mona Chitnis
Assignee: Mona Chitnis
Fix For: 4.1.0
Attachments: OOZIE-1581-2.patch, OOZIE-1581-final.patch,
OOZIE-1581-r6.patch, OOZIE-1581-r6.patch, OOZIE-1581-r7.patch,
OOZIE-1581.patch
Creating a combo JIRA for small performance optimizations.
1. changing from asynchronous action start to a synchronous one, to overcome
the undue delay in transitioning from ::start:: control node to the actual
first node, owing to a loaded queue. This delay has been observed to be close
to 30 min at times in stress conditions.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (OOZIE-1581) Workflow performance optimizations
[ https://issues.apache.org/jira/browse/OOZIE-1581?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13873761#comment-13873761 ] Rohini Palaniswamy commented on OOZIE-1581: --- +1 Workflow performance optimizations -- Key: OOZIE-1581 URL: https://issues.apache.org/jira/browse/OOZIE-1581 Project: Oozie Issue Type: Improvement Components: workflow Affects Versions: trunk Reporter: Mona Chitnis Assignee: Mona Chitnis Fix For: trunk Attachments: OOZIE-1581-2.patch, OOZIE-1581-final.patch, OOZIE-1581-r6.patch, OOZIE-1581-r6.patch, OOZIE-1581-r7.patch, OOZIE-1581.patch Creating a combo JIRA for small performance optimizations. 1. changing from asynchronous action start to a synchronous one, to overcome the undue delay in transitioning from ::start:: control node to the actual first node, owing to a loaded queue. This delay has been observed to be close to 30 min at times in stress conditions. -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Commented] (OOZIE-1581) Workflow performance optimizations
[
https://issues.apache.org/jira/browse/OOZIE-1581?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13873100#comment-13873100
]
Hadoop QA commented on OOZIE-1581:
--
Testing JIRA OOZIE-1581
Cleaning local git workspace
{color:green}+1 PATCH_APPLIES{color}
{color:green}+1 CLEAN{color}
{color:green}+1 RAW_PATCH_ANALYSIS{color}
.{color:green}+1{color} the patch does not introduce any @author tags
.{color:green}+1{color} the patch does not introduce any tabs
.{color:green}+1{color} the patch does not introduce any trailing spaces
.{color:green}+1{color} the patch does not introduce any line longer than
132
.{color:green}+1{color} the patch does adds/modifies 13 testcase(s)
{color:green}+1 RAT{color}
.{color:green}+1{color} the patch does not seem to introduce new RAT
warnings
{color:green}+1 JAVADOC{color}
.{color:green}+1{color} the patch does not seem to introduce new Javadoc
warnings
{color:green}+1 COMPILE{color}
.{color:green}+1{color} HEAD compiles
.{color:green}+1{color} patch compiles
.{color:green}+1{color} the patch does not seem to introduce new javac
warnings
{color:green}+1 BACKWARDS_COMPATIBILITY{color}
.{color:green}+1{color} the patch does not change any JPA
Entity/Colum/Basic/Lob/Transient annotations
.{color:green}+1{color} the patch does not modify JPA files
{color:red}-1 TESTS{color}
.Tests run: 1384
.Tests failed: 0
.Tests errors: 2
.The patch failed the following testcases:
.
{color:green}+1 DISTRO{color}
.{color:green}+1{color} distro tarball builds with the patch
{color:red}*-1 Overall result, please check the reported -1(s)*{color}
The full output of the test-patch run is available at
. https://builds.apache.org/job/oozie-trunk-precommit-build/1013/
Workflow performance optimizations
--
Key: OOZIE-1581
URL: https://issues.apache.org/jira/browse/OOZIE-1581
Project: Oozie
Issue Type: Improvement
Components: workflow
Affects Versions: trunk
Reporter: Mona Chitnis
Assignee: Mona Chitnis
Fix For: trunk
Attachments: OOZIE-1581-2.patch, OOZIE-1581-final.patch,
OOZIE-1581-r6.patch, OOZIE-1581-r6.patch, OOZIE-1581-r7.patch,
OOZIE-1581.patch
Creating a combo JIRA for small performance optimizations.
1. changing from asynchronous action start to a synchronous one, to overcome
the undue delay in transitioning from ::start:: control node to the actual
first node, owing to a loaded queue. This delay has been observed to be close
to 30 min at times in stress conditions.
--
This message was sent by Atlassian JIRA
(v6.1.5#6160)
[jira] [Commented] (OOZIE-1581) Workflow performance optimizations
[
https://issues.apache.org/jira/browse/OOZIE-1581?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13865906#comment-13865906
]
Hadoop QA commented on OOZIE-1581:
--
Testing JIRA OOZIE-1581
Cleaning local git workspace
{color:green}+1 PATCH_APPLIES{color}
{color:green}+1 CLEAN{color}
{color:green}+1 RAW_PATCH_ANALYSIS{color}
.{color:green}+1{color} the patch does not introduce any @author tags
.{color:green}+1{color} the patch does not introduce any tabs
.{color:green}+1{color} the patch does not introduce any trailing spaces
.{color:green}+1{color} the patch does not introduce any line longer than
132
.{color:green}+1{color} the patch does adds/modifies 13 testcase(s)
{color:green}+1 RAT{color}
.{color:green}+1{color} the patch does not seem to introduce new RAT
warnings
{color:green}+1 JAVADOC{color}
.{color:green}+1{color} the patch does not seem to introduce new Javadoc
warnings
{color:green}+1 COMPILE{color}
.{color:green}+1{color} HEAD compiles
.{color:green}+1{color} patch compiles
.{color:green}+1{color} the patch does not seem to introduce new javac
warnings
{color:green}+1 BACKWARDS_COMPATIBILITY{color}
.{color:green}+1{color} the patch does not change any JPA
Entity/Colum/Basic/Lob/Transient annotations
.{color:green}+1{color} the patch does not modify JPA files
{color:red}-1 TESTS{color}
.Tests run: 1382
.Tests failed: 0
.Tests errors: 2
.The patch failed the following testcases:
.
{color:green}+1 DISTRO{color}
.{color:green}+1{color} distro tarball builds with the patch
{color:red}*-1 Overall result, please check the reported -1(s)*{color}
The full output of the test-patch run is available at
. https://builds.apache.org/job/oozie-trunk-precommit-build/993/
Workflow performance optimizations
--
Key: OOZIE-1581
URL: https://issues.apache.org/jira/browse/OOZIE-1581
Project: Oozie
Issue Type: Improvement
Components: workflow
Affects Versions: trunk
Reporter: Mona Chitnis
Assignee: Mona Chitnis
Fix For: trunk
Attachments: OOZIE-1581-2.patch, OOZIE-1581-final.patch,
OOZIE-1581-r6.patch, OOZIE-1581-r6.patch, OOZIE-1581.patch
Creating a combo JIRA for small performance optimizations.
1. changing from asynchronous action start to a synchronous one, to overcome
the undue delay in transitioning from ::start:: control node to the actual
first node, owing to a loaded queue. This delay has been observed to be close
to 30 min at times in stress conditions.
--
This message was sent by Atlassian JIRA
(v6.1.5#6160)
[jira] [Commented] (OOZIE-1581) Workflow performance optimizations
[
https://issues.apache.org/jira/browse/OOZIE-1581?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13865025#comment-13865025
]
Hadoop QA commented on OOZIE-1581:
--
Testing JIRA OOZIE-1581
Cleaning local git workspace
{color:green}+1 PATCH_APPLIES{color}
{color:green}+1 CLEAN{color}
{color:red}-1 RAW_PATCH_ANALYSIS{color}
.{color:green}+1{color} the patch does not introduce any @author tags
.{color:green}+1{color} the patch does not introduce any tabs
.{color:green}+1{color} the patch does not introduce any trailing spaces
.{color:red}-1{color} the patch contains 1 line(s) longer than 132
characters
.{color:green}+1{color} the patch does adds/modifies 13 testcase(s)
{color:green}+1 RAT{color}
.{color:green}+1{color} the patch does not seem to introduce new RAT
warnings
{color:green}+1 JAVADOC{color}
.{color:green}+1{color} the patch does not seem to introduce new Javadoc
warnings
{color:green}+1 COMPILE{color}
.{color:green}+1{color} HEAD compiles
.{color:green}+1{color} patch compiles
.{color:green}+1{color} the patch does not seem to introduce new javac
warnings
{color:green}+1 BACKWARDS_COMPATIBILITY{color}
.{color:green}+1{color} the patch does not change any JPA
Entity/Colum/Basic/Lob/Transient annotations
.{color:green}+1{color} the patch does not modify JPA files
{color:red}-1 TESTS{color}
.Tests run: 1382
.Tests failed: 0
.Tests errors: 1
.The patch failed the following testcases:
.
{color:green}+1 DISTRO{color}
.{color:green}+1{color} distro tarball builds with the patch
{color:red}*-1 Overall result, please check the reported -1(s)*{color}
The full output of the test-patch run is available at
. https://builds.apache.org/job/oozie-trunk-precommit-build/989/
Workflow performance optimizations
--
Key: OOZIE-1581
URL: https://issues.apache.org/jira/browse/OOZIE-1581
Project: Oozie
Issue Type: Improvement
Components: workflow
Affects Versions: trunk
Reporter: Mona Chitnis
Assignee: Mona Chitnis
Fix For: trunk
Attachments: OOZIE-1581-2.patch, OOZIE-1581-final.patch,
OOZIE-1581-r6.patch, OOZIE-1581.patch
Creating a combo JIRA for small performance optimizations.
1. changing from asynchronous action start to a synchronous one, to overcome
the undue delay in transitioning from ::start:: control node to the actual
first node, owing to a loaded queue. This delay has been observed to be close
to 30 min at times in stress conditions.
--
This message was sent by Atlassian JIRA
(v6.1.5#6160)
[jira] [Commented] (OOZIE-1581) Workflow performance optimizations
[
https://issues.apache.org/jira/browse/OOZIE-1581?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13865029#comment-13865029
]
Hadoop QA commented on OOZIE-1581:
--
Testing JIRA OOZIE-1581
Cleaning local git workspace
{color:red}-1{color} Patch failed to apply to head of branch
Workflow performance optimizations
--
Key: OOZIE-1581
URL: https://issues.apache.org/jira/browse/OOZIE-1581
Project: Oozie
Issue Type: Improvement
Components: workflow
Affects Versions: trunk
Reporter: Mona Chitnis
Assignee: Mona Chitnis
Fix For: trunk
Attachments: OOZIE-1581-2.patch, OOZIE-1581-final.patch,
OOZIE-1581-r6.patch, OOZIE-1581.patch
Creating a combo JIRA for small performance optimizations.
1. changing from asynchronous action start to a synchronous one, to overcome
the undue delay in transitioning from ::start:: control node to the actual
first node, owing to a loaded queue. This delay has been observed to be close
to 30 min at times in stress conditions.
--
This message was sent by Atlassian JIRA
(v6.1.5#6160)
[jira] [Commented] (OOZIE-1581) Workflow performance optimizations
[
https://issues.apache.org/jira/browse/OOZIE-1581?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13861182#comment-13861182
]
Hadoop QA commented on OOZIE-1581:
--
Testing JIRA OOZIE-1581
Cleaning local git workspace
{color:green}+1 PATCH_APPLIES{color}
{color:green}+1 CLEAN{color}
{color:red}-1 RAW_PATCH_ANALYSIS{color}
.{color:green}+1{color} the patch does not introduce any @author tags
.{color:green}+1{color} the patch does not introduce any tabs
.{color:green}+1{color} the patch does not introduce any trailing spaces
.{color:red}-1{color} the patch contains 1 line(s) longer than 132
characters
.{color:green}+1{color} the patch does adds/modifies 13 testcase(s)
{color:green}+1 RAT{color}
.{color:green}+1{color} the patch does not seem to introduce new RAT
warnings
{color:green}+1 JAVADOC{color}
.{color:green}+1{color} the patch does not seem to introduce new Javadoc
warnings
{color:red}-1 COMPILE{color}
.{color:green}+1{color} HEAD compiles
.{color:red}-1{color} patch does not compile
.{color:green}+1{color} the patch does not seem to introduce new javac
warnings
{color:green}+1 BACKWARDS_COMPATIBILITY{color}
.{color:green}+1{color} the patch does not change any JPA
Entity/Colum/Basic/Lob/Transient annotations
.{color:green}+1{color} the patch does not modify JPA files
{color:red}-1 TESTS{color} - patch does not compile, cannot run testcases
{color:red}-1 DISTRO{color}
.{color:red}-1{color} distro tarball fails with the patch
{color:red}*-1 Overall result, please check the reported -1(s)*{color}
The full output of the test-patch run is available at
. https://builds.apache.org/job/oozie-trunk-precommit-build/983/
Workflow performance optimizations
--
Key: OOZIE-1581
URL: https://issues.apache.org/jira/browse/OOZIE-1581
Project: Oozie
Issue Type: Improvement
Components: workflow
Affects Versions: trunk
Reporter: Mona Chitnis
Assignee: Mona Chitnis
Fix For: trunk
Attachments: OOZIE-1581-2.patch, OOZIE-1581-final.patch,
OOZIE-1581-r6.patch, OOZIE-1581.patch
Creating a combo JIRA for small performance optimizations.
1. changing from asynchronous action start to a synchronous one, to overcome
the undue delay in transitioning from ::start:: control node to the actual
first node, owing to a loaded queue. This delay has been observed to be close
to 30 min at times in stress conditions.
--
This message was sent by Atlassian JIRA
(v6.1.5#6160)
[jira] [Commented] (OOZIE-1581) Workflow performance optimizations
[ https://issues.apache.org/jira/browse/OOZIE-1581?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13832076#comment-13832076 ] Mona Chitnis commented on OOZIE-1581: - failing test is flaky - java.lang.NullPointerException at org.apache.oozie.event.TestEventGeneration.testCoordinatorActionEvent(TestEventGeneration.java:326) will fix while committing. Workflow performance optimizations -- Key: OOZIE-1581 URL: https://issues.apache.org/jira/browse/OOZIE-1581 Project: Oozie Issue Type: Improvement Components: workflow Affects Versions: trunk Reporter: Mona Chitnis Assignee: Mona Chitnis Fix For: trunk Attachments: OOZIE-1581-2.patch, OOZIE-1581-final.patch, OOZIE-1581.patch Creating a combo JIRA for small performance optimizations. 1. changing from asynchronous action start to a synchronous one, to overcome the undue delay in transitioning from ::start:: control node to the actual first node, owing to a loaded queue. This delay has been observed to be close to 30 min at times in stress conditions. -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Commented] (OOZIE-1581) Workflow performance optimizations
[ https://issues.apache.org/jira/browse/OOZIE-1581?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13815219#comment-13815219 ] Virag Kothari commented on OOZIE-1581: -- Couple of things discussed with Mona: OOZIE-243 adds the control nodes to db and all the control nodes follow the command life cycle. Instead of optimizing the execution of commands for this nodes (async to sync), can we devise a way such that the control nodes can be configured to be not added as wf_actions in database? In this case, they wont go through the command life cycle like action nodes. While synchronously calling command, it would be good to check if we can pass the bean object to another command directly, so the other command doesn't need to perform db operations like loadState() and verifyPreCondition(). E.g CoordActionReady to CoordActionStart in the patch. Also, please check whether this type of optimization can be done at other places. Workflow performance optimizations -- Key: OOZIE-1581 URL: https://issues.apache.org/jira/browse/OOZIE-1581 Project: Oozie Issue Type: Improvement Components: workflow Affects Versions: trunk Reporter: Mona Chitnis Assignee: Mona Chitnis Fix For: trunk Attachments: OOZIE-1581-2.patch, OOZIE-1581.patch Creating a combo JIRA for small performance optimizations. 1. changing from asynchronous action start to a synchronous one, to overcome the undue delay in transitioning from ::start:: control node to the actual first node, owing to a loaded queue. This delay has been observed to be close to 30 min at times in stress conditions. -- This message was sent by Atlassian JIRA (v6.1#6144)
