[jira] [Commented] (OOZIE-1319) "LAST_ONLY" in execution control for coordinator job still runs all the actions
[ https://issues.apache.org/jira/browse/OOZIE-1319?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14562744#comment-14562744 ] zhang commented on OOZIE-1319: -- [~bowenzhangusa] in 4.1.0 version，Only when i restart the server it will skipped the WAITING or READY job. And why i can't find the handleLastOnly method in 4.1.0-branche source code. Thanks > "LAST_ONLY" in execution control for coordinator job still runs all the > actions > --- > > Key: OOZIE-1319 > URL: https://issues.apache.org/jira/browse/OOZIE-1319 > Project: Oozie > Issue Type: Bug >Reporter: Bowen Zhang >Assignee: Robert Kanter > Fix For: 4.1.0 > > Attachments: OOZIE-1319.patch, OOZIE-1319.patch, OOZIE-1319.patch, > OOZIE-1319.patch, OOZIE-1319.patch, OOZIE-1319.patch, OOZIE-1319.patch, > oozie-1319.patch > > > In execute() of CoordJobGetReadyActionsJPAExecutor.java, once we retrieve the > top item from a "LIFO" query result, we do not discard or delete the > remaining items from the result list. As a result, the next time execute() is > invoked, we will be retrieving the next item in line. Consequently, LAST_ONLY > strategy will also execute all ready actions for a given coordinator job, > making it no different than LIFO. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (OOZIE-1319) "LAST_ONLY" in execution control for coordinator job still runs all the actions
[ https://issues.apache.org/jira/browse/OOZIE-1319?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14072848#comment-14072848 ] Shwetha G S commented on OOZIE-1319: LAST_ONLY makes sense for coords with no inputs. Created OOZIE-1951 for running latest ready action. Thanks > "LAST_ONLY" in execution control for coordinator job still runs all the > actions > --- > > Key: OOZIE-1319 > URL: https://issues.apache.org/jira/browse/OOZIE-1319 > Project: Oozie > Issue Type: Bug >Reporter: Bowen Zhang >Assignee: Robert Kanter > Fix For: 4.1.0 > > Attachments: OOZIE-1319.patch, OOZIE-1319.patch, OOZIE-1319.patch, > OOZIE-1319.patch, OOZIE-1319.patch, OOZIE-1319.patch, OOZIE-1319.patch, > oozie-1319.patch > > > In execute() of CoordJobGetReadyActionsJPAExecutor.java, once we retrieve the > top item from a "LIFO" query result, we do not discard or delete the > remaining items from the result list. As a result, the next time execute() is > invoked, we will be retrieving the next item in line. Consequently, LAST_ONLY > strategy will also execute all ready actions for a given coordinator job, > making it no different than LIFO. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (OOZIE-1319) "LAST_ONLY" in execution control for coordinator job still runs all the actions
[
https://issues.apache.org/jira/browse/OOZIE-1319?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14072340#comment-14072340
]
Robert Kanter commented on OOZIE-1319:
--
{quote}Previously, LAST_ONLY was running the latest ready action. The gap was
that it was picking up even the old ready instances after that. {quote}
Actually, previously LAST_ONLY was basically the same as LIFO. And from what I
could tell, it hasn't worked properly for at least 3 years.
{quote}But the current implementation changes the meaning of LAST_ONLY to run
only the latest nominal time{quote}
The meaning of LAST_ONLY was never clearly specified. From the documentation,
it only had this: "discards all older materializations". That's pretty vague
and could mean a few different things. After Bowen's attempt at this JIRA, and
then my earlier attempts, it was getting too complicated and we kept find edge
cases that we couldn't fix. t finally decided that LAST_ONLY could be
interpreted as essentially the same as having a timeout equal to the frequency,
but with a "good" ending status (see [this
comment|https://issues.apache.org/jira/browse/OOZIE-1319?focusedCommentId=14003922&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-14003922]
and later).
{quote}Is there a usecase for the current implementation of LAST_ONLY. If there
is{quote}
LAST_ONLY is good if you have a coordinator where you don't necessarily care
about the specific instances that it runs, just that it runs. Specifically, we
had a customer who had a coordinator with LAST_ONLY and was expecting this
behavior. They brought down their Oozie server for a while for maintenance;
when they brought it up, they were expecting Oozie to ignore the "missed"
instantiations and just do the last one and continue normally after that, but
it instead played "catch up" and started them all. That's why I wanted to fix
this; otherwise, nobody has complained that LAST_ONLY hasn't worked in 3
years... I imagine most users don't even know that you can change it to LIFO or
LAST_ONLY.
{quote}I will add another enum which executes latest ready action and discards
the rest.{quote}
Feel free to add another execution strategy. [~bowenzhangusa] has already
created a new one called NONE in OOZIE-1875.
> "LAST_ONLY" in execution control for coordinator job still runs all the
> actions
> ---
>
> Key: OOZIE-1319
> URL: https://issues.apache.org/jira/browse/OOZIE-1319
> Project: Oozie
> Issue Type: Bug
>Reporter: Bowen Zhang
>Assignee: Robert Kanter
> Fix For: 4.1.0
>
> Attachments: OOZIE-1319.patch, OOZIE-1319.patch, OOZIE-1319.patch,
> OOZIE-1319.patch, OOZIE-1319.patch, OOZIE-1319.patch, OOZIE-1319.patch,
> oozie-1319.patch
>
>
> In execute() of CoordJobGetReadyActionsJPAExecutor.java, once we retrieve the
> top item from a "LIFO" query result, we do not discard or delete the
> remaining items from the result list. As a result, the next time execute() is
> invoked, we will be retrieving the next item in line. Consequently, LAST_ONLY
> strategy will also execute all ready actions for a given coordinator job,
> making it no different than LIFO.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (OOZIE-1319) "LAST_ONLY" in execution control for coordinator job still runs all the actions
[ https://issues.apache.org/jira/browse/OOZIE-1319?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14071498#comment-14071498 ] Shwetha G S commented on OOZIE-1319: [~rkanter], can you check this please? > "LAST_ONLY" in execution control for coordinator job still runs all the > actions > --- > > Key: OOZIE-1319 > URL: https://issues.apache.org/jira/browse/OOZIE-1319 > Project: Oozie > Issue Type: Bug >Reporter: Bowen Zhang >Assignee: Robert Kanter > Fix For: 4.1.0 > > Attachments: OOZIE-1319.patch, OOZIE-1319.patch, OOZIE-1319.patch, > OOZIE-1319.patch, OOZIE-1319.patch, OOZIE-1319.patch, OOZIE-1319.patch, > oozie-1319.patch > > > In execute() of CoordJobGetReadyActionsJPAExecutor.java, once we retrieve the > top item from a "LIFO" query result, we do not discard or delete the > remaining items from the result list. As a result, the next time execute() is > invoked, we will be retrieving the next item in line. Consequently, LAST_ONLY > strategy will also execute all ready actions for a given coordinator job, > making it no different than LIFO. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (OOZIE-1319) "LAST_ONLY" in execution control for coordinator job still runs all the actions
[ https://issues.apache.org/jira/browse/OOZIE-1319?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14066169#comment-14066169 ] Shwetha G S commented on OOZIE-1319: Previously, LAST_ONLY was running the latest ready action. The gap was that it was picking up even the old ready instances after that. So, I thought just skipping the old instances in CoordActionReadyXCommand would complete the feature. But the current implementation changes the meaning of LAST_ONLY to run only the latest nominal time(makes sense in case of backlog, but the backlog is in waiting actions which will happen probably because of materialisation delays which is internal to oozie?) Anyways, now that its implemented already, Is there a usecase for the current implementation of LAST_ONLY. If there is, I will add another enum which executes latest ready action and discards the rest. If there is no usecase, can we modify LAST_ONLY to run the latest ready action? Depending on what we decide, I will create the jira accordingly. Let me know. Thanks > "LAST_ONLY" in execution control for coordinator job still runs all the > actions > --- > > Key: OOZIE-1319 > URL: https://issues.apache.org/jira/browse/OOZIE-1319 > Project: Oozie > Issue Type: Bug >Reporter: Bowen Zhang >Assignee: Robert Kanter > Fix For: trunk > > Attachments: OOZIE-1319.patch, OOZIE-1319.patch, OOZIE-1319.patch, > OOZIE-1319.patch, OOZIE-1319.patch, OOZIE-1319.patch, OOZIE-1319.patch, > oozie-1319.patch > > > In execute() of CoordJobGetReadyActionsJPAExecutor.java, once we retrieve the > top item from a "LIFO" query result, we do not discard or delete the > remaining items from the result list. As a result, the next time execute() is > invoked, we will be retrieving the next item in line. Consequently, LAST_ONLY > strategy will also execute all ready actions for a given coordinator job, > making it no different than LIFO. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (OOZIE-1319) "LAST_ONLY" in execution control for coordinator job still runs all the actions
[
https://issues.apache.org/jira/browse/OOZIE-1319?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14065774#comment-14065774
]
Robert Kanter commented on OOZIE-1319:
--
{quote}CoordActionInputCheckXCommand, why the comparison with now and next
nominal time? {quote}
As for what the LAST_ONLY option was supposed to do, we never actually had any
very specific documentation. To try to keep this from becoming very
complicated (both to implement and to use), at some point I made the decision
that the LAST_ONLY will look at the nominal time of the next nominal time. In
essence, you can think of this like having the timeout set to the interval, but
instead of being TIMEDOUT (which results in a "bad" end state for the job), it
becomes SKIPPED (which results in a "good" end state for the job). I think
this decision makes sense anyway because if the next action is ready to run,
the previous action should have either already ran or should not run.
{quote}Take an example of an hourly coord and there are coord actions for 8th,
9th, 10th and 11th hour in WAITING state. The expectation of LAST_ONLY is that
among the ready actions(all dependencies met), it runs only the last one. But
with now and next nominal time check, CoordActionInputCheckXCommand marks 8th,
9th and 10th as SKIPPED and 11th will remain in WAITING state. If the
dependencies are available late consistently, the same thing continues(in 12th
hour, 11th will be marked SKIPPED) and so on and none of the instances for this
coord will run.{quote}
If the dependencies are met, then shouldn't the 11th transition from WAITING to
READY to RUNNING etc?
But yes, I suppose if you had a job where the data was consistently late by
over one interval of the frequency, and the timeout was set high enough, then
it is possible that no actions will actually run. I'm not sure of a good way
to handle this scenario, but if you have any ideas, please file a JIRA.
> "LAST_ONLY" in execution control for coordinator job still runs all the
> actions
> ---
>
> Key: OOZIE-1319
> URL: https://issues.apache.org/jira/browse/OOZIE-1319
> Project: Oozie
> Issue Type: Bug
>Reporter: Bowen Zhang
>Assignee: Robert Kanter
> Fix For: trunk
>
> Attachments: OOZIE-1319.patch, OOZIE-1319.patch, OOZIE-1319.patch,
> OOZIE-1319.patch, OOZIE-1319.patch, OOZIE-1319.patch, OOZIE-1319.patch,
> oozie-1319.patch
>
>
> In execute() of CoordJobGetReadyActionsJPAExecutor.java, once we retrieve the
> top item from a "LIFO" query result, we do not discard or delete the
> remaining items from the result list. As a result, the next time execute() is
> invoked, we will be retrieving the next item in line. Consequently, LAST_ONLY
> strategy will also execute all ready actions for a given coordinator job,
> making it no different than LIFO.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (OOZIE-1319) "LAST_ONLY" in execution control for coordinator job still runs all the actions
[ https://issues.apache.org/jira/browse/OOZIE-1319?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14064742#comment-14064742 ] Shwetha G S commented on OOZIE-1319: I have a question regarding this implementation(Sorry about the late ask) - Why does this require changes in CoordActionInputCheckXCommand, why the comparison with now and next nominal time? Take an example of an hourly coord and there are coord actions for 8th, 9th, 10th and 11th hour in WAITING state. The expectation of LAST_ONLY is that among the ready actions(all dependencies met), it runs only the last one. But with now and next nominal time check, CoordActionInputCheckXCommand marks 8th, 9th and 10th as SKIPPED and 11th will remain in WAITING state. If the dependencies are available late consistently, the same thing continues(in 12th hour, 11th will be marked SKIPPED) and so on and none of the instances for this coord will run. Instead, shouldn't the change be in CoordActionReadyXCommand which picks all ready actions and picks latest and discard the rest? > "LAST_ONLY" in execution control for coordinator job still runs all the > actions > --- > > Key: OOZIE-1319 > URL: https://issues.apache.org/jira/browse/OOZIE-1319 > Project: Oozie > Issue Type: Bug >Reporter: Bowen Zhang >Assignee: Robert Kanter > Fix For: trunk > > Attachments: OOZIE-1319.patch, OOZIE-1319.patch, OOZIE-1319.patch, > OOZIE-1319.patch, OOZIE-1319.patch, OOZIE-1319.patch, OOZIE-1319.patch, > oozie-1319.patch > > > In execute() of CoordJobGetReadyActionsJPAExecutor.java, once we retrieve the > top item from a "LIFO" query result, we do not discard or delete the > remaining items from the result list. As a result, the next time execute() is > invoked, we will be retrieving the next item in line. Consequently, LAST_ONLY > strategy will also execute all ready actions for a given coordinator job, > making it no different than LIFO. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (OOZIE-1319) "LAST_ONLY" in execution control for coordinator job still runs all the actions
[ https://issues.apache.org/jira/browse/OOZIE-1319?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14014248#comment-14014248 ] Bowen Zhang commented on OOZIE-1319: my bad, didn't see it. > "LAST_ONLY" in execution control for coordinator job still runs all the > actions > --- > > Key: OOZIE-1319 > URL: https://issues.apache.org/jira/browse/OOZIE-1319 > Project: Oozie > Issue Type: Bug >Reporter: Bowen Zhang >Assignee: Robert Kanter > Attachments: OOZIE-1319.patch, OOZIE-1319.patch, OOZIE-1319.patch, > OOZIE-1319.patch, OOZIE-1319.patch, OOZIE-1319.patch, OOZIE-1319.patch, > oozie-1319.patch > > > In execute() of CoordJobGetReadyActionsJPAExecutor.java, once we retrieve the > top item from a "LIFO" query result, we do not discard or delete the > remaining items from the result list. As a result, the next time execute() is > invoked, we will be retrieving the next item in line. Consequently, LAST_ONLY > strategy will also execute all ready actions for a given coordinator job, > making it no different than LIFO. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (OOZIE-1319) "LAST_ONLY" in execution control for coordinator job still runs all the actions
[ https://issues.apache.org/jira/browse/OOZIE-1319?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14014238#comment-14014238 ] Robert Kanter commented on OOZIE-1319: -- Thanks for the reviews and catching all of those edge cases. I'm glad we can finally get this in :) > "LAST_ONLY" in execution control for coordinator job still runs all the > actions > --- > > Key: OOZIE-1319 > URL: https://issues.apache.org/jira/browse/OOZIE-1319 > Project: Oozie > Issue Type: Bug >Reporter: Bowen Zhang >Assignee: Robert Kanter > Attachments: OOZIE-1319.patch, OOZIE-1319.patch, OOZIE-1319.patch, > OOZIE-1319.patch, OOZIE-1319.patch, OOZIE-1319.patch, OOZIE-1319.patch, > oozie-1319.patch > > > In execute() of CoordJobGetReadyActionsJPAExecutor.java, once we retrieve the > top item from a "LIFO" query result, we do not discard or delete the > remaining items from the result list. As a result, the next time execute() is > invoked, we will be retrieving the next item in line. Consequently, LAST_ONLY > strategy will also execute all ready actions for a given coordinator job, > making it no different than LIFO. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (OOZIE-1319) "LAST_ONLY" in execution control for coordinator job still runs all the actions
[
https://issues.apache.org/jira/browse/OOZIE-1319?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14014226#comment-14014226
]
Robert Kanter commented on OOZIE-1319:
--
It already checks that:
{code:java}
// If the next nominal time is after the job's end time, then this is the last
action, so return null
if (nextNominalTime.after(coordJob.getEndTime())) {
nextNominalTime = null;
}
return nextNominalTime;
{code}
> "LAST_ONLY" in execution control for coordinator job still runs all the
> actions
> ---
>
> Key: OOZIE-1319
> URL: https://issues.apache.org/jira/browse/OOZIE-1319
> Project: Oozie
> Issue Type: Bug
>Reporter: Bowen Zhang
>Assignee: Robert Kanter
> Attachments: OOZIE-1319.patch, OOZIE-1319.patch, OOZIE-1319.patch,
> OOZIE-1319.patch, OOZIE-1319.patch, OOZIE-1319.patch, OOZIE-1319.patch,
> oozie-1319.patch
>
>
> In execute() of CoordJobGetReadyActionsJPAExecutor.java, once we retrieve the
> top item from a "LIFO" query result, we do not discard or delete the
> remaining items from the result list. As a result, the next time execute() is
> invoked, we will be retrieving the next item in line. Consequently, LAST_ONLY
> strategy will also execute all ready actions for a given coordinator job,
> making it no different than LIFO.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (OOZIE-1319) "LAST_ONLY" in execution control for coordinator job still runs all the actions
[ https://issues.apache.org/jira/browse/OOZIE-1319?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14014224#comment-14014224 ] Bowen Zhang commented on OOZIE-1319: one last issue. When computing nextNominalTime, you need to make sure nextNominalTime is < end time of the coord job. Otherwise, you may wrongly skip the last action of the coord job. After that, +1. > "LAST_ONLY" in execution control for coordinator job still runs all the > actions > --- > > Key: OOZIE-1319 > URL: https://issues.apache.org/jira/browse/OOZIE-1319 > Project: Oozie > Issue Type: Bug >Reporter: Bowen Zhang >Assignee: Robert Kanter > Attachments: OOZIE-1319.patch, OOZIE-1319.patch, OOZIE-1319.patch, > OOZIE-1319.patch, OOZIE-1319.patch, OOZIE-1319.patch, OOZIE-1319.patch, > oozie-1319.patch > > > In execute() of CoordJobGetReadyActionsJPAExecutor.java, once we retrieve the > top item from a "LIFO" query result, we do not discard or delete the > remaining items from the result list. As a result, the next time execute() is > invoked, we will be retrieving the next item in line. Consequently, LAST_ONLY > strategy will also execute all ready actions for a given coordinator job, > making it no different than LIFO. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (OOZIE-1319) "LAST_ONLY" in execution control for coordinator job still runs all the actions
[ https://issues.apache.org/jira/browse/OOZIE-1319?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14013352#comment-14013352 ] Robert Kanter commented on OOZIE-1319: -- Test failures are unrelated. Lines that are too long are Named Queries. > "LAST_ONLY" in execution control for coordinator job still runs all the > actions > --- > > Key: OOZIE-1319 > URL: https://issues.apache.org/jira/browse/OOZIE-1319 > Project: Oozie > Issue Type: Bug >Reporter: Bowen Zhang >Assignee: Robert Kanter > Attachments: OOZIE-1319.patch, OOZIE-1319.patch, OOZIE-1319.patch, > OOZIE-1319.patch, OOZIE-1319.patch, OOZIE-1319.patch, OOZIE-1319.patch, > oozie-1319.patch > > > In execute() of CoordJobGetReadyActionsJPAExecutor.java, once we retrieve the > top item from a "LIFO" query result, we do not discard or delete the > remaining items from the result list. As a result, the next time execute() is > invoked, we will be retrieving the next item in line. Consequently, LAST_ONLY > strategy will also execute all ready actions for a given coordinator job, > making it no different than LIFO. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (OOZIE-1319) "LAST_ONLY" in execution control for coordinator job still runs all the actions
[
https://issues.apache.org/jira/browse/OOZIE-1319?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14013298#comment-14013298
]
Hadoop QA commented on OOZIE-1319:
--
Testing JIRA OOZIE-1319
Cleaning local git workspace
{color:green}+1 PATCH_APPLIES{color}
{color:green}+1 CLEAN{color}
{color:red}-1 RAW_PATCH_ANALYSIS{color}
.{color:green}+1{color} the patch does not introduce any @author tags
.{color:green}+1{color} the patch does not introduce any tabs
.{color:green}+1{color} the patch does not introduce any trailing spaces
.{color:red}-1{color} the patch contains 2 line(s) longer than 132
characters
.{color:green}+1{color} the patch does adds/modifies 7 testcase(s)
{color:green}+1 RAT{color}
.{color:green}+1{color} the patch does not seem to introduce new RAT
warnings
{color:green}+1 JAVADOC{color}
.{color:green}+1{color} the patch does not seem to introduce new Javadoc
warnings
{color:green}+1 COMPILE{color}
.{color:green}+1{color} HEAD compiles
.{color:green}+1{color} patch compiles
.{color:green}+1{color} the patch does not seem to introduce new javac
warnings
{color:green}+1 BACKWARDS_COMPATIBILITY{color}
.{color:green}+1{color} the patch does not change any JPA
Entity/Colum/Basic/Lob/Transient annotations
.{color:green}+1{color} the patch does not modify JPA files
{color:red}-1 TESTS{color}
.Tests run: 1458
.Tests failed: 1
.Tests errors: 1
.The patch failed the following testcases:
.
testBundleEngineKill(org.apache.oozie.servlet.TestV1JobServletBundleEngine)
{color:green}+1 DISTRO{color}
.{color:green}+1{color} distro tarball builds with the patch
{color:red}*-1 Overall result, please check the reported -1(s)*{color}
The full output of the test-patch run is available at
. https://builds.apache.org/job/oozie-trunk-precommit-build/1275/
> "LAST_ONLY" in execution control for coordinator job still runs all the
> actions
> ---
>
> Key: OOZIE-1319
> URL: https://issues.apache.org/jira/browse/OOZIE-1319
> Project: Oozie
> Issue Type: Bug
>Reporter: Bowen Zhang
>Assignee: Robert Kanter
> Attachments: OOZIE-1319.patch, OOZIE-1319.patch, OOZIE-1319.patch,
> OOZIE-1319.patch, OOZIE-1319.patch, OOZIE-1319.patch, OOZIE-1319.patch,
> oozie-1319.patch
>
>
> In execute() of CoordJobGetReadyActionsJPAExecutor.java, once we retrieve the
> top item from a "LIFO" query result, we do not discard or delete the
> remaining items from the result list. As a result, the next time execute() is
> invoked, we will be retrieving the next item in line. Consequently, LAST_ONLY
> strategy will also execute all ready actions for a given coordinator job,
> making it no different than LIFO.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (OOZIE-1319) "LAST_ONLY" in execution control for coordinator job still runs all the actions
[ https://issues.apache.org/jira/browse/OOZIE-1319?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14013090#comment-14013090 ] Bowen Zhang commented on OOZIE-1319: Sounds promising. please consider both integer and cron frequency. > "LAST_ONLY" in execution control for coordinator job still runs all the > actions > --- > > Key: OOZIE-1319 > URL: https://issues.apache.org/jira/browse/OOZIE-1319 > Project: Oozie > Issue Type: Bug >Reporter: Bowen Zhang >Assignee: Robert Kanter > Attachments: OOZIE-1319.patch, OOZIE-1319.patch, OOZIE-1319.patch, > OOZIE-1319.patch, OOZIE-1319.patch, OOZIE-1319.patch, oozie-1319.patch > > > In execute() of CoordJobGetReadyActionsJPAExecutor.java, once we retrieve the > top item from a "LIFO" query result, we do not discard or delete the > remaining items from the result list. As a result, the next time execute() is > invoked, we will be retrieving the next item in line. Consequently, LAST_ONLY > strategy will also execute all ready actions for a given coordinator job, > making it no different than LIFO. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (OOZIE-1319) "LAST_ONLY" in execution control for coordinator job still runs all the actions
[ https://issues.apache.org/jira/browse/OOZIE-1319?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14012987#comment-14012987 ] Robert Kanter commented on OOZIE-1319: -- To clarify, if I can get that idea to work, then I don't need to even retrieve the nextcoordaction at all and can delete that code. > "LAST_ONLY" in execution control for coordinator job still runs all the > actions > --- > > Key: OOZIE-1319 > URL: https://issues.apache.org/jira/browse/OOZIE-1319 > Project: Oozie > Issue Type: Bug >Reporter: Bowen Zhang >Assignee: Robert Kanter > Attachments: OOZIE-1319.patch, OOZIE-1319.patch, OOZIE-1319.patch, > OOZIE-1319.patch, OOZIE-1319.patch, OOZIE-1319.patch, oozie-1319.patch > > > In execute() of CoordJobGetReadyActionsJPAExecutor.java, once we retrieve the > top item from a "LIFO" query result, we do not discard or delete the > remaining items from the result list. As a result, the next time execute() is > invoked, we will be retrieving the next item in line. Consequently, LAST_ONLY > strategy will also execute all ready actions for a given coordinator job, > making it no different than LIFO. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (OOZIE-1319) "LAST_ONLY" in execution control for coordinator job still runs all the actions
[ https://issues.apache.org/jira/browse/OOZIE-1319?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14012985#comment-14012985 ] Robert Kanter commented on OOZIE-1319: -- That's going to get tricky. I have another idea to try first: Instead of looking at the nominal time of the next action in CoordActionInputCheckXCommand, I can have it calculate what the nominal time of the next action should be. Functionally, this is equivalent to what I have now, but the advantage is that it doesn't rely on the next action having been materialized yet. So in the last scenario that you gave, even though the action at 8:50am might see the nextcoordaction as null, it won't matter because it will compute what the nextcoordaction's nominal time is. I should be able to steal some logic from the CoordMaterializeTransitionXCommand for computing the next nominal time. > "LAST_ONLY" in execution control for coordinator job still runs all the > actions > --- > > Key: OOZIE-1319 > URL: https://issues.apache.org/jira/browse/OOZIE-1319 > Project: Oozie > Issue Type: Bug >Reporter: Bowen Zhang >Assignee: Robert Kanter > Attachments: OOZIE-1319.patch, OOZIE-1319.patch, OOZIE-1319.patch, > OOZIE-1319.patch, OOZIE-1319.patch, OOZIE-1319.patch, oozie-1319.patch > > > In execute() of CoordJobGetReadyActionsJPAExecutor.java, once we retrieve the > top item from a "LIFO" query result, we do not discard or delete the > remaining items from the result list. As a result, the next time execute() is > invoked, we will be retrieving the next item in line. Consequently, LAST_ONLY > strategy will also execute all ready actions for a given coordinator job, > making it no different than LIFO. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (OOZIE-1319) "LAST_ONLY" in execution control for coordinator job still runs all the actions
[ https://issues.apache.org/jira/browse/OOZIE-1319?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14012972#comment-14012972 ] Bowen Zhang commented on OOZIE-1319: Try to think if you can have any blocking mechanism between the two services. For examples, skip queuing CoordInputCheckXCommand for certain actions with LAST_ONLY execution order. And queue them only after certain conditions are satisfied. > "LAST_ONLY" in execution control for coordinator job still runs all the > actions > --- > > Key: OOZIE-1319 > URL: https://issues.apache.org/jira/browse/OOZIE-1319 > Project: Oozie > Issue Type: Bug >Reporter: Bowen Zhang >Assignee: Robert Kanter > Attachments: OOZIE-1319.patch, OOZIE-1319.patch, OOZIE-1319.patch, > OOZIE-1319.patch, OOZIE-1319.patch, OOZIE-1319.patch, oozie-1319.patch > > > In execute() of CoordJobGetReadyActionsJPAExecutor.java, once we retrieve the > top item from a "LIFO" query result, we do not discard or delete the > remaining items from the result list. As a result, the next time execute() is > invoked, we will be retrieving the next item in line. Consequently, LAST_ONLY > strategy will also execute all ready actions for a given coordinator job, > making it no different than LIFO. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (OOZIE-1319) "LAST_ONLY" in execution control for coordinator job still runs all the actions
[ https://issues.apache.org/jira/browse/OOZIE-1319?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14012960#comment-14012960 ] Robert Kanter commented on OOZIE-1319: -- Ah, that's a good point. I'll try to think of a solution to that problem. > "LAST_ONLY" in execution control for coordinator job still runs all the > actions > --- > > Key: OOZIE-1319 > URL: https://issues.apache.org/jira/browse/OOZIE-1319 > Project: Oozie > Issue Type: Bug >Reporter: Bowen Zhang >Assignee: Robert Kanter > Attachments: OOZIE-1319.patch, OOZIE-1319.patch, OOZIE-1319.patch, > OOZIE-1319.patch, OOZIE-1319.patch, OOZIE-1319.patch, oozie-1319.patch > > > In execute() of CoordJobGetReadyActionsJPAExecutor.java, once we retrieve the > top item from a "LIFO" query result, we do not discard or delete the > remaining items from the result list. As a result, the next time execute() is > invoked, we will be retrieving the next item in line. Consequently, LAST_ONLY > strategy will also execute all ready actions for a given coordinator job, > making it no different than LIFO. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (OOZIE-1319) "LAST_ONLY" in execution control for coordinator job still runs all the actions
[ https://issues.apache.org/jira/browse/OOZIE-1319?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14012931#comment-14012931 ] Bowen Zhang commented on OOZIE-1319: I think the reason you don't see the problem is that you only have one coordinator or few coordinators running. Therefore, Materialization happens really fast which beats the speed of recoveryservice. You can check this patch in, but certainly this is a bug. > "LAST_ONLY" in execution control for coordinator job still runs all the > actions > --- > > Key: OOZIE-1319 > URL: https://issues.apache.org/jira/browse/OOZIE-1319 > Project: Oozie > Issue Type: Bug >Reporter: Bowen Zhang >Assignee: Robert Kanter > Attachments: OOZIE-1319.patch, OOZIE-1319.patch, OOZIE-1319.patch, > OOZIE-1319.patch, OOZIE-1319.patch, OOZIE-1319.patch, oozie-1319.patch > > > In execute() of CoordJobGetReadyActionsJPAExecutor.java, once we retrieve the > top item from a "LIFO" query result, we do not discard or delete the > remaining items from the result list. As a result, the next time execute() is > invoked, we will be retrieving the next item in line. Consequently, LAST_ONLY > strategy will also execute all ready actions for a given coordinator job, > making it no different than LIFO. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (OOZIE-1319) "LAST_ONLY" in execution control for coordinator job still runs all the actions
[ https://issues.apache.org/jira/browse/OOZIE-1319?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14012897#comment-14012897 ] Robert Kanter commented on OOZIE-1319: -- That certainly seems like a valid concern, but it looks like it materializes the actions before the check happens so it's not actually a problem. > "LAST_ONLY" in execution control for coordinator job still runs all the > actions > --- > > Key: OOZIE-1319 > URL: https://issues.apache.org/jira/browse/OOZIE-1319 > Project: Oozie > Issue Type: Bug >Reporter: Bowen Zhang >Assignee: Robert Kanter > Attachments: OOZIE-1319.patch, OOZIE-1319.patch, OOZIE-1319.patch, > OOZIE-1319.patch, OOZIE-1319.patch, OOZIE-1319.patch, oozie-1319.patch > > > In execute() of CoordJobGetReadyActionsJPAExecutor.java, once we retrieve the > top item from a "LIFO" query result, we do not discard or delete the > remaining items from the result list. As a result, the next time execute() is > invoked, we will be retrieving the next item in line. Consequently, LAST_ONLY > strategy will also execute all ready actions for a given coordinator job, > making it no different than LIFO. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (OOZIE-1319) "LAST_ONLY" in execution control for coordinator job still runs all the actions
[ https://issues.apache.org/jira/browse/OOZIE-1319?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14012891#comment-14012891 ] Bowen Zhang commented on OOZIE-1319: But analytically, this does seem to be a problem. > "LAST_ONLY" in execution control for coordinator job still runs all the > actions > --- > > Key: OOZIE-1319 > URL: https://issues.apache.org/jira/browse/OOZIE-1319 > Project: Oozie > Issue Type: Bug >Reporter: Bowen Zhang >Assignee: Robert Kanter > Attachments: OOZIE-1319.patch, OOZIE-1319.patch, OOZIE-1319.patch, > OOZIE-1319.patch, OOZIE-1319.patch, OOZIE-1319.patch, oozie-1319.patch > > > In execute() of CoordJobGetReadyActionsJPAExecutor.java, once we retrieve the > top item from a "LIFO" query result, we do not discard or delete the > remaining items from the result list. As a result, the next time execute() is > invoked, we will be retrieving the next item in line. Consequently, LAST_ONLY > strategy will also execute all ready actions for a given coordinator job, > making it no different than LIFO. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (OOZIE-1319) "LAST_ONLY" in execution control for coordinator job still runs all the actions
[ https://issues.apache.org/jira/browse/OOZIE-1319?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14012828#comment-14012828 ] Robert Kanter commented on OOZIE-1319: -- I tried replicating that scenario (by playing with my system clock so I didn't have to wait so long) and it worked correctly; that doesn't seem to be a problem. Though I do have some tests to fix; it looks like OOZIE-1819, which got in when I last rebased the patch, somehow broke some of my tests. > "LAST_ONLY" in execution control for coordinator job still runs all the > actions > --- > > Key: OOZIE-1319 > URL: https://issues.apache.org/jira/browse/OOZIE-1319 > Project: Oozie > Issue Type: Bug >Reporter: Bowen Zhang >Assignee: Robert Kanter > Attachments: OOZIE-1319.patch, OOZIE-1319.patch, OOZIE-1319.patch, > OOZIE-1319.patch, OOZIE-1319.patch, OOZIE-1319.patch, oozie-1319.patch > > > In execute() of CoordJobGetReadyActionsJPAExecutor.java, once we retrieve the > top item from a "LIFO" query result, we do not discard or delete the > remaining items from the result list. As a result, the next time execute() is > invoked, we will be retrieving the next item in line. Consequently, LAST_ONLY > strategy will also execute all ready actions for a given coordinator job, > making it no different than LIFO. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (OOZIE-1319) "LAST_ONLY" in execution control for coordinator job still runs all the actions
[ https://issues.apache.org/jira/browse/OOZIE-1319?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14012582#comment-14012582 ] Bowen Zhang commented on OOZIE-1319: [~rkanter], I think my claim about problem arising from server backup is still legit. Consider the server goes down at 8:30 am and back up at 9:31am. The job has a frequency of 10 mins and starts at 8 am. what is expected is only the action at 9:30 am should run and the rest should be skipped. But in your code, the action at 8:50am would also run since CoordInputcheckXCommand sees the nextcoordaction is null before the materialization service materializes actions from 9 am till 10 am. As a result, the action at 8:50 am will be marked as READY. > "LAST_ONLY" in execution control for coordinator job still runs all the > actions > --- > > Key: OOZIE-1319 > URL: https://issues.apache.org/jira/browse/OOZIE-1319 > Project: Oozie > Issue Type: Bug >Reporter: Bowen Zhang >Assignee: Robert Kanter > Attachments: OOZIE-1319.patch, OOZIE-1319.patch, OOZIE-1319.patch, > OOZIE-1319.patch, OOZIE-1319.patch, OOZIE-1319.patch, oozie-1319.patch > > > In execute() of CoordJobGetReadyActionsJPAExecutor.java, once we retrieve the > top item from a "LIFO" query result, we do not discard or delete the > remaining items from the result list. As a result, the next time execute() is > invoked, we will be retrieving the next item in line. Consequently, LAST_ONLY > strategy will also execute all ready actions for a given coordinator job, > making it no different than LIFO. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (OOZIE-1319) "LAST_ONLY" in execution control for coordinator job still runs all the actions
[
https://issues.apache.org/jira/browse/OOZIE-1319?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14011994#comment-14011994
]
Hadoop QA commented on OOZIE-1319:
--
Testing JIRA OOZIE-1319
Cleaning local git workspace
{color:green}+1 PATCH_APPLIES{color}
{color:green}+1 CLEAN{color}
{color:red}-1 RAW_PATCH_ANALYSIS{color}
.{color:green}+1{color} the patch does not introduce any @author tags
.{color:green}+1{color} the patch does not introduce any tabs
.{color:green}+1{color} the patch does not introduce any trailing spaces
.{color:red}-1{color} the patch contains 5 line(s) longer than 132
characters
.{color:green}+1{color} the patch does adds/modifies 9 testcase(s)
{color:green}+1 RAT{color}
.{color:green}+1{color} the patch does not seem to introduce new RAT
warnings
{color:green}+1 JAVADOC{color}
.{color:green}+1{color} the patch does not seem to introduce new Javadoc
warnings
{color:green}+1 COMPILE{color}
.{color:green}+1{color} HEAD compiles
.{color:green}+1{color} patch compiles
.{color:green}+1{color} the patch does not seem to introduce new javac
warnings
{color:green}+1 BACKWARDS_COMPATIBILITY{color}
.{color:green}+1{color} the patch does not change any JPA
Entity/Colum/Basic/Lob/Transient annotations
.{color:green}+1{color} the patch does not modify JPA files
{color:red}-1 TESTS{color}
.Tests run: 1460
.Tests failed: 4
.Tests errors: 6
.The patch failed the following testcases:
.
testActionInputCheckLatestActionCreationTime(org.apache.oozie.command.coord.TestCoordActionInputCheckXCommand)
.
testBundleEngineSuspend(org.apache.oozie.servlet.TestV1JobServletBundleEngine)
.
testLastOnlyMaterialization(org.apache.oozie.command.coord.TestCoordMaterializeTransitionXCommand)
.
testActionInputCheckLatestActionCreationTime(org.apache.oozie.command.coord.TestCoordActionInputCheckXCommandNonUTC)
{color:green}+1 DISTRO{color}
.{color:green}+1{color} distro tarball builds with the patch
{color:red}*-1 Overall result, please check the reported -1(s)*{color}
The full output of the test-patch run is available at
. https://builds.apache.org/job/oozie-trunk-precommit-build/1270/
> "LAST_ONLY" in execution control for coordinator job still runs all the
> actions
> ---
>
> Key: OOZIE-1319
> URL: https://issues.apache.org/jira/browse/OOZIE-1319
> Project: Oozie
> Issue Type: Bug
>Reporter: Bowen Zhang
>Assignee: Robert Kanter
> Attachments: OOZIE-1319.patch, OOZIE-1319.patch, OOZIE-1319.patch,
> OOZIE-1319.patch, OOZIE-1319.patch, OOZIE-1319.patch, oozie-1319.patch
>
>
> In execute() of CoordJobGetReadyActionsJPAExecutor.java, once we retrieve the
> top item from a "LIFO" query result, we do not discard or delete the
> remaining items from the result list. As a result, the next time execute() is
> invoked, we will be retrieving the next item in line. Consequently, LAST_ONLY
> strategy will also execute all ready actions for a given coordinator job,
> making it no different than LIFO.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (OOZIE-1319) "LAST_ONLY" in execution control for coordinator job still runs all the actions
[ https://issues.apache.org/jira/browse/OOZIE-1319?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14011504#comment-14011504 ] Robert Kanter commented on OOZIE-1319: -- Ah, sorry, I misunderstood what you meant. I'll check that the test failures are unrelated and commit this soon. Thanks for the review > "LAST_ONLY" in execution control for coordinator job still runs all the > actions > --- > > Key: OOZIE-1319 > URL: https://issues.apache.org/jira/browse/OOZIE-1319 > Project: Oozie > Issue Type: Bug >Reporter: Bowen Zhang >Assignee: Robert Kanter > Attachments: OOZIE-1319.patch, OOZIE-1319.patch, OOZIE-1319.patch, > OOZIE-1319.patch, OOZIE-1319.patch, oozie-1319.patch > > > In execute() of CoordJobGetReadyActionsJPAExecutor.java, once we retrieve the > top item from a "LIFO" query result, we do not discard or delete the > remaining items from the result list. As a result, the next time execute() is > invoked, we will be retrieving the next item in line. Consequently, LAST_ONLY > strategy will also execute all ready actions for a given coordinator job, > making it no different than LIFO. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (OOZIE-1319) "LAST_ONLY" in execution control for coordinator job still runs all the actions
[
https://issues.apache.org/jira/browse/OOZIE-1319?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14011456#comment-14011456
]
Bowen Zhang commented on OOZIE-1319:
I didn't say commit 1305 with this one. I ssaid {quote}I want to merge 1305
into this feature once you check this in{quote}.
> "LAST_ONLY" in execution control for coordinator job still runs all the
> actions
> ---
>
> Key: OOZIE-1319
> URL: https://issues.apache.org/jira/browse/OOZIE-1319
> Project: Oozie
> Issue Type: Bug
>Reporter: Bowen Zhang
>Assignee: Robert Kanter
> Attachments: OOZIE-1319.patch, OOZIE-1319.patch, OOZIE-1319.patch,
> OOZIE-1319.patch, OOZIE-1319.patch, oozie-1319.patch
>
>
> In execute() of CoordJobGetReadyActionsJPAExecutor.java, once we retrieve the
> top item from a "LIFO" query result, we do not discard or delete the
> remaining items from the result list. As a result, the next time execute() is
> invoked, we will be retrieving the next item in line. Consequently, LAST_ONLY
> strategy will also execute all ready actions for a given coordinator job,
> making it no different than LIFO.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (OOZIE-1319) "LAST_ONLY" in execution control for coordinator job still runs all the actions
[
https://issues.apache.org/jira/browse/OOZIE-1319?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14011426#comment-14011426
]
Robert Kanter commented on OOZIE-1319:
--
{quote}Can you change the check for "LAST_ONLY" execution order to
"ignoreCase"?{quote}
When checking LAST_ONLY I use the enum, not the String, so that shouldn't be a
problem. Unless I did one of the checks the wrong way?
I don't think we should merge OOZIE-1305 into this JIRA; this patch is big and
complicated enough. But it definitely makes sense to re-use the SKIPPED status
I added for the NONE execution order. Agree?
> "LAST_ONLY" in execution control for coordinator job still runs all the
> actions
> ---
>
> Key: OOZIE-1319
> URL: https://issues.apache.org/jira/browse/OOZIE-1319
> Project: Oozie
> Issue Type: Bug
>Reporter: Bowen Zhang
>Assignee: Robert Kanter
> Attachments: OOZIE-1319.patch, OOZIE-1319.patch, OOZIE-1319.patch,
> OOZIE-1319.patch, OOZIE-1319.patch, oozie-1319.patch
>
>
> In execute() of CoordJobGetReadyActionsJPAExecutor.java, once we retrieve the
> top item from a "LIFO" query result, we do not discard or delete the
> remaining items from the result list. As a result, the next time execute() is
> invoked, we will be retrieving the next item in line. Consequently, LAST_ONLY
> strategy will also execute all ready actions for a given coordinator job,
> making it no different than LIFO.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (OOZIE-1319) "LAST_ONLY" in execution control for coordinator job still runs all the actions
[ https://issues.apache.org/jira/browse/OOZIE-1319?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14011370#comment-14011370 ] Bowen Zhang commented on OOZIE-1319: As a follow-on, I want to merge 1305 into this feature once you check this in. I am just adding a new order called "NONE". so, we don't need to change any DB. > "LAST_ONLY" in execution control for coordinator job still runs all the > actions > --- > > Key: OOZIE-1319 > URL: https://issues.apache.org/jira/browse/OOZIE-1319 > Project: Oozie > Issue Type: Bug >Reporter: Bowen Zhang >Assignee: Robert Kanter > Attachments: OOZIE-1319.patch, OOZIE-1319.patch, OOZIE-1319.patch, > OOZIE-1319.patch, OOZIE-1319.patch, oozie-1319.patch > > > In execute() of CoordJobGetReadyActionsJPAExecutor.java, once we retrieve the > top item from a "LIFO" query result, we do not discard or delete the > remaining items from the result list. As a result, the next time execute() is > invoked, we will be retrieving the next item in line. Consequently, LAST_ONLY > strategy will also execute all ready actions for a given coordinator job, > making it no different than LIFO. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (OOZIE-1319) "LAST_ONLY" in execution control for coordinator job still runs all the actions
[ https://issues.apache.org/jira/browse/OOZIE-1319?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14011367#comment-14011367 ] Bowen Zhang commented on OOZIE-1319: you are right on these two points. The logic LGTM. Can you change the check for "LAST_ONLY" execution order to "ignoreCase"? +1. > "LAST_ONLY" in execution control for coordinator job still runs all the > actions > --- > > Key: OOZIE-1319 > URL: https://issues.apache.org/jira/browse/OOZIE-1319 > Project: Oozie > Issue Type: Bug >Reporter: Bowen Zhang >Assignee: Robert Kanter > Attachments: OOZIE-1319.patch, OOZIE-1319.patch, OOZIE-1319.patch, > OOZIE-1319.patch, OOZIE-1319.patch, oozie-1319.patch > > > In execute() of CoordJobGetReadyActionsJPAExecutor.java, once we retrieve the > top item from a "LIFO" query result, we do not discard or delete the > remaining items from the result list. As a result, the next time execute() is > invoked, we will be retrieving the next item in line. Consequently, LAST_ONLY > strategy will also execute all ready actions for a given coordinator job, > making it no different than LIFO. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (OOZIE-1319) "LAST_ONLY" in execution control for coordinator job still runs all the actions
[
https://issues.apache.org/jira/browse/OOZIE-1319?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14010674#comment-14010674
]
Robert Kanter commented on OOZIE-1319:
--
{quote}When oozie server is back up after more than an hour of downtime, you
may only skip the first hour of actions, but you still will materialize and run
all actions with nominal times in the past one or several hours during downtime
since CoordActionInputCheckXCommand runs before
CoordMaterializationXCommand.{quote}
By CoordMaterializationXCommand, did you mean
CoordMaterializeTransitionXCommand? Doesn't CoordActionInputCheckXCommand get
run periodically to check if an action's dependencies are met?
{quote}Second, you CoordActionInputCheckXCommand relies on a big assumption
that the latest action will get called first by CoordActionInputCheckXCommand
since if the second to the latest one gets called first in
CoordActionInputCheckXCommand, it will mark the latest action which is supposed
to run as "SKIPPED".{quote}
I don't think that's the case. CoordActionInputCheckXCommand looks at the next
action, but it only edits the "current' action. That is, if it's checking
action 5, it will look at action 6's nominal time, but it would set action 5's
status only, not action 6's; that happens in a different call to
CoordActionInputCheckXCommand. I don't think the order makes a difference
here. Can you give an example of what you mean?
The new approach I used should inherently work because it's doing something
very similar to how the timeout stuff works, and that doesn't have any problems
AFAIK.
> "LAST_ONLY" in execution control for coordinator job still runs all the
> actions
> ---
>
> Key: OOZIE-1319
> URL: https://issues.apache.org/jira/browse/OOZIE-1319
> Project: Oozie
> Issue Type: Bug
>Reporter: Bowen Zhang
>Assignee: Robert Kanter
> Attachments: OOZIE-1319.patch, OOZIE-1319.patch, OOZIE-1319.patch,
> OOZIE-1319.patch, OOZIE-1319.patch, oozie-1319.patch
>
>
> In execute() of CoordJobGetReadyActionsJPAExecutor.java, once we retrieve the
> top item from a "LIFO" query result, we do not discard or delete the
> remaining items from the result list. As a result, the next time execute() is
> invoked, we will be retrieving the next item in line. Consequently, LAST_ONLY
> strategy will also execute all ready actions for a given coordinator job,
> making it no different than LIFO.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (OOZIE-1319) "LAST_ONLY" in execution control for coordinator job still runs all the actions
[ https://issues.apache.org/jira/browse/OOZIE-1319?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14010654#comment-14010654 ] Bowen Zhang commented on OOZIE-1319: I don't see you dealing with the race condition between RecoveryService and CoordMaterilalizeTriggerService.. When oozie server is back up after more than an hour of downtime, you may only skip the first hour of actions, but you still will materialize and run all actions with nominal times in the past one or several hours during downtime since CoordActionInputCheckXCommand runs before CoordMaterializationXCommand. Second, you CoordActionInputCheckXCommand relies on a big assumption that the latest action will get called first by CoordActionInputCheckXCommand since if the second to the latest one gets called first in CoordActionInputCheckXCommand, it will mark the latest action which is supposed to run as "SKIPPED". > "LAST_ONLY" in execution control for coordinator job still runs all the > actions > --- > > Key: OOZIE-1319 > URL: https://issues.apache.org/jira/browse/OOZIE-1319 > Project: Oozie > Issue Type: Bug >Reporter: Bowen Zhang >Assignee: Robert Kanter > Attachments: OOZIE-1319.patch, OOZIE-1319.patch, OOZIE-1319.patch, > OOZIE-1319.patch, OOZIE-1319.patch, oozie-1319.patch > > > In execute() of CoordJobGetReadyActionsJPAExecutor.java, once we retrieve the > top item from a "LIFO" query result, we do not discard or delete the > remaining items from the result list. As a result, the next time execute() is > invoked, we will be retrieving the next item in line. Consequently, LAST_ONLY > strategy will also execute all ready actions for a given coordinator job, > making it no different than LIFO. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (OOZIE-1319) "LAST_ONLY" in execution control for coordinator job still runs all the actions
[
https://issues.apache.org/jira/browse/OOZIE-1319?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14008432#comment-14008432
]
Hadoop QA commented on OOZIE-1319:
--
Testing JIRA OOZIE-1319
Cleaning local git workspace
{color:green}+1 PATCH_APPLIES{color}
{color:green}+1 CLEAN{color}
{color:red}-1 RAW_PATCH_ANALYSIS{color}
.{color:green}+1{color} the patch does not introduce any @author tags
.{color:green}+1{color} the patch does not introduce any tabs
.{color:green}+1{color} the patch does not introduce any trailing spaces
.{color:red}-1{color} the patch contains 5 line(s) longer than 132
characters
.{color:green}+1{color} the patch does adds/modifies 8 testcase(s)
{color:green}+1 RAT{color}
.{color:green}+1{color} the patch does not seem to introduce new RAT
warnings
{color:green}+1 JAVADOC{color}
.{color:green}+1{color} the patch does not seem to introduce new Javadoc
warnings
{color:green}+1 COMPILE{color}
.{color:green}+1{color} HEAD compiles
.{color:green}+1{color} patch compiles
.{color:green}+1{color} the patch does not seem to introduce new javac
warnings
{color:green}+1 BACKWARDS_COMPATIBILITY{color}
.{color:green}+1{color} the patch does not change any JPA
Entity/Colum/Basic/Lob/Transient annotations
.{color:green}+1{color} the patch does not modify JPA files
{color:red}-1 TESTS{color}
.Tests run: 1458
.Tests failed: 6
.Tests errors: 4
.The patch failed the following testcases:
.
testCoordStatusTransitServiceDoneWithError(org.apache.oozie.service.TestStatusTransitService)
.
testCoordinatorActionCommandsSubmitAndStart(org.apache.oozie.sla.TestSLAEventGeneration)
.
testBundleRerunInPausedWithError(org.apache.oozie.command.bundle.TestBundleRerunXCommand)
.
testCoordDefUnsupportedChange(org.apache.oozie.command.coord.TestCoordUpdateXCommand)
.
testTimeOutWithUnresolvedMissingDependencies(org.apache.oozie.command.coord.TestCoordPushDependencyCheckXCommand)
. testEngine(org.apache.oozie.TestCoordinatorEngine)
{color:green}+1 DISTRO{color}
.{color:green}+1{color} distro tarball builds with the patch
{color:red}*-1 Overall result, please check the reported -1(s)*{color}
The full output of the test-patch run is available at
. https://builds.apache.org/job/oozie-trunk-precommit-build/1263/
> "LAST_ONLY" in execution control for coordinator job still runs all the
> actions
> ---
>
> Key: OOZIE-1319
> URL: https://issues.apache.org/jira/browse/OOZIE-1319
> Project: Oozie
> Issue Type: Bug
>Reporter: Bowen Zhang
>Assignee: Robert Kanter
> Attachments: OOZIE-1319.patch, OOZIE-1319.patch, OOZIE-1319.patch,
> OOZIE-1319.patch, OOZIE-1319.patch, oozie-1319.patch
>
>
> In execute() of CoordJobGetReadyActionsJPAExecutor.java, once we retrieve the
> top item from a "LIFO" query result, we do not discard or delete the
> remaining items from the result list. As a result, the next time execute() is
> invoked, we will be retrieving the next item in line. Consequently, LAST_ONLY
> strategy will also execute all ready actions for a given coordinator job,
> making it no different than LIFO.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (OOZIE-1319) "LAST_ONLY" in execution control for coordinator job still runs all the actions
[
https://issues.apache.org/jira/browse/OOZIE-1319?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14003922#comment-14003922
]
Robert Kanter commented on OOZIE-1319:
--
Looking at this again from another perspective: isn't setting LAST_ONLY
effectively equivalent to doing FIFO with a timeout set to the amount of time
between actions? For example, if you have actions materializing every 10min
and also have a timeout of 10min, then when the time dependency for the next
action is met, the previous action would become TIMEDOUT. This behavior
already exists. So, perhaps we can re-use that logic so that when LAST_ONLY is
set, the timeout for the {{current_action}} is set to
{{MIN(timeout_set_by_the_user, next_action.nominal_time –
current_action.nominal_time)}}? We then wouldn't need most of the other
changes I've already made and we'd reuse the TIMEDOUT status instead of the new
SKIPPED status.
Thoughts?
> "LAST_ONLY" in execution control for coordinator job still runs all the
> actions
> ---
>
> Key: OOZIE-1319
> URL: https://issues.apache.org/jira/browse/OOZIE-1319
> Project: Oozie
> Issue Type: Bug
>Reporter: Bowen Zhang
>Assignee: Robert Kanter
> Attachments: OOZIE-1319.patch, OOZIE-1319.patch, OOZIE-1319.patch,
> OOZIE-1319.patch, oozie-1319.patch
>
>
> In execute() of CoordJobGetReadyActionsJPAExecutor.java, once we retrieve the
> top item from a "LIFO" query result, we do not discard or delete the
> remaining items from the result list. As a result, the next time execute() is
> invoked, we will be retrieving the next item in line. Consequently, LAST_ONLY
> strategy will also execute all ready actions for a given coordinator job,
> making it no different than LIFO.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (OOZIE-1319) "LAST_ONLY" in execution control for coordinator job still runs all the actions
[ https://issues.apache.org/jira/browse/OOZIE-1319?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13993721#comment-13993721 ] Bowen Zhang commented on OOZIE-1319: You can certainly do that. But, this is equivalent of having some blocking mechanism between RecoveryService and MaterializationService. And the "Ignore" should only apply when server is back up, not during regular recovery Runnable. Otherwise, job with "Last_only" never gets to run. So, we all need to just go back to my approach in oozie-1305. > "LAST_ONLY" in execution control for coordinator job still runs all the > actions > --- > > Key: OOZIE-1319 > URL: https://issues.apache.org/jira/browse/OOZIE-1319 > Project: Oozie > Issue Type: Bug >Reporter: Bowen Zhang >Assignee: Robert Kanter > Attachments: OOZIE-1319.patch, OOZIE-1319.patch, OOZIE-1319.patch, > OOZIE-1319.patch, oozie-1319.patch > > > In execute() of CoordJobGetReadyActionsJPAExecutor.java, once we retrieve the > top item from a "LIFO" query result, we do not discard or delete the > remaining items from the result list. As a result, the next time execute() is > invoked, we will be retrieving the next item in line. Consequently, LAST_ONLY > strategy will also execute all ready actions for a given coordinator job, > making it no different than LIFO. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (OOZIE-1319) "LAST_ONLY" in execution control for coordinator job still runs all the actions
[
https://issues.apache.org/jira/browse/OOZIE-1319?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13990279#comment-13990279
]
Bowen Zhang commented on OOZIE-1319:
[~chitnis], {quote}If materialization occurred at 10:55, why is 10:50 action
SKIPPED? it would be the last eligible action in this case right?{quote}, this
is the expected behavior, but not what his code is going to do. His new
function of handleLastOnly will mark all actions as "SKIPPED" during
materialization event.
{quote} e.g.
if action 10:50, n(SUBMITTED) = 0/1, own status - READY, action 10:40 -
SUBMITTED,
then action 10:40 -> SKIPPED, action 10:50 -> SUBMITTED
but, if action 10:50, n(SUBMITTED) = 1, own status - READY, action 10:40 -
RUNNING,
then action 10:50 would have to mark itself as SKIPPED right?
{quote}
These are all correct expected behavior. But the problem is in your first case,
marking action 10:40 from SUBMITTED to SKIPPED can't work 100% of the time
since at the mean time, RecoveryService may have already converted it into
RUNNING.
> "LAST_ONLY" in execution control for coordinator job still runs all the
> actions
> ---
>
> Key: OOZIE-1319
> URL: https://issues.apache.org/jira/browse/OOZIE-1319
> Project: Oozie
> Issue Type: Bug
>Reporter: Bowen Zhang
>Assignee: Robert Kanter
> Attachments: OOZIE-1319.patch, OOZIE-1319.patch, OOZIE-1319.patch,
> OOZIE-1319.patch, oozie-1319.patch
>
>
> In execute() of CoordJobGetReadyActionsJPAExecutor.java, once we retrieve the
> top item from a "LIFO" query result, we do not discard or delete the
> remaining items from the result list. As a result, the next time execute() is
> invoked, we will be retrieving the next item in line. Consequently, LAST_ONLY
> strategy will also execute all ready actions for a given coordinator job,
> making it no different than LIFO.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (OOZIE-1319) "LAST_ONLY" in execution control for coordinator job still runs all the actions
[ https://issues.apache.org/jira/browse/OOZIE-1319?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13990273#comment-13990273 ] Mona Chitnis commented on OOZIE-1319: - bq. A job with frequency of 10 mins and starts from 10 am till 12 pm. Say, none of the data dependency is fulfilled. Your code will break if all of the data dependencies are fulfilled between 10:55 am and 11am. Since at 10:55 am, you SKIPPED all actions in the first hour If materialization occurred at 10:55, why is 10:50 action SKIPPED? it would be the last eligible action in this case right? So we are going to have a case with multiple CoordActionReadyX commands queued up all looping over each other. To resolve this, you could have an action know the total SUBMITTED count so far and look only towards its predecessor. e.g. if action 10:50, n(SUBMITTED) = 0/1, own status - READY, action 10:40 - SUBMITTED, then action 10:40 -> SKIPPED, action 10:50 -> SUBMITTED but, if action 10:50, n(SUBMITTED) = 1, own status - READY, action 10:40 - RUNNING, then action 10:50 would have to mark itself as SKIPPED right? > "LAST_ONLY" in execution control for coordinator job still runs all the > actions > --- > > Key: OOZIE-1319 > URL: https://issues.apache.org/jira/browse/OOZIE-1319 > Project: Oozie > Issue Type: Bug >Reporter: Bowen Zhang >Assignee: Robert Kanter > Attachments: OOZIE-1319.patch, OOZIE-1319.patch, OOZIE-1319.patch, > OOZIE-1319.patch, oozie-1319.patch > > > In execute() of CoordJobGetReadyActionsJPAExecutor.java, once we retrieve the > top item from a "LIFO" query result, we do not discard or delete the > remaining items from the result list. As a result, the next time execute() is > invoked, we will be retrieving the next item in line. Consequently, LAST_ONLY > strategy will also execute all ready actions for a given coordinator job, > making it no different than LIFO. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (OOZIE-1319) "LAST_ONLY" in execution control for coordinator job still runs all the actions
[ https://issues.apache.org/jira/browse/OOZIE-1319?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13990054#comment-13990054 ] Robert Kanter commented on OOZIE-1319: -- You're right, those two cases will both be problematic. I'm not sure of a good way to solve them; any ideas? What if we made it LAST_ONLY a little more restrictive and have it effectively set the timeout equal to the frequency? I believe this would help prevent the second case because even if a bunch of data dependencies get satisfied at the same time, the older actions would have already TIMEDOUT, so you'd only have the current one. > "LAST_ONLY" in execution control for coordinator job still runs all the > actions > --- > > Key: OOZIE-1319 > URL: https://issues.apache.org/jira/browse/OOZIE-1319 > Project: Oozie > Issue Type: Bug >Reporter: Bowen Zhang >Assignee: Robert Kanter > Attachments: OOZIE-1319.patch, OOZIE-1319.patch, OOZIE-1319.patch, > OOZIE-1319.patch, oozie-1319.patch > > > In execute() of CoordJobGetReadyActionsJPAExecutor.java, once we retrieve the > top item from a "LIFO" query result, we do not discard or delete the > remaining items from the result list. As a result, the next time execute() is > invoked, we will be retrieving the next item in line. Consequently, LAST_ONLY > strategy will also execute all ready actions for a given coordinator job, > making it no different than LIFO. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (OOZIE-1319) "LAST_ONLY" in execution control for coordinator job still runs all the actions
[ https://issues.apache.org/jira/browse/OOZIE-1319?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13988812#comment-13988812 ] Bowen Zhang commented on OOZIE-1319: two things here: * A job with frequency of 10 mins and starts from 10 am till 12 pm. Say, none of the data dependency is fulfilled. Your code will break if all of the data dependencies are fulfilled between 10:55 am and 11am. Since at 10:55 am, you SKIPPED all actions in the first hour and materialize all future actions. But by definition, if all data dependencies are available at 10:56 am, then, the action at 10:50 am should run. * This is a bigger issue with your change in CoordActionReadyXCommand. Say, a job with 10 mins frequency starts from 10 am to 11am. And all the data dependencies are realized at 10:35 am. So by definition, actions at 10:00, 10:10, and 10:20 will be SKIPPED. But, do you realize that CoordActionReadyXCommand is invoked per action in CoordActionInputCheckXCommand right after each action is set to READY? And the query "GET_COORD_ACTIONS_WAITING_SUBMITTED_OLDER_THAN" has no order preference. As a result, all your "for loop" in ReadyXCommand will only contain one action which you set to SUBMITTED. It's very likely at the end, all actions will be run. > "LAST_ONLY" in execution control for coordinator job still runs all the > actions > --- > > Key: OOZIE-1319 > URL: https://issues.apache.org/jira/browse/OOZIE-1319 > Project: Oozie > Issue Type: Bug >Reporter: Bowen Zhang >Assignee: Robert Kanter > Attachments: OOZIE-1319.patch, OOZIE-1319.patch, OOZIE-1319.patch, > OOZIE-1319.patch, oozie-1319.patch > > > In execute() of CoordJobGetReadyActionsJPAExecutor.java, once we retrieve the > top item from a "LIFO" query result, we do not discard or delete the > remaining items from the result list. As a result, the next time execute() is > invoked, we will be retrieving the next item in line. Consequently, LAST_ONLY > strategy will also execute all ready actions for a given coordinator job, > making it no different than LIFO. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (OOZIE-1319) "LAST_ONLY" in execution control for coordinator job still runs all the actions
[ https://issues.apache.org/jira/browse/OOZIE-1319?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13983995#comment-13983995 ] Robert Kanter commented on OOZIE-1319: -- Test failures appear to be unrelated > "LAST_ONLY" in execution control for coordinator job still runs all the > actions > --- > > Key: OOZIE-1319 > URL: https://issues.apache.org/jira/browse/OOZIE-1319 > Project: Oozie > Issue Type: Bug >Reporter: Bowen Zhang >Assignee: Robert Kanter > Attachments: OOZIE-1319.patch, OOZIE-1319.patch, OOZIE-1319.patch, > OOZIE-1319.patch, oozie-1319.patch > > > In execute() of CoordJobGetReadyActionsJPAExecutor.java, once we retrieve the > top item from a "LIFO" query result, we do not discard or delete the > remaining items from the result list. As a result, the next time execute() is > invoked, we will be retrieving the next item in line. Consequently, LAST_ONLY > strategy will also execute all ready actions for a given coordinator job, > making it no different than LIFO. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (OOZIE-1319) "LAST_ONLY" in execution control for coordinator job still runs all the actions
[
https://issues.apache.org/jira/browse/OOZIE-1319?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13983896#comment-13983896
]
Hadoop QA commented on OOZIE-1319:
--
Testing JIRA OOZIE-1319
Cleaning local git workspace
{color:green}+1 PATCH_APPLIES{color}
{color:green}+1 CLEAN{color}
{color:red}-1 RAW_PATCH_ANALYSIS{color}
.{color:green}+1{color} the patch does not introduce any @author tags
.{color:green}+1{color} the patch does not introduce any tabs
.{color:green}+1{color} the patch does not introduce any trailing spaces
.{color:red}-1{color} the patch contains 3 line(s) longer than 132
characters
.{color:green}+1{color} the patch does adds/modifies 7 testcase(s)
{color:green}+1 RAT{color}
.{color:green}+1{color} the patch does not seem to introduce new RAT
warnings
{color:green}+1 JAVADOC{color}
.{color:green}+1{color} the patch does not seem to introduce new Javadoc
warnings
{color:red}-1 COMPILE{color}
.{color:green}+1{color} HEAD compiles
.{color:green}+1{color} patch compiles
.{color:red}-1{color} the patch seems to introduce 1 new javac warning(s)
{color:green}+1 BACKWARDS_COMPATIBILITY{color}
.{color:green}+1{color} the patch does not change any JPA
Entity/Colum/Basic/Lob/Transient annotations
.{color:green}+1{color} the patch does not modify JPA files
{color:red}-1 TESTS{color}
.Tests run: 1445
.Tests failed: 5
.Tests errors: 2
.The patch failed the following testcases:
.
testQueueUniquenessWithSameKey(org.apache.oozie.service.TestCallableQueueService)
. testJobLog(org.apache.oozie.client.TestOozieCLI)
.
testCoordinatorActionCommandsSubmitAndStart(org.apache.oozie.sla.TestSLAEventGeneration)
.
testTimeOutWithUnresolvedMissingDependencies(org.apache.oozie.command.coord.TestCoordPushDependencyCheckXCommand)
. testBundleId(org.apache.oozie.servlet.TestBulkMonitorWebServiceAPI)
{color:green}+1 DISTRO{color}
.{color:green}+1{color} distro tarball builds with the patch
{color:red}*-1 Overall result, please check the reported -1(s)*{color}
The full output of the test-patch run is available at
. https://builds.apache.org/job/oozie-trunk-precommit-build/1205/
> "LAST_ONLY" in execution control for coordinator job still runs all the
> actions
> ---
>
> Key: OOZIE-1319
> URL: https://issues.apache.org/jira/browse/OOZIE-1319
> Project: Oozie
> Issue Type: Bug
>Reporter: Bowen Zhang
>Assignee: Robert Kanter
> Attachments: OOZIE-1319.patch, OOZIE-1319.patch, OOZIE-1319.patch,
> OOZIE-1319.patch, oozie-1319.patch
>
>
> In execute() of CoordJobGetReadyActionsJPAExecutor.java, once we retrieve the
> top item from a "LIFO" query result, we do not discard or delete the
> remaining items from the result list. As a result, the next time execute() is
> invoked, we will be retrieving the next item in line. Consequently, LAST_ONLY
> strategy will also execute all ready actions for a given coordinator job,
> making it no different than LIFO.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (OOZIE-1319) "LAST_ONLY" in execution control for coordinator job still runs all the actions
[ https://issues.apache.org/jira/browse/OOZIE-1319?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13981821#comment-13981821 ] Robert Kanter commented on OOZIE-1319: -- Here's a rough idea of what I'm currently thinking: # When materializing actions in CoordMaterializeTransitionXCommand, if we're in the past and LAST_ONLY, then it needs to materialize *all* actions until "now" or the end of the job (whichever is earliest) and set all but the latest one to SKIPPED (so we ignore the window in this case). And when materializing more actions (at the next window), it should set any old WAITING ones to SKIPPED as well. This doesn't touch any actions materialized in the future. # When getting READY actions in CoordActionReadyXCommand, only the latest READY action should become SUBMITTED; the rest should be SKIPPED (re-use LIFO query here) Just a note to clarify: even with LAST_ONLY it is possible for multiple jobs to be RUNNING at the same time. e.g. if an older action that used to be the latest is still RUNNING when new actions come along, then that older action and the latest of the newer actions could both be RUNNING. > "LAST_ONLY" in execution control for coordinator job still runs all the > actions > --- > > Key: OOZIE-1319 > URL: https://issues.apache.org/jira/browse/OOZIE-1319 > Project: Oozie > Issue Type: Bug >Reporter: Bowen Zhang >Assignee: Robert Kanter > Attachments: OOZIE-1319.patch, OOZIE-1319.patch, OOZIE-1319.patch, > oozie-1319.patch > > > In execute() of CoordJobGetReadyActionsJPAExecutor.java, once we retrieve the > top item from a "LIFO" query result, we do not discard or delete the > remaining items from the result list. As a result, the next time execute() is > invoked, we will be retrieving the next item in line. Consequently, LAST_ONLY > strategy will also execute all ready actions for a given coordinator job, > making it no different than LIFO. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (OOZIE-1319) "LAST_ONLY" in execution control for coordinator job still runs all the actions
[
https://issues.apache.org/jira/browse/OOZIE-1319?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13981803#comment-13981803
]
Robert Kanter commented on OOZIE-1319:
--
First scenario: I understand now. You are correct and this doesn't work
properly because of the windowing. It needs to take into account all of the
backlogged actions.
Second scenario: I understand now. The CoordMaterializationService doesn't run
frequently enough against this coordinator job (and it's not supposed to) for
what I did to work properly.
{quote}Not true. What if at the same time, another coordinator job with "LIFO"
execution order is running? Will you "SKIPPED" all its actions?{quote}
I was being stupid here :)
The query should take the coordinator job id as an argument. Good catch; I was
only trying this with one coord job, so I didn't run into this being an issue.
I think you might be right and I need to look into maybe doing something in
CoordActionReadyXCommand or possibly both there and
CoordMaterializeTransitionXCommand. Whatever it is, it definitely needs to
take into account if a materialization is in the future or in the past because
the behavior should be different. Materialized actions in the past should be
SKIPPED (except for the latest one once we catch up or the job ends), while
materialized actions in the future should not be SKIPPED because we haven't
gotten there yet. And when materializing in the past, the materialization
window makes things trickier.
Do you agree?
> "LAST_ONLY" in execution control for coordinator job still runs all the
> actions
> ---
>
> Key: OOZIE-1319
> URL: https://issues.apache.org/jira/browse/OOZIE-1319
> Project: Oozie
> Issue Type: Bug
>Reporter: Bowen Zhang
>Assignee: Robert Kanter
> Attachments: OOZIE-1319.patch, OOZIE-1319.patch, OOZIE-1319.patch,
> oozie-1319.patch
>
>
> In execute() of CoordJobGetReadyActionsJPAExecutor.java, once we retrieve the
> top item from a "LIFO" query result, we do not discard or delete the
> remaining items from the result list. As a result, the next time execute() is
> invoked, we will be retrieving the next item in line. Consequently, LAST_ONLY
> strategy will also execute all ready actions for a given coordinator job,
> making it no different than LIFO.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (OOZIE-1319) "LAST_ONLY" in execution control for coordinator job still runs all the actions
[ https://issues.apache.org/jira/browse/OOZIE-1319?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13981788#comment-13981788 ] Bowen Zhang commented on OOZIE-1319: correction for my second point. In the second scenario, actions from 9:00 am to 9:50 am will be correctly skipped, but actions from 10:00 am till 10:50 am will run to "SUCCEDDED". > "LAST_ONLY" in execution control for coordinator job still runs all the > actions > --- > > Key: OOZIE-1319 > URL: https://issues.apache.org/jira/browse/OOZIE-1319 > Project: Oozie > Issue Type: Bug >Reporter: Bowen Zhang >Assignee: Robert Kanter > Attachments: OOZIE-1319.patch, OOZIE-1319.patch, OOZIE-1319.patch, > oozie-1319.patch > > > In execute() of CoordJobGetReadyActionsJPAExecutor.java, once we retrieve the > top item from a "LIFO" query result, we do not discard or delete the > remaining items from the result list. As a result, the next time execute() is > invoked, we will be retrieving the next item in line. Consequently, LAST_ONLY > strategy will also execute all ready actions for a given coordinator job, > making it no different than LIFO. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (OOZIE-1319) "LAST_ONLY" in execution control for coordinator job still runs all the actions
[
https://issues.apache.org/jira/browse/OOZIE-1319?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13981787#comment-13981787
]
Bowen Zhang commented on OOZIE-1319:
Let me give a more detailed description of the two scenarios
* same coordinator job. If oozie server is back up at 11am,
CoordMaterializationService will run immediately and create 6 actions from 9am
to 9:50am which the first 5 actions will be "SKIPPED". At the same time,
RecoveryService is very likely to kick off the action at 9:50am and turn the
action into "SUCCEEDED" even before your CoordMaterializationService runs again
to materialize actions from 10am to 10:50am where your
"GET_UNSCHEDULED_ACTIONS" won't be able to retrieve the action at 9:50am. As a
result, both actions at 9:50am and 10:50am are "SUCCEEDED". This is not the
correct behavior.
* same coordinator job with a data dependency. oozie server is always up from
the beginning. If data dependency is fulfilled at 10:35am, then by definition,
the action at 10:30 am should run and all the previous ones should be
"SKIPPED". But in your code, everything will run into "SUCCEEDED" since the
next CoordMaterializationService will run and try to materialize new actions at
10:55 am. At 10:30 am, there are already 12 actions in "WAITING" state in the
database since at 8:55 am and 9:55 am, actions ARE NOT marked "SKIPPED" during
materialization stage.
* {quote} It only runs "GET_UNSCHEDULED_ACTIONS" if the coordinator job's
execution strategy is LAST_ONLY; there's an if statement in handleLastOnly()
{quote} Not true. What if at the same time, another coordinator job with "LIFO"
execution order is running? Will you "SKIPPED" all its actions?
> "LAST_ONLY" in execution control for coordinator job still runs all the
> actions
> ---
>
> Key: OOZIE-1319
> URL: https://issues.apache.org/jira/browse/OOZIE-1319
> Project: Oozie
> Issue Type: Bug
>Reporter: Bowen Zhang
>Assignee: Robert Kanter
> Attachments: OOZIE-1319.patch, OOZIE-1319.patch, OOZIE-1319.patch,
> oozie-1319.patch
>
>
> In execute() of CoordJobGetReadyActionsJPAExecutor.java, once we retrieve the
> top item from a "LIFO" query result, we do not discard or delete the
> remaining items from the result list. As a result, the next time execute() is
> invoked, we will be retrieving the next item in line. Consequently, LAST_ONLY
> strategy will also execute all ready actions for a given coordinator job,
> making it no different than LIFO.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (OOZIE-1319) "LAST_ONLY" in execution control for coordinator job still runs all the actions
[
https://issues.apache.org/jira/browse/OOZIE-1319?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13981725#comment-13981725
]
Robert Kanter commented on OOZIE-1319:
--
{quote}After oozie server is back up, when you run the second materialization,
previously materialized actions may already be in RUNNING state or even
"SUCCEEDED" state, therefore your "get_unscheduled_actions" query won't
retrieve them.{quote}
Isn't that the correct behavior? If the action is already running, then I
think we should not kill it.
In the second scenario, there would only be one WAITING action becuase the
older ones should be SKIPPED. My understanding of LAST_ONLY, is that if there
are any WAITING or READY actions in the past, they should all be SKIPPED except
for the latest one. WAITING or READY actions in the future should be
unaffected (though my patch doesn't do this yet).
{quote}Third point, your "GET_UNSCHEDULED_ACTIONS" query is retrieving all
actions regardless the execution order of the parent coord jobs.{quote}
It only runs "GET_UNSCHEDULED_ACTIONS" if the coordinator job's execution
strategy is LAST_ONLY; there's an if statement in {{handleLastOnly()}}.
> "LAST_ONLY" in execution control for coordinator job still runs all the
> actions
> ---
>
> Key: OOZIE-1319
> URL: https://issues.apache.org/jira/browse/OOZIE-1319
> Project: Oozie
> Issue Type: Bug
>Reporter: Bowen Zhang
>Assignee: Robert Kanter
> Attachments: OOZIE-1319.patch, OOZIE-1319.patch, OOZIE-1319.patch,
> oozie-1319.patch
>
>
> In execute() of CoordJobGetReadyActionsJPAExecutor.java, once we retrieve the
> top item from a "LIFO" query result, we do not discard or delete the
> remaining items from the result list. As a result, the next time execute() is
> invoked, we will be retrieving the next item in line. Consequently, LAST_ONLY
> strategy will also execute all ready actions for a given coordinator job,
> making it no different than LIFO.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (OOZIE-1319) "LAST_ONLY" in execution control for coordinator job still runs all the actions
[
https://issues.apache.org/jira/browse/OOZIE-1319?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13981614#comment-13981614
]
Bowen Zhang commented on OOZIE-1319:
Neither of the two scenarios are guaranteed to work.
* The first scenario, {quote}In other words, when materializing actions, if it
sees any previously materialized actions READY/WAITING, they'll be set to
SKIPPED.{quote} After oozie server is back up, when you run the second
materialization, previously materialized actions may already be in RUNNING
state or even "SUCCEEDED" state, therefore your "get_unscheduled_actions" query
won't retrieve them.
* Second scenario, what if the data dependency is fulfilled at 10:30am? All the
WAITING actions will be in "READY" state. You need to wait till 10:55am to kick
off your materialization logic.
* Third point, your "GET_UNSCHEDULED_ACTIONS" query is retrieving all actions
regardless the execution order of the parent coord jobs.
> "LAST_ONLY" in execution control for coordinator job still runs all the
> actions
> ---
>
> Key: OOZIE-1319
> URL: https://issues.apache.org/jira/browse/OOZIE-1319
> Project: Oozie
> Issue Type: Bug
>Reporter: Bowen Zhang
>Assignee: Robert Kanter
> Attachments: OOZIE-1319.patch, OOZIE-1319.patch, OOZIE-1319.patch,
> oozie-1319.patch
>
>
> In execute() of CoordJobGetReadyActionsJPAExecutor.java, once we retrieve the
> top item from a "LIFO" query result, we do not discard or delete the
> remaining items from the result list. As a result, the next time execute() is
> invoked, we will be retrieving the next item in line. Consequently, LAST_ONLY
> strategy will also execute all ready actions for a given coordinator job,
> making it no different than LIFO.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (OOZIE-1319) "LAST_ONLY" in execution control for coordinator job still runs all the actions
[
https://issues.apache.org/jira/browse/OOZIE-1319?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13981526#comment-13981526
]
Robert Kanter commented on OOZIE-1319:
--
The second part of {{handleLastOnly()}} actually retrieves all READY or WAITING
actions already created and sets them to SKIPPED. In other words, when
materializing actions, if it sees any previously materialized actions
READY/WAITING, they'll be set to SKIPPED. So in those two scenarios, I believe
it will actually work correctly.
The only time we need to set an action to SKIPPED is if its READY/WAITING and a
newer action gets materialized. So I think putting the code in
CoordMaterializeTransitionXCommand makes sense with this in mind. I think the
issue now is that it doesn't take into account if we're materializing future
actions or not, in which case, it needs to not do this behavior.
> "LAST_ONLY" in execution control for coordinator job still runs all the
> actions
> ---
>
> Key: OOZIE-1319
> URL: https://issues.apache.org/jira/browse/OOZIE-1319
> Project: Oozie
> Issue Type: Bug
>Reporter: Bowen Zhang
>Assignee: Robert Kanter
> Attachments: OOZIE-1319.patch, OOZIE-1319.patch, OOZIE-1319.patch,
> oozie-1319.patch
>
>
> In execute() of CoordJobGetReadyActionsJPAExecutor.java, once we retrieve the
> top item from a "LIFO" query result, we do not discard or delete the
> remaining items from the result list. As a result, the next time execute() is
> invoked, we will be retrieving the next item in line. Consequently, LAST_ONLY
> strategy will also execute all ready actions for a given coordinator job,
> making it no different than LIFO.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (OOZIE-1319) "LAST_ONLY" in execution control for coordinator job still runs all the actions
[ https://issues.apache.org/jira/browse/OOZIE-1319?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13981488#comment-13981488 ] Bowen Zhang commented on OOZIE-1319: The right logic change should really happen in CoordActionReadyXCommand instead of materialization stage. Your new approach will not work in the following two scenarios: * If a coordinator job starts at 9am and ends at 12pm with frequency of 10 minutes. And oozie server is back up at 11am. It is supposed to only run the action at 10:50 am and forward. However, your new code will run the action at 10:50am AS WELL AS the action at 9:50am. * Same coordinator job with a data dependency and oozie server is always up. The job starts at 9am, but the data dependency fulfills only at 11am. ALL actions will run since they are all marked as "WAITING" and no logic in CoordActionREadyXCommand. > "LAST_ONLY" in execution control for coordinator job still runs all the > actions > --- > > Key: OOZIE-1319 > URL: https://issues.apache.org/jira/browse/OOZIE-1319 > Project: Oozie > Issue Type: Bug >Reporter: Bowen Zhang >Assignee: Robert Kanter > Attachments: OOZIE-1319.patch, OOZIE-1319.patch, OOZIE-1319.patch, > oozie-1319.patch > > > In execute() of CoordJobGetReadyActionsJPAExecutor.java, once we retrieve the > top item from a "LIFO" query result, we do not discard or delete the > remaining items from the result list. As a result, the next time execute() is > invoked, we will be retrieving the next item in line. Consequently, LAST_ONLY > strategy will also execute all ready actions for a given coordinator job, > making it no different than LIFO. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (OOZIE-1319) "LAST_ONLY" in execution control for coordinator job still runs all the actions
[
https://issues.apache.org/jira/browse/OOZIE-1319?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13981461#comment-13981461
]
Robert Kanter commented on OOZIE-1319:
--
I think this will work if I update {{handleLastOnly()}} to check if the nominal
time for the action is in the past. If it's in the past, then Oozie is doing
"catch up" and the current behavior of {{handleLastOnly()}} is correct. If
it's in the future, then {{handleLastOnly()}} should do nothing to it (so the
original behavior remains and it will be WAITING/READY). Does this sound
right?
I'll get rid of the JPAExecutor in the next patch.
> "LAST_ONLY" in execution control for coordinator job still runs all the
> actions
> ---
>
> Key: OOZIE-1319
> URL: https://issues.apache.org/jira/browse/OOZIE-1319
> Project: Oozie
> Issue Type: Bug
>Reporter: Bowen Zhang
>Assignee: Robert Kanter
> Attachments: OOZIE-1319.patch, OOZIE-1319.patch, OOZIE-1319.patch,
> oozie-1319.patch
>
>
> In execute() of CoordJobGetReadyActionsJPAExecutor.java, once we retrieve the
> top item from a "LIFO" query result, we do not discard or delete the
> remaining items from the result list. As a result, the next time execute() is
> invoked, we will be retrieving the next item in line. Consequently, LAST_ONLY
> strategy will also execute all ready actions for a given coordinator job,
> making it no different than LIFO.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (OOZIE-1319) "LAST_ONLY" in execution control for coordinator job still runs all the actions
[ https://issues.apache.org/jira/browse/OOZIE-1319?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13980516#comment-13980516 ] Bowen Zhang commented on OOZIE-1319: By spotting your patch, I see two things. * By definition, "execution order" means "the execution order if multiple instances of the coordinator job have satisfied their execution criteria" which means instances that fulfill both data and time dependency and ready to run. You added the "handleLastOnly()" in CoordMaterializeTransitionXCommand which changes the logic of regular materialization. After your changes, any coordinator job with "LAST_ONLY" execution will only execute the last action in EACH MATERIALIZATION WINDOW. For example, suppose right now is 9am and I submit a coordinator job with "LAST_ONLY" and 10 minutes frequency which starts at 10 am and ends at 1pm. As a result, among all the 18 actions, only actions with nominal time of 10:50am, 11:50am, and 12:50pm will run. All the others will be SKIPPED. That is not correct behavior. * We should not create JPAExecutor class anymore. Your query name "GET_UNSCHEDULED_ACTIONS" should be put into CoordActionQueryExecutor.java. > "LAST_ONLY" in execution control for coordinator job still runs all the > actions > --- > > Key: OOZIE-1319 > URL: https://issues.apache.org/jira/browse/OOZIE-1319 > Project: Oozie > Issue Type: Bug >Reporter: Bowen Zhang >Assignee: Robert Kanter > Attachments: OOZIE-1319.patch, OOZIE-1319.patch, OOZIE-1319.patch, > oozie-1319.patch > > > In execute() of CoordJobGetReadyActionsJPAExecutor.java, once we retrieve the > top item from a "LIFO" query result, we do not discard or delete the > remaining items from the result list. As a result, the next time execute() is > invoked, we will be retrieving the next item in line. Consequently, LAST_ONLY > strategy will also execute all ready actions for a given coordinator job, > making it no different than LIFO. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (OOZIE-1319) "LAST_ONLY" in execution control for coordinator job still runs all the actions
[
https://issues.apache.org/jira/browse/OOZIE-1319?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13980253#comment-13980253
]
Hadoop QA commented on OOZIE-1319:
--
Testing JIRA OOZIE-1319
Cleaning local git workspace
{color:green}+1 PATCH_APPLIES{color}
{color:green}+1 CLEAN{color}
{color:red}-1 RAW_PATCH_ANALYSIS{color}
.{color:green}+1{color} the patch does not introduce any @author tags
.{color:green}+1{color} the patch does not introduce any tabs
.{color:green}+1{color} the patch does not introduce any trailing spaces
.{color:red}-1{color} the patch contains 4 line(s) longer than 132
characters
.{color:green}+1{color} the patch does adds/modifies 4 testcase(s)
{color:green}+1 RAT{color}
.{color:green}+1{color} the patch does not seem to introduce new RAT
warnings
{color:green}+1 JAVADOC{color}
.{color:green}+1{color} the patch does not seem to introduce new Javadoc
warnings
{color:green}+1 COMPILE{color}
.{color:green}+1{color} HEAD compiles
.{color:green}+1{color} patch compiles
.{color:green}+1{color} the patch does not seem to introduce new javac
warnings
{color:green}+1 BACKWARDS_COMPATIBILITY{color}
.{color:green}+1{color} the patch does not change any JPA
Entity/Colum/Basic/Lob/Transient annotations
.{color:green}+1{color} the patch does not modify JPA files
{color:red}-1 TESTS{color}
.Tests run: 1426
.Tests failed: 3
.Tests errors: 4
.The patch failed the following testcases:
.
testConcurrencyReachedAndChooseNextEligible(org.apache.oozie.service.TestCallableQueueService)
.
testTimeOutWithUnresolvedMissingDependencies(org.apache.oozie.command.coord.TestCoordPushDependencyCheckXCommand)
.
testCoordinatorActionCommandsSubmitAndStart(org.apache.oozie.sla.TestSLAEventGeneration)
{color:green}+1 DISTRO{color}
.{color:green}+1{color} distro tarball builds with the patch
{color:red}*-1 Overall result, please check the reported -1(s)*{color}
The full output of the test-patch run is available at
. https://builds.apache.org/job/oozie-trunk-precommit-build/1197/
> "LAST_ONLY" in execution control for coordinator job still runs all the
> actions
> ---
>
> Key: OOZIE-1319
> URL: https://issues.apache.org/jira/browse/OOZIE-1319
> Project: Oozie
> Issue Type: Bug
>Reporter: Bowen Zhang
>Assignee: Robert Kanter
> Attachments: OOZIE-1319.patch, OOZIE-1319.patch, OOZIE-1319.patch,
> oozie-1319.patch
>
>
> In execute() of CoordJobGetReadyActionsJPAExecutor.java, once we retrieve the
> top item from a "LIFO" query result, we do not discard or delete the
> remaining items from the result list. As a result, the next time execute() is
> invoked, we will be retrieving the next item in line. Consequently, LAST_ONLY
> strategy will also execute all ready actions for a given coordinator job,
> making it no different than LIFO.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (OOZIE-1319) "LAST_ONLY" in execution control for coordinator job still runs all the actions
[
https://issues.apache.org/jira/browse/OOZIE-1319?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13973128#comment-13973128
]
Robert Kanter commented on OOZIE-1319:
--
{quote}Since skipped coord actions will not have any workflow jobs, just want
to make sure this is handled. Great if its already taken care. Thanks{quote}
Yup, actions are deleted when their associated jobs are deleted. For example,
all Coordinator actions, regardless of status or anything else, are deleted
when their associated Coordinator Job is deleted.
> "LAST_ONLY" in execution control for coordinator job still runs all the
> actions
> ---
>
> Key: OOZIE-1319
> URL: https://issues.apache.org/jira/browse/OOZIE-1319
> Project: Oozie
> Issue Type: Bug
>Reporter: Bowen Zhang
>Assignee: Robert Kanter
> Attachments: OOZIE-1319.patch, OOZIE-1319.patch, oozie-1319.patch
>
>
> In execute() of CoordJobGetReadyActionsJPAExecutor.java, once we retrieve the
> top item from a "LIFO" query result, we do not discard or delete the
> remaining items from the result list. As a result, the next time execute() is
> invoked, we will be retrieving the next item in line. Consequently, LAST_ONLY
> strategy will also execute all ready actions for a given coordinator job,
> making it no different than LIFO.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (OOZIE-1319) "LAST_ONLY" in execution control for coordinator job still runs all the actions
[
https://issues.apache.org/jira/browse/OOZIE-1319?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13972802#comment-13972802
]
Shwetha G S commented on OOZIE-1319:
{quote}
I'm still working on the patch, but it should delete the SKIPPED actions
because when looking for entries to delete, it only looks at the jobs; it then
deletes all actions belonging to the job without looking at their status. In
other words, the actions' statuses don't actually matter.
{quote}
Since skipped coord actions will not have any workflow jobs, just want to make
sure this is handled. Great if its already taken care. Thanks
> "LAST_ONLY" in execution control for coordinator job still runs all the
> actions
> ---
>
> Key: OOZIE-1319
> URL: https://issues.apache.org/jira/browse/OOZIE-1319
> Project: Oozie
> Issue Type: Bug
>Reporter: Bowen Zhang
>Assignee: Robert Kanter
> Attachments: OOZIE-1319.patch, OOZIE-1319.patch, oozie-1319.patch
>
>
> In execute() of CoordJobGetReadyActionsJPAExecutor.java, once we retrieve the
> top item from a "LIFO" query result, we do not discard or delete the
> remaining items from the result list. As a result, the next time execute() is
> invoked, we will be retrieving the next item in line. Consequently, LAST_ONLY
> strategy will also execute all ready actions for a given coordinator job,
> making it no different than LIFO.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (OOZIE-1319) "LAST_ONLY" in execution control for coordinator job still runs all the actions
[ https://issues.apache.org/jira/browse/OOZIE-1319?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13972320#comment-13972320 ] Robert Kanter commented on OOZIE-1319: -- The 4 lines that are too long are for namedQueries and twiki formatting. Test failure is unrelated. > "LAST_ONLY" in execution control for coordinator job still runs all the > actions > --- > > Key: OOZIE-1319 > URL: https://issues.apache.org/jira/browse/OOZIE-1319 > Project: Oozie > Issue Type: Bug >Reporter: Bowen Zhang >Assignee: Robert Kanter > Attachments: OOZIE-1319.patch, OOZIE-1319.patch, oozie-1319.patch > > > In execute() of CoordJobGetReadyActionsJPAExecutor.java, once we retrieve the > top item from a "LIFO" query result, we do not discard or delete the > remaining items from the result list. As a result, the next time execute() is > invoked, we will be retrieving the next item in line. Consequently, LAST_ONLY > strategy will also execute all ready actions for a given coordinator job, > making it no different than LIFO. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (OOZIE-1319) "LAST_ONLY" in execution control for coordinator job still runs all the actions
[
https://issues.apache.org/jira/browse/OOZIE-1319?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13972294#comment-13972294
]
Hadoop QA commented on OOZIE-1319:
--
Testing JIRA OOZIE-1319
Cleaning local git workspace
{color:green}+1 PATCH_APPLIES{color}
{color:green}+1 CLEAN{color}
{color:red}-1 RAW_PATCH_ANALYSIS{color}
.{color:green}+1{color} the patch does not introduce any @author tags
.{color:green}+1{color} the patch does not introduce any tabs
.{color:green}+1{color} the patch does not introduce any trailing spaces
.{color:red}-1{color} the patch contains 4 line(s) longer than 132
characters
.{color:green}+1{color} the patch does adds/modifies 4 testcase(s)
{color:green}+1 RAT{color}
.{color:green}+1{color} the patch does not seem to introduce new RAT
warnings
{color:green}+1 JAVADOC{color}
.{color:green}+1{color} the patch does not seem to introduce new Javadoc
warnings
{color:green}+1 COMPILE{color}
.{color:green}+1{color} HEAD compiles
.{color:green}+1{color} patch compiles
.{color:green}+1{color} the patch does not seem to introduce new javac
warnings
{color:green}+1 BACKWARDS_COMPATIBILITY{color}
.{color:green}+1{color} the patch does not change any JPA
Entity/Colum/Basic/Lob/Transient annotations
.{color:green}+1{color} the patch does not modify JPA files
{color:red}-1 TESTS{color}
.Tests run: 1422
.Tests failed: 0
.Tests errors: 1
.The patch failed the following testcases:
.
{color:green}+1 DISTRO{color}
.{color:green}+1{color} distro tarball builds with the patch
{color:red}*-1 Overall result, please check the reported -1(s)*{color}
The full output of the test-patch run is available at
. https://builds.apache.org/job/oozie-trunk-precommit-build/1176/
> "LAST_ONLY" in execution control for coordinator job still runs all the
> actions
> ---
>
> Key: OOZIE-1319
> URL: https://issues.apache.org/jira/browse/OOZIE-1319
> Project: Oozie
> Issue Type: Bug
>Reporter: Bowen Zhang
>Assignee: Robert Kanter
> Attachments: OOZIE-1319.patch, OOZIE-1319.patch, oozie-1319.patch
>
>
> In execute() of CoordJobGetReadyActionsJPAExecutor.java, once we retrieve the
> top item from a "LIFO" query result, we do not discard or delete the
> remaining items from the result list. As a result, the next time execute() is
> invoked, we will be retrieving the next item in line. Consequently, LAST_ONLY
> strategy will also execute all ready actions for a given coordinator job,
> making it no different than LIFO.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (OOZIE-1319) "LAST_ONLY" in execution control for coordinator job still runs all the actions
[
https://issues.apache.org/jira/browse/OOZIE-1319?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13972213#comment-13972213
]
Hadoop QA commented on OOZIE-1319:
--
Testing JIRA OOZIE-1319
Cleaning local git workspace
{color:red}-1{color} Patch failed to apply to head of branch
> "LAST_ONLY" in execution control for coordinator job still runs all the
> actions
> ---
>
> Key: OOZIE-1319
> URL: https://issues.apache.org/jira/browse/OOZIE-1319
> Project: Oozie
> Issue Type: Bug
>Reporter: Bowen Zhang
>Assignee: Robert Kanter
> Attachments: OOZIE-1319.patch, oozie-1319.patch
>
>
> In execute() of CoordJobGetReadyActionsJPAExecutor.java, once we retrieve the
> top item from a "LIFO" query result, we do not discard or delete the
> remaining items from the result list. As a result, the next time execute() is
> invoked, we will be retrieving the next item in line. Consequently, LAST_ONLY
> strategy will also execute all ready actions for a given coordinator job,
> making it no different than LIFO.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (OOZIE-1319) "LAST_ONLY" in execution control for coordinator job still runs all the actions
[ https://issues.apache.org/jira/browse/OOZIE-1319?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13971556#comment-13971556 ] Robert Kanter commented on OOZIE-1319: -- I'm still working on the patch, but it should delete the SKIPPED actions because when looking for entries to delete, it only looks at the jobs; it then deletes all actions belonging to the job without looking at their status. In other words, the actions' statuses don't actually matter. > "LAST_ONLY" in execution control for coordinator job still runs all the > actions > --- > > Key: OOZIE-1319 > URL: https://issues.apache.org/jira/browse/OOZIE-1319 > Project: Oozie > Issue Type: Bug >Reporter: Bowen Zhang >Assignee: Robert Kanter > Attachments: oozie-1319.patch > > > In execute() of CoordJobGetReadyActionsJPAExecutor.java, once we retrieve the > top item from a "LIFO" query result, we do not discard or delete the > remaining items from the result list. As a result, the next time execute() is > invoked, we will be retrieving the next item in line. Consequently, LAST_ONLY > strategy will also execute all ready actions for a given coordinator job, > making it no different than LIFO. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (OOZIE-1319) "LAST_ONLY" in execution control for coordinator job still runs all the actions
[ https://issues.apache.org/jira/browse/OOZIE-1319?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13970612#comment-13970612 ] Shwetha G S commented on OOZIE-1319: Does purge delete the old skipped coord actions? Please make sure that they are deleted, else it will just fill up the DB > "LAST_ONLY" in execution control for coordinator job still runs all the > actions > --- > > Key: OOZIE-1319 > URL: https://issues.apache.org/jira/browse/OOZIE-1319 > Project: Oozie > Issue Type: Bug >Reporter: Bowen Zhang >Assignee: Robert Kanter > Attachments: oozie-1319.patch > > > In execute() of CoordJobGetReadyActionsJPAExecutor.java, once we retrieve the > top item from a "LIFO" query result, we do not discard or delete the > remaining items from the result list. As a result, the next time execute() is > invoked, we will be retrieving the next item in line. Consequently, LAST_ONLY > strategy will also execute all ready actions for a given coordinator job, > making it no different than LIFO. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (OOZIE-1319) "LAST_ONLY" in execution control for coordinator job still runs all the actions
[ https://issues.apache.org/jira/browse/OOZIE-1319?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13967878#comment-13967878 ] Shaik Idris Ali commented on OOZIE-1319: We have some usecases related to realtime data pipeline with frequency around minutes(5), this feature is very handy specially when we have huge backlog of workflow instances, wherein we just pick the latest action and discard the rest. > "LAST_ONLY" in execution control for coordinator job still runs all the > actions > --- > > Key: OOZIE-1319 > URL: https://issues.apache.org/jira/browse/OOZIE-1319 > Project: Oozie > Issue Type: Bug >Reporter: Bowen Zhang >Assignee: Bowen Zhang > Attachments: oozie-1319.patch > > > In execute() of CoordJobGetReadyActionsJPAExecutor.java, once we retrieve the > top item from a "LIFO" query result, we do not discard or delete the > remaining items from the result list. As a result, the next time execute() is > invoked, we will be retrieving the next item in line. Consequently, LAST_ONLY > strategy will also execute all ready actions for a given coordinator job, > making it no different than LIFO. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (OOZIE-1319) "LAST_ONLY" in execution control for coordinator job still runs all the actions
[ https://issues.apache.org/jira/browse/OOZIE-1319?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13966928#comment-13966928 ] Bowen Zhang commented on OOZIE-1319: go ahead and take it. But, don't try too hard since I haven't encountered any customers using this property. > "LAST_ONLY" in execution control for coordinator job still runs all the > actions > --- > > Key: OOZIE-1319 > URL: https://issues.apache.org/jira/browse/OOZIE-1319 > Project: Oozie > Issue Type: Bug >Reporter: Bowen Zhang >Assignee: Bowen Zhang > Attachments: oozie-1319.patch > > > In execute() of CoordJobGetReadyActionsJPAExecutor.java, once we retrieve the > top item from a "LIFO" query result, we do not discard or delete the > remaining items from the result list. As a result, the next time execute() is > invoked, we will be retrieving the next item in line. Consequently, LAST_ONLY > strategy will also execute all ready actions for a given coordinator job, > making it no different than LIFO. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (OOZIE-1319) "LAST_ONLY" in execution control for coordinator job still runs all the actions
[
https://issues.apache.org/jira/browse/OOZIE-1319?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13966921#comment-13966921
]
Robert Kanter commented on OOZIE-1319:
--
{quote}I think the concurrency also need to be 2 if you get two actions in
waiting.{quote}
That just made both actions do RUNNING at the same time instead of in sequence
I think the problem is that the query used to get the Jobs is this (for FIFO;
FILO is just this with the order as "desc"):
{noformat}
select a.id, a.jobId, a.statusStr, a.pending, a.nominalTimestamp,
a.createdTimestamp from CoordinatorActionBean a where a.jobId = :jobId AND
a.statusStr = 'READY' order by a.nominalTimestamp
{noformat}
Even though I made the data available for both actions at the same time, Oozie
didn't transition them to READY at the same time, so when this query was
executed, there was only ever 1 action in READY -- that's why it didn't seem to
do anything. I was thinking that this may depend partly on the timing of the
actions transitioning from WAITING to READY, which doesn't happen
simultaneously for all actions.
I tried modifying the example to have 4 actions; I then waited for all of them
to be WAITING and put up the data; the \@4 went to RUNNING and the rest
eventually went to READY. I was then expecting those to go to SKIPPED because
the query should be looking for them; however, they stayed READY and just went
in sequence to RUNNING and SUCCEEDED.
tldr; the patch isn't working :(
{quote}Overall, this execution control is not really a well-defined
property.{quote}
My understanding of LAST_ONLY is that if multiple actions are WAITING or READY,
only one of them should actually run while the rest are skipped. What's not
clear to me is if the "LAST" means the oldest action (i.e. the _last_ one in
the list) or the newest action (i.e. the _last_ one to be added); though this
isn't too hard to switch around once we get the rest of this working. It may
make sense to add a "FIRST_ONLY" for the opposite or something.
[~bowenzhangusa], are you planning on still doing this? Or do you mind if I
take a stab at it?
> "LAST_ONLY" in execution control for coordinator job still runs all the
> actions
> ---
>
> Key: OOZIE-1319
> URL: https://issues.apache.org/jira/browse/OOZIE-1319
> Project: Oozie
> Issue Type: Bug
>Reporter: Bowen Zhang
>Assignee: Bowen Zhang
> Attachments: oozie-1319.patch
>
>
> In execute() of CoordJobGetReadyActionsJPAExecutor.java, once we retrieve the
> top item from a "LIFO" query result, we do not discard or delete the
> remaining items from the result list. As a result, the next time execute() is
> invoked, we will be retrieving the next item in line. Consequently, LAST_ONLY
> strategy will also execute all ready actions for a given coordinator job,
> making it no different than LIFO.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (OOZIE-1319) "LAST_ONLY" in execution control for coordinator job still runs all the actions
[ https://issues.apache.org/jira/browse/OOZIE-1319?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13966323#comment-13966323 ] Bowen Zhang commented on OOZIE-1319: I think the concurrency also need to be 2 if you get two actions in waiting. Overall, this execution control is not really a well-defined property. > "LAST_ONLY" in execution control for coordinator job still runs all the > actions > --- > > Key: OOZIE-1319 > URL: https://issues.apache.org/jira/browse/OOZIE-1319 > Project: Oozie > Issue Type: Bug >Reporter: Bowen Zhang >Assignee: Bowen Zhang > Attachments: oozie-1319.patch > > > In execute() of CoordJobGetReadyActionsJPAExecutor.java, once we retrieve the > top item from a "LIFO" query result, we do not discard or delete the > remaining items from the result list. As a result, the next time execute() is > invoked, we will be retrieving the next item in line. Consequently, LAST_ONLY > strategy will also execute all ready actions for a given coordinator job, > making it no different than LIFO. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (OOZIE-1319) "LAST_ONLY" in execution control for coordinator job still runs all the actions
[ https://issues.apache.org/jira/browse/OOZIE-1319?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13966113#comment-13966113 ] Robert Kanter commented on OOZIE-1319: -- I tried out the patch by modifying the aggregator example to be LAST_ONLY and removing the input data from HDFS. Once both actions were WAITING, I uploaded the data back into HDFS. I was expecting the first (\@1) action to go WAITING --> READY --> SKIPPED and the second (\@2) action to go WAITING --> READY --> RUNNING --> SUCCEEDED. However, they both went WAITING --> READY --> RUNNING --> SUCCEEDED; though they did it in LIFO order (that is, \@2 then \@1). So, it doesn't look like this is working properly. Looking at the code, I think it looks like it should work. [~bowenzhangusa], any ideas? Also, so I don't forget, the patch should also update the documentation to mention the SKIPPED status here: https://oozie.apache.org/docs/4.0.1/CoordinatorFunctionalSpec.html#a6.1.3.2._Coordinator_Action_Status and also where it talks about LAST_ONLY here: https://oozie.apache.org/docs/4.0.1/CoordinatorFunctionalSpec.html#a6.3._Synchronous_Coordinator_Application_Definition > "LAST_ONLY" in execution control for coordinator job still runs all the > actions > --- > > Key: OOZIE-1319 > URL: https://issues.apache.org/jira/browse/OOZIE-1319 > Project: Oozie > Issue Type: Bug >Reporter: Bowen Zhang >Assignee: Bowen Zhang > Attachments: oozie-1319.patch > > > In execute() of CoordJobGetReadyActionsJPAExecutor.java, once we retrieve the > top item from a "LIFO" query result, we do not discard or delete the > remaining items from the result list. As a result, the next time execute() is > invoked, we will be retrieving the next item in line. Consequently, LAST_ONLY > strategy will also execute all ready actions for a given coordinator job, > making it no different than LIFO. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (OOZIE-1319) "LAST_ONLY" in execution control for coordinator job still runs all the actions
[ https://issues.apache.org/jira/browse/OOZIE-1319?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13684042#comment-13684042 ] Alejandro Abdelnur commented on OOZIE-1319: --- Bowen, could you please check if the failure is related or not? > "LAST_ONLY" in execution control for coordinator job still runs all the > actions > --- > > Key: OOZIE-1319 > URL: https://issues.apache.org/jira/browse/OOZIE-1319 > Project: Oozie > Issue Type: Bug >Reporter: Bowen Zhang >Assignee: Bowen Zhang > Attachments: oozie-1319.patch > > > In execute() of CoordJobGetReadyActionsJPAExecutor.java, once we retrieve the > top item from a "LIFO" query result, we do not discard or delete the > remaining items from the result list. As a result, the next time execute() is > invoked, we will be retrieving the next item in line. Consequently, LAST_ONLY > strategy will also execute all ready actions for a given coordinator job, > making it no different than LIFO. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (OOZIE-1319) "LAST_ONLY" in execution control for coordinator job still runs all the actions
[
https://issues.apache.org/jira/browse/OOZIE-1319?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13683969#comment-13683969
]
Hadoop QA commented on OOZIE-1319:
--
Testing JIRA OOZIE-1319
Cleaning local svn workspace
{color:green}+1 PATCH_APPLIES{color}
{color:green}+1 CLEAN{color}
{color:red}-1 RAW_PATCH_ANALYSIS{color}
.{color:green}+1{color} the patch does not introduce any @author tags
.{color:green}+1{color} the patch does not introduce any tabs
.{color:green}+1{color} the patch does not introduce any trailing spaces
.{color:red}-1{color} the patch contains 3 line(s) longer than 132
characters
.{color:green}+1{color} the patch does adds/modifies 2 testcase(s)
{color:green}+1 RAT{color}
.{color:green}+1{color} the patch does not seem to introduce new RAT
warnings
{color:green}+1 JAVADOC{color}
.{color:green}+1{color} the patch does not seem to introduce new Javadoc
warnings
{color:green}+1 COMPILE{color}
.{color:green}+1{color} HEAD compiles
.{color:green}+1{color} patch compiles
.{color:green}+1{color} the patch does not seem to introduce new javac
warnings
{color:green}+1 BACKWARDS_COMPATIBILITY{color}
.{color:green}+1{color} the patch does not change any JPA
Entity/Colum/Basic/Lob/Transient annotations
.{color:green}+1{color} the patch does not modify JPA files
{color:red}-1 TESTS{color}
.Tests run: 1248
.Tests failed: 1
.Tests errors: 0
.The patch failed the following testcases:
. testCoordinatorActionEvent(org.apache.oozie.event.TestEventGeneration)
{color:green}+1 DISTRO{color}
.{color:green}+1{color} distro tarball builds with the patch
{color:red}*-1 Overall result, please check the reported -1(s)*{color}
The full output of the test-patch run is available at
. https://builds.apache.org/job/oozie-trunk-precommit-build/595/
> "LAST_ONLY" in execution control for coordinator job still runs all the
> actions
> ---
>
> Key: OOZIE-1319
> URL: https://issues.apache.org/jira/browse/OOZIE-1319
> Project: Oozie
> Issue Type: Bug
>Reporter: Bowen Zhang
>Assignee: Bowen Zhang
> Attachments: oozie-1319.patch
>
>
> In execute() of CoordJobGetReadyActionsJPAExecutor.java, once we retrieve the
> top item from a "LIFO" query result, we do not discard or delete the
> remaining items from the result list. As a result, the next time execute() is
> invoked, we will be retrieving the next item in line. Consequently, LAST_ONLY
> strategy will also execute all ready actions for a given coordinator job,
> making it no different than LIFO.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (OOZIE-1319) "LAST_ONLY" in execution control for coordinator job still runs all the actions
[
https://issues.apache.org/jira/browse/OOZIE-1319?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13682927#comment-13682927
]
Hadoop QA commented on OOZIE-1319:
--
Testing JIRA OOZIE-1319
Cleaning local svn workspace
{color:green}+1 PATCH_APPLIES{color}
{color:green}+1 CLEAN{color}
{color:red}-1 RAW_PATCH_ANALYSIS{color}
.{color:green}+1{color} the patch does not introduce any @author tags
.{color:green}+1{color} the patch does not introduce any tabs
.{color:green}+1{color} the patch does not introduce any trailing spaces
.{color:red}-1{color} the patch contains 3 line(s) longer than 132
characters
.{color:green}+1{color} the patch does adds/modifies 2 testcase(s)
{color:green}+1 RAT{color}
.{color:green}+1{color} the patch does not seem to introduce new RAT
warnings
{color:green}+1 JAVADOC{color}
.{color:green}+1{color} the patch does not seem to introduce new Javadoc
warnings
{color:green}+1 COMPILE{color}
.{color:green}+1{color} HEAD compiles
.{color:green}+1{color} patch compiles
.{color:green}+1{color} the patch does not seem to introduce new javac
warnings
{color:green}+1 BACKWARDS_COMPATIBILITY{color}
.{color:green}+1{color} the patch does not change any JPA
Entity/Colum/Basic/Lob/Transient annotations
.{color:green}+1{color} the patch does not modify JPA files
{color:red}-1 TESTS{color}
.Tests run: 1247
.Tests failed: 1
.Tests errors: 0
.The patch failed the following testcases:
. testCoordinatorActionEvent(org.apache.oozie.event.TestEventGeneration)
{color:green}+1 DISTRO{color}
.{color:green}+1{color} distro tarball builds with the patch
{color:red}*-1 Overall result, please check the reported -1(s)*{color}
The full output of the test-patch run is available at
. https://builds.apache.org/job/oozie-trunk-precommit-build/584/
> "LAST_ONLY" in execution control for coordinator job still runs all the
> actions
> ---
>
> Key: OOZIE-1319
> URL: https://issues.apache.org/jira/browse/OOZIE-1319
> Project: Oozie
> Issue Type: Bug
>Reporter: Bowen Zhang
>Assignee: Bowen Zhang
> Attachments: oozie-1319.patch
>
>
> In execute() of CoordJobGetReadyActionsJPAExecutor.java, once we retrieve the
> top item from a "LIFO" query result, we do not discard or delete the
> remaining items from the result list. As a result, the next time execute() is
> invoked, we will be retrieving the next item in line. Consequently, LAST_ONLY
> strategy will also execute all ready actions for a given coordinator job,
> making it no different than LIFO.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (OOZIE-1319) "LAST_ONLY" in execution control for coordinator job still runs all the actions
[ https://issues.apache.org/jira/browse/OOZIE-1319?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13682869#comment-13682869 ] Alejandro Abdelnur commented on OOZIE-1319: --- +1, pending jenkins, kicking off jenkins manually. > "LAST_ONLY" in execution control for coordinator job still runs all the > actions > --- > > Key: OOZIE-1319 > URL: https://issues.apache.org/jira/browse/OOZIE-1319 > Project: Oozie > Issue Type: Bug >Reporter: Bowen Zhang >Assignee: Bowen Zhang > Attachments: oozie-1319.patch > > > In execute() of CoordJobGetReadyActionsJPAExecutor.java, once we retrieve the > top item from a "LIFO" query result, we do not discard or delete the > remaining items from the result list. As a result, the next time execute() is > invoked, we will be retrieving the next item in line. Consequently, LAST_ONLY > strategy will also execute all ready actions for a given coordinator job, > making it no different than LIFO. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (OOZIE-1319) "LAST_ONLY" in execution control for coordinator job still runs all the actions
[ https://issues.apache.org/jira/browse/OOZIE-1319?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13669883#comment-13669883 ] Alejandro Abdelnur commented on OOZIE-1319: --- no need to reupload, i've kicked the build for this patch manually. > "LAST_ONLY" in execution control for coordinator job still runs all the > actions > --- > > Key: OOZIE-1319 > URL: https://issues.apache.org/jira/browse/OOZIE-1319 > Project: Oozie > Issue Type: Bug >Reporter: Bowen Zhang >Assignee: Bowen Zhang > Attachments: oozie-1319.patch > > > In execute() of CoordJobGetReadyActionsJPAExecutor.java, once we retrieve the > top item from a "LIFO" query result, we do not discard or delete the > remaining items from the result list. As a result, the next time execute() is > invoked, we will be retrieving the next item in line. Consequently, LAST_ONLY > strategy will also execute all ready actions for a given coordinator job, > making it no different than LIFO. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (OOZIE-1319) "LAST_ONLY" in execution control for coordinator job still runs all the actions
[ https://issues.apache.org/jira/browse/OOZIE-1319?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13669878#comment-13669878 ] Robert Kanter commented on OOZIE-1319: -- Probably not; can you reupload the patch to run the tests again? It's doing that thing were it can't lock some directory for the mini dfs cluster; hopefully its fixed itself by now. > "LAST_ONLY" in execution control for coordinator job still runs all the > actions > --- > > Key: OOZIE-1319 > URL: https://issues.apache.org/jira/browse/OOZIE-1319 > Project: Oozie > Issue Type: Bug >Reporter: Bowen Zhang >Assignee: Bowen Zhang > Attachments: oozie-1319.patch > > > In execute() of CoordJobGetReadyActionsJPAExecutor.java, once we retrieve the > top item from a "LIFO" query result, we do not discard or delete the > remaining items from the result list. As a result, the next time execute() is > invoked, we will be retrieving the next item in line. Consequently, LAST_ONLY > strategy will also execute all ready actions for a given coordinator job, > making it no different than LIFO. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (OOZIE-1319) "LAST_ONLY" in execution control for coordinator job still runs all the actions
[ https://issues.apache.org/jira/browse/OOZIE-1319?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13669824#comment-13669824 ] Bowen Zhang commented on OOZIE-1319: I don't think the test failures are related to my changes. > "LAST_ONLY" in execution control for coordinator job still runs all the > actions > --- > > Key: OOZIE-1319 > URL: https://issues.apache.org/jira/browse/OOZIE-1319 > Project: Oozie > Issue Type: Bug >Reporter: Bowen Zhang >Assignee: Bowen Zhang > Attachments: oozie-1319.patch > > > In execute() of CoordJobGetReadyActionsJPAExecutor.java, once we retrieve the > top item from a "LIFO" query result, we do not discard or delete the > remaining items from the result list. As a result, the next time execute() is > invoked, we will be retrieving the next item in line. Consequently, LAST_ONLY > strategy will also execute all ready actions for a given coordinator job, > making it no different than LIFO. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (OOZIE-1319) "LAST_ONLY" in execution control for coordinator job still runs all the actions
[
https://issues.apache.org/jira/browse/OOZIE-1319?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13668898#comment-13668898
]
Hadoop QA commented on OOZIE-1319:
--
Testing JIRA OOZIE-1319
Cleaning local svn workspace
{color:green}+1 PATCH_APPLIES{color}
{color:green}+1 CLEAN{color}
{color:red}-1 RAW_PATCH_ANALYSIS{color}
.{color:green}+1{color} the patch does not introduce any @author tags
.{color:green}+1{color} the patch does not introduce any tabs
.{color:green}+1{color} the patch does not introduce any trailing spaces
.{color:red}-1{color} the patch contains 3 line(s) longer than 132
characters
.{color:green}+1{color} the patch does adds/modifies 2 testcase(s)
{color:green}+1 RAT{color}
.{color:green}+1{color} the patch does not seem to introduce new RAT
warnings
{color:green}+1 JAVADOC{color}
.{color:green}+1{color} the patch does not seem to introduce new Javadoc
warnings
.{color:red}WARNING{color}: the current HEAD has 1 Javadoc warning(s)
{color:red}-1 COMPILE{color}
.{color:green}+1{color} HEAD compiles
.{color:green}+1{color} patch compiles
.{color:red}-1{color} the patch seems to introduce 1 new javac warning(s)
{color:green}+1 BACKWARDS_COMPATIBILITY{color}
.{color:green}+1{color} the patch does not change any JPA
Entity/Colum/Basic/Lob/Transient annotations
.{color:green}+1{color} the patch does not modify JPA files
{color:red}-1 TESTS{color}
.Tests run: 1240
.Tests failed: 0
.Tests errors: 1109
.The patch failed the following testcases:
.
{color:green}+1 DISTRO{color}
.{color:green}+1{color} distro tarball builds with the patch
{color:red}*-1 Overall result, please check the reported -1(s)*{color}
{color:red}. There is at least one warning, please check{color}
The full output of the test-patch run is available at
. https://builds.apache.org/job/oozie-trunk-precommit-build/546/
> "LAST_ONLY" in execution control for coordinator job still runs all the
> actions
> ---
>
> Key: OOZIE-1319
> URL: https://issues.apache.org/jira/browse/OOZIE-1319
> Project: Oozie
> Issue Type: Bug
>Reporter: Bowen Zhang
>Assignee: Bowen Zhang
> Attachments: oozie-1319.patch
>
>
> In execute() of CoordJobGetReadyActionsJPAExecutor.java, once we retrieve the
> top item from a "LIFO" query result, we do not discard or delete the
> remaining items from the result list. As a result, the next time execute() is
> invoked, we will be retrieving the next item in line. Consequently, LAST_ONLY
> strategy will also execute all ready actions for a given coordinator job,
> making it no different than LIFO.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (OOZIE-1319) "LAST_ONLY" in execution control for coordinator job still runs all the actions
[ https://issues.apache.org/jira/browse/OOZIE-1319?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13668829#comment-13668829 ] Robert Kanter commented on OOZIE-1319: -- I don't think this should affect the purge logic. I just took a quick look and it doesn't search for coordinator actions; the way it deletes coordinator actions is by searching for the underlying workflows and deleting all of the coordinator actions when we delete the coordinator job (after verifying some other stuff of course). > "LAST_ONLY" in execution control for coordinator job still runs all the > actions > --- > > Key: OOZIE-1319 > URL: https://issues.apache.org/jira/browse/OOZIE-1319 > Project: Oozie > Issue Type: Bug >Reporter: Bowen Zhang >Assignee: Bowen Zhang > Attachments: oozie-1319.patch > > > In execute() of CoordJobGetReadyActionsJPAExecutor.java, once we retrieve the > top item from a "LIFO" query result, we do not discard or delete the > remaining items from the result list. As a result, the next time execute() is > invoked, we will be retrieving the next item in line. Consequently, LAST_ONLY > strategy will also execute all ready actions for a given coordinator job, > making it no different than LIFO. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (OOZIE-1319) "LAST_ONLY" in execution control for coordinator job still runs all the actions
[ https://issues.apache.org/jira/browse/OOZIE-1319?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13668827#comment-13668827 ] Bowen Zhang commented on OOZIE-1319: [~tucu00], I will take a look at the purge logic. > "LAST_ONLY" in execution control for coordinator job still runs all the > actions > --- > > Key: OOZIE-1319 > URL: https://issues.apache.org/jira/browse/OOZIE-1319 > Project: Oozie > Issue Type: Bug >Reporter: Bowen Zhang >Assignee: Bowen Zhang > Attachments: oozie-1319.patch > > > In execute() of CoordJobGetReadyActionsJPAExecutor.java, once we retrieve the > top item from a "LIFO" query result, we do not discard or delete the > remaining items from the result list. As a result, the next time execute() is > invoked, we will be retrieving the next item in line. Consequently, LAST_ONLY > strategy will also execute all ready actions for a given coordinator job, > making it no different than LIFO. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (OOZIE-1319) "LAST_ONLY" in execution control for coordinator job still runs all the actions
[ https://issues.apache.org/jira/browse/OOZIE-1319?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13668822#comment-13668822 ] Alejandro Abdelnur commented on OOZIE-1319: --- Apologies for the delay. patch LGTM. One more thing we may have to take care is in the purge logic, can you please check this? Any reason why it has not been marked as 'submit patch' for test-patch to run? > "LAST_ONLY" in execution control for coordinator job still runs all the > actions > --- > > Key: OOZIE-1319 > URL: https://issues.apache.org/jira/browse/OOZIE-1319 > Project: Oozie > Issue Type: Bug >Reporter: Bowen Zhang >Assignee: Bowen Zhang > Attachments: oozie-1319.patch > > > In execute() of CoordJobGetReadyActionsJPAExecutor.java, once we retrieve the > top item from a "LIFO" query result, we do not discard or delete the > remaining items from the result list. As a result, the next time execute() is > invoked, we will be retrieving the next item in line. Consequently, LAST_ONLY > strategy will also execute all ready actions for a given coordinator job, > making it no different than LIFO. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (OOZIE-1319) "LAST_ONLY" in execution control for coordinator job still runs all the actions
[ https://issues.apache.org/jira/browse/OOZIE-1319?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13635657#comment-13635657 ] Alejandro Abdelnur commented on OOZIE-1319: --- It seems that a new 'SKIPPED' status makes more sense. We have to make sure this does not break things. > "LAST_ONLY" in execution control for coordinator job still runs all the > actions > --- > > Key: OOZIE-1319 > URL: https://issues.apache.org/jira/browse/OOZIE-1319 > Project: Oozie > Issue Type: Bug >Reporter: Bowen Zhang >Assignee: Bowen Zhang > > In execute() of CoordJobGetReadyActionsJPAExecutor.java, once we retrieve the > top item from a "LIFO" query result, we do not discard or delete the > remaining items from the result list. As a result, the next time execute() is > invoked, we will be retrieving the next item in line. Consequently, LAST_ONLY > strategy will also execute all ready actions for a given coordinator job, > making it no different than LIFO. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (OOZIE-1319) "LAST_ONLY" in execution control for coordinator job still runs all the actions
[ https://issues.apache.org/jira/browse/OOZIE-1319?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13630460#comment-13630460 ] Bowen Zhang commented on OOZIE-1319: [~tucu00], what should be the status of the actions we skipped using LAST_ONLY execution? Should we introduce a new status called "SKIPPED"? or do we use "KILLED". > "LAST_ONLY" in execution control for coordinator job still runs all the > actions > --- > > Key: OOZIE-1319 > URL: https://issues.apache.org/jira/browse/OOZIE-1319 > Project: Oozie > Issue Type: Bug >Reporter: Bowen Zhang > > In execute() of CoordJobGetReadyActionsJPAExecutor.java, once we retrieve the > top item from a "LIFO" query result, we do not discard or delete the > remaining items from the result list. As a result, the next time execute() is > invoked, we will be retrieving the next item in line. Consequently, LAST_ONLY > strategy will also execute all ready actions for a given coordinator job, > making it no different than LIFO. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (OOZIE-1319) "LAST_ONLY" in execution control for coordinator job still runs all the actions
[ https://issues.apache.org/jira/browse/OOZIE-1319?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13630438#comment-13630438 ] Alejandro Abdelnur commented on OOZIE-1319: --- it seems like bug > "LAST_ONLY" in execution control for coordinator job still runs all the > actions > --- > > Key: OOZIE-1319 > URL: https://issues.apache.org/jira/browse/OOZIE-1319 > Project: Oozie > Issue Type: Bug >Reporter: Bowen Zhang > > In execute() of CoordJobGetReadyActionsJPAExecutor.java, once we retrieve the > top item from a "LIFO" query result, we do not discard or delete the > remaining items from the result list. As a result, the next time execute() is > invoked, we will be retrieving the next item in line. Consequently, LAST_ONLY > strategy will also execute all ready actions for a given coordinator job, > making it no different than LIFO. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
