from:"Mona Chitnis"


 [ 
https://issues.apache.org/jira/browse/OOZIE-1984?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Mona Chitnis resolved OOZIE-1984.
-

Resolution: Fixed

Committed to trunk and 4.1.0. Thanks for review Ryota

> SLACalculator in HA mode performs duplicate operations on records with 
> completed jobs
> -
>
> Key: OOZIE-1984
> URL: https://issues.apache.org/jira/browse/OOZIE-1984
> Project: Oozie
>  Issue Type: Bug
>Affects Versions: trunk
>Reporter: Mona Chitnis
> Fix For: trunk, 4.1.0
>
> Attachments: OOZIE-1984-1.patch, OOZIE-1984.patch
>
>
> Scenario:
> SLA periodic run has already processed start,duration and end for a job's sla 
> entry. But job notification for that job came after this, and triggers the 
> sla listener.
> Buggy part:
> {code}
> SLACalculatorMemory.java
> else if 
> (Services.get().get(JobsConcurrencyService.class).isHighlyAvailableMode()) {
> // jobid might not exist in slaMap in HA Setting
> SLARegistrationBean slaRegBean = 
> SLARegistrationQueryExecutor.getInstance().get(
> SLARegQuery.GET_SLA_REG_ALL, jobId);
> if (slaRegBean != null) { // filter out jobs picked by SLA 
> job event listener
>   // but not actually configured for 
> SLA
> SLASummaryBean slaSummaryBean = 
> SLASummaryQueryExecutor.getInstance().get(
> SLASummaryQuery.GET_SLA_SUMMARY, jobId);
> slaCalc = new SLACalcStatus(slaSummaryBean, slaRegBean);
> if (slaCalc.getEventProcessed() < 7) {
> slaMap.put(jobId, slaCalc);
> }
> }
> }
> }
> if (slaCalc != null) {
> ..
> Object eventProcObj = ((SLASummaryQueryExecutor) 
> SLASummaryQueryExecutor.getInstance())
> 
> .getSingleValue(SLASummaryQuery.GET_SLA_SUMMARY_EVENTPROCESSED, jobId);
> byte eventProc = ((Byte) eventProcObj).byteValue();
> ..
> processJobEndSuccessSLA(slaCalc, startTime, endTime);
> {code}
> method processJobEndSuccesSLA goes ahead and checks second LSB bit of 
> eventProc and sends duration event _again_. So the bug here is two-fold:
>  * if all events are already processed, still invokes this function
>  * event processed is 8 (1000), so second LSB bit is unset and hence duration 
> processed.
> Fix - not invoke function when eventProc = 1000



--
This message was sent by Atlassian JIRA
(v6.2#6252)

[jira] [Updated] (OOZIE-1984) SLACalculator in HA mode performs duplicate operations on records with completed jobs


 [ 
https://issues.apache.org/jira/browse/OOZIE-1984?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Mona Chitnis updated OOZIE-1984:


Attachment: OOZIE-1984-1.patch

updated patch v-1

> SLACalculator in HA mode performs duplicate operations on records with 
> completed jobs
> -
>
> Key: OOZIE-1984
> URL: https://issues.apache.org/jira/browse/OOZIE-1984
> Project: Oozie
>  Issue Type: Bug
>Affects Versions: trunk
>Reporter: Mona Chitnis
> Fix For: trunk, 4.1.0
>
> Attachments: OOZIE-1984-1.patch, OOZIE-1984.patch
>
>
> Scenario:
> SLA periodic run has already processed start,duration and end for a job's sla 
> entry. But job notification for that job came after this, and triggers the 
> sla listener.
> Buggy part:
> {code}
> SLACalculatorMemory.java
> else if 
> (Services.get().get(JobsConcurrencyService.class).isHighlyAvailableMode()) {
> // jobid might not exist in slaMap in HA Setting
> SLARegistrationBean slaRegBean = 
> SLARegistrationQueryExecutor.getInstance().get(
> SLARegQuery.GET_SLA_REG_ALL, jobId);
> if (slaRegBean != null) { // filter out jobs picked by SLA 
> job event listener
>   // but not actually configured for 
> SLA
> SLASummaryBean slaSummaryBean = 
> SLASummaryQueryExecutor.getInstance().get(
> SLASummaryQuery.GET_SLA_SUMMARY, jobId);
> slaCalc = new SLACalcStatus(slaSummaryBean, slaRegBean);
> if (slaCalc.getEventProcessed() < 7) {
> slaMap.put(jobId, slaCalc);
> }
> }
> }
> }
> if (slaCalc != null) {
> ..
> Object eventProcObj = ((SLASummaryQueryExecutor) 
> SLASummaryQueryExecutor.getInstance())
> 
> .getSingleValue(SLASummaryQuery.GET_SLA_SUMMARY_EVENTPROCESSED, jobId);
> byte eventProc = ((Byte) eventProcObj).byteValue();
> ..
> processJobEndSuccessSLA(slaCalc, startTime, endTime);
> {code}
> method processJobEndSuccesSLA goes ahead and checks second LSB bit of 
> eventProc and sends duration event _again_. So the bug here is two-fold:
>  * if all events are already processed, still invokes this function
>  * event processed is 8 (1000), so second LSB bit is unset and hence duration 
> processed.
> Fix - not invoke function when eventProc = 1000



--
This message was sent by Atlassian JIRA
(v6.2#6252)

Re: Review Request 25166: OOZIE-1984 SLACalculator in HA mode performs duplicate operations on records with completed jobs

2014-08-28 Thread Mona Chitnis


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/25166/
---

(Updated Aug. 28, 2014, 10:55 p.m.)


Review request for oozie.


Changes
---

Ryota's comment addressed and after offline clarification


Bugs: OOZIE-1984
https://issues.apache.org/jira/browse/OOZIE-1984


Repository: oozie-git


Description
---

see jira


Diffs (updated)
-

  core/src/main/java/org/apache/oozie/sla/SLACalculatorMemory.java 3801325 

Diff: https://reviews.apache.org/r/25166/diff/


Testing
---

existing test pass. e-2-e test will follow in QA


Thanks,

Mona Chitnis

[jira] [Updated] (OOZIE-1984) SLACalculator in HA mode performs duplicate operations on records with completed jobs


 [ 
https://issues.apache.org/jira/browse/OOZIE-1984?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Mona Chitnis updated OOZIE-1984:


Attachment: OOZIE-1984.patch

> SLACalculator in HA mode performs duplicate operations on records with 
> completed jobs
> -
>
> Key: OOZIE-1984
> URL: https://issues.apache.org/jira/browse/OOZIE-1984
> Project: Oozie
>  Issue Type: Bug
>Affects Versions: trunk
>Reporter: Mona Chitnis
> Fix For: trunk, 4.1.0
>
> Attachments: OOZIE-1984.patch
>
>
> Scenario:
> SLA periodic run has already processed start,duration and end for a job's sla 
> entry. But job notification for that job came after this, and triggers the 
> sla listener.
> Buggy part:
> {code}
> SLACalculatorMemory.java
> else if 
> (Services.get().get(JobsConcurrencyService.class).isHighlyAvailableMode()) {
> // jobid might not exist in slaMap in HA Setting
> SLARegistrationBean slaRegBean = 
> SLARegistrationQueryExecutor.getInstance().get(
> SLARegQuery.GET_SLA_REG_ALL, jobId);
> if (slaRegBean != null) { // filter out jobs picked by SLA 
> job event listener
>   // but not actually configured for 
> SLA
> SLASummaryBean slaSummaryBean = 
> SLASummaryQueryExecutor.getInstance().get(
> SLASummaryQuery.GET_SLA_SUMMARY, jobId);
> slaCalc = new SLACalcStatus(slaSummaryBean, slaRegBean);
> if (slaCalc.getEventProcessed() < 7) {
> slaMap.put(jobId, slaCalc);
> }
> }
> }
> }
> if (slaCalc != null) {
> ..
> Object eventProcObj = ((SLASummaryQueryExecutor) 
> SLASummaryQueryExecutor.getInstance())
> 
> .getSingleValue(SLASummaryQuery.GET_SLA_SUMMARY_EVENTPROCESSED, jobId);
> byte eventProc = ((Byte) eventProcObj).byteValue();
> ..
> processJobEndSuccessSLA(slaCalc, startTime, endTime);
> {code}
> method processJobEndSuccesSLA goes ahead and checks second LSB bit of 
> eventProc and sends duration event _again_. So the bug here is two-fold:
>  * if all events are already processed, still invokes this function
>  * event processed is 8 (1000), so second LSB bit is unset and hence duration 
> processed.
> Fix - not invoke function when eventProc = 1000



--
This message was sent by Atlassian JIRA
(v6.2#6252)

Review Request 25166: OOZIE-1984 SLACalculator in HA mode performs duplicate operations on records with completed jobs

2014-08-28 Thread Mona Chitnis


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/25166/
---

Review request for oozie.


Bugs: OOZIE-1984
https://issues.apache.org/jira/browse/OOZIE-1984


Repository: oozie-git


Description
---

see jira


Diffs
-

  core/src/main/java/org/apache/oozie/sla/SLACalculatorMemory.java 3801325 

Diff: https://reviews.apache.org/r/25166/diff/


Testing
---

existing test pass. e-2-e test will follow in QA


Thanks,

Mona Chitnis

[jira] [Created] (OOZIE-1984) SLACalculator in HA mode performs duplicate operations on records with completed jobs

Mona Chitnis created OOZIE-1984:
---

 Summary: SLACalculator in HA mode performs duplicate operations on 
records with completed jobs
 Key: OOZIE-1984
 URL: https://issues.apache.org/jira/browse/OOZIE-1984
 Project: Oozie
  Issue Type: Bug
Affects Versions: trunk
Reporter: Mona Chitnis
 Fix For: trunk, 4.1.0


Scenario:

SLA periodic run has already processed start,duration and end for a job's sla 
entry. But job notification for that job came after this, and triggers the sla 
listener.

Buggy part:
{code}
SLACalculatorMemory.java

else if 
(Services.get().get(JobsConcurrencyService.class).isHighlyAvailableMode()) {
// jobid might not exist in slaMap in HA Setting
SLARegistrationBean slaRegBean = 
SLARegistrationQueryExecutor.getInstance().get(
SLARegQuery.GET_SLA_REG_ALL, jobId);
if (slaRegBean != null) { // filter out jobs picked by SLA job 
event listener
  // but not actually configured for SLA
SLASummaryBean slaSummaryBean = 
SLASummaryQueryExecutor.getInstance().get(
SLASummaryQuery.GET_SLA_SUMMARY, jobId);
slaCalc = new SLACalcStatus(slaSummaryBean, slaRegBean);
if (slaCalc.getEventProcessed() < 7) {
slaMap.put(jobId, slaCalc);
}
}
}
}
if (slaCalc != null) {
..
Object eventProcObj = ((SLASummaryQueryExecutor) 
SLASummaryQueryExecutor.getInstance())

.getSingleValue(SLASummaryQuery.GET_SLA_SUMMARY_EVENTPROCESSED, jobId);
byte eventProc = ((Byte) eventProcObj).byteValue();
..
processJobEndSuccessSLA(slaCalc, startTime, endTime);
{code}

method processJobEndSuccesSLA goes ahead and checks second LSB bit of eventProc 
and sends duration event _again_. So the bug here is two-fold:
 * if all events are already processed, still invokes this function
 * event processed is 8 (1000), so second LSB bit is unset and hence duration 
processed.

Fix - not invoke function when eventProc = 1000



--
This message was sent by Atlassian JIRA
(v6.2#6252)

Re: Review Request 24948: OOZIE-1940 StatusTransitService has race condition

2014-08-27 Thread Mona Chitnis


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/24948/#review51661
---


this is good cleanup and refactoring. Did cursory review to understand 
structural changes. Will review more carefully for any bugs introduced later 
today


core/src/main/java/org/apache/oozie/command/bundle/BundleStatusTransitXCommand.java
<https://reviews.apache.org/r/24948/#comment90178>

if bundleActionStatus map has some action as RUNNINGWITHERROR, why are we 
setting bundle job to PAUSED?



core/src/main/java/org/apache/oozie/command/bundle/BundleStatusTransitXCommand.java
<https://reviews.apache.org/r/24948/#comment90177>

typo in bottom



core/src/main/java/org/apache/oozie/command/coord/CoordStatusTransitXCommand.java
<https://reviews.apache.org/r/24948/#comment90179>

same typo


- Mona Chitnis


On Aug. 26, 2014, 12:30 a.m., Purshotam Shah wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/24948/
> ---
> 
> (Updated Aug. 26, 2014, 12:30 a.m.)
> 
> 
> Review request for oozie.
> 
> 
> Bugs: OOZIE-1940
> https://issues.apache.org/jira/browse/OOZIE-1940
> 
> 
> Repository: oozie-git
> 
> 
> Description
> ---
> 
> StatusTransitService has race condition
> 
> 
> Diffs
> -
> 
>   core/src/main/java/org/apache/oozie/BundleActionBean.java 5d85a4d 
>   core/src/main/java/org/apache/oozie/BundleJobBean.java 0f1670a 
>   core/src/main/java/org/apache/oozie/CoordinatorActionBean.java 795bf63 
>   core/src/main/java/org/apache/oozie/CoordinatorJobBean.java 8fd53f1 
>   core/src/main/java/org/apache/oozie/ErrorCode.java 88a2c67 
>   core/src/main/java/org/apache/oozie/command/StatusTransitXCommand.java 
> e69de29 
>   
> core/src/main/java/org/apache/oozie/command/bundle/BundleStatusTransitXCommand.java
>  e69de29 
>   
> core/src/main/java/org/apache/oozie/command/coord/CoordStatusTransitXCommand.java
>  e69de29 
>   
> core/src/main/java/org/apache/oozie/executor/jpa/BundleJobQueryExecutor.java 
> 36cd968 
>   
> core/src/main/java/org/apache/oozie/executor/jpa/CoordActionQueryExecutor.java
>  3008393 
>   core/src/main/java/org/apache/oozie/executor/jpa/CoordJobQueryExecutor.java 
> 04e6e29 
>   core/src/main/java/org/apache/oozie/service/StatusTransitService.java 
> 21ac25f 
>   core/src/test/java/org/apache/oozie/service/TestStatusTransitService.java 
> bb99138 
> 
> Diff: https://reviews.apache.org/r/24948/diff/
> 
> 
> Testing
> ---
> 
> UTC
> 
> 
> Thanks,
> 
> Purshotam Shah
> 
>

[jira] [Commented] (OOZIE-1940) StatusTransitService has race condition


[ 
https://issues.apache.org/jira/browse/OOZIE-1940?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14112362#comment-14112362
 ] 

Mona Chitnis commented on OOZIE-1940:
-

Agree with approach. Currently each run of Status Transit Service takes 
multiple seconds I believe. If it is going to hold the lock for that long, we 
have to asses the consequences on the other commands waiting for lock. E.g. 
Change command appearing to "hang" on user-facing CLI, because its 
synchronously trying to acquire lock held by STS. OOZIE-1885 should ideally 
reduce this overall time the lock is to be held by STS

> StatusTransitService has race condition
> ---
>
> Key: OOZIE-1940
> URL: https://issues.apache.org/jira/browse/OOZIE-1940
> Project: Oozie
>  Issue Type: Bug
>Reporter: Purshotam Shah
>
> StatusTransitService doesn't acquire lock while updating DB. 
> We noticed one such issue while doing HA testing, thanks to [~mchiang]
> We issue a change command to change pause time, which got executed on one 
> server. While change command was running on one server, other server started 
> executing StatusTransitService.
> Server 1 log
> {code}
> 2014-07-16 17:28:05,268  INFO StatusTransitService$StatusTransitRunnable:539 
> [pool-1-thread-13] - USER[-] GROUP[-] Acquired lock for 
> [org.apache.oozie.service.StatusTransitService]
> 2014-07-16 17:28:09,694  INFO StatusTransitService$StatusTransitRunnable:539 
> [pool-1-thread-13] - USER[-] GROUP[-] Set coordinator job 
> [0011385-140716042555-oozie-oozi-C] status to 'SUCCEEDED' from 'RUNNING' 
> 2014-07-16 17:28:15,416  INFO StatusTransitService$StatusTransitRunnable:539 
> [pool-1-thread-13] - USER[-] GROUP[-] Released lock for 
> [org.apache.oozie.service.StatusTransitService]
> {code}
> Server 2 log
> {code}
> 2014-07-16 17:28:06,499 DEBUG CoordChangeXCommand:545 [http-0.0.0.0-4443-5] - 
> USER[hadoopqa] GROUP[users] TOKEN[] APP[coordB180] 
> JOB[0011385-140716042555-oozie-oozi-C] ACTION[-] New pause/end date is : Wed 
> Jul 16 17:30:00 UTC 2014 and last action number is : 3
> 2014-07-16 17:28:06,508  INFO CoordChangeXCommand:539 [http-0.0.0.0-4443-5] - 
> USER[hadoopqa] GROUP[users] TOKEN[] APP[coordB180] 
> JOB[0011385-140716042555-oozie-oozi-C] ACTION[-] ENDED CoordChangeXCommand 
> for jobId=0011385-140716042555-oozie-oozi-C
> {code}
> CoordMaterializeTransitionXCommand has created all actions( few were in 
> waiting and few were in running state) and set doneMaterialization to true.
> Change command deletes all waiting coords, except 3 running/SUCCEEDED action 
> and reset doneMaterialization.
> StatusTransitService first loads a set of pending jobs and for each job it 
> make DB calls to check coord action status. Coord jobs are loaded only once 
> in beginning.
> This is what happened.
> 1.StatusTransitService loads the coord job which doneMaterialization is set 
> to true at 17:28:05,268 (server 1)
> 2.Change command deletes waiting cation and reset  doneMaterialization at  
> 17:28:06,508 (server 2)
> 3.StatusTransitService load actions for job, only 3 and in SUCCEEDED status. 
> It never reload the doneMaterialization at 17:28:09,694 (server 1)
> StatusTransitService overrides set job status to SUCCEEDED, bcz it's 
> doneMaterialization and all action are SUCCEEDED.



--
This message was sent by Atlassian JIRA
(v6.2#6252)

[jira] [Commented] (OOZIE-1940) StatusTransitService has race condition


[ 
https://issues.apache.org/jira/browse/OOZIE-1940?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14112359#comment-14112359
 ] 

Mona Chitnis commented on OOZIE-1940:
-

linking this as dependent of OOZIE-1885

> StatusTransitService has race condition
> ---
>
> Key: OOZIE-1940
> URL: https://issues.apache.org/jira/browse/OOZIE-1940
> Project: Oozie
>  Issue Type: Bug
>Reporter: Purshotam Shah
>
> StatusTransitService doesn't acquire lock while updating DB. 
> We noticed one such issue while doing HA testing, thanks to [~mchiang]
> We issue a change command to change pause time, which got executed on one 
> server. While change command was running on one server, other server started 
> executing StatusTransitService.
> Server 1 log
> {code}
> 2014-07-16 17:28:05,268  INFO StatusTransitService$StatusTransitRunnable:539 
> [pool-1-thread-13] - USER[-] GROUP[-] Acquired lock for 
> [org.apache.oozie.service.StatusTransitService]
> 2014-07-16 17:28:09,694  INFO StatusTransitService$StatusTransitRunnable:539 
> [pool-1-thread-13] - USER[-] GROUP[-] Set coordinator job 
> [0011385-140716042555-oozie-oozi-C] status to 'SUCCEEDED' from 'RUNNING' 
> 2014-07-16 17:28:15,416  INFO StatusTransitService$StatusTransitRunnable:539 
> [pool-1-thread-13] - USER[-] GROUP[-] Released lock for 
> [org.apache.oozie.service.StatusTransitService]
> {code}
> Server 2 log
> {code}
> 2014-07-16 17:28:06,499 DEBUG CoordChangeXCommand:545 [http-0.0.0.0-4443-5] - 
> USER[hadoopqa] GROUP[users] TOKEN[] APP[coordB180] 
> JOB[0011385-140716042555-oozie-oozi-C] ACTION[-] New pause/end date is : Wed 
> Jul 16 17:30:00 UTC 2014 and last action number is : 3
> 2014-07-16 17:28:06,508  INFO CoordChangeXCommand:539 [http-0.0.0.0-4443-5] - 
> USER[hadoopqa] GROUP[users] TOKEN[] APP[coordB180] 
> JOB[0011385-140716042555-oozie-oozi-C] ACTION[-] ENDED CoordChangeXCommand 
> for jobId=0011385-140716042555-oozie-oozi-C
> {code}
> CoordMaterializeTransitionXCommand has created all actions( few were in 
> waiting and few were in running state) and set doneMaterialization to true.
> Change command deletes all waiting coords, except 3 running/SUCCEEDED action 
> and reset doneMaterialization.
> StatusTransitService first loads a set of pending jobs and for each job it 
> make DB calls to check coord action status. Coord jobs are loaded only once 
> in beginning.
> This is what happened.
> 1.StatusTransitService loads the coord job which doneMaterialization is set 
> to true at 17:28:05,268 (server 1)
> 2.Change command deletes waiting cation and reset  doneMaterialization at  
> 17:28:06,508 (server 2)
> 3.StatusTransitService load actions for job, only 3 and in SUCCEEDED status. 
> It never reload the doneMaterialization at 17:28:09,694 (server 1)
> StatusTransitService overrides set job status to SUCCEEDED, bcz it's 
> doneMaterialization and all action are SUCCEEDED.



--
This message was sent by Atlassian JIRA
(v6.2#6252)

[jira] [Commented] (OOZIE-1885) Query optimization for StatusTransitService


[ 
https://issues.apache.org/jira/browse/OOZIE-1885?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14112357#comment-14112357
 ] 

Mona Chitnis commented on OOZIE-1885:
-

A join query is always more CPU and memory intensive. But it will probably cut 
down on the overall time it takes, because of the multiple queries in loop 
right now. Approach is fine but we should vet it with end-to-end test 
performance gains

> Query optimization for StatusTransitService
> ---
>
> Key: OOZIE-1885
> URL: https://issues.apache.org/jira/browse/OOZIE-1885
> Project: Oozie
>  Issue Type: Bug
>Reporter: Purshotam Shah
>
> {code}
>  private void coordTransit() throws JPAExecutorException, CommandException {
> List pendingJobCheckList = null;
> if (lastInstanceStartTime == null) {
> LOG.info("Running coordinator status service first instance");
> // this is the first instance, we need to check for all 
> pending jobs;
> pendingJobCheckList = jpaService.execute(new 
> CoordJobsGetPendingJPAExecutor(limit));
> }
> else {
> LOG.info("Running coordinator status service from last 
> instance time =  "
> + DateUtils.formatDateOozieTZ(lastInstanceStartTime));
> // this is not the first instance, we should only check jobs
> // that have actions or jobs been
> // updated >= start time of last service run;
> List actionsList = 
> CoordActionQueryExecutor.getInstance().getList(
> 
> CoordActionQuery.GET_COORD_ACTIONS_BY_LAST_MODIFIED_TIME, 
> lastInstanceStartTime);
> Set coordIds = new HashSet();
> for (CoordinatorActionBean action : actionsList) {
> coordIds.add(action.getJobId());
> }
> pendingJobCheckList = new ArrayList();
> for (String coordId : coordIds.toArray(new 
> String[coordIds.size()])) {
> CoordinatorJobBean coordJob;
> try {
> coordJob = 
> CoordJobQueryExecutor.getInstance().get(CoordJobQuery.GET_COORD_JOB, coordId);
> }
> catch (JPAExecutorException jpaee) {
> if (jpaee.getErrorCode().equals(ErrorCode.E0604)) {
> LOG.warn("Exception happened during 
> StatusTransitRunnable; Coordinator Job doesn't exist", jpaee);
> continue;
> } else {
> throw jpaee;
> }
> }
> // Running coord job might have pending false
> Job.Status coordJobStatus = coordJob.getStatus();
> if ((coordJob.isPending() || 
> coordJobStatus.equals(Job.Status.PAUSED)
> || coordJobStatus.equals(Job.Status.RUNNING)
> || 
> coordJobStatus.equals(Job.Status.RUNNINGWITHERROR)
> || 
> coordJobStatus.equals(Job.Status.PAUSEDWITHERROR))
> && !coordJobStatus.equals(Job.Status.IGNORED)) {
> pendingJobCheckList.add(coordJob);
> }
> }
> 
> pendingJobCheckList.addAll(CoordJobQueryExecutor.getInstance().getList(
> CoordJobQuery.GET_COORD_JOBS_CHANGED, 
> lastInstanceStartTime));
> }
> aggregateCoordJobsStatus(pendingJobCheckList);
> }
> }
> {code}
> This could be done in one sql, something like 
> select w.id, w.status, w.pending from CoordinatorJobBean w where 
> w.startTimestamp <= :matTime AND (w.statusStr = 'PREP' OR w.statusStr = 
> 'RUNNING' or w.statusStr = 'RUNNINGWITHERROR' or w.statusStr= 
> 'PAUSEDWITHERROR' and w.statusStr <> 'IGNORED') w.id in  ( select a.jobId 
> from CoordinatorActionBean a where a.lastModifiedTimestamp >= 
> :lastModifiedTime groupby a.jobId)
> Same for bundleTransit().



--
This message was sent by Atlassian JIRA
(v6.2#6252)

[jira] [Commented] (OOZIE-1847) HA - Oozie servers should shutdown (or go in safe mode) in case of ZK failure


[ 
https://issues.apache.org/jira/browse/OOZIE-1847?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14112333#comment-14112333
 ] 

Mona Chitnis commented on OOZIE-1847:
-

^^ in case of timeout > 3 seconds resulting in server shutdown and job failure

> HA - Oozie servers should shutdown (or go in safe mode) in case of ZK failure
> -
>
> Key: OOZIE-1847
> URL: https://issues.apache.org/jira/browse/OOZIE-1847
> Project: Oozie
>  Issue Type: Bug
>  Components: HA
>Reporter: Purshotam Shah
>Assignee: Purshotam Shah
> Attachments: OOZIE-1847-V1.patch
>
>




--
This message was sent by Atlassian JIRA
(v6.2#6252)

[jira] [Commented] (OOZIE-1847) HA - Oozie servers should shutdown (or go in safe mode) in case of ZK failure


[ 
https://issues.apache.org/jira/browse/OOZIE-1847?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14112331#comment-14112331
 ] 

Mona Chitnis commented on OOZIE-1847:
-

pretty straightforward patch and agree its needed. But in addition to printing 
in logs, should we bubble it up to action error message too? That way reason 
for a workflow failing can be pulled up from any of the client-facing APIs too 
- e.g. job-info, web-console, RESTful aPI etc

> HA - Oozie servers should shutdown (or go in safe mode) in case of ZK failure
> -
>
> Key: OOZIE-1847
> URL: https://issues.apache.org/jira/browse/OOZIE-1847
> Project: Oozie
>  Issue Type: Bug
>  Components: HA
>Reporter: Purshotam Shah
>Assignee: Purshotam Shah
> Attachments: OOZIE-1847-V1.patch
>
>




--
This message was sent by Atlassian JIRA
(v6.2#6252)

[jira] [Updated] (OOZIE-1976) Specifying coordinator input datasets in more logical ways

2014-08-20 Thread Mona Chitnis (JIRA)


 [ 
https://issues.apache.org/jira/browse/OOZIE-1976?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Mona Chitnis updated OOZIE-1976:


Attachment: OOZIE-1976-rough-design-2.pdf

New design spec uploaded (rought-design-2) with additions about
 * Wait-for in action
 * EL functions initial thoughts - implementation details will follow in code 
patch
 * HCatDependencyCache changes (for the in-memory push-based hcat dependencies)
 * Job info API (coord-action) changes for displaying Missing Dependency. It 
runs the risk of being verbose if optional dataset has lot of instances. Needs 
thought about how to possibly truncate there.
 

> Specifying coordinator input datasets in more logical ways
> --
>
> Key: OOZIE-1976
> URL: https://issues.apache.org/jira/browse/OOZIE-1976
> Project: Oozie
>  Issue Type: New Feature
>  Components: coordinator
>Affects Versions: trunk
>Reporter: Mona Chitnis
>    Assignee: Mona Chitnis
> Fix For: trunk
>
> Attachments: OOZIE-1976-rough-design-2.pdf, 
> OOZIE-1976-rough-design.pdf
>
>
> All dataset instances specified as input to coordinator, currently work on 
> AND logic i.e. ALL of them should be available for workflow to start. We 
> should enhance this to include more logical ways of specifying availability 
> criteria e.g.
>  * OR between instances
>  * minimum N out of K instances
>  * delta datasets (process data incrementally)
> Use-cases for this:
>  * Different datasets are BCP, and workflow can run with either, whichever 
> arrives earlier.
>  * Data is not guaranteed, and while $coord:latest allows skipping to 
> available ones, workflow will never trigger unless mentioned number of 
> instances are found.
>  * Workflow is like a ‘refining’ algorithm which should run after minimum 
> required datasets are ready, and should only process the delta for efficiency.
> This JIRA is to discuss the design and then the review the implementation for 
> some or all of the above features.



--
This message was sent by Atlassian JIRA
(v6.2#6252)

[jira] [Commented] (OOZIE-1976) Specifying coordinator input datasets in more logical ways

2014-08-20 Thread Mona Chitnis (JIRA)


[ 
https://issues.apache.org/jira/browse/OOZIE-1976?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14104618#comment-14104618
 ] 

Mona Chitnis commented on OOZIE-1976:
-

For Ryota's comment about priority, I think it complicates the missing 
dependencies field, now we require a structure to indicate something like 
{{P0=dep1,dep2#P1=dep3,dep4}} which in turn is nested under the AND/OR 
structure. So when dependencies are checked and found to exist, action will 
start only when all P0's are satisfied etc. I think this is essentially same as 
putting them in the  block instead of optional  block. For the N out 
of M case, it will start when _any_ instances >=n are available, using all M if 
all there, and not limit to N there.

Good pointer about EL functions, that one's going to be important and will 
probably need a few new ones.

> Specifying coordinator input datasets in more logical ways
> --
>
> Key: OOZIE-1976
> URL: https://issues.apache.org/jira/browse/OOZIE-1976
> Project: Oozie
>  Issue Type: New Feature
>  Components: coordinator
>    Affects Versions: trunk
>    Reporter: Mona Chitnis
>Assignee: Mona Chitnis
> Fix For: trunk
>
> Attachments: OOZIE-1976-rough-design.pdf
>
>
> All dataset instances specified as input to coordinator, currently work on 
> AND logic i.e. ALL of them should be available for workflow to start. We 
> should enhance this to include more logical ways of specifying availability 
> criteria e.g.
>  * OR between instances
>  * minimum N out of K instances
>  * delta datasets (process data incrementally)
> Use-cases for this:
>  * Different datasets are BCP, and workflow can run with either, whichever 
> arrives earlier.
>  * Data is not guaranteed, and while $coord:latest allows skipping to 
> available ones, workflow will never trigger unless mentioned number of 
> instances are found.
>  * Workflow is like a ‘refining’ algorithm which should run after minimum 
> required datasets are ready, and should only process the delta for efficiency.
> This JIRA is to discuss the design and then the review the implementation for 
> some or all of the above features.



--
This message was sent by Atlassian JIRA
(v6.2#6252)

[jira] [Commented] (OOZIE-1976) Specifying coordinator input datasets in more logical ways

2014-08-20 Thread Mona Chitnis (JIRA)


[ 
https://issues.apache.org/jira/browse/OOZIE-1976?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14104227#comment-14104227
 ] 

Mona Chitnis commented on OOZIE-1976:
-

Thanks Puru and Ryota. Will incorporate your comments and come up with new 
design specification. As for the 'explain', this can be done as part of 'info' 
command displaying missing dependency itself, rather than introducing another 
command

> Specifying coordinator input datasets in more logical ways
> --
>
> Key: OOZIE-1976
> URL: https://issues.apache.org/jira/browse/OOZIE-1976
> Project: Oozie
>  Issue Type: New Feature
>  Components: coordinator
>Affects Versions: trunk
>    Reporter: Mona Chitnis
>Assignee: Mona Chitnis
> Fix For: trunk
>
> Attachments: OOZIE-1976-rough-design.pdf
>
>
> All dataset instances specified as input to coordinator, currently work on 
> AND logic i.e. ALL of them should be available for workflow to start. We 
> should enhance this to include more logical ways of specifying availability 
> criteria e.g.
>  * OR between instances
>  * minimum N out of K instances
>  * delta datasets (process data incrementally)
> Use-cases for this:
>  * Different datasets are BCP, and workflow can run with either, whichever 
> arrives earlier.
>  * Data is not guaranteed, and while $coord:latest allows skipping to 
> available ones, workflow will never trigger unless mentioned number of 
> instances are found.
>  * Workflow is like a ‘refining’ algorithm which should run after minimum 
> required datasets are ready, and should only process the delta for efficiency.
> This JIRA is to discuss the design and then the review the implementation for 
> some or all of the above features.



--
This message was sent by Atlassian JIRA
(v6.2#6252)

[jira] [Updated] (OOZIE-1976) Specifying coordinator input datasets in more logical ways


 [ 
https://issues.apache.org/jira/browse/OOZIE-1976?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Mona Chitnis updated OOZIE-1976:


Attachment: OOZIE-1976-rough-design.pdf

Attaching rough design doc (pdf)

> Specifying coordinator input datasets in more logical ways
> --
>
> Key: OOZIE-1976
> URL: https://issues.apache.org/jira/browse/OOZIE-1976
> Project: Oozie
>  Issue Type: New Feature
>  Components: coordinator
>Affects Versions: trunk
>Reporter: Mona Chitnis
>    Assignee: Mona Chitnis
> Fix For: trunk
>
> Attachments: OOZIE-1976-rough-design.pdf
>
>
> All dataset instances specified as input to coordinator, currently work on 
> AND logic i.e. ALL of them should be available for workflow to start. We 
> should enhance this to include more logical ways of specifying availability 
> criteria e.g.
>  * OR between instances
>  * minimum N out of K instances
>  * delta datasets (process data incrementally)
> Use-cases for this:
>  * Different datasets are BCP, and workflow can run with either, whichever 
> arrives earlier.
>  * Data is not guaranteed, and while $coord:latest allows skipping to 
> available ones, workflow will never trigger unless mentioned number of 
> instances are found.
>  * Workflow is like a ‘refining’ algorithm which should run after minimum 
> required datasets are ready, and should only process the delta for efficiency.
> This JIRA is to discuss the design and then the review the implementation for 
> some or all of the above features.



--
This message was sent by Atlassian JIRA
(v6.2#6252)

[jira] [Updated] (OOZIE-1976) Specifying coordinator input datasets in more logical ways


 [ 
https://issues.apache.org/jira/browse/OOZIE-1976?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Mona Chitnis updated OOZIE-1976:


Description: 
All dataset instances specified as input to coordinator, currently work on AND 
logic i.e. ALL of them should be available for workflow to start. We should 
enhance this to include more logical ways of specifying availability criteria 
e.g.
 * OR between instances
 * minimum N out of K instances
 * delta datasets (process data incrementally)

Use-cases for this:
 * Different datasets are BCP, and workflow can run with either, whichever 
arrives earlier.
 * Data is not guaranteed, and while $coord:latest allows skipping to available 
ones, workflow will never trigger unless mentioned number of instances are 
found.
 * Workflow is like a ‘refining’ algorithm which should run after minimum 
required datasets are ready, and should only process the delta for efficiency.

This JIRA is to discuss the design and then the review the implementation for 
some or all of the above features.

  was:
All dataset instances specified as input to coordinator, currently work on AND 
logic i.e. ALL of them should be available for workflow to start. We should 
enhance this to include more logical ways of specifying availability criteria 
e.g.
 * OR between instances
 * minimum N out of K instances
 * delta datasets (process data incrementally)

Use-cases for this:
Different datasets are BCP, and workflow can run with either, whichever arrives 
earlier.
Data is not guaranteed, and while $coord:latest allows skipping to available 
ones, workflow will never trigger unless mentioned number of instances are 
found.
Workflow is like a ‘refining’ algorithm which should run after minimum required 
datasets are ready, and should only process the delta for efficiency.

This JIRA is to discuss the design and then the review the implementation for 
some or all of the above features.


> Specifying coordinator input datasets in more logical ways
> --
>
> Key: OOZIE-1976
> URL: https://issues.apache.org/jira/browse/OOZIE-1976
> Project: Oozie
>  Issue Type: New Feature
>  Components: coordinator
>Affects Versions: trunk
>Reporter: Mona Chitnis
>    Assignee: Mona Chitnis
> Fix For: trunk
>
>
> All dataset instances specified as input to coordinator, currently work on 
> AND logic i.e. ALL of them should be available for workflow to start. We 
> should enhance this to include more logical ways of specifying availability 
> criteria e.g.
>  * OR between instances
>  * minimum N out of K instances
>  * delta datasets (process data incrementally)
> Use-cases for this:
>  * Different datasets are BCP, and workflow can run with either, whichever 
> arrives earlier.
>  * Data is not guaranteed, and while $coord:latest allows skipping to 
> available ones, workflow will never trigger unless mentioned number of 
> instances are found.
>  * Workflow is like a ‘refining’ algorithm which should run after minimum 
> required datasets are ready, and should only process the delta for efficiency.
> This JIRA is to discuss the design and then the review the implementation for 
> some or all of the above features.



--
This message was sent by Atlassian JIRA
(v6.2#6252)

[jira] [Updated] (OOZIE-1976) Specifying coordinator input datasets in more logical ways


 [ 
https://issues.apache.org/jira/browse/OOZIE-1976?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Mona Chitnis updated OOZIE-1976:


Description: 
All dataset instances specified as input to coordinator, currently work on AND 
logic i.e. ALL of them should be available for workflow to start. We should 
enhance this to include more logical ways of specifying availability criteria 
e.g.
 * OR between instances
 * minimum N out of K instances
 * delta datasets (process data incrementally)

Use-cases for this:
Different datasets are BCP, and workflow can run with either, whichever arrives 
earlier.
Data is not guaranteed, and while $coord:latest allows skipping to available 
ones, workflow will never trigger unless mentioned number of instances are 
found.
Workflow is like a ‘refining’ algorithm which should run after minimum required 
datasets are ready, and should only process the delta for efficiency.

This JIRA is to discuss the design and then the review the implementation for 
some or all of the above features.

  was:
All dataset instances specified as input to coordinator, currently work on AND 
logic i.e. ALL of them should be available for workflow to start. We should 
enhance this to include more logical ways of specifying availability criteria 
e.g.
 * OR between instances
 * minimum N out of K instances
 * delta datasets (process data incrementally)

This JIRA is to discuss the design and then the review the implementation for 
some or all of the above features.


> Specifying coordinator input datasets in more logical ways
> --
>
> Key: OOZIE-1976
> URL: https://issues.apache.org/jira/browse/OOZIE-1976
> Project: Oozie
>  Issue Type: New Feature
>  Components: coordinator
>Affects Versions: trunk
>Reporter: Mona Chitnis
>    Assignee: Mona Chitnis
> Fix For: trunk
>
>
> All dataset instances specified as input to coordinator, currently work on 
> AND logic i.e. ALL of them should be available for workflow to start. We 
> should enhance this to include more logical ways of specifying availability 
> criteria e.g.
>  * OR between instances
>  * minimum N out of K instances
>  * delta datasets (process data incrementally)
> Use-cases for this:
> Different datasets are BCP, and workflow can run with either, whichever 
> arrives earlier.
> Data is not guaranteed, and while $coord:latest allows skipping to available 
> ones, workflow will never trigger unless mentioned number of instances are 
> found.
> Workflow is like a ‘refining’ algorithm which should run after minimum 
> required datasets are ready, and should only process the delta for efficiency.
> This JIRA is to discuss the design and then the review the implementation for 
> some or all of the above features.



--
This message was sent by Atlassian JIRA
(v6.2#6252)

[jira] [Created] (OOZIE-1976) Specifying coordinator input datasets in more logical ways

Mona Chitnis created OOZIE-1976:
---

 Summary: Specifying coordinator input datasets in more logical ways
 Key: OOZIE-1976
 URL: https://issues.apache.org/jira/browse/OOZIE-1976
 Project: Oozie
  Issue Type: New Feature
  Components: coordinator
Affects Versions: trunk
Reporter: Mona Chitnis
Assignee: Mona Chitnis
 Fix For: trunk


All dataset instances specified as input to coordinator, currently work on AND 
logic i.e. ALL of them should be available for workflow to start. We should 
enhance this to include more logical ways of specifying availability criteria 
e.g.
 * OR between instances
 * minimum N out of K instances
 * delta datasets (process data incrementally)

This JIRA is to discuss the design and then the review the implementation for 
some or all of the above features.



--
This message was sent by Atlassian JIRA
(v6.2#6252)

Re: Review Request 24487: OOZIE-1913 Devise a way to turn off SLA alerts for bundle/coordinator flexibly

2014-08-14 Thread Mona Chitnis


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/24487/
---

(Updated Aug. 14, 2014, 11:13 p.m.)


Review request for oozie.


Changes
---

updated patch to include unit tests, and fixes uncovered in the process


Bugs: OOZIE-1913
https://issues.apache.org/jira/browse/OOZIE-1913


Repository: oozie-git


Description
---

See Jira


Diffs (updated)
-

  client/src/main/java/org/apache/oozie/cli/OozieCLI.java 33935d3 
  client/src/main/java/org/apache/oozie/client/OozieClient.java b468186 
  
client/src/main/java/org/apache/oozie/client/event/jms/JMSHeaderConstants.java 
2f0a45c 
  client/src/main/java/org/apache/oozie/client/rest/RestConstants.java 5d3fc62 
  core/src/main/java/org/apache/oozie/CoordinatorActionBean.java 795bf63 
  core/src/main/java/org/apache/oozie/CoordinatorJobBean.java 8fd53f1 
  core/src/main/java/org/apache/oozie/command/SubmitTransitionXCommand.java 
5d3b6af 
  core/src/main/java/org/apache/oozie/command/bundle/BundleSubmitXCommand.java 
ffb2d08 
  
core/src/main/java/org/apache/oozie/command/coord/CoordMaterializeTransitionXCommand.java
 b4b2fef 
  core/src/main/java/org/apache/oozie/command/coord/CoordSubmitXCommand.java 
02b30ef 
  core/src/main/java/org/apache/oozie/coord/CoordUtils.java 26db068 
  
core/src/main/java/org/apache/oozie/executor/jpa/CoordActionQueryExecutor.java 
cd26e07 
  core/src/main/java/org/apache/oozie/executor/jpa/CoordJobQueryExecutor.java 
42a0968 
  core/src/main/java/org/apache/oozie/jms/JMSSLAEventListener.java 8296a6c 
  
core/src/main/java/org/apache/oozie/service/CoordMaterializeTriggerService.java 
3fbd092 
  core/src/main/java/org/apache/oozie/servlet/SLAServlet.java 8ca2e81 
  core/src/main/java/org/apache/oozie/servlet/V2SLAServlet.java 8620af5 
  core/src/main/java/org/apache/oozie/sla/SLACalcStatus.java 67d6237 
  core/src/main/java/org/apache/oozie/sla/SLACalculator.java 132d4df 
  core/src/main/java/org/apache/oozie/sla/SLACalculatorMemory.java 3801325 
  core/src/main/java/org/apache/oozie/sla/SLAOperations.java 0cad071 
  core/src/main/java/org/apache/oozie/sla/service/SLAService.java 2349329 
  core/src/main/java/org/apache/oozie/util/CoordActionsInDateRange.java fd21c45 
  core/src/main/resources/oozie-default.xml ebceaa7 
  core/src/test/java/org/apache/oozie/client/TestWorkflowClient.java e2e0f11 
  
core/src/test/java/org/apache/oozie/command/coord/TestCoordSubmitXCommand.java 
fedf4a8 
  core/src/test/java/org/apache/oozie/coord/TestCoordUtils.java a39efe3 
  core/src/test/java/org/apache/oozie/jms/TestJMSSLAEventListener.java fa26935 
  core/src/test/java/org/apache/oozie/servlet/TestV2SLAServlet.java 5a35fdb 
  core/src/test/java/org/apache/oozie/sla/TestSLACalculatorMemory.java 210c99e 

Diff: https://reviews.apache.org/r/24487/diff/


Testing (updated)
---

unit tests added, e-2-e test with CLI command done


Thanks,

Mona Chitnis

[jira] [Commented] (OOZIE-1913) Devise a way to turn off SLA alerts for bundle/coordinator flexibly

2014-08-13 Thread Mona Chitnis (JIRA)


[ 
https://issues.apache.org/jira/browse/OOZIE-1913?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14096118#comment-14096118
 ] 

Mona Chitnis commented on OOZIE-1913:
-

Want to mention another point:

this API also allows to disable for "ALL" sla instances for a coordinator or 
bundle. For bundle, that would mean all coordinators' all actions. 
SLARegistrationBean stores 'parentId' if the sla object pertains to 
coord-action/wf-action/bundle-action. To avoid heavy dB query in case of the 
suspend ALL for bundle(s) case, I want to change this 'parentId' to point to 
bundle jobId directly, if coordinator is part of a bundle. If not, it will be 
coord job id as it is now.

The impact this has is in JMSSLAEventListener, where topicName is set to this 
parentId. So topicName will get set to top-level bundle-id, and user will have 
to change topic name being listened to. Please give feedback if this is a 
reasonable approach. I will make sure appropriate JMS selector options are 
available, if user gives this bundle id topicName, but still wants to limit per 
coordinator job id. 

> Devise a way to turn off SLA alerts for bundle/coordinator flexibly
> ---
>
> Key: OOZIE-1913
> URL: https://issues.apache.org/jira/browse/OOZIE-1913
> Project: Oozie
>  Issue Type: Improvement
>Affects Versions: trunk
>Reporter: Mona Chitnis
>Assignee: Mona Chitnis
> Fix For: trunk
>
>
> From user:
> Need to turn off the SLA miss alerts in jobs when the bundle is suspended for
> grid upgrades and similar work so that when it's resumed we aren't flooded 
> with a bunch of alerts.



--
This message was sent by Atlassian JIRA
(v6.2#6252)

Re: Review Request 24487: OOZIE-1913 Devise a way to turn off SLA alerts for bundle/coordinator flexibly

2014-08-11 Thread Mona Chitnis


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/24487/#review50269
---



core/src/main/resources/oozie-default.xml
<https://reviews.apache.org/r/24487/#comment87956>

this change is part of OOZIE-1932 and will remove it in next patch version


- Mona Chitnis


On Aug. 8, 2014, 2:20 a.m., Mona Chitnis wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/24487/
> ---
> 
> (Updated Aug. 8, 2014, 2:20 a.m.)
> 
> 
> Review request for oozie.
> 
> 
> Bugs: OOZIE-1913
> https://issues.apache.org/jira/browse/OOZIE-1913
> 
> 
> Repository: oozie-git
> 
> 
> Description
> ---
> 
> See Jira
> 
> 
> Diffs
> -
> 
>   client/src/main/java/org/apache/oozie/cli/OozieCLI.java 33935d3 
>   client/src/main/java/org/apache/oozie/client/OozieClient.java b468186 
>   client/src/main/java/org/apache/oozie/client/rest/RestConstants.java 
> 5d3fc62 
>   core/src/main/java/org/apache/oozie/CoordinatorActionBean.java 795bf63 
>   core/src/main/java/org/apache/oozie/CoordinatorJobBean.java 8fd53f1 
>   core/src/main/java/org/apache/oozie/command/SubmitTransitionXCommand.java 
> 5d3b6af 
>   
> core/src/main/java/org/apache/oozie/command/bundle/BundleSubmitXCommand.java 
> ffb2d08 
>   
> core/src/main/java/org/apache/oozie/command/coord/CoordMaterializeTransitionXCommand.java
>  b4b2fef 
>   core/src/main/java/org/apache/oozie/command/coord/CoordSubmitXCommand.java 
> 02b30ef 
>   core/src/main/java/org/apache/oozie/coord/CoordUtils.java 26db068 
>   
> core/src/main/java/org/apache/oozie/executor/jpa/CoordActionQueryExecutor.java
>  cd26e07 
>   core/src/main/java/org/apache/oozie/executor/jpa/CoordJobQueryExecutor.java 
> 42a0968 
>   core/src/main/java/org/apache/oozie/servlet/SLAServlet.java 8ca2e81 
>   core/src/main/java/org/apache/oozie/servlet/V2SLAServlet.java 8620af5 
>   core/src/main/java/org/apache/oozie/sla/SLACalcStatus.java 67d6237 
>   core/src/main/java/org/apache/oozie/sla/SLACalculator.java 132d4df 
>   core/src/main/java/org/apache/oozie/sla/SLACalculatorMemory.java 3801325 
>   core/src/main/java/org/apache/oozie/sla/SLAOperations.java 0cad071 
>   core/src/main/java/org/apache/oozie/sla/service/SLAService.java 2349329 
>   core/src/main/java/org/apache/oozie/util/CoordActionsInDateRange.java 
> fd21c45 
>   core/src/main/resources/oozie-default.xml ebceaa7 
>   core/src/test/java/org/apache/oozie/client/TestWorkflowClient.java e2e0f11 
> 
> Diff: https://reviews.apache.org/r/24487/diff/
> 
> 
> Testing
> ---
> 
> ongoing
> 
> 
> Thanks,
> 
> Mona Chitnis
> 
>

Re: Review Request 24487: OOZIE-1913 Devise a way to turn off SLA alerts for bundle/coordinator flexibly

2014-08-08 Thread Mona Chitnis


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/24487/#review50101
---



client/src/main/java/org/apache/oozie/cli/OozieCLI.java
<https://reviews.apache.org/r/24487/#comment87652>

this has been removed



client/src/main/java/org/apache/oozie/cli/OozieCLI.java
<https://reviews.apache.org/r/24487/#comment87653>

this has been removed. The action ids/dates range is read as argument for 
option -suspendalerts itself


- Mona Chitnis


On Aug. 8, 2014, 2:20 a.m., Mona Chitnis wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/24487/
> ---
> 
> (Updated Aug. 8, 2014, 2:20 a.m.)
> 
> 
> Review request for oozie.
> 
> 
> Bugs: OOZIE-1913
> https://issues.apache.org/jira/browse/OOZIE-1913
> 
> 
> Repository: oozie-git
> 
> 
> Description
> ---
> 
> See Jira
> 
> 
> Diffs
> -
> 
>   client/src/main/java/org/apache/oozie/cli/OozieCLI.java 33935d3 
>   client/src/main/java/org/apache/oozie/client/OozieClient.java b468186 
>   client/src/main/java/org/apache/oozie/client/rest/RestConstants.java 
> 5d3fc62 
>   core/src/main/java/org/apache/oozie/CoordinatorActionBean.java 795bf63 
>   core/src/main/java/org/apache/oozie/CoordinatorJobBean.java 8fd53f1 
>   core/src/main/java/org/apache/oozie/command/SubmitTransitionXCommand.java 
> 5d3b6af 
>   
> core/src/main/java/org/apache/oozie/command/bundle/BundleSubmitXCommand.java 
> ffb2d08 
>   
> core/src/main/java/org/apache/oozie/command/coord/CoordMaterializeTransitionXCommand.java
>  b4b2fef 
>   core/src/main/java/org/apache/oozie/command/coord/CoordSubmitXCommand.java 
> 02b30ef 
>   core/src/main/java/org/apache/oozie/coord/CoordUtils.java 26db068 
>   
> core/src/main/java/org/apache/oozie/executor/jpa/CoordActionQueryExecutor.java
>  cd26e07 
>   core/src/main/java/org/apache/oozie/executor/jpa/CoordJobQueryExecutor.java 
> 42a0968 
>   core/src/main/java/org/apache/oozie/servlet/SLAServlet.java 8ca2e81 
>   core/src/main/java/org/apache/oozie/servlet/V2SLAServlet.java 8620af5 
>   core/src/main/java/org/apache/oozie/sla/SLACalcStatus.java 67d6237 
>   core/src/main/java/org/apache/oozie/sla/SLACalculator.java 132d4df 
>   core/src/main/java/org/apache/oozie/sla/SLACalculatorMemory.java 3801325 
>   core/src/main/java/org/apache/oozie/sla/SLAOperations.java 0cad071 
>   core/src/main/java/org/apache/oozie/sla/service/SLAService.java 2349329 
>   core/src/main/java/org/apache/oozie/util/CoordActionsInDateRange.java 
> fd21c45 
>   core/src/main/resources/oozie-default.xml ebceaa7 
>   core/src/test/java/org/apache/oozie/client/TestWorkflowClient.java e2e0f11 
> 
> Diff: https://reviews.apache.org/r/24487/diff/
> 
> 
> Testing
> ---
> 
> ongoing
> 
> 
> Thanks,
> 
> Mona Chitnis
> 
>

Re: Problem with Oozie DeadLock

2014-08-08 Thread Mona Chitnis

Hi Fabiano,

You should definitely be able to execute multiple Oozie jobs simultaneously.
The issue comes up when you have a small hadoop cluster setup, and thus very
small number of queue slots for submitting jobs to the ResourceManager.

Can you look into adding an additional queue by configuring your Hadoop cluster
through capacity-scheduler.xml? Then you can use the approach mentioned in
OOZIE-1673, to specify in your workflow's properties

oozie.launcher.mapreduce.job.queuename=queue1
mapreduce.job.queuename=queue2

and avoid deadlock situation.

You can also avoid deadlocks by tuning the memory requirements of your oozie
launcher and child jobs, to request for lower memory container slots, and
increase the number of jobs you can submit and execute that
way.http://docs.hortonworks.com/HDPDocuments/HDP2/HDP-2.0.9.1/bk_installing_manually_book/content/rpm-chap1-11.html

Mona Chitnis
Yahoo!

On Friday, August 8, 2014 6:58 AM, Gaetano Fabiano
wrote:

Hi All,
this is my first post on this mailing list I hope to stay here for long
time. I saw the project and I love it.
But recently I'm going crazy about issues.
Our problem is about Oozie Deadlock, we would like to execute more than one
Oozie job at the same time but when we try to execute its the entire
environment becames locked all.
We read a lot of post about this issue but without solution.

How is possible to set two different queue? We read a lot of post where
people suggest to use different queue for different execution
Where we could set these queue setting?
Is this the correct way to have different execution at the same time?

Our bewilderment is reading this issue
https://issues.apache.org/jira/browse/OOZIE-1673 and if is as described
the big doubt is about the usefulness of the enteire Oozie framework.
I hope someone can help us to resolve it and clear our mind about.
Any suggestion is welcome.

Regards
Gaetano

Dott. Gaetano Fabiano

via Timpone n° 79
87055 San Giovanni in Fiore (Cs) ITALY
mobile: +39 328 9469919
phone: +39 0984 991980
email: fabiano.gaet...@gmail.com
skype: deepyoudeep
skype:: gaetano.fab
msn: deep...@hotmail.it
twitter: @gaetanofabiano
gtalk/hangoout/Google+ fg.pa...@gmail.com

[jira] [Commented] (OOZIE-1913) Devise a way to turn off SLA alerts for bundle/coordinator flexibly

2014-08-08 Thread Mona Chitnis (JIRA)


[ 
https://issues.apache.org/jira/browse/OOZIE-1913?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14090999#comment-14090999
 ] 

Mona Chitnis commented on OOZIE-1913:
-

okay let me remove "-id" requirement. Regarding treating as job operation, I 
think it becomes ambiguous what type of alerts it means, so better to be clear 
with 'sla' command. Also, it removes the need to add additional param 'actions'.

But can rework this if there's a consensus about what api usage is more 
intuitive. Asking feedback from users too

> Devise a way to turn off SLA alerts for bundle/coordinator flexibly
> ---
>
> Key: OOZIE-1913
> URL: https://issues.apache.org/jira/browse/OOZIE-1913
> Project: Oozie
>  Issue Type: Improvement
>Affects Versions: trunk
>Reporter: Mona Chitnis
>Assignee: Mona Chitnis
> Fix For: trunk
>
>
> From user:
> Need to turn off the SLA miss alerts in jobs when the bundle is suspended for
> grid upgrades and similar work so that when it's resumed we aren't flooded 
> with a bunch of alerts.



--
This message was sent by Atlassian JIRA
(v6.2#6252)

[jira] [Updated] (OOZIE-1913) Devise a way to turn off SLA alerts for bundle/coordinator flexibly

2014-08-07 Thread Mona Chitnis (JIRA)


 [ 
https://issues.apache.org/jira/browse/OOZIE-1913?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Mona Chitnis updated OOZIE-1913:


Summary: Devise a way to turn off SLA alerts for bundle/coordinator 
flexibly  (was: Devise a way to turn off SLA alerts when bundle/coordinator 
suspended)

> Devise a way to turn off SLA alerts for bundle/coordinator flexibly
> ---
>
> Key: OOZIE-1913
> URL: https://issues.apache.org/jira/browse/OOZIE-1913
> Project: Oozie
>  Issue Type: Improvement
>Affects Versions: trunk
>    Reporter: Mona Chitnis
>Assignee: Mona Chitnis
> Fix For: trunk
>
>
> From user:
> Need to turn off the SLA miss alerts in jobs when the bundle is suspended for
> grid upgrades and similar work so that when it's resumed we aren't flooded 
> with a bunch of alerts.



--
This message was sent by Atlassian JIRA
(v6.2#6252)

Review Request 24487: OOZIE-1913 Devise a way to turn off SLA alerts for bundle/coordinator flexibly

2014-08-07 Thread Mona Chitnis


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/24487/
---

Review request for oozie.


Bugs: OOZIE-1913
https://issues.apache.org/jira/browse/OOZIE-1913


Repository: oozie-git


Description
---

See Jira


Diffs
-

  client/src/main/java/org/apache/oozie/cli/OozieCLI.java 33935d3 
  client/src/main/java/org/apache/oozie/client/OozieClient.java b468186 
  client/src/main/java/org/apache/oozie/client/rest/RestConstants.java 5d3fc62 
  core/src/main/java/org/apache/oozie/CoordinatorActionBean.java 795bf63 
  core/src/main/java/org/apache/oozie/CoordinatorJobBean.java 8fd53f1 
  core/src/main/java/org/apache/oozie/command/SubmitTransitionXCommand.java 
5d3b6af 
  core/src/main/java/org/apache/oozie/command/bundle/BundleSubmitXCommand.java 
ffb2d08 
  
core/src/main/java/org/apache/oozie/command/coord/CoordMaterializeTransitionXCommand.java
 b4b2fef 
  core/src/main/java/org/apache/oozie/command/coord/CoordSubmitXCommand.java 
02b30ef 
  core/src/main/java/org/apache/oozie/coord/CoordUtils.java 26db068 
  
core/src/main/java/org/apache/oozie/executor/jpa/CoordActionQueryExecutor.java 
cd26e07 
  core/src/main/java/org/apache/oozie/executor/jpa/CoordJobQueryExecutor.java 
42a0968 
  core/src/main/java/org/apache/oozie/servlet/SLAServlet.java 8ca2e81 
  core/src/main/java/org/apache/oozie/servlet/V2SLAServlet.java 8620af5 
  core/src/main/java/org/apache/oozie/sla/SLACalcStatus.java 67d6237 
  core/src/main/java/org/apache/oozie/sla/SLACalculator.java 132d4df 
  core/src/main/java/org/apache/oozie/sla/SLACalculatorMemory.java 3801325 
  core/src/main/java/org/apache/oozie/sla/SLAOperations.java 0cad071 
  core/src/main/java/org/apache/oozie/sla/service/SLAService.java 2349329 
  core/src/main/java/org/apache/oozie/util/CoordActionsInDateRange.java fd21c45 
  core/src/main/resources/oozie-default.xml ebceaa7 
  core/src/test/java/org/apache/oozie/client/TestWorkflowClient.java e2e0f11 

Diff: https://reviews.apache.org/r/24487/diff/


Testing
---

ongoing


Thanks,

Mona Chitnis

Re: 4.1 release

2014-08-07 Thread Mona Chitnis

OOZIE-1932 I will fix in a couple of days.

Thanks,

 
Mona Chitnis



On Thursday, August 7, 2014 10:25 AM, bowen zhang 
 wrote:
 


Hi guys,
The following link shows all the unresolved issues that will go into 4.1 
release. If anyone has another ticket that needs to go into 4.1, please make 
the fix version "4.1.0".
Issue Navigator - ASF JIRA
Thanks,
Bowen

  
             
Issue Navigator - ASF JIRA
Linked Applications Loading…… Dashboards Projects Issues Agile Help Online Help 
 
View on issues.apache.org Preview by Yahoo

Re: Review Request 24187: OOZIE-1958 address duplication of env variables in oozie.launcher.yarn.app.mapreduce.am.env when running with uber mode

2014-08-05 Thread Mona Chitnis


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/24187/#review49628
---


+1 pending minor comment about naming and checking via e-2-e test


core/src/main/java/org/apache/oozie/action/hadoop/JavaActionExecutor.java
<https://reviews.apache.org/r/24187/#comment86870>

keep naming consistent i.e.
launcherEnvMap and launcherEnvMapStr



core/src/main/java/org/apache/oozie/action/hadoop/JavaActionExecutor.java
<https://reviews.apache.org/r/24187/#comment86865>

false formatting change.. but its ok to include



core/src/main/java/org/apache/oozie/action/hadoop/JavaActionExecutor.java
<https://reviews.apache.org/r/24187/#comment86866>

same as above


- Mona Chitnis


On Aug. 1, 2014, 5:56 p.m., Ryota Egashira wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/24187/
> ---
> 
> (Updated Aug. 1, 2014, 5:56 p.m.)
> 
> 
> Review request for oozie.
> 
> 
> Bugs: OOZIE-1958
> https://issues.apache.org/jira/browse/OOZIE-1958
> 
> 
> Repository: oozie-git
> 
> 
> Description
> ---
> 
> https://issues.apache.org/jira/browse/OOZIE-1958?filter=-1
> 
> 
> Diffs
> -
> 
>   core/src/main/java/org/apache/oozie/action/hadoop/JavaActionExecutor.java 
> 94b55cf 
>   
> core/src/test/java/org/apache/oozie/action/hadoop/TestJavaActionExecutor.java 
> 72a137c 
> 
> Diff: https://reviews.apache.org/r/24187/diff/
> 
> 
> Testing
> ---
> 
> 
> Thanks,
> 
> Ryota Egashira
> 
>

Re: Thanks for fixing UTC.

2014-08-02 Thread Mona Chitnis

No problem. OOZIE-1811 did fix root cause of non-flaky tests failing randomly 
but there's one which is timing sensitive and needs to be fixed
org.apache.oozie.service.TestCallableQueueService.testConcurrencyReachedAndChooseNextEligible


and OOZIE-1952 will fix TestPurgeXCommand which uses old StoreService code.

And then we're done! :)

 
Mona Chitnis
Software Engineer, Hadoop Team
Yahoo!


On Friday, August 1, 2014 4:25 PM, Purshotam Shah 
 wrote:
 


Thanks Mona for fixing testcases. It feel so good to see no UTC failure.


On 8/1/14, 4:13 PM, "Hadoop QA (JIRA)"  wrote:

>
>    [ 
>https://issues.apache.org/jira/browse/OOZIE-1939?page=com.atlassian.jira.p
>lugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14083151#com
>ment-14083151 ] 
>
>Hadoop QA commented on OOZIE-1939:
>--
>
>Testing JIRA OOZIE-1939
>
>Cleaning local git workspace
>
>
>
>{color:green}+1
 PATCH_APPLIES{color}
>{color:green}+1 CLEAN{color}
>{color:green}+1 RAW_PATCH_ANALYSIS{color}
>.    {color:green}+1{color} the patch does not introduce any @author tags
>.    {color:green}+1{color} the patch does not introduce any tabs
>.    {color:green}+1{color} the patch does not introduce any trailing
>spaces
>.    {color:green}+1{color} the patch does not introduce any line longer
>than 132
>.    {color:green}+1{color} the patch does adds/modifies 1 testcase(s)
>{color:green}+1 RAT{color}
>. 
   {color:green}+1{color} the patch does not seem to introduce new RAT
>warnings
>{color:green}+1 JAVADOC{color}
>.    {color:green}+1{color} the patch does not seem to introduce new
>Javadoc warnings
>{color:green}+1 COMPILE{color}
>.    {color:green}+1{color} HEAD
 compiles
>.    {color:green}+1{color} patch compiles
>.    {color:green}+1{color} the patch does not seem to introduce new
>javac warnings
>{color:green}+1 BACKWARDS_COMPATIBILITY{color}
>.    {color:green}+1{color} the patch does not change any JPA
>Entity/Colum/Basic/Lob/Transient annotations
>.    {color:green}+1{color} the patch does not modify JPA files
>{color:green}+1 TESTS{color}
>.    Tests run: 1506
>{color:green}+1 DISTRO{color}
>.    {color:green}+1{color} distro tarball builds with
 the patch
>
>
>{color:green}*+1 Overall result, good!, no -1s*{color}
>
>
>The full output of the test-patch run is available at
>
>.  https://builds.apache.org/job/oozie-trunk-precommit-build/1377/
>
>> Incorrect job information is set while logging
>> --
>>
>>                 Key: OOZIE-1939
>>                 URL: https://issues.apache.org/jira/browse/OOZIE-1939
>>             Project: Oozie
>>          Issue Type: Bug
>>            Reporter: Purshotam Shah
>>            Assignee: Azrael
>>         Attachments: OOZIE-1939.1.patch, OOZIE-1939.2.patch
>>
>>
>> {code}
>> 2014-07-16 17:28:06,422 DEBUG
 CoordChangeXCommand:545
>>[http-0.0.0.0-4443-5] - USER[hadoopqa] GROUP[users] TOKEN[]
>>APP[coordB236] JOB[0011514-140716042555-oozie-oozi-C] ACTION[-] Acquired
>>lock for [0011385-140716042555-oozie-oozi-C] in [coord_change]
>> 2014-07-16 17:28:06,422 TRACE CoordChangeXCommand:548
>>[http-0.0.0.0-4443-5] - USER[hadoopqa] GROUP[users] TOKEN[]
>>APP[coordB236] JOB[0011514-140716042555-oozie-oozi-C] ACTION[-] Load
>>state for [0011385-140716042555-oozie-oozi-C]
>> {code}
>> {code}
>>     protected void loadState() throws CommandException {
>>         jpaService = Services.get().get(JPAService.class);
>>         if (jpaService == null) {
>>             LOG.error(ErrorCode.E0610);
>>         }
>>        
 try {
>>             coordJob =
>>CoordJobQueryExecutor.getInstance().get(CoordJobQuery.GET_COORD_JOB_MATER
>>IALIZE, jobId);
>>             prevStatus = coordJob.getStatus();
>>         }
>>         catch (JPAExecutorException jex) {
>>             throw new CommandException(jex);
>>         }
>>         // calculate start materialize and end materialize time
>>        
 calcMatdTime();
>>         LogUtils.setLogInfo(coordJob, logInfo);
>>     }
>> {code}
>> Most of the commands set jobinfo after loadstate, because of that few
>>log statements ( like acquiring lock, load state) logs with
 previous
>>jobinfo. 
>
>
>
>--
>This message was sent by Atlassian JIRA
>(v6.2#6252)

[jira] [Commented] (OOZIE-1932) Services should load CallableQueueService after MemoryLocksService


[ 
https://issues.apache.org/jira/browse/OOZIE-1932?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14082590#comment-14082590
 ] 

Mona Chitnis commented on OOZIE-1932:
-

okay thanks. will revise the order 

> Services should load CallableQueueService after MemoryLocksService
> --
>
> Key: OOZIE-1932
> URL: https://issues.apache.org/jira/browse/OOZIE-1932
> Project: Oozie
>  Issue Type: Bug
>Affects Versions: trunk
>    Reporter: Mona Chitnis
>Assignee: Mona Chitnis
> Fix For: 4.1.0
>
> Attachments: OOZIE-1932-2.patch, OOZIE-1932-addendum.patch, 
> OOZIE-1932.patch
>
>
> This is not a problem during startup but is during shutdown, as services are 
> destroyed in reverse order of initialization. Hence, when MemoryLocksService 
> destroy sets it to null, and commands are still executing due to 
> CallableQueueService still active, they all encounter NPEs during locking. 
> This is a simple fix in oozie-default.xml to set MemoryLocksService before in 
> the order of services loading.



--
This message was sent by Atlassian JIRA
(v6.2#6252)

[jira] [Updated] (OOZIE-1811) Current test failures in trunk


 [ 
https://issues.apache.org/jira/browse/OOZIE-1811?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Mona Chitnis updated OOZIE-1811:


Attachment: (was: OOZIE-1811-3.patch)

> Current test failures in trunk
> --
>
> Key: OOZIE-1811
> URL: https://issues.apache.org/jira/browse/OOZIE-1811
> Project: Oozie
>  Issue Type: Bug
>Affects Versions: trunk
>Reporter: Robert Kanter
>Assignee: Mona Chitnis
>Priority: Critical
> Attachments: OOZIE-1811-1.patch, OOZIE-1811-2.patch, 
> OOZIE-1811-3.patch
>
>
> There's a bunch of test failures currently in trunk; I'm not sure what 
> commit(s) is the cause, but I think it was somewhat recent.
> e.g. https://builds.apache.org/job/oozie-trunk-precommit-build/1199/
> Reproducible by running these tests, instead of having to run them all, which 
> takes a lot longer :)
> {noformat}
> mvn clean test 
> -Dtest=TestSubWorkflowActionExecutor,TestBunldeChangeXCommand,TestCoordUpdateXCommand,TestCoordJobQueryExecutor,TestStatusTransitService,TestSLAEventGeneration
> {noformat}
> {noformat}
> Results :
> Failed tests:   
> testCoordinatorActionCommandsSubmitAndStart(org.apache.oozie.sla.TestSLAEventGeneration):
>  expected:<...11921-oozie-rkan-C@1[]> but was:<...11921-oozie-rkan-C@1[2]>
>   
> testCoordStatusTransitServiceDoneWithError(org.apache.oozie.service.TestStatusTransitService):
>  expected: but was:
>   
> testBundleStatusTransitRunningFromKilled(org.apache.oozie.service.TestStatusTransitService):
>  expected: but was:
> Tests in error: 
>   testGetList(org.apache.oozie.executor.jpa.TestCoordJobQueryExecutor)
>   testInsert(org.apache.oozie.executor.jpa.TestCoordJobQueryExecutor)
> Tests run: 62, Failures: 3, Errors: 2, Skipped: 0
> {noformat}



--
This message was sent by Atlassian JIRA
(v6.2#6252)

[jira] [Updated] (OOZIE-1811) Current test failures in trunk


 [ 
https://issues.apache.org/jira/browse/OOZIE-1811?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Mona Chitnis updated OOZIE-1811:


Attachment: OOZIE-1811-3.patch

good catch! uploaded new patch

> Current test failures in trunk
> --
>
> Key: OOZIE-1811
> URL: https://issues.apache.org/jira/browse/OOZIE-1811
> Project: Oozie
>  Issue Type: Bug
>Affects Versions: trunk
>Reporter: Robert Kanter
>Assignee: Mona Chitnis
>Priority: Critical
> Attachments: OOZIE-1811-1.patch, OOZIE-1811-2.patch, 
> OOZIE-1811-3.patch
>
>
> There's a bunch of test failures currently in trunk; I'm not sure what 
> commit(s) is the cause, but I think it was somewhat recent.
> e.g. https://builds.apache.org/job/oozie-trunk-precommit-build/1199/
> Reproducible by running these tests, instead of having to run them all, which 
> takes a lot longer :)
> {noformat}
> mvn clean test 
> -Dtest=TestSubWorkflowActionExecutor,TestBunldeChangeXCommand,TestCoordUpdateXCommand,TestCoordJobQueryExecutor,TestStatusTransitService,TestSLAEventGeneration
> {noformat}
> {noformat}
> Results :
> Failed tests:   
> testCoordinatorActionCommandsSubmitAndStart(org.apache.oozie.sla.TestSLAEventGeneration):
>  expected:<...11921-oozie-rkan-C@1[]> but was:<...11921-oozie-rkan-C@1[2]>
>   
> testCoordStatusTransitServiceDoneWithError(org.apache.oozie.service.TestStatusTransitService):
>  expected: but was:
>   
> testBundleStatusTransitRunningFromKilled(org.apache.oozie.service.TestStatusTransitService):
>  expected: but was:
> Tests in error: 
>   testGetList(org.apache.oozie.executor.jpa.TestCoordJobQueryExecutor)
>   testInsert(org.apache.oozie.executor.jpa.TestCoordJobQueryExecutor)
> Tests run: 62, Failures: 3, Errors: 2, Skipped: 0
> {noformat}



--
This message was sent by Atlassian JIRA
(v6.2#6252)

[jira] [Updated] (OOZIE-1811) Current test failures in trunk


 [ 
https://issues.apache.org/jira/browse/OOZIE-1811?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Mona Chitnis updated OOZIE-1811:


Attachment: OOZIE-1811-3.patch

> Current test failures in trunk
> --
>
> Key: OOZIE-1811
> URL: https://issues.apache.org/jira/browse/OOZIE-1811
> Project: Oozie
>  Issue Type: Bug
>Affects Versions: trunk
>Reporter: Robert Kanter
>Assignee: Mona Chitnis
>Priority: Critical
> Attachments: OOZIE-1811-1.patch, OOZIE-1811-2.patch, 
> OOZIE-1811-3.patch
>
>
> There's a bunch of test failures currently in trunk; I'm not sure what 
> commit(s) is the cause, but I think it was somewhat recent.
> e.g. https://builds.apache.org/job/oozie-trunk-precommit-build/1199/
> Reproducible by running these tests, instead of having to run them all, which 
> takes a lot longer :)
> {noformat}
> mvn clean test 
> -Dtest=TestSubWorkflowActionExecutor,TestBunldeChangeXCommand,TestCoordUpdateXCommand,TestCoordJobQueryExecutor,TestStatusTransitService,TestSLAEventGeneration
> {noformat}
> {noformat}
> Results :
> Failed tests:   
> testCoordinatorActionCommandsSubmitAndStart(org.apache.oozie.sla.TestSLAEventGeneration):
>  expected:<...11921-oozie-rkan-C@1[]> but was:<...11921-oozie-rkan-C@1[2]>
>   
> testCoordStatusTransitServiceDoneWithError(org.apache.oozie.service.TestStatusTransitService):
>  expected: but was:
>   
> testBundleStatusTransitRunningFromKilled(org.apache.oozie.service.TestStatusTransitService):
>  expected: but was:
> Tests in error: 
>   testGetList(org.apache.oozie.executor.jpa.TestCoordJobQueryExecutor)
>   testInsert(org.apache.oozie.executor.jpa.TestCoordJobQueryExecutor)
> Tests run: 62, Failures: 3, Errors: 2, Skipped: 0
> {noformat}



--
This message was sent by Atlassian JIRA
(v6.2#6252)

[jira] [Updated] (OOZIE-1811) Current test failures in trunk


 [ 
https://issues.apache.org/jira/browse/OOZIE-1811?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Mona Chitnis updated OOZIE-1811:


Attachment: (was: OOZIE-1811-3.patch)

> Current test failures in trunk
> --
>
> Key: OOZIE-1811
> URL: https://issues.apache.org/jira/browse/OOZIE-1811
> Project: Oozie
>  Issue Type: Bug
>Affects Versions: trunk
>Reporter: Robert Kanter
>Assignee: Mona Chitnis
>Priority: Critical
> Attachments: OOZIE-1811-1.patch, OOZIE-1811-2.patch, 
> OOZIE-1811-3.patch
>
>
> There's a bunch of test failures currently in trunk; I'm not sure what 
> commit(s) is the cause, but I think it was somewhat recent.
> e.g. https://builds.apache.org/job/oozie-trunk-precommit-build/1199/
> Reproducible by running these tests, instead of having to run them all, which 
> takes a lot longer :)
> {noformat}
> mvn clean test 
> -Dtest=TestSubWorkflowActionExecutor,TestBunldeChangeXCommand,TestCoordUpdateXCommand,TestCoordJobQueryExecutor,TestStatusTransitService,TestSLAEventGeneration
> {noformat}
> {noformat}
> Results :
> Failed tests:   
> testCoordinatorActionCommandsSubmitAndStart(org.apache.oozie.sla.TestSLAEventGeneration):
>  expected:<...11921-oozie-rkan-C@1[]> but was:<...11921-oozie-rkan-C@1[2]>
>   
> testCoordStatusTransitServiceDoneWithError(org.apache.oozie.service.TestStatusTransitService):
>  expected: but was:
>   
> testBundleStatusTransitRunningFromKilled(org.apache.oozie.service.TestStatusTransitService):
>  expected: but was:
> Tests in error: 
>   testGetList(org.apache.oozie.executor.jpa.TestCoordJobQueryExecutor)
>   testInsert(org.apache.oozie.executor.jpa.TestCoordJobQueryExecutor)
> Tests run: 62, Failures: 3, Errors: 2, Skipped: 0
> {noformat}



--
This message was sent by Atlassian JIRA
(v6.2#6252)

[jira] [Updated] (OOZIE-1811) Current test failures in trunk


 [ 
https://issues.apache.org/jira/browse/OOZIE-1811?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Mona Chitnis updated OOZIE-1811:


Attachment: OOZIE-1811-3.patch

addressed review comments and fixed couple of classes missed in earlier patch - 
BatchQueryExecutor, SLA*QueryExecutors to be consistent

> Current test failures in trunk
> --
>
> Key: OOZIE-1811
> URL: https://issues.apache.org/jira/browse/OOZIE-1811
> Project: Oozie
>  Issue Type: Bug
>Affects Versions: trunk
>Reporter: Robert Kanter
>Assignee: Mona Chitnis
>Priority: Critical
> Attachments: OOZIE-1811-1.patch, OOZIE-1811-2.patch, 
> OOZIE-1811-3.patch
>
>
> There's a bunch of test failures currently in trunk; I'm not sure what 
> commit(s) is the cause, but I think it was somewhat recent.
> e.g. https://builds.apache.org/job/oozie-trunk-precommit-build/1199/
> Reproducible by running these tests, instead of having to run them all, which 
> takes a lot longer :)
> {noformat}
> mvn clean test 
> -Dtest=TestSubWorkflowActionExecutor,TestBunldeChangeXCommand,TestCoordUpdateXCommand,TestCoordJobQueryExecutor,TestStatusTransitService,TestSLAEventGeneration
> {noformat}
> {noformat}
> Results :
> Failed tests:   
> testCoordinatorActionCommandsSubmitAndStart(org.apache.oozie.sla.TestSLAEventGeneration):
>  expected:<...11921-oozie-rkan-C@1[]> but was:<...11921-oozie-rkan-C@1[2]>
>   
> testCoordStatusTransitServiceDoneWithError(org.apache.oozie.service.TestStatusTransitService):
>  expected: but was:
>   
> testBundleStatusTransitRunningFromKilled(org.apache.oozie.service.TestStatusTransitService):
>  expected: but was:
> Tests in error: 
>   testGetList(org.apache.oozie.executor.jpa.TestCoordJobQueryExecutor)
>   testInsert(org.apache.oozie.executor.jpa.TestCoordJobQueryExecutor)
> Tests run: 62, Failures: 3, Errors: 2, Skipped: 0
> {noformat}



--
This message was sent by Atlassian JIRA
(v6.2#6252)

[jira] [Commented] (OOZIE-1939) Incorrect job information is set while logging


[ 
https://issues.apache.org/jira/browse/OOZIE-1939?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14082317#comment-14082317
 ] 

Mona Chitnis commented on OOZIE-1939:
-

Yet it will work with threadlocal params too. Fix was done to minimize overall 
change and just clear prefix and set it to what object the thread is handling 
now. same will apply with threadlocal params too

> Incorrect job information is set while logging
> --
>
> Key: OOZIE-1939
> URL: https://issues.apache.org/jira/browse/OOZIE-1939
> Project: Oozie
>  Issue Type: Bug
>Reporter: Purshotam Shah
>Assignee: Azrael
> Attachments: OOZIE-1939.1.patch, OOZIE-1939.2.patch
>
>
> {code}
> 2014-07-16 17:28:06,422 DEBUG CoordChangeXCommand:545 [http-0.0.0.0-4443-5] - 
> USER[hadoopqa] GROUP[users] TOKEN[] APP[coordB236] 
> JOB[0011514-140716042555-oozie-oozi-C] ACTION[-] Acquired lock for 
> [0011385-140716042555-oozie-oozi-C] in [coord_change]
> 2014-07-16 17:28:06,422 TRACE CoordChangeXCommand:548 [http-0.0.0.0-4443-5] - 
> USER[hadoopqa] GROUP[users] TOKEN[] APP[coordB236] 
> JOB[0011514-140716042555-oozie-oozi-C] ACTION[-] Load state for 
> [0011385-140716042555-oozie-oozi-C]
> {code}
> {code}
> protected void loadState() throws CommandException {
> jpaService = Services.get().get(JPAService.class);
> if (jpaService == null) {
> LOG.error(ErrorCode.E0610);
> }
> try {
> coordJob = 
> CoordJobQueryExecutor.getInstance().get(CoordJobQuery.GET_COORD_JOB_MATERIALIZE,
>  jobId);
> prevStatus = coordJob.getStatus();
> }
> catch (JPAExecutorException jex) {
> throw new CommandException(jex);
> }
> // calculate start materialize and end materialize time
> calcMatdTime();
> LogUtils.setLogInfo(coordJob, logInfo);
> }
> {code}
> Most of the commands set jobinfo after loadstate, because of that few log 
> statements ( like acquiring lock, load state) logs with previous jobinfo. 



--
This message was sent by Atlassian JIRA
(v6.2#6252)

[jira] [Commented] (OOZIE-1811) Current test failures in trunk

2014-07-31 Thread Mona Chitnis (JIRA)


[ 
https://issues.apache.org/jira/browse/OOZIE-1811?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14081820#comment-14081820
 ] 

Mona Chitnis commented on OOZIE-1811:
-

{{. -1 the patch contains 2 line(s) with trailing spaces}} located and fixed in 
the xml file - {{coord-action-sla.xml}}

> Current test failures in trunk
> --
>
> Key: OOZIE-1811
> URL: https://issues.apache.org/jira/browse/OOZIE-1811
> Project: Oozie
>  Issue Type: Bug
>Affects Versions: trunk
>Reporter: Robert Kanter
>Assignee: Mona Chitnis
>Priority: Critical
> Attachments: OOZIE-1811-1.patch, OOZIE-1811-2.patch
>
>
> There's a bunch of test failures currently in trunk; I'm not sure what 
> commit(s) is the cause, but I think it was somewhat recent.
> e.g. https://builds.apache.org/job/oozie-trunk-precommit-build/1199/
> Reproducible by running these tests, instead of having to run them all, which 
> takes a lot longer :)
> {noformat}
> mvn clean test 
> -Dtest=TestSubWorkflowActionExecutor,TestBunldeChangeXCommand,TestCoordUpdateXCommand,TestCoordJobQueryExecutor,TestStatusTransitService,TestSLAEventGeneration
> {noformat}
> {noformat}
> Results :
> Failed tests:   
> testCoordinatorActionCommandsSubmitAndStart(org.apache.oozie.sla.TestSLAEventGeneration):
>  expected:<...11921-oozie-rkan-C@1[]> but was:<...11921-oozie-rkan-C@1[2]>
>   
> testCoordStatusTransitServiceDoneWithError(org.apache.oozie.service.TestStatusTransitService):
>  expected: but was:
>   
> testBundleStatusTransitRunningFromKilled(org.apache.oozie.service.TestStatusTransitService):
>  expected: but was:
> Tests in error: 
>   testGetList(org.apache.oozie.executor.jpa.TestCoordJobQueryExecutor)
>   testInsert(org.apache.oozie.executor.jpa.TestCoordJobQueryExecutor)
> Tests run: 62, Failures: 3, Errors: 2, Skipped: 0
> {noformat}



--
This message was sent by Atlassian JIRA
(v6.2#6252)

[jira] [Updated] (OOZIE-1932) Services should load CallableQueueService after MemoryLocksService

2014-07-31 Thread Mona Chitnis (JIRA)


 [ 
https://issues.apache.org/jira/browse/OOZIE-1932?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Mona Chitnis updated OOZIE-1932:


Attachment: OOZIE-1932-addendum.patch

attaching simple change to initialize CallableQueueService at the very end so 
that its destroyed first in order. unit test TestBulkMonitorWebServiceAPI 
failed in my local machine but not able to determine if cause related to this 
change. I will let the pre-commit build test run decide

> Services should load CallableQueueService after MemoryLocksService
> --
>
> Key: OOZIE-1932
> URL: https://issues.apache.org/jira/browse/OOZIE-1932
> Project: Oozie
>  Issue Type: Bug
>Affects Versions: trunk
>    Reporter: Mona Chitnis
>Assignee: Mona Chitnis
> Fix For: 4.1.0
>
> Attachments: OOZIE-1932-2.patch, OOZIE-1932-addendum.patch, 
> OOZIE-1932.patch
>
>
> This is not a problem during startup but is during shutdown, as services are 
> destroyed in reverse order of initialization. Hence, when MemoryLocksService 
> destroy sets it to null, and commands are still executing due to 
> CallableQueueService still active, they all encounter NPEs during locking. 
> This is a simple fix in oozie-default.xml to set MemoryLocksService before in 
> the order of services loading.



--
This message was sent by Atlassian JIRA
(v6.2#6252)

[jira] [Commented] (OOZIE-1811) Current test failures in trunk

2014-07-29 Thread Mona Chitnis (JIRA)


[ 
https://issues.apache.org/jira/browse/OOZIE-1811?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14078158#comment-14078158
 ] 

Mona Chitnis commented on OOZIE-1811:
-

above failures due to strange network error on the host. Happened before at 
https://builds.apache.org/job/oozie-trunk-precommit-build/1363/ too.

Ran the whole suit locally and only 1 failed, which I've mentioned is going to 
be part of OOZIE-1952.

> Current test failures in trunk
> --
>
> Key: OOZIE-1811
> URL: https://issues.apache.org/jira/browse/OOZIE-1811
> Project: Oozie
>  Issue Type: Bug
>Affects Versions: trunk
>Reporter: Robert Kanter
>Assignee: Mona Chitnis
>Priority: Critical
> Attachments: OOZIE-1811-1.patch, OOZIE-1811-2.patch
>
>
> There's a bunch of test failures currently in trunk; I'm not sure what 
> commit(s) is the cause, but I think it was somewhat recent.
> e.g. https://builds.apache.org/job/oozie-trunk-precommit-build/1199/
> Reproducible by running these tests, instead of having to run them all, which 
> takes a lot longer :)
> {noformat}
> mvn clean test 
> -Dtest=TestSubWorkflowActionExecutor,TestBunldeChangeXCommand,TestCoordUpdateXCommand,TestCoordJobQueryExecutor,TestStatusTransitService,TestSLAEventGeneration
> {noformat}
> {noformat}
> Results :
> Failed tests:   
> testCoordinatorActionCommandsSubmitAndStart(org.apache.oozie.sla.TestSLAEventGeneration):
>  expected:<...11921-oozie-rkan-C@1[]> but was:<...11921-oozie-rkan-C@1[2]>
>   
> testCoordStatusTransitServiceDoneWithError(org.apache.oozie.service.TestStatusTransitService):
>  expected: but was:
>   
> testBundleStatusTransitRunningFromKilled(org.apache.oozie.service.TestStatusTransitService):
>  expected: but was:
> Tests in error: 
>   testGetList(org.apache.oozie.executor.jpa.TestCoordJobQueryExecutor)
>   testInsert(org.apache.oozie.executor.jpa.TestCoordJobQueryExecutor)
> Tests run: 62, Failures: 3, Errors: 2, Skipped: 0
> {noformat}



--
This message was sent by Atlassian JIRA
(v6.2#6252)

[jira] [Reopened] (OOZIE-1932) Services should load CallableQueueService after MemoryLocksService

2014-07-29 Thread Mona Chitnis (JIRA)


 [ 
https://issues.apache.org/jira/browse/OOZIE-1932?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Mona Chitnis reopened OOZIE-1932:
-


Reopening issue to fix similar issue with URIHandlerService should be loaded 
before CallableQueueService, so that its closed before. This JIRA's scope to 
include a permanent fix to the services ordering to work for all cases, and 
avoid all NPEs and other issues with the services during server shutdown/startup

> Services should load CallableQueueService after MemoryLocksService
> --
>
> Key: OOZIE-1932
> URL: https://issues.apache.org/jira/browse/OOZIE-1932
> Project: Oozie
>  Issue Type: Bug
>Affects Versions: trunk
>    Reporter: Mona Chitnis
>Assignee: Mona Chitnis
> Fix For: 4.1.0
>
> Attachments: OOZIE-1932-2.patch, OOZIE-1932.patch
>
>
> This is not a problem during startup but is during shutdown, as services are 
> destroyed in reverse order of initialization. Hence, when MemoryLocksService 
> destroy sets it to null, and commands are still executing due to 
> CallableQueueService still active, they all encounter NPEs during locking. 
> This is a simple fix in oozie-default.xml to set MemoryLocksService before in 
> the order of services loading.



--
This message was sent by Atlassian JIRA
(v6.2#6252)

[jira] [Updated] (OOZIE-1811) Current test failures in trunk

2014-07-29 Thread Mona Chitnis (JIRA)


 [ 
https://issues.apache.org/jira/browse/OOZIE-1811?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Mona Chitnis updated OOZIE-1811:


Attachment: OOZIE-1811-2.patch

updated patch to apply cleanly to trunk HEAD

> Current test failures in trunk
> --
>
> Key: OOZIE-1811
> URL: https://issues.apache.org/jira/browse/OOZIE-1811
> Project: Oozie
>  Issue Type: Bug
>Affects Versions: trunk
>Reporter: Robert Kanter
>Assignee: Mona Chitnis
>Priority: Critical
> Attachments: OOZIE-1811-1.patch, OOZIE-1811-2.patch
>
>
> There's a bunch of test failures currently in trunk; I'm not sure what 
> commit(s) is the cause, but I think it was somewhat recent.
> e.g. https://builds.apache.org/job/oozie-trunk-precommit-build/1199/
> Reproducible by running these tests, instead of having to run them all, which 
> takes a lot longer :)
> {noformat}
> mvn clean test 
> -Dtest=TestSubWorkflowActionExecutor,TestBunldeChangeXCommand,TestCoordUpdateXCommand,TestCoordJobQueryExecutor,TestStatusTransitService,TestSLAEventGeneration
> {noformat}
> {noformat}
> Results :
> Failed tests:   
> testCoordinatorActionCommandsSubmitAndStart(org.apache.oozie.sla.TestSLAEventGeneration):
>  expected:<...11921-oozie-rkan-C@1[]> but was:<...11921-oozie-rkan-C@1[2]>
>   
> testCoordStatusTransitServiceDoneWithError(org.apache.oozie.service.TestStatusTransitService):
>  expected: but was:
>   
> testBundleStatusTransitRunningFromKilled(org.apache.oozie.service.TestStatusTransitService):
>  expected: but was:
> Tests in error: 
>   testGetList(org.apache.oozie.executor.jpa.TestCoordJobQueryExecutor)
>   testInsert(org.apache.oozie.executor.jpa.TestCoordJobQueryExecutor)
> Tests run: 62, Failures: 3, Errors: 2, Skipped: 0
> {noformat}



--
This message was sent by Atlassian JIRA
(v6.2#6252)

[jira] [Updated] (OOZIE-1811) Current test failures in trunk

2014-07-28 Thread Mona Chitnis (JIRA)


 [ 
https://issues.apache.org/jira/browse/OOZIE-1811?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Mona Chitnis updated OOZIE-1811:


Attachment: OOZIE-1811-1.patch

attaching patch which fixes the QueryExecutors and TestSLAEventGeneration. 
Errors related to StoreService usage in tests can be fixed as part of overall 
StoreService fix in OOZIE-1952

> Current test failures in trunk
> --
>
> Key: OOZIE-1811
> URL: https://issues.apache.org/jira/browse/OOZIE-1811
> Project: Oozie
>  Issue Type: Bug
>Affects Versions: trunk
>Reporter: Robert Kanter
>Assignee: Mona Chitnis
>Priority: Critical
> Attachments: OOZIE-1811-1.patch
>
>
> There's a bunch of test failures currently in trunk; I'm not sure what 
> commit(s) is the cause, but I think it was somewhat recent.
> e.g. https://builds.apache.org/job/oozie-trunk-precommit-build/1199/
> Reproducible by running these tests, instead of having to run them all, which 
> takes a lot longer :)
> {noformat}
> mvn clean test 
> -Dtest=TestSubWorkflowActionExecutor,TestBunldeChangeXCommand,TestCoordUpdateXCommand,TestCoordJobQueryExecutor,TestStatusTransitService,TestSLAEventGeneration
> {noformat}
> {noformat}
> Results :
> Failed tests:   
> testCoordinatorActionCommandsSubmitAndStart(org.apache.oozie.sla.TestSLAEventGeneration):
>  expected:<...11921-oozie-rkan-C@1[]> but was:<...11921-oozie-rkan-C@1[2]>
>   
> testCoordStatusTransitServiceDoneWithError(org.apache.oozie.service.TestStatusTransitService):
>  expected: but was:
>   
> testBundleStatusTransitRunningFromKilled(org.apache.oozie.service.TestStatusTransitService):
>  expected: but was:
> Tests in error: 
>   testGetList(org.apache.oozie.executor.jpa.TestCoordJobQueryExecutor)
>   testInsert(org.apache.oozie.executor.jpa.TestCoordJobQueryExecutor)
> Tests run: 62, Failures: 3, Errors: 2, Skipped: 0
> {noformat}



--
This message was sent by Atlassian JIRA
(v6.2#6252)

[jira] [Created] (OOZIE-1952) Cleanup duplicate/obsolete code - Command, StoreService

2014-07-28 Thread Mona Chitnis (JIRA)

Mona Chitnis created OOZIE-1952:
---

 Summary: Cleanup duplicate/obsolete code - Command, StoreService
 Key: OOZIE-1952
 URL: https://issues.apache.org/jira/browse/OOZIE-1952
 Project: Oozie
  Issue Type: Task
Reporter: Mona Chitnis


StoreService has been superceded by JPAService, and Command has been superceded 
by XCommand. These old classes have been lying around long enough and probably 
only referenced through unit tests, creating some confusion when tests have to 
be fixed for flaky failures



--
This message was sent by Atlassian JIRA
(v6.2#6252)

[jira] [Commented] (OOZIE-1811) Current test failures in trunk

2014-07-24 Thread Mona Chitnis (JIRA)


[ 
https://issues.apache.org/jira/browse/OOZIE-1811?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14074117#comment-14074117
 ] 

Mona Chitnis commented on OOZIE-1811:
-

I'd suggest getting rid of the static reference to JPAService in each of the 
Query Executors. We can always get the reference to it from the Services 
singleton, while executing the query. By keeping another static reference and 
manipulating it through the constructor and destroy(), we run the risk of 
nullifying it inadvertently. This is why suddenly so many tests are becoming 
flaky and it is very tough to detect exact patterns or even fix tests in a 
foolproof way. 

I ran the whole suit with the static reference removed and only 2 tests failed 
- which is quite an improvement!
{code}
Results :

Failed tests:   
testBundleId(org.apache.oozie.servlet.TestBulkMonitorWebServiceAPI): 
expected: but was:

Tests in error: 
  testSucCoordPurgeXCommand(org.apache.oozie.command.TestPurgeXCommand): E0604: 
Job does not exist [000-140724213655573-oozie-chit-C]
{code}

Test#2 here is failing with error StoreService cannot work without JPAService. 
We can replace usage of StoreService completely as it is superceded by 
JPAService anyway.
Test #1 doesnt really have any error except random assert fail, and this test 
is not usually flaky so can ignore


> Current test failures in trunk
> --
>
> Key: OOZIE-1811
> URL: https://issues.apache.org/jira/browse/OOZIE-1811
> Project: Oozie
>  Issue Type: Bug
>Affects Versions: trunk
>Reporter: Robert Kanter
>Assignee: Mona Chitnis
>Priority: Critical
>
> There's a bunch of test failures currently in trunk; I'm not sure what 
> commit(s) is the cause, but I think it was somewhat recent.
> e.g. https://builds.apache.org/job/oozie-trunk-precommit-build/1199/
> Reproducible by running these tests, instead of having to run them all, which 
> takes a lot longer :)
> {noformat}
> mvn clean test 
> -Dtest=TestSubWorkflowActionExecutor,TestBunldeChangeXCommand,TestCoordUpdateXCommand,TestCoordJobQueryExecutor,TestStatusTransitService,TestSLAEventGeneration
> {noformat}
> {noformat}
> Results :
> Failed tests:   
> testCoordinatorActionCommandsSubmitAndStart(org.apache.oozie.sla.TestSLAEventGeneration):
>  expected:<...11921-oozie-rkan-C@1[]> but was:<...11921-oozie-rkan-C@1[2]>
>   
> testCoordStatusTransitServiceDoneWithError(org.apache.oozie.service.TestStatusTransitService):
>  expected: but was:
>   
> testBundleStatusTransitRunningFromKilled(org.apache.oozie.service.TestStatusTransitService):
>  expected: but was:
> Tests in error: 
>   testGetList(org.apache.oozie.executor.jpa.TestCoordJobQueryExecutor)
>   testInsert(org.apache.oozie.executor.jpa.TestCoordJobQueryExecutor)
> Tests run: 62, Failures: 3, Errors: 2, Skipped: 0
> {noformat}



--
This message was sent by Atlassian JIRA
(v6.2#6252)

[jira] [Updated] (OOZIE-1944) Recursive variable resolution broken when same parameter name in config-default and action conf

2014-07-24 Thread Mona Chitnis (JIRA)


 [ 
https://issues.apache.org/jira/browse/OOZIE-1944?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Mona Chitnis updated OOZIE-1944:


Attachment: OOZIE-1944-2.patch

adding null check for configDefault which was causing TestWorkflowAppParser 
tests to fail

> Recursive variable resolution broken when same parameter name in 
> config-default and action conf
> ---
>
> Key: OOZIE-1944
> URL: https://issues.apache.org/jira/browse/OOZIE-1944
> Project: Oozie
>  Issue Type: Bug
>  Components: workflow
>Affects Versions: trunk
>    Reporter: Mona Chitnis
>Assignee: Mona Chitnis
> Fix For: 4.1.0
>
> Attachments: OOZIE-1944-1.patch, OOZIE-1944-2.patch
>
>
> Hitting error
> {code}
> can not create DagEngine for submitting jobs
> org.apache.oozie.DagEngineException: E0803: IO error, Variable
> substitution depth too large: 20 ${param}/000
> {code}
> when config-default.xml has
> {{param=default}}
> and action conf has
> {code}
> 
> ...
> 
> 
> param
> ${param}/000
> 
> 
> {code}



--
This message was sent by Atlassian JIRA
(v6.2#6252)

[jira] [Updated] (OOZIE-1944) Recursive variable resolution broken when same parameter name in config-default and action conf


 [ 
https://issues.apache.org/jira/browse/OOZIE-1944?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Mona Chitnis updated OOZIE-1944:


Attachment: OOZIE-1944-1.patch

Attaching patch. approach is to switch from using 
XConfiguration.injectDefaults() method to copy(), since the former does a 
Configuration.get() which tries to recursively resolve params. So simply, copy 
over defaults, global, and finally action , in this order of 
precedence

> Recursive variable resolution broken when same parameter name in 
> config-default and action conf
> ---
>
> Key: OOZIE-1944
> URL: https://issues.apache.org/jira/browse/OOZIE-1944
> Project: Oozie
>  Issue Type: Bug
>  Components: workflow
>Affects Versions: trunk
>    Reporter: Mona Chitnis
>Assignee: Mona Chitnis
> Fix For: 4.1.0
>
> Attachments: OOZIE-1944-1.patch
>
>
> Hitting error
> {code}
> can not create DagEngine for submitting jobs
> org.apache.oozie.DagEngineException: E0803: IO error, Variable
> substitution depth too large: 20 ${param}/000
> {code}
> when config-default.xml has
> {{param=default}}
> and action conf has
> {code}
> 
> ...
> 
> 
> param
> ${param}/000
> 
> 
> {code}



--
This message was sent by Atlassian JIRA
(v6.2#6252)

[jira] [Updated] (OOZIE-1944) Recursive variable resolution broken when same parameter name in config-default and action conf


 [ 
https://issues.apache.org/jira/browse/OOZIE-1944?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Mona Chitnis updated OOZIE-1944:


Fix Version/s: (was: trunk)

> Recursive variable resolution broken when same parameter name in 
> config-default and action conf
> ---
>
> Key: OOZIE-1944
> URL: https://issues.apache.org/jira/browse/OOZIE-1944
> Project: Oozie
>  Issue Type: Bug
>  Components: workflow
>Affects Versions: trunk
>    Reporter: Mona Chitnis
>Assignee: Mona Chitnis
> Fix For: 4.1.0
>
>
> Hitting error
> {code}
> can not create DagEngine for submitting jobs
> org.apache.oozie.DagEngineException: E0803: IO error, Variable
> substitution depth too large: 20 ${param}/000
> {code}
> when config-default.xml has
> {{param=default}}
> and action conf has
> {code}
> 
> ...
> 
> 
> param
> ${param}/000
> 
> 
> {code}



--
This message was sent by Atlassian JIRA
(v6.2#6252)

[jira] [Updated] (OOZIE-1872) TestCoordActionInputCheckXCommand.testActionInputCheckLatestActionCreationTime is failing for past couple of builds


 [ 
https://issues.apache.org/jira/browse/OOZIE-1872?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Mona Chitnis updated OOZIE-1872:


Fix Version/s: (was: trunk)

> TestCoordActionInputCheckXCommand.testActionInputCheckLatestActionCreationTime
>  is failing for past couple of builds
> ---
>
> Key: OOZIE-1872
> URL: https://issues.apache.org/jira/browse/OOZIE-1872
> Project: Oozie
>  Issue Type: Bug
>  Components: tests
>Affects Versions: trunk, 4.1.0
>Reporter: Rohini Palaniswamy
> Fix For: 4.1.0
>
> Attachments: OOZIE-1872-1.patch
>
>
> https://builds.apache.org/job/oozie-trunk-precommit-build/1291/testReport/junit/org.apache.oozie.command.coord/TestCoordActionInputCheckXCommand/testActionInputCheckLatestActionCreationTime/



--
This message was sent by Atlassian JIRA
(v6.2#6252)

[jira] [Updated] (OOZIE-1872) TestCoordActionInputCheckXCommand.testActionInputCheckLatestActionCreationTime is failing for past couple of builds


 [ 
https://issues.apache.org/jira/browse/OOZIE-1872?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Mona Chitnis updated OOZIE-1872:


Component/s: tests

> TestCoordActionInputCheckXCommand.testActionInputCheckLatestActionCreationTime
>  is failing for past couple of builds
> ---
>
> Key: OOZIE-1872
> URL: https://issues.apache.org/jira/browse/OOZIE-1872
> Project: Oozie
>  Issue Type: Bug
>  Components: tests
>Affects Versions: trunk, 4.1.0
>Reporter: Rohini Palaniswamy
> Fix For: trunk, 4.1.0
>
> Attachments: OOZIE-1872-1.patch
>
>
> https://builds.apache.org/job/oozie-trunk-precommit-build/1291/testReport/junit/org.apache.oozie.command.coord/TestCoordActionInputCheckXCommand/testActionInputCheckLatestActionCreationTime/



--
This message was sent by Atlassian JIRA
(v6.2#6252)

[jira] [Updated] (OOZIE-1872) TestCoordActionInputCheckXCommand.testActionInputCheckLatestActionCreationTime is failing for past couple of builds


 [ 
https://issues.apache.org/jira/browse/OOZIE-1872?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Mona Chitnis updated OOZIE-1872:


Attachment: OOZIE-1872-1.patch

attaching fix to test case. Root causes of failure were

 * materialize command was directly queuing input-check command with zero 
delay, with no changes in action-actual-time taking effect. Hence actual time 
was not updated to desired value, instead remaining at 'current time' and 
giving wrong dependency results
 * explicitly invoked input-check command was failing precondition verification 
due to earlier command transitioning action to FAILED.
* flakiness was due to timing issues of the direct vs explicit input-check 
commands

> TestCoordActionInputCheckXCommand.testActionInputCheckLatestActionCreationTime
>  is failing for past couple of builds
> ---
>
> Key: OOZIE-1872
> URL: https://issues.apache.org/jira/browse/OOZIE-1872
> Project: Oozie
>  Issue Type: Bug
>Affects Versions: trunk, 4.1.0
>Reporter: Rohini Palaniswamy
> Fix For: trunk, 4.1.0
>
> Attachments: OOZIE-1872-1.patch
>
>
> https://builds.apache.org/job/oozie-trunk-precommit-build/1291/testReport/junit/org.apache.oozie.command.coord/TestCoordActionInputCheckXCommand/testActionInputCheckLatestActionCreationTime/



--
This message was sent by Atlassian JIRA
(v6.2#6252)

[jira] [Created] (OOZIE-1945) NPE in JaveActionExecutor#check()

2014-07-22 Thread Mona Chitnis (JIRA)

Mona Chitnis created OOZIE-1945:
---

 Summary: NPE in JaveActionExecutor#check()
 Key: OOZIE-1945
 URL: https://issues.apache.org/jira/browse/OOZIE-1945
 Project: Oozie
  Issue Type: Bug
Affects Versions: trunk
Reporter: Mona Chitnis
Priority: Trivial
 Fix For: trunk, 4.1.0


in method check()
{code}
 String errorCode = props.getProperty("error.code");
if (errorCode.equals("0")) {
errorCode = "JA018";
}
if (errorCode.equals("-1")) {
errorCode = "JA019";
}
errorReason = props.getProperty("error.reason");
{code}
if error.code is null, these leads to NPEs
easy fix
{code}
if ("0".equals(errorCode))
...
{code}



--
This message was sent by Atlassian JIRA
(v6.2#6252)

[jira] [Updated] (OOZIE-1536) Coordinator action reruns start a new workflow

2014-07-22 Thread Mona Chitnis (JIRA)


 [ 
https://issues.apache.org/jira/browse/OOZIE-1536?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Mona Chitnis updated OOZIE-1536:


Assignee: (was: Mona Chitnis)

> Coordinator action reruns start a new workflow
> --
>
> Key: OOZIE-1536
> URL: https://issues.apache.org/jira/browse/OOZIE-1536
> Project: Oozie
>  Issue Type: Improvement
>Reporter: Srikanth Sundarrajan
>
> Coordinator action reruns start a new workflow and if existing workflow for 
> the action is in running state, the same is not checked. Coord rerun can 
> possibly do a workflow re-run to prevent this.



--
This message was sent by Atlassian JIRA
(v6.2#6252)

[jira] [Updated] (OOZIE-1944) Recursive variable resolution broken when same parameter name in config-default and action conf


 [ 
https://issues.apache.org/jira/browse/OOZIE-1944?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Mona Chitnis updated OOZIE-1944:


Fix Version/s: 4.1.0

> Recursive variable resolution broken when same parameter name in 
> config-default and action conf
> ---
>
> Key: OOZIE-1944
> URL: https://issues.apache.org/jira/browse/OOZIE-1944
> Project: Oozie
>  Issue Type: Bug
>  Components: workflow
>Affects Versions: trunk
>    Reporter: Mona Chitnis
>Assignee: Mona Chitnis
> Fix For: trunk, 4.1.0
>
>
> Hitting error
> {code}
> can not create DagEngine for submitting jobs
> org.apache.oozie.DagEngineException: E0803: IO error, Variable
> substitution depth too large: 20 ${param}/000
> {code}
> when config-default.xml has
> {{param=default}}
> and action conf has
> {code}
> 
> ...
> 
> 
> param
> ${param}/000
> 
> 
> {code}



--
This message was sent by Atlassian JIRA
(v6.2#6252)

[jira] [Created] (OOZIE-1944) Recursive variable resolution broken when same parameter name in config-default and action conf

Mona Chitnis created OOZIE-1944:
---

 Summary: Recursive variable resolution broken when same parameter 
name in config-default and action conf
 Key: OOZIE-1944
 URL: https://issues.apache.org/jira/browse/OOZIE-1944
 Project: Oozie
  Issue Type: Bug
  Components: workflow
Affects Versions: trunk
Reporter: Mona Chitnis
Assignee: Mona Chitnis
 Fix For: trunk


Hitting error
{code}
can not create DagEngine for submitting jobs
org.apache.oozie.DagEngineException: E0803: IO error, Variable
substitution depth too large: 20 ${param}/000
{code}

when config-default.xml has
{{param=default}}
and action conf has
{code}

...


param
${param}/000


{code}



--
This message was sent by Atlassian JIRA
(v6.2#6252)

[jira] [Updated] (OOZIE-1933) SLACalculatorMemory HA changes assume SLARegistrationBean exists for all jobs


 [ 
https://issues.apache.org/jira/browse/OOZIE-1933?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Mona Chitnis updated OOZIE-1933:


Attachment: OOZIE-1933-unit-tests-fix.patch

updated patch for cleanly apply to trunk

> SLACalculatorMemory HA changes assume SLARegistrationBean exists for all jobs
> -
>
> Key: OOZIE-1933
> URL: https://issues.apache.org/jira/browse/OOZIE-1933
> Project: Oozie
>  Issue Type: Bug
>Affects Versions: trunk
>    Reporter: Mona Chitnis
>Assignee: Mona Chitnis
> Fix For: trunk
>
> Attachments: OOZIE-1933-3.patch, OOZIE-1933-4-1.patch, 
> OOZIE-1933-unit-tests-fix.patch
>
>
> SLACalculatorMemory.addJobStatus()
> {code}
> else {
> // jobid might not exist in slaMap in HA Setting
> SLARegistrationBean slaRegBean = 
> SLARegistrationQueryExecutor.getInstance().get(
> SLARegQuery.GET_SLA_REG_ALL, jobId);
> SLASummaryBean slaSummaryBean = 
> SLASummaryQueryExecutor.getInstance().get(SLASummaryQuery.GET_SLA_SUMMARY,
> jobId);
> slaCalc = new SLACalcStatus(slaSummaryBean, slaRegBean);
> {code}
> Because of SLA Listener, job notification event triggers this even for jobs 
> with no SLA configured - leading to NPE in the SLACalcStatus constructor and 
> annoying exception stacktraces in logs
> Patch to also include log prefix addition to some SLACalculator log line



--
This message was sent by Atlassian JIRA
(v6.2#6252)

[jira] [Updated] (OOZIE-1933) SLACalculatorMemory HA changes assume SLARegistrationBean exists for all jobs


 [ 
https://issues.apache.org/jira/browse/OOZIE-1933?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Mona Chitnis updated OOZIE-1933:


Attachment: (was: sla_unit_tests-1.patch)

> SLACalculatorMemory HA changes assume SLARegistrationBean exists for all jobs
> -
>
> Key: OOZIE-1933
> URL: https://issues.apache.org/jira/browse/OOZIE-1933
> Project: Oozie
>  Issue Type: Bug
>Affects Versions: trunk
>    Reporter: Mona Chitnis
>Assignee: Mona Chitnis
> Fix For: trunk
>
> Attachments: OOZIE-1933-3.patch, OOZIE-1933-4-1.patch, 
> OOZIE-1933-unit-tests-fix.patch
>
>
> SLACalculatorMemory.addJobStatus()
> {code}
> else {
> // jobid might not exist in slaMap in HA Setting
> SLARegistrationBean slaRegBean = 
> SLARegistrationQueryExecutor.getInstance().get(
> SLARegQuery.GET_SLA_REG_ALL, jobId);
> SLASummaryBean slaSummaryBean = 
> SLASummaryQueryExecutor.getInstance().get(SLASummaryQuery.GET_SLA_SUMMARY,
> jobId);
> slaCalc = new SLACalcStatus(slaSummaryBean, slaRegBean);
> {code}
> Because of SLA Listener, job notification event triggers this even for jobs 
> with no SLA configured - leading to NPE in the SLACalcStatus constructor and 
> annoying exception stacktraces in logs
> Patch to also include log prefix addition to some SLACalculator log line



--
This message was sent by Atlassian JIRA
(v6.2#6252)

[jira] [Resolved] (OOZIE-1933) SLACalculatorMemory HA changes assume SLARegistrationBean exists for all jobs


 [ 
https://issues.apache.org/jira/browse/OOZIE-1933?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Mona Chitnis resolved OOZIE-1933.
-

Resolution: Fixed

failing unit tests fix committed to trunk after review. Thanks!

> SLACalculatorMemory HA changes assume SLARegistrationBean exists for all jobs
> -
>
> Key: OOZIE-1933
> URL: https://issues.apache.org/jira/browse/OOZIE-1933
> Project: Oozie
>  Issue Type: Bug
>Affects Versions: trunk
>    Reporter: Mona Chitnis
>Assignee: Mona Chitnis
> Fix For: trunk
>
> Attachments: OOZIE-1933-3.patch, OOZIE-1933-4-1.patch, 
> OOZIE-1933-unit-tests-fix.patch
>
>
> SLACalculatorMemory.addJobStatus()
> {code}
> else {
> // jobid might not exist in slaMap in HA Setting
> SLARegistrationBean slaRegBean = 
> SLARegistrationQueryExecutor.getInstance().get(
> SLARegQuery.GET_SLA_REG_ALL, jobId);
> SLASummaryBean slaSummaryBean = 
> SLASummaryQueryExecutor.getInstance().get(SLASummaryQuery.GET_SLA_SUMMARY,
> jobId);
> slaCalc = new SLACalcStatus(slaSummaryBean, slaRegBean);
> {code}
> Because of SLA Listener, job notification event triggers this even for jobs 
> with no SLA configured - leading to NPE in the SLACalcStatus constructor and 
> annoying exception stacktraces in logs
> Patch to also include log prefix addition to some SLACalculator log line



--
This message was sent by Atlassian JIRA
(v6.2#6252)

[jira] [Updated] (OOZIE-1933) SLACalculatorMemory HA changes assume SLARegistrationBean exists for all jobs


 [ 
https://issues.apache.org/jira/browse/OOZIE-1933?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Mona Chitnis updated OOZIE-1933:


Attachment: (was: sla_unit_tests.patch)

> SLACalculatorMemory HA changes assume SLARegistrationBean exists for all jobs
> -
>
> Key: OOZIE-1933
> URL: https://issues.apache.org/jira/browse/OOZIE-1933
> Project: Oozie
>  Issue Type: Bug
>Affects Versions: trunk
>    Reporter: Mona Chitnis
>Assignee: Mona Chitnis
> Fix For: trunk
>
> Attachments: OOZIE-1933-3.patch, OOZIE-1933-4-1.patch, 
> OOZIE-1933-unit-tests-fix.patch
>
>
> SLACalculatorMemory.addJobStatus()
> {code}
> else {
> // jobid might not exist in slaMap in HA Setting
> SLARegistrationBean slaRegBean = 
> SLARegistrationQueryExecutor.getInstance().get(
> SLARegQuery.GET_SLA_REG_ALL, jobId);
> SLASummaryBean slaSummaryBean = 
> SLASummaryQueryExecutor.getInstance().get(SLASummaryQuery.GET_SLA_SUMMARY,
> jobId);
> slaCalc = new SLACalcStatus(slaSummaryBean, slaRegBean);
> {code}
> Because of SLA Listener, job notification event triggers this even for jobs 
> with no SLA configured - leading to NPE in the SLACalcStatus constructor and 
> annoying exception stacktraces in logs
> Patch to also include log prefix addition to some SLACalculator log line



--
This message was sent by Atlassian JIRA
(v6.2#6252)

[jira] [Commented] (OOZIE-1811) Current test failures in trunk

2014-07-18 Thread Mona Chitnis (JIRA)


[ 
https://issues.apache.org/jira/browse/OOZIE-1811?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14066771#comment-14066771
 ] 

Mona Chitnis commented on OOZIE-1811:
-

{{org.apache.oozie.command.coord.TestCoordActionInputCheckXCommand.testActionInputCheckLatestCurrentTime}}
 also failing because JPAService null. Same class test but using latest 
calculation with rest to action creation time (old behavior) 
{{org.apache.oozie.command.coord.TestCoordActionInputCheckXCommand.testActionInputCheckLatestActionCreationTime}}
 however, failing with a dependency mismatch problem - OOZIE-1872

> Current test failures in trunk
> --
>
> Key: OOZIE-1811
> URL: https://issues.apache.org/jira/browse/OOZIE-1811
> Project: Oozie
>  Issue Type: Bug
>Affects Versions: trunk
>Reporter: Robert Kanter
>Assignee: Mona Chitnis
>Priority: Critical
>
> There's a bunch of test failures currently in trunk; I'm not sure what 
> commit(s) is the cause, but I think it was somewhat recent.
> e.g. https://builds.apache.org/job/oozie-trunk-precommit-build/1199/
> Reproducible by running these tests, instead of having to run them all, which 
> takes a lot longer :)
> {noformat}
> mvn clean test 
> -Dtest=TestSubWorkflowActionExecutor,TestBunldeChangeXCommand,TestCoordUpdateXCommand,TestCoordJobQueryExecutor,TestStatusTransitService,TestSLAEventGeneration
> {noformat}
> {noformat}
> Results :
> Failed tests:   
> testCoordinatorActionCommandsSubmitAndStart(org.apache.oozie.sla.TestSLAEventGeneration):
>  expected:<...11921-oozie-rkan-C@1[]> but was:<...11921-oozie-rkan-C@1[2]>
>   
> testCoordStatusTransitServiceDoneWithError(org.apache.oozie.service.TestStatusTransitService):
>  expected: but was:
>   
> testBundleStatusTransitRunningFromKilled(org.apache.oozie.service.TestStatusTransitService):
>  expected: but was:
> Tests in error: 
>   testGetList(org.apache.oozie.executor.jpa.TestCoordJobQueryExecutor)
>   testInsert(org.apache.oozie.executor.jpa.TestCoordJobQueryExecutor)
> Tests run: 62, Failures: 3, Errors: 2, Skipped: 0
> {noformat}



--
This message was sent by Atlassian JIRA
(v6.2#6252)

[jira] [Updated] (OOZIE-1933) SLACalculatorMemory HA changes assume SLARegistrationBean exists for all jobs


 [ 
https://issues.apache.org/jira/browse/OOZIE-1933?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Mona Chitnis updated OOZIE-1933:


Attachment: sla_unit_tests-1.patch

updated patch to include another broken testcase. All other failed tests pass 
locally and are known to be flaky

Test run:
{code}
Results :

Failed tests:   
testCoordinatorActionCommandsSubmitAndStart(org.apache.oozie.sla.TestSLAEventGeneration)
  testRecovery(org.apache.oozie.action.hadoop.TestJavaActionExecutor): 
expected:<[SUCCEED]ED> but was:<[FAILED/KILL]ED>
  
testCoordStatusTransitServiceBackwardSupport(org.apache.oozie.service.TestStatusTransitService)

Tests in error: 
  testOnJobEvent(org.apache.oozie.sla.TestSLAJobEventListener): invalid child 
id [wa1]
  
testActionReuseWfJobAppPath(org.apache.oozie.command.wf.TestActionStartXCommand):
 E0607: Other error in operation [action.start], null
  testWorkflowRun(org.apache.oozie.command.wf.TestLastModified): 
org.apache.oozie.DagEngineException: E0607: Other error in operation [start], 
null
  testSucJobPurgeXCommand(org.apache.oozie.command.TestPurgeXCommand): E0604: 
Job does not exist [001-140717193440158-oozie-chit-W]
  testSucCoordPurgeXCommand(org.apache.oozie.command.TestPurgeXCommand): E0604: 
Job does not exist [000-140717193442386-oozie-chit-C]
{code}


> SLACalculatorMemory HA changes assume SLARegistrationBean exists for all jobs
> -
>
> Key: OOZIE-1933
> URL: https://issues.apache.org/jira/browse/OOZIE-1933
> Project: Oozie
>  Issue Type: Bug
>Affects Versions: trunk
>Reporter: Mona Chitnis
>Assignee: Mona Chitnis
> Fix For: trunk
>
> Attachments: OOZIE-1933-3.patch, OOZIE-1933-4-1.patch, 
> sla_unit_tests-1.patch, sla_unit_tests.patch
>
>
> SLACalculatorMemory.addJobStatus()
> {code}
> else {
> // jobid might not exist in slaMap in HA Setting
> SLARegistrationBean slaRegBean = 
> SLARegistrationQueryExecutor.getInstance().get(
> SLARegQuery.GET_SLA_REG_ALL, jobId);
> SLASummaryBean slaSummaryBean = 
> SLASummaryQueryExecutor.getInstance().get(SLASummaryQuery.GET_SLA_SUMMARY,
> jobId);
> slaCalc = new SLACalcStatus(slaSummaryBean, slaRegBean);
> {code}
> Because of SLA Listener, job notification event triggers this even for jobs 
> with no SLA configured - leading to NPE in the SLACalcStatus constructor and 
> annoying exception stacktraces in logs
> Patch to also include log prefix addition to some SLACalculator log line



--
This message was sent by Atlassian JIRA
(v6.2#6252)

[jira] [Updated] (OOZIE-1933) SLACalculatorMemory HA changes assume SLARegistrationBean exists for all jobs


 [ 
https://issues.apache.org/jira/browse/OOZIE-1933?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Mona Chitnis updated OOZIE-1933:


Attachment: sla_unit_tests.patch

> SLACalculatorMemory HA changes assume SLARegistrationBean exists for all jobs
> -
>
> Key: OOZIE-1933
> URL: https://issues.apache.org/jira/browse/OOZIE-1933
> Project: Oozie
>  Issue Type: Bug
>Affects Versions: trunk
>    Reporter: Mona Chitnis
>Assignee: Mona Chitnis
> Fix For: trunk
>
> Attachments: OOZIE-1933-3.patch, OOZIE-1933-4-1.patch, 
> sla_unit_tests.patch
>
>
> SLACalculatorMemory.addJobStatus()
> {code}
> else {
> // jobid might not exist in slaMap in HA Setting
> SLARegistrationBean slaRegBean = 
> SLARegistrationQueryExecutor.getInstance().get(
> SLARegQuery.GET_SLA_REG_ALL, jobId);
> SLASummaryBean slaSummaryBean = 
> SLASummaryQueryExecutor.getInstance().get(SLASummaryQuery.GET_SLA_SUMMARY,
> jobId);
> slaCalc = new SLACalcStatus(slaSummaryBean, slaRegBean);
> {code}
> Because of SLA Listener, job notification event triggers this even for jobs 
> with no SLA configured - leading to NPE in the SLACalcStatus constructor and 
> annoying exception stacktraces in logs
> Patch to also include log prefix addition to some SLACalculator log line



--
This message was sent by Atlassian JIRA
(v6.2#6252)

[jira] [Reopened] (OOZIE-1933) SLACalculatorMemory HA changes assume SLARegistrationBean exists for all jobs


 [ 
https://issues.apache.org/jira/browse/OOZIE-1933?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Mona Chitnis reopened OOZIE-1933:
-


adding test cases broken by the patch

> SLACalculatorMemory HA changes assume SLARegistrationBean exists for all jobs
> -
>
> Key: OOZIE-1933
> URL: https://issues.apache.org/jira/browse/OOZIE-1933
> Project: Oozie
>  Issue Type: Bug
>Affects Versions: trunk
>    Reporter: Mona Chitnis
>Assignee: Mona Chitnis
> Fix For: trunk
>
> Attachments: OOZIE-1933-3.patch, OOZIE-1933-4-1.patch
>
>
> SLACalculatorMemory.addJobStatus()
> {code}
> else {
> // jobid might not exist in slaMap in HA Setting
> SLARegistrationBean slaRegBean = 
> SLARegistrationQueryExecutor.getInstance().get(
> SLARegQuery.GET_SLA_REG_ALL, jobId);
> SLASummaryBean slaSummaryBean = 
> SLASummaryQueryExecutor.getInstance().get(SLASummaryQuery.GET_SLA_SUMMARY,
> jobId);
> slaCalc = new SLACalcStatus(slaSummaryBean, slaRegBean);
> {code}
> Because of SLA Listener, job notification event triggers this even for jobs 
> with no SLA configured - leading to NPE in the SLACalcStatus constructor and 
> annoying exception stacktraces in logs
> Patch to also include log prefix addition to some SLACalculator log line



--
This message was sent by Atlassian JIRA
(v6.2#6252)

[jira] [Resolved] (OOZIE-1933) SLACalculatorMemory HA changes assume SLARegistrationBean exists for all jobs