oozie git commit: OOZIE-3103 Improve log message regarding max concurrency for coordinators (asasvari via rkanter)

2017-11-06 Thread rkanter 
Repository: oozie
Updated Branches:
  refs/heads/master 6edcc291b -> cc9f30607


OOZIE-3103 Improve log message regarding max concurrency for coordinators 
(asasvari via rkanter)


Project: http://git-wip-us.apache.org/repos/asf/oozie/repo
Commit: http://git-wip-us.apache.org/repos/asf/oozie/commit/cc9f3060
Tree: http://git-wip-us.apache.org/repos/asf/oozie/tree/cc9f3060
Diff: http://git-wip-us.apache.org/repos/asf/oozie/diff/cc9f3060

Branch: refs/heads/master
Commit: cc9f30607f5f0104be8e161d245f46b0e45c4eb5
Parents: 6edcc29
Author: Robert Kanter 
Authored: Mon Nov 6 10:06:33 2017 -0800
Committer: Robert Kanter 
Committed: Mon Nov 6 10:06:33 2017 -0800

--
 .../org/apache/oozie/command/coord/CoordActionReadyXCommand.java  | 3 ++-
 release-log.txt   | 1 +
 2 files changed, 3 insertions(+), 1 deletion(-)
--


http://git-wip-us.apache.org/repos/asf/oozie/blob/cc9f3060/core/src/main/java/org/apache/oozie/command/coord/CoordActionReadyXCommand.java
--
diff --git 
a/core/src/main/java/org/apache/oozie/command/coord/CoordActionReadyXCommand.java
 
b/core/src/main/java/org/apache/oozie/command/coord/CoordActionReadyXCommand.java
index 2d8af04..40bd931 100644
--- 
a/core/src/main/java/org/apache/oozie/command/coord/CoordActionReadyXCommand.java
+++ 
b/core/src/main/java/org/apache/oozie/command/coord/CoordActionReadyXCommand.java
@@ -100,7 +100,8 @@ public class CoordActionReadyXCommand extends 
CoordinatorXCommand {
 + numRunningJobs + ", numLeftover=" + numActionsToStart);
 // no actions to start
 if (numActionsToStart == 0) {
-log.warn("No actions to start for jobId=" + jobId + " as max 
concurrency reached!");
+log.info("Not starting any additional actions because max 
concurrency [{0}]" +
+" for coordinator [{1}] has been reached.", 
jobConcurrency, jobId);
 }
 }
 // get list of actions that are READY and fit in the concurrency and 
execution

http://git-wip-us.apache.org/repos/asf/oozie/blob/cc9f3060/release-log.txt
--
diff --git a/release-log.txt b/release-log.txt
index 93add2e..ec26359 100644
--- a/release-log.txt
+++ b/release-log.txt
@@ -1,5 +1,6 @@
 -- Oozie 5.0.0 release (trunk - unreleased)
 
+OOZIE-3103 Improve log message regarding max concurrency for coordinators 
(asasvari via rkanter)
 OOZIE-3102 External Link to "EL Expression Language Quick Reference" not found 
(asasvari)
 OOZIE-2618 Include hive-common in hive-beeline as compile component for 
hive-server2 (hive2 action) (wutaklon via andras.piros)
 OOZIE-1401 PurgeCommand should purge the workflow jobs w/o end_time (asasvari)

oozie git commit: OOZIE-3102 External Link to "EL Expression Language Quick Reference" not found (asasvari)

2017-11-06 Thread asasvari 
Repository: oozie
Updated Branches:
  refs/heads/master 97a8cba82 -> 6edcc291b


OOZIE-3102 External Link to "EL Expression Language Quick Reference" not found 
(asasvari)


Project: http://git-wip-us.apache.org/repos/asf/oozie/repo
Commit: http://git-wip-us.apache.org/repos/asf/oozie/commit/6edcc291
Tree: http://git-wip-us.apache.org/repos/asf/oozie/tree/6edcc291
Diff: http://git-wip-us.apache.org/repos/asf/oozie/diff/6edcc291

Branch: refs/heads/master
Commit: 6edcc291b1d13a2f950b8ae40669ccfe0cc94039
Parents: 97a8cba
Author: Attila Sasvari 
Authored: Sat Nov 4 10:42:23 2017 +0100
Committer: Attila Sasvari 
Committed: Mon Nov 6 15:28:33 2017 +0100

--
 docs/src/site/twiki/index.twiki | 2 +-
 release-log.txt | 1 +
 2 files changed, 2 insertions(+), 1 deletion(-)
--


http://git-wip-us.apache.org/repos/asf/oozie/blob/6edcc291/docs/src/site/twiki/index.twiki
--
diff --git a/docs/src/site/twiki/index.twiki b/docs/src/site/twiki/index.twiki
index b7c962d..3003fa9 100644
--- a/docs/src/site/twiki/index.twiki
+++ b/docs/src/site/twiki/index.twiki
@@ -41,7 +41,7 @@ Enough reading already? Follow the steps in 
[[DG_QuickStart][Oozie Quick Start]]
* [[WorkflowFunctionalSpec][Workflow Functional Specification]]
* [[CoordinatorFunctionalSpec][Coordinator Functional Specification]]
* [[BundleFunctionalSpec][Bundle Functional Specification]]
-   * [[http://docs.oracle.com/javaee/6/tutorial/doc/gjddd.html][EL Expression 
Language Quick Reference]]
+   * [[https://docs.oracle.com/javaee/7/tutorial/jsf-el.htm][EL Expression 
Language Quick Reference]]
* [[DG_CommandLineTool][Command Line Tool]]
* [[DG_WorkflowReRun][Workflow Re-runs Explained]]
* [[DG_HCatalogIntegration][HCatalog Integration Explained]]

http://git-wip-us.apache.org/repos/asf/oozie/blob/6edcc291/release-log.txt
--
diff --git a/release-log.txt b/release-log.txt
index 0b3c59a..93add2e 100644
--- a/release-log.txt
+++ b/release-log.txt
@@ -1,5 +1,6 @@
 -- Oozie 5.0.0 release (trunk - unreleased)
 
+OOZIE-3102 External Link to "EL Expression Language Quick Reference" not found 
(asasvari)
 OOZIE-2618 Include hive-common in hive-beeline as compile component for 
hive-server2 (hive2 action) (wutaklon via andras.piros)
 OOZIE-1401 PurgeCommand should purge the workflow jobs w/o end_time (asasvari)
 OOZIE-3084 Add missing JVM properties to Jetty startup script (andras.piros)

oozie git commit: OOZIE-1401 PurgeCommand should purge the workflow jobs w/o end_time (asasvari)

2017-11-06 Thread asasvari 
Repository: oozie
Updated Branches:
  refs/heads/master 37e9d7cf9 -> cab20408e


OOZIE-1401 PurgeCommand should purge the workflow jobs w/o end_time (asasvari)


Project: http://git-wip-us.apache.org/repos/asf/oozie/repo
Commit: http://git-wip-us.apache.org/repos/asf/oozie/commit/cab20408
Tree: http://git-wip-us.apache.org/repos/asf/oozie/tree/cab20408
Diff: http://git-wip-us.apache.org/repos/asf/oozie/diff/cab20408

Branch: refs/heads/master
Commit: cab20408e54b10c8a4c90a4e0ba6df2e99db45f9
Parents: 37e9d7c
Author: Attila Sasvari 
Authored: Sat Nov 4 22:21:36 2017 +0100
Committer: Attila Sasvari 
Committed: Mon Nov 6 10:34:36 2017 +0100

--
 .../org/apache/oozie/command/PurgeXCommand.java | 14 +--
 .../apache/oozie/command/TestPurgeXCommand.java | 26 +++-
 release-log.txt |  1 +
 3 files changed, 38 insertions(+), 3 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/oozie/blob/cab20408/core/src/main/java/org/apache/oozie/command/PurgeXCommand.java
--
diff --git a/core/src/main/java/org/apache/oozie/command/PurgeXCommand.java 
b/core/src/main/java/org/apache/oozie/command/PurgeXCommand.java
index 2ba1999..166c236 100644
--- a/core/src/main/java/org/apache/oozie/command/PurgeXCommand.java
+++ b/core/src/main/java/org/apache/oozie/command/PurgeXCommand.java
@@ -43,8 +43,10 @@ import org.eclipse.jgit.util.StringUtils;
 import java.util.ArrayList;
 import java.util.Collection;
 import java.util.Collections;
+import java.util.Date;
 import java.util.Iterator;
 import java.util.List;
+
 /**
  * This class is used to purge workflows, coordinators, and bundles.  It takes 
into account the relationships between workflows and
  * coordinators, and coordinators and bundles.  It also only acts on 'limit' 
number of items at a time to not overtax the DB and in
@@ -246,8 +248,16 @@ public class PurgeXCommand extends XCommand {
 List children = new ArrayList();
 long wfOlderThanMS = System.currentTimeMillis() - (wfOlderThan * 
DAY_IN_MS);
 for (WorkflowJobBean wfjBean : wfBeanList) {
-if (wfjBean.inTerminalState() && wfjBean.getEndTime().getTime() < 
wfOlderThanMS) {
-children.add(wfjBean.getId());
+final Date wfEndTime = wfjBean.getEndTime();
+final boolean isFinished = wfjBean.inTerminalState();
+if (isFinished && wfEndTime != null && wfEndTime.getTime() < 
wfOlderThanMS) {
+children.add(wfjBean.getId());
+}
+else {
+final Date lastModificationTime = 
wfjBean.getLastModifiedTime();
+if (isFinished && lastModificationTime != null && 
lastModificationTime.getTime() < wfOlderThanMS) {
+children.add(wfjBean.getId());
+}
 }
 }
 return children;

http://git-wip-us.apache.org/repos/asf/oozie/blob/cab20408/core/src/test/java/org/apache/oozie/command/TestPurgeXCommand.java
--
diff --git a/core/src/test/java/org/apache/oozie/command/TestPurgeXCommand.java 
b/core/src/test/java/org/apache/oozie/command/TestPurgeXCommand.java
index 20a59c0..af9591b 100644
--- a/core/src/test/java/org/apache/oozie/command/TestPurgeXCommand.java
+++ b/core/src/test/java/org/apache/oozie/command/TestPurgeXCommand.java
@@ -174,10 +174,34 @@ public class TestPurgeXCommand extends XDataTestCase {
 }
 
 /**
- * Test : purge killed wf job and action successfully
+ * Test : purge failed wf job with null end_time successfully
  *
  * @throws Exception
  */
+public void testFailedJobNullEndTimePurgeXCommand() throws Exception {
+final WorkflowJobBean job = 
this.addRecordToWfJobTable(WorkflowJob.Status.FAILED, 
WorkflowInstance.Status.FAILED);
+final Date endTime = job.getEndTime();
+job.setLastModifiedTime(endTime);
+job.setEndTime(null);
+
+final JPAService jpaService = Services.get().get(JPAService.class);
+final WorkflowJobGetJPAExecutor wfJobGetCmd = new 
WorkflowJobGetJPAExecutor(job.getId());
+
+new PurgeXCommand(7, 1, 1, 10).call();
+
+try {
+jpaService.execute(wfJobGetCmd);
+fail("Workflow Job should have been purged");
+} catch (JPAExecutorException je) {
+assertEquals(ErrorCode.E0604, je.getErrorCode());
+}
+}
+
+/**
+* Test : purge killed wf job and action successfully
+*
+* @throws Exception
+*/
 public void testKillJobPurgeXCommand() throws Exception {
 WorkflowJobBean job = 
this.addRecordToWfJobTable(WorkflowJob.Status.KILLED, 
WorkflowInstance.Status.KILLED);