[jira] [Created] (OOZIE-1865) Oozie servers can't talk to each other with Oozie HA and Kerberos

[Robert Kanter (JIRA)]

Robert Kanter created OOZIE-1865:


 Summary: Oozie servers can't talk to each other with Oozie HA and 
Kerberos
 Key: OOZIE-1865
 URL: https://issues.apache.org/jira/browse/OOZIE-1865
 Project: Oozie
  Issue Type: Bug
  Components: HA
Affects Versions: trunk
Reporter: Robert Kanter
Assignee: Robert Kanter


When you use Oozie HA with Kerberos, you have to set 
{{oozie.authentication.kerberos.principal}} to {{HTTP/}} 
instead of {{HTTP/}}.  This allows clients to connect to any 
of the Oozie servers through the load balancer.  However, it also blocks 
clients from directly talking to any of the Oozie servers.  In and of itself, 
that's okay, but it turns out that in most cases, it also blocks the Oozie 
servers from talking to each other, namely for log streaming, the 
sharelibupdate command, and collating instrumentation/metrics (OOZIE-1676).  

Ultimately, what we need to do is allow Oozie to use both 
{{HTTP/}} instead of {{HTTP/}} at the 
same time so that clients (including Oozie servers, users, Web UI, etc) can 
talk to Oozie both through the load balancer and directly.  If my understanding 
of HADOOP-10158 is correct, HADOOP-10158 adds this ability.  For this JIRA, we 
should update Oozie to take advantage of HADOOP-10158.  



--
This message was sent by Atlassian JIRA
(v6.2#6252)

[jira] Subscription: Oozie Patch Available

[jira]

Issue Subscription
Filter: Oozie Patch Available (39 issues)

Subscriber: ooziedaily

Key Summary
OOZIE-1860  Oozie job mapper launch fails due to null value returned from 
action file
https://issues.apache.org/jira/browse/OOZIE-1860
OOZIE-1855  TestPriorityDelayQueue#testPoll failed intermittently in Jenkins
https://issues.apache.org/jira/browse/OOZIE-1855
OOZIE-1830  Change hadoop-1 profile to use 1.2.1
https://issues.apache.org/jira/browse/OOZIE-1830
OOZIE-1829  URIHandlerService doesn't support URI schemes with query strings 
but no path segment
https://issues.apache.org/jira/browse/OOZIE-1829
OOZIE-1828  Introduce counters JobStatus terminal states metrics
https://issues.apache.org/jira/browse/OOZIE-1828
OOZIE-1821  Oozie java action fails due to AlreadyBeingCreatedException
https://issues.apache.org/jira/browse/OOZIE-1821
OOZIE-1817  Oozie timers are not biased
https://issues.apache.org/jira/browse/OOZIE-1817
OOZIE-1816  LogInfo uses action name instead of id
https://issues.apache.org/jira/browse/OOZIE-1816
OOZIE-1813  Add service to report/kill rogue bundles and coordinator jobs
https://issues.apache.org/jira/browse/OOZIE-1813
OOZIE-1812  Bundle status is always in RUNNING if one of the action status is 
in PREP
https://issues.apache.org/jira/browse/OOZIE-1812
OOZIE-1807  Make bundle change command synchronous
https://issues.apache.org/jira/browse/OOZIE-1807
OOZIE-1804  Improve documentation for Coordinator Specification
https://issues.apache.org/jira/browse/OOZIE-1804
OOZIE-1803  Improvement in Purge service
https://issues.apache.org/jira/browse/OOZIE-1803
OOZIE-1802  Support workflow action log
https://issues.apache.org/jira/browse/OOZIE-1802
OOZIE-1793  Improve find bugs reporting for Oozie
https://issues.apache.org/jira/browse/OOZIE-1793
OOZIE-1782  Workflow path not found is thrown as SC_UNAUTHORIZED
https://issues.apache.org/jira/browse/OOZIE-1782
OOZIE-1741  Add new coord EL function to get input partitions value string
https://issues.apache.org/jira/browse/OOZIE-1741
OOZIE-1724  Make it easier to specify the HCat hive-site.xml for the Oozie 
Server
https://issues.apache.org/jira/browse/OOZIE-1724
OOZIE-1705  Enable gc logs and print thread id in logs
https://issues.apache.org/jira/browse/OOZIE-1705
OOZIE-1688  New configuration to specify server-server authentication type.
https://issues.apache.org/jira/browse/OOZIE-1688
OOZIE-1686  Typo in DG_CommandLineTool
https://issues.apache.org/jira/browse/OOZIE-1686
OOZIE-1685  Oozie doesn’t process correctly workflows with a non-default name 
node
https://issues.apache.org/jira/browse/OOZIE-1685
OOZIE-1659  oozie-site is missing email-action-0.2 schema
https://issues.apache.org/jira/browse/OOZIE-1659
OOZIE-1654  Fix typo (inteval to interval)
https://issues.apache.org/jira/browse/OOZIE-1654
OOZIE-1638  Action retry does not use default retry max count.
https://issues.apache.org/jira/browse/OOZIE-1638
OOZIE-1636  OOZIE_SYS table engine should be innodb
https://issues.apache.org/jira/browse/OOZIE-1636
OOZIE-1624  Exclusion pattern for sharelib.
https://issues.apache.org/jira/browse/OOZIE-1624
OOZIE-1602  pom.xml should take use of the profiles when building hadooplibs
https://issues.apache.org/jira/browse/OOZIE-1602
OOZIE-1586  upgrade oozie to hive 13.1 (including hcatalog)
https://issues.apache.org/jira/browse/OOZIE-1586
OOZIE-1579  Add basic HTTP auth to Oozie CLI
https://issues.apache.org/jira/browse/OOZIE-1579
OOZIE-1567  Provide a wait tool in Oozie
https://issues.apache.org/jira/browse/OOZIE-1567
OOZIE-1533  Coordinator action materialization is too slow due to coarse job 
level locks
https://issues.apache.org/jira/browse/OOZIE-1533
OOZIE-1532  Purging should remove completed children job for long running 
coordinator jobs
https://issues.apache.org/jira/browse/OOZIE-1532
OOZIE-1376  Extending Oozie ACLs like admin groups and proxy users to support 
both groups and users
https://issues.apache.org/jira/browse/OOZIE-1376
OOZIE-1369  OozieDBCLI code should not hardcode the Oozie table filenames
https://issues.apache.org/jira/browse/OOZIE-1369
OOZIE-1232  GroupsService should be able to reference Hadoop configurations in 
Hadoop configuration folder (such as /etc/hadoop/conf)
https://issues.apache.org/jira/browse/OOZIE-1232
OOZIE-1074  workflowgenerator does not package a tar.gz file though it seems 
creating one is the intention
https://issues.apache.org/jira/browse/OOZIE-1074
OOZIE-891   Add pagination for all popup panels in Oozie Web UI
https://issues.apache.org/jira/browse/OOZIE-891
OOZIE-

Re: Review Request 17720: OOZIE-1678 HA support for SLA

[Ryota Egashira]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/17720/
---

(Updated May 31, 2014, 12:26 a.m.)


Review request for oozie.


Changes
---

fixed review comments


Bugs: OOZIE-1678
https://issues.apache.org/jira/browse/OOZIE-1678


Repository: oozie-git


Description
---

https://issues.apache.org/jira/browse/OOZIE-1678


Diffs (updated)
-

  core/src/main/java/org/apache/oozie/command/coord/CoordChangeXCommand.java 
ea7df17 
  
core/src/main/java/org/apache/oozie/executor/jpa/BundleActionQueryExecutor.java 
d2331e8 
  core/src/main/java/org/apache/oozie/executor/jpa/BundleJobQueryExecutor.java 
319aea0 
  
core/src/main/java/org/apache/oozie/executor/jpa/CoordActionQueryExecutor.java 
f5304ca 
  core/src/main/java/org/apache/oozie/executor/jpa/CoordJobQueryExecutor.java 
1a6ded7 
  core/src/main/java/org/apache/oozie/executor/jpa/QueryExecutor.java 536743b 
  
core/src/main/java/org/apache/oozie/executor/jpa/SLARegistrationQueryExecutor.java
 e3b115f 
  core/src/main/java/org/apache/oozie/executor/jpa/SLASummaryQueryExecutor.java 
79d11ed 
  
core/src/main/java/org/apache/oozie/executor/jpa/WorkflowActionQueryExecutor.java
 9156a27 
  
core/src/main/java/org/apache/oozie/executor/jpa/WorkflowJobQueryExecutor.java 
e7d42e9 
  
core/src/main/java/org/apache/oozie/executor/jpa/sla/SLARegistrationGetJPAExecutor.java
 7b4e177 
  
core/src/main/java/org/apache/oozie/executor/jpa/sla/SLARegistrationGetOnRestartJPAExecutor.java
 1510daf 
  
core/src/main/java/org/apache/oozie/executor/jpa/sla/SLASummaryGetJPAExecutor.java
 1f7cb4d 
  core/src/main/java/org/apache/oozie/service/JPAService.java 1b8f53e 
  core/src/main/java/org/apache/oozie/service/JobsConcurrencyService.java 
27c97e6 
  core/src/main/java/org/apache/oozie/service/ZKJobsConcurrencyService.java 
611b74c 
  core/src/main/java/org/apache/oozie/sla/SLACalcStatus.java ea53712 
  core/src/main/java/org/apache/oozie/sla/SLACalculatorMemory.java 618d899 
  core/src/main/java/org/apache/oozie/sla/SLAOperations.java b0da3dc 
  core/src/main/java/org/apache/oozie/sla/SLASummaryBean.java 0a70326 
  core/src/main/java/org/apache/oozie/sla/service/SLAService.java 7fcb334 
  
core/src/test/java/org/apache/oozie/command/coord/TestCoordChangeXCommand.java 
bc24235 
  
core/src/test/java/org/apache/oozie/executor/jpa/TestSLARegistrationQueryExecutor.java
 00fb677 
  
core/src/test/java/org/apache/oozie/executor/jpa/TestSLASummaryQueryExecutor.java
 2e170a4 
  
core/src/test/java/org/apache/oozie/executor/jpa/TestWorkflowJobsCountNotForPurgeFromWorkflowParentIdJPAExecutor.java
 ea36720 
  core/src/test/java/org/apache/oozie/service/TestHASLAService.java 
PRE-CREATION 
  core/src/test/java/org/apache/oozie/sla/TestSLACalculationJPAExecutor.java 
9270aa2 
  core/src/test/java/org/apache/oozie/sla/TestSLACalculatorMemory.java 9a16722 
  core/src/test/java/org/apache/oozie/sla/TestSLAEventGeneration.java cb4e434 
  core/src/test/java/org/apache/oozie/sla/TestSLAJobEventListener.java 3ce86ab 
  
core/src/test/java/org/apache/oozie/sla/TestSLARegistrationGetJPAExecutor.java 
40376a5 
  
core/src/test/java/org/apache/oozie/sla/TestSLARegistrationGetRecordsOnRestartJPAExecutor.java
 cd2fbae 
  core/src/test/java/org/apache/oozie/sla/TestSLAService.java 291d850 
  
core/src/test/java/org/apache/oozie/sla/TestSLASummaryGetOnRestartJPAExecutor.java
 3a9e72e 
  core/src/test/java/org/apache/oozie/test/ZKXTestCase.java 7bebaf0 
  core/src/test/resources/coord-action-sla.xml 4893c61 

Diff: https://reviews.apache.org/r/17720/diff/


Testing
---


Thanks,

Ryota Egashira

[jira] [Commented] (OOZIE-1812) Bundle status is always in RUNNING if one of the action status is in PREP

[Hadoop QA (JIRA)]

[ 
https://issues.apache.org/jira/browse/OOZIE-1812?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14014404#comment-14014404
 ] 

Hadoop QA commented on OOZIE-1812:
--

Testing JIRA OOZIE-1812

Cleaning local git workspace



{color:green}+1 PATCH_APPLIES{color}
{color:green}+1 CLEAN{color}
{color:green}+1 RAW_PATCH_ANALYSIS{color}
.{color:green}+1{color} the patch does not introduce any @author tags
.{color:green}+1{color} the patch does not introduce any tabs
.{color:green}+1{color} the patch does not introduce any trailing spaces
.{color:green}+1{color} the patch does not introduce any line longer than 
132
.{color:green}+1{color} the patch does adds/modifies 1 testcase(s)
{color:green}+1 RAT{color}
.{color:green}+1{color} the patch does not seem to introduce new RAT 
warnings
{color:green}+1 JAVADOC{color}
.{color:green}+1{color} the patch does not seem to introduce new Javadoc 
warnings
{color:green}+1 COMPILE{color}
.{color:green}+1{color} HEAD compiles
.{color:green}+1{color} patch compiles
.{color:green}+1{color} the patch does not seem to introduce new javac 
warnings
{color:green}+1 BACKWARDS_COMPATIBILITY{color}
.{color:green}+1{color} the patch does not change any JPA 
Entity/Colum/Basic/Lob/Transient annotations
.{color:green}+1{color} the patch does not modify JPA files
{color:red}-1 TESTS{color}
.Tests run: 1459
.Tests failed: 3
.Tests errors: 1

.The patch failed the following testcases:

.  
testTimeoutWithException(org.apache.oozie.command.coord.TestCoordActionInputCheckXCommandNonUTC)
.  
testBundleEngineKill(org.apache.oozie.servlet.TestV1JobServletBundleEngine)
.  
testActionInputCheckLatestActionCreationTime(org.apache.oozie.command.coord.TestCoordActionInputCheckXCommand)

{color:green}+1 DISTRO{color}
.{color:green}+1{color} distro tarball builds with the patch 


{color:red}*-1 Overall result, please check the reported -1(s)*{color}


The full output of the test-patch run is available at

.   https://builds.apache.org/job/oozie-trunk-precommit-build/1279/

> Bundle status is always in RUNNING if one of the action status is in PREP
> -
>
> Key: OOZIE-1812
> URL: https://issues.apache.org/jira/browse/OOZIE-1812
> Project: Oozie
>  Issue Type: Bug
>Affects Versions: trunk, 4.0.1
>Reporter: Rohini Palaniswamy
>Assignee: Rohini Palaniswamy
> Attachments: OOZIE-1812-V1.patch
>
>
> Even if there are FAILED or KILLED coordinator jobs, bundle action status 
> remains in RUNNING because the checkPrepStatus either does PREP or RUNNING 
> and does not take terminal states into account.



--
This message was sent by Atlassian JIRA
(v6.2#6252)

Build failed in Jenkins: oozie-trunk-precommit-build #1279

[Apache Jenkins Server]

See 

Changes:

[rkanter] OOZIE-1319 LAST_ONLY in execution control for coordinator job still 
runs all the actions (rkanter)

--
[...truncated 9706 lines...]
[INFO] share/lib already added, skipping
[INFO]  already added, skipping
[INFO] share already added, skipping
[INFO] share/lib already added, skipping
[INFO]  already added, skipping
[INFO] share already added, skipping
[INFO] share/lib already added, skipping
[INFO]  already added, skipping
[INFO] share already added, skipping
[INFO] share/lib already added, skipping
[INFO]  already added, skipping
[INFO] share already added, skipping
[INFO] share/lib already added, skipping
[INFO]  already added, skipping
[INFO] share already added, skipping
[INFO] share/lib already added, skipping
[INFO] 
[INFO] 
[INFO] Building Apache Oozie Tools 4.1.0-SNAPSHOT
[INFO] 
[WARNING] Failure to transfer asm:asm/maven-metadata.xml from 
file:../../local.repository/trunk was cached in the local repository, 
resolution will not be reattempted until the update interval of 
local.repository has elapsed or updates are forced. Original error: Could not 
transfer metadata asm:asm/maven-metadata.xml from/to local.repository 
(file:../../local.repository/trunk): No connector available to access 
repository local.repository (file:../../local.repository/trunk) of type legacy 
using the available factories WagonRepositoryConnectorFactory
[WARNING] Failure to transfer asm:asm/maven-metadata.xml from 
file:../../local.repository/trunk was cached in the local repository, 
resolution will not be reattempted until the update interval of 
local.repository has elapsed or updates are forced. Original error: Could not 
transfer metadata asm:asm/maven-metadata.xml from/to local.repository 
(file:../../local.repository/trunk): No connector available to access 
repository local.repository (file:../../local.repository/trunk) of type legacy 
using the available factories WagonRepositoryConnectorFactory
[INFO] 
[INFO] --- maven-resources-plugin:2.5:resources (default-resources) @ 
oozie-tools ---
[debug] execute contextualize
[INFO] Using 'UTF-8' encoding to copy filtered resources.
[INFO] skip non existing resourceDirectory 

[INFO] 
[INFO] --- maven-compiler-plugin:2.3.2:compile (default-compile) @ oozie-tools 
---
[INFO] Nothing to compile - all classes are up to date
[INFO] 
[INFO] --- maven-resources-plugin:2.5:testResources (default-testResources) @ 
oozie-tools ---
[debug] execute contextualize
[INFO] Using 'UTF-8' encoding to copy filtered resources.
[INFO] Copying 2 resources
[INFO] 
[INFO] --- maven-compiler-plugin:2.3.2:testCompile (default-testCompile) @ 
oozie-tools ---
[INFO] Nothing to compile - all classes are up to date
[INFO] 
[INFO] --- maven-surefire-plugin:2.12:test (default-test) @ oozie-tools ---
[INFO] Tests are skipped.
[INFO] 
[INFO] --- maven-jar-plugin:2.3.1:jar (default-jar) @ oozie-tools ---
[INFO] Building jar: 

[INFO] 
[INFO] --- maven-assembly-plugin:2.2.1:single (default-cli) @ oozie-tools ---
[INFO] Reading assembly descriptor: ../src/main/assemblies/tools.xml
[WARNING] The following patterns were never triggered in this artifact 
exclusion filter:
o  '*:*:pom:*'

[INFO] Copying files to 

[WARNING] Assembly file: 

 is not a regular file (it may be a directory). It cannot be attached to the 
project build for installation or deployment.
[INFO] 
[INFO] 
[INFO] Building Apache Oozie MiniOozie 4.1.0-SNAPSHOT
[INFO] 
[INFO] 
[INFO] --- maven-resources-plugin:2.5:resources (default-resources) @ 
oozie-mini ---
[debug] execute contextualize
[INFO] Using 'UTF-8' encoding to copy filtered resources.
[INFO] skip non existing resourceDirectory 

[INFO] 
[INFO] --- maven-compiler-plugin:2.3.2:compile (default-compile) @ oozie-mini 
---
[INFO] No sources to compile
[INFO] 
[INFO] --- maven-resources-plugin:2.5:testResources (default-testResources) @ 
oozie-mini ---
[debug] execute contextualize
[INFO] Using 'UTF-8' encoding to copy filtered resourc

Re: Review Request 21278: OOZIE-1817: Oozie timers are not biased

[Robert Kanter]

> On May 30, 2014, 9 p.m., Mona Chitnis wrote:
> > core/src/main/java/org/apache/oozie/servlet/V2AdminServlet.java, line 53
> > 
> >
> > where is the disabling of instrumentation endpoint taking place?

In V2AdminServlet, if (metricsInstrumentation == null) sendMetricsResponse(...) 
will response when a SC_SERVICE_UNAVAILABLE; sentInstrumentationReponse(...) 
will do the opposite.


> On May 30, 2014, 9 p.m., Mona Chitnis wrote:
> > core/src/main/java/org/apache/oozie/util/MetricsInstrumentation.java, line 
> > 165
> > 
> >
> > why use lock when guages is a concurrenthashmap?

Besides adding to the gauges hashmap, addVariable(...) also adds and/or removes 
the gauge to/from the metricsRegistry so that entire set of operations should 
be thread safe


> On May 30, 2014, 9 p.m., Mona Chitnis wrote:
> > core/src/main/java/org/apache/oozie/util/MetricsInstrumentation.java, line 
> > 221
> > 
> >
> > same as above

Same as before, but for histograms instead of gauges.


- Robert


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/21278/#review44419
---


On May 22, 2014, 11:27 p.m., Robert Kanter wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/21278/
> ---
> 
> (Updated May 22, 2014, 11:27 p.m.)
> 
> 
> Review request for oozie.
> 
> 
> Bugs: OOZIE-1817
> https://issues.apache.org/jira/browse/OOZIE-1817
> 
> 
> Repository: oozie-git
> 
> 
> Description
> ---
> 
> See 
> https://issues.apache.org/jira/browse/OOZIE-1817?focusedCommentId=13993838&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-13993838
> 
> 
> Diffs
> -
> 
>   client/src/main/java/org/apache/oozie/client/rest/RestConstants.java 
> 97fd465 
>   core/pom.xml c935dd7 
>   core/src/main/java/org/apache/oozie/command/Command.java d3f1011 
>   core/src/main/java/org/apache/oozie/command/XCommand.java b712ce0 
>   
> core/src/main/java/org/apache/oozie/command/coord/CoordActionInputCheckXCommand.java
>  e8667c1 
>   
> core/src/main/java/org/apache/oozie/command/coord/CoordActionMaterializeCommand.java
>  6962fb2 
>   
> core/src/main/java/org/apache/oozie/command/coord/CoordMaterializeTransitionXCommand.java
>  57cbb34 
>   core/src/main/java/org/apache/oozie/service/InstrumentationService.java 
> 80437b1 
>   
> core/src/main/java/org/apache/oozie/service/MetricsInstrumentationService.java
>  PRE-CREATION 
>   core/src/main/java/org/apache/oozie/service/Services.java 5feac7b 
>   core/src/main/java/org/apache/oozie/service/XLogService.java 403a089 
>   core/src/main/java/org/apache/oozie/servlet/BaseAdminServlet.java 29d7bd6 
>   core/src/main/java/org/apache/oozie/servlet/JsonRestServlet.java 5c05acd 
>   core/src/main/java/org/apache/oozie/servlet/V0AdminServlet.java 97bd81c 
>   core/src/main/java/org/apache/oozie/servlet/V1AdminServlet.java a47f737 
>   core/src/main/java/org/apache/oozie/servlet/V2AdminServlet.java 002a367 
>   core/src/main/java/org/apache/oozie/util/MetricsInstrumentation.java 
> PRE-CREATION 
>   core/src/main/java/org/apache/oozie/util/XLog.java 21e00c0 
>   core/src/main/resources/oozie-default.xml 9788daf 
>   core/src/test/java/org/apache/oozie/service/TestInstrumentationService.java 
> b5206cf 
>   
> core/src/test/java/org/apache/oozie/service/TestMetricsInstrumentationService.java
>  PRE-CREATION 
>   core/src/test/java/org/apache/oozie/util/TestInstrumentation.java 6040b47 
>   core/src/test/java/org/apache/oozie/util/TestMetricsInstrumentation.java 
> PRE-CREATION 
>   docs/src/site/twiki/AG_Install.twiki e343d7e 
>   docs/src/site/twiki/WebServicesAPI.twiki 5769924 
>   pom.xml b5e0e4e 
>   webapp/src/main/webapp/index.jsp 3c7ffe5 
>   webapp/src/main/webapp/oozie-console.js 764d888 
> 
> Diff: https://reviews.apache.org/r/21278/diff/
> 
> 
> Testing
> ---
> 
> Unit tests, also verified in a cluster
> 
> 
> Thanks,
> 
> Robert Kanter
> 
>

[jira] [Commented] (OOZIE-1848) Pig actions fail due to missing joda-time jar from pig sharelib

[Bowen Zhang (JIRA)]

[ 
https://issues.apache.org/jira/browse/OOZIE-1848?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14014313#comment-14014313
 ] 

Bowen Zhang commented on OOZIE-1848:


Committed to trunk.

> Pig actions fail due to missing joda-time jar from pig sharelib
> ---
>
> Key: OOZIE-1848
> URL: https://issues.apache.org/jira/browse/OOZIE-1848
> Project: Oozie
>  Issue Type: Bug
>Affects Versions: trunk
>Reporter: Robert Kanter
>Assignee: Bowen Zhang
>Priority: Blocker
>  Labels: newbie
> Attachments: oozie-1848.patch
>
>
> When I try to run the pig example, I get this Exception in the launcher log:
> {noformat}
> java.lang.NoClassDefFoundError: org/joda/time/ReadableInstant
>   at 
> org.apache.pig.newplan.logical.relational.LogicalSchema$LogicalFieldSchema.merge(LogicalSchema.java:404)
>   at 
> org.apache.pig.newplan.logical.relational.LogicalSchema.merge(LogicalSchema.java:768)
>   at 
> org.apache.pig.newplan.logical.relational.LOGenerate.getSchema(LOGenerate.java:158)
>   at 
> org.apache.pig.newplan.logical.optimizer.SchemaResetter.visit(SchemaResetter.java:123)
>   at 
> org.apache.pig.newplan.logical.relational.LOGenerate.accept(LOGenerate.java:245)
>   at 
> org.apache.pig.newplan.DependencyOrderWalker.walk(DependencyOrderWalker.java:75)
>   at 
> org.apache.pig.newplan.logical.optimizer.SchemaResetter.visit(SchemaResetter.java:114)
>   at 
> org.apache.pig.parser.LogicalPlanBuilder.buildForeachOp(LogicalPlanBuilder.java:1015)
>   at 
> org.apache.pig.parser.LogicalPlanGenerator.foreach_clause(LogicalPlanGenerator.java:15870)
>   at 
> org.apache.pig.parser.LogicalPlanGenerator.op_clause(LogicalPlanGenerator.java:1933)
>   at 
> org.apache.pig.parser.LogicalPlanGenerator.general_statement(LogicalPlanGenerator.java:1102)
>   at 
> org.apache.pig.parser.LogicalPlanGenerator.statement(LogicalPlanGenerator.java:560)
>   at 
> org.apache.pig.parser.LogicalPlanGenerator.query(LogicalPlanGenerator.java:421)
>   at 
> org.apache.pig.parser.QueryParserDriver.parse(QueryParserDriver.java:188)
>   at org.apache.pig.PigServer$Graph.parseQuery(PigServer.java:1678)
>   at org.apache.pig.PigServer$Graph.access$000(PigServer.java:1411)
>   at org.apache.pig.PigServer.parseAndBuild(PigServer.java:344)
>   at org.apache.pig.PigServer.executeBatch(PigServer.java:369)
>   at org.apache.pig.PigServer.executeBatch(PigServer.java:355)
>   at 
> org.apache.pig.tools.grunt.GruntParser.executeBatch(GruntParser.java:140)
>   at 
> org.apache.pig.tools.grunt.GruntParser.parseStopOnError(GruntParser.java:202)
>   at 
> org.apache.pig.tools.grunt.GruntParser.parseStopOnError(GruntParser.java:173)
>   at org.apache.pig.tools.grunt.Grunt.exec(Grunt.java:84)
>   at org.apache.pig.Main.run(Main.java:478)
>   at org.apache.pig.PigRunner.run(PigRunner.java:49)
>   at org.apache.oozie.action.hadoop.PigMain.runPigJob(PigMain.java:288)
>   at org.apache.oozie.action.hadoop.PigMain.run(PigMain.java:228)
>   at org.apache.oozie.action.hadoop.LauncherMain.run(LauncherMain.java:39)
>   at org.apache.oozie.action.hadoop.PigMain.main(PigMain.java:77)
>   at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>   at 
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
>   at 
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>   at java.lang.reflect.Method.invoke(Method.java:606)
>   at 
> org.apache.oozie.action.hadoop.LauncherMapper.map(LauncherMapper.java:226)
>   at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:50)
>   at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:430)
>   at org.apache.hadoop.mapred.MapTask.run(MapTask.java:366)
>   at org.apache.hadoop.mapred.Child$4.run(Child.java:255)
>   at java.security.AccessController.doPrivileged(Native Method)
>   at javax.security.auth.Subject.doAs(Subject.java:415)
>   at 
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1190)
>   at org.apache.hadoop.mapred.Child.main(Child.java:249)
> Caused by: java.lang.ClassNotFoundException: org.joda.time.ReadableInstant
>   at java.net.URLClassLoader$1.run(URLClassLoader.java:366)
>   at java.net.URLClassLoader$1.run(URLClassLoader.java:355)
>   at java.security.AccessController.doPrivileged(Native Method)
>   at java.net.URLClassLoader.findClass(URLClassLoader.java:354)
>   at java.lang.ClassLoader.loadClass(ClassLoader.java:425)
>   at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:308)
>   at java.lang.ClassLoader.loadClass(ClassLoader.java:358)
>   ... 42 more
> {noformat}
>

[jira] [Commented] (OOZIE-1864) Improve chid job id aggregation logic

[Robert Kanter (JIRA)]

[ 
https://issues.apache.org/jira/browse/OOZIE-1864?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14014311#comment-14014311
 ] 

Robert Kanter commented on OOZIE-1864:
--

Another way to get the child jobs is to use the YARN tags stuff added in 
OOZIE-1722.  The Oozie server can simply ask YARN for the jobs with the tag 
that the Oozie server already knows.  Though this only works for Hadoop 2.4.0+

> Improve chid job id aggregation logic
> -
>
> Key: OOZIE-1864
> URL: https://issues.apache.org/jira/browse/OOZIE-1864
> Project: Oozie
>  Issue Type: Bug
>Reporter: Purshotam Shah
>
> Improve chid job id aggregation logic
> Current chid job id aggregation logic
> Once launcher job complete submitting child job (jobs in case on pig), it 
> writes jobID to file.
> From Oozie server side, we collect childID in two ways
> 1. As soon as we submit launcher jobs, we check if launcher job terminated or 
> not. If it's terminated, we read child-id from file and populated to DB.  And 
> once kill command is issued we kill all child jobs.
> 2. We have a timer task (ActionCheckerService) which keeps on checking the 
> status of all running actions and if launcher job is terminated, it's update 
> the DB with childIDs.
> Jobend notification is rejected if action is not running.  
> Assume that launcher is killed after it has submitted child job.
> Child job will never be killed.
> To fix this, we should do following things.
> 1. If oozie receives job end notification and if launcher job is killed, 
> collect all child job and kill them if they are not killed.
> 2. Have a better way logic to collect child job id. Launcher job can call 
> callbackServlet ( may be periodically) to update child job ids. This could be 
> useful in pig jobs. In current  scenario we report child jobs job only when 
> launcher job completes.
>  



--
This message was sent by Atlassian JIRA
(v6.2#6252)

[jira] [Updated] (OOZIE-1812) Bundle status is always in RUNNING if one of the action status is in PREP

[Purshotam Shah (JIRA)]

 [ 
https://issues.apache.org/jira/browse/OOZIE-1812?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Purshotam Shah updated OOZIE-1812:
--

Attachment: OOZIE-1812-V1.patch

> Bundle status is always in RUNNING if one of the action status is in PREP
> -
>
> Key: OOZIE-1812
> URL: https://issues.apache.org/jira/browse/OOZIE-1812
> Project: Oozie
>  Issue Type: Bug
>Affects Versions: 4.0.1
>Reporter: Rohini Palaniswamy
>Assignee: Rohini Palaniswamy
> Attachments: OOZIE-1812-V1.patch
>
>
> Even if there are FAILED or KILLED coordinator jobs, bundle action status 
> remains in RUNNING because the checkPrepStatus either does PREP or RUNNING 
> and does not take terminal states into account.



--
This message was sent by Atlassian JIRA
(v6.2#6252)

[jira] [Commented] (OOZIE-1319) "LAST_ONLY" in execution control for coordinator job still runs all the actions

[Bowen Zhang (JIRA)]

[ 
https://issues.apache.org/jira/browse/OOZIE-1319?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14014248#comment-14014248
 ] 

Bowen Zhang commented on OOZIE-1319:


my bad, didn't see it.

> "LAST_ONLY" in execution control for coordinator job still runs all the 
> actions
> ---
>
> Key: OOZIE-1319
> URL: https://issues.apache.org/jira/browse/OOZIE-1319
> Project: Oozie
>  Issue Type: Bug
>Reporter: Bowen Zhang
>Assignee: Robert Kanter
> Attachments: OOZIE-1319.patch, OOZIE-1319.patch, OOZIE-1319.patch, 
> OOZIE-1319.patch, OOZIE-1319.patch, OOZIE-1319.patch, OOZIE-1319.patch, 
> oozie-1319.patch
>
>
> In execute() of CoordJobGetReadyActionsJPAExecutor.java, once we retrieve the 
> top item from a "LIFO" query result, we do not discard or delete the 
> remaining items from the result list. As a result, the next time execute() is 
> invoked, we will be retrieving the next item in line. Consequently, LAST_ONLY 
> strategy will also execute all ready actions for a given coordinator job, 
> making it no different than LIFO.



--
This message was sent by Atlassian JIRA
(v6.2#6252)

[jira] [Commented] (OOZIE-1319) "LAST_ONLY" in execution control for coordinator job still runs all the actions

[Robert Kanter (JIRA)]

[ 
https://issues.apache.org/jira/browse/OOZIE-1319?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14014238#comment-14014238
 ] 

Robert Kanter commented on OOZIE-1319:
--

Thanks for the reviews and catching all of those edge cases.  I'm glad we can 
finally get this in :)

> "LAST_ONLY" in execution control for coordinator job still runs all the 
> actions
> ---
>
> Key: OOZIE-1319
> URL: https://issues.apache.org/jira/browse/OOZIE-1319
> Project: Oozie
>  Issue Type: Bug
>Reporter: Bowen Zhang
>Assignee: Robert Kanter
> Attachments: OOZIE-1319.patch, OOZIE-1319.patch, OOZIE-1319.patch, 
> OOZIE-1319.patch, OOZIE-1319.patch, OOZIE-1319.patch, OOZIE-1319.patch, 
> oozie-1319.patch
>
>
> In execute() of CoordJobGetReadyActionsJPAExecutor.java, once we retrieve the 
> top item from a "LIFO" query result, we do not discard or delete the 
> remaining items from the result list. As a result, the next time execute() is 
> invoked, we will be retrieving the next item in line. Consequently, LAST_ONLY 
> strategy will also execute all ready actions for a given coordinator job, 
> making it no different than LIFO.



--
This message was sent by Atlassian JIRA
(v6.2#6252)

Build failed in Jenkins: oozie-trunk-precommit-build #1278

[Apache Jenkins Server]

See 

Changes:

[bowenzhangusa] OOZIE-1862 Add hadoop token file location for Hive/Tez jobs 
(venkatnrangan via bzhang)

--
[...truncated 9612 lines...]
[INFO]  already added, skipping
[INFO] share already added, skipping
[INFO] share/lib already added, skipping
[INFO]  already added, skipping
[INFO] share already added, skipping
[INFO] share/lib already added, skipping
[INFO]  already added, skipping
[INFO] share already added, skipping
[INFO] share/lib already added, skipping
[INFO]  already added, skipping
[INFO] share already added, skipping
[INFO] share/lib already added, skipping
[INFO]  already added, skipping
[INFO] share already added, skipping
[INFO] share/lib already added, skipping
[INFO] 
[INFO] 
[INFO] Building Apache Oozie Tools 4.1.0-SNAPSHOT
[INFO] 
[WARNING] Failure to transfer asm:asm/maven-metadata.xml from 
file:../../local.repository/trunk was cached in the local repository, 
resolution will not be reattempted until the update interval of 
local.repository has elapsed or updates are forced. Original error: Could not 
transfer metadata asm:asm/maven-metadata.xml from/to local.repository 
(file:../../local.repository/trunk): No connector available to access 
repository local.repository (file:../../local.repository/trunk) of type legacy 
using the available factories WagonRepositoryConnectorFactory
[WARNING] Failure to transfer asm:asm/maven-metadata.xml from 
file:../../local.repository/trunk was cached in the local repository, 
resolution will not be reattempted until the update interval of 
local.repository has elapsed or updates are forced. Original error: Could not 
transfer metadata asm:asm/maven-metadata.xml from/to local.repository 
(file:../../local.repository/trunk): No connector available to access 
repository local.repository (file:../../local.repository/trunk) of type legacy 
using the available factories WagonRepositoryConnectorFactory
[INFO] 
[INFO] --- maven-resources-plugin:2.5:resources (default-resources) @ 
oozie-tools ---
[debug] execute contextualize
[INFO] Using 'UTF-8' encoding to copy filtered resources.
[INFO] skip non existing resourceDirectory 

[INFO] 
[INFO] --- maven-compiler-plugin:2.3.2:compile (default-compile) @ oozie-tools 
---
[INFO] Nothing to compile - all classes are up to date
[INFO] 
[INFO] --- maven-resources-plugin:2.5:testResources (default-testResources) @ 
oozie-tools ---
[debug] execute contextualize
[INFO] Using 'UTF-8' encoding to copy filtered resources.
[INFO] Copying 2 resources
[INFO] 
[INFO] --- maven-compiler-plugin:2.3.2:testCompile (default-testCompile) @ 
oozie-tools ---
[INFO] Nothing to compile - all classes are up to date
[INFO] 
[INFO] --- maven-surefire-plugin:2.12:test (default-test) @ oozie-tools ---
[INFO] Tests are skipped.
[INFO] 
[INFO] --- maven-jar-plugin:2.3.1:jar (default-jar) @ oozie-tools ---
[INFO] Building jar: 

[INFO] 
[INFO] --- maven-assembly-plugin:2.2.1:single (default-cli) @ oozie-tools ---
[INFO] Reading assembly descriptor: ../src/main/assemblies/tools.xml
[WARNING] The following patterns were never triggered in this artifact 
exclusion filter:
o  '*:*:pom:*'

[INFO] Copying files to 

[WARNING] Assembly file: 

 is not a regular file (it may be a directory). It cannot be attached to the 
project build for installation or deployment.
[INFO] 
[INFO] 
[INFO] Building Apache Oozie MiniOozie 4.1.0-SNAPSHOT
[INFO] 
[INFO] 
[INFO] --- maven-resources-plugin:2.5:resources (default-resources) @ 
oozie-mini ---
[debug] execute contextualize
[INFO] Using 'UTF-8' encoding to copy filtered resources.
[INFO] skip non existing resourceDirectory 

[INFO] 
[INFO] --- maven-compiler-plugin:2.3.2:compile (default-compile) @ oozie-mini 
---
[INFO] No sources to compile
[INFO] 
[INFO] --- maven-resources-plugin:2.5:testResources (default-testResources) @ 
oozie-mini ---
[debug] execute contextualize
[INFO] Using 'UTF-8' encoding to copy filtered resources.
[INFO] Copying 3 resources
[INFO] 
[INFO] -

[jira] [Commented] (OOZIE-1813) Add service to report/kill rogue bundles and coordinator jobs

[Hadoop QA (JIRA)]

[ 
https://issues.apache.org/jira/browse/OOZIE-1813?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14014236#comment-14014236
 ] 

Hadoop QA commented on OOZIE-1813:
--

Testing JIRA OOZIE-1813

Cleaning local git workspace



{color:green}+1 PATCH_APPLIES{color}
{color:green}+1 CLEAN{color}
{color:red}-1 RAW_PATCH_ANALYSIS{color}
.{color:green}+1{color} the patch does not introduce any @author tags
.{color:green}+1{color} the patch does not introduce any tabs
.{color:green}+1{color} the patch does not introduce any trailing spaces
.{color:red}-1{color} the patch contains 2 line(s) longer than 132 
characters
.{color:green}+1{color} the patch does adds/modifies 1 testcase(s)
{color:green}+1 RAT{color}
.{color:green}+1{color} the patch does not seem to introduce new RAT 
warnings
{color:green}+1 JAVADOC{color}
.{color:green}+1{color} the patch does not seem to introduce new Javadoc 
warnings
{color:green}+1 COMPILE{color}
.{color:green}+1{color} HEAD compiles
.{color:green}+1{color} patch compiles
.{color:green}+1{color} the patch does not seem to introduce new javac 
warnings
{color:green}+1 BACKWARDS_COMPATIBILITY{color}
.{color:green}+1{color} the patch does not change any JPA 
Entity/Colum/Basic/Lob/Transient annotations
.{color:green}+1{color} the patch does not modify JPA files
{color:red}-1 TESTS{color}
.Tests run: 1454
.Tests failed: 4
.Tests errors: 0

.The patch failed the following testcases:

.  
testConcurrencyReachedAndChooseNextEligible(org.apache.oozie.service.TestCallableQueueService)
.  
testBundleEngineKill(org.apache.oozie.servlet.TestV1JobServletBundleEngine)
.  
testActionInputCheckLatestActionCreationTime(org.apache.oozie.command.coord.TestCoordActionInputCheckXCommand)
.  
testTimeOutWithUnresolvedMissingDependencies(org.apache.oozie.command.coord.TestCoordPushDependencyCheckXCommand)

{color:green}+1 DISTRO{color}
.{color:green}+1{color} distro tarball builds with the patch 


{color:red}*-1 Overall result, please check the reported -1(s)*{color}


The full output of the test-patch run is available at

.   https://builds.apache.org/job/oozie-trunk-precommit-build/1278/

> Add service to report/kill rogue bundles and coordinator jobs
> -
>
> Key: OOZIE-1813
> URL: https://issues.apache.org/jira/browse/OOZIE-1813
> Project: Oozie
>  Issue Type: Bug
>Reporter: Purshotam Shah
>Assignee: Purshotam Shah
> Attachments: OOZIE-1813-V2.patch, OOZIE-1813-V3.patch, 
> OOZIE-1813-V4.patch, OOZIE-1813-V5.patch, OOZIE-1813-V6.patch, 
> OOZIE-1813-V7.patch
>
>
> People leave their test coordinator and bundle jobs without ever killing them
> and they just eat up resources heavily. We should have a service which 
> periodically check for abandoned coords and report/kill them.
> We can add multiple logic to this like ( number of consecutive 
> failed/timedout action, total number of failed/timedout action). 
> To start with if number of coord action with failed/timedout status > defined 
> value, then coord is considered to be rogue.



--
This message was sent by Atlassian JIRA
(v6.2#6252)

[jira] [Commented] (OOZIE-1319) "LAST_ONLY" in execution control for coordinator job still runs all the actions

[Robert Kanter (JIRA)]

[ 
https://issues.apache.org/jira/browse/OOZIE-1319?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14014226#comment-14014226
 ] 

Robert Kanter commented on OOZIE-1319:
--

It already checks that:
{code:java}
// If the next nominal time is after the job's end time, then this is the last 
action, so return null
if (nextNominalTime.after(coordJob.getEndTime())) {
nextNominalTime = null;
}
return nextNominalTime;
{code}

> "LAST_ONLY" in execution control for coordinator job still runs all the 
> actions
> ---
>
> Key: OOZIE-1319
> URL: https://issues.apache.org/jira/browse/OOZIE-1319
> Project: Oozie
>  Issue Type: Bug
>Reporter: Bowen Zhang
>Assignee: Robert Kanter
> Attachments: OOZIE-1319.patch, OOZIE-1319.patch, OOZIE-1319.patch, 
> OOZIE-1319.patch, OOZIE-1319.patch, OOZIE-1319.patch, OOZIE-1319.patch, 
> oozie-1319.patch
>
>
> In execute() of CoordJobGetReadyActionsJPAExecutor.java, once we retrieve the 
> top item from a "LIFO" query result, we do not discard or delete the 
> remaining items from the result list. As a result, the next time execute() is 
> invoked, we will be retrieving the next item in line. Consequently, LAST_ONLY 
> strategy will also execute all ready actions for a given coordinator job, 
> making it no different than LIFO.



--
This message was sent by Atlassian JIRA
(v6.2#6252)

[jira] [Updated] (OOZIE-1864) Improve chid job id aggregation logic

[Purshotam Shah (JIRA)]

 [ 
https://issues.apache.org/jira/browse/OOZIE-1864?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Purshotam Shah updated OOZIE-1864:
--

Description: 
Improve chid job id aggregation logic


Current chid job id aggregation logic
Once launcher job complete submitting child job (jobs in case on pig), it 
writes jobID to file.

>From Oozie server side, we collect childID in two ways
1. As soon as we submit launcher jobs, we check if launcher job terminated or 
not. If it's terminated, we read child-id from file and populated to DB.  And 
once kill command is issued we kill all child jobs.
2. We have a timer task (ActionCheckerService) which keeps on checking the 
status of all running actions and if launcher job is terminated, it's update 
the DB with childIDs.

Jobend notification is rejected if action is not running.  

Assume that launcher is killed after it has submitted child job.
Child job will never be killed.


To fix this, we should do following things.

1. If oozie receives job end notification and if launcher job is killed, 
collect all child job and kill them if they are not killed.
2. Have a better way logic to collect child job id. Launcher job can call 
callbackServlet ( may be periodically) to update child job ids. This could be 
useful in pig jobs. In current  scenario we report child jobs job only when 
launcher job completes.
 

  was:
Current chid job id aggregation logic
Once launcher job complete submitting child job (jobs in case on pig), it
writes jobID to file.

>From Oozie server side, we collect childID in two ways
1. As soon as we submit launcher jobs, we check if launcher job terminated or
not. If it's terminated, we read child-id from file and populated to DB.  And
once kill command is issued we kill all child jobs.
2. We have a timer task (ActionCheckerService) which keeps on checking the
status of all running actions and if launcher job is terminated, it's update
the DB with childIDs.

Jobend notification is rejected if action is not running.  

Assume that launcher is killed after it has submitted child job.
Child job will never be killed.


To fix this, we should do following things.

1. If oozie receives job end notification and if launcher job is killed, collect
all child job and kill them if they are not killed.

2. Have a better way logic to collect child job id. Launcher job can call 
callbackServlet ( may be periodically) to
update child job ids. This could be useful in pig jobs. In current scenario we
report child jobs job only when launcher job completes.


> Improve chid job id aggregation logic
> -
>
> Key: OOZIE-1864
> URL: https://issues.apache.org/jira/browse/OOZIE-1864
> Project: Oozie
>  Issue Type: Bug
>Reporter: Purshotam Shah
>
> Improve chid job id aggregation logic
> Current chid job id aggregation logic
> Once launcher job complete submitting child job (jobs in case on pig), it 
> writes jobID to file.
> From Oozie server side, we collect childID in two ways
> 1. As soon as we submit launcher jobs, we check if launcher job terminated or 
> not. If it's terminated, we read child-id from file and populated to DB.  And 
> once kill command is issued we kill all child jobs.
> 2. We have a timer task (ActionCheckerService) which keeps on checking the 
> status of all running actions and if launcher job is terminated, it's update 
> the DB with childIDs.
> Jobend notification is rejected if action is not running.  
> Assume that launcher is killed after it has submitted child job.
> Child job will never be killed.
> To fix this, we should do following things.
> 1. If oozie receives job end notification and if launcher job is killed, 
> collect all child job and kill them if they are not killed.
> 2. Have a better way logic to collect child job id. Launcher job can call 
> callbackServlet ( may be periodically) to update child job ids. This could be 
> useful in pig jobs. In current  scenario we report child jobs job only when 
> launcher job completes.
>  



--
This message was sent by Atlassian JIRA
(v6.2#6252)

[jira] [Commented] (OOZIE-1319) "LAST_ONLY" in execution control for coordinator job still runs all the actions

[Bowen Zhang (JIRA)]

[ 
https://issues.apache.org/jira/browse/OOZIE-1319?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14014224#comment-14014224
 ] 

Bowen Zhang commented on OOZIE-1319:


one last issue. When computing nextNominalTime, you need to make sure 
nextNominalTime is < end time of the coord job. Otherwise, you may wrongly skip 
the last action of the coord job. After that, +1.

> "LAST_ONLY" in execution control for coordinator job still runs all the 
> actions
> ---
>
> Key: OOZIE-1319
> URL: https://issues.apache.org/jira/browse/OOZIE-1319
> Project: Oozie
>  Issue Type: Bug
>Reporter: Bowen Zhang
>Assignee: Robert Kanter
> Attachments: OOZIE-1319.patch, OOZIE-1319.patch, OOZIE-1319.patch, 
> OOZIE-1319.patch, OOZIE-1319.patch, OOZIE-1319.patch, OOZIE-1319.patch, 
> oozie-1319.patch
>
>
> In execute() of CoordJobGetReadyActionsJPAExecutor.java, once we retrieve the 
> top item from a "LIFO" query result, we do not discard or delete the 
> remaining items from the result list. As a result, the next time execute() is 
> invoked, we will be retrieving the next item in line. Consequently, LAST_ONLY 
> strategy will also execute all ready actions for a given coordinator job, 
> making it no different than LIFO.



--
This message was sent by Atlassian JIRA
(v6.2#6252)

[jira] [Updated] (OOZIE-1864) Improve chid job id aggregation logic

[Purshotam Shah (JIRA)]

 [ 
https://issues.apache.org/jira/browse/OOZIE-1864?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Purshotam Shah updated OOZIE-1864:
--

Description: 
Current chid job id aggregation logic
Once launcher job complete submitting child job (jobs in case on pig), it
writes jobID to file.

>From Oozie server side, we collect childID in two ways
1. As soon as we submit launcher jobs, we check if launcher job terminated or
not. If it's terminated, we read child-id from file and populated to DB.  And
once kill command is issued we kill all child jobs.
2. We have a timer task (ActionCheckerService) which keeps on checking the
status of all running actions and if launcher job is terminated, it's update
the DB with childIDs.

Jobend notification is rejected if action is not running.  

Assume that launcher is killed after it has submitted child job.
Child job will never be killed.


To fix this, we should do following things.

1. If oozie receives job end notification and if launcher job is killed, collect
all child job and kill them if they are not killed.

2. Have a better way logic to collect child job id. Launcher job can call 
callbackServlet ( may be periodically) to
update child job ids. This could be useful in pig jobs. In current scenario we
report child jobs job only when launcher job completes.

> Improve chid job id aggregation logic
> -
>
> Key: OOZIE-1864
> URL: https://issues.apache.org/jira/browse/OOZIE-1864
> Project: Oozie
>  Issue Type: Bug
>Reporter: Purshotam Shah
>
> Current chid job id aggregation logic
> Once launcher job complete submitting child job (jobs in case on pig), it
> writes jobID to file.
> From Oozie server side, we collect childID in two ways
> 1. As soon as we submit launcher jobs, we check if launcher job terminated or
> not. If it's terminated, we read child-id from file and populated to DB.  And
> once kill command is issued we kill all child jobs.
> 2. We have a timer task (ActionCheckerService) which keeps on checking the
> status of all running actions and if launcher job is terminated, it's update
> the DB with childIDs.
> Jobend notification is rejected if action is not running.  
> Assume that launcher is killed after it has submitted child job.
> Child job will never be killed.
> To fix this, we should do following things.
> 1. If oozie receives job end notification and if launcher job is killed, 
> collect
> all child job and kill them if they are not killed.
> 2. Have a better way logic to collect child job id. Launcher job can call 
> callbackServlet ( may be periodically) to
> update child job ids. This could be useful in pig jobs. In current scenario we
> report child jobs job only when launcher job completes.



--
This message was sent by Atlassian JIRA
(v6.2#6252)

[jira] [Created] (OOZIE-1864) Improve chid job id aggregation logic

[Purshotam Shah (JIRA)]

Purshotam Shah created OOZIE-1864:
-

 Summary: Improve chid job id aggregation logic
 Key: OOZIE-1864
 URL: https://issues.apache.org/jira/browse/OOZIE-1864
 Project: Oozie
  Issue Type: Bug
Reporter: Purshotam Shah






--
This message was sent by Atlassian JIRA
(v6.2#6252)

Re: Review Request 21278: OOZIE-1817: Oozie timers are not biased

[Mona Chitnis]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/21278/#review44419
---


some more comments in addition to the couple before


core/src/main/java/org/apache/oozie/servlet/V2AdminServlet.java


where is the disabling of instrumentation endpoint taking place?



core/src/main/java/org/apache/oozie/util/MetricsInstrumentation.java


nitpick- typo retrieve



core/src/main/java/org/apache/oozie/util/MetricsInstrumentation.java


why use lock when guages is a concurrenthashmap?



core/src/main/java/org/apache/oozie/util/MetricsInstrumentation.java


same as above


- Mona Chitnis


On May 22, 2014, 11:27 p.m., Robert Kanter wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/21278/
> ---
> 
> (Updated May 22, 2014, 11:27 p.m.)
> 
> 
> Review request for oozie.
> 
> 
> Bugs: OOZIE-1817
> https://issues.apache.org/jira/browse/OOZIE-1817
> 
> 
> Repository: oozie-git
> 
> 
> Description
> ---
> 
> See 
> https://issues.apache.org/jira/browse/OOZIE-1817?focusedCommentId=13993838&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-13993838
> 
> 
> Diffs
> -
> 
>   client/src/main/java/org/apache/oozie/client/rest/RestConstants.java 
> 97fd465 
>   core/pom.xml c935dd7 
>   core/src/main/java/org/apache/oozie/command/Command.java d3f1011 
>   core/src/main/java/org/apache/oozie/command/XCommand.java b712ce0 
>   
> core/src/main/java/org/apache/oozie/command/coord/CoordActionInputCheckXCommand.java
>  e8667c1 
>   
> core/src/main/java/org/apache/oozie/command/coord/CoordActionMaterializeCommand.java
>  6962fb2 
>   
> core/src/main/java/org/apache/oozie/command/coord/CoordMaterializeTransitionXCommand.java
>  57cbb34 
>   core/src/main/java/org/apache/oozie/service/InstrumentationService.java 
> 80437b1 
>   
> core/src/main/java/org/apache/oozie/service/MetricsInstrumentationService.java
>  PRE-CREATION 
>   core/src/main/java/org/apache/oozie/service/Services.java 5feac7b 
>   core/src/main/java/org/apache/oozie/service/XLogService.java 403a089 
>   core/src/main/java/org/apache/oozie/servlet/BaseAdminServlet.java 29d7bd6 
>   core/src/main/java/org/apache/oozie/servlet/JsonRestServlet.java 5c05acd 
>   core/src/main/java/org/apache/oozie/servlet/V0AdminServlet.java 97bd81c 
>   core/src/main/java/org/apache/oozie/servlet/V1AdminServlet.java a47f737 
>   core/src/main/java/org/apache/oozie/servlet/V2AdminServlet.java 002a367 
>   core/src/main/java/org/apache/oozie/util/MetricsInstrumentation.java 
> PRE-CREATION 
>   core/src/main/java/org/apache/oozie/util/XLog.java 21e00c0 
>   core/src/main/resources/oozie-default.xml 9788daf 
>   core/src/test/java/org/apache/oozie/service/TestInstrumentationService.java 
> b5206cf 
>   
> core/src/test/java/org/apache/oozie/service/TestMetricsInstrumentationService.java
>  PRE-CREATION 
>   core/src/test/java/org/apache/oozie/util/TestInstrumentation.java 6040b47 
>   core/src/test/java/org/apache/oozie/util/TestMetricsInstrumentation.java 
> PRE-CREATION 
>   docs/src/site/twiki/AG_Install.twiki e343d7e 
>   docs/src/site/twiki/WebServicesAPI.twiki 5769924 
>   pom.xml b5e0e4e 
>   webapp/src/main/webapp/index.jsp 3c7ffe5 
>   webapp/src/main/webapp/oozie-console.js 764d888 
> 
> Diff: https://reviews.apache.org/r/21278/diff/
> 
> 
> Testing
> ---
> 
> Unit tests, also verified in a cluster
> 
> 
> Thanks,
> 
> Robert Kanter
> 
>

[jira] [Commented] (OOZIE-1862) Add hadoop token file location for Hive/Tez jobs

[Venkat Ranganathan (JIRA)]

[ 
https://issues.apache.org/jira/browse/OOZIE-1862?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14014210#comment-14014210
 ] 

Venkat Ranganathan commented on OOZIE-1862:
---

Isn't Pig/Tez targeted for Pig 0.14 and not available?   Yes, we have to do the 
same when Pig with Tez is available.

> Add hadoop token file location for Hive/Tez jobs
> 
>
> Key: OOZIE-1862
> URL: https://issues.apache.org/jira/browse/OOZIE-1862
> Project: Oozie
>  Issue Type: Bug
>Reporter: Venkat Ranganathan
>Assignee: Venkat Ranganathan
> Fix For: 4.1.0
>
> Attachments: OOZIE-1862.diff
>
>
> We need to set the configuration parameter tez.credentials.path to the same 
> hadoop token file location as we set for mapreduce.job.credentials.binary
> This is needed to run Hive/Tez jobs



--
This message was sent by Atlassian JIRA
(v6.2#6252)

[jira] [Commented] (OOZIE-1862) Add hadoop token file location for Hive/Tez jobs

[Rohini Palaniswamy (JIRA)]

[ 
https://issues.apache.org/jira/browse/OOZIE-1862?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14014191#comment-14014191
 ] 

Rohini Palaniswamy commented on OOZIE-1862:
---

PigMain?

> Add hadoop token file location for Hive/Tez jobs
> 
>
> Key: OOZIE-1862
> URL: https://issues.apache.org/jira/browse/OOZIE-1862
> Project: Oozie
>  Issue Type: Bug
>Reporter: Venkat Ranganathan
>Assignee: Venkat Ranganathan
> Fix For: 4.1.0
>
> Attachments: OOZIE-1862.diff
>
>
> We need to set the configuration parameter tez.credentials.path to the same 
> hadoop token file location as we set for mapreduce.job.credentials.binary
> This is needed to run Hive/Tez jobs



--
This message was sent by Atlassian JIRA
(v6.2#6252)

[jira] [Updated] (OOZIE-1863) Bundle submit should fail without submitting any coord if one of coord has issue

[Purshotam Shah (JIRA)]

 [ 
https://issues.apache.org/jira/browse/OOZIE-1863?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Purshotam Shah updated OOZIE-1863:
--

Description: 
Currently, bundle submit command queues coord submit for all coords.

StatusTransitService, which normally runs after 5 min check the status of each 
submitted coord and if any of them has failed, it kills running bundle.
This approach has two issue
a. Bundle is status is shown as "killed" ( few submitted coord is also killed). 
It will be difficult for user to find out the root cause.
b. Few of the coord will be in running state for sometime and might corrupt or 
produce stale data.

Bundle submit should dryrun coord before queuing submit command.

We can further improve the bundle job submission logic.
We can call CoordSubmitXCommand synchronously and if any coord submit fails ( 
bcz of DB or other evn issues), kill other submitted coord jobs.

> Bundle submit should fail without submitting any coord if one of coord has 
> issue
> 
>
> Key: OOZIE-1863
> URL: https://issues.apache.org/jira/browse/OOZIE-1863
> Project: Oozie
>  Issue Type: Bug
>Reporter: Purshotam Shah
>
> Currently, bundle submit command queues coord submit for all coords.
> StatusTransitService, which normally runs after 5 min check the status of 
> each submitted coord and if any of them has failed, it kills running bundle.
> This approach has two issue
> a. Bundle is status is shown as "killed" ( few submitted coord is also 
> killed). It will be difficult for user to find out the root cause.
> b. Few of the coord will be in running state for sometime and might corrupt 
> or produce stale data.
> Bundle submit should dryrun coord before queuing submit command.
> We can further improve the bundle job submission logic.
> We can call CoordSubmitXCommand synchronously and if any coord submit fails ( 
> bcz of DB or other evn issues), kill other submitted coord jobs.



--
This message was sent by Atlassian JIRA
(v6.2#6252)

[jira] [Updated] (OOZIE-1813) Add service to report/kill rogue bundles and coordinator jobs

[Purshotam Shah (JIRA)]

 [ 
https://issues.apache.org/jira/browse/OOZIE-1813?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Purshotam Shah updated OOZIE-1813:
--

Attachment: OOZIE-1813-V7.patch

> Add service to report/kill rogue bundles and coordinator jobs
> -
>
> Key: OOZIE-1813
> URL: https://issues.apache.org/jira/browse/OOZIE-1813
> Project: Oozie
>  Issue Type: Bug
>Reporter: Purshotam Shah
>Assignee: Purshotam Shah
> Attachments: OOZIE-1813-V2.patch, OOZIE-1813-V3.patch, 
> OOZIE-1813-V4.patch, OOZIE-1813-V5.patch, OOZIE-1813-V6.patch, 
> OOZIE-1813-V7.patch
>
>
> People leave their test coordinator and bundle jobs without ever killing them
> and they just eat up resources heavily. We should have a service which 
> periodically check for abandoned coords and report/kill them.
> We can add multiple logic to this like ( number of consecutive 
> failed/timedout action, total number of failed/timedout action). 
> To start with if number of coord action with failed/timedout status > defined 
> value, then coord is considered to be rogue.



--
This message was sent by Atlassian JIRA
(v6.2#6252)

Re: Review Request 21423: OOZIE-1813 Add service to report/kill rogue bundles and coordinator jobs

[Purshotam Shah]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/21423/
---

(Updated May 30, 2014, 7:41 p.m.)


Review request for oozie.


Changes
---

Adding log statement.


Bugs: OOZIE-1813
https://issues.apache.org/jira/browse/OOZIE-1813


Repository: oozie-git


Description
---

Report will be sent in html format, where each job is clickable and will take 
you to job info. 

Coordinator id  Coordinator 
nameUser name   Group   Kill Status
000-140513184018277-oozie-puru-CCOORD-TEST  test
testg   Coord kill is disabled
001-140513184018277-oozie-puru-CCOORD-TEST  test
testg   Coord kill is disabled


Diffs (updated)
-

  core/src/main/java/org/apache/oozie/CoordinatorJobBean.java b77082c 
  core/src/main/java/org/apache/oozie/action/email/EmailActionExecutor.java 
332d02c 
  core/src/main/java/org/apache/oozie/command/wf/JobXCommand.java 1d7ca82 
  core/src/main/java/org/apache/oozie/executor/jpa/CoordJobQueryExecutor.java 
1a6ded7 
  core/src/main/java/org/apache/oozie/service/AbandonedCoordCheckerService.java 
e69de29 
  
core/src/test/java/org/apache/oozie/command/coord/TestAbandonedCoordChecker.java
 e69de29 

Diff: https://reviews.apache.org/r/21423/diff/


Testing
---

UTC


Thanks,

Purshotam Shah

[jira] [Created] (OOZIE-1863) Bundle submit should fail without submitting any coord if one of coord has issue

[Purshotam Shah (JIRA)]

Purshotam Shah created OOZIE-1863:
-

 Summary: Bundle submit should fail without submitting any coord if 
one of coord has issue
 Key: OOZIE-1863
 URL: https://issues.apache.org/jira/browse/OOZIE-1863
 Project: Oozie
  Issue Type: Bug
Reporter: Purshotam Shah






--
This message was sent by Atlassian JIRA
(v6.2#6252)

[jira] [Commented] (OOZIE-1817) Oozie timers are not biased

[Mona Chitnis (JIRA)]

[ 
https://issues.apache.org/jira/browse/OOZIE-1817?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14014093#comment-14014093
 ] 

Mona Chitnis commented on OOZIE-1817:
-

I agree we can get rid of the meters. they are not useful at the moment since 
our logging levels are not quite perfect, and also errors/warns here can be due 
to myriad reasons - user definitions etc, and do not necessarily pinpoint to 
system issues.

the JSON attachment show useful metrics around locks and times they were held. 
That will be useful.

taking a second look at code now

> Oozie timers are not biased
> ---
>
> Key: OOZIE-1817
> URL: https://issues.apache.org/jira/browse/OOZIE-1817
> Project: Oozie
>  Issue Type: Improvement
>  Components: monitoring
>Affects Versions: 4.0.0
>Reporter: Gilad Wolff
>Assignee: Robert Kanter
> Attachments: OOZIE-1817.patch, OOZIE-1817.patch, OOZIE-1817.patch, 
> OOZIE-1817.patch, OOZIE-1817.patch, OOZIE-1817.patch, OOZIE-1817.patch, 
> counters.jpg, gauges.jpg, histograms.jpg, json.txt, meters.jpg, timers.jpg
>
>
> Oozie timers are not biased, that is, the statistical metrics they expose are 
> over the run-time of the Oozie server instead of a window of time. This makes 
> them not very useful especially after the server has been running for a while 
> (codehale has very efficient and easy to use biased histograms that can be 
> used instead).



--
This message was sent by Atlassian JIRA
(v6.2#6252)

[jira] [Commented] (OOZIE-1862) Add hadoop token file location for Hive/Tez jobs

[Bowen Zhang (JIRA)]

[ 
https://issues.apache.org/jira/browse/OOZIE-1862?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14014052#comment-14014052
 ] 

Bowen Zhang commented on OOZIE-1862:


committed to trunk. Thanks Venkat.

> Add hadoop token file location for Hive/Tez jobs
> 
>
> Key: OOZIE-1862
> URL: https://issues.apache.org/jira/browse/OOZIE-1862
> Project: Oozie
>  Issue Type: Bug
>Reporter: Venkat Ranganathan
>Assignee: Venkat Ranganathan
> Fix For: 4.1.0
>
> Attachments: OOZIE-1862.diff
>
>
> We need to set the configuration parameter tez.credentials.path to the same 
> hadoop token file location as we set for mapreduce.job.credentials.binary
> This is needed to run Hive/Tez jobs



--
This message was sent by Atlassian JIRA
(v6.2#6252)

[jira] [Commented] (OOZIE-1685) Oozie doesn’t process correctly workflows with a non-default name node

[Hadoop QA (JIRA)]

[ 
https://issues.apache.org/jira/browse/OOZIE-1685?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14013983#comment-14013983
 ] 

Hadoop QA commented on OOZIE-1685:
--

Testing JIRA OOZIE-1685

Cleaning local git workspace



{color:green}+1 PATCH_APPLIES{color}
{color:green}+1 CLEAN{color}
{color:green}+1 RAW_PATCH_ANALYSIS{color}
.{color:green}+1{color} the patch does not introduce any @author tags
.{color:green}+1{color} the patch does not introduce any tabs
.{color:green}+1{color} the patch does not introduce any trailing spaces
.{color:green}+1{color} the patch does not introduce any line longer than 
132
.{color:green}+1{color} the patch does adds/modifies 4 testcase(s)
{color:green}+1 RAT{color}
.{color:green}+1{color} the patch does not seem to introduce new RAT 
warnings
{color:green}+1 JAVADOC{color}
.{color:green}+1{color} the patch does not seem to introduce new Javadoc 
warnings
{color:green}+1 COMPILE{color}
.{color:green}+1{color} HEAD compiles
.{color:green}+1{color} patch compiles
.{color:green}+1{color} the patch does not seem to introduce new javac 
warnings
{color:green}+1 BACKWARDS_COMPATIBILITY{color}
.{color:green}+1{color} the patch does not change any JPA 
Entity/Colum/Basic/Lob/Transient annotations
.{color:green}+1{color} the patch does not modify JPA files
{color:red}-1 TESTS{color}
.Tests run: 1452
.Tests failed: 3
.Tests errors: 2

.The patch failed the following testcases:

.  
testMaxMatThrottleNotPickedMultipleJobs(org.apache.oozie.service.TestCoordMaterializeTriggerService)
.  testSingleRecord(org.apache.oozie.servlet.TestBulkMonitorWebServiceAPI)
.  
testCoordDefUnsupportedChange(org.apache.oozie.command.coord.TestCoordUpdateXCommand)

{color:green}+1 DISTRO{color}
.{color:green}+1{color} distro tarball builds with the patch 


{color:red}*-1 Overall result, please check the reported -1(s)*{color}


The full output of the test-patch run is available at

.   https://builds.apache.org/job/oozie-trunk-precommit-build/1277/

> Oozie doesn’t process correctly workflows with a non-default name node
> --
>
> Key: OOZIE-1685
> URL: https://issues.apache.org/jira/browse/OOZIE-1685
> Project: Oozie
>  Issue Type: Bug
>  Components: core
>Affects Versions: trunk
> Environment: Any
>Reporter: Benjamin Zhitomirsky
> Fix For: trunk
>
> Attachments: Design of the fix OOZIE-1685-rev1.docx, 
> oozie-1685-trunk.patch, oozie-1685-trunk.patch, oozie-1685-trunk.patch, 
> oozie-1685.1.patch, oozie-1685.2.patch, oozie-1685.3.patch, 
> oozie-1685.4.patch, oozie-1685.5.patch
>
>   Original Estimate: 168h
>  Remaining Estimate: 168h
>
> When  element in Oozie workflow specifies a name node different 
> from the default one (specified in core-site.xml), the following 
> functionality doesn’t work properly:
> - Location of libraries specified via 
> oozie.service.WorkflowAppService.system.libpath. Oozie first (during launcher 
> configuration) tries to locate them using name node specified by the 
>  element, but later during job submission it expects this path to 
> be under the default Oozie name node 
> - Processing of the job-xml element if job xml is specified via absolute 
> path. Oozie tries locate it under the default Oozie name node instead of the 
> name-node specified in action.
> Specifying non-default name node makes a lot of sense in Azure environment, 
> because it allows to submit the same job to different Hadoop clusters.
> I will submit a fix for CR soon. Please refer attached short document for 
> more information.



--
This message was sent by Atlassian JIRA
(v6.2#6252)

Build failed in Jenkins: oozie-trunk-precommit-build #1277

[Apache Jenkins Server]

See 

--
[...truncated 9694 lines...]
[INFO] share/lib already added, skipping
[INFO]  already added, skipping
[INFO] share already added, skipping
[INFO] share/lib already added, skipping
[INFO]  already added, skipping
[INFO] share already added, skipping
[INFO] share/lib already added, skipping
[INFO]  already added, skipping
[INFO] share already added, skipping
[INFO] share/lib already added, skipping
[INFO]  already added, skipping
[INFO] share already added, skipping
[INFO] share/lib already added, skipping
[INFO]  already added, skipping
[INFO] share already added, skipping
[INFO] share/lib already added, skipping
[INFO] 
[INFO] 
[INFO] Building Apache Oozie Tools 4.1.0-SNAPSHOT
[INFO] 
[WARNING] Could not transfer metadata asm:asm/maven-metadata.xml from/to 
local.repository (file:../../local.repository/trunk): No connector available to 
access repository local.repository (file:../../local.repository/trunk) of type 
legacy using the available factories WagonRepositoryConnectorFactory
[WARNING] Could not transfer metadata asm:asm/maven-metadata.xml from/to 
local.repository (file:../../local.repository/trunk): No connector available to 
access repository local.repository (file:../../local.repository/trunk) of type 
legacy using the available factories WagonRepositoryConnectorFactory
[INFO] 
[INFO] --- maven-resources-plugin:2.5:resources (default-resources) @ 
oozie-tools ---
[debug] execute contextualize
[INFO] Using 'UTF-8' encoding to copy filtered resources.
[INFO] skip non existing resourceDirectory 

[INFO] 
[INFO] --- maven-compiler-plugin:2.3.2:compile (default-compile) @ oozie-tools 
---
[INFO] Nothing to compile - all classes are up to date
[INFO] 
[INFO] --- maven-resources-plugin:2.5:testResources (default-testResources) @ 
oozie-tools ---
[debug] execute contextualize
[INFO] Using 'UTF-8' encoding to copy filtered resources.
[INFO] Copying 2 resources
[INFO] 
[INFO] --- maven-compiler-plugin:2.3.2:testCompile (default-testCompile) @ 
oozie-tools ---
[INFO] Nothing to compile - all classes are up to date
[INFO] 
[INFO] --- maven-surefire-plugin:2.12:test (default-test) @ oozie-tools ---
[INFO] Tests are skipped.
[INFO] 
[INFO] --- maven-jar-plugin:2.3.1:jar (default-jar) @ oozie-tools ---
[INFO] Building jar: 

[INFO] 
[INFO] --- maven-assembly-plugin:2.2.1:single (default-cli) @ oozie-tools ---
[INFO] Reading assembly descriptor: ../src/main/assemblies/tools.xml
[WARNING] The following patterns were never triggered in this artifact 
exclusion filter:
o  '*:*:pom:*'

[INFO] Copying files to 

[WARNING] Assembly file: 

 is not a regular file (it may be a directory). It cannot be attached to the 
project build for installation or deployment.
[INFO] 
[INFO] 
[INFO] Building Apache Oozie MiniOozie 4.1.0-SNAPSHOT
[INFO] 
[INFO] 
[INFO] --- maven-resources-plugin:2.5:resources (default-resources) @ 
oozie-mini ---
[debug] execute contextualize
[INFO] Using 'UTF-8' encoding to copy filtered resources.
[INFO] skip non existing resourceDirectory 

[INFO] 
[INFO] --- maven-compiler-plugin:2.3.2:compile (default-compile) @ oozie-mini 
---
[INFO] No sources to compile
[INFO] 
[INFO] --- maven-resources-plugin:2.5:testResources (default-testResources) @ 
oozie-mini ---
[debug] execute contextualize
[INFO] Using 'UTF-8' encoding to copy filtered resources.
[INFO] Copying 3 resources
[INFO] 
[INFO] --- maven-compiler-plugin:2.3.2:testCompile (default-testCompile) @ 
oozie-mini ---
[INFO] Nothing to compile - all classes are up to date
[INFO] 
[INFO] --- maven-surefire-plugin:2.12:test (default-test) @ oozie-mini ---
[INFO] Tests are skipped.
[INFO] 
[INFO] --- maven-jar-plugin:2.3.1:jar (default-jar) @ oozie-mini ---
[WARNING] JAR will be empty - no content was marked for inclusion!
[INFO] Building jar: 

[INFO] 
[INFO] --- maven-assembly-plugin:2.2.1:single (default-cli) @ oozie

[jira] [Updated] (OOZIE-1703) User should be able to set coord end-time before start time

[Purshotam Shah (JIRA)]

 [ 
https://issues.apache.org/jira/browse/OOZIE-1703?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Purshotam Shah updated OOZIE-1703:
--

Attachment: OOZIE-1703-V7.patch

> User should be able to set coord end-time before start time
> ---
>
> Key: OOZIE-1703
> URL: https://issues.apache.org/jira/browse/OOZIE-1703
> Project: Oozie
>  Issue Type: Bug
>Reporter: Purshotam Shah
>Assignee: Purshotam Shah
> Fix For: trunk
>
> Attachments: OOZIE-1703-V5.patch, OOZIE-1703-V6.patch, 
> OOZIE-1703-V7.patch, OOZIE-1703.patch
>
>
> This is one of the important use-case in case of versioning.
> User can set end date of old coord before start date( if coord is supposed to 
> run in future), so that it doesn't run and user can safely upload new version.
> We should also lookahead at created actions that become invalid because of 
> the new end time.
> These actions should be deleted from DB.
> We  should also update the status of bundle. If it's in PREP and new end-date 
> is before kick off time, then job status will be set to SUCCEEDED. User can 
> again change the end date to future date, the status will be changed to 
> RUNNING.



--
This message was sent by Atlassian JIRA
(v6.2#6252)

[jira] [Reopened] (OOZIE-1703) User should be able to set coord end-time before start time

[Purshotam Shah (JIRA)]

 [ 
https://issues.apache.org/jira/browse/OOZIE-1703?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Purshotam Shah reopened OOZIE-1703:
---


> User should be able to set coord end-time before start time
> ---
>
> Key: OOZIE-1703
> URL: https://issues.apache.org/jira/browse/OOZIE-1703
> Project: Oozie
>  Issue Type: Bug
>Reporter: Purshotam Shah
>Assignee: Purshotam Shah
> Fix For: trunk
>
> Attachments: OOZIE-1703-V5.patch, OOZIE-1703-V6.patch, 
> OOZIE-1703-V7.patch, OOZIE-1703.patch
>
>
> This is one of the important use-case in case of versioning.
> User can set end date of old coord before start date( if coord is supposed to 
> run in future), so that it doesn't run and user can safely upload new version.
> We should also lookahead at created actions that become invalid because of 
> the new end time.
> These actions should be deleted from DB.
> We  should also update the status of bundle. If it's in PREP and new end-date 
> is before kick off time, then job status will be set to SUCCEEDED. User can 
> again change the end date to future date, the status will be changed to 
> RUNNING.



--
This message was sent by Atlassian JIRA
(v6.2#6252)

[jira] [Commented] (OOZIE-1703) User should be able to set coord end-time before start time

[Purshotam Shah (JIRA)]

[ 
https://issues.apache.org/jira/browse/OOZIE-1703?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14013931#comment-14013931
 ] 

Purshotam Shah commented on OOZIE-1703:
---

We noticed some issue while testing internally.

If we set endtime multiple times, coord remains in running state.

Attaching amendment patch.

> User should be able to set coord end-time before start time
> ---
>
> Key: OOZIE-1703
> URL: https://issues.apache.org/jira/browse/OOZIE-1703
> Project: Oozie
>  Issue Type: Bug
>Reporter: Purshotam Shah
>Assignee: Purshotam Shah
> Fix For: trunk
>
> Attachments: OOZIE-1703-V5.patch, OOZIE-1703-V6.patch, 
> OOZIE-1703-V7.patch, OOZIE-1703.patch
>
>
> This is one of the important use-case in case of versioning.
> User can set end date of old coord before start date( if coord is supposed to 
> run in future), so that it doesn't run and user can safely upload new version.
> We should also lookahead at created actions that become invalid because of 
> the new end time.
> These actions should be deleted from DB.
> We  should also update the status of bundle. If it's in PREP and new end-date 
> is before kick off time, then job status will be set to SUCCEEDED. User can 
> again change the end date to future date, the status will be changed to 
> RUNNING.



--
This message was sent by Atlassian JIRA
(v6.2#6252)

[jira] [Updated] (OOZIE-1685) Oozie doesn’t process correctly workflows with a non-default name node

[Benjamin Zhitomirsky (JIRA)]

 [ 
https://issues.apache.org/jira/browse/OOZIE-1685?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Benjamin Zhitomirsky updated OOZIE-1685:


Attachment: oozie-1685.5.patch

CR comments applied

> Oozie doesn’t process correctly workflows with a non-default name node
> --
>
> Key: OOZIE-1685
> URL: https://issues.apache.org/jira/browse/OOZIE-1685
> Project: Oozie
>  Issue Type: Bug
>  Components: core
>Affects Versions: trunk, 3.3.2, 4.0.0
> Environment: Any
>Reporter: Benjamin Zhitomirsky
> Fix For: trunk
>
> Attachments: Design of the fix OOZIE-1685-rev1.docx, 
> oozie-1685-trunk.patch, oozie-1685-trunk.patch, oozie-1685-trunk.patch, 
> oozie-1685.1.patch, oozie-1685.2.patch, oozie-1685.3.patch, 
> oozie-1685.4.patch, oozie-1685.5.patch
>
>   Original Estimate: 168h
>  Remaining Estimate: 168h
>
> When  element in Oozie workflow specifies a name node different 
> from the default one (specified in core-site.xml), the following 
> functionality doesn’t work properly:
> - Location of libraries specified via 
> oozie.service.WorkflowAppService.system.libpath. Oozie first (during launcher 
> configuration) tries to locate them using name node specified by the 
>  element, but later during job submission it expects this path to 
> be under the default Oozie name node 
> - Processing of the job-xml element if job xml is specified via absolute 
> path. Oozie tries locate it under the default Oozie name node instead of the 
> name-node specified in action.
> Specifying non-default name node makes a lot of sense in Azure environment, 
> because it allows to submit the same job to different Hadoop clusters.
> I will submit a fix for CR soon. Please refer attached short document for 
> more information.



--
This message was sent by Atlassian JIRA
(v6.2#6252)

Re: Review Request 19929: OOZIE-1685: Oozie doesn’t process correctly workflows with a non-default name node

[Benjamin Zhitomirsky]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/19929/
---

(Updated May 30, 2014, 2:32 p.m.)


Review request for oozie.


Changes
---

Fixed according to CR comments


Repository: oozie-git


Description
---

When  element in Oozie workflow specifies a name node different from 
the default one (specified in core-site.xml), the following functionality 
doesn’t work properly:
?Location of libraries specified via 
oozie.service.WorkflowAppService.system.libpath. Oozie first (during launcher 
configuration) tries to locate them using name node specified by the 
 element, but later during job submission it expects this path to be 
under the default Oozie name node
?Processing of the job-xml element if job xml is specified via absolute path. 
Oozie tries locate it under the default Oozie name node instead of the 
name-node specified in action.

Specifying non-default name node makes a lot of sense in Azure environment, 
because it allows to submit the same job to different Hadoop clusters.


Diffs (updated)
-

  core/src/main/java/org/apache/oozie/action/hadoop/JavaActionExecutor.java 
40add2c 
  core/src/main/java/org/apache/oozie/service/HadoopAccessorService.java 
bb68b0e 
  core/src/main/java/org/apache/oozie/service/ShareLibService.java 353b382 
  core/src/main/java/org/apache/oozie/util/JobUtils.java 135b096 
  core/src/test/java/org/apache/oozie/action/hadoop/ActionExecutorTestCase.java 
bc2c1b6 
  core/src/test/java/org/apache/oozie/action/hadoop/TestJavaActionExecutor.java 
390ad3f 
  core/src/test/java/org/apache/oozie/test/XFsTestCase.java 18cb742 
  core/src/test/java/org/apache/oozie/test/XTestCase.java 1536927 
  docs/src/site/twiki/WorkflowFunctionalSpec.twiki f7590d0 

Diff: https://reviews.apache.org/r/19929/diff/


Testing
---

On deployed Hadoop cluster. Two tests were added.


Thanks,

Benjamin Zhitomirsky

[jira] [Commented] (OOZIE-1860) Oozie job mapper launch fails due to null value returned from action file

[Hadoop QA (JIRA)]

[ 
https://issues.apache.org/jira/browse/OOZIE-1860?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14013413#comment-14013413
 ] 

Hadoop QA commented on OOZIE-1860:
--

Testing JIRA OOZIE-1860

Cleaning local git workspace



{color:green}+1 PATCH_APPLIES{color}
{color:green}+1 CLEAN{color}
{color:green}+1 RAW_PATCH_ANALYSIS{color}
.{color:green}+1{color} the patch does not introduce any @author tags
.{color:green}+1{color} the patch does not introduce any tabs
.{color:green}+1{color} the patch does not introduce any trailing spaces
.{color:green}+1{color} the patch does not introduce any line longer than 
132
.{color:green}+1{color} the patch does adds/modifies 1 testcase(s)
{color:green}+1 RAT{color}
.{color:green}+1{color} the patch does not seem to introduce new RAT 
warnings
{color:green}+1 JAVADOC{color}
.{color:green}+1{color} the patch does not seem to introduce new Javadoc 
warnings
{color:green}+1 COMPILE{color}
.{color:green}+1{color} HEAD compiles
.{color:green}+1{color} patch compiles
.{color:green}+1{color} the patch does not seem to introduce new javac 
warnings
{color:green}+1 BACKWARDS_COMPATIBILITY{color}
.{color:green}+1{color} the patch does not change any JPA 
Entity/Colum/Basic/Lob/Transient annotations
.{color:green}+1{color} the patch does not modify JPA files
{color:red}-1 TESTS{color} - patch does not compile, cannot run testcases
{color:green}+1 DISTRO{color}
.{color:green}+1{color} distro tarball builds with the patch 


{color:red}*-1 Overall result, please check the reported -1(s)*{color}


The full output of the test-patch run is available at

.   https://builds.apache.org/job/oozie-trunk-precommit-build/1276/

> Oozie job mapper launch fails due to null value returned from action file
> -
>
> Key: OOZIE-1860
> URL: https://issues.apache.org/jira/browse/OOZIE-1860
> Project: Oozie
>  Issue Type: Bug
>Affects Versions: 4.0.1
>Reporter: Osadchiy Artem
> Fix For: trunk
>
> Attachments: OOZIE-1860.patch, action.log, oozie.log
>
>
> Certain oozie workflows fail at launch mapper stage when the id retrieved 
> from the recovery action file returns a null value.
> Logs attached below



--
This message was sent by Atlassian JIRA
(v6.2#6252)

Build failed in Jenkins: oozie-trunk-precommit-build #1276

[Apache Jenkins Server]

See 

--
[...truncated 9031 lines...]
[INFO] share/lib already added, skipping
[INFO]  already added, skipping
[INFO] share already added, skipping
[INFO] share/lib already added, skipping
[INFO]  already added, skipping
[INFO] share already added, skipping
[INFO] share/lib already added, skipping
[INFO] Building tar : 

[INFO]  already added, skipping
[INFO] share already added, skipping
[INFO] share/lib already added, skipping
[INFO]  already added, skipping
[INFO] share already added, skipping
[INFO] share/lib already added, skipping
[INFO]  already added, skipping
[INFO] share already added, skipping
[INFO] share/lib already added, skipping
[INFO]  already added, skipping
[INFO] share already added, skipping
[INFO] share/lib already added, skipping
[INFO]  already added, skipping
[INFO] share already added, skipping
[INFO] share/lib already added, skipping
[INFO]  already added, skipping
[INFO] share already added, skipping
[INFO] share/lib already added, skipping
[INFO] 
[INFO] 
[INFO] Building Apache Oozie Tools 4.1.0-SNAPSHOT
[INFO] 
[WARNING] Could not transfer metadata asm:asm/maven-metadata.xml from/to 
local.repository (file:../../local.repository/trunk): No connector available to 
access repository local.repository (file:../../local.repository/trunk) of type 
legacy using the available factories WagonRepositoryConnectorFactory
[WARNING] Could not transfer metadata asm:asm/maven-metadata.xml from/to 
local.repository (file:../../local.repository/trunk): No connector available to 
access repository local.repository (file:../../local.repository/trunk) of type 
legacy using the available factories WagonRepositoryConnectorFactory
[INFO] 
[INFO] --- maven-resources-plugin:2.5:resources (default-resources) @ 
oozie-tools ---
[debug] execute contextualize
[INFO] Using 'UTF-8' encoding to copy filtered resources.
[INFO] skip non existing resourceDirectory 

[INFO] 
[INFO] --- maven-compiler-plugin:2.3.2:compile (default-compile) @ oozie-tools 
---
[INFO] Nothing to compile - all classes are up to date
[INFO] 
[INFO] --- maven-resources-plugin:2.5:testResources (default-testResources) @ 
oozie-tools ---
[debug] execute contextualize
[INFO] Using 'UTF-8' encoding to copy filtered resources.
[INFO] Copying 2 resources
[INFO] 
[INFO] --- maven-compiler-plugin:2.3.2:testCompile (default-testCompile) @ 
oozie-tools ---
[INFO] Nothing to compile - all classes are up to date
[INFO] 
[INFO] --- maven-surefire-plugin:2.12:test (default-test) @ oozie-tools ---
[INFO] Tests are skipped.
[INFO] 
[INFO] --- maven-jar-plugin:2.3.1:jar (default-jar) @ oozie-tools ---
[INFO] Building jar: 

[INFO] 
[INFO] --- maven-assembly-plugin:2.2.1:single (default-cli) @ oozie-tools ---
[INFO] Reading assembly descriptor: ../src/main/assemblies/tools.xml
[WARNING] The following patterns were never triggered in this artifact 
exclusion filter:
o  '*:*:pom:*'

[INFO] Copying files to 

[WARNING] Assembly file: 

 is not a regular file (it may be a directory). It cannot be attached to the 
project build for installation or deployment.
[INFO] 
[INFO] 
[INFO] Building Apache Oozie MiniOozie 4.1.0-SNAPSHOT
[INFO] 
[INFO] 
[INFO] --- maven-resources-plugin:2.5:resources (default-resources) @ 
oozie-mini ---
[debug] execute contextualize
[INFO] Using 'UTF-8' encoding to copy filtered resources.
[INFO] skip non existing resourceDirectory 

[INFO] 
[INFO] --- maven-compiler-plugin:2.3.2:compile (default-compile) @ oozie-mini 
---
[INFO] No sources to compile
[INFO] 
[INFO] --- maven-resources-plugin:2.5:testResources (default-testResources) @ 
oozie-mini ---
[debug] execute contextualize
[INFO] Using 'UTF-8' encoding to copy filtered resources.
[INFO] Copying 3 resources
[INFO] 
[INFO] --- maven-compiler-plugin:2.3.2:testCompile (default-testCompile) @ 
oozie-mini ---
[INFO] Nothing to compile - all classes are u

[jira] [Updated] (OOZIE-1860) Oozie job mapper launch fails due to null value returned from action file

[Osadchiy Artem (JIRA)]

 [ 
https://issues.apache.org/jira/browse/OOZIE-1860?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Osadchiy Artem updated OOZIE-1860:
--

Attachment: (was: OOZIE-1860.patch)

> Oozie job mapper launch fails due to null value returned from action file
> -
>
> Key: OOZIE-1860
> URL: https://issues.apache.org/jira/browse/OOZIE-1860
> Project: Oozie
>  Issue Type: Bug
>Affects Versions: 4.0.1
>Reporter: Osadchiy Artem
> Fix For: trunk
>
> Attachments: OOZIE-1860.patch, action.log, oozie.log
>
>
> Certain oozie workflows fail at launch mapper stage when the id retrieved 
> from the recovery action file returns a null value.
> Logs attached below



--
This message was sent by Atlassian JIRA
(v6.2#6252)

[jira] [Updated] (OOZIE-1860) Oozie job mapper launch fails due to null value returned from action file

[Osadchiy Artem (JIRA)]

 [ 
https://issues.apache.org/jira/browse/OOZIE-1860?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Osadchiy Artem updated OOZIE-1860:
--

Attachment: OOZIE-1860.patch

> Oozie job mapper launch fails due to null value returned from action file
> -
>
> Key: OOZIE-1860
> URL: https://issues.apache.org/jira/browse/OOZIE-1860
> Project: Oozie
>  Issue Type: Bug
>Affects Versions: 4.0.1
>Reporter: Osadchiy Artem
> Fix For: trunk
>
> Attachments: OOZIE-1860.patch, action.log, oozie.log
>
>
> Certain oozie workflows fail at launch mapper stage when the id retrieved 
> from the recovery action file returns a null value.
> Logs attached below



--
This message was sent by Atlassian JIRA
(v6.2#6252)