[jira] [Updated] (OOZIE-1813) Add service to report/kill rogue bundles and coordinator jobs

[Purshotam Shah (JIRA)]

 [ 
https://issues.apache.org/jira/browse/OOZIE-1813?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Purshotam Shah updated OOZIE-1813:
--

Attachment: (was: OOZIE-1813-V3.patch)

> Add service to report/kill rogue bundles and coordinator jobs
> -
>
> Key: OOZIE-1813
> URL: https://issues.apache.org/jira/browse/OOZIE-1813
> Project: Oozie
>  Issue Type: Bug
>Reporter: Purshotam Shah
>Assignee: Purshotam Shah
> Attachments: OOZIE-1813-V2.patch, OOZIE-1813-V3.patch
>
>
> People leave their test coordinator and bundle jobs without ever killing them
> and they just eat up resources heavily. We should have a service which 
> periodically check for abandoned coords and report/kill them.
> We can add multiple logic to this like ( number of consecutive 
> failed/timedout action, total number of failed/timedout action). 
> To start with if number of coord action with failed/timedout status > defined 
> value, then coord is considered to be rogue.



--
This message was sent by Atlassian JIRA
(v6.2#6252)

[jira] [Updated] (OOZIE-1737) Oozie log streaming is slow

[Purshotam Shah (JIRA)]

 [ 
https://issues.apache.org/jira/browse/OOZIE-1737?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Purshotam Shah updated OOZIE-1737:
--

Attachment: OOZIE-1737-V8.patch

> Oozie log streaming is slow
> ---
>
> Key: OOZIE-1737
> URL: https://issues.apache.org/jira/browse/OOZIE-1737
> Project: Oozie
>  Issue Type: Bug
>Reporter: Purshotam Shah
>Assignee: Purshotam Shah
> Fix For: trunk
>
> Attachments: OOZIE-1737-V5.patch, OOZIE-1737-V6.patch, 
> OOZIE-1737-V7.patch, OOZIE-1737-V8.patch
>
>
> Most of cases, Oozie search huge set of log files and return lot of logs.
> Most of our users has complained that log streaming is slow and they can't 
> easily find relevant information.
> To make log streaming fast and precise we should provide few options (which 
> user can specify for log streaming) like
> 1. Start time : Start date (if number, then offset(in hour) from 
> coord/wf/bundle start)
> 2. End time : End date (if number, then offset from coord/wf/bundle start)
> 3. Log level :
>Setting log level to ERROR, will only give erred log and exception.
>In most of cases user want to find out error message and exception.
> 4. Number of lines : limit number of lines of logs from start date.
> 5. Recent logs: In most of the case user is only interested in recent 
> activity.
> Similar to https://issues.apache.org/jira/browse/OOZIE-1716.
> User should be able to specify recent logs in hours.
> 6. Grep for pattern.
> + in addition to this, we also need to clean up log statements( which may 
> also include consolidating log statements).



--
This message was sent by Atlassian JIRA
(v6.2#6252)

[jira] [Commented] (OOZIE-1737) Oozie log streaming is slow

[Hadoop QA (JIRA)]

[ 
https://issues.apache.org/jira/browse/OOZIE-1737?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13999762#comment-13999762
 ] 

Hadoop QA commented on OOZIE-1737:
--

Testing JIRA OOZIE-1737

Cleaning local git workspace



{color:green}+1 PATCH_APPLIES{color}
{color:green}+1 CLEAN{color}
{color:green}+1 RAW_PATCH_ANALYSIS{color}
.{color:green}+1{color} the patch does not introduce any @author tags
.{color:green}+1{color} the patch does not introduce any tabs
.{color:green}+1{color} the patch does not introduce any trailing spaces
.{color:green}+1{color} the patch does not introduce any line longer than 
132
.{color:green}+1{color} the patch does adds/modifies 1 testcase(s)
{color:green}+1 RAT{color}
.{color:green}+1{color} the patch does not seem to introduce new RAT 
warnings
{color:green}+1 JAVADOC{color}
.{color:green}+1{color} the patch does not seem to introduce new Javadoc 
warnings
{color:green}+1 COMPILE{color}
.{color:green}+1{color} HEAD compiles
.{color:green}+1{color} patch compiles
.{color:green}+1{color} the patch does not seem to introduce new javac 
warnings
{color:green}+1 BACKWARDS_COMPATIBILITY{color}
.{color:green}+1{color} the patch does not change any JPA 
Entity/Colum/Basic/Lob/Transient annotations
.{color:green}+1{color} the patch does not modify JPA files
{color:red}-1 TESTS{color}
.Tests run: 1446
.Tests failed: 4
.Tests errors: 2

.The patch failed the following testcases:

.  
testCoordinatorActionCommandsSubmitAndStart(org.apache.oozie.sla.TestSLAEventGeneration)
.  
testBundleEngineKill(org.apache.oozie.servlet.TestV1JobServletBundleEngine)
.  
testBundleEngineResume(org.apache.oozie.servlet.TestV1JobServletBundleEngine)
.  
testTimeout(org.apache.oozie.command.coord.TestCoordActionInputCheckXCommand)

{color:green}+1 DISTRO{color}
.{color:green}+1{color} distro tarball builds with the patch 


{color:red}*-1 Overall result, please check the reported -1(s)*{color}


The full output of the test-patch run is available at

.   https://builds.apache.org/job/oozie-trunk-precommit-build/1242/

> Oozie log streaming is slow
> ---
>
> Key: OOZIE-1737
> URL: https://issues.apache.org/jira/browse/OOZIE-1737
> Project: Oozie
>  Issue Type: Bug
>Reporter: Purshotam Shah
>Assignee: Purshotam Shah
> Fix For: trunk
>
> Attachments: OOZIE-1737-V5.patch, OOZIE-1737-V6.patch, 
> OOZIE-1737-V7.patch, OOZIE-1737-V8.patch
>
>
> Most of cases, Oozie search huge set of log files and return lot of logs.
> Most of our users has complained that log streaming is slow and they can't 
> easily find relevant information.
> To make log streaming fast and precise we should provide few options (which 
> user can specify for log streaming) like
> 1. Start time : Start date (if number, then offset(in hour) from 
> coord/wf/bundle start)
> 2. End time : End date (if number, then offset from coord/wf/bundle start)
> 3. Log level :
>Setting log level to ERROR, will only give erred log and exception.
>In most of cases user want to find out error message and exception.
> 4. Number of lines : limit number of lines of logs from start date.
> 5. Recent logs: In most of the case user is only interested in recent 
> activity.
> Similar to https://issues.apache.org/jira/browse/OOZIE-1716.
> User should be able to specify recent logs in hours.
> 6. Grep for pattern.
> + in addition to this, we also need to clean up log statements( which may 
> also include consolidating log statements).



--
This message was sent by Atlassian JIRA
(v6.2#6252)

[jira] [Created] (OOZIE-1849) If the underlying job finishes while a Workflow is suspended, Oozie can take a while to realize it

[Robert Kanter (JIRA)]

Robert Kanter created OOZIE-1849:


 Summary: If the underlying job finishes while a Workflow is 
suspended, Oozie can take a while to realize it
 Key: OOZIE-1849
 URL: https://issues.apache.org/jira/browse/OOZIE-1849
 Project: Oozie
  Issue Type: Improvement
  Components: core
Affects Versions: 4.0.1
Reporter: Robert Kanter
Assignee: Robert Kanter


Suppose you have a Workflow and you suspend it while one of the actions is 
still RUNNING.  The underlying MR/Pig/etc job will continue running (as 
expected, because we can't pause those).  However, if that job finishes while 
the workflow is SUSPENDED, the CallbackServlet will receive the callback, but 
the ActionCheckXCommand won't update the action:
{noformat}
2014-05-16 17:40:57,959  INFO CallbackServlet:541 - SERVER[rkanter-mbp.local] 
USER[-] GROUP[-] TOKEN[-] APP[-] JOB[002-140516173529928-oozie-rkan-W] 
ACTION[002-140516173529928-oozie-rkan-W@mr-node] callback for action 
[002-140516173529928-oozie-rkan-W@mr-node]
2014-05-16 17:40:57,985  WARN ActionCheckXCommand:544 - 
SERVER[rkanter-mbp.local] USER[rkanter] GROUP[-] TOKEN[] APP[map-reduce-wf] 
JOB[002-140516173529928-oozie-rkan-W] 
ACTION[002-140516173529928-oozie-rkan-W@mr-node] E0818: Action 
[002-140516173529928-oozie-rkan-W@mr-node] status is running but WF Job 
[002-140516173529928-oozie-rkan-W] status is [SUSPENDED]. Expected status 
is RUNNING., Error Code: E0818
{noformat}
If you then resume the workflow, the action will stay RUNNING for up to 10 
minutes (the default fallback polling interval), at which point the 
ActionCheckerService will run an ActionCheckXCommand that will pass, check the 
job, and finally mark the action as SUCCESSFUL.

We should fix this by one of the following:
# ResumeXCommand should also queue a ActionCheckXCommand (if the workflow was 
SUSPENDED) so we don't have to wait for the ActionCheckerService
# ActionCheckXCommand's precondition check should allow SUSPENDED workflows



--
This message was sent by Atlassian JIRA
(v6.2#6252)

Re: Review Request 21278: OOZIE-1817: Oozie timers are not biased

[Robert Kanter]

> On May 15, 2014, 8:55 p.m., Gilad Wolff wrote:
> > core/src/main/java/org/apache/oozie/service/MetricsInstrumentationService.java,
> >  line 37
> > 
> >
> > nit: you don't really need this member. Use the base class member 
> > instead?
> 
> Robert Kanter wrote:
> MetricsInstrumentation has a destroy() method that we have to call to 
> stop the Metrics stuff from logging.  Instrumentation doesn't have this, so I 
> need it to be a MetricsInstrumentation object and not simply an 
> Instrumentation object (and I'd rather not deal with casting).

I ended up doing this because I made some other changes and didn't need the 
destroy() anymore :)


- Robert


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/21278/#review42740
---


On May 13, 2014, 5:40 p.m., Robert Kanter wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/21278/
> ---
> 
> (Updated May 13, 2014, 5:40 p.m.)
> 
> 
> Review request for oozie.
> 
> 
> Bugs: OOZIE-1817
> https://issues.apache.org/jira/browse/OOZIE-1817
> 
> 
> Repository: oozie-git
> 
> 
> Description
> ---
> 
> See 
> https://issues.apache.org/jira/browse/OOZIE-1817?focusedCommentId=13993838&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-13993838
> 
> 
> Diffs
> -
> 
>   client/src/main/java/org/apache/oozie/client/rest/RestConstants.java 
> 808f9b2 
>   core/pom.xml c935dd7 
>   core/src/main/java/org/apache/oozie/command/Command.java d3f1011 
>   core/src/main/java/org/apache/oozie/command/XCommand.java b712ce0 
>   
> core/src/main/java/org/apache/oozie/command/coord/CoordActionInputCheckXCommand.java
>  e8667c1 
>   
> core/src/main/java/org/apache/oozie/command/coord/CoordActionMaterializeCommand.java
>  6962fb2 
>   
> core/src/main/java/org/apache/oozie/command/coord/CoordMaterializeTransitionXCommand.java
>  57cbb34 
>   core/src/main/java/org/apache/oozie/service/InstrumentationService.java 
> 80437b1 
>   
> core/src/main/java/org/apache/oozie/service/MetricsInstrumentationService.java
>  PRE-CREATION 
>   core/src/main/java/org/apache/oozie/service/Services.java 5feac7b 
>   core/src/main/java/org/apache/oozie/service/XLogService.java 403a089 
>   core/src/main/java/org/apache/oozie/servlet/BaseAdminServlet.java 29d7bd6 
>   core/src/main/java/org/apache/oozie/servlet/JsonRestServlet.java 5c05acd 
>   core/src/main/java/org/apache/oozie/servlet/V0AdminServlet.java 97bd81c 
>   core/src/main/java/org/apache/oozie/servlet/V1AdminServlet.java a47f737 
>   core/src/main/java/org/apache/oozie/servlet/V2AdminServlet.java 002a367 
>   core/src/main/java/org/apache/oozie/util/MetricsInstrumentation.java 
> PRE-CREATION 
>   core/src/main/java/org/apache/oozie/util/XLogReporter.java PRE-CREATION 
>   core/src/test/java/org/apache/oozie/service/TestInstrumentationService.java 
> b5206cf 
>   
> core/src/test/java/org/apache/oozie/service/TestMetricsInstrumentationService.java
>  PRE-CREATION 
>   core/src/test/java/org/apache/oozie/util/TestInstrumentation.java 6040b47 
>   core/src/test/java/org/apache/oozie/util/TestMetricsInstrumentation.java 
> PRE-CREATION 
>   docs/src/site/twiki/AG_Install.twiki e343d7e 
>   docs/src/site/twiki/WebServicesAPI.twiki 6730255 
>   pom.xml b5e0e4e 
>   webapp/src/main/webapp/index.jsp 3c7ffe5 
>   webapp/src/main/webapp/oozie-console.js 764d888 
> 
> Diff: https://reviews.apache.org/r/21278/diff/
> 
> 
> Testing
> ---
> 
> Unit tests, also verified in a cluster
> 
> 
> Thanks,
> 
> Robert Kanter
> 
>

Re: Review Request 21278: OOZIE-1817: Oozie timers are not biased

[Robert Kanter]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/21278/
---

(Updated May 17, 2014, 12:09 a.m.)


Review request for oozie.


Changes
---

See 
https://issues.apache.org/jira/browse/OOZIE-1817?focusedCommentId=14000539&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-14000539


Bugs: OOZIE-1817
https://issues.apache.org/jira/browse/OOZIE-1817


Repository: oozie-git


Description
---

See 
https://issues.apache.org/jira/browse/OOZIE-1817?focusedCommentId=13993838&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-13993838


Diffs (updated)
-

  client/src/main/java/org/apache/oozie/client/rest/RestConstants.java 97fd465 
  core/pom.xml c935dd7 
  core/src/main/java/org/apache/oozie/command/Command.java d3f1011 
  core/src/main/java/org/apache/oozie/command/XCommand.java b712ce0 
  
core/src/main/java/org/apache/oozie/command/coord/CoordActionInputCheckXCommand.java
 e8667c1 
  
core/src/main/java/org/apache/oozie/command/coord/CoordActionMaterializeCommand.java
 6962fb2 
  
core/src/main/java/org/apache/oozie/command/coord/CoordMaterializeTransitionXCommand.java
 57cbb34 
  core/src/main/java/org/apache/oozie/service/InstrumentationService.java 
80437b1 
  
core/src/main/java/org/apache/oozie/service/MetricsInstrumentationService.java 
PRE-CREATION 
  core/src/main/java/org/apache/oozie/service/Services.java 5feac7b 
  core/src/main/java/org/apache/oozie/service/XLogService.java 403a089 
  core/src/main/java/org/apache/oozie/servlet/BaseAdminServlet.java 29d7bd6 
  core/src/main/java/org/apache/oozie/servlet/JsonRestServlet.java 5c05acd 
  core/src/main/java/org/apache/oozie/servlet/V0AdminServlet.java 97bd81c 
  core/src/main/java/org/apache/oozie/servlet/V1AdminServlet.java a47f737 
  core/src/main/java/org/apache/oozie/servlet/V2AdminServlet.java 002a367 
  core/src/main/java/org/apache/oozie/util/MetricsInstrumentation.java 
PRE-CREATION 
  core/src/main/java/org/apache/oozie/util/XLog.java 21e00c0 
  core/src/main/resources/oozie-default.xml 9788daf 
  core/src/test/java/org/apache/oozie/service/TestInstrumentationService.java 
b5206cf 
  
core/src/test/java/org/apache/oozie/service/TestMetricsInstrumentationService.java
 PRE-CREATION 
  core/src/test/java/org/apache/oozie/util/TestInstrumentation.java 6040b47 
  core/src/test/java/org/apache/oozie/util/TestMetricsInstrumentation.java 
PRE-CREATION 
  docs/src/site/twiki/AG_Install.twiki e343d7e 
  docs/src/site/twiki/WebServicesAPI.twiki 5769924 
  pom.xml b5e0e4e 
  webapp/src/main/webapp/index.jsp 3c7ffe5 
  webapp/src/main/webapp/oozie-console.js 764d888 

Diff: https://reviews.apache.org/r/21278/diff/


Testing
---

Unit tests, also verified in a cluster


Thanks,

Robert Kanter

[jira] [Created] (OOZIE-1848) Pig actions fail due to missing joda-time jar from pig sharelib

[Robert Kanter (JIRA)]

Robert Kanter created OOZIE-1848:


 Summary: Pig actions fail due to missing joda-time jar from pig 
sharelib
 Key: OOZIE-1848
 URL: https://issues.apache.org/jira/browse/OOZIE-1848
 Project: Oozie
  Issue Type: Bug
Affects Versions: trunk
Reporter: Robert Kanter
Priority: Blocker


When I try to run the pig example, I get this Exception in the launcher log:
{noformat}
java.lang.NoClassDefFoundError: org/joda/time/ReadableInstant
at 
org.apache.pig.newplan.logical.relational.LogicalSchema$LogicalFieldSchema.merge(LogicalSchema.java:404)
at 
org.apache.pig.newplan.logical.relational.LogicalSchema.merge(LogicalSchema.java:768)
at 
org.apache.pig.newplan.logical.relational.LOGenerate.getSchema(LOGenerate.java:158)
at 
org.apache.pig.newplan.logical.optimizer.SchemaResetter.visit(SchemaResetter.java:123)
at 
org.apache.pig.newplan.logical.relational.LOGenerate.accept(LOGenerate.java:245)
at 
org.apache.pig.newplan.DependencyOrderWalker.walk(DependencyOrderWalker.java:75)
at 
org.apache.pig.newplan.logical.optimizer.SchemaResetter.visit(SchemaResetter.java:114)
at 
org.apache.pig.parser.LogicalPlanBuilder.buildForeachOp(LogicalPlanBuilder.java:1015)
at 
org.apache.pig.parser.LogicalPlanGenerator.foreach_clause(LogicalPlanGenerator.java:15870)
at 
org.apache.pig.parser.LogicalPlanGenerator.op_clause(LogicalPlanGenerator.java:1933)
at 
org.apache.pig.parser.LogicalPlanGenerator.general_statement(LogicalPlanGenerator.java:1102)
at 
org.apache.pig.parser.LogicalPlanGenerator.statement(LogicalPlanGenerator.java:560)
at 
org.apache.pig.parser.LogicalPlanGenerator.query(LogicalPlanGenerator.java:421)
at 
org.apache.pig.parser.QueryParserDriver.parse(QueryParserDriver.java:188)
at org.apache.pig.PigServer$Graph.parseQuery(PigServer.java:1678)
at org.apache.pig.PigServer$Graph.access$000(PigServer.java:1411)
at org.apache.pig.PigServer.parseAndBuild(PigServer.java:344)
at org.apache.pig.PigServer.executeBatch(PigServer.java:369)
at org.apache.pig.PigServer.executeBatch(PigServer.java:355)
at 
org.apache.pig.tools.grunt.GruntParser.executeBatch(GruntParser.java:140)
at 
org.apache.pig.tools.grunt.GruntParser.parseStopOnError(GruntParser.java:202)
at 
org.apache.pig.tools.grunt.GruntParser.parseStopOnError(GruntParser.java:173)
at org.apache.pig.tools.grunt.Grunt.exec(Grunt.java:84)
at org.apache.pig.Main.run(Main.java:478)
at org.apache.pig.PigRunner.run(PigRunner.java:49)
at org.apache.oozie.action.hadoop.PigMain.runPigJob(PigMain.java:288)
at org.apache.oozie.action.hadoop.PigMain.run(PigMain.java:228)
at org.apache.oozie.action.hadoop.LauncherMain.run(LauncherMain.java:39)
at org.apache.oozie.action.hadoop.PigMain.main(PigMain.java:77)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at 
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at 
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at 
org.apache.oozie.action.hadoop.LauncherMapper.map(LauncherMapper.java:226)
at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:50)
at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:430)
at org.apache.hadoop.mapred.MapTask.run(MapTask.java:366)
at org.apache.hadoop.mapred.Child$4.run(Child.java:255)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:415)
at 
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1190)
at org.apache.hadoop.mapred.Child.main(Child.java:249)
Caused by: java.lang.ClassNotFoundException: org.joda.time.ReadableInstant
at java.net.URLClassLoader$1.run(URLClassLoader.java:366)
at java.net.URLClassLoader$1.run(URLClassLoader.java:355)
at java.security.AccessController.doPrivileged(Native Method)
at java.net.URLClassLoader.findClass(URLClassLoader.java:354)
at java.lang.ClassLoader.loadClass(ClassLoader.java:425)
at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:308)
at java.lang.ClassLoader.loadClass(ClassLoader.java:358)
... 42 more
{noformat}

It looks like we need to add the {{joda-time}} jar to the Pig sharelib



--
This message was sent by Atlassian JIRA
(v6.2#6252)

Re: Review Request 17720: OOZIE-1678 HA support for SLA

[Ryota Egashira]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/17720/
---

(Updated May 16, 2014, 11:46 p.m.)


Review request for oozie.


Changes
---

revised version based on comments.


Bugs: OOZIE-1678
https://issues.apache.org/jira/browse/OOZIE-1678


Repository: oozie-git


Description
---

https://issues.apache.org/jira/browse/OOZIE-1678


Diffs (updated)
-

  core/src/main/java/org/apache/oozie/command/coord/CoordChangeXCommand.java 
ea7df17 
  
core/src/main/java/org/apache/oozie/executor/jpa/BundleActionQueryExecutor.java 
d2331e8 
  core/src/main/java/org/apache/oozie/executor/jpa/BundleJobQueryExecutor.java 
319aea0 
  
core/src/main/java/org/apache/oozie/executor/jpa/CoordActionQueryExecutor.java 
f5304ca 
  core/src/main/java/org/apache/oozie/executor/jpa/CoordJobQueryExecutor.java 
1a6ded7 
  core/src/main/java/org/apache/oozie/executor/jpa/QueryExecutor.java 536743b 
  
core/src/main/java/org/apache/oozie/executor/jpa/SLARegistrationQueryExecutor.java
 e3b115f 
  core/src/main/java/org/apache/oozie/executor/jpa/SLASummaryQueryExecutor.java 
79d11ed 
  
core/src/main/java/org/apache/oozie/executor/jpa/WorkflowActionQueryExecutor.java
 9156a27 
  
core/src/main/java/org/apache/oozie/executor/jpa/WorkflowJobQueryExecutor.java 
e7d42e9 
  
core/src/main/java/org/apache/oozie/executor/jpa/sla/SLARegistrationGetJPAExecutor.java
 7b4e177 
  
core/src/main/java/org/apache/oozie/executor/jpa/sla/SLARegistrationGetOnRestartJPAExecutor.java
 1510daf 
  
core/src/main/java/org/apache/oozie/executor/jpa/sla/SLASummaryGetJPAExecutor.java
 1f7cb4d 
  core/src/main/java/org/apache/oozie/service/JPAService.java aba8709 
  core/src/main/java/org/apache/oozie/service/JobsConcurrencyService.java 
27c97e6 
  core/src/main/java/org/apache/oozie/service/ZKJobsConcurrencyService.java 
611b74c 
  core/src/main/java/org/apache/oozie/sla/SLACalcStatus.java ea53712 
  core/src/main/java/org/apache/oozie/sla/SLACalculatorMemory.java 618d899 
  core/src/main/java/org/apache/oozie/sla/SLAOperations.java b0da3dc 
  core/src/main/java/org/apache/oozie/sla/SLASummaryBean.java 0a70326 
  core/src/main/java/org/apache/oozie/sla/service/SLAService.java ea2983f 
  
core/src/test/java/org/apache/oozie/command/coord/TestCoordChangeXCommand.java 
bc24235 
  
core/src/test/java/org/apache/oozie/executor/jpa/TestSLARegistrationQueryExecutor.java
 00fb677 
  
core/src/test/java/org/apache/oozie/executor/jpa/TestSLASummaryQueryExecutor.java
 2e170a4 
  
core/src/test/java/org/apache/oozie/executor/jpa/TestWorkflowJobsCountNotForPurgeFromWorkflowParentIdJPAExecutor.java
 ea36720 
  core/src/test/java/org/apache/oozie/service/TestHASLAService.java 
PRE-CREATION 
  core/src/test/java/org/apache/oozie/sla/TestSLACalculationJPAExecutor.java 
9270aa2 
  core/src/test/java/org/apache/oozie/sla/TestSLACalculatorMemory.java 9a16722 
  core/src/test/java/org/apache/oozie/sla/TestSLAEventGeneration.java f3bfc29 
  core/src/test/java/org/apache/oozie/sla/TestSLAJobEventListener.java 3ce86ab 
  
core/src/test/java/org/apache/oozie/sla/TestSLARegistrationGetJPAExecutor.java 
40376a5 
  
core/src/test/java/org/apache/oozie/sla/TestSLARegistrationGetRecordsOnRestartJPAExecutor.java
 cd2fbae 
  core/src/test/java/org/apache/oozie/sla/TestSLAService.java 291d850 
  
core/src/test/java/org/apache/oozie/sla/TestSLASummaryGetOnRestartJPAExecutor.java
 3a9e72e 
  core/src/test/java/org/apache/oozie/test/ZKXTestCase.java 7bebaf0 
  core/src/test/resources/coord-action-sla.xml e88df6c 

Diff: https://reviews.apache.org/r/17720/diff/


Testing
---


Thanks,

Ryota Egashira

[jira] [Created] (OOZIE-1842) publish oozie artifacts

[Bowen Zhang (JIRA)]

Bowen Zhang created OOZIE-1842:
--

 Summary: publish oozie artifacts
 Key: OOZIE-1842
 URL: https://issues.apache.org/jira/browse/OOZIE-1842
 Project: Oozie
  Issue Type: New Feature
Reporter: Bowen Zhang
Assignee: Mona Chitnis






--
This message was sent by Atlassian JIRA
(v6.2#6252)

Re: Review Request 21423: OOZIE-1813 Add service to report/kill rogue bundles and coordinator jobs

[Purshotam Shah]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/21423/
---

(Updated May 17, 2014, 12:06 a.m.)


Review request for oozie.


Bugs: OOZIE-1813
https://issues.apache.org/jira/browse/OOZIE-1813


Repository: oozie-git


Description
---

Report will be sent in html format, where each job is clickable and will take 
you to job info. 

Coordinator id  Coordinator 
nameUser name   Group   Kill Status
000-140513184018277-oozie-puru-CCOORD-TEST  test
testg   Coord kill is disabled
001-140513184018277-oozie-puru-CCOORD-TEST  test
testg   Coord kill is disabled


Diffs (updated)
-

  core/src/main/java/org/apache/oozie/CoordinatorJobBean.java 7915698 
  core/src/main/java/org/apache/oozie/action/email/EmailActionExecutor.java 
332d02c 
  core/src/main/java/org/apache/oozie/command/wf/JobXCommand.java cafeaeb 
  core/src/main/java/org/apache/oozie/executor/jpa/CoordJobQueryExecutor.java 
1a6ded7 
  core/src/main/java/org/apache/oozie/service/AbandonedCoordCheckerService.java 
e69de29 
  
core/src/test/java/org/apache/oozie/command/coord/TestAbandonedCoordChecker.java
 e69de29 

Diff: https://reviews.apache.org/r/21423/diff/


Testing
---

UTC


Thanks,

Purshotam Shah

[jira] Subscription: Oozie Patch Available

[jira]

Issue Subscription
Filter: Oozie Patch Available (42 issues)

Subscriber: ooziedaily

Key Summary
OOZIE-1838  jdbc.connections.active sampler does not show up
https://issues.apache.org/jira/browse/OOZIE-1838
OOZIE-1830  Change hadoop-1 profile to use 1.2.1
https://issues.apache.org/jira/browse/OOZIE-1830
OOZIE-1829  URIHandlerService doesn't support URI schemes with query strings 
but no path segment
https://issues.apache.org/jira/browse/OOZIE-1829
OOZIE-1821  Oozie java action fails due to AlreadyBeingCreatedException
https://issues.apache.org/jira/browse/OOZIE-1821
OOZIE-1819  Avoid early queueing of CoordActionInputCheckXCommand
https://issues.apache.org/jira/browse/OOZIE-1819
OOZIE-1817  Oozie timers are not biased
https://issues.apache.org/jira/browse/OOZIE-1817
OOZIE-1816  LogInfo uses action name instead of id
https://issues.apache.org/jira/browse/OOZIE-1816
OOZIE-1813  Add service to report/kill rogue bundles and coordinator jobs
https://issues.apache.org/jira/browse/OOZIE-1813
OOZIE-1807  Make bundle change command synchronous
https://issues.apache.org/jira/browse/OOZIE-1807
OOZIE-1804  Improve documentation for Coordinator Specification
https://issues.apache.org/jira/browse/OOZIE-1804
OOZIE-1803  Improvement in Purge service
https://issues.apache.org/jira/browse/OOZIE-1803
OOZIE-1802  Support workflow action log
https://issues.apache.org/jira/browse/OOZIE-1802
OOZIE-1801  ZKLocksService instrumentation should say how many locks this 
server has
https://issues.apache.org/jira/browse/OOZIE-1801
OOZIE-1793  Improve find bugs reporting for Oozie
https://issues.apache.org/jira/browse/OOZIE-1793
OOZIE-1783  Sharelib purging only occurs at Oozie startup
https://issues.apache.org/jira/browse/OOZIE-1783
OOZIE-1782  Workflow path not found is thrown as SC_UNAUTHORIZED
https://issues.apache.org/jira/browse/OOZIE-1782
OOZIE-1775  TestEventGeneration.testCoordinatorActionEvent is failing and 
CoordRerunX should generate event
https://issues.apache.org/jira/browse/OOZIE-1775
OOZIE-1760  Set default shared lib dir for java action
https://issues.apache.org/jira/browse/OOZIE-1760
OOZIE-1741  Add new coord EL function to get input partitions value string
https://issues.apache.org/jira/browse/OOZIE-1741
OOZIE-1724  Make it easier to specify the HCat hive-site.xml for the Oozie 
Server
https://issues.apache.org/jira/browse/OOZIE-1724
OOZIE-1689  HA support for OOZIE-7(Ability to view the log information 
corresponding to particular coordinator action)
https://issues.apache.org/jira/browse/OOZIE-1689
OOZIE-1688  New configuration to specify server-server authentication type.
https://issues.apache.org/jira/browse/OOZIE-1688
OOZIE-1686  Typo in DG_CommandLineTool
https://issues.apache.org/jira/browse/OOZIE-1686
OOZIE-1685  Oozie doesn’t process correctly workflows with a non-default name 
node
https://issues.apache.org/jira/browse/OOZIE-1685
OOZIE-1676  Instrumentation and Configuration over the REST API and Web UI 
should include all Oozie servers
https://issues.apache.org/jira/browse/OOZIE-1676
OOZIE-1659  oozie-site is missing email-action-0.2 schema
https://issues.apache.org/jira/browse/OOZIE-1659
OOZIE-1654  Fix typo (inteval to interval)
https://issues.apache.org/jira/browse/OOZIE-1654
OOZIE-1638  Action retry does not use default retry max count.
https://issues.apache.org/jira/browse/OOZIE-1638
OOZIE-1636  OOZIE_SYS table engine should be innodb
https://issues.apache.org/jira/browse/OOZIE-1636
OOZIE-1624  Exclusion pattern for sharelib.
https://issues.apache.org/jira/browse/OOZIE-1624
OOZIE-1602  pom.xml should take use of the profiles when building hadooplibs
https://issues.apache.org/jira/browse/OOZIE-1602
OOZIE-1579  Add basic HTTP auth to Oozie CLI
https://issues.apache.org/jira/browse/OOZIE-1579
OOZIE-1567  Provide a wait tool in Oozie
https://issues.apache.org/jira/browse/OOZIE-1567
OOZIE-1533  Coordinator action materialization is too slow due to coarse job 
level locks
https://issues.apache.org/jira/browse/OOZIE-1533
OOZIE-1532  Purging should remove completed children job for long running 
coordinator jobs
https://issues.apache.org/jira/browse/OOZIE-1532
OOZIE-1376  Extending Oozie ACLs like admin groups and proxy users to support 
both groups and users
https://issues.apache.org/jira/browse/OOZIE-1376
OOZIE-1369  OozieDBCLI code should not hardcode the Oozie table filenames
https://issues.apache.org/jira/browse/OOZIE-1369
OOZIE-1319  "LAST_ONLY" in execution control for coordinator job still runs all 
the actions
https://issues.apache.org/jira/brow

[jira] [Updated] (OOZIE-1813) Add service to report/kill rogue bundles and coordinator jobs

[Purshotam Shah (JIRA)]

 [ 
https://issues.apache.org/jira/browse/OOZIE-1813?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Purshotam Shah updated OOZIE-1813:
--

Attachment: OOZIE-1813-V3.patch

> Add service to report/kill rogue bundles and coordinator jobs
> -
>
> Key: OOZIE-1813
> URL: https://issues.apache.org/jira/browse/OOZIE-1813
> Project: Oozie
>  Issue Type: Bug
>Reporter: Purshotam Shah
>Assignee: Purshotam Shah
> Attachments: OOZIE-1813-V2.patch, OOZIE-1813-V3.patch
>
>
> People leave their test coordinator and bundle jobs without ever killing them
> and they just eat up resources heavily. We should have a service which 
> periodically check for abandoned coords and report/kill them.
> We can add multiple logic to this like ( number of consecutive 
> failed/timedout action, total number of failed/timedout action). 
> To start with if number of coord action with failed/timedout status > defined 
> value, then coord is considered to be rogue.



--
This message was sent by Atlassian JIRA
(v6.2#6252)

[jira] [Commented] (OOZIE-1849) If the underlying job finishes while a Workflow is suspended, Oozie can take a while to realize it

[Robert Kanter (JIRA)]

[ 
https://issues.apache.org/jira/browse/OOZIE-1849?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14000577#comment-14000577
 ] 

Robert Kanter commented on OOZIE-1849:
--

I think we should do the second solution.  When you SUSPEND a workflow, any 
currently RUNNING actions are still listed as RUNNING, so I think it makes 
sense that they could transition to a terminal state from there.  The workflow 
is SUSPENDED, not the current action.

> If the underlying job finishes while a Workflow is suspended, Oozie can take 
> a while to realize it
> --
>
> Key: OOZIE-1849
> URL: https://issues.apache.org/jira/browse/OOZIE-1849
> Project: Oozie
>  Issue Type: Improvement
>  Components: core
>Affects Versions: 4.0.1
>Reporter: Robert Kanter
>Assignee: Robert Kanter
>
> Suppose you have a Workflow and you suspend it while one of the actions is 
> still RUNNING.  The underlying MR/Pig/etc job will continue running (as 
> expected, because we can't pause those).  However, if that job finishes while 
> the workflow is SUSPENDED, the CallbackServlet will receive the callback, but 
> the ActionCheckXCommand won't update the action:
> {noformat}
> 2014-05-16 17:40:57,959  INFO CallbackServlet:541 - SERVER[rkanter-mbp.local] 
> USER[-] GROUP[-] TOKEN[-] APP[-] JOB[002-140516173529928-oozie-rkan-W] 
> ACTION[002-140516173529928-oozie-rkan-W@mr-node] callback for action 
> [002-140516173529928-oozie-rkan-W@mr-node]
> 2014-05-16 17:40:57,985  WARN ActionCheckXCommand:544 - 
> SERVER[rkanter-mbp.local] USER[rkanter] GROUP[-] TOKEN[] APP[map-reduce-wf] 
> JOB[002-140516173529928-oozie-rkan-W] 
> ACTION[002-140516173529928-oozie-rkan-W@mr-node] E0818: Action 
> [002-140516173529928-oozie-rkan-W@mr-node] status is running but WF Job 
> [002-140516173529928-oozie-rkan-W] status is [SUSPENDED]. Expected status 
> is RUNNING., Error Code: E0818
> {noformat}
> If you then resume the workflow, the action will stay RUNNING for up to 10 
> minutes (the default fallback polling interval), at which point the 
> ActionCheckerService will run an ActionCheckXCommand that will pass, check 
> the job, and finally mark the action as SUCCESSFUL.
> We should fix this by one of the following:
> # ResumeXCommand should also queue a ActionCheckXCommand (if the workflow was 
> SUSPENDED) so we don't have to wait for the ActionCheckerService
> # ActionCheckXCommand's precondition check should allow SUSPENDED workflows



--
This message was sent by Atlassian JIRA
(v6.2#6252)

Re: Review Request 21278: OOZIE-1817: Oozie timers are not biased

[Robert Kanter]

> On May 15, 2014, 8:55 p.m., Gilad Wolff wrote:
> > core/src/main/java/org/apache/oozie/util/MetricsInstrumentation.java, line 
> > 147
> > 
> >
> > >= 0 ? what if something takes less than 1ms?
> 
> Robert Kanter wrote:
> The reason I added that was that if you add a cron that was never 
> actually run, without the if statement, you'll add an extra 0 value to the 
> timer.  I'll look into what to do about this.

I've removed that check.  The proper usage pattern of the cron object is to run 
them before adding them.


- Robert


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/21278/#review42740
---


On May 13, 2014, 5:40 p.m., Robert Kanter wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/21278/
> ---
> 
> (Updated May 13, 2014, 5:40 p.m.)
> 
> 
> Review request for oozie.
> 
> 
> Bugs: OOZIE-1817
> https://issues.apache.org/jira/browse/OOZIE-1817
> 
> 
> Repository: oozie-git
> 
> 
> Description
> ---
> 
> See 
> https://issues.apache.org/jira/browse/OOZIE-1817?focusedCommentId=13993838&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-13993838
> 
> 
> Diffs
> -
> 
>   client/src/main/java/org/apache/oozie/client/rest/RestConstants.java 
> 808f9b2 
>   core/pom.xml c935dd7 
>   core/src/main/java/org/apache/oozie/command/Command.java d3f1011 
>   core/src/main/java/org/apache/oozie/command/XCommand.java b712ce0 
>   
> core/src/main/java/org/apache/oozie/command/coord/CoordActionInputCheckXCommand.java
>  e8667c1 
>   
> core/src/main/java/org/apache/oozie/command/coord/CoordActionMaterializeCommand.java
>  6962fb2 
>   
> core/src/main/java/org/apache/oozie/command/coord/CoordMaterializeTransitionXCommand.java
>  57cbb34 
>   core/src/main/java/org/apache/oozie/service/InstrumentationService.java 
> 80437b1 
>   
> core/src/main/java/org/apache/oozie/service/MetricsInstrumentationService.java
>  PRE-CREATION 
>   core/src/main/java/org/apache/oozie/service/Services.java 5feac7b 
>   core/src/main/java/org/apache/oozie/service/XLogService.java 403a089 
>   core/src/main/java/org/apache/oozie/servlet/BaseAdminServlet.java 29d7bd6 
>   core/src/main/java/org/apache/oozie/servlet/JsonRestServlet.java 5c05acd 
>   core/src/main/java/org/apache/oozie/servlet/V0AdminServlet.java 97bd81c 
>   core/src/main/java/org/apache/oozie/servlet/V1AdminServlet.java a47f737 
>   core/src/main/java/org/apache/oozie/servlet/V2AdminServlet.java 002a367 
>   core/src/main/java/org/apache/oozie/util/MetricsInstrumentation.java 
> PRE-CREATION 
>   core/src/main/java/org/apache/oozie/util/XLogReporter.java PRE-CREATION 
>   core/src/test/java/org/apache/oozie/service/TestInstrumentationService.java 
> b5206cf 
>   
> core/src/test/java/org/apache/oozie/service/TestMetricsInstrumentationService.java
>  PRE-CREATION 
>   core/src/test/java/org/apache/oozie/util/TestInstrumentation.java 6040b47 
>   core/src/test/java/org/apache/oozie/util/TestMetricsInstrumentation.java 
> PRE-CREATION 
>   docs/src/site/twiki/AG_Install.twiki e343d7e 
>   docs/src/site/twiki/WebServicesAPI.twiki 6730255 
>   pom.xml b5e0e4e 
>   webapp/src/main/webapp/index.jsp 3c7ffe5 
>   webapp/src/main/webapp/oozie-console.js 764d888 
> 
> Diff: https://reviews.apache.org/r/21278/diff/
> 
> 
> Testing
> ---
> 
> Unit tests, also verified in a cluster
> 
> 
> Thanks,
> 
> Robert Kanter
> 
>

[jira] [Updated] (OOZIE-1817) Oozie timers are not biased

[Robert Kanter (JIRA)]

 [ 
https://issues.apache.org/jira/browse/OOZIE-1817?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Robert Kanter updated OOZIE-1817:
-

Attachment: OOZIE-1817.patch

The new patch:
- Does a bunch of misc improvements that Gilad suggested on ReviewBoard
- Removed XLogReporter for printing to the oozie-instrumentation log and 
instead does a pretty-print of the JSON shown on the metrics endpoint
-- I was able to remove XLogReporter, which being a copy-paste from Metrics 
code is "ugly"
-- The output doesn't have all the extra log4j prefix stuff now, which is more 
similar to the Instrumentation's output
-- Pretty-printing the JSON is actually twice as fast as XLogReporter.  I 
temporarily added some timers to check this.  Here's some of the metrics after 
running for ~30min:
||Metric|||XLogReporter (ms)||Pretty-Print JSON (ms)||
||min|31.0|10.0|
||max|274.0|170.0|
||mean|48.46|21.96551724137931|
||stddev|43.608551307357565|28.96239280591876|
-- This allowed me to simplify the code a lot

> Oozie timers are not biased
> ---
>
> Key: OOZIE-1817
> URL: https://issues.apache.org/jira/browse/OOZIE-1817
> Project: Oozie
>  Issue Type: Improvement
>  Components: monitoring
>Affects Versions: 4.0.0
>Reporter: Gilad Wolff
>Assignee: Robert Kanter
> Attachments: OOZIE-1817.patch, OOZIE-1817.patch, OOZIE-1817.patch, 
> OOZIE-1817.patch, OOZIE-1817.patch
>
>
> Oozie timers are not biased, that is, the statistical metrics they expose are 
> over the run-time of the Oozie server instead of a window of time. This makes 
> them not very useful especially after the server has been running for a while 
> (codehale has very efficient and easy to use biased histograms that can be 
> used instead).



--
This message was sent by Atlassian JIRA
(v6.2#6252)

Re: Review Request 17720: OOZIE-1678 HA support for SLA

[Ryota Egashira]

> On May 14, 2014, 4:53 p.m., Rohini Palaniswamy wrote:
> > core/src/main/java/org/apache/oozie/sla/SLACalculatorMemory.java, line 127
> > 
> >
> > Shouldn't we be calling slaCalcStatus.acquireLock() instead of calling 
> > directly?

because this case, need to get lock when SLACalcStatus not created yet at 
initialization


- Ryota


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/17720/#review42969
---


On May 16, 2014, 11:46 p.m., Ryota Egashira wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/17720/
> ---
> 
> (Updated May 16, 2014, 11:46 p.m.)
> 
> 
> Review request for oozie.
> 
> 
> Bugs: OOZIE-1678
> https://issues.apache.org/jira/browse/OOZIE-1678
> 
> 
> Repository: oozie-git
> 
> 
> Description
> ---
> 
> https://issues.apache.org/jira/browse/OOZIE-1678
> 
> 
> Diffs
> -
> 
>   core/src/main/java/org/apache/oozie/command/coord/CoordChangeXCommand.java 
> ea7df17 
>   
> core/src/main/java/org/apache/oozie/executor/jpa/BundleActionQueryExecutor.java
>  d2331e8 
>   
> core/src/main/java/org/apache/oozie/executor/jpa/BundleJobQueryExecutor.java 
> 319aea0 
>   
> core/src/main/java/org/apache/oozie/executor/jpa/CoordActionQueryExecutor.java
>  f5304ca 
>   core/src/main/java/org/apache/oozie/executor/jpa/CoordJobQueryExecutor.java 
> 1a6ded7 
>   core/src/main/java/org/apache/oozie/executor/jpa/QueryExecutor.java 536743b 
>   
> core/src/main/java/org/apache/oozie/executor/jpa/SLARegistrationQueryExecutor.java
>  e3b115f 
>   
> core/src/main/java/org/apache/oozie/executor/jpa/SLASummaryQueryExecutor.java 
> 79d11ed 
>   
> core/src/main/java/org/apache/oozie/executor/jpa/WorkflowActionQueryExecutor.java
>  9156a27 
>   
> core/src/main/java/org/apache/oozie/executor/jpa/WorkflowJobQueryExecutor.java
>  e7d42e9 
>   
> core/src/main/java/org/apache/oozie/executor/jpa/sla/SLARegistrationGetJPAExecutor.java
>  7b4e177 
>   
> core/src/main/java/org/apache/oozie/executor/jpa/sla/SLARegistrationGetOnRestartJPAExecutor.java
>  1510daf 
>   
> core/src/main/java/org/apache/oozie/executor/jpa/sla/SLASummaryGetJPAExecutor.java
>  1f7cb4d 
>   core/src/main/java/org/apache/oozie/service/JPAService.java aba8709 
>   core/src/main/java/org/apache/oozie/service/JobsConcurrencyService.java 
> 27c97e6 
>   core/src/main/java/org/apache/oozie/service/ZKJobsConcurrencyService.java 
> 611b74c 
>   core/src/main/java/org/apache/oozie/sla/SLACalcStatus.java ea53712 
>   core/src/main/java/org/apache/oozie/sla/SLACalculatorMemory.java 618d899 
>   core/src/main/java/org/apache/oozie/sla/SLAOperations.java b0da3dc 
>   core/src/main/java/org/apache/oozie/sla/SLASummaryBean.java 0a70326 
>   core/src/main/java/org/apache/oozie/sla/service/SLAService.java ea2983f 
>   
> core/src/test/java/org/apache/oozie/command/coord/TestCoordChangeXCommand.java
>  bc24235 
>   
> core/src/test/java/org/apache/oozie/executor/jpa/TestSLARegistrationQueryExecutor.java
>  00fb677 
>   
> core/src/test/java/org/apache/oozie/executor/jpa/TestSLASummaryQueryExecutor.java
>  2e170a4 
>   
> core/src/test/java/org/apache/oozie/executor/jpa/TestWorkflowJobsCountNotForPurgeFromWorkflowParentIdJPAExecutor.java
>  ea36720 
>   core/src/test/java/org/apache/oozie/service/TestHASLAService.java 
> PRE-CREATION 
>   core/src/test/java/org/apache/oozie/sla/TestSLACalculationJPAExecutor.java 
> 9270aa2 
>   core/src/test/java/org/apache/oozie/sla/TestSLACalculatorMemory.java 
> 9a16722 
>   core/src/test/java/org/apache/oozie/sla/TestSLAEventGeneration.java f3bfc29 
>   core/src/test/java/org/apache/oozie/sla/TestSLAJobEventListener.java 
> 3ce86ab 
>   
> core/src/test/java/org/apache/oozie/sla/TestSLARegistrationGetJPAExecutor.java
>  40376a5 
>   
> core/src/test/java/org/apache/oozie/sla/TestSLARegistrationGetRecordsOnRestartJPAExecutor.java
>  cd2fbae 
>   core/src/test/java/org/apache/oozie/sla/TestSLAService.java 291d850 
>   
> core/src/test/java/org/apache/oozie/sla/TestSLASummaryGetOnRestartJPAExecutor.java
>  3a9e72e 
>   core/src/test/java/org/apache/oozie/test/ZKXTestCase.java 7bebaf0 
>   core/src/test/resources/coord-action-sla.xml e88df6c 
> 
> Diff: https://reviews.apache.org/r/17720/diff/
> 
> 
> Testing
> ---
> 
> 
> Thanks,
> 
> Ryota Egashira
> 
>

[jira] [Commented] (OOZIE-1829) URIHandlerService doesn't support URI schemes with query strings but no path segment

[Hadoop QA (JIRA)]

[ 
https://issues.apache.org/jira/browse/OOZIE-1829?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13999264#comment-13999264
 ] 

Hadoop QA commented on OOZIE-1829:
--

Testing JIRA OOZIE-1829

Cleaning local git workspace



{color:red}-1{color} Patch failed to apply to head of branch



> URIHandlerService doesn't support URI schemes with query strings but no path 
> segment
> 
>
> Key: OOZIE-1829
> URL: https://issues.apache.org/jira/browse/OOZIE-1829
> Project: Oozie
>  Issue Type: Bug
>  Components: core
>Affects Versions: 4.0.1
>Reporter: Ben Roling
> Attachments: OOZIE-1829.patch
>
>
> While working on a prototype of integration between Oozie and the Kite SDK 
> (see https://issues.cloudera.org/browse/CDK-385), I came to find that 
> URIHandlerService.getAuthorityWithSchema(String uri) doesn't support URI 
> schemes where there is a query string, but no path segment.
> I am currently prototyping Kite Dataset URIs and in my prototype, a Dataset 
> URI for a dataset in a Hive/HCatalog DatasetRepository with managed 
> Hive/HCatalog tables could look like this:
> repo:hive://localhost:9043?dataset-name=Person&partition-key=\[201405091300\]
> I am attempting to create an Oozie dataset around this Kite dataset and to 
> make that happen I have implemented Oozie's URIHandler API for the Kite 
> "repo" URI scheme.  When I attempted to run my first coordinator, it failed.
> The coordinator has the following dataset definition:
> {code}
>  initial-instance="2014-04-24T00:00Z" timezone="UTC">
>   
> repo:hive://localhost:9083?dataset-name=Person&partition-key=[${YEAR}${MONTH}${DAY}${HOUR}${MINUTE}]
>   
> 
> {code}
> This dataset is used as an output of the coordinator.
> When the coordinator is submitted it fails with the following exception:
> {code}
> 2014-05-09 10:57:34,991 ERROR 
> org.apache.oozie.command.coord.CoordMaterializeTransitionXCommand: 
> SERVER[localhost.localdomain] USER[cloudera] GROUP[-] TOKEN[] APP[Person-c] 
> JOB[013-140508121805317-oozie-oozi-C] ACTION[-] Exception occurred:E0906: 
> URI parsing error : 
> repo:hive://localhost:9083?dataset-name=Person&partition-key=[${YEAR}${MONTH}${DAY}${HOUR}${MINUTE}]
>  Making the job failed 
> org.apache.oozie.dependency.URIHandlerException: E0906: URI parsing error : 
> repo:hive://localhost:9083?dataset-name=Person&partition-key=[${YEAR}${MONTH}${DAY}${HOUR}${MINUTE}]
>   at 
> org.apache.oozie.service.URIHandlerService.getAuthorityWithScheme(URIHandlerService.java:216)
>   at 
> org.apache.oozie.command.coord.CoordCommandUtils.materializeDataEvents(CoordCommandUtils.java:582)
>   at 
> org.apache.oozie.command.coord.CoordCommandUtils.materializeOneInstance(CoordCommandUtils.java:451)
>   at 
> org.apache.oozie.command.coord.CoordMaterializeTransitionXCommand.materializeActions(CoordMaterializeTransitionXCommand.java:386)
>   at 
> org.apache.oozie.command.coord.CoordMaterializeTransitionXCommand.materialize(CoordMaterializeTransitionXCommand.java:267)
>   at 
> org.apache.oozie.command.MaterializeTransitionXCommand.execute(MaterializeTransitionXCommand.java:72)
>   at 
> org.apache.oozie.command.MaterializeTransitionXCommand.execute(MaterializeTransitionXCommand.java:28)
>   at org.apache.oozie.command.XCommand.call(XCommand.java:280)
>   at 
> org.apache.oozie.service.CallableQueueService$CallableWrapper.run(CallableQueueService.java:174)
>   at 
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
>   at 
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
>   at java.lang.Thread.run(Thread.java:744)
> Caused by: java.net.URISyntaxException: Illegal character in opaque part at 
> index 63: 
> repo:hive://localhost:9083?dataset-name=Person&partition-key=[${YEAR}${MONTH}${DAY}${HOUR}${MINUTE}]
>   at java.net.URI$Parser.fail(URI.java:2829)
>   at java.net.URI$Parser.checkChars(URI.java:3002)
>   at java.net.URI$Parser.parse(URI.java:3039)
>   at java.net.URI.(URI.java:595)
>   at 
> org.apache.oozie.service.URIHandlerService.getAuthorityWithScheme(URIHandlerService.java:209)
>   ... 11 more
> {code}
> The problem is that URIHandlerService.getAuthorityWithScheme(String uri) 
> doesn't consider the possibility that the URI might have a query string and 
> no path segment.  As a result, it ends up trying to create a URI from the 
> entire URI template, which blows up due to $ in the template parameters.



--
This message was sent by Atlassian JIRA
(v6.2#6252)

Re: publish oozie artifacts

[Mona Chitnis]

Thanks for the JIRA. Getting started on this task today and will update on
JIRA for feedback.

‹
Mona

On 5/15/14, 10:42 AM, "bowen zhang"  wrote:

>Hi Mona,
>I created this ticket:https://issues.apache.org/jira/browse/OOZIE-1842.
>Do you have any updates on this?
>Bowen
>
>On Tuesday, April 29, 2014 11:52 AM, Mona Chitnis 
>wrote:
> 
>Thanks Robert. I'll look into it this week.--Monahref="https://overview.mail.yahoo.com?.src=iOS";>Sent from Yahoo
>Mail for iPhone

[jira] [Created] (OOZIE-1847) HA - Oozie servers should shutdown (or go in safe mode) in case ZK failure

[Purshotam Shah (JIRA)]

Purshotam Shah created OOZIE-1847:
-

 Summary: HA - Oozie servers should shutdown (or go in safe mode) 
in case ZK failure
 Key: OOZIE-1847
 URL: https://issues.apache.org/jira/browse/OOZIE-1847
 Project: Oozie
  Issue Type: Bug
Reporter: Purshotam Shah






--
This message was sent by Atlassian JIRA
(v6.2#6252)

Re: What are the plans for an Oozie 4.1.0 release?

[bowen zhang]

For backward compatibility, we need to resolve 
https://issues.apache.org/jira/browse/OOZIE-1789 for this release since this is 
about sharelib installation.

 
On Friday, May 16, 2014 3:58 PM, Alejandro Abdelnur  wrote:
 


the fact that other projects stay un the minor version to 'allow' incompat 
changes whenever they want, well it is a matter of those projects and their 
communities. 

i'm proud that oozie has a great track of record in that front, including among 
(yes among, not only between) major versions. and we should keep it that way. 

regarding db schema upgrades, until now we were saying that it should be only 
in major upgrades.  given the fact that the dbtools manages upgrades in a very 
unpainful way, and this has happened for a while now, i think it should be fine 
for a minor release to require a dbtool run. we just have to make sure this is 
documented, and that if the db has not been upgraded the new oozie does not 
start (which i think it is already happening)

thx


Alejandro
(phone typing)


> On May 14, 2014, at 16:48, Robert Kanter  wrote:
> 
> During the 3.x releases, we had an optional database change for MySQL to
> change TEXT to MEDIUMTEXT; I don't think there was a required one, but I
> don't remember.
> 
> In any case, that's a good point about the other projects and how they
> handle these things.  So I'd agree then that we can have a database
> upgrade.  Bowen, please make sure to verify that the database upgrade works
> for 2.x to 4.1, 3.x to 4.1, and 4.0 to 4.1 and for each type.
> 
> 
> On Wed, May 14, 2014 at 11:07 AM, Rohini Palaniswamy <
> rohini.adi...@gmail.com> wrote:
> 
>> Not sure. I remember someone mentioning that schema upgrade should have
>> major version incremented in case of Oozie. But there was one release (3.1
>> ->3.2 or something before I started working on Oozie) which had schema
>> change just incrementing minor version. If there was an already agreed
>> procedure we should follow that else I am fine either way and ok going with
>> 4.1 (as long it is atleast minor version and not patch version that we are
>> changing) considering every other hadoop project still have major version
>> as 0 and only keep updating minor version. For eg: hive 0.12 -> 0.13 is a
>> major release with schema changes. If we keep incrementing major version
>> very often it feels slightly weird when comparing to other hadoop projects
>> which are still in 0.x :).
>> 
>> 
>> On Tue, May 13, 2014 at 3:22 PM, Robert Kanter 
>> wrote:
>> 
>>> I agree; we'd also have to likely make two versions of each patch going
>>> forward for 4.x and 5.x/trunk otherwise.
>>> But are we "allowed" to make Oozie 4.0.x --> 4.1.0 require a database
>>> upgrade?
>>> 
>>> 
>>> On Tue, May 13, 2014 at 2:30 PM, Rohini Palaniswamy <
>>> rohini.adi...@gmail.com
 wrote:
>>> 
 It might be tough work as the CLOB->BLOB conversion change touches a
>> lot
>>> of
 classes.
 
 
 On Sun, May 11, 2014 at 9:25 PM, Robert Kanter 
 wrote:
 
> Hi everyone,
> 
> Should we start thinking about a 4.1.0 release?  It's been a while
>>> since
> 4.0.0 (4.0.1 was to fix some critical things like not being able to
> compile) and I think we have about 200 additional JIRAs in master.
> 
> Are there any specific features that we'd want to put in 4.1.0 and
>> wait
> for?
> 
> Another thing to keep in mind is that we can't just take everything
>> in
> master; I forget what exactly, but there's some JIRAs that change the
> database that we'll have to save for 5.0.0.
> 
> Does anybody want to volunteer to drive the 4.1.0 release?
> 
> 
> thanks
> - Robert
>> 

[jira] [Updated] (OOZIE-1813) Add service to report/kill rogue bundles and coordinator jobs

[Purshotam Shah (JIRA)]

 [ 
https://issues.apache.org/jira/browse/OOZIE-1813?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Purshotam Shah updated OOZIE-1813:
--

Attachment: OOZIE-1813-V3.patch

> Add service to report/kill rogue bundles and coordinator jobs
> -
>
> Key: OOZIE-1813
> URL: https://issues.apache.org/jira/browse/OOZIE-1813
> Project: Oozie
>  Issue Type: Bug
>Reporter: Purshotam Shah
>Assignee: Purshotam Shah
> Attachments: OOZIE-1813-V2.patch, OOZIE-1813-V3.patch
>
>
> People leave their test coordinator and bundle jobs without ever killing them
> and they just eat up resources heavily. We should have a service which 
> periodically check for abandoned coords and report/kill them.
> We can add multiple logic to this like ( number of consecutive 
> failed/timedout action, total number of failed/timedout action). 
> To start with if number of coord action with failed/timedout status > defined 
> value, then coord is considered to be rogue.



--
This message was sent by Atlassian JIRA
(v6.2#6252)

Build failed in Jenkins: oozie-trunk-precommit-build #1239

[Apache Jenkins Server]

See 

--
[...truncated 9675 lines...]
[INFO] share already added, skipping
[INFO] share/lib already added, skipping
[INFO]  already added, skipping
[INFO] share already added, skipping
[INFO] share/lib already added, skipping
[INFO]  already added, skipping
[INFO] share already added, skipping
[INFO] share/lib already added, skipping
[INFO]  already added, skipping
[INFO] share already added, skipping
[INFO] share/lib already added, skipping
[INFO]  already added, skipping
[INFO] share already added, skipping
[INFO] share/lib already added, skipping
[INFO]  already added, skipping
[INFO] share already added, skipping
[INFO] share/lib already added, skipping
[INFO] 
[INFO] 
[INFO] Building Apache Oozie Tools 4.1.0-SNAPSHOT
[INFO] 
[WARNING] Failure to transfer asm:asm/maven-metadata.xml from 
file:../../local.repository/trunk was cached in the local repository, 
resolution will not be reattempted until the update interval of 
local.repository has elapsed or updates are forced. Original error: Could not 
transfer metadata asm:asm/maven-metadata.xml from/to local.repository 
(file:../../local.repository/trunk): No connector available to access 
repository local.repository (file:../../local.repository/trunk) of type legacy 
using the available factories WagonRepositoryConnectorFactory
[WARNING] Failure to transfer asm:asm/maven-metadata.xml from 
file:../../local.repository/trunk was cached in the local repository, 
resolution will not be reattempted until the update interval of 
local.repository has elapsed or updates are forced. Original error: Could not 
transfer metadata asm:asm/maven-metadata.xml from/to local.repository 
(file:../../local.repository/trunk): No connector available to access 
repository local.repository (file:../../local.repository/trunk) of type legacy 
using the available factories WagonRepositoryConnectorFactory
[INFO] 
[INFO] --- maven-resources-plugin:2.5:resources (default-resources) @ 
oozie-tools ---
[debug] execute contextualize
[INFO] Using 'UTF-8' encoding to copy filtered resources.
[INFO] skip non existing resourceDirectory 

[INFO] 
[INFO] --- maven-compiler-plugin:2.3.2:compile (default-compile) @ oozie-tools 
---
[INFO] Nothing to compile - all classes are up to date
[INFO] 
[INFO] --- maven-resources-plugin:2.5:testResources (default-testResources) @ 
oozie-tools ---
[debug] execute contextualize
[INFO] Using 'UTF-8' encoding to copy filtered resources.
[INFO] Copying 2 resources
[INFO] 
[INFO] --- maven-compiler-plugin:2.3.2:testCompile (default-testCompile) @ 
oozie-tools ---
[INFO] Nothing to compile - all classes are up to date
[INFO] 
[INFO] --- maven-surefire-plugin:2.12:test (default-test) @ oozie-tools ---
[INFO] Tests are skipped.
[INFO] 
[INFO] --- maven-jar-plugin:2.3.1:jar (default-jar) @ oozie-tools ---
[INFO] Building jar: 

[INFO] 
[INFO] --- maven-assembly-plugin:2.2.1:single (default-cli) @ oozie-tools ---
[INFO] Reading assembly descriptor: ../src/main/assemblies/tools.xml
[WARNING] The following patterns were never triggered in this artifact 
exclusion filter:
o  '*:*:pom:*'

[INFO] Copying files to 

[WARNING] Assembly file: 

 is not a regular file (it may be a directory). It cannot be attached to the 
project build for installation or deployment.
[INFO] 
[INFO] 
[INFO] Building Apache Oozie MiniOozie 4.1.0-SNAPSHOT
[INFO] 
[INFO] 
[INFO] --- maven-resources-plugin:2.5:resources (default-resources) @ 
oozie-mini ---
[debug] execute contextualize
[INFO] Using 'UTF-8' encoding to copy filtered resources.
[INFO] skip non existing resourceDirectory 

[INFO] 
[INFO] --- maven-compiler-plugin:2.3.2:compile (default-compile) @ oozie-mini 
---
[INFO] No sources to compile
[INFO] 
[INFO] --- maven-resources-plugin:2.5:testResources (default-testResources) @ 
oozie-mini ---
[debug] execute contextualize
[INFO] Using 'UTF-8' encoding to copy filtered resources.
[INFO] Copying 3 resources
[INFO] 
[INFO] --- maven-compiler-plugin:2.3.2:testCompile (

Re: Review Request 21278: OOZIE-1817: Oozie timers are not biased

[Gilad Wolff]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/21278/#review42740
---

Ship it!


Looks great, thanks for doing this. I have a few comments nothing major. 

G.


core/src/main/java/org/apache/oozie/command/Command.java


So does it make sense to normalize command metrics? Currently Command and 
XCommand create different metrics. XCommand has metrics for failure (xexception 
and exception) and an 'executions' counter. Command does not have these.



core/src/main/java/org/apache/oozie/command/XCommand.java


nit pick: acquireLock does the same, remove it from there as well?



core/src/main/java/org/apache/oozie/command/XCommand.java


does it make sense to add a counter here (it can be the 'exceptions' one or 
a new one?



core/src/main/java/org/apache/oozie/command/XCommand.java


I think it makes more sense to only time the executions that succeeded. I 
think that in general you don't want the timer to get data from failed 
executions that may be very short.



core/src/main/java/org/apache/oozie/command/coord/CoordMaterializeTransitionXCommand.java


move it into the try so you won't add failed runs?



core/src/main/java/org/apache/oozie/service/InstrumentationService.java


nit: can't you use (instrumentation == null) for this?



core/src/main/java/org/apache/oozie/service/InstrumentationService.java


nit: what if you are already initialized? Is that okay to call init twice? 
Also, if you remove the isEnabled you'll need to assign the member at the end 
to make sure that you only initializes it when init was successful.



core/src/main/java/org/apache/oozie/service/MetricsInstrumentationService.java


super nit: rename constant to include the unit?



core/src/main/java/org/apache/oozie/service/MetricsInstrumentationService.java


nit: you don't really need this member. Use the base class member instead?



core/src/main/java/org/apache/oozie/service/MetricsInstrumentationService.java


same comment about using the instrumentation member itself.



core/src/main/java/org/apache/oozie/servlet/BaseAdminServlet.java


nit: Why not have the same signature and pass the Instrumentation object?



core/src/main/java/org/apache/oozie/servlet/V2AdminServlet.java


nit: I am not sure if it is possible but the actual metric instrumentation 
instance can change (by calling destroy and init on the metric instrumentation 
service). So maybe instead of caching a copy of the instrumentation object 
check the object that is passed in the methods instead?



core/src/main/java/org/apache/oozie/util/MetricsInstrumentation.java


super nit: this controls not only the logging frequency but also the actual 
collection frequency, no?



core/src/main/java/org/apache/oozie/util/MetricsInstrumentation.java


nit: comment or add a readability constant for the no-samples boolean?



core/src/main/java/org/apache/oozie/util/MetricsInstrumentation.java


can the sliding time window be made configurable?



core/src/main/java/org/apache/oozie/util/MetricsInstrumentation.java


why not use loading caches for these as well?



core/src/main/java/org/apache/oozie/util/MetricsInstrumentation.java


nit: Maybe make it into a constant? I believe it is used in a few places.



core/src/main/java/org/apache/oozie/util/MetricsInstrumentation.java


>= 0 ? what if something takes less than 1ms? 



core/src/main/java/org/apache/oozie/util/MetricsInstrumentation.java


this can throw if the gauge's name already exist. Why not use a loading 
cache as you do for counters and timers? This is also true for histograms.



core/src/main/java/org/apache/oozie/util/MetricsInstrumentation.java


super nit: mention that the histogram is biased (decaying)?



docs/src/site/twiki/WebServicesAPI.twiki

[jira] [Updated] (OOZIE-1844) HA - Lock mechanism for CoordMaterializeTriggerService ( may be for other services as well)

[Purshotam Shah (JIRA)]

 [ 
https://issues.apache.org/jira/browse/OOZIE-1844?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Purshotam Shah updated OOZIE-1844:
--

Description: 
Currently we check if job id belong to this server by using modulus operation.

This may not be optimum way to do.
1. We are not processing MATERIALIZATION_SYSTEM_LIMIT, each server is only 
doing half (in case of two servers) processing. We can always double the limit. 
But as we add  new system, we need to restart whole cluster to increase the 
limit.

2. The job sequence id is shared among wf,coord,bundle. So, we could have a 
case where coord with odd/even id is more. In that case we are not distribute 
load. One server will always do more processing.

3. We also have different frequency for different coord jobs. Job with 1 min or 
5 min frequency will put more load on system. In this approach one particular 
job will always run in one system and eventually putting more load on one 
server. 

4. This logic may distribute load for even number of  servers, but not for odd 
number of servers.

May be simple way to optimize is to have a lock mechanism, each 
CoordMaterializeTriggerService will obtain a lock and materialize coord. If 
lock is held by other system, then it will wait for other system to release 
lock. In this way coord jobs will get distributed among servers.

> HA -  Lock mechanism for CoordMaterializeTriggerService ( may be for other 
> services as well)
> 
>
> Key: OOZIE-1844
> URL: https://issues.apache.org/jira/browse/OOZIE-1844
> Project: Oozie
>  Issue Type: Bug
>Reporter: Purshotam Shah
>
> Currently we check if job id belong to this server by using modulus operation.
> This may not be optimum way to do.
> 1. We are not processing MATERIALIZATION_SYSTEM_LIMIT, each server is only 
> doing half (in case of two servers) processing. We can always double the 
> limit. But as we add  new system, we need to restart whole cluster to 
> increase the limit.
> 2. The job sequence id is shared among wf,coord,bundle. So, we could have a 
> case where coord with odd/even id is more. In that case we are not distribute 
> load. One server will always do more processing.
> 3. We also have different frequency for different coord jobs. Job with 1 min 
> or 5 min frequency will put more load on system. In this approach one 
> particular job will always run in one system and eventually putting more load 
> on one server. 
> 4. This logic may distribute load for even number of  servers, but not for 
> odd number of servers.
> May be simple way to optimize is to have a lock mechanism, each 
> CoordMaterializeTriggerService will obtain a lock and materialize coord. If 
> lock is held by other system, then it will wait for other system to release 
> lock. In this way coord jobs will get distributed among servers.



--
This message was sent by Atlassian JIRA
(v6.2#6252)

Oozie BOF meetup at Hadoop Summit

[Mona Chitnis]

Hi Oozie Devs,

I’m in charge of organizing the Oozie BOF meetup for the Summit coming soon. 
They have given us two options for the date

  1.  Tues June 3 (first day of conf) 7:30p
  2.  Thu June 5 (last day of conf) - sharing room with Falcon BOF 3:30p onwards

I prefer option 1 though i’m not sure if 7:30 might be too late. Please chip 
asap in your thoughts so I can finalize it.

—
Mona

Build failed in Jenkins: oozie-trunk-precommit-build #1241

[Apache Jenkins Server]

See 

--
[...truncated 9638 lines...]
[INFO] share/lib already added, skipping
[INFO]  already added, skipping
[INFO] share already added, skipping
[INFO] share/lib already added, skipping
[INFO]  already added, skipping
[INFO] share already added, skipping
[INFO] share/lib already added, skipping
[INFO]  already added, skipping
[INFO] share already added, skipping
[INFO] share/lib already added, skipping
[INFO]  already added, skipping
[INFO] share already added, skipping
[INFO] share/lib already added, skipping
[INFO]  already added, skipping
[INFO] share already added, skipping
[INFO] share/lib already added, skipping
[INFO] 
[INFO] 
[INFO] Building Apache Oozie Tools 4.1.0-SNAPSHOT
[INFO] 
[WARNING] Failure to transfer asm:asm/maven-metadata.xml from 
file:../../local.repository/trunk was cached in the local repository, 
resolution will not be reattempted until the update interval of 
local.repository has elapsed or updates are forced. Original error: Could not 
transfer metadata asm:asm/maven-metadata.xml from/to local.repository 
(file:../../local.repository/trunk): No connector available to access 
repository local.repository (file:../../local.repository/trunk) of type legacy 
using the available factories WagonRepositoryConnectorFactory
[WARNING] Failure to transfer asm:asm/maven-metadata.xml from 
file:../../local.repository/trunk was cached in the local repository, 
resolution will not be reattempted until the update interval of 
local.repository has elapsed or updates are forced. Original error: Could not 
transfer metadata asm:asm/maven-metadata.xml from/to local.repository 
(file:../../local.repository/trunk): No connector available to access 
repository local.repository (file:../../local.repository/trunk) of type legacy 
using the available factories WagonRepositoryConnectorFactory
[INFO] 
[INFO] --- maven-resources-plugin:2.5:resources (default-resources) @ 
oozie-tools ---
[debug] execute contextualize
[INFO] Using 'UTF-8' encoding to copy filtered resources.
[INFO] skip non existing resourceDirectory 

[INFO] 
[INFO] --- maven-compiler-plugin:2.3.2:compile (default-compile) @ oozie-tools 
---
[INFO] Nothing to compile - all classes are up to date
[INFO] 
[INFO] --- maven-resources-plugin:2.5:testResources (default-testResources) @ 
oozie-tools ---
[debug] execute contextualize
[INFO] Using 'UTF-8' encoding to copy filtered resources.
[INFO] Copying 2 resources
[INFO] 
[INFO] --- maven-compiler-plugin:2.3.2:testCompile (default-testCompile) @ 
oozie-tools ---
[INFO] Nothing to compile - all classes are up to date
[INFO] 
[INFO] --- maven-surefire-plugin:2.12:test (default-test) @ oozie-tools ---
[INFO] Tests are skipped.
[INFO] 
[INFO] --- maven-jar-plugin:2.3.1:jar (default-jar) @ oozie-tools ---
[INFO] Building jar: 

[INFO] 
[INFO] --- maven-assembly-plugin:2.2.1:single (default-cli) @ oozie-tools ---
[INFO] Reading assembly descriptor: ../src/main/assemblies/tools.xml
[WARNING] The following patterns were never triggered in this artifact 
exclusion filter:
o  '*:*:pom:*'

[INFO] Copying files to 

[WARNING] Assembly file: 

 is not a regular file (it may be a directory). It cannot be attached to the 
project build for installation or deployment.
[INFO] 
[INFO] 
[INFO] Building Apache Oozie MiniOozie 4.1.0-SNAPSHOT
[INFO] 
[INFO] 
[INFO] --- maven-resources-plugin:2.5:resources (default-resources) @ 
oozie-mini ---
[debug] execute contextualize
[INFO] Using 'UTF-8' encoding to copy filtered resources.
[INFO] skip non existing resourceDirectory 

[INFO] 
[INFO] --- maven-compiler-plugin:2.3.2:compile (default-compile) @ oozie-mini 
---
[INFO] No sources to compile
[INFO] 
[INFO] --- maven-resources-plugin:2.5:testResources (default-testResources) @ 
oozie-mini ---
[debug] execute contextualize
[INFO] Using 'UTF-8' encoding to copy filtered resources.
[INFO] Copying 3 resources
[INFO] 
[INFO] --- maven-compiler-plugin:2.3.2:testCompile (default-testCompile) @ 
oozie-mini --

Re: Review Request 21278: OOZIE-1817: Oozie timers are not biased

[Robert Kanter]

> On May 15, 2014, 8:55 p.m., Gilad Wolff wrote:
> > Looks great, thanks for doing this. I have a few comments nothing major. 
> > 
> > G.

Thanks.  I've answered most of the issues below and made code changes or 
explained why not.  There's a few things I'll have to look into more and I need 
to do some testing for some of the changes.  I'll try to post an updated patch 
tomorrow.


> On May 15, 2014, 8:55 p.m., Gilad Wolff wrote:
> > core/src/main/java/org/apache/oozie/command/Command.java, line 195
> > 
> >
> > So does it make sense to normalize command metrics? Currently Command 
> > and XCommand create different metrics. XCommand has metrics for failure 
> > (xexception and exception) and an 'executions' counter. Command does not 
> > have these.

I checked and we currently only have one Command 
(CoordActionMaterializeCommand) that is using Command and not XCommand.  Plus, 
it's only being used during a dryrun operation so it's not frequently used.  I 
think it would be better to just convert CoordActionMaterializeCommand to an 
CoordActionMaterializeXCommand and remove Command completely; I've opened 
OOZIE-1846 for that.


> On May 15, 2014, 8:55 p.m., Gilad Wolff wrote:
> > core/src/main/java/org/apache/oozie/command/XCommand.java, line 332
> > 
> >
> > I think it makes more sense to only time the executions that succeeded. 
> > I think that in general you don't want the timer to get data from failed 
> > executions that may be very short.

The "call" timer includes other stuff than just the main part of the command, 
including loading state, verifying a precondition, etc.  I think we should keep 
the "call" timer as-is so it includes these other things.  For example, if a 
command fails its precondition a lot, the timer should show that (failing the 
precondition throws an exception).

However, the "execute" timer only includes the main part of the command and 
only gets updated if there's no exceptions.  So I think this timer covers 
essentially what you're looking for here.  


> On May 15, 2014, 8:55 p.m., Gilad Wolff wrote:
> > core/src/main/java/org/apache/oozie/service/InstrumentationService.java, 
> > line 51
> > 
> >
> > nit: what if you are already initialized? Is that okay to call init 
> > twice? Also, if you remove the isEnabled you'll need to assign the member 
> > at the end to make sure that you only initializes it when init was 
> > successful.

You're not allowed to init() twice without first doing destroy().  I imagine 
that many of the Services would have problems if you tried to do that.  We 
don't have anything enforcing this really (and it's actually caused problems in 
tests before), but for now we just assume you don't.  

That's is a good point about assigning the member at the end.


> On May 15, 2014, 8:55 p.m., Gilad Wolff wrote:
> > core/src/main/java/org/apache/oozie/service/MetricsInstrumentationService.java,
> >  line 34
> > 
> >
> > super nit: rename constant to include the unit?

I know Hadoop usually does that, but Oozie doesn't.  We just mention it in the 
description in oozie-default/oozie-site and/or the documentation.  I had 
forgotten to add this to oozie-default.xml, so thanks for reminding me though.


> On May 15, 2014, 8:55 p.m., Gilad Wolff wrote:
> > core/src/main/java/org/apache/oozie/service/MetricsInstrumentationService.java,
> >  line 37
> > 
> >
> > nit: you don't really need this member. Use the base class member 
> > instead?

MetricsInstrumentation has a destroy() method that we have to call to stop the 
Metrics stuff from logging.  Instrumentation doesn't have this, so I need it to 
be a MetricsInstrumentation object and not simply an Instrumentation object 
(and I'd rather not deal with casting).


> On May 15, 2014, 8:55 p.m., Gilad Wolff wrote:
> > core/src/main/java/org/apache/oozie/servlet/BaseAdminServlet.java, line 397
> > 
> >
> > nit: Why not have the same signature and pass the Instrumentation 
> > object?

We generate the json from the Instrumentation and MetricsInstrumentation 
differently, so this can't be done without a lot of refactoring.  I think it's 
simpler to just have two methods.


> On May 15, 2014, 8:55 p.m., Gilad Wolff wrote:
> > core/src/main/java/org/apache/oozie/servlet/V2AdminServlet.java, line 48
> > 
> >
> > nit: I am not sure if it is possible but the actual metric 
> > instrumentation instance can change (by calling destroy and init on the 
> > metric instrumentation service)

[jira] [Assigned] (OOZIE-1840) oozie needs to accept doas query param in place of doAs

[Bowen Zhang (JIRA)]

 [ 
https://issues.apache.org/jira/browse/OOZIE-1840?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Bowen Zhang reassigned OOZIE-1840:
--

Assignee: Bowen Zhang

> oozie needs to accept  doas query param in place of doAs
> 
>
> Key: OOZIE-1840
> URL: https://issues.apache.org/jira/browse/OOZIE-1840
> Project: Oozie
>  Issue Type: Bug
>  Components: security
>Reporter: Dilli Arumugam
>Assignee: Bowen Zhang
>
> At present, Oozie accepts  proxy user name as value of doAs query parameter.
> WebHDFS accepts proxy user name as value doas(case ignored) query parameter. 
> For exampe, both doAs and doas are valid query parameter names for WebHDFS.
> To be consistent with WebHDFS, Oozie needs to accept proxy user name as value 
> of doas(case ignored) query parameter.



--
This message was sent by Atlassian JIRA
(v6.2#6252)

Re: Review Request 21423: OOZIE-1813 Add service to report/kill rogue bundles and coordinator jobs

[Robert Kanter]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/21423/#review43175
---



core/src/main/java/org/apache/oozie/service/AbandonedCoordCheckerService.java


What happens if Oozie isn't configured to send emails?  We should handle 
that gracefully.


- Robert Kanter


On May 14, 2014, 6:42 p.m., Purshotam Shah wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/21423/
> ---
> 
> (Updated May 14, 2014, 6:42 p.m.)
> 
> 
> Review request for oozie.
> 
> 
> Repository: oozie-git
> 
> 
> Description
> ---
> 
> Report will be sent in html format, where each job is clickable and will take 
> you to job info. 
> 
> Coordinator id
> Coordinator nameUser name   Group   Kill Status
> 000-140513184018277-oozie-puru-C  COORD-TEST  test
> testg   Coord kill is disabled
> 001-140513184018277-oozie-puru-C  COORD-TEST  test
> testg   Coord kill is disabled
> 
> 
> Diffs
> -
> 
>   core/src/main/java/org/apache/oozie/CoordinatorJobBean.java 7915698 
>   core/src/main/java/org/apache/oozie/action/email/EmailActionExecutor.java 
> 332d02c 
>   core/src/main/java/org/apache/oozie/command/wf/JobXCommand.java cafeaeb 
>   core/src/main/java/org/apache/oozie/executor/jpa/CoordJobQueryExecutor.java 
> 1a6ded7 
>   
> core/src/main/java/org/apache/oozie/service/AbandonedCoordCheckerService.java 
> e69de29 
>   
> core/src/test/java/org/apache/oozie/command/coord/TestAbandonedCoordChecker.java
>  e69de29 
> 
> Diff: https://reviews.apache.org/r/21423/diff/
> 
> 
> Testing
> ---
> 
> UTC
> 
> 
> Thanks,
> 
> Purshotam Shah
> 
>

[jira] [Commented] (OOZIE-1813) Add service to report/kill rogue bundles and coordinator jobs

[Purshotam Shah (JIRA)]

[ 
https://issues.apache.org/jira/browse/OOZIE-1813?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14000379#comment-14000379
 ] 

Purshotam Shah commented on OOZIE-1813:
---

. -1 the patch contains 2 line(s) longer than 132 characters
Are namedQuery.

> Add service to report/kill rogue bundles and coordinator jobs
> -
>
> Key: OOZIE-1813
> URL: https://issues.apache.org/jira/browse/OOZIE-1813
> Project: Oozie
>  Issue Type: Bug
>Reporter: Purshotam Shah
>Assignee: Purshotam Shah
> Attachments: OOZIE-1813-V2.patch, OOZIE-1813-V3.patch
>
>
> People leave their test coordinator and bundle jobs without ever killing them
> and they just eat up resources heavily. We should have a service which 
> periodically check for abandoned coords and report/kill them.
> We can add multiple logic to this like ( number of consecutive 
> failed/timedout action, total number of failed/timedout action). 
> To start with if number of coord action with failed/timedout status > defined 
> value, then coord is considered to be rogue.



--
This message was sent by Atlassian JIRA
(v6.2#6252)

Build failed in Jenkins: oozie-trunk-precommit-build #1236

[Apache Jenkins Server]

See 

--
[...truncated 2840 lines...]
[INFO] --- maven-compiler-plugin:2.3.2:testCompile (default-testCompile) @ 
oozie-examples ---
[INFO] Compiling 1 source file to 

[INFO] 
[INFO] --- maven-surefire-plugin:2.12:test (default-test) @ oozie-examples ---
[INFO] Tests are skipped.
[INFO] 
[INFO] 
[INFO] Building Apache Oozie Share Lib 4.1.0-SNAPSHOT
[INFO] 
[INFO] 
[INFO] --- maven-clean-plugin:2.4.1:clean (default-clean) @ oozie-sharelib ---
[INFO] Deleting 

[INFO] 
[INFO] 
[INFO] Building Apache Oozie Tools 4.1.0-SNAPSHOT
[INFO] 
[WARNING] Could not transfer metadata asm:asm/maven-metadata.xml from/to 
local.repository (file:../../local.repository/trunk): No connector available to 
access repository local.repository (file:../../local.repository/trunk) of type 
legacy using the available factories WagonRepositoryConnectorFactory
[WARNING] Could not transfer metadata asm:asm/maven-metadata.xml from/to 
local.repository (file:../../local.repository/trunk): No connector available to 
access repository local.repository (file:../../local.repository/trunk) of type 
legacy using the available factories WagonRepositoryConnectorFactory
[INFO] 
[INFO] --- maven-clean-plugin:2.4.1:clean (default-clean) @ oozie-tools ---
[INFO] Deleting 

[INFO] 
[INFO] --- maven-resources-plugin:2.5:resources (default-resources) @ 
oozie-tools ---
[debug] execute contextualize
[INFO] Using 'UTF-8' encoding to copy filtered resources.
[INFO] skip non existing resourceDirectory 

[INFO] 
[INFO] --- maven-compiler-plugin:2.3.2:compile (default-compile) @ oozie-tools 
---
[INFO] Compiling 2 source files to 

[INFO] 
[INFO] --- maven-resources-plugin:2.5:testResources (default-testResources) @ 
oozie-tools ---
[debug] execute contextualize
[INFO] Using 'UTF-8' encoding to copy filtered resources.
[INFO] Copying 2 resources
[INFO] 
[INFO] --- maven-compiler-plugin:2.3.2:testCompile (default-testCompile) @ 
oozie-tools ---
[INFO] Compiling 6 source files to 

[INFO] 
[INFO] --- maven-surefire-plugin:2.12:test (default-test) @ oozie-tools ---
[INFO] Tests are skipped.
[INFO] 
[INFO] 
[INFO] Building Apache Oozie MiniOozie 4.1.0-SNAPSHOT
[INFO] 
[INFO] 
[INFO] --- maven-clean-plugin:2.4.1:clean (default-clean) @ oozie-mini ---
[INFO] Deleting 

[INFO] 
[INFO] --- maven-resources-plugin:2.5:resources (default-resources) @ 
oozie-mini ---
[debug] execute contextualize
[INFO] Using 'UTF-8' encoding to copy filtered resources.
[INFO] skip non existing resourceDirectory 

[INFO] 
[INFO] --- maven-compiler-plugin:2.3.2:compile (default-compile) @ oozie-mini 
---
[INFO] No sources to compile
[INFO] 
[INFO] --- maven-resources-plugin:2.5:testResources (default-testResources) @ 
oozie-mini ---
[debug] execute contextualize
[INFO] Using 'UTF-8' encoding to copy filtered resources.
[INFO] Copying 3 resources
[INFO] 
[INFO] --- maven-compiler-plugin:2.3.2:testCompile (default-testCompile) @ 
oozie-mini ---
[INFO] Compiling 1 source file to 

[INFO] 
[INFO] --- maven-surefire-plugin:2.12:test (default-test) @ oozie-mini ---
[INFO] Tests are skipped.
[INFO] 
[INFO] 
[INFO] Building Apache Oozie WorkflowGenerator 4.1.0-SNAPSHOT
[INFO] 
Downloading: 
https://repository.apache.org/content/groups/snapshots/com/orange/links/gwt-links/1.3-SNAPSHOT/maven-metadata.x

Re: Review Request 21278: OOZIE-1817: Oozie timers are not biased

[Robert Kanter]

> On May 15, 2014, 8:55 p.m., Gilad Wolff wrote:
> > core/src/main/java/org/apache/oozie/util/MetricsInstrumentation.java, line 
> > 105
> > 
> >
> > can the sliding time window be made configurable?
> 
> Robert Kanter wrote:
> I need to investigate this some more.  I'm wondering if this is the right 
> type of Reservoir to be using here.
> 
> According to 
> http://metrics.codahale.com/manual/core/#exponentially-decaying-reservoirs 
> SlidingTimeWindowReservoir can be memory intensive and they recommend 
> ExponentiallyDecayingReservoir instead (which does roughly the last 5min with 
> some fancy algorithm).  That's also what I'm using for the 
> samplers/histograms.
> 
> There's also the SlidingWindowReservoir, which stores the last N 
> measurements (SlidingTimeWindowReservoir does the last N seconds/minutes).  
> 
> Thoughts?

I'm going to use the ExponentiallyDecayingReservoir because of the memory 
usage.  We don't want someone configuring the SlidingTimeWindowReservoir to eat 
up a ton of memory.


- Robert


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/21278/#review42740
---


On May 13, 2014, 5:40 p.m., Robert Kanter wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/21278/
> ---
> 
> (Updated May 13, 2014, 5:40 p.m.)
> 
> 
> Review request for oozie.
> 
> 
> Bugs: OOZIE-1817
> https://issues.apache.org/jira/browse/OOZIE-1817
> 
> 
> Repository: oozie-git
> 
> 
> Description
> ---
> 
> See 
> https://issues.apache.org/jira/browse/OOZIE-1817?focusedCommentId=13993838&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-13993838
> 
> 
> Diffs
> -
> 
>   client/src/main/java/org/apache/oozie/client/rest/RestConstants.java 
> 808f9b2 
>   core/pom.xml c935dd7 
>   core/src/main/java/org/apache/oozie/command/Command.java d3f1011 
>   core/src/main/java/org/apache/oozie/command/XCommand.java b712ce0 
>   
> core/src/main/java/org/apache/oozie/command/coord/CoordActionInputCheckXCommand.java
>  e8667c1 
>   
> core/src/main/java/org/apache/oozie/command/coord/CoordActionMaterializeCommand.java
>  6962fb2 
>   
> core/src/main/java/org/apache/oozie/command/coord/CoordMaterializeTransitionXCommand.java
>  57cbb34 
>   core/src/main/java/org/apache/oozie/service/InstrumentationService.java 
> 80437b1 
>   
> core/src/main/java/org/apache/oozie/service/MetricsInstrumentationService.java
>  PRE-CREATION 
>   core/src/main/java/org/apache/oozie/service/Services.java 5feac7b 
>   core/src/main/java/org/apache/oozie/service/XLogService.java 403a089 
>   core/src/main/java/org/apache/oozie/servlet/BaseAdminServlet.java 29d7bd6 
>   core/src/main/java/org/apache/oozie/servlet/JsonRestServlet.java 5c05acd 
>   core/src/main/java/org/apache/oozie/servlet/V0AdminServlet.java 97bd81c 
>   core/src/main/java/org/apache/oozie/servlet/V1AdminServlet.java a47f737 
>   core/src/main/java/org/apache/oozie/servlet/V2AdminServlet.java 002a367 
>   core/src/main/java/org/apache/oozie/util/MetricsInstrumentation.java 
> PRE-CREATION 
>   core/src/main/java/org/apache/oozie/util/XLogReporter.java PRE-CREATION 
>   core/src/test/java/org/apache/oozie/service/TestInstrumentationService.java 
> b5206cf 
>   
> core/src/test/java/org/apache/oozie/service/TestMetricsInstrumentationService.java
>  PRE-CREATION 
>   core/src/test/java/org/apache/oozie/util/TestInstrumentation.java 6040b47 
>   core/src/test/java/org/apache/oozie/util/TestMetricsInstrumentation.java 
> PRE-CREATION 
>   docs/src/site/twiki/AG_Install.twiki e343d7e 
>   docs/src/site/twiki/WebServicesAPI.twiki 6730255 
>   pom.xml b5e0e4e 
>   webapp/src/main/webapp/index.jsp 3c7ffe5 
>   webapp/src/main/webapp/oozie-console.js 764d888 
> 
> Diff: https://reviews.apache.org/r/21278/diff/
> 
> 
> Testing
> ---
> 
> Unit tests, also verified in a cluster
> 
> 
> Thanks,
> 
> Robert Kanter
> 
>

[jira] [Commented] (OOZIE-1842) publish oozie artifacts

[Suresh Srinivas (JIRA)]

[ 
https://issues.apache.org/jira/browse/OOZIE-1842?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13998959#comment-13998959
 ] 

Suresh Srinivas commented on OOZIE-1842:


Downstream projects such as falcon can benefit from publishing oozie artifacts 
and get rid of work arounds such as 
[this|https://github.com/apache/incubator-falcon/blob/master/build-tools/src/bin/build-oozie.sh].
 Can this jira be taken as higher priority. [~tucu00], [~chitnis], any help 
here would be appreciated. Let me know if in anyway I can help.

> publish oozie artifacts
> ---
>
> Key: OOZIE-1842
> URL: https://issues.apache.org/jira/browse/OOZIE-1842
> Project: Oozie
>  Issue Type: New Feature
>Reporter: Bowen Zhang
>Assignee: Mona Chitnis
>




--
This message was sent by Atlassian JIRA
(v6.2#6252)

[jira] [Commented] (OOZIE-1783) Sharelib purging only occurs at Oozie startup

[Purshotam Shah (JIRA)]

[ 
https://issues.apache.org/jira/browse/OOZIE-1783?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14000305#comment-14000305
 ] 

Purshotam Shah commented on OOZIE-1783:
---

User can still use new sharelib behavior ( sharelib with timestamps).

I was talking more in terms of HA.  
Not only for launchersharelib, even different timestamp of sharelib can be used 
by different host.

This is not common scenario, it might happens. One Example, sharelib update 
didn't get propagated to one host (bcz of env issue). Each host might submit 
job with different sharelib jars for short interval. If purging happens at that 
time, then it will purge older sharelib, which is being used and all submitted 
will start failing.

If we are not in favor of having different sharelib path for each host, them 
somehow we need to have a flag saying that which sharelib is used by each host. 
Having a new path in ZK and adding timestamp of sharelib and launcherlib being 
used by all host, might help.


> Sharelib purging only occurs at Oozie startup
> -
>
> Key: OOZIE-1783
> URL: https://issues.apache.org/jira/browse/OOZIE-1783
> Project: Oozie
>  Issue Type: Bug
>Affects Versions: trunk
>Reporter: Robert Kanter
>Assignee: Robert Kanter
>Priority: Critical
> Attachments: OOZIE-1783.patch, OOZIE-1783.patch
>
>
> The purgeLibs method only gets called on startup of the first Oozie server.  
> This means that if you update the sharelib without restarting Oozie, then 
> Oozie will never clean up the old sharelibs and launcherlibs until you 
> restart the server.  In fact, with Oozie HA, it will never clean up unless 
> you restart all of the servers at the same time (in other words, a rolling 
> restart or just restarting one Oozie will never cause purgeLibs to get 
> called).  
> This should be moved into a background thread (i.e. scheduled with 
> SchedulerService) to run periodically.  Once a day is probably good enough, 
> but we could make it configurable if needed.  



--
This message was sent by Atlassian JIRA
(v6.2#6252)

[jira] [Updated] (OOZIE-1845) Coord submit command doesn't honor conf "materialization_window" while calling CoordMaterializeTransitionXCommand

[Purshotam Shah (JIRA)]

 [ 
https://issues.apache.org/jira/browse/OOZIE-1845?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Purshotam Shah updated OOZIE-1845:
--

Description: 
{code}
 /**
 * Queue MaterializeTransitionXCommand
 */
protected void queueMaterializeTransitionXCommand(String jobId) {
// submit a command to materialize jobs for the next 1 hour (3600 secs)
// so we don't wait 10 mins for the Service to run.
queue(new CoordMaterializeTransitionXCommand(jobId, 3600), 100);
}
{code}

> Coord submit command doesn't honor conf "materialization_window" while 
> calling CoordMaterializeTransitionXCommand 
> --
>
> Key: OOZIE-1845
> URL: https://issues.apache.org/jira/browse/OOZIE-1845
> Project: Oozie
>  Issue Type: Bug
>Reporter: Purshotam Shah
>
> {code}
>  /**
>  * Queue MaterializeTransitionXCommand
>  */
> protected void queueMaterializeTransitionXCommand(String jobId) {
> // submit a command to materialize jobs for the next 1 hour (3600 
> secs)
> // so we don't wait 10 mins for the Service to run.
> queue(new CoordMaterializeTransitionXCommand(jobId, 3600), 100);
> }
> {code}



--
This message was sent by Atlassian JIRA
(v6.2#6252)

[jira] [Updated] (OOZIE-1527) Fix scalability issues with coordinator materialization

[Purshotam Shah (JIRA)]

 [ 
https://issues.apache.org/jira/browse/OOZIE-1527?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Purshotam Shah updated OOZIE-1527:
--

Attachment: OOZIE-1527-V3.patch

> Fix scalability issues with coordinator materialization
> ---
>
> Key: OOZIE-1527
> URL: https://issues.apache.org/jira/browse/OOZIE-1527
> Project: Oozie
>  Issue Type: Bug
>  Components: coordinator
>Affects Versions: trunk
>Reporter: Mona Chitnis
>Assignee: Purshotam Shah
> Fix For: trunk
>
> Attachments: OOZIE-1527-V2.patch, OOZIE-1527-V3.patch
>
>   Original Estimate: 96h
>  Remaining Estimate: 96h
>
> In certain situations when there is a large number of coordinators in the 
> system, they have been observed to create huge backlog in materialization, 
> and progressing very slow compared to expected. This patch can be looked upon 
> as both a bug-fix or an enhancement addressing following points:
> 1. 'materialization.system.limit' leads to bringing Coord jobs in LRU 
> fashion, but some of them may already be maxing out at actions to materialize 
> (= throttle), and < limit jobs may actually undergo materialization. This 
> patch does a second iteration of loading jobs to get materialized to reduce 
> backlog
> 2. 'materialization.window' being 1 hour may work in most cases, but hourly 
> jobs are seen to face significant slowdown at times, by lot of other minute 
> jobs getting materialized. Therefore, window can be doubled (i.e. 2 hours) 
> when job is hourly/daily.
> 3. For hourly coordinators, it is consistently seen that materialization 
> occurs only near the end of the hour. e.g. for action whose nominal time is 
> 2:00, action creation time is 1:59, if nominal time - 3:00, creation time is 
> 2:58 and so on. If window is an hour in the future, doesn't explain why 
> materialization won't occur anytime in the middle of the preceding hour.



--
This message was sent by Atlassian JIRA
(v6.2#6252)

[jira] [Updated] (OOZIE-1844) HA - Lock mechanism for CoordMaterializeTriggerService ( may be for other services as well)

[Purshotam Shah (JIRA)]

 [ 
https://issues.apache.org/jira/browse/OOZIE-1844?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Purshotam Shah updated OOZIE-1844:
--

Description: 
Currently we check if job id belong to this server by using modulus operation.

This may not be optimum way to do.
1. We are not processing MATERIALIZATION_SYSTEM_LIMIT, each server is only 
doing half (in case of two servers) processing. We can always double the limit. 
But as we add  new system, we need to restart whole cluster to increase the 
limit.

2. The job sequence id is shared among wf,coord,bundle. So, we could have a 
case where coord with odd/even id is more. In that case we are not distribute 
load. One server will always do more processing.

3. We also have different frequency for different coord jobs. Job with 1 min or 
5 min frequency will put more load on system. In this approach one particular 
job will always run in one system and eventually putting more load on one 
server. 

May be simple way to optimize is to have a lock mechanism, each 
CoordMaterializeTriggerService will obtain a lock and materialize coord. If 
lock is held by other system, then it will wait for other system to release 
lock. In this way coord jobs will get distributed among servers.

  was:
Currently we check if job id belong to this server by using modulus operation.

This may not be optimum way to do.
1. We are not processing MATERIALIZATION_SYSTEM_LIMIT, each server is only 
doing half (in case of two servers) processing. We can always double the limit. 
But as we add  new system, we need to restart whole cluster to increase the 
limit.

2. The job sequence id is shared among wf,coord,bundle. So, we could have a 
case where coord with odd/even id is more. In that case we are not distribute 
load. One server will always do more processing.

3. We also have different frequency for different coord jobs. Job with 1 min or 
5 min frequency will put more load on system. In this approach one particular 
job will always run in one system and eventually putting more load on one 
server. 

4. This logic may distribute load for even number of  servers, but not for odd 
number of servers.

May be simple way to optimize is to have a lock mechanism, each 
CoordMaterializeTriggerService will obtain a lock and materialize coord. If 
lock is held by other system, then it will wait for other system to release 
lock. In this way coord jobs will get distributed among servers.


> HA -  Lock mechanism for CoordMaterializeTriggerService ( may be for other 
> services as well)
> 
>
> Key: OOZIE-1844
> URL: https://issues.apache.org/jira/browse/OOZIE-1844
> Project: Oozie
>  Issue Type: Bug
>Reporter: Purshotam Shah
>
> Currently we check if job id belong to this server by using modulus operation.
> This may not be optimum way to do.
> 1. We are not processing MATERIALIZATION_SYSTEM_LIMIT, each server is only 
> doing half (in case of two servers) processing. We can always double the 
> limit. But as we add  new system, we need to restart whole cluster to 
> increase the limit.
> 2. The job sequence id is shared among wf,coord,bundle. So, we could have a 
> case where coord with odd/even id is more. In that case we are not distribute 
> load. One server will always do more processing.
> 3. We also have different frequency for different coord jobs. Job with 1 min 
> or 5 min frequency will put more load on system. In this approach one 
> particular job will always run in one system and eventually putting more load 
> on one server. 
> May be simple way to optimize is to have a lock mechanism, each 
> CoordMaterializeTriggerService will obtain a lock and materialize coord. If 
> lock is held by other system, then it will wait for other system to release 
> lock. In this way coord jobs will get distributed among servers.



--
This message was sent by Atlassian JIRA
(v6.2#6252)

[jira] [Commented] (OOZIE-1838) jdbc.connections.active sampler does not show up

[Gilad Wolff (JIRA)]

[ 
https://issues.apache.org/jira/browse/OOZIE-1838?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13999270#comment-13999270
 ] 

Gilad Wolff commented on OOZIE-1838:


Patch looks good. One comment is that I would make the period and the interval 
configurable.

> jdbc.connections.active sampler does not show up
> 
>
> Key: OOZIE-1838
> URL: https://issues.apache.org/jira/browse/OOZIE-1838
> Project: Oozie
>  Issue Type: Bug
>  Components: core
>Affects Versions: 4.0.0
>Reporter: Robert Kanter
>Assignee: Robert Kanter
> Attachments: OOZIE-1838.patch
>
>
> I'm not sure how long this has been missing, but I don't ever remember seeing 
> a "jdbc.connections.active" sampler in the instrumentation.  However, I did 
> stumble upon it in the code.  It's supposed to sample the number of active 
> JDBC connections.



--
This message was sent by Atlassian JIRA
(v6.2#6252)

[jira] [Commented] (OOZIE-1737) Oozie log streaming is slow

[Purshotam Shah (JIRA)]

[ 
https://issues.apache.org/jira/browse/OOZIE-1737?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13999427#comment-13999427
 ] 

Purshotam Shah commented on OOZIE-1737:
---

We noticed two bugs as part of  regression, attaching amendment patch for that.
1. start/end offset is not displayed as part of debug message.
2. Log statement before start time is displayed.

> Oozie log streaming is slow
> ---
>
> Key: OOZIE-1737
> URL: https://issues.apache.org/jira/browse/OOZIE-1737
> Project: Oozie
>  Issue Type: Bug
>Reporter: Purshotam Shah
>Assignee: Purshotam Shah
> Fix For: trunk
>
> Attachments: OOZIE-1737-V5.patch, OOZIE-1737-V6.patch, 
> OOZIE-1737-V7.patch, OOZIE-1737-V8.patch
>
>
> Most of cases, Oozie search huge set of log files and return lot of logs.
> Most of our users has complained that log streaming is slow and they can't 
> easily find relevant information.
> To make log streaming fast and precise we should provide few options (which 
> user can specify for log streaming) like
> 1. Start time : Start date (if number, then offset(in hour) from 
> coord/wf/bundle start)
> 2. End time : End date (if number, then offset from coord/wf/bundle start)
> 3. Log level :
>Setting log level to ERROR, will only give erred log and exception.
>In most of cases user want to find out error message and exception.
> 4. Number of lines : limit number of lines of logs from start date.
> 5. Recent logs: In most of the case user is only interested in recent 
> activity.
> Similar to https://issues.apache.org/jira/browse/OOZIE-1716.
> User should be able to specify recent logs in hours.
> 6. Grep for pattern.
> + in addition to this, we also need to clean up log statements( which may 
> also include consolidating log statements).



--
This message was sent by Atlassian JIRA
(v6.2#6252)

Re: publish oozie artifacts

[bowen zhang]

Hi Mona,
I created this ticket:https://issues.apache.org/jira/browse/OOZIE-1842. Do you 
have any updates on this?
Bowen

On Tuesday, April 29, 2014 11:52 AM, Mona Chitnis  wrote:
 
Thanks Robert. I'll look into it this week.--Monahttps://overview.mail.yahoo.com?.src=iOS";>Sent from Yahoo Mail 
for iPhone

[jira] [Commented] (OOZIE-1841) Allow for specifications of wf/coordinator propertey files from web api

[Purshotam Shah (JIRA)]

[ 
https://issues.apache.org/jira/browse/OOZIE-1841?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13999501#comment-13999501
 ] 

Purshotam Shah commented on OOZIE-1841:
---

Are you looking for something like 
https://issues.apache.org/jira/browse/OOZIE-1769(An option to update coord 
properties/definition)

> Allow for specifications of wf/coordinator propertey files from web api
> ---
>
> Key: OOZIE-1841
> URL: https://issues.apache.org/jira/browse/OOZIE-1841
> Project: Oozie
>  Issue Type: New Feature
>Reporter: Josh Kurz
>
> I would like to be able to post to oozie/v1/jobs and just set a configuration 
> path. The xml files for each coordinator job that could be created are 
> already created and are on the uzi box (for my use case). Is there a way to 
> just tell oozie that I would like to use one of those pre made xml files vs 
> submitting xml with an http post. 
> curl -X POST -H "Content-Type: application/json" -d {"configUrl": 
> "/path/to/uzi/configuration"} "http://uzibox:11000/oozie/v1/jobs?action=start";



--
This message was sent by Atlassian JIRA
(v6.2#6252)

[jira] [Updated] (OOZIE-1843) Bulk update for coord last modified time for CoordMaterializeTriggerService

[Purshotam Shah (JIRA)]

 [ 
https://issues.apache.org/jira/browse/OOZIE-1843?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Purshotam Shah updated OOZIE-1843:
--

Description: 
{code}
for (CoordinatorJobBean coordJob : materializeJobs) {
if 
(Services.get().get(JobsConcurrencyService.class).isJobIdForThisServer(coordJob.getId()))
 {
Services.get().get(InstrumentationService.class).get()
.incr(INSTRUMENTATION_GROUP, 
INSTR_MAT_JOBS_COUNTER, 1);
queueCallable(new 
CoordMaterializeTransitionXCommand(coordJob.getId(), materializationWindow));
coordJob.setLastModifiedTime(new Date());
CoordJobQueryExecutor.getInstance().executeUpdate(

CoordJobQueryExecutor.CoordJobQuery.UPDATE_COORD_JOB_LAST_MODIFIED_TIME, 
coordJob);
 }
{code}

> Bulk update for coord last modified time for CoordMaterializeTriggerService
> ---
>
> Key: OOZIE-1843
> URL: https://issues.apache.org/jira/browse/OOZIE-1843
> Project: Oozie
>  Issue Type: Bug
>Reporter: Purshotam Shah
>
> {code}
> for (CoordinatorJobBean coordJob : materializeJobs) {
> if 
> (Services.get().get(JobsConcurrencyService.class).isJobIdForThisServer(coordJob.getId()))
>  {
> Services.get().get(InstrumentationService.class).get()
> .incr(INSTRUMENTATION_GROUP, 
> INSTR_MAT_JOBS_COUNTER, 1);
> queueCallable(new 
> CoordMaterializeTransitionXCommand(coordJob.getId(), materializationWindow));
> coordJob.setLastModifiedTime(new Date());
> CoordJobQueryExecutor.getInstance().executeUpdate(
> 
> CoordJobQueryExecutor.CoordJobQuery.UPDATE_COORD_JOB_LAST_MODIFIED_TIME, 
> coordJob);
>  }
> {code}



--
This message was sent by Atlassian JIRA
(v6.2#6252)

[jira] [Commented] (OOZIE-1813) Add service to report/kill rogue bundles and coordinator jobs

[Hadoop QA (JIRA)]

[ 
https://issues.apache.org/jira/browse/OOZIE-1813?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13999553#comment-13999553
 ] 

Hadoop QA commented on OOZIE-1813:
--

Testing JIRA OOZIE-1813

Cleaning local git workspace



{color:green}+1 PATCH_APPLIES{color}
{color:green}+1 CLEAN{color}
{color:red}-1 RAW_PATCH_ANALYSIS{color}
.{color:green}+1{color} the patch does not introduce any @author tags
.{color:green}+1{color} the patch does not introduce any tabs
.{color:green}+1{color} the patch does not introduce any trailing spaces
.{color:red}-1{color} the patch contains 2 line(s) longer than 132 
characters
.{color:green}+1{color} the patch does adds/modifies 1 testcase(s)
{color:red}-1 RAT{color}
.{color:red}-1{color} the patch seems to introduce 1 new RAT warning(s)
{color:green}+1 JAVADOC{color}
.{color:green}+1{color} the patch does not seem to introduce new Javadoc 
warnings
{color:green}+1 COMPILE{color}
.{color:green}+1{color} HEAD compiles
.{color:green}+1{color} patch compiles
.{color:green}+1{color} the patch does not seem to introduce new javac 
warnings
{color:green}+1 BACKWARDS_COMPATIBILITY{color}
.{color:green}+1{color} the patch does not change any JPA 
Entity/Colum/Basic/Lob/Transient annotations
.{color:green}+1{color} the patch does not modify JPA files
{color:red}-1 TESTS{color}
.Tests run: 1450
.Tests failed: 1
.Tests errors: 7

.The patch failed the following testcases:

.  
testCoordinatorActionCommandsSubmitAndStart(org.apache.oozie.sla.TestSLAEventGeneration)

{color:green}+1 DISTRO{color}
.{color:green}+1{color} distro tarball builds with the patch 


{color:red}*-1 Overall result, please check the reported -1(s)*{color}


The full output of the test-patch run is available at

.   https://builds.apache.org/job/oozie-trunk-precommit-build/1237/

> Add service to report/kill rogue bundles and coordinator jobs
> -
>
> Key: OOZIE-1813
> URL: https://issues.apache.org/jira/browse/OOZIE-1813
> Project: Oozie
>  Issue Type: Bug
>Reporter: Purshotam Shah
>Assignee: Purshotam Shah
> Attachments: OOZIE-1813-V2.patch
>
>
> People leave their test coordinator and bundle jobs without ever killing them
> and they just eat up resources heavily. We should have a service which 
> periodically check for abandoned coords and report/kill them.
> We can add multiple logic to this like ( number of consecutive 
> failed/timedout action, total number of failed/timedout action). 
> To start with if number of coord action with failed/timedout status > defined 
> value, then coord is considered to be rogue.



--
This message was sent by Atlassian JIRA
(v6.2#6252)

[jira] [Reopened] (OOZIE-1737) Oozie log streaming is slow

[Purshotam Shah (JIRA)]

 [ 
https://issues.apache.org/jira/browse/OOZIE-1737?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Purshotam Shah reopened OOZIE-1737:
---


> Oozie log streaming is slow
> ---
>
> Key: OOZIE-1737
> URL: https://issues.apache.org/jira/browse/OOZIE-1737
> Project: Oozie
>  Issue Type: Bug
>Reporter: Purshotam Shah
>Assignee: Purshotam Shah
> Fix For: trunk
>
> Attachments: OOZIE-1737-V5.patch, OOZIE-1737-V6.patch, 
> OOZIE-1737-V7.patch, OOZIE-1737-V8.patch
>
>
> Most of cases, Oozie search huge set of log files and return lot of logs.
> Most of our users has complained that log streaming is slow and they can't 
> easily find relevant information.
> To make log streaming fast and precise we should provide few options (which 
> user can specify for log streaming) like
> 1. Start time : Start date (if number, then offset(in hour) from 
> coord/wf/bundle start)
> 2. End time : End date (if number, then offset from coord/wf/bundle start)
> 3. Log level :
>Setting log level to ERROR, will only give erred log and exception.
>In most of cases user want to find out error message and exception.
> 4. Number of lines : limit number of lines of logs from start date.
> 5. Recent logs: In most of the case user is only interested in recent 
> activity.
> Similar to https://issues.apache.org/jira/browse/OOZIE-1716.
> User should be able to specify recent logs in hours.
> 6. Grep for pattern.
> + in addition to this, we also need to clean up log statements( which may 
> also include consolidating log statements).



--
This message was sent by Atlassian JIRA
(v6.2#6252)

Build failed in Jenkins: oozie-trunk-precommit-build #1244

[Apache Jenkins Server]

See 

--
[...truncated 9594 lines...]
[INFO] share/lib already added, skipping
[INFO]  already added, skipping
[INFO] share already added, skipping
[INFO] share/lib already added, skipping
[INFO]  already added, skipping
[INFO] share already added, skipping
[INFO] share/lib already added, skipping
[INFO]  already added, skipping
[INFO] share already added, skipping
[INFO] share/lib already added, skipping
[INFO]  already added, skipping
[INFO] share already added, skipping
[INFO] share/lib already added, skipping
[INFO] 
[INFO] 
[INFO] Building Apache Oozie Tools 4.1.0-SNAPSHOT
[INFO] 
[WARNING] Failure to transfer asm:asm/maven-metadata.xml from 
file:../../local.repository/trunk was cached in the local repository, 
resolution will not be reattempted until the update interval of 
local.repository has elapsed or updates are forced. Original error: Could not 
transfer metadata asm:asm/maven-metadata.xml from/to local.repository 
(file:../../local.repository/trunk): No connector available to access 
repository local.repository (file:../../local.repository/trunk) of type legacy 
using the available factories WagonRepositoryConnectorFactory
[WARNING] Failure to transfer asm:asm/maven-metadata.xml from 
file:../../local.repository/trunk was cached in the local repository, 
resolution will not be reattempted until the update interval of 
local.repository has elapsed or updates are forced. Original error: Could not 
transfer metadata asm:asm/maven-metadata.xml from/to local.repository 
(file:../../local.repository/trunk): No connector available to access 
repository local.repository (file:../../local.repository/trunk) of type legacy 
using the available factories WagonRepositoryConnectorFactory
[INFO] 
[INFO] --- maven-resources-plugin:2.5:resources (default-resources) @ 
oozie-tools ---
[debug] execute contextualize
[INFO] Using 'UTF-8' encoding to copy filtered resources.
[INFO] skip non existing resourceDirectory 

[INFO] 
[INFO] --- maven-compiler-plugin:2.3.2:compile (default-compile) @ oozie-tools 
---
[INFO] Nothing to compile - all classes are up to date
[INFO] 
[INFO] --- maven-resources-plugin:2.5:testResources (default-testResources) @ 
oozie-tools ---
[debug] execute contextualize
[INFO] Using 'UTF-8' encoding to copy filtered resources.
[INFO] Copying 2 resources
[INFO] 
[INFO] --- maven-compiler-plugin:2.3.2:testCompile (default-testCompile) @ 
oozie-tools ---
[INFO] Nothing to compile - all classes are up to date
[INFO] 
[INFO] --- maven-surefire-plugin:2.12:test (default-test) @ oozie-tools ---
[INFO] Tests are skipped.
[INFO] 
[INFO] --- maven-jar-plugin:2.3.1:jar (default-jar) @ oozie-tools ---
[INFO] Building jar: 

[INFO] 
[INFO] --- maven-assembly-plugin:2.2.1:single (default-cli) @ oozie-tools ---
[INFO] Reading assembly descriptor: ../src/main/assemblies/tools.xml
[WARNING] The following patterns were never triggered in this artifact 
exclusion filter:
o  '*:*:pom:*'

[INFO] Copying files to 

[WARNING] Assembly file: 

 is not a regular file (it may be a directory). It cannot be attached to the 
project build for installation or deployment.
[INFO] 
[INFO] 
[INFO] Building Apache Oozie MiniOozie 4.1.0-SNAPSHOT
[INFO] 
[INFO] 
[INFO] --- maven-resources-plugin:2.5:resources (default-resources) @ 
oozie-mini ---
[debug] execute contextualize
[INFO] Using 'UTF-8' encoding to copy filtered resources.
[INFO] skip non existing resourceDirectory 

[INFO] 
[INFO] --- maven-compiler-plugin:2.3.2:compile (default-compile) @ oozie-mini 
---
[INFO] No sources to compile
[INFO] 
[INFO] --- maven-resources-plugin:2.5:testResources (default-testResources) @ 
oozie-mini ---
[debug] execute contextualize
[INFO] Using 'UTF-8' encoding to copy filtered resources.
[INFO] Copying 3 resources
[INFO] 
[INFO] --- maven-compiler-plugin:2.3.2:testCompile (default-testCompile) @ 
oozie-mini ---
[INFO] Nothing to compile - all classes are up to date
[INFO] 
[INFO] --- maven-surefire-plugin:2.12:test (d

[jira] [Created] (OOZIE-1846) Convert CoordActionMaterializeCommand to an XCommand and remove Command

[Robert Kanter (JIRA)]

Robert Kanter created OOZIE-1846:


 Summary: Convert CoordActionMaterializeCommand to an XCommand and 
remove Command
 Key: OOZIE-1846
 URL: https://issues.apache.org/jira/browse/OOZIE-1846
 Project: Oozie
  Issue Type: Improvement
  Components: core
Affects Versions: 4.0.1
Reporter: Robert Kanter


{{CoordActionMaterializeCommand}} is only used in one place where we do a 
coordinator dryrun ({{CoordSubmitXCommand.getDryRun(...)}}); otherwise, 
{{CoordMaterializeTransitionXCommand}} is normally used.

{{CoordActionMaterializeCommand}} appears to be the only class left still based 
on {{Command}}; everything else is using {{XCommand}}.  We should update the 
code that uses {{CoordActionMaterializeCommand}} to use 
{{CoordMaterializeTransitionXCommand}} instead.  Then we could completely 
remove {{Comamnd}}.  




--
This message was sent by Atlassian JIRA
(v6.2#6252)

Re: Review Request 21423: OOZIE-1813 Add service to report/kill rogue bundles and coordinator jobs

[Purshotam Shah]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/21423/
---

(Updated May 16, 2014, 9:51 p.m.)


Review request for oozie.


Repository: oozie-git


Description
---

Report will be sent in html format, where each job is clickable and will take 
you to job info. 

Coordinator id  Coordinator 
nameUser name   Group   Kill Status
000-140513184018277-oozie-puru-CCOORD-TEST  test
testg   Coord kill is disabled
001-140513184018277-oozie-puru-CCOORD-TEST  test
testg   Coord kill is disabled


Diffs (updated)
-

  core/src/main/java/org/apache/oozie/CoordinatorJobBean.java 7915698 
  core/src/main/java/org/apache/oozie/action/email/EmailActionExecutor.java 
332d02c 
  core/src/main/java/org/apache/oozie/command/wf/JobXCommand.java cafeaeb 
  core/src/main/java/org/apache/oozie/executor/jpa/CoordJobQueryExecutor.java 
1a6ded7 
  core/src/main/java/org/apache/oozie/service/AbandonedCoordCheckerService.java 
e69de29 
  
core/src/test/java/org/apache/oozie/command/coord/TestAbandonedCoordChecker.java
 e69de29 

Diff: https://reviews.apache.org/r/21423/diff/


Testing
---

UTC


Thanks,

Purshotam Shah

[jira] [Commented] (OOZIE-1817) Oozie timers are not biased

[Hadoop QA (JIRA)]

[ 
https://issues.apache.org/jira/browse/OOZIE-1817?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13999730#comment-13999730
 ] 

Hadoop QA commented on OOZIE-1817:
--

Testing JIRA OOZIE-1817

Cleaning local git workspace



{color:green}+1 PATCH_APPLIES{color}
{color:green}+1 CLEAN{color}
{color:green}+1 RAW_PATCH_ANALYSIS{color}
.{color:green}+1{color} the patch does not introduce any @author tags
.{color:green}+1{color} the patch does not introduce any tabs
.{color:green}+1{color} the patch does not introduce any trailing spaces
.{color:green}+1{color} the patch does not introduce any line longer than 
132
.{color:green}+1{color} the patch does adds/modifies 4 testcase(s)
{color:green}+1 RAT{color}
.{color:green}+1{color} the patch does not seem to introduce new RAT 
warnings
{color:green}+1 JAVADOC{color}
.{color:green}+1{color} the patch does not seem to introduce new Javadoc 
warnings
{color:green}+1 COMPILE{color}
.{color:green}+1{color} HEAD compiles
.{color:green}+1{color} patch compiles
.{color:green}+1{color} the patch does not seem to introduce new javac 
warnings
{color:green}+1 BACKWARDS_COMPATIBILITY{color}
.{color:green}+1{color} the patch does not change any JPA 
Entity/Colum/Basic/Lob/Transient annotations
.{color:green}+1{color} the patch does not modify JPA files
{color:red}-1 TESTS{color}
.Tests run: 1454
.Tests failed: 3
.Tests errors: 2

.The patch failed the following testcases:

.  testJobLog(org.apache.oozie.client.TestOozieCLI)
.  
testCoordinatorActionCommandsSubmitAndStart(org.apache.oozie.sla.TestSLAEventGeneration)
.  
testBundleEngineKill(org.apache.oozie.servlet.TestV1JobServletBundleEngine)

{color:green}+1 DISTRO{color}
.{color:green}+1{color} distro tarball builds with the patch 


{color:red}*-1 Overall result, please check the reported -1(s)*{color}


The full output of the test-patch run is available at

.   https://builds.apache.org/job/oozie-trunk-precommit-build/1241/

> Oozie timers are not biased
> ---
>
> Key: OOZIE-1817
> URL: https://issues.apache.org/jira/browse/OOZIE-1817
> Project: Oozie
>  Issue Type: Improvement
>  Components: monitoring
>Affects Versions: 4.0.0
>Reporter: Gilad Wolff
>Assignee: Robert Kanter
> Attachments: OOZIE-1817.patch, OOZIE-1817.patch, OOZIE-1817.patch, 
> OOZIE-1817.patch
>
>
> Oozie timers are not biased, that is, the statistical metrics they expose are 
> over the run-time of the Oozie server instead of a window of time. This makes 
> them not very useful especially after the server has been running for a while 
> (codehale has very efficient and easy to use biased histograms that can be 
> used instead).



--
This message was sent by Atlassian JIRA
(v6.2#6252)

Re: Review Request 21423: OOZIE-1813 Add service to report/kill rogue bundles and coordinator jobs

[Purshotam Shah]

> On May 15, 2014, 10:54 p.m., Robert Kanter wrote:
> > core/src/main/java/org/apache/oozie/service/AbandonedCoordCheckerService.java,
> >  line 149
> > 
> >
> > What happens if Oozie isn't configured to send emails?  We should 
> > handle that gracefully.

I see your point. We will not send email if "TO_ADDRESS = CONF_PREFIX + 
"email.address"" is not configured.


- Purshotam


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/21423/#review43175
---


On May 14, 2014, 6:42 p.m., Purshotam Shah wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/21423/
> ---
> 
> (Updated May 14, 2014, 6:42 p.m.)
> 
> 
> Review request for oozie.
> 
> 
> Repository: oozie-git
> 
> 
> Description
> ---
> 
> Report will be sent in html format, where each job is clickable and will take 
> you to job info. 
> 
> Coordinator id
> Coordinator nameUser name   Group   Kill Status
> 000-140513184018277-oozie-puru-C  COORD-TEST  test
> testg   Coord kill is disabled
> 001-140513184018277-oozie-puru-C  COORD-TEST  test
> testg   Coord kill is disabled
> 
> 
> Diffs
> -
> 
>   core/src/main/java/org/apache/oozie/CoordinatorJobBean.java 7915698 
>   core/src/main/java/org/apache/oozie/action/email/EmailActionExecutor.java 
> 332d02c 
>   core/src/main/java/org/apache/oozie/command/wf/JobXCommand.java cafeaeb 
>   core/src/main/java/org/apache/oozie/executor/jpa/CoordJobQueryExecutor.java 
> 1a6ded7 
>   
> core/src/main/java/org/apache/oozie/service/AbandonedCoordCheckerService.java 
> e69de29 
>   
> core/src/test/java/org/apache/oozie/command/coord/TestAbandonedCoordChecker.java
>  e69de29 
> 
> Diff: https://reviews.apache.org/r/21423/diff/
> 
> 
> Testing
> ---
> 
> UTC
> 
> 
> Thanks,
> 
> Purshotam Shah
> 
>

[jira] [Commented] (OOZIE-1783) Sharelib purging only occurs at Oozie startup

[Purshotam Shah (JIRA)]

[ 
https://issues.apache.org/jira/browse/OOZIE-1783?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14000318#comment-14000318
 ] 

Purshotam Shah commented on OOZIE-1783:
---

[~rohini]
Yes, we need to move  purgeLibs(fs, LAUNCHER_PREFIX, current) out of 
isFirstServer().

Purging sharelib in HA can be slightly complicated, we can create a different 
JIRA for that.

> Sharelib purging only occurs at Oozie startup
> -
>
> Key: OOZIE-1783
> URL: https://issues.apache.org/jira/browse/OOZIE-1783
> Project: Oozie
>  Issue Type: Bug
>Affects Versions: trunk
>Reporter: Robert Kanter
>Assignee: Robert Kanter
>Priority: Critical
> Attachments: OOZIE-1783.patch, OOZIE-1783.patch
>
>
> The purgeLibs method only gets called on startup of the first Oozie server.  
> This means that if you update the sharelib without restarting Oozie, then 
> Oozie will never clean up the old sharelibs and launcherlibs until you 
> restart the server.  In fact, with Oozie HA, it will never clean up unless 
> you restart all of the servers at the same time (in other words, a rolling 
> restart or just restarting one Oozie will never cause purgeLibs to get 
> called).  
> This should be moved into a background thread (i.e. scheduled with 
> SchedulerService) to run periodically.  Once a day is probably good enough, 
> but we could make it configurable if needed.  



--
This message was sent by Atlassian JIRA
(v6.2#6252)

[jira] [Commented] (OOZIE-1783) Sharelib purging only occurs at Oozie startup

[Purshotam Shah (JIRA)]

[ 
https://issues.apache.org/jira/browse/OOZIE-1783?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=1436#comment-1436
 ] 

Purshotam Shah commented on OOZIE-1783:
---

There might be other issue with sharelib purging in HA.

Current purging logic
--
 Keep all share-lib between current timestamp and 7days old + 1 latest sharelib 
older than 7 days.
 https://issues.apache.org/jira/browse/OOZIE-1761
---


Each server should maintain their own sharelib, otherwise sharelib use by one 
server might be purged. 
Ex. two servers are running and both using different sharelib older then 7 
days, one used sharelib will be purged.

At Yahoo, each server in HA has their own sharelib ( by setting different path 
for oozie.service.WorkflowAppService.system.libpath). 
For this approach we need to take out isFirstServer logic. So that each server 
can purge their own sharelib.

{code}
if 
(Services.get().get(JobsConcurrencyService.class).isFirstServer()) {
final Date current = 
Calendar.getInstance(TimeZone.getTimeZone("GMT")).getTime();
purgeLibs(fs, LAUNCHER_PREFIX, current);
purgeLibs(fs, SHARED_LIB_PREFIX, current);
}
{code}


Other solution, sharelib service itself suffix hostname to path and works on 
path which has hostname ( for all operations retrieving sharelib, update and 
purging).
In this case also we need to take away isFirstServer logic.



> Sharelib purging only occurs at Oozie startup
> -
>
> Key: OOZIE-1783
> URL: https://issues.apache.org/jira/browse/OOZIE-1783
> Project: Oozie
>  Issue Type: Bug
>Affects Versions: trunk
>Reporter: Robert Kanter
>Assignee: Robert Kanter
>Priority: Critical
> Attachments: OOZIE-1783.patch, OOZIE-1783.patch
>
>
> The purgeLibs method only gets called on startup of the first Oozie server.  
> This means that if you update the sharelib without restarting Oozie, then 
> Oozie will never clean up the old sharelibs and launcherlibs until you 
> restart the server.  In fact, with Oozie HA, it will never clean up unless 
> you restart all of the servers at the same time (in other words, a rolling 
> restart or just restarting one Oozie will never cause purgeLibs to get 
> called).  
> This should be moved into a background thread (i.e. scheduled with 
> SchedulerService) to run periodically.  Once a day is probably good enough, 
> but we could make it configurable if needed.  



--
This message was sent by Atlassian JIRA
(v6.2#6252)

Build failed in Jenkins: oozie-trunk-precommit-build #1240

[Apache Jenkins Server]

See 

--
[...truncated 9579 lines...]
[INFO] share/lib already added, skipping
[INFO]  already added, skipping
[INFO] share already added, skipping
[INFO] share/lib already added, skipping
[INFO]  already added, skipping
[INFO] share already added, skipping
[INFO] share/lib already added, skipping
[INFO] Building tar : 

[INFO]  already added, skipping
[INFO] share already added, skipping
[INFO] share/lib already added, skipping
[INFO]  already added, skipping
[INFO] share already added, skipping
[INFO] share/lib already added, skipping
[INFO]  already added, skipping
[INFO] share already added, skipping
[INFO] share/lib already added, skipping
[INFO]  already added, skipping
[INFO] share already added, skipping
[INFO] share/lib already added, skipping
[INFO]  already added, skipping
[INFO] share already added, skipping
[INFO] share/lib already added, skipping
[INFO]  already added, skipping
[INFO] share already added, skipping
[INFO] share/lib already added, skipping
[INFO] 
[INFO] 
[INFO] Building Apache Oozie Tools 4.1.0-SNAPSHOT
[INFO] 
[WARNING] Failure to transfer asm:asm/maven-metadata.xml from 
file:../../local.repository/trunk was cached in the local repository, 
resolution will not be reattempted until the update interval of 
local.repository has elapsed or updates are forced. Original error: Could not 
transfer metadata asm:asm/maven-metadata.xml from/to local.repository 
(file:../../local.repository/trunk): No connector available to access 
repository local.repository (file:../../local.repository/trunk) of type legacy 
using the available factories WagonRepositoryConnectorFactory
[WARNING] Failure to transfer asm:asm/maven-metadata.xml from 
file:../../local.repository/trunk was cached in the local repository, 
resolution will not be reattempted until the update interval of 
local.repository has elapsed or updates are forced. Original error: Could not 
transfer metadata asm:asm/maven-metadata.xml from/to local.repository 
(file:../../local.repository/trunk): No connector available to access 
repository local.repository (file:../../local.repository/trunk) of type legacy 
using the available factories WagonRepositoryConnectorFactory
[INFO] 
[INFO] --- maven-resources-plugin:2.5:resources (default-resources) @ 
oozie-tools ---
[debug] execute contextualize
[INFO] Using 'UTF-8' encoding to copy filtered resources.
[INFO] skip non existing resourceDirectory 

[INFO] 
[INFO] --- maven-compiler-plugin:2.3.2:compile (default-compile) @ oozie-tools 
---
[INFO] Nothing to compile - all classes are up to date
[INFO] 
[INFO] --- maven-resources-plugin:2.5:testResources (default-testResources) @ 
oozie-tools ---
[debug] execute contextualize
[INFO] Using 'UTF-8' encoding to copy filtered resources.
[INFO] Copying 2 resources
[INFO] 
[INFO] --- maven-compiler-plugin:2.3.2:testCompile (default-testCompile) @ 
oozie-tools ---
[INFO] Nothing to compile - all classes are up to date
[INFO] 
[INFO] --- maven-surefire-plugin:2.12:test (default-test) @ oozie-tools ---
[INFO] Tests are skipped.
[INFO] 
[INFO] --- maven-jar-plugin:2.3.1:jar (default-jar) @ oozie-tools ---
[INFO] Building jar: 

[INFO] 
[INFO] --- maven-assembly-plugin:2.2.1:single (default-cli) @ oozie-tools ---
[INFO] Reading assembly descriptor: ../src/main/assemblies/tools.xml
[WARNING] The following patterns were never triggered in this artifact 
exclusion filter:
o  '*:*:pom:*'

[INFO] Copying files to 

[WARNING] Assembly file: 

 is not a regular file (it may be a directory). It cannot be attached to the 
project build for installation or deployment.
[INFO] 
[INFO] 
[INFO] Building Apache Oozie MiniOozie 4.1.0-SNAPSHOT
[INFO] 
[INFO] 
[INFO] --- maven-resources-plugin:2.5:resources (default-resources) @ 
oozie-mini ---
[debug] execute contextualize
[INFO] Using 'UTF-8' encoding to copy filtered resources.
[INFO] skip non existing resourceDirectory 

[jira] [Created] (OOZIE-1843) Bulk update for coord last modified time for CoordMaterializeTriggerService

[Purshotam Shah (JIRA)]

Purshotam Shah created OOZIE-1843:
-

 Summary: Bulk update for coord last modified time for 
CoordMaterializeTriggerService
 Key: OOZIE-1843
 URL: https://issues.apache.org/jira/browse/OOZIE-1843
 Project: Oozie
  Issue Type: Bug
Reporter: Purshotam Shah






--
This message was sent by Atlassian JIRA
(v6.2#6252)

[jira] [Commented] (OOZIE-1783) Sharelib purging only occurs at Oozie startup

[Rohini Palaniswamy (JIRA)]

[ 
https://issues.apache.org/jira/browse/OOZIE-1783?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14000282#comment-14000282
 ] 

Rohini Palaniswamy commented on OOZIE-1783:
---

[~puru],
   Good point. But cannot expect everyone to have two different sharelibs. Our 
case is different as we don't have sharelibs in a directory but pick from a 
metafile and use the sharelib directory only for shipping launcher libs. In 
other cases they upload only to hdfs once and share it across all servers. 
There was also a jira to keep the old sharelib behavior where the directory has 
no timestamps. So I think we need to take all these cases into account and do 
the fix.

My thought is we should keep the isFirstServer() check and have a different sub 
directory with hostname below oozie.service.WorkflowAppService.system.libpath 
directory (like your suggesstion) under which launcher libs will be shipped (if 
ship launcher is true). sharelib will continue to reside in the top-level of 
libpath directory.  purgeLibs(fs, LAUNCHER_PREFIX, current); can then be done 
outside the isFirstServer() check.

> Sharelib purging only occurs at Oozie startup
> -
>
> Key: OOZIE-1783
> URL: https://issues.apache.org/jira/browse/OOZIE-1783
> Project: Oozie
>  Issue Type: Bug
>Affects Versions: trunk
>Reporter: Robert Kanter
>Assignee: Robert Kanter
>Priority: Critical
> Attachments: OOZIE-1783.patch, OOZIE-1783.patch
>
>
> The purgeLibs method only gets called on startup of the first Oozie server.  
> This means that if you update the sharelib without restarting Oozie, then 
> Oozie will never clean up the old sharelibs and launcherlibs until you 
> restart the server.  In fact, with Oozie HA, it will never clean up unless 
> you restart all of the servers at the same time (in other words, a rolling 
> restart or just restarting one Oozie will never cause purgeLibs to get 
> called).  
> This should be moved into a background thread (i.e. scheduled with 
> SchedulerService) to run periodically.  Once a day is probably good enough, 
> but we could make it configurable if needed.  



--
This message was sent by Atlassian JIRA
(v6.2#6252)

[jira] [Commented] (OOZIE-1835) NullPointerException from SLAEmailEventListener

[Hadoop QA (JIRA)]

[ 
https://issues.apache.org/jira/browse/OOZIE-1835?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13999877#comment-13999877
 ] 

Hadoop QA commented on OOZIE-1835:
--

Testing JIRA OOZIE-1835

Cleaning local git workspace



{color:green}+1 PATCH_APPLIES{color}
{color:green}+1 CLEAN{color}
{color:green}+1 RAW_PATCH_ANALYSIS{color}
.{color:green}+1{color} the patch does not introduce any @author tags
.{color:green}+1{color} the patch does not introduce any tabs
.{color:green}+1{color} the patch does not introduce any trailing spaces
.{color:green}+1{color} the patch does not introduce any line longer than 
132
.{color:green}+1{color} the patch does adds/modifies 1 testcase(s)
{color:green}+1 RAT{color}
.{color:green}+1{color} the patch does not seem to introduce new RAT 
warnings
{color:green}+1 JAVADOC{color}
.{color:green}+1{color} the patch does not seem to introduce new Javadoc 
warnings
{color:green}+1 COMPILE{color}
.{color:green}+1{color} HEAD compiles
.{color:green}+1{color} patch compiles
.{color:green}+1{color} the patch does not seem to introduce new javac 
warnings
{color:green}+1 BACKWARDS_COMPATIBILITY{color}
.{color:green}+1{color} the patch does not change any JPA 
Entity/Colum/Basic/Lob/Transient annotations
.{color:green}+1{color} the patch does not modify JPA files
{color:red}-1 TESTS{color}
.Tests run: 1447
.Tests failed: 6
.Tests errors: 3

.The patch failed the following testcases:

.  
testConcurrencyReachedAndChooseNextEligible(org.apache.oozie.service.TestCallableQueueService)
.  
testCoordinatorActionCommandsSubmitAndStart(org.apache.oozie.sla.TestSLAEventGeneration)
.  
testBundleEngineKill(org.apache.oozie.servlet.TestV1JobServletBundleEngine)
.  
testBundleEngineResume(org.apache.oozie.servlet.TestV1JobServletBundleEngine)
.  
testBundleEngineSuspend(org.apache.oozie.servlet.TestV1JobServletBundleEngine)
.  
testTimeout(org.apache.oozie.command.coord.TestCoordActionInputCheckXCommand)

{color:green}+1 DISTRO{color}
.{color:green}+1{color} distro tarball builds with the patch 


{color:red}*-1 Overall result, please check the reported -1(s)*{color}


The full output of the test-patch run is available at

.   https://builds.apache.org/job/oozie-trunk-precommit-build/1244/

> NullPointerException from SLAEmailEventListener
> ---
>
> Key: OOZIE-1835
> URL: https://issues.apache.org/jira/browse/OOZIE-1835
> Project: Oozie
>  Issue Type: Bug
>  Components: core
>Affects Versions: 4.0.0
>Reporter: Robert Kanter
>Assignee: Robert Kanter
> Attachments: OOZIE-1835.patch
>
>
> Both the documentation and specification say that the "alert-contact" SLA 
> field is optional, at least when you don't want to send email notifications.  
> However, if you don't include it, you get a NullPointerException (though it 
> doesn't seem to cause any problems):
> {noformat}
> 2014-04-17 22:02:16,664 ERROR 
> org.apache.oozie.sla.listener.SLAEmailEventListener: 
> SERVER[rkanter-5.ent.cloudera.com] USER[-] GROUP[-] TOKEN[-] APP[-] JOB[-] 
> ACTION[-] Failed to send EndMiss alert email
> java.lang.NullPointerException
> at 
> org.apache.oozie.sla.listener.SLAEmailEventListener.parseAddress(SLAEmailEventListener.java:231)
> at 
> org.apache.oozie.sla.listener.SLAEmailEventListener.setMessageHeader(SLAEmailEventListener.java:262)
> at 
> org.apache.oozie.sla.listener.SLAEmailEventListener.sendSLAEmail(SLAEmailEventListener.java:163)
> at 
> org.apache.oozie.sla.listener.SLAEmailEventListener.onEndMiss(SLAEmailEventListener.java:200)
> at 
> org.apache.oozie.service.EventHandlerService$EventWorker.invokeSLAEventListener(EventHandlerService.java:297)
> at 
> org.apache.oozie.service.EventHandlerService$EventWorker.run(EventHandlerService.java:241)
> at 
> org.apache.oozie.service.SchedulerService$2.run(SchedulerService.java:174)
> at 
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
> at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:304)
> at 
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:178)
> at 
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
> at 
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
> at 
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
> at java.lang.Thread.run(Thread.java:744)
> {noformat}
> We should give a friendlier WARN message instead.



--
This message was sent by Atlassian JIRA
(v

[jira] [Commented] (OOZIE-1783) Sharelib purging only occurs at Oozie startup

[Robert Kanter (JIRA)]

[ 
https://issues.apache.org/jira/browse/OOZIE-1783?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14000263#comment-14000263
 ] 

Robert Kanter commented on OOZIE-1783:
--

Why would you want each server to have their own sharelib?  They should have 
identical sharelibs or else if you submit to different Oozie servers you will 
have different jars...

> Sharelib purging only occurs at Oozie startup
> -
>
> Key: OOZIE-1783
> URL: https://issues.apache.org/jira/browse/OOZIE-1783
> Project: Oozie
>  Issue Type: Bug
>Affects Versions: trunk
>Reporter: Robert Kanter
>Assignee: Robert Kanter
>Priority: Critical
> Attachments: OOZIE-1783.patch, OOZIE-1783.patch
>
>
> The purgeLibs method only gets called on startup of the first Oozie server.  
> This means that if you update the sharelib without restarting Oozie, then 
> Oozie will never clean up the old sharelibs and launcherlibs until you 
> restart the server.  In fact, with Oozie HA, it will never clean up unless 
> you restart all of the servers at the same time (in other words, a rolling 
> restart or just restarting one Oozie will never cause purgeLibs to get 
> called).  
> This should be moved into a background thread (i.e. scheduled with 
> SchedulerService) to run periodically.  Once a day is probably good enough, 
> but we could make it configurable if needed.  



--
This message was sent by Atlassian JIRA
(v6.2#6252)

[jira] [Commented] (OOZIE-1841) Allow for specifications of wf/coordinator propertey files from web api

[Josh Kurz (JIRA)]

[ 
https://issues.apache.org/jira/browse/OOZIE-1841?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14000211#comment-14000211
 ] 

Josh Kurz commented on OOZIE-1841:
--

If this sounds like a worthy feature to master, I would gladly work on the 
commit.

> Allow for specifications of wf/coordinator propertey files from web api
> ---
>
> Key: OOZIE-1841
> URL: https://issues.apache.org/jira/browse/OOZIE-1841
> Project: Oozie
>  Issue Type: New Feature
>Reporter: Josh Kurz
>
> I would like to be able to post to oozie/v1/jobs and just set a configuration 
> path. The xml files for each coordinator job that could be created are 
> already created and are on the uzi box (for my use case). Is there a way to 
> just tell oozie that I would like to use one of those pre made xml files vs 
> submitting xml with an http post. 
> curl -X POST -H "Content-Type: application/json" -d {"configUrl": 
> "/path/to/uzi/configuration"} "http://uzibox:11000/oozie/v1/jobs?action=start";



--
This message was sent by Atlassian JIRA
(v6.2#6252)

[jira] [Commented] (OOZIE-1783) Sharelib purging only occurs at Oozie startup

[Rohini Palaniswamy (JIRA)]

[ 
https://issues.apache.org/jira/browse/OOZIE-1783?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=1322#comment-1322
 ] 

Rohini Palaniswamy commented on OOZIE-1783:
---

+1

> Sharelib purging only occurs at Oozie startup
> -
>
> Key: OOZIE-1783
> URL: https://issues.apache.org/jira/browse/OOZIE-1783
> Project: Oozie
>  Issue Type: Bug
>Affects Versions: trunk
>Reporter: Robert Kanter
>Assignee: Robert Kanter
>Priority: Critical
> Attachments: OOZIE-1783.patch
>
>
> The purgeLibs method only gets called on startup of the first Oozie server.  
> This means that if you update the sharelib without restarting Oozie, then 
> Oozie will never clean up the old sharelibs and launcherlibs until you 
> restart the server.  In fact, with Oozie HA, it will never clean up unless 
> you restart all of the servers at the same time (in other words, a rolling 
> restart or just restarting one Oozie will never cause purgeLibs to get 
> called).  
> This should be moved into a background thread (i.e. scheduled with 
> SchedulerService) to run periodically.  Once a day is probably good enough, 
> but we could make it configurable if needed.  



--
This message was sent by Atlassian JIRA
(v6.2#6252)

[jira] [Updated] (OOZIE-1586) upgrade oozie to hive 13.1 (including hcatalog)

[Ravi Prakash (JIRA)]

 [ 
https://issues.apache.org/jira/browse/OOZIE-1586?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ravi Prakash updated OOZIE-1586:


Attachment: OOZIE-1586.patch

Here are the changes I had to make in order for Oozie to build and work with 
Hadoop 2.4, Hive 0.13.1 and Pig 0.12.1

> upgrade oozie to hive 13.1 (including hcatalog)
> ---
>
> Key: OOZIE-1586
> URL: https://issues.apache.org/jira/browse/OOZIE-1586
> Project: Oozie
>  Issue Type: Improvement
>Reporter: Bowen Zhang
>Assignee: Bowen Zhang
> Attachments: OOZIE-1586.patch
>
>




--
This message was sent by Atlassian JIRA
(v6.2#6252)

[jira] [Resolved] (OOZIE-1648) Update to Hive 0.13 when its available

[Robert Kanter (JIRA)]

 [ 
https://issues.apache.org/jira/browse/OOZIE-1648?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Robert Kanter resolved OOZIE-1648.
--

Resolution: Duplicate

> Update to Hive 0.13 when its available
> --
>
> Key: OOZIE-1648
> URL: https://issues.apache.org/jira/browse/OOZIE-1648
> Project: Oozie
>  Issue Type: Improvement
>Affects Versions: trunk
>Reporter: Robert Kanter
>
> We're currently on Hive 0.10; we should update to Hive 0.13 when it becomes 
> available.  This will also impact the HCatalog integration because HCat 
> packaging has changed.



--
This message was sent by Atlassian JIRA
(v6.2#6252)

[jira] [Commented] (OOZIE-1841) Allow for specifications of wf/coordinator propertey files from web api

[Josh Kurz (JIRA)]

[ 
https://issues.apache.org/jira/browse/OOZIE-1841?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14000206#comment-14000206
 ] 

Josh Kurz commented on OOZIE-1841:
--

1769 is about updating an existing coordinator and is a nice feature, but I 
would like to submit a new coordinator with the API and not have to submit a 
configuration xml with the post. I would rather just submit a json payload with 
a url that points to a properties file that has all of the information already. 
This post could also have information inside of it, that would overwrite items 
that are set in the properties file. 

Use case: (submit a coordinator)

post to /jobs with xml config || post to /jobs with json payload that points to 
location of properties file.

This would just allow a simpler way to specify properties when submitting jobs 
IMO. Currently we run jobs from the cmd on the oozie box itself. These 
properties files are created via chef for each possible job that we would like 
to run and are used in the oozie commands. We are building a UI that allows us 
to simply create our own jobs, which works with the oozie api. So instead of 
recreating these properties files in xml format to submit from the UI, I 
believe it would be much easier to just point to their location when submitting 
a job.  

> Allow for specifications of wf/coordinator propertey files from web api
> ---
>
> Key: OOZIE-1841
> URL: https://issues.apache.org/jira/browse/OOZIE-1841
> Project: Oozie
>  Issue Type: New Feature
>Reporter: Josh Kurz
>
> I would like to be able to post to oozie/v1/jobs and just set a configuration 
> path. The xml files for each coordinator job that could be created are 
> already created and are on the uzi box (for my use case). Is there a way to 
> just tell oozie that I would like to use one of those pre made xml files vs 
> submitting xml with an http post. 
> curl -X POST -H "Content-Type: application/json" -d {"configUrl": 
> "/path/to/uzi/configuration"} "http://uzibox:11000/oozie/v1/jobs?action=start";



--
This message was sent by Atlassian JIRA
(v6.2#6252)

[jira] [Commented] (OOZIE-1586) upgrade oozie to hive 13.1 (including hcatalog)

[Robert Kanter (JIRA)]

[ 
https://issues.apache.org/jira/browse/OOZIE-1586?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14000174#comment-14000174
 ] 

Robert Kanter commented on OOZIE-1586:
--

Ravi, thanks for putting up a patch for this.  A few points:
- It doesn't look like Hive 0.13.1 has been released yet and I don't see it at 
the 
[maven.org|http://search.maven.org/#search%7Cga%7C1%7Cg%3A%22org.apache.hive%22];
 are you building 13.1 locally and installing in your maven cache?
- Does Hive 0.13.1 require Hadoop 2.4.0 instead of 2.3.0?  Though I suppose 
it's good to upgrade Hadoop anyway; but we should probably update the name of 
this JIRA then
- Should we also get rid of hcatalog-0.5?
- We've already defined version 3.4 for antlr in the root pom, so you can 
simply remove the version for the antlr dependency in the hive sharelib to make 
it use 3.4
- For all of the hive dependency versions in hive-hcatalog-0.13.1's pom, I 
think we should hardcode the version to {{0.13.1}} instead of {{hive.version}} 
as it will mix jars if the user changes {{hive.version}}
- Have you done any testing?
-- Hive action
-- HCat integration
-- Hadoop 2.4.0

> upgrade oozie to hive 13.1 (including hcatalog)
> ---
>
> Key: OOZIE-1586
> URL: https://issues.apache.org/jira/browse/OOZIE-1586
> Project: Oozie
>  Issue Type: Improvement
>Reporter: Bowen Zhang
>Assignee: Bowen Zhang
> Attachments: OOZIE-1586.patch
>
>




--
This message was sent by Atlassian JIRA
(v6.2#6252)

[jira] [Commented] (OOZIE-1527) Fix scalability issues with coordinator materialization

[Purshotam Shah (JIRA)]

[ 
https://issues.apache.org/jira/browse/OOZIE-1527?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13999136#comment-13999136
 ] 

Purshotam Shah commented on OOZIE-1527:
---

{quote}
Previously lookup interval was scheduling interval. If it was set to 2 mins and 
scheduling interval is not overriden (default 5 mins), then actions can 
materialize slowly. Need to make scheduling interval=min(lookup interval, 
scheduling interval). Reopening the jira to fix that.
{quote}
If scheduling interval is specified then it should fall back on lookup interval.

> Fix scalability issues with coordinator materialization
> ---
>
> Key: OOZIE-1527
> URL: https://issues.apache.org/jira/browse/OOZIE-1527
> Project: Oozie
>  Issue Type: Bug
>  Components: coordinator
>Affects Versions: trunk
>Reporter: Mona Chitnis
>Assignee: Purshotam Shah
> Fix For: trunk
>
> Attachments: OOZIE-1527-V2.patch, OOZIE-1527-V3.patch
>
>   Original Estimate: 96h
>  Remaining Estimate: 96h
>
> In certain situations when there is a large number of coordinators in the 
> system, they have been observed to create huge backlog in materialization, 
> and progressing very slow compared to expected. This patch can be looked upon 
> as both a bug-fix or an enhancement addressing following points:
> 1. 'materialization.system.limit' leads to bringing Coord jobs in LRU 
> fashion, but some of them may already be maxing out at actions to materialize 
> (= throttle), and < limit jobs may actually undergo materialization. This 
> patch does a second iteration of loading jobs to get materialized to reduce 
> backlog
> 2. 'materialization.window' being 1 hour may work in most cases, but hourly 
> jobs are seen to face significant slowdown at times, by lot of other minute 
> jobs getting materialized. Therefore, window can be doubled (i.e. 2 hours) 
> when job is hourly/daily.
> 3. For hourly coordinators, it is consistently seen that materialization 
> occurs only near the end of the hour. e.g. for action whose nominal time is 
> 2:00, action creation time is 1:59, if nominal time - 3:00, creation time is 
> 2:58 and so on. If window is an hour in the future, doesn't explain why 
> materialization won't occur anytime in the middle of the preceding hour.



--
This message was sent by Atlassian JIRA
(v6.2#6252)

[jira] [Updated] (OOZIE-1829) URIHandlerService doesn't support URI schemes with query strings but no path segment

[Ben Roling (JIRA)]

 [ 
https://issues.apache.org/jira/browse/OOZIE-1829?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ben Roling updated OOZIE-1829:
--

Attachment: OOZIE-1829.patch

Patch for issue.

> URIHandlerService doesn't support URI schemes with query strings but no path 
> segment
> 
>
> Key: OOZIE-1829
> URL: https://issues.apache.org/jira/browse/OOZIE-1829
> Project: Oozie
>  Issue Type: Bug
>  Components: core
>Affects Versions: 4.0.1
>Reporter: Ben Roling
> Attachments: OOZIE-1829.patch
>
>
> While working on a prototype of integration between Oozie and the Kite SDK 
> (see https://issues.cloudera.org/browse/CDK-385), I came to find that 
> URIHandlerService.getAuthorityWithSchema(String uri) doesn't support URI 
> schemes where there is a query string, but no path segment.
> I am currently prototyping Kite Dataset URIs and in my prototype, a Dataset 
> URI for a dataset in a Hive/HCatalog DatasetRepository with managed 
> Hive/HCatalog tables could look like this:
> repo:hive://localhost:9043?dataset-name=Person&partition-key=\[201405091300\]
> I am attempting to create an Oozie dataset around this Kite dataset and to 
> make that happen I have implemented Oozie's URIHandler API for the Kite 
> "repo" URI scheme.  When I attempted to run my first coordinator, it failed.
> The coordinator has the following dataset definition:
> {code}
>  initial-instance="2014-04-24T00:00Z" timezone="UTC">
>   
> repo:hive://localhost:9083?dataset-name=Person&partition-key=[${YEAR}${MONTH}${DAY}${HOUR}${MINUTE}]
>   
> 
> {code}
> This dataset is used as an output of the coordinator.
> When the coordinator is submitted it fails with the following exception:
> {code}
> 2014-05-09 10:57:34,991 ERROR 
> org.apache.oozie.command.coord.CoordMaterializeTransitionXCommand: 
> SERVER[localhost.localdomain] USER[cloudera] GROUP[-] TOKEN[] APP[Person-c] 
> JOB[013-140508121805317-oozie-oozi-C] ACTION[-] Exception occurred:E0906: 
> URI parsing error : 
> repo:hive://localhost:9083?dataset-name=Person&partition-key=[${YEAR}${MONTH}${DAY}${HOUR}${MINUTE}]
>  Making the job failed 
> org.apache.oozie.dependency.URIHandlerException: E0906: URI parsing error : 
> repo:hive://localhost:9083?dataset-name=Person&partition-key=[${YEAR}${MONTH}${DAY}${HOUR}${MINUTE}]
>   at 
> org.apache.oozie.service.URIHandlerService.getAuthorityWithScheme(URIHandlerService.java:216)
>   at 
> org.apache.oozie.command.coord.CoordCommandUtils.materializeDataEvents(CoordCommandUtils.java:582)
>   at 
> org.apache.oozie.command.coord.CoordCommandUtils.materializeOneInstance(CoordCommandUtils.java:451)
>   at 
> org.apache.oozie.command.coord.CoordMaterializeTransitionXCommand.materializeActions(CoordMaterializeTransitionXCommand.java:386)
>   at 
> org.apache.oozie.command.coord.CoordMaterializeTransitionXCommand.materialize(CoordMaterializeTransitionXCommand.java:267)
>   at 
> org.apache.oozie.command.MaterializeTransitionXCommand.execute(MaterializeTransitionXCommand.java:72)
>   at 
> org.apache.oozie.command.MaterializeTransitionXCommand.execute(MaterializeTransitionXCommand.java:28)
>   at org.apache.oozie.command.XCommand.call(XCommand.java:280)
>   at 
> org.apache.oozie.service.CallableQueueService$CallableWrapper.run(CallableQueueService.java:174)
>   at 
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
>   at 
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
>   at java.lang.Thread.run(Thread.java:744)
> Caused by: java.net.URISyntaxException: Illegal character in opaque part at 
> index 63: 
> repo:hive://localhost:9083?dataset-name=Person&partition-key=[${YEAR}${MONTH}${DAY}${HOUR}${MINUTE}]
>   at java.net.URI$Parser.fail(URI.java:2829)
>   at java.net.URI$Parser.checkChars(URI.java:3002)
>   at java.net.URI$Parser.parse(URI.java:3039)
>   at java.net.URI.(URI.java:595)
>   at 
> org.apache.oozie.service.URIHandlerService.getAuthorityWithScheme(URIHandlerService.java:209)
>   ... 11 more
> {code}
> The problem is that URIHandlerService.getAuthorityWithScheme(String uri) 
> doesn't consider the possibility that the URI might have a query string and 
> no path segment.  As a result, it ends up trying to create a URI from the 
> entire URI template, which blows up due to $ in the template parameters.



--
This message was sent by Atlassian JIRA
(v6.2#6252)

Re: Review Request 21046: OOZIE-1715 Distributed ID sequence for HA

[Purshotam Shah]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/21046/
---

(Updated May 15, 2014, 6:43 p.m.)


Review request for oozie.


Bugs: OOZIE-1715
https://issues.apache.org/jira/browse/OOZIE-1715


Repository: oozie-git


Description
---

OOZIE-1715 Distributed ID sequence for HA


Diffs (updated)
-

  core/src/main/java/org/apache/oozie/service/DistributedUUIDService.java 
e69de29 
  core/src/main/java/org/apache/oozie/service/UUIDService.java 836815d 
  core/src/main/java/org/apache/oozie/service/ZKUUIDService.java e69de29 
  core/src/main/java/org/apache/oozie/util/ZKUtils.java 56055b8 
  core/src/test/java/org/apache/oozie/service/TestDistributedUUIDService.java 
e69de29 
  core/src/test/java/org/apache/oozie/service/TestZKUUIDService.java e69de29 

Diff: https://reviews.apache.org/r/21046/diff/


Testing
---

UTC


Thanks,

Purshotam Shah

Re: Review Request 21046: OOZIE-1715 Distributed ID sequence for HA

[Purshotam Shah]

> On May 13, 2014, 9:55 p.m., Robert Kanter wrote:
> > ID's usually look like this: 000-140505141334644-oozie-oozi-W
> > where the second value is a timestamp and the first value is the increasing 
> > part.  
> > 
> > With the ZKUUIDService, will all Oozie servers use the same timestamp part? 
> >  If not, then having the first part shared among the servers doesn't get 
> > you too much; so I think we need to have a way of making the timestamp 
> > shared as well.  
> > 
> > Also, when you reset the sequence, isn't it possible to still cause 
> > collisions if someone has a really old job still sitting around?  I agree 
> > that it's probably not likely given the number of jobs they'd have to run 
> > without cleaning up, but still.  If you also reset the timestamp to "now" 
> > when reseting the atomic long, that should take care of this.

>With the ZKUUIDService, will all Oozie servers use the same timestamp part?  
>If not, then having the first part shared among the servers doesn't get you 
>too much; so >I think we need to have a way of making the timestamp shared as 
>well.  

In HA Timestamp may tell, where job was submitted.


>Also, when you reset the sequence, isn't it possible to still cause collisions 
>if someone has a really old job still sitting around?  I agree that it's 
>probably not >likely given the number of jobs they'd have to run without 
>cleaning up, but still.  If you also reset the timestamp to "now" when 
>reseting the atomic long, that >should take care of this.

Once we restart, timestamp will change. If one knows that there are huge jobs 
submission and less purging, they can increase max count. 
Reseting timestamp to "now" might take out the purpose of appending timestamp 
to jobid.

I believe taking out timestamp for HA is debatable. I will create a JIRA and if 
all agrees we can take it out.


> On May 13, 2014, 9:55 p.m., Robert Kanter wrote:
> > core/src/main/java/org/apache/oozie/service/ZKUUIDService.java, line 55
> > 
> >
> > I'm not sure we should make the path configurable; I'm concerned that 
> > will just lead to a user accidentally setting them different in different 
> > servers and then running into problems because Oozie servers are using 
> > different paths.  Or what happens if they try to use a patch Oozie is 
> > already using for something else?
> > 
> > Also, please verify that when you run Oozie with ZKUUIDService, the 
> > path gets created under the namespace set for the server (default "oozie") 
> > and not simply at the root.  e.g. /oozie/job_id_sequence
> > 
> > Also, can you add a class javadoc comment to ZKUUIDService mentioning 
> > the path it's using (see ZKUtils and ZKLocksService for an example of what 
> > I mean)

>I'm not sure we should make the path configurable; I'm concerned that will 
>just lead to a user accidentally setting them different in different servers 
>and then >running into problems because Oozie servers are using different 
>paths.  Or what happens if they try to use a patch Oozie is already using for 
>something else?

Even Namespace is configurable, it should be ok. Most of the people will verify 
their setting before moving to production.


I have verified and our QE will also verify that. Path are created under 
namespace.


> On May 13, 2014, 9:55 p.m., Robert Kanter wrote:
> > core/src/main/java/org/apache/oozie/service/ZKUUIDService.java, line 90
> > 
> >
> > I think you mean "increment" not "reset" here?

I mean reset, we try to reset three time, before throwing exception.


- Purshotam


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/21046/#review42891
---


On May 13, 2014, 5:32 p.m., Purshotam Shah wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/21046/
> ---
> 
> (Updated May 13, 2014, 5:32 p.m.)
> 
> 
> Review request for oozie.
> 
> 
> Bugs: OOZIE-1715
> https://issues.apache.org/jira/browse/OOZIE-1715
> 
> 
> Repository: oozie-git
> 
> 
> Description
> ---
> 
> OOZIE-1715 Distributed ID sequence for HA
> 
> 
> Diffs
> -
> 
>   core/src/main/java/org/apache/oozie/service/DistributedUUIDService.java 
> e69de29 
>   core/src/main/java/org/apache/oozie/service/UUIDService.java 836815d 
>   core/src/main/java/org/apache/oozie/service/ZKUUIDService.java e69de29 
>   core/src/main/java/org/apache/oozie/util/ZKUtils.java 56055b8 
>   core/src/test/java/org/apache/oozie/service/TestDistributedUUIDService.java 
> e69de29 
>   core/src/test/java/org/apache/oozie/service/TestZKUUIDService.java e69de29 
> 
> Di

[jira] [Commented] (OOZIE-1803) Improvement in Purge service

[Hadoop QA (JIRA)]

[ 
https://issues.apache.org/jira/browse/OOZIE-1803?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13999313#comment-13999313
 ] 

Hadoop QA commented on OOZIE-1803:
--

Testing JIRA OOZIE-1803

Cleaning local git workspace



{color:red}-1{color} Patch failed to apply to head of branch



> Improvement in Purge service
> 
>
> Key: OOZIE-1803
> URL: https://issues.apache.org/jira/browse/OOZIE-1803
> Project: Oozie
>  Issue Type: Improvement
>  Components: core
>Reporter: Jaydeep Vishwakarma
>Assignee: Jaydeep Vishwakarma
> Attachments: purgeservice.patch
>
>
> Current purge service of oozie have some performance issues and it might help 
> to look at the queries and indexes to improve the the purge service.



--
This message was sent by Atlassian JIRA
(v6.2#6252)

Re: What are the plans for an Oozie 4.1.0 release?

[Alejandro Abdelnur]

the fact that other projects stay un the minor version to 'allow' incompat 
changes whenever they want, well it is a matter of those projects and their 
communities. 

i'm proud that oozie has a great track of record in that front, including among 
(yes among, not only between) major versions. and we should keep it that way. 

regarding db schema upgrades, until now we were saying that it should be only 
in major upgrades.  given the fact that the dbtools manages upgrades in a very 
unpainful way, and this has happened for a while now, i think it should be fine 
for a minor release to require a dbtool run. we just have to make sure this is 
documented, and that if the db has not been upgraded the new oozie does not 
start (which i think it is already happening)

thx


Alejandro
(phone typing)

> On May 14, 2014, at 16:48, Robert Kanter  wrote:
> 
> During the 3.x releases, we had an optional database change for MySQL to
> change TEXT to MEDIUMTEXT; I don't think there was a required one, but I
> don't remember.
> 
> In any case, that's a good point about the other projects and how they
> handle these things.  So I'd agree then that we can have a database
> upgrade.  Bowen, please make sure to verify that the database upgrade works
> for 2.x to 4.1, 3.x to 4.1, and 4.0 to 4.1 and for each type.
> 
> 
> On Wed, May 14, 2014 at 11:07 AM, Rohini Palaniswamy <
> rohini.adi...@gmail.com> wrote:
> 
>> Not sure. I remember someone mentioning that schema upgrade should have
>> major version incremented in case of Oozie. But there was one release (3.1
>> ->3.2 or something before I started working on Oozie) which had schema
>> change just incrementing minor version. If there was an already agreed
>> procedure we should follow that else I am fine either way and ok going with
>> 4.1 (as long it is atleast minor version and not patch version that we are
>> changing) considering every other hadoop project still have major version
>> as 0 and only keep updating minor version. For eg: hive 0.12 -> 0.13 is a
>> major release with schema changes. If we keep incrementing major version
>> very often it feels slightly weird when comparing to other hadoop projects
>> which are still in 0.x :).
>> 
>> 
>> On Tue, May 13, 2014 at 3:22 PM, Robert Kanter 
>> wrote:
>> 
>>> I agree; we'd also have to likely make two versions of each patch going
>>> forward for 4.x and 5.x/trunk otherwise.
>>> But are we "allowed" to make Oozie 4.0.x --> 4.1.0 require a database
>>> upgrade?
>>> 
>>> 
>>> On Tue, May 13, 2014 at 2:30 PM, Rohini Palaniswamy <
>>> rohini.adi...@gmail.com
 wrote:
>>> 
 It might be tough work as the CLOB->BLOB conversion change touches a
>> lot
>>> of
 classes.
 
 
 On Sun, May 11, 2014 at 9:25 PM, Robert Kanter 
 wrote:
 
> Hi everyone,
> 
> Should we start thinking about a 4.1.0 release?  It's been a while
>>> since
> 4.0.0 (4.0.1 was to fix some critical things like not being able to
> compile) and I think we have about 200 additional JIRAs in master.
> 
> Are there any specific features that we'd want to put in 4.1.0 and
>> wait
> for?
> 
> Another thing to keep in mind is that we can't just take everything
>> in
> master; I forget what exactly, but there's some JIRAs that change the
> database that we'll have to save for 5.0.0.
> 
> Does anybody want to volunteer to drive the 4.1.0 release?
> 
> 
> thanks
> - Robert
>> 

[jira] [Commented] (OOZIE-1586) upgrade oozie to hive 13.1 (including hcatalog)

[Ravi Prakash (JIRA)]

[ 
https://issues.apache.org/jira/browse/OOZIE-1586?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=1440#comment-1440
 ] 

Ravi Prakash commented on OOZIE-1586:
-

These would be applied on top of release-4.0.1


> upgrade oozie to hive 13.1 (including hcatalog)
> ---
>
> Key: OOZIE-1586
> URL: https://issues.apache.org/jira/browse/OOZIE-1586
> Project: Oozie
>  Issue Type: Improvement
>Reporter: Bowen Zhang
>Assignee: Bowen Zhang
> Attachments: OOZIE-1586.patch
>
>




--
This message was sent by Atlassian JIRA
(v6.2#6252)

[jira] [Commented] (OOZIE-1842) publish oozie artifacts

[Robert Kanter (JIRA)]

[ 
https://issues.apache.org/jira/browse/OOZIE-1842?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=1453#comment-1453
 ] 

Robert Kanter commented on OOZIE-1842:
--

I think I gave this link on an email somewhere, but here's the directions for 
publishing to ASF's maven repo:
http://www.apache.org/dev/publishing-maven-artifacts.html

We'll want to do this for each release going forward, so please update the "How 
to Release" page in the cwiki (I'd give a more direct link, but it's currently 
not loading for me; maybe it's down right now):
https://cwiki.apache.org/confluence/display/OOZIE/Developer+Resources

> publish oozie artifacts
> ---
>
> Key: OOZIE-1842
> URL: https://issues.apache.org/jira/browse/OOZIE-1842
> Project: Oozie
>  Issue Type: New Feature
>Reporter: Bowen Zhang
>Assignee: Mona Chitnis
>




--
This message was sent by Atlassian JIRA
(v6.2#6252)

[jira] [Updated] (OOZIE-1783) Sharelib purging only occurs at Oozie startup

[Robert Kanter (JIRA)]

 [ 
https://issues.apache.org/jira/browse/OOZIE-1783?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Robert Kanter updated OOZIE-1783:
-

Attachment: OOZIE-1783.patch

New patch fixes related test failure in TestOozieShareLibCLI; I had to add the 
SchedulerService to the list of Services it was starting.  Also, 
TestOozieShareLibCLI was never destroying the Services, so I added that.

> Sharelib purging only occurs at Oozie startup
> -
>
> Key: OOZIE-1783
> URL: https://issues.apache.org/jira/browse/OOZIE-1783
> Project: Oozie
>  Issue Type: Bug
>Affects Versions: trunk
>Reporter: Robert Kanter
>Assignee: Robert Kanter
>Priority: Critical
> Attachments: OOZIE-1783.patch, OOZIE-1783.patch
>
>
> The purgeLibs method only gets called on startup of the first Oozie server.  
> This means that if you update the sharelib without restarting Oozie, then 
> Oozie will never clean up the old sharelibs and launcherlibs until you 
> restart the server.  In fact, with Oozie HA, it will never clean up unless 
> you restart all of the servers at the same time (in other words, a rolling 
> restart or just restarting one Oozie will never cause purgeLibs to get 
> called).  
> This should be moved into a background thread (i.e. scheduled with 
> SchedulerService) to run periodically.  Once a day is probably good enough, 
> but we could make it configurable if needed.  



--
This message was sent by Atlassian JIRA
(v6.2#6252)

[jira] [Commented] (OOZIE-1838) jdbc.connections.active sampler does not show up

[Robert Kanter (JIRA)]

[ 
https://issues.apache.org/jira/browse/OOZIE-1838?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=1407#comment-1407
 ] 

Robert Kanter commented on OOZIE-1838:
--

Test failures are unrelated.
persistence.xml is modified to on purpose to switch from 
InstrumentedBasicDataSource to BasicDataSource.

> jdbc.connections.active sampler does not show up
> 
>
> Key: OOZIE-1838
> URL: https://issues.apache.org/jira/browse/OOZIE-1838
> Project: Oozie
>  Issue Type: Bug
>  Components: core
>Affects Versions: 4.0.0
>Reporter: Robert Kanter
>Assignee: Robert Kanter
> Attachments: OOZIE-1838.patch
>
>
> I'm not sure how long this has been missing, but I don't ever remember seeing 
> a "jdbc.connections.active" sampler in the instrumentation.  However, I did 
> stumble upon it in the code.  It's supposed to sample the number of active 
> JDBC connections.



--
This message was sent by Atlassian JIRA
(v6.2#6252)

[jira] [Commented] (OOZIE-1835) NullPointerException from SLAEmailEventListener

[Rohini Palaniswamy (JIRA)]

[ 
https://issues.apache.org/jira/browse/OOZIE-1835?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=1320#comment-1320
 ] 

Rohini Palaniswamy commented on OOZIE-1835:
---

+1

> NullPointerException from SLAEmailEventListener
> ---
>
> Key: OOZIE-1835
> URL: https://issues.apache.org/jira/browse/OOZIE-1835
> Project: Oozie
>  Issue Type: Bug
>  Components: core
>Affects Versions: 4.0.0
>Reporter: Robert Kanter
>Assignee: Robert Kanter
> Attachments: OOZIE-1835.patch
>
>
> Both the documentation and specification say that the "alert-contact" SLA 
> field is optional, at least when you don't want to send email notifications.  
> However, if you don't include it, you get a NullPointerException (though it 
> doesn't seem to cause any problems):
> {noformat}
> 2014-04-17 22:02:16,664 ERROR 
> org.apache.oozie.sla.listener.SLAEmailEventListener: 
> SERVER[rkanter-5.ent.cloudera.com] USER[-] GROUP[-] TOKEN[-] APP[-] JOB[-] 
> ACTION[-] Failed to send EndMiss alert email
> java.lang.NullPointerException
> at 
> org.apache.oozie.sla.listener.SLAEmailEventListener.parseAddress(SLAEmailEventListener.java:231)
> at 
> org.apache.oozie.sla.listener.SLAEmailEventListener.setMessageHeader(SLAEmailEventListener.java:262)
> at 
> org.apache.oozie.sla.listener.SLAEmailEventListener.sendSLAEmail(SLAEmailEventListener.java:163)
> at 
> org.apache.oozie.sla.listener.SLAEmailEventListener.onEndMiss(SLAEmailEventListener.java:200)
> at 
> org.apache.oozie.service.EventHandlerService$EventWorker.invokeSLAEventListener(EventHandlerService.java:297)
> at 
> org.apache.oozie.service.EventHandlerService$EventWorker.run(EventHandlerService.java:241)
> at 
> org.apache.oozie.service.SchedulerService$2.run(SchedulerService.java:174)
> at 
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
> at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:304)
> at 
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:178)
> at 
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
> at 
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
> at 
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
> at java.lang.Thread.run(Thread.java:744)
> {noformat}
> We should give a friendlier WARN message instead.



--
This message was sent by Atlassian JIRA
(v6.2#6252)

[jira] [Commented] (OOZIE-1829) URIHandlerService doesn't support URI schemes with query strings but no path segment

[Robert Kanter (JIRA)]

[ 
https://issues.apache.org/jira/browse/OOZIE-1829?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=1307#comment-1307
 ] 

Robert Kanter commented on OOZIE-1829:
--

You have to specify the {{--no-prefix}} argument to git when generating patches 
so it doesn't include the "a" and "b" things; then the patch should apply.  Can 
you regenerate the patch and reupload it?  

> URIHandlerService doesn't support URI schemes with query strings but no path 
> segment
> 
>
> Key: OOZIE-1829
> URL: https://issues.apache.org/jira/browse/OOZIE-1829
> Project: Oozie
>  Issue Type: Bug
>  Components: core
>Affects Versions: 4.0.1
>Reporter: Ben Roling
> Attachments: OOZIE-1829.patch
>
>
> While working on a prototype of integration between Oozie and the Kite SDK 
> (see https://issues.cloudera.org/browse/CDK-385), I came to find that 
> URIHandlerService.getAuthorityWithSchema(String uri) doesn't support URI 
> schemes where there is a query string, but no path segment.
> I am currently prototyping Kite Dataset URIs and in my prototype, a Dataset 
> URI for a dataset in a Hive/HCatalog DatasetRepository with managed 
> Hive/HCatalog tables could look like this:
> repo:hive://localhost:9043?dataset-name=Person&partition-key=\[201405091300\]
> I am attempting to create an Oozie dataset around this Kite dataset and to 
> make that happen I have implemented Oozie's URIHandler API for the Kite 
> "repo" URI scheme.  When I attempted to run my first coordinator, it failed.
> The coordinator has the following dataset definition:
> {code}
>  initial-instance="2014-04-24T00:00Z" timezone="UTC">
>   
> repo:hive://localhost:9083?dataset-name=Person&partition-key=[${YEAR}${MONTH}${DAY}${HOUR}${MINUTE}]
>   
> 
> {code}
> This dataset is used as an output of the coordinator.
> When the coordinator is submitted it fails with the following exception:
> {code}
> 2014-05-09 10:57:34,991 ERROR 
> org.apache.oozie.command.coord.CoordMaterializeTransitionXCommand: 
> SERVER[localhost.localdomain] USER[cloudera] GROUP[-] TOKEN[] APP[Person-c] 
> JOB[013-140508121805317-oozie-oozi-C] ACTION[-] Exception occurred:E0906: 
> URI parsing error : 
> repo:hive://localhost:9083?dataset-name=Person&partition-key=[${YEAR}${MONTH}${DAY}${HOUR}${MINUTE}]
>  Making the job failed 
> org.apache.oozie.dependency.URIHandlerException: E0906: URI parsing error : 
> repo:hive://localhost:9083?dataset-name=Person&partition-key=[${YEAR}${MONTH}${DAY}${HOUR}${MINUTE}]
>   at 
> org.apache.oozie.service.URIHandlerService.getAuthorityWithScheme(URIHandlerService.java:216)
>   at 
> org.apache.oozie.command.coord.CoordCommandUtils.materializeDataEvents(CoordCommandUtils.java:582)
>   at 
> org.apache.oozie.command.coord.CoordCommandUtils.materializeOneInstance(CoordCommandUtils.java:451)
>   at 
> org.apache.oozie.command.coord.CoordMaterializeTransitionXCommand.materializeActions(CoordMaterializeTransitionXCommand.java:386)
>   at 
> org.apache.oozie.command.coord.CoordMaterializeTransitionXCommand.materialize(CoordMaterializeTransitionXCommand.java:267)
>   at 
> org.apache.oozie.command.MaterializeTransitionXCommand.execute(MaterializeTransitionXCommand.java:72)
>   at 
> org.apache.oozie.command.MaterializeTransitionXCommand.execute(MaterializeTransitionXCommand.java:28)
>   at org.apache.oozie.command.XCommand.call(XCommand.java:280)
>   at 
> org.apache.oozie.service.CallableQueueService$CallableWrapper.run(CallableQueueService.java:174)
>   at 
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
>   at 
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
>   at java.lang.Thread.run(Thread.java:744)
> Caused by: java.net.URISyntaxException: Illegal character in opaque part at 
> index 63: 
> repo:hive://localhost:9083?dataset-name=Person&partition-key=[${YEAR}${MONTH}${DAY}${HOUR}${MINUTE}]
>   at java.net.URI$Parser.fail(URI.java:2829)
>   at java.net.URI$Parser.checkChars(URI.java:3002)
>   at java.net.URI$Parser.parse(URI.java:3039)
>   at java.net.URI.(URI.java:595)
>   at 
> org.apache.oozie.service.URIHandlerService.getAuthorityWithScheme(URIHandlerService.java:209)
>   ... 11 more
> {code}
> The problem is that URIHandlerService.getAuthorityWithScheme(String uri) 
> doesn't consider the possibility that the URI might have a query string and 
> no path segment.  As a result, it ends up trying to create a URI from the 
> entire URI template, which blows up due to $ in the template parameters.



--
This message was sent by Atlassian JIRA
(v6.2#6252)

Build failed in Jenkins: oozie-trunk-precommit-build #1222

[Apache Jenkins Server]

See 

--
[...truncated 8723 lines...]
[INFO] share/lib already added, skipping
[INFO]  already added, skipping
[INFO] share already added, skipping
[INFO] share/lib already added, skipping
[INFO]  already added, skipping
[INFO] share already added, skipping
[INFO] share/lib already added, skipping
[INFO] Building tar : 

[INFO]  already added, skipping
[INFO] share already added, skipping
[INFO] share/lib already added, skipping
[INFO]  already added, skipping
[INFO] share already added, skipping
[INFO] share/lib already added, skipping
[INFO]  already added, skipping
[INFO] share already added, skipping
[INFO] share/lib already added, skipping
[INFO]  already added, skipping
[INFO] share already added, skipping
[INFO] share/lib already added, skipping
[INFO]  already added, skipping
[INFO] share already added, skipping
[INFO] share/lib already added, skipping
[INFO]  already added, skipping
[INFO] share already added, skipping
[INFO] share/lib already added, skipping
[INFO] 
[INFO] 
[INFO] Building Apache Oozie Tools 4.1.0-SNAPSHOT
[INFO] 
[WARNING] Could not transfer metadata asm:asm/maven-metadata.xml from/to 
local.repository (file:../../local.repository/trunk): No connector available to 
access repository local.repository (file:../../local.repository/trunk) of type 
legacy using the available factories WagonRepositoryConnectorFactory
[WARNING] Could not transfer metadata asm:asm/maven-metadata.xml from/to 
local.repository (file:../../local.repository/trunk): No connector available to 
access repository local.repository (file:../../local.repository/trunk) of type 
legacy using the available factories WagonRepositoryConnectorFactory
[INFO] 
[INFO] --- maven-resources-plugin:2.5:resources (default-resources) @ 
oozie-tools ---
[debug] execute contextualize
[INFO] Using 'UTF-8' encoding to copy filtered resources.
[INFO] skip non existing resourceDirectory 

[INFO] 
[INFO] --- maven-compiler-plugin:2.3.2:compile (default-compile) @ oozie-tools 
---
[INFO] Nothing to compile - all classes are up to date
[INFO] 
[INFO] --- maven-resources-plugin:2.5:testResources (default-testResources) @ 
oozie-tools ---
[debug] execute contextualize
[INFO] Using 'UTF-8' encoding to copy filtered resources.
[INFO] Copying 2 resources
[INFO] 
[INFO] --- maven-compiler-plugin:2.3.2:testCompile (default-testCompile) @ 
oozie-tools ---
[INFO] Nothing to compile - all classes are up to date
[INFO] 
[INFO] --- maven-surefire-plugin:2.12:test (default-test) @ oozie-tools ---
[INFO] Tests are skipped.
[INFO] 
[INFO] --- maven-jar-plugin:2.3.1:jar (default-jar) @ oozie-tools ---
[INFO] Building jar: 

[INFO] 
[INFO] --- maven-assembly-plugin:2.2.1:single (default-cli) @ oozie-tools ---
[INFO] Reading assembly descriptor: ../src/main/assemblies/tools.xml
[WARNING] The following patterns were never triggered in this artifact 
exclusion filter:
o  '*:*:pom:*'

[INFO] Copying files to 

[WARNING] Assembly file: 

 is not a regular file (it may be a directory). It cannot be attached to the 
project build for installation or deployment.
[INFO] 
[INFO] 
[INFO] Building Apache Oozie MiniOozie 4.1.0-SNAPSHOT
[INFO] 
[INFO] 
[INFO] --- maven-resources-plugin:2.5:resources (default-resources) @ 
oozie-mini ---
[debug] execute contextualize
[INFO] Using 'UTF-8' encoding to copy filtered resources.
[INFO] skip non existing resourceDirectory 

[INFO] 
[INFO] --- maven-compiler-plugin:2.3.2:compile (default-compile) @ oozie-mini 
---
[INFO] No sources to compile
[INFO] 
[INFO] --- maven-resources-plugin:2.5:testResources (default-testResources) @ 
oozie-mini ---
[debug] execute contextualize
[INFO] Using 'UTF-8' encoding to copy filtered resources.
[INFO] Copying 3 resources
[INFO] 
[INFO] --- maven-compiler-plugin:2.3.2:testCompile (default-testCompile) @ 
oozie-mini ---
[INFO] Nothing to compile - all classes are u

[jira] [Updated] (OOZIE-1803) Improvement in Purge service

[Jaydeep Vishwakarma (JIRA)]

 [ 
https://issues.apache.org/jira/browse/OOZIE-1803?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jaydeep Vishwakarma updated OOZIE-1803:
---

Attachment: purgeservice.patch

bulk delete 
adding index 

> Improvement in Purge service
> 
>
> Key: OOZIE-1803
> URL: https://issues.apache.org/jira/browse/OOZIE-1803
> Project: Oozie
>  Issue Type: Improvement
>  Components: core
>Reporter: Jaydeep Vishwakarma
>Assignee: Jaydeep Vishwakarma
> Attachments: purgeservice.patch
>
>
> Current purge service of oozie have some performance issues and it might help 
> to look at the queries and indexes to improve the the purge service.



--
This message was sent by Atlassian JIRA
(v6.2#6252)

[jira] [Commented] (OOZIE-1688) New configuration to specify server-server authentication type.

[Hadoop QA (JIRA)]

[ 
https://issues.apache.org/jira/browse/OOZIE-1688?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13999702#comment-13999702
 ] 

Hadoop QA commented on OOZIE-1688:
--

Testing JIRA OOZIE-1688

Cleaning local git workspace



{color:green}+1 PATCH_APPLIES{color}
{color:green}+1 CLEAN{color}
{color:red}-1 RAW_PATCH_ANALYSIS{color}
.{color:green}+1{color} the patch does not introduce any @author tags
.{color:green}+1{color} the patch does not introduce any tabs
.{color:green}+1{color} the patch does not introduce any trailing spaces
.{color:green}+1{color} the patch does not introduce any line longer than 
132
.{color:red}-1{color} the patch does not add/modify any testcase
{color:green}+1 RAT{color}
.{color:green}+1{color} the patch does not seem to introduce new RAT 
warnings
{color:green}+1 JAVADOC{color}
.{color:green}+1{color} the patch does not seem to introduce new Javadoc 
warnings
{color:green}+1 COMPILE{color}
.{color:green}+1{color} HEAD compiles
.{color:green}+1{color} patch compiles
.{color:green}+1{color} the patch does not seem to introduce new javac 
warnings
{color:green}+1 BACKWARDS_COMPATIBILITY{color}
.{color:green}+1{color} the patch does not change any JPA 
Entity/Colum/Basic/Lob/Transient annotations
.{color:green}+1{color} the patch does not modify JPA files
{color:red}-1 TESTS{color}
.Tests run: 1446
.Tests failed: 3
.Tests errors: 2

.The patch failed the following testcases:

.  
testCoordinatorActionCommandsSubmitAndStart(org.apache.oozie.sla.TestSLAEventGeneration)
.  
testBundleEngineKill(org.apache.oozie.servlet.TestV1JobServletBundleEngine)
.  
testBundleEngineResume(org.apache.oozie.servlet.TestV1JobServletBundleEngine)

{color:red}-1 DISTRO{color}
.{color:red}-1{color} distro tarball fails with the patch


{color:red}*-1 Overall result, please check the reported -1(s)*{color}


The full output of the test-patch run is available at

.   https://builds.apache.org/job/oozie-trunk-precommit-build/1240/

> New configuration to specify server-server authentication type.
> ---
>
> Key: OOZIE-1688
> URL: https://issues.apache.org/jira/browse/OOZIE-1688
> Project: Oozie
>  Issue Type: Bug
>  Components: HA
>Reporter: Purshotam Shah
>Assignee: Purshotam Shah
> Attachments: OOZIE-1688-V4.patch, OOZIE-1688-v1.patch, 
> OOZIE-1688-v1.patch, OOZIE-1688-v2.patch, OOZIE-1688-v3.patch, 
> OOZIE-1688.patch
>
>
> Currently conf "oozie.authentication.type" is used by specify authentication 
> type for client-server as well as server-server.
> Client-server and server-server communication can use different type of 
> authentication.
> We need to have  different conf "oozie.server.authentication.type" to specify 
> server-server authentication type.



--
This message was sent by Atlassian JIRA
(v6.2#6252)

[jira] [Created] (OOZIE-1845) Coord submit command doesn't honor conf "materialization_window" while calling CoordMaterializeTransitionXCommand

[Purshotam Shah (JIRA)]

Purshotam Shah created OOZIE-1845:
-

 Summary: Coord submit command doesn't honor conf 
"materialization_window" while calling CoordMaterializeTransitionXCommand 
 Key: OOZIE-1845
 URL: https://issues.apache.org/jira/browse/OOZIE-1845
 Project: Oozie
  Issue Type: Bug
Reporter: Purshotam Shah






--
This message was sent by Atlassian JIRA
(v6.2#6252)

[jira] [Commented] (OOZIE-1775) TestEventGeneration.testCoordinatorActionEvent is failing and CoordRerunX should generate event

[Hadoop QA (JIRA)]

[ 
https://issues.apache.org/jira/browse/OOZIE-1775?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13999620#comment-13999620
 ] 

Hadoop QA commented on OOZIE-1775:
--

Testing JIRA OOZIE-1775

Cleaning local git workspace



{color:green}+1 PATCH_APPLIES{color}
{color:green}+1 CLEAN{color}
{color:green}+1 RAW_PATCH_ANALYSIS{color}
.{color:green}+1{color} the patch does not introduce any @author tags
.{color:green}+1{color} the patch does not introduce any tabs
.{color:green}+1{color} the patch does not introduce any trailing spaces
.{color:green}+1{color} the patch does not introduce any line longer than 
132
.{color:green}+1{color} the patch does adds/modifies 2 testcase(s)
{color:green}+1 RAT{color}
.{color:green}+1{color} the patch does not seem to introduce new RAT 
warnings
{color:green}+1 JAVADOC{color}
.{color:green}+1{color} the patch does not seem to introduce new Javadoc 
warnings
{color:green}+1 COMPILE{color}
.{color:green}+1{color} HEAD compiles
.{color:green}+1{color} patch compiles
.{color:green}+1{color} the patch does not seem to introduce new javac 
warnings
{color:green}+1 BACKWARDS_COMPATIBILITY{color}
.{color:green}+1{color} the patch does not change any JPA 
Entity/Colum/Basic/Lob/Transient annotations
.{color:green}+1{color} the patch does not modify JPA files
{color:red}-1 TESTS{color}
.Tests run: 1446
.Tests failed: 1
.Tests errors: 4

.The patch failed the following testcases:

.  
testBundleStatusNotTransitionFromKilled(org.apache.oozie.service.TestStatusTransitService)

{color:green}+1 DISTRO{color}
.{color:green}+1{color} distro tarball builds with the patch 


{color:red}*-1 Overall result, please check the reported -1(s)*{color}


The full output of the test-patch run is available at

.   https://builds.apache.org/job/oozie-trunk-precommit-build/1238/

> TestEventGeneration.testCoordinatorActionEvent is failing and CoordRerunX 
> should generate event
> ---
>
> Key: OOZIE-1775
> URL: https://issues.apache.org/jira/browse/OOZIE-1775
> Project: Oozie
>  Issue Type: Task
>Affects Versions: trunk
>Reporter: Mona Chitnis
>Assignee: Mona Chitnis
> Fix For: trunk
>
> Attachments: OOZIE-1175-2.patch, OOZIE-1775-1.patch, 
> OOZIE-1775-3.patch
>
>
> test is not flaky but consistently failing. this is due to 
> CoordResumeXCommand not publishing job event afterwards, for the resumed 
> coord actions.



--
This message was sent by Atlassian JIRA
(v6.2#6252)

[jira] [Commented] (OOZIE-1838) jdbc.connections.active sampler does not show up

[Hadoop QA (JIRA)]

[ 
https://issues.apache.org/jira/browse/OOZIE-1838?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13999661#comment-13999661
 ] 

Hadoop QA commented on OOZIE-1838:
--

Testing JIRA OOZIE-1838

Cleaning local git workspace



{color:green}+1 PATCH_APPLIES{color}
{color:green}+1 CLEAN{color}
{color:red}-1 RAW_PATCH_ANALYSIS{color}
.{color:green}+1{color} the patch does not introduce any @author tags
.{color:green}+1{color} the patch does not introduce any tabs
.{color:green}+1{color} the patch does not introduce any trailing spaces
.{color:green}+1{color} the patch does not introduce any line longer than 
132
.{color:red}-1{color} the patch does not add/modify any testcase
{color:green}+1 RAT{color}
.{color:green}+1{color} the patch does not seem to introduce new RAT 
warnings
{color:green}+1 JAVADOC{color}
.{color:green}+1{color} the patch does not seem to introduce new Javadoc 
warnings
{color:green}+1 COMPILE{color}
.{color:green}+1{color} HEAD compiles
.{color:green}+1{color} patch compiles
.{color:green}+1{color} the patch does not seem to introduce new javac 
warnings
{color:red}-1 BACKWARDS_COMPATIBILITY{color}
.{color:green}+1{color} the patch does not change any JPA 
Entity/Colum/Basic/Lob/Transient annotations
.{color:red}-1{color} the patch modifies 1 JPA file(s), persistence.xml or 
*-orm.xml
{color:red}-1 TESTS{color}
.Tests run: 1446
.Tests failed: 2
.Tests errors: 2

.The patch failed the following testcases:

.  
testCoordinatorActionCommandsSubmitAndStart(org.apache.oozie.sla.TestSLAEventGeneration)
.  
testBundleEngineResume(org.apache.oozie.servlet.TestV1JobServletBundleEngine)

{color:green}+1 DISTRO{color}
.{color:green}+1{color} distro tarball builds with the patch 


{color:red}*-1 Overall result, please check the reported -1(s)*{color}


The full output of the test-patch run is available at

.   https://builds.apache.org/job/oozie-trunk-precommit-build/1239/

> jdbc.connections.active sampler does not show up
> 
>
> Key: OOZIE-1838
> URL: https://issues.apache.org/jira/browse/OOZIE-1838
> Project: Oozie
>  Issue Type: Bug
>  Components: core
>Affects Versions: 4.0.0
>Reporter: Robert Kanter
>Assignee: Robert Kanter
> Attachments: OOZIE-1838.patch
>
>
> I'm not sure how long this has been missing, but I don't ever remember seeing 
> a "jdbc.connections.active" sampler in the instrumentation.  However, I did 
> stumble upon it in the code.  It's supposed to sample the number of active 
> JDBC connections.



--
This message was sent by Atlassian JIRA
(v6.2#6252)

[jira] [Commented] (OOZIE-1783) Sharelib purging only occurs at Oozie startup

[Hadoop QA (JIRA)]

[ 
https://issues.apache.org/jira/browse/OOZIE-1783?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13999798#comment-13999798
 ] 

Hadoop QA commented on OOZIE-1783:
--

Testing JIRA OOZIE-1783

Cleaning local git workspace



{color:green}+1 PATCH_APPLIES{color}
{color:green}+1 CLEAN{color}
{color:green}+1 RAW_PATCH_ANALYSIS{color}
.{color:green}+1{color} the patch does not introduce any @author tags
.{color:green}+1{color} the patch does not introduce any tabs
.{color:green}+1{color} the patch does not introduce any trailing spaces
.{color:green}+1{color} the patch does not introduce any line longer than 
132
.{color:green}+1{color} the patch does adds/modifies 1 testcase(s)
{color:green}+1 RAT{color}
.{color:green}+1{color} the patch does not seem to introduce new RAT 
warnings
{color:green}+1 JAVADOC{color}
.{color:green}+1{color} the patch does not seem to introduce new Javadoc 
warnings
{color:green}+1 COMPILE{color}
.{color:green}+1{color} HEAD compiles
.{color:green}+1{color} patch compiles
.{color:green}+1{color} the patch does not seem to introduce new javac 
warnings
{color:green}+1 BACKWARDS_COMPATIBILITY{color}
.{color:green}+1{color} the patch does not change any JPA 
Entity/Colum/Basic/Lob/Transient annotations
.{color:green}+1{color} the patch does not modify JPA files
{color:red}-1 TESTS{color}
.Tests run: 1446
.Tests failed: 3
.Tests errors: 3

.The patch failed the following testcases:

.  
testCoordinatorActionCommandsSubmitAndStart(org.apache.oozie.sla.TestSLAEventGeneration)
.  
testBundleEngineKill(org.apache.oozie.servlet.TestV1JobServletBundleEngine)
.  
testBundleEngineResume(org.apache.oozie.servlet.TestV1JobServletBundleEngine)

{color:green}+1 DISTRO{color}
.{color:green}+1{color} distro tarball builds with the patch 


{color:red}*-1 Overall result, please check the reported -1(s)*{color}


The full output of the test-patch run is available at

.   https://builds.apache.org/job/oozie-trunk-precommit-build/1243/

> Sharelib purging only occurs at Oozie startup
> -
>
> Key: OOZIE-1783
> URL: https://issues.apache.org/jira/browse/OOZIE-1783
> Project: Oozie
>  Issue Type: Bug
>Affects Versions: trunk
>Reporter: Robert Kanter
>Assignee: Robert Kanter
>Priority: Critical
> Attachments: OOZIE-1783.patch
>
>
> The purgeLibs method only gets called on startup of the first Oozie server.  
> This means that if you update the sharelib without restarting Oozie, then 
> Oozie will never clean up the old sharelibs and launcherlibs until you 
> restart the server.  In fact, with Oozie HA, it will never clean up unless 
> you restart all of the servers at the same time (in other words, a rolling 
> restart or just restarting one Oozie will never cause purgeLibs to get 
> called).  
> This should be moved into a background thread (i.e. scheduled with 
> SchedulerService) to run periodically.  Once a day is probably good enough, 
> but we could make it configurable if needed.  



--
This message was sent by Atlassian JIRA
(v6.2#6252)

Build failed in Jenkins: oozie-trunk-precommit-build #1243

[Apache Jenkins Server]

See 

--
[...truncated 9592 lines...]
[INFO] share/lib already added, skipping
[INFO]  already added, skipping
[INFO] share already added, skipping
[INFO] share/lib already added, skipping
[INFO]  already added, skipping
[INFO] share already added, skipping
[INFO] share/lib already added, skipping
[INFO]  already added, skipping
[INFO] share already added, skipping
[INFO] share/lib already added, skipping
[INFO]  already added, skipping
[INFO] share already added, skipping
[INFO] share/lib already added, skipping
[INFO]  already added, skipping
[INFO] share already added, skipping
[INFO] share/lib already added, skipping
[INFO] 
[INFO] 
[INFO] Building Apache Oozie Tools 4.1.0-SNAPSHOT
[INFO] 
[WARNING] Failure to transfer asm:asm/maven-metadata.xml from 
file:../../local.repository/trunk was cached in the local repository, 
resolution will not be reattempted until the update interval of 
local.repository has elapsed or updates are forced. Original error: Could not 
transfer metadata asm:asm/maven-metadata.xml from/to local.repository 
(file:../../local.repository/trunk): No connector available to access 
repository local.repository (file:../../local.repository/trunk) of type legacy 
using the available factories WagonRepositoryConnectorFactory
[WARNING] Failure to transfer asm:asm/maven-metadata.xml from 
file:../../local.repository/trunk was cached in the local repository, 
resolution will not be reattempted until the update interval of 
local.repository has elapsed or updates are forced. Original error: Could not 
transfer metadata asm:asm/maven-metadata.xml from/to local.repository 
(file:../../local.repository/trunk): No connector available to access 
repository local.repository (file:../../local.repository/trunk) of type legacy 
using the available factories WagonRepositoryConnectorFactory
[INFO] 
[INFO] --- maven-resources-plugin:2.5:resources (default-resources) @ 
oozie-tools ---
[debug] execute contextualize
[INFO] Using 'UTF-8' encoding to copy filtered resources.
[INFO] skip non existing resourceDirectory 

[INFO] 
[INFO] --- maven-compiler-plugin:2.3.2:compile (default-compile) @ oozie-tools 
---
[INFO] Nothing to compile - all classes are up to date
[INFO] 
[INFO] --- maven-resources-plugin:2.5:testResources (default-testResources) @ 
oozie-tools ---
[debug] execute contextualize
[INFO] Using 'UTF-8' encoding to copy filtered resources.
[INFO] Copying 2 resources
[INFO] 
[INFO] --- maven-compiler-plugin:2.3.2:testCompile (default-testCompile) @ 
oozie-tools ---
[INFO] Nothing to compile - all classes are up to date
[INFO] 
[INFO] --- maven-surefire-plugin:2.12:test (default-test) @ oozie-tools ---
[INFO] Tests are skipped.
[INFO] 
[INFO] --- maven-jar-plugin:2.3.1:jar (default-jar) @ oozie-tools ---
[INFO] Building jar: 

[INFO] 
[INFO] --- maven-assembly-plugin:2.2.1:single (default-cli) @ oozie-tools ---
[INFO] Reading assembly descriptor: ../src/main/assemblies/tools.xml
[WARNING] The following patterns were never triggered in this artifact 
exclusion filter:
o  '*:*:pom:*'

[INFO] Copying files to 

[WARNING] Assembly file: 

 is not a regular file (it may be a directory). It cannot be attached to the 
project build for installation or deployment.
[INFO] 
[INFO] 
[INFO] Building Apache Oozie MiniOozie 4.1.0-SNAPSHOT
[INFO] 
[INFO] 
[INFO] --- maven-resources-plugin:2.5:resources (default-resources) @ 
oozie-mini ---
[debug] execute contextualize
[INFO] Using 'UTF-8' encoding to copy filtered resources.
[INFO] skip non existing resourceDirectory 

[INFO] 
[INFO] --- maven-compiler-plugin:2.3.2:compile (default-compile) @ oozie-mini 
---
[INFO] No sources to compile
[INFO] 
[INFO] --- maven-resources-plugin:2.5:testResources (default-testResources) @ 
oozie-mini ---
[debug] execute contextualize
[INFO] Using 'UTF-8' encoding to copy filtered resources.
[INFO] Copying 3 resources
[INFO] 
[INFO] --- maven-compiler-plugin:2.3.2:testCompile (default-testCompile) @ 
oozie-mini --

Re: What are the plans for an Oozie 4.1.0 release?

[bowen zhang]

>From 4.0 to 4.1, we should not have any DB upgrade and DB version should stay 
>at 2.0, shouldn't it?

On Wednesday, May 14, 2014 4:48 PM, Robert Kanter  wrote:
 
During the 3.x releases, we had an optional database change for MySQL to
change TEXT to MEDIUMTEXT; I don't think there was a required one, but I
don't remember.

In any case, that's a good point about the other projects and how they
handle these things.  So I'd agree then that we can have a database
upgrade.  Bowen, please make sure to verify that the database upgrade works
for 2.x to 4.1, 3.x to 4.1, and 4.0 to 4.1 and for each type.



On Wed, May 14, 2014 at 11:07 AM, Rohini Palaniswamy <
rohini.adi...@gmail.com> wrote:

> Not sure. I remember someone mentioning that schema upgrade should have
> major version incremented in case of Oozie. But there was one release (3.1
> ->3.2 or something before I started working on Oozie) which had schema
> change just incrementing minor version. If there was an already agreed
> procedure we should follow that else I am fine either way and ok going with
> 4.1 (as long it is atleast minor version and not patch version that we are
> changing) considering every other hadoop project still have major version
> as 0 and only keep updating minor version. For eg: hive 0.12 -> 0.13 is a
> major release with schema changes. If we keep incrementing major version
> very often it feels slightly weird when comparing to other hadoop projects
> which are still in 0.x :).
>
>
> On Tue, May 13, 2014 at 3:22 PM, Robert Kanter 
> wrote:
>
> > I agree; we'd also have to likely make two versions of each patch going
> > forward for 4.x and 5.x/trunk otherwise.
> > But are we "allowed" to make Oozie 4.0.x --> 4.1.0 require a database
> > upgrade?
> >
> >
> > On Tue, May 13, 2014 at 2:30 PM, Rohini Palaniswamy <
> > rohini.adi...@gmail.com
> > > wrote:
> >
> > > It might be tough work as the CLOB->BLOB conversion change touches a
> lot
> > of
> > > classes.
> > >
> > >
> > > On Sun, May 11, 2014 at 9:25 PM, Robert Kanter 
> > > wrote:
> > >
> > > > Hi everyone,
> > > >
> > > > Should we start thinking about a 4.1.0 release?  It's been a while
> > since
> > > > 4.0.0 (4.0.1 was to fix some critical things like not being able to
> > > > compile) and I think we have about 200 additional JIRAs in master.
> > > >
> > > > Are there any specific features that we'd want to put in 4.1.0 and
> wait
> > > > for?
> > > >
> > > > Another thing to keep in mind is that we can't just take everything
> in
> > > > master; I forget what exactly, but there's some JIRAs that change the
> > > > database that we'll have to save for 5.0.0.
> > > >
> > > > Does anybody want to volunteer to drive the 4.1.0 release?
> > > >
> > > >
> > > > thanks
> > > > - Robert
> > > >
> > >
> >
>

[jira] [Updated] (OOZIE-1835) NullPointerException from SLAEmailEventListener

[Robert Kanter (JIRA)]

 [ 
https://issues.apache.org/jira/browse/OOZIE-1835?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Robert Kanter updated OOZIE-1835:
-

Attachment: OOZIE-1835.patch

The patch makes it print a log message when no destination address is provided. 
 It will still log an IllegalArgumentException when an address is provided, but 
is malformed.

> NullPointerException from SLAEmailEventListener
> ---
>
> Key: OOZIE-1835
> URL: https://issues.apache.org/jira/browse/OOZIE-1835
> Project: Oozie
>  Issue Type: Bug
>  Components: core
>Affects Versions: 4.0.0
>Reporter: Robert Kanter
>Assignee: Robert Kanter
> Attachments: OOZIE-1835.patch
>
>
> Both the documentation and specification say that the "alert-contact" SLA 
> field is optional, at least when you don't want to send email notifications.  
> However, if you don't include it, you get a NullPointerException (though it 
> doesn't seem to cause any problems):
> {noformat}
> 2014-04-17 22:02:16,664 ERROR 
> org.apache.oozie.sla.listener.SLAEmailEventListener: 
> SERVER[rkanter-5.ent.cloudera.com] USER[-] GROUP[-] TOKEN[-] APP[-] JOB[-] 
> ACTION[-] Failed to send EndMiss alert email
> java.lang.NullPointerException
> at 
> org.apache.oozie.sla.listener.SLAEmailEventListener.parseAddress(SLAEmailEventListener.java:231)
> at 
> org.apache.oozie.sla.listener.SLAEmailEventListener.setMessageHeader(SLAEmailEventListener.java:262)
> at 
> org.apache.oozie.sla.listener.SLAEmailEventListener.sendSLAEmail(SLAEmailEventListener.java:163)
> at 
> org.apache.oozie.sla.listener.SLAEmailEventListener.onEndMiss(SLAEmailEventListener.java:200)
> at 
> org.apache.oozie.service.EventHandlerService$EventWorker.invokeSLAEventListener(EventHandlerService.java:297)
> at 
> org.apache.oozie.service.EventHandlerService$EventWorker.run(EventHandlerService.java:241)
> at 
> org.apache.oozie.service.SchedulerService$2.run(SchedulerService.java:174)
> at 
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
> at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:304)
> at 
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:178)
> at 
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
> at 
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
> at 
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
> at java.lang.Thread.run(Thread.java:744)
> {noformat}
> We should give a friendlier WARN message instead.



--
This message was sent by Atlassian JIRA
(v6.2#6252)

[jira] [Updated] (OOZIE-1783) Sharelib purging only occurs at Oozie startup

[Robert Kanter (JIRA)]

 [ 
https://issues.apache.org/jira/browse/OOZIE-1783?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Robert Kanter updated OOZIE-1783:
-

Attachment: OOZIE-1783.patch

The patch looks a lot longer than it really is because I found a bunch of cases 
in TestShareLibService where we did {{services.init()}} and later 
{{services.destroy()}} but didn't use a try-finally block so Services could 
have leaked when there were test failures.  The patch fixes these, which show 
up as a bunch of changes but I didn't really change anything here.

Other than that, the patch basically just takes the existing call to 
{{purgeLibs}}, puts it into a Runnable, and schedules that to run at 
{{oozie.service.ShareLibService.purge.interval}}, which defaults to 1 day.  I 
also updated {{TestShareLibService.testPurgeJar}} to wait for the purge 
runnable to run.

> Sharelib purging only occurs at Oozie startup
> -
>
> Key: OOZIE-1783
> URL: https://issues.apache.org/jira/browse/OOZIE-1783
> Project: Oozie
>  Issue Type: Bug
>Affects Versions: trunk
>Reporter: Robert Kanter
>Assignee: Robert Kanter
>Priority: Critical
> Attachments: OOZIE-1783.patch
>
>
> The purgeLibs method only gets called on startup of the first Oozie server.  
> This means that if you update the sharelib without restarting Oozie, then 
> Oozie will never clean up the old sharelibs and launcherlibs until you 
> restart the server.  In fact, with Oozie HA, it will never clean up unless 
> you restart all of the servers at the same time (in other words, a rolling 
> restart or just restarting one Oozie will never cause purgeLibs to get 
> called).  
> This should be moved into a background thread (i.e. scheduled with 
> SchedulerService) to run periodically.  Once a day is probably good enough, 
> but we could make it configurable if needed.  



--
This message was sent by Atlassian JIRA
(v6.2#6252)

Build failed in Jenkins: oozie-trunk-precommit-build #1238

[Apache Jenkins Server]

See 

--
[...truncated 9681 lines...]
[INFO]  already added, skipping
[INFO] share already added, skipping
[INFO] share/lib already added, skipping
[INFO]  already added, skipping
[INFO] share already added, skipping
[INFO] share/lib already added, skipping
[INFO]  already added, skipping
[INFO] share already added, skipping
[INFO] share/lib already added, skipping
[INFO]  already added, skipping
[INFO] share already added, skipping
[INFO] share/lib already added, skipping
[INFO]  already added, skipping
[INFO] share already added, skipping
[INFO] share/lib already added, skipping
[INFO]  already added, skipping
[INFO] share already added, skipping
[INFO] share/lib already added, skipping
[INFO] 
[INFO] 
[INFO] Building Apache Oozie Tools 4.1.0-SNAPSHOT
[INFO] 
[WARNING] Could not transfer metadata asm:asm/maven-metadata.xml from/to 
local.repository (file:../../local.repository/trunk): No connector available to 
access repository local.repository (file:../../local.repository/trunk) of type 
legacy using the available factories WagonRepositoryConnectorFactory
[WARNING] Could not transfer metadata asm:asm/maven-metadata.xml from/to 
local.repository (file:../../local.repository/trunk): No connector available to 
access repository local.repository (file:../../local.repository/trunk) of type 
legacy using the available factories WagonRepositoryConnectorFactory
[INFO] 
[INFO] --- maven-resources-plugin:2.5:resources (default-resources) @ 
oozie-tools ---
[debug] execute contextualize
[INFO] Using 'UTF-8' encoding to copy filtered resources.
[INFO] skip non existing resourceDirectory 

[INFO] 
[INFO] --- maven-compiler-plugin:2.3.2:compile (default-compile) @ oozie-tools 
---
[INFO] Nothing to compile - all classes are up to date
[INFO] 
[INFO] --- maven-resources-plugin:2.5:testResources (default-testResources) @ 
oozie-tools ---
[debug] execute contextualize
[INFO] Using 'UTF-8' encoding to copy filtered resources.
[INFO] Copying 2 resources
[INFO] 
[INFO] --- maven-compiler-plugin:2.3.2:testCompile (default-testCompile) @ 
oozie-tools ---
[INFO] Nothing to compile - all classes are up to date
[INFO] 
[INFO] --- maven-surefire-plugin:2.12:test (default-test) @ oozie-tools ---
[INFO] Tests are skipped.
[INFO] 
[INFO] --- maven-jar-plugin:2.3.1:jar (default-jar) @ oozie-tools ---
[INFO] Building jar: 

[INFO] 
[INFO] --- maven-assembly-plugin:2.2.1:single (default-cli) @ oozie-tools ---
[INFO] Reading assembly descriptor: ../src/main/assemblies/tools.xml
[WARNING] The following patterns were never triggered in this artifact 
exclusion filter:
o  '*:*:pom:*'

[INFO] Copying files to 

[WARNING] Assembly file: 

 is not a regular file (it may be a directory). It cannot be attached to the 
project build for installation or deployment.
[INFO] 
[INFO] 
[INFO] Building Apache Oozie MiniOozie 4.1.0-SNAPSHOT
[INFO] 
[INFO] 
[INFO] --- maven-resources-plugin:2.5:resources (default-resources) @ 
oozie-mini ---
[debug] execute contextualize
[INFO] Using 'UTF-8' encoding to copy filtered resources.
[INFO] skip non existing resourceDirectory 

[INFO] 
[INFO] --- maven-compiler-plugin:2.3.2:compile (default-compile) @ oozie-mini 
---
[INFO] No sources to compile
[INFO] 
[INFO] --- maven-resources-plugin:2.5:testResources (default-testResources) @ 
oozie-mini ---
[debug] execute contextualize
[INFO] Using 'UTF-8' encoding to copy filtered resources.
[INFO] Copying 3 resources
[INFO] 
[INFO] --- maven-compiler-plugin:2.3.2:testCompile (default-testCompile) @ 
oozie-mini ---
[INFO] Nothing to compile - all classes are up to date
[INFO] 
[INFO] --- maven-surefire-plugin:2.12:test (default-test) @ oozie-mini ---
[INFO] Tests are skipped.
[INFO] 
[INFO] --- maven-jar-plugin:2.3.1:jar (default-jar) @ oozie-mini ---
[WARNING] JAR will be empty - no content was marked for inclusion!
[INFO] Building jar: 

[INFO]

Build failed in Jenkins: oozie-trunk-precommit-build #1237

[Apache Jenkins Server]

See 

--
[...truncated 8966 lines...]
[INFO]  already added, skipping
[INFO] share already added, skipping
[INFO] share/lib already added, skipping
[INFO]  already added, skipping
[INFO] share already added, skipping
[INFO] share/lib already added, skipping
[INFO]  already added, skipping
[INFO] share already added, skipping
[INFO] share/lib already added, skipping
[INFO]  already added, skipping
[INFO] share already added, skipping
[INFO] share/lib already added, skipping
[INFO]  already added, skipping
[INFO] share already added, skipping
[INFO] share/lib already added, skipping
[INFO]  already added, skipping
[INFO] share already added, skipping
[INFO] share/lib already added, skipping
[INFO] 
[INFO] 
[INFO] Building Apache Oozie Tools 4.1.0-SNAPSHOT
[INFO] 
[WARNING] Could not transfer metadata asm:asm/maven-metadata.xml from/to 
local.repository (file:../../local.repository/trunk): No connector available to 
access repository local.repository (file:../../local.repository/trunk) of type 
legacy using the available factories WagonRepositoryConnectorFactory
[WARNING] Could not transfer metadata asm:asm/maven-metadata.xml from/to 
local.repository (file:../../local.repository/trunk): No connector available to 
access repository local.repository (file:../../local.repository/trunk) of type 
legacy using the available factories WagonRepositoryConnectorFactory
[INFO] 
[INFO] --- maven-resources-plugin:2.5:resources (default-resources) @ 
oozie-tools ---
[debug] execute contextualize
[INFO] Using 'UTF-8' encoding to copy filtered resources.
[INFO] skip non existing resourceDirectory 

[INFO] 
[INFO] --- maven-compiler-plugin:2.3.2:compile (default-compile) @ oozie-tools 
---
[INFO] Nothing to compile - all classes are up to date
[INFO] 
[INFO] --- maven-resources-plugin:2.5:testResources (default-testResources) @ 
oozie-tools ---
[debug] execute contextualize
[INFO] Using 'UTF-8' encoding to copy filtered resources.
[INFO] Copying 2 resources
[INFO] 
[INFO] --- maven-compiler-plugin:2.3.2:testCompile (default-testCompile) @ 
oozie-tools ---
[INFO] Nothing to compile - all classes are up to date
[INFO] 
[INFO] --- maven-surefire-plugin:2.12:test (default-test) @ oozie-tools ---
[INFO] Tests are skipped.
[INFO] 
[INFO] --- maven-jar-plugin:2.3.1:jar (default-jar) @ oozie-tools ---
[INFO] Building jar: 

[INFO] 
[INFO] --- maven-assembly-plugin:2.2.1:single (default-cli) @ oozie-tools ---
[INFO] Reading assembly descriptor: ../src/main/assemblies/tools.xml
[WARNING] The following patterns were never triggered in this artifact 
exclusion filter:
o  '*:*:pom:*'

[INFO] Copying files to 

[WARNING] Assembly file: 

 is not a regular file (it may be a directory). It cannot be attached to the 
project build for installation or deployment.
[INFO] 
[INFO] 
[INFO] Building Apache Oozie MiniOozie 4.1.0-SNAPSHOT
[INFO] 
[INFO] 
[INFO] --- maven-resources-plugin:2.5:resources (default-resources) @ 
oozie-mini ---
[debug] execute contextualize
[INFO] Using 'UTF-8' encoding to copy filtered resources.
[INFO] skip non existing resourceDirectory 

[INFO] 
[INFO] --- maven-compiler-plugin:2.3.2:compile (default-compile) @ oozie-mini 
---
[INFO] No sources to compile
[INFO] 
[INFO] --- maven-resources-plugin:2.5:testResources (default-testResources) @ 
oozie-mini ---
[debug] execute contextualize
[INFO] Using 'UTF-8' encoding to copy filtered resources.
[INFO] Copying 3 resources
[INFO] 
[INFO] --- maven-compiler-plugin:2.3.2:testCompile (default-testCompile) @ 
oozie-mini ---
[INFO] Nothing to compile - all classes are up to date
[INFO] 
[INFO] --- maven-surefire-plugin:2.12:test (default-test) @ oozie-mini ---
[INFO] Tests are skipped.
[INFO] 
[INFO] --- maven-jar-plugin:2.3.1:jar (default-jar) @ oozie-mini ---
[WARNING] JAR will be empty - no content was marked for inclusion!
[INFO] Building jar: 

[INFO]

Build failed in Jenkins: oozie-trunk-precommit-build #1235

[Apache Jenkins Server]

See 

Changes:

[rohinip] OOZIE-1831 Oozie upgrade fails if workflow jobs are in running or 
suspended state (satish.mittal via rohini)

[rohinip] OOZIE-1825 Optimize wf_jobs protoconf storage (puru via rohini)

[ryota.egashira] OOZIE-1791 add IGNORED status to Coordinator Job and Action 
(ryota)

[rkanter] OOZIE-1826 Add thread which detects JVM pauses (rkanter)

[ryota.egashira] OOZIE-1809 offset and len options are ignored in oozie job 
-info for workflow (ryota)

--
[...truncated 2818 lines...]
[INFO] --- maven-surefire-plugin:2.12:test (default-test) @ oozie-webapp ---
[INFO] Tests are skipped.
[INFO] 
[INFO] 
[INFO] Building Apache Oozie Examples 4.1.0-SNAPSHOT
[INFO] 
[INFO] 
[INFO] --- maven-clean-plugin:2.4.1:clean (default-clean) @ oozie-examples ---
[INFO] Deleting 

[INFO] 
[INFO] --- maven-resources-plugin:2.5:resources (default-resources) @ 
oozie-examples ---
[debug] execute contextualize
[INFO] Using 'UTF-8' encoding to copy filtered resources.
[INFO] skip non existing resourceDirectory 

[INFO] 
[INFO] --- maven-compiler-plugin:2.3.2:compile (default-compile) @ 
oozie-examples ---
[INFO] Compiling 11 source files to 

[INFO] 
[INFO] --- maven-resources-plugin:2.5:testResources (default-testResources) @ 
oozie-examples ---
[debug] execute contextualize
[INFO] Using 'UTF-8' encoding to copy filtered resources.
[INFO] Copying 7 resources
[INFO] 
[INFO] --- maven-compiler-plugin:2.3.2:testCompile (default-testCompile) @ 
oozie-examples ---
[INFO] Compiling 1 source file to 

[INFO] 
[INFO] --- maven-surefire-plugin:2.12:test (default-test) @ oozie-examples ---
[INFO] Tests are skipped.
[INFO] 
[INFO] 
[INFO] Building Apache Oozie Share Lib 4.1.0-SNAPSHOT
[INFO] 
[INFO] 
[INFO] --- maven-clean-plugin:2.4.1:clean (default-clean) @ oozie-sharelib ---
[INFO] Deleting 

[INFO] 
[INFO] 
[INFO] Building Apache Oozie Tools 4.1.0-SNAPSHOT
[INFO] 
[WARNING] Could not transfer metadata asm:asm/maven-metadata.xml from/to 
local.repository (file:../../local.repository/trunk): No connector available to 
access repository local.repository (file:../../local.repository/trunk) of type 
legacy using the available factories WagonRepositoryConnectorFactory
[WARNING] Could not transfer metadata asm:asm/maven-metadata.xml from/to 
local.repository (file:../../local.repository/trunk): No connector available to 
access repository local.repository (file:../../local.repository/trunk) of type 
legacy using the available factories WagonRepositoryConnectorFactory
[INFO] 
[INFO] --- maven-clean-plugin:2.4.1:clean (default-clean) @ oozie-tools ---
[INFO] Deleting 

[INFO] 
[INFO] --- maven-resources-plugin:2.5:resources (default-resources) @ 
oozie-tools ---
[debug] execute contextualize
[INFO] Using 'UTF-8' encoding to copy filtered resources.
[INFO] skip non existing resourceDirectory 

[INFO] 
[INFO] --- maven-compiler-plugin:2.3.2:compile (default-compile) @ oozie-tools 
---
[INFO] Compiling 2 source files to 

[INFO] 
[INFO] --- maven-resources-plugin:2.5:testResources (default-testResources) @ 
oozie-tools ---
[debug] execute contextualize
[INFO] Using 'UTF-8' encoding to copy filtered resources.
[INFO] Copying 2 resources
[INFO] 
[INFO] --- maven-compiler-plugin:2.3.2:testCompile (default-testCompile) @ 
oozie-tools ---
[INFO] Compiling 6 source files to 

[INFO] 
[INFO] --- maven-surefire-plugin:2.12:test (default-test) @ oozie-tools ---
[INFO] Tests are skipped.
[INFO]  

[jira] [Created] (OOZIE-1844) HA - Lock mechanism for CoordMaterializeTriggerService ( may be for other services as well)

[Purshotam Shah (JIRA)]

Purshotam Shah created OOZIE-1844:
-

 Summary: HA -  Lock mechanism for CoordMaterializeTriggerService ( 
may be for other services as well)
 Key: OOZIE-1844
 URL: https://issues.apache.org/jira/browse/OOZIE-1844
 Project: Oozie
  Issue Type: Bug
Reporter: Purshotam Shah






--
This message was sent by Atlassian JIRA
(v6.2#6252)

[jira] Subscription: Oozie Patch Available

[jira]

Issue Subscription
Filter: Oozie Patch Available (39 issues)

Subscriber: ooziedaily

Key Summary
OOZIE-1838  jdbc.connections.active sampler does not show up
https://issues.apache.org/jira/browse/OOZIE-1838
OOZIE-1830  Change hadoop-1 profile to use 1.2.1
https://issues.apache.org/jira/browse/OOZIE-1830
OOZIE-1821  Oozie java action fails due to AlreadyBeingCreatedException
https://issues.apache.org/jira/browse/OOZIE-1821
OOZIE-1819  Avoid early queueing of CoordActionInputCheckXCommand
https://issues.apache.org/jira/browse/OOZIE-1819
OOZIE-1817  Oozie timers are not biased
https://issues.apache.org/jira/browse/OOZIE-1817
OOZIE-1816  LogInfo uses action name instead of id
https://issues.apache.org/jira/browse/OOZIE-1816
OOZIE-1813  Add service to report/kill rogue bundles and coordinator jobs
https://issues.apache.org/jira/browse/OOZIE-1813
OOZIE-1807  Make bundle change command synchronous
https://issues.apache.org/jira/browse/OOZIE-1807
OOZIE-1804  Improve documentation for Coordinator Specification
https://issues.apache.org/jira/browse/OOZIE-1804
OOZIE-1802  Support workflow action log
https://issues.apache.org/jira/browse/OOZIE-1802
OOZIE-1801  ZKLocksService instrumentation should say how many locks this 
server has
https://issues.apache.org/jira/browse/OOZIE-1801
OOZIE-1793  Improve find bugs reporting for Oozie
https://issues.apache.org/jira/browse/OOZIE-1793
OOZIE-1782  Workflow path not found is thrown as SC_UNAUTHORIZED
https://issues.apache.org/jira/browse/OOZIE-1782
OOZIE-1775  TestEventGeneration.testCoordinatorActionEvent is failing and 
CoordRerunX should generate event
https://issues.apache.org/jira/browse/OOZIE-1775
OOZIE-1760  Set default shared lib dir for java action
https://issues.apache.org/jira/browse/OOZIE-1760
OOZIE-1741  Add new coord EL function to get input partitions value string
https://issues.apache.org/jira/browse/OOZIE-1741
OOZIE-1724  Make it easier to specify the HCat hive-site.xml for the Oozie 
Server
https://issues.apache.org/jira/browse/OOZIE-1724
OOZIE-1689  HA support for OOZIE-7(Ability to view the log information 
corresponding to particular coordinator action)
https://issues.apache.org/jira/browse/OOZIE-1689
OOZIE-1688  New configuration to specify server-server authentication type.
https://issues.apache.org/jira/browse/OOZIE-1688
OOZIE-1686  Typo in DG_CommandLineTool
https://issues.apache.org/jira/browse/OOZIE-1686
OOZIE-1685  Oozie doesn’t process correctly workflows with a non-default name 
node
https://issues.apache.org/jira/browse/OOZIE-1685
OOZIE-1676  Instrumentation and Configuration over the REST API and Web UI 
should include all Oozie servers
https://issues.apache.org/jira/browse/OOZIE-1676
OOZIE-1659  oozie-site is missing email-action-0.2 schema
https://issues.apache.org/jira/browse/OOZIE-1659
OOZIE-1654  Fix typo (inteval to interval)
https://issues.apache.org/jira/browse/OOZIE-1654
OOZIE-1638  Action retry does not use default retry max count.
https://issues.apache.org/jira/browse/OOZIE-1638
OOZIE-1636  OOZIE_SYS table engine should be innodb
https://issues.apache.org/jira/browse/OOZIE-1636
OOZIE-1624  Exclusion pattern for sharelib.
https://issues.apache.org/jira/browse/OOZIE-1624
OOZIE-1602  pom.xml should take use of the profiles when building hadooplibs
https://issues.apache.org/jira/browse/OOZIE-1602
OOZIE-1579  Add basic HTTP auth to Oozie CLI
https://issues.apache.org/jira/browse/OOZIE-1579
OOZIE-1567  Provide a wait tool in Oozie
https://issues.apache.org/jira/browse/OOZIE-1567
OOZIE-1533  Coordinator action materialization is too slow due to coarse job 
level locks
https://issues.apache.org/jira/browse/OOZIE-1533
OOZIE-1532  Purging should remove completed children job for long running 
coordinator jobs
https://issues.apache.org/jira/browse/OOZIE-1532
OOZIE-1376  Extending Oozie ACLs like admin groups and proxy users to support 
both groups and users
https://issues.apache.org/jira/browse/OOZIE-1376
OOZIE-1369  OozieDBCLI code should not hardcode the Oozie table filenames
https://issues.apache.org/jira/browse/OOZIE-1369
OOZIE-1319  "LAST_ONLY" in execution control for coordinator job still runs all 
the actions
https://issues.apache.org/jira/browse/OOZIE-1319
OOZIE-1232  GroupsService should be able to reference Hadoop configurations in 
Hadoop configuration folder (such as /etc/hadoop/conf)
https://issues.apache.org/jira/browse/OOZIE-1232
OOZIE-1074  workflowgenerator does not package a tar.gz file though it seems 
creating one is the intention
https://issues.apache.org/jira/browse/OOZIE-1074
O

Build failed in Jenkins: oozie-trunk-precommit-build #1242

[Apache Jenkins Server]

See 

--
[...truncated 9596 lines...]
[INFO]  already added, skipping
[INFO] share already added, skipping
[INFO] share/lib already added, skipping
[INFO]  already added, skipping
[INFO] share already added, skipping
[INFO] share/lib already added, skipping
[INFO]  already added, skipping
[INFO] share already added, skipping
[INFO] share/lib already added, skipping
[INFO]  already added, skipping
[INFO] share already added, skipping
[INFO] share/lib already added, skipping
[INFO]  already added, skipping
[INFO] share already added, skipping
[INFO] share/lib already added, skipping
[INFO] 
[INFO] 
[INFO] Building Apache Oozie Tools 4.1.0-SNAPSHOT
[INFO] 
[WARNING] Failure to transfer asm:asm/maven-metadata.xml from 
file:../../local.repository/trunk was cached in the local repository, 
resolution will not be reattempted until the update interval of 
local.repository has elapsed or updates are forced. Original error: Could not 
transfer metadata asm:asm/maven-metadata.xml from/to local.repository 
(file:../../local.repository/trunk): No connector available to access 
repository local.repository (file:../../local.repository/trunk) of type legacy 
using the available factories WagonRepositoryConnectorFactory
[WARNING] Failure to transfer asm:asm/maven-metadata.xml from 
file:../../local.repository/trunk was cached in the local repository, 
resolution will not be reattempted until the update interval of 
local.repository has elapsed or updates are forced. Original error: Could not 
transfer metadata asm:asm/maven-metadata.xml from/to local.repository 
(file:../../local.repository/trunk): No connector available to access 
repository local.repository (file:../../local.repository/trunk) of type legacy 
using the available factories WagonRepositoryConnectorFactory
[INFO] 
[INFO] --- maven-resources-plugin:2.5:resources (default-resources) @ 
oozie-tools ---
[debug] execute contextualize
[INFO] Using 'UTF-8' encoding to copy filtered resources.
[INFO] skip non existing resourceDirectory 

[INFO] 
[INFO] --- maven-compiler-plugin:2.3.2:compile (default-compile) @ oozie-tools 
---
[INFO] Nothing to compile - all classes are up to date
[INFO] 
[INFO] --- maven-resources-plugin:2.5:testResources (default-testResources) @ 
oozie-tools ---
[debug] execute contextualize
[INFO] Using 'UTF-8' encoding to copy filtered resources.
[INFO] Copying 2 resources
[INFO] 
[INFO] --- maven-compiler-plugin:2.3.2:testCompile (default-testCompile) @ 
oozie-tools ---
[INFO] Nothing to compile - all classes are up to date
[INFO] 
[INFO] --- maven-surefire-plugin:2.12:test (default-test) @ oozie-tools ---
[INFO] Tests are skipped.
[INFO] 
[INFO] --- maven-jar-plugin:2.3.1:jar (default-jar) @ oozie-tools ---
[INFO] Building jar: 

[INFO] 
[INFO] --- maven-assembly-plugin:2.2.1:single (default-cli) @ oozie-tools ---
[INFO] Reading assembly descriptor: ../src/main/assemblies/tools.xml
[WARNING] The following patterns were never triggered in this artifact 
exclusion filter:
o  '*:*:pom:*'

[INFO] Copying files to 

[WARNING] Assembly file: 

 is not a regular file (it may be a directory). It cannot be attached to the 
project build for installation or deployment.
[INFO] 
[INFO] 
[INFO] Building Apache Oozie MiniOozie 4.1.0-SNAPSHOT
[INFO] 
[INFO] 
[INFO] --- maven-resources-plugin:2.5:resources (default-resources) @ 
oozie-mini ---
[debug] execute contextualize
[INFO] Using 'UTF-8' encoding to copy filtered resources.
[INFO] skip non existing resourceDirectory 

[INFO] 
[INFO] --- maven-compiler-plugin:2.3.2:compile (default-compile) @ oozie-mini 
---
[INFO] No sources to compile
[INFO] 
[INFO] --- maven-resources-plugin:2.5:testResources (default-testResources) @ 
oozie-mini ---
[debug] execute contextualize
[INFO] Using 'UTF-8' encoding to copy filtered resources.
[INFO] Copying 3 resources
[INFO] 
[INFO] --- maven-compiler-plugin:2.3.2:testCompile (default-testCompile) @ 
oozie-mini ---
[INFO] Nothing to compile - all classes

Review Request 21278: OOZIE-1817: Oozie timers are not biased

[Robert Kanter]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/21278/
---

Review request for oozie.


Bugs: OOZIE-1817
https://issues.apache.org/jira/browse/OOZIE-1817


Repository: oozie-git


Description
---

See 
https://issues.apache.org/jira/browse/OOZIE-1817?focusedCommentId=13993838&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-13993838


Diffs
-

  client/src/main/java/org/apache/oozie/client/rest/RestConstants.java 808f9b2 
  core/pom.xml c935dd7 
  core/src/main/java/org/apache/oozie/command/Command.java d3f1011 
  core/src/main/java/org/apache/oozie/command/XCommand.java b712ce0 
  
core/src/main/java/org/apache/oozie/command/coord/CoordActionInputCheckXCommand.java
 e8667c1 
  
core/src/main/java/org/apache/oozie/command/coord/CoordActionMaterializeCommand.java
 6962fb2 
  
core/src/main/java/org/apache/oozie/command/coord/CoordMaterializeTransitionXCommand.java
 57cbb34 
  core/src/main/java/org/apache/oozie/service/InstrumentationService.java 
80437b1 
  
core/src/main/java/org/apache/oozie/service/MetricsInstrumentationService.java 
PRE-CREATION 
  core/src/main/java/org/apache/oozie/service/Services.java 5feac7b 
  core/src/main/java/org/apache/oozie/service/XLogService.java 403a089 
  core/src/main/java/org/apache/oozie/servlet/BaseAdminServlet.java 29d7bd6 
  core/src/main/java/org/apache/oozie/servlet/JsonRestServlet.java 5c05acd 
  core/src/main/java/org/apache/oozie/servlet/V0AdminServlet.java 97bd81c 
  core/src/main/java/org/apache/oozie/servlet/V1AdminServlet.java a47f737 
  core/src/main/java/org/apache/oozie/servlet/V2AdminServlet.java 002a367 
  core/src/main/java/org/apache/oozie/util/MetricsInstrumentation.java 
PRE-CREATION 
  core/src/main/java/org/apache/oozie/util/XLogReporter.java PRE-CREATION 
  core/src/test/java/org/apache/oozie/service/TestInstrumentationService.java 
b5206cf 
  
core/src/test/java/org/apache/oozie/service/TestMetricsInstrumentationService.java
 PRE-CREATION 
  core/src/test/java/org/apache/oozie/util/TestInstrumentation.java 6040b47 
  core/src/test/java/org/apache/oozie/util/TestMetricsInstrumentation.java 
PRE-CREATION 
  docs/src/site/twiki/AG_Install.twiki e343d7e 
  docs/src/site/twiki/WebServicesAPI.twiki 6730255 
  pom.xml b5e0e4e 
  webapp/src/main/webapp/index.jsp 3c7ffe5 
  webapp/src/main/webapp/oozie-console.js 764d888 

Diff: https://reviews.apache.org/r/21278/diff/


Testing
---

Unit tests, also verified in a cluster


Thanks,

Robert Kanter

[jira] [Commented] (OOZIE-1833) Oozie build error on old maven-project-info-reports-plugin

[Robert Kanter (JIRA)]

[ 
https://issues.apache.org/jira/browse/OOZIE-1833?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13998286#comment-13998286
 ] 

Robert Kanter commented on OOZIE-1833:
--

Due to the way Oozie includes Hadoop jars, you can't simply change 
"hadoop.version" on the CLI; that won't compile as you saw.  

We have 4 profiles you can use to compile against different versions of Hadoop, 
and as of Oozie 4.0.1, they map like this:
- Hadoop 1.1.1 :: (default, no profile)
- Hadoop 0.23.5 :: {{hadoop-23}}
- Hadoop 2.3.0 :: {{hadoop-2}}
- Hadoop 3.0.0-SNAPSHOT (trunk) :: {{hadoop-3}}

So, for what you want, Hadoop 2.3.0:
{noformat}
bin/mkdistro.sh -Phadoop-2 -DskipTests
{noformat}

> Oozie build error on old maven-project-info-reports-plugin
> --
>
> Key: OOZIE-1833
> URL: https://issues.apache.org/jira/browse/OOZIE-1833
> Project: Oozie
>  Issue Type: Improvement
>  Components: build, bundle
>Affects Versions: 4.0.0, 4.0.1
> Environment: Osx 10.9.2 jdk 1.6.0_65-b14462-11M4609
>Reporter: Antonio Conte
>
> during build with 
> bin/mkdistro.sh -Dhadoop.version=2.3.0 -DskipTests
> build fails on "Apache Oozie Share Lib Distcp"
> [INFO] --- maven-project-info-reports-plugin:2.3:dependencies (default) @ 
> oozie-webapp ---
> [ERROR] Artifact: org.apache.hadoop:avro:jar:1.3.2 has no file.
> to solve please update main pom.xml with 
> line:805
> 
> org.apache.maven.plugins
> maven-project-info-reports-plugin
> 2.7
> 



--
This message was sent by Atlassian JIRA
(v6.2#6252)