Re: Review Request 60544: OOZIE-2854 Oozie should handle transient database problems

[Peter Cseh]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/60544/#review180170
---




core/src/main/java/org/apache/oozie/util/db/FailingMySQLDriverWrapper.java
Lines 27 (patched)


This class is not used anywhere. Please remove it or document what it can 
be used for (e.g. on-site testing)


- Peter Cseh


On July 6, 2017, 5:36 p.m., András Piros wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/60544/
> ---
> 
> (Updated July 6, 2017, 5:36 p.m.)
> 
> 
> Review request for oozie, Attila Sasvari, Peter Cseh, Peter Bacsko, and 
> Robert Kanter.
> 
> 
> Repository: oozie-git
> 
> 
> Description
> ---
> 
> https://issues.apache.org/jira/browse/OOZIE-2854
> 
> 
> Diffs
> -
> 
>   core/pom.xml acddf349a89cf09a7fc4f384ebcaec56dfd0ab48 
>   
> core/src/main/java/org/apache/oozie/executor/jpa/JsonBeanPersisterExecutor.java
>  PRE-CREATION 
>   core/src/main/java/org/apache/oozie/executor/jpa/QueryExecutor.java 
> 8d94c23e40d1281864db40e141b200ca207a6324 
>   core/src/main/java/org/apache/oozie/service/JPAService.java 
> 028381d3b72bcc3b8c2cd27cacb3e0ac6d48d146 
>   core/src/main/java/org/apache/oozie/sla/SLASummaryBean.java 
> cfe1522a4b1f89085eb29e7f1281c2abd631bdc2 
>   core/src/main/java/org/apache/oozie/store/WorkflowStore.java 
> c565e74893b863caef6c93015cfe38fe520d04ec 
>   core/src/main/java/org/apache/oozie/util/db/BasicDataSourceWrapper.java 
> PRE-CREATION 
>   core/src/main/java/org/apache/oozie/util/db/DatabaseRetryPredicate.java 
> PRE-CREATION 
>   core/src/main/java/org/apache/oozie/util/db/FailingConnectionWrapper.java 
> PRE-CREATION 
>   core/src/main/java/org/apache/oozie/util/db/FailingHSQLDBDriverWrapper.java 
> PRE-CREATION 
>   core/src/main/java/org/apache/oozie/util/db/FailingMySQLDriverWrapper.java 
> PRE-CREATION 
>   core/src/main/java/org/apache/oozie/util/db/OperationRetryHandler.java 
> PRE-CREATION 
>   
> core/src/main/java/org/apache/oozie/util/db/PersistenceExceptionSubclassFilterRetryPredicate.java
>  PRE-CREATION 
>   core/src/main/java/org/apache/oozie/util/db/RetryAttemptState.java 
> PRE-CREATION 
>   core/src/main/java/org/apache/oozie/util/db/RuntimeExceptionInjector.java 
> PRE-CREATION 
>   core/src/main/resources/META-INF/persistence.xml 
> bad9278597fcd4f93b4cc482afae8af14beaa922 
>   core/src/main/resources/oozie-default.xml 
> 5629a897b4a1834554a1a5f7a21984ea857ed133 
>   core/src/main/resources/oozie-log4j.properties 
> c065f3cd4c5a3df1308b69d7c16e8fcfa8796efc 
>   core/src/test/java/org/apache/oozie/command/SkipCommitFaultInjection.java  
>   core/src/test/java/org/apache/oozie/test/XTestCase.java 
> 161927ac8f1132b3080d2924844826fcc7b807a5 
>   
> core/src/test/java/org/apache/oozie/util/db/TestOozieDmlStatementPredicate.java
>  PRE-CREATION 
>   core/src/test/java/org/apache/oozie/util/db/TestOperationRetryHandler.java 
> PRE-CREATION 
>   
> core/src/test/java/org/apache/oozie/util/db/TestPersistenceExceptionSubclassFilterRetryPredicate.java
>  PRE-CREATION 
>   core/src/test/java/org/apache/oozie/util/db/TestRetryAttemptState.java 
> PRE-CREATION 
>   docs/src/site/twiki/AG_Install.twiki 
> 2125442c273661d54f3566cbe41ead822c1f7baa 
>   minitest/pom.xml 9515284bb5f32c279a93161c10e6571680e4f9fc 
>   
> minitest/src/test/java/org/apache/oozie/test/TestParallelJPAOperationRetries.java
>  PRE-CREATION 
>   minitest/src/test/java/org/apache/oozie/test/TestWorkflowRetries.java 
> PRE-CREATION 
>   minitest/src/test/java/org/apache/oozie/test/WorkflowTest.java 
> 2845f0af6efb9ef75fdbfcb326115c62e6fb3bdd 
>   minitest/src/test/resources/hsqldb-oozie-site.xml 
> fa5fe9c3185e973e8247d7bf10b126119d9c02c9 
>   minitest/src/test/resources/oozie-log4j.properties 
> c142d725140930bfa89cd2b163d0768a4c3a750a 
>   minitest/src/test/resources/parallel-fs-and-shell.xml PRE-CREATION 
>   minitest/src/test/resources/wf-test.xml 
> 20c4946862039a65c76ed7f49991345e90a694de 
>   pom.xml 16c5137d44d7db891da46f80adb51c85e4c1b214 
> 
> 
> Diff: https://reviews.apache.org/r/60544/diff/7/
> 
> 
> Testing
> ---
> 
> Tests covered in code:
> 
> Unit tests
> ==
> 
> * testing the retry handler, the retry predicate filter, and parallel calls 
> to JPA `EntityManager` (mostly Oozie database reads and writes) when 
> injecting failures
> 
> Integration tests
> =
> 
> * using the `MiniOozieTestCase` framework
> * fixing it so that also asynchronous workflow applications (the ones that 
> use `CallableQueueService`) can be run
> * following workflow scenarios:
> * a very simple one consisting only of a `` and an `` node
> * a more sophisticated one consisting of multiple synchronous `` nodes 
> an

Re: Review Request 60544: OOZIE-2854 Oozie should handle transient database problems

[Peter Cseh]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/60544/#review180169
---




core/src/main/java/org/apache/oozie/util/db/FailingConnectionWrapper.java
Lines 49 (patched)


please move the Failing* classes to src/test from src/main. As far as I can 
see they are used only in tests.



core/src/main/resources/oozie-log4j.properties
Lines 62 (patched)


Do we want to increase the log level for this? Isn't this just a leftover 
from testing?



docs/src/site/twiki/AG_Install.twiki
Lines 211 (patched)


Please don't use "w/" here



docs/src/site/twiki/AG_Install.twiki
Lines 211-213 (patched)


Please describe how the retry logic is working instead of how it's tested.


- Peter Cseh


On July 6, 2017, 5:36 p.m., András Piros wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/60544/
> ---
> 
> (Updated July 6, 2017, 5:36 p.m.)
> 
> 
> Review request for oozie, Attila Sasvari, Peter Cseh, Peter Bacsko, and 
> Robert Kanter.
> 
> 
> Repository: oozie-git
> 
> 
> Description
> ---
> 
> https://issues.apache.org/jira/browse/OOZIE-2854
> 
> 
> Diffs
> -
> 
>   core/pom.xml acddf349a89cf09a7fc4f384ebcaec56dfd0ab48 
>   
> core/src/main/java/org/apache/oozie/executor/jpa/JsonBeanPersisterExecutor.java
>  PRE-CREATION 
>   core/src/main/java/org/apache/oozie/executor/jpa/QueryExecutor.java 
> 8d94c23e40d1281864db40e141b200ca207a6324 
>   core/src/main/java/org/apache/oozie/service/JPAService.java 
> 028381d3b72bcc3b8c2cd27cacb3e0ac6d48d146 
>   core/src/main/java/org/apache/oozie/sla/SLASummaryBean.java 
> cfe1522a4b1f89085eb29e7f1281c2abd631bdc2 
>   core/src/main/java/org/apache/oozie/store/WorkflowStore.java 
> c565e74893b863caef6c93015cfe38fe520d04ec 
>   core/src/main/java/org/apache/oozie/util/db/BasicDataSourceWrapper.java 
> PRE-CREATION 
>   core/src/main/java/org/apache/oozie/util/db/DatabaseRetryPredicate.java 
> PRE-CREATION 
>   core/src/main/java/org/apache/oozie/util/db/FailingConnectionWrapper.java 
> PRE-CREATION 
>   core/src/main/java/org/apache/oozie/util/db/FailingHSQLDBDriverWrapper.java 
> PRE-CREATION 
>   core/src/main/java/org/apache/oozie/util/db/FailingMySQLDriverWrapper.java 
> PRE-CREATION 
>   core/src/main/java/org/apache/oozie/util/db/OperationRetryHandler.java 
> PRE-CREATION 
>   
> core/src/main/java/org/apache/oozie/util/db/PersistenceExceptionSubclassFilterRetryPredicate.java
>  PRE-CREATION 
>   core/src/main/java/org/apache/oozie/util/db/RetryAttemptState.java 
> PRE-CREATION 
>   core/src/main/java/org/apache/oozie/util/db/RuntimeExceptionInjector.java 
> PRE-CREATION 
>   core/src/main/resources/META-INF/persistence.xml 
> bad9278597fcd4f93b4cc482afae8af14beaa922 
>   core/src/main/resources/oozie-default.xml 
> 5629a897b4a1834554a1a5f7a21984ea857ed133 
>   core/src/main/resources/oozie-log4j.properties 
> c065f3cd4c5a3df1308b69d7c16e8fcfa8796efc 
>   core/src/test/java/org/apache/oozie/command/SkipCommitFaultInjection.java  
>   core/src/test/java/org/apache/oozie/test/XTestCase.java 
> 161927ac8f1132b3080d2924844826fcc7b807a5 
>   
> core/src/test/java/org/apache/oozie/util/db/TestOozieDmlStatementPredicate.java
>  PRE-CREATION 
>   core/src/test/java/org/apache/oozie/util/db/TestOperationRetryHandler.java 
> PRE-CREATION 
>   
> core/src/test/java/org/apache/oozie/util/db/TestPersistenceExceptionSubclassFilterRetryPredicate.java
>  PRE-CREATION 
>   core/src/test/java/org/apache/oozie/util/db/TestRetryAttemptState.java 
> PRE-CREATION 
>   docs/src/site/twiki/AG_Install.twiki 
> 2125442c273661d54f3566cbe41ead822c1f7baa 
>   minitest/pom.xml 9515284bb5f32c279a93161c10e6571680e4f9fc 
>   
> minitest/src/test/java/org/apache/oozie/test/TestParallelJPAOperationRetries.java
>  PRE-CREATION 
>   minitest/src/test/java/org/apache/oozie/test/TestWorkflowRetries.java 
> PRE-CREATION 
>   minitest/src/test/java/org/apache/oozie/test/WorkflowTest.java 
> 2845f0af6efb9ef75fdbfcb326115c62e6fb3bdd 
>   minitest/src/test/resources/hsqldb-oozie-site.xml 
> fa5fe9c3185e973e8247d7bf10b126119d9c02c9 
>   minitest/src/test/resources/oozie-log4j.properties 
> c142d725140930bfa89cd2b163d0768a4c3a750a 
>   minitest/src/test/resources/parallel-fs-and-shell.xml PRE-CREATION 
>   minitest/src/test/resources/wf-test.xml 
> 20c4946862039a65c76ed7f49991345e90a694de 
>   pom.xml 16c5137d44d7db891da46f80adb51c85e4c1b214 
> 
> 
> Diff: https://reviews.apache.org/r/60544/diff/7/
> 
> 
> Testing
> ---
> 
> Tests covered in code:
> 
> Unit tests
> ==
> 

[jira] [Assigned] (OOZIE-2984) Parse spark-defaults.conf values with spaces without needing the quotes.

[Andras Piros (JIRA)]

 [ 
https://issues.apache.org/jira/browse/OOZIE-2984?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Andras Piros reassigned OOZIE-2984:
---

Assignee: Andras Piros

> Parse spark-defaults.conf values with spaces without needing the quotes.
> 
>
> Key: OOZIE-2984
> URL: https://issues.apache.org/jira/browse/OOZIE-2984
> Project: Oozie
>  Issue Type: Bug
>Affects Versions: trunk
>Reporter: Fahd Siddiqui
>Assignee: Andras Piros
>
> Oozie requires multiple java opts to be enclosed in quotes. For example:
> {code}
> spark.driver.extraJavaOptions="-Xmn2703m -XX:SurvivorRatio=2 
> -XX:ParallelGCThreads=20"
> {code}
> However the above breaks spark-shell. 
> {code}
> $ ./spark2-shell
> Invalid initial young generation size: -Xmn2212m -XX:SurvivorRatio=2 
> -XX:ParallelGCThreads=20
> Error: Could not create the Java Virtual Machine.
> Error: A fatal exception has occurred. Program will exit.
> {code}
> To fix spark-shell, we have to remove the quotes, which errors out Oozie:
> {code}
> Error: Unrecognized option '-XX:SurvivorRatio=2'
> ...
> --conf
> spark.executor.extraJavaOptions=-Xmn2703m 
> -Dlog4j.configuration=spark-log4j.properties
> -XX:SurvivorRatio=2
> {code}
> Oozie should be able to parse spark-defaults.conf values with spaces without 
> needing the quotes.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

Re: Review Request 60544: OOZIE-2854 Oozie should handle transient database problems

[András Piros]

> On July 10, 2017, 7:17 p.m., Robert Kanter wrote:
> > core/src/main/resources/oozie-default.xml
> > Lines 1496 (patched)
> > 
> >
> > oozie.service.JPAService.jdbc.driver definately does work because it's 
> > set to Derby by default and we clearly can change it to other databases.  
> > Why say this doesn't have a real effect?

Actually it's `openJpa.connectionProperties="DriverClassName=..."`, making it 
more clear.


- András


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/60544/#review180082
---


On July 6, 2017, 5:36 p.m., András Piros wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/60544/
> ---
> 
> (Updated July 6, 2017, 5:36 p.m.)
> 
> 
> Review request for oozie, Attila Sasvari, Peter Cseh, Peter Bacsko, and 
> Robert Kanter.
> 
> 
> Repository: oozie-git
> 
> 
> Description
> ---
> 
> https://issues.apache.org/jira/browse/OOZIE-2854
> 
> 
> Diffs
> -
> 
>   core/pom.xml acddf349a89cf09a7fc4f384ebcaec56dfd0ab48 
>   
> core/src/main/java/org/apache/oozie/executor/jpa/JsonBeanPersisterExecutor.java
>  PRE-CREATION 
>   core/src/main/java/org/apache/oozie/executor/jpa/QueryExecutor.java 
> 8d94c23e40d1281864db40e141b200ca207a6324 
>   core/src/main/java/org/apache/oozie/service/JPAService.java 
> 028381d3b72bcc3b8c2cd27cacb3e0ac6d48d146 
>   core/src/main/java/org/apache/oozie/sla/SLASummaryBean.java 
> cfe1522a4b1f89085eb29e7f1281c2abd631bdc2 
>   core/src/main/java/org/apache/oozie/store/WorkflowStore.java 
> c565e74893b863caef6c93015cfe38fe520d04ec 
>   core/src/main/java/org/apache/oozie/util/db/BasicDataSourceWrapper.java 
> PRE-CREATION 
>   core/src/main/java/org/apache/oozie/util/db/DatabaseRetryPredicate.java 
> PRE-CREATION 
>   core/src/main/java/org/apache/oozie/util/db/FailingConnectionWrapper.java 
> PRE-CREATION 
>   core/src/main/java/org/apache/oozie/util/db/FailingHSQLDBDriverWrapper.java 
> PRE-CREATION 
>   core/src/main/java/org/apache/oozie/util/db/FailingMySQLDriverWrapper.java 
> PRE-CREATION 
>   core/src/main/java/org/apache/oozie/util/db/OperationRetryHandler.java 
> PRE-CREATION 
>   
> core/src/main/java/org/apache/oozie/util/db/PersistenceExceptionSubclassFilterRetryPredicate.java
>  PRE-CREATION 
>   core/src/main/java/org/apache/oozie/util/db/RetryAttemptState.java 
> PRE-CREATION 
>   core/src/main/java/org/apache/oozie/util/db/RuntimeExceptionInjector.java 
> PRE-CREATION 
>   core/src/main/resources/META-INF/persistence.xml 
> bad9278597fcd4f93b4cc482afae8af14beaa922 
>   core/src/main/resources/oozie-default.xml 
> 5629a897b4a1834554a1a5f7a21984ea857ed133 
>   core/src/main/resources/oozie-log4j.properties 
> c065f3cd4c5a3df1308b69d7c16e8fcfa8796efc 
>   core/src/test/java/org/apache/oozie/command/SkipCommitFaultInjection.java  
>   core/src/test/java/org/apache/oozie/test/XTestCase.java 
> 161927ac8f1132b3080d2924844826fcc7b807a5 
>   
> core/src/test/java/org/apache/oozie/util/db/TestOozieDmlStatementPredicate.java
>  PRE-CREATION 
>   core/src/test/java/org/apache/oozie/util/db/TestOperationRetryHandler.java 
> PRE-CREATION 
>   
> core/src/test/java/org/apache/oozie/util/db/TestPersistenceExceptionSubclassFilterRetryPredicate.java
>  PRE-CREATION 
>   core/src/test/java/org/apache/oozie/util/db/TestRetryAttemptState.java 
> PRE-CREATION 
>   docs/src/site/twiki/AG_Install.twiki 
> 2125442c273661d54f3566cbe41ead822c1f7baa 
>   minitest/pom.xml 9515284bb5f32c279a93161c10e6571680e4f9fc 
>   
> minitest/src/test/java/org/apache/oozie/test/TestParallelJPAOperationRetries.java
>  PRE-CREATION 
>   minitest/src/test/java/org/apache/oozie/test/TestWorkflowRetries.java 
> PRE-CREATION 
>   minitest/src/test/java/org/apache/oozie/test/WorkflowTest.java 
> 2845f0af6efb9ef75fdbfcb326115c62e6fb3bdd 
>   minitest/src/test/resources/hsqldb-oozie-site.xml 
> fa5fe9c3185e973e8247d7bf10b126119d9c02c9 
>   minitest/src/test/resources/oozie-log4j.properties 
> c142d725140930bfa89cd2b163d0768a4c3a750a 
>   minitest/src/test/resources/parallel-fs-and-shell.xml PRE-CREATION 
>   minitest/src/test/resources/wf-test.xml 
> 20c4946862039a65c76ed7f49991345e90a694de 
>   pom.xml 16c5137d44d7db891da46f80adb51c85e4c1b214 
> 
> 
> Diff: https://reviews.apache.org/r/60544/diff/7/
> 
> 
> Testing
> ---
> 
> Tests covered in code:
> 
> Unit tests
> ==
> 
> * testing the retry handler, the retry predicate filter, and parallel calls 
> to JPA `EntityManager` (mostly Oozie database reads and writes) when 
> injecting failures
> 
> Integration tests
> =
> 
> * using the `MiniOozieTestCase` framework
> * fixing it so that also asynchronous workflow appl

Re: Review Request 60544: OOZIE-2854 Oozie should handle transient database problems

[András Piros]

> On July 11, 2017, 7:47 a.m., Peter Cseh wrote:
> > core/src/main/java/org/apache/oozie/util/db/FailingConnectionWrapper.java
> > Lines 49 (patched)
> > 
> >
> > please move the Failing* classes to src/test from src/main. As far as I 
> > can see they are used only in tests.

Cannot do so. To be able to test failing behavior w/ MySQL runtime, we need 
`FailingMySQLDriverWrapper` and `FailingConnectionWrapper` on the runtime Oozie 
classpath. Best way to achieve that is to have them within `oozie-core` sources.

We don't have a classloading problem because most of the time 
`FailingMySQLDriverWrapper` won't be loaded (only if we set 
`oozie.service.JPAService.jdbc.driver` explicitly), and only this class 
`extends com.mysql.jdbc.Driver`.


- András


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/60544/#review180169
---


On July 6, 2017, 5:36 p.m., András Piros wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/60544/
> ---
> 
> (Updated July 6, 2017, 5:36 p.m.)
> 
> 
> Review request for oozie, Attila Sasvari, Peter Cseh, Peter Bacsko, and 
> Robert Kanter.
> 
> 
> Repository: oozie-git
> 
> 
> Description
> ---
> 
> https://issues.apache.org/jira/browse/OOZIE-2854
> 
> 
> Diffs
> -
> 
>   core/pom.xml acddf349a89cf09a7fc4f384ebcaec56dfd0ab48 
>   
> core/src/main/java/org/apache/oozie/executor/jpa/JsonBeanPersisterExecutor.java
>  PRE-CREATION 
>   core/src/main/java/org/apache/oozie/executor/jpa/QueryExecutor.java 
> 8d94c23e40d1281864db40e141b200ca207a6324 
>   core/src/main/java/org/apache/oozie/service/JPAService.java 
> 028381d3b72bcc3b8c2cd27cacb3e0ac6d48d146 
>   core/src/main/java/org/apache/oozie/sla/SLASummaryBean.java 
> cfe1522a4b1f89085eb29e7f1281c2abd631bdc2 
>   core/src/main/java/org/apache/oozie/store/WorkflowStore.java 
> c565e74893b863caef6c93015cfe38fe520d04ec 
>   core/src/main/java/org/apache/oozie/util/db/BasicDataSourceWrapper.java 
> PRE-CREATION 
>   core/src/main/java/org/apache/oozie/util/db/DatabaseRetryPredicate.java 
> PRE-CREATION 
>   core/src/main/java/org/apache/oozie/util/db/FailingConnectionWrapper.java 
> PRE-CREATION 
>   core/src/main/java/org/apache/oozie/util/db/FailingHSQLDBDriverWrapper.java 
> PRE-CREATION 
>   core/src/main/java/org/apache/oozie/util/db/FailingMySQLDriverWrapper.java 
> PRE-CREATION 
>   core/src/main/java/org/apache/oozie/util/db/OperationRetryHandler.java 
> PRE-CREATION 
>   
> core/src/main/java/org/apache/oozie/util/db/PersistenceExceptionSubclassFilterRetryPredicate.java
>  PRE-CREATION 
>   core/src/main/java/org/apache/oozie/util/db/RetryAttemptState.java 
> PRE-CREATION 
>   core/src/main/java/org/apache/oozie/util/db/RuntimeExceptionInjector.java 
> PRE-CREATION 
>   core/src/main/resources/META-INF/persistence.xml 
> bad9278597fcd4f93b4cc482afae8af14beaa922 
>   core/src/main/resources/oozie-default.xml 
> 5629a897b4a1834554a1a5f7a21984ea857ed133 
>   core/src/main/resources/oozie-log4j.properties 
> c065f3cd4c5a3df1308b69d7c16e8fcfa8796efc 
>   core/src/test/java/org/apache/oozie/command/SkipCommitFaultInjection.java  
>   core/src/test/java/org/apache/oozie/test/XTestCase.java 
> 161927ac8f1132b3080d2924844826fcc7b807a5 
>   
> core/src/test/java/org/apache/oozie/util/db/TestOozieDmlStatementPredicate.java
>  PRE-CREATION 
>   core/src/test/java/org/apache/oozie/util/db/TestOperationRetryHandler.java 
> PRE-CREATION 
>   
> core/src/test/java/org/apache/oozie/util/db/TestPersistenceExceptionSubclassFilterRetryPredicate.java
>  PRE-CREATION 
>   core/src/test/java/org/apache/oozie/util/db/TestRetryAttemptState.java 
> PRE-CREATION 
>   docs/src/site/twiki/AG_Install.twiki 
> 2125442c273661d54f3566cbe41ead822c1f7baa 
>   minitest/pom.xml 9515284bb5f32c279a93161c10e6571680e4f9fc 
>   
> minitest/src/test/java/org/apache/oozie/test/TestParallelJPAOperationRetries.java
>  PRE-CREATION 
>   minitest/src/test/java/org/apache/oozie/test/TestWorkflowRetries.java 
> PRE-CREATION 
>   minitest/src/test/java/org/apache/oozie/test/WorkflowTest.java 
> 2845f0af6efb9ef75fdbfcb326115c62e6fb3bdd 
>   minitest/src/test/resources/hsqldb-oozie-site.xml 
> fa5fe9c3185e973e8247d7bf10b126119d9c02c9 
>   minitest/src/test/resources/oozie-log4j.properties 
> c142d725140930bfa89cd2b163d0768a4c3a750a 
>   minitest/src/test/resources/parallel-fs-and-shell.xml PRE-CREATION 
>   minitest/src/test/resources/wf-test.xml 
> 20c4946862039a65c76ed7f49991345e90a694de 
>   pom.xml 16c5137d44d7db891da46f80adb51c85e4c1b214 
> 
> 
> Diff: https://reviews.apache.org/r/60544/diff/7/
> 
> 
> Testing
> ---
> 
> Tests covered in code:
> 
> Unit tests
> ===

[jira] [Commented] (OOZIE-2984) Parse spark-defaults.conf values with spaces without needing the quotes.

[Peter Cseh (JIRA)]

[ 
https://issues.apache.org/jira/browse/OOZIE-2984?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16082113#comment-16082113
 ] 

Peter Cseh commented on OOZIE-2984:
---

To solve this properly we should understand what values are permitted in the 
Spark configuration file. E.g: Are quotes permitted inside an option like 
{{spark.driver.extraJavaOptions=-Xmn2703m -XX:SurvivorRatio=2 
-XX:ParallelGCThreads=20 -Dsome="example java option"}}?


> Parse spark-defaults.conf values with spaces without needing the quotes.
> 
>
> Key: OOZIE-2984
> URL: https://issues.apache.org/jira/browse/OOZIE-2984
> Project: Oozie
>  Issue Type: Bug
>Affects Versions: trunk
>Reporter: Fahd Siddiqui
>Assignee: Andras Piros
>
> Oozie requires multiple java opts to be enclosed in quotes. For example:
> {code}
> spark.driver.extraJavaOptions="-Xmn2703m -XX:SurvivorRatio=2 
> -XX:ParallelGCThreads=20"
> {code}
> However the above breaks spark-shell. 
> {code}
> $ ./spark2-shell
> Invalid initial young generation size: -Xmn2212m -XX:SurvivorRatio=2 
> -XX:ParallelGCThreads=20
> Error: Could not create the Java Virtual Machine.
> Error: A fatal exception has occurred. Program will exit.
> {code}
> To fix spark-shell, we have to remove the quotes, which errors out Oozie:
> {code}
> Error: Unrecognized option '-XX:SurvivorRatio=2'
> ...
> --conf
> spark.executor.extraJavaOptions=-Xmn2703m 
> -Dlog4j.configuration=spark-log4j.properties
> -XX:SurvivorRatio=2
> {code}
> Oozie should be able to parse spark-defaults.conf values with spaces without 
> needing the quotes.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Updated] (OOZIE-2854) Oozie should handle transient database problems

[Andras Piros (JIRA)]

 [ 
https://issues.apache.org/jira/browse/OOZIE-2854?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Andras Piros updated OOZIE-2854:

Attachment: OOZIE-2854.012.patch

Addressing another round on review comments.

> Oozie should handle transient database problems
> ---
>
> Key: OOZIE-2854
> URL: https://issues.apache.org/jira/browse/OOZIE-2854
> Project: Oozie
>  Issue Type: Improvement
>  Components: core
>Reporter: Peter Bacsko
>Assignee: Andras Piros
> Attachments: OOZIE-2854-001.patch, OOZIE-2854-002.patch, 
> OOZIE-2854-003.patch, OOZIE-2854-004.patch, OOZIE-2854-005.patch, 
> OOZIE-2854.006.patch, OOZIE-2854.007.patch, OOZIE-2854.008.patch, 
> OOZIE-2854.009.patch, OOZIE-2854.010.patch, OOZIE-2854.011.patch, 
> OOZIE-2854.012.patch, OOZIE-2854-POC-001.patch
>
>
> There can be problems when Oozie cannot update the database properly. 
> Recently, we have experienced erratic behavior with two setups:
> * MySQL with the Galera cluster manager. Galera uses cluster-wide optimistic 
> locking which might cause a transaction to rollback if there are two or more 
> parallel transaction running and one of them cannot complete because of a 
> conflict.
> * MySQL with Percona XtraDB Cluster. If one of the MySQL instances is killed, 
> Oozie might get "Communications link failure" exception during the failover.
> The problem is that failed DB transactions later might cause a workflow 
> (which are started/re-started by RecoveryService) to get stuck. It's not 
> clear to us how this happens but it has to do with the fact that certain DB 
> updates are not executed.
> The solution is to use some sort of retry logic with exponential backoff if 
> the DB update fails. We could start with a 100ms wait time which is doubled 
> at every retry. The operation can be considered a failure if it still fails 
> after 10 attempts. These values could be configurable. We should discuss 
> initial values in the scope of this JIRA.
> Note that this solution is to handle *transient* failures. If the DB is down 
> for a longer period of time, we have to accept that the internal state of 
> Oozie is corrupted.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

Re: Review Request 60544: OOZIE-2854 Oozie should handle transient database problems

[András Piros]

> On July 11, 2017, 7:53 a.m., Peter Cseh wrote:
> > core/src/main/java/org/apache/oozie/util/db/FailingMySQLDriverWrapper.java
> > Lines 27 (patched)
> > 
> >
> > This class is not used anywhere. Please remove it or document what it 
> > can be used for (e.g. on-site testing)

Documenting usage (error injecting MySQL behavior to test runtime) on the class 
level, and inside TWiki entry.


- András


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/60544/#review180170
---


On July 6, 2017, 5:36 p.m., András Piros wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/60544/
> ---
> 
> (Updated July 6, 2017, 5:36 p.m.)
> 
> 
> Review request for oozie, Attila Sasvari, Peter Cseh, Peter Bacsko, and 
> Robert Kanter.
> 
> 
> Repository: oozie-git
> 
> 
> Description
> ---
> 
> https://issues.apache.org/jira/browse/OOZIE-2854
> 
> 
> Diffs
> -
> 
>   core/pom.xml acddf349a89cf09a7fc4f384ebcaec56dfd0ab48 
>   
> core/src/main/java/org/apache/oozie/executor/jpa/JsonBeanPersisterExecutor.java
>  PRE-CREATION 
>   core/src/main/java/org/apache/oozie/executor/jpa/QueryExecutor.java 
> 8d94c23e40d1281864db40e141b200ca207a6324 
>   core/src/main/java/org/apache/oozie/service/JPAService.java 
> 028381d3b72bcc3b8c2cd27cacb3e0ac6d48d146 
>   core/src/main/java/org/apache/oozie/sla/SLASummaryBean.java 
> cfe1522a4b1f89085eb29e7f1281c2abd631bdc2 
>   core/src/main/java/org/apache/oozie/store/WorkflowStore.java 
> c565e74893b863caef6c93015cfe38fe520d04ec 
>   core/src/main/java/org/apache/oozie/util/db/BasicDataSourceWrapper.java 
> PRE-CREATION 
>   core/src/main/java/org/apache/oozie/util/db/DatabaseRetryPredicate.java 
> PRE-CREATION 
>   core/src/main/java/org/apache/oozie/util/db/FailingConnectionWrapper.java 
> PRE-CREATION 
>   core/src/main/java/org/apache/oozie/util/db/FailingHSQLDBDriverWrapper.java 
> PRE-CREATION 
>   core/src/main/java/org/apache/oozie/util/db/FailingMySQLDriverWrapper.java 
> PRE-CREATION 
>   core/src/main/java/org/apache/oozie/util/db/OperationRetryHandler.java 
> PRE-CREATION 
>   
> core/src/main/java/org/apache/oozie/util/db/PersistenceExceptionSubclassFilterRetryPredicate.java
>  PRE-CREATION 
>   core/src/main/java/org/apache/oozie/util/db/RetryAttemptState.java 
> PRE-CREATION 
>   core/src/main/java/org/apache/oozie/util/db/RuntimeExceptionInjector.java 
> PRE-CREATION 
>   core/src/main/resources/META-INF/persistence.xml 
> bad9278597fcd4f93b4cc482afae8af14beaa922 
>   core/src/main/resources/oozie-default.xml 
> 5629a897b4a1834554a1a5f7a21984ea857ed133 
>   core/src/main/resources/oozie-log4j.properties 
> c065f3cd4c5a3df1308b69d7c16e8fcfa8796efc 
>   core/src/test/java/org/apache/oozie/command/SkipCommitFaultInjection.java  
>   core/src/test/java/org/apache/oozie/test/XTestCase.java 
> 161927ac8f1132b3080d2924844826fcc7b807a5 
>   
> core/src/test/java/org/apache/oozie/util/db/TestOozieDmlStatementPredicate.java
>  PRE-CREATION 
>   core/src/test/java/org/apache/oozie/util/db/TestOperationRetryHandler.java 
> PRE-CREATION 
>   
> core/src/test/java/org/apache/oozie/util/db/TestPersistenceExceptionSubclassFilterRetryPredicate.java
>  PRE-CREATION 
>   core/src/test/java/org/apache/oozie/util/db/TestRetryAttemptState.java 
> PRE-CREATION 
>   docs/src/site/twiki/AG_Install.twiki 
> 2125442c273661d54f3566cbe41ead822c1f7baa 
>   minitest/pom.xml 9515284bb5f32c279a93161c10e6571680e4f9fc 
>   
> minitest/src/test/java/org/apache/oozie/test/TestParallelJPAOperationRetries.java
>  PRE-CREATION 
>   minitest/src/test/java/org/apache/oozie/test/TestWorkflowRetries.java 
> PRE-CREATION 
>   minitest/src/test/java/org/apache/oozie/test/WorkflowTest.java 
> 2845f0af6efb9ef75fdbfcb326115c62e6fb3bdd 
>   minitest/src/test/resources/hsqldb-oozie-site.xml 
> fa5fe9c3185e973e8247d7bf10b126119d9c02c9 
>   minitest/src/test/resources/oozie-log4j.properties 
> c142d725140930bfa89cd2b163d0768a4c3a750a 
>   minitest/src/test/resources/parallel-fs-and-shell.xml PRE-CREATION 
>   minitest/src/test/resources/wf-test.xml 
> 20c4946862039a65c76ed7f49991345e90a694de 
>   pom.xml 16c5137d44d7db891da46f80adb51c85e4c1b214 
> 
> 
> Diff: https://reviews.apache.org/r/60544/diff/7/
> 
> 
> Testing
> ---
> 
> Tests covered in code:
> 
> Unit tests
> ==
> 
> * testing the retry handler, the retry predicate filter, and parallel calls 
> to JPA `EntityManager` (mostly Oozie database reads and writes) when 
> injecting failures
> 
> Integration tests
> =
> 
> * using the `MiniOozieTestCase` framework
> * fixing it so that also asynchronous workflow applications (the ones that 
> use `Cal

Re: Review Request 60544: OOZIE-2854 Oozie should handle transient database problems

[András Piros]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/60544/
---

(Updated July 11, 2017, 12:01 p.m.)


Review request for oozie, Attila Sasvari, Peter Cseh, Peter Bacsko, and Robert 
Kanter.


Changes
---

Addressing another round on review comments.


Repository: oozie-git


Description
---

https://issues.apache.org/jira/browse/OOZIE-2854


Diffs (updated)
-

  core/pom.xml acddf349a89cf09a7fc4f384ebcaec56dfd0ab48 
  
core/src/main/java/org/apache/oozie/executor/jpa/JsonBeanPersisterExecutor.java 
PRE-CREATION 
  core/src/main/java/org/apache/oozie/executor/jpa/QueryExecutor.java 
8d94c23e40d1281864db40e141b200ca207a6324 
  core/src/main/java/org/apache/oozie/service/JPAService.java 
028381d3b72bcc3b8c2cd27cacb3e0ac6d48d146 
  core/src/main/java/org/apache/oozie/sla/SLASummaryBean.java 
cfe1522a4b1f89085eb29e7f1281c2abd631bdc2 
  core/src/main/java/org/apache/oozie/store/WorkflowStore.java 
c565e74893b863caef6c93015cfe38fe520d04ec 
  core/src/main/java/org/apache/oozie/util/db/BasicDataSourceWrapper.java 
PRE-CREATION 
  core/src/main/java/org/apache/oozie/util/db/DatabaseRetryPredicate.java 
PRE-CREATION 
  core/src/main/java/org/apache/oozie/util/db/FailingConnectionWrapper.java 
PRE-CREATION 
  core/src/main/java/org/apache/oozie/util/db/FailingHSQLDBDriverWrapper.java 
PRE-CREATION 
  core/src/main/java/org/apache/oozie/util/db/FailingMySQLDriverWrapper.java 
PRE-CREATION 
  core/src/main/java/org/apache/oozie/util/db/OperationRetryHandler.java 
PRE-CREATION 
  
core/src/main/java/org/apache/oozie/util/db/PersistenceExceptionSubclassFilterRetryPredicate.java
 PRE-CREATION 
  core/src/main/java/org/apache/oozie/util/db/RetryAttemptState.java 
PRE-CREATION 
  core/src/main/java/org/apache/oozie/util/db/RuntimeExceptionInjector.java 
PRE-CREATION 
  core/src/main/resources/META-INF/persistence.xml 
bad9278597fcd4f93b4cc482afae8af14beaa922 
  core/src/main/resources/oozie-default.xml 
5629a897b4a1834554a1a5f7a21984ea857ed133 
  core/src/main/resources/oozie-log4j.properties 
c065f3cd4c5a3df1308b69d7c16e8fcfa8796efc 
  core/src/test/java/org/apache/oozie/command/SkipCommitFaultInjection.java  
  core/src/test/java/org/apache/oozie/test/XTestCase.java 
2816e5d80efd89151606b364376120a76b5c5a4a 
  
core/src/test/java/org/apache/oozie/util/db/TestOozieDmlStatementPredicate.java 
PRE-CREATION 
  core/src/test/java/org/apache/oozie/util/db/TestOperationRetryHandler.java 
PRE-CREATION 
  
core/src/test/java/org/apache/oozie/util/db/TestPersistenceExceptionSubclassFilterRetryPredicate.java
 PRE-CREATION 
  core/src/test/java/org/apache/oozie/util/db/TestRetryAttemptState.java 
PRE-CREATION 
  docs/src/site/twiki/AG_Install.twiki 2125442c273661d54f3566cbe41ead822c1f7baa 
  minitest/pom.xml 9515284bb5f32c279a93161c10e6571680e4f9fc 
  
minitest/src/test/java/org/apache/oozie/test/TestParallelJPAOperationRetries.java
 PRE-CREATION 
  minitest/src/test/java/org/apache/oozie/test/TestWorkflowRetries.java 
PRE-CREATION 
  minitest/src/test/java/org/apache/oozie/test/WorkflowTest.java 
2845f0af6efb9ef75fdbfcb326115c62e6fb3bdd 
  minitest/src/test/resources/hsqldb-oozie-site.xml 
fa5fe9c3185e973e8247d7bf10b126119d9c02c9 
  minitest/src/test/resources/oozie-log4j.properties 
c142d725140930bfa89cd2b163d0768a4c3a750a 
  minitest/src/test/resources/parallel-fs-and-shell.xml PRE-CREATION 
  minitest/src/test/resources/wf-test.xml 
20c4946862039a65c76ed7f49991345e90a694de 
  pom.xml 16c5137d44d7db891da46f80adb51c85e4c1b214 


Diff: https://reviews.apache.org/r/60544/diff/8/

Changes: https://reviews.apache.org/r/60544/diff/7-8/


Testing
---

Tests covered in code:

Unit tests
==

* testing the retry handler, the retry predicate filter, and parallel calls to 
JPA `EntityManager` (mostly Oozie database reads and writes) when injecting 
failures

Integration tests
=

* using the `MiniOozieTestCase` framework
* fixing it so that also asynchronous workflow applications (the ones that use 
`CallableQueueService`) can be run
* following workflow scenarios:
* a very simple one consisting only of a `` and an `` node
* a more sophisticated one consisting of multiple synchronous `` nodes and 
a `` node
* the ultimate one consisting of a `` node, and two branches of an 
`` and an asynchronous `` nodes

Test cases run:
```
mvn clean test 
-Dtest=TestOperationRetryHandler,TestPersistenceExceptionSubclassFilterRetryPredicate,TestParallelJPAOperationRetries,TestWorkflow,TestWorkflowRetries,TestJPAService,TestRetryAttemptState
```

Functional and stress tests performed on a 4-node MySQL cluster. MySQL daemon 
has been stopped / killed / restarted several times. Also firewall rules have 
been modified temporarily to simulate network outages.


Thanks,

András Piros

Re: Review Request 60544: OOZIE-2854 Oozie should handle transient database problems

[Peter Cseh]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/60544/#review180184
---




core/src/main/java/org/apache/oozie/util/db/OperationRetryHandler.java
Lines 79-86 (patched)


These log messages about being on the blacklist and handling rerties are 
seem to be mixed up


- Peter Cseh


On July 11, 2017, 12:01 p.m., András Piros wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/60544/
> ---
> 
> (Updated July 11, 2017, 12:01 p.m.)
> 
> 
> Review request for oozie, Attila Sasvari, Peter Cseh, Peter Bacsko, and 
> Robert Kanter.
> 
> 
> Repository: oozie-git
> 
> 
> Description
> ---
> 
> https://issues.apache.org/jira/browse/OOZIE-2854
> 
> 
> Diffs
> -
> 
>   core/pom.xml acddf349a89cf09a7fc4f384ebcaec56dfd0ab48 
>   
> core/src/main/java/org/apache/oozie/executor/jpa/JsonBeanPersisterExecutor.java
>  PRE-CREATION 
>   core/src/main/java/org/apache/oozie/executor/jpa/QueryExecutor.java 
> 8d94c23e40d1281864db40e141b200ca207a6324 
>   core/src/main/java/org/apache/oozie/service/JPAService.java 
> 028381d3b72bcc3b8c2cd27cacb3e0ac6d48d146 
>   core/src/main/java/org/apache/oozie/sla/SLASummaryBean.java 
> cfe1522a4b1f89085eb29e7f1281c2abd631bdc2 
>   core/src/main/java/org/apache/oozie/store/WorkflowStore.java 
> c565e74893b863caef6c93015cfe38fe520d04ec 
>   core/src/main/java/org/apache/oozie/util/db/BasicDataSourceWrapper.java 
> PRE-CREATION 
>   core/src/main/java/org/apache/oozie/util/db/DatabaseRetryPredicate.java 
> PRE-CREATION 
>   core/src/main/java/org/apache/oozie/util/db/FailingConnectionWrapper.java 
> PRE-CREATION 
>   core/src/main/java/org/apache/oozie/util/db/FailingHSQLDBDriverWrapper.java 
> PRE-CREATION 
>   core/src/main/java/org/apache/oozie/util/db/FailingMySQLDriverWrapper.java 
> PRE-CREATION 
>   core/src/main/java/org/apache/oozie/util/db/OperationRetryHandler.java 
> PRE-CREATION 
>   
> core/src/main/java/org/apache/oozie/util/db/PersistenceExceptionSubclassFilterRetryPredicate.java
>  PRE-CREATION 
>   core/src/main/java/org/apache/oozie/util/db/RetryAttemptState.java 
> PRE-CREATION 
>   core/src/main/java/org/apache/oozie/util/db/RuntimeExceptionInjector.java 
> PRE-CREATION 
>   core/src/main/resources/META-INF/persistence.xml 
> bad9278597fcd4f93b4cc482afae8af14beaa922 
>   core/src/main/resources/oozie-default.xml 
> 5629a897b4a1834554a1a5f7a21984ea857ed133 
>   core/src/main/resources/oozie-log4j.properties 
> c065f3cd4c5a3df1308b69d7c16e8fcfa8796efc 
>   core/src/test/java/org/apache/oozie/command/SkipCommitFaultInjection.java  
>   core/src/test/java/org/apache/oozie/test/XTestCase.java 
> 2816e5d80efd89151606b364376120a76b5c5a4a 
>   
> core/src/test/java/org/apache/oozie/util/db/TestOozieDmlStatementPredicate.java
>  PRE-CREATION 
>   core/src/test/java/org/apache/oozie/util/db/TestOperationRetryHandler.java 
> PRE-CREATION 
>   
> core/src/test/java/org/apache/oozie/util/db/TestPersistenceExceptionSubclassFilterRetryPredicate.java
>  PRE-CREATION 
>   core/src/test/java/org/apache/oozie/util/db/TestRetryAttemptState.java 
> PRE-CREATION 
>   docs/src/site/twiki/AG_Install.twiki 
> 2125442c273661d54f3566cbe41ead822c1f7baa 
>   minitest/pom.xml 9515284bb5f32c279a93161c10e6571680e4f9fc 
>   
> minitest/src/test/java/org/apache/oozie/test/TestParallelJPAOperationRetries.java
>  PRE-CREATION 
>   minitest/src/test/java/org/apache/oozie/test/TestWorkflowRetries.java 
> PRE-CREATION 
>   minitest/src/test/java/org/apache/oozie/test/WorkflowTest.java 
> 2845f0af6efb9ef75fdbfcb326115c62e6fb3bdd 
>   minitest/src/test/resources/hsqldb-oozie-site.xml 
> fa5fe9c3185e973e8247d7bf10b126119d9c02c9 
>   minitest/src/test/resources/oozie-log4j.properties 
> c142d725140930bfa89cd2b163d0768a4c3a750a 
>   minitest/src/test/resources/parallel-fs-and-shell.xml PRE-CREATION 
>   minitest/src/test/resources/wf-test.xml 
> 20c4946862039a65c76ed7f49991345e90a694de 
>   pom.xml 16c5137d44d7db891da46f80adb51c85e4c1b214 
> 
> 
> Diff: https://reviews.apache.org/r/60544/diff/8/
> 
> 
> Testing
> ---
> 
> Tests covered in code:
> 
> Unit tests
> ==
> 
> * testing the retry handler, the retry predicate filter, and parallel calls 
> to JPA `EntityManager` (mostly Oozie database reads and writes) when 
> injecting failures
> 
> Integration tests
> =
> 
> * using the `MiniOozieTestCase` framework
> * fixing it so that also asynchronous workflow applications (the ones that 
> use `CallableQueueService`) can be run
> * following workflow scenarios:
> * a very simple one consisting only of a `` and an `` node
> * a more sophisticated one consisting of multiple synchronous `` nodes 
> and a `` node
>

[jira] [Comment Edited] (OOZIE-2978) Remove code that handles Pig versions before 0.8

[Attila Sasvari (JIRA)]

[ 
https://issues.apache.org/jira/browse/OOZIE-2978?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16082157#comment-16082157
 ] 

Attila Sasvari edited comment on OOZIE-2978 at 7/11/17 12:50 PM:
-

[~pbacsko]I added back code block, and decided to keep comment about embedded 
python


was (Author: asasvari):
@pbacsko I added back code block, and decided to keep comment about embedded 
python

> Remove code that handles Pig versions before 0.8 
> -
>
> Key: OOZIE-2978
> URL: https://issues.apache.org/jira/browse/OOZIE-2978
> Project: Oozie
>  Issue Type: Bug
>Reporter: Attila Sasvari
>Assignee: Attila Sasvari
> Attachments: OOZIE-2978-01.patch, OOZIE-2978-02.patch, 
> OOZIE-2978-03.patch, OOZIE-2978-04.patch, OOZIE-2978-05.patch
>
>
> Oozie submits Pig jobs according to given Pig version. For example, Pig 
> versions before 0.8 are treated in the following way:
>  
> [Main.main(args)|https://github.com/apache/oozie/blob/678f1b2b2cc9a2456c7e3d9f693d2637f1f03257/sharelib/pig/src/main/java/org/apache/oozie/action/hadoop/PigMain.java#L382]
> However, we dropped support for *Hadoop 1*, *Hadoop 0.23* support via 
> {{OOZIE-2316}}. I wonder whether submitting jobs using Pig version before 0.8 
> will work at all:
> - 
> https://pig.apache.org/releases.html#13+May%2C+2010%3A+release+0.7.0+available
>  says "This release works with Hadoop 0.20". Pig *0.10.1* is the first pig 
> release that  works with Hadoop 2.
> - Looking at [PIG-2125|https://issues.apache.org/jira/browse/PIG-2125], I 
> believe the answer is NO.  
> Therefore we can safely delete corresponding parts from the codebase. There 
> is also another class {{PigMainWithOldAPI}} that is only used by tests in 
> {{TestPigMainWithOldAPI}}. So we can remove these too.
>  



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Updated] (OOZIE-2978) Remove code that handles Pig versions before 0.8

[Attila Sasvari (JIRA)]

 [ 
https://issues.apache.org/jira/browse/OOZIE-2978?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Attila Sasvari updated OOZIE-2978:
--
Attachment: OOZIE-2978-05.patch

@pbacsko I added back code block, and decided to keep comment about embedded 
python

> Remove code that handles Pig versions before 0.8 
> -
>
> Key: OOZIE-2978
> URL: https://issues.apache.org/jira/browse/OOZIE-2978
> Project: Oozie
>  Issue Type: Bug
>Reporter: Attila Sasvari
>Assignee: Attila Sasvari
> Attachments: OOZIE-2978-01.patch, OOZIE-2978-02.patch, 
> OOZIE-2978-03.patch, OOZIE-2978-04.patch, OOZIE-2978-05.patch
>
>
> Oozie submits Pig jobs according to given Pig version. For example, Pig 
> versions before 0.8 are treated in the following way:
>  
> [Main.main(args)|https://github.com/apache/oozie/blob/678f1b2b2cc9a2456c7e3d9f693d2637f1f03257/sharelib/pig/src/main/java/org/apache/oozie/action/hadoop/PigMain.java#L382]
> However, we dropped support for *Hadoop 1*, *Hadoop 0.23* support via 
> {{OOZIE-2316}}. I wonder whether submitting jobs using Pig version before 0.8 
> will work at all:
> - 
> https://pig.apache.org/releases.html#13+May%2C+2010%3A+release+0.7.0+available
>  says "This release works with Hadoop 0.20". Pig *0.10.1* is the first pig 
> release that  works with Hadoop 2.
> - Looking at [PIG-2125|https://issues.apache.org/jira/browse/PIG-2125], I 
> believe the answer is NO.  
> Therefore we can safely delete corresponding parts from the codebase. There 
> is also another class {{PigMainWithOldAPI}} that is only used by tests in 
> {{TestPigMainWithOldAPI}}. So we can remove these too.
>  



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

Re: Review Request 60544: OOZIE-2854 Oozie should handle transient database problems

[Peter Cseh]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/60544/#review180182
---




core/src/main/java/org/apache/oozie/util/db/OperationRetryHandler.java
Lines 41 (patched)


Based on the doc 0 should be allowed, shouldn't they?


- Peter Cseh


On July 11, 2017, 12:01 p.m., András Piros wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/60544/
> ---
> 
> (Updated July 11, 2017, 12:01 p.m.)
> 
> 
> Review request for oozie, Attila Sasvari, Peter Cseh, Peter Bacsko, and 
> Robert Kanter.
> 
> 
> Repository: oozie-git
> 
> 
> Description
> ---
> 
> https://issues.apache.org/jira/browse/OOZIE-2854
> 
> 
> Diffs
> -
> 
>   core/pom.xml acddf349a89cf09a7fc4f384ebcaec56dfd0ab48 
>   
> core/src/main/java/org/apache/oozie/executor/jpa/JsonBeanPersisterExecutor.java
>  PRE-CREATION 
>   core/src/main/java/org/apache/oozie/executor/jpa/QueryExecutor.java 
> 8d94c23e40d1281864db40e141b200ca207a6324 
>   core/src/main/java/org/apache/oozie/service/JPAService.java 
> 028381d3b72bcc3b8c2cd27cacb3e0ac6d48d146 
>   core/src/main/java/org/apache/oozie/sla/SLASummaryBean.java 
> cfe1522a4b1f89085eb29e7f1281c2abd631bdc2 
>   core/src/main/java/org/apache/oozie/store/WorkflowStore.java 
> c565e74893b863caef6c93015cfe38fe520d04ec 
>   core/src/main/java/org/apache/oozie/util/db/BasicDataSourceWrapper.java 
> PRE-CREATION 
>   core/src/main/java/org/apache/oozie/util/db/DatabaseRetryPredicate.java 
> PRE-CREATION 
>   core/src/main/java/org/apache/oozie/util/db/FailingConnectionWrapper.java 
> PRE-CREATION 
>   core/src/main/java/org/apache/oozie/util/db/FailingHSQLDBDriverWrapper.java 
> PRE-CREATION 
>   core/src/main/java/org/apache/oozie/util/db/FailingMySQLDriverWrapper.java 
> PRE-CREATION 
>   core/src/main/java/org/apache/oozie/util/db/OperationRetryHandler.java 
> PRE-CREATION 
>   
> core/src/main/java/org/apache/oozie/util/db/PersistenceExceptionSubclassFilterRetryPredicate.java
>  PRE-CREATION 
>   core/src/main/java/org/apache/oozie/util/db/RetryAttemptState.java 
> PRE-CREATION 
>   core/src/main/java/org/apache/oozie/util/db/RuntimeExceptionInjector.java 
> PRE-CREATION 
>   core/src/main/resources/META-INF/persistence.xml 
> bad9278597fcd4f93b4cc482afae8af14beaa922 
>   core/src/main/resources/oozie-default.xml 
> 5629a897b4a1834554a1a5f7a21984ea857ed133 
>   core/src/main/resources/oozie-log4j.properties 
> c065f3cd4c5a3df1308b69d7c16e8fcfa8796efc 
>   core/src/test/java/org/apache/oozie/command/SkipCommitFaultInjection.java  
>   core/src/test/java/org/apache/oozie/test/XTestCase.java 
> 2816e5d80efd89151606b364376120a76b5c5a4a 
>   
> core/src/test/java/org/apache/oozie/util/db/TestOozieDmlStatementPredicate.java
>  PRE-CREATION 
>   core/src/test/java/org/apache/oozie/util/db/TestOperationRetryHandler.java 
> PRE-CREATION 
>   
> core/src/test/java/org/apache/oozie/util/db/TestPersistenceExceptionSubclassFilterRetryPredicate.java
>  PRE-CREATION 
>   core/src/test/java/org/apache/oozie/util/db/TestRetryAttemptState.java 
> PRE-CREATION 
>   docs/src/site/twiki/AG_Install.twiki 
> 2125442c273661d54f3566cbe41ead822c1f7baa 
>   minitest/pom.xml 9515284bb5f32c279a93161c10e6571680e4f9fc 
>   
> minitest/src/test/java/org/apache/oozie/test/TestParallelJPAOperationRetries.java
>  PRE-CREATION 
>   minitest/src/test/java/org/apache/oozie/test/TestWorkflowRetries.java 
> PRE-CREATION 
>   minitest/src/test/java/org/apache/oozie/test/WorkflowTest.java 
> 2845f0af6efb9ef75fdbfcb326115c62e6fb3bdd 
>   minitest/src/test/resources/hsqldb-oozie-site.xml 
> fa5fe9c3185e973e8247d7bf10b126119d9c02c9 
>   minitest/src/test/resources/oozie-log4j.properties 
> c142d725140930bfa89cd2b163d0768a4c3a750a 
>   minitest/src/test/resources/parallel-fs-and-shell.xml PRE-CREATION 
>   minitest/src/test/resources/wf-test.xml 
> 20c4946862039a65c76ed7f49991345e90a694de 
>   pom.xml 16c5137d44d7db891da46f80adb51c85e4c1b214 
> 
> 
> Diff: https://reviews.apache.org/r/60544/diff/8/
> 
> 
> Testing
> ---
> 
> Tests covered in code:
> 
> Unit tests
> ==
> 
> * testing the retry handler, the retry predicate filter, and parallel calls 
> to JPA `EntityManager` (mostly Oozie database reads and writes) when 
> injecting failures
> 
> Integration tests
> =
> 
> * using the `MiniOozieTestCase` framework
> * fixing it so that also asynchronous workflow applications (the ones that 
> use `CallableQueueService`) can be run
> * following workflow scenarios:
> * a very simple one consisting only of a `` and an `` node
> * a more sophisticated one consisting of multiple synchronous `` nodes 
> and a `` node
> * the ultimate one consisting of a `` node

[jira] [Updated] (OOZIE-2601) Ability to use local paths for the sharelib

[Attila Sasvari (JIRA)]

 [ 
https://issues.apache.org/jira/browse/OOZIE-2601?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Attila Sasvari updated OOZIE-2601:
--
Attachment: OOZIE-2601-03.patch

> Ability to use local paths for the sharelib
> ---
>
> Key: OOZIE-2601
> URL: https://issues.apache.org/jira/browse/OOZIE-2601
> Project: Oozie
>  Issue Type: New Feature
>Affects Versions: oya
>Reporter: Robert Kanter
>Assignee: Attila Sasvari
> Fix For: oya
>
> Attachments: OOZIE-2601-01.patch, OOZIE-2601-03.patch
>
>
> With OOZIE-2590, as part of OOZIE-1770 Oozie on Yarn work, Oozie now has full 
> control over the classpath given to the Launcher AM.  In a cluster where all 
> nodes have everything installed locally (in the same paths), it should be 
> possible to have the Launcher AM reference the local jars instead of having 
> to localize them from HDFS.
> For example, if you have Hive installed on all nodes at {{/usr/lib/hive/}} 
> and all Hive jars under {{/usr/lib/hive/lib/}}, we could have the Launcher AM 
> add {{/usr/lib/hive/lib}} to its classpath.  This saves on the overhead of 
> localizing the same jars from the hive sharelib in HDFS.  
> I think the best way to implement this is to augment the [Sharelib Mapping 
> File|https://oozie.apache.org/docs/4.2.0/AG_Install.html#Oozie_Share_Lib] 
> feature to accept {{file:///}} paths.
> If we had this also work with the "oozie" sharelib and the Oozie jars in the 
> individual sharelibs (e.g. have the Mapping file take comma-separated 
> dirs/jars), then in a cluster with everything installed on all of the nodes, 
> you wouldn't need to bother with the sharelib at all!



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

Build failed in Jenkins: oozie-trunk-precommit-build #3964

[Apache Jenkins Server]

See 


--
[...truncated 334.34 KB...]
[INFO] Copying jackson-core-asl-1.9.13.jar to 

[INFO] Copying commons-jexl-2.1.1.jar to 

[INFO] Copying oozie-sharelib-hcatalog-5.0.0-SNAPSHOT.jar to 

[INFO] Copying hadoop-mapreduce-client-jobclient-2.6.0.jar to 

[INFO] Copying gson-2.7.jar to 

[INFO] Copying oozie-core-5.0.0-SNAPSHOT.jar to 

[INFO] Copying commons-pool-1.5.4.jar to 

[INFO] Copying htrace-core-3.0.4.jar to 

[INFO] Copying jaxb-impl-2.2.3-1.jar to 

[INFO] Copying jets3t-0.9.0.jar to 

[INFO] Copying datanucleus-rdbms-3.2.9.jar to 

[INFO] Copying hive-common-1.2.0.jar to 

[INFO] Copying snappy-java-1.0.4.1.jar to 

[INFO] Copying javax.annotation-api-1.2.jar to 

[INFO] Copying commons-compiler-2.7.6.jar to 

[INFO] Copying ant-1.9.1.jar to 

[INFO] Copying hadoop-annotations-2.6.0.jar to 

[INFO] Copying janino-2.7.6.jar to 

[INFO] Copying activemq-client-5.13.3.jar to 

[INFO] Copying guice-3.0.jar to 

[INFO] Copying httpclient-4.3.6.jar to 

[INFO] Copying xz-1.0.jar to 

[INFO] Copying oozie-client-5.0.0-SNAPSHOT.jar to 

[INFO] Copying hadoop-mapreduce-client-common-2.6.0.jar to 

[INFO] Copying leveldbjni-all-1.8.jar to 

[INFO] Copying jetty-util-6.1.26.jar to 

[INFO] Copying velocity-1.5.jar to 

[INFO] Copying jackson-core-2.4.2.jar to 

[INFO] Copying apache-log4j-extras-1.2.17.jar to 

[INFO] Copying netty-3.7.0.Final.jar to 

[INFO] Copying xmlenc-0.52.jar to 

[INFO] Copyin

Re: Review Request 60544: OOZIE-2854 Oozie should handle transient database problems

[András Piros]

> On July 11, 2017, 12:31 p.m., Peter Cseh wrote:
> > core/src/main/java/org/apache/oozie/util/db/OperationRetryHandler.java
> > Lines 41 (patched)
> > 
> >
> > Based on the doc 0 should be allowed, shouldn't they?

I've modified the functionality: `maxRetryCount` and `maxWaitTime` should not 
be negative. If either is `0`, `OperationRetryHandler` does only one try. See 
`TestOperationRetryHandler#testWhenConfiguredNotToRetryThenTriesOnce()` for 
details.


- András


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/60544/#review180182
---


On July 11, 2017, 12:01 p.m., András Piros wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/60544/
> ---
> 
> (Updated July 11, 2017, 12:01 p.m.)
> 
> 
> Review request for oozie, Attila Sasvari, Peter Cseh, Peter Bacsko, and 
> Robert Kanter.
> 
> 
> Repository: oozie-git
> 
> 
> Description
> ---
> 
> https://issues.apache.org/jira/browse/OOZIE-2854
> 
> 
> Diffs
> -
> 
>   core/pom.xml acddf349a89cf09a7fc4f384ebcaec56dfd0ab48 
>   
> core/src/main/java/org/apache/oozie/executor/jpa/JsonBeanPersisterExecutor.java
>  PRE-CREATION 
>   core/src/main/java/org/apache/oozie/executor/jpa/QueryExecutor.java 
> 8d94c23e40d1281864db40e141b200ca207a6324 
>   core/src/main/java/org/apache/oozie/service/JPAService.java 
> 028381d3b72bcc3b8c2cd27cacb3e0ac6d48d146 
>   core/src/main/java/org/apache/oozie/sla/SLASummaryBean.java 
> cfe1522a4b1f89085eb29e7f1281c2abd631bdc2 
>   core/src/main/java/org/apache/oozie/store/WorkflowStore.java 
> c565e74893b863caef6c93015cfe38fe520d04ec 
>   core/src/main/java/org/apache/oozie/util/db/BasicDataSourceWrapper.java 
> PRE-CREATION 
>   core/src/main/java/org/apache/oozie/util/db/DatabaseRetryPredicate.java 
> PRE-CREATION 
>   core/src/main/java/org/apache/oozie/util/db/FailingConnectionWrapper.java 
> PRE-CREATION 
>   core/src/main/java/org/apache/oozie/util/db/FailingHSQLDBDriverWrapper.java 
> PRE-CREATION 
>   core/src/main/java/org/apache/oozie/util/db/FailingMySQLDriverWrapper.java 
> PRE-CREATION 
>   core/src/main/java/org/apache/oozie/util/db/OperationRetryHandler.java 
> PRE-CREATION 
>   
> core/src/main/java/org/apache/oozie/util/db/PersistenceExceptionSubclassFilterRetryPredicate.java
>  PRE-CREATION 
>   core/src/main/java/org/apache/oozie/util/db/RetryAttemptState.java 
> PRE-CREATION 
>   core/src/main/java/org/apache/oozie/util/db/RuntimeExceptionInjector.java 
> PRE-CREATION 
>   core/src/main/resources/META-INF/persistence.xml 
> bad9278597fcd4f93b4cc482afae8af14beaa922 
>   core/src/main/resources/oozie-default.xml 
> 5629a897b4a1834554a1a5f7a21984ea857ed133 
>   core/src/main/resources/oozie-log4j.properties 
> c065f3cd4c5a3df1308b69d7c16e8fcfa8796efc 
>   core/src/test/java/org/apache/oozie/command/SkipCommitFaultInjection.java  
>   core/src/test/java/org/apache/oozie/test/XTestCase.java 
> 2816e5d80efd89151606b364376120a76b5c5a4a 
>   
> core/src/test/java/org/apache/oozie/util/db/TestOozieDmlStatementPredicate.java
>  PRE-CREATION 
>   core/src/test/java/org/apache/oozie/util/db/TestOperationRetryHandler.java 
> PRE-CREATION 
>   
> core/src/test/java/org/apache/oozie/util/db/TestPersistenceExceptionSubclassFilterRetryPredicate.java
>  PRE-CREATION 
>   core/src/test/java/org/apache/oozie/util/db/TestRetryAttemptState.java 
> PRE-CREATION 
>   docs/src/site/twiki/AG_Install.twiki 
> 2125442c273661d54f3566cbe41ead822c1f7baa 
>   minitest/pom.xml 9515284bb5f32c279a93161c10e6571680e4f9fc 
>   
> minitest/src/test/java/org/apache/oozie/test/TestParallelJPAOperationRetries.java
>  PRE-CREATION 
>   minitest/src/test/java/org/apache/oozie/test/TestWorkflowRetries.java 
> PRE-CREATION 
>   minitest/src/test/java/org/apache/oozie/test/WorkflowTest.java 
> 2845f0af6efb9ef75fdbfcb326115c62e6fb3bdd 
>   minitest/src/test/resources/hsqldb-oozie-site.xml 
> fa5fe9c3185e973e8247d7bf10b126119d9c02c9 
>   minitest/src/test/resources/oozie-log4j.properties 
> c142d725140930bfa89cd2b163d0768a4c3a750a 
>   minitest/src/test/resources/parallel-fs-and-shell.xml PRE-CREATION 
>   minitest/src/test/resources/wf-test.xml 
> 20c4946862039a65c76ed7f49991345e90a694de 
>   pom.xml 16c5137d44d7db891da46f80adb51c85e4c1b214 
> 
> 
> Diff: https://reviews.apache.org/r/60544/diff/8/
> 
> 
> Testing
> ---
> 
> Tests covered in code:
> 
> Unit tests
> ==
> 
> * testing the retry handler, the retry predicate filter, and parallel calls 
> to JPA `EntityManager` (mostly Oozie database reads and writes) when 
> injecting failures
> 
> Integration tests
> =
> 
> * using the `MiniOozieTestCase` framework
> * fixing it s

[jira] [Commented] (OOZIE-2601) Ability to use local paths for the sharelib

[Hadoop QA (JIRA)]

[ 
https://issues.apache.org/jira/browse/OOZIE-2601?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16082301#comment-16082301
 ] 

Hadoop QA commented on OOZIE-2601:
--

Testing JIRA OOZIE-2601

Cleaning local git workspace



{color:green}+1 PATCH_APPLIES{color}
{color:green}+1 CLEAN{color}
{color:green}+1 RAW_PATCH_ANALYSIS{color}
.{color:green}+1{color} the patch does not introduce any @author tags
.{color:green}+1{color} the patch does not introduce any tabs
.{color:green}+1{color} the patch does not introduce any trailing spaces
.{color:green}+1{color} the patch does not introduce any line longer than 
132
.{color:green}+1{color} the patch does adds/modifies 1 testcase(s)
{color:green}+1 RAT{color}
.{color:green}+1{color} the patch does not seem to introduce new RAT 
warnings
{color:green}+1 JAVADOC{color}
.{color:green}+1{color} the patch does not seem to introduce new Javadoc 
warnings
.{color:red}WARNING{color}: the current HEAD has 1 Javadoc warning(s)
{color:green}+1 COMPILE{color}
.{color:green}+1{color} HEAD compiles
.{color:green}+1{color} patch compiles
.{color:green}+1{color} the patch does not seem to introduce new javac 
warnings
{color:green}+1{color} There are no new bugs found in total.
{color:green}+1 BACKWARDS_COMPATIBILITY{color}
.{color:green}+1{color} the patch does not change any JPA 
Entity/Colum/Basic/Lob/Transient annotations
.{color:green}+1{color} the patch does not modify JPA files
{color:green}+1 TESTS{color}
.Tests run: 1967
.Tests rerun: 55
.Tests failed at first run: 
org.apache.oozie.service.TestPartitionDependencyManagerEhcache,org.apache.oozie.service.TestPartitionDependencyManagerService,org.apache.oozie.action.hadoop.TestJavaActionExecutor,org.apache.oozie.action.hadoop.TestHdfsOperations,
{color:green}+1 DISTRO{color}
.{color:green}+1{color} distro tarball builds with the patch 


{color:green}*+1 Overall result, good!, no -1s*{color}

{color:red}. There is at least one warning, please check{color}

The full output of the test-patch run is available at

. https://builds.apache.org/job/oozie-trunk-precommit-build/3965/

> Ability to use local paths for the sharelib
> ---
>
> Key: OOZIE-2601
> URL: https://issues.apache.org/jira/browse/OOZIE-2601
> Project: Oozie
>  Issue Type: New Feature
>Affects Versions: oya
>Reporter: Robert Kanter
>Assignee: Attila Sasvari
> Fix For: oya
>
> Attachments: OOZIE-2601-01.patch, OOZIE-2601-03.patch
>
>
> With OOZIE-2590, as part of OOZIE-1770 Oozie on Yarn work, Oozie now has full 
> control over the classpath given to the Launcher AM.  In a cluster where all 
> nodes have everything installed locally (in the same paths), it should be 
> possible to have the Launcher AM reference the local jars instead of having 
> to localize them from HDFS.
> For example, if you have Hive installed on all nodes at {{/usr/lib/hive/}} 
> and all Hive jars under {{/usr/lib/hive/lib/}}, we could have the Launcher AM 
> add {{/usr/lib/hive/lib}} to its classpath.  This saves on the overhead of 
> localizing the same jars from the hive sharelib in HDFS.  
> I think the best way to implement this is to augment the [Sharelib Mapping 
> File|https://oozie.apache.org/docs/4.2.0/AG_Install.html#Oozie_Share_Lib] 
> feature to accept {{file:///}} paths.
> If we had this also work with the "oozie" sharelib and the Oozie jars in the 
> individual sharelibs (e.g. have the Mapping file take comma-separated 
> dirs/jars), then in a cluster with everything installed on all of the nodes, 
> you wouldn't need to bother with the sharelib at all!



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

Jenkins build is back to normal : oozie-trunk-precommit-build #3965

[Apache Jenkins Server]

See 

Re: Review Request 60544: OOZIE-2854 Oozie should handle transient database problems

[András Piros]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/60544/
---

(Updated July 11, 2017, 2:41 p.m.)


Review request for oozie, Attila Sasvari, Peter Cseh, Peter Bacsko, and Robert 
Kanter.


Changes
---

Addressing review comments.


Repository: oozie-git


Description
---

https://issues.apache.org/jira/browse/OOZIE-2854


Diffs (updated)
-

  core/pom.xml acddf349a89cf09a7fc4f384ebcaec56dfd0ab48 
  
core/src/main/java/org/apache/oozie/executor/jpa/JsonBeanPersisterExecutor.java 
PRE-CREATION 
  core/src/main/java/org/apache/oozie/executor/jpa/QueryExecutor.java 
8d94c23e40d1281864db40e141b200ca207a6324 
  core/src/main/java/org/apache/oozie/service/JPAService.java 
028381d3b72bcc3b8c2cd27cacb3e0ac6d48d146 
  core/src/main/java/org/apache/oozie/sla/SLASummaryBean.java 
cfe1522a4b1f89085eb29e7f1281c2abd631bdc2 
  core/src/main/java/org/apache/oozie/store/WorkflowStore.java 
c565e74893b863caef6c93015cfe38fe520d04ec 
  core/src/main/java/org/apache/oozie/util/db/BasicDataSourceWrapper.java 
PRE-CREATION 
  core/src/main/java/org/apache/oozie/util/db/DatabaseRetryPredicate.java 
PRE-CREATION 
  core/src/main/java/org/apache/oozie/util/db/FailingConnectionWrapper.java 
PRE-CREATION 
  core/src/main/java/org/apache/oozie/util/db/FailingHSQLDBDriverWrapper.java 
PRE-CREATION 
  core/src/main/java/org/apache/oozie/util/db/FailingMySQLDriverWrapper.java 
PRE-CREATION 
  core/src/main/java/org/apache/oozie/util/db/OperationRetryHandler.java 
PRE-CREATION 
  
core/src/main/java/org/apache/oozie/util/db/PersistenceExceptionSubclassFilterRetryPredicate.java
 PRE-CREATION 
  core/src/main/java/org/apache/oozie/util/db/RetryAttemptState.java 
PRE-CREATION 
  core/src/main/java/org/apache/oozie/util/db/RuntimeExceptionInjector.java 
PRE-CREATION 
  core/src/main/resources/META-INF/persistence.xml 
bad9278597fcd4f93b4cc482afae8af14beaa922 
  core/src/main/resources/oozie-default.xml 
5629a897b4a1834554a1a5f7a21984ea857ed133 
  core/src/main/resources/oozie-log4j.properties 
c065f3cd4c5a3df1308b69d7c16e8fcfa8796efc 
  core/src/test/java/org/apache/oozie/command/SkipCommitFaultInjection.java  
  core/src/test/java/org/apache/oozie/test/XTestCase.java 
2816e5d80efd89151606b364376120a76b5c5a4a 
  
core/src/test/java/org/apache/oozie/util/db/TestOozieDmlStatementPredicate.java 
PRE-CREATION 
  core/src/test/java/org/apache/oozie/util/db/TestOperationRetryHandler.java 
PRE-CREATION 
  
core/src/test/java/org/apache/oozie/util/db/TestPersistenceExceptionSubclassFilterRetryPredicate.java
 PRE-CREATION 
  core/src/test/java/org/apache/oozie/util/db/TestRetryAttemptState.java 
PRE-CREATION 
  docs/src/site/twiki/AG_Install.twiki 2125442c273661d54f3566cbe41ead822c1f7baa 
  minitest/pom.xml 9515284bb5f32c279a93161c10e6571680e4f9fc 
  
minitest/src/test/java/org/apache/oozie/test/TestParallelJPAOperationRetries.java
 PRE-CREATION 
  minitest/src/test/java/org/apache/oozie/test/TestWorkflowRetries.java 
PRE-CREATION 
  minitest/src/test/java/org/apache/oozie/test/WorkflowTest.java 
2845f0af6efb9ef75fdbfcb326115c62e6fb3bdd 
  minitest/src/test/resources/hsqldb-oozie-site.xml 
fa5fe9c3185e973e8247d7bf10b126119d9c02c9 
  minitest/src/test/resources/oozie-log4j.properties 
c142d725140930bfa89cd2b163d0768a4c3a750a 
  minitest/src/test/resources/parallel-fs-and-shell.xml PRE-CREATION 
  minitest/src/test/resources/wf-test.xml 
20c4946862039a65c76ed7f49991345e90a694de 
  pom.xml 16c5137d44d7db891da46f80adb51c85e4c1b214 


Diff: https://reviews.apache.org/r/60544/diff/9/

Changes: https://reviews.apache.org/r/60544/diff/8-9/


Testing
---

Tests covered in code:

Unit tests
==

* testing the retry handler, the retry predicate filter, and parallel calls to 
JPA `EntityManager` (mostly Oozie database reads and writes) when injecting 
failures

Integration tests
=

* using the `MiniOozieTestCase` framework
* fixing it so that also asynchronous workflow applications (the ones that use 
`CallableQueueService`) can be run
* following workflow scenarios:
* a very simple one consisting only of a `` and an `` node
* a more sophisticated one consisting of multiple synchronous `` nodes and 
a `` node
* the ultimate one consisting of a `` node, and two branches of an 
`` and an asynchronous `` nodes

Test cases run:
```
mvn clean test 
-Dtest=TestOperationRetryHandler,TestPersistenceExceptionSubclassFilterRetryPredicate,TestParallelJPAOperationRetries,TestWorkflow,TestWorkflowRetries,TestJPAService,TestRetryAttemptState
```

Functional and stress tests performed on a 4-node MySQL cluster. MySQL daemon 
has been stopped / killed / restarted several times. Also firewall rules have 
been modified temporarily to simulate network outages.


Thanks,

András Piros

Build failed in Jenkins: oozie-trunk-precommit-build #3966

[Apache Jenkins Server]

See 


--
[...truncated 355.29 KB...]
[INFO] Copying ivy-2.4.0.jar to 

[INFO] Copying jackson-core-asl-1.9.13.jar to 

[INFO] Copying commons-jexl-2.1.1.jar to 

[INFO] Copying oozie-sharelib-hcatalog-5.0.0-SNAPSHOT.jar to 

[INFO] Copying hadoop-mapreduce-client-jobclient-2.6.0.jar to 

[INFO] Copying gson-2.7.jar to 

[INFO] Copying oozie-core-5.0.0-SNAPSHOT.jar to 

[INFO] Copying commons-pool-1.5.4.jar to 

[INFO] Copying htrace-core-3.0.4.jar to 

[INFO] Copying jaxb-impl-2.2.3-1.jar to 

[INFO] Copying jets3t-0.9.0.jar to 

[INFO] Copying datanucleus-rdbms-3.2.9.jar to 

[INFO] Copying hive-common-1.2.0.jar to 

[INFO] Copying snappy-java-1.0.4.1.jar to 

[INFO] Copying javax.annotation-api-1.2.jar to 

[INFO] Copying commons-compiler-2.7.6.jar to 

[INFO] Copying ant-1.9.1.jar to 

[INFO] Copying hadoop-annotations-2.6.0.jar to 

[INFO] Copying janino-2.7.6.jar to 

[INFO] Copying activemq-client-5.13.3.jar to 

[INFO] Copying guice-3.0.jar to 

[INFO] Copying httpclient-4.3.6.jar to 

[INFO] Copying xz-1.0.jar to 

[INFO] Copying oozie-client-5.0.0-SNAPSHOT.jar to 

[INFO] Copying hadoop-mapreduce-client-common-2.6.0.jar to 

[INFO] Copying leveldbjni-all-1.8.jar to 

[INFO] Copying jetty-util-6.1.26.jar to 

[INFO] Copying velocity-1.5.jar to 

[INFO] Copying jackson-core-2.4.2.jar to 

[INFO] Copying apache-log4j-extras-1.2.17.jar to 

[INFO] Copying netty-3.7.0.Final.jar to 

[INFO] Copying xm

[jira] [Commented] (OOZIE-2978) Remove code that handles Pig versions before 0.8

[Hadoop QA (JIRA)]

[ 
https://issues.apache.org/jira/browse/OOZIE-2978?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16082317#comment-16082317
 ] 

Hadoop QA commented on OOZIE-2978:
--

Testing JIRA OOZIE-2978

Cleaning local git workspace



{color:green}+1 PATCH_APPLIES{color}
{color:green}+1 CLEAN{color}
{color:red}-1 RAW_PATCH_ANALYSIS{color}
.{color:green}+1{color} the patch does not introduce any @author tags
.{color:green}+1{color} the patch does not introduce any tabs
.{color:green}+1{color} the patch does not introduce any trailing spaces
.{color:green}+1{color} the patch does not introduce any line longer than 
132
.{color:red}-1{color} the patch does not add/modify any testcase
{color:green}+1 RAT{color}
.{color:green}+1{color} the patch does not seem to introduce new RAT 
warnings
{color:green}+1 JAVADOC{color}
.{color:green}+1{color} the patch does not seem to introduce new Javadoc 
warnings
.{color:red}WARNING{color}: the current HEAD has 1 Javadoc warning(s)
{color:green}+1 COMPILE{color}
.{color:green}+1{color} HEAD compiles
.{color:green}+1{color} patch compiles
.{color:green}+1{color} the patch does not seem to introduce new javac 
warnings
{color:green}+1{color} There are no new bugs found in total.
{color:green}+1 BACKWARDS_COMPATIBILITY{color}
.{color:green}+1{color} the patch does not change any JPA 
Entity/Colum/Basic/Lob/Transient annotations
.{color:green}+1{color} the patch does not modify JPA files
{color:green}+1 TESTS{color}
.Tests run: 1965
.Tests rerun: 79
.Tests failed at first run: 
org.apache.oozie.action.hadoop.TestJavaActionExecutor,org.apache.oozie.jms.TestJMSJobEventListener,org.apache.oozie.action.hadoop.TestLauncherAM,
{color:green}+1 DISTRO{color}
.{color:green}+1{color} distro tarball builds with the patch 


{color:red}*-1 Overall result, please check the reported -1(s)*{color}

{color:red}. There is at least one warning, please check{color}

The full output of the test-patch run is available at

. https://builds.apache.org/job/oozie-trunk-precommit-build/3966/

> Remove code that handles Pig versions before 0.8 
> -
>
> Key: OOZIE-2978
> URL: https://issues.apache.org/jira/browse/OOZIE-2978
> Project: Oozie
>  Issue Type: Bug
>Reporter: Attila Sasvari
>Assignee: Attila Sasvari
> Attachments: OOZIE-2978-01.patch, OOZIE-2978-02.patch, 
> OOZIE-2978-03.patch, OOZIE-2978-04.patch, OOZIE-2978-05.patch
>
>
> Oozie submits Pig jobs according to given Pig version. For example, Pig 
> versions before 0.8 are treated in the following way:
>  
> [Main.main(args)|https://github.com/apache/oozie/blob/678f1b2b2cc9a2456c7e3d9f693d2637f1f03257/sharelib/pig/src/main/java/org/apache/oozie/action/hadoop/PigMain.java#L382]
> However, we dropped support for *Hadoop 1*, *Hadoop 0.23* support via 
> {{OOZIE-2316}}. I wonder whether submitting jobs using Pig version before 0.8 
> will work at all:
> - 
> https://pig.apache.org/releases.html#13+May%2C+2010%3A+release+0.7.0+available
>  says "This release works with Hadoop 0.20". Pig *0.10.1* is the first pig 
> release that  works with Hadoop 2.
> - Looking at [PIG-2125|https://issues.apache.org/jira/browse/PIG-2125], I 
> believe the answer is NO.  
> Therefore we can safely delete corresponding parts from the codebase. There 
> is also another class {{PigMainWithOldAPI}} that is only used by tests in 
> {{TestPigMainWithOldAPI}}. So we can remove these too.
>  



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Updated] (OOZIE-2854) Oozie should handle transient database problems

[Andras Piros (JIRA)]

 [ 
https://issues.apache.org/jira/browse/OOZIE-2854?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Andras Piros updated OOZIE-2854:

Attachment: OOZIE-2854.013.patch

Addressing review comments.

> Oozie should handle transient database problems
> ---
>
> Key: OOZIE-2854
> URL: https://issues.apache.org/jira/browse/OOZIE-2854
> Project: Oozie
>  Issue Type: Improvement
>  Components: core
>Reporter: Peter Bacsko
>Assignee: Andras Piros
> Attachments: OOZIE-2854-001.patch, OOZIE-2854-002.patch, 
> OOZIE-2854-003.patch, OOZIE-2854-004.patch, OOZIE-2854-005.patch, 
> OOZIE-2854.006.patch, OOZIE-2854.007.patch, OOZIE-2854.008.patch, 
> OOZIE-2854.009.patch, OOZIE-2854.010.patch, OOZIE-2854.011.patch, 
> OOZIE-2854.012.patch, OOZIE-2854.013.patch, OOZIE-2854-POC-001.patch
>
>
> There can be problems when Oozie cannot update the database properly. 
> Recently, we have experienced erratic behavior with two setups:
> * MySQL with the Galera cluster manager. Galera uses cluster-wide optimistic 
> locking which might cause a transaction to rollback if there are two or more 
> parallel transaction running and one of them cannot complete because of a 
> conflict.
> * MySQL with Percona XtraDB Cluster. If one of the MySQL instances is killed, 
> Oozie might get "Communications link failure" exception during the failover.
> The problem is that failed DB transactions later might cause a workflow 
> (which are started/re-started by RecoveryService) to get stuck. It's not 
> clear to us how this happens but it has to do with the fact that certain DB 
> updates are not executed.
> The solution is to use some sort of retry logic with exponential backoff if 
> the DB update fails. We could start with a 100ms wait time which is doubled 
> at every retry. The operation can be considered a failure if it still fails 
> after 10 attempts. These values could be configurable. We should discuss 
> initial values in the scope of this JIRA.
> Note that this solution is to handle *transient* failures. If the DB is down 
> for a longer period of time, we have to accept that the internal state of 
> Oozie is corrupted.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (OOZIE-2854) Oozie should handle transient database problems

[Hadoop QA (JIRA)]

[ 
https://issues.apache.org/jira/browse/OOZIE-2854?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16082265#comment-16082265
 ] 

Hadoop QA commented on OOZIE-2854:
--

Testing JIRA OOZIE-2854

Cleaning local git workspace



{color:green}+1 PATCH_APPLIES{color}
{color:green}+1 CLEAN{color}
{color:red}-1 RAW_PATCH_ANALYSIS{color}
.{color:green}+1{color} the patch does not introduce any @author tags
.{color:green}+1{color} the patch does not introduce any tabs
.{color:green}+1{color} the patch does not introduce any trailing spaces
.{color:red}-1{color} the patch contains 1 line(s) longer than 132 
characters
.{color:green}+1{color} the patch does adds/modifies 12 testcase(s)
{color:green}+1 RAT{color}
.{color:green}+1{color} the patch does not seem to introduce new RAT 
warnings
{color:green}+1 JAVADOC{color}
.{color:green}+1{color} the patch does not seem to introduce new Javadoc 
warnings
.{color:red}WARNING{color}: the current HEAD has 1 Javadoc warning(s)
{color:green}+1 COMPILE{color}
.{color:green}+1{color} HEAD compiles
.{color:green}+1{color} patch compiles
.{color:green}+1{color} the patch does not seem to introduce new javac 
warnings
{color:green}+1{color} There are no new bugs found in total.
{color:red}-1 BACKWARDS_COMPATIBILITY{color}
.{color:green}+1{color} the patch does not change any JPA 
Entity/Colum/Basic/Lob/Transient annotations
.{color:red}-1{color} the patch modifies 1 JPA file(s), persistence.xml or 
*-orm.xml
{color:green}+1 TESTS{color}
.Tests run: 1558
.Tests rerun: 3
.Tests failed at first run: 
org.apache.oozie.action.hadoop.TestHdfsOperations,
{color:green}+1 DISTRO{color}
.{color:green}+1{color} distro tarball builds with the patch 


{color:red}*-1 Overall result, please check the reported -1(s)*{color}

{color:red}. There is at least one warning, please check{color}

The full output of the test-patch run is available at

. https://builds.apache.org/job/oozie-trunk-precommit-build/3964/

> Oozie should handle transient database problems
> ---
>
> Key: OOZIE-2854
> URL: https://issues.apache.org/jira/browse/OOZIE-2854
> Project: Oozie
>  Issue Type: Improvement
>  Components: core
>Reporter: Peter Bacsko
>Assignee: Andras Piros
> Attachments: OOZIE-2854-001.patch, OOZIE-2854-002.patch, 
> OOZIE-2854-003.patch, OOZIE-2854-004.patch, OOZIE-2854-005.patch, 
> OOZIE-2854.006.patch, OOZIE-2854.007.patch, OOZIE-2854.008.patch, 
> OOZIE-2854.009.patch, OOZIE-2854.010.patch, OOZIE-2854.011.patch, 
> OOZIE-2854.012.patch, OOZIE-2854-POC-001.patch
>
>
> There can be problems when Oozie cannot update the database properly. 
> Recently, we have experienced erratic behavior with two setups:
> * MySQL with the Galera cluster manager. Galera uses cluster-wide optimistic 
> locking which might cause a transaction to rollback if there are two or more 
> parallel transaction running and one of them cannot complete because of a 
> conflict.
> * MySQL with Percona XtraDB Cluster. If one of the MySQL instances is killed, 
> Oozie might get "Communications link failure" exception during the failover.
> The problem is that failed DB transactions later might cause a workflow 
> (which are started/re-started by RecoveryService) to get stuck. It's not 
> clear to us how this happens but it has to do with the fact that certain DB 
> updates are not executed.
> The solution is to use some sort of retry logic with exponential backoff if 
> the DB update fails. We could start with a 100ms wait time which is doubled 
> at every retry. The operation can be considered a failure if it still fails 
> after 10 attempts. These values could be configurable. We should discuss 
> initial values in the scope of this JIRA.
> Note that this solution is to handle *transient* failures. If the DB is down 
> for a longer period of time, we have to accept that the internal state of 
> Oozie is corrupted.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (OOZIE-2601) Ability to use local paths for the sharelib

[Attila Sasvari (JIRA)]

[ 
https://issues.apache.org/jira/browse/OOZIE-2601?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16082472#comment-16082472
 ] 

Attila Sasvari commented on OOZIE-2601:
---

Executed successfully example workflows:
custom-main, demo, disctp, java-main, map-reduce, pig, streaming, subwf

> Ability to use local paths for the sharelib
> ---
>
> Key: OOZIE-2601
> URL: https://issues.apache.org/jira/browse/OOZIE-2601
> Project: Oozie
>  Issue Type: New Feature
>Affects Versions: oya
>Reporter: Robert Kanter
>Assignee: Attila Sasvari
> Fix For: oya
>
> Attachments: OOZIE-2601-01.patch, OOZIE-2601-03.patch, 
> OOZIE-2601-05.patch
>
>
> With OOZIE-2590, as part of OOZIE-1770 Oozie on Yarn work, Oozie now has full 
> control over the classpath given to the Launcher AM.  In a cluster where all 
> nodes have everything installed locally (in the same paths), it should be 
> possible to have the Launcher AM reference the local jars instead of having 
> to localize them from HDFS.
> For example, if you have Hive installed on all nodes at {{/usr/lib/hive/}} 
> and all Hive jars under {{/usr/lib/hive/lib/}}, we could have the Launcher AM 
> add {{/usr/lib/hive/lib}} to its classpath.  This saves on the overhead of 
> localizing the same jars from the hive sharelib in HDFS.  
> I think the best way to implement this is to augment the [Sharelib Mapping 
> File|https://oozie.apache.org/docs/4.2.0/AG_Install.html#Oozie_Share_Lib] 
> feature to accept {{file:///}} paths.
> If we had this also work with the "oozie" sharelib and the Oozie jars in the 
> individual sharelibs (e.g. have the Mapping file take comma-separated 
> dirs/jars), then in a cluster with everything installed on all of the nodes, 
> you wouldn't need to bother with the sharelib at all!



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (OOZIE-2004) Improve Oozie version info output

[Artem Ervits (JIRA)]

[ 
https://issues.apache.org/jira/browse/OOZIE-2004?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16082457#comment-16082457
 ] 

Artem Ervits commented on OOZIE-2004:
-

[~rkanter] [~gezapeti] can I get a yay or nay on this, I have some bandwidth 
and want to close some of my issues in "patch available" stage.

> Improve Oozie version info output
> -
>
> Key: OOZIE-2004
> URL: https://issues.apache.org/jira/browse/OOZIE-2004
> Project: Oozie
>  Issue Type: Improvement
>  Components: build
>Affects Versions: 4.0.1
>Reporter: Chunjun Xiao
>Assignee: Artem Ervits
>Priority: Minor
> Fix For: 5.0.0
>
> Attachments: HBASE-2004-2.patch, OOZIE-2004-0.patch, 
> OOZIE-2004-1.patch
>
>
> Current oozie version info output is quite simple:
> {code}
> [root@svm5 ~] # oozie version
> Oozie client build version: 4.0.1
> [root@svm5 ~] #
> {code}
> Can we improve it to follow the convention of  Hadoop/HBase, as:
> {code}
> [root@svm5 ~] # oozie version
> Oozie 4.0.1
> Source code repository: ssh://git@XXX/oozie.git -r 
> 3171cb656a4e4c1b2c1e9c6a47abed7df5bbcc8d
> Compiled by root on Sun Jul  6 04:38:08 PDT 2014
> From source with checksum XXX
> [root@svm5 ~] #
> {code}



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (OOZIE-2877) Oozie Git Action

[Andras Piros (JIRA)]

[ 
https://issues.apache.org/jira/browse/OOZIE-2877?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16082487#comment-16082487
 ] 

Andras Piros commented on OOZIE-2877:
-

Thanks [~clayb], waiting on your new ReviewBoard request!

> Oozie Git Action
> 
>
> Key: OOZIE-2877
> URL: https://issues.apache.org/jira/browse/OOZIE-2877
> Project: Oozie
>  Issue Type: Sub-task
>  Components: action
>Reporter: Clay B.
>  Labels: action
> Fix For: trunk
>
> Attachments: 0001-OOZIE-2877-Oozie-Git-Action.patch, 
> 0002-OOZIE-2877-Oozie-Git-Action.patch, 
> 0003-OOZIE-2877-Oozie-Git-Action.patch, 0004-OOZIE-2877-Oozie-Git-Action.patch
>
>
> To aide in deploying ASCII artifacts to clusters, let's provide a tie-in for 
> a source-code management system. Git would be my preferred choice.
> Ideally, this could handle a user's key material e.g. for an ssh key to pull 
> down from a secured repository. This would free users from handling their own 
> key staging and clean-up on YARN nodes and only require them to store the key 
> secured in HDFS.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Updated] (OOZIE-2601) Ability to use local paths for the sharelib

[Attila Sasvari (JIRA)]

 [ 
https://issues.apache.org/jira/browse/OOZIE-2601?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Attila Sasvari updated OOZIE-2601:
--
Attachment: OOZIE-2601-05.patch

> Ability to use local paths for the sharelib
> ---
>
> Key: OOZIE-2601
> URL: https://issues.apache.org/jira/browse/OOZIE-2601
> Project: Oozie
>  Issue Type: New Feature
>Affects Versions: oya
>Reporter: Robert Kanter
>Assignee: Attila Sasvari
> Fix For: oya
>
> Attachments: OOZIE-2601-01.patch, OOZIE-2601-03.patch, 
> OOZIE-2601-05.patch
>
>
> With OOZIE-2590, as part of OOZIE-1770 Oozie on Yarn work, Oozie now has full 
> control over the classpath given to the Launcher AM.  In a cluster where all 
> nodes have everything installed locally (in the same paths), it should be 
> possible to have the Launcher AM reference the local jars instead of having 
> to localize them from HDFS.
> For example, if you have Hive installed on all nodes at {{/usr/lib/hive/}} 
> and all Hive jars under {{/usr/lib/hive/lib/}}, we could have the Launcher AM 
> add {{/usr/lib/hive/lib}} to its classpath.  This saves on the overhead of 
> localizing the same jars from the hive sharelib in HDFS.  
> I think the best way to implement this is to augment the [Sharelib Mapping 
> File|https://oozie.apache.org/docs/4.2.0/AG_Install.html#Oozie_Share_Lib] 
> feature to accept {{file:///}} paths.
> If we had this also work with the "oozie" sharelib and the Oozie jars in the 
> individual sharelibs (e.g. have the Mapping file take comma-separated 
> dirs/jars), then in a cluster with everything installed on all of the nodes, 
> you wouldn't need to bother with the sharelib at all!



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

Build failed in Jenkins: oozie-trunk-precommit-build #3967

[Apache Jenkins Server]

See 


--
[...truncated 331.72 KB...]
[INFO] Copying ivy-2.4.0.jar to 

[INFO] Copying jackson-core-asl-1.9.13.jar to 

[INFO] Copying commons-jexl-2.1.1.jar to 

[INFO] Copying oozie-sharelib-hcatalog-5.0.0-SNAPSHOT.jar to 

[INFO] Copying hadoop-mapreduce-client-jobclient-2.6.0.jar to 

[INFO] Copying gson-2.7.jar to 

[INFO] Copying oozie-core-5.0.0-SNAPSHOT.jar to 

[INFO] Copying commons-pool-1.5.4.jar to 

[INFO] Copying htrace-core-3.0.4.jar to 

[INFO] Copying jaxb-impl-2.2.3-1.jar to 

[INFO] Copying jets3t-0.9.0.jar to 

[INFO] Copying datanucleus-rdbms-3.2.9.jar to 

[INFO] Copying hive-common-1.2.0.jar to 

[INFO] Copying snappy-java-1.0.4.1.jar to 

[INFO] Copying javax.annotation-api-1.2.jar to 

[INFO] Copying commons-compiler-2.7.6.jar to 

[INFO] Copying ant-1.9.1.jar to 

[INFO] Copying hadoop-annotations-2.6.0.jar to 

[INFO] Copying janino-2.7.6.jar to 

[INFO] Copying activemq-client-5.13.3.jar to 

[INFO] Copying guice-3.0.jar to 

[INFO] Copying httpclient-4.3.6.jar to 

[INFO] Copying xz-1.0.jar to 

[INFO] Copying oozie-client-5.0.0-SNAPSHOT.jar to 

[INFO] Copying hadoop-mapreduce-client-common-2.6.0.jar to 

[INFO] Copying leveldbjni-all-1.8.jar to 

[INFO] Copying jetty-util-6.1.26.jar to 

[INFO] Copying velocity-1.5.jar to 

[INFO] Copying jackson-core-2.4.2.jar to 

[INFO] Copying apache-log4j-extras-1.2.17.jar to 

[INFO] Copying netty-3.7.0.Final.jar to 

[INFO] Copying xm

[jira] [Commented] (OOZIE-2854) Oozie should handle transient database problems

[Hadoop QA (JIRA)]

[ 
https://issues.apache.org/jira/browse/OOZIE-2854?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16082504#comment-16082504
 ] 

Hadoop QA commented on OOZIE-2854:
--

Testing JIRA OOZIE-2854

Cleaning local git workspace



{color:green}+1 PATCH_APPLIES{color}
{color:green}+1 CLEAN{color}
{color:red}-1 RAW_PATCH_ANALYSIS{color}
.{color:green}+1{color} the patch does not introduce any @author tags
.{color:green}+1{color} the patch does not introduce any tabs
.{color:green}+1{color} the patch does not introduce any trailing spaces
.{color:red}-1{color} the patch contains 1 line(s) longer than 132 
characters
.{color:green}+1{color} the patch does adds/modifies 12 testcase(s)
{color:green}+1 RAT{color}
.{color:green}+1{color} the patch does not seem to introduce new RAT 
warnings
{color:green}+1 JAVADOC{color}
.{color:green}+1{color} the patch does not seem to introduce new Javadoc 
warnings
.{color:red}WARNING{color}: the current HEAD has 1 Javadoc warning(s)
{color:green}+1 COMPILE{color}
.{color:green}+1{color} HEAD compiles
.{color:green}+1{color} patch compiles
.{color:green}+1{color} the patch does not seem to introduce new javac 
warnings
{color:green}+1{color} There are no new bugs found in total.
{color:red}-1 BACKWARDS_COMPATIBILITY{color}
.{color:green}+1{color} the patch does not change any JPA 
Entity/Colum/Basic/Lob/Transient annotations
.{color:red}-1{color} the patch modifies 1 JPA file(s), persistence.xml or 
*-orm.xml
{color:green}+1 TESTS{color}
.Tests run: 1080
.Tests rerun: 22
.Tests failed at first run: org.apache.oozie.action.hadoop.TestLauncherAM,
{color:green}+1 DISTRO{color}
.{color:green}+1{color} distro tarball builds with the patch 


{color:red}*-1 Overall result, please check the reported -1(s)*{color}

{color:red}. There is at least one warning, please check{color}

The full output of the test-patch run is available at

. https://builds.apache.org/job/oozie-trunk-precommit-build/3967/

> Oozie should handle transient database problems
> ---
>
> Key: OOZIE-2854
> URL: https://issues.apache.org/jira/browse/OOZIE-2854
> Project: Oozie
>  Issue Type: Improvement
>  Components: core
>Reporter: Peter Bacsko
>Assignee: Andras Piros
> Attachments: OOZIE-2854-001.patch, OOZIE-2854-002.patch, 
> OOZIE-2854-003.patch, OOZIE-2854-004.patch, OOZIE-2854-005.patch, 
> OOZIE-2854.006.patch, OOZIE-2854.007.patch, OOZIE-2854.008.patch, 
> OOZIE-2854.009.patch, OOZIE-2854.010.patch, OOZIE-2854.011.patch, 
> OOZIE-2854.012.patch, OOZIE-2854.013.patch, OOZIE-2854-POC-001.patch
>
>
> There can be problems when Oozie cannot update the database properly. 
> Recently, we have experienced erratic behavior with two setups:
> * MySQL with the Galera cluster manager. Galera uses cluster-wide optimistic 
> locking which might cause a transaction to rollback if there are two or more 
> parallel transaction running and one of them cannot complete because of a 
> conflict.
> * MySQL with Percona XtraDB Cluster. If one of the MySQL instances is killed, 
> Oozie might get "Communications link failure" exception during the failover.
> The problem is that failed DB transactions later might cause a workflow 
> (which are started/re-started by RecoveryService) to get stuck. It's not 
> clear to us how this happens but it has to do with the fact that certain DB 
> updates are not executed.
> The solution is to use some sort of retry logic with exponential backoff if 
> the DB update fails. We could start with a 100ms wait time which is doubled 
> at every retry. The operation can be considered a failure if it still fails 
> after 10 attempts. These values could be configurable. We should discuss 
> initial values in the scope of this JIRA.
> Note that this solution is to handle *transient* failures. If the DB is down 
> for a longer period of time, we have to accept that the internal state of 
> Oozie is corrupted.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

Jenkins build is back to normal : oozie-trunk-precommit-build #3968

[Apache Jenkins Server]

See 

[jira] [Commented] (OOZIE-2601) Ability to use local paths for the sharelib

[Hadoop QA (JIRA)]

[ 
https://issues.apache.org/jira/browse/OOZIE-2601?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16082699#comment-16082699
 ] 

Hadoop QA commented on OOZIE-2601:
--

Testing JIRA OOZIE-2601

Cleaning local git workspace



{color:green}+1 PATCH_APPLIES{color}
{color:green}+1 CLEAN{color}
{color:green}+1 RAW_PATCH_ANALYSIS{color}
.{color:green}+1{color} the patch does not introduce any @author tags
.{color:green}+1{color} the patch does not introduce any tabs
.{color:green}+1{color} the patch does not introduce any trailing spaces
.{color:green}+1{color} the patch does not introduce any line longer than 
132
.{color:green}+1{color} the patch does adds/modifies 1 testcase(s)
{color:green}+1 RAT{color}
.{color:green}+1{color} the patch does not seem to introduce new RAT 
warnings
{color:green}+1 JAVADOC{color}
.{color:green}+1{color} the patch does not seem to introduce new Javadoc 
warnings
.{color:red}WARNING{color}: the current HEAD has 1 Javadoc warning(s)
{color:green}+1 COMPILE{color}
.{color:green}+1{color} HEAD compiles
.{color:green}+1{color} patch compiles
.{color:green}+1{color} the patch does not seem to introduce new javac 
warnings
{color:green}+1{color} There are no new bugs found in total.
{color:green}+1 BACKWARDS_COMPATIBILITY{color}
.{color:green}+1{color} the patch does not change any JPA 
Entity/Colum/Basic/Lob/Transient annotations
.{color:green}+1{color} the patch does not modify JPA files
{color:green}+1 TESTS{color}
.Tests run: 1967
.Tests rerun: 18
.Tests failed at first run: 
org.apache.oozie.action.hadoop.TestHdfsOperations,org.apache.oozie.jms.TestJMSJobEventListener,
{color:green}+1 DISTRO{color}
.{color:green}+1{color} distro tarball builds with the patch 


{color:green}*+1 Overall result, good!, no -1s*{color}

{color:red}. There is at least one warning, please check{color}

The full output of the test-patch run is available at

. https://builds.apache.org/job/oozie-trunk-precommit-build/3968/

> Ability to use local paths for the sharelib
> ---
>
> Key: OOZIE-2601
> URL: https://issues.apache.org/jira/browse/OOZIE-2601
> Project: Oozie
>  Issue Type: New Feature
>Affects Versions: oya
>Reporter: Robert Kanter
>Assignee: Attila Sasvari
> Fix For: oya
>
> Attachments: OOZIE-2601-01.patch, OOZIE-2601-03.patch, 
> OOZIE-2601-05.patch
>
>
> With OOZIE-2590, as part of OOZIE-1770 Oozie on Yarn work, Oozie now has full 
> control over the classpath given to the Launcher AM.  In a cluster where all 
> nodes have everything installed locally (in the same paths), it should be 
> possible to have the Launcher AM reference the local jars instead of having 
> to localize them from HDFS.
> For example, if you have Hive installed on all nodes at {{/usr/lib/hive/}} 
> and all Hive jars under {{/usr/lib/hive/lib/}}, we could have the Launcher AM 
> add {{/usr/lib/hive/lib}} to its classpath.  This saves on the overhead of 
> localizing the same jars from the hive sharelib in HDFS.  
> I think the best way to implement this is to augment the [Sharelib Mapping 
> File|https://oozie.apache.org/docs/4.2.0/AG_Install.html#Oozie_Share_Lib] 
> feature to accept {{file:///}} paths.
> If we had this also work with the "oozie" sharelib and the Oozie jars in the 
> individual sharelibs (e.g. have the Mapping file take comma-separated 
> dirs/jars), then in a cluster with everything installed on all of the nodes, 
> you wouldn't need to bother with the sharelib at all!



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] Subscription: Oozie Patch Available

[jira]

Issue Subscription
Filter: Oozie Patch Available (110 issues)

Subscriber: ooziedaily

Key Summary
OOZIE-2978  Remove code that handles Pig versions before 0.8 
https://issues.apache.org/jira/browse/OOZIE-2978
OOZIE-2977  error: self-closing element not allowed
https://issues.apache.org/jira/browse/OOZIE-2977
OOZIE-2975  code clean up in pig sharelib, replace Exception with more 
explicit, add try with resources, StringBuilder instead of StringBuffer
https://issues.apache.org/jira/browse/OOZIE-2975
OOZIE-2974  Change TestLiteWorkflowAppParser so that it works with Hadoop 3
https://issues.apache.org/jira/browse/OOZIE-2974
OOZIE-2969  Drop support for Java 1.7
https://issues.apache.org/jira/browse/OOZIE-2969
OOZIE-2964  getting at 
org.apache.maven.plugin.javadoc.AbstractJavadocMojo.executeJavadocCommandLine(AbstractJavadocMojo.java:5163)
https://issues.apache.org/jira/browse/OOZIE-2964
OOZIE-2963  getting error in build ArtifactNotFoundException: Could not find 
artifact org.pentaho:pentaho-aggdesigner-algorithm:jar:5.1.5-jhyde
https://issues.apache.org/jira/browse/OOZIE-2963
OOZIE-2962  bump maven-javadoc-plugin to 2.10.4
https://issues.apache.org/jira/browse/OOZIE-2962
OOZIE-2961  build contains multiple warnings for is not a parameter name
https://issues.apache.org/jira/browse/OOZIE-2961
OOZIE-2960  many warnings for @return tag has no arguments
https://issues.apache.org/jira/browse/OOZIE-2960
OOZIE-2957  Documentation states that starting a coordinator is possible
https://issues.apache.org/jira/browse/OOZIE-2957
OOZIE-2956  Fix Findbugs warnings related to reliance on default encoding in 
oozie-core
https://issues.apache.org/jira/browse/OOZIE-2956
OOZIE-2955  Fix Findbugs warnings related to reliance on default encoding in 
oozie-client
https://issues.apache.org/jira/browse/OOZIE-2955
OOZIE-2954  Fix Checkstyle issues in oozie-client
https://issues.apache.org/jira/browse/OOZIE-2954
OOZIE-2953  Fix Checkstyle issues in oozie-tools
https://issues.apache.org/jira/browse/OOZIE-2953
OOZIE-2952  Fix Findbugs warnings in oozie-sharelib-oozie
https://issues.apache.org/jira/browse/OOZIE-2952
OOZIE-2949  Escape quotes whitespaces in Sqoop  field
https://issues.apache.org/jira/browse/OOZIE-2949
OOZIE-2942  Fix Findbugs warnings in oozie-examples
https://issues.apache.org/jira/browse/OOZIE-2942
OOZIE-2937  Remove redundant groupId from the child pom's
https://issues.apache.org/jira/browse/OOZIE-2937
OOZIE-2934  Fix "Exceptional return value of java.io.File.mkdirs() ignored" 
Findbugs error in oozie-sharelib-spark
https://issues.apache.org/jira/browse/OOZIE-2934
OOZIE-2931  multiple warnings during build for "no @param for"
https://issues.apache.org/jira/browse/OOZIE-2931
OOZIE-2927  Append new line character for Hive2 query using query tag
https://issues.apache.org/jira/browse/OOZIE-2927
OOZIE-2916  Set a job name for the MR Action's child job
https://issues.apache.org/jira/browse/OOZIE-2916
OOZIE-2914  Consolidate Trim 
https://issues.apache.org/jira/browse/OOZIE-2914
OOZIE-2883  OOZIE throw the error "Missing 
[oozie.service.ProxyUserService.proxyuser.oozie.service.ProxyUserService.proxyuser.mr.groups]
 property"
https://issues.apache.org/jira/browse/OOZIE-2883
OOZIE-2877  Oozie Git Action
https://issues.apache.org/jira/browse/OOZIE-2877
OOZIE-2867  Timezone handling for Coordinators: emphasize "Continent/City" 
format
https://issues.apache.org/jira/browse/OOZIE-2867
OOZIE-2854  Oozie should handle transient database problems
https://issues.apache.org/jira/browse/OOZIE-2854
OOZIE-2834  ParameterVerifier logging non-useful warning for workflow definition
https://issues.apache.org/jira/browse/OOZIE-2834
OOZIE-2833  when using uber mode the regex pattern used in the 
extractHeapSizeMB method does not allow heap sizes specified in bytes.
https://issues.apache.org/jira/browse/OOZIE-2833
OOZIE-2829  Improve sharelib upload to accept multiple source folders
https://issues.apache.org/jira/browse/OOZIE-2829
OOZIE-2826  Falcon feed fails to aws s3; Oozie joda time version does not meet 
required jar version 2.2 or later
https://issues.apache.org/jira/browse/OOZIE-2826
OOZIE-2812  SparkConfigurationService should support loading configurations 
from multiple Spark versions
https://issues.apache.org/jira/browse/OOZIE-2812
OOZIE-2795  Create lib directory or symlink for Oozie CLI during packaging
https://issues.apache.org/jira/browse/OOZIE-2795
OOZIE-2791  ShareLib installation may fail on busy Hadoop clusters
https://issues.apache.org/jira/browse/OOZIE-2791
OOZIE-2784  Include WEEK as a parameter in the C