Re: Review Request 68317: Sqoop Action should support tez delegation tokens for hive-import

[Peter Cseh via Review Board]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/68317/#review207151
---




sharelib/sqoop/src/test/java/org/apache/oozie/action/hadoop/TestSqoopMain.java
Lines 32 (patched)


We could use 
https://github.com/apache/hadoop/blob/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/security/UserGroupInformation.java#L289
 here



sharelib/sqoop/src/test/java/org/apache/oozie/action/hadoop/TestSqoopMain.java
Lines 33 (patched)


Please use the constant from LauncherAM: 
https://github.com/apache/oozie/blob/master/sharelib/oozie/src/main/java/org/apache/oozie/action/hadoop/LauncherAM.java#L90



sharelib/sqoop/src/test/java/org/apache/oozie/action/hadoop/TestSqoopMain.java
Lines 66-68 (patched)


Please put this into a finally or @after block so the cleanup happens even 
if the test fails


- Peter Cseh


On Aug. 13, 2018, 8:20 a.m., Denes Bodo wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/68317/
> ---
> 
> (Updated Aug. 13, 2018, 8:20 a.m.)
> 
> 
> Review request for oozie and András Piros.
> 
> 
> Bugs: OOZIE-3326
> https://issues.apache.org/jira/browse/OOZIE-3326
> 
> 
> Repository: oozie-git
> 
> 
> Description
> ---
> 
> SqoopMain needs to support tez delegation tokens for hive-imports. 
> Implementation is similar to that of HiveMain and Hive2Main.
> 
> At present, hive-import will fail to start a tez session in secure 
> environment.
> 
> 
> Diffs
> -
> 
>   sharelib/oozie/src/main/java/org/apache/oozie/action/hadoop/LauncherAM.java 
> 4a41941 
>   
> sharelib/oozie/src/main/java/org/apache/oozie/action/hadoop/LauncherMain.java 
> e6e182c 
>   
> sharelib/oozie/src/main/java/org/apache/oozie/action/hadoop/SystemEnvironment.java
>  PRE-CREATION 
>   sharelib/sqoop/src/main/java/org/apache/oozie/action/hadoop/SqoopMain.java 
> 27f9306 
>   
> sharelib/sqoop/src/test/java/org/apache/oozie/action/hadoop/TestSqoopMain.java
>  d6f96d5 
> 
> 
> Diff: https://reviews.apache.org/r/68317/diff/1/
> 
> 
> Testing
> ---
> 
> 
> Thanks,
> 
> Denes Bodo
> 
>

Re: Review Request 68102: OOZIE-3193 Applications are not killed when submitted via subworkflow

[Peter Cseh via Review Board]

> On July 30, 2018, 10:46 a.m., András Piros wrote:
> > core/src/test/java/org/apache/oozie/action/oozie/TestSubWorkflowActionExecutor.java
> > Lines 708-712 (patched)
> > 
> >
> > Please use latest schema version `1.0`.

I don't think we should give up test coverage for older schemas. We should keep 
around the old ones in test and make sure to add enough covergave for new 
schemas when we add them.


- Peter


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/68102/#review206592
---


On July 30, 2018, 8:42 a.m., Kinga Marton wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/68102/
> ---
> 
> (Updated July 30, 2018, 8:42 a.m.)
> 
> 
> Review request for oozie, András Piros and Peter Cseh.
> 
> 
> Repository: oozie-git
> 
> 
> Description
> ---
> 
> Tags were not generated properly for external child jobs launched from subWFs 
> and they were not killed.
> 
> 
> Diffs
> -
> 
>   core/src/main/java/org/apache/oozie/action/ActionExecutor.java 71bd36ae6 
>   core/src/main/java/org/apache/oozie/action/hadoop/JavaActionExecutor.java 
> a1a9671ab 
>   core/src/test/java/org/apache/oozie/action/hadoop/LauncherMainTester.java 
> fdca5706e 
>   
> core/src/test/java/org/apache/oozie/action/hadoop/SleepMapperReducerForTest.java
>  PRE-CREATION 
>   
> core/src/test/java/org/apache/oozie/action/oozie/TestSubWorkflowActionExecutor.java
>  e074d482f 
> 
> 
> Diff: https://reviews.apache.org/r/68102/diff/1/
> 
> 
> Testing
> ---
> 
> Tested manually.
> Added Junit test.
> 
> 
> Thanks,
> 
> Kinga Marton
> 
>

Re: Review Request 64029: OOZIE-2339 Provide an API for writing jobs based on the XSD schemas

[Peter Cseh via Review Board]

> On June 14, 2018, 7:50 a.m., Peter Cseh wrote:
> > mvn clean install -DskipTests fails for me with a ton of checkstyle errors. 
> > The strange thing is that ./bin/mkdistro.sh -DskipTests does not. Can you 
> > check on this as well?

This is fine with the latest patch, sorry for the false alarm.


- Peter


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/64029/#review204762
---


On June 12, 2018, 1:42 p.m., András Piros wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/64029/
> ---
> 
> (Updated June 12, 2018, 1:42 p.m.)
> 
> 
> Review request for oozie, Attila Sasvari, Peter Cseh, and Robert Kanter.
> 
> 
> Repository: oozie-git
> 
> 
> Description
> ---
> 
> OOZIE-2339 Provide an API for writing jobs based on the XSD schemas
> 
> 
> Diffs
> -
> 
>   client/pom.xml 7d36e2d5941e598520a653a9eb4b293c763378e3 
>   client/src/main/java/org/apache/oozie/cli/OozieCLI.java 
> 08e2b91e6d9209278f9e086e23cfb4e555f7bd52 
>   client/src/main/java/org/apache/oozie/client/ApiJarFactory.java 
> PRE-CREATION 
>   client/src/main/java/org/apache/oozie/client/ApiJarLoader.java PRE-CREATION 
>   client/src/main/java/org/apache/oozie/client/OozieClient.java 
> 4c8175628ba5ee346d97ad984532662032b88acc 
>   core/pom.xml 45880a619e70a1643f4eb5ceeb61abb2090551f1 
>   core/src/main/java/org/apache/oozie/BaseLocalOozieClient.java 
> 38fb006980c0acb582b4f91175be1ced8a1afa79 
>   core/src/main/java/org/apache/oozie/local/LocalOozie.java 
> 9ab646cd53223d903e387eb92aa103a7b6577d0f 
>   core/src/main/java/org/apache/oozie/servlet/BaseJobServlet.java 
> e1bd3cf61aabfa4311881b2944fc9f10ffe9145d 
>   core/src/main/java/org/apache/oozie/servlet/BaseJobsServlet.java 
> d4b08711223356c8d4f58a3391b6b87efa0a2b73 
>   core/src/main/java/org/apache/oozie/servlet/ServletUtilities.java 
> 3eb32d50d07687c136fcc0e467a055bdc1e7ec60 
>   core/src/main/java/org/apache/oozie/servlet/V0JobsServlet.java 
> 1d800949cbff34a95f2ae111364eb8cbc6a5f5fb 
>   core/src/main/java/org/apache/oozie/servlet/V1JobsServlet.java 
> c1ca65fd2f8bf0f3e5c618dddcba3bf52eaf5420 
>   core/src/main/java/org/apache/oozie/servlet/V2ValidateServlet.java 
> b86fa6ace43aead115302aad3ae15368b15f1923 
>   core/src/main/resources/oozie-default.xml 
> 8d7465c27bfe0a25b07daf556a36c6ecc2569cb6 
>   core/src/test/java/org/apache/oozie/client/TestOozieCLI.java 
> 3395cc25d27805444ba0b6463357c4ecff977ac1 
>   core/src/test/java/org/apache/oozie/servlet/TestV1JobsServlet.java 
> fb481e367653187a94199069199f1d1b06d1cecf 
>   core/src/test/java/org/apache/oozie/test/MiniOozieTestCase.java 
> 0a030d660f9c545b520afcce458a27ba2a367658 
>   docs/src/site/twiki/DG_CommandLineTool.twiki 
> 9a174596dd78fb15989daf4e88936a9a2b09ef13 
>   docs/src/site/twiki/DG_CustomActionExecutor.twiki 
> 4acbf0d40ae36c441f8510e107243603fa0e001b 
>   docs/src/site/twiki/DG_Examples.twiki 
> 13dfa28c0eecab70321886b2806a791346c46d3b 
>   docs/src/site/twiki/DG_FluentJobAPI.twiki PRE-CREATION 
>   examples/pom.xml c13febc81cf5b26b4af90885787113e247ae7cb8 
>   
> examples/src/main/java/org/apache/oozie/example/fluentjob/CredentialsRetrying.java
>  PRE-CREATION 
>   examples/src/main/java/org/apache/oozie/example/fluentjob/Global.java 
> PRE-CREATION 
>   examples/src/main/java/org/apache/oozie/example/fluentjob/JavaMain.java 
> PRE-CREATION 
>   
> examples/src/main/java/org/apache/oozie/example/fluentjob/MultipleShellActions.java
>  PRE-CREATION 
>   examples/src/main/java/org/apache/oozie/example/fluentjob/Parameters.java 
> PRE-CREATION 
>   examples/src/main/java/org/apache/oozie/example/fluentjob/Shell.java 
> PRE-CREATION 
>   examples/src/main/java/org/apache/oozie/example/fluentjob/Spark.java 
> PRE-CREATION 
>   findbugs-filter.xml 03ee4d103f46487493fd725224cf4912ac7bc7dd 
>   fluent-job/fluent-job-api/pom.xml PRE-CREATION 
>   
> fluent-job/fluent-job-api/src/main/java/org/apache/oozie/fluentjob/api/Condition.java
>  PRE-CREATION 
>   
> fluent-job/fluent-job-api/src/main/java/org/apache/oozie/fluentjob/api/GraphVisualization.java
>  PRE-CREATION 
>   
> fluent-job/fluent-job-api/src/main/java/org/apache/oozie/fluentjob/api/ModifyOnce.java
>  PRE-CREATION 
>   
> fluent-job/fluent-job-api/src/main/java/org/apache/oozie/fluentjob/api/action/ActionAttributes.java
>  PRE-CREATION 
>   
> fluent-job/fluent-job-api/src/main/java/org/apache/oozie/fluentjob/api/action/ActionAttributesBuilder.java
>  PRE-CREATION 
>   
> fluent-job/fluent-job-api/src/main/java/org/apache/oozie/fluentjob/api/action/Builder.java
>  PRE-CREATION 
>   
> fluent-job/fluent-job-api/src/main/java/org/apache/oozie/fluentjob/api/action/ChFSBase.java
>  PRE-CREATION 
>   
> 

Re: Review Request 64029: OOZIE-2339 Provide an API for writing jobs based on the XSD schemas

[Peter Cseh via Review Board]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/64029/#review204766
---




fluent-job/fluent-job-client/src/test/java/org/apache/oozie/jobs/client/minitest/TestJavaAction.java
Lines 80 (patched)


I have checked on this and found that the generated xml is:
 
 
 When there are no credentials set, the "cred" tag should not be generated.


- Peter Cseh


On June 12, 2018, 1:42 p.m., András Piros wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/64029/
> ---
> 
> (Updated June 12, 2018, 1:42 p.m.)
> 
> 
> Review request for oozie, Attila Sasvari, Peter Cseh, and Robert Kanter.
> 
> 
> Repository: oozie-git
> 
> 
> Description
> ---
> 
> OOZIE-2339 Provide an API for writing jobs based on the XSD schemas
> 
> 
> Diffs
> -
> 
>   client/pom.xml 7d36e2d5941e598520a653a9eb4b293c763378e3 
>   client/src/main/java/org/apache/oozie/cli/OozieCLI.java 
> 08e2b91e6d9209278f9e086e23cfb4e555f7bd52 
>   client/src/main/java/org/apache/oozie/client/ApiJarFactory.java 
> PRE-CREATION 
>   client/src/main/java/org/apache/oozie/client/ApiJarLoader.java PRE-CREATION 
>   client/src/main/java/org/apache/oozie/client/OozieClient.java 
> 4c8175628ba5ee346d97ad984532662032b88acc 
>   core/pom.xml 45880a619e70a1643f4eb5ceeb61abb2090551f1 
>   core/src/main/java/org/apache/oozie/BaseLocalOozieClient.java 
> 38fb006980c0acb582b4f91175be1ced8a1afa79 
>   core/src/main/java/org/apache/oozie/local/LocalOozie.java 
> 9ab646cd53223d903e387eb92aa103a7b6577d0f 
>   core/src/main/java/org/apache/oozie/servlet/BaseJobServlet.java 
> e1bd3cf61aabfa4311881b2944fc9f10ffe9145d 
>   core/src/main/java/org/apache/oozie/servlet/BaseJobsServlet.java 
> d4b08711223356c8d4f58a3391b6b87efa0a2b73 
>   core/src/main/java/org/apache/oozie/servlet/ServletUtilities.java 
> 3eb32d50d07687c136fcc0e467a055bdc1e7ec60 
>   core/src/main/java/org/apache/oozie/servlet/V0JobsServlet.java 
> 1d800949cbff34a95f2ae111364eb8cbc6a5f5fb 
>   core/src/main/java/org/apache/oozie/servlet/V1JobsServlet.java 
> c1ca65fd2f8bf0f3e5c618dddcba3bf52eaf5420 
>   core/src/main/java/org/apache/oozie/servlet/V2ValidateServlet.java 
> b86fa6ace43aead115302aad3ae15368b15f1923 
>   core/src/main/resources/oozie-default.xml 
> 8d7465c27bfe0a25b07daf556a36c6ecc2569cb6 
>   core/src/test/java/org/apache/oozie/client/TestOozieCLI.java 
> 3395cc25d27805444ba0b6463357c4ecff977ac1 
>   core/src/test/java/org/apache/oozie/servlet/TestV1JobsServlet.java 
> fb481e367653187a94199069199f1d1b06d1cecf 
>   core/src/test/java/org/apache/oozie/test/MiniOozieTestCase.java 
> 0a030d660f9c545b520afcce458a27ba2a367658 
>   docs/src/site/twiki/DG_CommandLineTool.twiki 
> 9a174596dd78fb15989daf4e88936a9a2b09ef13 
>   docs/src/site/twiki/DG_CustomActionExecutor.twiki 
> 4acbf0d40ae36c441f8510e107243603fa0e001b 
>   docs/src/site/twiki/DG_Examples.twiki 
> 13dfa28c0eecab70321886b2806a791346c46d3b 
>   docs/src/site/twiki/DG_FluentJobAPI.twiki PRE-CREATION 
>   examples/pom.xml c13febc81cf5b26b4af90885787113e247ae7cb8 
>   
> examples/src/main/java/org/apache/oozie/example/fluentjob/CredentialsRetrying.java
>  PRE-CREATION 
>   examples/src/main/java/org/apache/oozie/example/fluentjob/Global.java 
> PRE-CREATION 
>   examples/src/main/java/org/apache/oozie/example/fluentjob/JavaMain.java 
> PRE-CREATION 
>   
> examples/src/main/java/org/apache/oozie/example/fluentjob/MultipleShellActions.java
>  PRE-CREATION 
>   examples/src/main/java/org/apache/oozie/example/fluentjob/Parameters.java 
> PRE-CREATION 
>   examples/src/main/java/org/apache/oozie/example/fluentjob/Shell.java 
> PRE-CREATION 
>   examples/src/main/java/org/apache/oozie/example/fluentjob/Spark.java 
> PRE-CREATION 
>   findbugs-filter.xml 03ee4d103f46487493fd725224cf4912ac7bc7dd 
>   fluent-job/fluent-job-api/pom.xml PRE-CREATION 
>   
> fluent-job/fluent-job-api/src/main/java/org/apache/oozie/fluentjob/api/Condition.java
>  PRE-CREATION 
>   
> fluent-job/fluent-job-api/src/main/java/org/apache/oozie/fluentjob/api/GraphVisualization.java
>  PRE-CREATION 
>   
> fluent-job/fluent-job-api/src/main/java/org/apache/oozie/fluentjob/api/ModifyOnce.java
>  PRE-CREATION 
>   
> fluent-job/fluent-job-api/src/main/java/org/apache/oozie/fluentjob/api/action/ActionAttributes.java
>  PRE-CREATION 
>   
> fluent-job/fluent-job-api/src/main/java/org/apache/oozie/fluentjob/api/action/ActionAttributesBuilder.java
>  PRE-CREATION 
>   
> fluent-job/fluent-job-api/src/main/java/org/apache/oozie/fluentjob/api/action/Builder.java
>  PRE-CREATION 
>   
> fluent-job/fluent-job-api/src/main/java/org/apache/oozie/fluentjob/api/action/ChFSBase.java
>  PRE-CREATION 
>   
> 

Re: Review Request 64029: OOZIE-2339 Provide an API for writing jobs based on the XSD schemas

[Peter Cseh via Review Board]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/64029/#review204762
---



mvn clean install -DskipTests fails for me with a ton of checkstyle errors. The 
strange thing is that ./bin/mkdistro.sh -DskipTests does not. Can you check on 
this as well?

- Peter Cseh


On June 12, 2018, 1:42 p.m., András Piros wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/64029/
> ---
> 
> (Updated June 12, 2018, 1:42 p.m.)
> 
> 
> Review request for oozie, Attila Sasvari, Peter Cseh, and Robert Kanter.
> 
> 
> Repository: oozie-git
> 
> 
> Description
> ---
> 
> OOZIE-2339 Provide an API for writing jobs based on the XSD schemas
> 
> 
> Diffs
> -
> 
>   client/pom.xml 7d36e2d5941e598520a653a9eb4b293c763378e3 
>   client/src/main/java/org/apache/oozie/cli/OozieCLI.java 
> 08e2b91e6d9209278f9e086e23cfb4e555f7bd52 
>   client/src/main/java/org/apache/oozie/client/ApiJarFactory.java 
> PRE-CREATION 
>   client/src/main/java/org/apache/oozie/client/ApiJarLoader.java PRE-CREATION 
>   client/src/main/java/org/apache/oozie/client/OozieClient.java 
> 4c8175628ba5ee346d97ad984532662032b88acc 
>   core/pom.xml 45880a619e70a1643f4eb5ceeb61abb2090551f1 
>   core/src/main/java/org/apache/oozie/BaseLocalOozieClient.java 
> 38fb006980c0acb582b4f91175be1ced8a1afa79 
>   core/src/main/java/org/apache/oozie/local/LocalOozie.java 
> 9ab646cd53223d903e387eb92aa103a7b6577d0f 
>   core/src/main/java/org/apache/oozie/servlet/BaseJobServlet.java 
> e1bd3cf61aabfa4311881b2944fc9f10ffe9145d 
>   core/src/main/java/org/apache/oozie/servlet/BaseJobsServlet.java 
> d4b08711223356c8d4f58a3391b6b87efa0a2b73 
>   core/src/main/java/org/apache/oozie/servlet/ServletUtilities.java 
> 3eb32d50d07687c136fcc0e467a055bdc1e7ec60 
>   core/src/main/java/org/apache/oozie/servlet/V0JobsServlet.java 
> 1d800949cbff34a95f2ae111364eb8cbc6a5f5fb 
>   core/src/main/java/org/apache/oozie/servlet/V1JobsServlet.java 
> c1ca65fd2f8bf0f3e5c618dddcba3bf52eaf5420 
>   core/src/main/java/org/apache/oozie/servlet/V2ValidateServlet.java 
> b86fa6ace43aead115302aad3ae15368b15f1923 
>   core/src/main/resources/oozie-default.xml 
> 8d7465c27bfe0a25b07daf556a36c6ecc2569cb6 
>   core/src/test/java/org/apache/oozie/client/TestOozieCLI.java 
> 3395cc25d27805444ba0b6463357c4ecff977ac1 
>   core/src/test/java/org/apache/oozie/servlet/TestV1JobsServlet.java 
> fb481e367653187a94199069199f1d1b06d1cecf 
>   core/src/test/java/org/apache/oozie/test/MiniOozieTestCase.java 
> 0a030d660f9c545b520afcce458a27ba2a367658 
>   docs/src/site/twiki/DG_CommandLineTool.twiki 
> 9a174596dd78fb15989daf4e88936a9a2b09ef13 
>   docs/src/site/twiki/DG_CustomActionExecutor.twiki 
> 4acbf0d40ae36c441f8510e107243603fa0e001b 
>   docs/src/site/twiki/DG_Examples.twiki 
> 13dfa28c0eecab70321886b2806a791346c46d3b 
>   docs/src/site/twiki/DG_FluentJobAPI.twiki PRE-CREATION 
>   examples/pom.xml c13febc81cf5b26b4af90885787113e247ae7cb8 
>   
> examples/src/main/java/org/apache/oozie/example/fluentjob/CredentialsRetrying.java
>  PRE-CREATION 
>   examples/src/main/java/org/apache/oozie/example/fluentjob/Global.java 
> PRE-CREATION 
>   examples/src/main/java/org/apache/oozie/example/fluentjob/JavaMain.java 
> PRE-CREATION 
>   
> examples/src/main/java/org/apache/oozie/example/fluentjob/MultipleShellActions.java
>  PRE-CREATION 
>   examples/src/main/java/org/apache/oozie/example/fluentjob/Parameters.java 
> PRE-CREATION 
>   examples/src/main/java/org/apache/oozie/example/fluentjob/Shell.java 
> PRE-CREATION 
>   examples/src/main/java/org/apache/oozie/example/fluentjob/Spark.java 
> PRE-CREATION 
>   findbugs-filter.xml 03ee4d103f46487493fd725224cf4912ac7bc7dd 
>   fluent-job/fluent-job-api/pom.xml PRE-CREATION 
>   
> fluent-job/fluent-job-api/src/main/java/org/apache/oozie/fluentjob/api/Condition.java
>  PRE-CREATION 
>   
> fluent-job/fluent-job-api/src/main/java/org/apache/oozie/fluentjob/api/GraphVisualization.java
>  PRE-CREATION 
>   
> fluent-job/fluent-job-api/src/main/java/org/apache/oozie/fluentjob/api/ModifyOnce.java
>  PRE-CREATION 
>   
> fluent-job/fluent-job-api/src/main/java/org/apache/oozie/fluentjob/api/action/ActionAttributes.java
>  PRE-CREATION 
>   
> fluent-job/fluent-job-api/src/main/java/org/apache/oozie/fluentjob/api/action/ActionAttributesBuilder.java
>  PRE-CREATION 
>   
> fluent-job/fluent-job-api/src/main/java/org/apache/oozie/fluentjob/api/action/Builder.java
>  PRE-CREATION 
>   
> fluent-job/fluent-job-api/src/main/java/org/apache/oozie/fluentjob/api/action/ChFSBase.java
>  PRE-CREATION 
>   
> fluent-job/fluent-job-api/src/main/java/org/apache/oozie/fluentjob/api/action/ChFSBaseBuilder.java
>  PRE-CREATION 
>   
> 

Re: Review Request 64029: OOZIE-2339 Provide an API for writing jobs based on the XSD schemas

[Peter Cseh via Review Board]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/64029/#review204761
---




fluent-job/fluent-job-api/src/main/java/org/apache/oozie/fluentjob/api/workflow/Credentials.java
Lines 39-45 (patched)


This should be a map instead of a List.
Credentials then could be referenced by their names.

Or better, this part could be autogenerated based on the credentials every 
action has. Basically we only need a CredentialBuilder to create Credential 
objects and add those to actions. 
The generator part could collect the used credentials in the workflow and 
just put in the  section automatically.


- Peter Cseh


On June 12, 2018, 1:42 p.m., András Piros wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/64029/
> ---
> 
> (Updated June 12, 2018, 1:42 p.m.)
> 
> 
> Review request for oozie, Attila Sasvari, Peter Cseh, and Robert Kanter.
> 
> 
> Repository: oozie-git
> 
> 
> Description
> ---
> 
> OOZIE-2339 Provide an API for writing jobs based on the XSD schemas
> 
> 
> Diffs
> -
> 
>   client/pom.xml 7d36e2d5941e598520a653a9eb4b293c763378e3 
>   client/src/main/java/org/apache/oozie/cli/OozieCLI.java 
> 08e2b91e6d9209278f9e086e23cfb4e555f7bd52 
>   client/src/main/java/org/apache/oozie/client/ApiJarFactory.java 
> PRE-CREATION 
>   client/src/main/java/org/apache/oozie/client/ApiJarLoader.java PRE-CREATION 
>   client/src/main/java/org/apache/oozie/client/OozieClient.java 
> 4c8175628ba5ee346d97ad984532662032b88acc 
>   core/pom.xml 45880a619e70a1643f4eb5ceeb61abb2090551f1 
>   core/src/main/java/org/apache/oozie/BaseLocalOozieClient.java 
> 38fb006980c0acb582b4f91175be1ced8a1afa79 
>   core/src/main/java/org/apache/oozie/local/LocalOozie.java 
> 9ab646cd53223d903e387eb92aa103a7b6577d0f 
>   core/src/main/java/org/apache/oozie/servlet/BaseJobServlet.java 
> e1bd3cf61aabfa4311881b2944fc9f10ffe9145d 
>   core/src/main/java/org/apache/oozie/servlet/BaseJobsServlet.java 
> d4b08711223356c8d4f58a3391b6b87efa0a2b73 
>   core/src/main/java/org/apache/oozie/servlet/ServletUtilities.java 
> 3eb32d50d07687c136fcc0e467a055bdc1e7ec60 
>   core/src/main/java/org/apache/oozie/servlet/V0JobsServlet.java 
> 1d800949cbff34a95f2ae111364eb8cbc6a5f5fb 
>   core/src/main/java/org/apache/oozie/servlet/V1JobsServlet.java 
> c1ca65fd2f8bf0f3e5c618dddcba3bf52eaf5420 
>   core/src/main/java/org/apache/oozie/servlet/V2ValidateServlet.java 
> b86fa6ace43aead115302aad3ae15368b15f1923 
>   core/src/main/resources/oozie-default.xml 
> 8d7465c27bfe0a25b07daf556a36c6ecc2569cb6 
>   core/src/test/java/org/apache/oozie/client/TestOozieCLI.java 
> 3395cc25d27805444ba0b6463357c4ecff977ac1 
>   core/src/test/java/org/apache/oozie/servlet/TestV1JobsServlet.java 
> fb481e367653187a94199069199f1d1b06d1cecf 
>   core/src/test/java/org/apache/oozie/test/MiniOozieTestCase.java 
> 0a030d660f9c545b520afcce458a27ba2a367658 
>   docs/src/site/twiki/DG_CommandLineTool.twiki 
> 9a174596dd78fb15989daf4e88936a9a2b09ef13 
>   docs/src/site/twiki/DG_CustomActionExecutor.twiki 
> 4acbf0d40ae36c441f8510e107243603fa0e001b 
>   docs/src/site/twiki/DG_Examples.twiki 
> 13dfa28c0eecab70321886b2806a791346c46d3b 
>   docs/src/site/twiki/DG_FluentJobAPI.twiki PRE-CREATION 
>   examples/pom.xml c13febc81cf5b26b4af90885787113e247ae7cb8 
>   
> examples/src/main/java/org/apache/oozie/example/fluentjob/CredentialsRetrying.java
>  PRE-CREATION 
>   examples/src/main/java/org/apache/oozie/example/fluentjob/Global.java 
> PRE-CREATION 
>   examples/src/main/java/org/apache/oozie/example/fluentjob/JavaMain.java 
> PRE-CREATION 
>   
> examples/src/main/java/org/apache/oozie/example/fluentjob/MultipleShellActions.java
>  PRE-CREATION 
>   examples/src/main/java/org/apache/oozie/example/fluentjob/Parameters.java 
> PRE-CREATION 
>   examples/src/main/java/org/apache/oozie/example/fluentjob/Shell.java 
> PRE-CREATION 
>   examples/src/main/java/org/apache/oozie/example/fluentjob/Spark.java 
> PRE-CREATION 
>   findbugs-filter.xml 03ee4d103f46487493fd725224cf4912ac7bc7dd 
>   fluent-job/fluent-job-api/pom.xml PRE-CREATION 
>   
> fluent-job/fluent-job-api/src/main/java/org/apache/oozie/fluentjob/api/Condition.java
>  PRE-CREATION 
>   
> fluent-job/fluent-job-api/src/main/java/org/apache/oozie/fluentjob/api/GraphVisualization.java
>  PRE-CREATION 
>   
> fluent-job/fluent-job-api/src/main/java/org/apache/oozie/fluentjob/api/ModifyOnce.java
>  PRE-CREATION 
>   
> fluent-job/fluent-job-api/src/main/java/org/apache/oozie/fluentjob/api/action/ActionAttributes.java
>  PRE-CREATION 
>   
> 

Re: Review Request 67314: Oozie Sqoop action with command splits the select clause into multiple parts due to delimiter being space

[Peter Cseh via Review Board]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/67314/#review204362
---




core/src/main/java/org/apache/oozie/action/hadoop/SqoopActionExecutor.java
Lines 95 (patched)


Is "contains" strong enough here?
I can think of two scenarios when it might screw up the parsin:
1) a table name contains a "-" character
2) a query like "select * where price > -3 and price < 0



core/src/main/java/org/apache/oozie/action/hadoop/SqoopActionExecutor.java
Lines 98 (patched)


This changes behavior (for the better I thnik :)) Please update the 
documentation as well.



sharelib/sqoop/src/test/java/org/apache/oozie/action/hadoop/TestSqoopActionExecutor.java
Lines 50 (patched)


Please add a test case where this select is quoted


- Peter Cseh


On May 25, 2018, 7:37 a.m., Denes Bodo wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/67314/
> ---
> 
> (Updated May 25, 2018, 7:37 a.m.)
> 
> 
> Review request for oozie and András Piros.
> 
> 
> Bugs: OOZIE-3218
> https://issues.apache.org/jira/browse/OOZIE-3218
> 
> 
> Repository: oozie-git
> 
> 
> Description
> ---
> 
> When running a Oozie Sqoop action which has command with --query in place the 
> query is split into multiple parts causing "Unrecognized argument:" and 
> in-turn fails.
> 
> 
> Diffs
> -
> 
>   core/src/main/java/org/apache/oozie/action/hadoop/SqoopActionExecutor.java 
> 69d5e7e 
>   
> sharelib/sqoop/src/test/java/org/apache/oozie/action/hadoop/TestSqoopActionExecutor.java
>  edfe0c7 
> 
> 
> Diff: https://reviews.apache.org/r/67314/diff/1/
> 
> 
> Testing
> ---
> 
> 
> Thanks,
> 
> Denes Bodo
> 
>

Re: Review Request 66930: OOZIE-2791 ShareLib installation may fail on busy Hadoop clusters

[Peter Cseh via Review Board]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/66930/#review204339
---




tools/src/main/java/org/apache/oozie/tools/OozieSharelibCLI.java
Lines 252 (patched)


Why is a delete call here? Is it to clean up for retries in case the 
previous attempt created the file but failed during upload?


- Peter Cseh


On June 5, 2018, 8:37 a.m., Kinga Marton wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/66930/
> ---
> 
> (Updated June 5, 2018, 8:37 a.m.)
> 
> 
> Review request for oozie, András Piros and Peter Cseh.
> 
> 
> Repository: oozie-git
> 
> 
> Description
> ---
> 
> On a busy Hadoop cluster it can happen that users cannot install properly 
> Oozie ShareLib.
> 
> 
> Diffs
> -
> 
>   tools/src/main/java/org/apache/oozie/tools/OozieSharelibCLI.java dce1c559 
>   tools/src/test/java/org/apache/oozie/tools/TestBlockSizeCalculator.java 
> PRE-CREATION 
>   tools/src/test/java/org/apache/oozie/tools/TestConcurrentCopyFromLocal.java 
> PRE-CREATION 
>   tools/src/test/java/org/apache/oozie/tools/TestCopyTaskCallable.java 
> PRE-CREATION 
>   tools/src/test/java/org/apache/oozie/tools/TestOozieSharelibCLI.java 
> ccad273b 
> 
> 
> Diff: https://reviews.apache.org/r/66930/diff/4/
> 
> 
> Testing
> ---
> 
> Tested manually
> 
> 
> Thanks,
> 
> Kinga Marton
> 
>

Re: Review Request 66930: OOZIE-2791 ShareLib installation may fail on busy Hadoop clusters

[Peter Cseh via Review Board]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/66930/#review204336
---




tools/src/main/java/org/apache/oozie/tools/OozieSharelibCLI.java
Line 254 (original), 424 (patched)


We could check for failedCopyTasks.isEmpty() here, can't we?



tools/src/main/java/org/apache/oozie/tools/OozieSharelibCLI.java
Lines 429 (patched)


the "t" in the message might be different here than the original "t" that 
was null-checked. Please use an error from the failed task list.


- Peter Cseh


On June 5, 2018, 8:37 a.m., Kinga Marton wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/66930/
> ---
> 
> (Updated June 5, 2018, 8:37 a.m.)
> 
> 
> Review request for oozie, András Piros and Peter Cseh.
> 
> 
> Repository: oozie-git
> 
> 
> Description
> ---
> 
> On a busy Hadoop cluster it can happen that users cannot install properly 
> Oozie ShareLib.
> 
> 
> Diffs
> -
> 
>   tools/src/main/java/org/apache/oozie/tools/OozieSharelibCLI.java dce1c559 
>   tools/src/test/java/org/apache/oozie/tools/TestBlockSizeCalculator.java 
> PRE-CREATION 
>   tools/src/test/java/org/apache/oozie/tools/TestConcurrentCopyFromLocal.java 
> PRE-CREATION 
>   tools/src/test/java/org/apache/oozie/tools/TestCopyTaskCallable.java 
> PRE-CREATION 
>   tools/src/test/java/org/apache/oozie/tools/TestOozieSharelibCLI.java 
> ccad273b 
> 
> 
> Diff: https://reviews.apache.org/r/66930/diff/4/
> 
> 
> Testing
> ---
> 
> Tested manually
> 
> 
> Thanks,
> 
> Kinga Marton
> 
>

Re: Review Request 67112: OOZIE-3228 [Spark action] Can't load properties from spark-defaults.conf

[Peter Cseh via Review Board]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/67112/#review203207
---




sharelib/spark/src/main/java/org/apache/oozie/action/hadoop/SparkArgsExtractor.java
Lines 379-398 (patched)


Can you also add this on the twiki level? Most users don't check the source 
code if something happens they don't expect.


- Peter Cseh


On May 16, 2018, 9:23 a.m., András Piros wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/67112/
> ---
> 
> (Updated May 16, 2018, 9:23 a.m.)
> 
> 
> Review request for oozie and Peter Cseh.
> 
> 
> Repository: oozie-git
> 
> 
> Description
> ---
> 
> OOZIE-3228 [Spark action] Can't load properties from spark-defaults.conf
> 
> 
> Diffs
> -
> 
>   
> sharelib/spark/src/main/java/org/apache/oozie/action/hadoop/SparkArgsExtractor.java
>  5af39cda52b403d8989fdd5d4f2524395a55580a 
>   
> sharelib/spark/src/test/java/org/apache/oozie/action/hadoop/TestSparkArgsExtractor.java
>  6a9baa5c7997198ab69f039f6a8884e49b13cad3 
> 
> 
> Diff: https://reviews.apache.org/r/67112/diff/3/
> 
> 
> Testing
> ---
> 
> `mvn clean test -Dtest=TestSparkArgsExtractor`
> 
> 
> Thanks,
> 
> András Piros
> 
>

Re: Review Request 67112: OOZIE-3228 [Spark action] Can't load properties from spark-defaults.conf

[Peter Cseh via Review Board]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/67112/#review203009
---




sharelib/spark/src/main/java/org/apache/oozie/action/hadoop/SparkArgsExtractor.java
Line 41 (original), 41 (patched)


Please don't use *



sharelib/spark/src/main/java/org/apache/oozie/action/hadoop/SparkArgsExtractor.java
Lines 85 (patched)


This one is used only in tests if I see correctly. Please move it ther.



sharelib/spark/src/main/java/org/apache/oozie/action/hadoop/SparkArgsExtractor.java
Lines 375-378 (patched)


Please add documentation about the affected properties and some reasoning 
on why are we doing this.



sharelib/spark/src/main/java/org/apache/oozie/action/hadoop/SparkArgsExtractor.java
Lines 381-389 (patched)


This part feels similar to parts of 
https://github.com/apache/oozie/blob/master/core/src/main/java/org/apache/oozie/action/hadoop/LauncherConfigurationInjector.java

Is it possible to reuse code from there?
It's possible I'm missing something that makes that not feasable.


- Peter Cseh


On May 14, 2018, 11:04 a.m., András Piros wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/67112/
> ---
> 
> (Updated May 14, 2018, 11:04 a.m.)
> 
> 
> Review request for oozie and Peter Cseh.
> 
> 
> Repository: oozie-git
> 
> 
> Description
> ---
> 
> OOZIE-3228 [Spark action] Can't load properties from spark-defaults.conf
> 
> 
> Diffs
> -
> 
>   
> sharelib/spark/src/main/java/org/apache/oozie/action/hadoop/SparkArgsExtractor.java
>  2f3cfbe4dfa2c2015686d2f2ca09ec4de544f423 
>   
> sharelib/spark/src/test/java/org/apache/oozie/action/hadoop/TestSparkArgsExtractor.java
>  6a9baa5c7997198ab69f039f6a8884e49b13cad3 
> 
> 
> Diff: https://reviews.apache.org/r/67112/diff/1/
> 
> 
> Testing
> ---
> 
> `mvn clean test -Dtest=TestSparkArgsExtractor`
> 
> 
> Thanks,
> 
> András Piros
> 
>

Re: Review Request 59620: This review board request is for an action to provide a Git action for Oozie

[Peter Cseh via Review Board]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/59620/#review202546
---




examples/src/main/apps/git/workflow.xml
Lines 24-25 (patched)


Let's check out apache/Oozie from github instead :)


- Peter Cseh


On May 4, 2018, 5:43 p.m., Clay B. wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/59620/
> ---
> 
> (Updated May 4, 2018, 5:43 p.m.)
> 
> 
> Review request for oozie and András Piros.
> 
> 
> Bugs: OOZIE-2877
> https://issues.apache.org/jira/browse/OOZIE-2877
> 
> 
> Repository: oozie-git
> 
> 
> Description
> ---
> 
> OOZIE-2877 - Oozie Git Action
> 
> 
> Diffs
> -
> 
>   client/src/main/java/org/apache/oozie/cli/OozieCLI.java 
> 4abc7502c0c9d8b59ded2baaed30c407ad073008 
>   client/src/main/resources/git-action-0.1.xsd PRE-CREATION 
>   core/src/main/java/org/apache/oozie/action/hadoop/GitActionExecutor.java 
> PRE-CREATION 
>   core/src/main/resources/oozie-default.xml 
> 6f8925828090cee29a818de30a7c31db0617d816 
>   docs/src/site/twiki/WorkflowFunctionalSpec.twiki 
> 76cbe21ecd79eae86885a8db77831bf2c40f2d4f 
>   examples/src/main/apps/git/job.properties PRE-CREATION 
>   examples/src/main/apps/git/workflow.xml PRE-CREATION 
>   pom.xml d9fe1b20f571b1a6afacb3ce53f4d409dd3d60b1 
>   sharelib/git/pom.xml PRE-CREATION 
>   sharelib/git/src/main/java/org/apache/oozie/action/hadoop/GitMain.java 
> PRE-CREATION 
>   
> sharelib/git/src/test/java/org/apache/oozie/action/hadoop/TestGitActionExecutor.java
>  PRE-CREATION 
>   
> sharelib/git/src/test/java/org/apache/oozie/action/hadoop/TestGitMainGetKey.java
>  PRE-CREATION 
>   
> sharelib/git/src/test/java/org/apache/oozie/action/hadoop/TestIntegrationGitActionExecutor.java
>  PRE-CREATION 
>   sharelib/pom.xml 6a0864d4e5b52160dc4e1a5cb9dfc34a037b67c8 
>   src/main/assemblies/sharelib.xml 07dc69c8276895b254e5af0cc021cce6ebad18f4 
>   webapp/pom.xml 797996912b6e6381b261a69f8eb1e012fe488fdf 
> 
> 
> Diff: https://reviews.apache.org/r/59620/diff/5/
> 
> 
> Testing
> ---
> 
> Tested using unit and integration tests. Still need to:
> * Test on a cluster
> * Test with an authenticated SSH hosted Git repo
> 
> Sumitted a request to the JGit community as their branch pulling code seems 
> to have an 
> [issue](https://dev.eclipse.org/mhonarc/lists/jgit-dev/msg03343.html).
> 
> Andras has tested against his patch 011 with results at 
> https://issues.apache.org/jira/browse/OOZIE-2877?focusedCommentId=16459523=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-16459523
> 
> 
> File Attachments
> 
> 
> 0001-OOZIE-2877-Oozie-Git-Action.patch
>   
> https://reviews.apache.org/media/uploaded/files/2017/05/29/24f90a78-3dc1-49fe-bf29-5927a3cd5e72__0001-OOZIE-2877-Oozie-Git-Action.patch
> Patch
>   
> https://reviews.apache.org/media/uploaded/files/2017/05/29/dd23dd72-67e0-456f-9b52-e566d8d17d16__0001-OOZIE-2877-Oozie-Git-Action.patch
> 
> 
> Thanks,
> 
> Clay B.
> 
>

Re: Review Request 59620: This review board request is for an action to provide a Git action for Oozie

[Peter Cseh via Review Board]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/59620/#review202545
---




core/src/main/java/org/apache/oozie/action/hadoop/GitActionExecutor.java
Lines 55-56 (patched)


oozie.oozie? seems redundant :)


- Peter Cseh


On May 4, 2018, 5:43 p.m., Clay B. wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/59620/
> ---
> 
> (Updated May 4, 2018, 5:43 p.m.)
> 
> 
> Review request for oozie and András Piros.
> 
> 
> Bugs: OOZIE-2877
> https://issues.apache.org/jira/browse/OOZIE-2877
> 
> 
> Repository: oozie-git
> 
> 
> Description
> ---
> 
> OOZIE-2877 - Oozie Git Action
> 
> 
> Diffs
> -
> 
>   client/src/main/java/org/apache/oozie/cli/OozieCLI.java 
> 4abc7502c0c9d8b59ded2baaed30c407ad073008 
>   client/src/main/resources/git-action-0.1.xsd PRE-CREATION 
>   core/src/main/java/org/apache/oozie/action/hadoop/GitActionExecutor.java 
> PRE-CREATION 
>   core/src/main/resources/oozie-default.xml 
> 6f8925828090cee29a818de30a7c31db0617d816 
>   docs/src/site/twiki/WorkflowFunctionalSpec.twiki 
> 76cbe21ecd79eae86885a8db77831bf2c40f2d4f 
>   examples/src/main/apps/git/job.properties PRE-CREATION 
>   examples/src/main/apps/git/workflow.xml PRE-CREATION 
>   pom.xml d9fe1b20f571b1a6afacb3ce53f4d409dd3d60b1 
>   sharelib/git/pom.xml PRE-CREATION 
>   sharelib/git/src/main/java/org/apache/oozie/action/hadoop/GitMain.java 
> PRE-CREATION 
>   
> sharelib/git/src/test/java/org/apache/oozie/action/hadoop/TestGitActionExecutor.java
>  PRE-CREATION 
>   
> sharelib/git/src/test/java/org/apache/oozie/action/hadoop/TestGitMainGetKey.java
>  PRE-CREATION 
>   
> sharelib/git/src/test/java/org/apache/oozie/action/hadoop/TestIntegrationGitActionExecutor.java
>  PRE-CREATION 
>   sharelib/pom.xml 6a0864d4e5b52160dc4e1a5cb9dfc34a037b67c8 
>   src/main/assemblies/sharelib.xml 07dc69c8276895b254e5af0cc021cce6ebad18f4 
>   webapp/pom.xml 797996912b6e6381b261a69f8eb1e012fe488fdf 
> 
> 
> Diff: https://reviews.apache.org/r/59620/diff/5/
> 
> 
> Testing
> ---
> 
> Tested using unit and integration tests. Still need to:
> * Test on a cluster
> * Test with an authenticated SSH hosted Git repo
> 
> Sumitted a request to the JGit community as their branch pulling code seems 
> to have an 
> [issue](https://dev.eclipse.org/mhonarc/lists/jgit-dev/msg03343.html).
> 
> Andras has tested against his patch 011 with results at 
> https://issues.apache.org/jira/browse/OOZIE-2877?focusedCommentId=16459523=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-16459523
> 
> 
> File Attachments
> 
> 
> 0001-OOZIE-2877-Oozie-Git-Action.patch
>   
> https://reviews.apache.org/media/uploaded/files/2017/05/29/24f90a78-3dc1-49fe-bf29-5927a3cd5e72__0001-OOZIE-2877-Oozie-Git-Action.patch
> Patch
>   
> https://reviews.apache.org/media/uploaded/files/2017/05/29/dd23dd72-67e0-456f-9b52-e566d8d17d16__0001-OOZIE-2877-Oozie-Git-Action.patch
> 
> 
> Thanks,
> 
> Clay B.
> 
>

Re: Review Request 59620: This review board request is for an action to provide a Git action for Oozie

[Peter Cseh via Review Board]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/59620/#review202124
---




client/src/main/java/org/apache/oozie/cli/OozieCLI.java
Lines 2168-2169 (patched)


Let's start with schema version 1.0



client/src/main/resources/git-action-0.1.xsd
Lines 26-30 (patched)


Plese use the new oozie-common namespace to make launcher config possible 
and use resoure-manager and instead of job-tracker. Take a look at the 
spark-action-1.0.xsd for reference



client/src/main/resources/git-action-0.1.xsd
Lines 32 (patched)


I don't know if it's possible to define uris outside 
oozie.service.HadoopAccessorService.nameNode.whitelist here.

Can you please add a test for that?



sharelib/git/src/main/java/org/apache/oozie/action/hadoop/GitMain.java
Lines 56 (patched)


I don't know if we shoudl provide compatibility to all the way back to 
mapred.job.tracker. I'd prefer to only look for yarn.resourcemanager.address.



sharelib/git/src/main/java/org/apache/oozie/action/hadoop/GitMain.java
Lines 64 (patched)


This looks unneccessary. Where it's used?



sharelib/git/src/main/java/org/apache/oozie/action/hadoop/GitMain.java
Lines 75-82 (patched)


This is not used anywhere if I see it correctly



sharelib/git/src/main/java/org/apache/oozie/action/hadoop/GitMain.java
Lines 137-145 (patched)


I wonder how strict we should be with credential files. We might want to do 
a ssh-like check. Ssh fails if your .pem file is not readably only by your 
user, that's why we're setting the permissions here.

At least we could print out a warning to help users avoid leaking 
credentials to everyone from HDFS

Or we can go ssh-level strict and have an option to do fall back to 
warnings only.



sharelib/git/src/main/java/org/apache/oozie/action/hadoop/GitMain.java
Lines 229-236 (patched)


Please extract a method for creating local tmp folders.


- Peter Cseh


On July 7, 2017, 1:27 a.m., Clay B. wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/59620/
> ---
> 
> (Updated July 7, 2017, 1:27 a.m.)
> 
> 
> Review request for oozie and András Piros.
> 
> 
> Bugs: OOZIE-2877
> https://issues.apache.org/jira/browse/OOZIE-2877
> 
> 
> Repository: oozie-git
> 
> 
> Description
> ---
> 
> OOZIE-2877 - Oozie Git Action
> 
> 
> Diffs
> -
> 
>   client/src/main/java/org/apache/oozie/cli/OozieCLI.java 4adf1a8 
>   client/src/main/resources/git-action-0.1.xsd PRE-CREATION 
>   core/src/main/java/org/apache/oozie/action/hadoop/GitActionExecutor.java 
> PRE-CREATION 
>   core/src/main/resources/oozie-default.xml 5629a89 
>   docs/src/site/twiki/WorkflowFunctionalSpec.twiki 6bd3e5a 
>   pom.xml 16c5137 
>   sharelib/git/pom.xml PRE-CREATION 
>   sharelib/git/src/main/java/org/apache/oozie/action/hadoop/GitMain.java 
> PRE-CREATION 
>   
> sharelib/git/src/test/java/org/apache/oozie/action/hadoop/TestGitActionExecutor.java
>  PRE-CREATION 
>   
> sharelib/git/src/test/java/org/apache/oozie/action/hadoop/TestGitMainGetKey.java
>  PRE-CREATION 
>   
> sharelib/git/src/test/java/org/apache/oozie/action/hadoop/TestIntegrationGitActionExecutor.java
>  PRE-CREATION 
>   sharelib/pom.xml d794246 
>   src/main/assemblies/sharelib.xml 07dc69c 
>   webapp/pom.xml ee6341c 
> 
> 
> Diff: https://reviews.apache.org/r/59620/diff/2/
> 
> 
> Testing
> ---
> 
> Tested using unit and integration tests. Still need to:
> * Test on a cluster
> * Test with an authenticated SSH hosted Git repo
> 
> Sumitted a request to the JGit community as their branch pulling code seems 
> to have an 
> [issue](https://dev.eclipse.org/mhonarc/lists/jgit-dev/msg03343.html).
> 
> 
> File Attachments
> 
> 
> 0001-OOZIE-2877-Oozie-Git-Action.patch
>   
> https://reviews.apache.org/media/uploaded/files/2017/05/29/24f90a78-3dc1-49fe-bf29-5927a3cd5e72__0001-OOZIE-2877-Oozie-Git-Action.patch
> Patch
>   
> https://reviews.apache.org/media/uploaded/files/2017/05/29/dd23dd72-67e0-456f-9b52-e566d8d17d16__0001-OOZIE-2877-Oozie-Git-Action.patch
> 
> 
> Thanks,
> 
> Clay B.
> 
>

Re: Review Request 66084: Oozie-core fails with checkstyle errors.

[Peter Cseh via Review Board]

> On March 19, 2018, 8:37 a.m., Attila Sasvari wrote:
> > core/src/main/resources/jetty-logging.properties
> > Line 1 (original), 1 (patched)
> > 
> >
> > Please leave .properties files intact. ``/**/`` is not a valid comment. 
> > Format: 
> > https://docs.oracle.com/javase/8/docs/api/java/util/Properties.html#load-java.io.Reader-
> 
> Alisha Prabhu wrote:
> Have now kept all the .properties files intact.
> 
> Peter Cseh wrote:
> I think the proper solution here is to don't use CheckStyle for header 
> checking. We have RAT to check the headers.
> Please remove this section from the checkstyle config: 
> https://github.com/apache/oozie/blob/master/src/main/resources/checkstyle.xml#L29-L31
>  and the corresponging file: 
> https://github.com/apache/oozie/blob/master/src/main/resources/checkstyle-header.txt
> 
> Alisha Prabhu wrote:
> Uploaded the patch by making the above changes.

It looks good! I've attached this patch to the Jira so the pre-commit hook can 
kick in.


- Peter


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/66084/#review199384
---


On March 20, 2018, 5:36 a.m., Alisha Prabhu wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/66084/
> ---
> 
> (Updated March 20, 2018, 5:36 a.m.)
> 
> 
> Review request for oozie.
> 
> 
> Bugs: OOZIE-3176
> https://issues.apache.org/jira/browse/OOZIE-3176
> 
> 
> Repository: oozie-git
> 
> 
> Description
> ---
> 
> There are multiple checkstyle violations in the Oozie-core module.
> Maven command used is mvn clean install -DskipTests
> 
> 
> Diffs
> -
> 
>   core/src/main/java/org/apache/oozie/BundleActionBean.java eeec549 
>   core/src/main/java/org/apache/oozie/BundleJobBean.java dec0967 
>   core/src/main/java/org/apache/oozie/CoordinatorActionBean.java ddec1e6 
>   core/src/main/java/org/apache/oozie/CoordinatorJobBean.java 5999f40 
>   core/src/main/java/org/apache/oozie/CoordinatorJobInfo.java 04c5bc9 
>   core/src/main/java/org/apache/oozie/DagELFunctions.java 22ac624 
>   core/src/main/java/org/apache/oozie/SLAEventBean.java b30867b 
>   core/src/main/java/org/apache/oozie/WorkflowActionBean.java 6b05b89 
>   core/src/main/java/org/apache/oozie/WorkflowJobBean.java cb06bec 
>   core/src/main/java/org/apache/oozie/action/hadoop/JavaActionExecutor.java 
> 122dfd0 
>   core/src/main/java/org/apache/oozie/client/rest/JsonBean.java c8bd7c2 
>   
> core/src/main/java/org/apache/oozie/command/bundle/BundleJobSuspendXCommand.java
>  669ac5e 
>   
> core/src/main/java/org/apache/oozie/command/coord/CoordActionInputCheckXCommand.java
>  179cc45 
>   
> core/src/main/java/org/apache/oozie/command/coord/CoordActionStartXCommand.java
>  16d7036 
>   core/src/main/java/org/apache/oozie/command/coord/CoordChangeXCommand.java 
> ff7bf8a 
>   core/src/main/java/org/apache/oozie/command/coord/CoordCommandUtils.java 
> fbc6152 
>   
> core/src/main/java/org/apache/oozie/command/coord/CoordMaterializeTransitionXCommand.java
>  2b91253 
>   
> core/src/main/java/org/apache/oozie/command/coord/CoordPushDependencyCheckXCommand.java
>  1e864ea 
>   core/src/main/java/org/apache/oozie/command/coord/CoordResumeXCommand.java 
> 1998924 
>   core/src/main/java/org/apache/oozie/command/coord/CoordSubmitXCommand.java 
> 42bdc99 
>   core/src/main/java/org/apache/oozie/command/coord/CoordSuspendXCommand.java 
> 1195ef0 
>   core/src/main/java/org/apache/oozie/command/wf/ActionEndXCommand.java 
> 86ee1cc 
>   core/src/main/java/org/apache/oozie/command/wf/ActionKillXCommand.java 
> 61891b8 
>   core/src/main/java/org/apache/oozie/command/wf/ActionStartXCommand.java 
> 71c29cd 
>   core/src/main/java/org/apache/oozie/command/wf/ActionXCommand.java 29ccd32 
>   core/src/main/java/org/apache/oozie/coord/CoordUtils.java 2874bf6 
>   core/src/main/java/org/apache/oozie/coord/HCatELFunctions.java f40f406 
>   
> core/src/main/java/org/apache/oozie/executor/jpa/BundleJobsGetRunningOrPendingJPAExecutor.java
>  18ceb10 
>   
> core/src/main/java/org/apache/oozie/executor/jpa/CoordActionRemoveJPAExecutor.java
>  02a4f68 
>   
> core/src/main/java/org/apache/oozie/executor/jpa/CoordJobGetRunningActionsCountJPAExecutor.java
>  274b3c1 
>   
> core/src/main/java/org/apache/oozie/executor/jpa/WorkflowActionRetryManualGetJPAExecutor.java
>  51449db 
>   
> core/src/main/java/org/apache/oozie/executor/jpa/WorkflowInfoWithActionsSubsetGetJPAExecutor.java
>  43c7d08 
>   
> core/src/main/java/org/apache/oozie/executor/jpa/WorkflowJobsBasicInfoFromParentId.java
>  4b8adfb 
>   core/src/main/java/org/apache/oozie/service/AuthorizationService.java 
> d76e1ea 
>   

Re: Review Request 66262: [OOZIE-2726] Flaky test due to daylight saving changes. Review request #2

[Peter Cseh via Review Board]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/66262/#review199959
---




core/src/test/java/org/apache/oozie/command/coord/TestCoordMaterializeTransitionXCommand.java
Line 468 (original), 479 (patched)


This was 4 hours before. Why is this changed?


- Peter Cseh


On March 24, 2018, 11:53 a.m., András Piros wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/66262/
> ---
> 
> (Updated March 24, 2018, 11:53 a.m.)
> 
> 
> Review request for oozie, Peter Cseh and Satish Saley.
> 
> 
> Repository: oozie-git
> 
> 
> Description
> ---
> 
> [OOZIE-2726] Flaky test due to daylight saving changes. Review request #2.
> 
> Based on the [diff#2](https://reviews.apache.org/r/53775/diff/2#index_header) 
> of the [review request #1](https://reviews.apache.org/r/53775).
> 
> 
> Diffs
> -
> 
>   
> core/src/main/java/org/apache/oozie/command/coord/CoordMaterializeTransitionXCommand.java
>  2b9125387b4e356e408e112c55af7bb3f1c39d3e 
>   
> core/src/main/java/org/apache/oozie/command/coord/DaylightOffsetCalculator.java
>  PRE-CREATION 
>   
> core/src/test/java/org/apache/oozie/command/coord/TestCoordMaterializeTransitionXCommand.java
>  61bbbfe3d1dde89b875c30c75e875efc9227f05d 
>   
> core/src/test/java/org/apache/oozie/command/coord/TestDaylightOffsetCalculator.java
>  PRE-CREATION 
> 
> 
> Diff: https://reviews.apache.org/r/66262/diff/2/
> 
> 
> Testing
> ---
> 
> `TestCoordMaterializeTransitionXCommand`, `TestDaylightOffsetCalculator`
> 
> 
> Thanks,
> 
> András Piros
> 
>

Re: Review Request 66262: [OOZIE-2726] Flaky test due to daylight saving changes. Review request #2

[Peter Cseh via Review Board]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/66262/#review199958
---




core/src/main/java/org/apache/oozie/command/coord/DaylightOffsetCalculator.java
Lines 54-58 (patched)


Isn't this handled in getDSTOffset as part of returning 0 there? I don't 
think the performance would require this additional logic, but it makes things 
a bit confusing


- Peter Cseh


On March 24, 2018, 11:53 a.m., András Piros wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/66262/
> ---
> 
> (Updated March 24, 2018, 11:53 a.m.)
> 
> 
> Review request for oozie, Peter Cseh and Satish Saley.
> 
> 
> Repository: oozie-git
> 
> 
> Description
> ---
> 
> [OOZIE-2726] Flaky test due to daylight saving changes. Review request #2.
> 
> Based on the [diff#2](https://reviews.apache.org/r/53775/diff/2#index_header) 
> of the [review request #1](https://reviews.apache.org/r/53775).
> 
> 
> Diffs
> -
> 
>   
> core/src/main/java/org/apache/oozie/command/coord/CoordMaterializeTransitionXCommand.java
>  2b9125387b4e356e408e112c55af7bb3f1c39d3e 
>   
> core/src/main/java/org/apache/oozie/command/coord/DaylightOffsetCalculator.java
>  PRE-CREATION 
>   
> core/src/test/java/org/apache/oozie/command/coord/TestCoordMaterializeTransitionXCommand.java
>  61bbbfe3d1dde89b875c30c75e875efc9227f05d 
>   
> core/src/test/java/org/apache/oozie/command/coord/TestDaylightOffsetCalculator.java
>  PRE-CREATION 
> 
> 
> Diff: https://reviews.apache.org/r/66262/diff/2/
> 
> 
> Testing
> ---
> 
> `TestCoordMaterializeTransitionXCommand`, `TestDaylightOffsetCalculator`
> 
> 
> Thanks,
> 
> András Piros
> 
>

Re: Review Request 66250: [OOZIE-2645] Deprecate Instrumentation in favor of Metrics

[Peter Cseh via Review Board]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/66250/#review199953
---




docs/src/site/twiki/AG_Install.twiki
Lines 919-921 (original), 927-937 (patched)


Shouldn't it be enabled by adding the deprecated Service to the 
services.ext?



docs/src/site/twiki/AG_Install.twiki
Lines 919-921 (original), 927-937 (patched)


Shouldn't it be enabled by adding the deprecated Service to the 
services.ext?


- Peter Cseh


On March 24, 2018, 6:15 p.m., András Piros wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/66250/
> ---
> 
> (Updated March 24, 2018, 6:15 p.m.)
> 
> 
> Review request for oozie, Peter Cseh and Robert Kanter.
> 
> 
> Repository: oozie-git
> 
> 
> Description
> ---
> 
> [OOZIE-2645] Deprecate Instrumentation in favor of Metrics
> 
> 
> Diffs
> -
> 
>   core/src/main/java/org/apache/oozie/service/InstrumentationService.java 
> afa35113c7fc258d0da75bdbf183fc6f2a1e54ee 
>   core/src/main/java/org/apache/oozie/util/Instrumentation.java 
> a57b6655da6a6c28eb857691336ab27508ee8092 
>   core/src/main/java/org/apache/oozie/util/MetricsInstrumentation.java 
> 1288a3f980435164c38b09edbbbcad51577e694d 
>   core/src/main/resources/oozie-default.xml 
> 966dd29b20ffb4a627418d62403c0cec166dd147 
>   core/src/test/java/org/apache/oozie/service/TestInstrumentationService.java 
> b31816e15f5f419e52042187931674fbd3cd4c5b 
>   core/src/test/java/org/apache/oozie/servlet/TestAdminServlet.java 
> f91d22f668b14bde7437e9a3f854dd87a5565e00 
>   core/src/test/java/org/apache/oozie/servlet/TestV1AdminServlet.java 
> 534f5727ce398d6dbf8473bf4442b76e7c2630c4 
>   core/src/test/java/org/apache/oozie/util/TestMetricsInstrumentation.java 
> ab974436769b786d99d10fc3aa951c2a3b16d398 
>   core/src/test/resources/oozie-site-instrumentation.xml 
> ab3f0f5b99aabf149d409c762efd3feb51fe9f6d 
>   docs/src/site/twiki/AG_Install.twiki 
> 8e2d795da838d184940bc177e95e9dfb8b52cc54 
>   docs/src/site/twiki/AG_Monitoring.twiki 
> 523f85133055b80b32b2cf73f1e8c25751cf9fc4 
>   docs/src/site/twiki/DG_CommandLineTool.twiki 
> a94201ded8451cc725b40e9f73182ecaa52be88a 
>   docs/src/site/twiki/WebServicesAPI.twiki 
> c662a5eefdfc012ea6c13256a3df426e1a073965 
> 
> 
> Diff: https://reviews.apache.org/r/66250/diff/3/
> 
> 
> Testing
> ---
> 
> Generated `mkdistro` and saw that `/admin/v2/metrics` is available, and 
> `/admin/instrumentation` isn't.
> 
> 
> Thanks,
> 
> András Piros
> 
>

Re: Review Request 66181: OOZIE-3056 Implement new mechanism to specify ShareLibs for workflow actions

[Peter Cseh via Review Board]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/66181/
---

(Updated March 23, 2018, 7:47 a.m.)


Review request for oozie, András Piros and Attila Sasvari.


Bugs: OOZIE-3056
https://issues.apache.org/jira/browse/OOZIE-3056


Repository: oozie-git


Description
---

Refactored out Sharelib name resolution from JavaAE as an additional step to 
pick that giant class to pieces.
Added logic to process the  properties added in  sections.


Diffs (updated)
-

  core/src/main/java/org/apache/oozie/action/hadoop/JavaActionExecutor.java 
122dfd02e17456b9aab924504faee790813fb6a1 
  core/src/main/java/org/apache/oozie/action/hadoop/SharelibResolver.java 
PRE-CREATION 
  core/src/test/java/org/apache/oozie/action/hadoop/TestJavaActionExecutor.java 
b55a3cd7f8064cd898edc686891c9f6ebb118c42 
  core/src/test/java/org/apache/oozie/action/hadoop/TestSharelibConfigs.java 
PRE-CREATION 
  core/src/test/java/org/apache/oozie/action/hadoop/TestSharelibResolver.java 
PRE-CREATION 
  core/src/test/java/org/apache/oozie/action/hadoop/TestWorkflowHelper.java 
PRE-CREATION 


Diff: https://reviews.apache.org/r/66181/diff/4/

Changes: https://reviews.apache.org/r/66181/diff/3-4/


Testing
---

Added Junit tests to validate that 
1) the configs are propagating and overwriting each other correctly when an xml 
is formatted
2) the configs are resolved in the correct order for sharelib names


Thanks,

Peter Cseh

Re: Review Request 66181: OOZIE-3056 Implement new mechanism to specify ShareLibs for workflow actions

[Peter Cseh via Review Board]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/66181/
---

(Updated March 22, 2018, 2:07 p.m.)


Review request for oozie, András Piros and Attila Sasvari.


Bugs: OOZIE-3056
https://issues.apache.org/jira/browse/OOZIE-3056


Repository: oozie-git


Description
---

Refactored out Sharelib name resolution from JavaAE as an additional step to 
pick that giant class to pieces.
Added logic to process the  properties added in  sections.


Diffs (updated)
-

  core/src/main/java/org/apache/oozie/action/hadoop/JavaActionExecutor.java 
122dfd02e17456b9aab924504faee790813fb6a1 
  core/src/main/java/org/apache/oozie/action/hadoop/SharelibResolver.java 
PRE-CREATION 
  core/src/test/java/org/apache/oozie/action/hadoop/TestJavaActionExecutor.java 
b55a3cd7f8064cd898edc686891c9f6ebb118c42 
  core/src/test/java/org/apache/oozie/action/hadoop/TestSharelibConfigs.java 
PRE-CREATION 
  core/src/test/java/org/apache/oozie/action/hadoop/TestSharelibResolver.java 
PRE-CREATION 
  core/src/test/java/org/apache/oozie/action/hadoop/TestWorkflowHelper.java 
PRE-CREATION 
  docs/src/site/twiki/AG_Install.twiki 8e2d795da838d184940bc177e95e9dfb8b52cc54 
  docs/src/site/twiki/DG_DistCpActionExtension.twiki 
8bab3da448f5491164406860309cfce76b85e464 
  docs/src/site/twiki/DG_Examples.twiki 
13dfa28c0eecab70321886b2806a791346c46d3b 
  docs/src/site/twiki/DG_Hive2ActionExtension.twiki 
efbe56dabdd7e54f56d2351694ed3b5d722c902b 
  docs/src/site/twiki/DG_HiveActionExtension.twiki 
aaa74fa7b7046394c265128c4517ba667508b0f0 
  docs/src/site/twiki/DG_ShellActionExtension.twiki 
5894c280a52b28e7507b2b5f078a0cc6d65cf93c 
  docs/src/site/twiki/DG_SparkActionExtension.twiki 
66076ff9d3098aeb06301458b900da8d7f5fb5db 
  docs/src/site/twiki/DG_SqoopActionExtension.twiki 
0317d0ceccf065a954976dcf6ef1cf995a169d14 
  docs/src/site/twiki/DG_SshActionExtension.twiki 
99b96f7fa398de9a1771359894f46352e9dbd9cd 
  docs/src/site/twiki/ENG_MiniOozie.twiki 
0b1628986bf31a4a0874be7539fae0af57e51bd0 
  docs/src/site/twiki/WorkflowFunctionalSpec.twiki 
5df6024bf22c56f1fffee6487d18157c8485d520 
  release-log.txt 76401f5eb92cdea16bc8e33aa92f3d8e5702a28a 


Diff: https://reviews.apache.org/r/66181/diff/2/

Changes: https://reviews.apache.org/r/66181/diff/1-2/


Testing
---

Added Junit tests to validate that 
1) the configs are propagating and overwriting each other correctly when an xml 
is formatted
2) the configs are resolved in the correct order for sharelib names


Thanks,

Peter Cseh

Re: Review Request 66181: OOZIE-3056 Implement new mechanism to specify ShareLibs for workflow actions

[Peter Cseh via Review Board]

> On March 21, 2018, 1:28 p.m., András Piros wrote:
> > core/src/main/java/org/apache/oozie/action/hadoop/JavaActionExecutor.java
> > Lines 1862-1875 (original), 1846-1848 (patched)
> > 
> >
> > Remove `catch` block as it's already handled by 
> > `SharelibResolver#resolve()`.

It turns out that I don't need the try-catch in the Resolver.


> On March 21, 2018, 1:28 p.m., András Piros wrote:
> > core/src/main/java/org/apache/oozie/action/hadoop/SharelibResolver.java
> > Lines 76-78 (patched)
> > 
> >
> > What about `[""]`, that is, a `String[]` consisting of only empty 
> > `String` instances?

conf.getStrings() does not give back empty strings in the arrays.


> On March 21, 2018, 1:28 p.m., András Piros wrote:
> > core/src/test/java/org/apache/oozie/action/hadoop/TestJavaActionExecutor.java
> > Lines 1251-1265 (patched)
> > 
> >
> > What about a new test class that `extends MiniOozieTestCase` instead? 
> > `TestJavaActionExecutor` is lengthy enough already.

Done.


- Peter


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/66181/#review199656
---


On March 20, 2018, 9:04 p.m., Peter Cseh wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/66181/
> ---
> 
> (Updated March 20, 2018, 9:04 p.m.)
> 
> 
> Review request for oozie, András Piros and Attila Sasvari.
> 
> 
> Bugs: OOZIE-3056
> https://issues.apache.org/jira/browse/OOZIE-3056
> 
> 
> Repository: oozie-git
> 
> 
> Description
> ---
> 
> Refactored out Sharelib name resolution from JavaAE as an additional step to 
> pick that giant class to pieces.
> Added logic to process the  properties added in  sections.
> 
> 
> Diffs
> -
> 
>   core/src/main/java/org/apache/oozie/action/hadoop/JavaActionExecutor.java 
> 122dfd02e17456b9aab924504faee790813fb6a1 
>   core/src/main/java/org/apache/oozie/action/hadoop/SharelibResolver.java 
> PRE-CREATION 
>   
> core/src/test/java/org/apache/oozie/action/hadoop/TestJavaActionExecutor.java 
> b55a3cd7f8064cd898edc686891c9f6ebb118c42 
>   core/src/test/java/org/apache/oozie/action/hadoop/TestSharelibResolver.java 
> PRE-CREATION 
> 
> 
> Diff: https://reviews.apache.org/r/66181/diff/1/
> 
> 
> Testing
> ---
> 
> Added Junit tests to validate that 
> 1) the configs are propagating and overwriting each other correctly when an 
> xml is formatted
> 2) the configs are resolved in the correct order for sharelib names
> 
> 
> Thanks,
> 
> Peter Cseh
> 
>

Review Request 66181: OOZIE-3056 Implement new mechanism to specify ShareLibs for workflow actions

[Peter Cseh via Review Board]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/66181/
---

Review request for oozie, András Piros and Attila Sasvari.


Bugs: OOZIE-3056
https://issues.apache.org/jira/browse/OOZIE-3056


Repository: oozie-git


Description
---

Refactored out Sharelib name resolution from JavaAE as an additional step to 
pick that giant class to pieces.
Added logic to process the  properties added in  sections.


Diffs
-

  core/src/main/java/org/apache/oozie/action/hadoop/JavaActionExecutor.java 
122dfd02e17456b9aab924504faee790813fb6a1 
  core/src/main/java/org/apache/oozie/action/hadoop/SharelibResolver.java 
PRE-CREATION 
  core/src/test/java/org/apache/oozie/action/hadoop/TestJavaActionExecutor.java 
b55a3cd7f8064cd898edc686891c9f6ebb118c42 
  core/src/test/java/org/apache/oozie/action/hadoop/TestSharelibResolver.java 
PRE-CREATION 


Diff: https://reviews.apache.org/r/66181/diff/1/


Testing
---

Added Junit tests to validate that 
1) the configs are propagating and overwriting each other correctly when an xml 
is formatted
2) the configs are resolved in the correct order for sharelib names


Thanks,

Peter Cseh

Re: Review Request 66149: OOZIE-2600 OYA: Update Documentation

[Peter Cseh via Review Board]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/66149/#review199552
---




docs/src/site/twiki/WorkflowFunctionalSpec.twiki
Line 8 (original), 8 (patched)


Please update the version here as well to 5.0


- Peter Cseh


On March 20, 2018, 2:48 p.m., András Piros wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/66149/
> ---
> 
> (Updated March 20, 2018, 2:48 p.m.)
> 
> 
> Review request for oozie, Peter Cseh and Robert Kanter.
> 
> 
> Repository: oozie-git
> 
> 
> Description
> ---
> 
> OOZIE-2600 OYA: Update Documentation
> 
> 
> Diffs
> -
> 
>   docs/src/site/twiki/AG_Install.twiki 
> 15042151f11666517f328f700cca022375bad8ae 
>   docs/src/site/twiki/DG_DistCpActionExtension.twiki 
> 260cd2578e7a4d78067512461fac72f47d32e7cb 
>   docs/src/site/twiki/DG_Examples.twiki 
> 87cac48bcb7bfb2c6628ad33429a036d0d2b676e 
>   docs/src/site/twiki/DG_Hive2ActionExtension.twiki 
> 094307fd81e57f51c25e6c6e30a2c7564a2bba76 
>   docs/src/site/twiki/DG_HiveActionExtension.twiki 
> a0d7b578ba0dd30fa8cd8c0b8a6c796c20e78c93 
>   docs/src/site/twiki/DG_ShellActionExtension.twiki 
> 493254a6f5d77eaa3dfa56ba39047e0847fa15ee 
>   docs/src/site/twiki/DG_SparkActionExtension.twiki 
> 294e7dd566b8849742e42962ce164c71b9caaed9 
>   docs/src/site/twiki/DG_SqoopActionExtension.twiki 
> 906ba5476f7628b3bf70dc3d6d5ef4a48418ea94 
>   docs/src/site/twiki/DG_SshActionExtension.twiki 
> c2e33c90c154df1e978e09c1534474a107191cd5 
>   docs/src/site/twiki/ENG_MiniOozie.twiki 
> 2c397264e1d758bae0c0590289d329a82fa3fd8c 
>   docs/src/site/twiki/WorkflowFunctionalSpec.twiki 
> 038c430d37887617070f2cb13dd0eba0962600d2 
> 
> 
> Diff: https://reviews.apache.org/r/66149/diff/3/
> 
> 
> Testing
> ---
> 
> This is a documentation-only task, hence no testing except for generating the 
> documentation, and checking, is done:
> ```
> mvn clean install -DskipTests -Denforcer.skip=true -Dcheckstyle.skip=true 
> -Dfindbugs.skip=true -PgenerateDocs
> ```
> 
> 
> Thanks,
> 
> András Piros
> 
>

Re: Review Request 66149: OOZIE-2600 OYA: Update Documentation

[Peter Cseh via Review Board]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/66149/#review199531
---




docs/src/site/twiki/AG_Install.twiki
Lines 307 (patched)


We should mention that it's on by default



docs/src/site/twiki/AG_Install.twiki
Lines 325-326 (patched)


We should mention that the list can be extended in oozie-site.xml and there 
are examples for this in oozie-default.xml



docs/src/site/twiki/AG_Install.twiki
Lines 329 (patched)


oozie.launcher.prepend, isn't it?
Also, we should mention the default value


- Peter Cseh


On March 19, 2018, 8:17 p.m., András Piros wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/66149/
> ---
> 
> (Updated March 19, 2018, 8:17 p.m.)
> 
> 
> Review request for oozie, Peter Cseh and Robert Kanter.
> 
> 
> Repository: oozie-git
> 
> 
> Description
> ---
> 
> OOZIE-2600 OYA: Update Documentation
> 
> 
> Diffs
> -
> 
>   docs/src/site/twiki/AG_Install.twiki 
> 15042151f11666517f328f700cca022375bad8ae 
>   docs/src/site/twiki/DG_DistCpActionExtension.twiki 
> 260cd2578e7a4d78067512461fac72f47d32e7cb 
>   docs/src/site/twiki/DG_Examples.twiki 
> 87cac48bcb7bfb2c6628ad33429a036d0d2b676e 
>   docs/src/site/twiki/DG_Hive2ActionExtension.twiki 
> 094307fd81e57f51c25e6c6e30a2c7564a2bba76 
>   docs/src/site/twiki/DG_HiveActionExtension.twiki 
> a0d7b578ba0dd30fa8cd8c0b8a6c796c20e78c93 
>   docs/src/site/twiki/DG_ShellActionExtension.twiki 
> 493254a6f5d77eaa3dfa56ba39047e0847fa15ee 
>   docs/src/site/twiki/DG_SparkActionExtension.twiki 
> 294e7dd566b8849742e42962ce164c71b9caaed9 
>   docs/src/site/twiki/DG_SqoopActionExtension.twiki 
> 906ba5476f7628b3bf70dc3d6d5ef4a48418ea94 
>   docs/src/site/twiki/DG_SshActionExtension.twiki 
> c2e33c90c154df1e978e09c1534474a107191cd5 
>   docs/src/site/twiki/ENG_MiniOozie.twiki 
> 2c397264e1d758bae0c0590289d329a82fa3fd8c 
>   docs/src/site/twiki/WorkflowFunctionalSpec.twiki 
> 038c430d37887617070f2cb13dd0eba0962600d2 
> 
> 
> Diff: https://reviews.apache.org/r/66149/diff/1/
> 
> 
> Testing
> ---
> 
> This is a documentation-only task, hence no testing except for generating the 
> documentation, and checking, is done:
> ```
> mvn clean install -DskipTests -Denforcer.skip=true -Dcheckstyle.skip=true 
> -Dfindbugs.skip=true -PgenerateDocs
> ```
> 
> 
> Thanks,
> 
> András Piros
> 
>

Re: Review Request 66084: Oozie-core fails with checkstyle errors.

[Peter Cseh via Review Board]

> On March 19, 2018, 8:37 a.m., Attila Sasvari wrote:
> > core/src/main/resources/jetty-logging.properties
> > Line 1 (original), 1 (patched)
> > 
> >
> > Please leave .properties files intact. ``/**/`` is not a valid comment. 
> > Format: 
> > https://docs.oracle.com/javase/8/docs/api/java/util/Properties.html#load-java.io.Reader-
> 
> Alisha Prabhu wrote:
> Have now kept all the .properties files intact.

I think the proper solution here is to don't use CheckStyle for header 
checking. We have RAT to check the headers.
Please remove this section from the checkstyle config: 
https://github.com/apache/oozie/blob/master/src/main/resources/checkstyle.xml#L29-L31
 and the corresponging file: 
https://github.com/apache/oozie/blob/master/src/main/resources/checkstyle-header.txt


- Peter


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/66084/#review199384
---


On March 19, 2018, 12:32 p.m., Alisha Prabhu wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/66084/
> ---
> 
> (Updated March 19, 2018, 12:32 p.m.)
> 
> 
> Review request for oozie.
> 
> 
> Bugs: OOZIE-3176
> https://issues.apache.org/jira/browse/OOZIE-3176
> 
> 
> Repository: oozie-git
> 
> 
> Description
> ---
> 
> There are multiple checkstyle violations in the Oozie-core module.
> Maven command used is mvn clean install -DskipTests
> 
> 
> Diffs
> -
> 
>   core/src/main/java/org/apache/oozie/BundleActionBean.java eeec549 
>   core/src/main/java/org/apache/oozie/BundleJobBean.java dec0967 
>   core/src/main/java/org/apache/oozie/CoordinatorActionBean.java ddec1e6 
>   core/src/main/java/org/apache/oozie/CoordinatorJobBean.java 5999f40 
>   core/src/main/java/org/apache/oozie/CoordinatorJobInfo.java 04c5bc9 
>   core/src/main/java/org/apache/oozie/DagELFunctions.java 22ac624 
>   core/src/main/java/org/apache/oozie/SLAEventBean.java b30867b 
>   core/src/main/java/org/apache/oozie/WorkflowActionBean.java 6b05b89 
>   core/src/main/java/org/apache/oozie/WorkflowJobBean.java cb06bec 
>   core/src/main/java/org/apache/oozie/action/hadoop/JavaActionExecutor.java 
> 122dfd0 
>   core/src/main/java/org/apache/oozie/client/rest/JsonBean.java c8bd7c2 
>   
> core/src/main/java/org/apache/oozie/command/bundle/BundleJobSuspendXCommand.java
>  669ac5e 
>   
> core/src/main/java/org/apache/oozie/command/coord/CoordActionInputCheckXCommand.java
>  179cc45 
>   
> core/src/main/java/org/apache/oozie/command/coord/CoordActionStartXCommand.java
>  16d7036 
>   core/src/main/java/org/apache/oozie/command/coord/CoordChangeXCommand.java 
> ff7bf8a 
>   core/src/main/java/org/apache/oozie/command/coord/CoordCommandUtils.java 
> fbc6152 
>   
> core/src/main/java/org/apache/oozie/command/coord/CoordMaterializeTransitionXCommand.java
>  2b91253 
>   
> core/src/main/java/org/apache/oozie/command/coord/CoordPushDependencyCheckXCommand.java
>  1e864ea 
>   core/src/main/java/org/apache/oozie/command/coord/CoordResumeXCommand.java 
> 1998924 
>   core/src/main/java/org/apache/oozie/command/coord/CoordSubmitXCommand.java 
> 42bdc99 
>   core/src/main/java/org/apache/oozie/command/coord/CoordSuspendXCommand.java 
> 1195ef0 
>   core/src/main/java/org/apache/oozie/command/wf/ActionEndXCommand.java 
> 86ee1cc 
>   core/src/main/java/org/apache/oozie/command/wf/ActionKillXCommand.java 
> 61891b8 
>   core/src/main/java/org/apache/oozie/command/wf/ActionStartXCommand.java 
> 71c29cd 
>   core/src/main/java/org/apache/oozie/command/wf/ActionXCommand.java 29ccd32 
>   core/src/main/java/org/apache/oozie/coord/CoordUtils.java 2874bf6 
>   core/src/main/java/org/apache/oozie/coord/HCatELFunctions.java f40f406 
>   
> core/src/main/java/org/apache/oozie/executor/jpa/BundleJobsGetRunningOrPendingJPAExecutor.java
>  18ceb10 
>   
> core/src/main/java/org/apache/oozie/executor/jpa/CoordActionRemoveJPAExecutor.java
>  02a4f68 
>   
> core/src/main/java/org/apache/oozie/executor/jpa/CoordJobGetRunningActionsCountJPAExecutor.java
>  274b3c1 
>   
> core/src/main/java/org/apache/oozie/executor/jpa/WorkflowActionRetryManualGetJPAExecutor.java
>  51449db 
>   
> core/src/main/java/org/apache/oozie/executor/jpa/WorkflowInfoWithActionsSubsetGetJPAExecutor.java
>  43c7d08 
>   
> core/src/main/java/org/apache/oozie/executor/jpa/WorkflowJobsBasicInfoFromParentId.java
>  4b8adfb 
>   core/src/main/java/org/apache/oozie/service/AuthorizationService.java 
> d76e1ea 
>   core/src/main/java/org/apache/oozie/service/PurgeService.java c2c966b 
>   core/src/main/java/org/apache/oozie/servlet/JsonRestServlet.java 404d3cd 
>   

Re: Review Request 65606: OOZIE-2814 OYA: Update example workflows to newest schemas

[Peter Cseh via Review Board]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/65606/#review197498
---




core/src/main/java/org/apache/oozie/workflow/lite/LiteWorkflowAppParser.java
Line 109 (original), 109 (patched)


Please deprecate this constant



core/src/main/java/org/apache/oozie/workflow/lite/LiteWorkflowAppParser.java
Line 123 (original), 126 (patched)


Please remove defaultJobTracker and use the defaultRM only



core/src/main/resources/oozie-default.xml
Lines 3035-3042 (original), 3035-3042 (patched)


I think this could be removed from oozie-default.xml as part of this cleanup


- Peter Cseh


On Feb. 12, 2018, 4:19 p.m., Attila Sasvari wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/65606/
> ---
> 
> (Updated Feb. 12, 2018, 4:19 p.m.)
> 
> 
> Review request for oozie.
> 
> 
> Repository: oozie-git
> 
> 
> Description
> ---
> 
> OOZIE-2687 will add in a whole set of newer schemas that are more 
> Yarn-centric. Most, if not all, examples currently use pretty old versions of 
> the schemas. We should take this opportunity to update the examples to the 
> newest schemas added by OOZIE-2687, which should help demonstrate them.
> 
> It may make sense to keep the older schemas too, to show that older workflows 
> still work. We have some examples where we have two workflow XML files that 
> do the same thing in different ways.
> 
> 
> Diffs
> -
> 
>   core/src/main/java/org/apache/oozie/action/hadoop/JavaActionExecutor.java 
> bc0f4050bb3788988d3a5352a15220fbc4fa219d 
>   
> core/src/main/java/org/apache/oozie/workflow/lite/LiteWorkflowAppParser.java 
> c236dafa5965403c33e1a400509e7c374fe2637c 
>   core/src/main/resources/oozie-default.xml 
> e53b17b3cf201b4cf7bc5371503e66327eae2126 
>   
> core/src/test/java/org/apache/oozie/action/hadoop/TestJavaActionExecutor.java 
> 1cc569d8d7a0e61905ad6c4e940cefb00da1343d 
>   
> core/src/test/java/org/apache/oozie/workflow/lite/TestLiteWorkflowAppParser.java
>  2cdbde4d2fc86a07b35beadc56012c438baf8d54 
>   examples/src/main/apps/aggregator/coordinator-with-offset.xml 
> 4c351677f4afc6e85f807106d426bb2f922a28b1 
>   examples/src/main/apps/aggregator/coordinator.xml 
> 948edc1adbe09dfb6f7bffb5eebc82f3670658cc 
>   examples/src/main/apps/aggregator/job-with-offset.properties 
> 9a28ae13e50c40f70db4b0f583a63eb42f97388d 
>   examples/src/main/apps/aggregator/job.properties 
> aaf4edf7b9d030b5044610698b27a564b008f77b 
>   examples/src/main/apps/aggregator/workflow.xml 
> 4a801090e31acba23ea49da1c59a8ce4b951c654 
>   examples/src/main/apps/bundle/job.properties 
> c06edeb3f42e2eba3367b9486982dabb4caf6436 
>   examples/src/main/apps/coord-input-logic/job.properties 
> 2cd137b94999c7e1f68bb54928f1c3ce2838d59d 
>   examples/src/main/apps/coord-input-logic/workflow.xml 
> 80d31457eff6e78710ad1fb4d4f1b372c0d1e43c 
>   examples/src/main/apps/cron-schedule/coordinator.xml 
> 48a7de76245e02397b9736a95fa0132504067f9a 
>   examples/src/main/apps/cron-schedule/job.properties 
> 5b5eb43ca2bf9d8cd1b263ca3b94a552b562a20d 
>   examples/src/main/apps/cron-schedule/workflow.xml 
> b53922c8f40e691cb637b91cb182172794296a04 
>   examples/src/main/apps/cron/coordinator.xml 
> 744d14e870b59d0b89f65a78fea63fffa3ff4b8e 
>   examples/src/main/apps/cron/job.properties 
> 42875fd5a5ce393cd11717bcc55fc1bb91e0fcd3 
>   examples/src/main/apps/cron/workflow.xml 
> 63dff976f9e35a26b0ad09078778748d3d02bc3b 
>   examples/src/main/apps/custom-main/job.properties 
> b897512a1c6246308ca17b019ce400f3ec06cd7b 
>   examples/src/main/apps/custom-main/workflow.xml 
> 828dfaa0594aac0539c4e08ae8e37cb89d17eb54 
>   examples/src/main/apps/datelist-java-main/job.properties 
> e8356e3a4a2e1ae41e09f5ea124fb5360bd6e8b6 
>   examples/src/main/apps/datelist-java-main/workflow.xml 
> b59a12ebd1fc21c9e26f6a5b0348eb9c76375c03 
>   examples/src/main/apps/demo/job.properties 
> d70d3208df77e0b612005196907cecd97d7c5d21 
>   examples/src/main/apps/demo/job.windows.properties 
> ea948b20009685e1286053549fc1974669819682 
>   examples/src/main/apps/demo/workflow.xml 
> 0ceca507adc68c0674b794420c9eb733cb320879 
>   examples/src/main/apps/distcp/job.properties 
> ec9422caddcdecfa6ee9a58868cf68d7457ad4db 
>   examples/src/main/apps/distcp/workflow.xml 
> 6deecb377901613a5bb28fc18bf5beff2d30f252 
>   examples/src/main/apps/hadoop-el/job.properties 
> 1d13c27056ac58ad359b5298f12ec046bb4cc61d 
>   examples/src/main/apps/hadoop-el/workflow.xml 
> d4094d44136a7de2f85f5c9be042d5aab24b90be 
>   examples/src/main/apps/hcatalog/job.properties 
> 

Re: Review Request 65636: OOZIE-3179 - Adding a configurable config-default.xml location to a workflow

[Peter Cseh via Review Board]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/65636/#review197491
---




core/src/test/java/org/apache/oozie/command/wf/TestSubmitXCommand.java
Line 423 (original), 506-518 (patched)


I don't see the case where the same property is set in 
extra-config-default.xml and config-default.xml as well and config-default.xml 
takes precedence


- Peter Cseh


On Feb. 13, 2018, 3:07 p.m., Jason Phelps wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/65636/
> ---
> 
> (Updated Feb. 13, 2018, 3:07 p.m.)
> 
> 
> Review request for oozie.
> 
> 
> Repository: oozie-git
> 
> 
> Description
> ---
> 
> Added another client property: `oozie.default.configuration.path` which 
> allows for a comma seperated list of HDFS XML property files to be applied to 
> the job. This is useful if multiple workflows have shared properties (i.e. 
> database names, table names for Sqoop or Hive, spark arguments, etc). This is 
> similar to the `oozie.libpath` property which allows user's to define common 
> paths workflows to load jars.
> 
> 
> Diffs
> -
> 
>   client/src/main/java/org/apache/oozie/client/OozieClient.java e581e50 
>   core/src/main/java/org/apache/oozie/command/wf/SubmitXCommand.java a4d3e2f 
>   core/src/test/java/org/apache/oozie/command/wf/TestSubmitXCommand.java 
> e71a662 
> 
> 
> Diff: https://reviews.apache.org/r/65636/diff/1/
> 
> 
> Testing
> ---
> 
> 
> Thanks,
> 
> Jason Phelps
> 
>

Re: Review Request 65606: OOZIE-2814 OYA: Update example workflows to newest schemas

[Peter Cseh via Review Board]

> On Feb. 12, 2018, 8:05 p.m., András Piros wrote:
> > core/src/test/java/org/apache/oozie/workflow/lite/TestLiteWorkflowAppParser.java
> > Lines 1680 (patched)
> > 
> >
> > Would give a catchier name and remove comment.
> > 
> > What about more test cases:
> > 
> > * both `resource-manager` and `job-tracker` present at the same level
> > * both `resource-manager` and `job-tracker` present at different levels
> > * none of these are present

I think it's better to keep both RM and JT as a single variable in the code as 
they have the same meaning. JT is only there for compatiblity reasons, but it 
should overwrite RM values of lower levels.


- Peter


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/65606/#review197301
---


On Feb. 12, 2018, 4:19 p.m., Attila Sasvari wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/65606/
> ---
> 
> (Updated Feb. 12, 2018, 4:19 p.m.)
> 
> 
> Review request for oozie.
> 
> 
> Repository: oozie-git
> 
> 
> Description
> ---
> 
> OOZIE-2687 will add in a whole set of newer schemas that are more 
> Yarn-centric. Most, if not all, examples currently use pretty old versions of 
> the schemas. We should take this opportunity to update the examples to the 
> newest schemas added by OOZIE-2687, which should help demonstrate them.
> 
> It may make sense to keep the older schemas too, to show that older workflows 
> still work. We have some examples where we have two workflow XML files that 
> do the same thing in different ways.
> 
> 
> Diffs
> -
> 
>   core/src/main/java/org/apache/oozie/action/hadoop/JavaActionExecutor.java 
> bc0f4050bb3788988d3a5352a15220fbc4fa219d 
>   
> core/src/main/java/org/apache/oozie/workflow/lite/LiteWorkflowAppParser.java 
> c236dafa5965403c33e1a400509e7c374fe2637c 
>   
> core/src/test/java/org/apache/oozie/action/hadoop/TestJavaActionExecutor.java 
> 1cc569d8d7a0e61905ad6c4e940cefb00da1343d 
>   
> core/src/test/java/org/apache/oozie/workflow/lite/TestLiteWorkflowAppParser.java
>  2cdbde4d2fc86a07b35beadc56012c438baf8d54 
>   examples/src/main/apps/aggregator/coordinator-with-offset.xml 
> 4c351677f4afc6e85f807106d426bb2f922a28b1 
>   examples/src/main/apps/aggregator/coordinator.xml 
> 948edc1adbe09dfb6f7bffb5eebc82f3670658cc 
>   examples/src/main/apps/aggregator/job-with-offset.properties 
> 9a28ae13e50c40f70db4b0f583a63eb42f97388d 
>   examples/src/main/apps/aggregator/job.properties 
> aaf4edf7b9d030b5044610698b27a564b008f77b 
>   examples/src/main/apps/aggregator/workflow.xml 
> 4a801090e31acba23ea49da1c59a8ce4b951c654 
>   examples/src/main/apps/bundle/job.properties 
> c06edeb3f42e2eba3367b9486982dabb4caf6436 
>   examples/src/main/apps/coord-input-logic/job.properties 
> 2cd137b94999c7e1f68bb54928f1c3ce2838d59d 
>   examples/src/main/apps/coord-input-logic/workflow.xml 
> 80d31457eff6e78710ad1fb4d4f1b372c0d1e43c 
>   examples/src/main/apps/cron-schedule/coordinator.xml 
> 48a7de76245e02397b9736a95fa0132504067f9a 
>   examples/src/main/apps/cron-schedule/job.properties 
> 5b5eb43ca2bf9d8cd1b263ca3b94a552b562a20d 
>   examples/src/main/apps/cron-schedule/workflow.xml 
> b53922c8f40e691cb637b91cb182172794296a04 
>   examples/src/main/apps/cron/coordinator.xml 
> 744d14e870b59d0b89f65a78fea63fffa3ff4b8e 
>   examples/src/main/apps/cron/job.properties 
> 42875fd5a5ce393cd11717bcc55fc1bb91e0fcd3 
>   examples/src/main/apps/cron/workflow.xml 
> 63dff976f9e35a26b0ad09078778748d3d02bc3b 
>   examples/src/main/apps/custom-main/job.properties 
> b897512a1c6246308ca17b019ce400f3ec06cd7b 
>   examples/src/main/apps/custom-main/workflow.xml 
> 828dfaa0594aac0539c4e08ae8e37cb89d17eb54 
>   examples/src/main/apps/datelist-java-main/job.properties 
> e8356e3a4a2e1ae41e09f5ea124fb5360bd6e8b6 
>   examples/src/main/apps/datelist-java-main/workflow.xml 
> b59a12ebd1fc21c9e26f6a5b0348eb9c76375c03 
>   examples/src/main/apps/demo/job.properties 
> d70d3208df77e0b612005196907cecd97d7c5d21 
>   examples/src/main/apps/demo/job.windows.properties 
> ea948b20009685e1286053549fc1974669819682 
>   examples/src/main/apps/demo/workflow.xml 
> 0ceca507adc68c0674b794420c9eb733cb320879 
>   examples/src/main/apps/distcp/job.properties 
> ec9422caddcdecfa6ee9a58868cf68d7457ad4db 
>   examples/src/main/apps/distcp/workflow.xml 
> 6deecb377901613a5bb28fc18bf5beff2d30f252 
>   examples/src/main/apps/hadoop-el/job.properties 
> 1d13c27056ac58ad359b5298f12ec046bb4cc61d 
>   examples/src/main/apps/hadoop-el/workflow.xml 
> d4094d44136a7de2f85f5c9be042d5aab24b90be 
>   examples/src/main/apps/hcatalog/job.properties 
> 

Re: Review Request 65535: OOZIE-3173 Coordinator job with frequency using cron syntax creates only one action in catchup mode

[Peter Cseh via Review Board]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/65535/#review196925
---




core/src/main/java/org/apache/oozie/command/coord/CoordMaterializeTransitionXCommand.java
Line 507 (original), 509-514 (patched)


This was probably left here during development.



core/src/test/java/org/apache/oozie/command/coord/TestCoordMaterializeTransitionXCommand.java
Lines 331 (patched)


Could you add error messages to the asserts so it's easier to understand 
what is the error if something goes wrong?



core/src/test/java/org/apache/oozie/command/coord/TestCoordMaterializeTransitionXCommand.java
Lines 344-345 (patched)


What is the purpose of this two lines?



core/src/test/java/org/apache/oozie/command/coord/TestCoordMaterializeTransitionXCommand.java
Lines 373-374 (patched)


Same as above


- Peter Cseh


On Feb. 6, 2018, 7:34 p.m., András Piros wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/65535/
> ---
> 
> (Updated Feb. 6, 2018, 7:34 p.m.)
> 
> 
> Review request for oozie, Attila Sasvari and Peter Cseh.
> 
> 
> Repository: oozie-git
> 
> 
> Description
> ---
> 
> OOZIE-3173 Coordinator job with frequency using cron syntax creates only one 
> action in catchup mode
> 
> 
> Diffs
> -
> 
>   
> core/src/main/java/org/apache/oozie/command/coord/CoordMaterializeTransitionXCommand.java
>  91d3508d128032ea5928cd5c6de482a65008c2bc 
>   
> core/src/test/java/org/apache/oozie/command/coord/TestCoordMaterializeTransitionXCommand.java
>  2a648d7c35c1a396e099ab27ab33d91e95d119d5 
> 
> 
> Diff: https://reviews.apache.org/r/65535/diff/1/
> 
> 
> Testing
> ---
> 
> New unit tests:
> 
> * 
> `TestCoordMaterializeTransitionXCommand#testCronFrequencyCatchupThrottleLessThanDuration()`
> * 
> `TestCoordMaterializeTransitionXCommand#testCronFrequencyCatchupThrottleEqualsDuration`
> * 
> `TestCoordMaterializeTransitionXCommand#testCronFrequencyCatchupThrottleMoreThanDuration`
> 
> 
> Thanks,
> 
> András Piros
> 
>

Re: Review Request 65385: OOZIE-1717 add indices to speed up db queries

[Peter Cseh via Review Board]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/65385/#review196803
---


Ship it!




Ship It!

- Peter Cseh


On Feb. 5, 2018, 9:52 a.m., Attila Sasvari wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/65385/
> ---
> 
> (Updated Feb. 5, 2018, 9:52 a.m.)
> 
> 
> Review request for oozie.
> 
> 
> Repository: oozie-git
> 
> 
> Description
> ---
> 
> Add indices to speed up SELECT operations.
> 
> 
> Index user_name column
> 
> for purge service OOZIE-1803:
> CREATE INDEX "OOZIEDB"."BUNDLE_JOBS_END_TIME_IDX" ON "BUNDLE_JOBS" 
> ("END_TIME");
> CREATE INDEX "OOZIEDB"."COORD_JOBS_END_TIME_IDX" ON "COORD_JOBS" ("END_TIME");
> CREATE INDEX "OOZIEDB"."WF_JOBS_END_TIME_IDX" ON "WF_JOBS" ("END_TIME");
> CREATE INDEX "OOZIEDB"."COORD_ACTIONS_JOB_ID_IDX" ON "COORD_ACTIONS" 
> ("JOB_ID");
> CREATE INDEX "OOZIEDB"."COORD_JOBS_BUNDLE_ID_IDX" ON "COORD_JOBS" 
> ("BUNDLE_ID");
> CREATE INDEX "OOZIEDB"."WF_ACTIONS_WF_ID_IDX" ON "WF_ACTIONS" ("WF_ID");
> CREATE INDEX "OOZIEDB"."WF_JOBS_PARENT_ID_IDX" ON "WF_JOBS" ("PARENT_ID");
> CREATE INDEX "OOZIEDB"."BUNDLE_ACTIONS_BUNDLE_ID_IDX" ON "BUNDLE_ACTIONS" 
> ("BUNDLE_ID");
> 
> Composite indices to speed up operations on the web UI
> 
> 
> Diffs
> -
> 
>   core/src/main/java/org/apache/oozie/BundleJobBean.java 
> b42f12841c2f9cb6d508cd2ac2f393e6407d1cf0 
>   core/src/main/java/org/apache/oozie/CoordinatorJobBean.java 
> 99c7d3c474d9e1e22e2f5a927c03d71b9292b663 
>   core/src/main/java/org/apache/oozie/WorkflowJobBean.java 
> cd9a6df42e552a0607606a244b81978747d89b41 
>   tools/src/main/java/org/apache/oozie/tools/OozieDBCLI.java 
> a8b2234966c967f9ad9d722628bd486012b14bd3 
> 
> 
> Diff: https://reviews.apache.org/r/65385/diff/3/
> 
> 
> Testing
> ---
> 
> 
> Thanks,
> 
> Attila Sasvari
> 
>

Re: Review Request 65177: >git status< should be clean after >mvn test< was called

[Peter Cseh via Review Board]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/65177/#review195561
---




sharelib/spark/src/test/java/org/apache/oozie/action/hadoop/TestJarFilter.java
Line 47 (original), 49 (patched)


can we create the jar in a temporary folder or in target/ instead of 
deleting it after the fact?


- Peter Cseh


On Jan. 16, 2018, 2:45 p.m., Kinga Marton wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/65177/
> ---
> 
> (Updated Jan. 16, 2018, 2:45 p.m.)
> 
> 
> Review request for oozie, András Piros, Attila Sasvari, and Peter Cseh.
> 
> 
> Repository: oozie-git
> 
> 
> Description
> ---
> 
> After running TestDistcpMain, local files created are not removed. Subsequent 
> test runs will reuse those files and fail:
> `$ ls -lrt /home/asasvari/workspace/oozie/core/distcp*`
> `-rw-rw-r-- 1 asasvari asasvari 1025 Dec 21 18:25 
> /home/asasvari/workspace/oozie/core/distcp-log4j.properties`
> `-rw-rw-r-- 1 asasvari asasvari 5756 Dec 21 18:26 
> /home/asasvari/workspace/oozie/core/distcp-oozie-1513876931701.log`
> Those files should be removed after test execution. It would be better to use 
> the oozie/core/target directory so that mvn could clean it.
> 
> 
> Diffs
> -
> 
>   core/src/test/java/org/apache/oozie/action/hadoop/TestLauncher.java 
> a3b3b7b3 
>   core/src/test/java/org/apache/oozie/client/TestOozieCLI.java 8c2aa982 
>   core/src/test/java/org/apache/oozie/jms/TestJMSJobEventListener.java 
> 728916ec 
>   core/src/test/java/org/apache/oozie/service/TestJMSAccessorService.java 
> 505049d2 
>   core/src/test/java/org/apache/oozie/test/XTestCase.java 8cecb89b 
>   
> sharelib/hive/src/test/java/org/apache/oozie/action/hadoop/TestHiveMain.java 
> 35c2865e 
>   sharelib/pig/src/test/java/org/apache/oozie/action/hadoop/TestPigMain.java 
> 9a185c91 
>   
> sharelib/spark/src/test/java/org/apache/oozie/action/hadoop/TestJarFilter.java
>  ff1b3cef 
>   
> sharelib/spark/src/test/java/org/apache/oozie/action/hadoop/TestSparkMain.java
>  b9f37c84 
>   tools/src/test/java/org/apache/oozie/tools/TestOozieMySqlDBCLI.java 
> a7df5c79 
> 
> 
> Diff: https://reviews.apache.org/r/65177/diff/1/
> 
> 
> Testing
> ---
> 
> I`ve run a mvn clean test and the repository remained clean.
> 
> 
> Thanks,
> 
> Kinga Marton
> 
>

Re: Review Request 65177: >git status< should be clean after >mvn test< was called

[Peter Cseh via Review Board]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/65177/#review195560
---




core/src/test/java/org/apache/oozie/test/XTestCase.java
Lines 501 (patched)


Can you create a function that returns the "files to delete" list? I think 
it would be nicer to just overwrite that methong in Pig and Spark tests to add 
more files there.


- Peter Cseh


On Jan. 16, 2018, 2:45 p.m., Kinga Marton wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/65177/
> ---
> 
> (Updated Jan. 16, 2018, 2:45 p.m.)
> 
> 
> Review request for oozie, András Piros, Attila Sasvari, and Peter Cseh.
> 
> 
> Repository: oozie-git
> 
> 
> Description
> ---
> 
> After running TestDistcpMain, local files created are not removed. Subsequent 
> test runs will reuse those files and fail:
> `$ ls -lrt /home/asasvari/workspace/oozie/core/distcp*`
> `-rw-rw-r-- 1 asasvari asasvari 1025 Dec 21 18:25 
> /home/asasvari/workspace/oozie/core/distcp-log4j.properties`
> `-rw-rw-r-- 1 asasvari asasvari 5756 Dec 21 18:26 
> /home/asasvari/workspace/oozie/core/distcp-oozie-1513876931701.log`
> Those files should be removed after test execution. It would be better to use 
> the oozie/core/target directory so that mvn could clean it.
> 
> 
> Diffs
> -
> 
>   core/src/test/java/org/apache/oozie/action/hadoop/TestLauncher.java 
> a3b3b7b3 
>   core/src/test/java/org/apache/oozie/client/TestOozieCLI.java 8c2aa982 
>   core/src/test/java/org/apache/oozie/jms/TestJMSJobEventListener.java 
> 728916ec 
>   core/src/test/java/org/apache/oozie/service/TestJMSAccessorService.java 
> 505049d2 
>   core/src/test/java/org/apache/oozie/test/XTestCase.java 8cecb89b 
>   
> sharelib/hive/src/test/java/org/apache/oozie/action/hadoop/TestHiveMain.java 
> 35c2865e 
>   sharelib/pig/src/test/java/org/apache/oozie/action/hadoop/TestPigMain.java 
> 9a185c91 
>   
> sharelib/spark/src/test/java/org/apache/oozie/action/hadoop/TestJarFilter.java
>  ff1b3cef 
>   
> sharelib/spark/src/test/java/org/apache/oozie/action/hadoop/TestSparkMain.java
>  b9f37c84 
>   tools/src/test/java/org/apache/oozie/tools/TestOozieMySqlDBCLI.java 
> a7df5c79 
> 
> 
> Diff: https://reviews.apache.org/r/65177/diff/1/
> 
> 
> Testing
> ---
> 
> I`ve run a mvn clean test and the repository remained clean.
> 
> 
> Thanks,
> 
> Kinga Marton
> 
>

Re: Review Request 64245: OOZIE-2681 fix javadoc to compile on JDK8 again

[Peter Cseh via Review Board]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/64245/
---

(Updated Dec. 5, 2017, 1:42 p.m.)


Review request for oozie.


Bugs: OOZIE-2681
https://issues.apache.org/jira/browse/OOZIE-2681


Repository: oozie-git


Description
---

OOZIE-2681
fix javadoc to compile on JDK8 again


Diffs (updated)
-

  client/src/main/java/org/apache/oozie/cli/CLIParser.java 
3930276f6dea23a774fed51421b9151bc158be29 
  client/src/main/java/org/apache/oozie/cli/OozieCLI.java 
d837c84af4667c94c1a234370457d5c075065628 
  client/src/main/java/org/apache/oozie/client/Job.java 
277e112722e6ee838aed3977ce3429d98ae34745 
  client/src/main/java/org/apache/oozie/client/OozieClient.java 
7fb1cf32cc034234a23c40556b93ea2e638ca7af 
  client/src/main/java/org/apache/oozie/client/event/jms/JMSMessagingUtils.java 
d1588e930c9df40ec77f428267c840944a81a81d 
  
client/src/main/java/org/apache/oozie/client/event/jms/MessageDeserializer.java 
fe31061fe6fbcefa36415cca55e409fcf25ad9ec 
  
client/src/main/java/org/apache/oozie/client/event/message/CoordinatorActionMessage.java
 d6209c529b8be82f40feb74ce4173537b84e68f2 
  client/src/main/java/org/apache/oozie/client/event/message/EventMessage.java 
382bb82bf32469c5836a9b0fc70cf573d6b94560 
  client/src/main/java/org/apache/oozie/client/event/message/JobMessage.java 
c51b2f94dd98b99b2f1dd74fdc76e6f87c5a517a 
  client/src/main/java/org/apache/oozie/client/event/message/SLAMessage.java 
96f9d675db2cd3d307bfa1c25794bfd50b58e820 
  
client/src/main/java/org/apache/oozie/client/event/message/WorkflowJobMessage.java
 b204ccd78a4274a2fae9c0d402ecd035a69729e6 
  core/src/main/java/org/apache/oozie/BaseEngine.java 
6a8ebfffb2bf83dae86336858fc4da64636564e9 
  core/src/main/java/org/apache/oozie/BinaryBlob.java 
d1971fc75571cbf7be20175a25b308b1ed122d4c 
  core/src/main/java/org/apache/oozie/BundleActionBean.java 
ccc18d9f15f745b22f67dbd8dd21f8dffd613863 
  core/src/main/java/org/apache/oozie/BundleJobBean.java 
5868412514209b7010e6d58fe6341650fea0b92b 
  core/src/main/java/org/apache/oozie/CoordinatorEngine.java 
89fda05f2dee90f9c654f199532b73beb87d8b02 
  core/src/main/java/org/apache/oozie/CoordinatorJobBean.java 
b6e0b7e4c39363968a07a5770d23791d0c16264e 
  core/src/main/java/org/apache/oozie/DagEngine.java 
01cf44922dd81d3cbcc34e3e4f1bb7b2a6be9ad0 
  core/src/main/java/org/apache/oozie/LocalOozieClient.java 
123725f5d54e01d930d2c29538f848fcc2f5db7c 
  core/src/main/java/org/apache/oozie/LocalOozieClientBundle.java 
3c229bea77f86c4241f837d70cb4d1059e7eb24d 
  core/src/main/java/org/apache/oozie/LocalOozieClientCoord.java 
d1e6972ac47cd33fcc7f8b358713cf0ee94ab3ec 
  core/src/main/java/org/apache/oozie/StringBlob.java 
9e8d3837b89ebe399a253c708946126924180d9a 
  core/src/main/java/org/apache/oozie/WorkflowActionBean.java 
14f36ed67a893cd94a7dbf7340caecf6af165e78 
  core/src/main/java/org/apache/oozie/client/rest/BulkResponseImpl.java 
374c6ef38d78573683bea90332cb737848f8d6c8 
  core/src/main/java/org/apache/oozie/client/rest/JMSConnectionInfoBean.java 
12576407285bc4b5d8cfa0ec03a2fe049661c743 
  core/src/main/java/org/apache/oozie/compression/CodecFactory.java 
72cf38598f5a9989f6d1c84db287f74c5785edab 
  core/src/main/java/org/apache/oozie/coord/CoordELEvaluator.java 
4230a201cc1d581957162355bb77955055e68d45 
  core/src/main/java/org/apache/oozie/coord/CoordELFunctions.java 
76d837c1f16834b58cb2589378de1fd83afe9d05 
  
core/src/main/java/org/apache/oozie/coord/input/dependency/CoordInputDependency.java
 4a2a0b088945ba36cf2530e6382ae33fa51a574e 
  
core/src/main/java/org/apache/oozie/coord/input/logic/CoordInputLogicEvaluatorPhaseOne.java
 52e03d9839f01e8fcbc9c0fdee179467b9ac5c98 
  
core/src/main/java/org/apache/oozie/coord/input/logic/CoordInputLogicEvaluatorUtil.java
 653fb2a32bf01ad297125cc66df6b29bd52607e6 
  core/src/main/java/org/apache/oozie/dependency/DependencyChecker.java 
bdd854ffb7025c71ee04d433b2dd7d60afcfa07c 
  core/src/main/java/org/apache/oozie/dependency/URIHandler.java 
da68b0005d130bd42cf2f4cc72729115a06901b4 
  core/src/main/java/org/apache/oozie/dependency/hcat/HCatDependencyCache.java 
a5a31e4c2abbb469704e6ac2c0a68ec1812e93a7 
  core/src/main/java/org/apache/oozie/service/UUIDService.java 
faddd6d217252e4d63821416c335783552a21d17 
  core/src/main/java/org/apache/oozie/servlet/BaseAdminServlet.java 
97c2007fbe3cf51a9fa6cb8ccc48919c251af4fd 
  core/src/main/java/org/apache/oozie/servlet/BaseJobServlet.java 
b6a896f8201705116e1e13e643a67c9b93860d20 
  core/src/main/java/org/apache/oozie/servlet/ErrorServlet.java 
94e93bfcb2f39e8ab31a9ce22246c2cd86cb907d 
  core/src/main/java/org/apache/oozie/servlet/JsonRestServlet.java 
0f2279651b4f1caf073967dedb166c3750c4c853 
  core/src/main/java/org/apache/oozie/servlet/ServletUtilities.java 
a7b092124675b4372599749df34eb2d5d263897f 
  

Re: Review Request 64245: OOZIE-2681 fix javadoc to compile on JDK8 again

[Peter Cseh via Review Board]

> On Dec. 1, 2017, 3:51 p.m., Peter Bacsko wrote:
> > sharelib/hcatalog/src/main/java/org/apache/oozie/util/HCatURI.java
> > Line 292 (original), 293 (patched)
> > 
> >
> > Nit: this isn't too helpful

The function itself is not that helpful :)


- Peter


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/64245/#review192496
---


On Dec. 1, 2017, 11:45 a.m., Peter Cseh wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/64245/
> ---
> 
> (Updated Dec. 1, 2017, 11:45 a.m.)
> 
> 
> Review request for oozie.
> 
> 
> Bugs: OOZIE-2681
> https://issues.apache.org/jira/browse/OOZIE-2681
> 
> 
> Repository: oozie-git
> 
> 
> Description
> ---
> 
> OOZIE-2681
> fix javadoc to compile on JDK8 again
> 
> 
> Diffs
> -
> 
>   client/src/main/java/org/apache/oozie/cli/CLIParser.java 
> 3930276f6dea23a774fed51421b9151bc158be29 
>   client/src/main/java/org/apache/oozie/cli/OozieCLI.java 
> d837c84af4667c94c1a234370457d5c075065628 
>   client/src/main/java/org/apache/oozie/client/Job.java 
> 277e112722e6ee838aed3977ce3429d98ae34745 
>   client/src/main/java/org/apache/oozie/client/OozieClient.java 
> 7fb1cf32cc034234a23c40556b93ea2e638ca7af 
>   
> client/src/main/java/org/apache/oozie/client/event/jms/JMSMessagingUtils.java 
> d1588e930c9df40ec77f428267c840944a81a81d 
>   
> client/src/main/java/org/apache/oozie/client/event/jms/MessageDeserializer.java
>  fe31061fe6fbcefa36415cca55e409fcf25ad9ec 
>   
> client/src/main/java/org/apache/oozie/client/event/message/CoordinatorActionMessage.java
>  d6209c529b8be82f40feb74ce4173537b84e68f2 
>   
> client/src/main/java/org/apache/oozie/client/event/message/EventMessage.java 
> 382bb82bf32469c5836a9b0fc70cf573d6b94560 
>   client/src/main/java/org/apache/oozie/client/event/message/JobMessage.java 
> c51b2f94dd98b99b2f1dd74fdc76e6f87c5a517a 
>   client/src/main/java/org/apache/oozie/client/event/message/SLAMessage.java 
> 96f9d675db2cd3d307bfa1c25794bfd50b58e820 
>   
> client/src/main/java/org/apache/oozie/client/event/message/WorkflowJobMessage.java
>  b204ccd78a4274a2fae9c0d402ecd035a69729e6 
>   core/src/main/java/org/apache/oozie/BaseEngine.java 
> 6a8ebfffb2bf83dae86336858fc4da64636564e9 
>   core/src/main/java/org/apache/oozie/BinaryBlob.java 
> d1971fc75571cbf7be20175a25b308b1ed122d4c 
>   core/src/main/java/org/apache/oozie/BundleActionBean.java 
> ccc18d9f15f745b22f67dbd8dd21f8dffd613863 
>   core/src/main/java/org/apache/oozie/BundleJobBean.java 
> 5868412514209b7010e6d58fe6341650fea0b92b 
>   core/src/main/java/org/apache/oozie/CoordinatorEngine.java 
> 89fda05f2dee90f9c654f199532b73beb87d8b02 
>   core/src/main/java/org/apache/oozie/CoordinatorJobBean.java 
> b6e0b7e4c39363968a07a5770d23791d0c16264e 
>   core/src/main/java/org/apache/oozie/DagEngine.java 
> 01cf44922dd81d3cbcc34e3e4f1bb7b2a6be9ad0 
>   core/src/main/java/org/apache/oozie/LocalOozieClient.java 
> 123725f5d54e01d930d2c29538f848fcc2f5db7c 
>   core/src/main/java/org/apache/oozie/LocalOozieClientBundle.java 
> 3c229bea77f86c4241f837d70cb4d1059e7eb24d 
>   core/src/main/java/org/apache/oozie/LocalOozieClientCoord.java 
> d1e6972ac47cd33fcc7f8b358713cf0ee94ab3ec 
>   core/src/main/java/org/apache/oozie/StringBlob.java 
> 9e8d3837b89ebe399a253c708946126924180d9a 
>   core/src/main/java/org/apache/oozie/WorkflowActionBean.java 
> 14f36ed67a893cd94a7dbf7340caecf6af165e78 
>   core/src/main/java/org/apache/oozie/client/rest/BulkResponseImpl.java 
> 374c6ef38d78573683bea90332cb737848f8d6c8 
>   core/src/main/java/org/apache/oozie/client/rest/JMSConnectionInfoBean.java 
> 12576407285bc4b5d8cfa0ec03a2fe049661c743 
>   core/src/main/java/org/apache/oozie/compression/CodecFactory.java 
> 72cf38598f5a9989f6d1c84db287f74c5785edab 
>   core/src/main/java/org/apache/oozie/coord/CoordELEvaluator.java 
> 4230a201cc1d581957162355bb77955055e68d45 
>   core/src/main/java/org/apache/oozie/coord/CoordELFunctions.java 
> 76d837c1f16834b58cb2589378de1fd83afe9d05 
>   
> core/src/main/java/org/apache/oozie/coord/input/dependency/CoordInputDependency.java
>  4a2a0b088945ba36cf2530e6382ae33fa51a574e 
>   
> core/src/main/java/org/apache/oozie/coord/input/logic/CoordInputLogicEvaluatorPhaseOne.java
>  52e03d9839f01e8fcbc9c0fdee179467b9ac5c98 
>   
> core/src/main/java/org/apache/oozie/coord/input/logic/CoordInputLogicEvaluatorUtil.java
>  653fb2a32bf01ad297125cc66df6b29bd52607e6 
>   core/src/main/java/org/apache/oozie/dependency/DependencyChecker.java 
> bdd854ffb7025c71ee04d433b2dd7d60afcfa07c 
>   core/src/main/java/org/apache/oozie/dependency/URIHandler.java 
> da68b0005d130bd42cf2f4cc72729115a06901b4 
>   
> 

Review Request 64245: OOZIE-2681 fix javadoc to compile on JDK8 again

[Peter Cseh via Review Board]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/64245/
---

Review request for oozie.


Bugs: OOZIE-2681
https://issues.apache.org/jira/browse/OOZIE-2681


Repository: oozie-git


Description
---

OOZIE-2681
fix javadoc to compile on JDK8 again


Diffs
-

  client/src/main/java/org/apache/oozie/cli/CLIParser.java 
3930276f6dea23a774fed51421b9151bc158be29 
  client/src/main/java/org/apache/oozie/cli/OozieCLI.java 
d837c84af4667c94c1a234370457d5c075065628 
  client/src/main/java/org/apache/oozie/client/Job.java 
277e112722e6ee838aed3977ce3429d98ae34745 
  client/src/main/java/org/apache/oozie/client/OozieClient.java 
7fb1cf32cc034234a23c40556b93ea2e638ca7af 
  client/src/main/java/org/apache/oozie/client/event/jms/JMSMessagingUtils.java 
d1588e930c9df40ec77f428267c840944a81a81d 
  
client/src/main/java/org/apache/oozie/client/event/jms/MessageDeserializer.java 
fe31061fe6fbcefa36415cca55e409fcf25ad9ec 
  
client/src/main/java/org/apache/oozie/client/event/message/CoordinatorActionMessage.java
 d6209c529b8be82f40feb74ce4173537b84e68f2 
  client/src/main/java/org/apache/oozie/client/event/message/EventMessage.java 
382bb82bf32469c5836a9b0fc70cf573d6b94560 
  client/src/main/java/org/apache/oozie/client/event/message/JobMessage.java 
c51b2f94dd98b99b2f1dd74fdc76e6f87c5a517a 
  client/src/main/java/org/apache/oozie/client/event/message/SLAMessage.java 
96f9d675db2cd3d307bfa1c25794bfd50b58e820 
  
client/src/main/java/org/apache/oozie/client/event/message/WorkflowJobMessage.java
 b204ccd78a4274a2fae9c0d402ecd035a69729e6 
  core/src/main/java/org/apache/oozie/BaseEngine.java 
6a8ebfffb2bf83dae86336858fc4da64636564e9 
  core/src/main/java/org/apache/oozie/BinaryBlob.java 
d1971fc75571cbf7be20175a25b308b1ed122d4c 
  core/src/main/java/org/apache/oozie/BundleActionBean.java 
ccc18d9f15f745b22f67dbd8dd21f8dffd613863 
  core/src/main/java/org/apache/oozie/BundleJobBean.java 
5868412514209b7010e6d58fe6341650fea0b92b 
  core/src/main/java/org/apache/oozie/CoordinatorEngine.java 
89fda05f2dee90f9c654f199532b73beb87d8b02 
  core/src/main/java/org/apache/oozie/CoordinatorJobBean.java 
b6e0b7e4c39363968a07a5770d23791d0c16264e 
  core/src/main/java/org/apache/oozie/DagEngine.java 
01cf44922dd81d3cbcc34e3e4f1bb7b2a6be9ad0 
  core/src/main/java/org/apache/oozie/LocalOozieClient.java 
123725f5d54e01d930d2c29538f848fcc2f5db7c 
  core/src/main/java/org/apache/oozie/LocalOozieClientBundle.java 
3c229bea77f86c4241f837d70cb4d1059e7eb24d 
  core/src/main/java/org/apache/oozie/LocalOozieClientCoord.java 
d1e6972ac47cd33fcc7f8b358713cf0ee94ab3ec 
  core/src/main/java/org/apache/oozie/StringBlob.java 
9e8d3837b89ebe399a253c708946126924180d9a 
  core/src/main/java/org/apache/oozie/WorkflowActionBean.java 
14f36ed67a893cd94a7dbf7340caecf6af165e78 
  core/src/main/java/org/apache/oozie/client/rest/BulkResponseImpl.java 
374c6ef38d78573683bea90332cb737848f8d6c8 
  core/src/main/java/org/apache/oozie/client/rest/JMSConnectionInfoBean.java 
12576407285bc4b5d8cfa0ec03a2fe049661c743 
  core/src/main/java/org/apache/oozie/compression/CodecFactory.java 
72cf38598f5a9989f6d1c84db287f74c5785edab 
  core/src/main/java/org/apache/oozie/coord/CoordELEvaluator.java 
4230a201cc1d581957162355bb77955055e68d45 
  core/src/main/java/org/apache/oozie/coord/CoordELFunctions.java 
76d837c1f16834b58cb2589378de1fd83afe9d05 
  
core/src/main/java/org/apache/oozie/coord/input/dependency/CoordInputDependency.java
 4a2a0b088945ba36cf2530e6382ae33fa51a574e 
  
core/src/main/java/org/apache/oozie/coord/input/logic/CoordInputLogicEvaluatorPhaseOne.java
 52e03d9839f01e8fcbc9c0fdee179467b9ac5c98 
  
core/src/main/java/org/apache/oozie/coord/input/logic/CoordInputLogicEvaluatorUtil.java
 653fb2a32bf01ad297125cc66df6b29bd52607e6 
  core/src/main/java/org/apache/oozie/dependency/DependencyChecker.java 
bdd854ffb7025c71ee04d433b2dd7d60afcfa07c 
  core/src/main/java/org/apache/oozie/dependency/URIHandler.java 
da68b0005d130bd42cf2f4cc72729115a06901b4 
  core/src/main/java/org/apache/oozie/dependency/hcat/HCatDependencyCache.java 
a5a31e4c2abbb469704e6ac2c0a68ec1812e93a7 
  core/src/main/java/org/apache/oozie/service/UUIDService.java 
faddd6d217252e4d63821416c335783552a21d17 
  core/src/main/java/org/apache/oozie/servlet/BaseAdminServlet.java 
97c2007fbe3cf51a9fa6cb8ccc48919c251af4fd 
  core/src/main/java/org/apache/oozie/servlet/BaseJobServlet.java 
b6a896f8201705116e1e13e643a67c9b93860d20 
  core/src/main/java/org/apache/oozie/servlet/ErrorServlet.java 
94e93bfcb2f39e8ab31a9ce22246c2cd86cb907d 
  core/src/main/java/org/apache/oozie/servlet/JsonRestServlet.java 
0f2279651b4f1caf073967dedb166c3750c4c853 
  core/src/main/java/org/apache/oozie/servlet/ServletUtilities.java 
a7b092124675b4372599749df34eb2d5d263897f 
  core/src/main/java/org/apache/oozie/test/EmbeddedServletContainer.java 

Re: Review Request 63837: SparkConfigrationService overwrites properties provided via --properties-file option in SparkAction

[Peter Cseh via Review Board]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/63837/
---

(Updated Nov. 30, 2017, 9:19 p.m.)


Review request for oozie.


Bugs: OOZIE-3112
https://issues.apache.org/jira/browse/OOZIE-3112


Repository: oozie-git


Description
---

SparkConfigurationService injects the config values as --conf to SparkSubmit. 
This will overwrite propties provided in the --properies-file option which is 
not the expected behavior.


Diffs (updated)
-

  core/src/main/java/org/apache/oozie/action/hadoop/SparkActionExecutor.java 
2338ad28bf1a57b2fceab586372b237973d90f80 
  docs/src/site/twiki/DG_SparkActionExtension.twiki 
45b60b86bee4cfe2de2d374f03b602ddacdc5e65 
  sharelib/spark/pom.xml 65f641a927ea2fe3f357333dc0fcd518c2ad270f 
  
sharelib/spark/src/main/java/org/apache/oozie/action/hadoop/SparkArgsExtractor.java
 052950fa36a0b40f8629ce91c1d135062a6571a9 
  
sharelib/spark/src/test/java/org/apache/oozie/action/hadoop/TestSparkActionExecutor.java
 d97f1f06b5832b1f09fab30b08ecff31db3221f6 
  
sharelib/spark/src/test/java/org/apache/oozie/action/hadoop/TestSparkArgsExtractor.java
 574bf240304379d5d26af8a3e5c079d4ad707b7b 


Diff: https://reviews.apache.org/r/63837/diff/7/

Changes: https://reviews.apache.org/r/63837/diff/6-7/


Testing
---


Thanks,

Peter Cseh

Re: Review Request 63837: SparkConfigrationService overwrites properties provided via --properties-file option in SparkAction

[Peter Cseh via Review Board]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/63837/
---

(Updated Nov. 30, 2017, 1:18 p.m.)


Review request for oozie.


Bugs: OOZIE-3112
https://issues.apache.org/jira/browse/OOZIE-3112


Repository: oozie-git


Description
---

SparkConfigurationService injects the config values as --conf to SparkSubmit. 
This will overwrite propties provided in the --properies-file option which is 
not the expected behavior.


Diffs (updated)
-

  core/src/main/java/org/apache/oozie/action/hadoop/SparkActionExecutor.java 
2338ad28bf1a57b2fceab586372b237973d90f80 
  docs/src/site/twiki/DG_SparkActionExtension.twiki 
45b60b86bee4cfe2de2d374f03b602ddacdc5e65 
  sharelib/spark/pom.xml 65f641a927ea2fe3f357333dc0fcd518c2ad270f 
  
sharelib/spark/src/main/java/org/apache/oozie/action/hadoop/SparkArgsExtractor.java
 052950fa36a0b40f8629ce91c1d135062a6571a9 
  
sharelib/spark/src/test/java/org/apache/oozie/action/hadoop/TestSparkActionExecutor.java
 d97f1f06b5832b1f09fab30b08ecff31db3221f6 
  
sharelib/spark/src/test/java/org/apache/oozie/action/hadoop/TestSparkArgsExtractor.java
 574bf240304379d5d26af8a3e5c079d4ad707b7b 


Diff: https://reviews.apache.org/r/63837/diff/6/

Changes: https://reviews.apache.org/r/63837/diff/5-6/


Testing
---


Thanks,

Peter Cseh

Re: Review Request 63875: OOZIE-2900 Retrieve tokens for oozie.launcher.mapreduce.job.hdfs-servers before submission

[Peter Cseh via Review Board]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/63875/#review191662
---




core/src/main/java/org/apache/oozie/action/hadoop/JavaActionExecutor.java
Lines 1379-1386 (patched)


Can this part be pushed down to DistcpActionExecutor? It does not feel like 
other actions would have to work with these properties.
Also, please cosider adding some logging here, at least on debug level to 
make it easier to see what's happening.


- Peter Cseh


On Nov. 21, 2017, 3:49 p.m., Attila Sasvari wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/63875/
> ---
> 
> (Updated Nov. 21, 2017, 3:49 p.m.)
> 
> 
> Review request for oozie, Peter Bacsko and Robert Kanter.
> 
> 
> Repository: oozie-git
> 
> 
> Description
> ---
> 
> Before Oozie on YARN, ``JobSubmitter`` from MapReduce (more precisely 
> ``TokenCache.obtainTokensForNamenodes``) took care of obtaining delegation 
> tokens for HDFS nodes specified by 
> ``oozie.launcher.mapreduce.job.hdfs-servers`` before submitting the Oozie 
> launcher job.
> 
> Oozie launcher is now a Yarn Application Master. It needs HDFS delegation 
> tokens to be able to copy files between secure clusters via the Oozie DistCp 
> action. 
> 
> Changes:
> - ``JavaActionExecutor`` was modified to handle Distcp related parameters 
> like (``oozie.launcher.mapreduce.job.hdfs-servers`` and 
> ``oozie.launcher.mapreduce.job.hdfs-servers.token-renewal.exclude``)
> - ``HDFSCredentials`` was changed to reuse 
> ``TokenCache.obtainTokensForNamenodes`` to obtain HDFS delegation tokens.
> 
> 
> Diffs
> -
> 
>   core/src/main/java/org/apache/oozie/action/hadoop/HDFSCredentials.java 
> 92a7ebe9a7876b6400d80356d5c826e77575e2ab 
>   core/src/main/java/org/apache/oozie/action/hadoop/JavaActionExecutor.java 
> a1df304914b73d406e986409a8053c2a48e1bd38 
> 
> 
> Diff: https://reviews.apache.org/r/63875/diff/3/
> 
> 
> Testing
> ---
> 
> Tested on a secure cluster that Oozie dist cp action can copy file from 
> another secure cluster where different Kerberos realm was used.
> 
> - workflow:
> ```
> 
> 
> 
> Action failed, error 
> message[${wf:errorMessage(wf:lastErrorNode())}]
> 
> 
> 
> ${jobTracker}
> ${nameNode}
> 
> 
>   
> 
> oozie.launcher.mapreduce.job.dfs.namenode.kerberos.principal.pattern
> *
>   
> 
>   oozie.launcher.mapreduce.job.hdfs-servers
>   hdfs://oozie.test1.com:8020,hdfs://remote.test2.com:8020
> 
>  
> 
> 
> oozie.launcher.mapreduce.job.hdfs-servers.token-renewal.exclude
> remote.test2.com
> 
> 
>   hdfs://remote.test2.com:8020/tmp/1
>   hdfs://oozie.test1.com:8020/tmp/2
> 
> 
> 
> 
> 
> 
> ```
> 
> Prior to executing the workflow I had to setup cross realm trust between the 
> test secure clusters. It involved:
> - changing Kerberos configuration ``/etc/krb5.conf`` (adding realms and 
> setting additional properties like ``udp_preference_limit = 1``)
> - regenerating service credentials
> - changing HDFS settings (such as 
> ``dfs.namenode.kerberos.principal.pattern``) and setting hadoop auth to local 
> rule like ``RULE:[2:$1](.*)s/(.*)/$1/g``
> - additional configuration to enable trust between the test hadoop clusters
> 
> 
> Thanks,
> 
> Attila Sasvari
> 
>

Re: Review Request 63875: OOZIE-2900 Retrieve tokens for oozie.launcher.mapreduce.job.hdfs-servers before submission

[Peter Cseh via Review Board]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/63875/#review191334
---




core/src/main/java/org/apache/oozie/action/hadoop/HDFSCredentials.java
Lines 72-76 (patched)


Can you add some info-level logging or validation of the prperties? E.g. 
what happens if the JOB_NAMENODES property not null, but empty?



core/src/main/java/org/apache/oozie/action/hadoop/HDFSCredentials.java
Lines 110 (patched)


Please add logging similart to the other code path here.



core/src/main/java/org/apache/oozie/action/hadoop/JavaActionExecutor.java
Lines 1375 (patched)


Please create a constant for this as well.



core/src/main/java/org/apache/oozie/action/hadoop/JavaActionExecutor.java
Lines 1378 (patched)


You've created a constant for this but haven't used it. Please do so.


- Peter Cseh


On Nov. 16, 2017, 1:50 p.m., Attila Sasvari wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/63875/
> ---
> 
> (Updated Nov. 16, 2017, 1:50 p.m.)
> 
> 
> Review request for oozie, Peter Bacsko and Robert Kanter.
> 
> 
> Repository: oozie-git
> 
> 
> Description
> ---
> 
> Before Oozie on YARN, ``JobSubmitter`` from MapReduce (more precisely 
> ``TokenCache.obtainTokensForNamenodes``) took care of obtaining delegation 
> tokens for HDFS nodes specified by 
> ``oozie.launcher.mapreduce.job.hdfs-servers`` before submitting the Oozie 
> launcher job.
> 
> Oozie launcher is now a Yarn Application Master. It needs HDFS delegation 
> tokens to be able to copy files between secure clusters via the Oozie DistCp 
> action. 
> 
> Changes:
> - ``JavaActionExecutor`` was modified to handle Distcp related parameters 
> like (``oozie.launcher.mapreduce.job.hdfs-servers`` and 
> ``oozie.launcher.mapreduce.job.hdfs-servers.token-renewal.exclude``)
> - ``HDFSCredentials`` was changed to reuse 
> ``TokenCache.obtainTokensForNamenodes`` to obtain HDFS delegation tokens.
> 
> 
> Diffs
> -
> 
>   core/src/main/java/org/apache/oozie/action/hadoop/HDFSCredentials.java 
> 92a7ebe 
>   core/src/main/java/org/apache/oozie/action/hadoop/JavaActionExecutor.java 
> 8cb76cf 
> 
> 
> Diff: https://reviews.apache.org/r/63875/diff/1/
> 
> 
> Testing
> ---
> 
> Tested on a secure cluster that Oozie dist cp action can copy file from 
> another secure cluster where different Kerberos realm was used.
> 
> - workflow:
> ```
> 
> 
> 
> Action failed, error 
> message[${wf:errorMessage(wf:lastErrorNode())}]
> 
> 
> 
> ${jobTracker}
> ${nameNode}
> 
> 
>   
> 
> oozie.launcher.mapreduce.job.dfs.namenode.kerberos.principal.pattern
> *
>   
> 
>   oozie.launcher.mapreduce.job.hdfs-servers
>   hdfs://oozie.test1.com:8020,hdfs://remote.test2.com:8020
> 
>  
> 
> 
> oozie.launcher.mapreduce.job.hdfs-servers.token-renewal.exclude
> remote.test2.com
> 
> 
>   hdfs://remote.test2.com:8020/tmp/1
>   hdfs://oozie.test1.com:8020/tmp/2
> 
> 
> 
> 
> 
> 
> ```
> 
> Prior to executing the workflow I had to setup cross realm trust between the 
> test secure clusters. It involved:
> - changing Kerberos configuration ``/etc/krb5.conf`` (adding realms and 
> setting additional properties like ``udp_preference_limit = 1``)
> - regenerating service credentials
> - changing HDFS settings (such as 
> ``dfs.namenode.kerberos.principal.pattern``) and setting hadoop auth to local 
> rule like ``RULE:[2:$1](.*)s/(.*)/$1/g``
> - additional configuration to enable trust between the test hadoop clusters
> 
> 
> Thanks,
> 
> Attila Sasvari
> 
>

Re: Review Request 63875: OOZIE-2900 Retrieve tokens for oozie.launcher.mapreduce.job.hdfs-servers before submission

[Peter Cseh via Review Board]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/63875/#review191332
---




core/src/main/java/org/apache/oozie/action/hadoop/HDFSCredentials.java
Lines 63-64 (patched)


You could use UserGroupInformationService here.


- Peter Cseh


On Nov. 16, 2017, 1:50 p.m., Attila Sasvari wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/63875/
> ---
> 
> (Updated Nov. 16, 2017, 1:50 p.m.)
> 
> 
> Review request for oozie, Peter Bacsko and Robert Kanter.
> 
> 
> Repository: oozie-git
> 
> 
> Description
> ---
> 
> Before Oozie on YARN, ``JobSubmitter`` from MapReduce (more precisely 
> ``TokenCache.obtainTokensForNamenodes``) took care of obtaining delegation 
> tokens for HDFS nodes specified by 
> ``oozie.launcher.mapreduce.job.hdfs-servers`` before submitting the Oozie 
> launcher job.
> 
> Oozie launcher is now a Yarn Application Master. It needs HDFS delegation 
> tokens to be able to copy files between secure clusters via the Oozie DistCp 
> action. 
> 
> Changes:
> - ``JavaActionExecutor`` was modified to handle Distcp related parameters 
> like (``oozie.launcher.mapreduce.job.hdfs-servers`` and 
> ``oozie.launcher.mapreduce.job.hdfs-servers.token-renewal.exclude``)
> - ``HDFSCredentials`` was changed to reuse 
> ``TokenCache.obtainTokensForNamenodes`` to obtain HDFS delegation tokens.
> 
> 
> Diffs
> -
> 
>   core/src/main/java/org/apache/oozie/action/hadoop/HDFSCredentials.java 
> 92a7ebe 
>   core/src/main/java/org/apache/oozie/action/hadoop/JavaActionExecutor.java 
> 8cb76cf 
> 
> 
> Diff: https://reviews.apache.org/r/63875/diff/1/
> 
> 
> Testing
> ---
> 
> Tested on a secure cluster that Oozie dist cp action can copy file from 
> another secure cluster where different Kerberos realm was used.
> 
> - workflow:
> ```
> 
> 
> 
> Action failed, error 
> message[${wf:errorMessage(wf:lastErrorNode())}]
> 
> 
> 
> ${jobTracker}
> ${nameNode}
> 
> 
>   
> 
> oozie.launcher.mapreduce.job.dfs.namenode.kerberos.principal.pattern
> *
>   
> 
>   oozie.launcher.mapreduce.job.hdfs-servers
>   hdfs://oozie.test1.com:8020,hdfs://remote.test2.com:8020
> 
>  
> 
> 
> oozie.launcher.mapreduce.job.hdfs-servers.token-renewal.exclude
> remote.test2.com
> 
> 
>   hdfs://remote.test2.com:8020/tmp/1
>   hdfs://oozie.test1.com:8020/tmp/2
> 
> 
> 
> 
> 
> 
> ```
> 
> Prior to executing the workflow I had to setup cross realm trust between the 
> test secure clusters. It involved:
> - changing Kerberos configuration ``/etc/krb5.conf`` (adding realms and 
> setting additional properties like ``udp_preference_limit = 1``)
> - regenerating service credentials
> - changing HDFS settings (such as 
> ``dfs.namenode.kerberos.principal.pattern``) and setting hadoop auth to local 
> rule like ``RULE:[2:$1](.*)s/(.*)/$1/g``
> - additional configuration to enable trust between the test hadoop clusters
> 
> 
> Thanks,
> 
> Attila Sasvari
> 
>

Re: Review Request 63836: OOZIE-2869 Ability to add/remove MapReduce jars from action classpath.

[Peter Cseh via Review Board]

> On Nov. 15, 2017, 2:11 p.m., András Piros wrote:
> > Would be advisable to have a test case covering different `needed.for` 
> > scenarios, that is, the new functionality.
> 
> Peter Cseh wrote:
> Yeah, I agree. I'm not sure I can test this properly. I'm opening up a 
> Jira to create an AppSubmissionContextFactory or something similar so we can 
> write tests against that part.

OOZIE-3131


- Peter


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/63836/#review191060
---


On Nov. 15, 2017, 1:34 p.m., Peter Cseh wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/63836/
> ---
> 
> (Updated Nov. 15, 2017, 1:34 p.m.)
> 
> 
> Review request for oozie.
> 
> 
> Bugs: OOZIE-2869
> https://issues.apache.org/jira/browse/OOZIE-2869
> 
> 
> Repository: oozie-git
> 
> 
> Description
> ---
> 
> I've introduced a property to provide the possibility to add/remove jars from 
> the classpath for each action type using the 
> "oozie.action.mapreduce.needed.for" prefix.
> 
> 
> Diffs
> -
> 
>   core/src/main/java/org/apache/oozie/action/hadoop/DistcpActionExecutor.java 
> 961994ef14c5ef86be6609cde68bd98b39af8a2f 
>   core/src/main/java/org/apache/oozie/action/hadoop/HiveActionExecutor.java 
> d8c613bbbf9ed0d6b741a83861aeb82009084e3c 
>   core/src/main/java/org/apache/oozie/action/hadoop/JavaActionExecutor.java 
> 8cb76cf086264ac09d8c9bbff22989d6e5eac69f 
>   
> core/src/main/java/org/apache/oozie/action/hadoop/MapReduceActionExecutor.java
>  233dce7838b8d0ff0a18317d00768e87a0e5d7c8 
>   core/src/main/java/org/apache/oozie/action/hadoop/PigActionExecutor.java 
> 6541d52e35689a3776fa2ab3e6eb420c14c16a12 
>   core/src/main/java/org/apache/oozie/action/hadoop/SparkActionExecutor.java 
> 80d64ec80e683ea571748c2df976cf6bd22063cd 
>   core/src/main/java/org/apache/oozie/action/hadoop/SqoopActionExecutor.java 
> a0dfd31775507aeafaf09f1a316da0f52eac0a22 
>   core/src/main/resources/oozie-default.xml 
> 91d281125b236a9641faff54a69947b4440254d5 
>   sharelib/spark/pom.xml b40a3528137cedf58c0591cf9d010c38eec75d44 
> 
> 
> Diff: https://reviews.apache.org/r/63836/diff/1/
> 
> 
> Testing
> ---
> 
> 
> Thanks,
> 
> Peter Cseh
> 
>

Re: Review Request 63836: OOZIE-2869 Ability to add/remove MapReduce jars from action classpath.

[Peter Cseh via Review Board]

> On Nov. 15, 2017, 2:11 p.m., András Piros wrote:
> > Would be advisable to have a test case covering different `needed.for` 
> > scenarios, that is, the new functionality.

Yeah, I agree. I'm not sure I can test this properly. I'm opening up a Jira to 
create an AppSubmissionContextFactory or something similar so we can write 
tests against that part.


- Peter


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/63836/#review191060
---


On Nov. 15, 2017, 1:34 p.m., Peter Cseh wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/63836/
> ---
> 
> (Updated Nov. 15, 2017, 1:34 p.m.)
> 
> 
> Review request for oozie.
> 
> 
> Bugs: OOZIE-2869
> https://issues.apache.org/jira/browse/OOZIE-2869
> 
> 
> Repository: oozie-git
> 
> 
> Description
> ---
> 
> I've introduced a property to provide the possibility to add/remove jars from 
> the classpath for each action type using the 
> "oozie.action.mapreduce.needed.for" prefix.
> 
> 
> Diffs
> -
> 
>   core/src/main/java/org/apache/oozie/action/hadoop/DistcpActionExecutor.java 
> 961994ef14c5ef86be6609cde68bd98b39af8a2f 
>   core/src/main/java/org/apache/oozie/action/hadoop/HiveActionExecutor.java 
> d8c613bbbf9ed0d6b741a83861aeb82009084e3c 
>   core/src/main/java/org/apache/oozie/action/hadoop/JavaActionExecutor.java 
> 8cb76cf086264ac09d8c9bbff22989d6e5eac69f 
>   
> core/src/main/java/org/apache/oozie/action/hadoop/MapReduceActionExecutor.java
>  233dce7838b8d0ff0a18317d00768e87a0e5d7c8 
>   core/src/main/java/org/apache/oozie/action/hadoop/PigActionExecutor.java 
> 6541d52e35689a3776fa2ab3e6eb420c14c16a12 
>   core/src/main/java/org/apache/oozie/action/hadoop/SparkActionExecutor.java 
> 80d64ec80e683ea571748c2df976cf6bd22063cd 
>   core/src/main/java/org/apache/oozie/action/hadoop/SqoopActionExecutor.java 
> a0dfd31775507aeafaf09f1a316da0f52eac0a22 
>   core/src/main/resources/oozie-default.xml 
> 91d281125b236a9641faff54a69947b4440254d5 
>   sharelib/spark/pom.xml b40a3528137cedf58c0591cf9d010c38eec75d44 
> 
> 
> Diff: https://reviews.apache.org/r/63836/diff/1/
> 
> 
> Testing
> ---
> 
> 
> Thanks,
> 
> Peter Cseh
> 
>

Re: Review Request 63837: SparkConfigrationService overwrites properties provided via --properties-file option in SparkAction

[Peter Cseh via Review Board]

> On Nov. 16, 2017, 10:18 a.m., András Piros wrote:
> > sharelib/spark/src/main/java/org/apache/oozie/action/hadoop/SparkArgsExtractor.java
> > Lines 342-343 (patched)
> > 
> >
> > Typo: `persisted`

Thanks.


- Peter


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/63837/#review191164
---


On Nov. 16, 2017, 10:04 a.m., Peter Cseh wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/63837/
> ---
> 
> (Updated Nov. 16, 2017, 10:04 a.m.)
> 
> 
> Review request for oozie.
> 
> 
> Bugs: OOZIE-3112
> https://issues.apache.org/jira/browse/OOZIE-3112
> 
> 
> Repository: oozie-git
> 
> 
> Description
> ---
> 
> SparkConfigurationService injects the config values as --conf to SparkSubmit. 
> This will overwrite propties provided in the --properies-file option which is 
> not the expected behavior.
> 
> 
> Diffs
> -
> 
>   core/src/main/java/org/apache/oozie/action/hadoop/SparkActionExecutor.java 
> 80d64ec80 
>   docs/src/site/twiki/DG_SparkActionExtension.twiki 45b60b86b 
>   
> sharelib/spark/src/main/java/org/apache/oozie/action/hadoop/SparkArgsExtractor.java
>  052950fa3 
>   
> sharelib/spark/src/test/java/org/apache/oozie/action/hadoop/TestSparkActionExecutor.java
>  d97f1f06b 
>   
> sharelib/spark/src/test/java/org/apache/oozie/action/hadoop/TestSparkArgsExtractor.java
>  574bf2403 
> 
> 
> Diff: https://reviews.apache.org/r/63837/diff/5/
> 
> 
> Testing
> ---
> 
> 
> Thanks,
> 
> Peter Cseh
> 
>

Re: Review Request 63837: SparkConfigrationService overwrites properties provided via --properties-file option in SparkAction

[Peter Cseh via Review Board]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/63837/
---

(Updated Nov. 16, 2017, 10:04 a.m.)


Review request for oozie.


Bugs: OOZIE-3112
https://issues.apache.org/jira/browse/OOZIE-3112


Repository: oozie-git


Description
---

SparkConfigurationService injects the config values as --conf to SparkSubmit. 
This will overwrite propties provided in the --properies-file option which is 
not the expected behavior.


Diffs (updated)
-

  core/src/main/java/org/apache/oozie/action/hadoop/SparkActionExecutor.java 
80d64ec80 
  docs/src/site/twiki/DG_SparkActionExtension.twiki 45b60b86b 
  
sharelib/spark/src/main/java/org/apache/oozie/action/hadoop/SparkArgsExtractor.java
 052950fa3 
  
sharelib/spark/src/test/java/org/apache/oozie/action/hadoop/TestSparkActionExecutor.java
 d97f1f06b 
  
sharelib/spark/src/test/java/org/apache/oozie/action/hadoop/TestSparkArgsExtractor.java
 574bf2403 


Diff: https://reviews.apache.org/r/63837/diff/5/

Changes: https://reviews.apache.org/r/63837/diff/4-5/


Testing
---


Thanks,

Peter Cseh

Re: Review Request 63837: SparkConfigrationService overwrites properties provided via --properties-file option in SparkAction

[Peter Cseh via Review Board]

> On Nov. 15, 2017, 3:29 p.m., András Piros wrote:
> > sharelib/spark/src/main/java/org/apache/oozie/action/hadoop/SparkArgsExtractor.java
> > Lines 381 (patched)
> > 
> >
> > Typo: `Could not...`

I really have to learn to type...


> On Nov. 15, 2017, 3:29 p.m., András Piros wrote:
> > sharelib/spark/src/test/java/org/apache/oozie/action/hadoop/TestSparkActionExecutor.java
> > Line 107 (original), 104 (patched)
> > 
> >
> > When modified, please set it back to a runnable test method: `public 
> > void test()`

This is a helper function called from other tests. It shouldn't be called on 
it's own.


- Peter


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/63837/#review191066
---


On Nov. 16, 2017, 9:55 a.m., Peter Cseh wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/63837/
> ---
> 
> (Updated Nov. 16, 2017, 9:55 a.m.)
> 
> 
> Review request for oozie.
> 
> 
> Bugs: OOZIE-3112
> https://issues.apache.org/jira/browse/OOZIE-3112
> 
> 
> Repository: oozie-git
> 
> 
> Description
> ---
> 
> SparkConfigurationService injects the config values as --conf to SparkSubmit. 
> This will overwrite propties provided in the --properies-file option which is 
> not the expected behavior.
> 
> 
> Diffs
> -
> 
>   core/src/main/java/org/apache/oozie/action/hadoop/SparkActionExecutor.java 
> 80d64ec80 
>   docs/src/site/twiki/DG_SparkActionExtension.twiki 45b60b86b 
>   
> sharelib/spark/src/main/java/org/apache/oozie/action/hadoop/SparkArgsExtractor.java
>  052950fa3 
>   
> sharelib/spark/src/test/java/org/apache/oozie/action/hadoop/TestSparkActionExecutor.java
>  d97f1f06b 
>   
> sharelib/spark/src/test/java/org/apache/oozie/action/hadoop/TestSparkArgsExtractor.java
>  574bf2403 
> 
> 
> Diff: https://reviews.apache.org/r/63837/diff/4/
> 
> 
> Testing
> ---
> 
> 
> Thanks,
> 
> Peter Cseh
> 
>

Re: Review Request 63837: SparkConfigrationService overwrites properties provided via --properties-file option in SparkAction

[Peter Cseh via Review Board]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/63837/
---

(Updated Nov. 15, 2017, 2:55 p.m.)


Review request for oozie.


Bugs: OOZIE-3112
https://issues.apache.org/jira/browse/OOZIE-3112


Repository: oozie-git


Description
---

SparkConfigurationService injects the config values as --conf to SparkSubmit. 
This will overwrite propties provided in the --properies-file option which is 
not the expected behavior.


Diffs (updated)
-

  core/src/main/java/org/apache/oozie/action/hadoop/SparkActionExecutor.java 
80d64ec80e683ea571748c2df976cf6bd22063cd 
  docs/src/site/twiki/DG_SparkActionExtension.twiki 
45b60b86bee4cfe2de2d374f03b602ddacdc5e65 
  
sharelib/spark/src/main/java/org/apache/oozie/action/hadoop/SparkArgsExtractor.java
 052950fa36a0b40f8629ce91c1d135062a6571a9 
  
sharelib/spark/src/test/java/org/apache/oozie/action/hadoop/TestSparkActionExecutor.java
 d97f1f06b5832b1f09fab30b08ecff31db3221f6 
  
sharelib/spark/src/test/java/org/apache/oozie/action/hadoop/TestSparkArgsExtractor.java
 574bf240304379d5d26af8a3e5c079d4ad707b7b 


Diff: https://reviews.apache.org/r/63837/diff/3/

Changes: https://reviews.apache.org/r/63837/diff/2-3/


Testing
---


Thanks,

Peter Cseh

Re: Review Request 63837: SparkConfigrationService overwrites properties provided via --properties-file option in SparkAction

[Peter Cseh via Review Board]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/63837/
---

(Updated Nov. 15, 2017, 2:54 p.m.)


Review request for oozie.


Bugs: OOZIE-3112
https://issues.apache.org/jira/browse/OOZIE-3112


Repository: oozie-git


Description
---

SparkConfigurationService injects the config values as --conf to SparkSubmit. 
This will overwrite propties provided in the --properies-file option which is 
not the expected behavior.


Diffs (updated)
-

  client/src/main/java/org/apache/oozie/cli/CLIParser.java 
3930276f6dea23a774fed51421b9151bc158be29 
  client/src/main/java/org/apache/oozie/client/AuthOozieClient.java 
98f421f325fb44c89900db9b7a7763850106cd5f 
  client/src/main/java/org/apache/oozie/client/BundleJob.java 
0aede7191134f0089597d379b672ef6ab6ffb3d2 
  client/src/main/java/org/apache/oozie/client/WorkflowAction.java 
088d8f57dc00054142e84106a37597879bfcbeef 
  client/src/main/java/org/apache/oozie/client/WorkflowJob.java 
2d1cb4c2879a12c16cc5876841debf24f9a2024f 
  client/src/main/java/org/apache/oozie/client/rest/JsonTags.java 
e041354920c2ce501ef8784298193c1d1f11888d 
  client/src/main/java/org/apache/oozie/client/rest/JsonUtils.java 
b13ad3a618561235f446438cbdea422c07e536db 
  client/src/main/resources/oozie-common-1.0.xsd 
79e026dca028ffbf12528d0df7cae2a18989ec83 
  core/pom.xml 12e426e29866f57d6ea2bb41ef87d24ed87e6473 
  core/src/main/java/org/apache/oozie/action/email/EmailActionExecutor.java 
d59f1d76ce8069bd835bef944e1dbeec7be41ded 
  core/src/main/java/org/apache/oozie/action/hadoop/HCatCredentials.java 
9260d83f27c3d666f63c79063657822ce5a471c1 
  core/src/main/java/org/apache/oozie/action/hadoop/HbaseCredentials.java 
4d63c04a06b32d3c73aa307b43f722666470ae88 
  core/src/main/java/org/apache/oozie/action/hadoop/JavaActionExecutor.java 
8cb76cf086264ac09d8c9bbff22989d6e5eac69f 
  core/src/main/java/org/apache/oozie/action/hadoop/SparkActionExecutor.java 
80d64ec80e683ea571748c2df976cf6bd22063cd 
  core/src/main/java/org/apache/oozie/action/hadoop/YarnACLHandler.java 
46e621f1773db33b76410cf6fe700722c9b42006 
  core/src/main/java/org/apache/oozie/command/PurgeXCommand.java 
166c236f77c472b269cfd82c464259a40e6202d3 
  core/src/main/java/org/apache/oozie/command/bundle/BundleJobException.java 
d112fb46d9a0681a6e226f9ef1123416b3467b4d 
  
core/src/main/java/org/apache/oozie/command/bundle/BundleJobResumeXCommand.java 
e2bd69f80459e1b99d7d48ef8b6f45f29235c0f6 
  core/src/main/java/org/apache/oozie/command/bundle/BundleKillXCommand.java 
4e71791d0a2ca70796f241e9b792e31312418c74 
  
core/src/main/java/org/apache/oozie/command/coord/CoordActionReadyXCommand.java 
40bd93184b9840bc96396bd3f592720dbabc2972 
  
core/src/main/java/org/apache/oozie/executor/jpa/CoordJobDeleteJPAExecutor.java 
PRE-CREATION 
  core/src/main/java/org/apache/oozie/servlet/ErrorServlet.java 
94e93bfcb2f39e8ab31a9ce22246c2cd86cb907d 
  core/src/main/java/org/apache/oozie/servlet/JsonRestServlet.java 
03cf35c55e839a9cd2c6ffc0140970dc2689d3b9 
  core/src/main/java/org/apache/oozie/test/EmbeddedServletContainer.java 
59d20b577ef82f401dfe996fb666f70c298ade14 
  core/src/main/java/org/apache/oozie/workflow/lite/LauncherConfigHandler.java 
a6b8585c1125c1f25b866f54bd94aa80d41dbc9b 
  core/src/main/resources/oozie-default.xml 
91d281125b236a9641faff54a69947b4440254d5 
  core/src/test/java/org/apache/oozie/action/hadoop/ActionExecutorTestCase.java 
f39bba2c691435354dac6da7794e5142b511d937 
  core/src/test/java/org/apache/oozie/action/hadoop/LauncherMainTester.java 
fdca5706efbc13741477cee186550d07eee9f737 
  core/src/test/java/org/apache/oozie/action/hadoop/TestJavaActionExecutor.java 
1cc569d8d7a0e61905ad6c4e940cefb00da1343d 
  
core/src/test/java/org/apache/oozie/action/hadoop/TestLauncherConfigurationInjector.java
 bcc55bff18784c92be15b46b66c3344d822e0f2a 
  core/src/test/java/org/apache/oozie/action/hadoop/TestYarnACLHandler.java 
bcb36d39fe92ad90ab539b9cb7d0d29878552af8 
  core/src/test/java/org/apache/oozie/command/TestPurgeXCommand.java 
af9591bb5b5fe05e4832e3a0015c53fb1cb8c023 
  
core/src/test/java/org/apache/oozie/executor/jpa/TestCoordJobDeleteJPAExecutor.java
 PRE-CREATION 
  core/src/test/java/org/apache/oozie/service/TestJobsConcurrencyService.java 
513784a8516035a09fca8824390d661ac062be8d 
  core/src/test/java/org/apache/oozie/service/TestSchemaService.java 
d0a3fdf2bb566f75cfa0f48bcb448e5171d76b08 
  core/src/test/java/org/apache/oozie/servlet/TestErrorServlet.java 
cf5547b4a8ff8585587b64817fce41b44def829c 
  core/src/test/java/org/apache/oozie/servlet/TestJsonRestServlet.java 
60eadbb5325c1334e0ca909eb8281bae8cf2aaa6 
  core/src/test/java/org/apache/oozie/test/XTestCase.java 
8cecb89be8d6c45ed7bd9d5d4eafad2044454362 
  core/src/test/java/org/apache/oozie/test/ZKXTestCase.java 
abc7c9fb9262646887e8a7cc6fcb9785d7a12c0d 
  

Review Request 63837: SparkConfigrationService overwrites properties provided via --properties-file option in SparkAction

[Peter Cseh via Review Board]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/63837/
---

Review request for oozie.


Bugs: OOZIE-3112
https://issues.apache.org/jira/browse/OOZIE-3112


Repository: oozie-git


Description
---

SparkConfigurationService injects the config values as --conf to SparkSubmit. 
This will overwrite propties provided in the --properies-file option which is 
not the expected behavior.


Diffs
-

  client/src/main/java/org/apache/oozie/cli/CLIParser.java 
3930276f6dea23a774fed51421b9151bc158be29 
  client/src/main/java/org/apache/oozie/client/AuthOozieClient.java 
98f421f325fb44c89900db9b7a7763850106cd5f 
  client/src/main/java/org/apache/oozie/client/BundleJob.java 
0aede7191134f0089597d379b672ef6ab6ffb3d2 
  client/src/main/java/org/apache/oozie/client/WorkflowAction.java 
088d8f57dc00054142e84106a37597879bfcbeef 
  client/src/main/java/org/apache/oozie/client/WorkflowJob.java 
2d1cb4c2879a12c16cc5876841debf24f9a2024f 
  client/src/main/java/org/apache/oozie/client/rest/JsonTags.java 
e041354920c2ce501ef8784298193c1d1f11888d 
  client/src/main/java/org/apache/oozie/client/rest/JsonUtils.java 
b13ad3a618561235f446438cbdea422c07e536db 
  client/src/main/resources/oozie-common-1.0.xsd 
79e026dca028ffbf12528d0df7cae2a18989ec83 
  core/pom.xml 12e426e29866f57d6ea2bb41ef87d24ed87e6473 
  core/src/main/java/org/apache/oozie/action/email/EmailActionExecutor.java 
d59f1d76ce8069bd835bef944e1dbeec7be41ded 
  core/src/main/java/org/apache/oozie/action/hadoop/HCatCredentials.java 
9260d83f27c3d666f63c79063657822ce5a471c1 
  core/src/main/java/org/apache/oozie/action/hadoop/HbaseCredentials.java 
4d63c04a06b32d3c73aa307b43f722666470ae88 
  core/src/main/java/org/apache/oozie/action/hadoop/JavaActionExecutor.java 
8cb76cf086264ac09d8c9bbff22989d6e5eac69f 
  core/src/main/java/org/apache/oozie/action/hadoop/SparkActionExecutor.java 
80d64ec80e683ea571748c2df976cf6bd22063cd 
  core/src/main/java/org/apache/oozie/action/hadoop/YarnACLHandler.java 
46e621f1773db33b76410cf6fe700722c9b42006 
  core/src/main/java/org/apache/oozie/command/PurgeXCommand.java 
166c236f77c472b269cfd82c464259a40e6202d3 
  core/src/main/java/org/apache/oozie/command/bundle/BundleJobException.java 
d112fb46d9a0681a6e226f9ef1123416b3467b4d 
  
core/src/main/java/org/apache/oozie/command/bundle/BundleJobResumeXCommand.java 
e2bd69f80459e1b99d7d48ef8b6f45f29235c0f6 
  core/src/main/java/org/apache/oozie/command/bundle/BundleKillXCommand.java 
4e71791d0a2ca70796f241e9b792e31312418c74 
  
core/src/main/java/org/apache/oozie/command/coord/CoordActionReadyXCommand.java 
40bd93184b9840bc96396bd3f592720dbabc2972 
  
core/src/main/java/org/apache/oozie/executor/jpa/CoordJobDeleteJPAExecutor.java 
PRE-CREATION 
  core/src/main/java/org/apache/oozie/servlet/ErrorServlet.java 
94e93bfcb2f39e8ab31a9ce22246c2cd86cb907d 
  core/src/main/java/org/apache/oozie/servlet/JsonRestServlet.java 
03cf35c55e839a9cd2c6ffc0140970dc2689d3b9 
  core/src/main/java/org/apache/oozie/test/EmbeddedServletContainer.java 
59d20b577ef82f401dfe996fb666f70c298ade14 
  core/src/main/java/org/apache/oozie/workflow/lite/LauncherConfigHandler.java 
a6b8585c1125c1f25b866f54bd94aa80d41dbc9b 
  core/src/main/resources/oozie-default.xml 
91d281125b236a9641faff54a69947b4440254d5 
  core/src/test/java/org/apache/oozie/action/hadoop/ActionExecutorTestCase.java 
f39bba2c691435354dac6da7794e5142b511d937 
  core/src/test/java/org/apache/oozie/action/hadoop/LauncherMainTester.java 
fdca5706efbc13741477cee186550d07eee9f737 
  core/src/test/java/org/apache/oozie/action/hadoop/TestJavaActionExecutor.java 
1cc569d8d7a0e61905ad6c4e940cefb00da1343d 
  
core/src/test/java/org/apache/oozie/action/hadoop/TestLauncherConfigurationInjector.java
 bcc55bff18784c92be15b46b66c3344d822e0f2a 
  core/src/test/java/org/apache/oozie/action/hadoop/TestYarnACLHandler.java 
bcb36d39fe92ad90ab539b9cb7d0d29878552af8 
  core/src/test/java/org/apache/oozie/command/TestPurgeXCommand.java 
af9591bb5b5fe05e4832e3a0015c53fb1cb8c023 
  
core/src/test/java/org/apache/oozie/executor/jpa/TestCoordJobDeleteJPAExecutor.java
 PRE-CREATION 
  core/src/test/java/org/apache/oozie/service/TestJobsConcurrencyService.java 
513784a8516035a09fca8824390d661ac062be8d 
  core/src/test/java/org/apache/oozie/service/TestSchemaService.java 
d0a3fdf2bb566f75cfa0f48bcb448e5171d76b08 
  core/src/test/java/org/apache/oozie/servlet/TestErrorServlet.java 
cf5547b4a8ff8585587b64817fce41b44def829c 
  core/src/test/java/org/apache/oozie/servlet/TestJsonRestServlet.java 
60eadbb5325c1334e0ca909eb8281bae8cf2aaa6 
  core/src/test/java/org/apache/oozie/test/XTestCase.java 
8cecb89be8d6c45ed7bd9d5d4eafad2044454362 
  core/src/test/java/org/apache/oozie/test/ZKXTestCase.java 
abc7c9fb9262646887e8a7cc6fcb9785d7a12c0d 
  
core/src/test/java/org/apache/oozie/workflow/lite/TestLiteWorkflowAppParser.java
 

Review Request 63836: OOZIE-2869 Ability to add/remove MapReduce jars from action classpath.

[Peter Cseh via Review Board]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/63836/
---

Review request for oozie.


Bugs: OOZIE-2869
https://issues.apache.org/jira/browse/OOZIE-2869


Repository: oozie-git


Description
---

I've introduced a property to provide the possibility to add/remove jars from 
the classpath for each action type using the 
"oozie.action.mapreduce.needed.for" prefix.


Diffs
-

  core/src/main/java/org/apache/oozie/action/hadoop/DistcpActionExecutor.java 
961994ef14c5ef86be6609cde68bd98b39af8a2f 
  core/src/main/java/org/apache/oozie/action/hadoop/HiveActionExecutor.java 
d8c613bbbf9ed0d6b741a83861aeb82009084e3c 
  core/src/main/java/org/apache/oozie/action/hadoop/JavaActionExecutor.java 
8cb76cf086264ac09d8c9bbff22989d6e5eac69f 
  
core/src/main/java/org/apache/oozie/action/hadoop/MapReduceActionExecutor.java 
233dce7838b8d0ff0a18317d00768e87a0e5d7c8 
  core/src/main/java/org/apache/oozie/action/hadoop/PigActionExecutor.java 
6541d52e35689a3776fa2ab3e6eb420c14c16a12 
  core/src/main/java/org/apache/oozie/action/hadoop/SparkActionExecutor.java 
80d64ec80e683ea571748c2df976cf6bd22063cd 
  core/src/main/java/org/apache/oozie/action/hadoop/SqoopActionExecutor.java 
a0dfd31775507aeafaf09f1a316da0f52eac0a22 
  core/src/main/resources/oozie-default.xml 
91d281125b236a9641faff54a69947b4440254d5 
  sharelib/spark/pom.xml b40a3528137cedf58c0591cf9d010c38eec75d44 


Diff: https://reviews.apache.org/r/63836/diff/1/


Testing
---


Thanks,

Peter Cseh

Re: Review Request 63327: OOZIE-2897 LauncherAM should support ACLs

[Peter Cseh via Review Board]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/63327/#review190306
---


Ship it!




Ship It!

- Peter Cseh


On Nov. 7, 2017, 12:36 p.m., Peter Bacsko wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/63327/
> ---
> 
> (Updated Nov. 7, 2017, 12:36 p.m.)
> 
> 
> Review request for oozie, András Piros, Attila Sasvari, Peter Cseh, and 
> Robert Kanter.
> 
> 
> Repository: oozie-git
> 
> 
> Description
> ---
> 
> https://issues.apache.org/jira/browse/OOZIE-2897
> 
> 
> Diffs
> -
> 
>   client/src/main/resources/oozie-common-1.0.xsd ddae91224 
>   core/src/main/java/org/apache/oozie/action/hadoop/JavaActionExecutor.java 
> 6a25e5c89 
>   core/src/main/java/org/apache/oozie/action/hadoop/YarnACLHandler.java 
> PRE-CREATION 
>   
> core/src/main/java/org/apache/oozie/workflow/lite/LauncherConfigHandler.java 
> c36774239 
>   core/src/main/resources/oozie-default.xml 8285df0a7 
>   
> core/src/test/java/org/apache/oozie/action/hadoop/ActionExecutorTestCase.java 
> 36b302667 
>   core/src/test/java/org/apache/oozie/action/hadoop/LauncherMainTester.java 
> 43c71b0ad 
>   
> core/src/test/java/org/apache/oozie/action/hadoop/TestJavaActionExecutor.java 
> adb2168ca 
>   
> core/src/test/java/org/apache/oozie/action/hadoop/TestLauncherConfigurationInjector.java
>  4c5b0f972 
>   core/src/test/java/org/apache/oozie/action/hadoop/TestYarnACLHandler.java 
> PRE-CREATION 
>   core/src/test/java/org/apache/oozie/service/TestSchemaService.java 
> 940868aeb 
>   core/src/test/java/org/apache/oozie/test/XTestCase.java 912c3f316 
>   
> core/src/test/java/org/apache/oozie/workflow/lite/TestLiteWorkflowAppParser.java
>  a361078ac 
>   core/src/test/resources/fair-scheduler-alloc.xml PRE-CREATION 
>   core/src/test/resources/wf-schema-global-launcherconf.xml 9cd4f6c37 
> 
> 
> Diff: https://reviews.apache.org/r/63327/diff/4/
> 
> 
> Testing
> ---
> 
> - Unit test added
> - Verified on a 4-node cluster
> 
> 
> Thanks,
> 
> Peter Bacsko
> 
>

Re: Review Request 63327: OOZIE-2897 LauncherAM should support ACLs

[Peter Cseh via Review Board]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/63327/#review190152
---




core/src/test/java/org/apache/oozie/action/hadoop/TestJavaActionExecutor.java
Lines 92 (patched)


Unused import.



core/src/test/java/org/apache/oozie/test/XTestCase.java
Lines 1028-1030 (original), 1029-1031 (patched)


These properties should be set in case of capacity scheduler used.


- Peter Cseh


On Nov. 6, 2017, 12:08 p.m., Peter Bacsko wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/63327/
> ---
> 
> (Updated Nov. 6, 2017, 12:08 p.m.)
> 
> 
> Review request for oozie, András Piros, Attila Sasvari, Peter Cseh, and 
> Robert Kanter.
> 
> 
> Repository: oozie-git
> 
> 
> Description
> ---
> 
> https://issues.apache.org/jira/browse/OOZIE-2897
> 
> 
> Diffs
> -
> 
>   client/src/main/resources/oozie-common-1.0.xsd ddae91224 
>   core/src/main/java/org/apache/oozie/action/hadoop/JavaActionExecutor.java 
> 6a25e5c89 
>   core/src/main/java/org/apache/oozie/action/hadoop/YarnACLHandler.java 
> PRE-CREATION 
>   
> core/src/main/java/org/apache/oozie/workflow/lite/LauncherConfigHandler.java 
> c36774239 
>   core/src/main/resources/oozie-default.xml 8285df0a7 
>   
> core/src/test/java/org/apache/oozie/action/hadoop/ActionExecutorTestCase.java 
> 36b302667 
>   core/src/test/java/org/apache/oozie/action/hadoop/LauncherMainTester.java 
> 43c71b0ad 
>   
> core/src/test/java/org/apache/oozie/action/hadoop/TestJavaActionExecutor.java 
> adb2168ca 
>   
> core/src/test/java/org/apache/oozie/action/hadoop/TestLauncherConfigurationInjector.java
>  4c5b0f972 
>   core/src/test/java/org/apache/oozie/action/hadoop/TestYarnACLHandler.java 
> PRE-CREATION 
>   core/src/test/java/org/apache/oozie/service/TestSchemaService.java 
> 940868aeb 
>   core/src/test/java/org/apache/oozie/test/XTestCase.java 912c3f316 
>   
> core/src/test/java/org/apache/oozie/workflow/lite/TestLiteWorkflowAppParser.java
>  a361078ac 
>   core/src/test/resources/fair-scheduler-alloc.xml PRE-CREATION 
>   core/src/test/resources/wf-schema-global-launcherconf.xml 9cd4f6c37 
> 
> 
> Diff: https://reviews.apache.org/r/63327/diff/3/
> 
> 
> Testing
> ---
> 
> - Unit test added
> - Verified on a 4-node cluster
> 
> 
> Thanks,
> 
> Peter Bacsko
> 
>

Re: Review Request 63327: OOZIE-2897 LauncherAM should support ACLs

[Peter Cseh via Review Board]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/63327/#review189933
---




core/src/main/java/org/apache/oozie/action/hadoop/YarnACLHandler.java
Lines 38-40 (patched)


Why are you checking oozie.launcher.override here?


- Peter Cseh


On Oct. 31, 2017, 3:02 p.m., Peter Bacsko wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/63327/
> ---
> 
> (Updated Oct. 31, 2017, 3:02 p.m.)
> 
> 
> Review request for oozie, András Piros, Attila Sasvari, Peter Cseh, and 
> Robert Kanter.
> 
> 
> Repository: oozie-git
> 
> 
> Description
> ---
> 
> https://issues.apache.org/jira/browse/OOZIE-2897
> 
> 
> Diffs
> -
> 
>   client/src/main/resources/oozie-common-1.0.xsd ddae91224 
>   core/src/main/java/org/apache/oozie/action/hadoop/JavaActionExecutor.java 
> 6a25e5c89 
>   core/src/main/java/org/apache/oozie/action/hadoop/YarnACLHandler.java 
> PRE-CREATION 
>   
> core/src/main/java/org/apache/oozie/workflow/lite/LauncherConfigHandler.java 
> c36774239 
>   core/src/main/resources/oozie-default.xml b2036997d 
>   
> core/src/test/java/org/apache/oozie/action/hadoop/ActionExecutorTestCase.java 
> 36b302667 
>   core/src/test/java/org/apache/oozie/action/hadoop/LauncherMainTester.java 
> 43c71b0ad 
>   
> core/src/test/java/org/apache/oozie/action/hadoop/TestJavaActionExecutor.java 
> adb2168ca 
>   
> core/src/test/java/org/apache/oozie/action/hadoop/TestLauncherConfigurationInjector.java
>  4c5b0f972 
>   core/src/test/java/org/apache/oozie/action/hadoop/TestYarnACLHandler.java 
> PRE-CREATION 
>   core/src/test/java/org/apache/oozie/service/TestSchemaService.java 
> 940868aeb 
>   core/src/test/java/org/apache/oozie/test/XTestCase.java 584aa12ef 
>   
> core/src/test/java/org/apache/oozie/workflow/lite/TestLiteWorkflowAppParser.java
>  a361078ac 
>   core/src/test/resources/fair-scheduler-alloc.xml PRE-CREATION 
>   core/src/test/resources/wf-schema-global-launcherconf.xml 9cd4f6c37 
> 
> 
> Diff: https://reviews.apache.org/r/63327/diff/2/
> 
> 
> Testing
> ---
> 
> - Unit test added
> - Verified on a 4-node cluster
> 
> 
> Thanks,
> 
> Peter Bacsko
> 
>

Re: Review Request 62936: OOZIE-2896 Ensure compatibility for existing LauncherMapper settings

[Peter Cseh via Review Board]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/62936/#review189412
---


Ship it!




Ship It!

- Peter Cseh


On Oct. 27, 2017, 10:01 a.m., András Piros wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/62936/
> ---
> 
> (Updated Oct. 27, 2017, 10:01 a.m.)
> 
> 
> Review request for oozie, Attila Sasvari, Peter Cseh, and Peter Bacsko.
> 
> 
> Repository: oozie-git
> 
> 
> Description
> ---
> 
> OOZIE-2896 Ensure compatibility for existing LauncherMapper settings
> 
> 
> Diffs
> -
> 
>   core/src/main/java/org/apache/oozie/action/hadoop/JavaActionExecutor.java 
> 76d0daa4f7528de6fb3c8993bae4dd28b6f7865b 
>   
> core/src/main/java/org/apache/oozie/action/hadoop/LauncherConfigurationInjector.java
>  PRE-CREATION 
>   core/src/main/java/org/apache/oozie/service/ConfigurationService.java 
> a51933022c14ea4ce06a65cc3c123bed8a8f7f34 
>   core/src/main/resources/oozie-default.xml 
> 9ba8fd4a0d9dd5f0bd6a64af3724c2f910d86457 
>   
> core/src/test/java/org/apache/oozie/action/hadoop/TestJavaActionExecutor.java 
> a7bd357f8938d148a150db43135a7a8fe7f0da4c 
>   
> core/src/test/java/org/apache/oozie/action/hadoop/TestLauncherConfigurationInjector.java
>  PRE-CREATION 
>   sharelib/oozie/src/main/java/org/apache/oozie/action/hadoop/LauncherAM.java 
> ee1a32ac9b071009e1dbe78396520d778f58bddc 
> 
> 
> Diff: https://reviews.apache.org/r/62936/diff/6/
> 
> 
> Testing
> ---
> 
> JUnit tests:
> 
> * `TestJavaActionExecutor`
> * `TestLauncherConfigurationFilter`
> 
> Functional tests: submit and run `examples/apps/java-main/workflow.xml` in 
> three modes:
> 
> * only `oozie.launcher.` parameters are present: `oozie.launcher.` parameters 
> are passed through
> * `oozie.launcher.` and override (`yarn.`, `mapreduce.`, and `mapred.`) 
> parameters are also present: `oozie.launcher.` parameters are passed through
> * only override (`yarn.`, `mapreduce.`, and `mapred.`) parameters are 
> present: override parameters are passed through
> 
> 
> Thanks,
> 
> András Piros
> 
>

Re: Review Request 62936: OOZIE-2896 Ensure compatibility for existing LauncherMapper settings

[Peter Cseh via Review Board]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/62936/#review189406
---




core/src/main/java/org/apache/oozie/action/hadoop/LauncherConfigurationInjector.java
Lines 316 (patched)


Does this mean that we're not overwriting things if there was already a 
value given for it?



core/src/main/resources/oozie-default.xml
Lines 3120 (patched)


I don't like that it's max.attempts in most cases and max-attempts here.


- Peter Cseh


On Oct. 26, 2017, 5:20 p.m., András Piros wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/62936/
> ---
> 
> (Updated Oct. 26, 2017, 5:20 p.m.)
> 
> 
> Review request for oozie, Attila Sasvari, Peter Cseh, and Peter Bacsko.
> 
> 
> Repository: oozie-git
> 
> 
> Description
> ---
> 
> OOZIE-2896 Ensure compatibility for existing LauncherMapper settings
> 
> 
> Diffs
> -
> 
>   core/src/main/java/org/apache/oozie/action/hadoop/JavaActionExecutor.java 
> 76d0daa4f7528de6fb3c8993bae4dd28b6f7865b 
>   
> core/src/main/java/org/apache/oozie/action/hadoop/LauncherConfigurationInjector.java
>  PRE-CREATION 
>   core/src/main/java/org/apache/oozie/service/ConfigurationService.java 
> a51933022c14ea4ce06a65cc3c123bed8a8f7f34 
>   core/src/main/resources/oozie-default.xml 
> 9ba8fd4a0d9dd5f0bd6a64af3724c2f910d86457 
>   
> core/src/test/java/org/apache/oozie/action/hadoop/TestJavaActionExecutor.java 
> a7bd357f8938d148a150db43135a7a8fe7f0da4c 
>   
> core/src/test/java/org/apache/oozie/action/hadoop/TestLauncherConfigurationInjector.java
>  PRE-CREATION 
>   sharelib/oozie/src/main/java/org/apache/oozie/action/hadoop/LauncherAM.java 
> ee1a32ac9b071009e1dbe78396520d778f58bddc 
> 
> 
> Diff: https://reviews.apache.org/r/62936/diff/5/
> 
> 
> Testing
> ---
> 
> JUnit tests:
> 
> * `TestJavaActionExecutor`
> * `TestLauncherConfigurationFilter`
> 
> Functional tests: submit and run `examples/apps/java-main/workflow.xml` in 
> three modes:
> 
> * only `oozie.launcher.` parameters are present: `oozie.launcher.` parameters 
> are passed through
> * `oozie.launcher.` and override (`yarn.`, `mapreduce.`, and `mapred.`) 
> parameters are also present: `oozie.launcher.` parameters are passed through
> * only override (`yarn.`, `mapreduce.`, and `mapred.`) parameters are 
> present: override parameters are passed through
> 
> 
> Thanks,
> 
> András Piros
> 
>

Re: Review Request 62459: OOZIE-2296: Add an Oozie diagnostic bundle tool

[Peter Cseh via Review Board]

> On Oct. 6, 2017, 12:26 a.m., Robert Kanter wrote:
> > tools/src/main/java/org/apache/oozie/tools/diag/AppInfoCollector.java
> > Lines 173-174 (patched)
> > 
> >
> > LogAggregationUtils is marked @Private so we shouldn't 
> > be using it.  Hadoop can change things incompatibly here.
> 
> Attila Sasvari wrote:
> What do you think about borrowing/inlining/copying those functions from 
> Hadoop 2.6 too? I was thinking about it and it is probably better than 
> bringing back the initial ``ExecutorService`` based approach. I am thinking 
> of a class like:
> 
> ```java
> // TODO: once OOZIE-2983 ("Stream the Launcher AM Logs") is done, remove 
> it.
> public class OozieLauncherLogFetcher {
> private static final String TMP_FILE_SUFFIX = ".tmp";
> final private Configuration hadoopConfig;
> 
> public OozieLauncherLogFetcher(final Configuration hadoopConfig) {
> this.hadoopConfig = hadoopConfig;
> }
> 
> // Borrowed code from 
> org.apache.hadoop.yarn.logaggregation.LogCLIHelpers
> private static void logDirNotExist(String remoteAppLogDir) {
> System.out.println(remoteAppLogDir + "does not exist.");
> System.out.println("Log aggregation has not completed or is not 
> enabled.");
> }
> // Borrowed code from 
> org.apache.hadoop.yarn.logaggregation.LogCLIHelpers
> private static void emptyLogDir(String remoteAppLogDir) {
> System.out.println(remoteAppLogDir + "does not have any log 
> files.");
> }
> // Borrowed code from 
> org.apache.hadoop.yarn.logaggregation.LogAggregationUtils
> public static String getRemoteNodeLogDirSuffix(Configuration conf) {
> return conf.get(YarnConfiguration.NM_REMOTE_APP_LOG_DIR_SUFFIX, 
> YarnConfiguration.DEFAULT_NM_REMOTE_APP_LOG_DIR_SUFFIX);
> }
> // Borrowed code from 
> org.apache.hadoop.yarn.logaggregation.LogAggregationUtils
> public static Path getRemoteLogSuffixedDir(Path remoteRootLogDir, 
> String user, String suffix) {
> return suffix != null && !suffix.isEmpty() ? new 
> Path(getRemoteLogUserDir(remoteRootLogDir, user), suffix) :
> getRemoteLogUserDir(remoteRootLogDir, user);
> }
> // Borrowed code from 
> org.apache.hadoop.yarn.logaggregation.LogAggregationUtils
> public static Path getRemoteLogUserDir(Path remoteRootLogDir, String 
> user) {
> return new Path(remoteRootLogDir, user);
> }
> 
> // Borrowed code from 
> org.apache.hadoop.yarn.logaggregation.LogAggregationUtils
> public static Path getRemoteAppLogDir(Path remoteRootLogDir, 
> ApplicationId appId, String user, String suffix) {
> return new Path(getRemoteLogSuffixedDir(remoteRootLogDir, user, 
> suffix), appId.toString());
> }
> 
> // Borrowed code from 
> org.apache.hadoop.yarn.logaggregation.LogCLIHelpers
> public int dumpAllContainersLogs(ApplicationId appId, String 
> appOwner, PrintStream out) throws IOException {
> Path remoteRootLogDir = new 
> Path(hadoopConfig.get(YarnConfiguration.NM_REMOTE_APP_LOG_DIR,
> YarnConfiguration.DEFAULT_NM_REMOTE_APP_LOG_DIR));
> String logDirSuffix = getRemoteNodeLogDirSuffix(hadoopConfig);
> Path remoteAppLogDir = getRemoteAppLogDir(remoteRootLogDir, 
> appId, appOwner, logDirSuffix);
> 
> RemoteIterator nodeFiles;
> try {
> Path qualifiedLogDir = 
> FileContext.getFileContext(hadoopConfig).makeQualified(remoteAppLogDir);
> nodeFiles = 
> FileContext.getFileContext(qualifiedLogDir.toUri(), 
> hadoopConfig).listStatus(remoteAppLogDir);
> } catch (FileNotFoundException fileNotFoundException) {
> logDirNotExist(remoteAppLogDir.toString());
> return -1;
> }
> 
> boolean foundAnyLogs = false;
> 
> while(true) {
> FileStatus thisNodeFile;
> do {
> if (!nodeFiles.hasNext()) {
> if (!foundAnyLogs) {
> emptyLogDir(remoteAppLogDir.toString());
> return -1;
> }
> 
> return 0;
> }
> 
> thisNodeFile = (FileStatus)nodeFiles.next();
> } 
> while(thisNodeFile.getPath().getName().endsWith(TMP_FILE_SUFFIX));
> 
> AggregatedLogFormat.LogReader reader = new 
> AggregatedLogFormat.LogReader(hadoopConfig, thisNodeFile.getPath());
> 
> try {
> AggregatedLogFormat.LogKey key = new 
> AggregatedLogFormat.LogKey();
> DataInputStream 

Re: Review Request 62936: OOZIE-2896 Ensure compatibility for existing LauncherMapper settings

[Peter Cseh via Review Board]

> On Oct. 12, 2017, 11:36 p.m., Robert Kanter wrote:
> > core/src/main/java/org/apache/oozie/action/hadoop/JavaActionExecutor.java
> > Lines 1197 (patched)
> > 
> >
> > This is fine to leave for a different JIRA, but we should consider 
> > dropping the "oozie:launcher:" part of the name.  The job type is now 
> > OOZIE, so it's probably redundent and we can shorten the names.

OOZIE-2893


- Peter


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/62936/#review187893
---


On Oct. 12, 2017, 3:33 p.m., András Piros wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/62936/
> ---
> 
> (Updated Oct. 12, 2017, 3:33 p.m.)
> 
> 
> Review request for oozie, Attila Sasvari, Peter Cseh, and Peter Bacsko.
> 
> 
> Repository: oozie-git
> 
> 
> Description
> ---
> 
> OOZIE-2896 Ensure compatibility for existing LauncherMapper settings
> 
> 
> Diffs
> -
> 
>   core/src/main/java/org/apache/oozie/action/hadoop/JavaActionExecutor.java 
> 76d0daa4f7528de6fb3c8993bae4dd28b6f7865b 
>   
> core/src/main/java/org/apache/oozie/action/hadoop/LauncherConfigurationFilter.java
>  PRE-CREATION 
>   core/src/main/resources/oozie-default.xml 
> 2389b993b7b1769521d6dd0c7386e99e7516210a 
>   
> core/src/test/java/org/apache/oozie/action/hadoop/TestJavaActionExecutor.java 
> a7bd357f8938d148a150db43135a7a8fe7f0da4c 
>   
> core/src/test/java/org/apache/oozie/action/hadoop/TestLauncherConfigurationFilter.java
>  PRE-CREATION 
>   sharelib/oozie/src/main/java/org/apache/oozie/action/hadoop/LauncherAM.java 
> ee1a32ac9b071009e1dbe78396520d778f58bddc 
> 
> 
> Diff: https://reviews.apache.org/r/62936/diff/1/
> 
> 
> Testing
> ---
> 
> JUnit tests:
> 
> * `TestJavaActionExecutor`
> * `TestLauncherConfigurationFilter`
> 
> Functional tests: submit and run `examples/apps/java-main/workflow.xml` in 
> three modes:
> 
> * only `oozie.launcher.` parameters are present: `oozie.launcher.` parameters 
> are passed through
> * `oozie.launcher.` and override (`yarn.`, `mapreduce.`, and `mapred.`) 
> parameters are also present: `oozie.launcher.` parameters are passed through
> * only override (`yarn.`, `mapreduce.`, and `mapred.`) parameters are 
> present: override parameters are passed through
> 
> 
> Thanks,
> 
> András Piros
> 
>

Re: Review Request 62936: OOZIE-2896 Ensure compatibility for existing LauncherMapper settings

[Peter Cseh via Review Board]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/62936/#review187806
---




core/src/main/java/org/apache/oozie/action/hadoop/JavaActionExecutor.java
Lines 1174-1175 (original), 1148-1183 (patched)


Why move this huge block? Things like this make backports harder.



core/src/main/java/org/apache/oozie/action/hadoop/JavaActionExecutor.java
Line 1183 (original), 1259 (patched)


Shouldn't the ",CLA" part be kept here? 
If it's just the log level, the appender should stay the same.



core/src/main/java/org/apache/oozie/action/hadoop/LauncherConfigurationFilter.java
Lines 32-52 (patched)


Please use the configuration.getValByRegex() and use 
oozie.launcher.override.* and oozie.launcher.prepend.*

Also, we're calling the LauncherAM as "launcher" in the configfiles, not 
"am" we should use the same properties from LauncherAM like 
LauncherAM.OOZIE_LAUNCHER_VCORES_PROPERTY and use the prefixes for it.

You could do ALL the overrites first and all the prepends later.

This way when we add a new property, it will just work. (I may come handy 
when resourcetypes are released in Hadoop)



core/src/main/java/org/apache/oozie/action/hadoop/LauncherConfigurationFilter.java
Lines 61 (patched)


this method does not filter. It injects/copies properties over.



core/src/test/java/org/apache/oozie/action/hadoop/TestJavaActionExecutor.java
Line 2183 (original), 2183 (patched)


What is the extra property appearing here? The converted attempt count?



core/src/test/java/org/apache/oozie/action/hadoop/TestJavaActionExecutor.java
Lines 2548-2563 (patched)


There are a whole lot of things set here without any assertion in the test 
down the road.
Either remove the unneccessary properties or assert them.


- Peter Cseh


On Oct. 12, 2017, 3:33 p.m., András Piros wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/62936/
> ---
> 
> (Updated Oct. 12, 2017, 3:33 p.m.)
> 
> 
> Review request for oozie, Attila Sasvari, Peter Cseh, and Peter Bacsko.
> 
> 
> Repository: oozie-git
> 
> 
> Description
> ---
> 
> OOZIE-2896 Ensure compatibility for existing LauncherMapper settings
> 
> 
> Diffs
> -
> 
>   core/src/main/java/org/apache/oozie/action/hadoop/JavaActionExecutor.java 
> 76d0daa4f7528de6fb3c8993bae4dd28b6f7865b 
>   
> core/src/main/java/org/apache/oozie/action/hadoop/LauncherConfigurationFilter.java
>  PRE-CREATION 
>   core/src/main/resources/oozie-default.xml 
> 2389b993b7b1769521d6dd0c7386e99e7516210a 
>   
> core/src/test/java/org/apache/oozie/action/hadoop/TestJavaActionExecutor.java 
> a7bd357f8938d148a150db43135a7a8fe7f0da4c 
>   
> core/src/test/java/org/apache/oozie/action/hadoop/TestLauncherConfigurationFilter.java
>  PRE-CREATION 
>   sharelib/oozie/src/main/java/org/apache/oozie/action/hadoop/LauncherAM.java 
> ee1a32ac9b071009e1dbe78396520d778f58bddc 
> 
> 
> Diff: https://reviews.apache.org/r/62936/diff/1/
> 
> 
> Testing
> ---
> 
> JUnit tests:
> 
> * `TestJavaActionExecutor`
> * `TestLauncherConfigurationFilter`
> 
> Functional tests: submit and run `examples/apps/java-main/workflow.xml` in 
> three modes:
> 
> * only `oozie.launcher.` parameters are present: `oozie.launcher.` parameters 
> are passed through
> * `oozie.launcher.` and override (`yarn.`, `mapreduce.`, and `mapred.`) 
> parameters are also present: `oozie.launcher.` parameters are passed through
> * only override (`yarn.`, `mapreduce.`, and `mapred.`) parameters are 
> present: override parameters are passed through
> 
> 
> Thanks,
> 
> András Piros
> 
>