Review Request 54705: OOZIE-1770: Create Oozie Application Master for YARN

[Peter Cseh]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/54705/
---

Review request for oozie, Purshotam Shah, Robert Kanter, and Rohini Palaniswamy.


Repository: oozie-git


Description
---

Replacing the launcher MR job with a launcher AM


Diffs
-

  client/pom.xml f99366c90887a5a45f3942291c90e1cdbfd84b3f 
  client/src/main/java/org/apache/oozie/client/XOozieClient.java 
8538ec7c742cc013f7c18eaff3ecf61c2835cc8a 
  core/pom.xml 1dd104581ccab7976d788b674d2200e48a70c27f 
  core/src/main/java/org/apache/oozie/WorkflowJobBean.java 
55d79a5d5e5e4efab9eb832e9341a16bcf8959cc 
  core/src/main/java/org/apache/oozie/action/ActionExecutor.java 
1d6456b236bf9e27c0f55d9f9e3b4bd94517c797 
  core/src/main/java/org/apache/oozie/action/hadoop/Credentials.java 
eadb47b07d157f45f086517ab41c4b9826f003f0 
  core/src/main/java/org/apache/oozie/action/hadoop/CredentialsProvider.java 
6fe22fbd2fca5d873d58ece60f20d0f131371475 
  core/src/main/java/org/apache/oozie/action/hadoop/DistcpActionExecutor.java 
78cd257a836707c1efeac13566d6285d42970545 
  core/src/main/java/org/apache/oozie/action/hadoop/FsActionExecutor.java 
27654741101d2056b25626fbbe345183eafd612d 
  core/src/main/java/org/apache/oozie/action/hadoop/FsELFunctions.java 
801bfe6eb212b549888f0e9fb7df7b65bcf59393 
  core/src/main/java/org/apache/oozie/action/hadoop/HCatCredentialHelper.java 
36ad1df727e869b8051481ad969771e91bc6ae07 
  core/src/main/java/org/apache/oozie/action/hadoop/HadoopELFunctions.java 
ad2a71d641caa71efc098ad00bfbab3cec4d9506 
  core/src/main/java/org/apache/oozie/action/hadoop/HbaseCredentials.java 
307f565786ad77f5a35b5790bd5b052b04c6eac8 
  core/src/main/java/org/apache/oozie/action/hadoop/Hive2ActionExecutor.java 
2aed936af75da38c433be5d202bdc01c2f612c76 
  core/src/main/java/org/apache/oozie/action/hadoop/HiveActionExecutor.java 
8e2453ef4f78f4a5f1f95b1ca2793f5d7e07ad81 
  core/src/main/java/org/apache/oozie/action/hadoop/JavaActionExecutor.java 
4beeb965a461e246479c3e8ad88f92ecafca13c1 
  core/src/main/java/org/apache/oozie/action/hadoop/LauncherMapperHelper.java 
9609fdc2f3440fb176b659ef16c2a23451724cf9 
  
core/src/main/java/org/apache/oozie/action/hadoop/MapReduceActionExecutor.java 
55c937249ac35e14bd63dd0c1fa962ea636afa3a 
  core/src/main/java/org/apache/oozie/action/hadoop/OozieJobInfo.java 
581d3b3219eff882f961c8e495611e873c6a5a52 
  core/src/main/java/org/apache/oozie/action/hadoop/PigActionExecutor.java 
8b2dc16db80ad602911f1f64e60cfad41b198922 
  
core/src/main/java/org/apache/oozie/action/hadoop/ScriptLanguageActionExecutor.java
 f2541265ca71fb7f3b053c59775ef5e23d0d8f94 
  core/src/main/java/org/apache/oozie/action/hadoop/ShellActionExecutor.java 
b9ffa7ab7ec87db8a76a3653c74974d296c8cd17 
  core/src/main/java/org/apache/oozie/action/hadoop/SparkActionExecutor.java 
8e8d7d308b294efd91e3057e0d88ee06643ab2d4 
  core/src/main/java/org/apache/oozie/action/hadoop/SqoopActionExecutor.java 
6cee32aa8f5b8b36c9c7a98bb738f50b18b64624 
  core/src/main/java/org/apache/oozie/command/wf/SubmitHttpXCommand.java 
d2a2742ae418070a1cd0a889f14194224bc944a6 
  core/src/main/java/org/apache/oozie/command/wf/SubmitMRXCommand.java 
cc61d3de813c75c7f1cf4d405fc043253e3ca4ea 
  
core/src/main/java/org/apache/oozie/command/wf/SubmitScriptLanguageXCommand.java
 9d413059e99bc676feb63e02a2f704048eb4a355 
  core/src/main/java/org/apache/oozie/command/wf/SubmitSqoopXCommand.java 
51b739e21c802e09f4110b740138f4ff5c341fe4 
  core/src/main/java/org/apache/oozie/service/HadoopAccessorService.java 
5377127736cf189bcf4aa3d62bbbeab6b3a29d52 
  core/src/main/java/org/apache/oozie/service/Services.java 
829d5f5cd90c86e6cba9854dc0b4b7df570b8900 
  core/src/main/java/org/apache/oozie/service/ShareLibService.java 
b79bd37bb2b44348f220128334497d788dd922d4 
  core/src/main/java/org/apache/oozie/servlet/CallbackServlet.java 
612302173e0f485345558b0c96a8a75d2f43d238 
  core/src/main/java/org/apache/oozie/util/ClasspathUtils.java PRE-CREATION 
  core/src/main/java/org/apache/oozie/util/JobUtils.java 
a7a53b355d194b55959ede20c85d38e02294853d 
  core/src/main/resources/META-INF/persistence.xml 
edda2d5e22856b0250a53bfde7c73fd96f85c6e9 
  core/src/main/resources/oozie-default.xml 
2d7650acee73af713e2e3ea70975e74d1866cfd4 
  core/src/main/resources/oozie-log4j.properties 
c86b30143f5f58135493d9e6bdba29f402a0b1fd 
  core/src/test/java/org/apache/oozie/QueryServlet.java PRE-CREATION 
  core/src/test/java/org/apache/oozie/action/hadoop/BlockingMapper.java 
PRE-CREATION 
  core/src/test/java/org/apache/oozie/action/hadoop/LauncherMainTester.java 
4baed6e42beccd2fdb55ab1e6fb150c0d8821a9c 
  core/src/test/java/org/apache/oozie/action/hadoop/MapperReducerForTest.java 
8f08ddd9fa077fc9526814fd25e7477850c51737 
  
core/src/test/java/org/apache/oozie/action/hadoop/TestDistCpActionExecutor.java 
e8a140f330062c5ce8324112aae83b3d0ae1f772 
  co

Re: Review Request 54705: OOZIE-1770: Create Oozie Application Master for YARN

[Peter Cseh]

> On March 26, 2017, 12:37 a.m., Rohini Palaniswamy wrote:
> > core/src/main/java/org/apache/oozie/action/hadoop/SparkActionExecutor.java
> > Lines 112-114 (original)
> > 
> >
> > You can't remove this. Will breaking existing users. Since it starts 
> > with oozie.launcher, config deprecation will not handle it. Basically any 
> > oozie.launcher hadoop 1.x settings cannot be removed because of that.

This is setting "mapreduce.task.classpath.user.precedence" in the launcher job 
conf, it was parsed by the launcher Mapper and the LauncherAM is not doing 
anything with this property at the moment. Should we add similar functionality 
into LauncherAM? Is this a must have to support without MR launcher?


> On March 26, 2017, 12:37 a.m., Rohini Palaniswamy wrote:
> > core/src/main/java/org/apache/oozie/action/hadoop/SqoopActionExecutor.java
> > Line 89 (original), 87 (patched)
> > 
> >
> > Revert back to old code. Just remove the final instead of creating 
> > another temporary list.

merge issue. will check


> On March 26, 2017, 12:37 a.m., Rohini Palaniswamy wrote:
> > core/src/main/java/org/apache/oozie/action/hadoop/SqoopActionExecutor.java
> > Lines 109-111 (original)
> > 
> >
> > Why is this condition not necessary any more?

merge issue. will check


> On March 26, 2017, 12:37 a.m., Rohini Palaniswamy wrote:
> > core/src/main/java/org/apache/oozie/action/ssh/SshActionExecutor.java
> > Line 69 (original), 69 (patched)
> > 
> >
> > Revert. UNKNOWN_ERROR is correct

merge issue. will check


> On March 26, 2017, 12:37 a.m., Rohini Palaniswamy wrote:
> > core/src/main/java/org/apache/oozie/command/PurgeXCommand.java
> > Line 410 (original), 410 (patched)
> > 
> >
> > Why remove entity key?

merge issue. will check


> On March 26, 2017, 12:37 a.m., Rohini Palaniswamy wrote:
> > core/src/main/java/org/apache/oozie/command/PurgeXCommand.java
> > Line 418 (original), 418 (patched)
> > 
> >
> > Why is lock not required anymore?

merge issue. will check


> On March 26, 2017, 12:37 a.m., Rohini Palaniswamy wrote:
> > core/src/main/java/org/apache/oozie/dependency/HCatURIHandler.java
> > Line 182 (original)
> > 
> >
> > Why is support for dropping table being removed?

merge issue. will check


> On March 26, 2017, 12:37 a.m., Rohini Palaniswamy wrote:
> > core/src/main/java/org/apache/oozie/dependency/HCatURIHandler.java
> > Line 345 (original)
> > 
> >
> > Why is support for checking exists on a table being removed?

merge issue. will check


> On March 26, 2017, 12:37 a.m., Rohini Palaniswamy wrote:
> > core/src/main/java/org/apache/oozie/servlet/JVMInfo.java
> > Line 123 (original), 123 (patched)
> > 
> >
> > Revert. The original spelling of Daemon is correct.

merge issue. will check


> On March 26, 2017, 12:37 a.m., Rohini Palaniswamy wrote:
> > core/src/main/java/org/apache/oozie/sla/listener/SLAEmailEventListener.java
> > Line 386 (original), 386 (patched)
> > 
> >
> > Revert. threw exception is correct.

merge issue. will check


- Peter


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/54705/#review170061
---


On March 22, 2017, 8:34 p.m., Peter Cseh wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/54705/
> ---
> 
> (Updated March 22, 2017, 8:34 p.m.)
> 
> 
> Review request for oozie, Purshotam Shah, Robert Kanter, and Rohini 
> Palaniswamy.
> 
> 
> Bugs: OOZIE-1779
> https://issues.apache.org/jira/browse/OOZIE-1779
> 
> 
> Repository: oozie-git
> 
> 
> Description
> ---
> 
> Replacing the launcher MR job with a launcher AM
> 
> 
> Diffs
> -
> 
>   bin/test-patch-20-tests 22f472c07e80fdb38c632329a5707998ab352172 
>   client/pom.xml f99366c90887a5a45f3942291c90e1cdbfd84b3f 
>   client/src/main/java/org/apache/oozie/cli/OozieCLI.java 
> 38fb84e8a8918aee6ae57d19a29cdf181aba1955 
>   client/src/main/java/org/apache/oozie/cli/ValidationUtil.java 
> b1da2af070970add01031d6208a395520cd705

Re: Review Request 54705: OOZIE-1770: Create Oozie Application Master for YARN

[Peter Bacsko]

> On jan. 6, 2017, 7:19 de, Robert Kanter wrote:
> > core/src/main/java/org/apache/oozie/action/hadoop/SparkActionExecutor.java
> > Lines 156-158 (patched)
> > 
> >
> > We shouldn't need the MR jars in the Spark Action :)
> 
> Peter Cseh wrote:
> Oozie's SparkExample needs MR jars on the classpath.
> 
> Robert Kanter wrote:
> That's surprising, but okay.

I'm not sure which Spark code I ran, but it missed 
org.apache.hadoop.mapred.TextInputFormat


> On jan. 6, 2017, 7:19 de, Robert Kanter wrote:
> > core/src/test/java/org/apache/oozie/action/hadoop/TestJavaActionExecutor.java
> > Lines 1787-1821 (original), 1640-1674 (patched)
> > 
> >
> > What's the plan here?
> 
> Peter Bacsko wrote:
> Perhaps we don't need this test anymore.
> 
> I'm not entirely familiar with MapReduce ACLs but I think it's related to 
> the LauncherMapper. But we don't have a mapper anymore, but a standalone AM. 
> Do we still care about MR ACLs?
> 
> Robert Kanter wrote:
> I'm not either, but it sounds like this doesn't apply anymore so we 
> should delete it.

Checked with Geza, will be removed


- Peter


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/54705/#review160656
---


On márc. 22, 2017, 8:34 du, Peter Cseh wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/54705/
> ---
> 
> (Updated márc. 22, 2017, 8:34 du)
> 
> 
> Review request for oozie, Purshotam Shah, Robert Kanter, and Rohini 
> Palaniswamy.
> 
> 
> Bugs: OOZIE-1779
> https://issues.apache.org/jira/browse/OOZIE-1779
> 
> 
> Repository: oozie-git
> 
> 
> Description
> ---
> 
> Replacing the launcher MR job with a launcher AM
> 
> 
> Diffs
> -
> 
>   bin/test-patch-20-tests 22f472c07e80fdb38c632329a5707998ab352172 
>   client/pom.xml f99366c90887a5a45f3942291c90e1cdbfd84b3f 
>   client/src/main/java/org/apache/oozie/cli/OozieCLI.java 
> 38fb84e8a8918aee6ae57d19a29cdf181aba1955 
>   client/src/main/java/org/apache/oozie/cli/ValidationUtil.java 
> b1da2af070970add01031d6208a395520cd7055c 
>   client/src/main/java/org/apache/oozie/client/OozieClient.java 
> 73708080e624e775f7d058cd3377ad69d8399264 
>   client/src/main/java/org/apache/oozie/client/XOozieClient.java 
> 8538ec7c742cc013f7c18eaff3ecf61c2835cc8a 
>   client/src/main/java/org/apache/oozie/client/rest/JsonTags.java 
> ca168e084998b5dd564536078a7a69efd7291828 
>   client/src/main/java/org/apache/oozie/client/rest/RestConstants.java 
> 8ddb1f873f0c69aadf5f4a5161d8b21f7392fa1e 
>   core/pom.xml 1dd104581ccab7976d788b674d2200e48a70c27f 
>   core/src/main/java/org/apache/oozie/WorkflowJobBean.java 
> 55d79a5d5e5e4efab9eb832e9341a16bcf8959cc 
>   core/src/main/java/org/apache/oozie/action/ActionExecutor.java 
> 1d6456b236bf9e27c0f55d9f9e3b4bd94517c797 
>   core/src/main/java/org/apache/oozie/action/hadoop/Credentials.java 
> eadb47b07d157f45f086517ab41c4b9826f003f0 
>   core/src/main/java/org/apache/oozie/action/hadoop/CredentialsProvider.java 
> 6fe22fbd2fca5d873d58ece60f20d0f131371475 
>   
> core/src/main/java/org/apache/oozie/action/hadoop/CredentialsProviderFactory.java
>  PRE-CREATION 
>   core/src/main/java/org/apache/oozie/action/hadoop/DistcpActionExecutor.java 
> 78cd257a836707c1efeac13566d6285d42970545 
>   core/src/main/java/org/apache/oozie/action/hadoop/FsActionExecutor.java 
> 27654741101d2056b25626fbbe345183eafd612d 
>   core/src/main/java/org/apache/oozie/action/hadoop/FsELFunctions.java 
> 801bfe6eb212b549888f0e9fb7df7b65bcf59393 
>   core/src/main/java/org/apache/oozie/action/hadoop/HCatCredentialHelper.java 
> 36ad1df727e869b8051481ad969771e91bc6ae07 
>   core/src/main/java/org/apache/oozie/action/hadoop/HCatCredentials.java 
> d7689a924059e553abaeed374eed4072933292cb 
>   core/src/main/java/org/apache/oozie/action/hadoop/HadoopELFunctions.java 
> ad2a71d641caa71efc098ad00bfbab3cec4d9506 
>   core/src/main/java/org/apache/oozie/action/hadoop/HbaseCredentials.java 
> 307f565786ad77f5a35b5790bd5b052b04c6eac8 
>   core/src/main/java/org/apache/oozie/action/hadoop/Hive2ActionExecutor.java 
> 2aed936af75da38c433be5d202bdc01c2f612c76 
>   core/src/main/java/org/apache/oozie/action/hadoop/Hive2Credentials.java 
> bd2f1f768c5b04bef4a33613baede9202144c747 
>   core/src/main/java/org/apache/oozie/action/hadoop/HiveActionExecutor.java 
> 8e2453ef4f78f4a5f1f95b1ca2793f5d7e07ad81 
>   core/src/main/java/org/apache/oozie/action/hadoop/JavaActionExecutor.java 
> 39daca7b3050c259a36d54710b4c1279f3acdeac 
>   core/src/main/java/org/apache/oozie/action/hadoop/LauncherMapperHelper.java 
> 9609fdc2f3

Re: Review Request 54705: OOZIE-1770: Create Oozie Application Master for YARN

[Robert Kanter]

> On March 26, 2017, 12:37 a.m., Rohini Palaniswamy wrote:
> > core/src/main/java/org/apache/oozie/action/hadoop/SparkActionExecutor.java
> > Lines 112-114 (original)
> > 
> >
> > You can't remove this. Will breaking existing users. Since it starts 
> > with oozie.launcher, config deprecation will not handle it. Basically any 
> > oozie.launcher hadoop 1.x settings cannot be removed because of that.
> 
> Peter Cseh wrote:
> This is setting "mapreduce.task.classpath.user.precedence" in the 
> launcher job conf, it was parsed by the launcher Mapper and the LauncherAM is 
> not doing anything with this property at the moment. Should we add similar 
> functionality into LauncherAM? Is this a must have to support without MR 
> launcher?

I'd rather not have to support an equivalent of 
"mapreduce.task.classpath.user.precedence" in the Oozie AM itself.  It's a 
really ugly hacky thing.  If we want to go down this route, a better fix would 
be to provide complete classpath isolation somehow; but I think that could be a 
future feature.


- Robert


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/54705/#review170061
---


On March 22, 2017, 8:34 p.m., Peter Cseh wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/54705/
> ---
> 
> (Updated March 22, 2017, 8:34 p.m.)
> 
> 
> Review request for oozie, Purshotam Shah, Robert Kanter, and Rohini 
> Palaniswamy.
> 
> 
> Bugs: OOZIE-1779
> https://issues.apache.org/jira/browse/OOZIE-1779
> 
> 
> Repository: oozie-git
> 
> 
> Description
> ---
> 
> Replacing the launcher MR job with a launcher AM
> 
> 
> Diffs
> -
> 
>   bin/test-patch-20-tests 22f472c07e80fdb38c632329a5707998ab352172 
>   client/pom.xml f99366c90887a5a45f3942291c90e1cdbfd84b3f 
>   client/src/main/java/org/apache/oozie/cli/OozieCLI.java 
> 38fb84e8a8918aee6ae57d19a29cdf181aba1955 
>   client/src/main/java/org/apache/oozie/cli/ValidationUtil.java 
> b1da2af070970add01031d6208a395520cd7055c 
>   client/src/main/java/org/apache/oozie/client/OozieClient.java 
> 73708080e624e775f7d058cd3377ad69d8399264 
>   client/src/main/java/org/apache/oozie/client/XOozieClient.java 
> 8538ec7c742cc013f7c18eaff3ecf61c2835cc8a 
>   client/src/main/java/org/apache/oozie/client/rest/JsonTags.java 
> ca168e084998b5dd564536078a7a69efd7291828 
>   client/src/main/java/org/apache/oozie/client/rest/RestConstants.java 
> 8ddb1f873f0c69aadf5f4a5161d8b21f7392fa1e 
>   core/pom.xml 1dd104581ccab7976d788b674d2200e48a70c27f 
>   core/src/main/java/org/apache/oozie/WorkflowJobBean.java 
> 55d79a5d5e5e4efab9eb832e9341a16bcf8959cc 
>   core/src/main/java/org/apache/oozie/action/ActionExecutor.java 
> 1d6456b236bf9e27c0f55d9f9e3b4bd94517c797 
>   core/src/main/java/org/apache/oozie/action/hadoop/Credentials.java 
> eadb47b07d157f45f086517ab41c4b9826f003f0 
>   core/src/main/java/org/apache/oozie/action/hadoop/CredentialsProvider.java 
> 6fe22fbd2fca5d873d58ece60f20d0f131371475 
>   
> core/src/main/java/org/apache/oozie/action/hadoop/CredentialsProviderFactory.java
>  PRE-CREATION 
>   core/src/main/java/org/apache/oozie/action/hadoop/DistcpActionExecutor.java 
> 78cd257a836707c1efeac13566d6285d42970545 
>   core/src/main/java/org/apache/oozie/action/hadoop/FsActionExecutor.java 
> 27654741101d2056b25626fbbe345183eafd612d 
>   core/src/main/java/org/apache/oozie/action/hadoop/FsELFunctions.java 
> 801bfe6eb212b549888f0e9fb7df7b65bcf59393 
>   core/src/main/java/org/apache/oozie/action/hadoop/HCatCredentialHelper.java 
> 36ad1df727e869b8051481ad969771e91bc6ae07 
>   core/src/main/java/org/apache/oozie/action/hadoop/HCatCredentials.java 
> d7689a924059e553abaeed374eed4072933292cb 
>   core/src/main/java/org/apache/oozie/action/hadoop/HadoopELFunctions.java 
> ad2a71d641caa71efc098ad00bfbab3cec4d9506 
>   core/src/main/java/org/apache/oozie/action/hadoop/HbaseCredentials.java 
> 307f565786ad77f5a35b5790bd5b052b04c6eac8 
>   core/src/main/java/org/apache/oozie/action/hadoop/Hive2ActionExecutor.java 
> 2aed936af75da38c433be5d202bdc01c2f612c76 
>   core/src/main/java/org/apache/oozie/action/hadoop/Hive2Credentials.java 
> bd2f1f768c5b04bef4a33613baede9202144c747 
>   core/src/main/java/org/apache/oozie/action/hadoop/HiveActionExecutor.java 
> 8e2453ef4f78f4a5f1f95b1ca2793f5d7e07ad81 
>   core/src/main/java/org/apache/oozie/action/hadoop/JavaActionExecutor.java 
> 39daca7b3050c259a36d54710b4c1279f3acdeac 
>   core/src/main/java/org/apache/oozie/action/hadoop/LauncherMapperHelper.java 
> 9609fdc2f3440fb176b659ef16c2a23451724cf9 
>   
> core/src/main/java/org/apache/oozie/action/hadoop/MapReduceActionExecutor.java
>  55c937249ac35e14

Re: Review Request 54705: OOZIE-1770: Create Oozie Application Master for YARN

[Peter Cseh]

> On Jan. 6, 2017, 7:19 a.m., Robert Kanter wrote:
> > core/src/main/java/org/apache/oozie/action/hadoop/LauncherMapperHelper.java
> > Line 148 (original), 148 (patched)
> > 
> >
> > I don't think we need this anymore?  Probably not for a long time.

I've tried it out and it's not working anyways. It could be removed as part of 
OOZIE-2601


> On Jan. 6, 2017, 7:19 a.m., Robert Kanter wrote:
> > docs/src/site/twiki/DG_QuickStart.twiki
> > Lines 49-50 (original), 45-46 (patched)
> > 
> >
> > Are these still separate?  If so, there's no reason to split these from 
> > hadoop.version anymore now that Hadoop 1 is gone.

https://issues.apache.org/jira/browse/OOZIE-2884


> On Jan. 6, 2017, 7:19 a.m., Robert Kanter wrote:
> > docs/src/site/twiki/ENG_Building.twiki
> > Lines 220-221 (original), 216-217 (patched)
> > 
> >
> > Same comment as in DG_QuickStart

https://issues.apache.org/jira/browse/OOZIE-2884


- Peter


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/54705/#review160656
---


On March 22, 2017, 8:34 p.m., Peter Cseh wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/54705/
> ---
> 
> (Updated March 22, 2017, 8:34 p.m.)
> 
> 
> Review request for oozie, Purshotam Shah, Robert Kanter, and Rohini 
> Palaniswamy.
> 
> 
> Bugs: OOZIE-1779
> https://issues.apache.org/jira/browse/OOZIE-1779
> 
> 
> Repository: oozie-git
> 
> 
> Description
> ---
> 
> Replacing the launcher MR job with a launcher AM
> 
> 
> Diffs
> -
> 
>   bin/test-patch-20-tests 22f472c07e80fdb38c632329a5707998ab352172 
>   client/pom.xml f99366c90887a5a45f3942291c90e1cdbfd84b3f 
>   client/src/main/java/org/apache/oozie/cli/OozieCLI.java 
> 38fb84e8a8918aee6ae57d19a29cdf181aba1955 
>   client/src/main/java/org/apache/oozie/cli/ValidationUtil.java 
> b1da2af070970add01031d6208a395520cd7055c 
>   client/src/main/java/org/apache/oozie/client/OozieClient.java 
> 73708080e624e775f7d058cd3377ad69d8399264 
>   client/src/main/java/org/apache/oozie/client/XOozieClient.java 
> 8538ec7c742cc013f7c18eaff3ecf61c2835cc8a 
>   client/src/main/java/org/apache/oozie/client/rest/JsonTags.java 
> ca168e084998b5dd564536078a7a69efd7291828 
>   client/src/main/java/org/apache/oozie/client/rest/RestConstants.java 
> 8ddb1f873f0c69aadf5f4a5161d8b21f7392fa1e 
>   core/pom.xml 1dd104581ccab7976d788b674d2200e48a70c27f 
>   core/src/main/java/org/apache/oozie/WorkflowJobBean.java 
> 55d79a5d5e5e4efab9eb832e9341a16bcf8959cc 
>   core/src/main/java/org/apache/oozie/action/ActionExecutor.java 
> 1d6456b236bf9e27c0f55d9f9e3b4bd94517c797 
>   core/src/main/java/org/apache/oozie/action/hadoop/Credentials.java 
> eadb47b07d157f45f086517ab41c4b9826f003f0 
>   core/src/main/java/org/apache/oozie/action/hadoop/CredentialsProvider.java 
> 6fe22fbd2fca5d873d58ece60f20d0f131371475 
>   
> core/src/main/java/org/apache/oozie/action/hadoop/CredentialsProviderFactory.java
>  PRE-CREATION 
>   core/src/main/java/org/apache/oozie/action/hadoop/DistcpActionExecutor.java 
> 78cd257a836707c1efeac13566d6285d42970545 
>   core/src/main/java/org/apache/oozie/action/hadoop/FsActionExecutor.java 
> 27654741101d2056b25626fbbe345183eafd612d 
>   core/src/main/java/org/apache/oozie/action/hadoop/FsELFunctions.java 
> 801bfe6eb212b549888f0e9fb7df7b65bcf59393 
>   core/src/main/java/org/apache/oozie/action/hadoop/HCatCredentialHelper.java 
> 36ad1df727e869b8051481ad969771e91bc6ae07 
>   core/src/main/java/org/apache/oozie/action/hadoop/HCatCredentials.java 
> d7689a924059e553abaeed374eed4072933292cb 
>   core/src/main/java/org/apache/oozie/action/hadoop/HadoopELFunctions.java 
> ad2a71d641caa71efc098ad00bfbab3cec4d9506 
>   core/src/main/java/org/apache/oozie/action/hadoop/HbaseCredentials.java 
> 307f565786ad77f5a35b5790bd5b052b04c6eac8 
>   core/src/main/java/org/apache/oozie/action/hadoop/Hive2ActionExecutor.java 
> 2aed936af75da38c433be5d202bdc01c2f612c76 
>   core/src/main/java/org/apache/oozie/action/hadoop/Hive2Credentials.java 
> bd2f1f768c5b04bef4a33613baede9202144c747 
>   core/src/main/java/org/apache/oozie/action/hadoop/HiveActionExecutor.java 
> 8e2453ef4f78f4a5f1f95b1ca2793f5d7e07ad81 
>   core/src/main/java/org/apache/oozie/action/hadoop/JavaActionExecutor.java 
> 39daca7b3050c259a36d54710b4c1279f3acdeac 
>   core/src/main/java/org/apache/oozie/action/hadoop/LauncherMapperHelper.java 
> 9609fdc2f3440fb176b659ef16c2a23451724cf9 
>   
> core/src/main/java/org/apache/oozie/action/hadoop/MapReduceActionExe

Re: Review Request 54705: OOZIE-1770: Create Oozie Application Master for YARN

[Peter Cseh]

> On Jan. 6, 2017, 4:37 p.m., Peter Bacsko wrote:
> > core/src/main/java/org/apache/oozie/action/hadoop/JavaActionExecutor.java
> > Lines 104-110 (original), 129-135 (patched)
> > 
> >
> > Since we use our own App master, do we care about all these properties?
> > 
> > I checked MapReduceActionExecutor which calls super.setupActionConf() 
> > and super.setupLauncherConf(). But even if we use these props, it should be 
> > relevant only for the MR action.
> > 
> > Thoughts?
> 
> Robert Kanter wrote:
> We had discussed doing some backwards compatibility for some of these 
> things for the launcher.  It might make sense to delete these for now (or 
> leave them?), and OOZIE-2687 can deal with them later.

Will be removed after https://issues.apache.org/jira/browse/OOZIE-2687


- Peter


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/54705/#review160725
---


On March 22, 2017, 8:34 p.m., Peter Cseh wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/54705/
> ---
> 
> (Updated March 22, 2017, 8:34 p.m.)
> 
> 
> Review request for oozie, Purshotam Shah, Robert Kanter, and Rohini 
> Palaniswamy.
> 
> 
> Bugs: OOZIE-1779
> https://issues.apache.org/jira/browse/OOZIE-1779
> 
> 
> Repository: oozie-git
> 
> 
> Description
> ---
> 
> Replacing the launcher MR job with a launcher AM
> 
> 
> Diffs
> -
> 
>   bin/test-patch-20-tests 22f472c07e80fdb38c632329a5707998ab352172 
>   client/pom.xml f99366c90887a5a45f3942291c90e1cdbfd84b3f 
>   client/src/main/java/org/apache/oozie/cli/OozieCLI.java 
> 38fb84e8a8918aee6ae57d19a29cdf181aba1955 
>   client/src/main/java/org/apache/oozie/cli/ValidationUtil.java 
> b1da2af070970add01031d6208a395520cd7055c 
>   client/src/main/java/org/apache/oozie/client/OozieClient.java 
> 73708080e624e775f7d058cd3377ad69d8399264 
>   client/src/main/java/org/apache/oozie/client/XOozieClient.java 
> 8538ec7c742cc013f7c18eaff3ecf61c2835cc8a 
>   client/src/main/java/org/apache/oozie/client/rest/JsonTags.java 
> ca168e084998b5dd564536078a7a69efd7291828 
>   client/src/main/java/org/apache/oozie/client/rest/RestConstants.java 
> 8ddb1f873f0c69aadf5f4a5161d8b21f7392fa1e 
>   core/pom.xml 1dd104581ccab7976d788b674d2200e48a70c27f 
>   core/src/main/java/org/apache/oozie/WorkflowJobBean.java 
> 55d79a5d5e5e4efab9eb832e9341a16bcf8959cc 
>   core/src/main/java/org/apache/oozie/action/ActionExecutor.java 
> 1d6456b236bf9e27c0f55d9f9e3b4bd94517c797 
>   core/src/main/java/org/apache/oozie/action/hadoop/Credentials.java 
> eadb47b07d157f45f086517ab41c4b9826f003f0 
>   core/src/main/java/org/apache/oozie/action/hadoop/CredentialsProvider.java 
> 6fe22fbd2fca5d873d58ece60f20d0f131371475 
>   
> core/src/main/java/org/apache/oozie/action/hadoop/CredentialsProviderFactory.java
>  PRE-CREATION 
>   core/src/main/java/org/apache/oozie/action/hadoop/DistcpActionExecutor.java 
> 78cd257a836707c1efeac13566d6285d42970545 
>   core/src/main/java/org/apache/oozie/action/hadoop/FsActionExecutor.java 
> 27654741101d2056b25626fbbe345183eafd612d 
>   core/src/main/java/org/apache/oozie/action/hadoop/FsELFunctions.java 
> 801bfe6eb212b549888f0e9fb7df7b65bcf59393 
>   core/src/main/java/org/apache/oozie/action/hadoop/HCatCredentialHelper.java 
> 36ad1df727e869b8051481ad969771e91bc6ae07 
>   core/src/main/java/org/apache/oozie/action/hadoop/HCatCredentials.java 
> d7689a924059e553abaeed374eed4072933292cb 
>   core/src/main/java/org/apache/oozie/action/hadoop/HadoopELFunctions.java 
> ad2a71d641caa71efc098ad00bfbab3cec4d9506 
>   core/src/main/java/org/apache/oozie/action/hadoop/HbaseCredentials.java 
> 307f565786ad77f5a35b5790bd5b052b04c6eac8 
>   core/src/main/java/org/apache/oozie/action/hadoop/Hive2ActionExecutor.java 
> 2aed936af75da38c433be5d202bdc01c2f612c76 
>   core/src/main/java/org/apache/oozie/action/hadoop/Hive2Credentials.java 
> bd2f1f768c5b04bef4a33613baede9202144c747 
>   core/src/main/java/org/apache/oozie/action/hadoop/HiveActionExecutor.java 
> 8e2453ef4f78f4a5f1f95b1ca2793f5d7e07ad81 
>   core/src/main/java/org/apache/oozie/action/hadoop/JavaActionExecutor.java 
> 39daca7b3050c259a36d54710b4c1279f3acdeac 
>   core/src/main/java/org/apache/oozie/action/hadoop/LauncherMapperHelper.java 
> 9609fdc2f3440fb176b659ef16c2a23451724cf9 
>   
> core/src/main/java/org/apache/oozie/action/hadoop/MapReduceActionExecutor.java
>  55c937249ac35e14bd63dd0c1fa962ea636afa3a 
>   core/src/main/java/org/apache/oozie/action/hadoop/OozieJobInfo.java 
> 581d3b3219eff882f961c8e495611e873c6a5a52 
>   core/src/main/java/org/apache/oozie/action/hadoop/PigActionExecutor.java 
> 8b2dc16db80ad602911f1f64

Re: Review Request 54705: OOZIE-1770: Create Oozie Application Master for YARN

[Peter Cseh]

> On March 26, 2017, 12:37 a.m., Rohini Palaniswamy wrote:
> > core/src/main/java/org/apache/oozie/servlet/BaseAdminServlet.java
> > Lines 86-87 (original), 71 (patched)
> > 
> >
> > You should retain the code of checking the resource name and throwing 
> > error here which is currently commented out instead of doing it in 
> > setOozieMode()

Merge issue


- Peter


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/54705/#review170061
---


On March 22, 2017, 8:34 p.m., Peter Cseh wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/54705/
> ---
> 
> (Updated March 22, 2017, 8:34 p.m.)
> 
> 
> Review request for oozie, Purshotam Shah, Robert Kanter, and Rohini 
> Palaniswamy.
> 
> 
> Bugs: OOZIE-1779
> https://issues.apache.org/jira/browse/OOZIE-1779
> 
> 
> Repository: oozie-git
> 
> 
> Description
> ---
> 
> Replacing the launcher MR job with a launcher AM
> 
> 
> Diffs
> -
> 
>   bin/test-patch-20-tests 22f472c07e80fdb38c632329a5707998ab352172 
>   client/pom.xml f99366c90887a5a45f3942291c90e1cdbfd84b3f 
>   client/src/main/java/org/apache/oozie/cli/OozieCLI.java 
> 38fb84e8a8918aee6ae57d19a29cdf181aba1955 
>   client/src/main/java/org/apache/oozie/cli/ValidationUtil.java 
> b1da2af070970add01031d6208a395520cd7055c 
>   client/src/main/java/org/apache/oozie/client/OozieClient.java 
> 73708080e624e775f7d058cd3377ad69d8399264 
>   client/src/main/java/org/apache/oozie/client/XOozieClient.java 
> 8538ec7c742cc013f7c18eaff3ecf61c2835cc8a 
>   client/src/main/java/org/apache/oozie/client/rest/JsonTags.java 
> ca168e084998b5dd564536078a7a69efd7291828 
>   client/src/main/java/org/apache/oozie/client/rest/RestConstants.java 
> 8ddb1f873f0c69aadf5f4a5161d8b21f7392fa1e 
>   core/pom.xml 1dd104581ccab7976d788b674d2200e48a70c27f 
>   core/src/main/java/org/apache/oozie/WorkflowJobBean.java 
> 55d79a5d5e5e4efab9eb832e9341a16bcf8959cc 
>   core/src/main/java/org/apache/oozie/action/ActionExecutor.java 
> 1d6456b236bf9e27c0f55d9f9e3b4bd94517c797 
>   core/src/main/java/org/apache/oozie/action/hadoop/Credentials.java 
> eadb47b07d157f45f086517ab41c4b9826f003f0 
>   core/src/main/java/org/apache/oozie/action/hadoop/CredentialsProvider.java 
> 6fe22fbd2fca5d873d58ece60f20d0f131371475 
>   
> core/src/main/java/org/apache/oozie/action/hadoop/CredentialsProviderFactory.java
>  PRE-CREATION 
>   core/src/main/java/org/apache/oozie/action/hadoop/DistcpActionExecutor.java 
> 78cd257a836707c1efeac13566d6285d42970545 
>   core/src/main/java/org/apache/oozie/action/hadoop/FsActionExecutor.java 
> 27654741101d2056b25626fbbe345183eafd612d 
>   core/src/main/java/org/apache/oozie/action/hadoop/FsELFunctions.java 
> 801bfe6eb212b549888f0e9fb7df7b65bcf59393 
>   core/src/main/java/org/apache/oozie/action/hadoop/HCatCredentialHelper.java 
> 36ad1df727e869b8051481ad969771e91bc6ae07 
>   core/src/main/java/org/apache/oozie/action/hadoop/HCatCredentials.java 
> d7689a924059e553abaeed374eed4072933292cb 
>   core/src/main/java/org/apache/oozie/action/hadoop/HadoopELFunctions.java 
> ad2a71d641caa71efc098ad00bfbab3cec4d9506 
>   core/src/main/java/org/apache/oozie/action/hadoop/HbaseCredentials.java 
> 307f565786ad77f5a35b5790bd5b052b04c6eac8 
>   core/src/main/java/org/apache/oozie/action/hadoop/Hive2ActionExecutor.java 
> 2aed936af75da38c433be5d202bdc01c2f612c76 
>   core/src/main/java/org/apache/oozie/action/hadoop/Hive2Credentials.java 
> bd2f1f768c5b04bef4a33613baede9202144c747 
>   core/src/main/java/org/apache/oozie/action/hadoop/HiveActionExecutor.java 
> 8e2453ef4f78f4a5f1f95b1ca2793f5d7e07ad81 
>   core/src/main/java/org/apache/oozie/action/hadoop/JavaActionExecutor.java 
> 39daca7b3050c259a36d54710b4c1279f3acdeac 
>   core/src/main/java/org/apache/oozie/action/hadoop/LauncherMapperHelper.java 
> 9609fdc2f3440fb176b659ef16c2a23451724cf9 
>   
> core/src/main/java/org/apache/oozie/action/hadoop/MapReduceActionExecutor.java
>  55c937249ac35e14bd63dd0c1fa962ea636afa3a 
>   core/src/main/java/org/apache/oozie/action/hadoop/OozieJobInfo.java 
> 581d3b3219eff882f961c8e495611e873c6a5a52 
>   core/src/main/java/org/apache/oozie/action/hadoop/PigActionExecutor.java 
> 8b2dc16db80ad602911f1f64e60cfad41b198922 
>   
> core/src/main/java/org/apache/oozie/action/hadoop/ScriptLanguageActionExecutor.java
>  92e149d5cac6509893c8507b776263a38312a53b 
>   core/src/main/java/org/apache/oozie/action/hadoop/ShellActionExecutor.java 
> b9ffa7ab7ec87db8a76a3653c74974d296c8cd17 
>   core/src/main/java/org/apache/oozie/action/hadoop/SparkActionExecutor.java 
> 1a3197abb52f80b545dcc8634a8cac7a281a9eac 
>   core/src/main/java/org/apache/oozie/acti

Re: Review Request 54705: OOZIE-1770: Create Oozie Application Master for YARN

[Peter Cseh]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/54705/
---

(Updated May 9, 2017, 10:48 p.m.)


Review request for oozie, Purshotam Shah, Robert Kanter, and Rohini Palaniswamy.


Bugs: OOZIE-1779
https://issues.apache.org/jira/browse/OOZIE-1779


Repository: oozie-git


Description
---

Replacing the launcher MR job with a launcher AM


Diffs (updated)
-

  client/pom.xml f99366c90887a5a45f3942291c90e1cdbfd84b3f 
  client/src/main/java/org/apache/oozie/cli/OozieCLI.java 
38fb84e8a8918aee6ae57d19a29cdf181aba1955 
  client/src/main/java/org/apache/oozie/client/XOozieClient.java 
8538ec7c742cc013f7c18eaff3ecf61c2835cc8a 
  core/pom.xml 14aa03483e107db326628ba627cc3ee9f5fd3f3d 
  core/src/main/java/org/apache/oozie/CoordinatorEngine.java.orig 
2c04beaa5be1d9deca8effbd138f50c84d8b85a6 
  core/src/main/java/org/apache/oozie/WorkflowJobBean.java 
55d79a5d5e5e4efab9eb832e9341a16bcf8959cc 
  core/src/main/java/org/apache/oozie/action/ActionExecutor.java 
1d6456b236bf9e27c0f55d9f9e3b4bd94517c797 
  core/src/main/java/org/apache/oozie/action/hadoop/Credentials.java 
eadb47b07d157f45f086517ab41c4b9826f003f0 
  core/src/main/java/org/apache/oozie/action/hadoop/CredentialsProvider.java 
6fe22fbd2fca5d873d58ece60f20d0f131371475 
  
core/src/main/java/org/apache/oozie/action/hadoop/CredentialsProviderFactory.java
 PRE-CREATION 
  core/src/main/java/org/apache/oozie/action/hadoop/DistcpActionExecutor.java 
78cd257a836707c1efeac13566d6285d42970545 
  core/src/main/java/org/apache/oozie/action/hadoop/FsActionExecutor.java 
27654741101d2056b25626fbbe345183eafd612d 
  core/src/main/java/org/apache/oozie/action/hadoop/FsELFunctions.java 
801bfe6eb212b549888f0e9fb7df7b65bcf59393 
  core/src/main/java/org/apache/oozie/action/hadoop/HCatCredentialHelper.java 
36ad1df727e869b8051481ad969771e91bc6ae07 
  core/src/main/java/org/apache/oozie/action/hadoop/HCatCredentials.java 
d7689a924059e553abaeed374eed4072933292cb 
  core/src/main/java/org/apache/oozie/action/hadoop/HadoopELFunctions.java 
ad2a71d641caa71efc098ad00bfbab3cec4d9506 
  core/src/main/java/org/apache/oozie/action/hadoop/HbaseCredentials.java 
307f565786ad77f5a35b5790bd5b052b04c6eac8 
  core/src/main/java/org/apache/oozie/action/hadoop/Hive2ActionExecutor.java 
2aed936af75da38c433be5d202bdc01c2f612c76 
  core/src/main/java/org/apache/oozie/action/hadoop/Hive2Credentials.java 
bd2f1f768c5b04bef4a33613baede9202144c747 
  core/src/main/java/org/apache/oozie/action/hadoop/HiveActionExecutor.java 
8e2453ef4f78f4a5f1f95b1ca2793f5d7e07ad81 
  core/src/main/java/org/apache/oozie/action/hadoop/JavaActionExecutor.java 
f62c99700f715d7afaa35b5657a837f9dc884103 
  core/src/main/java/org/apache/oozie/action/hadoop/LauncherMapperHelper.java 
9609fdc2f3440fb176b659ef16c2a23451724cf9 
  
core/src/main/java/org/apache/oozie/action/hadoop/MapReduceActionExecutor.java 
55c937249ac35e14bd63dd0c1fa962ea636afa3a 
  core/src/main/java/org/apache/oozie/action/hadoop/OozieJobInfo.java 
581d3b3219eff882f961c8e495611e873c6a5a52 
  core/src/main/java/org/apache/oozie/action/hadoop/PigActionExecutor.java 
8b2dc16db80ad602911f1f64e60cfad41b198922 
  
core/src/main/java/org/apache/oozie/action/hadoop/ScriptLanguageActionExecutor.java
 92e149d5cac6509893c8507b776263a38312a53b 
  core/src/main/java/org/apache/oozie/action/hadoop/ShellActionExecutor.java 
b9ffa7ab7ec87db8a76a3653c74974d296c8cd17 
  core/src/main/java/org/apache/oozie/action/hadoop/SparkActionExecutor.java 
1a3197abb52f80b545dcc8634a8cac7a281a9eac 
  core/src/main/java/org/apache/oozie/action/hadoop/SqoopActionExecutor.java 
22e28746a9a39ae6c1e22326cc4f8f759d820b20 
  core/src/main/java/org/apache/oozie/client/rest/BulkResponseImpl.java 
fb021bd022d199f93fc5674843514abf45ae12a5 
  core/src/main/java/org/apache/oozie/command/XCommand.java 
bdf13f65ea25e8e0f5125e2798568a2c03fffe81 
  core/src/main/java/org/apache/oozie/command/coord/CoordSubmitXCommand.java 
969336d27ce5cd6535c926b0883ff5b7f610ce72 
  core/src/main/java/org/apache/oozie/command/wf/SubmitHttpXCommand.java 
d2a2742ae418070a1cd0a889f14194224bc944a6 
  core/src/main/java/org/apache/oozie/command/wf/SubmitMRXCommand.java 
cc61d3de813c75c7f1cf4d405fc043253e3ca4ea 
  
core/src/main/java/org/apache/oozie/command/wf/SubmitScriptLanguageXCommand.java
 9d413059e99bc676feb63e02a2f704048eb4a355 
  core/src/main/java/org/apache/oozie/command/wf/SubmitSqoopXCommand.java 
51b739e21c802e09f4110b740138f4ff5c341fe4 
  
core/src/main/java/org/apache/oozie/coord/input/dependency/AbstractCoordInputDependency.java
 ace120d10e3546201aa4145df56abf7933fb24b8 
  
core/src/main/java/org/apache/oozie/dependency/hcat/EhcacheHCatDependencyCache.java
 6f0abf6e094ed04158652191f2b8bb3b0cd32e90 
  core/src/main/java/org/apache/oozie/service/CallableQueueService.java 
a86a8d0889138cd1564ca9253e5b319ea52e55cc 
  core/src/main/java/org/apache/o

Re: Review Request 54705: OOZIE-1770: Create Oozie Application Master for YARN

[Peter Cseh]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/54705/
---

(Updated May 10, 2017, 9:13 p.m.)


Review request for oozie, Purshotam Shah, Robert Kanter, and Rohini Palaniswamy.


Bugs: OOZIE-1779
https://issues.apache.org/jira/browse/OOZIE-1779


Repository: oozie-git


Description
---

Replacing the launcher MR job with a launcher AM


Diffs (updated)
-

  client/pom.xml f99366c90887a5a45f3942291c90e1cdbfd84b3f 
  client/src/main/java/org/apache/oozie/cli/OozieCLI.java 
38fb84e8a8918aee6ae57d19a29cdf181aba1955 
  client/src/main/java/org/apache/oozie/client/XOozieClient.java 
8538ec7c742cc013f7c18eaff3ecf61c2835cc8a 
  core/pom.xml 14aa03483e107db326628ba627cc3ee9f5fd3f3d 
  core/src/main/java/org/apache/oozie/WorkflowJobBean.java 
55d79a5d5e5e4efab9eb832e9341a16bcf8959cc 
  core/src/main/java/org/apache/oozie/action/ActionExecutor.java 
1d6456b236bf9e27c0f55d9f9e3b4bd94517c797 
  core/src/main/java/org/apache/oozie/action/hadoop/Credentials.java 
eadb47b07d157f45f086517ab41c4b9826f003f0 
  core/src/main/java/org/apache/oozie/action/hadoop/CredentialsProvider.java 
6fe22fbd2fca5d873d58ece60f20d0f131371475 
  
core/src/main/java/org/apache/oozie/action/hadoop/CredentialsProviderFactory.java
 PRE-CREATION 
  core/src/main/java/org/apache/oozie/action/hadoop/DistcpActionExecutor.java 
78cd257a836707c1efeac13566d6285d42970545 
  core/src/main/java/org/apache/oozie/action/hadoop/FsActionExecutor.java 
27654741101d2056b25626fbbe345183eafd612d 
  core/src/main/java/org/apache/oozie/action/hadoop/FsELFunctions.java 
801bfe6eb212b549888f0e9fb7df7b65bcf59393 
  core/src/main/java/org/apache/oozie/action/hadoop/HCatCredentialHelper.java 
36ad1df727e869b8051481ad969771e91bc6ae07 
  core/src/main/java/org/apache/oozie/action/hadoop/HCatCredentials.java 
d7689a924059e553abaeed374eed4072933292cb 
  core/src/main/java/org/apache/oozie/action/hadoop/HadoopELFunctions.java 
ad2a71d641caa71efc098ad00bfbab3cec4d9506 
  core/src/main/java/org/apache/oozie/action/hadoop/HbaseCredentials.java 
307f565786ad77f5a35b5790bd5b052b04c6eac8 
  core/src/main/java/org/apache/oozie/action/hadoop/Hive2ActionExecutor.java 
2aed936af75da38c433be5d202bdc01c2f612c76 
  core/src/main/java/org/apache/oozie/action/hadoop/Hive2Credentials.java 
bd2f1f768c5b04bef4a33613baede9202144c747 
  core/src/main/java/org/apache/oozie/action/hadoop/HiveActionExecutor.java 
8e2453ef4f78f4a5f1f95b1ca2793f5d7e07ad81 
  core/src/main/java/org/apache/oozie/action/hadoop/JavaActionExecutor.java 
d60a5c7b996a4085f9bc77d82f12ad6925280b1a 
  core/src/main/java/org/apache/oozie/action/hadoop/LauncherMapperHelper.java 
72ed2f150be73f0a5e8b71b0fb6cc3594dc23078 
  
core/src/main/java/org/apache/oozie/action/hadoop/MapReduceActionExecutor.java 
55c937249ac35e14bd63dd0c1fa962ea636afa3a 
  core/src/main/java/org/apache/oozie/action/hadoop/OozieJobInfo.java 
581d3b3219eff882f961c8e495611e873c6a5a52 
  core/src/main/java/org/apache/oozie/action/hadoop/PigActionExecutor.java 
8b2dc16db80ad602911f1f64e60cfad41b198922 
  
core/src/main/java/org/apache/oozie/action/hadoop/ScriptLanguageActionExecutor.java
 92e149d5cac6509893c8507b776263a38312a53b 
  core/src/main/java/org/apache/oozie/action/hadoop/ShellActionExecutor.java 
b9ffa7ab7ec87db8a76a3653c74974d296c8cd17 
  core/src/main/java/org/apache/oozie/action/hadoop/SparkActionExecutor.java 
1a3197abb52f80b545dcc8634a8cac7a281a9eac 
  core/src/main/java/org/apache/oozie/action/hadoop/SqoopActionExecutor.java 
22e28746a9a39ae6c1e22326cc4f8f759d820b20 
  core/src/main/java/org/apache/oozie/client/rest/BulkResponseImpl.java 
fb021bd022d199f93fc5674843514abf45ae12a5 
  core/src/main/java/org/apache/oozie/command/XCommand.java 
bdf13f65ea25e8e0f5125e2798568a2c03fffe81 
  core/src/main/java/org/apache/oozie/command/wf/SubmitHttpXCommand.java 
d2a2742ae418070a1cd0a889f14194224bc944a6 
  core/src/main/java/org/apache/oozie/command/wf/SubmitMRXCommand.java 
cc61d3de813c75c7f1cf4d405fc043253e3ca4ea 
  
core/src/main/java/org/apache/oozie/command/wf/SubmitScriptLanguageXCommand.java
 9d413059e99bc676feb63e02a2f704048eb4a355 
  core/src/main/java/org/apache/oozie/command/wf/SubmitSqoopXCommand.java 
51b739e21c802e09f4110b740138f4ff5c341fe4 
  
core/src/main/java/org/apache/oozie/coord/input/dependency/AbstractCoordInputDependency.java
 ace120d10e3546201aa4145df56abf7933fb24b8 
  
core/src/main/java/org/apache/oozie/dependency/hcat/EhcacheHCatDependencyCache.java
 6f0abf6e094ed04158652191f2b8bb3b0cd32e90 
  core/src/main/java/org/apache/oozie/service/CallableQueueService.java 
a86a8d0889138cd1564ca9253e5b319ea52e55cc 
  core/src/main/java/org/apache/oozie/service/EventHandlerService.java 
22c6fb0965f45c5cf2322c251d4a8aa77d263458 
  core/src/main/java/org/apache/oozie/service/HadoopAccessorService.java 
23a9d92bf56a625469ec247cfb9727972511b101 
  core/src/main/java/org/apache/

Re: Review Request 54705: OOZIE-1770: Create Oozie Application Master for YARN

[Peter Cseh]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/54705/
---

(Updated May 10, 2017, 9:24 p.m.)


Review request for oozie, Purshotam Shah, Robert Kanter, and Rohini Palaniswamy.


Bugs: OOZIE-1770
https://issues.apache.org/jira/browse/OOZIE-1770


Repository: oozie-git


Description
---

Replacing the launcher MR job with a launcher AM


Diffs
-

  client/pom.xml f99366c90887a5a45f3942291c90e1cdbfd84b3f 
  client/src/main/java/org/apache/oozie/cli/OozieCLI.java 
38fb84e8a8918aee6ae57d19a29cdf181aba1955 
  client/src/main/java/org/apache/oozie/client/XOozieClient.java 
8538ec7c742cc013f7c18eaff3ecf61c2835cc8a 
  core/pom.xml 14aa03483e107db326628ba627cc3ee9f5fd3f3d 
  core/src/main/java/org/apache/oozie/WorkflowJobBean.java 
55d79a5d5e5e4efab9eb832e9341a16bcf8959cc 
  core/src/main/java/org/apache/oozie/action/ActionExecutor.java 
1d6456b236bf9e27c0f55d9f9e3b4bd94517c797 
  core/src/main/java/org/apache/oozie/action/hadoop/Credentials.java 
eadb47b07d157f45f086517ab41c4b9826f003f0 
  core/src/main/java/org/apache/oozie/action/hadoop/CredentialsProvider.java 
6fe22fbd2fca5d873d58ece60f20d0f131371475 
  
core/src/main/java/org/apache/oozie/action/hadoop/CredentialsProviderFactory.java
 PRE-CREATION 
  core/src/main/java/org/apache/oozie/action/hadoop/DistcpActionExecutor.java 
78cd257a836707c1efeac13566d6285d42970545 
  core/src/main/java/org/apache/oozie/action/hadoop/FsActionExecutor.java 
27654741101d2056b25626fbbe345183eafd612d 
  core/src/main/java/org/apache/oozie/action/hadoop/FsELFunctions.java 
801bfe6eb212b549888f0e9fb7df7b65bcf59393 
  core/src/main/java/org/apache/oozie/action/hadoop/HCatCredentialHelper.java 
36ad1df727e869b8051481ad969771e91bc6ae07 
  core/src/main/java/org/apache/oozie/action/hadoop/HCatCredentials.java 
d7689a924059e553abaeed374eed4072933292cb 
  core/src/main/java/org/apache/oozie/action/hadoop/HadoopELFunctions.java 
ad2a71d641caa71efc098ad00bfbab3cec4d9506 
  core/src/main/java/org/apache/oozie/action/hadoop/HbaseCredentials.java 
307f565786ad77f5a35b5790bd5b052b04c6eac8 
  core/src/main/java/org/apache/oozie/action/hadoop/Hive2ActionExecutor.java 
2aed936af75da38c433be5d202bdc01c2f612c76 
  core/src/main/java/org/apache/oozie/action/hadoop/Hive2Credentials.java 
bd2f1f768c5b04bef4a33613baede9202144c747 
  core/src/main/java/org/apache/oozie/action/hadoop/HiveActionExecutor.java 
8e2453ef4f78f4a5f1f95b1ca2793f5d7e07ad81 
  core/src/main/java/org/apache/oozie/action/hadoop/JavaActionExecutor.java 
d60a5c7b996a4085f9bc77d82f12ad6925280b1a 
  core/src/main/java/org/apache/oozie/action/hadoop/LauncherMapperHelper.java 
72ed2f150be73f0a5e8b71b0fb6cc3594dc23078 
  
core/src/main/java/org/apache/oozie/action/hadoop/MapReduceActionExecutor.java 
55c937249ac35e14bd63dd0c1fa962ea636afa3a 
  core/src/main/java/org/apache/oozie/action/hadoop/OozieJobInfo.java 
581d3b3219eff882f961c8e495611e873c6a5a52 
  core/src/main/java/org/apache/oozie/action/hadoop/PigActionExecutor.java 
8b2dc16db80ad602911f1f64e60cfad41b198922 
  
core/src/main/java/org/apache/oozie/action/hadoop/ScriptLanguageActionExecutor.java
 92e149d5cac6509893c8507b776263a38312a53b 
  core/src/main/java/org/apache/oozie/action/hadoop/ShellActionExecutor.java 
b9ffa7ab7ec87db8a76a3653c74974d296c8cd17 
  core/src/main/java/org/apache/oozie/action/hadoop/SparkActionExecutor.java 
1a3197abb52f80b545dcc8634a8cac7a281a9eac 
  core/src/main/java/org/apache/oozie/action/hadoop/SqoopActionExecutor.java 
22e28746a9a39ae6c1e22326cc4f8f759d820b20 
  core/src/main/java/org/apache/oozie/client/rest/BulkResponseImpl.java 
fb021bd022d199f93fc5674843514abf45ae12a5 
  core/src/main/java/org/apache/oozie/command/XCommand.java 
bdf13f65ea25e8e0f5125e2798568a2c03fffe81 
  core/src/main/java/org/apache/oozie/command/wf/SubmitHttpXCommand.java 
d2a2742ae418070a1cd0a889f14194224bc944a6 
  core/src/main/java/org/apache/oozie/command/wf/SubmitMRXCommand.java 
cc61d3de813c75c7f1cf4d405fc043253e3ca4ea 
  
core/src/main/java/org/apache/oozie/command/wf/SubmitScriptLanguageXCommand.java
 9d413059e99bc676feb63e02a2f704048eb4a355 
  core/src/main/java/org/apache/oozie/command/wf/SubmitSqoopXCommand.java 
51b739e21c802e09f4110b740138f4ff5c341fe4 
  
core/src/main/java/org/apache/oozie/coord/input/dependency/AbstractCoordInputDependency.java
 ace120d10e3546201aa4145df56abf7933fb24b8 
  
core/src/main/java/org/apache/oozie/dependency/hcat/EhcacheHCatDependencyCache.java
 6f0abf6e094ed04158652191f2b8bb3b0cd32e90 
  core/src/main/java/org/apache/oozie/service/CallableQueueService.java 
a86a8d0889138cd1564ca9253e5b319ea52e55cc 
  core/src/main/java/org/apache/oozie/service/EventHandlerService.java 
22c6fb0965f45c5cf2322c251d4a8aa77d263458 
  core/src/main/java/org/apache/oozie/service/HadoopAccessorService.java 
23a9d92bf56a625469ec247cfb9727972511b101 
  core/src/main/java/org/apache/oozie/serv

Re: Review Request 54705: OOZIE-1770: Create Oozie Application Master for YARN

[Peter Cseh]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/54705/
---

(Updated May 10, 2017, 10:46 p.m.)


Review request for oozie, Purshotam Shah, Robert Kanter, and Rohini Palaniswamy.


Bugs: OOZIE-1770
https://issues.apache.org/jira/browse/OOZIE-1770


Repository: oozie-git


Description
---

Replacing the launcher MR job with a launcher AM


Diffs (updated)
-

  client/pom.xml f99366c90887a5a45f3942291c90e1cdbfd84b3f 
  client/src/main/java/org/apache/oozie/cli/OozieCLI.java 
38fb84e8a8918aee6ae57d19a29cdf181aba1955 
  client/src/main/java/org/apache/oozie/client/XOozieClient.java 
8538ec7c742cc013f7c18eaff3ecf61c2835cc8a 
  core/pom.xml 14aa03483e107db326628ba627cc3ee9f5fd3f3d 
  core/src/main/java/org/apache/oozie/WorkflowJobBean.java 
55d79a5d5e5e4efab9eb832e9341a16bcf8959cc 
  core/src/main/java/org/apache/oozie/action/ActionExecutor.java 
1d6456b236bf9e27c0f55d9f9e3b4bd94517c797 
  core/src/main/java/org/apache/oozie/action/hadoop/Credentials.java 
eadb47b07d157f45f086517ab41c4b9826f003f0 
  core/src/main/java/org/apache/oozie/action/hadoop/CredentialsProvider.java 
6fe22fbd2fca5d873d58ece60f20d0f131371475 
  
core/src/main/java/org/apache/oozie/action/hadoop/CredentialsProviderFactory.java
 PRE-CREATION 
  core/src/main/java/org/apache/oozie/action/hadoop/DistcpActionExecutor.java 
78cd257a836707c1efeac13566d6285d42970545 
  core/src/main/java/org/apache/oozie/action/hadoop/FsActionExecutor.java 
27654741101d2056b25626fbbe345183eafd612d 
  core/src/main/java/org/apache/oozie/action/hadoop/FsELFunctions.java 
801bfe6eb212b549888f0e9fb7df7b65bcf59393 
  core/src/main/java/org/apache/oozie/action/hadoop/HCatCredentialHelper.java 
36ad1df727e869b8051481ad969771e91bc6ae07 
  core/src/main/java/org/apache/oozie/action/hadoop/HCatCredentials.java 
d7689a924059e553abaeed374eed4072933292cb 
  core/src/main/java/org/apache/oozie/action/hadoop/HadoopELFunctions.java 
ad2a71d641caa71efc098ad00bfbab3cec4d9506 
  core/src/main/java/org/apache/oozie/action/hadoop/HbaseCredentials.java 
307f565786ad77f5a35b5790bd5b052b04c6eac8 
  core/src/main/java/org/apache/oozie/action/hadoop/Hive2ActionExecutor.java 
2aed936af75da38c433be5d202bdc01c2f612c76 
  core/src/main/java/org/apache/oozie/action/hadoop/Hive2Credentials.java 
bd2f1f768c5b04bef4a33613baede9202144c747 
  core/src/main/java/org/apache/oozie/action/hadoop/HiveActionExecutor.java 
8e2453ef4f78f4a5f1f95b1ca2793f5d7e07ad81 
  core/src/main/java/org/apache/oozie/action/hadoop/JavaActionExecutor.java 
d60a5c7b996a4085f9bc77d82f12ad6925280b1a 
  core/src/main/java/org/apache/oozie/action/hadoop/LauncherMapperHelper.java 
72ed2f150be73f0a5e8b71b0fb6cc3594dc23078 
  
core/src/main/java/org/apache/oozie/action/hadoop/MapReduceActionExecutor.java 
55c937249ac35e14bd63dd0c1fa962ea636afa3a 
  core/src/main/java/org/apache/oozie/action/hadoop/OozieJobInfo.java 
581d3b3219eff882f961c8e495611e873c6a5a52 
  core/src/main/java/org/apache/oozie/action/hadoop/PigActionExecutor.java 
8b2dc16db80ad602911f1f64e60cfad41b198922 
  
core/src/main/java/org/apache/oozie/action/hadoop/ScriptLanguageActionExecutor.java
 92e149d5cac6509893c8507b776263a38312a53b 
  core/src/main/java/org/apache/oozie/action/hadoop/ShellActionExecutor.java 
b9ffa7ab7ec87db8a76a3653c74974d296c8cd17 
  core/src/main/java/org/apache/oozie/action/hadoop/SparkActionExecutor.java 
1a3197abb52f80b545dcc8634a8cac7a281a9eac 
  core/src/main/java/org/apache/oozie/action/hadoop/SqoopActionExecutor.java 
22e28746a9a39ae6c1e22326cc4f8f759d820b20 
  core/src/main/java/org/apache/oozie/client/rest/BulkResponseImpl.java 
fb021bd022d199f93fc5674843514abf45ae12a5 
  core/src/main/java/org/apache/oozie/command/XCommand.java 
bdf13f65ea25e8e0f5125e2798568a2c03fffe81 
  core/src/main/java/org/apache/oozie/command/wf/SubmitHttpXCommand.java 
d2a2742ae418070a1cd0a889f14194224bc944a6 
  core/src/main/java/org/apache/oozie/command/wf/SubmitMRXCommand.java 
cc61d3de813c75c7f1cf4d405fc043253e3ca4ea 
  
core/src/main/java/org/apache/oozie/command/wf/SubmitScriptLanguageXCommand.java
 9d413059e99bc676feb63e02a2f704048eb4a355 
  core/src/main/java/org/apache/oozie/command/wf/SubmitSqoopXCommand.java 
51b739e21c802e09f4110b740138f4ff5c341fe4 
  
core/src/main/java/org/apache/oozie/coord/input/dependency/AbstractCoordInputDependency.java
 ace120d10e3546201aa4145df56abf7933fb24b8 
  
core/src/main/java/org/apache/oozie/dependency/hcat/EhcacheHCatDependencyCache.java
 6f0abf6e094ed04158652191f2b8bb3b0cd32e90 
  core/src/main/java/org/apache/oozie/service/CallableQueueService.java 
a86a8d0889138cd1564ca9253e5b319ea52e55cc 
  core/src/main/java/org/apache/oozie/service/EventHandlerService.java 
22c6fb0965f45c5cf2322c251d4a8aa77d263458 
  core/src/main/java/org/apache/oozie/service/HadoopAccessorService.java 
23a9d92bf56a625469ec247cfb9727972511b101 
  core/src/main/java/org/apache

Re: Review Request 54705: OOZIE-1770: Create Oozie Application Master for YARN

[Robert Kanter]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/54705/#review174588
---




core/src/main/java/org/apache/oozie/action/hadoop/CredentialsProvider.java
Lines 29 (patched)


"... should modify the configuration which..."



core/src/main/java/org/apache/oozie/action/hadoop/CredentialsProviderFactory.java
Lines 108 (patched)


I imagine there's one of these in some Commons library or something?  At 
the very least, this should go into a Utils class.



core/src/main/java/org/apache/oozie/action/hadoop/DistcpActionExecutor.java
Line 47 (original)


I think this is the only place where getClassNamesByType is called, in 
which case, we can also delete that.

While I can see that this code isn't currently doing anything, so it should 
be safe to remove, it would be good to figure out what it was supposed to be 
doing originally.



core/src/main/java/org/apache/oozie/action/hadoop/FsELFunctions.java
Lines 189 (patched)


Do we need this?



core/src/main/java/org/apache/oozie/action/hadoop/HCatCredentialHelper.java
Line 49 (original), 49 (patched)


Not "job conf" anymore.



core/src/main/java/org/apache/oozie/action/hadoop/HCatCredentialHelper.java
Line 92 (original), 92 (patched)


We should use the {0} syntax to avoid String concatination.



core/src/main/java/org/apache/oozie/action/hadoop/HCatCredentialHelper.java
Line 102 (original), 102 (patched)


Same here.  Also check for others, especially on busy code paths.



core/src/main/java/org/apache/oozie/action/hadoop/HbaseCredentials.java
Lines 105-106 (patched)


Unnecessary new lines


- Robert Kanter


On May 10, 2017, 10:46 p.m., Peter Cseh wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/54705/
> ---
> 
> (Updated May 10, 2017, 10:46 p.m.)
> 
> 
> Review request for oozie, Purshotam Shah, Robert Kanter, and Rohini 
> Palaniswamy.
> 
> 
> Bugs: OOZIE-1770
> https://issues.apache.org/jira/browse/OOZIE-1770
> 
> 
> Repository: oozie-git
> 
> 
> Description
> ---
> 
> Replacing the launcher MR job with a launcher AM
> 
> 
> Diffs
> -
> 
>   client/pom.xml f99366c90887a5a45f3942291c90e1cdbfd84b3f 
>   client/src/main/java/org/apache/oozie/cli/OozieCLI.java 
> 38fb84e8a8918aee6ae57d19a29cdf181aba1955 
>   client/src/main/java/org/apache/oozie/client/XOozieClient.java 
> 8538ec7c742cc013f7c18eaff3ecf61c2835cc8a 
>   core/pom.xml 14aa03483e107db326628ba627cc3ee9f5fd3f3d 
>   core/src/main/java/org/apache/oozie/WorkflowJobBean.java 
> 55d79a5d5e5e4efab9eb832e9341a16bcf8959cc 
>   core/src/main/java/org/apache/oozie/action/ActionExecutor.java 
> 1d6456b236bf9e27c0f55d9f9e3b4bd94517c797 
>   core/src/main/java/org/apache/oozie/action/hadoop/Credentials.java 
> eadb47b07d157f45f086517ab41c4b9826f003f0 
>   core/src/main/java/org/apache/oozie/action/hadoop/CredentialsProvider.java 
> 6fe22fbd2fca5d873d58ece60f20d0f131371475 
>   
> core/src/main/java/org/apache/oozie/action/hadoop/CredentialsProviderFactory.java
>  PRE-CREATION 
>   core/src/main/java/org/apache/oozie/action/hadoop/DistcpActionExecutor.java 
> 78cd257a836707c1efeac13566d6285d42970545 
>   core/src/main/java/org/apache/oozie/action/hadoop/FsActionExecutor.java 
> 27654741101d2056b25626fbbe345183eafd612d 
>   core/src/main/java/org/apache/oozie/action/hadoop/FsELFunctions.java 
> 801bfe6eb212b549888f0e9fb7df7b65bcf59393 
>   core/src/main/java/org/apache/oozie/action/hadoop/HCatCredentialHelper.java 
> 36ad1df727e869b8051481ad969771e91bc6ae07 
>   core/src/main/java/org/apache/oozie/action/hadoop/HCatCredentials.java 
> d7689a924059e553abaeed374eed4072933292cb 
>   core/src/main/java/org/apache/oozie/action/hadoop/HadoopELFunctions.java 
> ad2a71d641caa71efc098ad00bfbab3cec4d9506 
>   core/src/main/java/org/apache/oozie/action/hadoop/HbaseCredentials.java 
> 307f565786ad77f5a35b5790bd5b052b04c6eac8 
>   core/src/main/java/org/apache/oozie/action/hadoop/Hive2ActionExecutor.java 
> 2aed936af75da38c433be5d202bdc01c2f612c76 
>   core/src/main/java/org/apache/oozie/action/hadoop/Hive2Credentials.java 
> bd2f1f768c5b04bef4a33613baede9202144c747 
>   core/src/main/java/org/apache/oozie/action/hadoop/HiveActionExecutor.java 
> 8e2453ef4f78f4a5f1f95b1ca2793f5d7e07ad81 
>   core/src/main/java/org/apache/oozie/action/hadoop/JavaActionE

Re: Review Request 54705: OOZIE-1770: Create Oozie Application Master for YARN

[Robert Kanter]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/54705/#review174589
---




core/src/main/java/org/apache/oozie/action/hadoop/JavaActionExecutor.java
Line 1166 (original), 989 (patched)


Have we figured this out?



core/src/main/java/org/apache/oozie/action/hadoop/JavaActionExecutor.java
Line 1239 (original), 1058 (patched)


Can you create a followup JIRA to decide the name for the jobs?  We might 
want to have some discussion around this, so it would be easier to spin off 
into a new JIRA.



core/src/main/java/org/apache/oozie/action/hadoop/JavaActionExecutor.java
Lines 1066 (patched)


It's my understanding that the priority doesn't do anything in Yarn today.  
YARNRunner currently uses 0 for the MR AM, so let's also use 0.

https://github.com/apache/hadoop/blob/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/main/java/org/apache/hadoop/mapred/YARNRunner.java#L130

I think we can get rid of the TODO, and replace it with a comment like "MR 
AM uses 0" or something like that so we remember why we set 0.  Should probably 
make it a constant too.



core/src/main/java/org/apache/oozie/action/hadoop/JavaActionExecutor.java
Lines 1069 (patched)


Please file a followup JIRA for this.



core/src/main/java/org/apache/oozie/action/hadoop/JavaActionExecutor.java
Lines 1344 (patched)


Also say what the default behavior is (i.e. returns false).



core/src/main/java/org/apache/oozie/action/hadoop/JavaActionExecutor.java
Lines 1382 (patched)


Should we make this a warning?



core/src/main/java/org/apache/oozie/action/hadoop/LauncherMapperHelper.java
Line 56 (original), 54 (patched)


Remove TODO?



core/src/main/java/org/apache/oozie/action/hadoop/MapReduceActionExecutor.java
Line 102 (original), 119 (patched)


I think we still need this here, but it got me thinking - do we need to 
Oozie AM to cancel the tokens before it exits?



core/src/main/java/org/apache/oozie/action/hadoop/MapReduceActionExecutor.java
Lines 136 (patched)


Is this needed here?  We already inject it on line 120.



core/src/main/java/org/apache/oozie/action/hadoop/SparkActionExecutor.java
Line 39 (original), 39 (patched)


I think we can remove this.  It's set on the Oozie AM's config, which 
doesn't do anything with this.



core/src/main/java/org/apache/oozie/command/wf/SubmitMRXCommand.java
Lines 47 (patched)


Delete


- Robert Kanter


On May 10, 2017, 10:46 p.m., Peter Cseh wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/54705/
> ---
> 
> (Updated May 10, 2017, 10:46 p.m.)
> 
> 
> Review request for oozie, Purshotam Shah, Robert Kanter, and Rohini 
> Palaniswamy.
> 
> 
> Bugs: OOZIE-1770
> https://issues.apache.org/jira/browse/OOZIE-1770
> 
> 
> Repository: oozie-git
> 
> 
> Description
> ---
> 
> Replacing the launcher MR job with a launcher AM
> 
> 
> Diffs
> -
> 
>   client/pom.xml f99366c90887a5a45f3942291c90e1cdbfd84b3f 
>   client/src/main/java/org/apache/oozie/cli/OozieCLI.java 
> 38fb84e8a8918aee6ae57d19a29cdf181aba1955 
>   client/src/main/java/org/apache/oozie/client/XOozieClient.java 
> 8538ec7c742cc013f7c18eaff3ecf61c2835cc8a 
>   core/pom.xml 14aa03483e107db326628ba627cc3ee9f5fd3f3d 
>   core/src/main/java/org/apache/oozie/WorkflowJobBean.java 
> 55d79a5d5e5e4efab9eb832e9341a16bcf8959cc 
>   core/src/main/java/org/apache/oozie/action/ActionExecutor.java 
> 1d6456b236bf9e27c0f55d9f9e3b4bd94517c797 
>   core/src/main/java/org/apache/oozie/action/hadoop/Credentials.java 
> eadb47b07d157f45f086517ab41c4b9826f003f0 
>   core/src/main/java/org/apache/oozie/action/hadoop/CredentialsProvider.java 
> 6fe22fbd2fca5d873d58ece60f20d0f131371475 
>   
> core/src/main/java/org/apache/oozie/action/hadoop/CredentialsProviderFactory.java
>  PRE-CREATION 
>   core/src/main/java/org/apache/oozie/action/hadoop/DistcpActionExecutor.java 
> 78cd257a836707c1efeac13566d6285d42970545 
>   core/src/main/java/org/apache/oozie/action/hadoop/FsActionExecutor.java 
> 27654741101d2056b25626fbbe345183eafd612d 
>   core/src/main/java/org/apache/oozie/action/hadoo

Re: Review Request 54705: OOZIE-1770: Create Oozie Application Master for YARN

[Robert Kanter]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/54705/#review174591
---




core/src/main/java/org/apache/oozie/service/HadoopAccessorService.java
Lines 528-532 (original), 535-540 (patched)


There's no group



core/src/main/java/org/apache/oozie/service/HadoopAccessorService.java
Lines 574-577 (patched)


This can use the | syntax for multiple Exception types in a catch.



core/src/main/java/org/apache/oozie/service/Services.java
Line 308 (original), 307 (patched)


Why printStackTrace?



core/src/main/java/org/apache/oozie/servlet/CallbackServlet.java
Lines 43 (patched)


Do we need this?



core/src/main/java/org/apache/oozie/util/ClasspathUtils.java
Lines 44-46 (patched)


I'm not sure we need these 3.  We only localize files into the PWD in the 
Oozie AM.



core/src/main/java/org/apache/oozie/util/Instrumentation.java
Lines 35 (patched)


Unnecessary change



core/src/main/java/org/apache/oozie/util/Instrumentation.java
Lines 699 (patched)


Unnecessary change



core/src/main/resources/oozie-default.xml
Line 1863 (original), 1838 (patched)


This can also be removed - it has no bearing now.


- Robert Kanter


On May 10, 2017, 10:46 p.m., Peter Cseh wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/54705/
> ---
> 
> (Updated May 10, 2017, 10:46 p.m.)
> 
> 
> Review request for oozie, Purshotam Shah, Robert Kanter, and Rohini 
> Palaniswamy.
> 
> 
> Bugs: OOZIE-1770
> https://issues.apache.org/jira/browse/OOZIE-1770
> 
> 
> Repository: oozie-git
> 
> 
> Description
> ---
> 
> Replacing the launcher MR job with a launcher AM
> 
> 
> Diffs
> -
> 
>   client/pom.xml f99366c90887a5a45f3942291c90e1cdbfd84b3f 
>   client/src/main/java/org/apache/oozie/cli/OozieCLI.java 
> 38fb84e8a8918aee6ae57d19a29cdf181aba1955 
>   client/src/main/java/org/apache/oozie/client/XOozieClient.java 
> 8538ec7c742cc013f7c18eaff3ecf61c2835cc8a 
>   core/pom.xml 14aa03483e107db326628ba627cc3ee9f5fd3f3d 
>   core/src/main/java/org/apache/oozie/WorkflowJobBean.java 
> 55d79a5d5e5e4efab9eb832e9341a16bcf8959cc 
>   core/src/main/java/org/apache/oozie/action/ActionExecutor.java 
> 1d6456b236bf9e27c0f55d9f9e3b4bd94517c797 
>   core/src/main/java/org/apache/oozie/action/hadoop/Credentials.java 
> eadb47b07d157f45f086517ab41c4b9826f003f0 
>   core/src/main/java/org/apache/oozie/action/hadoop/CredentialsProvider.java 
> 6fe22fbd2fca5d873d58ece60f20d0f131371475 
>   
> core/src/main/java/org/apache/oozie/action/hadoop/CredentialsProviderFactory.java
>  PRE-CREATION 
>   core/src/main/java/org/apache/oozie/action/hadoop/DistcpActionExecutor.java 
> 78cd257a836707c1efeac13566d6285d42970545 
>   core/src/main/java/org/apache/oozie/action/hadoop/FsActionExecutor.java 
> 27654741101d2056b25626fbbe345183eafd612d 
>   core/src/main/java/org/apache/oozie/action/hadoop/FsELFunctions.java 
> 801bfe6eb212b549888f0e9fb7df7b65bcf59393 
>   core/src/main/java/org/apache/oozie/action/hadoop/HCatCredentialHelper.java 
> 36ad1df727e869b8051481ad969771e91bc6ae07 
>   core/src/main/java/org/apache/oozie/action/hadoop/HCatCredentials.java 
> d7689a924059e553abaeed374eed4072933292cb 
>   core/src/main/java/org/apache/oozie/action/hadoop/HadoopELFunctions.java 
> ad2a71d641caa71efc098ad00bfbab3cec4d9506 
>   core/src/main/java/org/apache/oozie/action/hadoop/HbaseCredentials.java 
> 307f565786ad77f5a35b5790bd5b052b04c6eac8 
>   core/src/main/java/org/apache/oozie/action/hadoop/Hive2ActionExecutor.java 
> 2aed936af75da38c433be5d202bdc01c2f612c76 
>   core/src/main/java/org/apache/oozie/action/hadoop/Hive2Credentials.java 
> bd2f1f768c5b04bef4a33613baede9202144c747 
>   core/src/main/java/org/apache/oozie/action/hadoop/HiveActionExecutor.java 
> 8e2453ef4f78f4a5f1f95b1ca2793f5d7e07ad81 
>   core/src/main/java/org/apache/oozie/action/hadoop/JavaActionExecutor.java 
> d60a5c7b996a4085f9bc77d82f12ad6925280b1a 
>   core/src/main/java/org/apache/oozie/action/hadoop/LauncherMapperHelper.java 
> 72ed2f150be73f0a5e8b71b0fb6cc3594dc23078 
>   
> core/src/main/java/org/apache/oozie/action/hadoop/MapReduceActionExecutor.java
>  55c937249ac35e14bd63dd0c1fa962ea636afa3a 
>   core/src/main/java/org/apache/oozie/action/hadoop/OozieJobInfo.java 
> 581d3b3219eff882f961c8e495611e873c6a5a52 
>   core/src/main/j

Re: Review Request 54705: OOZIE-1770: Create Oozie Application Master for YARN

[Robert Kanter]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/54705/#review174829
---




core/src/test/java/org/apache/oozie/test/XTestCase.java
Lines 1238 (patched)


We should use Configuration



docs/src/site/twiki/ENG_Building.twiki
Line 115 (original), 115 (patched)


missing space in "version(Hadoop-2)"


- Robert Kanter


On May 10, 2017, 10:46 p.m., Peter Cseh wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/54705/
> ---
> 
> (Updated May 10, 2017, 10:46 p.m.)
> 
> 
> Review request for oozie, Purshotam Shah, Robert Kanter, and Rohini 
> Palaniswamy.
> 
> 
> Bugs: OOZIE-1770
> https://issues.apache.org/jira/browse/OOZIE-1770
> 
> 
> Repository: oozie-git
> 
> 
> Description
> ---
> 
> Replacing the launcher MR job with a launcher AM
> 
> 
> Diffs
> -
> 
>   client/pom.xml f99366c90887a5a45f3942291c90e1cdbfd84b3f 
>   client/src/main/java/org/apache/oozie/cli/OozieCLI.java 
> 38fb84e8a8918aee6ae57d19a29cdf181aba1955 
>   client/src/main/java/org/apache/oozie/client/XOozieClient.java 
> 8538ec7c742cc013f7c18eaff3ecf61c2835cc8a 
>   core/pom.xml 14aa03483e107db326628ba627cc3ee9f5fd3f3d 
>   core/src/main/java/org/apache/oozie/WorkflowJobBean.java 
> 55d79a5d5e5e4efab9eb832e9341a16bcf8959cc 
>   core/src/main/java/org/apache/oozie/action/ActionExecutor.java 
> 1d6456b236bf9e27c0f55d9f9e3b4bd94517c797 
>   core/src/main/java/org/apache/oozie/action/hadoop/Credentials.java 
> eadb47b07d157f45f086517ab41c4b9826f003f0 
>   core/src/main/java/org/apache/oozie/action/hadoop/CredentialsProvider.java 
> 6fe22fbd2fca5d873d58ece60f20d0f131371475 
>   
> core/src/main/java/org/apache/oozie/action/hadoop/CredentialsProviderFactory.java
>  PRE-CREATION 
>   core/src/main/java/org/apache/oozie/action/hadoop/DistcpActionExecutor.java 
> 78cd257a836707c1efeac13566d6285d42970545 
>   core/src/main/java/org/apache/oozie/action/hadoop/FsActionExecutor.java 
> 27654741101d2056b25626fbbe345183eafd612d 
>   core/src/main/java/org/apache/oozie/action/hadoop/FsELFunctions.java 
> 801bfe6eb212b549888f0e9fb7df7b65bcf59393 
>   core/src/main/java/org/apache/oozie/action/hadoop/HCatCredentialHelper.java 
> 36ad1df727e869b8051481ad969771e91bc6ae07 
>   core/src/main/java/org/apache/oozie/action/hadoop/HCatCredentials.java 
> d7689a924059e553abaeed374eed4072933292cb 
>   core/src/main/java/org/apache/oozie/action/hadoop/HadoopELFunctions.java 
> ad2a71d641caa71efc098ad00bfbab3cec4d9506 
>   core/src/main/java/org/apache/oozie/action/hadoop/HbaseCredentials.java 
> 307f565786ad77f5a35b5790bd5b052b04c6eac8 
>   core/src/main/java/org/apache/oozie/action/hadoop/Hive2ActionExecutor.java 
> 2aed936af75da38c433be5d202bdc01c2f612c76 
>   core/src/main/java/org/apache/oozie/action/hadoop/Hive2Credentials.java 
> bd2f1f768c5b04bef4a33613baede9202144c747 
>   core/src/main/java/org/apache/oozie/action/hadoop/HiveActionExecutor.java 
> 8e2453ef4f78f4a5f1f95b1ca2793f5d7e07ad81 
>   core/src/main/java/org/apache/oozie/action/hadoop/JavaActionExecutor.java 
> d60a5c7b996a4085f9bc77d82f12ad6925280b1a 
>   core/src/main/java/org/apache/oozie/action/hadoop/LauncherMapperHelper.java 
> 72ed2f150be73f0a5e8b71b0fb6cc3594dc23078 
>   
> core/src/main/java/org/apache/oozie/action/hadoop/MapReduceActionExecutor.java
>  55c937249ac35e14bd63dd0c1fa962ea636afa3a 
>   core/src/main/java/org/apache/oozie/action/hadoop/OozieJobInfo.java 
> 581d3b3219eff882f961c8e495611e873c6a5a52 
>   core/src/main/java/org/apache/oozie/action/hadoop/PigActionExecutor.java 
> 8b2dc16db80ad602911f1f64e60cfad41b198922 
>   
> core/src/main/java/org/apache/oozie/action/hadoop/ScriptLanguageActionExecutor.java
>  92e149d5cac6509893c8507b776263a38312a53b 
>   core/src/main/java/org/apache/oozie/action/hadoop/ShellActionExecutor.java 
> b9ffa7ab7ec87db8a76a3653c74974d296c8cd17 
>   core/src/main/java/org/apache/oozie/action/hadoop/SparkActionExecutor.java 
> 1a3197abb52f80b545dcc8634a8cac7a281a9eac 
>   core/src/main/java/org/apache/oozie/action/hadoop/SqoopActionExecutor.java 
> 22e28746a9a39ae6c1e22326cc4f8f759d820b20 
>   core/src/main/java/org/apache/oozie/client/rest/BulkResponseImpl.java 
> fb021bd022d199f93fc5674843514abf45ae12a5 
>   core/src/main/java/org/apache/oozie/command/XCommand.java 
> bdf13f65ea25e8e0f5125e2798568a2c03fffe81 
>   core/src/main/java/org/apache/oozie/command/wf/SubmitHttpXCommand.java 
> d2a2742ae418070a1cd0a889f14194224bc944a6 
>   core/src/main/java/org/apache/oozie/command/wf/SubmitMRXCommand.java 
> cc61d3de813c75c7f1cf4d405fc043253e3ca4ea 
>   
> core/src/main/java/org/apache/oozie/command/wf/SubmitScriptLanguageXC

Re: Review Request 54705: OOZIE-1770: Create Oozie Application Master for YARN

[Robert Kanter]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/54705/#review174833
---




sharelib/hive/src/main/java/org/apache/oozie/action/hadoop/HiveMain.java
Lines 312-318 (original)


Don't we need this to capture System.exit calls?



sharelib/hive2/src/main/java/org/apache/oozie/action/hadoop/Hive2Main.java
Lines 247-253 (original)


Same as previous


- Robert Kanter


On May 10, 2017, 10:46 p.m., Peter Cseh wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/54705/
> ---
> 
> (Updated May 10, 2017, 10:46 p.m.)
> 
> 
> Review request for oozie, Purshotam Shah, Robert Kanter, and Rohini 
> Palaniswamy.
> 
> 
> Bugs: OOZIE-1770
> https://issues.apache.org/jira/browse/OOZIE-1770
> 
> 
> Repository: oozie-git
> 
> 
> Description
> ---
> 
> Replacing the launcher MR job with a launcher AM
> 
> 
> Diffs
> -
> 
>   client/pom.xml f99366c90887a5a45f3942291c90e1cdbfd84b3f 
>   client/src/main/java/org/apache/oozie/cli/OozieCLI.java 
> 38fb84e8a8918aee6ae57d19a29cdf181aba1955 
>   client/src/main/java/org/apache/oozie/client/XOozieClient.java 
> 8538ec7c742cc013f7c18eaff3ecf61c2835cc8a 
>   core/pom.xml 14aa03483e107db326628ba627cc3ee9f5fd3f3d 
>   core/src/main/java/org/apache/oozie/WorkflowJobBean.java 
> 55d79a5d5e5e4efab9eb832e9341a16bcf8959cc 
>   core/src/main/java/org/apache/oozie/action/ActionExecutor.java 
> 1d6456b236bf9e27c0f55d9f9e3b4bd94517c797 
>   core/src/main/java/org/apache/oozie/action/hadoop/Credentials.java 
> eadb47b07d157f45f086517ab41c4b9826f003f0 
>   core/src/main/java/org/apache/oozie/action/hadoop/CredentialsProvider.java 
> 6fe22fbd2fca5d873d58ece60f20d0f131371475 
>   
> core/src/main/java/org/apache/oozie/action/hadoop/CredentialsProviderFactory.java
>  PRE-CREATION 
>   core/src/main/java/org/apache/oozie/action/hadoop/DistcpActionExecutor.java 
> 78cd257a836707c1efeac13566d6285d42970545 
>   core/src/main/java/org/apache/oozie/action/hadoop/FsActionExecutor.java 
> 27654741101d2056b25626fbbe345183eafd612d 
>   core/src/main/java/org/apache/oozie/action/hadoop/FsELFunctions.java 
> 801bfe6eb212b549888f0e9fb7df7b65bcf59393 
>   core/src/main/java/org/apache/oozie/action/hadoop/HCatCredentialHelper.java 
> 36ad1df727e869b8051481ad969771e91bc6ae07 
>   core/src/main/java/org/apache/oozie/action/hadoop/HCatCredentials.java 
> d7689a924059e553abaeed374eed4072933292cb 
>   core/src/main/java/org/apache/oozie/action/hadoop/HadoopELFunctions.java 
> ad2a71d641caa71efc098ad00bfbab3cec4d9506 
>   core/src/main/java/org/apache/oozie/action/hadoop/HbaseCredentials.java 
> 307f565786ad77f5a35b5790bd5b052b04c6eac8 
>   core/src/main/java/org/apache/oozie/action/hadoop/Hive2ActionExecutor.java 
> 2aed936af75da38c433be5d202bdc01c2f612c76 
>   core/src/main/java/org/apache/oozie/action/hadoop/Hive2Credentials.java 
> bd2f1f768c5b04bef4a33613baede9202144c747 
>   core/src/main/java/org/apache/oozie/action/hadoop/HiveActionExecutor.java 
> 8e2453ef4f78f4a5f1f95b1ca2793f5d7e07ad81 
>   core/src/main/java/org/apache/oozie/action/hadoop/JavaActionExecutor.java 
> d60a5c7b996a4085f9bc77d82f12ad6925280b1a 
>   core/src/main/java/org/apache/oozie/action/hadoop/LauncherMapperHelper.java 
> 72ed2f150be73f0a5e8b71b0fb6cc3594dc23078 
>   
> core/src/main/java/org/apache/oozie/action/hadoop/MapReduceActionExecutor.java
>  55c937249ac35e14bd63dd0c1fa962ea636afa3a 
>   core/src/main/java/org/apache/oozie/action/hadoop/OozieJobInfo.java 
> 581d3b3219eff882f961c8e495611e873c6a5a52 
>   core/src/main/java/org/apache/oozie/action/hadoop/PigActionExecutor.java 
> 8b2dc16db80ad602911f1f64e60cfad41b198922 
>   
> core/src/main/java/org/apache/oozie/action/hadoop/ScriptLanguageActionExecutor.java
>  92e149d5cac6509893c8507b776263a38312a53b 
>   core/src/main/java/org/apache/oozie/action/hadoop/ShellActionExecutor.java 
> b9ffa7ab7ec87db8a76a3653c74974d296c8cd17 
>   core/src/main/java/org/apache/oozie/action/hadoop/SparkActionExecutor.java 
> 1a3197abb52f80b545dcc8634a8cac7a281a9eac 
>   core/src/main/java/org/apache/oozie/action/hadoop/SqoopActionExecutor.java 
> 22e28746a9a39ae6c1e22326cc4f8f759d820b20 
>   core/src/main/java/org/apache/oozie/client/rest/BulkResponseImpl.java 
> fb021bd022d199f93fc5674843514abf45ae12a5 
>   core/src/main/java/org/apache/oozie/command/XCommand.java 
> bdf13f65ea25e8e0f5125e2798568a2c03fffe81 
>   core/src/main/java/org/apache/oozie/command/wf/SubmitHttpXCommand.java 
> d2a2742ae418070a1cd0a889f14194224bc944a6 
>   core/src/main/java/org/apache/oozie/command/wf/SubmitMRXCommand.java 
> cc61d3de813c75c7f1cf4d405fc043253e3ca4ea 
>   
> core/src/main/java/or

Re: Review Request 54705: OOZIE-1770: Create Oozie Application Master for YARN

[Robert Kanter]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/54705/#review174834
---




sharelib/oozie/src/main/java/org/apache/oozie/action/hadoop/HdfsOperations.java
Lines 61 (patched)


Replace with finalPath.toUri() - no need to create a duplicate Path object.



sharelib/oozie/src/main/java/org/apache/oozie/action/hadoop/LauncherAM.java
Lines 284 (patched)


What should we do about this?  I don't think there's really a reason the 
user should play with this, so I'm thinking we should leave this as hardcoded 
and remove the TODO.



sharelib/oozie/src/main/java/org/apache/oozie/action/hadoop/LauncherMain.java
Lines 222-226 (patched)


Can be combined into a multi-catch



sharelib/oozie/src/main/java/org/apache/oozie/action/hadoop/LauncherMain.java
Lines 260-264 (patched)


Multicatch



sharelib/oozie/src/main/java/org/apache/oozie/action/hadoop/LauncherMapper.java
Lines 57 (patched)


We should delete this class.



sharelib/oozie/src/test/java/org/apache/oozie/action/hadoop/TestLauncherAM.java
Lines 379 (patched)


Please file a followup JIRA for this if we're not going to figure it out 
now.


- Robert Kanter


On May 10, 2017, 10:46 p.m., Peter Cseh wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/54705/
> ---
> 
> (Updated May 10, 2017, 10:46 p.m.)
> 
> 
> Review request for oozie, Purshotam Shah, Robert Kanter, and Rohini 
> Palaniswamy.
> 
> 
> Bugs: OOZIE-1770
> https://issues.apache.org/jira/browse/OOZIE-1770
> 
> 
> Repository: oozie-git
> 
> 
> Description
> ---
> 
> Replacing the launcher MR job with a launcher AM
> 
> 
> Diffs
> -
> 
>   client/pom.xml f99366c90887a5a45f3942291c90e1cdbfd84b3f 
>   client/src/main/java/org/apache/oozie/cli/OozieCLI.java 
> 38fb84e8a8918aee6ae57d19a29cdf181aba1955 
>   client/src/main/java/org/apache/oozie/client/XOozieClient.java 
> 8538ec7c742cc013f7c18eaff3ecf61c2835cc8a 
>   core/pom.xml 14aa03483e107db326628ba627cc3ee9f5fd3f3d 
>   core/src/main/java/org/apache/oozie/WorkflowJobBean.java 
> 55d79a5d5e5e4efab9eb832e9341a16bcf8959cc 
>   core/src/main/java/org/apache/oozie/action/ActionExecutor.java 
> 1d6456b236bf9e27c0f55d9f9e3b4bd94517c797 
>   core/src/main/java/org/apache/oozie/action/hadoop/Credentials.java 
> eadb47b07d157f45f086517ab41c4b9826f003f0 
>   core/src/main/java/org/apache/oozie/action/hadoop/CredentialsProvider.java 
> 6fe22fbd2fca5d873d58ece60f20d0f131371475 
>   
> core/src/main/java/org/apache/oozie/action/hadoop/CredentialsProviderFactory.java
>  PRE-CREATION 
>   core/src/main/java/org/apache/oozie/action/hadoop/DistcpActionExecutor.java 
> 78cd257a836707c1efeac13566d6285d42970545 
>   core/src/main/java/org/apache/oozie/action/hadoop/FsActionExecutor.java 
> 27654741101d2056b25626fbbe345183eafd612d 
>   core/src/main/java/org/apache/oozie/action/hadoop/FsELFunctions.java 
> 801bfe6eb212b549888f0e9fb7df7b65bcf59393 
>   core/src/main/java/org/apache/oozie/action/hadoop/HCatCredentialHelper.java 
> 36ad1df727e869b8051481ad969771e91bc6ae07 
>   core/src/main/java/org/apache/oozie/action/hadoop/HCatCredentials.java 
> d7689a924059e553abaeed374eed4072933292cb 
>   core/src/main/java/org/apache/oozie/action/hadoop/HadoopELFunctions.java 
> ad2a71d641caa71efc098ad00bfbab3cec4d9506 
>   core/src/main/java/org/apache/oozie/action/hadoop/HbaseCredentials.java 
> 307f565786ad77f5a35b5790bd5b052b04c6eac8 
>   core/src/main/java/org/apache/oozie/action/hadoop/Hive2ActionExecutor.java 
> 2aed936af75da38c433be5d202bdc01c2f612c76 
>   core/src/main/java/org/apache/oozie/action/hadoop/Hive2Credentials.java 
> bd2f1f768c5b04bef4a33613baede9202144c747 
>   core/src/main/java/org/apache/oozie/action/hadoop/HiveActionExecutor.java 
> 8e2453ef4f78f4a5f1f95b1ca2793f5d7e07ad81 
>   core/src/main/java/org/apache/oozie/action/hadoop/JavaActionExecutor.java 
> d60a5c7b996a4085f9bc77d82f12ad6925280b1a 
>   core/src/main/java/org/apache/oozie/action/hadoop/LauncherMapperHelper.java 
> 72ed2f150be73f0a5e8b71b0fb6cc3594dc23078 
>   
> core/src/main/java/org/apache/oozie/action/hadoop/MapReduceActionExecutor.java
>  55c937249ac35e14bd63dd0c1fa962ea636afa3a 
>   core/src/main/java/org/apache/oozie/action/hadoop/OozieJobInfo.java 
> 581d3b3219eff882f961c8e495611e873c6a5a52 
>   core/src/main/java/org/apache/oozie/action/hadoop/PigActionExecutor.java 
> 8b2dc16db80ad602911f1f64e60cfad41b198922 
>   
> core/src/main/j

Re: Review Request 54705: OOZIE-1770: Create Oozie Application Master for YARN

[Rohini Palaniswamy]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/54705/#review174883
---



Have asked for new jiras for enhancements, missing scenarios or changes that 
require more work. Please reply with the new jira link before closing those 
issues in review board for easier referencing. Also add a TODO in the code with 
the jira number.

Some more action items:
1) Getting tokens for ATS (Application Timeline Server) is missing. Please 
create a separate jira for that.
2) Getting tokens for oozie.launcher.mapreduce.job.hdfs-servers is missing. 
Cannot do cross cluster distcp without that. Please create a separate jira for 
that.
3) There should be code to get the Oozie AM application state from AHS 
(Application History Server) in check() if the completed job is gone from RM 
memory. For mapreduce launcher jobs, JobClient goes to Job History server to 
fetch that information. Please create a separate jira for that.


core/src/main/java/org/apache/oozie/action/hadoop/Credentials.java
Line 25 (original), 28 (patched)


Shouldn't this be just a new class instead of moving Credentials as they 
are totally unrelated?



core/src/main/java/org/apache/oozie/action/hadoop/JavaActionExecutor.java
Lines 156-158 (original), 177-179 (patched)


Do we still need the input and output format classes?



core/src/main/java/org/apache/oozie/action/hadoop/JavaActionExecutor.java
Line 325 (original)


We still need the java opts, memory.mb and env settings specified by the 
user using oozie.launcher. settings applied to the Oozie AM. Please create a 
new jira for this.



core/src/main/java/org/apache/oozie/action/hadoop/JavaActionExecutor.java
Line 423 (original), 321 (patched)


This should not be required anymore. The Oozie AM code should be doing the 
same thing as done in MAPREDUCE-6472.



core/src/main/java/org/apache/oozie/action/hadoop/JavaActionExecutor.java
Line 1147 (original)


Can you retain this debug message? Has come in useful debugging slowness in 
app submissions



core/src/main/java/org/apache/oozie/action/hadoop/JavaActionExecutor.java
Lines 1133 (patched)


You have to translate them from mapreduce settings instead of setting own 
defaults. Else it will break lot of existing user jobs which have been tuned to 
have more memory.  Please create a new jira for this (same issue as mentioned 
in updateConfForUberMode comment)



core/src/main/java/org/apache/oozie/action/hadoop/JavaActionExecutor.java
Lines 1292-1293 (original), 1187-1188 (patched)


Nitpick. Unnecessary local variable assignment. 
CredentialsProviderFactory.getInstance().createCredentialsProvider(credProps.getType());



core/src/main/java/org/apache/oozie/action/hadoop/JavaActionExecutor.java
Line 1622 (original), 1514 (patched)


Put just the kill call within the loop inside try catch block and log a 
warning if it cannot be killed. Killing an already completed child application 
may throw exception and may prevent other child applications from being killed.



core/src/main/java/org/apache/oozie/action/hadoop/MapReduceActionExecutor.java
Lines 413 (patched)


This code will have race condition issues. super.check() will check the 
status of launcher AM if action data file is not found. While it starts the 
check, if it completes then it will read the NEW_ID from action data and mark 
the action as completed even though the mapreduce job is just launched and is 
still running.



core/src/main/java/org/apache/oozie/action/hadoop/PigActionExecutor.java
Line 130 (original), 126 (patched)


Can you rename method as addMapreduceToClasspath?



core/src/main/java/org/apache/oozie/command/XCommand.java
Line 247 (original), 247 (patched)


Unnecessary thread local call when there are no commands to be queued. Can 
we make getInterruptTypes as static method in CallableQueueService keeping the 
HashSet as private static?



core/src/main/java/org/apache/oozie/service/HadoopAccessorService.java
Lines 777 (patched)


os needs to be closed.



core/src/test/java/org/apache/oozie/action/hadoop/TestJavaActionExecutor.java
Line 546 (original)


Why is this test removed? This is important.



core/src/test/ja

Re: Review Request 54705: OOZIE-1770: Create Oozie Application Master for YARN

[Rohini Palaniswamy]

> On May 12, 2017, 8:27 p.m., Robert Kanter wrote:
> > sharelib/oozie/src/main/java/org/apache/oozie/action/hadoop/LauncherAM.java
> > Lines 284 (patched)
> > 
> >
> > What should we do about this?  I don't think there's really a reason 
> > the user should play with this, so I'm thinking we should leave this as 
> > hardcoded and remove the TODO.

For cases like this, please use the value of mapreduce settings if it is set 
instead of coming up with a new setting name or trying to hard code some value. 
In our clusters we use a higher value to put less load on RM.


- Rohini


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/54705/#review174834
---


On May 10, 2017, 10:46 p.m., Peter Cseh wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/54705/
> ---
> 
> (Updated May 10, 2017, 10:46 p.m.)
> 
> 
> Review request for oozie, Purshotam Shah, Robert Kanter, and Rohini 
> Palaniswamy.
> 
> 
> Bugs: OOZIE-1770
> https://issues.apache.org/jira/browse/OOZIE-1770
> 
> 
> Repository: oozie-git
> 
> 
> Description
> ---
> 
> Replacing the launcher MR job with a launcher AM
> 
> 
> Diffs
> -
> 
>   client/pom.xml f99366c90887a5a45f3942291c90e1cdbfd84b3f 
>   client/src/main/java/org/apache/oozie/cli/OozieCLI.java 
> 38fb84e8a8918aee6ae57d19a29cdf181aba1955 
>   client/src/main/java/org/apache/oozie/client/XOozieClient.java 
> 8538ec7c742cc013f7c18eaff3ecf61c2835cc8a 
>   core/pom.xml 14aa03483e107db326628ba627cc3ee9f5fd3f3d 
>   core/src/main/java/org/apache/oozie/WorkflowJobBean.java 
> 55d79a5d5e5e4efab9eb832e9341a16bcf8959cc 
>   core/src/main/java/org/apache/oozie/action/ActionExecutor.java 
> 1d6456b236bf9e27c0f55d9f9e3b4bd94517c797 
>   core/src/main/java/org/apache/oozie/action/hadoop/Credentials.java 
> eadb47b07d157f45f086517ab41c4b9826f003f0 
>   core/src/main/java/org/apache/oozie/action/hadoop/CredentialsProvider.java 
> 6fe22fbd2fca5d873d58ece60f20d0f131371475 
>   
> core/src/main/java/org/apache/oozie/action/hadoop/CredentialsProviderFactory.java
>  PRE-CREATION 
>   core/src/main/java/org/apache/oozie/action/hadoop/DistcpActionExecutor.java 
> 78cd257a836707c1efeac13566d6285d42970545 
>   core/src/main/java/org/apache/oozie/action/hadoop/FsActionExecutor.java 
> 27654741101d2056b25626fbbe345183eafd612d 
>   core/src/main/java/org/apache/oozie/action/hadoop/FsELFunctions.java 
> 801bfe6eb212b549888f0e9fb7df7b65bcf59393 
>   core/src/main/java/org/apache/oozie/action/hadoop/HCatCredentialHelper.java 
> 36ad1df727e869b8051481ad969771e91bc6ae07 
>   core/src/main/java/org/apache/oozie/action/hadoop/HCatCredentials.java 
> d7689a924059e553abaeed374eed4072933292cb 
>   core/src/main/java/org/apache/oozie/action/hadoop/HadoopELFunctions.java 
> ad2a71d641caa71efc098ad00bfbab3cec4d9506 
>   core/src/main/java/org/apache/oozie/action/hadoop/HbaseCredentials.java 
> 307f565786ad77f5a35b5790bd5b052b04c6eac8 
>   core/src/main/java/org/apache/oozie/action/hadoop/Hive2ActionExecutor.java 
> 2aed936af75da38c433be5d202bdc01c2f612c76 
>   core/src/main/java/org/apache/oozie/action/hadoop/Hive2Credentials.java 
> bd2f1f768c5b04bef4a33613baede9202144c747 
>   core/src/main/java/org/apache/oozie/action/hadoop/HiveActionExecutor.java 
> 8e2453ef4f78f4a5f1f95b1ca2793f5d7e07ad81 
>   core/src/main/java/org/apache/oozie/action/hadoop/JavaActionExecutor.java 
> d60a5c7b996a4085f9bc77d82f12ad6925280b1a 
>   core/src/main/java/org/apache/oozie/action/hadoop/LauncherMapperHelper.java 
> 72ed2f150be73f0a5e8b71b0fb6cc3594dc23078 
>   
> core/src/main/java/org/apache/oozie/action/hadoop/MapReduceActionExecutor.java
>  55c937249ac35e14bd63dd0c1fa962ea636afa3a 
>   core/src/main/java/org/apache/oozie/action/hadoop/OozieJobInfo.java 
> 581d3b3219eff882f961c8e495611e873c6a5a52 
>   core/src/main/java/org/apache/oozie/action/hadoop/PigActionExecutor.java 
> 8b2dc16db80ad602911f1f64e60cfad41b198922 
>   
> core/src/main/java/org/apache/oozie/action/hadoop/ScriptLanguageActionExecutor.java
>  92e149d5cac6509893c8507b776263a38312a53b 
>   core/src/main/java/org/apache/oozie/action/hadoop/ShellActionExecutor.java 
> b9ffa7ab7ec87db8a76a3653c74974d296c8cd17 
>   core/src/main/java/org/apache/oozie/action/hadoop/SparkActionExecutor.java 
> 1a3197abb52f80b545dcc8634a8cac7a281a9eac 
>   core/src/main/java/org/apache/oozie/action/hadoop/SqoopActionExecutor.java 
> 22e28746a9a39ae6c1e22326cc4f8f759d820b20 
>   core/src/main/java/org/apache/oozie/client/rest/BulkResponseImpl.java 
> fb021bd022d199f93fc5674843514abf45ae12a5 
>   core/src/main/java/org/apache/oozie/command/XCommand.java 
> bdf13f65ea25e8e0f5125e2798568a2c03fffe81 
>  

Re: Review Request 54705: OOZIE-1770: Create Oozie Application Master for YARN

[Rohini Palaniswamy]

> On May 11, 2017, 12:41 a.m., Robert Kanter wrote:
> > core/src/main/java/org/apache/oozie/action/hadoop/JavaActionExecutor.java
> > Line 1166 (original), 989 (patched)
> > 
> >
> > Have we figured this out?

Yes it is needed. Many people launch MR jobs from java actions today.


> On May 11, 2017, 12:41 a.m., Robert Kanter wrote:
> > core/src/main/java/org/apache/oozie/action/hadoop/JavaActionExecutor.java
> > Lines 1066 (patched)
> > 
> >
> > It's my understanding that the priority doesn't do anything in Yarn 
> > today.  YARNRunner currently uses 0 for the MR AM, so let's also use 0.
> > 
> > https://github.com/apache/hadoop/blob/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/main/java/org/apache/hadoop/mapred/YARNRunner.java#L130
> > 
> > I think we can get rid of the TODO, and replace it with a comment like 
> > "MR AM uses 0" or something like that so we remember why we set 0.  Should 
> > probably make it a constant too.

mapred.job.priority is used by some users. Oozie AM needs to support that. 
Please file a new jira for it


> On May 11, 2017, 12:41 a.m., Robert Kanter wrote:
> > core/src/main/java/org/apache/oozie/action/hadoop/JavaActionExecutor.java
> > Lines 1069 (patched)
> > 
> >
> > Please file a followup JIRA for this.

This is important and needs to be fixed. 95% of our users configure a different 
queue using oozie.launcher settings.


> On May 11, 2017, 12:41 a.m., Robert Kanter wrote:
> > core/src/main/java/org/apache/oozie/action/hadoop/MapReduceActionExecutor.java
> > Line 102 (original), 119 (patched)
> > 
> >
> > I think we still need this here, but it got me thinking - do we need to 
> > Oozie AM to cancel the tokens before it exits?

Yes. We do need to have the delegation tokens cancelled. AM itself should not 
be cancelling the tokens. It has tell the RM via 
ApplicationSubmissionContext.setCancelTokensWhenComplete and RM has to do it.


- Rohini


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/54705/#review174589
---


On May 10, 2017, 10:46 p.m., Peter Cseh wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/54705/
> ---
> 
> (Updated May 10, 2017, 10:46 p.m.)
> 
> 
> Review request for oozie, Purshotam Shah, Robert Kanter, and Rohini 
> Palaniswamy.
> 
> 
> Bugs: OOZIE-1770
> https://issues.apache.org/jira/browse/OOZIE-1770
> 
> 
> Repository: oozie-git
> 
> 
> Description
> ---
> 
> Replacing the launcher MR job with a launcher AM
> 
> 
> Diffs
> -
> 
>   client/pom.xml f99366c90887a5a45f3942291c90e1cdbfd84b3f 
>   client/src/main/java/org/apache/oozie/cli/OozieCLI.java 
> 38fb84e8a8918aee6ae57d19a29cdf181aba1955 
>   client/src/main/java/org/apache/oozie/client/XOozieClient.java 
> 8538ec7c742cc013f7c18eaff3ecf61c2835cc8a 
>   core/pom.xml 14aa03483e107db326628ba627cc3ee9f5fd3f3d 
>   core/src/main/java/org/apache/oozie/WorkflowJobBean.java 
> 55d79a5d5e5e4efab9eb832e9341a16bcf8959cc 
>   core/src/main/java/org/apache/oozie/action/ActionExecutor.java 
> 1d6456b236bf9e27c0f55d9f9e3b4bd94517c797 
>   core/src/main/java/org/apache/oozie/action/hadoop/Credentials.java 
> eadb47b07d157f45f086517ab41c4b9826f003f0 
>   core/src/main/java/org/apache/oozie/action/hadoop/CredentialsProvider.java 
> 6fe22fbd2fca5d873d58ece60f20d0f131371475 
>   
> core/src/main/java/org/apache/oozie/action/hadoop/CredentialsProviderFactory.java
>  PRE-CREATION 
>   core/src/main/java/org/apache/oozie/action/hadoop/DistcpActionExecutor.java 
> 78cd257a836707c1efeac13566d6285d42970545 
>   core/src/main/java/org/apache/oozie/action/hadoop/FsActionExecutor.java 
> 27654741101d2056b25626fbbe345183eafd612d 
>   core/src/main/java/org/apache/oozie/action/hadoop/FsELFunctions.java 
> 801bfe6eb212b549888f0e9fb7df7b65bcf59393 
>   core/src/main/java/org/apache/oozie/action/hadoop/HCatCredentialHelper.java 
> 36ad1df727e869b8051481ad969771e91bc6ae07 
>   core/src/main/java/org/apache/oozie/action/hadoop/HCatCredentials.java 
> d7689a924059e553abaeed374eed4072933292cb 
>   core/src/main/java/org/apache/oozie/action/hadoop/HadoopELFunctions.java 
> ad2a71d641caa71efc098ad00bfbab3cec4d9506 
>   core/src/main/java/org/apache/oozie/action/hadoop/HbaseCredentials.java 
> 307f565786ad77f5a35b5790bd5b052b04c6eac8 
>   core/src/main/java/org/apache/oozie/action/hadoop/Hive2ActionExecutor.java 
> 2aed936af75da3

Re: Review Request 54705: OOZIE-1770: Create Oozie Application Master for YARN

[Rohini Palaniswamy]

> On March 26, 2017, 12:37 a.m., Rohini Palaniswamy wrote:
> > core/src/main/java/org/apache/oozie/action/hadoop/SparkActionExecutor.java
> > Lines 112-114 (original)
> > 
> >
> > You can't remove this. Will breaking existing users. Since it starts 
> > with oozie.launcher, config deprecation will not handle it. Basically any 
> > oozie.launcher hadoop 1.x settings cannot be removed because of that.
> 
> Peter Cseh wrote:
> This is setting "mapreduce.task.classpath.user.precedence" in the 
> launcher job conf, it was parsed by the launcher Mapper and the LauncherAM is 
> not doing anything with this property at the moment. Should we add similar 
> functionality into LauncherAM? Is this a must have to support without MR 
> launcher?
> 
> Robert Kanter wrote:
> I'd rather not have to support an equivalent of 
> "mapreduce.task.classpath.user.precedence" in the Oozie AM itself.  It's a 
> really ugly hacky thing.  If we want to go down this route, a better fix 
> would be to provide complete classpath isolation somehow; but I think that 
> could be a future feature.

We need to provide feature parity for both 
mapreduce.task.classpath.user.precedence (simple hack, but very useful and 
addressed most of cases) and 
mapreduce.job.classloader/mapreduce.job.classloader.system.classes (this 
provided the isolation in the mapreduce world). Please create a separate jira 
for this issue that can be addressed after the merge. Have seen a lot of users 
use this feature due to one conflict or another. One of the most common cases 
that requires this is avro.


> On March 26, 2017, 12:37 a.m., Rohini Palaniswamy wrote:
> > core/src/main/java/org/apache/oozie/action/hadoop/SqoopActionExecutor.java
> > Line 89 (original), 87 (patched)
> > 
> >
> > Revert back to old code. Just remove the final instead of creating 
> > another temporary list.
> 
> Peter Cseh wrote:
> merge issue. will check

You can totally avoid the array list to array conversion by using 
String.split() instead of StringTokenizer. 

https://docs.oracle.com/javase/7/docs/api/java/util/StringTokenizer.html
StringTokenizer is a legacy class that is retained for compatibility reasons 
although its use is discouraged in new code.


- Rohini


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/54705/#review170061
---


On May 10, 2017, 10:46 p.m., Peter Cseh wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/54705/
> ---
> 
> (Updated May 10, 2017, 10:46 p.m.)
> 
> 
> Review request for oozie, Purshotam Shah, Robert Kanter, and Rohini 
> Palaniswamy.
> 
> 
> Bugs: OOZIE-1770
> https://issues.apache.org/jira/browse/OOZIE-1770
> 
> 
> Repository: oozie-git
> 
> 
> Description
> ---
> 
> Replacing the launcher MR job with a launcher AM
> 
> 
> Diffs
> -
> 
>   client/pom.xml f99366c90887a5a45f3942291c90e1cdbfd84b3f 
>   client/src/main/java/org/apache/oozie/cli/OozieCLI.java 
> 38fb84e8a8918aee6ae57d19a29cdf181aba1955 
>   client/src/main/java/org/apache/oozie/client/XOozieClient.java 
> 8538ec7c742cc013f7c18eaff3ecf61c2835cc8a 
>   core/pom.xml 14aa03483e107db326628ba627cc3ee9f5fd3f3d 
>   core/src/main/java/org/apache/oozie/WorkflowJobBean.java 
> 55d79a5d5e5e4efab9eb832e9341a16bcf8959cc 
>   core/src/main/java/org/apache/oozie/action/ActionExecutor.java 
> 1d6456b236bf9e27c0f55d9f9e3b4bd94517c797 
>   core/src/main/java/org/apache/oozie/action/hadoop/Credentials.java 
> eadb47b07d157f45f086517ab41c4b9826f003f0 
>   core/src/main/java/org/apache/oozie/action/hadoop/CredentialsProvider.java 
> 6fe22fbd2fca5d873d58ece60f20d0f131371475 
>   
> core/src/main/java/org/apache/oozie/action/hadoop/CredentialsProviderFactory.java
>  PRE-CREATION 
>   core/src/main/java/org/apache/oozie/action/hadoop/DistcpActionExecutor.java 
> 78cd257a836707c1efeac13566d6285d42970545 
>   core/src/main/java/org/apache/oozie/action/hadoop/FsActionExecutor.java 
> 27654741101d2056b25626fbbe345183eafd612d 
>   core/src/main/java/org/apache/oozie/action/hadoop/FsELFunctions.java 
> 801bfe6eb212b549888f0e9fb7df7b65bcf59393 
>   core/src/main/java/org/apache/oozie/action/hadoop/HCatCredentialHelper.java 
> 36ad1df727e869b8051481ad969771e91bc6ae07 
>   core/src/main/java/org/apache/oozie/action/hadoop/HCatCredentials.java 
> d7689a924059e553abaeed374eed4072933292cb 
>   core/src/main/java/org/apache/oozie/action/hadoop/HadoopELFunctions.java 
> ad2a71d641caa71efc098ad00bfbab3cec4d9506 
>   core/src/main/java/org/apache/oozie/action/hadoop/HbaseCredentials.java 
> 307f565786ad77f5a35b5790bd5b052

Re: Review Request 54705: OOZIE-1770: Create Oozie Application Master for YARN

[Rohini Palaniswamy]

> On May 14, 2017, 4:16 a.m., Rohini Palaniswamy wrote:
> > core/src/main/java/org/apache/oozie/action/hadoop/JavaActionExecutor.java
> > Line 325 (original)
> > 
> >
> > We still need the java opts, memory.mb and env settings specified by 
> > the user using oozie.launcher. settings applied to the Oozie AM. Please 
> > create a new jira for this.

This is different from OOZIE-2687 which introduces a option xml section to 
simplify specification of those launcher settings. Existing oozie.launcher MR 
settings specified by users should continute to work as is without user having 
to rewrite their workflow.xml. Else this will be a major backward 
incompatibility.


- Rohini


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/54705/#review174883
---


On May 10, 2017, 10:46 p.m., Peter Cseh wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/54705/
> ---
> 
> (Updated May 10, 2017, 10:46 p.m.)
> 
> 
> Review request for oozie, Purshotam Shah, Robert Kanter, and Rohini 
> Palaniswamy.
> 
> 
> Bugs: OOZIE-1770
> https://issues.apache.org/jira/browse/OOZIE-1770
> 
> 
> Repository: oozie-git
> 
> 
> Description
> ---
> 
> Replacing the launcher MR job with a launcher AM
> 
> 
> Diffs
> -
> 
>   client/pom.xml f99366c90887a5a45f3942291c90e1cdbfd84b3f 
>   client/src/main/java/org/apache/oozie/cli/OozieCLI.java 
> 38fb84e8a8918aee6ae57d19a29cdf181aba1955 
>   client/src/main/java/org/apache/oozie/client/XOozieClient.java 
> 8538ec7c742cc013f7c18eaff3ecf61c2835cc8a 
>   core/pom.xml 14aa03483e107db326628ba627cc3ee9f5fd3f3d 
>   core/src/main/java/org/apache/oozie/WorkflowJobBean.java 
> 55d79a5d5e5e4efab9eb832e9341a16bcf8959cc 
>   core/src/main/java/org/apache/oozie/action/ActionExecutor.java 
> 1d6456b236bf9e27c0f55d9f9e3b4bd94517c797 
>   core/src/main/java/org/apache/oozie/action/hadoop/Credentials.java 
> eadb47b07d157f45f086517ab41c4b9826f003f0 
>   core/src/main/java/org/apache/oozie/action/hadoop/CredentialsProvider.java 
> 6fe22fbd2fca5d873d58ece60f20d0f131371475 
>   
> core/src/main/java/org/apache/oozie/action/hadoop/CredentialsProviderFactory.java
>  PRE-CREATION 
>   core/src/main/java/org/apache/oozie/action/hadoop/DistcpActionExecutor.java 
> 78cd257a836707c1efeac13566d6285d42970545 
>   core/src/main/java/org/apache/oozie/action/hadoop/FsActionExecutor.java 
> 27654741101d2056b25626fbbe345183eafd612d 
>   core/src/main/java/org/apache/oozie/action/hadoop/FsELFunctions.java 
> 801bfe6eb212b549888f0e9fb7df7b65bcf59393 
>   core/src/main/java/org/apache/oozie/action/hadoop/HCatCredentialHelper.java 
> 36ad1df727e869b8051481ad969771e91bc6ae07 
>   core/src/main/java/org/apache/oozie/action/hadoop/HCatCredentials.java 
> d7689a924059e553abaeed374eed4072933292cb 
>   core/src/main/java/org/apache/oozie/action/hadoop/HadoopELFunctions.java 
> ad2a71d641caa71efc098ad00bfbab3cec4d9506 
>   core/src/main/java/org/apache/oozie/action/hadoop/HbaseCredentials.java 
> 307f565786ad77f5a35b5790bd5b052b04c6eac8 
>   core/src/main/java/org/apache/oozie/action/hadoop/Hive2ActionExecutor.java 
> 2aed936af75da38c433be5d202bdc01c2f612c76 
>   core/src/main/java/org/apache/oozie/action/hadoop/Hive2Credentials.java 
> bd2f1f768c5b04bef4a33613baede9202144c747 
>   core/src/main/java/org/apache/oozie/action/hadoop/HiveActionExecutor.java 
> 8e2453ef4f78f4a5f1f95b1ca2793f5d7e07ad81 
>   core/src/main/java/org/apache/oozie/action/hadoop/JavaActionExecutor.java 
> d60a5c7b996a4085f9bc77d82f12ad6925280b1a 
>   core/src/main/java/org/apache/oozie/action/hadoop/LauncherMapperHelper.java 
> 72ed2f150be73f0a5e8b71b0fb6cc3594dc23078 
>   
> core/src/main/java/org/apache/oozie/action/hadoop/MapReduceActionExecutor.java
>  55c937249ac35e14bd63dd0c1fa962ea636afa3a 
>   core/src/main/java/org/apache/oozie/action/hadoop/OozieJobInfo.java 
> 581d3b3219eff882f961c8e495611e873c6a5a52 
>   core/src/main/java/org/apache/oozie/action/hadoop/PigActionExecutor.java 
> 8b2dc16db80ad602911f1f64e60cfad41b198922 
>   
> core/src/main/java/org/apache/oozie/action/hadoop/ScriptLanguageActionExecutor.java
>  92e149d5cac6509893c8507b776263a38312a53b 
>   core/src/main/java/org/apache/oozie/action/hadoop/ShellActionExecutor.java 
> b9ffa7ab7ec87db8a76a3653c74974d296c8cd17 
>   core/src/main/java/org/apache/oozie/action/hadoop/SparkActionExecutor.java 
> 1a3197abb52f80b545dcc8634a8cac7a281a9eac 
>   core/src/main/java/org/apache/oozie/action/hadoop/SqoopActionExecutor.java 
> 22e28746a9a39ae6c1e22326cc4f8f759d820b20 
>   core/src/main/java/org/apache/oozie/client/rest/BulkResponseImpl.java 
> fb021bd022d199f93fc5674843514abf45ae12a5 
>   core/src

Re: Review Request 54705: OOZIE-1770: Create Oozie Application Master for YARN

[Peter Cseh]

> On May 10, 2017, 11:54 p.m., Robert Kanter wrote:
> > core/src/main/java/org/apache/oozie/action/hadoop/CredentialsProviderFactory.java
> > Lines 108 (patched)
> > 
> >
> > I imagine there's one of these in some Commons library or something?  
> > At the very least, this should go into a Utils class.

This is comming from here: 
https://github.com/apache/oozie/blame/master/core/src/main/java/org/apache/oozie/action/hadoop/CredentialsProvider.java#L91
 
We'll look for a better place for it.


- Peter


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/54705/#review174588
---


On May 10, 2017, 10:46 p.m., Peter Cseh wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/54705/
> ---
> 
> (Updated May 10, 2017, 10:46 p.m.)
> 
> 
> Review request for oozie, Purshotam Shah, Robert Kanter, and Rohini 
> Palaniswamy.
> 
> 
> Bugs: OOZIE-1770
> https://issues.apache.org/jira/browse/OOZIE-1770
> 
> 
> Repository: oozie-git
> 
> 
> Description
> ---
> 
> Replacing the launcher MR job with a launcher AM
> 
> 
> Diffs
> -
> 
>   client/pom.xml f99366c90887a5a45f3942291c90e1cdbfd84b3f 
>   client/src/main/java/org/apache/oozie/cli/OozieCLI.java 
> 38fb84e8a8918aee6ae57d19a29cdf181aba1955 
>   client/src/main/java/org/apache/oozie/client/XOozieClient.java 
> 8538ec7c742cc013f7c18eaff3ecf61c2835cc8a 
>   core/pom.xml 14aa03483e107db326628ba627cc3ee9f5fd3f3d 
>   core/src/main/java/org/apache/oozie/WorkflowJobBean.java 
> 55d79a5d5e5e4efab9eb832e9341a16bcf8959cc 
>   core/src/main/java/org/apache/oozie/action/ActionExecutor.java 
> 1d6456b236bf9e27c0f55d9f9e3b4bd94517c797 
>   core/src/main/java/org/apache/oozie/action/hadoop/Credentials.java 
> eadb47b07d157f45f086517ab41c4b9826f003f0 
>   core/src/main/java/org/apache/oozie/action/hadoop/CredentialsProvider.java 
> 6fe22fbd2fca5d873d58ece60f20d0f131371475 
>   
> core/src/main/java/org/apache/oozie/action/hadoop/CredentialsProviderFactory.java
>  PRE-CREATION 
>   core/src/main/java/org/apache/oozie/action/hadoop/DistcpActionExecutor.java 
> 78cd257a836707c1efeac13566d6285d42970545 
>   core/src/main/java/org/apache/oozie/action/hadoop/FsActionExecutor.java 
> 27654741101d2056b25626fbbe345183eafd612d 
>   core/src/main/java/org/apache/oozie/action/hadoop/FsELFunctions.java 
> 801bfe6eb212b549888f0e9fb7df7b65bcf59393 
>   core/src/main/java/org/apache/oozie/action/hadoop/HCatCredentialHelper.java 
> 36ad1df727e869b8051481ad969771e91bc6ae07 
>   core/src/main/java/org/apache/oozie/action/hadoop/HCatCredentials.java 
> d7689a924059e553abaeed374eed4072933292cb 
>   core/src/main/java/org/apache/oozie/action/hadoop/HadoopELFunctions.java 
> ad2a71d641caa71efc098ad00bfbab3cec4d9506 
>   core/src/main/java/org/apache/oozie/action/hadoop/HbaseCredentials.java 
> 307f565786ad77f5a35b5790bd5b052b04c6eac8 
>   core/src/main/java/org/apache/oozie/action/hadoop/Hive2ActionExecutor.java 
> 2aed936af75da38c433be5d202bdc01c2f612c76 
>   core/src/main/java/org/apache/oozie/action/hadoop/Hive2Credentials.java 
> bd2f1f768c5b04bef4a33613baede9202144c747 
>   core/src/main/java/org/apache/oozie/action/hadoop/HiveActionExecutor.java 
> 8e2453ef4f78f4a5f1f95b1ca2793f5d7e07ad81 
>   core/src/main/java/org/apache/oozie/action/hadoop/JavaActionExecutor.java 
> d60a5c7b996a4085f9bc77d82f12ad6925280b1a 
>   core/src/main/java/org/apache/oozie/action/hadoop/LauncherMapperHelper.java 
> 72ed2f150be73f0a5e8b71b0fb6cc3594dc23078 
>   
> core/src/main/java/org/apache/oozie/action/hadoop/MapReduceActionExecutor.java
>  55c937249ac35e14bd63dd0c1fa962ea636afa3a 
>   core/src/main/java/org/apache/oozie/action/hadoop/OozieJobInfo.java 
> 581d3b3219eff882f961c8e495611e873c6a5a52 
>   core/src/main/java/org/apache/oozie/action/hadoop/PigActionExecutor.java 
> 8b2dc16db80ad602911f1f64e60cfad41b198922 
>   
> core/src/main/java/org/apache/oozie/action/hadoop/ScriptLanguageActionExecutor.java
>  92e149d5cac6509893c8507b776263a38312a53b 
>   core/src/main/java/org/apache/oozie/action/hadoop/ShellActionExecutor.java 
> b9ffa7ab7ec87db8a76a3653c74974d296c8cd17 
>   core/src/main/java/org/apache/oozie/action/hadoop/SparkActionExecutor.java 
> 1a3197abb52f80b545dcc8634a8cac7a281a9eac 
>   core/src/main/java/org/apache/oozie/action/hadoop/SqoopActionExecutor.java 
> 22e28746a9a39ae6c1e22326cc4f8f759d820b20 
>   core/src/main/java/org/apache/oozie/client/rest/BulkResponseImpl.java 
> fb021bd022d199f93fc5674843514abf45ae12a5 
>   core/src/main/java/org/apache/oozie/command/XCommand.java 
> bdf13f65ea25e8e0f5125e2798568a2c03fffe81 
>   core/src/main/java/org/apache/oozie/command/wf/SubmitHttpXCommand.java 
> d

Re: Review Request 54705: OOZIE-1770: Create Oozie Application Master for YARN

[Peter Cseh]

> On May 11, 2017, 12:41 a.m., Robert Kanter wrote:
> > core/src/main/java/org/apache/oozie/action/hadoop/JavaActionExecutor.java
> > Lines 1066 (patched)
> > 
> >
> > It's my understanding that the priority doesn't do anything in Yarn 
> > today.  YARNRunner currently uses 0 for the MR AM, so let's also use 0.
> > 
> > https://github.com/apache/hadoop/blob/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/main/java/org/apache/hadoop/mapred/YARNRunner.java#L130
> > 
> > I think we can get rid of the TODO, and replace it with a comment like 
> > "MR AM uses 0" or something like that so we remember why we set 0.  Should 
> > probably make it a constant too.
> 
> Rohini Palaniswamy wrote:
> mapred.job.priority is used by some users. Oozie AM needs to support 
> that. Please file a new jira for it

OOZIE-2892


> On May 11, 2017, 12:41 a.m., Robert Kanter wrote:
> > core/src/main/java/org/apache/oozie/action/hadoop/JavaActionExecutor.java
> > Lines 1069 (patched)
> > 
> >
> > Please file a followup JIRA for this.
> 
> Rohini Palaniswamy wrote:
> This is important and needs to be fixed. 95% of our users configure a 
> different queue using oozie.launcher settings.

OOZIE-2896


- Peter


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/54705/#review174589
---


On May 10, 2017, 10:46 p.m., Peter Cseh wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/54705/
> ---
> 
> (Updated May 10, 2017, 10:46 p.m.)
> 
> 
> Review request for oozie, Purshotam Shah, Robert Kanter, and Rohini 
> Palaniswamy.
> 
> 
> Bugs: OOZIE-1770
> https://issues.apache.org/jira/browse/OOZIE-1770
> 
> 
> Repository: oozie-git
> 
> 
> Description
> ---
> 
> Replacing the launcher MR job with a launcher AM
> 
> 
> Diffs
> -
> 
>   client/pom.xml f99366c90887a5a45f3942291c90e1cdbfd84b3f 
>   client/src/main/java/org/apache/oozie/cli/OozieCLI.java 
> 38fb84e8a8918aee6ae57d19a29cdf181aba1955 
>   client/src/main/java/org/apache/oozie/client/XOozieClient.java 
> 8538ec7c742cc013f7c18eaff3ecf61c2835cc8a 
>   core/pom.xml 14aa03483e107db326628ba627cc3ee9f5fd3f3d 
>   core/src/main/java/org/apache/oozie/WorkflowJobBean.java 
> 55d79a5d5e5e4efab9eb832e9341a16bcf8959cc 
>   core/src/main/java/org/apache/oozie/action/ActionExecutor.java 
> 1d6456b236bf9e27c0f55d9f9e3b4bd94517c797 
>   core/src/main/java/org/apache/oozie/action/hadoop/Credentials.java 
> eadb47b07d157f45f086517ab41c4b9826f003f0 
>   core/src/main/java/org/apache/oozie/action/hadoop/CredentialsProvider.java 
> 6fe22fbd2fca5d873d58ece60f20d0f131371475 
>   
> core/src/main/java/org/apache/oozie/action/hadoop/CredentialsProviderFactory.java
>  PRE-CREATION 
>   core/src/main/java/org/apache/oozie/action/hadoop/DistcpActionExecutor.java 
> 78cd257a836707c1efeac13566d6285d42970545 
>   core/src/main/java/org/apache/oozie/action/hadoop/FsActionExecutor.java 
> 27654741101d2056b25626fbbe345183eafd612d 
>   core/src/main/java/org/apache/oozie/action/hadoop/FsELFunctions.java 
> 801bfe6eb212b549888f0e9fb7df7b65bcf59393 
>   core/src/main/java/org/apache/oozie/action/hadoop/HCatCredentialHelper.java 
> 36ad1df727e869b8051481ad969771e91bc6ae07 
>   core/src/main/java/org/apache/oozie/action/hadoop/HCatCredentials.java 
> d7689a924059e553abaeed374eed4072933292cb 
>   core/src/main/java/org/apache/oozie/action/hadoop/HadoopELFunctions.java 
> ad2a71d641caa71efc098ad00bfbab3cec4d9506 
>   core/src/main/java/org/apache/oozie/action/hadoop/HbaseCredentials.java 
> 307f565786ad77f5a35b5790bd5b052b04c6eac8 
>   core/src/main/java/org/apache/oozie/action/hadoop/Hive2ActionExecutor.java 
> 2aed936af75da38c433be5d202bdc01c2f612c76 
>   core/src/main/java/org/apache/oozie/action/hadoop/Hive2Credentials.java 
> bd2f1f768c5b04bef4a33613baede9202144c747 
>   core/src/main/java/org/apache/oozie/action/hadoop/HiveActionExecutor.java 
> 8e2453ef4f78f4a5f1f95b1ca2793f5d7e07ad81 
>   core/src/main/java/org/apache/oozie/action/hadoop/JavaActionExecutor.java 
> d60a5c7b996a4085f9bc77d82f12ad6925280b1a 
>   core/src/main/java/org/apache/oozie/action/hadoop/LauncherMapperHelper.java 
> 72ed2f150be73f0a5e8b71b0fb6cc3594dc23078 
>   
> core/src/main/java/org/apache/oozie/action/hadoop/MapReduceActionExecutor.java
>  55c937249ac35e14bd63dd0c1fa962ea636afa3a 
>   core/src/main/java/org/apache/oozie/action/hadoop/OozieJobInfo.java 
> 581d3b3219eff882f961c8e495611e873c6a5a52 
>   core/src/main/java/org/apache/oozie/action/hadoop/PigActionExecutor.java 
> 8b2dc16db80ad6029

Re: Review Request 54705: OOZIE-1770: Create Oozie Application Master for YARN

[Peter Cseh]

> On May 11, 2017, 12:41 a.m., Robert Kanter wrote:
> > core/src/main/java/org/apache/oozie/action/hadoop/MapReduceActionExecutor.java
> > Line 102 (original), 119 (patched)
> > 
> >
> > I think we still need this here, but it got me thinking - do we need to 
> > Oozie AM to cancel the tokens before it exits?
> 
> Rohini Palaniswamy wrote:
> Yes. We do need to have the delegation tokens cancelled. AM itself should 
> not be cancelling the tokens. It has tell the RM via 
> ApplicationSubmissionContext.setCancelTokensWhenComplete and RM has to do it.

If we set appContext.setCancelTokensWhenComplete(true) in 
createAppSubmissionContext, it will cancel the tokens for MR action because the 
LauncherAM exits early in that case. I think it will cause issues there.
Created  OOZIE-2904 to address this.


- Peter


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/54705/#review174589
---


On May 10, 2017, 10:46 p.m., Peter Cseh wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/54705/
> ---
> 
> (Updated May 10, 2017, 10:46 p.m.)
> 
> 
> Review request for oozie, Purshotam Shah, Robert Kanter, and Rohini 
> Palaniswamy.
> 
> 
> Bugs: OOZIE-1770
> https://issues.apache.org/jira/browse/OOZIE-1770
> 
> 
> Repository: oozie-git
> 
> 
> Description
> ---
> 
> Replacing the launcher MR job with a launcher AM
> 
> 
> Diffs
> -
> 
>   client/pom.xml f99366c90887a5a45f3942291c90e1cdbfd84b3f 
>   client/src/main/java/org/apache/oozie/cli/OozieCLI.java 
> 38fb84e8a8918aee6ae57d19a29cdf181aba1955 
>   client/src/main/java/org/apache/oozie/client/XOozieClient.java 
> 8538ec7c742cc013f7c18eaff3ecf61c2835cc8a 
>   core/pom.xml 14aa03483e107db326628ba627cc3ee9f5fd3f3d 
>   core/src/main/java/org/apache/oozie/WorkflowJobBean.java 
> 55d79a5d5e5e4efab9eb832e9341a16bcf8959cc 
>   core/src/main/java/org/apache/oozie/action/ActionExecutor.java 
> 1d6456b236bf9e27c0f55d9f9e3b4bd94517c797 
>   core/src/main/java/org/apache/oozie/action/hadoop/Credentials.java 
> eadb47b07d157f45f086517ab41c4b9826f003f0 
>   core/src/main/java/org/apache/oozie/action/hadoop/CredentialsProvider.java 
> 6fe22fbd2fca5d873d58ece60f20d0f131371475 
>   
> core/src/main/java/org/apache/oozie/action/hadoop/CredentialsProviderFactory.java
>  PRE-CREATION 
>   core/src/main/java/org/apache/oozie/action/hadoop/DistcpActionExecutor.java 
> 78cd257a836707c1efeac13566d6285d42970545 
>   core/src/main/java/org/apache/oozie/action/hadoop/FsActionExecutor.java 
> 27654741101d2056b25626fbbe345183eafd612d 
>   core/src/main/java/org/apache/oozie/action/hadoop/FsELFunctions.java 
> 801bfe6eb212b549888f0e9fb7df7b65bcf59393 
>   core/src/main/java/org/apache/oozie/action/hadoop/HCatCredentialHelper.java 
> 36ad1df727e869b8051481ad969771e91bc6ae07 
>   core/src/main/java/org/apache/oozie/action/hadoop/HCatCredentials.java 
> d7689a924059e553abaeed374eed4072933292cb 
>   core/src/main/java/org/apache/oozie/action/hadoop/HadoopELFunctions.java 
> ad2a71d641caa71efc098ad00bfbab3cec4d9506 
>   core/src/main/java/org/apache/oozie/action/hadoop/HbaseCredentials.java 
> 307f565786ad77f5a35b5790bd5b052b04c6eac8 
>   core/src/main/java/org/apache/oozie/action/hadoop/Hive2ActionExecutor.java 
> 2aed936af75da38c433be5d202bdc01c2f612c76 
>   core/src/main/java/org/apache/oozie/action/hadoop/Hive2Credentials.java 
> bd2f1f768c5b04bef4a33613baede9202144c747 
>   core/src/main/java/org/apache/oozie/action/hadoop/HiveActionExecutor.java 
> 8e2453ef4f78f4a5f1f95b1ca2793f5d7e07ad81 
>   core/src/main/java/org/apache/oozie/action/hadoop/JavaActionExecutor.java 
> d60a5c7b996a4085f9bc77d82f12ad6925280b1a 
>   core/src/main/java/org/apache/oozie/action/hadoop/LauncherMapperHelper.java 
> 72ed2f150be73f0a5e8b71b0fb6cc3594dc23078 
>   
> core/src/main/java/org/apache/oozie/action/hadoop/MapReduceActionExecutor.java
>  55c937249ac35e14bd63dd0c1fa962ea636afa3a 
>   core/src/main/java/org/apache/oozie/action/hadoop/OozieJobInfo.java 
> 581d3b3219eff882f961c8e495611e873c6a5a52 
>   core/src/main/java/org/apache/oozie/action/hadoop/PigActionExecutor.java 
> 8b2dc16db80ad602911f1f64e60cfad41b198922 
>   
> core/src/main/java/org/apache/oozie/action/hadoop/ScriptLanguageActionExecutor.java
>  92e149d5cac6509893c8507b776263a38312a53b 
>   core/src/main/java/org/apache/oozie/action/hadoop/ShellActionExecutor.java 
> b9ffa7ab7ec87db8a76a3653c74974d296c8cd17 
>   core/src/main/java/org/apache/oozie/action/hadoop/SparkActionExecutor.java 
> 1a3197abb52f80b545dcc8634a8cac7a281a9eac 
>   core/src/main/java/org/apache/oozie/action/hadoop/SqoopActionExecutor.java 
> 22e28746a9a39ae6c1e22326cc4f8f759d

Re: Review Request 54705: OOZIE-1770: Create Oozie Application Master for YARN

[Peter Cseh]

- Peter


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/54705/#review174589
---


On May 10, 2017, 10:46 p.m., Peter Cseh wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/54705/
> ---
> 
> (Updated May 10, 2017, 10:46 p.m.)
> 
> 
> Review request for oozie, Purshotam Shah, Robert Kanter, and Rohini 
> Palaniswamy.
> 
> 
> Bugs: OOZIE-1770
> https://issues.apache.org/jira/browse/OOZIE-1770
> 
> 
> Repository: oozie-git
> 
> 
> Description
> ---
> 
> Replacing the launcher MR job with a launcher AM
> 
> 
> Diffs
> -
> 
>   client/pom.xml f99366c90887a5a45f3942291c90e1cdbfd84b3f 
>   client/src/main/java/org/apache/oozie/cli/OozieCLI.java 
> 38fb84e8a8918aee6ae57d19a29cdf181aba1955 
>   client/src/main/java/org/apache/oozie/client/XOozieClient.java 
> 8538ec7c742cc013f7c18eaff3ecf61c2835cc8a 
>   core/pom.xml 14aa03483e107db326628ba627cc3ee9f5fd3f3d 
>   core/src/main/java/org/apache/oozie/WorkflowJobBean.java 
> 55d79a5d5e5e4efab9eb832e9341a16bcf8959cc 
>   core/src/main/java/org/apache/oozie/action/ActionExecutor.java 
> 1d6456b236bf9e27c0f55d9f9e3b4bd94517c797 
>   core/src/main/java/org/apache/oozie/action/hadoop/Credentials.java 
> eadb47b07d157f45f086517ab41c4b9826f003f0 
>   core/src/main/java/org/apache/oozie/action/hadoop/CredentialsProvider.java 
> 6fe22fbd2fca5d873d58ece60f20d0f131371475 
>   
> core/src/main/java/org/apache/oozie/action/hadoop/CredentialsProviderFactory.java
>  PRE-CREATION 
>   core/src/main/java/org/apache/oozie/action/hadoop/DistcpActionExecutor.java 
> 78cd257a836707c1efeac13566d6285d42970545 
>   core/src/main/java/org/apache/oozie/action/hadoop/FsActionExecutor.java 
> 27654741101d2056b25626fbbe345183eafd612d 
>   core/src/main/java/org/apache/oozie/action/hadoop/FsELFunctions.java 
> 801bfe6eb212b549888f0e9fb7df7b65bcf59393 
>   core/src/main/java/org/apache/oozie/action/hadoop/HCatCredentialHelper.java 
> 36ad1df727e869b8051481ad969771e91bc6ae07 
>   core/src/main/java/org/apache/oozie/action/hadoop/HCatCredentials.java 
> d7689a924059e553abaeed374eed4072933292cb 
>   core/src/main/java/org/apache/oozie/action/hadoop/HadoopELFunctions.java 
> ad2a71d641caa71efc098ad00bfbab3cec4d9506 
>   core/src/main/java/org/apache/oozie/action/hadoop/HbaseCredentials.java 
> 307f565786ad77f5a35b5790bd5b052b04c6eac8 
>   core/src/main/java/org/apache/oozie/action/hadoop/Hive2ActionExecutor.java 
> 2aed936af75da38c433be5d202bdc01c2f612c76 
>   core/src/main/java/org/apache/oozie/action/hadoop/Hive2Credentials.java 
> bd2f1f768c5b04bef4a33613baede9202144c747 
>   core/src/main/java/org/apache/oozie/action/hadoop/HiveActionExecutor.java 
> 8e2453ef4f78f4a5f1f95b1ca2793f5d7e07ad81 
>   core/src/main/java/org/apache/oozie/action/hadoop/JavaActionExecutor.java 
> d60a5c7b996a4085f9bc77d82f12ad6925280b1a 
>   core/src/main/java/org/apache/oozie/action/hadoop/LauncherMapperHelper.java 
> 72ed2f150be73f0a5e8b71b0fb6cc3594dc23078 
>   
> core/src/main/java/org/apache/oozie/action/hadoop/MapReduceActionExecutor.java
>  55c937249ac35e14bd63dd0c1fa962ea636afa3a 
>   core/src/main/java/org/apache/oozie/action/hadoop/OozieJobInfo.java 
> 581d3b3219eff882f961c8e495611e873c6a5a52 
>   core/src/main/java/org/apache/oozie/action/hadoop/PigActionExecutor.java 
> 8b2dc16db80ad602911f1f64e60cfad41b198922 
>   
> core/src/main/java/org/apache/oozie/action/hadoop/ScriptLanguageActionExecutor.java
>  92e149d5cac6509893c8507b776263a38312a53b 
>   core/src/main/java/org/apache/oozie/action/hadoop/ShellActionExecutor.java 
> b9ffa7ab7ec87db8a76a3653c74974d296c8cd17 
>   core/src/main/java/org/apache/oozie/action/hadoop/SparkActionExecutor.java 
> 1a3197abb52f80b545dcc8634a8cac7a281a9eac 
>   core/src/main/java/org/apache/oozie/action/hadoop/SqoopActionExecutor.java 
> 22e28746a9a39ae6c1e22326cc4f8f759d820b20 
>   core/src/main/java/org/apache/oozie/client/rest/BulkResponseImpl.java 
> fb021bd022d199f93fc5674843514abf45ae12a5 
>   core/src/main/java/org/apache/oozie/command/XCommand.java 
> bdf13f65ea25e8e0f5125e2798568a2c03fffe81 
>   core/src/main/java/org/apache/oozie/command/wf/SubmitHttpXCommand.java 
> d2a2742ae418070a1cd0a889f14194224bc944a6 
>   core/src/main/java/org/apache/oozie/command/wf/SubmitMRXCommand.java 
> cc61d3de813c75c7f1cf4d405fc043253e3ca4ea 
>   
> core/src/main/java/org/apache/oozie/command/wf/SubmitScriptLanguageXCommand.java
>  9d413059e99bc676feb63e02a2f704048eb4a355 
>   core/src/main/java/org/apache/oozie/command/wf/SubmitSqoopXCommand.java 
> 51b739e21c802e09f4110b740138f4ff5c341fe4 
>   
> core/src/main/java/org/apache/oozie/coord/input/dependency/AbstractCoordInputDependency.java
>  ace120d10e3546201aa4145df56abf7933fb24b8 
>   
> core/src/main/

Re: Review Request 54705: OOZIE-1770: Create Oozie Application Master for YARN

[Peter Cseh]

> On May 11, 2017, 6:14 a.m., Robert Kanter wrote:
> > core/src/main/java/org/apache/oozie/service/HadoopAccessorService.java
> > Lines 528-532 (original), 535-540 (patched)
> > 
> >
> > There's no group

It mentioned user/group originally. UserGroupInformatio can have an implicit 
support for groups also. 
https://github.com/apache/oozie/blob/master/core/src/main/java/org/apache/oozie/service/HadoopAccessorService.java#L495


- Peter


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/54705/#review174591
---


On May 10, 2017, 10:46 p.m., Peter Cseh wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/54705/
> ---
> 
> (Updated May 10, 2017, 10:46 p.m.)
> 
> 
> Review request for oozie, Purshotam Shah, Robert Kanter, and Rohini 
> Palaniswamy.
> 
> 
> Bugs: OOZIE-1770
> https://issues.apache.org/jira/browse/OOZIE-1770
> 
> 
> Repository: oozie-git
> 
> 
> Description
> ---
> 
> Replacing the launcher MR job with a launcher AM
> 
> 
> Diffs
> -
> 
>   client/pom.xml f99366c90887a5a45f3942291c90e1cdbfd84b3f 
>   client/src/main/java/org/apache/oozie/cli/OozieCLI.java 
> 38fb84e8a8918aee6ae57d19a29cdf181aba1955 
>   client/src/main/java/org/apache/oozie/client/XOozieClient.java 
> 8538ec7c742cc013f7c18eaff3ecf61c2835cc8a 
>   core/pom.xml 14aa03483e107db326628ba627cc3ee9f5fd3f3d 
>   core/src/main/java/org/apache/oozie/WorkflowJobBean.java 
> 55d79a5d5e5e4efab9eb832e9341a16bcf8959cc 
>   core/src/main/java/org/apache/oozie/action/ActionExecutor.java 
> 1d6456b236bf9e27c0f55d9f9e3b4bd94517c797 
>   core/src/main/java/org/apache/oozie/action/hadoop/Credentials.java 
> eadb47b07d157f45f086517ab41c4b9826f003f0 
>   core/src/main/java/org/apache/oozie/action/hadoop/CredentialsProvider.java 
> 6fe22fbd2fca5d873d58ece60f20d0f131371475 
>   
> core/src/main/java/org/apache/oozie/action/hadoop/CredentialsProviderFactory.java
>  PRE-CREATION 
>   core/src/main/java/org/apache/oozie/action/hadoop/DistcpActionExecutor.java 
> 78cd257a836707c1efeac13566d6285d42970545 
>   core/src/main/java/org/apache/oozie/action/hadoop/FsActionExecutor.java 
> 27654741101d2056b25626fbbe345183eafd612d 
>   core/src/main/java/org/apache/oozie/action/hadoop/FsELFunctions.java 
> 801bfe6eb212b549888f0e9fb7df7b65bcf59393 
>   core/src/main/java/org/apache/oozie/action/hadoop/HCatCredentialHelper.java 
> 36ad1df727e869b8051481ad969771e91bc6ae07 
>   core/src/main/java/org/apache/oozie/action/hadoop/HCatCredentials.java 
> d7689a924059e553abaeed374eed4072933292cb 
>   core/src/main/java/org/apache/oozie/action/hadoop/HadoopELFunctions.java 
> ad2a71d641caa71efc098ad00bfbab3cec4d9506 
>   core/src/main/java/org/apache/oozie/action/hadoop/HbaseCredentials.java 
> 307f565786ad77f5a35b5790bd5b052b04c6eac8 
>   core/src/main/java/org/apache/oozie/action/hadoop/Hive2ActionExecutor.java 
> 2aed936af75da38c433be5d202bdc01c2f612c76 
>   core/src/main/java/org/apache/oozie/action/hadoop/Hive2Credentials.java 
> bd2f1f768c5b04bef4a33613baede9202144c747 
>   core/src/main/java/org/apache/oozie/action/hadoop/HiveActionExecutor.java 
> 8e2453ef4f78f4a5f1f95b1ca2793f5d7e07ad81 
>   core/src/main/java/org/apache/oozie/action/hadoop/JavaActionExecutor.java 
> d60a5c7b996a4085f9bc77d82f12ad6925280b1a 
>   core/src/main/java/org/apache/oozie/action/hadoop/LauncherMapperHelper.java 
> 72ed2f150be73f0a5e8b71b0fb6cc3594dc23078 
>   
> core/src/main/java/org/apache/oozie/action/hadoop/MapReduceActionExecutor.java
>  55c937249ac35e14bd63dd0c1fa962ea636afa3a 
>   core/src/main/java/org/apache/oozie/action/hadoop/OozieJobInfo.java 
> 581d3b3219eff882f961c8e495611e873c6a5a52 
>   core/src/main/java/org/apache/oozie/action/hadoop/PigActionExecutor.java 
> 8b2dc16db80ad602911f1f64e60cfad41b198922 
>   
> core/src/main/java/org/apache/oozie/action/hadoop/ScriptLanguageActionExecutor.java
>  92e149d5cac6509893c8507b776263a38312a53b 
>   core/src/main/java/org/apache/oozie/action/hadoop/ShellActionExecutor.java 
> b9ffa7ab7ec87db8a76a3653c74974d296c8cd17 
>   core/src/main/java/org/apache/oozie/action/hadoop/SparkActionExecutor.java 
> 1a3197abb52f80b545dcc8634a8cac7a281a9eac 
>   core/src/main/java/org/apache/oozie/action/hadoop/SqoopActionExecutor.java 
> 22e28746a9a39ae6c1e22326cc4f8f759d820b20 
>   core/src/main/java/org/apache/oozie/client/rest/BulkResponseImpl.java 
> fb021bd022d199f93fc5674843514abf45ae12a5 
>   core/src/main/java/org/apache/oozie/command/XCommand.java 
> bdf13f65ea25e8e0f5125e2798568a2c03fffe81 
>   core/src/main/java/org/apache/oozie/command/wf/SubmitHttpXCommand.java 
> d2a2742ae418070a1cd0a889f14194224bc944a6 
>   core/src/main/java/org/

Re: Review Request 54705: OOZIE-1770: Create Oozie Application Master for YARN

[Peter Cseh]

> On May 12, 2017, 6:35 p.m., Robert Kanter wrote:
> > sharelib/hive/src/main/java/org/apache/oozie/action/hadoop/HiveMain.java
> > Lines 312-318 (original)
> > 
> >
> > Don't we need this to capture System.exit calls?

It's handled in LauncherAM 
https://github.com/apache/oozie/blob/oya/sharelib/oozie/src/main/java/org/apache/oozie/action/hadoop/LauncherAM.java#L387


- Peter


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/54705/#review174833
---


On May 10, 2017, 10:46 p.m., Peter Cseh wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/54705/
> ---
> 
> (Updated May 10, 2017, 10:46 p.m.)
> 
> 
> Review request for oozie, Purshotam Shah, Robert Kanter, and Rohini 
> Palaniswamy.
> 
> 
> Bugs: OOZIE-1770
> https://issues.apache.org/jira/browse/OOZIE-1770
> 
> 
> Repository: oozie-git
> 
> 
> Description
> ---
> 
> Replacing the launcher MR job with a launcher AM
> 
> 
> Diffs
> -
> 
>   client/pom.xml f99366c90887a5a45f3942291c90e1cdbfd84b3f 
>   client/src/main/java/org/apache/oozie/cli/OozieCLI.java 
> 38fb84e8a8918aee6ae57d19a29cdf181aba1955 
>   client/src/main/java/org/apache/oozie/client/XOozieClient.java 
> 8538ec7c742cc013f7c18eaff3ecf61c2835cc8a 
>   core/pom.xml 14aa03483e107db326628ba627cc3ee9f5fd3f3d 
>   core/src/main/java/org/apache/oozie/WorkflowJobBean.java 
> 55d79a5d5e5e4efab9eb832e9341a16bcf8959cc 
>   core/src/main/java/org/apache/oozie/action/ActionExecutor.java 
> 1d6456b236bf9e27c0f55d9f9e3b4bd94517c797 
>   core/src/main/java/org/apache/oozie/action/hadoop/Credentials.java 
> eadb47b07d157f45f086517ab41c4b9826f003f0 
>   core/src/main/java/org/apache/oozie/action/hadoop/CredentialsProvider.java 
> 6fe22fbd2fca5d873d58ece60f20d0f131371475 
>   
> core/src/main/java/org/apache/oozie/action/hadoop/CredentialsProviderFactory.java
>  PRE-CREATION 
>   core/src/main/java/org/apache/oozie/action/hadoop/DistcpActionExecutor.java 
> 78cd257a836707c1efeac13566d6285d42970545 
>   core/src/main/java/org/apache/oozie/action/hadoop/FsActionExecutor.java 
> 27654741101d2056b25626fbbe345183eafd612d 
>   core/src/main/java/org/apache/oozie/action/hadoop/FsELFunctions.java 
> 801bfe6eb212b549888f0e9fb7df7b65bcf59393 
>   core/src/main/java/org/apache/oozie/action/hadoop/HCatCredentialHelper.java 
> 36ad1df727e869b8051481ad969771e91bc6ae07 
>   core/src/main/java/org/apache/oozie/action/hadoop/HCatCredentials.java 
> d7689a924059e553abaeed374eed4072933292cb 
>   core/src/main/java/org/apache/oozie/action/hadoop/HadoopELFunctions.java 
> ad2a71d641caa71efc098ad00bfbab3cec4d9506 
>   core/src/main/java/org/apache/oozie/action/hadoop/HbaseCredentials.java 
> 307f565786ad77f5a35b5790bd5b052b04c6eac8 
>   core/src/main/java/org/apache/oozie/action/hadoop/Hive2ActionExecutor.java 
> 2aed936af75da38c433be5d202bdc01c2f612c76 
>   core/src/main/java/org/apache/oozie/action/hadoop/Hive2Credentials.java 
> bd2f1f768c5b04bef4a33613baede9202144c747 
>   core/src/main/java/org/apache/oozie/action/hadoop/HiveActionExecutor.java 
> 8e2453ef4f78f4a5f1f95b1ca2793f5d7e07ad81 
>   core/src/main/java/org/apache/oozie/action/hadoop/JavaActionExecutor.java 
> d60a5c7b996a4085f9bc77d82f12ad6925280b1a 
>   core/src/main/java/org/apache/oozie/action/hadoop/LauncherMapperHelper.java 
> 72ed2f150be73f0a5e8b71b0fb6cc3594dc23078 
>   
> core/src/main/java/org/apache/oozie/action/hadoop/MapReduceActionExecutor.java
>  55c937249ac35e14bd63dd0c1fa962ea636afa3a 
>   core/src/main/java/org/apache/oozie/action/hadoop/OozieJobInfo.java 
> 581d3b3219eff882f961c8e495611e873c6a5a52 
>   core/src/main/java/org/apache/oozie/action/hadoop/PigActionExecutor.java 
> 8b2dc16db80ad602911f1f64e60cfad41b198922 
>   
> core/src/main/java/org/apache/oozie/action/hadoop/ScriptLanguageActionExecutor.java
>  92e149d5cac6509893c8507b776263a38312a53b 
>   core/src/main/java/org/apache/oozie/action/hadoop/ShellActionExecutor.java 
> b9ffa7ab7ec87db8a76a3653c74974d296c8cd17 
>   core/src/main/java/org/apache/oozie/action/hadoop/SparkActionExecutor.java 
> 1a3197abb52f80b545dcc8634a8cac7a281a9eac 
>   core/src/main/java/org/apache/oozie/action/hadoop/SqoopActionExecutor.java 
> 22e28746a9a39ae6c1e22326cc4f8f759d820b20 
>   core/src/main/java/org/apache/oozie/client/rest/BulkResponseImpl.java 
> fb021bd022d199f93fc5674843514abf45ae12a5 
>   core/src/main/java/org/apache/oozie/command/XCommand.java 
> bdf13f65ea25e8e0f5125e2798568a2c03fffe81 
>   core/src/main/java/org/apache/oozie/command/wf/SubmitHttpXCommand.java 
> d2a2742ae418070a1cd0a889f14194224bc944a6 
>   core/src/main/java/org/apache/oozie/command/wf/SubmitMRXCommand.java 
> cc61d3de81

Re: Review Request 54705: OOZIE-1770: Create Oozie Application Master for YARN

[Peter Cseh]

> On May 12, 2017, 8:27 p.m., Robert Kanter wrote:
> > sharelib/oozie/src/main/java/org/apache/oozie/action/hadoop/LauncherAM.java
> > Lines 284 (patched)
> > 
> >
> > What should we do about this?  I don't think there's really a reason 
> > the user should play with this, so I'm thinking we should leave this as 
> > hardcoded and remove the TODO.
> 
> Rohini Palaniswamy wrote:
> For cases like this, please use the value of mapreduce settings if it is 
> set instead of coming up with a new setting name or trying to hard code some 
> value. In our clusters we use a higher value to put less load on RM.

OOZIE-2894


- Peter


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/54705/#review174834
---


On May 10, 2017, 10:46 p.m., Peter Cseh wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/54705/
> ---
> 
> (Updated May 10, 2017, 10:46 p.m.)
> 
> 
> Review request for oozie, Purshotam Shah, Robert Kanter, and Rohini 
> Palaniswamy.
> 
> 
> Bugs: OOZIE-1770
> https://issues.apache.org/jira/browse/OOZIE-1770
> 
> 
> Repository: oozie-git
> 
> 
> Description
> ---
> 
> Replacing the launcher MR job with a launcher AM
> 
> 
> Diffs
> -
> 
>   client/pom.xml f99366c90887a5a45f3942291c90e1cdbfd84b3f 
>   client/src/main/java/org/apache/oozie/cli/OozieCLI.java 
> 38fb84e8a8918aee6ae57d19a29cdf181aba1955 
>   client/src/main/java/org/apache/oozie/client/XOozieClient.java 
> 8538ec7c742cc013f7c18eaff3ecf61c2835cc8a 
>   core/pom.xml 14aa03483e107db326628ba627cc3ee9f5fd3f3d 
>   core/src/main/java/org/apache/oozie/WorkflowJobBean.java 
> 55d79a5d5e5e4efab9eb832e9341a16bcf8959cc 
>   core/src/main/java/org/apache/oozie/action/ActionExecutor.java 
> 1d6456b236bf9e27c0f55d9f9e3b4bd94517c797 
>   core/src/main/java/org/apache/oozie/action/hadoop/Credentials.java 
> eadb47b07d157f45f086517ab41c4b9826f003f0 
>   core/src/main/java/org/apache/oozie/action/hadoop/CredentialsProvider.java 
> 6fe22fbd2fca5d873d58ece60f20d0f131371475 
>   
> core/src/main/java/org/apache/oozie/action/hadoop/CredentialsProviderFactory.java
>  PRE-CREATION 
>   core/src/main/java/org/apache/oozie/action/hadoop/DistcpActionExecutor.java 
> 78cd257a836707c1efeac13566d6285d42970545 
>   core/src/main/java/org/apache/oozie/action/hadoop/FsActionExecutor.java 
> 27654741101d2056b25626fbbe345183eafd612d 
>   core/src/main/java/org/apache/oozie/action/hadoop/FsELFunctions.java 
> 801bfe6eb212b549888f0e9fb7df7b65bcf59393 
>   core/src/main/java/org/apache/oozie/action/hadoop/HCatCredentialHelper.java 
> 36ad1df727e869b8051481ad969771e91bc6ae07 
>   core/src/main/java/org/apache/oozie/action/hadoop/HCatCredentials.java 
> d7689a924059e553abaeed374eed4072933292cb 
>   core/src/main/java/org/apache/oozie/action/hadoop/HadoopELFunctions.java 
> ad2a71d641caa71efc098ad00bfbab3cec4d9506 
>   core/src/main/java/org/apache/oozie/action/hadoop/HbaseCredentials.java 
> 307f565786ad77f5a35b5790bd5b052b04c6eac8 
>   core/src/main/java/org/apache/oozie/action/hadoop/Hive2ActionExecutor.java 
> 2aed936af75da38c433be5d202bdc01c2f612c76 
>   core/src/main/java/org/apache/oozie/action/hadoop/Hive2Credentials.java 
> bd2f1f768c5b04bef4a33613baede9202144c747 
>   core/src/main/java/org/apache/oozie/action/hadoop/HiveActionExecutor.java 
> 8e2453ef4f78f4a5f1f95b1ca2793f5d7e07ad81 
>   core/src/main/java/org/apache/oozie/action/hadoop/JavaActionExecutor.java 
> d60a5c7b996a4085f9bc77d82f12ad6925280b1a 
>   core/src/main/java/org/apache/oozie/action/hadoop/LauncherMapperHelper.java 
> 72ed2f150be73f0a5e8b71b0fb6cc3594dc23078 
>   
> core/src/main/java/org/apache/oozie/action/hadoop/MapReduceActionExecutor.java
>  55c937249ac35e14bd63dd0c1fa962ea636afa3a 
>   core/src/main/java/org/apache/oozie/action/hadoop/OozieJobInfo.java 
> 581d3b3219eff882f961c8e495611e873c6a5a52 
>   core/src/main/java/org/apache/oozie/action/hadoop/PigActionExecutor.java 
> 8b2dc16db80ad602911f1f64e60cfad41b198922 
>   
> core/src/main/java/org/apache/oozie/action/hadoop/ScriptLanguageActionExecutor.java
>  92e149d5cac6509893c8507b776263a38312a53b 
>   core/src/main/java/org/apache/oozie/action/hadoop/ShellActionExecutor.java 
> b9ffa7ab7ec87db8a76a3653c74974d296c8cd17 
>   core/src/main/java/org/apache/oozie/action/hadoop/SparkActionExecutor.java 
> 1a3197abb52f80b545dcc8634a8cac7a281a9eac 
>   core/src/main/java/org/apache/oozie/action/hadoop/SqoopActionExecutor.java 
> 22e28746a9a39ae6c1e22326cc4f8f759d820b20 
>   core/src/main/java/org/apache/oozie/client/rest/BulkResponseImpl.java 
> fb021bd022d199f93fc5674843514abf45ae12a5 
>   core/src/main/java/org/apache/oozie/command/XCommand.ja

Re: Review Request 54705: OOZIE-1770: Create Oozie Application Master for YARN

[Peter Cseh]

> On May 12, 2017, 8:27 p.m., Robert Kanter wrote:
> > sharelib/oozie/src/test/java/org/apache/oozie/action/hadoop/TestLauncherAM.java
> > Lines 379 (patched)
> > 
> >
> > Please file a followup JIRA for this if we're not going to figure it 
> > out now.

https://issues.apache.org/jira/browse/OOZIE-2895


- Peter


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/54705/#review174834
---


On May 10, 2017, 10:46 p.m., Peter Cseh wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/54705/
> ---
> 
> (Updated May 10, 2017, 10:46 p.m.)
> 
> 
> Review request for oozie, Purshotam Shah, Robert Kanter, and Rohini 
> Palaniswamy.
> 
> 
> Bugs: OOZIE-1770
> https://issues.apache.org/jira/browse/OOZIE-1770
> 
> 
> Repository: oozie-git
> 
> 
> Description
> ---
> 
> Replacing the launcher MR job with a launcher AM
> 
> 
> Diffs
> -
> 
>   client/pom.xml f99366c90887a5a45f3942291c90e1cdbfd84b3f 
>   client/src/main/java/org/apache/oozie/cli/OozieCLI.java 
> 38fb84e8a8918aee6ae57d19a29cdf181aba1955 
>   client/src/main/java/org/apache/oozie/client/XOozieClient.java 
> 8538ec7c742cc013f7c18eaff3ecf61c2835cc8a 
>   core/pom.xml 14aa03483e107db326628ba627cc3ee9f5fd3f3d 
>   core/src/main/java/org/apache/oozie/WorkflowJobBean.java 
> 55d79a5d5e5e4efab9eb832e9341a16bcf8959cc 
>   core/src/main/java/org/apache/oozie/action/ActionExecutor.java 
> 1d6456b236bf9e27c0f55d9f9e3b4bd94517c797 
>   core/src/main/java/org/apache/oozie/action/hadoop/Credentials.java 
> eadb47b07d157f45f086517ab41c4b9826f003f0 
>   core/src/main/java/org/apache/oozie/action/hadoop/CredentialsProvider.java 
> 6fe22fbd2fca5d873d58ece60f20d0f131371475 
>   
> core/src/main/java/org/apache/oozie/action/hadoop/CredentialsProviderFactory.java
>  PRE-CREATION 
>   core/src/main/java/org/apache/oozie/action/hadoop/DistcpActionExecutor.java 
> 78cd257a836707c1efeac13566d6285d42970545 
>   core/src/main/java/org/apache/oozie/action/hadoop/FsActionExecutor.java 
> 27654741101d2056b25626fbbe345183eafd612d 
>   core/src/main/java/org/apache/oozie/action/hadoop/FsELFunctions.java 
> 801bfe6eb212b549888f0e9fb7df7b65bcf59393 
>   core/src/main/java/org/apache/oozie/action/hadoop/HCatCredentialHelper.java 
> 36ad1df727e869b8051481ad969771e91bc6ae07 
>   core/src/main/java/org/apache/oozie/action/hadoop/HCatCredentials.java 
> d7689a924059e553abaeed374eed4072933292cb 
>   core/src/main/java/org/apache/oozie/action/hadoop/HadoopELFunctions.java 
> ad2a71d641caa71efc098ad00bfbab3cec4d9506 
>   core/src/main/java/org/apache/oozie/action/hadoop/HbaseCredentials.java 
> 307f565786ad77f5a35b5790bd5b052b04c6eac8 
>   core/src/main/java/org/apache/oozie/action/hadoop/Hive2ActionExecutor.java 
> 2aed936af75da38c433be5d202bdc01c2f612c76 
>   core/src/main/java/org/apache/oozie/action/hadoop/Hive2Credentials.java 
> bd2f1f768c5b04bef4a33613baede9202144c747 
>   core/src/main/java/org/apache/oozie/action/hadoop/HiveActionExecutor.java 
> 8e2453ef4f78f4a5f1f95b1ca2793f5d7e07ad81 
>   core/src/main/java/org/apache/oozie/action/hadoop/JavaActionExecutor.java 
> d60a5c7b996a4085f9bc77d82f12ad6925280b1a 
>   core/src/main/java/org/apache/oozie/action/hadoop/LauncherMapperHelper.java 
> 72ed2f150be73f0a5e8b71b0fb6cc3594dc23078 
>   
> core/src/main/java/org/apache/oozie/action/hadoop/MapReduceActionExecutor.java
>  55c937249ac35e14bd63dd0c1fa962ea636afa3a 
>   core/src/main/java/org/apache/oozie/action/hadoop/OozieJobInfo.java 
> 581d3b3219eff882f961c8e495611e873c6a5a52 
>   core/src/main/java/org/apache/oozie/action/hadoop/PigActionExecutor.java 
> 8b2dc16db80ad602911f1f64e60cfad41b198922 
>   
> core/src/main/java/org/apache/oozie/action/hadoop/ScriptLanguageActionExecutor.java
>  92e149d5cac6509893c8507b776263a38312a53b 
>   core/src/main/java/org/apache/oozie/action/hadoop/ShellActionExecutor.java 
> b9ffa7ab7ec87db8a76a3653c74974d296c8cd17 
>   core/src/main/java/org/apache/oozie/action/hadoop/SparkActionExecutor.java 
> 1a3197abb52f80b545dcc8634a8cac7a281a9eac 
>   core/src/main/java/org/apache/oozie/action/hadoop/SqoopActionExecutor.java 
> 22e28746a9a39ae6c1e22326cc4f8f759d820b20 
>   core/src/main/java/org/apache/oozie/client/rest/BulkResponseImpl.java 
> fb021bd022d199f93fc5674843514abf45ae12a5 
>   core/src/main/java/org/apache/oozie/command/XCommand.java 
> bdf13f65ea25e8e0f5125e2798568a2c03fffe81 
>   core/src/main/java/org/apache/oozie/command/wf/SubmitHttpXCommand.java 
> d2a2742ae418070a1cd0a889f14194224bc944a6 
>   core/src/main/java/org/apache/oozie/command/wf/SubmitMRXCommand.java 
> cc61d3de813c75c7f1cf4d405fc043253e3ca4ea 
>   
> core/src/main/java/org/apa

Re: Review Request 54705: OOZIE-1770: Create Oozie Application Master for YARN

[Peter Cseh]

> On May 14, 2017, 4:16 a.m., Rohini Palaniswamy wrote:
> > Have asked for new jiras for enhancements, missing scenarios or changes 
> > that require more work. Please reply with the new jira link before closing 
> > those issues in review board for easier referencing. Also add a TODO in the 
> > code with the jira number.
> > 
> > Some more action items:
> > 1) Getting tokens for ATS (Application Timeline Server) is missing. Please 
> > create a separate jira for that.
> > 2) Getting tokens for oozie.launcher.mapreduce.job.hdfs-servers is missing. 
> > Cannot do cross cluster distcp without that. Please create a separate jira 
> > for that.
> > 3) There should be code to get the Oozie AM application state from AHS 
> > (Application History Server) in check() if the completed job is gone from 
> > RM memory. For mapreduce launcher jobs, JobClient goes to Job History 
> > server to fetch that information. Please create a separate jira for that.

https://issues.apache.org/jira/browse/OOZIE-2899
https://issues.apache.org/jira/browse/OOZIE-2900
https://issues.apache.org/jira/browse/OOZIE-2901


- Peter


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/54705/#review174883
---


On May 10, 2017, 10:46 p.m., Peter Cseh wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/54705/
> ---
> 
> (Updated May 10, 2017, 10:46 p.m.)
> 
> 
> Review request for oozie, Purshotam Shah, Robert Kanter, and Rohini 
> Palaniswamy.
> 
> 
> Bugs: OOZIE-1770
> https://issues.apache.org/jira/browse/OOZIE-1770
> 
> 
> Repository: oozie-git
> 
> 
> Description
> ---
> 
> Replacing the launcher MR job with a launcher AM
> 
> 
> Diffs
> -
> 
>   client/pom.xml f99366c90887a5a45f3942291c90e1cdbfd84b3f 
>   client/src/main/java/org/apache/oozie/cli/OozieCLI.java 
> 38fb84e8a8918aee6ae57d19a29cdf181aba1955 
>   client/src/main/java/org/apache/oozie/client/XOozieClient.java 
> 8538ec7c742cc013f7c18eaff3ecf61c2835cc8a 
>   core/pom.xml 14aa03483e107db326628ba627cc3ee9f5fd3f3d 
>   core/src/main/java/org/apache/oozie/WorkflowJobBean.java 
> 55d79a5d5e5e4efab9eb832e9341a16bcf8959cc 
>   core/src/main/java/org/apache/oozie/action/ActionExecutor.java 
> 1d6456b236bf9e27c0f55d9f9e3b4bd94517c797 
>   core/src/main/java/org/apache/oozie/action/hadoop/Credentials.java 
> eadb47b07d157f45f086517ab41c4b9826f003f0 
>   core/src/main/java/org/apache/oozie/action/hadoop/CredentialsProvider.java 
> 6fe22fbd2fca5d873d58ece60f20d0f131371475 
>   
> core/src/main/java/org/apache/oozie/action/hadoop/CredentialsProviderFactory.java
>  PRE-CREATION 
>   core/src/main/java/org/apache/oozie/action/hadoop/DistcpActionExecutor.java 
> 78cd257a836707c1efeac13566d6285d42970545 
>   core/src/main/java/org/apache/oozie/action/hadoop/FsActionExecutor.java 
> 27654741101d2056b25626fbbe345183eafd612d 
>   core/src/main/java/org/apache/oozie/action/hadoop/FsELFunctions.java 
> 801bfe6eb212b549888f0e9fb7df7b65bcf59393 
>   core/src/main/java/org/apache/oozie/action/hadoop/HCatCredentialHelper.java 
> 36ad1df727e869b8051481ad969771e91bc6ae07 
>   core/src/main/java/org/apache/oozie/action/hadoop/HCatCredentials.java 
> d7689a924059e553abaeed374eed4072933292cb 
>   core/src/main/java/org/apache/oozie/action/hadoop/HadoopELFunctions.java 
> ad2a71d641caa71efc098ad00bfbab3cec4d9506 
>   core/src/main/java/org/apache/oozie/action/hadoop/HbaseCredentials.java 
> 307f565786ad77f5a35b5790bd5b052b04c6eac8 
>   core/src/main/java/org/apache/oozie/action/hadoop/Hive2ActionExecutor.java 
> 2aed936af75da38c433be5d202bdc01c2f612c76 
>   core/src/main/java/org/apache/oozie/action/hadoop/Hive2Credentials.java 
> bd2f1f768c5b04bef4a33613baede9202144c747 
>   core/src/main/java/org/apache/oozie/action/hadoop/HiveActionExecutor.java 
> 8e2453ef4f78f4a5f1f95b1ca2793f5d7e07ad81 
>   core/src/main/java/org/apache/oozie/action/hadoop/JavaActionExecutor.java 
> d60a5c7b996a4085f9bc77d82f12ad6925280b1a 
>   core/src/main/java/org/apache/oozie/action/hadoop/LauncherMapperHelper.java 
> 72ed2f150be73f0a5e8b71b0fb6cc3594dc23078 
>   
> core/src/main/java/org/apache/oozie/action/hadoop/MapReduceActionExecutor.java
>  55c937249ac35e14bd63dd0c1fa962ea636afa3a 
>   core/src/main/java/org/apache/oozie/action/hadoop/OozieJobInfo.java 
> 581d3b3219eff882f961c8e495611e873c6a5a52 
>   core/src/main/java/org/apache/oozie/action/hadoop/PigActionExecutor.java 
> 8b2dc16db80ad602911f1f64e60cfad41b198922 
>   
> core/src/main/java/org/apache/oozie/action/hadoop/ScriptLanguageActionExecutor.java
>  92e149d5cac6509893c8507b776263a38312a53b 
>   core/src/main/java/org/apache/oozie/action/hadoop/ShellActionExecutor.java 
> b9ffa7ab7ec87db8a76a3653c74974d296c8cd17 
>   core/src/main/java/o

Re: Review Request 54705: OOZIE-1770: Create Oozie Application Master for YARN

[Peter Cseh]

> On May 14, 2017, 4:16 a.m., Rohini Palaniswamy wrote:
> > core/src/main/java/org/apache/oozie/action/hadoop/Credentials.java
> > Line 25 (original), 28 (patched)
> > 
> >
> > Shouldn't this be just a new class instead of moving Credentials as 
> > they are totally unrelated?

This was not a rename, git is messing up something. We'll check it.


- Peter


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/54705/#review174883
---


On May 10, 2017, 10:46 p.m., Peter Cseh wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/54705/
> ---
> 
> (Updated May 10, 2017, 10:46 p.m.)
> 
> 
> Review request for oozie, Purshotam Shah, Robert Kanter, and Rohini 
> Palaniswamy.
> 
> 
> Bugs: OOZIE-1770
> https://issues.apache.org/jira/browse/OOZIE-1770
> 
> 
> Repository: oozie-git
> 
> 
> Description
> ---
> 
> Replacing the launcher MR job with a launcher AM
> 
> 
> Diffs
> -
> 
>   client/pom.xml f99366c90887a5a45f3942291c90e1cdbfd84b3f 
>   client/src/main/java/org/apache/oozie/cli/OozieCLI.java 
> 38fb84e8a8918aee6ae57d19a29cdf181aba1955 
>   client/src/main/java/org/apache/oozie/client/XOozieClient.java 
> 8538ec7c742cc013f7c18eaff3ecf61c2835cc8a 
>   core/pom.xml 14aa03483e107db326628ba627cc3ee9f5fd3f3d 
>   core/src/main/java/org/apache/oozie/WorkflowJobBean.java 
> 55d79a5d5e5e4efab9eb832e9341a16bcf8959cc 
>   core/src/main/java/org/apache/oozie/action/ActionExecutor.java 
> 1d6456b236bf9e27c0f55d9f9e3b4bd94517c797 
>   core/src/main/java/org/apache/oozie/action/hadoop/Credentials.java 
> eadb47b07d157f45f086517ab41c4b9826f003f0 
>   core/src/main/java/org/apache/oozie/action/hadoop/CredentialsProvider.java 
> 6fe22fbd2fca5d873d58ece60f20d0f131371475 
>   
> core/src/main/java/org/apache/oozie/action/hadoop/CredentialsProviderFactory.java
>  PRE-CREATION 
>   core/src/main/java/org/apache/oozie/action/hadoop/DistcpActionExecutor.java 
> 78cd257a836707c1efeac13566d6285d42970545 
>   core/src/main/java/org/apache/oozie/action/hadoop/FsActionExecutor.java 
> 27654741101d2056b25626fbbe345183eafd612d 
>   core/src/main/java/org/apache/oozie/action/hadoop/FsELFunctions.java 
> 801bfe6eb212b549888f0e9fb7df7b65bcf59393 
>   core/src/main/java/org/apache/oozie/action/hadoop/HCatCredentialHelper.java 
> 36ad1df727e869b8051481ad969771e91bc6ae07 
>   core/src/main/java/org/apache/oozie/action/hadoop/HCatCredentials.java 
> d7689a924059e553abaeed374eed4072933292cb 
>   core/src/main/java/org/apache/oozie/action/hadoop/HadoopELFunctions.java 
> ad2a71d641caa71efc098ad00bfbab3cec4d9506 
>   core/src/main/java/org/apache/oozie/action/hadoop/HbaseCredentials.java 
> 307f565786ad77f5a35b5790bd5b052b04c6eac8 
>   core/src/main/java/org/apache/oozie/action/hadoop/Hive2ActionExecutor.java 
> 2aed936af75da38c433be5d202bdc01c2f612c76 
>   core/src/main/java/org/apache/oozie/action/hadoop/Hive2Credentials.java 
> bd2f1f768c5b04bef4a33613baede9202144c747 
>   core/src/main/java/org/apache/oozie/action/hadoop/HiveActionExecutor.java 
> 8e2453ef4f78f4a5f1f95b1ca2793f5d7e07ad81 
>   core/src/main/java/org/apache/oozie/action/hadoop/JavaActionExecutor.java 
> d60a5c7b996a4085f9bc77d82f12ad6925280b1a 
>   core/src/main/java/org/apache/oozie/action/hadoop/LauncherMapperHelper.java 
> 72ed2f150be73f0a5e8b71b0fb6cc3594dc23078 
>   
> core/src/main/java/org/apache/oozie/action/hadoop/MapReduceActionExecutor.java
>  55c937249ac35e14bd63dd0c1fa962ea636afa3a 
>   core/src/main/java/org/apache/oozie/action/hadoop/OozieJobInfo.java 
> 581d3b3219eff882f961c8e495611e873c6a5a52 
>   core/src/main/java/org/apache/oozie/action/hadoop/PigActionExecutor.java 
> 8b2dc16db80ad602911f1f64e60cfad41b198922 
>   
> core/src/main/java/org/apache/oozie/action/hadoop/ScriptLanguageActionExecutor.java
>  92e149d5cac6509893c8507b776263a38312a53b 
>   core/src/main/java/org/apache/oozie/action/hadoop/ShellActionExecutor.java 
> b9ffa7ab7ec87db8a76a3653c74974d296c8cd17 
>   core/src/main/java/org/apache/oozie/action/hadoop/SparkActionExecutor.java 
> 1a3197abb52f80b545dcc8634a8cac7a281a9eac 
>   core/src/main/java/org/apache/oozie/action/hadoop/SqoopActionExecutor.java 
> 22e28746a9a39ae6c1e22326cc4f8f759d820b20 
>   core/src/main/java/org/apache/oozie/client/rest/BulkResponseImpl.java 
> fb021bd022d199f93fc5674843514abf45ae12a5 
>   core/src/main/java/org/apache/oozie/command/XCommand.java 
> bdf13f65ea25e8e0f5125e2798568a2c03fffe81 
>   core/src/main/java/org/apache/oozie/command/wf/SubmitHttpXCommand.java 
> d2a2742ae418070a1cd0a889f14194224bc944a6 
>   core/src/main/java/org/apache/oozie/command/wf/SubmitMRXCommand.java 
> cc61d3de813c75c7f1cf4d405fc043253e

Re: Review Request 54705: OOZIE-1770: Create Oozie Application Master for YARN

[Peter Cseh]

> On May 14, 2017, 4:16 a.m., Rohini Palaniswamy wrote:
> > core/src/main/java/org/apache/oozie/action/hadoop/JavaActionExecutor.java
> > Line 325 (original)
> > 
> >
> > We still need the java opts, memory.mb and env settings specified by 
> > the user using oozie.launcher. settings applied to the Oozie AM. Please 
> > create a new jira for this.
> 
> Rohini Palaniswamy wrote:
> This is different from OOZIE-2687 which introduces a option xml section 
> to simplify specification of those launcher settings. Existing oozie.launcher 
> MR settings specified by users should continute to work as is without user 
> having to rewrite their workflow.xml. Else this will be a major backward 
> incompatibility.

OOZIE-2896


- Peter


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/54705/#review174883
---


On May 10, 2017, 10:46 p.m., Peter Cseh wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/54705/
> ---
> 
> (Updated May 10, 2017, 10:46 p.m.)
> 
> 
> Review request for oozie, Purshotam Shah, Robert Kanter, and Rohini 
> Palaniswamy.
> 
> 
> Bugs: OOZIE-1770
> https://issues.apache.org/jira/browse/OOZIE-1770
> 
> 
> Repository: oozie-git
> 
> 
> Description
> ---
> 
> Replacing the launcher MR job with a launcher AM
> 
> 
> Diffs
> -
> 
>   client/pom.xml f99366c90887a5a45f3942291c90e1cdbfd84b3f 
>   client/src/main/java/org/apache/oozie/cli/OozieCLI.java 
> 38fb84e8a8918aee6ae57d19a29cdf181aba1955 
>   client/src/main/java/org/apache/oozie/client/XOozieClient.java 
> 8538ec7c742cc013f7c18eaff3ecf61c2835cc8a 
>   core/pom.xml 14aa03483e107db326628ba627cc3ee9f5fd3f3d 
>   core/src/main/java/org/apache/oozie/WorkflowJobBean.java 
> 55d79a5d5e5e4efab9eb832e9341a16bcf8959cc 
>   core/src/main/java/org/apache/oozie/action/ActionExecutor.java 
> 1d6456b236bf9e27c0f55d9f9e3b4bd94517c797 
>   core/src/main/java/org/apache/oozie/action/hadoop/Credentials.java 
> eadb47b07d157f45f086517ab41c4b9826f003f0 
>   core/src/main/java/org/apache/oozie/action/hadoop/CredentialsProvider.java 
> 6fe22fbd2fca5d873d58ece60f20d0f131371475 
>   
> core/src/main/java/org/apache/oozie/action/hadoop/CredentialsProviderFactory.java
>  PRE-CREATION 
>   core/src/main/java/org/apache/oozie/action/hadoop/DistcpActionExecutor.java 
> 78cd257a836707c1efeac13566d6285d42970545 
>   core/src/main/java/org/apache/oozie/action/hadoop/FsActionExecutor.java 
> 27654741101d2056b25626fbbe345183eafd612d 
>   core/src/main/java/org/apache/oozie/action/hadoop/FsELFunctions.java 
> 801bfe6eb212b549888f0e9fb7df7b65bcf59393 
>   core/src/main/java/org/apache/oozie/action/hadoop/HCatCredentialHelper.java 
> 36ad1df727e869b8051481ad969771e91bc6ae07 
>   core/src/main/java/org/apache/oozie/action/hadoop/HCatCredentials.java 
> d7689a924059e553abaeed374eed4072933292cb 
>   core/src/main/java/org/apache/oozie/action/hadoop/HadoopELFunctions.java 
> ad2a71d641caa71efc098ad00bfbab3cec4d9506 
>   core/src/main/java/org/apache/oozie/action/hadoop/HbaseCredentials.java 
> 307f565786ad77f5a35b5790bd5b052b04c6eac8 
>   core/src/main/java/org/apache/oozie/action/hadoop/Hive2ActionExecutor.java 
> 2aed936af75da38c433be5d202bdc01c2f612c76 
>   core/src/main/java/org/apache/oozie/action/hadoop/Hive2Credentials.java 
> bd2f1f768c5b04bef4a33613baede9202144c747 
>   core/src/main/java/org/apache/oozie/action/hadoop/HiveActionExecutor.java 
> 8e2453ef4f78f4a5f1f95b1ca2793f5d7e07ad81 
>   core/src/main/java/org/apache/oozie/action/hadoop/JavaActionExecutor.java 
> d60a5c7b996a4085f9bc77d82f12ad6925280b1a 
>   core/src/main/java/org/apache/oozie/action/hadoop/LauncherMapperHelper.java 
> 72ed2f150be73f0a5e8b71b0fb6cc3594dc23078 
>   
> core/src/main/java/org/apache/oozie/action/hadoop/MapReduceActionExecutor.java
>  55c937249ac35e14bd63dd0c1fa962ea636afa3a 
>   core/src/main/java/org/apache/oozie/action/hadoop/OozieJobInfo.java 
> 581d3b3219eff882f961c8e495611e873c6a5a52 
>   core/src/main/java/org/apache/oozie/action/hadoop/PigActionExecutor.java 
> 8b2dc16db80ad602911f1f64e60cfad41b198922 
>   
> core/src/main/java/org/apache/oozie/action/hadoop/ScriptLanguageActionExecutor.java
>  92e149d5cac6509893c8507b776263a38312a53b 
>   core/src/main/java/org/apache/oozie/action/hadoop/ShellActionExecutor.java 
> b9ffa7ab7ec87db8a76a3653c74974d296c8cd17 
>   core/src/main/java/org/apache/oozie/action/hadoop/SparkActionExecutor.java 
> 1a3197abb52f80b545dcc8634a8cac7a281a9eac 
>   core/src/main/java/org/apache/oozie/action/hadoop/SqoopActionExecutor.java 
> 22e28746a9a39ae6c1e22326cc4f8f759d820b20 
>   core/src/main/java/org/apache/oozie/client/rest/BulkResponseImpl.java 
>

Re: Review Request 54705: OOZIE-1770: Create Oozie Application Master for YARN

[Peter Cseh]

> On May 14, 2017, 4:16 a.m., Rohini Palaniswamy wrote:
> > core/src/main/java/org/apache/oozie/service/HadoopAccessorService.java
> > Lines 777 (patched)
> > 
> >
> > os needs to be closed.

It's closed by java in try-with-resouces 
https://docs.oracle.com/javase/tutorial/essential/exceptions/tryResourceClose.html


- Peter


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/54705/#review174883
---


On May 10, 2017, 10:46 p.m., Peter Cseh wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/54705/
> ---
> 
> (Updated May 10, 2017, 10:46 p.m.)
> 
> 
> Review request for oozie, Purshotam Shah, Robert Kanter, and Rohini 
> Palaniswamy.
> 
> 
> Bugs: OOZIE-1770
> https://issues.apache.org/jira/browse/OOZIE-1770
> 
> 
> Repository: oozie-git
> 
> 
> Description
> ---
> 
> Replacing the launcher MR job with a launcher AM
> 
> 
> Diffs
> -
> 
>   client/pom.xml f99366c90887a5a45f3942291c90e1cdbfd84b3f 
>   client/src/main/java/org/apache/oozie/cli/OozieCLI.java 
> 38fb84e8a8918aee6ae57d19a29cdf181aba1955 
>   client/src/main/java/org/apache/oozie/client/XOozieClient.java 
> 8538ec7c742cc013f7c18eaff3ecf61c2835cc8a 
>   core/pom.xml 14aa03483e107db326628ba627cc3ee9f5fd3f3d 
>   core/src/main/java/org/apache/oozie/WorkflowJobBean.java 
> 55d79a5d5e5e4efab9eb832e9341a16bcf8959cc 
>   core/src/main/java/org/apache/oozie/action/ActionExecutor.java 
> 1d6456b236bf9e27c0f55d9f9e3b4bd94517c797 
>   core/src/main/java/org/apache/oozie/action/hadoop/Credentials.java 
> eadb47b07d157f45f086517ab41c4b9826f003f0 
>   core/src/main/java/org/apache/oozie/action/hadoop/CredentialsProvider.java 
> 6fe22fbd2fca5d873d58ece60f20d0f131371475 
>   
> core/src/main/java/org/apache/oozie/action/hadoop/CredentialsProviderFactory.java
>  PRE-CREATION 
>   core/src/main/java/org/apache/oozie/action/hadoop/DistcpActionExecutor.java 
> 78cd257a836707c1efeac13566d6285d42970545 
>   core/src/main/java/org/apache/oozie/action/hadoop/FsActionExecutor.java 
> 27654741101d2056b25626fbbe345183eafd612d 
>   core/src/main/java/org/apache/oozie/action/hadoop/FsELFunctions.java 
> 801bfe6eb212b549888f0e9fb7df7b65bcf59393 
>   core/src/main/java/org/apache/oozie/action/hadoop/HCatCredentialHelper.java 
> 36ad1df727e869b8051481ad969771e91bc6ae07 
>   core/src/main/java/org/apache/oozie/action/hadoop/HCatCredentials.java 
> d7689a924059e553abaeed374eed4072933292cb 
>   core/src/main/java/org/apache/oozie/action/hadoop/HadoopELFunctions.java 
> ad2a71d641caa71efc098ad00bfbab3cec4d9506 
>   core/src/main/java/org/apache/oozie/action/hadoop/HbaseCredentials.java 
> 307f565786ad77f5a35b5790bd5b052b04c6eac8 
>   core/src/main/java/org/apache/oozie/action/hadoop/Hive2ActionExecutor.java 
> 2aed936af75da38c433be5d202bdc01c2f612c76 
>   core/src/main/java/org/apache/oozie/action/hadoop/Hive2Credentials.java 
> bd2f1f768c5b04bef4a33613baede9202144c747 
>   core/src/main/java/org/apache/oozie/action/hadoop/HiveActionExecutor.java 
> 8e2453ef4f78f4a5f1f95b1ca2793f5d7e07ad81 
>   core/src/main/java/org/apache/oozie/action/hadoop/JavaActionExecutor.java 
> d60a5c7b996a4085f9bc77d82f12ad6925280b1a 
>   core/src/main/java/org/apache/oozie/action/hadoop/LauncherMapperHelper.java 
> 72ed2f150be73f0a5e8b71b0fb6cc3594dc23078 
>   
> core/src/main/java/org/apache/oozie/action/hadoop/MapReduceActionExecutor.java
>  55c937249ac35e14bd63dd0c1fa962ea636afa3a 
>   core/src/main/java/org/apache/oozie/action/hadoop/OozieJobInfo.java 
> 581d3b3219eff882f961c8e495611e873c6a5a52 
>   core/src/main/java/org/apache/oozie/action/hadoop/PigActionExecutor.java 
> 8b2dc16db80ad602911f1f64e60cfad41b198922 
>   
> core/src/main/java/org/apache/oozie/action/hadoop/ScriptLanguageActionExecutor.java
>  92e149d5cac6509893c8507b776263a38312a53b 
>   core/src/main/java/org/apache/oozie/action/hadoop/ShellActionExecutor.java 
> b9ffa7ab7ec87db8a76a3653c74974d296c8cd17 
>   core/src/main/java/org/apache/oozie/action/hadoop/SparkActionExecutor.java 
> 1a3197abb52f80b545dcc8634a8cac7a281a9eac 
>   core/src/main/java/org/apache/oozie/action/hadoop/SqoopActionExecutor.java 
> 22e28746a9a39ae6c1e22326cc4f8f759d820b20 
>   core/src/main/java/org/apache/oozie/client/rest/BulkResponseImpl.java 
> fb021bd022d199f93fc5674843514abf45ae12a5 
>   core/src/main/java/org/apache/oozie/command/XCommand.java 
> bdf13f65ea25e8e0f5125e2798568a2c03fffe81 
>   core/src/main/java/org/apache/oozie/command/wf/SubmitHttpXCommand.java 
> d2a2742ae418070a1cd0a889f14194224bc944a6 
>   core/src/main/java/org/apache/oozie/command/wf/SubmitMRXCommand.java 
> cc61d3de813c75c7f1cf4d405fc043253e3ca4ea 
>   
> core/src/main/

Re: Review Request 54705: OOZIE-1770: Create Oozie Application Master for YARN

[Peter Cseh]

> On May 14, 2017, 4:16 a.m., Rohini Palaniswamy wrote:
> > hadooplibs/hadoop-utils-1/src/main/java/org/apache/oozie/action/hadoop/LauncherMainHadoopUtils.java
> > Line 30 (original), 22 (patched)
> > 
> >
> > Shouldn't this be just a new class instead of moving 
> > LauncherMainHadoopUtils as they are totally unrelated?

Will check.


> On May 14, 2017, 4:16 a.m., Rohini Palaniswamy wrote:
> > sharelib/oozie/src/main/java/org/apache/oozie/action/hadoop/LauncherAM.java
> > Lines 295 (patched)
> > 
> >
> > Can't we use log4j (syslog instead of stdout) for Oozie AM messages? 
> > There will be no timestamp with S.O.P and debugging will be really hard. 
> > Can be a new jira.

https://issues.apache.org/jira/browse/OOZIE-2702 and 
https://issues.apache.org/jira/browse/OOZIE-2898 are opened to address this


> On May 14, 2017, 4:16 a.m., Rohini Palaniswamy wrote:
> > sharelib/oozie/src/main/java/org/apache/oozie/action/hadoop/LauncherAMCallbackNotifier.java
> > Lines 93 (patched)
> > 
> >
> > Use log4j instead of S.O.P and S.E.P

https://issues.apache.org/jira/browse/OOZIE-2898


> On May 14, 2017, 4:16 a.m., Rohini Palaniswamy wrote:
> > sharelib/sqoop/src/main/java/org/apache/oozie/action/hadoop/SqoopMain.java
> > Lines 172-175 (original)
> > 
> >
> > Shouldn't we retain this?

It's handled in LauncherAM for every kind of Main class.


- Peter


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/54705/#review174883
---


On May 10, 2017, 10:46 p.m., Peter Cseh wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/54705/
> ---
> 
> (Updated May 10, 2017, 10:46 p.m.)
> 
> 
> Review request for oozie, Purshotam Shah, Robert Kanter, and Rohini 
> Palaniswamy.
> 
> 
> Bugs: OOZIE-1770
> https://issues.apache.org/jira/browse/OOZIE-1770
> 
> 
> Repository: oozie-git
> 
> 
> Description
> ---
> 
> Replacing the launcher MR job with a launcher AM
> 
> 
> Diffs
> -
> 
>   client/pom.xml f99366c90887a5a45f3942291c90e1cdbfd84b3f 
>   client/src/main/java/org/apache/oozie/cli/OozieCLI.java 
> 38fb84e8a8918aee6ae57d19a29cdf181aba1955 
>   client/src/main/java/org/apache/oozie/client/XOozieClient.java 
> 8538ec7c742cc013f7c18eaff3ecf61c2835cc8a 
>   core/pom.xml 14aa03483e107db326628ba627cc3ee9f5fd3f3d 
>   core/src/main/java/org/apache/oozie/WorkflowJobBean.java 
> 55d79a5d5e5e4efab9eb832e9341a16bcf8959cc 
>   core/src/main/java/org/apache/oozie/action/ActionExecutor.java 
> 1d6456b236bf9e27c0f55d9f9e3b4bd94517c797 
>   core/src/main/java/org/apache/oozie/action/hadoop/Credentials.java 
> eadb47b07d157f45f086517ab41c4b9826f003f0 
>   core/src/main/java/org/apache/oozie/action/hadoop/CredentialsProvider.java 
> 6fe22fbd2fca5d873d58ece60f20d0f131371475 
>   
> core/src/main/java/org/apache/oozie/action/hadoop/CredentialsProviderFactory.java
>  PRE-CREATION 
>   core/src/main/java/org/apache/oozie/action/hadoop/DistcpActionExecutor.java 
> 78cd257a836707c1efeac13566d6285d42970545 
>   core/src/main/java/org/apache/oozie/action/hadoop/FsActionExecutor.java 
> 27654741101d2056b25626fbbe345183eafd612d 
>   core/src/main/java/org/apache/oozie/action/hadoop/FsELFunctions.java 
> 801bfe6eb212b549888f0e9fb7df7b65bcf59393 
>   core/src/main/java/org/apache/oozie/action/hadoop/HCatCredentialHelper.java 
> 36ad1df727e869b8051481ad969771e91bc6ae07 
>   core/src/main/java/org/apache/oozie/action/hadoop/HCatCredentials.java 
> d7689a924059e553abaeed374eed4072933292cb 
>   core/src/main/java/org/apache/oozie/action/hadoop/HadoopELFunctions.java 
> ad2a71d641caa71efc098ad00bfbab3cec4d9506 
>   core/src/main/java/org/apache/oozie/action/hadoop/HbaseCredentials.java 
> 307f565786ad77f5a35b5790bd5b052b04c6eac8 
>   core/src/main/java/org/apache/oozie/action/hadoop/Hive2ActionExecutor.java 
> 2aed936af75da38c433be5d202bdc01c2f612c76 
>   core/src/main/java/org/apache/oozie/action/hadoop/Hive2Credentials.java 
> bd2f1f768c5b04bef4a33613baede9202144c747 
>   core/src/main/java/org/apache/oozie/action/hadoop/HiveActionExecutor.java 
> 8e2453ef4f78f4a5f1f95b1ca2793f5d7e07ad81 
>   core/src/main/java/org/apache/oozie/action/hadoop/JavaActionExecutor.java 
> d60a5c7b996a4085f9bc77d82f12ad6925280b1a 
>   core/src/main/java/org/apache/oozie/action/hadoop/LauncherMapperHelper.java 
> 72ed2f150be73f0a5e8b71b0fb6cc3594dc23078 
>   
> core/src/main/java/org/apache/oozie/action/hadoop/MapReduceActionExecutor.java
>  

Re: Review Request 54705: OOZIE-1770: Create Oozie Application Master for YARN

[Peter Bacsko]

> On máj. 14, 2017, 4:16 de, Rohini Palaniswamy wrote:
> > core/src/test/java/org/apache/oozie/action/hadoop/TestJavaActionExecutor.java
> > Line 1788 (original)
> > 
> >
> > Oozie AM needs to support the mapreduce view and modify acls. This 
> > feature is heavily used by users for security in a multi-tenant 
> > environment. Please create a new jira for this to be addressed after merge.

Created follow-up JIRA: https://issues.apache.org/jira/browse/OOZIE-2897


> On máj. 14, 2017, 4:16 de, Rohini Palaniswamy wrote:
> > hadooplibs/hadoop-utils-1/src/main/java/org/apache/oozie/action/hadoop/LauncherMainHadoopUtils.java
> > Line 37 (original), 24 (patched)
> > 
> >
> > Any reason for not having this method as static and instead creating a 
> > new instance of LauncherAMCallbackNotifierFactory everytime?

Unit tests. We insert a mock factory because we don't want to invoke methods on 
a real callback notifier instance.


> On máj. 14, 2017, 4:16 de, Rohini Palaniswamy wrote:
> > hadooplibs/hadoop-utils-1/src/main/java/org/apache/oozie/hadoop/utils/HadoopShims.java
> > Line 44 (original), 26 (patched)
> > 
> >
> > Create a new class instead of moving HadoopShims as they are totally 
> > unrelated

This wasn't a move, I suppose Git thought it was a file rename/move.


> On máj. 14, 2017, 4:16 de, Rohini Palaniswamy wrote:
> > sharelib/oozie/src/main/java/org/apache/oozie/action/hadoop/HdfsOperations.java
> > Lines 64 (patched)
> > 
> >
> > Why do we need a factory for creating a SequenceFile? It is just a few 
> > lines of code and it is a direct Hadoop API. If there were different 
> > formats for the file  or different ways to create a sequence file it would 
> > make sense to have a factory.

This helps greatly with writing unit tests for LauncherAM. An implementation of 
SequenceFile requires a Hadoop configuration, wants to create a FileSystem 
object, etc. Much easier with mocks.


> On máj. 14, 2017, 4:16 de, Rohini Palaniswamy wrote:
> > sharelib/oozie/src/main/java/org/apache/oozie/action/hadoop/PrepareActionsDriver.java
> > Line 41 (original), 41-42 (patched)
> > 
> >
> > This is a internal class. Why not totally get rid of it instead of 
> > deprecating?

Created follow-up JIRA: https://issues.apache.org/jira/browse/OOZIE-2907


> On máj. 14, 2017, 4:16 de, Rohini Palaniswamy wrote:
> > sharelib/oozie/src/test/java/org/apache/oozie/action/hadoop/TestHdfsOperations.java
> > Lines 49 (patched)
> > 
> >
> > Do we need this test class? uploadActionDataToHDFS is already tested 
> > during execution of many tests - mapreduce action tests or when we check 
> > for external ids. Just trying to avoid unnecessary tests as test run time 
> > is constantly going up.

This is a super-fast unit test (doesn't extend XTestCase). IMO it's worth 
keeping it.


- Peter


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/54705/#review174883
---


On máj. 10, 2017, 10:46 du, Peter Cseh wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/54705/
> ---
> 
> (Updated máj. 10, 2017, 10:46 du)
> 
> 
> Review request for oozie, Purshotam Shah, Robert Kanter, and Rohini 
> Palaniswamy.
> 
> 
> Bugs: OOZIE-1770
> https://issues.apache.org/jira/browse/OOZIE-1770
> 
> 
> Repository: oozie-git
> 
> 
> Description
> ---
> 
> Replacing the launcher MR job with a launcher AM
> 
> 
> Diffs
> -
> 
>   client/pom.xml f99366c90887a5a45f3942291c90e1cdbfd84b3f 
>   client/src/main/java/org/apache/oozie/cli/OozieCLI.java 
> 38fb84e8a8918aee6ae57d19a29cdf181aba1955 
>   client/src/main/java/org/apache/oozie/client/XOozieClient.java 
> 8538ec7c742cc013f7c18eaff3ecf61c2835cc8a 
>   core/pom.xml 14aa03483e107db326628ba627cc3ee9f5fd3f3d 
>   core/src/main/java/org/apache/oozie/WorkflowJobBean.java 
> 55d79a5d5e5e4efab9eb832e9341a16bcf8959cc 
>   core/src/main/java/org/apache/oozie/action/ActionExecutor.java 
> 1d6456b236bf9e27c0f55d9f9e3b4bd94517c797 
>   core/src/main/java/org/apache/oozie/action/hadoop/Credentials.java 
> eadb47b07d157f45f086517ab41c4b9826f003f0 
>   core/src/main/java/org/apache/oozie/action/hadoop/CredentialsProvider.java 
> 6fe22fbd2fca5d873d58ece60f20d0f131371475 
>   
> core/src/main/java/org/apache/oozie/action/hadoop/Crede

Re: Review Request 54705: OOZIE-1770: Create Oozie Application Master for YARN

[Peter Bacsko]

> On máj. 14, 2017, 4:16 de, Rohini Palaniswamy wrote:
> > core/src/test/java/org/apache/oozie/action/hadoop/TestShellActionExecutor.java
> > Line 51 (original), 45 (patched)
> > 
> >
> > Why is ./ required instead of just script.sh?

This was a tricky thing that I didn't understand before. We have to update 
$PATH to contain the current working directory. I can imagine there are users 
who depend on this.

OOZIE-2896 will address handling MR properties (ShellActionExecutor updates 
"mapred.map.env" but it's ignored right now).

But it's better to update ShellActionExecutor itself, that is, extend 
addActionSpecificEnvVars(). Will do this soon.


- Peter


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/54705/#review174883
---


On máj. 10, 2017, 10:46 du, Peter Cseh wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/54705/
> ---
> 
> (Updated máj. 10, 2017, 10:46 du)
> 
> 
> Review request for oozie, Purshotam Shah, Robert Kanter, and Rohini 
> Palaniswamy.
> 
> 
> Bugs: OOZIE-1770
> https://issues.apache.org/jira/browse/OOZIE-1770
> 
> 
> Repository: oozie-git
> 
> 
> Description
> ---
> 
> Replacing the launcher MR job with a launcher AM
> 
> 
> Diffs
> -
> 
>   client/pom.xml f99366c90887a5a45f3942291c90e1cdbfd84b3f 
>   client/src/main/java/org/apache/oozie/cli/OozieCLI.java 
> 38fb84e8a8918aee6ae57d19a29cdf181aba1955 
>   client/src/main/java/org/apache/oozie/client/XOozieClient.java 
> 8538ec7c742cc013f7c18eaff3ecf61c2835cc8a 
>   core/pom.xml 14aa03483e107db326628ba627cc3ee9f5fd3f3d 
>   core/src/main/java/org/apache/oozie/WorkflowJobBean.java 
> 55d79a5d5e5e4efab9eb832e9341a16bcf8959cc 
>   core/src/main/java/org/apache/oozie/action/ActionExecutor.java 
> 1d6456b236bf9e27c0f55d9f9e3b4bd94517c797 
>   core/src/main/java/org/apache/oozie/action/hadoop/Credentials.java 
> eadb47b07d157f45f086517ab41c4b9826f003f0 
>   core/src/main/java/org/apache/oozie/action/hadoop/CredentialsProvider.java 
> 6fe22fbd2fca5d873d58ece60f20d0f131371475 
>   
> core/src/main/java/org/apache/oozie/action/hadoop/CredentialsProviderFactory.java
>  PRE-CREATION 
>   core/src/main/java/org/apache/oozie/action/hadoop/DistcpActionExecutor.java 
> 78cd257a836707c1efeac13566d6285d42970545 
>   core/src/main/java/org/apache/oozie/action/hadoop/FsActionExecutor.java 
> 27654741101d2056b25626fbbe345183eafd612d 
>   core/src/main/java/org/apache/oozie/action/hadoop/FsELFunctions.java 
> 801bfe6eb212b549888f0e9fb7df7b65bcf59393 
>   core/src/main/java/org/apache/oozie/action/hadoop/HCatCredentialHelper.java 
> 36ad1df727e869b8051481ad969771e91bc6ae07 
>   core/src/main/java/org/apache/oozie/action/hadoop/HCatCredentials.java 
> d7689a924059e553abaeed374eed4072933292cb 
>   core/src/main/java/org/apache/oozie/action/hadoop/HadoopELFunctions.java 
> ad2a71d641caa71efc098ad00bfbab3cec4d9506 
>   core/src/main/java/org/apache/oozie/action/hadoop/HbaseCredentials.java 
> 307f565786ad77f5a35b5790bd5b052b04c6eac8 
>   core/src/main/java/org/apache/oozie/action/hadoop/Hive2ActionExecutor.java 
> 2aed936af75da38c433be5d202bdc01c2f612c76 
>   core/src/main/java/org/apache/oozie/action/hadoop/Hive2Credentials.java 
> bd2f1f768c5b04bef4a33613baede9202144c747 
>   core/src/main/java/org/apache/oozie/action/hadoop/HiveActionExecutor.java 
> 8e2453ef4f78f4a5f1f95b1ca2793f5d7e07ad81 
>   core/src/main/java/org/apache/oozie/action/hadoop/JavaActionExecutor.java 
> d60a5c7b996a4085f9bc77d82f12ad6925280b1a 
>   core/src/main/java/org/apache/oozie/action/hadoop/LauncherMapperHelper.java 
> 72ed2f150be73f0a5e8b71b0fb6cc3594dc23078 
>   
> core/src/main/java/org/apache/oozie/action/hadoop/MapReduceActionExecutor.java
>  55c937249ac35e14bd63dd0c1fa962ea636afa3a 
>   core/src/main/java/org/apache/oozie/action/hadoop/OozieJobInfo.java 
> 581d3b3219eff882f961c8e495611e873c6a5a52 
>   core/src/main/java/org/apache/oozie/action/hadoop/PigActionExecutor.java 
> 8b2dc16db80ad602911f1f64e60cfad41b198922 
>   
> core/src/main/java/org/apache/oozie/action/hadoop/ScriptLanguageActionExecutor.java
>  92e149d5cac6509893c8507b776263a38312a53b 
>   core/src/main/java/org/apache/oozie/action/hadoop/ShellActionExecutor.java 
> b9ffa7ab7ec87db8a76a3653c74974d296c8cd17 
>   core/src/main/java/org/apache/oozie/action/hadoop/SparkActionExecutor.java 
> 1a3197abb52f80b545dcc8634a8cac7a281a9eac 
>   core/src/main/java/org/apache/oozie/action/hadoop/SqoopActionExecutor.java 
> 22e28746a9a39ae6c1e22326cc4f8f759d820b20 
>   core/src/main/java/org/apache/oozie/client/rest/BulkResponseImpl.java 
> fb021bd022d199f93fc5674843514abf45ae12a5 
>   core/src/main/java/org/apache

Re: Review Request 54705: OOZIE-1770: Create Oozie Application Master for YARN

[Peter Bacsko]

> On máj. 14, 2017, 4:16 de, Rohini Palaniswamy wrote:
> > core/src/main/java/org/apache/oozie/action/hadoop/JavaActionExecutor.java
> > Line 423 (original), 321 (patched)
> > 
> >
> > This should not be required anymore. The Oozie AM code should be doing 
> > the same thing as done in MAPREDUCE-6472.

Right - deleted


> On máj. 14, 2017, 4:16 de, Rohini Palaniswamy wrote:
> > core/src/main/java/org/apache/oozie/action/hadoop/PigActionExecutor.java
> > Line 130 (original), 126 (patched)
> > 
> >
> > Can you rename method as addMapreduceToClasspath?

Renamed to needToAddMapReduceToClassPath()


> On máj. 14, 2017, 4:16 de, Rohini Palaniswamy wrote:
> > sharelib/oozie/pom.xml
> > Line 66 (original), 66 (patched)
> > 
> >
> > Shouldn't scope be test?

Yep, added


> On máj. 14, 2017, 4:16 de, Rohini Palaniswamy wrote:
> > sharelib/oozie/src/main/java/org/apache/oozie/action/hadoop/LauncherAM.java
> > Lines 119-127 (patched)
> > 
> >
> > We cannot be doing createRemoteUser with submitter.user. Oozie AM 
> > should be launched as the ugi the whole process should run as similar to 
> > current mapreduce launcher. Proxy users only are supported when run from 
> > specific machines as they are super users (For eg: Oozie is specified as a 
> > super user in NN, RM and JT configs and oozie server hosts are specified 
> > there so that do as from any other host is rejected). Basically there 
> > should not be any ugi.doAs() in launcher code.

I don't completely follow this reasoning. This is very similar to MRAppMaster: 
https://github.com/apache/hadoop/blob/3ed3062fe3979ff55a411b730a8eee2b2c96d6b3/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/MRAppMaster.java#L1718-L1726

As far as I understand, proxy user support are only required on hosts where 
Oozie is running so that Oozie can retrieve the tokens for the impersonated 
user. 

There are two things which we probably should do differently:
1) For some reason, I had to use getLoginUser() in a Kerberized cluster because 
createRemoteUser() didn't work (RPC call got stuck in RetryInvocationHandler). 
I haven't been able to figure out what's missing, could be something minor
2) As you pointed out in a different comment, a single doAs() call should be 
sufficient for the whole LauncherAM, there's no need for separate calls

I opened a JIRA for these improvements: 
https://issues.apache.org/jira/browse/OOZIE-2909


- Peter


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/54705/#review174883
---


On máj. 10, 2017, 10:46 du, Peter Cseh wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/54705/
> ---
> 
> (Updated máj. 10, 2017, 10:46 du)
> 
> 
> Review request for oozie, Purshotam Shah, Robert Kanter, and Rohini 
> Palaniswamy.
> 
> 
> Bugs: OOZIE-1770
> https://issues.apache.org/jira/browse/OOZIE-1770
> 
> 
> Repository: oozie-git
> 
> 
> Description
> ---
> 
> Replacing the launcher MR job with a launcher AM
> 
> 
> Diffs
> -
> 
>   client/pom.xml f99366c90887a5a45f3942291c90e1cdbfd84b3f 
>   client/src/main/java/org/apache/oozie/cli/OozieCLI.java 
> 38fb84e8a8918aee6ae57d19a29cdf181aba1955 
>   client/src/main/java/org/apache/oozie/client/XOozieClient.java 
> 8538ec7c742cc013f7c18eaff3ecf61c2835cc8a 
>   core/pom.xml 14aa03483e107db326628ba627cc3ee9f5fd3f3d 
>   core/src/main/java/org/apache/oozie/WorkflowJobBean.java 
> 55d79a5d5e5e4efab9eb832e9341a16bcf8959cc 
>   core/src/main/java/org/apache/oozie/action/ActionExecutor.java 
> 1d6456b236bf9e27c0f55d9f9e3b4bd94517c797 
>   core/src/main/java/org/apache/oozie/action/hadoop/Credentials.java 
> eadb47b07d157f45f086517ab41c4b9826f003f0 
>   core/src/main/java/org/apache/oozie/action/hadoop/CredentialsProvider.java 
> 6fe22fbd2fca5d873d58ece60f20d0f131371475 
>   
> core/src/main/java/org/apache/oozie/action/hadoop/CredentialsProviderFactory.java
>  PRE-CREATION 
>   core/src/main/java/org/apache/oozie/action/hadoop/DistcpActionExecutor.java 
> 78cd257a836707c1efeac13566d6285d42970545 
>   core/src/main/java/org/apache/oozie/action/hadoop/FsActionExecutor.java 
> 27654741101d2056b25626fbbe345183eafd612d 
>   core/src/main/java/org/apache/oozie/action/hadoop/FsELFunctions.java 
> 801bfe6eb212b549888f0e9fb7df7b65bcf59393 
>   core/src/main/java/org/apache/oozie/action/hadoop/HCatCredentialHelper

Re: Review Request 54705: OOZIE-1770: Create Oozie Application Master for YARN

[Peter Bacsko]

> On máj. 14, 2017, 4:16 de, Rohini Palaniswamy wrote:
> > core/src/test/java/org/apache/oozie/action/hadoop/TestJavaActionExecutor.java
> > Line 546 (original)
> > 
> >
> > Why is this test removed? This is important.

Yes, it is - good catch. It's not obvious to me how this disappeared and why.
Created follow-up JIRA: https://issues.apache.org/jira/browse/OOZIE-2910


> On máj. 14, 2017, 4:16 de, Rohini Palaniswamy wrote:
> > core/src/test/java/org/apache/oozie/command/wf/TestWorkflowActionKillXCommand.java
> > Line 120 (original)
> > 
> >
> > Why remove this test? This is important.

Created follow-up JIRA: https://issues.apache.org/jira/browse/OOZIE-2911


> On máj. 14, 2017, 4:16 de, Rohini Palaniswamy wrote:
> > sharelib/oozie/src/main/java/org/apache/oozie/action/hadoop/HdfsOperations.java
> > Lines 55 (patched)
> > 
> >
> > Why are we doing ugi.doAs for all filesytem operations in this class? 
> > If they are all being called inside the launcher, ugi.doAs should not be 
> > required because Oozie AM should be running as the user ugi.  If it is 
> > broken for some reason then it is really bad as user calls in java action 
> > will also be impacted.

Yep, single doAs is enough -> OOZIE-2909


- Peter


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/54705/#review174883
---


On máj. 10, 2017, 10:46 du, Peter Cseh wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/54705/
> ---
> 
> (Updated máj. 10, 2017, 10:46 du)
> 
> 
> Review request for oozie, Purshotam Shah, Robert Kanter, and Rohini 
> Palaniswamy.
> 
> 
> Bugs: OOZIE-1770
> https://issues.apache.org/jira/browse/OOZIE-1770
> 
> 
> Repository: oozie-git
> 
> 
> Description
> ---
> 
> Replacing the launcher MR job with a launcher AM
> 
> 
> Diffs
> -
> 
>   client/pom.xml f99366c90887a5a45f3942291c90e1cdbfd84b3f 
>   client/src/main/java/org/apache/oozie/cli/OozieCLI.java 
> 38fb84e8a8918aee6ae57d19a29cdf181aba1955 
>   client/src/main/java/org/apache/oozie/client/XOozieClient.java 
> 8538ec7c742cc013f7c18eaff3ecf61c2835cc8a 
>   core/pom.xml 14aa03483e107db326628ba627cc3ee9f5fd3f3d 
>   core/src/main/java/org/apache/oozie/WorkflowJobBean.java 
> 55d79a5d5e5e4efab9eb832e9341a16bcf8959cc 
>   core/src/main/java/org/apache/oozie/action/ActionExecutor.java 
> 1d6456b236bf9e27c0f55d9f9e3b4bd94517c797 
>   core/src/main/java/org/apache/oozie/action/hadoop/Credentials.java 
> eadb47b07d157f45f086517ab41c4b9826f003f0 
>   core/src/main/java/org/apache/oozie/action/hadoop/CredentialsProvider.java 
> 6fe22fbd2fca5d873d58ece60f20d0f131371475 
>   
> core/src/main/java/org/apache/oozie/action/hadoop/CredentialsProviderFactory.java
>  PRE-CREATION 
>   core/src/main/java/org/apache/oozie/action/hadoop/DistcpActionExecutor.java 
> 78cd257a836707c1efeac13566d6285d42970545 
>   core/src/main/java/org/apache/oozie/action/hadoop/FsActionExecutor.java 
> 27654741101d2056b25626fbbe345183eafd612d 
>   core/src/main/java/org/apache/oozie/action/hadoop/FsELFunctions.java 
> 801bfe6eb212b549888f0e9fb7df7b65bcf59393 
>   core/src/main/java/org/apache/oozie/action/hadoop/HCatCredentialHelper.java 
> 36ad1df727e869b8051481ad969771e91bc6ae07 
>   core/src/main/java/org/apache/oozie/action/hadoop/HCatCredentials.java 
> d7689a924059e553abaeed374eed4072933292cb 
>   core/src/main/java/org/apache/oozie/action/hadoop/HadoopELFunctions.java 
> ad2a71d641caa71efc098ad00bfbab3cec4d9506 
>   core/src/main/java/org/apache/oozie/action/hadoop/HbaseCredentials.java 
> 307f565786ad77f5a35b5790bd5b052b04c6eac8 
>   core/src/main/java/org/apache/oozie/action/hadoop/Hive2ActionExecutor.java 
> 2aed936af75da38c433be5d202bdc01c2f612c76 
>   core/src/main/java/org/apache/oozie/action/hadoop/Hive2Credentials.java 
> bd2f1f768c5b04bef4a33613baede9202144c747 
>   core/src/main/java/org/apache/oozie/action/hadoop/HiveActionExecutor.java 
> 8e2453ef4f78f4a5f1f95b1ca2793f5d7e07ad81 
>   core/src/main/java/org/apache/oozie/action/hadoop/JavaActionExecutor.java 
> d60a5c7b996a4085f9bc77d82f12ad6925280b1a 
>   core/src/main/java/org/apache/oozie/action/hadoop/LauncherMapperHelper.java 
> 72ed2f150be73f0a5e8b71b0fb6cc3594dc23078 
>   
> core/src/main/java/org/apache/oozie/action/hadoop/MapReduceActionExecutor.java
>  55c937249ac35e14bd63dd0c1fa962ea636afa3a 
>   core/src/main/java/org/apache/oozie/action/hadoop/OozieJobInfo.java 
> 581d3b3219eff882f961c8e495611e873c6a5a52 
>   core/src/main/java/org/apache/oozie/action/hadoop/PigA

Re: Review Request 54705: OOZIE-1770: Create Oozie Application Master for YARN

[Peter Bacsko]

> On máj. 14, 2017, 4:16 de, Rohini Palaniswamy wrote:
> > core/src/test/java/org/apache/oozie/action/hadoop/TestJavaActionExecutor.java
> > Line 2110 (original)
> > 
> >
> > Instead of getting rid of the testUpdateConfForUberMode* tests 
> > altogether, it would be better to adapt them to the Oozie AM (once it is 
> > fixed to handle oozie.launcher configurations) as the tests are still 
> > valid. Comment them out for now with a TODO and address in the new jira 
> > after the merge.

https://issues.apache.org/jira/browse/OOZIE-2912


> On máj. 14, 2017, 4:16 de, Rohini Palaniswamy wrote:
> > core/src/test/java/org/apache/oozie/action/hadoop/TestShellActionExecutor.java
> > Line 436 (original)
> > 
> >
> > Please adapt this test to the Oozie AM instead of removing it similar 
> > to testUpdateConfForUberMode tests

https://issues.apache.org/jira/browse/OOZIE-2912


> On máj. 14, 2017, 4:16 de, Rohini Palaniswamy wrote:
> > sharelib/oozie/src/main/java/org/apache/oozie/action/hadoop/LauncherAMCallbackNotifier.java
> > Lines 39-44 (patched)
> > 
> >
> > Would be better to reuse mapreduce.job.end-notification. as prefix. 
> > Else will break backward compatibility and existing users will have to 
> > change if they have configured them.

This will be handled in OOZIE-2896


- Peter


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/54705/#review174883
---


On máj. 10, 2017, 10:46 du, Peter Cseh wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/54705/
> ---
> 
> (Updated máj. 10, 2017, 10:46 du)
> 
> 
> Review request for oozie, Purshotam Shah, Robert Kanter, and Rohini 
> Palaniswamy.
> 
> 
> Bugs: OOZIE-1770
> https://issues.apache.org/jira/browse/OOZIE-1770
> 
> 
> Repository: oozie-git
> 
> 
> Description
> ---
> 
> Replacing the launcher MR job with a launcher AM
> 
> 
> Diffs
> -
> 
>   client/pom.xml f99366c90887a5a45f3942291c90e1cdbfd84b3f 
>   client/src/main/java/org/apache/oozie/cli/OozieCLI.java 
> 38fb84e8a8918aee6ae57d19a29cdf181aba1955 
>   client/src/main/java/org/apache/oozie/client/XOozieClient.java 
> 8538ec7c742cc013f7c18eaff3ecf61c2835cc8a 
>   core/pom.xml 14aa03483e107db326628ba627cc3ee9f5fd3f3d 
>   core/src/main/java/org/apache/oozie/WorkflowJobBean.java 
> 55d79a5d5e5e4efab9eb832e9341a16bcf8959cc 
>   core/src/main/java/org/apache/oozie/action/ActionExecutor.java 
> 1d6456b236bf9e27c0f55d9f9e3b4bd94517c797 
>   core/src/main/java/org/apache/oozie/action/hadoop/Credentials.java 
> eadb47b07d157f45f086517ab41c4b9826f003f0 
>   core/src/main/java/org/apache/oozie/action/hadoop/CredentialsProvider.java 
> 6fe22fbd2fca5d873d58ece60f20d0f131371475 
>   
> core/src/main/java/org/apache/oozie/action/hadoop/CredentialsProviderFactory.java
>  PRE-CREATION 
>   core/src/main/java/org/apache/oozie/action/hadoop/DistcpActionExecutor.java 
> 78cd257a836707c1efeac13566d6285d42970545 
>   core/src/main/java/org/apache/oozie/action/hadoop/FsActionExecutor.java 
> 27654741101d2056b25626fbbe345183eafd612d 
>   core/src/main/java/org/apache/oozie/action/hadoop/FsELFunctions.java 
> 801bfe6eb212b549888f0e9fb7df7b65bcf59393 
>   core/src/main/java/org/apache/oozie/action/hadoop/HCatCredentialHelper.java 
> 36ad1df727e869b8051481ad969771e91bc6ae07 
>   core/src/main/java/org/apache/oozie/action/hadoop/HCatCredentials.java 
> d7689a924059e553abaeed374eed4072933292cb 
>   core/src/main/java/org/apache/oozie/action/hadoop/HadoopELFunctions.java 
> ad2a71d641caa71efc098ad00bfbab3cec4d9506 
>   core/src/main/java/org/apache/oozie/action/hadoop/HbaseCredentials.java 
> 307f565786ad77f5a35b5790bd5b052b04c6eac8 
>   core/src/main/java/org/apache/oozie/action/hadoop/Hive2ActionExecutor.java 
> 2aed936af75da38c433be5d202bdc01c2f612c76 
>   core/src/main/java/org/apache/oozie/action/hadoop/Hive2Credentials.java 
> bd2f1f768c5b04bef4a33613baede9202144c747 
>   core/src/main/java/org/apache/oozie/action/hadoop/HiveActionExecutor.java 
> 8e2453ef4f78f4a5f1f95b1ca2793f5d7e07ad81 
>   core/src/main/java/org/apache/oozie/action/hadoop/JavaActionExecutor.java 
> d60a5c7b996a4085f9bc77d82f12ad6925280b1a 
>   core/src/main/java/org/apache/oozie/action/hadoop/LauncherMapperHelper.java 
> 72ed2f150be73f0a5e8b71b0fb6cc3594dc23078 
>   
> core/src/main/java/org/apache/oozie/action/hadoop/MapReduceActionExecutor.java
>  55c937249ac35e14bd63dd0c1fa962ea636afa3a 
>   core/src/main/java/org/apache/oozie/action/hadoop/OozieJobInfo.java 
> 581d3b3219eff882f961c8e49

Re: Review Request 54705: OOZIE-1770: Create Oozie Application Master for YARN

[Peter Cseh]

> On May 14, 2017, 4:16 a.m., Rohini Palaniswamy wrote:
> > core/src/test/java/org/apache/oozie/action/hadoop/TestJavaActionExecutor.java
> > Line 2076 (original), 1791 (patched)
> > 
> >
> > -Xmx2048m should work if we are honoring the current configuration of 
> > users increasing launcher memory.

will be part of https://issues.apache.org/jira/browse/OOZIE-2896


- Peter


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/54705/#review174883
---


On May 10, 2017, 10:46 p.m., Peter Cseh wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/54705/
> ---
> 
> (Updated May 10, 2017, 10:46 p.m.)
> 
> 
> Review request for oozie, Purshotam Shah, Robert Kanter, and Rohini 
> Palaniswamy.
> 
> 
> Bugs: OOZIE-1770
> https://issues.apache.org/jira/browse/OOZIE-1770
> 
> 
> Repository: oozie-git
> 
> 
> Description
> ---
> 
> Replacing the launcher MR job with a launcher AM
> 
> 
> Diffs
> -
> 
>   client/pom.xml f99366c90887a5a45f3942291c90e1cdbfd84b3f 
>   client/src/main/java/org/apache/oozie/cli/OozieCLI.java 
> 38fb84e8a8918aee6ae57d19a29cdf181aba1955 
>   client/src/main/java/org/apache/oozie/client/XOozieClient.java 
> 8538ec7c742cc013f7c18eaff3ecf61c2835cc8a 
>   core/pom.xml 14aa03483e107db326628ba627cc3ee9f5fd3f3d 
>   core/src/main/java/org/apache/oozie/WorkflowJobBean.java 
> 55d79a5d5e5e4efab9eb832e9341a16bcf8959cc 
>   core/src/main/java/org/apache/oozie/action/ActionExecutor.java 
> 1d6456b236bf9e27c0f55d9f9e3b4bd94517c797 
>   core/src/main/java/org/apache/oozie/action/hadoop/Credentials.java 
> eadb47b07d157f45f086517ab41c4b9826f003f0 
>   core/src/main/java/org/apache/oozie/action/hadoop/CredentialsProvider.java 
> 6fe22fbd2fca5d873d58ece60f20d0f131371475 
>   
> core/src/main/java/org/apache/oozie/action/hadoop/CredentialsProviderFactory.java
>  PRE-CREATION 
>   core/src/main/java/org/apache/oozie/action/hadoop/DistcpActionExecutor.java 
> 78cd257a836707c1efeac13566d6285d42970545 
>   core/src/main/java/org/apache/oozie/action/hadoop/FsActionExecutor.java 
> 27654741101d2056b25626fbbe345183eafd612d 
>   core/src/main/java/org/apache/oozie/action/hadoop/FsELFunctions.java 
> 801bfe6eb212b549888f0e9fb7df7b65bcf59393 
>   core/src/main/java/org/apache/oozie/action/hadoop/HCatCredentialHelper.java 
> 36ad1df727e869b8051481ad969771e91bc6ae07 
>   core/src/main/java/org/apache/oozie/action/hadoop/HCatCredentials.java 
> d7689a924059e553abaeed374eed4072933292cb 
>   core/src/main/java/org/apache/oozie/action/hadoop/HadoopELFunctions.java 
> ad2a71d641caa71efc098ad00bfbab3cec4d9506 
>   core/src/main/java/org/apache/oozie/action/hadoop/HbaseCredentials.java 
> 307f565786ad77f5a35b5790bd5b052b04c6eac8 
>   core/src/main/java/org/apache/oozie/action/hadoop/Hive2ActionExecutor.java 
> 2aed936af75da38c433be5d202bdc01c2f612c76 
>   core/src/main/java/org/apache/oozie/action/hadoop/Hive2Credentials.java 
> bd2f1f768c5b04bef4a33613baede9202144c747 
>   core/src/main/java/org/apache/oozie/action/hadoop/HiveActionExecutor.java 
> 8e2453ef4f78f4a5f1f95b1ca2793f5d7e07ad81 
>   core/src/main/java/org/apache/oozie/action/hadoop/JavaActionExecutor.java 
> d60a5c7b996a4085f9bc77d82f12ad6925280b1a 
>   core/src/main/java/org/apache/oozie/action/hadoop/LauncherMapperHelper.java 
> 72ed2f150be73f0a5e8b71b0fb6cc3594dc23078 
>   
> core/src/main/java/org/apache/oozie/action/hadoop/MapReduceActionExecutor.java
>  55c937249ac35e14bd63dd0c1fa962ea636afa3a 
>   core/src/main/java/org/apache/oozie/action/hadoop/OozieJobInfo.java 
> 581d3b3219eff882f961c8e495611e873c6a5a52 
>   core/src/main/java/org/apache/oozie/action/hadoop/PigActionExecutor.java 
> 8b2dc16db80ad602911f1f64e60cfad41b198922 
>   
> core/src/main/java/org/apache/oozie/action/hadoop/ScriptLanguageActionExecutor.java
>  92e149d5cac6509893c8507b776263a38312a53b 
>   core/src/main/java/org/apache/oozie/action/hadoop/ShellActionExecutor.java 
> b9ffa7ab7ec87db8a76a3653c74974d296c8cd17 
>   core/src/main/java/org/apache/oozie/action/hadoop/SparkActionExecutor.java 
> 1a3197abb52f80b545dcc8634a8cac7a281a9eac 
>   core/src/main/java/org/apache/oozie/action/hadoop/SqoopActionExecutor.java 
> 22e28746a9a39ae6c1e22326cc4f8f759d820b20 
>   core/src/main/java/org/apache/oozie/client/rest/BulkResponseImpl.java 
> fb021bd022d199f93fc5674843514abf45ae12a5 
>   core/src/main/java/org/apache/oozie/command/XCommand.java 
> bdf13f65ea25e8e0f5125e2798568a2c03fffe81 
>   core/src/main/java/org/apache/oozie/command/wf/SubmitHttpXCommand.java 
> d2a2742ae418070a1cd0a889f14194224bc944a6 
>   core/src/main/java/org/apache/oozie/command/wf/SubmitMRXCommand.java 
> cc61d3de813c

Re: Review Request 54705: OOZIE-1770: Create Oozie Application Master for YARN

[Peter Cseh]

> On March 26, 2017, 12:37 a.m., Rohini Palaniswamy wrote:
> > core/src/main/java/org/apache/oozie/action/hadoop/SqoopActionExecutor.java
> > Line 89 (original), 87 (patched)
> > 
> >
> > Revert back to old code. Just remove the final instead of creating 
> > another temporary list.
> 
> Peter Cseh wrote:
> merge issue. will check
> 
> Rohini Palaniswamy wrote:
> You can totally avoid the array list to array conversion by using 
> String.split() instead of StringTokenizer. 
> 
> https://docs.oracle.com/javase/7/docs/api/java/util/StringTokenizer.html
> StringTokenizer is a legacy class that is retained for compatibility 
> reasons although its use is discouraged in new code.

This is not in the latest patch.


- Peter


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/54705/#review170061
---


On May 10, 2017, 10:46 p.m., Peter Cseh wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/54705/
> ---
> 
> (Updated May 10, 2017, 10:46 p.m.)
> 
> 
> Review request for oozie, Purshotam Shah, Robert Kanter, and Rohini 
> Palaniswamy.
> 
> 
> Bugs: OOZIE-1770
> https://issues.apache.org/jira/browse/OOZIE-1770
> 
> 
> Repository: oozie-git
> 
> 
> Description
> ---
> 
> Replacing the launcher MR job with a launcher AM
> 
> 
> Diffs
> -
> 
>   client/pom.xml f99366c90887a5a45f3942291c90e1cdbfd84b3f 
>   client/src/main/java/org/apache/oozie/cli/OozieCLI.java 
> 38fb84e8a8918aee6ae57d19a29cdf181aba1955 
>   client/src/main/java/org/apache/oozie/client/XOozieClient.java 
> 8538ec7c742cc013f7c18eaff3ecf61c2835cc8a 
>   core/pom.xml 14aa03483e107db326628ba627cc3ee9f5fd3f3d 
>   core/src/main/java/org/apache/oozie/WorkflowJobBean.java 
> 55d79a5d5e5e4efab9eb832e9341a16bcf8959cc 
>   core/src/main/java/org/apache/oozie/action/ActionExecutor.java 
> 1d6456b236bf9e27c0f55d9f9e3b4bd94517c797 
>   core/src/main/java/org/apache/oozie/action/hadoop/Credentials.java 
> eadb47b07d157f45f086517ab41c4b9826f003f0 
>   core/src/main/java/org/apache/oozie/action/hadoop/CredentialsProvider.java 
> 6fe22fbd2fca5d873d58ece60f20d0f131371475 
>   
> core/src/main/java/org/apache/oozie/action/hadoop/CredentialsProviderFactory.java
>  PRE-CREATION 
>   core/src/main/java/org/apache/oozie/action/hadoop/DistcpActionExecutor.java 
> 78cd257a836707c1efeac13566d6285d42970545 
>   core/src/main/java/org/apache/oozie/action/hadoop/FsActionExecutor.java 
> 27654741101d2056b25626fbbe345183eafd612d 
>   core/src/main/java/org/apache/oozie/action/hadoop/FsELFunctions.java 
> 801bfe6eb212b549888f0e9fb7df7b65bcf59393 
>   core/src/main/java/org/apache/oozie/action/hadoop/HCatCredentialHelper.java 
> 36ad1df727e869b8051481ad969771e91bc6ae07 
>   core/src/main/java/org/apache/oozie/action/hadoop/HCatCredentials.java 
> d7689a924059e553abaeed374eed4072933292cb 
>   core/src/main/java/org/apache/oozie/action/hadoop/HadoopELFunctions.java 
> ad2a71d641caa71efc098ad00bfbab3cec4d9506 
>   core/src/main/java/org/apache/oozie/action/hadoop/HbaseCredentials.java 
> 307f565786ad77f5a35b5790bd5b052b04c6eac8 
>   core/src/main/java/org/apache/oozie/action/hadoop/Hive2ActionExecutor.java 
> 2aed936af75da38c433be5d202bdc01c2f612c76 
>   core/src/main/java/org/apache/oozie/action/hadoop/Hive2Credentials.java 
> bd2f1f768c5b04bef4a33613baede9202144c747 
>   core/src/main/java/org/apache/oozie/action/hadoop/HiveActionExecutor.java 
> 8e2453ef4f78f4a5f1f95b1ca2793f5d7e07ad81 
>   core/src/main/java/org/apache/oozie/action/hadoop/JavaActionExecutor.java 
> d60a5c7b996a4085f9bc77d82f12ad6925280b1a 
>   core/src/main/java/org/apache/oozie/action/hadoop/LauncherMapperHelper.java 
> 72ed2f150be73f0a5e8b71b0fb6cc3594dc23078 
>   
> core/src/main/java/org/apache/oozie/action/hadoop/MapReduceActionExecutor.java
>  55c937249ac35e14bd63dd0c1fa962ea636afa3a 
>   core/src/main/java/org/apache/oozie/action/hadoop/OozieJobInfo.java 
> 581d3b3219eff882f961c8e495611e873c6a5a52 
>   core/src/main/java/org/apache/oozie/action/hadoop/PigActionExecutor.java 
> 8b2dc16db80ad602911f1f64e60cfad41b198922 
>   
> core/src/main/java/org/apache/oozie/action/hadoop/ScriptLanguageActionExecutor.java
>  92e149d5cac6509893c8507b776263a38312a53b 
>   core/src/main/java/org/apache/oozie/action/hadoop/ShellActionExecutor.java 
> b9ffa7ab7ec87db8a76a3653c74974d296c8cd17 
>   core/src/main/java/org/apache/oozie/action/hadoop/SparkActionExecutor.java 
> 1a3197abb52f80b545dcc8634a8cac7a281a9eac 
>   core/src/main/java/org/apache/oozie/action/hadoop/SqoopActionExecutor.java 
> 22e28746a9a39ae6c1e22326cc4f8f759d820b20 
>   core/src/main/java/org/apache/oozie/client/rest/BulkResponseI

Re: Review Request 54705: OOZIE-1770: Create Oozie Application Master for YARN

[Peter Cseh]

> On March 26, 2017, 12:37 a.m., Rohini Palaniswamy wrote:
> > core/src/main/java/org/apache/oozie/action/hadoop/SparkActionExecutor.java
> > Lines 112-114 (original)
> > 
> >
> > You can't remove this. Will breaking existing users. Since it starts 
> > with oozie.launcher, config deprecation will not handle it. Basically any 
> > oozie.launcher hadoop 1.x settings cannot be removed because of that.
> 
> Peter Cseh wrote:
> This is setting "mapreduce.task.classpath.user.precedence" in the 
> launcher job conf, it was parsed by the launcher Mapper and the LauncherAM is 
> not doing anything with this property at the moment. Should we add similar 
> functionality into LauncherAM? Is this a must have to support without MR 
> launcher?
> 
> Robert Kanter wrote:
> I'd rather not have to support an equivalent of 
> "mapreduce.task.classpath.user.precedence" in the Oozie AM itself.  It's a 
> really ugly hacky thing.  If we want to go down this route, a better fix 
> would be to provide complete classpath isolation somehow; but I think that 
> could be a future feature.
> 
> Rohini Palaniswamy wrote:
> We need to provide feature parity for both 
> mapreduce.task.classpath.user.precedence (simple hack, but very useful and 
> addressed most of cases) and 
> mapreduce.job.classloader/mapreduce.job.classloader.system.classes (this 
> provided the isolation in the mapreduce world). Please create a separate jira 
> for this issue that can be addressed after the merge. Have seen a lot of 
> users use this feature due to one conflict or another. One of the most common 
> cases that requires this is avro.

https://issues.apache.org/jira/browse/OOZIE-2891


- Peter


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/54705/#review170061
---


On May 10, 2017, 10:46 p.m., Peter Cseh wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/54705/
> ---
> 
> (Updated May 10, 2017, 10:46 p.m.)
> 
> 
> Review request for oozie, Purshotam Shah, Robert Kanter, and Rohini 
> Palaniswamy.
> 
> 
> Bugs: OOZIE-1770
> https://issues.apache.org/jira/browse/OOZIE-1770
> 
> 
> Repository: oozie-git
> 
> 
> Description
> ---
> 
> Replacing the launcher MR job with a launcher AM
> 
> 
> Diffs
> -
> 
>   client/pom.xml f99366c90887a5a45f3942291c90e1cdbfd84b3f 
>   client/src/main/java/org/apache/oozie/cli/OozieCLI.java 
> 38fb84e8a8918aee6ae57d19a29cdf181aba1955 
>   client/src/main/java/org/apache/oozie/client/XOozieClient.java 
> 8538ec7c742cc013f7c18eaff3ecf61c2835cc8a 
>   core/pom.xml 14aa03483e107db326628ba627cc3ee9f5fd3f3d 
>   core/src/main/java/org/apache/oozie/WorkflowJobBean.java 
> 55d79a5d5e5e4efab9eb832e9341a16bcf8959cc 
>   core/src/main/java/org/apache/oozie/action/ActionExecutor.java 
> 1d6456b236bf9e27c0f55d9f9e3b4bd94517c797 
>   core/src/main/java/org/apache/oozie/action/hadoop/Credentials.java 
> eadb47b07d157f45f086517ab41c4b9826f003f0 
>   core/src/main/java/org/apache/oozie/action/hadoop/CredentialsProvider.java 
> 6fe22fbd2fca5d873d58ece60f20d0f131371475 
>   
> core/src/main/java/org/apache/oozie/action/hadoop/CredentialsProviderFactory.java
>  PRE-CREATION 
>   core/src/main/java/org/apache/oozie/action/hadoop/DistcpActionExecutor.java 
> 78cd257a836707c1efeac13566d6285d42970545 
>   core/src/main/java/org/apache/oozie/action/hadoop/FsActionExecutor.java 
> 27654741101d2056b25626fbbe345183eafd612d 
>   core/src/main/java/org/apache/oozie/action/hadoop/FsELFunctions.java 
> 801bfe6eb212b549888f0e9fb7df7b65bcf59393 
>   core/src/main/java/org/apache/oozie/action/hadoop/HCatCredentialHelper.java 
> 36ad1df727e869b8051481ad969771e91bc6ae07 
>   core/src/main/java/org/apache/oozie/action/hadoop/HCatCredentials.java 
> d7689a924059e553abaeed374eed4072933292cb 
>   core/src/main/java/org/apache/oozie/action/hadoop/HadoopELFunctions.java 
> ad2a71d641caa71efc098ad00bfbab3cec4d9506 
>   core/src/main/java/org/apache/oozie/action/hadoop/HbaseCredentials.java 
> 307f565786ad77f5a35b5790bd5b052b04c6eac8 
>   core/src/main/java/org/apache/oozie/action/hadoop/Hive2ActionExecutor.java 
> 2aed936af75da38c433be5d202bdc01c2f612c76 
>   core/src/main/java/org/apache/oozie/action/hadoop/Hive2Credentials.java 
> bd2f1f768c5b04bef4a33613baede9202144c747 
>   core/src/main/java/org/apache/oozie/action/hadoop/HiveActionExecutor.java 
> 8e2453ef4f78f4a5f1f95b1ca2793f5d7e07ad81 
>   core/src/main/java/org/apache/oozie/action/hadoop/JavaActionExecutor.java 
> d60a5c7b996a4085f9bc77d82f12ad6925280b1a 
>   core/src/main/java/org/apache/oozie/action/hadoop/LauncherMapperHelper.java 
> 72ed2f150be73f0a5e8b71b0fb6cc3594dc23078 
>

Re: Review Request 54705: OOZIE-1770: Create Oozie Application Master for YARN

[Peter Cseh]

> On May 10, 2017, 11:54 p.m., Robert Kanter wrote:
> > core/src/main/java/org/apache/oozie/action/hadoop/DistcpActionExecutor.java
> > Line 47 (original)
> > 
> >
> > I think this is the only place where getClassNamesByType is called, in 
> > which case, we can also delete that.
> > 
> > While I can see that this code isn't currently doing anything, so it 
> > should be safe to remove, it would be good to figure out what it was 
> > supposed to be doing originally.

I'm not that familiar with Distcp so I don't know if removing this will cause 
issues or not. Created https://issues.apache.org/jira/browse/OOZIE-2913


- Peter


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/54705/#review174588
---


On May 10, 2017, 10:46 p.m., Peter Cseh wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/54705/
> ---
> 
> (Updated May 10, 2017, 10:46 p.m.)
> 
> 
> Review request for oozie, Purshotam Shah, Robert Kanter, and Rohini 
> Palaniswamy.
> 
> 
> Bugs: OOZIE-1770
> https://issues.apache.org/jira/browse/OOZIE-1770
> 
> 
> Repository: oozie-git
> 
> 
> Description
> ---
> 
> Replacing the launcher MR job with a launcher AM
> 
> 
> Diffs
> -
> 
>   client/pom.xml f99366c90887a5a45f3942291c90e1cdbfd84b3f 
>   client/src/main/java/org/apache/oozie/cli/OozieCLI.java 
> 38fb84e8a8918aee6ae57d19a29cdf181aba1955 
>   client/src/main/java/org/apache/oozie/client/XOozieClient.java 
> 8538ec7c742cc013f7c18eaff3ecf61c2835cc8a 
>   core/pom.xml 14aa03483e107db326628ba627cc3ee9f5fd3f3d 
>   core/src/main/java/org/apache/oozie/WorkflowJobBean.java 
> 55d79a5d5e5e4efab9eb832e9341a16bcf8959cc 
>   core/src/main/java/org/apache/oozie/action/ActionExecutor.java 
> 1d6456b236bf9e27c0f55d9f9e3b4bd94517c797 
>   core/src/main/java/org/apache/oozie/action/hadoop/Credentials.java 
> eadb47b07d157f45f086517ab41c4b9826f003f0 
>   core/src/main/java/org/apache/oozie/action/hadoop/CredentialsProvider.java 
> 6fe22fbd2fca5d873d58ece60f20d0f131371475 
>   
> core/src/main/java/org/apache/oozie/action/hadoop/CredentialsProviderFactory.java
>  PRE-CREATION 
>   core/src/main/java/org/apache/oozie/action/hadoop/DistcpActionExecutor.java 
> 78cd257a836707c1efeac13566d6285d42970545 
>   core/src/main/java/org/apache/oozie/action/hadoop/FsActionExecutor.java 
> 27654741101d2056b25626fbbe345183eafd612d 
>   core/src/main/java/org/apache/oozie/action/hadoop/FsELFunctions.java 
> 801bfe6eb212b549888f0e9fb7df7b65bcf59393 
>   core/src/main/java/org/apache/oozie/action/hadoop/HCatCredentialHelper.java 
> 36ad1df727e869b8051481ad969771e91bc6ae07 
>   core/src/main/java/org/apache/oozie/action/hadoop/HCatCredentials.java 
> d7689a924059e553abaeed374eed4072933292cb 
>   core/src/main/java/org/apache/oozie/action/hadoop/HadoopELFunctions.java 
> ad2a71d641caa71efc098ad00bfbab3cec4d9506 
>   core/src/main/java/org/apache/oozie/action/hadoop/HbaseCredentials.java 
> 307f565786ad77f5a35b5790bd5b052b04c6eac8 
>   core/src/main/java/org/apache/oozie/action/hadoop/Hive2ActionExecutor.java 
> 2aed936af75da38c433be5d202bdc01c2f612c76 
>   core/src/main/java/org/apache/oozie/action/hadoop/Hive2Credentials.java 
> bd2f1f768c5b04bef4a33613baede9202144c747 
>   core/src/main/java/org/apache/oozie/action/hadoop/HiveActionExecutor.java 
> 8e2453ef4f78f4a5f1f95b1ca2793f5d7e07ad81 
>   core/src/main/java/org/apache/oozie/action/hadoop/JavaActionExecutor.java 
> d60a5c7b996a4085f9bc77d82f12ad6925280b1a 
>   core/src/main/java/org/apache/oozie/action/hadoop/LauncherMapperHelper.java 
> 72ed2f150be73f0a5e8b71b0fb6cc3594dc23078 
>   
> core/src/main/java/org/apache/oozie/action/hadoop/MapReduceActionExecutor.java
>  55c937249ac35e14bd63dd0c1fa962ea636afa3a 
>   core/src/main/java/org/apache/oozie/action/hadoop/OozieJobInfo.java 
> 581d3b3219eff882f961c8e495611e873c6a5a52 
>   core/src/main/java/org/apache/oozie/action/hadoop/PigActionExecutor.java 
> 8b2dc16db80ad602911f1f64e60cfad41b198922 
>   
> core/src/main/java/org/apache/oozie/action/hadoop/ScriptLanguageActionExecutor.java
>  92e149d5cac6509893c8507b776263a38312a53b 
>   core/src/main/java/org/apache/oozie/action/hadoop/ShellActionExecutor.java 
> b9ffa7ab7ec87db8a76a3653c74974d296c8cd17 
>   core/src/main/java/org/apache/oozie/action/hadoop/SparkActionExecutor.java 
> 1a3197abb52f80b545dcc8634a8cac7a281a9eac 
>   core/src/main/java/org/apache/oozie/action/hadoop/SqoopActionExecutor.java 
> 22e28746a9a39ae6c1e22326cc4f8f759d820b20 
>   core/src/main/java/org/apache/oozie/client/rest/BulkResponseImpl.java 
> fb021bd022d199f93fc5674843514abf45ae12a5 
>   core/src/main/java/org/apache/oozie/command/XComm

Re: Review Request 54705: OOZIE-1770: Create Oozie Application Master for YARN

[Peter Cseh]

> On May 10, 2017, 11:54 p.m., Robert Kanter wrote:
> > core/src/main/java/org/apache/oozie/action/hadoop/CredentialsProviderFactory.java
> > Lines 108 (patched)
> > 
> >
> > I imagine there's one of these in some Commons library or something?  
> > At the very least, this should go into a Utils class.
> 
> Peter Cseh wrote:
> This is comming from here: 
> https://github.com/apache/oozie/blame/master/core/src/main/java/org/apache/oozie/action/hadoop/CredentialsProvider.java#L91
>  
> We'll look for a better place for it.

We've found this piece of code in DistCpActionExecutor as well.
 OOZIE-2914 - Consolidate Trim


- Peter


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/54705/#review174588
---


On May 10, 2017, 10:46 p.m., Peter Cseh wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/54705/
> ---
> 
> (Updated May 10, 2017, 10:46 p.m.)
> 
> 
> Review request for oozie, Purshotam Shah, Robert Kanter, and Rohini 
> Palaniswamy.
> 
> 
> Bugs: OOZIE-1770
> https://issues.apache.org/jira/browse/OOZIE-1770
> 
> 
> Repository: oozie-git
> 
> 
> Description
> ---
> 
> Replacing the launcher MR job with a launcher AM
> 
> 
> Diffs
> -
> 
>   client/pom.xml f99366c90887a5a45f3942291c90e1cdbfd84b3f 
>   client/src/main/java/org/apache/oozie/cli/OozieCLI.java 
> 38fb84e8a8918aee6ae57d19a29cdf181aba1955 
>   client/src/main/java/org/apache/oozie/client/XOozieClient.java 
> 8538ec7c742cc013f7c18eaff3ecf61c2835cc8a 
>   core/pom.xml 14aa03483e107db326628ba627cc3ee9f5fd3f3d 
>   core/src/main/java/org/apache/oozie/WorkflowJobBean.java 
> 55d79a5d5e5e4efab9eb832e9341a16bcf8959cc 
>   core/src/main/java/org/apache/oozie/action/ActionExecutor.java 
> 1d6456b236bf9e27c0f55d9f9e3b4bd94517c797 
>   core/src/main/java/org/apache/oozie/action/hadoop/Credentials.java 
> eadb47b07d157f45f086517ab41c4b9826f003f0 
>   core/src/main/java/org/apache/oozie/action/hadoop/CredentialsProvider.java 
> 6fe22fbd2fca5d873d58ece60f20d0f131371475 
>   
> core/src/main/java/org/apache/oozie/action/hadoop/CredentialsProviderFactory.java
>  PRE-CREATION 
>   core/src/main/java/org/apache/oozie/action/hadoop/DistcpActionExecutor.java 
> 78cd257a836707c1efeac13566d6285d42970545 
>   core/src/main/java/org/apache/oozie/action/hadoop/FsActionExecutor.java 
> 27654741101d2056b25626fbbe345183eafd612d 
>   core/src/main/java/org/apache/oozie/action/hadoop/FsELFunctions.java 
> 801bfe6eb212b549888f0e9fb7df7b65bcf59393 
>   core/src/main/java/org/apache/oozie/action/hadoop/HCatCredentialHelper.java 
> 36ad1df727e869b8051481ad969771e91bc6ae07 
>   core/src/main/java/org/apache/oozie/action/hadoop/HCatCredentials.java 
> d7689a924059e553abaeed374eed4072933292cb 
>   core/src/main/java/org/apache/oozie/action/hadoop/HadoopELFunctions.java 
> ad2a71d641caa71efc098ad00bfbab3cec4d9506 
>   core/src/main/java/org/apache/oozie/action/hadoop/HbaseCredentials.java 
> 307f565786ad77f5a35b5790bd5b052b04c6eac8 
>   core/src/main/java/org/apache/oozie/action/hadoop/Hive2ActionExecutor.java 
> 2aed936af75da38c433be5d202bdc01c2f612c76 
>   core/src/main/java/org/apache/oozie/action/hadoop/Hive2Credentials.java 
> bd2f1f768c5b04bef4a33613baede9202144c747 
>   core/src/main/java/org/apache/oozie/action/hadoop/HiveActionExecutor.java 
> 8e2453ef4f78f4a5f1f95b1ca2793f5d7e07ad81 
>   core/src/main/java/org/apache/oozie/action/hadoop/JavaActionExecutor.java 
> d60a5c7b996a4085f9bc77d82f12ad6925280b1a 
>   core/src/main/java/org/apache/oozie/action/hadoop/LauncherMapperHelper.java 
> 72ed2f150be73f0a5e8b71b0fb6cc3594dc23078 
>   
> core/src/main/java/org/apache/oozie/action/hadoop/MapReduceActionExecutor.java
>  55c937249ac35e14bd63dd0c1fa962ea636afa3a 
>   core/src/main/java/org/apache/oozie/action/hadoop/OozieJobInfo.java 
> 581d3b3219eff882f961c8e495611e873c6a5a52 
>   core/src/main/java/org/apache/oozie/action/hadoop/PigActionExecutor.java 
> 8b2dc16db80ad602911f1f64e60cfad41b198922 
>   
> core/src/main/java/org/apache/oozie/action/hadoop/ScriptLanguageActionExecutor.java
>  92e149d5cac6509893c8507b776263a38312a53b 
>   core/src/main/java/org/apache/oozie/action/hadoop/ShellActionExecutor.java 
> b9ffa7ab7ec87db8a76a3653c74974d296c8cd17 
>   core/src/main/java/org/apache/oozie/action/hadoop/SparkActionExecutor.java 
> 1a3197abb52f80b545dcc8634a8cac7a281a9eac 
>   core/src/main/java/org/apache/oozie/action/hadoop/SqoopActionExecutor.java 
> 22e28746a9a39ae6c1e22326cc4f8f759d820b20 
>   core/src/main/java/org/apache/oozie/client/rest/BulkResponseImpl.java 
> fb021bd022d199f93fc5674843514abf45ae12a5 
>   core/src/main/java/org/apache/oozie/command/XCom

Re: Review Request 54705: OOZIE-1770: Create Oozie Application Master for YARN

[Peter Cseh]

> On May 14, 2017, 4:16 a.m., Rohini Palaniswamy wrote:
> > core/src/main/java/org/apache/oozie/command/XCommand.java
> > Line 247 (original), 247 (patched)
> > 
> >
> > Unnecessary thread local call when there are no commands to be queued. 
> > Can we make getInterruptTypes as static method in CallableQueueService 
> > keeping the HashSet as private static?

The set is only populated when CallableQueueService.init is called on an 
instance. I would prefer to keep it an instance variable so we can't call it 
when the service is not initialized properly. I know that it's a rare case but 
calling a map.get() should not result in performance degradation.


- Peter


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/54705/#review174883
---


On May 10, 2017, 10:46 p.m., Peter Cseh wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/54705/
> ---
> 
> (Updated May 10, 2017, 10:46 p.m.)
> 
> 
> Review request for oozie, Purshotam Shah, Robert Kanter, and Rohini 
> Palaniswamy.
> 
> 
> Bugs: OOZIE-1770
> https://issues.apache.org/jira/browse/OOZIE-1770
> 
> 
> Repository: oozie-git
> 
> 
> Description
> ---
> 
> Replacing the launcher MR job with a launcher AM
> 
> 
> Diffs
> -
> 
>   client/pom.xml f99366c90887a5a45f3942291c90e1cdbfd84b3f 
>   client/src/main/java/org/apache/oozie/cli/OozieCLI.java 
> 38fb84e8a8918aee6ae57d19a29cdf181aba1955 
>   client/src/main/java/org/apache/oozie/client/XOozieClient.java 
> 8538ec7c742cc013f7c18eaff3ecf61c2835cc8a 
>   core/pom.xml 14aa03483e107db326628ba627cc3ee9f5fd3f3d 
>   core/src/main/java/org/apache/oozie/WorkflowJobBean.java 
> 55d79a5d5e5e4efab9eb832e9341a16bcf8959cc 
>   core/src/main/java/org/apache/oozie/action/ActionExecutor.java 
> 1d6456b236bf9e27c0f55d9f9e3b4bd94517c797 
>   core/src/main/java/org/apache/oozie/action/hadoop/Credentials.java 
> eadb47b07d157f45f086517ab41c4b9826f003f0 
>   core/src/main/java/org/apache/oozie/action/hadoop/CredentialsProvider.java 
> 6fe22fbd2fca5d873d58ece60f20d0f131371475 
>   
> core/src/main/java/org/apache/oozie/action/hadoop/CredentialsProviderFactory.java
>  PRE-CREATION 
>   core/src/main/java/org/apache/oozie/action/hadoop/DistcpActionExecutor.java 
> 78cd257a836707c1efeac13566d6285d42970545 
>   core/src/main/java/org/apache/oozie/action/hadoop/FsActionExecutor.java 
> 27654741101d2056b25626fbbe345183eafd612d 
>   core/src/main/java/org/apache/oozie/action/hadoop/FsELFunctions.java 
> 801bfe6eb212b549888f0e9fb7df7b65bcf59393 
>   core/src/main/java/org/apache/oozie/action/hadoop/HCatCredentialHelper.java 
> 36ad1df727e869b8051481ad969771e91bc6ae07 
>   core/src/main/java/org/apache/oozie/action/hadoop/HCatCredentials.java 
> d7689a924059e553abaeed374eed4072933292cb 
>   core/src/main/java/org/apache/oozie/action/hadoop/HadoopELFunctions.java 
> ad2a71d641caa71efc098ad00bfbab3cec4d9506 
>   core/src/main/java/org/apache/oozie/action/hadoop/HbaseCredentials.java 
> 307f565786ad77f5a35b5790bd5b052b04c6eac8 
>   core/src/main/java/org/apache/oozie/action/hadoop/Hive2ActionExecutor.java 
> 2aed936af75da38c433be5d202bdc01c2f612c76 
>   core/src/main/java/org/apache/oozie/action/hadoop/Hive2Credentials.java 
> bd2f1f768c5b04bef4a33613baede9202144c747 
>   core/src/main/java/org/apache/oozie/action/hadoop/HiveActionExecutor.java 
> 8e2453ef4f78f4a5f1f95b1ca2793f5d7e07ad81 
>   core/src/main/java/org/apache/oozie/action/hadoop/JavaActionExecutor.java 
> d60a5c7b996a4085f9bc77d82f12ad6925280b1a 
>   core/src/main/java/org/apache/oozie/action/hadoop/LauncherMapperHelper.java 
> 72ed2f150be73f0a5e8b71b0fb6cc3594dc23078 
>   
> core/src/main/java/org/apache/oozie/action/hadoop/MapReduceActionExecutor.java
>  55c937249ac35e14bd63dd0c1fa962ea636afa3a 
>   core/src/main/java/org/apache/oozie/action/hadoop/OozieJobInfo.java 
> 581d3b3219eff882f961c8e495611e873c6a5a52 
>   core/src/main/java/org/apache/oozie/action/hadoop/PigActionExecutor.java 
> 8b2dc16db80ad602911f1f64e60cfad41b198922 
>   
> core/src/main/java/org/apache/oozie/action/hadoop/ScriptLanguageActionExecutor.java
>  92e149d5cac6509893c8507b776263a38312a53b 
>   core/src/main/java/org/apache/oozie/action/hadoop/ShellActionExecutor.java 
> b9ffa7ab7ec87db8a76a3653c74974d296c8cd17 
>   core/src/main/java/org/apache/oozie/action/hadoop/SparkActionExecutor.java 
> 1a3197abb52f80b545dcc8634a8cac7a281a9eac 
>   core/src/main/java/org/apache/oozie/action/hadoop/SqoopActionExecutor.java 
> 22e28746a9a39ae6c1e22326cc4f8f759d820b20 
>   core/src/main/java/org/apache/oozie/client/rest/BulkResponseImpl.java 
> fb021bd022d199f93fc5674843514abf45ae12a5 
>   core/src/main/java/

Re: Review Request 54705: OOZIE-1770: Create Oozie Application Master for YARN

[Peter Bacsko]

> On máj. 14, 2017, 4:16 de, Rohini Palaniswamy wrote:
> > core/src/main/java/org/apache/oozie/action/hadoop/MapReduceActionExecutor.java
> > Lines 413 (patched)
> > 
> >
> > This code will have race condition issues. super.check() will check the 
> > status of launcher AM if action data file is not found. While it starts the 
> > check, if it completes then it will read the NEW_ID from action data and 
> > mark the action as completed even though the mapreduce job is just launched 
> > and is still running.

That's right. I think the whole logic inside the check() methods have to be 
rewritten and refactored properly. It's too convoluted and responsibilities are 
in the wrong place.

Created follow-up JIRA: https://issues.apache.org/jira/browse/OOZIE-2915


- Peter


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/54705/#review174883
---


On máj. 10, 2017, 10:46 du, Peter Cseh wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/54705/
> ---
> 
> (Updated máj. 10, 2017, 10:46 du)
> 
> 
> Review request for oozie, Purshotam Shah, Robert Kanter, and Rohini 
> Palaniswamy.
> 
> 
> Bugs: OOZIE-1770
> https://issues.apache.org/jira/browse/OOZIE-1770
> 
> 
> Repository: oozie-git
> 
> 
> Description
> ---
> 
> Replacing the launcher MR job with a launcher AM
> 
> 
> Diffs
> -
> 
>   client/pom.xml f99366c90887a5a45f3942291c90e1cdbfd84b3f 
>   client/src/main/java/org/apache/oozie/cli/OozieCLI.java 
> 38fb84e8a8918aee6ae57d19a29cdf181aba1955 
>   client/src/main/java/org/apache/oozie/client/XOozieClient.java 
> 8538ec7c742cc013f7c18eaff3ecf61c2835cc8a 
>   core/pom.xml 14aa03483e107db326628ba627cc3ee9f5fd3f3d 
>   core/src/main/java/org/apache/oozie/WorkflowJobBean.java 
> 55d79a5d5e5e4efab9eb832e9341a16bcf8959cc 
>   core/src/main/java/org/apache/oozie/action/ActionExecutor.java 
> 1d6456b236bf9e27c0f55d9f9e3b4bd94517c797 
>   core/src/main/java/org/apache/oozie/action/hadoop/Credentials.java 
> eadb47b07d157f45f086517ab41c4b9826f003f0 
>   core/src/main/java/org/apache/oozie/action/hadoop/CredentialsProvider.java 
> 6fe22fbd2fca5d873d58ece60f20d0f131371475 
>   
> core/src/main/java/org/apache/oozie/action/hadoop/CredentialsProviderFactory.java
>  PRE-CREATION 
>   core/src/main/java/org/apache/oozie/action/hadoop/DistcpActionExecutor.java 
> 78cd257a836707c1efeac13566d6285d42970545 
>   core/src/main/java/org/apache/oozie/action/hadoop/FsActionExecutor.java 
> 27654741101d2056b25626fbbe345183eafd612d 
>   core/src/main/java/org/apache/oozie/action/hadoop/FsELFunctions.java 
> 801bfe6eb212b549888f0e9fb7df7b65bcf59393 
>   core/src/main/java/org/apache/oozie/action/hadoop/HCatCredentialHelper.java 
> 36ad1df727e869b8051481ad969771e91bc6ae07 
>   core/src/main/java/org/apache/oozie/action/hadoop/HCatCredentials.java 
> d7689a924059e553abaeed374eed4072933292cb 
>   core/src/main/java/org/apache/oozie/action/hadoop/HadoopELFunctions.java 
> ad2a71d641caa71efc098ad00bfbab3cec4d9506 
>   core/src/main/java/org/apache/oozie/action/hadoop/HbaseCredentials.java 
> 307f565786ad77f5a35b5790bd5b052b04c6eac8 
>   core/src/main/java/org/apache/oozie/action/hadoop/Hive2ActionExecutor.java 
> 2aed936af75da38c433be5d202bdc01c2f612c76 
>   core/src/main/java/org/apache/oozie/action/hadoop/Hive2Credentials.java 
> bd2f1f768c5b04bef4a33613baede9202144c747 
>   core/src/main/java/org/apache/oozie/action/hadoop/HiveActionExecutor.java 
> 8e2453ef4f78f4a5f1f95b1ca2793f5d7e07ad81 
>   core/src/main/java/org/apache/oozie/action/hadoop/JavaActionExecutor.java 
> d60a5c7b996a4085f9bc77d82f12ad6925280b1a 
>   core/src/main/java/org/apache/oozie/action/hadoop/LauncherMapperHelper.java 
> 72ed2f150be73f0a5e8b71b0fb6cc3594dc23078 
>   
> core/src/main/java/org/apache/oozie/action/hadoop/MapReduceActionExecutor.java
>  55c937249ac35e14bd63dd0c1fa962ea636afa3a 
>   core/src/main/java/org/apache/oozie/action/hadoop/OozieJobInfo.java 
> 581d3b3219eff882f961c8e495611e873c6a5a52 
>   core/src/main/java/org/apache/oozie/action/hadoop/PigActionExecutor.java 
> 8b2dc16db80ad602911f1f64e60cfad41b198922 
>   
> core/src/main/java/org/apache/oozie/action/hadoop/ScriptLanguageActionExecutor.java
>  92e149d5cac6509893c8507b776263a38312a53b 
>   core/src/main/java/org/apache/oozie/action/hadoop/ShellActionExecutor.java 
> b9ffa7ab7ec87db8a76a3653c74974d296c8cd17 
>   core/src/main/java/org/apache/oozie/action/hadoop/SparkActionExecutor.java 
> 1a3197abb52f80b545dcc8634a8cac7a281a9eac 
>   core/src/main/java/org/apache/oozie/action/hadoop/SqoopActionExecutor.java 
> 22e28746a9a39ae6c1e22326cc4f8f759d820b20 
>   core/src/main/java/org/apache/ooz

Re: Review Request 54705: OOZIE-1770: Create Oozie Application Master for YARN

[Peter Cseh]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/54705/
---

(Updated May 20, 2017, 11:55 a.m.)


Review request for oozie, Purshotam Shah, Robert Kanter, and Rohini Palaniswamy.


Bugs: OOZIE-1770
https://issues.apache.org/jira/browse/OOZIE-1770


Repository: oozie-git


Description
---

Replacing the launcher MR job with a launcher AM


Diffs (updated)
-

  client/pom.xml f99366c90887a5a45f3942291c90e1cdbfd84b3f 
  client/src/main/java/org/apache/oozie/cli/OozieCLI.java 
38fb84e8a8918aee6ae57d19a29cdf181aba1955 
  client/src/main/java/org/apache/oozie/client/XOozieClient.java 
8538ec7c742cc013f7c18eaff3ecf61c2835cc8a 
  core/pom.xml 14aa03483e107db326628ba627cc3ee9f5fd3f3d 
  core/src/main/java/org/apache/oozie/WorkflowJobBean.java 
55d79a5d5e5e4efab9eb832e9341a16bcf8959cc 
  core/src/main/java/org/apache/oozie/action/ActionExecutor.java 
1d6456b236bf9e27c0f55d9f9e3b4bd94517c797 
  core/src/main/java/org/apache/oozie/action/hadoop/Credentials.java 
eadb47b07d157f45f086517ab41c4b9826f003f0 
  core/src/main/java/org/apache/oozie/action/hadoop/CredentialsProvider.java 
6fe22fbd2fca5d873d58ece60f20d0f131371475 
  
core/src/main/java/org/apache/oozie/action/hadoop/CredentialsProviderFactory.java
 PRE-CREATION 
  core/src/main/java/org/apache/oozie/action/hadoop/DistcpActionExecutor.java 
78cd257a836707c1efeac13566d6285d42970545 
  core/src/main/java/org/apache/oozie/action/hadoop/FsActionExecutor.java 
27654741101d2056b25626fbbe345183eafd612d 
  core/src/main/java/org/apache/oozie/action/hadoop/FsELFunctions.java 
801bfe6eb212b549888f0e9fb7df7b65bcf59393 
  core/src/main/java/org/apache/oozie/action/hadoop/HCatCredentialHelper.java 
36ad1df727e869b8051481ad969771e91bc6ae07 
  core/src/main/java/org/apache/oozie/action/hadoop/HCatCredentials.java 
d7689a924059e553abaeed374eed4072933292cb 
  core/src/main/java/org/apache/oozie/action/hadoop/HadoopELFunctions.java 
ad2a71d641caa71efc098ad00bfbab3cec4d9506 
  core/src/main/java/org/apache/oozie/action/hadoop/HbaseCredentials.java 
307f565786ad77f5a35b5790bd5b052b04c6eac8 
  core/src/main/java/org/apache/oozie/action/hadoop/Hive2ActionExecutor.java 
2aed936af75da38c433be5d202bdc01c2f612c76 
  core/src/main/java/org/apache/oozie/action/hadoop/Hive2Credentials.java 
bd2f1f768c5b04bef4a33613baede9202144c747 
  core/src/main/java/org/apache/oozie/action/hadoop/HiveActionExecutor.java 
8e2453ef4f78f4a5f1f95b1ca2793f5d7e07ad81 
  core/src/main/java/org/apache/oozie/action/hadoop/JavaActionExecutor.java 
06ae5fd9a52099c5bc63b038d336ce96103c8819 
  core/src/main/java/org/apache/oozie/action/hadoop/LauncherMapperHelper.java 
72ed2f150be73f0a5e8b71b0fb6cc3594dc23078 
  
core/src/main/java/org/apache/oozie/action/hadoop/MapReduceActionExecutor.java 
55c937249ac35e14bd63dd0c1fa962ea636afa3a 
  core/src/main/java/org/apache/oozie/action/hadoop/OozieJobInfo.java 
581d3b3219eff882f961c8e495611e873c6a5a52 
  core/src/main/java/org/apache/oozie/action/hadoop/PigActionExecutor.java 
8b2dc16db80ad602911f1f64e60cfad41b198922 
  
core/src/main/java/org/apache/oozie/action/hadoop/ScriptLanguageActionExecutor.java
 92e149d5cac6509893c8507b776263a38312a53b 
  core/src/main/java/org/apache/oozie/action/hadoop/ShellActionExecutor.java 
b9ffa7ab7ec87db8a76a3653c74974d296c8cd17 
  core/src/main/java/org/apache/oozie/action/hadoop/SparkActionExecutor.java 
1a3197abb52f80b545dcc8634a8cac7a281a9eac 
  core/src/main/java/org/apache/oozie/action/hadoop/SqoopActionExecutor.java 
22e28746a9a39ae6c1e22326cc4f8f759d820b20 
  core/src/main/java/org/apache/oozie/client/rest/BulkResponseImpl.java 
fb021bd022d199f93fc5674843514abf45ae12a5 
  core/src/main/java/org/apache/oozie/command/XCommand.java 
bdf13f65ea25e8e0f5125e2798568a2c03fffe81 
  core/src/main/java/org/apache/oozie/command/wf/SubmitHttpXCommand.java 
d2a2742ae418070a1cd0a889f14194224bc944a6 
  core/src/main/java/org/apache/oozie/command/wf/SubmitMRXCommand.java 
cc61d3de813c75c7f1cf4d405fc043253e3ca4ea 
  
core/src/main/java/org/apache/oozie/command/wf/SubmitScriptLanguageXCommand.java
 9d413059e99bc676feb63e02a2f704048eb4a355 
  core/src/main/java/org/apache/oozie/command/wf/SubmitSqoopXCommand.java 
51b739e21c802e09f4110b740138f4ff5c341fe4 
  
core/src/main/java/org/apache/oozie/coord/input/dependency/AbstractCoordInputDependency.java
 ace120d10e3546201aa4145df56abf7933fb24b8 
  
core/src/main/java/org/apache/oozie/dependency/hcat/EhcacheHCatDependencyCache.java
 6f0abf6e094ed04158652191f2b8bb3b0cd32e90 
  core/src/main/java/org/apache/oozie/service/CallableQueueService.java 
a86a8d0889138cd1564ca9253e5b319ea52e55cc 
  core/src/main/java/org/apache/oozie/service/EventHandlerService.java 
22c6fb0965f45c5cf2322c251d4a8aa77d263458 
  core/src/main/java/org/apache/oozie/service/HadoopAccessorService.java 
23a9d92bf56a625469ec247cfb9727972511b101 
  core/src/main/java/org/apache

Re: Review Request 54705: OOZIE-1770: Create Oozie Application Master for YARN

[Rohini Palaniswamy]

> On May 14, 2017, 4:16 a.m., Rohini Palaniswamy wrote:
> > core/src/main/java/org/apache/oozie/action/hadoop/JavaActionExecutor.java
> > Line 423 (original), 321 (patched)
> > 
> >
> > This should not be required anymore. The Oozie AM code should be doing 
> > the same thing as done in MAPREDUCE-6472.
> 
> Peter Bacsko wrote:
> Right - deleted

Code to be added to Oozie AM is still not done. Refer 
https://issues.apache.org/jira/secure/attachment/12755788/MAPREDUCE-6472.20150914-2.patch


- Rohini


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/54705/#review174883
---


On May 20, 2017, 11:55 a.m., Peter Cseh wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/54705/
> ---
> 
> (Updated May 20, 2017, 11:55 a.m.)
> 
> 
> Review request for oozie, Purshotam Shah, Robert Kanter, and Rohini 
> Palaniswamy.
> 
> 
> Bugs: OOZIE-1770
> https://issues.apache.org/jira/browse/OOZIE-1770
> 
> 
> Repository: oozie-git
> 
> 
> Description
> ---
> 
> Replacing the launcher MR job with a launcher AM
> 
> 
> Diffs
> -
> 
>   client/pom.xml f99366c90887a5a45f3942291c90e1cdbfd84b3f 
>   client/src/main/java/org/apache/oozie/cli/OozieCLI.java 
> 38fb84e8a8918aee6ae57d19a29cdf181aba1955 
>   client/src/main/java/org/apache/oozie/client/XOozieClient.java 
> 8538ec7c742cc013f7c18eaff3ecf61c2835cc8a 
>   core/pom.xml 14aa03483e107db326628ba627cc3ee9f5fd3f3d 
>   core/src/main/java/org/apache/oozie/WorkflowJobBean.java 
> 55d79a5d5e5e4efab9eb832e9341a16bcf8959cc 
>   core/src/main/java/org/apache/oozie/action/ActionExecutor.java 
> 1d6456b236bf9e27c0f55d9f9e3b4bd94517c797 
>   core/src/main/java/org/apache/oozie/action/hadoop/Credentials.java 
> eadb47b07d157f45f086517ab41c4b9826f003f0 
>   core/src/main/java/org/apache/oozie/action/hadoop/CredentialsProvider.java 
> 6fe22fbd2fca5d873d58ece60f20d0f131371475 
>   
> core/src/main/java/org/apache/oozie/action/hadoop/CredentialsProviderFactory.java
>  PRE-CREATION 
>   core/src/main/java/org/apache/oozie/action/hadoop/DistcpActionExecutor.java 
> 78cd257a836707c1efeac13566d6285d42970545 
>   core/src/main/java/org/apache/oozie/action/hadoop/FsActionExecutor.java 
> 27654741101d2056b25626fbbe345183eafd612d 
>   core/src/main/java/org/apache/oozie/action/hadoop/FsELFunctions.java 
> 801bfe6eb212b549888f0e9fb7df7b65bcf59393 
>   core/src/main/java/org/apache/oozie/action/hadoop/HCatCredentialHelper.java 
> 36ad1df727e869b8051481ad969771e91bc6ae07 
>   core/src/main/java/org/apache/oozie/action/hadoop/HCatCredentials.java 
> d7689a924059e553abaeed374eed4072933292cb 
>   core/src/main/java/org/apache/oozie/action/hadoop/HadoopELFunctions.java 
> ad2a71d641caa71efc098ad00bfbab3cec4d9506 
>   core/src/main/java/org/apache/oozie/action/hadoop/HbaseCredentials.java 
> 307f565786ad77f5a35b5790bd5b052b04c6eac8 
>   core/src/main/java/org/apache/oozie/action/hadoop/Hive2ActionExecutor.java 
> 2aed936af75da38c433be5d202bdc01c2f612c76 
>   core/src/main/java/org/apache/oozie/action/hadoop/Hive2Credentials.java 
> bd2f1f768c5b04bef4a33613baede9202144c747 
>   core/src/main/java/org/apache/oozie/action/hadoop/HiveActionExecutor.java 
> 8e2453ef4f78f4a5f1f95b1ca2793f5d7e07ad81 
>   core/src/main/java/org/apache/oozie/action/hadoop/JavaActionExecutor.java 
> 06ae5fd9a52099c5bc63b038d336ce96103c8819 
>   core/src/main/java/org/apache/oozie/action/hadoop/LauncherMapperHelper.java 
> 72ed2f150be73f0a5e8b71b0fb6cc3594dc23078 
>   
> core/src/main/java/org/apache/oozie/action/hadoop/MapReduceActionExecutor.java
>  55c937249ac35e14bd63dd0c1fa962ea636afa3a 
>   core/src/main/java/org/apache/oozie/action/hadoop/OozieJobInfo.java 
> 581d3b3219eff882f961c8e495611e873c6a5a52 
>   core/src/main/java/org/apache/oozie/action/hadoop/PigActionExecutor.java 
> 8b2dc16db80ad602911f1f64e60cfad41b198922 
>   
> core/src/main/java/org/apache/oozie/action/hadoop/ScriptLanguageActionExecutor.java
>  92e149d5cac6509893c8507b776263a38312a53b 
>   core/src/main/java/org/apache/oozie/action/hadoop/ShellActionExecutor.java 
> b9ffa7ab7ec87db8a76a3653c74974d296c8cd17 
>   core/src/main/java/org/apache/oozie/action/hadoop/SparkActionExecutor.java 
> 1a3197abb52f80b545dcc8634a8cac7a281a9eac 
>   core/src/main/java/org/apache/oozie/action/hadoop/SqoopActionExecutor.java 
> 22e28746a9a39ae6c1e22326cc4f8f759d820b20 
>   core/src/main/java/org/apache/oozie/client/rest/BulkResponseImpl.java 
> fb021bd022d199f93fc5674843514abf45ae12a5 
>   core/src/main/java/org/apache/oozie/command/XCommand.java 
> bdf13f65ea25e8e0f5125e2798568a2c03fffe81 
>   core/src/main/java/org/apache/oozie/command/wf/SubmitHttpXCommand.java 
> 

Re: Review Request 54705: OOZIE-1770: Create Oozie Application Master for YARN

[Rohini Palaniswamy]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/54705/#review175722
---




core/src/main/java/org/apache/oozie/action/hadoop/CredentialsProviderFactory.java
Line 49 (original), 50 (patched)


Why would this require a ConcurrentHashMap?



core/src/main/java/org/apache/oozie/action/hadoop/HCatCredentialHelper.java
Line 48 (original), 48 (patched)


to credentials



core/src/main/java/org/apache/oozie/action/hadoop/HCatCredentialHelper.java
Line 66 (original), 67 (patched)


to credentials



core/src/main/java/org/apache/oozie/action/hadoop/JavaActionExecutor.java
Lines 959 (patched)


Fix typo. Creating YarnClient



core/src/main/java/org/apache/oozie/action/hadoop/ShellActionExecutor.java
Line 145 (original), 148 (patched)


This method can be removed now.



sharelib/oozie/src/test/java/org/apache/oozie/action/hadoop/TestLauncherMapper.java
Line 45 (original), 44 (patched)


Why are they methods in this class renamed to start with testLauncherMapper 
when the test class itself is TestLauncherMapper?  

Are these tests still relevant as LauncherMapper is not used anymore? 
Shouldn't they be changed to adapt to Oozie AM instead? Please revert current 
changes and create a jira for it if that is the case.


- Rohini Palaniswamy


On May 20, 2017, 11:55 a.m., Peter Cseh wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/54705/
> ---
> 
> (Updated May 20, 2017, 11:55 a.m.)
> 
> 
> Review request for oozie, Purshotam Shah, Robert Kanter, and Rohini 
> Palaniswamy.
> 
> 
> Bugs: OOZIE-1770
> https://issues.apache.org/jira/browse/OOZIE-1770
> 
> 
> Repository: oozie-git
> 
> 
> Description
> ---
> 
> Replacing the launcher MR job with a launcher AM
> 
> 
> Diffs
> -
> 
>   client/pom.xml f99366c90887a5a45f3942291c90e1cdbfd84b3f 
>   client/src/main/java/org/apache/oozie/cli/OozieCLI.java 
> 38fb84e8a8918aee6ae57d19a29cdf181aba1955 
>   client/src/main/java/org/apache/oozie/client/XOozieClient.java 
> 8538ec7c742cc013f7c18eaff3ecf61c2835cc8a 
>   core/pom.xml 14aa03483e107db326628ba627cc3ee9f5fd3f3d 
>   core/src/main/java/org/apache/oozie/WorkflowJobBean.java 
> 55d79a5d5e5e4efab9eb832e9341a16bcf8959cc 
>   core/src/main/java/org/apache/oozie/action/ActionExecutor.java 
> 1d6456b236bf9e27c0f55d9f9e3b4bd94517c797 
>   core/src/main/java/org/apache/oozie/action/hadoop/Credentials.java 
> eadb47b07d157f45f086517ab41c4b9826f003f0 
>   core/src/main/java/org/apache/oozie/action/hadoop/CredentialsProvider.java 
> 6fe22fbd2fca5d873d58ece60f20d0f131371475 
>   
> core/src/main/java/org/apache/oozie/action/hadoop/CredentialsProviderFactory.java
>  PRE-CREATION 
>   core/src/main/java/org/apache/oozie/action/hadoop/DistcpActionExecutor.java 
> 78cd257a836707c1efeac13566d6285d42970545 
>   core/src/main/java/org/apache/oozie/action/hadoop/FsActionExecutor.java 
> 27654741101d2056b25626fbbe345183eafd612d 
>   core/src/main/java/org/apache/oozie/action/hadoop/FsELFunctions.java 
> 801bfe6eb212b549888f0e9fb7df7b65bcf59393 
>   core/src/main/java/org/apache/oozie/action/hadoop/HCatCredentialHelper.java 
> 36ad1df727e869b8051481ad969771e91bc6ae07 
>   core/src/main/java/org/apache/oozie/action/hadoop/HCatCredentials.java 
> d7689a924059e553abaeed374eed4072933292cb 
>   core/src/main/java/org/apache/oozie/action/hadoop/HadoopELFunctions.java 
> ad2a71d641caa71efc098ad00bfbab3cec4d9506 
>   core/src/main/java/org/apache/oozie/action/hadoop/HbaseCredentials.java 
> 307f565786ad77f5a35b5790bd5b052b04c6eac8 
>   core/src/main/java/org/apache/oozie/action/hadoop/Hive2ActionExecutor.java 
> 2aed936af75da38c433be5d202bdc01c2f612c76 
>   core/src/main/java/org/apache/oozie/action/hadoop/Hive2Credentials.java 
> bd2f1f768c5b04bef4a33613baede9202144c747 
>   core/src/main/java/org/apache/oozie/action/hadoop/HiveActionExecutor.java 
> 8e2453ef4f78f4a5f1f95b1ca2793f5d7e07ad81 
>   core/src/main/java/org/apache/oozie/action/hadoop/JavaActionExecutor.java 
> 06ae5fd9a52099c5bc63b038d336ce96103c8819 
>   core/src/main/java/org/apache/oozie/action/hadoop/LauncherMapperHelper.java 
> 72ed2f150be73f0a5e8b71b0fb6cc3594dc23078 
>   
> core/src/main/java/org/apache/oozie/action/hadoop/MapReduceActionExecutor.java
>  55c937249ac35e14bd63dd0c1fa962ea636afa3a 
>   core/src/main/java/org/apache/oozie/action/hadoop/OozieJobInfo.java 
> 581d3b3219eff882f961c8e495611e873c6a5a52 
>   core/

Re: Review Request 54705: OOZIE-1770: Create Oozie Application Master for YARN

[Peter Cseh]

> On May 22, 2017, 11:21 p.m., Rohini Palaniswamy wrote:
> > sharelib/oozie/src/test/java/org/apache/oozie/action/hadoop/TestLauncherMapper.java
> > Line 45 (original), 44 (patched)
> > 
> >
> > Why are they methods in this class renamed to start with 
> > testLauncherMapper when the test class itself is TestLauncherMapper?  
> > 
> > Are these tests still relevant as LauncherMapper is not used anymore? 
> > Shouldn't they be changed to adapt to Oozie AM instead? Please revert 
> > current changes and create a jira for it if that is the case.

Opened up https://issues.apache.org/jira/browse/OOZIE-2918 .


- Peter


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/54705/#review175722
---


On May 20, 2017, 11:55 a.m., Peter Cseh wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/54705/
> ---
> 
> (Updated May 20, 2017, 11:55 a.m.)
> 
> 
> Review request for oozie, Purshotam Shah, Robert Kanter, and Rohini 
> Palaniswamy.
> 
> 
> Bugs: OOZIE-1770
> https://issues.apache.org/jira/browse/OOZIE-1770
> 
> 
> Repository: oozie-git
> 
> 
> Description
> ---
> 
> Replacing the launcher MR job with a launcher AM
> 
> 
> Diffs
> -
> 
>   client/pom.xml f99366c90887a5a45f3942291c90e1cdbfd84b3f 
>   client/src/main/java/org/apache/oozie/cli/OozieCLI.java 
> 38fb84e8a8918aee6ae57d19a29cdf181aba1955 
>   client/src/main/java/org/apache/oozie/client/XOozieClient.java 
> 8538ec7c742cc013f7c18eaff3ecf61c2835cc8a 
>   core/pom.xml 14aa03483e107db326628ba627cc3ee9f5fd3f3d 
>   core/src/main/java/org/apache/oozie/WorkflowJobBean.java 
> 55d79a5d5e5e4efab9eb832e9341a16bcf8959cc 
>   core/src/main/java/org/apache/oozie/action/ActionExecutor.java 
> 1d6456b236bf9e27c0f55d9f9e3b4bd94517c797 
>   core/src/main/java/org/apache/oozie/action/hadoop/Credentials.java 
> eadb47b07d157f45f086517ab41c4b9826f003f0 
>   core/src/main/java/org/apache/oozie/action/hadoop/CredentialsProvider.java 
> 6fe22fbd2fca5d873d58ece60f20d0f131371475 
>   
> core/src/main/java/org/apache/oozie/action/hadoop/CredentialsProviderFactory.java
>  PRE-CREATION 
>   core/src/main/java/org/apache/oozie/action/hadoop/DistcpActionExecutor.java 
> 78cd257a836707c1efeac13566d6285d42970545 
>   core/src/main/java/org/apache/oozie/action/hadoop/FsActionExecutor.java 
> 27654741101d2056b25626fbbe345183eafd612d 
>   core/src/main/java/org/apache/oozie/action/hadoop/FsELFunctions.java 
> 801bfe6eb212b549888f0e9fb7df7b65bcf59393 
>   core/src/main/java/org/apache/oozie/action/hadoop/HCatCredentialHelper.java 
> 36ad1df727e869b8051481ad969771e91bc6ae07 
>   core/src/main/java/org/apache/oozie/action/hadoop/HCatCredentials.java 
> d7689a924059e553abaeed374eed4072933292cb 
>   core/src/main/java/org/apache/oozie/action/hadoop/HadoopELFunctions.java 
> ad2a71d641caa71efc098ad00bfbab3cec4d9506 
>   core/src/main/java/org/apache/oozie/action/hadoop/HbaseCredentials.java 
> 307f565786ad77f5a35b5790bd5b052b04c6eac8 
>   core/src/main/java/org/apache/oozie/action/hadoop/Hive2ActionExecutor.java 
> 2aed936af75da38c433be5d202bdc01c2f612c76 
>   core/src/main/java/org/apache/oozie/action/hadoop/Hive2Credentials.java 
> bd2f1f768c5b04bef4a33613baede9202144c747 
>   core/src/main/java/org/apache/oozie/action/hadoop/HiveActionExecutor.java 
> 8e2453ef4f78f4a5f1f95b1ca2793f5d7e07ad81 
>   core/src/main/java/org/apache/oozie/action/hadoop/JavaActionExecutor.java 
> 06ae5fd9a52099c5bc63b038d336ce96103c8819 
>   core/src/main/java/org/apache/oozie/action/hadoop/LauncherMapperHelper.java 
> 72ed2f150be73f0a5e8b71b0fb6cc3594dc23078 
>   
> core/src/main/java/org/apache/oozie/action/hadoop/MapReduceActionExecutor.java
>  55c937249ac35e14bd63dd0c1fa962ea636afa3a 
>   core/src/main/java/org/apache/oozie/action/hadoop/OozieJobInfo.java 
> 581d3b3219eff882f961c8e495611e873c6a5a52 
>   core/src/main/java/org/apache/oozie/action/hadoop/PigActionExecutor.java 
> 8b2dc16db80ad602911f1f64e60cfad41b198922 
>   
> core/src/main/java/org/apache/oozie/action/hadoop/ScriptLanguageActionExecutor.java
>  92e149d5cac6509893c8507b776263a38312a53b 
>   core/src/main/java/org/apache/oozie/action/hadoop/ShellActionExecutor.java 
> b9ffa7ab7ec87db8a76a3653c74974d296c8cd17 
>   core/src/main/java/org/apache/oozie/action/hadoop/SparkActionExecutor.java 
> 1a3197abb52f80b545dcc8634a8cac7a281a9eac 
>   core/src/main/java/org/apache/oozie/action/hadoop/SqoopActionExecutor.java 
> 22e28746a9a39ae6c1e22326cc4f8f759d820b20 
>   core/src/main/java/org/apache/oozie/client/rest/BulkResponseImpl.java 
> fb021bd022d199f93fc5674843514abf45ae12a5 
>   core/src/main/java/org/apache/oozie/command/XCommand.java 
> 

Re: Review Request 54705: OOZIE-1770: Create Oozie Application Master for YARN

[Peter Cseh]

> On May 22, 2017, 11:21 p.m., Rohini Palaniswamy wrote:
> > core/src/main/java/org/apache/oozie/action/hadoop/CredentialsProviderFactory.java
> > Line 49 (original), 50 (patched)
> > 
> >
> > Why would this require a ConcurrentHashMap?

It a precaution to avoid issues when getInstance() is called from multiple 
threads and the map is filled up from two threads. It shouldn't cause any 
issues with performance or threads this way.


- Peter


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/54705/#review175722
---


On May 20, 2017, 11:55 a.m., Peter Cseh wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/54705/
> ---
> 
> (Updated May 20, 2017, 11:55 a.m.)
> 
> 
> Review request for oozie, Purshotam Shah, Robert Kanter, and Rohini 
> Palaniswamy.
> 
> 
> Bugs: OOZIE-1770
> https://issues.apache.org/jira/browse/OOZIE-1770
> 
> 
> Repository: oozie-git
> 
> 
> Description
> ---
> 
> Replacing the launcher MR job with a launcher AM
> 
> 
> Diffs
> -
> 
>   client/pom.xml f99366c90887a5a45f3942291c90e1cdbfd84b3f 
>   client/src/main/java/org/apache/oozie/cli/OozieCLI.java 
> 38fb84e8a8918aee6ae57d19a29cdf181aba1955 
>   client/src/main/java/org/apache/oozie/client/XOozieClient.java 
> 8538ec7c742cc013f7c18eaff3ecf61c2835cc8a 
>   core/pom.xml 14aa03483e107db326628ba627cc3ee9f5fd3f3d 
>   core/src/main/java/org/apache/oozie/WorkflowJobBean.java 
> 55d79a5d5e5e4efab9eb832e9341a16bcf8959cc 
>   core/src/main/java/org/apache/oozie/action/ActionExecutor.java 
> 1d6456b236bf9e27c0f55d9f9e3b4bd94517c797 
>   core/src/main/java/org/apache/oozie/action/hadoop/Credentials.java 
> eadb47b07d157f45f086517ab41c4b9826f003f0 
>   core/src/main/java/org/apache/oozie/action/hadoop/CredentialsProvider.java 
> 6fe22fbd2fca5d873d58ece60f20d0f131371475 
>   
> core/src/main/java/org/apache/oozie/action/hadoop/CredentialsProviderFactory.java
>  PRE-CREATION 
>   core/src/main/java/org/apache/oozie/action/hadoop/DistcpActionExecutor.java 
> 78cd257a836707c1efeac13566d6285d42970545 
>   core/src/main/java/org/apache/oozie/action/hadoop/FsActionExecutor.java 
> 27654741101d2056b25626fbbe345183eafd612d 
>   core/src/main/java/org/apache/oozie/action/hadoop/FsELFunctions.java 
> 801bfe6eb212b549888f0e9fb7df7b65bcf59393 
>   core/src/main/java/org/apache/oozie/action/hadoop/HCatCredentialHelper.java 
> 36ad1df727e869b8051481ad969771e91bc6ae07 
>   core/src/main/java/org/apache/oozie/action/hadoop/HCatCredentials.java 
> d7689a924059e553abaeed374eed4072933292cb 
>   core/src/main/java/org/apache/oozie/action/hadoop/HadoopELFunctions.java 
> ad2a71d641caa71efc098ad00bfbab3cec4d9506 
>   core/src/main/java/org/apache/oozie/action/hadoop/HbaseCredentials.java 
> 307f565786ad77f5a35b5790bd5b052b04c6eac8 
>   core/src/main/java/org/apache/oozie/action/hadoop/Hive2ActionExecutor.java 
> 2aed936af75da38c433be5d202bdc01c2f612c76 
>   core/src/main/java/org/apache/oozie/action/hadoop/Hive2Credentials.java 
> bd2f1f768c5b04bef4a33613baede9202144c747 
>   core/src/main/java/org/apache/oozie/action/hadoop/HiveActionExecutor.java 
> 8e2453ef4f78f4a5f1f95b1ca2793f5d7e07ad81 
>   core/src/main/java/org/apache/oozie/action/hadoop/JavaActionExecutor.java 
> 06ae5fd9a52099c5bc63b038d336ce96103c8819 
>   core/src/main/java/org/apache/oozie/action/hadoop/LauncherMapperHelper.java 
> 72ed2f150be73f0a5e8b71b0fb6cc3594dc23078 
>   
> core/src/main/java/org/apache/oozie/action/hadoop/MapReduceActionExecutor.java
>  55c937249ac35e14bd63dd0c1fa962ea636afa3a 
>   core/src/main/java/org/apache/oozie/action/hadoop/OozieJobInfo.java 
> 581d3b3219eff882f961c8e495611e873c6a5a52 
>   core/src/main/java/org/apache/oozie/action/hadoop/PigActionExecutor.java 
> 8b2dc16db80ad602911f1f64e60cfad41b198922 
>   
> core/src/main/java/org/apache/oozie/action/hadoop/ScriptLanguageActionExecutor.java
>  92e149d5cac6509893c8507b776263a38312a53b 
>   core/src/main/java/org/apache/oozie/action/hadoop/ShellActionExecutor.java 
> b9ffa7ab7ec87db8a76a3653c74974d296c8cd17 
>   core/src/main/java/org/apache/oozie/action/hadoop/SparkActionExecutor.java 
> 1a3197abb52f80b545dcc8634a8cac7a281a9eac 
>   core/src/main/java/org/apache/oozie/action/hadoop/SqoopActionExecutor.java 
> 22e28746a9a39ae6c1e22326cc4f8f759d820b20 
>   core/src/main/java/org/apache/oozie/client/rest/BulkResponseImpl.java 
> fb021bd022d199f93fc5674843514abf45ae12a5 
>   core/src/main/java/org/apache/oozie/command/XCommand.java 
> bdf13f65ea25e8e0f5125e2798568a2c03fffe81 
>   core/src/main/java/org/apache/oozie/command/wf/SubmitHttpXCommand.java 
> d2a2742ae418070a1cd0a889f14194224bc944a6 
>   core/src/main/jav

Re: Review Request 54705: OOZIE-1770: Create Oozie Application Master for YARN

[Peter Cseh]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/54705/
---

(Updated May 23, 2017, 4:43 p.m.)


Review request for oozie, Purshotam Shah, Robert Kanter, and Rohini Palaniswamy.


Bugs: OOZIE-1770
https://issues.apache.org/jira/browse/OOZIE-1770


Repository: oozie-git


Description
---

Replacing the launcher MR job with a launcher AM


Diffs (updated)
-

  client/pom.xml f99366c90887a5a45f3942291c90e1cdbfd84b3f 
  client/src/main/java/org/apache/oozie/cli/OozieCLI.java 
38fb84e8a8918aee6ae57d19a29cdf181aba1955 
  client/src/main/java/org/apache/oozie/client/XOozieClient.java 
8538ec7c742cc013f7c18eaff3ecf61c2835cc8a 
  core/pom.xml 14aa03483e107db326628ba627cc3ee9f5fd3f3d 
  core/src/main/java/org/apache/oozie/WorkflowJobBean.java 
55d79a5d5e5e4efab9eb832e9341a16bcf8959cc 
  core/src/main/java/org/apache/oozie/action/ActionExecutor.java 
1d6456b236bf9e27c0f55d9f9e3b4bd94517c797 
  core/src/main/java/org/apache/oozie/action/hadoop/Credentials.java 
eadb47b07d157f45f086517ab41c4b9826f003f0 
  core/src/main/java/org/apache/oozie/action/hadoop/CredentialsProvider.java 
6fe22fbd2fca5d873d58ece60f20d0f131371475 
  
core/src/main/java/org/apache/oozie/action/hadoop/CredentialsProviderFactory.java
 PRE-CREATION 
  core/src/main/java/org/apache/oozie/action/hadoop/DistcpActionExecutor.java 
78cd257a836707c1efeac13566d6285d42970545 
  core/src/main/java/org/apache/oozie/action/hadoop/FsActionExecutor.java 
27654741101d2056b25626fbbe345183eafd612d 
  core/src/main/java/org/apache/oozie/action/hadoop/FsELFunctions.java 
801bfe6eb212b549888f0e9fb7df7b65bcf59393 
  core/src/main/java/org/apache/oozie/action/hadoop/HCatCredentialHelper.java 
36ad1df727e869b8051481ad969771e91bc6ae07 
  core/src/main/java/org/apache/oozie/action/hadoop/HCatCredentials.java 
d7689a924059e553abaeed374eed4072933292cb 
  core/src/main/java/org/apache/oozie/action/hadoop/HadoopELFunctions.java 
ad2a71d641caa71efc098ad00bfbab3cec4d9506 
  core/src/main/java/org/apache/oozie/action/hadoop/HbaseCredentials.java 
307f565786ad77f5a35b5790bd5b052b04c6eac8 
  core/src/main/java/org/apache/oozie/action/hadoop/Hive2ActionExecutor.java 
2aed936af75da38c433be5d202bdc01c2f612c76 
  core/src/main/java/org/apache/oozie/action/hadoop/Hive2Credentials.java 
bd2f1f768c5b04bef4a33613baede9202144c747 
  core/src/main/java/org/apache/oozie/action/hadoop/HiveActionExecutor.java 
8e2453ef4f78f4a5f1f95b1ca2793f5d7e07ad81 
  core/src/main/java/org/apache/oozie/action/hadoop/JavaActionExecutor.java 
06ae5fd9a52099c5bc63b038d336ce96103c8819 
  core/src/main/java/org/apache/oozie/action/hadoop/LauncherMapperHelper.java 
72ed2f150be73f0a5e8b71b0fb6cc3594dc23078 
  
core/src/main/java/org/apache/oozie/action/hadoop/MapReduceActionExecutor.java 
55c937249ac35e14bd63dd0c1fa962ea636afa3a 
  core/src/main/java/org/apache/oozie/action/hadoop/OozieJobInfo.java 
581d3b3219eff882f961c8e495611e873c6a5a52 
  core/src/main/java/org/apache/oozie/action/hadoop/PigActionExecutor.java 
8b2dc16db80ad602911f1f64e60cfad41b198922 
  
core/src/main/java/org/apache/oozie/action/hadoop/ScriptLanguageActionExecutor.java
 92e149d5cac6509893c8507b776263a38312a53b 
  core/src/main/java/org/apache/oozie/action/hadoop/ShellActionExecutor.java 
b9ffa7ab7ec87db8a76a3653c74974d296c8cd17 
  core/src/main/java/org/apache/oozie/action/hadoop/SparkActionExecutor.java 
1a3197abb52f80b545dcc8634a8cac7a281a9eac 
  core/src/main/java/org/apache/oozie/action/hadoop/SqoopActionExecutor.java 
22e28746a9a39ae6c1e22326cc4f8f759d820b20 
  core/src/main/java/org/apache/oozie/client/rest/BulkResponseImpl.java 
fb021bd022d199f93fc5674843514abf45ae12a5 
  core/src/main/java/org/apache/oozie/command/XCommand.java 
bdf13f65ea25e8e0f5125e2798568a2c03fffe81 
  core/src/main/java/org/apache/oozie/command/wf/SubmitHttpXCommand.java 
d2a2742ae418070a1cd0a889f14194224bc944a6 
  core/src/main/java/org/apache/oozie/command/wf/SubmitMRXCommand.java 
cc61d3de813c75c7f1cf4d405fc043253e3ca4ea 
  
core/src/main/java/org/apache/oozie/command/wf/SubmitScriptLanguageXCommand.java
 9d413059e99bc676feb63e02a2f704048eb4a355 
  core/src/main/java/org/apache/oozie/command/wf/SubmitSqoopXCommand.java 
51b739e21c802e09f4110b740138f4ff5c341fe4 
  
core/src/main/java/org/apache/oozie/coord/input/dependency/AbstractCoordInputDependency.java
 ace120d10e3546201aa4145df56abf7933fb24b8 
  
core/src/main/java/org/apache/oozie/dependency/hcat/EhcacheHCatDependencyCache.java
 6f0abf6e094ed04158652191f2b8bb3b0cd32e90 
  core/src/main/java/org/apache/oozie/service/CallableQueueService.java 
a86a8d0889138cd1564ca9253e5b319ea52e55cc 
  core/src/main/java/org/apache/oozie/service/EventHandlerService.java 
22c6fb0965f45c5cf2322c251d4a8aa77d263458 
  core/src/main/java/org/apache/oozie/service/HadoopAccessorService.java 
23a9d92bf56a625469ec247cfb9727972511b101 
  core/src/main/java/org/apache/

Re: Review Request 54705: OOZIE-1770: Create Oozie Application Master for YARN

[Rohini Palaniswamy]

> On May 22, 2017, 11:21 p.m., Rohini Palaniswamy wrote:
> > core/src/main/java/org/apache/oozie/action/hadoop/CredentialsProviderFactory.java
> > Line 49 (original), 50 (patched)
> > 
> >
> > Why would this require a ConcurrentHashMap?
> 
> Peter Cseh wrote:
> It a precaution to avoid issues when getInstance() is called from 
> multiple threads and the map is filled up from two threads. It shouldn't 
> cause any issues with performance or threads this way.

Using a ConcurrentHashMap does not help in anyway. That variable is local to 
the new CredentialsProviderFactory being created. Two threads cannot fill the 
same map as the constructors will create two different instances. If two 
threads call getInstance at same time and if instance is null, both will create 
a instance and the last one will win. Ideally we should avoid that by 
synchronizing, but in this case based on the nature of data it is not a 
problem. Please switch it back to HashMap.


> On May 22, 2017, 11:21 p.m., Rohini Palaniswamy wrote:
> > core/src/main/java/org/apache/oozie/action/hadoop/ShellActionExecutor.java
> > Line 145 (original), 148 (patched)
> > 
> >
> > This method can be removed now.

It is still not removed.


- Rohini


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/54705/#review175722
---


On May 23, 2017, 4:43 p.m., Peter Cseh wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/54705/
> ---
> 
> (Updated May 23, 2017, 4:43 p.m.)
> 
> 
> Review request for oozie, Purshotam Shah, Robert Kanter, and Rohini 
> Palaniswamy.
> 
> 
> Bugs: OOZIE-1770
> https://issues.apache.org/jira/browse/OOZIE-1770
> 
> 
> Repository: oozie-git
> 
> 
> Description
> ---
> 
> Replacing the launcher MR job with a launcher AM
> 
> 
> Diffs
> -
> 
>   client/pom.xml f99366c90887a5a45f3942291c90e1cdbfd84b3f 
>   client/src/main/java/org/apache/oozie/cli/OozieCLI.java 
> 38fb84e8a8918aee6ae57d19a29cdf181aba1955 
>   client/src/main/java/org/apache/oozie/client/XOozieClient.java 
> 8538ec7c742cc013f7c18eaff3ecf61c2835cc8a 
>   core/pom.xml 14aa03483e107db326628ba627cc3ee9f5fd3f3d 
>   core/src/main/java/org/apache/oozie/WorkflowJobBean.java 
> 55d79a5d5e5e4efab9eb832e9341a16bcf8959cc 
>   core/src/main/java/org/apache/oozie/action/ActionExecutor.java 
> 1d6456b236bf9e27c0f55d9f9e3b4bd94517c797 
>   core/src/main/java/org/apache/oozie/action/hadoop/Credentials.java 
> eadb47b07d157f45f086517ab41c4b9826f003f0 
>   core/src/main/java/org/apache/oozie/action/hadoop/CredentialsProvider.java 
> 6fe22fbd2fca5d873d58ece60f20d0f131371475 
>   
> core/src/main/java/org/apache/oozie/action/hadoop/CredentialsProviderFactory.java
>  PRE-CREATION 
>   core/src/main/java/org/apache/oozie/action/hadoop/DistcpActionExecutor.java 
> 78cd257a836707c1efeac13566d6285d42970545 
>   core/src/main/java/org/apache/oozie/action/hadoop/FsActionExecutor.java 
> 27654741101d2056b25626fbbe345183eafd612d 
>   core/src/main/java/org/apache/oozie/action/hadoop/FsELFunctions.java 
> 801bfe6eb212b549888f0e9fb7df7b65bcf59393 
>   core/src/main/java/org/apache/oozie/action/hadoop/HCatCredentialHelper.java 
> 36ad1df727e869b8051481ad969771e91bc6ae07 
>   core/src/main/java/org/apache/oozie/action/hadoop/HCatCredentials.java 
> d7689a924059e553abaeed374eed4072933292cb 
>   core/src/main/java/org/apache/oozie/action/hadoop/HadoopELFunctions.java 
> ad2a71d641caa71efc098ad00bfbab3cec4d9506 
>   core/src/main/java/org/apache/oozie/action/hadoop/HbaseCredentials.java 
> 307f565786ad77f5a35b5790bd5b052b04c6eac8 
>   core/src/main/java/org/apache/oozie/action/hadoop/Hive2ActionExecutor.java 
> 2aed936af75da38c433be5d202bdc01c2f612c76 
>   core/src/main/java/org/apache/oozie/action/hadoop/Hive2Credentials.java 
> bd2f1f768c5b04bef4a33613baede9202144c747 
>   core/src/main/java/org/apache/oozie/action/hadoop/HiveActionExecutor.java 
> 8e2453ef4f78f4a5f1f95b1ca2793f5d7e07ad81 
>   core/src/main/java/org/apache/oozie/action/hadoop/JavaActionExecutor.java 
> 06ae5fd9a52099c5bc63b038d336ce96103c8819 
>   core/src/main/java/org/apache/oozie/action/hadoop/LauncherMapperHelper.java 
> 72ed2f150be73f0a5e8b71b0fb6cc3594dc23078 
>   
> core/src/main/java/org/apache/oozie/action/hadoop/MapReduceActionExecutor.java
>  55c937249ac35e14bd63dd0c1fa962ea636afa3a 
>   core/src/main/java/org/apache/oozie/action/hadoop/OozieJobInfo.java 
> 581d3b3219eff882f961c8e495611e873c6a5a52 
>   core/src/main/java/org/apache/oozie/action/hadoop/PigActionExecutor.java 
> 8b2dc16db80ad602911f1f64e60cfad41b198922 
>   
> core/sr

Re: Review Request 54705: OOZIE-1770: Create Oozie Application Master for YARN

[Peter Cseh]

> On May 22, 2017, 11:21 p.m., Rohini Palaniswamy wrote:
> > core/src/main/java/org/apache/oozie/action/hadoop/ShellActionExecutor.java
> > Line 145 (original), 148 (patched)
> > 
> >
> > This method can be removed now.
> 
> Rohini Palaniswamy wrote:
> It is still not removed.

Hmm It's not there on the branch: 
https://github.com/apache/oozie/blob/oya/core/src/main/java/org/apache/oozie/action/hadoop/ShellActionExecutor.java
Let me double-check what's going on in the diff.


- Peter


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/54705/#review175722
---


On May 23, 2017, 4:43 p.m., Peter Cseh wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/54705/
> ---
> 
> (Updated May 23, 2017, 4:43 p.m.)
> 
> 
> Review request for oozie, Purshotam Shah, Robert Kanter, and Rohini 
> Palaniswamy.
> 
> 
> Bugs: OOZIE-1770
> https://issues.apache.org/jira/browse/OOZIE-1770
> 
> 
> Repository: oozie-git
> 
> 
> Description
> ---
> 
> Replacing the launcher MR job with a launcher AM
> 
> 
> Diffs
> -
> 
>   client/pom.xml f99366c90887a5a45f3942291c90e1cdbfd84b3f 
>   client/src/main/java/org/apache/oozie/cli/OozieCLI.java 
> 38fb84e8a8918aee6ae57d19a29cdf181aba1955 
>   client/src/main/java/org/apache/oozie/client/XOozieClient.java 
> 8538ec7c742cc013f7c18eaff3ecf61c2835cc8a 
>   core/pom.xml 14aa03483e107db326628ba627cc3ee9f5fd3f3d 
>   core/src/main/java/org/apache/oozie/WorkflowJobBean.java 
> 55d79a5d5e5e4efab9eb832e9341a16bcf8959cc 
>   core/src/main/java/org/apache/oozie/action/ActionExecutor.java 
> 1d6456b236bf9e27c0f55d9f9e3b4bd94517c797 
>   core/src/main/java/org/apache/oozie/action/hadoop/Credentials.java 
> eadb47b07d157f45f086517ab41c4b9826f003f0 
>   core/src/main/java/org/apache/oozie/action/hadoop/CredentialsProvider.java 
> 6fe22fbd2fca5d873d58ece60f20d0f131371475 
>   
> core/src/main/java/org/apache/oozie/action/hadoop/CredentialsProviderFactory.java
>  PRE-CREATION 
>   core/src/main/java/org/apache/oozie/action/hadoop/DistcpActionExecutor.java 
> 78cd257a836707c1efeac13566d6285d42970545 
>   core/src/main/java/org/apache/oozie/action/hadoop/FsActionExecutor.java 
> 27654741101d2056b25626fbbe345183eafd612d 
>   core/src/main/java/org/apache/oozie/action/hadoop/FsELFunctions.java 
> 801bfe6eb212b549888f0e9fb7df7b65bcf59393 
>   core/src/main/java/org/apache/oozie/action/hadoop/HCatCredentialHelper.java 
> 36ad1df727e869b8051481ad969771e91bc6ae07 
>   core/src/main/java/org/apache/oozie/action/hadoop/HCatCredentials.java 
> d7689a924059e553abaeed374eed4072933292cb 
>   core/src/main/java/org/apache/oozie/action/hadoop/HadoopELFunctions.java 
> ad2a71d641caa71efc098ad00bfbab3cec4d9506 
>   core/src/main/java/org/apache/oozie/action/hadoop/HbaseCredentials.java 
> 307f565786ad77f5a35b5790bd5b052b04c6eac8 
>   core/src/main/java/org/apache/oozie/action/hadoop/Hive2ActionExecutor.java 
> 2aed936af75da38c433be5d202bdc01c2f612c76 
>   core/src/main/java/org/apache/oozie/action/hadoop/Hive2Credentials.java 
> bd2f1f768c5b04bef4a33613baede9202144c747 
>   core/src/main/java/org/apache/oozie/action/hadoop/HiveActionExecutor.java 
> 8e2453ef4f78f4a5f1f95b1ca2793f5d7e07ad81 
>   core/src/main/java/org/apache/oozie/action/hadoop/JavaActionExecutor.java 
> 06ae5fd9a52099c5bc63b038d336ce96103c8819 
>   core/src/main/java/org/apache/oozie/action/hadoop/LauncherMapperHelper.java 
> 72ed2f150be73f0a5e8b71b0fb6cc3594dc23078 
>   
> core/src/main/java/org/apache/oozie/action/hadoop/MapReduceActionExecutor.java
>  55c937249ac35e14bd63dd0c1fa962ea636afa3a 
>   core/src/main/java/org/apache/oozie/action/hadoop/OozieJobInfo.java 
> 581d3b3219eff882f961c8e495611e873c6a5a52 
>   core/src/main/java/org/apache/oozie/action/hadoop/PigActionExecutor.java 
> 8b2dc16db80ad602911f1f64e60cfad41b198922 
>   
> core/src/main/java/org/apache/oozie/action/hadoop/ScriptLanguageActionExecutor.java
>  92e149d5cac6509893c8507b776263a38312a53b 
>   core/src/main/java/org/apache/oozie/action/hadoop/ShellActionExecutor.java 
> b9ffa7ab7ec87db8a76a3653c74974d296c8cd17 
>   core/src/main/java/org/apache/oozie/action/hadoop/SparkActionExecutor.java 
> 1a3197abb52f80b545dcc8634a8cac7a281a9eac 
>   core/src/main/java/org/apache/oozie/action/hadoop/SqoopActionExecutor.java 
> 22e28746a9a39ae6c1e22326cc4f8f759d820b20 
>   core/src/main/java/org/apache/oozie/client/rest/BulkResponseImpl.java 
> fb021bd022d199f93fc5674843514abf45ae12a5 
>   core/src/main/java/org/apache/oozie/command/XCommand.java 
> bdf13f65ea25e8e0f5125e2798568a2c03fffe81 
>   core/src/main/java/org/apache/oozie/command/wf/SubmitHttpXCommand.java 
> d2a2742ae418070

Re: Review Request 54705: OOZIE-1770: Create Oozie Application Master for YARN

[Peter Cseh]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/54705/
---

(Updated May 24, 2017, 12:05 p.m.)


Review request for oozie, Purshotam Shah, Robert Kanter, and Rohini Palaniswamy.


Bugs: OOZIE-1770
https://issues.apache.org/jira/browse/OOZIE-1770


Repository: oozie-git


Description
---

Replacing the launcher MR job with a launcher AM


Diffs (updated)
-

  client/pom.xml f99366c90887a5a45f3942291c90e1cdbfd84b3f 
  client/src/main/java/org/apache/oozie/cli/OozieCLI.java 
38fb84e8a8918aee6ae57d19a29cdf181aba1955 
  client/src/main/java/org/apache/oozie/client/XOozieClient.java 
8538ec7c742cc013f7c18eaff3ecf61c2835cc8a 
  core/pom.xml 14aa03483e107db326628ba627cc3ee9f5fd3f3d 
  core/src/main/java/org/apache/oozie/WorkflowJobBean.java 
55d79a5d5e5e4efab9eb832e9341a16bcf8959cc 
  core/src/main/java/org/apache/oozie/action/ActionExecutor.java 
1d6456b236bf9e27c0f55d9f9e3b4bd94517c797 
  core/src/main/java/org/apache/oozie/action/hadoop/Credentials.java 
eadb47b07d157f45f086517ab41c4b9826f003f0 
  core/src/main/java/org/apache/oozie/action/hadoop/CredentialsProvider.java 
6fe22fbd2fca5d873d58ece60f20d0f131371475 
  
core/src/main/java/org/apache/oozie/action/hadoop/CredentialsProviderFactory.java
 PRE-CREATION 
  core/src/main/java/org/apache/oozie/action/hadoop/DistcpActionExecutor.java 
78cd257a836707c1efeac13566d6285d42970545 
  core/src/main/java/org/apache/oozie/action/hadoop/FsActionExecutor.java 
27654741101d2056b25626fbbe345183eafd612d 
  core/src/main/java/org/apache/oozie/action/hadoop/FsELFunctions.java 
801bfe6eb212b549888f0e9fb7df7b65bcf59393 
  core/src/main/java/org/apache/oozie/action/hadoop/HCatCredentialHelper.java 
36ad1df727e869b8051481ad969771e91bc6ae07 
  core/src/main/java/org/apache/oozie/action/hadoop/HCatCredentials.java 
d7689a924059e553abaeed374eed4072933292cb 
  core/src/main/java/org/apache/oozie/action/hadoop/HadoopELFunctions.java 
ad2a71d641caa71efc098ad00bfbab3cec4d9506 
  core/src/main/java/org/apache/oozie/action/hadoop/HbaseCredentials.java 
307f565786ad77f5a35b5790bd5b052b04c6eac8 
  core/src/main/java/org/apache/oozie/action/hadoop/Hive2ActionExecutor.java 
2aed936af75da38c433be5d202bdc01c2f612c76 
  core/src/main/java/org/apache/oozie/action/hadoop/Hive2Credentials.java 
bd2f1f768c5b04bef4a33613baede9202144c747 
  core/src/main/java/org/apache/oozie/action/hadoop/HiveActionExecutor.java 
8e2453ef4f78f4a5f1f95b1ca2793f5d7e07ad81 
  core/src/main/java/org/apache/oozie/action/hadoop/JavaActionExecutor.java 
06ae5fd9a52099c5bc63b038d336ce96103c8819 
  core/src/main/java/org/apache/oozie/action/hadoop/LauncherMapperHelper.java 
72ed2f150be73f0a5e8b71b0fb6cc3594dc23078 
  
core/src/main/java/org/apache/oozie/action/hadoop/MapReduceActionExecutor.java 
55c937249ac35e14bd63dd0c1fa962ea636afa3a 
  core/src/main/java/org/apache/oozie/action/hadoop/OozieJobInfo.java 
581d3b3219eff882f961c8e495611e873c6a5a52 
  core/src/main/java/org/apache/oozie/action/hadoop/PigActionExecutor.java 
8b2dc16db80ad602911f1f64e60cfad41b198922 
  
core/src/main/java/org/apache/oozie/action/hadoop/ScriptLanguageActionExecutor.java
 92e149d5cac6509893c8507b776263a38312a53b 
  core/src/main/java/org/apache/oozie/action/hadoop/ShellActionExecutor.java 
b9ffa7ab7ec87db8a76a3653c74974d296c8cd17 
  core/src/main/java/org/apache/oozie/action/hadoop/SparkActionExecutor.java 
1a3197abb52f80b545dcc8634a8cac7a281a9eac 
  core/src/main/java/org/apache/oozie/action/hadoop/SqoopActionExecutor.java 
22e28746a9a39ae6c1e22326cc4f8f759d820b20 
  core/src/main/java/org/apache/oozie/client/rest/BulkResponseImpl.java 
fb021bd022d199f93fc5674843514abf45ae12a5 
  core/src/main/java/org/apache/oozie/command/XCommand.java 
bdf13f65ea25e8e0f5125e2798568a2c03fffe81 
  core/src/main/java/org/apache/oozie/command/wf/SubmitHttpXCommand.java 
d2a2742ae418070a1cd0a889f14194224bc944a6 
  core/src/main/java/org/apache/oozie/command/wf/SubmitMRXCommand.java 
cc61d3de813c75c7f1cf4d405fc043253e3ca4ea 
  
core/src/main/java/org/apache/oozie/command/wf/SubmitScriptLanguageXCommand.java
 9d413059e99bc676feb63e02a2f704048eb4a355 
  core/src/main/java/org/apache/oozie/command/wf/SubmitSqoopXCommand.java 
51b739e21c802e09f4110b740138f4ff5c341fe4 
  
core/src/main/java/org/apache/oozie/coord/input/dependency/AbstractCoordInputDependency.java
 ace120d10e3546201aa4145df56abf7933fb24b8 
  
core/src/main/java/org/apache/oozie/dependency/hcat/EhcacheHCatDependencyCache.java
 6f0abf6e094ed04158652191f2b8bb3b0cd32e90 
  core/src/main/java/org/apache/oozie/service/CallableQueueService.java 
a86a8d0889138cd1564ca9253e5b319ea52e55cc 
  core/src/main/java/org/apache/oozie/service/EventHandlerService.java 
22c6fb0965f45c5cf2322c251d4a8aa77d263458 
  core/src/main/java/org/apache/oozie/service/HadoopAccessorService.java 
23a9d92bf56a625469ec247cfb9727972511b101 
  core/src/main/java/org/apache

Re: Review Request 54705: OOZIE-1770: Create Oozie Application Master for YARN

[Peter Cseh]

> On May 22, 2017, 11:21 p.m., Rohini Palaniswamy wrote:
> > core/src/main/java/org/apache/oozie/action/hadoop/ShellActionExecutor.java
> > Line 145 (original), 148 (patched)
> > 
> >
> > This method can be removed now.
> 
> Rohini Palaniswamy wrote:
> It is still not removed.
> 
> Peter Cseh wrote:
> Hmm It's not there on the branch: 
> https://github.com/apache/oozie/blob/oya/core/src/main/java/org/apache/oozie/action/hadoop/ShellActionExecutor.java
> Let me double-check what's going on in the diff.

I looks like it's removed now
https://reviews.apache.org/r/54705/diff/11?page=2#25


- Peter


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/54705/#review175722
---


On May 24, 2017, 12:05 p.m., Peter Cseh wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/54705/
> ---
> 
> (Updated May 24, 2017, 12:05 p.m.)
> 
> 
> Review request for oozie, Purshotam Shah, Robert Kanter, and Rohini 
> Palaniswamy.
> 
> 
> Bugs: OOZIE-1770
> https://issues.apache.org/jira/browse/OOZIE-1770
> 
> 
> Repository: oozie-git
> 
> 
> Description
> ---
> 
> Replacing the launcher MR job with a launcher AM
> 
> 
> Diffs
> -
> 
>   client/pom.xml f99366c90887a5a45f3942291c90e1cdbfd84b3f 
>   client/src/main/java/org/apache/oozie/cli/OozieCLI.java 
> 38fb84e8a8918aee6ae57d19a29cdf181aba1955 
>   client/src/main/java/org/apache/oozie/client/XOozieClient.java 
> 8538ec7c742cc013f7c18eaff3ecf61c2835cc8a 
>   core/pom.xml 14aa03483e107db326628ba627cc3ee9f5fd3f3d 
>   core/src/main/java/org/apache/oozie/WorkflowJobBean.java 
> 55d79a5d5e5e4efab9eb832e9341a16bcf8959cc 
>   core/src/main/java/org/apache/oozie/action/ActionExecutor.java 
> 1d6456b236bf9e27c0f55d9f9e3b4bd94517c797 
>   core/src/main/java/org/apache/oozie/action/hadoop/Credentials.java 
> eadb47b07d157f45f086517ab41c4b9826f003f0 
>   core/src/main/java/org/apache/oozie/action/hadoop/CredentialsProvider.java 
> 6fe22fbd2fca5d873d58ece60f20d0f131371475 
>   
> core/src/main/java/org/apache/oozie/action/hadoop/CredentialsProviderFactory.java
>  PRE-CREATION 
>   core/src/main/java/org/apache/oozie/action/hadoop/DistcpActionExecutor.java 
> 78cd257a836707c1efeac13566d6285d42970545 
>   core/src/main/java/org/apache/oozie/action/hadoop/FsActionExecutor.java 
> 27654741101d2056b25626fbbe345183eafd612d 
>   core/src/main/java/org/apache/oozie/action/hadoop/FsELFunctions.java 
> 801bfe6eb212b549888f0e9fb7df7b65bcf59393 
>   core/src/main/java/org/apache/oozie/action/hadoop/HCatCredentialHelper.java 
> 36ad1df727e869b8051481ad969771e91bc6ae07 
>   core/src/main/java/org/apache/oozie/action/hadoop/HCatCredentials.java 
> d7689a924059e553abaeed374eed4072933292cb 
>   core/src/main/java/org/apache/oozie/action/hadoop/HadoopELFunctions.java 
> ad2a71d641caa71efc098ad00bfbab3cec4d9506 
>   core/src/main/java/org/apache/oozie/action/hadoop/HbaseCredentials.java 
> 307f565786ad77f5a35b5790bd5b052b04c6eac8 
>   core/src/main/java/org/apache/oozie/action/hadoop/Hive2ActionExecutor.java 
> 2aed936af75da38c433be5d202bdc01c2f612c76 
>   core/src/main/java/org/apache/oozie/action/hadoop/Hive2Credentials.java 
> bd2f1f768c5b04bef4a33613baede9202144c747 
>   core/src/main/java/org/apache/oozie/action/hadoop/HiveActionExecutor.java 
> 8e2453ef4f78f4a5f1f95b1ca2793f5d7e07ad81 
>   core/src/main/java/org/apache/oozie/action/hadoop/JavaActionExecutor.java 
> 06ae5fd9a52099c5bc63b038d336ce96103c8819 
>   core/src/main/java/org/apache/oozie/action/hadoop/LauncherMapperHelper.java 
> 72ed2f150be73f0a5e8b71b0fb6cc3594dc23078 
>   
> core/src/main/java/org/apache/oozie/action/hadoop/MapReduceActionExecutor.java
>  55c937249ac35e14bd63dd0c1fa962ea636afa3a 
>   core/src/main/java/org/apache/oozie/action/hadoop/OozieJobInfo.java 
> 581d3b3219eff882f961c8e495611e873c6a5a52 
>   core/src/main/java/org/apache/oozie/action/hadoop/PigActionExecutor.java 
> 8b2dc16db80ad602911f1f64e60cfad41b198922 
>   
> core/src/main/java/org/apache/oozie/action/hadoop/ScriptLanguageActionExecutor.java
>  92e149d5cac6509893c8507b776263a38312a53b 
>   core/src/main/java/org/apache/oozie/action/hadoop/ShellActionExecutor.java 
> b9ffa7ab7ec87db8a76a3653c74974d296c8cd17 
>   core/src/main/java/org/apache/oozie/action/hadoop/SparkActionExecutor.java 
> 1a3197abb52f80b545dcc8634a8cac7a281a9eac 
>   core/src/main/java/org/apache/oozie/action/hadoop/SqoopActionExecutor.java 
> 22e28746a9a39ae6c1e22326cc4f8f759d820b20 
>   core/src/main/java/org/apache/oozie/client/rest/BulkResponseImpl.java 
> fb021bd022d199f93fc5674843514abf45ae12a5 
>   core/src/main/java/org/apache/oozie/command/XCommand.java 
> bdf13f65ea25e

Re: Review Request 54705: OOZIE-1770: Create Oozie Application Master for YARN

[Rohini Palaniswamy]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/54705/#review175946
---


Ship it!




Ship It!

- Rohini Palaniswamy


On May 24, 2017, 12:05 p.m., Peter Cseh wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/54705/
> ---
> 
> (Updated May 24, 2017, 12:05 p.m.)
> 
> 
> Review request for oozie, Purshotam Shah, Robert Kanter, and Rohini 
> Palaniswamy.
> 
> 
> Bugs: OOZIE-1770
> https://issues.apache.org/jira/browse/OOZIE-1770
> 
> 
> Repository: oozie-git
> 
> 
> Description
> ---
> 
> Replacing the launcher MR job with a launcher AM
> 
> 
> Diffs
> -
> 
>   client/pom.xml f99366c90887a5a45f3942291c90e1cdbfd84b3f 
>   client/src/main/java/org/apache/oozie/cli/OozieCLI.java 
> 38fb84e8a8918aee6ae57d19a29cdf181aba1955 
>   client/src/main/java/org/apache/oozie/client/XOozieClient.java 
> 8538ec7c742cc013f7c18eaff3ecf61c2835cc8a 
>   core/pom.xml 14aa03483e107db326628ba627cc3ee9f5fd3f3d 
>   core/src/main/java/org/apache/oozie/WorkflowJobBean.java 
> 55d79a5d5e5e4efab9eb832e9341a16bcf8959cc 
>   core/src/main/java/org/apache/oozie/action/ActionExecutor.java 
> 1d6456b236bf9e27c0f55d9f9e3b4bd94517c797 
>   core/src/main/java/org/apache/oozie/action/hadoop/Credentials.java 
> eadb47b07d157f45f086517ab41c4b9826f003f0 
>   core/src/main/java/org/apache/oozie/action/hadoop/CredentialsProvider.java 
> 6fe22fbd2fca5d873d58ece60f20d0f131371475 
>   
> core/src/main/java/org/apache/oozie/action/hadoop/CredentialsProviderFactory.java
>  PRE-CREATION 
>   core/src/main/java/org/apache/oozie/action/hadoop/DistcpActionExecutor.java 
> 78cd257a836707c1efeac13566d6285d42970545 
>   core/src/main/java/org/apache/oozie/action/hadoop/FsActionExecutor.java 
> 27654741101d2056b25626fbbe345183eafd612d 
>   core/src/main/java/org/apache/oozie/action/hadoop/FsELFunctions.java 
> 801bfe6eb212b549888f0e9fb7df7b65bcf59393 
>   core/src/main/java/org/apache/oozie/action/hadoop/HCatCredentialHelper.java 
> 36ad1df727e869b8051481ad969771e91bc6ae07 
>   core/src/main/java/org/apache/oozie/action/hadoop/HCatCredentials.java 
> d7689a924059e553abaeed374eed4072933292cb 
>   core/src/main/java/org/apache/oozie/action/hadoop/HadoopELFunctions.java 
> ad2a71d641caa71efc098ad00bfbab3cec4d9506 
>   core/src/main/java/org/apache/oozie/action/hadoop/HbaseCredentials.java 
> 307f565786ad77f5a35b5790bd5b052b04c6eac8 
>   core/src/main/java/org/apache/oozie/action/hadoop/Hive2ActionExecutor.java 
> 2aed936af75da38c433be5d202bdc01c2f612c76 
>   core/src/main/java/org/apache/oozie/action/hadoop/Hive2Credentials.java 
> bd2f1f768c5b04bef4a33613baede9202144c747 
>   core/src/main/java/org/apache/oozie/action/hadoop/HiveActionExecutor.java 
> 8e2453ef4f78f4a5f1f95b1ca2793f5d7e07ad81 
>   core/src/main/java/org/apache/oozie/action/hadoop/JavaActionExecutor.java 
> 06ae5fd9a52099c5bc63b038d336ce96103c8819 
>   core/src/main/java/org/apache/oozie/action/hadoop/LauncherMapperHelper.java 
> 72ed2f150be73f0a5e8b71b0fb6cc3594dc23078 
>   
> core/src/main/java/org/apache/oozie/action/hadoop/MapReduceActionExecutor.java
>  55c937249ac35e14bd63dd0c1fa962ea636afa3a 
>   core/src/main/java/org/apache/oozie/action/hadoop/OozieJobInfo.java 
> 581d3b3219eff882f961c8e495611e873c6a5a52 
>   core/src/main/java/org/apache/oozie/action/hadoop/PigActionExecutor.java 
> 8b2dc16db80ad602911f1f64e60cfad41b198922 
>   
> core/src/main/java/org/apache/oozie/action/hadoop/ScriptLanguageActionExecutor.java
>  92e149d5cac6509893c8507b776263a38312a53b 
>   core/src/main/java/org/apache/oozie/action/hadoop/ShellActionExecutor.java 
> b9ffa7ab7ec87db8a76a3653c74974d296c8cd17 
>   core/src/main/java/org/apache/oozie/action/hadoop/SparkActionExecutor.java 
> 1a3197abb52f80b545dcc8634a8cac7a281a9eac 
>   core/src/main/java/org/apache/oozie/action/hadoop/SqoopActionExecutor.java 
> 22e28746a9a39ae6c1e22326cc4f8f759d820b20 
>   core/src/main/java/org/apache/oozie/client/rest/BulkResponseImpl.java 
> fb021bd022d199f93fc5674843514abf45ae12a5 
>   core/src/main/java/org/apache/oozie/command/XCommand.java 
> bdf13f65ea25e8e0f5125e2798568a2c03fffe81 
>   core/src/main/java/org/apache/oozie/command/wf/SubmitHttpXCommand.java 
> d2a2742ae418070a1cd0a889f14194224bc944a6 
>   core/src/main/java/org/apache/oozie/command/wf/SubmitMRXCommand.java 
> cc61d3de813c75c7f1cf4d405fc043253e3ca4ea 
>   
> core/src/main/java/org/apache/oozie/command/wf/SubmitScriptLanguageXCommand.java
>  9d413059e99bc676feb63e02a2f704048eb4a355 
>   core/src/main/java/org/apache/oozie/command/wf/SubmitSqoopXCommand.java 
> 51b739e21c802e09f4110b740138f4ff5c341fe4 
>   
> core/src/main/java/org/apache/oozie/coord/input/dependency/AbstractCoordInputDependency.java
>  ace120d10e3546201aa4145df56a

Re: Review Request 54705: OOZIE-1770: Create Oozie Application Master for YARN

[Robert Kanter]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/54705/#review176104
---


Ship it!




Ship It!

- Robert Kanter


On May 24, 2017, 12:05 p.m., Peter Cseh wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/54705/
> ---
> 
> (Updated May 24, 2017, 12:05 p.m.)
> 
> 
> Review request for oozie, Purshotam Shah, Robert Kanter, and Rohini 
> Palaniswamy.
> 
> 
> Bugs: OOZIE-1770
> https://issues.apache.org/jira/browse/OOZIE-1770
> 
> 
> Repository: oozie-git
> 
> 
> Description
> ---
> 
> Replacing the launcher MR job with a launcher AM
> 
> 
> Diffs
> -
> 
>   client/pom.xml f99366c90887a5a45f3942291c90e1cdbfd84b3f 
>   client/src/main/java/org/apache/oozie/cli/OozieCLI.java 
> 38fb84e8a8918aee6ae57d19a29cdf181aba1955 
>   client/src/main/java/org/apache/oozie/client/XOozieClient.java 
> 8538ec7c742cc013f7c18eaff3ecf61c2835cc8a 
>   core/pom.xml 14aa03483e107db326628ba627cc3ee9f5fd3f3d 
>   core/src/main/java/org/apache/oozie/WorkflowJobBean.java 
> 55d79a5d5e5e4efab9eb832e9341a16bcf8959cc 
>   core/src/main/java/org/apache/oozie/action/ActionExecutor.java 
> 1d6456b236bf9e27c0f55d9f9e3b4bd94517c797 
>   core/src/main/java/org/apache/oozie/action/hadoop/Credentials.java 
> eadb47b07d157f45f086517ab41c4b9826f003f0 
>   core/src/main/java/org/apache/oozie/action/hadoop/CredentialsProvider.java 
> 6fe22fbd2fca5d873d58ece60f20d0f131371475 
>   
> core/src/main/java/org/apache/oozie/action/hadoop/CredentialsProviderFactory.java
>  PRE-CREATION 
>   core/src/main/java/org/apache/oozie/action/hadoop/DistcpActionExecutor.java 
> 78cd257a836707c1efeac13566d6285d42970545 
>   core/src/main/java/org/apache/oozie/action/hadoop/FsActionExecutor.java 
> 27654741101d2056b25626fbbe345183eafd612d 
>   core/src/main/java/org/apache/oozie/action/hadoop/FsELFunctions.java 
> 801bfe6eb212b549888f0e9fb7df7b65bcf59393 
>   core/src/main/java/org/apache/oozie/action/hadoop/HCatCredentialHelper.java 
> 36ad1df727e869b8051481ad969771e91bc6ae07 
>   core/src/main/java/org/apache/oozie/action/hadoop/HCatCredentials.java 
> d7689a924059e553abaeed374eed4072933292cb 
>   core/src/main/java/org/apache/oozie/action/hadoop/HadoopELFunctions.java 
> ad2a71d641caa71efc098ad00bfbab3cec4d9506 
>   core/src/main/java/org/apache/oozie/action/hadoop/HbaseCredentials.java 
> 307f565786ad77f5a35b5790bd5b052b04c6eac8 
>   core/src/main/java/org/apache/oozie/action/hadoop/Hive2ActionExecutor.java 
> 2aed936af75da38c433be5d202bdc01c2f612c76 
>   core/src/main/java/org/apache/oozie/action/hadoop/Hive2Credentials.java 
> bd2f1f768c5b04bef4a33613baede9202144c747 
>   core/src/main/java/org/apache/oozie/action/hadoop/HiveActionExecutor.java 
> 8e2453ef4f78f4a5f1f95b1ca2793f5d7e07ad81 
>   core/src/main/java/org/apache/oozie/action/hadoop/JavaActionExecutor.java 
> 06ae5fd9a52099c5bc63b038d336ce96103c8819 
>   core/src/main/java/org/apache/oozie/action/hadoop/LauncherMapperHelper.java 
> 72ed2f150be73f0a5e8b71b0fb6cc3594dc23078 
>   
> core/src/main/java/org/apache/oozie/action/hadoop/MapReduceActionExecutor.java
>  55c937249ac35e14bd63dd0c1fa962ea636afa3a 
>   core/src/main/java/org/apache/oozie/action/hadoop/OozieJobInfo.java 
> 581d3b3219eff882f961c8e495611e873c6a5a52 
>   core/src/main/java/org/apache/oozie/action/hadoop/PigActionExecutor.java 
> 8b2dc16db80ad602911f1f64e60cfad41b198922 
>   
> core/src/main/java/org/apache/oozie/action/hadoop/ScriptLanguageActionExecutor.java
>  92e149d5cac6509893c8507b776263a38312a53b 
>   core/src/main/java/org/apache/oozie/action/hadoop/ShellActionExecutor.java 
> b9ffa7ab7ec87db8a76a3653c74974d296c8cd17 
>   core/src/main/java/org/apache/oozie/action/hadoop/SparkActionExecutor.java 
> 1a3197abb52f80b545dcc8634a8cac7a281a9eac 
>   core/src/main/java/org/apache/oozie/action/hadoop/SqoopActionExecutor.java 
> 22e28746a9a39ae6c1e22326cc4f8f759d820b20 
>   core/src/main/java/org/apache/oozie/client/rest/BulkResponseImpl.java 
> fb021bd022d199f93fc5674843514abf45ae12a5 
>   core/src/main/java/org/apache/oozie/command/XCommand.java 
> bdf13f65ea25e8e0f5125e2798568a2c03fffe81 
>   core/src/main/java/org/apache/oozie/command/wf/SubmitHttpXCommand.java 
> d2a2742ae418070a1cd0a889f14194224bc944a6 
>   core/src/main/java/org/apache/oozie/command/wf/SubmitMRXCommand.java 
> cc61d3de813c75c7f1cf4d405fc043253e3ca4ea 
>   
> core/src/main/java/org/apache/oozie/command/wf/SubmitScriptLanguageXCommand.java
>  9d413059e99bc676feb63e02a2f704048eb4a355 
>   core/src/main/java/org/apache/oozie/command/wf/SubmitSqoopXCommand.java 
> 51b739e21c802e09f4110b740138f4ff5c341fe4 
>   
> core/src/main/java/org/apache/oozie/coord/input/dependency/AbstractCoordInputDependency.java
>  ace120d10e3546201aa4145df56abf793

Re: Review Request 54705: OOZIE-1770: Create Oozie Application Master for YARN

[Peter Cseh]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/54705/
---

(Updated Jan. 5, 2017, 1:16 p.m.)


Review request for oozie, Purshotam Shah, Robert Kanter, and Rohini Palaniswamy.


Repository: oozie-git


Description
---

Replacing the launcher MR job with a launcher AM


Diffs (updated)
-

  client/pom.xml f99366c90887a5a45f3942291c90e1cdbfd84b3f 
  client/src/main/java/org/apache/oozie/cli/OozieCLI.java 
e67fae9e2c7b32c1f209c24b4aba98daba96c06e 
  client/src/main/java/org/apache/oozie/client/XOozieClient.java 
8538ec7c742cc013f7c18eaff3ecf61c2835cc8a 
  core/pom.xml 1dd104581ccab7976d788b674d2200e48a70c27f 
  core/src/main/java/org/apache/oozie/WorkflowJobBean.java 
55d79a5d5e5e4efab9eb832e9341a16bcf8959cc 
  core/src/main/java/org/apache/oozie/action/ActionExecutor.java 
1d6456b236bf9e27c0f55d9f9e3b4bd94517c797 
  core/src/main/java/org/apache/oozie/action/hadoop/Credentials.java 
eadb47b07d157f45f086517ab41c4b9826f003f0 
  core/src/main/java/org/apache/oozie/action/hadoop/CredentialsProvider.java 
6fe22fbd2fca5d873d58ece60f20d0f131371475 
  core/src/main/java/org/apache/oozie/action/hadoop/DistcpActionExecutor.java 
78cd257a836707c1efeac13566d6285d42970545 
  core/src/main/java/org/apache/oozie/action/hadoop/FsActionExecutor.java 
27654741101d2056b25626fbbe345183eafd612d 
  core/src/main/java/org/apache/oozie/action/hadoop/FsELFunctions.java 
801bfe6eb212b549888f0e9fb7df7b65bcf59393 
  core/src/main/java/org/apache/oozie/action/hadoop/HCatCredentialHelper.java 
36ad1df727e869b8051481ad969771e91bc6ae07 
  core/src/main/java/org/apache/oozie/action/hadoop/HadoopELFunctions.java 
ad2a71d641caa71efc098ad00bfbab3cec4d9506 
  core/src/main/java/org/apache/oozie/action/hadoop/HbaseCredentials.java 
307f565786ad77f5a35b5790bd5b052b04c6eac8 
  core/src/main/java/org/apache/oozie/action/hadoop/Hive2ActionExecutor.java 
2aed936af75da38c433be5d202bdc01c2f612c76 
  core/src/main/java/org/apache/oozie/action/hadoop/HiveActionExecutor.java 
8e2453ef4f78f4a5f1f95b1ca2793f5d7e07ad81 
  core/src/main/java/org/apache/oozie/action/hadoop/JavaActionExecutor.java 
4beeb965a461e246479c3e8ad88f92ecafca13c1 
  core/src/main/java/org/apache/oozie/action/hadoop/LauncherMapperHelper.java 
9609fdc2f3440fb176b659ef16c2a23451724cf9 
  
core/src/main/java/org/apache/oozie/action/hadoop/MapReduceActionExecutor.java 
55c937249ac35e14bd63dd0c1fa962ea636afa3a 
  core/src/main/java/org/apache/oozie/action/hadoop/OozieJobInfo.java 
581d3b3219eff882f961c8e495611e873c6a5a52 
  core/src/main/java/org/apache/oozie/action/hadoop/PigActionExecutor.java 
8b2dc16db80ad602911f1f64e60cfad41b198922 
  
core/src/main/java/org/apache/oozie/action/hadoop/ScriptLanguageActionExecutor.java
 f2541265ca71fb7f3b053c59775ef5e23d0d8f94 
  core/src/main/java/org/apache/oozie/action/hadoop/ShellActionExecutor.java 
b9ffa7ab7ec87db8a76a3653c74974d296c8cd17 
  core/src/main/java/org/apache/oozie/action/hadoop/SparkActionExecutor.java 
8e8d7d308b294efd91e3057e0d88ee06643ab2d4 
  core/src/main/java/org/apache/oozie/action/hadoop/SqoopActionExecutor.java 
6cee32aa8f5b8b36c9c7a98bb738f50b18b64624 
  core/src/main/java/org/apache/oozie/client/rest/BulkResponseImpl.java 
fb021bd022d199f93fc5674843514abf45ae12a5 
  core/src/main/java/org/apache/oozie/command/XCommand.java 
bdf13f65ea25e8e0f5125e2798568a2c03fffe81 
  core/src/main/java/org/apache/oozie/command/wf/SubmitHttpXCommand.java 
d2a2742ae418070a1cd0a889f14194224bc944a6 
  core/src/main/java/org/apache/oozie/command/wf/SubmitMRXCommand.java 
cc61d3de813c75c7f1cf4d405fc043253e3ca4ea 
  
core/src/main/java/org/apache/oozie/command/wf/SubmitScriptLanguageXCommand.java
 9d413059e99bc676feb63e02a2f704048eb4a355 
  core/src/main/java/org/apache/oozie/command/wf/SubmitSqoopXCommand.java 
51b739e21c802e09f4110b740138f4ff5c341fe4 
  
core/src/main/java/org/apache/oozie/coord/input/dependency/AbstractCoordInputDependency.java
 0da60ec70373399414a637c22753da7e2a7e6618 
  
core/src/main/java/org/apache/oozie/dependency/hcat/EhcacheHCatDependencyCache.java
 3bc467535202e13387b1d29ac678573f4154c522 
  core/src/main/java/org/apache/oozie/service/CallableQueueService.java 
a86a8d0889138cd1564ca9253e5b319ea52e55cc 
  core/src/main/java/org/apache/oozie/service/EventHandlerService.java 
22c6fb0965f45c5cf2322c251d4a8aa77d263458 
  core/src/main/java/org/apache/oozie/service/HadoopAccessorService.java 
5377127736cf189bcf4aa3d62bbbeab6b3a29d52 
  core/src/main/java/org/apache/oozie/service/Services.java 
829d5f5cd90c86e6cba9854dc0b4b7df570b8900 
  core/src/main/java/org/apache/oozie/service/ShareLibService.java 
b79bd37bb2b44348f220128334497d788dd922d4 
  core/src/main/java/org/apache/oozie/servlet/CallbackServlet.java 
612302173e0f485345558b0c96a8a75d2f43d238 
  core/src/main/java/org/apache/oozie/store/OozieSchema.java 
23dd1e508b6c721a10c9fe828baa5cd0d95064b4 
  core/src

Re: Review Request 54705: OOZIE-1770: Create Oozie Application Master for YARN

[Robert Kanter]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/54705/#review160656
---




client/src/main/java/org/apache/oozie/client/XOozieClient.java (line 119)


"ResourceManager is not specified in conf"



core/src/main/java/org/apache/oozie/WorkflowJobBean.java (line 607)


I was looking at 
http://stackoverflow.com/questions/933447/how-do-you-cast-a-list-of-supertypes-to-a-list-of-subtypes.

This still generics an unchecked warning, so is there an advantage to this 
ugly double casting vs just doing (List)?



core/src/main/java/org/apache/oozie/action/hadoop/JavaActionExecutor.java (line 
250)


This should be changed to use Configuration.



core/src/main/java/org/apache/oozie/action/hadoop/JavaActionExecutor.java (line 
253)


JobConf is an MR thing.  We should use a Configuration



core/src/main/java/org/apache/oozie/action/hadoop/JavaActionExecutor.java (line 
254)


?



core/src/main/java/org/apache/oozie/action/hadoop/JavaActionExecutor.java 
(lines 262 - 266)


We'll probably need to change the OozieActionConfigurator interface 
to use a Configuration object instead of a JobConf object.



core/src/main/java/org/apache/oozie/action/hadoop/JavaActionExecutor.java 


Yay!!!



core/src/main/java/org/apache/oozie/action/hadoop/JavaActionExecutor.java 
(lines 991 - 1001)


Instead of doing this (which I know is equivalent to what we currently do), 
would it better to call check(...) instead?  I think it handles stuff 
better (e.g. fallback to data file, etc).  Or maybe we can extract out the 
common code into a new method and have them both call that, if we don't want 
all of check(...) here.



core/src/main/java/org/apache/oozie/action/hadoop/JavaActionExecutor.java 
(lines 1113 - 1117)


Do we need this?



core/src/main/java/org/apache/oozie/action/hadoop/JavaActionExecutor.java (line 
1115)


Does this mean the launcher logs can't be larger than 1GB?



core/src/main/java/org/apache/oozie/action/hadoop/JavaActionExecutor.java (line 
1341)


Do we still need this method?  We have createYarnClient now.



core/src/main/java/org/apache/oozie/action/hadoop/JavaActionExecutor.java (line 
1396)


We should try to replace this with Configuration



core/src/main/java/org/apache/oozie/action/hadoop/JavaActionExecutor.java (line 
1538)


Let's give a more helpful message here.



core/src/main/java/org/apache/oozie/action/hadoop/LauncherMapperHelper.java 
(line 148)


I don't think we need this anymore?  Probably not for a long time.



core/src/main/java/org/apache/oozie/action/hadoop/SparkActionExecutor.java 
(lines 156 - 158)


We shouldn't need the MR jars in the Spark Action :)



core/src/main/java/org/apache/oozie/command/wf/SubmitMRXCommand.java (line 48)


This isn't deprecated, is it?



core/src/main/java/org/apache/oozie/service/HadoopAccessorService.java (line 
538)


(I think I originally wrote this Javadoc but,) I think we should add an 
additional sentence:

Return a YarnClient created with the provided user and configuration.  The 
caller is responsible for closing it when done.



core/src/main/resources/oozie-log4j.properties (line 35)


This is the issue we discussed in today's meeting about the logger breaking 
things, right?

In any case, I don't think we want to get rid of this.



core/src/test/java/org/apache/oozie/action/hadoop/TestJavaActionExecutor.java 
(lines 268 - 269)


?



core/src/test/java/org/apache/oozie/action/hadoop/TestJavaActionExecutor.java 
(lines 1640 - 1674)


What's the plan here?



core/src/test/java/org/apache/oozie/client/TestOozieCLI.java (lines 137 - 138)


900

Re: Review Request 54705: OOZIE-1770: Create Oozie Application Master for YARN

[Peter Cseh]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/54705/
---

(Updated Jan. 6, 2017, 1:59 p.m.)


Review request for oozie, Purshotam Shah, Robert Kanter, and Rohini Palaniswamy.


Repository: oozie-git


Description
---

Replacing the launcher MR job with a launcher AM


Diffs (updated)
-

  client/pom.xml f99366c90887a5a45f3942291c90e1cdbfd84b3f 
  client/src/main/java/org/apache/oozie/cli/OozieCLI.java 
e67fae9e2c7b32c1f209c24b4aba98daba96c06e 
  client/src/main/java/org/apache/oozie/client/XOozieClient.java 
8538ec7c742cc013f7c18eaff3ecf61c2835cc8a 
  core/pom.xml 1dd104581ccab7976d788b674d2200e48a70c27f 
  core/src/main/java/org/apache/oozie/WorkflowJobBean.java 
55d79a5d5e5e4efab9eb832e9341a16bcf8959cc 
  core/src/main/java/org/apache/oozie/action/ActionExecutor.java 
1d6456b236bf9e27c0f55d9f9e3b4bd94517c797 
  core/src/main/java/org/apache/oozie/action/hadoop/Credentials.java 
eadb47b07d157f45f086517ab41c4b9826f003f0 
  core/src/main/java/org/apache/oozie/action/hadoop/CredentialsProvider.java 
6fe22fbd2fca5d873d58ece60f20d0f131371475 
  core/src/main/java/org/apache/oozie/action/hadoop/DistcpActionExecutor.java 
78cd257a836707c1efeac13566d6285d42970545 
  core/src/main/java/org/apache/oozie/action/hadoop/FsActionExecutor.java 
27654741101d2056b25626fbbe345183eafd612d 
  core/src/main/java/org/apache/oozie/action/hadoop/FsELFunctions.java 
801bfe6eb212b549888f0e9fb7df7b65bcf59393 
  core/src/main/java/org/apache/oozie/action/hadoop/HCatCredentialHelper.java 
36ad1df727e869b8051481ad969771e91bc6ae07 
  core/src/main/java/org/apache/oozie/action/hadoop/HadoopELFunctions.java 
ad2a71d641caa71efc098ad00bfbab3cec4d9506 
  core/src/main/java/org/apache/oozie/action/hadoop/HbaseCredentials.java 
307f565786ad77f5a35b5790bd5b052b04c6eac8 
  core/src/main/java/org/apache/oozie/action/hadoop/Hive2ActionExecutor.java 
2aed936af75da38c433be5d202bdc01c2f612c76 
  core/src/main/java/org/apache/oozie/action/hadoop/HiveActionExecutor.java 
8e2453ef4f78f4a5f1f95b1ca2793f5d7e07ad81 
  core/src/main/java/org/apache/oozie/action/hadoop/JavaActionExecutor.java 
4beeb965a461e246479c3e8ad88f92ecafca13c1 
  core/src/main/java/org/apache/oozie/action/hadoop/LauncherMapperHelper.java 
9609fdc2f3440fb176b659ef16c2a23451724cf9 
  
core/src/main/java/org/apache/oozie/action/hadoop/MapReduceActionExecutor.java 
55c937249ac35e14bd63dd0c1fa962ea636afa3a 
  core/src/main/java/org/apache/oozie/action/hadoop/OozieJobInfo.java 
581d3b3219eff882f961c8e495611e873c6a5a52 
  core/src/main/java/org/apache/oozie/action/hadoop/PigActionExecutor.java 
8b2dc16db80ad602911f1f64e60cfad41b198922 
  
core/src/main/java/org/apache/oozie/action/hadoop/ScriptLanguageActionExecutor.java
 f2541265ca71fb7f3b053c59775ef5e23d0d8f94 
  core/src/main/java/org/apache/oozie/action/hadoop/ShellActionExecutor.java 
b9ffa7ab7ec87db8a76a3653c74974d296c8cd17 
  core/src/main/java/org/apache/oozie/action/hadoop/SparkActionExecutor.java 
8e8d7d308b294efd91e3057e0d88ee06643ab2d4 
  core/src/main/java/org/apache/oozie/action/hadoop/SqoopActionExecutor.java 
6cee32aa8f5b8b36c9c7a98bb738f50b18b64624 
  core/src/main/java/org/apache/oozie/client/rest/BulkResponseImpl.java 
fb021bd022d199f93fc5674843514abf45ae12a5 
  core/src/main/java/org/apache/oozie/command/XCommand.java 
bdf13f65ea25e8e0f5125e2798568a2c03fffe81 
  core/src/main/java/org/apache/oozie/command/wf/SubmitHttpXCommand.java 
d2a2742ae418070a1cd0a889f14194224bc944a6 
  core/src/main/java/org/apache/oozie/command/wf/SubmitMRXCommand.java 
cc61d3de813c75c7f1cf4d405fc043253e3ca4ea 
  
core/src/main/java/org/apache/oozie/command/wf/SubmitScriptLanguageXCommand.java
 9d413059e99bc676feb63e02a2f704048eb4a355 
  core/src/main/java/org/apache/oozie/command/wf/SubmitSqoopXCommand.java 
51b739e21c802e09f4110b740138f4ff5c341fe4 
  
core/src/main/java/org/apache/oozie/coord/input/dependency/AbstractCoordInputDependency.java
 0da60ec70373399414a637c22753da7e2a7e6618 
  
core/src/main/java/org/apache/oozie/dependency/hcat/EhcacheHCatDependencyCache.java
 3bc467535202e13387b1d29ac678573f4154c522 
  core/src/main/java/org/apache/oozie/service/CallableQueueService.java 
a86a8d0889138cd1564ca9253e5b319ea52e55cc 
  core/src/main/java/org/apache/oozie/service/EventHandlerService.java 
22c6fb0965f45c5cf2322c251d4a8aa77d263458 
  core/src/main/java/org/apache/oozie/service/HadoopAccessorService.java 
5377127736cf189bcf4aa3d62bbbeab6b3a29d52 
  core/src/main/java/org/apache/oozie/service/Services.java 
829d5f5cd90c86e6cba9854dc0b4b7df570b8900 
  core/src/main/java/org/apache/oozie/service/ShareLibService.java 
b79bd37bb2b44348f220128334497d788dd922d4 
  core/src/main/java/org/apache/oozie/servlet/CallbackServlet.java 
612302173e0f485345558b0c96a8a75d2f43d238 
  core/src/main/java/org/apache/oozie/store/OozieSchema.java 
23dd1e508b6c721a10c9fe828baa5cd0d95064b4 
  core/src

Re: Review Request 54705: OOZIE-1770: Create Oozie Application Master for YARN

[Peter Bacsko]

> On jan. 6, 2017, 7:19 de, Robert Kanter wrote:
> > core/src/main/java/org/apache/oozie/action/hadoop/JavaActionExecutor.java, 
> > lines 1328-1332
> > 
> >
> > Do we need this?

We might not need -Dlog4j.debug (although I think it's useful when we have 
logging problems) but the rest was copy-pasted from MRApps.java or already 
existed (your early code). The hadoop.root.logger and hadoop.root.logfile props 
are definitely necessary, perhaps the root loggers could be configurable.


> On jan. 6, 2017, 7:19 de, Robert Kanter wrote:
> > core/src/main/java/org/apache/oozie/action/hadoop/JavaActionExecutor.java, 
> > line 1330
> > 
> >
> > Does this mean the launcher logs can't be larger than 1GB?

I don't know - this value was already here :)


> On jan. 6, 2017, 7:19 de, Robert Kanter wrote:
> > core/src/main/java/org/apache/oozie/action/hadoop/JavaActionExecutor.java, 
> > line 1556
> > 
> >
> > Do we still need this method?  We have createYarnClient now.

It's not used directly by JavaActionExecutor, but MapReduceActionExecutor and 
SqoopActionExecutor need job clients in check() / end ().


> On jan. 6, 2017, 7:19 de, Robert Kanter wrote:
> > core/src/main/resources/oozie-log4j.properties, lines 36-56
> > 
> >
> > This is the issue we discussed in today's meeting about the logger 
> > breaking things, right?
> > 
> > In any case, I don't think we want to get rid of this.

We will restore this to the original.


> On jan. 6, 2017, 7:19 de, Robert Kanter wrote:
> > core/src/test/java/org/apache/oozie/action/hadoop/TestJavaActionExecutor.java,
> >  lines 279-280
> > 
> >
> > ?

Ah, I forgot about this.


> On jan. 6, 2017, 7:19 de, Robert Kanter wrote:
> > core/src/test/java/org/apache/oozie/action/hadoop/TestJavaActionExecutor.java,
> >  lines 1790-1824
> > 
> >
> > What's the plan here?

Perhaps we don't need this test anymore.

I'm not entirely familiar with MapReduce ACLs but I think it's related to the 
LauncherMapper. But we don't have a mapper anymore, but a standalone AM. Do we 
still care about MR ACLs?


> On jan. 6, 2017, 7:19 de, Robert Kanter wrote:
> > sharelib/hive2/src/main/java/org/apache/oozie/action/hadoop/Hive2Main.java, 
> > line 272
> > 
> >
> > I'm not sure we need to print the security manager here

Right, I forgot to remove this.


> On jan. 6, 2017, 7:19 de, Robert Kanter wrote:
> > docs/src/site/twiki/DG_QuickStart.twiki, line 51
> > 
> >
> > 0.12.1 < 0.16.0, right?

This might be due to a merge problem, will check.


> On jan. 6, 2017, 7:19 de, Robert Kanter wrote:
> > core/src/main/java/org/apache/oozie/action/hadoop/JavaActionExecutor.java, 
> > lines 1195-1205
> > 
> >
> > Instead of doing this (which I know is equivalent to what we currently 
> > do), would it better to call check(...) instead?  I think it 
> > handles stuff better (e.g. fallback to data file, etc).  Or maybe we can 
> > extract out the common code into a new method and have them both call that, 
> > if we don't want all of check(...) here.

You mean to call check() when the Launcher is already running?


> On jan. 6, 2017, 7:19 de, Robert Kanter wrote:
> > core/src/main/java/org/apache/oozie/WorkflowJobBean.java, line 608
> > 
> >
> > I was looking at 
> > http://stackoverflow.com/questions/933447/how-do-you-cast-a-list-of-supertypes-to-a-list-of-subtypes.
> > 
> > This still generics an unchecked warning, so is there an advantage to 
> > this ugly double casting vs just doing (List)?

I think I was just experimenting with something and ended up committing this. 
Agree, this has no advantage at all.


- Peter


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/54705/#review160656
---


On jan. 6, 2017, 1:59 du, Peter Cseh wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/54705/
> ---
> 
> (Updated jan. 6, 2017, 1:59 du)
> 
> 
> Review request for oozie

Re: Review Request 54705: OOZIE-1770: Create Oozie Application Master for YARN

[Peter Bacsko]

> On jan. 6, 2017, 7:19 de, Robert Kanter wrote:
> > hadooplibs/hadoop-utils-1/src/main/java/org/apache/oozie/action/hadoop/LauncherMainHadoopUtils.java,
> >  line 30
> > 
> >
> > Why do we need a factory for this?  Creating one seems pretty 
> > straightforward.

It helps unit testing. When you write unit test for LauncherAM, you want to 
avoid creating real LauncherCallbackNotifier instances because when you invoke 
notifyURL() it would connect to a real URL. If you mock the factory, it can 
return notifier mocks and you can do Mockito.verify() on them.

I know it seems like overkill but I think it's a good pattern to enhance 
testability.


> On jan. 6, 2017, 7:19 de, Robert Kanter wrote:
> > findbugs-filter.xml, line 18
> > 
> >
> > I'm fine with ignoring some findbugs stuff (Hadoop has a ton of these), 
> > but I think we should add a comment for each exclusion explaining it.  As 
> > it stands, I only know the class and findbugs pattern name.  It would be 
> > nice to at least narrow it down to the method and a reason why we can 
> > exclude it.

OK, I'll add some comments.


- Peter


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/54705/#review160656
---


On jan. 6, 2017, 1:59 du, Peter Cseh wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/54705/
> ---
> 
> (Updated jan. 6, 2017, 1:59 du)
> 
> 
> Review request for oozie, Purshotam Shah, Robert Kanter, and Rohini 
> Palaniswamy.
> 
> 
> Repository: oozie-git
> 
> 
> Description
> ---
> 
> Replacing the launcher MR job with a launcher AM
> 
> 
> Diffs
> -
> 
>   client/pom.xml f99366c90887a5a45f3942291c90e1cdbfd84b3f 
>   client/src/main/java/org/apache/oozie/cli/OozieCLI.java 
> e67fae9e2c7b32c1f209c24b4aba98daba96c06e 
>   client/src/main/java/org/apache/oozie/client/XOozieClient.java 
> 8538ec7c742cc013f7c18eaff3ecf61c2835cc8a 
>   core/pom.xml 1dd104581ccab7976d788b674d2200e48a70c27f 
>   core/src/main/java/org/apache/oozie/WorkflowJobBean.java 
> 55d79a5d5e5e4efab9eb832e9341a16bcf8959cc 
>   core/src/main/java/org/apache/oozie/action/ActionExecutor.java 
> 1d6456b236bf9e27c0f55d9f9e3b4bd94517c797 
>   core/src/main/java/org/apache/oozie/action/hadoop/Credentials.java 
> eadb47b07d157f45f086517ab41c4b9826f003f0 
>   core/src/main/java/org/apache/oozie/action/hadoop/CredentialsProvider.java 
> 6fe22fbd2fca5d873d58ece60f20d0f131371475 
>   core/src/main/java/org/apache/oozie/action/hadoop/DistcpActionExecutor.java 
> 78cd257a836707c1efeac13566d6285d42970545 
>   core/src/main/java/org/apache/oozie/action/hadoop/FsActionExecutor.java 
> 27654741101d2056b25626fbbe345183eafd612d 
>   core/src/main/java/org/apache/oozie/action/hadoop/FsELFunctions.java 
> 801bfe6eb212b549888f0e9fb7df7b65bcf59393 
>   core/src/main/java/org/apache/oozie/action/hadoop/HCatCredentialHelper.java 
> 36ad1df727e869b8051481ad969771e91bc6ae07 
>   core/src/main/java/org/apache/oozie/action/hadoop/HadoopELFunctions.java 
> ad2a71d641caa71efc098ad00bfbab3cec4d9506 
>   core/src/main/java/org/apache/oozie/action/hadoop/HbaseCredentials.java 
> 307f565786ad77f5a35b5790bd5b052b04c6eac8 
>   core/src/main/java/org/apache/oozie/action/hadoop/Hive2ActionExecutor.java 
> 2aed936af75da38c433be5d202bdc01c2f612c76 
>   core/src/main/java/org/apache/oozie/action/hadoop/HiveActionExecutor.java 
> 8e2453ef4f78f4a5f1f95b1ca2793f5d7e07ad81 
>   core/src/main/java/org/apache/oozie/action/hadoop/JavaActionExecutor.java 
> 4beeb965a461e246479c3e8ad88f92ecafca13c1 
>   core/src/main/java/org/apache/oozie/action/hadoop/LauncherMapperHelper.java 
> 9609fdc2f3440fb176b659ef16c2a23451724cf9 
>   
> core/src/main/java/org/apache/oozie/action/hadoop/MapReduceActionExecutor.java
>  55c937249ac35e14bd63dd0c1fa962ea636afa3a 
>   core/src/main/java/org/apache/oozie/action/hadoop/OozieJobInfo.java 
> 581d3b3219eff882f961c8e495611e873c6a5a52 
>   core/src/main/java/org/apache/oozie/action/hadoop/PigActionExecutor.java 
> 8b2dc16db80ad602911f1f64e60cfad41b198922 
>   
> core/src/main/java/org/apache/oozie/action/hadoop/ScriptLanguageActionExecutor.java
>  f2541265ca71fb7f3b053c59775ef5e23d0d8f94 
>   core/src/main/java/org/apache/oozie/action/hadoop/ShellActionExecutor.java 
> b9ffa7ab7ec87db8a76a3653c74974d296c8cd17 
>   core/src/main/java/org/apache/oozie/action/hadoop/SparkActionExecutor.java 
> 8e8d7d308b294efd91e3057e0d88ee06643ab2d4 
>   core/src/main/java/org/apache/oozie/action/hadoop/SqoopActionExecutor.java 
> 6cee32aa8f5b8b36c9c7a98bb738f50b18b64624 
>   core/src/main/java/org/apache/oozie/client/rest/BulkResp

Re: Review Request 54705: OOZIE-1770: Create Oozie Application Master for YARN

[Peter Bacsko]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/54705/#review160725
---



Random question - Since we're abandoning LauncherMapper, I think we should 
delete it. But then how we'll provide fixes for this class in the future?


core/src/main/java/org/apache/oozie/action/hadoop/JavaActionExecutor.java 
(lines 129 - 135)


Since we use our own App master, do we care about all these properties?

I checked MapReduceActionExecutor which calls super.setupActionConf() and 
super.setupLauncherConf(). But even if we use these props, it should be 
relevant only for the MR action.

Thoughts?



core/src/main/java/org/apache/oozie/action/hadoop/JavaActionExecutor.java (line 
1074)


This will be addressed in OOZIE-2687. For the time being IMO it's ok to 
have 0 here.



core/src/main/java/org/apache/oozie/action/hadoop/JavaActionExecutor.java (line 
1141)


Same here -- OOZIE-2687



core/src/main/java/org/apache/oozie/action/hadoop/LauncherMapperHelper.java 
(line 56)


I'm going to rename this, it's still used but we don't have a 
LauncherMapper anymore.


estion

- Peter Bacsko


On jan. 6, 2017, 1:59 du, Peter Cseh wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/54705/
> ---
> 
> (Updated jan. 6, 2017, 1:59 du)
> 
> 
> Review request for oozie, Purshotam Shah, Robert Kanter, and Rohini 
> Palaniswamy.
> 
> 
> Repository: oozie-git
> 
> 
> Description
> ---
> 
> Replacing the launcher MR job with a launcher AM
> 
> 
> Diffs
> -
> 
>   client/pom.xml f99366c90887a5a45f3942291c90e1cdbfd84b3f 
>   client/src/main/java/org/apache/oozie/cli/OozieCLI.java 
> e67fae9e2c7b32c1f209c24b4aba98daba96c06e 
>   client/src/main/java/org/apache/oozie/client/XOozieClient.java 
> 8538ec7c742cc013f7c18eaff3ecf61c2835cc8a 
>   core/pom.xml 1dd104581ccab7976d788b674d2200e48a70c27f 
>   core/src/main/java/org/apache/oozie/WorkflowJobBean.java 
> 55d79a5d5e5e4efab9eb832e9341a16bcf8959cc 
>   core/src/main/java/org/apache/oozie/action/ActionExecutor.java 
> 1d6456b236bf9e27c0f55d9f9e3b4bd94517c797 
>   core/src/main/java/org/apache/oozie/action/hadoop/Credentials.java 
> eadb47b07d157f45f086517ab41c4b9826f003f0 
>   core/src/main/java/org/apache/oozie/action/hadoop/CredentialsProvider.java 
> 6fe22fbd2fca5d873d58ece60f20d0f131371475 
>   core/src/main/java/org/apache/oozie/action/hadoop/DistcpActionExecutor.java 
> 78cd257a836707c1efeac13566d6285d42970545 
>   core/src/main/java/org/apache/oozie/action/hadoop/FsActionExecutor.java 
> 27654741101d2056b25626fbbe345183eafd612d 
>   core/src/main/java/org/apache/oozie/action/hadoop/FsELFunctions.java 
> 801bfe6eb212b549888f0e9fb7df7b65bcf59393 
>   core/src/main/java/org/apache/oozie/action/hadoop/HCatCredentialHelper.java 
> 36ad1df727e869b8051481ad969771e91bc6ae07 
>   core/src/main/java/org/apache/oozie/action/hadoop/HadoopELFunctions.java 
> ad2a71d641caa71efc098ad00bfbab3cec4d9506 
>   core/src/main/java/org/apache/oozie/action/hadoop/HbaseCredentials.java 
> 307f565786ad77f5a35b5790bd5b052b04c6eac8 
>   core/src/main/java/org/apache/oozie/action/hadoop/Hive2ActionExecutor.java 
> 2aed936af75da38c433be5d202bdc01c2f612c76 
>   core/src/main/java/org/apache/oozie/action/hadoop/HiveActionExecutor.java 
> 8e2453ef4f78f4a5f1f95b1ca2793f5d7e07ad81 
>   core/src/main/java/org/apache/oozie/action/hadoop/JavaActionExecutor.java 
> 4beeb965a461e246479c3e8ad88f92ecafca13c1 
>   core/src/main/java/org/apache/oozie/action/hadoop/LauncherMapperHelper.java 
> 9609fdc2f3440fb176b659ef16c2a23451724cf9 
>   
> core/src/main/java/org/apache/oozie/action/hadoop/MapReduceActionExecutor.java
>  55c937249ac35e14bd63dd0c1fa962ea636afa3a 
>   core/src/main/java/org/apache/oozie/action/hadoop/OozieJobInfo.java 
> 581d3b3219eff882f961c8e495611e873c6a5a52 
>   core/src/main/java/org/apache/oozie/action/hadoop/PigActionExecutor.java 
> 8b2dc16db80ad602911f1f64e60cfad41b198922 
>   
> core/src/main/java/org/apache/oozie/action/hadoop/ScriptLanguageActionExecutor.java
>  f2541265ca71fb7f3b053c59775ef5e23d0d8f94 
>   core/src/main/java/org/apache/oozie/action/hadoop/ShellActionExecutor.java 
> b9ffa7ab7ec87db8a76a3653c74974d296c8cd17 
>   core/src/main/java/org/apache/oozie/action/hadoop/SparkActionExecutor.java 
> 8e8d7d308b294efd91e3057e0d88ee06643ab2d4 
>   core/src/main/java/org/apache/oozie/action/hadoop/SqoopActionExecutor.java 
> 6cee32aa8f5b8b36c9c7a98bb738f50b18b64624 
>   core/src/main/java/org/apache/oozie/client/rest/BulkResponseImpl.java 
> fb

Re: Review Request 54705: OOZIE-1770: Create Oozie Application Master for YARN

[Robert Kanter]

> On Jan. 6, 2017, 4:37 p.m., Peter Bacsko wrote:
> > core/src/main/java/org/apache/oozie/action/hadoop/JavaActionExecutor.java, 
> > lines 134-140
> > 
> >
> > Since we use our own App master, do we care about all these properties?
> > 
> > I checked MapReduceActionExecutor which calls super.setupActionConf() 
> > and super.setupLauncherConf(). But even if we use these props, it should be 
> > relevant only for the MR action.
> > 
> > Thoughts?

We had discussed doing some backwards compatibility for some of these things 
for the launcher.  It might make sense to delete these for now (or leave 
them?), and OOZIE-2687 can deal with them later.


> On Jan. 6, 2017, 4:37 p.m., Peter Bacsko wrote:
> > core/src/main/java/org/apache/oozie/action/hadoop/JavaActionExecutor.java, 
> > line 1289
> > 
> >
> > This will be addressed in OOZIE-2687. For the time being IMO it's ok to 
> > have 0 here.

By the way, Yarn currently ignores the priority.  But I think we should still 
add it (via OOZIE-2687) in case Yarn starts honoring it in the future.


- Robert


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/54705/#review160725
---


On Jan. 6, 2017, 1:59 p.m., Peter Cseh wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/54705/
> ---
> 
> (Updated Jan. 6, 2017, 1:59 p.m.)
> 
> 
> Review request for oozie, Purshotam Shah, Robert Kanter, and Rohini 
> Palaniswamy.
> 
> 
> Repository: oozie-git
> 
> 
> Description
> ---
> 
> Replacing the launcher MR job with a launcher AM
> 
> 
> Diffs
> -
> 
>   client/pom.xml f99366c90887a5a45f3942291c90e1cdbfd84b3f 
>   client/src/main/java/org/apache/oozie/cli/OozieCLI.java 
> e67fae9e2c7b32c1f209c24b4aba98daba96c06e 
>   client/src/main/java/org/apache/oozie/client/XOozieClient.java 
> 8538ec7c742cc013f7c18eaff3ecf61c2835cc8a 
>   core/pom.xml 1dd104581ccab7976d788b674d2200e48a70c27f 
>   core/src/main/java/org/apache/oozie/WorkflowJobBean.java 
> 55d79a5d5e5e4efab9eb832e9341a16bcf8959cc 
>   core/src/main/java/org/apache/oozie/action/ActionExecutor.java 
> 1d6456b236bf9e27c0f55d9f9e3b4bd94517c797 
>   core/src/main/java/org/apache/oozie/action/hadoop/Credentials.java 
> eadb47b07d157f45f086517ab41c4b9826f003f0 
>   core/src/main/java/org/apache/oozie/action/hadoop/CredentialsProvider.java 
> 6fe22fbd2fca5d873d58ece60f20d0f131371475 
>   core/src/main/java/org/apache/oozie/action/hadoop/DistcpActionExecutor.java 
> 78cd257a836707c1efeac13566d6285d42970545 
>   core/src/main/java/org/apache/oozie/action/hadoop/FsActionExecutor.java 
> 27654741101d2056b25626fbbe345183eafd612d 
>   core/src/main/java/org/apache/oozie/action/hadoop/FsELFunctions.java 
> 801bfe6eb212b549888f0e9fb7df7b65bcf59393 
>   core/src/main/java/org/apache/oozie/action/hadoop/HCatCredentialHelper.java 
> 36ad1df727e869b8051481ad969771e91bc6ae07 
>   core/src/main/java/org/apache/oozie/action/hadoop/HadoopELFunctions.java 
> ad2a71d641caa71efc098ad00bfbab3cec4d9506 
>   core/src/main/java/org/apache/oozie/action/hadoop/HbaseCredentials.java 
> 307f565786ad77f5a35b5790bd5b052b04c6eac8 
>   core/src/main/java/org/apache/oozie/action/hadoop/Hive2ActionExecutor.java 
> 2aed936af75da38c433be5d202bdc01c2f612c76 
>   core/src/main/java/org/apache/oozie/action/hadoop/HiveActionExecutor.java 
> 8e2453ef4f78f4a5f1f95b1ca2793f5d7e07ad81 
>   core/src/main/java/org/apache/oozie/action/hadoop/JavaActionExecutor.java 
> 4beeb965a461e246479c3e8ad88f92ecafca13c1 
>   core/src/main/java/org/apache/oozie/action/hadoop/LauncherMapperHelper.java 
> 9609fdc2f3440fb176b659ef16c2a23451724cf9 
>   
> core/src/main/java/org/apache/oozie/action/hadoop/MapReduceActionExecutor.java
>  55c937249ac35e14bd63dd0c1fa962ea636afa3a 
>   core/src/main/java/org/apache/oozie/action/hadoop/OozieJobInfo.java 
> 581d3b3219eff882f961c8e495611e873c6a5a52 
>   core/src/main/java/org/apache/oozie/action/hadoop/PigActionExecutor.java 
> 8b2dc16db80ad602911f1f64e60cfad41b198922 
>   
> core/src/main/java/org/apache/oozie/action/hadoop/ScriptLanguageActionExecutor.java
>  f2541265ca71fb7f3b053c59775ef5e23d0d8f94 
>   core/src/main/java/org/apache/oozie/action/hadoop/ShellActionExecutor.java 
> b9ffa7ab7ec87db8a76a3653c74974d296c8cd17 
>   core/src/main/java/org/apache/oozie/action/hadoop/SparkActionExecutor.java 
> 8e8d7d308b294efd91e3057e0d88ee06643ab2d4 
>   core/src/main/java/org/apache/oozie/action/hadoop/SqoopActionExecutor.java 
> 6cee32aa8f5b8b36c9c7a98bb738f50b18b64624 
>   core/src/main/java/org/apache/oozie/client/rest/BulkResponseImpl.java 
> fb02

Re: Review Request 54705: OOZIE-1770: Create Oozie Application Master for YARN

[Peter Bacsko]

> On jan. 6, 2017, 7:19 de, Robert Kanter wrote:
> > core/src/main/java/org/apache/oozie/command/wf/SubmitMRXCommand.java, line 
> > 48
> > 
> >
> > This isn't deprecated, is it?

This is a result of Attila's commit: 
https://issues.apache.org/jira/browse/OOZIE-2316 although I'm not sure why this 
is in the map, because both XOozieClient.RM and "yarn.resourcemanager.address" 
are the same. I'll ask him.


- Peter


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/54705/#review160656
---


On jan. 6, 2017, 1:59 du, Peter Cseh wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/54705/
> ---
> 
> (Updated jan. 6, 2017, 1:59 du)
> 
> 
> Review request for oozie, Purshotam Shah, Robert Kanter, and Rohini 
> Palaniswamy.
> 
> 
> Repository: oozie-git
> 
> 
> Description
> ---
> 
> Replacing the launcher MR job with a launcher AM
> 
> 
> Diffs
> -
> 
>   client/pom.xml f99366c90887a5a45f3942291c90e1cdbfd84b3f 
>   client/src/main/java/org/apache/oozie/cli/OozieCLI.java 
> e67fae9e2c7b32c1f209c24b4aba98daba96c06e 
>   client/src/main/java/org/apache/oozie/client/XOozieClient.java 
> 8538ec7c742cc013f7c18eaff3ecf61c2835cc8a 
>   core/pom.xml 1dd104581ccab7976d788b674d2200e48a70c27f 
>   core/src/main/java/org/apache/oozie/WorkflowJobBean.java 
> 55d79a5d5e5e4efab9eb832e9341a16bcf8959cc 
>   core/src/main/java/org/apache/oozie/action/ActionExecutor.java 
> 1d6456b236bf9e27c0f55d9f9e3b4bd94517c797 
>   core/src/main/java/org/apache/oozie/action/hadoop/Credentials.java 
> eadb47b07d157f45f086517ab41c4b9826f003f0 
>   core/src/main/java/org/apache/oozie/action/hadoop/CredentialsProvider.java 
> 6fe22fbd2fca5d873d58ece60f20d0f131371475 
>   core/src/main/java/org/apache/oozie/action/hadoop/DistcpActionExecutor.java 
> 78cd257a836707c1efeac13566d6285d42970545 
>   core/src/main/java/org/apache/oozie/action/hadoop/FsActionExecutor.java 
> 27654741101d2056b25626fbbe345183eafd612d 
>   core/src/main/java/org/apache/oozie/action/hadoop/FsELFunctions.java 
> 801bfe6eb212b549888f0e9fb7df7b65bcf59393 
>   core/src/main/java/org/apache/oozie/action/hadoop/HCatCredentialHelper.java 
> 36ad1df727e869b8051481ad969771e91bc6ae07 
>   core/src/main/java/org/apache/oozie/action/hadoop/HadoopELFunctions.java 
> ad2a71d641caa71efc098ad00bfbab3cec4d9506 
>   core/src/main/java/org/apache/oozie/action/hadoop/HbaseCredentials.java 
> 307f565786ad77f5a35b5790bd5b052b04c6eac8 
>   core/src/main/java/org/apache/oozie/action/hadoop/Hive2ActionExecutor.java 
> 2aed936af75da38c433be5d202bdc01c2f612c76 
>   core/src/main/java/org/apache/oozie/action/hadoop/HiveActionExecutor.java 
> 8e2453ef4f78f4a5f1f95b1ca2793f5d7e07ad81 
>   core/src/main/java/org/apache/oozie/action/hadoop/JavaActionExecutor.java 
> 4beeb965a461e246479c3e8ad88f92ecafca13c1 
>   core/src/main/java/org/apache/oozie/action/hadoop/LauncherMapperHelper.java 
> 9609fdc2f3440fb176b659ef16c2a23451724cf9 
>   
> core/src/main/java/org/apache/oozie/action/hadoop/MapReduceActionExecutor.java
>  55c937249ac35e14bd63dd0c1fa962ea636afa3a 
>   core/src/main/java/org/apache/oozie/action/hadoop/OozieJobInfo.java 
> 581d3b3219eff882f961c8e495611e873c6a5a52 
>   core/src/main/java/org/apache/oozie/action/hadoop/PigActionExecutor.java 
> 8b2dc16db80ad602911f1f64e60cfad41b198922 
>   
> core/src/main/java/org/apache/oozie/action/hadoop/ScriptLanguageActionExecutor.java
>  f2541265ca71fb7f3b053c59775ef5e23d0d8f94 
>   core/src/main/java/org/apache/oozie/action/hadoop/ShellActionExecutor.java 
> b9ffa7ab7ec87db8a76a3653c74974d296c8cd17 
>   core/src/main/java/org/apache/oozie/action/hadoop/SparkActionExecutor.java 
> 8e8d7d308b294efd91e3057e0d88ee06643ab2d4 
>   core/src/main/java/org/apache/oozie/action/hadoop/SqoopActionExecutor.java 
> 6cee32aa8f5b8b36c9c7a98bb738f50b18b64624 
>   core/src/main/java/org/apache/oozie/client/rest/BulkResponseImpl.java 
> fb021bd022d199f93fc5674843514abf45ae12a5 
>   core/src/main/java/org/apache/oozie/command/XCommand.java 
> bdf13f65ea25e8e0f5125e2798568a2c03fffe81 
>   core/src/main/java/org/apache/oozie/command/wf/SubmitHttpXCommand.java 
> d2a2742ae418070a1cd0a889f14194224bc944a6 
>   core/src/main/java/org/apache/oozie/command/wf/SubmitMRXCommand.java 
> cc61d3de813c75c7f1cf4d405fc043253e3ca4ea 
>   
> core/src/main/java/org/apache/oozie/command/wf/SubmitScriptLanguageXCommand.java
>  9d413059e99bc676feb63e02a2f704048eb4a355 
>   core/src/main/java/org/apache/oozie/command/wf/SubmitSqoopXCommand.java 
> 51b739e21c802e09f4110b740138f4ff5c341fe4 
>   
> core/src/main/java/org/apache/oozie/coord/input/dependency/AbstractCoordInputDependency.java
>  0da60ec7

Re: Review Request 54705: OOZIE-1770: Create Oozie Application Master for YARN

[Attila Sasvari]

> On Jan. 6, 2017, 7:19 a.m., Robert Kanter wrote:
> > core/src/main/java/org/apache/oozie/command/wf/SubmitMRXCommand.java, line 
> > 48
> > 
> >
> > This isn't deprecated, is it?
> 
> Peter Bacsko wrote:
> This is a result of Attila's commit: 
> https://issues.apache.org/jira/browse/OOZIE-2316 although I'm not sure why 
> this is in the map, because both XOozieClient.RM and 
> "yarn.resourcemanager.address" are the same. I'll ask him.

It is mistake that somehow slipped through the previous review.


- Attila


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/54705/#review160656
---


On Jan. 6, 2017, 1:59 p.m., Peter Cseh wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/54705/
> ---
> 
> (Updated Jan. 6, 2017, 1:59 p.m.)
> 
> 
> Review request for oozie, Purshotam Shah, Robert Kanter, and Rohini 
> Palaniswamy.
> 
> 
> Repository: oozie-git
> 
> 
> Description
> ---
> 
> Replacing the launcher MR job with a launcher AM
> 
> 
> Diffs
> -
> 
>   client/pom.xml f99366c90887a5a45f3942291c90e1cdbfd84b3f 
>   client/src/main/java/org/apache/oozie/cli/OozieCLI.java 
> e67fae9e2c7b32c1f209c24b4aba98daba96c06e 
>   client/src/main/java/org/apache/oozie/client/XOozieClient.java 
> 8538ec7c742cc013f7c18eaff3ecf61c2835cc8a 
>   core/pom.xml 1dd104581ccab7976d788b674d2200e48a70c27f 
>   core/src/main/java/org/apache/oozie/WorkflowJobBean.java 
> 55d79a5d5e5e4efab9eb832e9341a16bcf8959cc 
>   core/src/main/java/org/apache/oozie/action/ActionExecutor.java 
> 1d6456b236bf9e27c0f55d9f9e3b4bd94517c797 
>   core/src/main/java/org/apache/oozie/action/hadoop/Credentials.java 
> eadb47b07d157f45f086517ab41c4b9826f003f0 
>   core/src/main/java/org/apache/oozie/action/hadoop/CredentialsProvider.java 
> 6fe22fbd2fca5d873d58ece60f20d0f131371475 
>   core/src/main/java/org/apache/oozie/action/hadoop/DistcpActionExecutor.java 
> 78cd257a836707c1efeac13566d6285d42970545 
>   core/src/main/java/org/apache/oozie/action/hadoop/FsActionExecutor.java 
> 27654741101d2056b25626fbbe345183eafd612d 
>   core/src/main/java/org/apache/oozie/action/hadoop/FsELFunctions.java 
> 801bfe6eb212b549888f0e9fb7df7b65bcf59393 
>   core/src/main/java/org/apache/oozie/action/hadoop/HCatCredentialHelper.java 
> 36ad1df727e869b8051481ad969771e91bc6ae07 
>   core/src/main/java/org/apache/oozie/action/hadoop/HadoopELFunctions.java 
> ad2a71d641caa71efc098ad00bfbab3cec4d9506 
>   core/src/main/java/org/apache/oozie/action/hadoop/HbaseCredentials.java 
> 307f565786ad77f5a35b5790bd5b052b04c6eac8 
>   core/src/main/java/org/apache/oozie/action/hadoop/Hive2ActionExecutor.java 
> 2aed936af75da38c433be5d202bdc01c2f612c76 
>   core/src/main/java/org/apache/oozie/action/hadoop/HiveActionExecutor.java 
> 8e2453ef4f78f4a5f1f95b1ca2793f5d7e07ad81 
>   core/src/main/java/org/apache/oozie/action/hadoop/JavaActionExecutor.java 
> 4beeb965a461e246479c3e8ad88f92ecafca13c1 
>   core/src/main/java/org/apache/oozie/action/hadoop/LauncherMapperHelper.java 
> 9609fdc2f3440fb176b659ef16c2a23451724cf9 
>   
> core/src/main/java/org/apache/oozie/action/hadoop/MapReduceActionExecutor.java
>  55c937249ac35e14bd63dd0c1fa962ea636afa3a 
>   core/src/main/java/org/apache/oozie/action/hadoop/OozieJobInfo.java 
> 581d3b3219eff882f961c8e495611e873c6a5a52 
>   core/src/main/java/org/apache/oozie/action/hadoop/PigActionExecutor.java 
> 8b2dc16db80ad602911f1f64e60cfad41b198922 
>   
> core/src/main/java/org/apache/oozie/action/hadoop/ScriptLanguageActionExecutor.java
>  f2541265ca71fb7f3b053c59775ef5e23d0d8f94 
>   core/src/main/java/org/apache/oozie/action/hadoop/ShellActionExecutor.java 
> b9ffa7ab7ec87db8a76a3653c74974d296c8cd17 
>   core/src/main/java/org/apache/oozie/action/hadoop/SparkActionExecutor.java 
> 8e8d7d308b294efd91e3057e0d88ee06643ab2d4 
>   core/src/main/java/org/apache/oozie/action/hadoop/SqoopActionExecutor.java 
> 6cee32aa8f5b8b36c9c7a98bb738f50b18b64624 
>   core/src/main/java/org/apache/oozie/client/rest/BulkResponseImpl.java 
> fb021bd022d199f93fc5674843514abf45ae12a5 
>   core/src/main/java/org/apache/oozie/command/XCommand.java 
> bdf13f65ea25e8e0f5125e2798568a2c03fffe81 
>   core/src/main/java/org/apache/oozie/command/wf/SubmitHttpXCommand.java 
> d2a2742ae418070a1cd0a889f14194224bc944a6 
>   core/src/main/java/org/apache/oozie/command/wf/SubmitMRXCommand.java 
> cc61d3de813c75c7f1cf4d405fc043253e3ca4ea 
>   
> core/src/main/java/org/apache/oozie/command/wf/SubmitScriptLanguageXCommand.java
>  9d413059e99bc676feb63e02a2f704048eb4a355 
>   core/src/main/java/org/apache/oozie/command/wf/SubmitSqoopXCommand.java 
> 51b739e21c802e09f4110b740138f4ff5c341fe4 
>  

Re: Review Request 54705: OOZIE-1770: Create Oozie Application Master for YARN

[Peter Cseh]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/54705/
---

(Updated Jan. 23, 2017, 11:43 a.m.)


Review request for oozie, Purshotam Shah, Robert Kanter, and Rohini Palaniswamy.


Repository: oozie-git


Description
---

Replacing the launcher MR job with a launcher AM


Diffs (updated)
-

  client/pom.xml f99366c90887a5a45f3942291c90e1cdbfd84b3f 
  client/src/main/java/org/apache/oozie/cli/OozieCLI.java 
6e30d7eb667fe74440281fab3ca06c9f4df7bcf7 
  client/src/main/java/org/apache/oozie/client/XOozieClient.java 
8538ec7c742cc013f7c18eaff3ecf61c2835cc8a 
  core/pom.xml 1dd104581ccab7976d788b674d2200e48a70c27f 
  core/src/main/java/org/apache/oozie/WorkflowJobBean.java 
55d79a5d5e5e4efab9eb832e9341a16bcf8959cc 
  core/src/main/java/org/apache/oozie/action/ActionExecutor.java 
1d6456b236bf9e27c0f55d9f9e3b4bd94517c797 
  core/src/main/java/org/apache/oozie/action/hadoop/Credentials.java 
eadb47b07d157f45f086517ab41c4b9826f003f0 
  core/src/main/java/org/apache/oozie/action/hadoop/CredentialsProvider.java 
6fe22fbd2fca5d873d58ece60f20d0f131371475 
  
core/src/main/java/org/apache/oozie/action/hadoop/CredentialsProviderFactory.java
 PRE-CREATION 
  core/src/main/java/org/apache/oozie/action/hadoop/DistcpActionExecutor.java 
78cd257a836707c1efeac13566d6285d42970545 
  core/src/main/java/org/apache/oozie/action/hadoop/FsActionExecutor.java 
27654741101d2056b25626fbbe345183eafd612d 
  core/src/main/java/org/apache/oozie/action/hadoop/FsELFunctions.java 
801bfe6eb212b549888f0e9fb7df7b65bcf59393 
  core/src/main/java/org/apache/oozie/action/hadoop/HCatCredentialHelper.java 
36ad1df727e869b8051481ad969771e91bc6ae07 
  core/src/main/java/org/apache/oozie/action/hadoop/HCatCredentials.java 
d7689a924059e553abaeed374eed4072933292cb 
  core/src/main/java/org/apache/oozie/action/hadoop/HadoopELFunctions.java 
ad2a71d641caa71efc098ad00bfbab3cec4d9506 
  core/src/main/java/org/apache/oozie/action/hadoop/HbaseCredentials.java 
307f565786ad77f5a35b5790bd5b052b04c6eac8 
  core/src/main/java/org/apache/oozie/action/hadoop/Hive2ActionExecutor.java 
2aed936af75da38c433be5d202bdc01c2f612c76 
  core/src/main/java/org/apache/oozie/action/hadoop/Hive2Credentials.java 
bd2f1f768c5b04bef4a33613baede9202144c747 
  core/src/main/java/org/apache/oozie/action/hadoop/HiveActionExecutor.java 
8e2453ef4f78f4a5f1f95b1ca2793f5d7e07ad81 
  core/src/main/java/org/apache/oozie/action/hadoop/JavaActionExecutor.java 
4beeb965a461e246479c3e8ad88f92ecafca13c1 
  core/src/main/java/org/apache/oozie/action/hadoop/LauncherMapperHelper.java 
9609fdc2f3440fb176b659ef16c2a23451724cf9 
  
core/src/main/java/org/apache/oozie/action/hadoop/MapReduceActionExecutor.java 
55c937249ac35e14bd63dd0c1fa962ea636afa3a 
  core/src/main/java/org/apache/oozie/action/hadoop/OozieJobInfo.java 
581d3b3219eff882f961c8e495611e873c6a5a52 
  core/src/main/java/org/apache/oozie/action/hadoop/PigActionExecutor.java 
8b2dc16db80ad602911f1f64e60cfad41b198922 
  
core/src/main/java/org/apache/oozie/action/hadoop/ScriptLanguageActionExecutor.java
 f2541265ca71fb7f3b053c59775ef5e23d0d8f94 
  core/src/main/java/org/apache/oozie/action/hadoop/ShellActionExecutor.java 
b9ffa7ab7ec87db8a76a3653c74974d296c8cd17 
  core/src/main/java/org/apache/oozie/action/hadoop/SparkActionExecutor.java 
8e8d7d308b294efd91e3057e0d88ee06643ab2d4 
  core/src/main/java/org/apache/oozie/action/hadoop/SqoopActionExecutor.java 
6cee32aa8f5b8b36c9c7a98bb738f50b18b64624 
  core/src/main/java/org/apache/oozie/client/rest/BulkResponseImpl.java 
fb021bd022d199f93fc5674843514abf45ae12a5 
  core/src/main/java/org/apache/oozie/command/XCommand.java 
bdf13f65ea25e8e0f5125e2798568a2c03fffe81 
  core/src/main/java/org/apache/oozie/command/wf/SubmitHttpXCommand.java 
d2a2742ae418070a1cd0a889f14194224bc944a6 
  core/src/main/java/org/apache/oozie/command/wf/SubmitMRXCommand.java 
cc61d3de813c75c7f1cf4d405fc043253e3ca4ea 
  
core/src/main/java/org/apache/oozie/command/wf/SubmitScriptLanguageXCommand.java
 9d413059e99bc676feb63e02a2f704048eb4a355 
  core/src/main/java/org/apache/oozie/command/wf/SubmitSqoopXCommand.java 
51b739e21c802e09f4110b740138f4ff5c341fe4 
  
core/src/main/java/org/apache/oozie/coord/input/dependency/AbstractCoordInputDependency.java
 0da60ec70373399414a637c22753da7e2a7e6618 
  
core/src/main/java/org/apache/oozie/dependency/hcat/EhcacheHCatDependencyCache.java
 3bc467535202e13387b1d29ac678573f4154c522 
  core/src/main/java/org/apache/oozie/service/CallableQueueService.java 
a86a8d0889138cd1564ca9253e5b319ea52e55cc 
  core/src/main/java/org/apache/oozie/service/EventHandlerService.java 
22c6fb0965f45c5cf2322c251d4a8aa77d263458 
  core/src/main/java/org/apache/oozie/service/HadoopAccessorService.java 
5377127736cf189bcf4aa3d62bbbeab6b3a29d52 
  core/src/main/java/org/apache/oozie/service/Services.java 
829d5f5cd90c86e6cba9854dc0b4b7df570b8900 

Re: Review Request 54705: OOZIE-1770: Create Oozie Application Master for YARN

[Peter Cseh]

> On Jan. 6, 2017, 7:19 a.m., Robert Kanter wrote:
> > core/src/main/java/org/apache/oozie/action/hadoop/SparkActionExecutor.java, 
> > lines 160-162
> > 
> >
> > We shouldn't need the MR jars in the Spark Action :)

Oozie's SparkExample needs MR jars on the classpath.


- Peter


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/54705/#review160656
---


On Jan. 23, 2017, 11:43 a.m., Peter Cseh wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/54705/
> ---
> 
> (Updated Jan. 23, 2017, 11:43 a.m.)
> 
> 
> Review request for oozie, Purshotam Shah, Robert Kanter, and Rohini 
> Palaniswamy.
> 
> 
> Repository: oozie-git
> 
> 
> Description
> ---
> 
> Replacing the launcher MR job with a launcher AM
> 
> 
> Diffs
> -
> 
>   client/pom.xml f99366c90887a5a45f3942291c90e1cdbfd84b3f 
>   client/src/main/java/org/apache/oozie/cli/OozieCLI.java 
> 6e30d7eb667fe74440281fab3ca06c9f4df7bcf7 
>   client/src/main/java/org/apache/oozie/client/XOozieClient.java 
> 8538ec7c742cc013f7c18eaff3ecf61c2835cc8a 
>   core/pom.xml 1dd104581ccab7976d788b674d2200e48a70c27f 
>   core/src/main/java/org/apache/oozie/WorkflowJobBean.java 
> 55d79a5d5e5e4efab9eb832e9341a16bcf8959cc 
>   core/src/main/java/org/apache/oozie/action/ActionExecutor.java 
> 1d6456b236bf9e27c0f55d9f9e3b4bd94517c797 
>   core/src/main/java/org/apache/oozie/action/hadoop/Credentials.java 
> eadb47b07d157f45f086517ab41c4b9826f003f0 
>   core/src/main/java/org/apache/oozie/action/hadoop/CredentialsProvider.java 
> 6fe22fbd2fca5d873d58ece60f20d0f131371475 
>   
> core/src/main/java/org/apache/oozie/action/hadoop/CredentialsProviderFactory.java
>  PRE-CREATION 
>   core/src/main/java/org/apache/oozie/action/hadoop/DistcpActionExecutor.java 
> 78cd257a836707c1efeac13566d6285d42970545 
>   core/src/main/java/org/apache/oozie/action/hadoop/FsActionExecutor.java 
> 27654741101d2056b25626fbbe345183eafd612d 
>   core/src/main/java/org/apache/oozie/action/hadoop/FsELFunctions.java 
> 801bfe6eb212b549888f0e9fb7df7b65bcf59393 
>   core/src/main/java/org/apache/oozie/action/hadoop/HCatCredentialHelper.java 
> 36ad1df727e869b8051481ad969771e91bc6ae07 
>   core/src/main/java/org/apache/oozie/action/hadoop/HCatCredentials.java 
> d7689a924059e553abaeed374eed4072933292cb 
>   core/src/main/java/org/apache/oozie/action/hadoop/HadoopELFunctions.java 
> ad2a71d641caa71efc098ad00bfbab3cec4d9506 
>   core/src/main/java/org/apache/oozie/action/hadoop/HbaseCredentials.java 
> 307f565786ad77f5a35b5790bd5b052b04c6eac8 
>   core/src/main/java/org/apache/oozie/action/hadoop/Hive2ActionExecutor.java 
> 2aed936af75da38c433be5d202bdc01c2f612c76 
>   core/src/main/java/org/apache/oozie/action/hadoop/Hive2Credentials.java 
> bd2f1f768c5b04bef4a33613baede9202144c747 
>   core/src/main/java/org/apache/oozie/action/hadoop/HiveActionExecutor.java 
> 8e2453ef4f78f4a5f1f95b1ca2793f5d7e07ad81 
>   core/src/main/java/org/apache/oozie/action/hadoop/JavaActionExecutor.java 
> 4beeb965a461e246479c3e8ad88f92ecafca13c1 
>   core/src/main/java/org/apache/oozie/action/hadoop/LauncherMapperHelper.java 
> 9609fdc2f3440fb176b659ef16c2a23451724cf9 
>   
> core/src/main/java/org/apache/oozie/action/hadoop/MapReduceActionExecutor.java
>  55c937249ac35e14bd63dd0c1fa962ea636afa3a 
>   core/src/main/java/org/apache/oozie/action/hadoop/OozieJobInfo.java 
> 581d3b3219eff882f961c8e495611e873c6a5a52 
>   core/src/main/java/org/apache/oozie/action/hadoop/PigActionExecutor.java 
> 8b2dc16db80ad602911f1f64e60cfad41b198922 
>   
> core/src/main/java/org/apache/oozie/action/hadoop/ScriptLanguageActionExecutor.java
>  f2541265ca71fb7f3b053c59775ef5e23d0d8f94 
>   core/src/main/java/org/apache/oozie/action/hadoop/ShellActionExecutor.java 
> b9ffa7ab7ec87db8a76a3653c74974d296c8cd17 
>   core/src/main/java/org/apache/oozie/action/hadoop/SparkActionExecutor.java 
> 8e8d7d308b294efd91e3057e0d88ee06643ab2d4 
>   core/src/main/java/org/apache/oozie/action/hadoop/SqoopActionExecutor.java 
> 6cee32aa8f5b8b36c9c7a98bb738f50b18b64624 
>   core/src/main/java/org/apache/oozie/client/rest/BulkResponseImpl.java 
> fb021bd022d199f93fc5674843514abf45ae12a5 
>   core/src/main/java/org/apache/oozie/command/XCommand.java 
> bdf13f65ea25e8e0f5125e2798568a2c03fffe81 
>   core/src/main/java/org/apache/oozie/command/wf/SubmitHttpXCommand.java 
> d2a2742ae418070a1cd0a889f14194224bc944a6 
>   core/src/main/java/org/apache/oozie/command/wf/SubmitMRXCommand.java 
> cc61d3de813c75c7f1cf4d405fc043253e3ca4ea 
>   
> core/src/main/java/org/apache/oozie/command/wf/SubmitScriptLanguageXCommand.java
>  9d413059e99bc676feb63e02a2f704048eb4a355 
>   core/src/

Re: Review Request 54705: OOZIE-1770: Create Oozie Application Master for YARN

[Robert Kanter]

> On Jan. 6, 2017, 7:19 a.m., Robert Kanter wrote:
> > core/src/main/java/org/apache/oozie/action/hadoop/JavaActionExecutor.java
> > Lines 1143-1148 (original), 991-1001 (patched)
> > 
> >
> > Instead of doing this (which I know is equivalent to what we currently 
> > do), would it better to call check(...) instead?  I think it 
> > handles stuff better (e.g. fallback to data file, etc).  Or maybe we can 
> > extract out the common code into a new method and have them both call that, 
> > if we don't want all of check(...) here.
> 
> Peter Bacsko wrote:
> You mean to call check() when the Launcher is already running?

Hmm, I don't remember what I was getting at here; let's ignore this for now.


> On Jan. 6, 2017, 7:19 a.m., Robert Kanter wrote:
> > core/src/main/java/org/apache/oozie/action/hadoop/JavaActionExecutor.java
> > Lines 1113-1117 (patched)
> > 
> >
> > Do we need this?
> 
> Peter Bacsko wrote:
> We might not need -Dlog4j.debug (although I think it's useful when we 
> have logging problems) but the rest was copy-pasted from MRApps.java or 
> already existed (your early code). The hadoop.root.logger and 
> hadoop.root.logfile props are definitely necessary, perhaps the root loggers 
> could be configurable.

The other stuff sounds fine then, but the log4j.debug=true turns on 
debug logging for Log4j itself, right?  I don't think we need that.


> On Jan. 6, 2017, 7:19 a.m., Robert Kanter wrote:
> > core/src/main/java/org/apache/oozie/action/hadoop/JavaActionExecutor.java
> > Lines 1115 (patched)
> > 
> >
> > Does this mean the launcher logs can't be larger than 1GB?
> 
> Peter Bacsko wrote:
> I don't know - this value was already here :)

It looks like this gets passed to the log4j config for the container.  It 
essentially sets the max filesize for when to rollover the log file.  Looking 
at how MRApps sets it 
(https://github.com/apache/hadoop/blob/f67237cbe7bc48a1b9088e990800b37529f1db2a/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-common/src/main/java/org/apache/hadoop/mapreduce/v2/util/MRApps.java#L710)
 to MRJobConfig.MR_AM_LOG_KB, which is 0 by default (indicating no rollover).  
Given that, I'm thinking we should set this to 0 so we don't rollover - I'm not 
sure how well Yarn will handle a rolled over log file.


> On Jan. 6, 2017, 7:19 a.m., Robert Kanter wrote:
> > core/src/main/java/org/apache/oozie/action/hadoop/SparkActionExecutor.java
> > Lines 156-158 (patched)
> > 
> >
> > We shouldn't need the MR jars in the Spark Action :)
> 
> Peter Cseh wrote:
> Oozie's SparkExample needs MR jars on the classpath.

That's surprising, but okay.


> On Jan. 6, 2017, 7:19 a.m., Robert Kanter wrote:
> > core/src/test/java/org/apache/oozie/action/hadoop/TestJavaActionExecutor.java
> > Lines 1787-1821 (original), 1640-1674 (patched)
> > 
> >
> > What's the plan here?
> 
> Peter Bacsko wrote:
> Perhaps we don't need this test anymore.
> 
> I'm not entirely familiar with MapReduce ACLs but I think it's related to 
> the LauncherMapper. But we don't have a mapper anymore, but a standalone AM. 
> Do we still care about MR ACLs?

I'm not either, but it sounds like this doesn't apply anymore so we should 
delete it.


- Robert


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/54705/#review160656
---


On Jan. 23, 2017, 11:43 a.m., Peter Cseh wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/54705/
> ---
> 
> (Updated Jan. 23, 2017, 11:43 a.m.)
> 
> 
> Review request for oozie, Purshotam Shah, Robert Kanter, and Rohini 
> Palaniswamy.
> 
> 
> Repository: oozie-git
> 
> 
> Description
> ---
> 
> Replacing the launcher MR job with a launcher AM
> 
> 
> Diffs
> -
> 
>   client/pom.xml f99366c90887a5a45f3942291c90e1cdbfd84b3f 
>   client/src/main/java/org/apache/oozie/cli/OozieCLI.java 
> 6e30d7eb667fe74440281fab3ca06c9f4df7bcf7 
>   client/src/main/java/org/apache/oozie/client/XOozieClient.java 
> 8538ec7c742cc013f7c18eaff3ecf61c2835cc8a 
>   core/pom.xml 1dd104581ccab7976d788b674d2200e48a70c27f 
>   core/src/main/java/org/apache/oozie/WorkflowJobBean.java 
> 55d79a5d5e5e4efab9eb832e9341a16bcf8959cc 
>   core/src/main/java/org/apache/oozie/action/ActionExecutor.java 
> 1d6456b236bf9e27c0f55d9f9e3b4bd94

Re: Review Request 54705: OOZIE-1770: Create Oozie Application Master for YARN

[Abhishek Bafna]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/54705/#review169460
---




core/src/main/java/org/apache/oozie/action/hadoop/CredentialsProviderFactory.java
Lines 37 (patched)


Instead of '+' can we use '{0}' index based parameters.



core/src/main/java/org/apache/oozie/action/hadoop/JavaActionExecutor.java
Lines 1376 (patched)


should not this be a error level message instead of debug?



core/src/main/java/org/apache/oozie/action/hadoop/MapReduceActionExecutor.java
Lines 319 (patched)


Should we log previous value as well here before updating it?

Can this be defined as constant "job.end.notification.url" and then used?



core/src/main/java/org/apache/oozie/service/Services.java
Lines 307 (patched)


Can we use a log instance to capture the stack trace?



core/src/main/java/org/apache/oozie/service/Services.java
Line 308 (original), 308 (patched)


Can this be converted into index and parameter based log statement?



core/src/main/resources/META-INF/persistence.xml
Lines 79 (patched)


Will this print user credential information as well?

From 
here:https://openjpa.apache.org/builds/1.2.3/apache-openjpa/docs/ref_guide_dbsetup.html

PrintParameters: A boolean indicating whether SQL parameter values will be 
included in exception text and when logging is enabled. Since the parameter 
values may contain sensitive information the default value is false.



core/src/test/java/org/apache/oozie/action/hadoop/TestJavaActionExecutor.java
Line 122 (original), 113 (patched)


I believe "yarn.resourcemanager.address" is defined as constant some where, 
can we use that. If not, please define as constant. Thanks.



core/src/test/java/org/apache/oozie/action/hadoop/TestJavaActionExecutor.java
Line 2890 (original), 2256 (patched)


"yarn.resourcemanager.address" this is hardcoded many places, please 
replace it by constant.



core/src/test/java/org/apache/oozie/client/TestOozieCLI.java
Line 137 (original), 137 (patched)


Can you keep the port number same as previous, like it is kept below?



core/src/test/java/org/apache/oozie/command/wf/HangServlet.java
Lines 49 (patched)


Please use index and parameter based logging.



sharelib/oozie/src/main/java/org/apache/oozie/action/hadoop/JavaMain.java
Line 51 (original), 51 (patched)


Why have we moved back to old way instead os using the utility method?



sharelib/oozie/src/test/java/org/apache/oozie/action/hadoop/LauncherAMTestMainClass.java
Lines 35 (patched)


Can this be converted into a switch case?
Why not check lenght of the array equal to 1 instead of > 0.



sharelib/sqoop/src/main/java/org/apache/oozie/action/hadoop/SqoopMain.java
Line 169 (original), 169 (patched)


With this, we will loose the masking of the sensitive information like 
password etc?

Any specific reason for using doing it this way and not using the utility 
method?


- Abhishek Bafna


On Jan. 23, 2017, 11:43 a.m., Peter Cseh wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/54705/
> ---
> 
> (Updated Jan. 23, 2017, 11:43 a.m.)
> 
> 
> Review request for oozie, Purshotam Shah, Robert Kanter, and Rohini 
> Palaniswamy.
> 
> 
> Repository: oozie-git
> 
> 
> Description
> ---
> 
> Replacing the launcher MR job with a launcher AM
> 
> 
> Diffs
> -
> 
>   client/pom.xml f99366c90887a5a45f3942291c90e1cdbfd84b3f 
>   client/src/main/java/org/apache/oozie/cli/OozieCLI.java 
> 6e30d7eb667fe74440281fab3ca06c9f4df7bcf7 
>   client/src/main/java/org/apache/oozie/client/XOozieClient.java 
> 8538ec7c742cc013f7c18eaff3ecf61c2835cc8a 
>   core/pom.xml 1dd104581ccab7976d788b674d2200e48a70c27f 
>   core/src/main/java/org/apache/oozie/WorkflowJobBean.java 
> 55d79a5d5e5e4efab9eb832e9341a16bcf8959cc 
>   core/src/main/java/org/apache/oozie/action/ActionExecutor.java 
> 1d6456b236bf9e27c0f55d9f9e3b4bd94517c797 
>   core/src/main/java/org/apache/oozie/action/hadoop/Credentials.java 
> eadb47b07d157f45f086517ab41c4b9826f003f0 
>   core/sr

Re: Review Request 54705: OOZIE-1770: Create Oozie Application Master for YARN

[Robert Kanter]

> On March 20, 2017, 5:34 p.m., Abhishek Bafna wrote:
> > core/src/test/java/org/apache/oozie/action/hadoop/TestJavaActionExecutor.java
> > Line 122 (original), 113 (patched)
> > 
> >
> > I believe "yarn.resourcemanager.address" is defined as constant some 
> > where, can we use that. If not, please define as constant. Thanks.

There's sort of two camps on this, but for tests, I like to hardcode these 
sorts of things.  That way, if we (accidently) change the constant, it breaks 
the test so we notice.


> On March 20, 2017, 5:34 p.m., Abhishek Bafna wrote:
> > core/src/test/java/org/apache/oozie/client/TestOozieCLI.java
> > Line 137 (original), 137 (patched)
> > 
> >
> > Can you keep the port number same as previous, like it is kept below?

9000 is a really really old NN port.  I think we should use the normal ports.  
This is also a problem with some of the examples in our docs.


- Robert


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/54705/#review169460
---


On Jan. 23, 2017, 11:43 a.m., Peter Cseh wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/54705/
> ---
> 
> (Updated Jan. 23, 2017, 11:43 a.m.)
> 
> 
> Review request for oozie, Purshotam Shah, Robert Kanter, and Rohini 
> Palaniswamy.
> 
> 
> Repository: oozie-git
> 
> 
> Description
> ---
> 
> Replacing the launcher MR job with a launcher AM
> 
> 
> Diffs
> -
> 
>   client/pom.xml f99366c90887a5a45f3942291c90e1cdbfd84b3f 
>   client/src/main/java/org/apache/oozie/cli/OozieCLI.java 
> 6e30d7eb667fe74440281fab3ca06c9f4df7bcf7 
>   client/src/main/java/org/apache/oozie/client/XOozieClient.java 
> 8538ec7c742cc013f7c18eaff3ecf61c2835cc8a 
>   core/pom.xml 1dd104581ccab7976d788b674d2200e48a70c27f 
>   core/src/main/java/org/apache/oozie/WorkflowJobBean.java 
> 55d79a5d5e5e4efab9eb832e9341a16bcf8959cc 
>   core/src/main/java/org/apache/oozie/action/ActionExecutor.java 
> 1d6456b236bf9e27c0f55d9f9e3b4bd94517c797 
>   core/src/main/java/org/apache/oozie/action/hadoop/Credentials.java 
> eadb47b07d157f45f086517ab41c4b9826f003f0 
>   core/src/main/java/org/apache/oozie/action/hadoop/CredentialsProvider.java 
> 6fe22fbd2fca5d873d58ece60f20d0f131371475 
>   
> core/src/main/java/org/apache/oozie/action/hadoop/CredentialsProviderFactory.java
>  PRE-CREATION 
>   core/src/main/java/org/apache/oozie/action/hadoop/DistcpActionExecutor.java 
> 78cd257a836707c1efeac13566d6285d42970545 
>   core/src/main/java/org/apache/oozie/action/hadoop/FsActionExecutor.java 
> 27654741101d2056b25626fbbe345183eafd612d 
>   core/src/main/java/org/apache/oozie/action/hadoop/FsELFunctions.java 
> 801bfe6eb212b549888f0e9fb7df7b65bcf59393 
>   core/src/main/java/org/apache/oozie/action/hadoop/HCatCredentialHelper.java 
> 36ad1df727e869b8051481ad969771e91bc6ae07 
>   core/src/main/java/org/apache/oozie/action/hadoop/HCatCredentials.java 
> d7689a924059e553abaeed374eed4072933292cb 
>   core/src/main/java/org/apache/oozie/action/hadoop/HadoopELFunctions.java 
> ad2a71d641caa71efc098ad00bfbab3cec4d9506 
>   core/src/main/java/org/apache/oozie/action/hadoop/HbaseCredentials.java 
> 307f565786ad77f5a35b5790bd5b052b04c6eac8 
>   core/src/main/java/org/apache/oozie/action/hadoop/Hive2ActionExecutor.java 
> 2aed936af75da38c433be5d202bdc01c2f612c76 
>   core/src/main/java/org/apache/oozie/action/hadoop/Hive2Credentials.java 
> bd2f1f768c5b04bef4a33613baede9202144c747 
>   core/src/main/java/org/apache/oozie/action/hadoop/HiveActionExecutor.java 
> 8e2453ef4f78f4a5f1f95b1ca2793f5d7e07ad81 
>   core/src/main/java/org/apache/oozie/action/hadoop/JavaActionExecutor.java 
> 4beeb965a461e246479c3e8ad88f92ecafca13c1 
>   core/src/main/java/org/apache/oozie/action/hadoop/LauncherMapperHelper.java 
> 9609fdc2f3440fb176b659ef16c2a23451724cf9 
>   
> core/src/main/java/org/apache/oozie/action/hadoop/MapReduceActionExecutor.java
>  55c937249ac35e14bd63dd0c1fa962ea636afa3a 
>   core/src/main/java/org/apache/oozie/action/hadoop/OozieJobInfo.java 
> 581d3b3219eff882f961c8e495611e873c6a5a52 
>   core/src/main/java/org/apache/oozie/action/hadoop/PigActionExecutor.java 
> 8b2dc16db80ad602911f1f64e60cfad41b198922 
>   
> core/src/main/java/org/apache/oozie/action/hadoop/ScriptLanguageActionExecutor.java
>  f2541265ca71fb7f3b053c59775ef5e23d0d8f94 
>   core/src/main/java/org/apache/oozie/action/hadoop/ShellActionExecutor.java 
> b9ffa7ab7ec87db8a76a3653c74974d296c8cd17 
>   core/src/main/java/org/apache/oozie/action/hadoop/SparkActionExecutor.java 
> 8e8d7d308b294efd91e3057e0d88ee06643ab2d4 
>   core/src/main/java/org/ap

Re: Review Request 54705: OOZIE-1770: Create Oozie Application Master for YARN

[Peter Cseh]

> On Jan. 6, 2017, 7:19 a.m., Robert Kanter wrote:
> > core/src/main/java/org/apache/oozie/action/hadoop/JavaActionExecutor.java
> > Lines 262-266 (patched)
> > 
> >
> > We'll probably need to change the OozieActionConfigurator 
> > interface to use a Configuration object instead of a 
> > JobConf object.

The OozieActionConfigurator is used only for MR actions based on it's apidoc. 
Is the launchermapper itself configurable via this? If so we might want to drop 
that option. If it's related only for MR actions, we can keep the JobConf here.


- Peter


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/54705/#review160656
---


On Jan. 23, 2017, 11:43 a.m., Peter Cseh wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/54705/
> ---
> 
> (Updated Jan. 23, 2017, 11:43 a.m.)
> 
> 
> Review request for oozie, Purshotam Shah, Robert Kanter, and Rohini 
> Palaniswamy.
> 
> 
> Repository: oozie-git
> 
> 
> Description
> ---
> 
> Replacing the launcher MR job with a launcher AM
> 
> 
> Diffs
> -
> 
>   client/pom.xml f99366c90887a5a45f3942291c90e1cdbfd84b3f 
>   client/src/main/java/org/apache/oozie/cli/OozieCLI.java 
> 6e30d7eb667fe74440281fab3ca06c9f4df7bcf7 
>   client/src/main/java/org/apache/oozie/client/XOozieClient.java 
> 8538ec7c742cc013f7c18eaff3ecf61c2835cc8a 
>   core/pom.xml 1dd104581ccab7976d788b674d2200e48a70c27f 
>   core/src/main/java/org/apache/oozie/WorkflowJobBean.java 
> 55d79a5d5e5e4efab9eb832e9341a16bcf8959cc 
>   core/src/main/java/org/apache/oozie/action/ActionExecutor.java 
> 1d6456b236bf9e27c0f55d9f9e3b4bd94517c797 
>   core/src/main/java/org/apache/oozie/action/hadoop/Credentials.java 
> eadb47b07d157f45f086517ab41c4b9826f003f0 
>   core/src/main/java/org/apache/oozie/action/hadoop/CredentialsProvider.java 
> 6fe22fbd2fca5d873d58ece60f20d0f131371475 
>   
> core/src/main/java/org/apache/oozie/action/hadoop/CredentialsProviderFactory.java
>  PRE-CREATION 
>   core/src/main/java/org/apache/oozie/action/hadoop/DistcpActionExecutor.java 
> 78cd257a836707c1efeac13566d6285d42970545 
>   core/src/main/java/org/apache/oozie/action/hadoop/FsActionExecutor.java 
> 27654741101d2056b25626fbbe345183eafd612d 
>   core/src/main/java/org/apache/oozie/action/hadoop/FsELFunctions.java 
> 801bfe6eb212b549888f0e9fb7df7b65bcf59393 
>   core/src/main/java/org/apache/oozie/action/hadoop/HCatCredentialHelper.java 
> 36ad1df727e869b8051481ad969771e91bc6ae07 
>   core/src/main/java/org/apache/oozie/action/hadoop/HCatCredentials.java 
> d7689a924059e553abaeed374eed4072933292cb 
>   core/src/main/java/org/apache/oozie/action/hadoop/HadoopELFunctions.java 
> ad2a71d641caa71efc098ad00bfbab3cec4d9506 
>   core/src/main/java/org/apache/oozie/action/hadoop/HbaseCredentials.java 
> 307f565786ad77f5a35b5790bd5b052b04c6eac8 
>   core/src/main/java/org/apache/oozie/action/hadoop/Hive2ActionExecutor.java 
> 2aed936af75da38c433be5d202bdc01c2f612c76 
>   core/src/main/java/org/apache/oozie/action/hadoop/Hive2Credentials.java 
> bd2f1f768c5b04bef4a33613baede9202144c747 
>   core/src/main/java/org/apache/oozie/action/hadoop/HiveActionExecutor.java 
> 8e2453ef4f78f4a5f1f95b1ca2793f5d7e07ad81 
>   core/src/main/java/org/apache/oozie/action/hadoop/JavaActionExecutor.java 
> 4beeb965a461e246479c3e8ad88f92ecafca13c1 
>   core/src/main/java/org/apache/oozie/action/hadoop/LauncherMapperHelper.java 
> 9609fdc2f3440fb176b659ef16c2a23451724cf9 
>   
> core/src/main/java/org/apache/oozie/action/hadoop/MapReduceActionExecutor.java
>  55c937249ac35e14bd63dd0c1fa962ea636afa3a 
>   core/src/main/java/org/apache/oozie/action/hadoop/OozieJobInfo.java 
> 581d3b3219eff882f961c8e495611e873c6a5a52 
>   core/src/main/java/org/apache/oozie/action/hadoop/PigActionExecutor.java 
> 8b2dc16db80ad602911f1f64e60cfad41b198922 
>   
> core/src/main/java/org/apache/oozie/action/hadoop/ScriptLanguageActionExecutor.java
>  f2541265ca71fb7f3b053c59775ef5e23d0d8f94 
>   core/src/main/java/org/apache/oozie/action/hadoop/ShellActionExecutor.java 
> b9ffa7ab7ec87db8a76a3653c74974d296c8cd17 
>   core/src/main/java/org/apache/oozie/action/hadoop/SparkActionExecutor.java 
> 8e8d7d308b294efd91e3057e0d88ee06643ab2d4 
>   core/src/main/java/org/apache/oozie/action/hadoop/SqoopActionExecutor.java 
> 6cee32aa8f5b8b36c9c7a98bb738f50b18b64624 
>   core/src/main/java/org/apache/oozie/client/rest/BulkResponseImpl.java 
> fb021bd022d199f93fc5674843514abf45ae12a5 
>   core/src/main/java/org/apache/oozie/command/XCommand.java 
> bdf13f65ea25e8e0f5125e2798568a2c03fffe81 
>   core/src/main/java/org/apache/oozie/command/wf/SubmitHttpXCommand.java 
> d2a2

Re: Review Request 54705: OOZIE-1770: Create Oozie Application Master for YARN

[Peter Cseh]

> On March 20, 2017, 5:34 p.m., Abhishek Bafna wrote:
> > core/src/main/java/org/apache/oozie/action/hadoop/JavaActionExecutor.java
> > Lines 1376 (patched)
> > 
> >
> > should not this be a error level message instead of debug?

We have a secondary mechanism to check the action status based on a file on 
HDFS. We'll throw an exception in line 1391 if the fallback mechanism fails. 
I've increased the log level to info to make it easier to find out why the 
initial check failed.


> On March 20, 2017, 5:34 p.m., Abhishek Bafna wrote:
> > core/src/main/java/org/apache/oozie/service/Services.java
> > Lines 307 (patched)
> > 
> >
> > Can we use a log instance to capture the stack trace?

I think it gets logged because of the exeption later. I've added a log entry to 
make it sure.


> On March 20, 2017, 5:34 p.m., Abhishek Bafna wrote:
> > core/src/main/resources/META-INF/persistence.xml
> > Lines 79 (patched)
> > 
> >
> > Will this print user credential information as well?
> > 
> > From 
> > here:https://openjpa.apache.org/builds/1.2.3/apache-openjpa/docs/ref_guide_dbsetup.html
> > 
> > PrintParameters: A boolean indicating whether SQL parameter values will 
> > be included in exception text and when logging is enabled. Since the 
> > parameter values may contain sensitive information the default value is 
> > false.

Good catch! This was added for development reasons.


> On March 20, 2017, 5:34 p.m., Abhishek Bafna wrote:
> > core/src/test/java/org/apache/oozie/action/hadoop/TestJavaActionExecutor.java
> > Line 122 (original), 113 (patched)
> > 
> >
> > I believe "yarn.resourcemanager.address" is defined as constant some 
> > where, can we use that. If not, please define as constant. Thanks.
> 
> Robert Kanter wrote:
> There's sort of two camps on this, but for tests, I like to hardcode 
> these sorts of things.  That way, if we (accidently) change the constant, it 
> breaks the test so we notice.

I think I'll create a constant for it in the test though.


- Peter


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/54705/#review169460
---


On Jan. 23, 2017, 11:43 a.m., Peter Cseh wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/54705/
> ---
> 
> (Updated Jan. 23, 2017, 11:43 a.m.)
> 
> 
> Review request for oozie, Purshotam Shah, Robert Kanter, and Rohini 
> Palaniswamy.
> 
> 
> Repository: oozie-git
> 
> 
> Description
> ---
> 
> Replacing the launcher MR job with a launcher AM
> 
> 
> Diffs
> -
> 
>   client/pom.xml f99366c90887a5a45f3942291c90e1cdbfd84b3f 
>   client/src/main/java/org/apache/oozie/cli/OozieCLI.java 
> 6e30d7eb667fe74440281fab3ca06c9f4df7bcf7 
>   client/src/main/java/org/apache/oozie/client/XOozieClient.java 
> 8538ec7c742cc013f7c18eaff3ecf61c2835cc8a 
>   core/pom.xml 1dd104581ccab7976d788b674d2200e48a70c27f 
>   core/src/main/java/org/apache/oozie/WorkflowJobBean.java 
> 55d79a5d5e5e4efab9eb832e9341a16bcf8959cc 
>   core/src/main/java/org/apache/oozie/action/ActionExecutor.java 
> 1d6456b236bf9e27c0f55d9f9e3b4bd94517c797 
>   core/src/main/java/org/apache/oozie/action/hadoop/Credentials.java 
> eadb47b07d157f45f086517ab41c4b9826f003f0 
>   core/src/main/java/org/apache/oozie/action/hadoop/CredentialsProvider.java 
> 6fe22fbd2fca5d873d58ece60f20d0f131371475 
>   
> core/src/main/java/org/apache/oozie/action/hadoop/CredentialsProviderFactory.java
>  PRE-CREATION 
>   core/src/main/java/org/apache/oozie/action/hadoop/DistcpActionExecutor.java 
> 78cd257a836707c1efeac13566d6285d42970545 
>   core/src/main/java/org/apache/oozie/action/hadoop/FsActionExecutor.java 
> 27654741101d2056b25626fbbe345183eafd612d 
>   core/src/main/java/org/apache/oozie/action/hadoop/FsELFunctions.java 
> 801bfe6eb212b549888f0e9fb7df7b65bcf59393 
>   core/src/main/java/org/apache/oozie/action/hadoop/HCatCredentialHelper.java 
> 36ad1df727e869b8051481ad969771e91bc6ae07 
>   core/src/main/java/org/apache/oozie/action/hadoop/HCatCredentials.java 
> d7689a924059e553abaeed374eed4072933292cb 
>   core/src/main/java/org/apache/oozie/action/hadoop/HadoopELFunctions.java 
> ad2a71d641caa71efc098ad00bfbab3cec4d9506 
>   core/src/main/java/org/apache/oozie/action/hadoop/HbaseCredentials.java 
> 307f565786ad77f5a35b5790bd5b052b04c6eac8 
>   core/src/main/java/org/apache/oozie/action/hadoop/Hive2ActionExecutor.java 
> 2aed936af75da38c433be5d202bdc01c2f612c76 

Re: Review Request 54705: OOZIE-1770: Create Oozie Application Master for YARN

[Peter Cseh]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/54705/
---

(Updated March 22, 2017, 8:34 p.m.)


Review request for oozie, Purshotam Shah, Robert Kanter, and Rohini Palaniswamy.


Repository: oozie-git


Description
---

Replacing the launcher MR job with a launcher AM


Diffs (updated)
-

  bin/test-patch-20-tests 22f472c07e80fdb38c632329a5707998ab352172 
  client/pom.xml f99366c90887a5a45f3942291c90e1cdbfd84b3f 
  client/src/main/java/org/apache/oozie/cli/OozieCLI.java 
38fb84e8a8918aee6ae57d19a29cdf181aba1955 
  client/src/main/java/org/apache/oozie/cli/ValidationUtil.java 
b1da2af070970add01031d6208a395520cd7055c 
  client/src/main/java/org/apache/oozie/client/OozieClient.java 
73708080e624e775f7d058cd3377ad69d8399264 
  client/src/main/java/org/apache/oozie/client/XOozieClient.java 
8538ec7c742cc013f7c18eaff3ecf61c2835cc8a 
  client/src/main/java/org/apache/oozie/client/rest/JsonTags.java 
ca168e084998b5dd564536078a7a69efd7291828 
  client/src/main/java/org/apache/oozie/client/rest/RestConstants.java 
8ddb1f873f0c69aadf5f4a5161d8b21f7392fa1e 
  core/pom.xml 1dd104581ccab7976d788b674d2200e48a70c27f 
  core/src/main/java/org/apache/oozie/WorkflowJobBean.java 
55d79a5d5e5e4efab9eb832e9341a16bcf8959cc 
  core/src/main/java/org/apache/oozie/action/ActionExecutor.java 
1d6456b236bf9e27c0f55d9f9e3b4bd94517c797 
  core/src/main/java/org/apache/oozie/action/hadoop/Credentials.java 
eadb47b07d157f45f086517ab41c4b9826f003f0 
  core/src/main/java/org/apache/oozie/action/hadoop/CredentialsProvider.java 
6fe22fbd2fca5d873d58ece60f20d0f131371475 
  
core/src/main/java/org/apache/oozie/action/hadoop/CredentialsProviderFactory.java
 PRE-CREATION 
  core/src/main/java/org/apache/oozie/action/hadoop/DistcpActionExecutor.java 
78cd257a836707c1efeac13566d6285d42970545 
  core/src/main/java/org/apache/oozie/action/hadoop/FsActionExecutor.java 
27654741101d2056b25626fbbe345183eafd612d 
  core/src/main/java/org/apache/oozie/action/hadoop/FsELFunctions.java 
801bfe6eb212b549888f0e9fb7df7b65bcf59393 
  core/src/main/java/org/apache/oozie/action/hadoop/HCatCredentialHelper.java 
36ad1df727e869b8051481ad969771e91bc6ae07 
  core/src/main/java/org/apache/oozie/action/hadoop/HCatCredentials.java 
d7689a924059e553abaeed374eed4072933292cb 
  core/src/main/java/org/apache/oozie/action/hadoop/HadoopELFunctions.java 
ad2a71d641caa71efc098ad00bfbab3cec4d9506 
  core/src/main/java/org/apache/oozie/action/hadoop/HbaseCredentials.java 
307f565786ad77f5a35b5790bd5b052b04c6eac8 
  core/src/main/java/org/apache/oozie/action/hadoop/Hive2ActionExecutor.java 
2aed936af75da38c433be5d202bdc01c2f612c76 
  core/src/main/java/org/apache/oozie/action/hadoop/Hive2Credentials.java 
bd2f1f768c5b04bef4a33613baede9202144c747 
  core/src/main/java/org/apache/oozie/action/hadoop/HiveActionExecutor.java 
8e2453ef4f78f4a5f1f95b1ca2793f5d7e07ad81 
  core/src/main/java/org/apache/oozie/action/hadoop/JavaActionExecutor.java 
39daca7b3050c259a36d54710b4c1279f3acdeac 
  core/src/main/java/org/apache/oozie/action/hadoop/LauncherMapperHelper.java 
9609fdc2f3440fb176b659ef16c2a23451724cf9 
  
core/src/main/java/org/apache/oozie/action/hadoop/MapReduceActionExecutor.java 
55c937249ac35e14bd63dd0c1fa962ea636afa3a 
  core/src/main/java/org/apache/oozie/action/hadoop/OozieJobInfo.java 
581d3b3219eff882f961c8e495611e873c6a5a52 
  core/src/main/java/org/apache/oozie/action/hadoop/PigActionExecutor.java 
8b2dc16db80ad602911f1f64e60cfad41b198922 
  
core/src/main/java/org/apache/oozie/action/hadoop/ScriptLanguageActionExecutor.java
 92e149d5cac6509893c8507b776263a38312a53b 
  core/src/main/java/org/apache/oozie/action/hadoop/ShellActionExecutor.java 
b9ffa7ab7ec87db8a76a3653c74974d296c8cd17 
  core/src/main/java/org/apache/oozie/action/hadoop/SparkActionExecutor.java 
1a3197abb52f80b545dcc8634a8cac7a281a9eac 
  core/src/main/java/org/apache/oozie/action/hadoop/SqoopActionExecutor.java 
22e28746a9a39ae6c1e22326cc4f8f759d820b20 
  core/src/main/java/org/apache/oozie/action/ssh/SshActionExecutor.java 
5fd30f7858c5c4c70dcbdb6df86904518d92f7f3 
  core/src/main/java/org/apache/oozie/client/rest/BulkResponseImpl.java 
fb021bd022d199f93fc5674843514abf45ae12a5 
  core/src/main/java/org/apache/oozie/command/PurgeXCommand.java 
2ba1999ba7d5e892bd7e37d5fd8d1f926a84b48b 
  core/src/main/java/org/apache/oozie/command/XCommand.java 
bdf13f65ea25e8e0f5125e2798568a2c03fffe81 
  core/src/main/java/org/apache/oozie/command/wf/SubmitHttpXCommand.java 
d2a2742ae418070a1cd0a889f14194224bc944a6 
  core/src/main/java/org/apache/oozie/command/wf/SubmitMRXCommand.java 
cc61d3de813c75c7f1cf4d405fc043253e3ca4ea 
  
core/src/main/java/org/apache/oozie/command/wf/SubmitScriptLanguageXCommand.java
 9d413059e99bc676feb63e02a2f704048eb4a355 
  core/src/main/java/org/apache/oozie/command/wf/SubmitSqoopXCommand.java 
51b739e21c802e09f4110b740138f4

Re: Review Request 54705: OOZIE-1770: Create Oozie Application Master for YARN

[Rohini Palaniswamy]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/54705/#review170061
---



Reviews for Pages 1-3 except JavaActionExecutor

Note: On a time crunch and doing review of couple of pages at a time to avoid 
postponing this for too long. Have also not had time to go through history of 
previous review comments from others. So there might be duplicates.


core/src/main/java/org/apache/oozie/action/hadoop/CredentialsProviderFactory.java
Lines 34 (patched)


If you are calling this a factory, you should look into removing the 
constructor by type and making it a true Factory. Cache the credential type and 
the corresponding Class. createCredentialsProvider() will then become 
createCredentialsProvider(String type) which will just look up the cache and 
return a new instance of the Class. 

Parsing ConfigurationService.getStrings(CRED_KEY) and loading the class 
everytime is a lot of waste.



core/src/main/java/org/apache/oozie/action/hadoop/LauncherMapperHelper.java
Lines 54 (patched)


You could just call it LauncherHelper.



core/src/main/java/org/apache/oozie/action/hadoop/MapReduceActionExecutor.java
Lines 319 (patched)


Typo. original



core/src/main/java/org/apache/oozie/action/hadoop/MapReduceActionExecutor.java
Lines 370 (patched)


Remove this warn and add e to the next warning logging.



core/src/main/java/org/apache/oozie/action/hadoop/MapReduceActionExecutor.java
Lines 406-407 (patched)


1) Can we kill directly if externalChild id is not empty (MR job already 
launched) else call  LauncherMainHadoopUtils.killChildYarnJobs? If it is empty 
we have to call killChildYarnJobs as the launcher AM could have just launched 
the mapreduce job before kill reached it.
2) Why are we duplicating the killChildYarnJobs code here?



core/src/main/java/org/apache/oozie/action/hadoop/SparkActionExecutor.java
Lines 112-114 (original)


You can't remove this. Will breaking existing users. Since it starts with 
oozie.launcher, config deprecation will not handle it. Basically any 
oozie.launcher hadoop 1.x settings cannot be removed because of that.



core/src/main/java/org/apache/oozie/action/hadoop/SqoopActionExecutor.java
Line 89 (original), 87 (patched)


Revert back to old code. Just remove the final instead of creating another 
temporary list.



core/src/main/java/org/apache/oozie/action/hadoop/SqoopActionExecutor.java
Lines 109-111 (original)


Why is this condition not necessary any more?



core/src/main/java/org/apache/oozie/action/ssh/SshActionExecutor.java
Line 69 (original), 69 (patched)


Revert. UNKNOWN_ERROR is correct



core/src/main/java/org/apache/oozie/command/PurgeXCommand.java
Line 410 (original), 410 (patched)


Why remove entity key?



core/src/main/java/org/apache/oozie/command/PurgeXCommand.java
Line 418 (original), 418 (patched)


Why is lock not required anymore?



core/src/main/java/org/apache/oozie/dependency/HCatURIHandler.java
Line 182 (original)


Why is support for dropping table being removed?



core/src/main/java/org/apache/oozie/dependency/HCatURIHandler.java
Line 345 (original)


Why is support for checking exists on a table being removed?



core/src/main/java/org/apache/oozie/service/HadoopAccessorService.java
Lines 776-777 (patched)


This is going to fill up the temp disk space pretty quickly. Please avoid 
creating the local file and write directly to the FSDataOutputStream after 
creating the file in HDFS.



core/src/main/java/org/apache/oozie/service/Services.java
Lines 308-309 (patched)


log.fatal("Runtime Exception during Services Load. Check your list of '{0}' 
or '{1}'", CONF_SERVICE_CLASSES, CONF_SERVICE_EXT_CLASSES, rex);



core/src/main/java/org/apache/oozie/servlet/BaseAdminServlet.java
Lines 86-87 (original), 71 (patched)


You should retain the code of checking the resource name and throwing error 
here which is currently commented out instead of doing it in setOozieMode()



core/src/main/java/org/apache/oozie/servlet/JVMIn