[jira] Subscription: Oozie Patch Available

2017-09-06 Thread jira 
Issue Subscription
Filter: Oozie Patch Available (106 issues)

Subscriber: ooziedaily

Key Summary
OOZIE-3041  TestWorkflowActionRetryInfoXCommand fails in oozie core module.
https://issues.apache.org/jira/browse/OOZIE-3041
OOZIE-3031  Coord job with only unresolved dependencies doesn't timeout
https://issues.apache.org/jira/browse/OOZIE-3031
OOZIE-3022  fix for warning has no file and won't be listed in dependency files 
details
https://issues.apache.org/jira/browse/OOZIE-3022
OOZIE-3002  address findbugs errors in client lib
https://issues.apache.org/jira/browse/OOZIE-3002
OOZIE-3001  core library has many instances of warnings with trailing spaces 
and lines longer than 132 chars
https://issues.apache.org/jira/browse/OOZIE-3001
OOZIE-2997  files contain trailing white spaces in client lib
https://issues.apache.org/jira/browse/OOZIE-2997
OOZIE-2996  add option for -UseGCOverheadLimit to maven opts as sometimes local 
testing fails
https://issues.apache.org/jira/browse/OOZIE-2996
OOZIE-2978  Remove code that handles Pig versions before 0.8 
https://issues.apache.org/jira/browse/OOZIE-2978
OOZIE-2975  code clean up in pig sharelib, replace Exception with more 
explicit, add try with resources, StringBuilder instead of StringBuffer
https://issues.apache.org/jira/browse/OOZIE-2975
OOZIE-2969  Drop support for Java 1.7
https://issues.apache.org/jira/browse/OOZIE-2969
OOZIE-2964  Add -Xdoclint:none to javadoc opts to avoid warnings
https://issues.apache.org/jira/browse/OOZIE-2964
OOZIE-2963  getting error in build ArtifactNotFoundException: Could not find 
artifact org.pentaho:pentaho-aggdesigner-algorithm:jar:5.1.5-jhyde
https://issues.apache.org/jira/browse/OOZIE-2963
OOZIE-2962  bump maven-javadoc-plugin to 2.10.4
https://issues.apache.org/jira/browse/OOZIE-2962
OOZIE-2957  Documentation states that starting a coordinator is possible
https://issues.apache.org/jira/browse/OOZIE-2957
OOZIE-2956  Fix Findbugs warnings related to reliance on default encoding in 
oozie-core
https://issues.apache.org/jira/browse/OOZIE-2956
OOZIE-2955  Fix Findbugs warnings related to reliance on default encoding in 
oozie-client
https://issues.apache.org/jira/browse/OOZIE-2955
OOZIE-2954  Fix Checkstyle issues in oozie-client
https://issues.apache.org/jira/browse/OOZIE-2954
OOZIE-2953  Fix Checkstyle issues in oozie-tools
https://issues.apache.org/jira/browse/OOZIE-2953
OOZIE-2952  Fix Findbugs warnings in oozie-sharelib-oozie
https://issues.apache.org/jira/browse/OOZIE-2952
OOZIE-2949  Escape quotes whitespaces in Sqoop  field
https://issues.apache.org/jira/browse/OOZIE-2949
OOZIE-2942  Fix Findbugs warnings in oozie-examples
https://issues.apache.org/jira/browse/OOZIE-2942
OOZIE-2937  Remove redundant groupId from the child pom's
https://issues.apache.org/jira/browse/OOZIE-2937
OOZIE-2934  Fix "Exceptional return value of java.io.File.mkdirs() ignored" 
Findbugs error in oozie-sharelib-spark
https://issues.apache.org/jira/browse/OOZIE-2934
OOZIE-2927  Append new line character for Hive2 query using query tag
https://issues.apache.org/jira/browse/OOZIE-2927
OOZIE-2916  Set a job name for the MR Action's child job
https://issues.apache.org/jira/browse/OOZIE-2916
OOZIE-2914  Consolidate Trim 
https://issues.apache.org/jira/browse/OOZIE-2914
OOZIE-2909  LauncherAM: rewrite UGI calls
https://issues.apache.org/jira/browse/OOZIE-2909
OOZIE-2883  OOZIE throw the error "Missing 
[oozie.service.ProxyUserService.proxyuser.oozie.service.ProxyUserService.proxyuser.mr.groups]
 property"
https://issues.apache.org/jira/browse/OOZIE-2883
OOZIE-2877  Oozie Git Action
https://issues.apache.org/jira/browse/OOZIE-2877
OOZIE-2867  Timezone handling for Coordinators: emphasize "Continent/City" 
format
https://issues.apache.org/jira/browse/OOZIE-2867
OOZIE-2834  ParameterVerifier logging non-useful warning for workflow definition
https://issues.apache.org/jira/browse/OOZIE-2834
OOZIE-2833  when using uber mode the regex pattern used in the 
extractHeapSizeMB method does not allow heap sizes specified in bytes.
https://issues.apache.org/jira/browse/OOZIE-2833
OOZIE-2829  Improve sharelib upload to accept multiple source folders
https://issues.apache.org/jira/browse/OOZIE-2829
OOZIE-2826  Falcon feed fails to aws s3; Oozie joda time version does not meet 
required jar version 2.2 or later
https://issues.apache.org/jira/browse/OOZIE-2826
OOZIE-2812  SparkConfigurationService should support loading configurations 
from multiple Spark versions
https://issues.apache.org/jira/browse/OOZIE-2812
OOZIE-2795  Create lib directory or symlink f

[jira] [Created] (OOZIE-3049) Provision to pass Key alias with Oozie SSL

2017-09-06 Thread HARIKRISHNAN Ck (JIRA) 
HARIKRISHNAN Ck created OOZIE-3049:
--

 Summary: Provision to pass Key alias with Oozie SSL
 Key: OOZIE-3049
 URL: https://issues.apache.org/jira/browse/OOZIE-3049
 Project: Oozie
  Issue Type: Improvement
Reporter: HARIKRISHNAN Ck
Priority: Minor


Currently, there are provisions to set the keyStore and keyPassword. Per Oozie 
documentation:

OOZIE_HTTPS_KEYSTORE_FILE : The location of the keystore file containing the 
certificate information. Default value ${HOME}/.keystore (i.e. the home dir of 
the Oozie user).

OOZIE_HTTPS_KEYSTORE_PASS : The password of the keystore file. Default value 
password 

However, there is no provision to set the keySotreAlias. In a Production 
Cluster there are good chances of having multiple keys in a kesytore and it's 
likely possible that multiple keys are present.
Per tomcat documentation : 
keyAlias: The alias used for the server key and certificate in the keystore. If 
not specified, the first key read from the keystore will be used. The order in 
which keys are read from the keystore is implementation dependent. It may not 
be the case that keys are read from the keystore in the same order as they were 
added. If more than one key is present in the keystore it is strongly 
recommended that a keyAlias is configured to ensure that the correct key is 
used.


 The work around would be to edit the server.xml and add the keyAlias field. 
Editing "server.xml" is not a recommended solution as this has to be done every 
time oozie-setup.sh script is executed also these steps are not documented any 
where. We should have a variable which can be set in the oozie-env.sh





--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Updated] (OOZIE-3048) Check El Functions for the coordinator action

2017-09-06 Thread Satish Subhashrao Saley (JIRA) 

 [ 
https://issues.apache.org/jira/browse/OOZIE-3048?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Satish Subhashrao Saley updated OOZIE-3048:
---
Description: In OOZIE-2873, we tried introduce check for EL functions 
before submitting the coordinator. But this not possible as described 
[here|https://issues.apache.org/jira/browse/OOZIE-2873?focusedCommentId=15999124&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-15999124].
 Instead, we can do this check in CoordActionInputCheckXCommand when we resolve 
coordinator configuration

> Check El Functions for the coordinator action
> -
>
> Key: OOZIE-3048
> URL: https://issues.apache.org/jira/browse/OOZIE-3048
> Project: Oozie
>  Issue Type: Bug
>Reporter: Satish Subhashrao Saley
>
> In OOZIE-2873, we tried introduce check for EL functions before submitting 
> the coordinator. But this not possible as described 
> [here|https://issues.apache.org/jira/browse/OOZIE-2873?focusedCommentId=15999124&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-15999124].
>  Instead, we can do this check in CoordActionInputCheckXCommand when we 
> resolve coordinator configuration



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Assigned] (OOZIE-3048) Check El Functions for the coordinator action

2017-09-06 Thread Satish Subhashrao Saley (JIRA) 

 [ 
https://issues.apache.org/jira/browse/OOZIE-3048?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Satish Subhashrao Saley reassigned OOZIE-3048:
--

Assignee: Satish Subhashrao Saley

> Check El Functions for the coordinator action
> -
>
> Key: OOZIE-3048
> URL: https://issues.apache.org/jira/browse/OOZIE-3048
> Project: Oozie
>  Issue Type: Bug
>Reporter: Satish Subhashrao Saley
>Assignee: Satish Subhashrao Saley
>
> In OOZIE-2873, we tried introduce check for EL functions before submitting 
> the coordinator. But this not possible as described 
> [here|https://issues.apache.org/jira/browse/OOZIE-2873?focusedCommentId=15999124&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-15999124].
>  Instead, we can do this check in CoordActionInputCheckXCommand when we 
> resolve coordinator configuration.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Updated] (OOZIE-3048) Check El Functions for the coordinator action

2017-09-06 Thread Satish Subhashrao Saley (JIRA) 

 [ 
https://issues.apache.org/jira/browse/OOZIE-3048?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Satish Subhashrao Saley updated OOZIE-3048:
---
Description: In OOZIE-2873, we tried introduce check for EL functions 
before submitting the coordinator. But this not possible as described 
[here|https://issues.apache.org/jira/browse/OOZIE-2873?focusedCommentId=15999124&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-15999124].
 Instead, we can do this check in CoordActionInputCheckXCommand when we resolve 
coordinator configuration.  (was: In OOZIE-2873, we tried introduce check for 
EL functions before submitting the coordinator. But this not possible as 
described 
[here|https://issues.apache.org/jira/browse/OOZIE-2873?focusedCommentId=15999124&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-15999124].
 Instead, we can do this check in CoordActionInputCheckXCommand when we resolve 
coordinator configuration)

> Check El Functions for the coordinator action
> -
>
> Key: OOZIE-3048
> URL: https://issues.apache.org/jira/browse/OOZIE-3048
> Project: Oozie
>  Issue Type: Bug
>Reporter: Satish Subhashrao Saley
>
> In OOZIE-2873, we tried introduce check for EL functions before submitting 
> the coordinator. But this not possible as described 
> [here|https://issues.apache.org/jira/browse/OOZIE-2873?focusedCommentId=15999124&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-15999124].
>  Instead, we can do this check in CoordActionInputCheckXCommand when we 
> resolve coordinator configuration.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Created] (OOZIE-3048) Check El Functions for the coordinator action

2017-09-06 Thread Satish Subhashrao Saley (JIRA) 
Satish Subhashrao Saley created OOZIE-3048:
--

 Summary: Check El Functions for the coordinator action
 Key: OOZIE-3048
 URL: https://issues.apache.org/jira/browse/OOZIE-3048
 Project: Oozie
  Issue Type: Bug
Reporter: Satish Subhashrao Saley






--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Updated] (OOZIE-2909) LauncherAM: rewrite UGI calls

2017-09-06 Thread Peter Cseh (JIRA) 

 [ 
https://issues.apache.org/jira/browse/OOZIE-2909?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Peter Cseh updated OOZIE-2909:
--
Attachment: OOZIE-2909-003.patch

Adding new patch.

> LauncherAM: rewrite UGI calls
> -
>
> Key: OOZIE-2909
> URL: https://issues.apache.org/jira/browse/OOZIE-2909
> Project: Oozie
>  Issue Type: Sub-task
>Reporter: Peter Bacsko
>Assignee: Peter Cseh
>Priority: Blocker
> Attachments: OOZIE-2909-001.patch, OOZIE-2909-002.patch, 
> OOZIE-2909-003.patch
>
>
> There are two problems in LauncherAM:
> 1) In a kerberized cluster, we have to use {{UGI.getLoginUser()}} even though 
> a call to {{createRemoteUser()}} should be enough, just like in 
> {{MRAppMaster.java}}. Reference code:
> https://github.com/apache/hadoop/blob/3ed3062fe3979ff55a411b730a8eee2b2c96d6b3/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/MRAppMaster.java#L1718-L1726
> 2) It's enough if we wrap the whole LauncherAM logic in a single {{doAs()}} 
> call. No need for multiple {{doAs()}}, just makes the whole thing more 
> complicated.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

Re: Review Request 61560: OOZIE-2909 LauncherAM: rewrite UGI calls

2017-09-06 Thread András Piros 

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/61560/#review184696
---




core/src/main/java/org/apache/oozie/service/HadoopAccessorService.java
Lines 558-570 (patched)


I'd extract those three methods, and this one, to a new service called 
`HadoopTokenService`. It's best readable and tested then.



core/src/main/java/org/apache/oozie/service/HadoopAccessorService.java
Lines 584-594 (patched)


Getting the `Text`, and calling `credentials.addToken()` can be refactored 
to a common class, and to get the token for the `Text` can be done within three 
different `Callable` calls for the three methods.



core/src/main/java/org/apache/oozie/service/HadoopAccessorService.java
Lines 607-617 (patched)


Getting the `Text`, and calling `credentials.addToken()` can be refactored 
to a common class, and to get the token for the `Text` can be done within three 
different `Callable` calls for the three methods.



core/src/main/java/org/apache/oozie/service/HadoopAccessorService.java
Lines 681-688 (patched)


Getting the `Text`, and calling `credentials.addToken()` can be refactored 
to a common class, and to get the token for the `Text` can be done within three 
different `Callable` calls for the three methods.



sharelib/oozie/src/main/java/org/apache/oozie/action/hadoop/LauncherAM.java
Lines 148-149 (original), 155-169 (patched)


Should be extracted to another method.


- András Piros


On Sept. 6, 2017, 2:25 p.m., Peter Cseh wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/61560/
> ---
> 
> (Updated Sept. 6, 2017, 2:25 p.m.)
> 
> 
> Review request for oozie, Attila Sasvari, Peter Bacsko, Robert Kanter, and 
> Rohini Palaniswamy.
> 
> 
> Bugs: OOZIE-2909
> https://issues.apache.org/jira/browse/OOZIE-2909
> 
> 
> Repository: oozie-git
> 
> 
> Description
> ---
> 
> Changing UGI calls and cleaning up things around requesting tokens.
> 
> 
> Diffs
> -
> 
>   core/src/main/java/org/apache/oozie/ErrorCode.java 
> 662e1edc9c4b23b3606c751bf5ed4b531ee7ac62 
>   
> core/src/main/java/org/apache/oozie/action/hadoop/CredentialsProviderFactory.java
>  5ca8d3e80942840a5bba51a06694350929044042 
>   core/src/main/java/org/apache/oozie/action/hadoop/HCatCredentialHelper.java 
> 9804c7b673a899cd9c778850c9a64731cc642fb3 
>   core/src/main/java/org/apache/oozie/action/hadoop/HbaseCredentials.java 
> 4add5f14cb20ab77c755b81eff939335c0c5fddc 
>   core/src/main/java/org/apache/oozie/action/hadoop/Hive2Credentials.java 
> 0b495f75842b0033a3337ef6728a334bf5651770 
>   core/src/main/java/org/apache/oozie/action/hadoop/JavaActionExecutor.java 
> bca79aa052521ea4f6f16e76bd69f84fb16be790 
>   core/src/main/java/org/apache/oozie/service/HadoopAccessorService.java 
> 187cee2290ba4bbe358913d28a052d9c75f2369a 
>   core/src/test/java/org/apache/oozie/service/TestHadoopAccessorService.java 
> 960c2f9750062c98fdf7a2b456d3668069e07ca5 
>   
> sharelib/oozie/src/main/java/org/apache/oozie/action/hadoop/AMRMClientAsyncFactory.java
>  b4cbb4b1d9b52aaebfb562c02edc91e15a2a9a2e 
>   
> sharelib/oozie/src/main/java/org/apache/oozie/action/hadoop/HdfsOperations.java
>  874d371664ced3ae19a7b5d615c67d4c72e7cfdd 
>   sharelib/oozie/src/main/java/org/apache/oozie/action/hadoop/LauncherAM.java 
> 6a98d6ef3a09a7a75272f3f0f9a9c2fc5472e76d 
>   sharelib/oozie/src/main/java/org/apache/oozie/action/hadoop/ShellMain.java 
> bde7f1daf123d663d47f5d2fb2350cfa1ac33ed8 
>   
> sharelib/oozie/src/test/java/org/apache/oozie/action/hadoop/TestHdfsOperations.java
>  68c0f4bafe0322ff0d1007297cd6e8f38c92020a 
>   
> sharelib/oozie/src/test/java/org/apache/oozie/action/hadoop/TestLauncherAM.java
>  37af3dd9ed43f4504ff3909484237e2d05665c6c 
> 
> 
> Diff: https://reviews.apache.org/r/61560/diff/3/
> 
> 
> Testing
> ---
> 
> We're running tests on a Kerberized cluster with these changes.
> 
> 
> Thanks,
> 
> Peter Cseh
> 
>

Re: Review Request 61560: OOZIE-2909 LauncherAM: rewrite UGI calls

2017-09-06 Thread Peter Cseh 

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/61560/
---

(Updated Sept. 6, 2017, 2:25 p.m.)


Review request for oozie, Attila Sasvari, Peter Bacsko, Robert Kanter, and 
Rohini Palaniswamy.


Bugs: OOZIE-2909
https://issues.apache.org/jira/browse/OOZIE-2909


Repository: oozie-git


Description
---

Changing UGI calls and cleaning up things around requesting tokens.


Diffs (updated)
-

  core/src/main/java/org/apache/oozie/ErrorCode.java 
662e1edc9c4b23b3606c751bf5ed4b531ee7ac62 
  
core/src/main/java/org/apache/oozie/action/hadoop/CredentialsProviderFactory.java
 5ca8d3e80942840a5bba51a06694350929044042 
  core/src/main/java/org/apache/oozie/action/hadoop/HCatCredentialHelper.java 
9804c7b673a899cd9c778850c9a64731cc642fb3 
  core/src/main/java/org/apache/oozie/action/hadoop/HbaseCredentials.java 
4add5f14cb20ab77c755b81eff939335c0c5fddc 
  core/src/main/java/org/apache/oozie/action/hadoop/Hive2Credentials.java 
0b495f75842b0033a3337ef6728a334bf5651770 
  core/src/main/java/org/apache/oozie/action/hadoop/JavaActionExecutor.java 
bca79aa052521ea4f6f16e76bd69f84fb16be790 
  core/src/main/java/org/apache/oozie/service/HadoopAccessorService.java 
187cee2290ba4bbe358913d28a052d9c75f2369a 
  core/src/test/java/org/apache/oozie/service/TestHadoopAccessorService.java 
960c2f9750062c98fdf7a2b456d3668069e07ca5 
  
sharelib/oozie/src/main/java/org/apache/oozie/action/hadoop/AMRMClientAsyncFactory.java
 b4cbb4b1d9b52aaebfb562c02edc91e15a2a9a2e 
  
sharelib/oozie/src/main/java/org/apache/oozie/action/hadoop/HdfsOperations.java 
874d371664ced3ae19a7b5d615c67d4c72e7cfdd 
  sharelib/oozie/src/main/java/org/apache/oozie/action/hadoop/LauncherAM.java 
6a98d6ef3a09a7a75272f3f0f9a9c2fc5472e76d 
  sharelib/oozie/src/main/java/org/apache/oozie/action/hadoop/ShellMain.java 
bde7f1daf123d663d47f5d2fb2350cfa1ac33ed8 
  
sharelib/oozie/src/test/java/org/apache/oozie/action/hadoop/TestHdfsOperations.java
 68c0f4bafe0322ff0d1007297cd6e8f38c92020a 
  
sharelib/oozie/src/test/java/org/apache/oozie/action/hadoop/TestLauncherAM.java 
37af3dd9ed43f4504ff3909484237e2d05665c6c 


Diff: https://reviews.apache.org/r/61560/diff/3/

Changes: https://reviews.apache.org/r/61560/diff/2-3/


Testing
---

We're running tests on a Kerberized cluster with these changes.


Thanks,

Peter Cseh

Re: Review Request 61560: OOZIE-2909 LauncherAM: rewrite UGI calls

2017-09-06 Thread Peter Cseh 


> On Sept. 5, 2017, 5:05 p.m., Robert Kanter wrote:
> > core/src/main/java/org/apache/oozie/service/HadoopAccessorService.java
> > Lines 555 (patched)
> > 
> >
> > Do we always want to get an MR token?  Certainly not for Spark or 
> > Shell, but Java, Hive (Hive-on-Spark), and Pig (Pig-on-Spark), are more 
> > complicated.  I guess it's simpler to just always add the MR token.
> 
> Peter Cseh wrote:
> I think we should always try to get one. It might be better just to log a 
> warning if we failed instead of failing the action.

I'd rather implement the warning part in a follow-up jira and keep it as is now.


- Peter


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/61560/#review184556
---


On Sept. 5, 2017, 1:56 p.m., Peter Cseh wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/61560/
> ---
> 
> (Updated Sept. 5, 2017, 1:56 p.m.)
> 
> 
> Review request for oozie, Attila Sasvari, Peter Bacsko, Robert Kanter, and 
> Rohini Palaniswamy.
> 
> 
> Bugs: OOZIE-2909
> https://issues.apache.org/jira/browse/OOZIE-2909
> 
> 
> Repository: oozie-git
> 
> 
> Description
> ---
> 
> Changing UGI calls and cleaning up things around requesting tokens.
> 
> 
> Diffs
> -
> 
>   core/src/main/java/org/apache/oozie/ErrorCode.java 
> 662e1edc9c4b23b3606c751bf5ed4b531ee7ac62 
>   
> core/src/main/java/org/apache/oozie/action/hadoop/CredentialsProviderFactory.java
>  5ca8d3e80942840a5bba51a06694350929044042 
>   core/src/main/java/org/apache/oozie/action/hadoop/HCatCredentialHelper.java 
> 9804c7b673a899cd9c778850c9a64731cc642fb3 
>   core/src/main/java/org/apache/oozie/action/hadoop/HbaseCredentials.java 
> 4add5f14cb20ab77c755b81eff939335c0c5fddc 
>   core/src/main/java/org/apache/oozie/action/hadoop/Hive2Credentials.java 
> 0b495f75842b0033a3337ef6728a334bf5651770 
>   core/src/main/java/org/apache/oozie/action/hadoop/JavaActionExecutor.java 
> bca79aa052521ea4f6f16e76bd69f84fb16be790 
>   core/src/main/java/org/apache/oozie/service/HadoopAccessorService.java 
> 187cee2290ba4bbe358913d28a052d9c75f2369a 
>   core/src/test/java/org/apache/oozie/service/TestHadoopAccessorService.java 
> 960c2f9750062c98fdf7a2b456d3668069e07ca5 
>   
> sharelib/oozie/src/main/java/org/apache/oozie/action/hadoop/AMRMCallBackHandler.java
>  e6c9d04653be984bdd5aa77bee5f28b187e3de5d 
>   
> sharelib/oozie/src/main/java/org/apache/oozie/action/hadoop/AMRMClientAsyncFactory.java
>  b4cbb4b1d9b52aaebfb562c02edc91e15a2a9a2e 
>   
> sharelib/oozie/src/main/java/org/apache/oozie/action/hadoop/HdfsOperations.java
>  874d371664ced3ae19a7b5d615c67d4c72e7cfdd 
>   sharelib/oozie/src/main/java/org/apache/oozie/action/hadoop/LauncherAM.java 
> 6a98d6ef3a09a7a75272f3f0f9a9c2fc5472e76d 
>   sharelib/oozie/src/main/java/org/apache/oozie/action/hadoop/ShellMain.java 
> bde7f1daf123d663d47f5d2fb2350cfa1ac33ed8 
>   
> sharelib/oozie/src/test/java/org/apache/oozie/action/hadoop/TestHdfsOperations.java
>  68c0f4bafe0322ff0d1007297cd6e8f38c92020a 
>   
> sharelib/oozie/src/test/java/org/apache/oozie/action/hadoop/TestLauncherAM.java
>  37af3dd9ed43f4504ff3909484237e2d05665c6c 
> 
> 
> Diff: https://reviews.apache.org/r/61560/diff/2/
> 
> 
> Testing
> ---
> 
> We're running tests on a Kerberized cluster with these changes.
> 
> 
> Thanks,
> 
> Peter Cseh
> 
>

Re: Review Request 61560: OOZIE-2909 LauncherAM: rewrite UGI calls

2017-09-06 Thread Peter Cseh 


> On Aug. 22, 2017, 8:34 p.m., Robert Kanter wrote:
> > core/src/main/java/org/apache/oozie/action/hadoop/HCatCredentialHelper.java
> > Line 66 (original), 65 (patched)
> > 
> >
> > Have we verified that HCat/HMS is okay with changing the alias?  I 
> > don't know if maybe they were doing something funny with it.
> 
> Peter Cseh wrote:
> I know there is a reference for it in Sqoop 
> (https://issues.apache.org/jira/browse/SQOOP-3177) , but It does not seem to 
> work anyways.
> 
> Peter Cseh wrote:
> I'm reverting the token alias changes

No, I'm keeping them. See Hive2Credentials for reasons


> On Aug. 22, 2017, 8:34 p.m., Robert Kanter wrote:
> > core/src/main/java/org/apache/oozie/action/hadoop/Hive2Credentials.java
> > Line 82 (original), 81 (patched)
> > 
> >
> > Have we verified that HS2 is okay with changing the alias?  I don't 
> > know if maybe they were doing something funny with it.
> 
> Peter Cseh wrote:
> I could not find a reference for this alias in Hive's codebase.

We have used the alias "  Text fauxAlias = new Text(tk.getKind() + "_" + 
tk.getService());" before OYA. 
https://github.com/apache/oozie/blob/53b1d1e4392cfd35d10860e3b1437273d909a316/core/src/main/java/org/apache/oozie/action/hadoop/JavaActionExecutor.java#L1178-L1196


- Peter


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/61560/#review183516
---


On Sept. 5, 2017, 1:56 p.m., Peter Cseh wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/61560/
> ---
> 
> (Updated Sept. 5, 2017, 1:56 p.m.)
> 
> 
> Review request for oozie, Attila Sasvari, Peter Bacsko, Robert Kanter, and 
> Rohini Palaniswamy.
> 
> 
> Bugs: OOZIE-2909
> https://issues.apache.org/jira/browse/OOZIE-2909
> 
> 
> Repository: oozie-git
> 
> 
> Description
> ---
> 
> Changing UGI calls and cleaning up things around requesting tokens.
> 
> 
> Diffs
> -
> 
>   core/src/main/java/org/apache/oozie/ErrorCode.java 
> 662e1edc9c4b23b3606c751bf5ed4b531ee7ac62 
>   
> core/src/main/java/org/apache/oozie/action/hadoop/CredentialsProviderFactory.java
>  5ca8d3e80942840a5bba51a06694350929044042 
>   core/src/main/java/org/apache/oozie/action/hadoop/HCatCredentialHelper.java 
> 9804c7b673a899cd9c778850c9a64731cc642fb3 
>   core/src/main/java/org/apache/oozie/action/hadoop/HbaseCredentials.java 
> 4add5f14cb20ab77c755b81eff939335c0c5fddc 
>   core/src/main/java/org/apache/oozie/action/hadoop/Hive2Credentials.java 
> 0b495f75842b0033a3337ef6728a334bf5651770 
>   core/src/main/java/org/apache/oozie/action/hadoop/JavaActionExecutor.java 
> bca79aa052521ea4f6f16e76bd69f84fb16be790 
>   core/src/main/java/org/apache/oozie/service/HadoopAccessorService.java 
> 187cee2290ba4bbe358913d28a052d9c75f2369a 
>   core/src/test/java/org/apache/oozie/service/TestHadoopAccessorService.java 
> 960c2f9750062c98fdf7a2b456d3668069e07ca5 
>   
> sharelib/oozie/src/main/java/org/apache/oozie/action/hadoop/AMRMCallBackHandler.java
>  e6c9d04653be984bdd5aa77bee5f28b187e3de5d 
>   
> sharelib/oozie/src/main/java/org/apache/oozie/action/hadoop/AMRMClientAsyncFactory.java
>  b4cbb4b1d9b52aaebfb562c02edc91e15a2a9a2e 
>   
> sharelib/oozie/src/main/java/org/apache/oozie/action/hadoop/HdfsOperations.java
>  874d371664ced3ae19a7b5d615c67d4c72e7cfdd 
>   sharelib/oozie/src/main/java/org/apache/oozie/action/hadoop/LauncherAM.java 
> 6a98d6ef3a09a7a75272f3f0f9a9c2fc5472e76d 
>   sharelib/oozie/src/main/java/org/apache/oozie/action/hadoop/ShellMain.java 
> bde7f1daf123d663d47f5d2fb2350cfa1ac33ed8 
>   
> sharelib/oozie/src/test/java/org/apache/oozie/action/hadoop/TestHdfsOperations.java
>  68c0f4bafe0322ff0d1007297cd6e8f38c92020a 
>   
> sharelib/oozie/src/test/java/org/apache/oozie/action/hadoop/TestLauncherAM.java
>  37af3dd9ed43f4504ff3909484237e2d05665c6c 
> 
> 
> Diff: https://reviews.apache.org/r/61560/diff/2/
> 
> 
> Testing
> ---
> 
> We're running tests on a Kerberized cluster with these changes.
> 
> 
> Thanks,
> 
> Peter Cseh
> 
>

Re: Review Request 61560: OOZIE-2909 LauncherAM: rewrite UGI calls

2017-09-06 Thread Peter Cseh 


> On Sept. 5, 2017, 5:05 p.m., Robert Kanter wrote:
> > sharelib/oozie/src/main/java/org/apache/oozie/action/hadoop/LauncherAM.java
> > Lines 196 (patched)
> > 
> >
> > Aren't we already in a doAs?  It looks like this is so we can remove 
> > the AMRM token.  I wonder if instead of doing a nested doAs here, we could 
> > instead simply remove the AMRM token from the current UGI?  If not, then 
> > it's fine to leave this as-is.

Without the AM_RM token we can't register and unregister the application in the 
RM. There may be a cleaner way to do this, but we'll need two doAs calls. With 
the AM_RM token customer code could request more containers and do such things 
we don't really want them to do.


- Peter


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/61560/#review184556
---


On Sept. 5, 2017, 1:56 p.m., Peter Cseh wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/61560/
> ---
> 
> (Updated Sept. 5, 2017, 1:56 p.m.)
> 
> 
> Review request for oozie, Attila Sasvari, Peter Bacsko, Robert Kanter, and 
> Rohini Palaniswamy.
> 
> 
> Bugs: OOZIE-2909
> https://issues.apache.org/jira/browse/OOZIE-2909
> 
> 
> Repository: oozie-git
> 
> 
> Description
> ---
> 
> Changing UGI calls and cleaning up things around requesting tokens.
> 
> 
> Diffs
> -
> 
>   core/src/main/java/org/apache/oozie/ErrorCode.java 
> 662e1edc9c4b23b3606c751bf5ed4b531ee7ac62 
>   
> core/src/main/java/org/apache/oozie/action/hadoop/CredentialsProviderFactory.java
>  5ca8d3e80942840a5bba51a06694350929044042 
>   core/src/main/java/org/apache/oozie/action/hadoop/HCatCredentialHelper.java 
> 9804c7b673a899cd9c778850c9a64731cc642fb3 
>   core/src/main/java/org/apache/oozie/action/hadoop/HbaseCredentials.java 
> 4add5f14cb20ab77c755b81eff939335c0c5fddc 
>   core/src/main/java/org/apache/oozie/action/hadoop/Hive2Credentials.java 
> 0b495f75842b0033a3337ef6728a334bf5651770 
>   core/src/main/java/org/apache/oozie/action/hadoop/JavaActionExecutor.java 
> bca79aa052521ea4f6f16e76bd69f84fb16be790 
>   core/src/main/java/org/apache/oozie/service/HadoopAccessorService.java 
> 187cee2290ba4bbe358913d28a052d9c75f2369a 
>   core/src/test/java/org/apache/oozie/service/TestHadoopAccessorService.java 
> 960c2f9750062c98fdf7a2b456d3668069e07ca5 
>   
> sharelib/oozie/src/main/java/org/apache/oozie/action/hadoop/AMRMCallBackHandler.java
>  e6c9d04653be984bdd5aa77bee5f28b187e3de5d 
>   
> sharelib/oozie/src/main/java/org/apache/oozie/action/hadoop/AMRMClientAsyncFactory.java
>  b4cbb4b1d9b52aaebfb562c02edc91e15a2a9a2e 
>   
> sharelib/oozie/src/main/java/org/apache/oozie/action/hadoop/HdfsOperations.java
>  874d371664ced3ae19a7b5d615c67d4c72e7cfdd 
>   sharelib/oozie/src/main/java/org/apache/oozie/action/hadoop/LauncherAM.java 
> 6a98d6ef3a09a7a75272f3f0f9a9c2fc5472e76d 
>   sharelib/oozie/src/main/java/org/apache/oozie/action/hadoop/ShellMain.java 
> bde7f1daf123d663d47f5d2fb2350cfa1ac33ed8 
>   
> sharelib/oozie/src/test/java/org/apache/oozie/action/hadoop/TestHdfsOperations.java
>  68c0f4bafe0322ff0d1007297cd6e8f38c92020a 
>   
> sharelib/oozie/src/test/java/org/apache/oozie/action/hadoop/TestLauncherAM.java
>  37af3dd9ed43f4504ff3909484237e2d05665c6c 
> 
> 
> Diff: https://reviews.apache.org/r/61560/diff/2/
> 
> 
> Testing
> ---
> 
> We're running tests on a Kerberized cluster with these changes.
> 
> 
> Thanks,
> 
> Peter Cseh
> 
>

Re: Review Request 61560: OOZIE-2909 LauncherAM: rewrite UGI calls

2017-09-06 Thread Peter Cseh 


> On Sept. 5, 2017, 5:05 p.m., Robert Kanter wrote:
> > core/src/main/java/org/apache/oozie/service/HadoopAccessorService.java
> > Lines 555 (patched)
> > 
> >
> > Do we always want to get an MR token?  Certainly not for Spark or 
> > Shell, but Java, Hive (Hive-on-Spark), and Pig (Pig-on-Spark), are more 
> > complicated.  I guess it's simpler to just always add the MR token.

I think we should always try to get one. It might be better just to log a 
warning if we failed instead of failing the action.


- Peter


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/61560/#review184556
---


On Sept. 5, 2017, 1:56 p.m., Peter Cseh wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/61560/
> ---
> 
> (Updated Sept. 5, 2017, 1:56 p.m.)
> 
> 
> Review request for oozie, Attila Sasvari, Peter Bacsko, Robert Kanter, and 
> Rohini Palaniswamy.
> 
> 
> Bugs: OOZIE-2909
> https://issues.apache.org/jira/browse/OOZIE-2909
> 
> 
> Repository: oozie-git
> 
> 
> Description
> ---
> 
> Changing UGI calls and cleaning up things around requesting tokens.
> 
> 
> Diffs
> -
> 
>   core/src/main/java/org/apache/oozie/ErrorCode.java 
> 662e1edc9c4b23b3606c751bf5ed4b531ee7ac62 
>   
> core/src/main/java/org/apache/oozie/action/hadoop/CredentialsProviderFactory.java
>  5ca8d3e80942840a5bba51a06694350929044042 
>   core/src/main/java/org/apache/oozie/action/hadoop/HCatCredentialHelper.java 
> 9804c7b673a899cd9c778850c9a64731cc642fb3 
>   core/src/main/java/org/apache/oozie/action/hadoop/HbaseCredentials.java 
> 4add5f14cb20ab77c755b81eff939335c0c5fddc 
>   core/src/main/java/org/apache/oozie/action/hadoop/Hive2Credentials.java 
> 0b495f75842b0033a3337ef6728a334bf5651770 
>   core/src/main/java/org/apache/oozie/action/hadoop/JavaActionExecutor.java 
> bca79aa052521ea4f6f16e76bd69f84fb16be790 
>   core/src/main/java/org/apache/oozie/service/HadoopAccessorService.java 
> 187cee2290ba4bbe358913d28a052d9c75f2369a 
>   core/src/test/java/org/apache/oozie/service/TestHadoopAccessorService.java 
> 960c2f9750062c98fdf7a2b456d3668069e07ca5 
>   
> sharelib/oozie/src/main/java/org/apache/oozie/action/hadoop/AMRMCallBackHandler.java
>  e6c9d04653be984bdd5aa77bee5f28b187e3de5d 
>   
> sharelib/oozie/src/main/java/org/apache/oozie/action/hadoop/AMRMClientAsyncFactory.java
>  b4cbb4b1d9b52aaebfb562c02edc91e15a2a9a2e 
>   
> sharelib/oozie/src/main/java/org/apache/oozie/action/hadoop/HdfsOperations.java
>  874d371664ced3ae19a7b5d615c67d4c72e7cfdd 
>   sharelib/oozie/src/main/java/org/apache/oozie/action/hadoop/LauncherAM.java 
> 6a98d6ef3a09a7a75272f3f0f9a9c2fc5472e76d 
>   sharelib/oozie/src/main/java/org/apache/oozie/action/hadoop/ShellMain.java 
> bde7f1daf123d663d47f5d2fb2350cfa1ac33ed8 
>   
> sharelib/oozie/src/test/java/org/apache/oozie/action/hadoop/TestHdfsOperations.java
>  68c0f4bafe0322ff0d1007297cd6e8f38c92020a 
>   
> sharelib/oozie/src/test/java/org/apache/oozie/action/hadoop/TestLauncherAM.java
>  37af3dd9ed43f4504ff3909484237e2d05665c6c 
> 
> 
> Diff: https://reviews.apache.org/r/61560/diff/2/
> 
> 
> Testing
> ---
> 
> We're running tests on a Kerberized cluster with these changes.
> 
> 
> Thanks,
> 
> Peter Cseh
> 
>

Re: Review Request 61560: OOZIE-2909 LauncherAM: rewrite UGI calls

2017-09-06 Thread Peter Cseh 


> On Aug. 22, 2017, 8:34 p.m., Robert Kanter wrote:
> > core/src/main/java/org/apache/oozie/action/hadoop/HCatCredentialHelper.java
> > Line 66 (original), 65 (patched)
> > 
> >
> > Have we verified that HCat/HMS is okay with changing the alias?  I 
> > don't know if maybe they were doing something funny with it.
> 
> Peter Cseh wrote:
> I know there is a reference for it in Sqoop 
> (https://issues.apache.org/jira/browse/SQOOP-3177) , but It does not seem to 
> work anyways.

I'm reverting the token alias changes


- Peter


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/61560/#review183516
---


On Sept. 5, 2017, 1:56 p.m., Peter Cseh wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/61560/
> ---
> 
> (Updated Sept. 5, 2017, 1:56 p.m.)
> 
> 
> Review request for oozie, Attila Sasvari, Peter Bacsko, Robert Kanter, and 
> Rohini Palaniswamy.
> 
> 
> Bugs: OOZIE-2909
> https://issues.apache.org/jira/browse/OOZIE-2909
> 
> 
> Repository: oozie-git
> 
> 
> Description
> ---
> 
> Changing UGI calls and cleaning up things around requesting tokens.
> 
> 
> Diffs
> -
> 
>   core/src/main/java/org/apache/oozie/ErrorCode.java 
> 662e1edc9c4b23b3606c751bf5ed4b531ee7ac62 
>   
> core/src/main/java/org/apache/oozie/action/hadoop/CredentialsProviderFactory.java
>  5ca8d3e80942840a5bba51a06694350929044042 
>   core/src/main/java/org/apache/oozie/action/hadoop/HCatCredentialHelper.java 
> 9804c7b673a899cd9c778850c9a64731cc642fb3 
>   core/src/main/java/org/apache/oozie/action/hadoop/HbaseCredentials.java 
> 4add5f14cb20ab77c755b81eff939335c0c5fddc 
>   core/src/main/java/org/apache/oozie/action/hadoop/Hive2Credentials.java 
> 0b495f75842b0033a3337ef6728a334bf5651770 
>   core/src/main/java/org/apache/oozie/action/hadoop/JavaActionExecutor.java 
> bca79aa052521ea4f6f16e76bd69f84fb16be790 
>   core/src/main/java/org/apache/oozie/service/HadoopAccessorService.java 
> 187cee2290ba4bbe358913d28a052d9c75f2369a 
>   core/src/test/java/org/apache/oozie/service/TestHadoopAccessorService.java 
> 960c2f9750062c98fdf7a2b456d3668069e07ca5 
>   
> sharelib/oozie/src/main/java/org/apache/oozie/action/hadoop/AMRMCallBackHandler.java
>  e6c9d04653be984bdd5aa77bee5f28b187e3de5d 
>   
> sharelib/oozie/src/main/java/org/apache/oozie/action/hadoop/AMRMClientAsyncFactory.java
>  b4cbb4b1d9b52aaebfb562c02edc91e15a2a9a2e 
>   
> sharelib/oozie/src/main/java/org/apache/oozie/action/hadoop/HdfsOperations.java
>  874d371664ced3ae19a7b5d615c67d4c72e7cfdd 
>   sharelib/oozie/src/main/java/org/apache/oozie/action/hadoop/LauncherAM.java 
> 6a98d6ef3a09a7a75272f3f0f9a9c2fc5472e76d 
>   sharelib/oozie/src/main/java/org/apache/oozie/action/hadoop/ShellMain.java 
> bde7f1daf123d663d47f5d2fb2350cfa1ac33ed8 
>   
> sharelib/oozie/src/test/java/org/apache/oozie/action/hadoop/TestHdfsOperations.java
>  68c0f4bafe0322ff0d1007297cd6e8f38c92020a 
>   
> sharelib/oozie/src/test/java/org/apache/oozie/action/hadoop/TestLauncherAM.java
>  37af3dd9ed43f4504ff3909484237e2d05665c6c 
> 
> 
> Diff: https://reviews.apache.org/r/61560/diff/2/
> 
> 
> Testing
> ---
> 
> We're running tests on a Kerberized cluster with these changes.
> 
> 
> Thanks,
> 
> Peter Cseh
> 
>

[jira] [Commented] (OOZIE-2687) Create XML schema for launcher configurations

2017-09-06 Thread Peter Cseh (JIRA) 

[ 
https://issues.apache.org/jira/browse/OOZIE-2687?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16155308#comment-16155308
 ] 

Peter Cseh commented on OOZIE-2687:
---

I agree with bumping 1.0 in every schema so it's easier to use them together.
I think we can deprecate things later, but this is also a good idea.
I prefer strict ordering in schemas as things would get messy around fields 
like . 



> Create XML schema for launcher configurations
> -
>
> Key: OOZIE-2687
> URL: https://issues.apache.org/jira/browse/OOZIE-2687
> Project: Oozie
>  Issue Type: Sub-task
>Affects Versions: 5.0.0
>Reporter: Peter Cseh
>Assignee: Attila Sasvari
>Priority: Critical
> Attachments: OOZIE-2687-001.patch, OOZIE-2687-002.patch, 
> OOZIE-2687-003.patch
>
>
> It was possible to manipulate the MapperLauncher's environment through 
> properties like:
> # mapreduce.map.memory.mb 
> # mapreduce.map.cpu.vcores
> # mapred.child.env
> # mapred.child.java.opts  
> # mapred.job.queue.name  - ability to set launcher queue
> E.g. We were using mapred.child.env to pass SPARK_HOME to the LauncherMapper 
> and make PySpark work. 
> Fixing OOZIE-2596 added a hack. We should decide how we support or break 
> compatibility and how we allow the manipulation of the Launcher environment.
> Verify if the new launcher section in global applies to actions in 
> sub-workflows as well. It did not use to work before and was only fixed in 
> OOZIE-2030. It would be good to have that testcase 
> (TestSubWorkflowActionExecutor. testParentGlobalConf) updated with the new 
> launcher section as well.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

Re: Review Request 61529: OOZIE-2687 Create XML schema for launcher configurations

2017-09-06 Thread Peter Cseh 

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/61529/#review184668
---




core/src/test/java/org/apache/oozie/service/TestSchemaService.java
Lines 143-148 (patched)


I don't like the fact that we have to use the workflow: prefix here for 
launcher properties. If anything I'd use the launcher: prefix, but ideally 
there should be no need for prefixes.


- Peter Cseh


On Sept. 5, 2017, 3:52 p.m., Attila Sasvari wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/61529/
> ---
> 
> (Updated Sept. 5, 2017, 3:52 p.m.)
> 
> 
> Review request for oozie and Peter Cseh.
> 
> 
> Repository: oozie-git
> 
> 
> Description
> ---
> 
> New workflow xsd to allow changing launcher configuration (e.g. setting java 
> system properties)
> 
> 
> Diffs
> -
> 
>   client/src/main/resources/hive-action-1.0.xsd PRE-CREATION 
>   client/src/main/resources/hive2-action-1.0.xsd PRE-CREATION 
>   client/src/main/resources/oozie-common-1.0.xsd PRE-CREATION 
>   client/src/main/resources/oozie-workflow-1.0.xsd PRE-CREATION 
>   client/src/main/resources/shell-action-1.0.xsd PRE-CREATION 
>   client/src/main/resources/spark-action-1.0.xsd PRE-CREATION 
>   client/src/main/resources/sqoop-action-1.0.xsd PRE-CREATION 
>   core/src/main/java/org/apache/oozie/action/hadoop/JavaActionExecutor.java 
> bca79aa052521ea4f6f16e76bd69f84fb16be790 
>   core/src/main/java/org/apache/oozie/action/hadoop/SqoopActionExecutor.java 
> 8fdc50cce15271c13b20d1aaab2ffb95cb8fa711 
>   core/src/main/java/org/apache/oozie/action/ssh/SshActionExecutor.java 
> 7e3348550e2ef44ae4bd6c5a7a417052bf9c108a 
>   core/src/main/java/org/apache/oozie/jms/MessageReceiver.java 
> 47bfd2bcf9b53eed0882c51cdb5c530e0f6ccfe7 
>   core/src/main/java/org/apache/oozie/service/LiteWorkflowStoreService.java 
> ffc29af2f834da6d0890ea7215c5a62fd7cd693e 
>   core/src/main/java/org/apache/oozie/service/SchemaService.java 
> 137e2c0b4840c6d3858683db1aa38f54bd55be92 
>   core/src/main/java/org/apache/oozie/util/WritableUtils.java 
> aa027e37ba23d4f481698e0bcd93c26c763a0b1f 
>   core/src/main/java/org/apache/oozie/util/schema/Input.java PRE-CREATION 
>   core/src/main/java/org/apache/oozie/util/schema/ResourceResolver.java 
> PRE-CREATION 
>   
> core/src/main/java/org/apache/oozie/workflow/lite/LauncherConfigHandler.java 
> PRE-CREATION 
>   
> core/src/main/java/org/apache/oozie/workflow/lite/LiteWorkflowAppParser.java 
> a74e5c759fe3e336e7e98c61b8f5ac52efa4100c 
>   core/src/main/java/org/apache/oozie/workflow/lite/LiteWorkflowLib.java 
> 23df0867e5984c518e8424d79365997bec28f619 
>   core/src/main/resources/oozie-default.xml 
> 0d174b5069aff36ca3a9d48722c5bba69623a046 
>   
> core/src/test/java/org/apache/oozie/action/hadoop/ActionExecutorTestCase.java 
> d74160a09ee237f4067e00ded526e7ec94528a6b 
>   core/src/test/java/org/apache/oozie/action/hadoop/LauncherMainTester.java 
> 6cee7a8948ca7f2c7aaf44d4f172d18a1308a482 
>   
> core/src/test/java/org/apache/oozie/action/hadoop/TestJavaActionExecutor.java 
> ce674adf9cffa20690083e6298875a6f9047a109 
>   core/src/test/java/org/apache/oozie/service/TestSchemaService.java 
> 88a10dbead32879334c3953a90a411a3a18fc0b3 
>   core/src/test/java/org/apache/oozie/util/TestMetricsInstrumentation.java 
> a882c82b25154338f89e8a90f831b6f0300fbfe7 
>   
> core/src/test/java/org/apache/oozie/workflow/lite/TestLiteWorkflowAppParser.java
>  21332404bea79fe0a3cea045d57b3e52a1f15cae 
>   core/src/test/resources/wf-schema-global-launcherconf-override.xml 
> PRE-CREATION 
>   core/src/test/resources/wf-schema-global-launcherconf.xml PRE-CREATION 
>   release-log.txt e2311eab6392ca08ad855da887e391f3f14a7408 
>   
> sharelib/distcp/src/main/java/org/apache/oozie/action/hadoop/DistcpMain.java 
> d17d5beb76754655720a0e29079daf676c507841 
>   sharelib/hive/src/main/java/org/apache/oozie/action/hadoop/HiveMain.java 
> fb3fabf027a1dda537d2bd86d574abaf95a685db 
>   sharelib/oozie/src/main/java/org/apache/oozie/action/hadoop/LauncherAM.java 
> 6a98d6ef3a09a7a75272f3f0f9a9c2fc5472e76d 
>   
> sharelib/oozie/src/main/java/org/apache/oozie/action/hadoop/LauncherMain.java 
> f1f52c6e876fae56c90979baef3d31a0f5ba2396 
>   
> sharelib/oozie/src/main/java/org/apache/oozie/action/hadoop/MapReduceMain.java
>  77aaa62e23185e6df053f01fd1c4db1f2e972f6b 
>   
> sharelib/oozie/src/main/java/org/apache/oozie/action/hadoop/PasswordMasker.java
>  6a73bf9b8b1d0c5cbb008594c75c9e779298a29d 
>   
> sharelib/oozie/src/test/java/org/apache/oozie/action/hadoop/TestLauncherMain.java
>  a793df1330d10828101abf9107d565db7972a95f 
>   sharelib/pig/src/main/java/org/apache/oozie/action/hadoop/PigMain.java 
> 13ee04f4482

Re: Review Request 61529: OOZIE-2687 Create XML schema for launcher configurations

2017-09-06 Thread Peter Bacsko 

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/61529/#review184655
---




core/src/main/java/org/apache/oozie/action/hadoop/JavaActionExecutor.java
Lines 1296 (patched)


Unnecessary comment



core/src/main/java/org/apache/oozie/action/hadoop/JavaActionExecutor.java
Lines 1363 (patched)


I haven't seen a testcase which covers this path, would be useful to have 
one



core/src/main/java/org/apache/oozie/action/ssh/SshActionExecutor.java
Line 179 (original), 179 (patched)


Could you explain what this rename is about? Now We have "processSettings" 
instead of "processw everywhere.



core/src/main/resources/oozie-default.xml
Lines 3087 (patched)


Perhaps this property could be called

oozie.launcher.default.memory.mb

so users won't accidentally define it in different units.


- Peter Bacsko


On szept. 5, 2017, 3:52 du, Attila Sasvari wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/61529/
> ---
> 
> (Updated szept. 5, 2017, 3:52 du)
> 
> 
> Review request for oozie and Peter Cseh.
> 
> 
> Repository: oozie-git
> 
> 
> Description
> ---
> 
> New workflow xsd to allow changing launcher configuration (e.g. setting java 
> system properties)
> 
> 
> Diffs
> -
> 
>   client/src/main/resources/hive-action-1.0.xsd PRE-CREATION 
>   client/src/main/resources/hive2-action-1.0.xsd PRE-CREATION 
>   client/src/main/resources/oozie-common-1.0.xsd PRE-CREATION 
>   client/src/main/resources/oozie-workflow-1.0.xsd PRE-CREATION 
>   client/src/main/resources/shell-action-1.0.xsd PRE-CREATION 
>   client/src/main/resources/spark-action-1.0.xsd PRE-CREATION 
>   client/src/main/resources/sqoop-action-1.0.xsd PRE-CREATION 
>   core/src/main/java/org/apache/oozie/action/hadoop/JavaActionExecutor.java 
> bca79aa052521ea4f6f16e76bd69f84fb16be790 
>   core/src/main/java/org/apache/oozie/action/hadoop/SqoopActionExecutor.java 
> 8fdc50cce15271c13b20d1aaab2ffb95cb8fa711 
>   core/src/main/java/org/apache/oozie/action/ssh/SshActionExecutor.java 
> 7e3348550e2ef44ae4bd6c5a7a417052bf9c108a 
>   core/src/main/java/org/apache/oozie/jms/MessageReceiver.java 
> 47bfd2bcf9b53eed0882c51cdb5c530e0f6ccfe7 
>   core/src/main/java/org/apache/oozie/service/LiteWorkflowStoreService.java 
> ffc29af2f834da6d0890ea7215c5a62fd7cd693e 
>   core/src/main/java/org/apache/oozie/service/SchemaService.java 
> 137e2c0b4840c6d3858683db1aa38f54bd55be92 
>   core/src/main/java/org/apache/oozie/util/WritableUtils.java 
> aa027e37ba23d4f481698e0bcd93c26c763a0b1f 
>   core/src/main/java/org/apache/oozie/util/schema/Input.java PRE-CREATION 
>   core/src/main/java/org/apache/oozie/util/schema/ResourceResolver.java 
> PRE-CREATION 
>   
> core/src/main/java/org/apache/oozie/workflow/lite/LauncherConfigHandler.java 
> PRE-CREATION 
>   
> core/src/main/java/org/apache/oozie/workflow/lite/LiteWorkflowAppParser.java 
> a74e5c759fe3e336e7e98c61b8f5ac52efa4100c 
>   core/src/main/java/org/apache/oozie/workflow/lite/LiteWorkflowLib.java 
> 23df0867e5984c518e8424d79365997bec28f619 
>   core/src/main/resources/oozie-default.xml 
> 0d174b5069aff36ca3a9d48722c5bba69623a046 
>   
> core/src/test/java/org/apache/oozie/action/hadoop/ActionExecutorTestCase.java 
> d74160a09ee237f4067e00ded526e7ec94528a6b 
>   core/src/test/java/org/apache/oozie/action/hadoop/LauncherMainTester.java 
> 6cee7a8948ca7f2c7aaf44d4f172d18a1308a482 
>   
> core/src/test/java/org/apache/oozie/action/hadoop/TestJavaActionExecutor.java 
> ce674adf9cffa20690083e6298875a6f9047a109 
>   core/src/test/java/org/apache/oozie/service/TestSchemaService.java 
> 88a10dbead32879334c3953a90a411a3a18fc0b3 
>   core/src/test/java/org/apache/oozie/util/TestMetricsInstrumentation.java 
> a882c82b25154338f89e8a90f831b6f0300fbfe7 
>   
> core/src/test/java/org/apache/oozie/workflow/lite/TestLiteWorkflowAppParser.java
>  21332404bea79fe0a3cea045d57b3e52a1f15cae 
>   core/src/test/resources/wf-schema-global-launcherconf-override.xml 
> PRE-CREATION 
>   core/src/test/resources/wf-schema-global-launcherconf.xml PRE-CREATION 
>   release-log.txt e2311eab6392ca08ad855da887e391f3f14a7408 
>   
> sharelib/distcp/src/main/java/org/apache/oozie/action/hadoop/DistcpMain.java 
> d17d5beb76754655720a0e29079daf676c507841 
>   sharelib/hive/src/main/java/org/apache/oozie/action/hadoop/HiveMain.java 
> fb3fabf027a1dda537d2bd86d574abaf95a685db 
>   sharelib/oozie/src/main/java/org/apache/oozie/action/hadoop/LauncherAM.java 
> 6a98d6ef3a09a7a75272f3f0f9a9c2fc5472e76d 
>

Build failed in Jenkins: oozie-trunk-precommit-build #4049

2017-09-06 Thread Apache Jenkins Server 
See 


Changes:

[pbacsko] OOZIE-3026 fix openjpa enhancer stage during build for logging

[pbacsko] OOZIE-3035 HDFS HA and log aggregation: getting HDFS delegation token

[gezapeti] OOZIE-2858 HiveMain, ShellMain and SparkMain should not overwrite

--
[...truncated 1.59 MB...]
[INFO] Copying jpam-1.1.jar to 

[INFO] Copying hive-service-1.2.0.jar to 

[INFO] Copying hive-hcatalog-server-extensions-1.2.0.jar to 

[INFO] Copying hadoop-mapreduce-client-core-2.6.0.jar to 

[INFO] Copying json-20090211.jar to 

[INFO] Copying avro-1.7.4.jar to 

[INFO] Copying jetty-util-9.2.19.v20160908.jar to 

[INFO] Copying apache-curator-2.6.0.pom to 

[INFO] 
[INFO] --- maven-assembly-plugin:2.2.1:single (default-cli) @ oozie-server ---
[INFO] Reading assembly descriptor: ../src/main/assemblies/empty.xml
[INFO] 
[INFO] 
[INFO] Building Apache Oozie Distro 5.0.0-SNAPSHOT
[INFO] 
[INFO] 
[INFO] --- maven-remote-resources-plugin:1.5:process (default) @ oozie-distro 
---
[INFO] 
[INFO] --- maven-resources-plugin:2.5:resources (default-resources) @ 
oozie-distro ---
[debug] execute contextualize
[INFO] Using 'UTF-8' encoding to copy filtered resources.
[INFO] skip non existing resourceDirectory 

[INFO] Copying 3 resources
[INFO] 
[INFO] --- maven-compiler-plugin:2.3.2:compile (default-compile) @ oozie-distro 
---
[INFO] No sources to compile
[INFO] 
[INFO] --- maven-resources-plugin:2.5:testResources (default-testResources) @ 
oozie-distro ---
[debug] execute contextualize
[INFO] Using 'UTF-8' encoding to copy filtered resources.
[INFO] skip non existing resourceDirectory 

[INFO] Copying 3 resources
[INFO] 
[INFO] --- maven-compiler-plugin:2.3.2:testCompile (default-testCompile) @ 
oozie-distro ---
[INFO] No sources to compile
[INFO] 
[INFO] --- maven-surefire-plugin:2.12.2:test (default-test) @ oozie-distro ---
[INFO] Tests are skipped.
[INFO] 
[INFO] --- maven-jar-plugin:2.3.1:jar (default-jar) @ oozie-distro ---
[INFO] Building jar: 

[INFO] 
[INFO] --- maven-site-plugin:2.0-beta-6:attach-descriptor (attach-descriptor) @ 
oozie-distro ---
[INFO] 
[INFO] --- maven-assembly-plugin:2.2.1:single (default-cli) @ oozie-distro ---
[INFO] Reading assembly descriptor: ../src/main/assemblies/distro.xml
[INFO] Copying files to 

[INFO] Building tar : 

[INFO] 
[INFO] 
[INFO] Building Apache Oozie ZooKeeper Security Tests 5.0.0-SNAPSHOT
[INFO] 
[INFO] 
[INFO] --- maven-remote-resources-plugin:1.5:process (default) @ 
oozie-zookeeper-security-tests ---
[INFO] 
[INFO] --- maven-resources-plugin:2.5:resources (default-resources) @ 
oozie-zookeeper-security-tests ---
[debug] execute contextualize
[INFO] Using 'UTF-8' encoding to copy filtered resources.
[INFO] skip non existing resourceDirectory 

[INFO] Copying 3 resources
[INFO] 
[INFO] --- maven-compiler-plugin:2.3.2:compile (default-compile) @ 
oozie-zookeeper-security-tests ---
[INFO] No sources to compile
[INFO] 
[INFO] --- maven-resources-plugin:2.5:testResources (default-testResources) @ 
oozie-zookeeper-security-tests ---
[debug] execute contextualize
[INFO] Using 'UTF-8' encoding to

[jira] [Commented] (OOZIE-2909) LauncherAM: rewrite UGI calls

2017-09-06 Thread Hadoop QA (JIRA) 

[ 
https://issues.apache.org/jira/browse/OOZIE-2909?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16155254#comment-16155254
 ] 

Hadoop QA commented on OOZIE-2909:
--

Testing JIRA OOZIE-2909

Cleaning local git workspace



{color:green}+1 PATCH_APPLIES{color}
{color:green}+1 CLEAN{color}
{color:red}-1 RAW_PATCH_ANALYSIS{color}
.{color:green}+1{color} the patch does not introduce any @author tags
.{color:green}+1{color} the patch does not introduce any tabs
.{color:green}+1{color} the patch does not introduce any trailing spaces
.{color:red}-1{color} the patch contains 5 line(s) longer than 132 
characters
.{color:green}+1{color} the patch does adds/modifies 3 testcase(s)
{color:green}+1 RAT{color}
.{color:green}+1{color} the patch does not seem to introduce new RAT 
warnings
{color:green}+1 JAVADOC{color}
.{color:green}+1{color} the patch does not seem to introduce new Javadoc 
warnings
.{color:red}WARNING{color}: the current HEAD has 77 Javadoc warning(s)
{color:green}+1 COMPILE{color}
.{color:green}+1{color} HEAD compiles
.{color:green}+1{color} patch compiles
.{color:green}+1{color} the patch does not seem to introduce new javac 
warnings
{color:orange}0{color} There are [7] new bugs found in total that would be nice 
to have fixed.
. {color:orange}0{color} There are [4] new bugs found in [core] that would be 
nice to have fixed.
. You can find the FindBugs diff here: core/findbugs-new.html
. {color:green}+1{color} There are no new bugs found in [server].
. {color:green}+1{color} There are no new bugs found in [client].
. {color:green}+1{color} There are no new bugs found in [docs].
. {color:green}+1{color} There are no new bugs found in [examples].
. {color:green}+1{color} There are no new bugs found in [sharelib/hive].
. {color:green}+1{color} There are no new bugs found in [sharelib/pig].
. {color:green}+1{color} There are no new bugs found in [sharelib/distcp].
. {color:green}+1{color} There are no new bugs found in [sharelib/streaming].
. {color:green}+1{color} There are no new bugs found in [sharelib/spark].
. {color:green}+1{color} There are no new bugs found in [sharelib/hcatalog].
. {color:green}+1{color} There are no new bugs found in [sharelib/sqoop].
. {color:green}+1{color} There are no new bugs found in [sharelib/hive2].
. {color:orange}0{color} There are [3] new bugs found in [sharelib/oozie] that 
would be nice to have fixed.
. You can find the FindBugs diff here: sharelib/oozie/findbugs-new.html
. {color:green}+1{color} There are no new bugs found in [tools].
{color:green}+1 BACKWARDS_COMPATIBILITY{color}
.{color:green}+1{color} the patch does not change any JPA 
Entity/Colum/Basic/Lob/Transient annotations
.{color:green}+1{color} the patch does not modify JPA files
{color:red}-1 TESTS{color}
.Tests run: 2012
.Tests failed: 1
.Tests errors: 1

.The patch failed the following testcases:

.  testEscapedSingleQuotes(org.apache.oozie.TestErrorCode)

.Tests failing with errors:
.  testJMXInstrumentation(org.apache.oozie.util.TestMetricsInstrumentation)

{color:green}+1 DISTRO{color}
.{color:green}+1{color} distro tarball builds with the patch 


{color:red}*-1 Overall result, please check the reported -1(s)*{color}

{color:red}. There is at least one warning, please check{color}

The full output of the test-patch run is available at

. https://builds.apache.org/job/oozie-trunk-precommit-build/4049/

> LauncherAM: rewrite UGI calls
> -
>
> Key: OOZIE-2909
> URL: https://issues.apache.org/jira/browse/OOZIE-2909
> Project: Oozie
>  Issue Type: Sub-task
>Reporter: Peter Bacsko
>Assignee: Peter Cseh
>Priority: Blocker
> Attachments: OOZIE-2909-001.patch, OOZIE-2909-002.patch
>
>
> There are two problems in LauncherAM:
> 1) In a kerberized cluster, we have to use {{UGI.getLoginUser()}} even though 
> a call to {{createRemoteUser()}} should be enough, just like in 
> {{MRAppMaster.java}}. Reference code:
> https://github.com/apache/hadoop/blob/3ed3062fe3979ff55a411b730a8eee2b2c96d6b3/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/MRAppMaster.java#L1718-L1726
> 2) It's enough if we wrap the whole LauncherAM logic in a single {{doAs()}} 
> call. No need for multiple {{doAs()}}, just makes the whole thing more 
> complicated.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

Re: Review Request 61529: OOZIE-2687 Create XML schema for launcher configurations

2017-09-06 Thread Peter Bacsko 

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/61529/#review184650
---




core/src/test/java/org/apache/oozie/action/hadoop/TestJavaActionExecutor.java
Lines 2268 (patched)


Have you considered creating new tests?

Is this coverage enough?



core/src/test/java/org/apache/oozie/service/TestSchemaService.java
Lines 101 (patched)


Wouldn't it be better to store all these dummy workflows in an external 
file?


- Peter Bacsko


On szept. 5, 2017, 3:52 du, Attila Sasvari wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/61529/
> ---
> 
> (Updated szept. 5, 2017, 3:52 du)
> 
> 
> Review request for oozie and Peter Cseh.
> 
> 
> Repository: oozie-git
> 
> 
> Description
> ---
> 
> New workflow xsd to allow changing launcher configuration (e.g. setting java 
> system properties)
> 
> 
> Diffs
> -
> 
>   client/src/main/resources/hive-action-1.0.xsd PRE-CREATION 
>   client/src/main/resources/hive2-action-1.0.xsd PRE-CREATION 
>   client/src/main/resources/oozie-common-1.0.xsd PRE-CREATION 
>   client/src/main/resources/oozie-workflow-1.0.xsd PRE-CREATION 
>   client/src/main/resources/shell-action-1.0.xsd PRE-CREATION 
>   client/src/main/resources/spark-action-1.0.xsd PRE-CREATION 
>   client/src/main/resources/sqoop-action-1.0.xsd PRE-CREATION 
>   core/src/main/java/org/apache/oozie/action/hadoop/JavaActionExecutor.java 
> bca79aa052521ea4f6f16e76bd69f84fb16be790 
>   core/src/main/java/org/apache/oozie/action/hadoop/SqoopActionExecutor.java 
> 8fdc50cce15271c13b20d1aaab2ffb95cb8fa711 
>   core/src/main/java/org/apache/oozie/action/ssh/SshActionExecutor.java 
> 7e3348550e2ef44ae4bd6c5a7a417052bf9c108a 
>   core/src/main/java/org/apache/oozie/jms/MessageReceiver.java 
> 47bfd2bcf9b53eed0882c51cdb5c530e0f6ccfe7 
>   core/src/main/java/org/apache/oozie/service/LiteWorkflowStoreService.java 
> ffc29af2f834da6d0890ea7215c5a62fd7cd693e 
>   core/src/main/java/org/apache/oozie/service/SchemaService.java 
> 137e2c0b4840c6d3858683db1aa38f54bd55be92 
>   core/src/main/java/org/apache/oozie/util/WritableUtils.java 
> aa027e37ba23d4f481698e0bcd93c26c763a0b1f 
>   core/src/main/java/org/apache/oozie/util/schema/Input.java PRE-CREATION 
>   core/src/main/java/org/apache/oozie/util/schema/ResourceResolver.java 
> PRE-CREATION 
>   
> core/src/main/java/org/apache/oozie/workflow/lite/LauncherConfigHandler.java 
> PRE-CREATION 
>   
> core/src/main/java/org/apache/oozie/workflow/lite/LiteWorkflowAppParser.java 
> a74e5c759fe3e336e7e98c61b8f5ac52efa4100c 
>   core/src/main/java/org/apache/oozie/workflow/lite/LiteWorkflowLib.java 
> 23df0867e5984c518e8424d79365997bec28f619 
>   core/src/main/resources/oozie-default.xml 
> 0d174b5069aff36ca3a9d48722c5bba69623a046 
>   
> core/src/test/java/org/apache/oozie/action/hadoop/ActionExecutorTestCase.java 
> d74160a09ee237f4067e00ded526e7ec94528a6b 
>   core/src/test/java/org/apache/oozie/action/hadoop/LauncherMainTester.java 
> 6cee7a8948ca7f2c7aaf44d4f172d18a1308a482 
>   
> core/src/test/java/org/apache/oozie/action/hadoop/TestJavaActionExecutor.java 
> ce674adf9cffa20690083e6298875a6f9047a109 
>   core/src/test/java/org/apache/oozie/service/TestSchemaService.java 
> 88a10dbead32879334c3953a90a411a3a18fc0b3 
>   core/src/test/java/org/apache/oozie/util/TestMetricsInstrumentation.java 
> a882c82b25154338f89e8a90f831b6f0300fbfe7 
>   
> core/src/test/java/org/apache/oozie/workflow/lite/TestLiteWorkflowAppParser.java
>  21332404bea79fe0a3cea045d57b3e52a1f15cae 
>   core/src/test/resources/wf-schema-global-launcherconf-override.xml 
> PRE-CREATION 
>   core/src/test/resources/wf-schema-global-launcherconf.xml PRE-CREATION 
>   release-log.txt e2311eab6392ca08ad855da887e391f3f14a7408 
>   
> sharelib/distcp/src/main/java/org/apache/oozie/action/hadoop/DistcpMain.java 
> d17d5beb76754655720a0e29079daf676c507841 
>   sharelib/hive/src/main/java/org/apache/oozie/action/hadoop/HiveMain.java 
> fb3fabf027a1dda537d2bd86d574abaf95a685db 
>   sharelib/oozie/src/main/java/org/apache/oozie/action/hadoop/LauncherAM.java 
> 6a98d6ef3a09a7a75272f3f0f9a9c2fc5472e76d 
>   
> sharelib/oozie/src/main/java/org/apache/oozie/action/hadoop/LauncherMain.java 
> f1f52c6e876fae56c90979baef3d31a0f5ba2396 
>   
> sharelib/oozie/src/main/java/org/apache/oozie/action/hadoop/MapReduceMain.java
>  77aaa62e23185e6df053f01fd1c4db1f2e972f6b 
>   
> sharelib/oozie/src/main/java/org/apache/oozie/action/hadoop/PasswordMasker.java
>  6a73bf9b8b1d0c5cbb008594c75c9e779298a29d 
>   
> sharelib/oozie/src/test/java/org/apache/oozie/action/hadoop/TestLauncherMain.java
>  a793df1330

Re: Review Request 61529: OOZIE-2687 Create XML schema for launcher configurations

2017-09-06 Thread Peter Bacsko 

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/61529/#review184649
---




client/src/main/resources/oozie-common-1.0.xsd
Lines 24 (patched)


Do we want a sequence here? As Robert mentioned, an xs:choice could be 
better.



core/src/main/java/org/apache/oozie/action/hadoop/JavaActionExecutor.java
Lines 26 (patched)


Avoid *



core/src/main/java/org/apache/oozie/action/hadoop/JavaActionExecutor.java
Lines 33 (patched)


Avoid *



core/src/main/java/org/apache/oozie/action/hadoop/JavaActionExecutor.java
Lines 865 (patched)


Identation seems odd



core/src/main/java/org/apache/oozie/action/hadoop/JavaActionExecutor.java
Lines 981 (patched)


This should not be logged on an INFO level



core/src/main/java/org/apache/oozie/action/hadoop/JavaActionExecutor.java
Line 1238 (original), 1269 (patched)


In theory we don't have to handle the case when we have both java-opt and 
java-opts, because the schema validator should take care of it.

Still, I'd be more confident if we explicitly checked this condition (that 
is, one or the another). We can add a comment that it's just a safety net.



core/src/main/java/org/apache/oozie/util/WritableUtils.java
Lines 191 (patched)


I remember creating this method for serializing something, but later 
abandoned that approach.

If it's not necessary, please remote this,



core/src/main/java/org/apache/oozie/util/WritableUtils.java
Lines 245 (patched)


Same thing here.


- Peter Bacsko


On szept. 5, 2017, 3:52 du, Attila Sasvari wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/61529/
> ---
> 
> (Updated szept. 5, 2017, 3:52 du)
> 
> 
> Review request for oozie and Peter Cseh.
> 
> 
> Repository: oozie-git
> 
> 
> Description
> ---
> 
> New workflow xsd to allow changing launcher configuration (e.g. setting java 
> system properties)
> 
> 
> Diffs
> -
> 
>   client/src/main/resources/hive-action-1.0.xsd PRE-CREATION 
>   client/src/main/resources/hive2-action-1.0.xsd PRE-CREATION 
>   client/src/main/resources/oozie-common-1.0.xsd PRE-CREATION 
>   client/src/main/resources/oozie-workflow-1.0.xsd PRE-CREATION 
>   client/src/main/resources/shell-action-1.0.xsd PRE-CREATION 
>   client/src/main/resources/spark-action-1.0.xsd PRE-CREATION 
>   client/src/main/resources/sqoop-action-1.0.xsd PRE-CREATION 
>   core/src/main/java/org/apache/oozie/action/hadoop/JavaActionExecutor.java 
> bca79aa052521ea4f6f16e76bd69f84fb16be790 
>   core/src/main/java/org/apache/oozie/action/hadoop/SqoopActionExecutor.java 
> 8fdc50cce15271c13b20d1aaab2ffb95cb8fa711 
>   core/src/main/java/org/apache/oozie/action/ssh/SshActionExecutor.java 
> 7e3348550e2ef44ae4bd6c5a7a417052bf9c108a 
>   core/src/main/java/org/apache/oozie/jms/MessageReceiver.java 
> 47bfd2bcf9b53eed0882c51cdb5c530e0f6ccfe7 
>   core/src/main/java/org/apache/oozie/service/LiteWorkflowStoreService.java 
> ffc29af2f834da6d0890ea7215c5a62fd7cd693e 
>   core/src/main/java/org/apache/oozie/service/SchemaService.java 
> 137e2c0b4840c6d3858683db1aa38f54bd55be92 
>   core/src/main/java/org/apache/oozie/util/WritableUtils.java 
> aa027e37ba23d4f481698e0bcd93c26c763a0b1f 
>   core/src/main/java/org/apache/oozie/util/schema/Input.java PRE-CREATION 
>   core/src/main/java/org/apache/oozie/util/schema/ResourceResolver.java 
> PRE-CREATION 
>   
> core/src/main/java/org/apache/oozie/workflow/lite/LauncherConfigHandler.java 
> PRE-CREATION 
>   
> core/src/main/java/org/apache/oozie/workflow/lite/LiteWorkflowAppParser.java 
> a74e5c759fe3e336e7e98c61b8f5ac52efa4100c 
>   core/src/main/java/org/apache/oozie/workflow/lite/LiteWorkflowLib.java 
> 23df0867e5984c518e8424d79365997bec28f619 
>   core/src/main/resources/oozie-default.xml 
> 0d174b5069aff36ca3a9d48722c5bba69623a046 
>   
> core/src/test/java/org/apache/oozie/action/hadoop/ActionExecutorTestCase.java 
> d74160a09ee237f4067e00ded526e7ec94528a6b 
>   core/src/test/java/org/apache/oozie/action/hadoop/LauncherMainTester.java 
> 6cee7a8948ca7f2c7aaf44d4f172d18a1308a482 
>   
> core/src/test/java/org/apache/oozie/action/hadoop/TestJavaActionExecutor.java 
> ce674adf9cffa20690083e6298875a6f9047a109 
>   core/src/test/java/org/apache/oozie/service/TestSchemaService.java 
> 88a10dbead32879334c3953a90a411a3a18fc0b

Re: Review Request 61529: OOZIE-2687 Create XML schema for launcher configurations

2017-09-06 Thread Peter Cseh 

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/61529/#review184643
---




core/src/main/java/org/apache/oozie/workflow/lite/LauncherConfigHandler.java
Lines 59 (patched)


In oozie-default.xml you state that this is in Mb. Would it be better to 
call this memory.mb or just leave it like this?


- Peter Cseh


On Sept. 5, 2017, 3:52 p.m., Attila Sasvari wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/61529/
> ---
> 
> (Updated Sept. 5, 2017, 3:52 p.m.)
> 
> 
> Review request for oozie and Peter Cseh.
> 
> 
> Repository: oozie-git
> 
> 
> Description
> ---
> 
> New workflow xsd to allow changing launcher configuration (e.g. setting java 
> system properties)
> 
> 
> Diffs
> -
> 
>   client/src/main/resources/hive-action-1.0.xsd PRE-CREATION 
>   client/src/main/resources/hive2-action-1.0.xsd PRE-CREATION 
>   client/src/main/resources/oozie-common-1.0.xsd PRE-CREATION 
>   client/src/main/resources/oozie-workflow-1.0.xsd PRE-CREATION 
>   client/src/main/resources/shell-action-1.0.xsd PRE-CREATION 
>   client/src/main/resources/spark-action-1.0.xsd PRE-CREATION 
>   client/src/main/resources/sqoop-action-1.0.xsd PRE-CREATION 
>   core/src/main/java/org/apache/oozie/action/hadoop/JavaActionExecutor.java 
> bca79aa052521ea4f6f16e76bd69f84fb16be790 
>   core/src/main/java/org/apache/oozie/action/hadoop/SqoopActionExecutor.java 
> 8fdc50cce15271c13b20d1aaab2ffb95cb8fa711 
>   core/src/main/java/org/apache/oozie/action/ssh/SshActionExecutor.java 
> 7e3348550e2ef44ae4bd6c5a7a417052bf9c108a 
>   core/src/main/java/org/apache/oozie/jms/MessageReceiver.java 
> 47bfd2bcf9b53eed0882c51cdb5c530e0f6ccfe7 
>   core/src/main/java/org/apache/oozie/service/LiteWorkflowStoreService.java 
> ffc29af2f834da6d0890ea7215c5a62fd7cd693e 
>   core/src/main/java/org/apache/oozie/service/SchemaService.java 
> 137e2c0b4840c6d3858683db1aa38f54bd55be92 
>   core/src/main/java/org/apache/oozie/util/WritableUtils.java 
> aa027e37ba23d4f481698e0bcd93c26c763a0b1f 
>   core/src/main/java/org/apache/oozie/util/schema/Input.java PRE-CREATION 
>   core/src/main/java/org/apache/oozie/util/schema/ResourceResolver.java 
> PRE-CREATION 
>   
> core/src/main/java/org/apache/oozie/workflow/lite/LauncherConfigHandler.java 
> PRE-CREATION 
>   
> core/src/main/java/org/apache/oozie/workflow/lite/LiteWorkflowAppParser.java 
> a74e5c759fe3e336e7e98c61b8f5ac52efa4100c 
>   core/src/main/java/org/apache/oozie/workflow/lite/LiteWorkflowLib.java 
> 23df0867e5984c518e8424d79365997bec28f619 
>   core/src/main/resources/oozie-default.xml 
> 0d174b5069aff36ca3a9d48722c5bba69623a046 
>   
> core/src/test/java/org/apache/oozie/action/hadoop/ActionExecutorTestCase.java 
> d74160a09ee237f4067e00ded526e7ec94528a6b 
>   core/src/test/java/org/apache/oozie/action/hadoop/LauncherMainTester.java 
> 6cee7a8948ca7f2c7aaf44d4f172d18a1308a482 
>   
> core/src/test/java/org/apache/oozie/action/hadoop/TestJavaActionExecutor.java 
> ce674adf9cffa20690083e6298875a6f9047a109 
>   core/src/test/java/org/apache/oozie/service/TestSchemaService.java 
> 88a10dbead32879334c3953a90a411a3a18fc0b3 
>   core/src/test/java/org/apache/oozie/util/TestMetricsInstrumentation.java 
> a882c82b25154338f89e8a90f831b6f0300fbfe7 
>   
> core/src/test/java/org/apache/oozie/workflow/lite/TestLiteWorkflowAppParser.java
>  21332404bea79fe0a3cea045d57b3e52a1f15cae 
>   core/src/test/resources/wf-schema-global-launcherconf-override.xml 
> PRE-CREATION 
>   core/src/test/resources/wf-schema-global-launcherconf.xml PRE-CREATION 
>   release-log.txt e2311eab6392ca08ad855da887e391f3f14a7408 
>   
> sharelib/distcp/src/main/java/org/apache/oozie/action/hadoop/DistcpMain.java 
> d17d5beb76754655720a0e29079daf676c507841 
>   sharelib/hive/src/main/java/org/apache/oozie/action/hadoop/HiveMain.java 
> fb3fabf027a1dda537d2bd86d574abaf95a685db 
>   sharelib/oozie/src/main/java/org/apache/oozie/action/hadoop/LauncherAM.java 
> 6a98d6ef3a09a7a75272f3f0f9a9c2fc5472e76d 
>   
> sharelib/oozie/src/main/java/org/apache/oozie/action/hadoop/LauncherMain.java 
> f1f52c6e876fae56c90979baef3d31a0f5ba2396 
>   
> sharelib/oozie/src/main/java/org/apache/oozie/action/hadoop/MapReduceMain.java
>  77aaa62e23185e6df053f01fd1c4db1f2e972f6b 
>   
> sharelib/oozie/src/main/java/org/apache/oozie/action/hadoop/PasswordMasker.java
>  6a73bf9b8b1d0c5cbb008594c75c9e779298a29d 
>   
> sharelib/oozie/src/test/java/org/apache/oozie/action/hadoop/TestLauncherMain.java
>  a793df1330d10828101abf9107d565db7972a95f 
>   sharelib/pig/src/main/java/org/apache/oozie/action/hadoop/PigMain.java 
> 13ee04f4482471b61ee363f9cb98f04436c9e580 
>   sharelib/pig/src/test/jav

[jira] [Updated] (OOZIE-2687) Create XML schema for launcher configurations

2017-09-06 Thread Attila Sasvari (JIRA) 

 [ 
https://issues.apache.org/jira/browse/OOZIE-2687?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Attila Sasvari updated OOZIE-2687:
--
Attachment: OOZIE-2687-003.patch

> Create XML schema for launcher configurations
> -
>
> Key: OOZIE-2687
> URL: https://issues.apache.org/jira/browse/OOZIE-2687
> Project: Oozie
>  Issue Type: Sub-task
>Affects Versions: 5.0.0
>Reporter: Peter Cseh
>Assignee: Attila Sasvari
>Priority: Critical
> Attachments: OOZIE-2687-001.patch, OOZIE-2687-002.patch, 
> OOZIE-2687-003.patch
>
>
> It was possible to manipulate the MapperLauncher's environment through 
> properties like:
> # mapreduce.map.memory.mb 
> # mapreduce.map.cpu.vcores
> # mapred.child.env
> # mapred.child.java.opts  
> # mapred.job.queue.name  - ability to set launcher queue
> E.g. We were using mapred.child.env to pass SPARK_HOME to the LauncherMapper 
> and make PySpark work. 
> Fixing OOZIE-2596 added a hack. We should decide how we support or break 
> compatibility and how we allow the manipulation of the Launcher environment.
> Verify if the new launcher section in global applies to actions in 
> sub-workflows as well. It did not use to work before and was only fixed in 
> OOZIE-2030. It would be good to have that testcase 
> (TestSubWorkflowActionExecutor. testParentGlobalConf) updated with the new 
> launcher section as well.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)