[jira] Subscription: Oozie Patch Available

2017-10-19 Thread jira 
Issue Subscription
Filter: Oozie Patch Available (108 issues)

Subscriber: ooziedaily

Key Summary
OOZIE-3094  fix for grammar mistake
https://issues.apache.org/jira/browse/OOZIE-3094
OOZIE-3084  Add missing JVM properties to Jetty startup script
https://issues.apache.org/jira/browse/OOZIE-3084
OOZIE-3083  Make improved version Info backward compatible
https://issues.apache.org/jira/browse/OOZIE-3083
OOZIE-3071  Oozie 4.3 Spark sharelib ueses a different version of commons-lang3 
than Spark 2.2.0
https://issues.apache.org/jira/browse/OOZIE-3071
OOZIE-3063  Sanitizing variables that are part of openjpa.ConnectionProperties
https://issues.apache.org/jira/browse/OOZIE-3063
OOZIE-3062  Set HADOOP_CONF_DIR for spark action
https://issues.apache.org/jira/browse/OOZIE-3062
OOZIE-3022  fix for warning has no file and won't be listed in dependency files 
details
https://issues.apache.org/jira/browse/OOZIE-3022
OOZIE-3002  address findbugs errors in client lib
https://issues.apache.org/jira/browse/OOZIE-3002
OOZIE-3001  core library has many instances of warnings with trailing spaces 
and lines longer than 132 chars
https://issues.apache.org/jira/browse/OOZIE-3001
OOZIE-2997  files contain trailing white spaces in client lib
https://issues.apache.org/jira/browse/OOZIE-2997
OOZIE-2996  add option for -UseGCOverheadLimit to maven opts as sometimes local 
testing fails
https://issues.apache.org/jira/browse/OOZIE-2996
OOZIE-2975  code clean up in pig sharelib, replace Exception with more 
explicit, add try with resources, StringBuilder instead of StringBuffer
https://issues.apache.org/jira/browse/OOZIE-2975
OOZIE-2969  Drop support for Java 1.7
https://issues.apache.org/jira/browse/OOZIE-2969
OOZIE-2964  Add -Xdoclint:none to javadoc opts to avoid warnings
https://issues.apache.org/jira/browse/OOZIE-2964
OOZIE-2962  bump maven-javadoc-plugin to 2.10.4
https://issues.apache.org/jira/browse/OOZIE-2962
OOZIE-2957  Documentation states that starting a coordinator is possible
https://issues.apache.org/jira/browse/OOZIE-2957
OOZIE-2956  Fix Findbugs warnings related to reliance on default encoding in 
oozie-core
https://issues.apache.org/jira/browse/OOZIE-2956
OOZIE-2955  Fix Findbugs warnings related to reliance on default encoding in 
oozie-client
https://issues.apache.org/jira/browse/OOZIE-2955
OOZIE-2954  Fix Checkstyle issues in oozie-client
https://issues.apache.org/jira/browse/OOZIE-2954
OOZIE-2953  Fix Checkstyle issues in oozie-tools
https://issues.apache.org/jira/browse/OOZIE-2953
OOZIE-2952  Fix Findbugs warnings in oozie-sharelib-oozie
https://issues.apache.org/jira/browse/OOZIE-2952
OOZIE-2949  Escape quotes whitespaces in Sqoop  field
https://issues.apache.org/jira/browse/OOZIE-2949
OOZIE-2942  Fix Findbugs warnings in oozie-examples
https://issues.apache.org/jira/browse/OOZIE-2942
OOZIE-2937  Remove redundant groupId from the child pom's
https://issues.apache.org/jira/browse/OOZIE-2937
OOZIE-2934  Fix "Exceptional return value of java.io.File.mkdirs() ignored" 
Findbugs error in oozie-sharelib-spark
https://issues.apache.org/jira/browse/OOZIE-2934
OOZIE-2927  Append new line character for Hive2 query using query tag
https://issues.apache.org/jira/browse/OOZIE-2927
OOZIE-2914  Consolidate Trim 
https://issues.apache.org/jira/browse/OOZIE-2914
OOZIE-2896  Ensure compatibility for existing LauncherMapper settings
https://issues.apache.org/jira/browse/OOZIE-2896
OOZIE-2883  OOZIE throw the error "Missing 
[oozie.service.ProxyUserService.proxyuser.oozie.service.ProxyUserService.proxyuser.mr.groups]
 property"
https://issues.apache.org/jira/browse/OOZIE-2883
OOZIE-2877  Oozie Git Action
https://issues.apache.org/jira/browse/OOZIE-2877
OOZIE-2867  Timezone handling for Coordinators: emphasize "Continent/City" 
format
https://issues.apache.org/jira/browse/OOZIE-2867
OOZIE-2834  ParameterVerifier logging non-useful warning for workflow definition
https://issues.apache.org/jira/browse/OOZIE-2834
OOZIE-2833  when using uber mode the regex pattern used in the 
extractHeapSizeMB method does not allow heap sizes specified in bytes.
https://issues.apache.org/jira/browse/OOZIE-2833
OOZIE-2829  Improve sharelib upload to accept multiple source folders
https://issues.apache.org/jira/browse/OOZIE-2829
OOZIE-2826  Falcon feed fails to aws s3; Oozie joda time version does not meet 
required jar version 2.2 or later
https://issues.apache.org/jira/browse/OOZIE-2826
OOZIE-2812  SparkConfigurationService should support loading configurations 
from multiple Spark versions
https://issues.apache.org/jira/br

[jira] [Commented] (OOZIE-3094) fix for grammar mistake

2017-10-19 Thread Hadoop QA (JIRA) 

[ 
https://issues.apache.org/jira/browse/OOZIE-3094?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16212193#comment-16212193
 ] 

Hadoop QA commented on OOZIE-3094:
--

Testing JIRA OOZIE-3094

Cleaning local git workspace



{color:green}+1 PATCH_APPLIES{color}
{color:green}+1 CLEAN{color}
{color:red}-1 RAW_PATCH_ANALYSIS{color}
.{color:green}+1{color} the patch does not introduce any @author tags
.{color:green}+1{color} the patch does not introduce any tabs
.{color:green}+1{color} the patch does not introduce any trailing spaces
.{color:green}+1{color} the patch does not introduce any line longer than 
132
.{color:red}-1{color} the patch does not add/modify any testcase
{color:green}+1 RAT{color}
.{color:green}+1{color} the patch does not seem to introduce new RAT 
warnings
{color:green}+1 JAVADOC{color}
.{color:green}+1{color} the patch does not seem to introduce new Javadoc 
warnings
.{color:red}WARNING{color}: the current HEAD has 77 Javadoc warning(s)
{color:green}+1 COMPILE{color}
.{color:green}+1{color} HEAD compiles
.{color:green}+1{color} patch compiles
.{color:green}+1{color} the patch does not seem to introduce new javac 
warnings
{color:green}+1{color} There are no new bugs found in total.
. {color:green}+1{color} There are no new bugs found in [core].
. {color:green}+1{color} There are no new bugs found in [docs].
. {color:green}+1{color} There are no new bugs found in [sharelib/hive2].
. {color:green}+1{color} There are no new bugs found in [sharelib/streaming].
. {color:green}+1{color} There are no new bugs found in [sharelib/oozie].
. {color:green}+1{color} There are no new bugs found in [sharelib/spark].
. {color:green}+1{color} There are no new bugs found in [sharelib/sqoop].
. {color:green}+1{color} There are no new bugs found in [sharelib/hive].
. {color:green}+1{color} There are no new bugs found in [sharelib/distcp].
. {color:green}+1{color} There are no new bugs found in [sharelib/pig].
. {color:green}+1{color} There are no new bugs found in [sharelib/hcatalog].
. {color:green}+1{color} There are no new bugs found in [server].
. {color:green}+1{color} There are no new bugs found in [client].
. {color:green}+1{color} There are no new bugs found in [tools].
. {color:green}+1{color} There are no new bugs found in [examples].
{color:green}+1 BACKWARDS_COMPATIBILITY{color}
.{color:green}+1{color} the patch does not change any JPA 
Entity/Colum/Basic/Lob/Transient annotations
.{color:green}+1{color} the patch does not modify JPA files
{color:red}-1 TESTS{color} - patch does not compile, cannot run testcases
{color:green}+1 DISTRO{color}
.{color:green}+1{color} distro tarball builds with the patch 


{color:red}*-1 Overall result, please check the reported -1(s)*{color}

{color:red}. There is at least one warning, please check{color}

The full output of the test-patch run is available at

. https://builds.apache.org/job/PreCommit-OOZIE-Build/123/

> fix for grammar mistake
> ---
>
> Key: OOZIE-3094
> URL: https://issues.apache.org/jira/browse/OOZIE-3094
> Project: Oozie
>  Issue Type: Improvement
>  Components: docs
>Affects Versions: 4.3.0
>Reporter: Artem Ervits
>Assignee: Artem Ervits
>Priority: Trivial
> Fix For: 5.0.0b1
>
> Attachments: OOZIE-3094.patch
>
>
> HBase does not require any additional properties since the hbase-site.xml on 
> the Oozie server provides necessary information to the obtain delegation 
> token; though properties can be overwritten here if desired. 
> https://oozie.apache.org/docs/4.3.0/DG_ActionAuthentication.html



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

Failed: OOZIE-3094 PreCommit Build #123

2017-10-19 Thread Apache Jenkins Server 
Jira: https://issues.apache.org/jira/browse/OOZIE-3094
Build: https://builds.apache.org/job/PreCommit-OOZIE-Build/123/

###
## LAST 60 LINES OF THE CONSOLE 
###
[...truncated 1.53 MB...]
+1 the patch does not introduce any line longer than 132
-1 the patch does not add/modify any testcase
+1 RAT
+1 the patch does not seem to introduce new RAT warnings
+1 JAVADOC
+1 the patch does not seem to introduce new Javadoc warnings
WARNING: the current HEAD has 77 Javadoc warning(s)
+1 COMPILE
+1 HEAD compiles
+1 patch compiles
+1 the patch does not seem to introduce new javac warnings
+1 There are no new bugs found in total.
 +1 There are no new bugs found in [core].
 +1 There are no new bugs found in [docs].
 +1 There are no new bugs found in [sharelib/hive2].
 +1 There are no new bugs found in [sharelib/streaming].
 +1 There are no new bugs found in [sharelib/oozie].
 +1 There are no new bugs found in [sharelib/spark].
 +1 There are no new bugs found in [sharelib/sqoop].
 +1 There are no new bugs found in [sharelib/hive].
 +1 There are no new bugs found in [sharelib/distcp].
 +1 There are no new bugs found in [sharelib/pig].
 +1 There are no new bugs found in [sharelib/hcatalog].
 +1 There are no new bugs found in [server].
 +1 There are no new bugs found in [client].
 +1 There are no new bugs found in [tools].
 +1 There are no new bugs found in [examples].
+1 BACKWARDS_COMPATIBILITY
+1 the patch does not change any JPA Entity/Colum/Basic/Lob/Transient 
annotations
+1 the patch does not modify JPA files
-1 TESTS - patch does not compile, cannot run testcases
+1 DISTRO
+1 distro tarball builds with the patch 


-1 Overall result, please check the reported -1(s)

 There is at least one warning, please check

The full output of the test-patch run is available at

 https://builds.apache.org/job/PreCommit-OOZIE-Build/123/

  % Total% Received % Xferd  Average Speed   TimeTime Time  Current
 Dload  Upload   Total   SpentLeft  Speed
  0 00 00 0  0  0 --:--:-- --:--:-- --:--:-- 0  
0 00 00 0  0  0 --:--:-- --:--:-- --:--:-- 0
  0 00 00 0  0  0 --:--:-- --:--:-- --:--:-- 
0100 3706k  100 3706k0 0  3356k  0  0:00:01  0:00:01 --:--:-- 6775k
Adding comment to JIRA
Comment added.

test-patch exit code: 1

Build step 'Execute shell' marked build as failure
[description-setter] Description set: OOZIE-3094
Archiving artifacts
[Fast Archiver] Compressed 1.80 MB of artifacts by 64.3% relative to #122
Recording test results
Email was triggered for: Failure - Any
Sending email for trigger: Failure - Any



###
## FAILED TESTS (if any) 
##
All tests passed

[jira] [Updated] (OOZIE-3094) fix for grammar mistake

2017-10-19 Thread Artem Ervits (JIRA) 

 [ 
https://issues.apache.org/jira/browse/OOZIE-3094?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Artem Ervits updated OOZIE-3094:

Attachment: OOZIE-3094.patch

> fix for grammar mistake
> ---
>
> Key: OOZIE-3094
> URL: https://issues.apache.org/jira/browse/OOZIE-3094
> Project: Oozie
>  Issue Type: Improvement
>  Components: docs
>Affects Versions: 4.3.0
>Reporter: Artem Ervits
>Assignee: Artem Ervits
>Priority: Trivial
> Fix For: 5.0.0b1
>
> Attachments: OOZIE-3094.patch
>
>
> HBase does not require any additional properties since the hbase-site.xml on 
> the Oozie server provides necessary information to the obtain delegation 
> token; though properties can be overwritten here if desired. 
> https://oozie.apache.org/docs/4.3.0/DG_ActionAuthentication.html



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Updated] (OOZIE-3094) fix for grammar mistake

2017-10-19 Thread Artem Ervits (JIRA) 

 [ 
https://issues.apache.org/jira/browse/OOZIE-3094?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Artem Ervits updated OOZIE-3094:

Attachment: (was: OOZIE-3094.patch)

> fix for grammar mistake
> ---
>
> Key: OOZIE-3094
> URL: https://issues.apache.org/jira/browse/OOZIE-3094
> Project: Oozie
>  Issue Type: Improvement
>  Components: docs
>Affects Versions: 4.3.0
>Reporter: Artem Ervits
>Assignee: Artem Ervits
>Priority: Trivial
> Fix For: 5.0.0b1
>
> Attachments: OOZIE-3094.patch
>
>
> HBase does not require any additional properties since the hbase-site.xml on 
> the Oozie server provides necessary information to the obtain delegation 
> token; though properties can be overwritten here if desired. 
> https://oozie.apache.org/docs/4.3.0/DG_ActionAuthentication.html



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Updated] (OOZIE-3094) fix for grammar mistake

2017-10-19 Thread Artem Ervits (JIRA) 

 [ 
https://issues.apache.org/jira/browse/OOZIE-3094?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Artem Ervits updated OOZIE-3094:

Attachment: OOZIE-3094.patch

> fix for grammar mistake
> ---
>
> Key: OOZIE-3094
> URL: https://issues.apache.org/jira/browse/OOZIE-3094
> Project: Oozie
>  Issue Type: Improvement
>  Components: docs
>Affects Versions: 4.3.0
>Reporter: Artem Ervits
>Assignee: Artem Ervits
>Priority: Trivial
> Fix For: 5.0.0b1
>
> Attachments: OOZIE-3094.patch
>
>
> HBase does not require any additional properties since the hbase-site.xml on 
> the Oozie server provides necessary information to the obtain delegation 
> token; though properties can be overwritten here if desired. 
> https://oozie.apache.org/docs/4.3.0/DG_ActionAuthentication.html



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Created] (OOZIE-3094) fix for grammar mistake

2017-10-19 Thread Artem Ervits (JIRA) 
Artem Ervits created OOZIE-3094:
---

 Summary: fix for grammar mistake
 Key: OOZIE-3094
 URL: https://issues.apache.org/jira/browse/OOZIE-3094
 Project: Oozie
  Issue Type: Improvement
  Components: docs
Affects Versions: 4.3.0
Reporter: Artem Ervits
Assignee: Artem Ervits
Priority: Trivial
 Fix For: 5.0.0b1


HBase does not require any additional properties since the hbase-site.xml on 
the Oozie server provides necessary information to the obtain delegation token; 
though properties can be overwritten here if desired. 
https://oozie.apache.org/docs/4.3.0/DG_ActionAuthentication.html



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Created] (OOZIE-3093) Test Scenarios for Oozie 5.0

2017-10-19 Thread Artem Ervits (JIRA) 
Artem Ervits created OOZIE-3093:
---

 Summary: Test Scenarios for Oozie 5.0
 Key: OOZIE-3093
 URL: https://issues.apache.org/jira/browse/OOZIE-3093
 Project: Oozie
  Issue Type: Test
Affects Versions: 5.0.0b1
Reporter: Artem Ervits
Priority: Blocker


This is umbrella jira to track test scenarios on Oozie 5.0. In my discussion 
with [~gezapeti] we can have a joint effort to test different test cases. 
[~clayb] please provide your input as well.

1. Oozie HA
2. Oozie secure cluster
3. Run all examples
4. Execute against every Hadoop profile
etc.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Updated] (OOZIE-2896) Ensure compatibility for existing LauncherMapper settings

2017-10-19 Thread Robert Kanter (JIRA) 

 [ 
https://issues.apache.org/jira/browse/OOZIE-2896?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Robert Kanter updated OOZIE-2896:
-
Fix Version/s: (was: 5.0.0)
   5.0.0b1

> Ensure compatibility for existing LauncherMapper settings
> -
>
> Key: OOZIE-2896
> URL: https://issues.apache.org/jira/browse/OOZIE-2896
> Project: Oozie
>  Issue Type: Sub-task
>Affects Versions: 5.0.0
>Reporter: Peter Bacsko
>Assignee: Andras Piros
>Priority: Blocker
> Fix For: 5.0.0b1
>
> Attachments: OOZIE-2896.001.patch, OOZIE-2896.002.patch, 
> OOZIE-2896.002.patch, OOZIE-2896.003.patch, OOZIE-2896.004.patch, 
> OOZIE-2896.005.patch, OOZIE-2896.006.patch
>
>
> There are a couple of LauncherMapper settings that we should support for 
> backward compatibility reasons.
> For example, {{oozie.launcher.mapreduce.map.memory.mb}}, 
> {{mapred.child.env}}, {{mapred.map.env}}, etc.
> We also have to discuss the list of properties we're going to support, since 
> there are many of them.
> List of hadoop properties under this task:
> mapreduce.job.queuename
> mapreduce.map.memory.mb
> mapreduce.map.java.opts
> mapreduce.map.env
> mapreduce.job.priority
> yarn.app.mapreduce.am.scheduler.heartbeat.interval-ms
> yarn.app.mapreduce.am.resource.mb
> yarn.app.mapreduce.am.command-opts
> yarn.app.mapreduce.am.env
> ACL related properties should go under OOZIE-2897:
> mapreduce.job.acl-view-job
> mapreduce.job.acl-modfy-job



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Updated] (OOZIE-2973) Make sure Oozie works with Hadoop 3

2017-10-19 Thread Robert Kanter (JIRA) 

 [ 
https://issues.apache.org/jira/browse/OOZIE-2973?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Robert Kanter updated OOZIE-2973:
-
Fix Version/s: 5.0.0b1

> Make sure Oozie works with Hadoop 3 
> 
>
> Key: OOZIE-2973
> URL: https://issues.apache.org/jira/browse/OOZIE-2973
> Project: Oozie
>  Issue Type: Bug
>Reporter: Attila Sasvari
> Fix For: 5.0.0b1
>
>
> This JIRA is to track that Oozie works with Hadoop 3. 
> - Build Oozie with latest hadoop 3 (e.g. assuming Apache Hadoop 3.0.0-alpha3 
> pass {{-D-Dhadoop.version=3.0.0-alpha3}} to maven) and execute tests
> - It would be nice to create an Apache jenkins job that runs Oozie tests with 
> hadoop 3 (also with other component) 



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Updated] (OOZIE-2973) Make sure Oozie works with Hadoop 3

2017-10-19 Thread Robert Kanter (JIRA) 

 [ 
https://issues.apache.org/jira/browse/OOZIE-2973?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Robert Kanter updated OOZIE-2973:
-
Priority: Blocker  (was: Major)

> Make sure Oozie works with Hadoop 3 
> 
>
> Key: OOZIE-2973
> URL: https://issues.apache.org/jira/browse/OOZIE-2973
> Project: Oozie
>  Issue Type: Bug
>Reporter: Attila Sasvari
>Priority: Blocker
> Fix For: 5.0.0b1
>
>
> This JIRA is to track that Oozie works with Hadoop 3. 
> - Build Oozie with latest hadoop 3 (e.g. assuming Apache Hadoop 3.0.0-alpha3 
> pass {{-D-Dhadoop.version=3.0.0-alpha3}} to maven) and execute tests
> - It would be nice to create an Apache jenkins job that runs Oozie tests with 
> hadoop 3 (also with other component) 



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

Re: Welcome new Oozie Committer - Andras Piros

2017-10-19 Thread Attila Sasvari 
Congratulations Andras! Keep up the good work.

2017. okt. 16. du. 8:44 ezt írta ("Robert Kanter" ):

Hi everyone,

It is my pleasure to announce that the Oozie PMC has invited
Andras Piros to become an Oozie committer and he has
accepted our invitation.

Please join me in congratulating him.
Congrats!


- Robert, on behalf of the Oozie PMC

Re: Review Request 62936: OOZIE-2896 Ensure compatibility for existing LauncherMapper settings

2017-10-19 Thread Peter Bacsko via Review Board 

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/62936/#review188680
---




core/src/main/java/org/apache/oozie/action/hadoop/JavaActionExecutor.java
Lines 133 (patched)


Any plans to add this to the XML schema? Or it's not that useful?



core/src/main/java/org/apache/oozie/action/hadoop/LauncherConfigurationInjector.java
Lines 83-99 (patched)


Now that's what I'm talkin' about! :D



core/src/main/java/org/apache/oozie/action/hadoop/LauncherConfigurationInjector.java
Lines 224-241 (patched)


Good comment again.

Perhaps it would be worthwile to point out that 
"oozie.launcher.override.XX" will be "oozie.launcher.XX" just to be absolutely 
clear.



core/src/test/java/org/apache/oozie/action/hadoop/TestJavaActionExecutor.java
Lines 2564 (patched)


Add assertion failure message


- Peter Bacsko


On okt. 19, 2017, 10:04 de, András Piros wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/62936/
> ---
> 
> (Updated okt. 19, 2017, 10:04 de)
> 
> 
> Review request for oozie, Attila Sasvari, Peter Cseh, and Peter Bacsko.
> 
> 
> Repository: oozie-git
> 
> 
> Description
> ---
> 
> OOZIE-2896 Ensure compatibility for existing LauncherMapper settings
> 
> 
> Diffs
> -
> 
>   core/src/main/java/org/apache/oozie/action/hadoop/JavaActionExecutor.java 
> 76d0daa4f7528de6fb3c8993bae4dd28b6f7865b 
>   
> core/src/main/java/org/apache/oozie/action/hadoop/LauncherConfigurationInjector.java
>  PRE-CREATION 
>   core/src/main/java/org/apache/oozie/service/ConfigurationService.java 
> a51933022c14ea4ce06a65cc3c123bed8a8f7f34 
>   core/src/main/resources/oozie-default.xml 
> 9ba8fd4a0d9dd5f0bd6a64af3724c2f910d86457 
>   
> core/src/test/java/org/apache/oozie/action/hadoop/TestJavaActionExecutor.java 
> a7bd357f8938d148a150db43135a7a8fe7f0da4c 
>   
> core/src/test/java/org/apache/oozie/action/hadoop/TestLauncherConfigurationInjector.java
>  PRE-CREATION 
>   sharelib/oozie/src/main/java/org/apache/oozie/action/hadoop/LauncherAM.java 
> ee1a32ac9b071009e1dbe78396520d778f58bddc 
> 
> 
> Diff: https://reviews.apache.org/r/62936/diff/4/
> 
> 
> Testing
> ---
> 
> JUnit tests:
> 
> * `TestJavaActionExecutor`
> * `TestLauncherConfigurationFilter`
> 
> Functional tests: submit and run `examples/apps/java-main/workflow.xml` in 
> three modes:
> 
> * only `oozie.launcher.` parameters are present: `oozie.launcher.` parameters 
> are passed through
> * `oozie.launcher.` and override (`yarn.`, `mapreduce.`, and `mapred.`) 
> parameters are also present: `oozie.launcher.` parameters are passed through
> * only override (`yarn.`, `mapreduce.`, and `mapred.`) parameters are 
> present: override parameters are passed through
> 
> 
> Thanks,
> 
> András Piros
> 
>

Re: Review Request 62936: OOZIE-2896 Ensure compatibility for existing LauncherMapper settings

2017-10-19 Thread Peter Bacsko via Review Board 

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/62936/#review188678
---




core/src/test/java/org/apache/oozie/action/hadoop/TestLauncherConfigurationInjector.java
Lines 26 (patched)


There are a lot of hard-coded stuff in this class, especially test data.

I know that we don't import the constant fields which define properties, 
but I think at least the test-specific stuff could be extracted to final 
fields, especially if it's repeated.



core/src/test/java/org/apache/oozie/action/hadoop/TestLauncherConfigurationInjector.java
Lines 130 (patched)


Are you intentionally using "true" here, and "false" above?



core/src/test/java/org/apache/oozie/action/hadoop/TestLauncherConfigurationInjector.java
Lines 132 (patched)


Instead of Integer.toString(4), wouldn't it be easier just to simply type 
"4"? Does this have an advantage?


- Peter Bacsko


On okt. 19, 2017, 10:04 de, András Piros wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/62936/
> ---
> 
> (Updated okt. 19, 2017, 10:04 de)
> 
> 
> Review request for oozie, Attila Sasvari, Peter Cseh, and Peter Bacsko.
> 
> 
> Repository: oozie-git
> 
> 
> Description
> ---
> 
> OOZIE-2896 Ensure compatibility for existing LauncherMapper settings
> 
> 
> Diffs
> -
> 
>   core/src/main/java/org/apache/oozie/action/hadoop/JavaActionExecutor.java 
> 76d0daa4f7528de6fb3c8993bae4dd28b6f7865b 
>   
> core/src/main/java/org/apache/oozie/action/hadoop/LauncherConfigurationInjector.java
>  PRE-CREATION 
>   core/src/main/java/org/apache/oozie/service/ConfigurationService.java 
> a51933022c14ea4ce06a65cc3c123bed8a8f7f34 
>   core/src/main/resources/oozie-default.xml 
> 9ba8fd4a0d9dd5f0bd6a64af3724c2f910d86457 
>   
> core/src/test/java/org/apache/oozie/action/hadoop/TestJavaActionExecutor.java 
> a7bd357f8938d148a150db43135a7a8fe7f0da4c 
>   
> core/src/test/java/org/apache/oozie/action/hadoop/TestLauncherConfigurationInjector.java
>  PRE-CREATION 
>   sharelib/oozie/src/main/java/org/apache/oozie/action/hadoop/LauncherAM.java 
> ee1a32ac9b071009e1dbe78396520d778f58bddc 
> 
> 
> Diff: https://reviews.apache.org/r/62936/diff/4/
> 
> 
> Testing
> ---
> 
> JUnit tests:
> 
> * `TestJavaActionExecutor`
> * `TestLauncherConfigurationFilter`
> 
> Functional tests: submit and run `examples/apps/java-main/workflow.xml` in 
> three modes:
> 
> * only `oozie.launcher.` parameters are present: `oozie.launcher.` parameters 
> are passed through
> * `oozie.launcher.` and override (`yarn.`, `mapreduce.`, and `mapred.`) 
> parameters are also present: `oozie.launcher.` parameters are passed through
> * only override (`yarn.`, `mapreduce.`, and `mapred.`) parameters are 
> present: override parameters are passed through
> 
> 
> Thanks,
> 
> András Piros
> 
>

[jira] [Updated] (OOZIE-3092) Unit tests are timing out

2017-10-19 Thread Andras Piros (JIRA) 

 [ 
https://issues.apache.org/jira/browse/OOZIE-3092?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Andras Piros updated OOZIE-3092:

Attachment: deploy-mvn-artifacts.sh

Deploy from local Maven repo to {{dist_test}} cache dir.

> Unit tests are timing out
> -
>
> Key: OOZIE-3092
> URL: https://issues.apache.org/jira/browse/OOZIE-3092
> Project: Oozie
>  Issue Type: Bug
>  Components: tests
>Affects Versions: 5.0.0
>Reporter: Andras Piros
>Assignee: Andras Piros
> Fix For: 5.0.0
>
> Attachments: deploy-mvn-artifacts.sh
>
>
> Here is what I've found so far:
> {noformat}
> TIMING OUT
> --
> http://dist-test.cloudera.org/job?job_id=oozie.forsage.1508330800.52107
> commit 430e51070fe3b0fd2ed919318eff7783684fbe35
> Author: satishsaley 
> Date:   Tue Oct 10 09:32:59 2017 -0700
> OOZIE-3077 SLA documentation fixes (satishsaley)
> http://dist-test.cloudera.org/job?job_id=oozie.forsage.1508332800.52476
> commit 660b3151ce1b808e922d708f14b460a715fa6a33
> Author: satishsaley 
> Date:   Mon Oct 9 09:57:37 2017 -0700
> OOZIE-3079 Filtering coordinators returns bundle id as null (satishsaley)
> http://dist-test.cloudera.org/job?job_id=oozie.forsage.1508328464.51669
> commit be293c046d7d83765c65140921c3de5ca3d7b55a
> Author: Attila Sasvari 
> Date:   Mon Oct 9 16:55:44 2017 +0200
> OOZIE-3078 PasswordMasker throws NPE with null arguments (asasvari)
> http://dist-test.cloudera.org/job?job_id=oozie.forsage.1508397233.53944
> commit ceefa2f6eba5ddc1fe68af427f504f863b249b6d
> Author: Gezapeti Cseh 
> Date:   Fri Sep 22 13:35:53 2017 +0200
> OOZIE-2909 amend Fix license headers (andras.piros via gezapeti)
> http://dist-test.cloudera.org/job?job_id=oozie.forsage.1508398792.54406
> commit 71d3ddc31cfb983dc83c51882f99c4429253230a
> Author: Gezapeti Cseh 
> Date:   Mon Sep 4 10:27:27 2017 +0200
> OOZIE-2858 HiveMain, ShellMain and SparkMain should not overwrite 
> properties and config files locally (gezapeti)
> http://dist-test.cloudera.org/job?job_id=oozie.forsage.1508408287.56112
> commit 9dee9d572ac6058b1730962352c1f1a63e1c2114
> Author: Gezapeti Cseh 
> Date:   Tue Aug 1 10:41:45 2017 +0200
> OOZIE-2960 Fix warnings for @return tag has no argument (Artem Ervits via 
> gezapeti)
> {noformat}



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Created] (OOZIE-3092) Unit tests are timing out

2017-10-19 Thread Andras Piros (JIRA) 
Andras Piros created OOZIE-3092:
---

 Summary: Unit tests are timing out
 Key: OOZIE-3092
 URL: https://issues.apache.org/jira/browse/OOZIE-3092
 Project: Oozie
  Issue Type: Bug
  Components: tests
Affects Versions: 5.0.0
Reporter: Andras Piros
Assignee: Andras Piros
 Fix For: 5.0.0


Here is what I've found so far:

{noformat}
TIMING OUT
--
http://dist-test.cloudera.org/job?job_id=oozie.forsage.1508330800.52107
commit 430e51070fe3b0fd2ed919318eff7783684fbe35
Author: satishsaley 
Date:   Tue Oct 10 09:32:59 2017 -0700

OOZIE-3077 SLA documentation fixes (satishsaley)

http://dist-test.cloudera.org/job?job_id=oozie.forsage.1508332800.52476
commit 660b3151ce1b808e922d708f14b460a715fa6a33
Author: satishsaley 
Date:   Mon Oct 9 09:57:37 2017 -0700

OOZIE-3079 Filtering coordinators returns bundle id as null (satishsaley)

http://dist-test.cloudera.org/job?job_id=oozie.forsage.1508328464.51669
commit be293c046d7d83765c65140921c3de5ca3d7b55a
Author: Attila Sasvari 
Date:   Mon Oct 9 16:55:44 2017 +0200

OOZIE-3078 PasswordMasker throws NPE with null arguments (asasvari)

http://dist-test.cloudera.org/job?job_id=oozie.forsage.1508397233.53944
commit ceefa2f6eba5ddc1fe68af427f504f863b249b6d
Author: Gezapeti Cseh 
Date:   Fri Sep 22 13:35:53 2017 +0200

OOZIE-2909 amend Fix license headers (andras.piros via gezapeti)

http://dist-test.cloudera.org/job?job_id=oozie.forsage.1508398792.54406
commit 71d3ddc31cfb983dc83c51882f99c4429253230a
Author: Gezapeti Cseh 
Date:   Mon Sep 4 10:27:27 2017 +0200

OOZIE-2858 HiveMain, ShellMain and SparkMain should not overwrite 
properties and config files locally (gezapeti)

http://dist-test.cloudera.org/job?job_id=oozie.forsage.1508408287.56112
commit 9dee9d572ac6058b1730962352c1f1a63e1c2114
Author: Gezapeti Cseh 
Date:   Tue Aug 1 10:41:45 2017 +0200

OOZIE-2960 Fix warnings for @return tag has no argument (Artem Ervits via 
gezapeti)
{noformat}



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (OOZIE-2896) Ensure compatibility for existing LauncherMapper settings

2017-10-19 Thread Hadoop QA (JIRA) 

[ 
https://issues.apache.org/jira/browse/OOZIE-2896?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16210917#comment-16210917
 ] 

Hadoop QA commented on OOZIE-2896:
--

Testing JIRA OOZIE-2896

Cleaning local git workspace



{color:green}+1 PATCH_APPLIES{color}
{color:green}+1 CLEAN{color}
{color:green}+1 RAW_PATCH_ANALYSIS{color}
.{color:green}+1{color} the patch does not introduce any @author tags
.{color:green}+1{color} the patch does not introduce any tabs
.{color:green}+1{color} the patch does not introduce any trailing spaces
.{color:green}+1{color} the patch does not introduce any line longer than 
132
.{color:green}+1{color} the patch does adds/modifies 2 testcase(s)
{color:green}+1 RAT{color}
.{color:green}+1{color} the patch does not seem to introduce new RAT 
warnings
{color:green}+1 JAVADOC{color}
.{color:green}+1{color} the patch does not seem to introduce new Javadoc 
warnings
.{color:red}WARNING{color}: the current HEAD has 77 Javadoc warning(s)
{color:green}+1 COMPILE{color}
.{color:green}+1{color} HEAD compiles
.{color:green}+1{color} patch compiles
.{color:green}+1{color} the patch does not seem to introduce new javac 
warnings
{color:green}+1{color} There are no new bugs found in total.
. {color:green}+1{color} There are no new bugs found in [core].
. {color:green}+1{color} There are no new bugs found in [server].
. {color:green}+1{color} There are no new bugs found in [client].
. {color:green}+1{color} There are no new bugs found in [docs].
. {color:green}+1{color} There are no new bugs found in [examples].
. {color:green}+1{color} There are no new bugs found in [sharelib/hive].
. {color:green}+1{color} There are no new bugs found in [sharelib/pig].
. {color:green}+1{color} There are no new bugs found in [sharelib/distcp].
. {color:green}+1{color} There are no new bugs found in [sharelib/streaming].
. {color:green}+1{color} There are no new bugs found in [sharelib/spark].
. {color:green}+1{color} There are no new bugs found in [sharelib/hcatalog].
. {color:green}+1{color} There are no new bugs found in [sharelib/sqoop].
. {color:green}+1{color} There are no new bugs found in [sharelib/hive2].
. {color:green}+1{color} There are no new bugs found in [sharelib/oozie].
. {color:green}+1{color} There are no new bugs found in [tools].
{color:green}+1 BACKWARDS_COMPATIBILITY{color}
.{color:green}+1{color} the patch does not change any JPA 
Entity/Colum/Basic/Lob/Transient annotations
.{color:green}+1{color} the patch does not modify JPA files
{color:green}+1 TESTS{color}
.Tests run: 1829
.Tests rerun: 78
.Tests failed at first run: 
org.apache.oozie.servlet.TestV1JobServlet,org.apache.oozie.util.TestInstrumentation,org.apache.oozie.command.bundle.TestBundleRerunXCommand,org.apache.oozie.servlet.TestV2SLAServlet,org.apache.oozie.jms.TestDefaultConnectionContext,org.apache.oozie.TestDagEngine,org.apache.oozie.jms.TestJMSJobEventListener,org.apache.oozie.compression.TestCodecFactory,org.apache.oozie.servlet.TestV2ValidateServlet,org.apache.oozie.command.coord.TestCoordActionsKillXCommand,org.apache.oozie.TestBundleJobBean,org.apache.oozie.jms.TestJMSSLAEventListener,
{color:green}+1 DISTRO{color}
.{color:green}+1{color} distro tarball builds with the patch 


{color:green}*+1 Overall result, good!, no -1s*{color}

{color:red}. There is at least one warning, please check{color}

The full output of the test-patch run is available at

. https://builds.apache.org/job/PreCommit-OOZIE-Build/122/

> Ensure compatibility for existing LauncherMapper settings
> -
>
> Key: OOZIE-2896
> URL: https://issues.apache.org/jira/browse/OOZIE-2896
> Project: Oozie
>  Issue Type: Sub-task
>Affects Versions: 5.0.0
>Reporter: Peter Bacsko
>Assignee: Andras Piros
>Priority: Blocker
> Fix For: 5.0.0
>
> Attachments: OOZIE-2896.001.patch, OOZIE-2896.002.patch, 
> OOZIE-2896.002.patch, OOZIE-2896.003.patch, OOZIE-2896.004.patch, 
> OOZIE-2896.005.patch, OOZIE-2896.006.patch
>
>
> There are a couple of LauncherMapper settings that we should support for 
> backward compatibility reasons.
> For example, {{oozie.launcher.mapreduce.map.memory.mb}}, 
> {{mapred.child.env}}, {{mapred.map.env}}, etc.
> We also have to discuss the list of properties we're going to support, since 
> there are many of them.
> List of hadoop properties under this task:
> mapreduce.job.queuename
> mapreduce.map.memory.mb
> mapreduce.map.java.opts
> mapreduce.map.env
> mapreduce.job.priority
> yarn.app.mapreduce.am.scheduler.heartbeat.interval-ms
> yarn.app.mapreduce.am.resource.mb
> yarn.app.mapreduce.am.command-opts
> yarn.app.mapreduce.am.env
> ACL related properties should go under OOZI

Success: OOZIE-2896 PreCommit Build #122

2017-10-19 Thread Apache Jenkins Server 
Jira: https://issues.apache.org/jira/browse/OOZIE-2896
Build: https://builds.apache.org/job/PreCommit-OOZIE-Build/122/

###
## LAST 60 LINES OF THE CONSOLE 
###
[...truncated 1.58 MB...]
+1 RAT
+1 the patch does not seem to introduce new RAT warnings
+1 JAVADOC
+1 the patch does not seem to introduce new Javadoc warnings
WARNING: the current HEAD has 77 Javadoc warning(s)
+1 COMPILE
+1 HEAD compiles
+1 patch compiles
+1 the patch does not seem to introduce new javac warnings
+1 There are no new bugs found in total.
 +1 There are no new bugs found in [core].
 +1 There are no new bugs found in [server].
 +1 There are no new bugs found in [client].
 +1 There are no new bugs found in [docs].
 +1 There are no new bugs found in [examples].
 +1 There are no new bugs found in [sharelib/hive].
 +1 There are no new bugs found in [sharelib/pig].
 +1 There are no new bugs found in [sharelib/distcp].
 +1 There are no new bugs found in [sharelib/streaming].
 +1 There are no new bugs found in [sharelib/spark].
 +1 There are no new bugs found in [sharelib/hcatalog].
 +1 There are no new bugs found in [sharelib/sqoop].
 +1 There are no new bugs found in [sharelib/hive2].
 +1 There are no new bugs found in [sharelib/oozie].
 +1 There are no new bugs found in [tools].
+1 BACKWARDS_COMPATIBILITY
+1 the patch does not change any JPA Entity/Colum/Basic/Lob/Transient 
annotations
+1 the patch does not modify JPA files
+1 TESTS
Tests run: 1829
Tests rerun: 78
Tests failed at first run: 
org.apache.oozie.servlet.TestV1JobServlet,org.apache.oozie.util.TestInstrumentation,org.apache.oozie.command.bundle.TestBundleRerunXCommand,org.apache.oozie.servlet.TestV2SLAServlet,org.apache.oozie.jms.TestDefaultConnectionContext,org.apache.oozie.TestDagEngine,org.apache.oozie.jms.TestJMSJobEventListener,org.apache.oozie.compression.TestCodecFactory,org.apache.oozie.servlet.TestV2ValidateServlet,org.apache.oozie.command.coord.TestCoordActionsKillXCommand,org.apache.oozie.TestBundleJobBean,org.apache.oozie.jms.TestJMSSLAEventListener,
+1 DISTRO
+1 distro tarball builds with the patch 


+1 Overall result, good!, no -1s

 There is at least one warning, please check

The full output of the test-patch run is available at

 https://builds.apache.org/job/PreCommit-OOZIE-Build/122/

  % Total% Received % Xferd  Average Speed   TimeTime Time  Current
 Dload  Upload   Total   SpentLeft  Speed
  0 00 00 0  0  0 --:--:-- --:--:-- --:--:-- 0  
0 00 00 0  0  0 --:--:-- --:--:-- --:--:-- 0
  0 00 00 0  0  0 --:--:-- --:--:-- --:--:-- 
0100 3706k  100 3706k0 0  3176k  0  0:00:01  0:00:01 --:--:-- 5147k
Adding comment to JIRA
Comment added.

test-patch exit code: 0

[description-setter] Description set: OOZIE-2896
Archiving artifacts
[Fast Archiver] Compressed 1.80 MB of artifacts by 36.4% relative to #108
Recording test results
Email was triggered for: Success
Sending email for trigger: Success



###
## FAILED TESTS (if any) 
##
All tests passed

[jira] [Created] (OOZIE-3091) Oozie Sqoop Avro Import fails with "java.lang.NoClassDefFoundError: org/apache/avro/mapred/AvroWrapper"

2017-10-19 Thread Prabhu Joseph (JIRA) 
Prabhu Joseph created OOZIE-3091:


 Summary: Oozie Sqoop Avro Import fails with 
"java.lang.NoClassDefFoundError: org/apache/avro/mapred/AvroWrapper"
 Key: OOZIE-3091
 URL: https://issues.apache.org/jira/browse/OOZIE-3091
 Project: Oozie
  Issue Type: Bug
  Components: action
Affects Versions: 4.2.0
Reporter: Prabhu Joseph


Oozie Sqoop Action which does Import as avro fails with below. 
avro-mapred-1.8.0-hadoop2.jar need to be included in Oozie Sqoop Sharelib

{code}

2017-10-19 09:45:25,349 WARN [main] org.apache.hadoop.mapred.YarnChild: 
Exception running child : java.lang.RuntimeException: 
java.lang.reflect.InvocationTargetException
at 
org.apache.hadoop.util.ReflectionUtils.newInstance(ReflectionUtils.java:134)
at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:745)
at org.apache.hadoop.mapred.MapTask.run(MapTask.java:341)
at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:170)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:422)
at 
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1866)
at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:164)
Caused by: java.lang.reflect.InvocationTargetException
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
at 
sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
at 
sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
at 
org.apache.hadoop.util.ReflectionUtils.newInstance(ReflectionUtils.java:132)
... 7 more
Caused by: java.lang.NoClassDefFoundError: org/apache/avro/mapred/AvroWrapper
at 
org.apache.sqoop.mapreduce.AvroImportMapper.(AvroImportMapper.java:43)
... 12 more
Caused by: java.lang.ClassNotFoundException: org.apache.avro.mapred.AvroWrapper
at java.net.URLClassLoader.findClass(URLClassLoader.java:381)
at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:331)
at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
... 13 more
{code}



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Updated] (OOZIE-2896) Ensure compatibility for existing LauncherMapper settings

2017-10-19 Thread Andras Piros (JIRA) 

 [ 
https://issues.apache.org/jira/browse/OOZIE-2896?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Andras Piros updated OOZIE-2896:

Attachment: OOZIE-2896.006.patch

More review comments.

> Ensure compatibility for existing LauncherMapper settings
> -
>
> Key: OOZIE-2896
> URL: https://issues.apache.org/jira/browse/OOZIE-2896
> Project: Oozie
>  Issue Type: Sub-task
>Affects Versions: 5.0.0
>Reporter: Peter Bacsko
>Assignee: Andras Piros
>Priority: Blocker
> Fix For: 5.0.0
>
> Attachments: OOZIE-2896.001.patch, OOZIE-2896.002.patch, 
> OOZIE-2896.002.patch, OOZIE-2896.003.patch, OOZIE-2896.004.patch, 
> OOZIE-2896.005.patch, OOZIE-2896.006.patch
>
>
> There are a couple of LauncherMapper settings that we should support for 
> backward compatibility reasons.
> For example, {{oozie.launcher.mapreduce.map.memory.mb}}, 
> {{mapred.child.env}}, {{mapred.map.env}}, etc.
> We also have to discuss the list of properties we're going to support, since 
> there are many of them.
> List of hadoop properties under this task:
> mapreduce.job.queuename
> mapreduce.map.memory.mb
> mapreduce.map.java.opts
> mapreduce.map.env
> mapreduce.job.priority
> yarn.app.mapreduce.am.scheduler.heartbeat.interval-ms
> yarn.app.mapreduce.am.resource.mb
> yarn.app.mapreduce.am.command-opts
> yarn.app.mapreduce.am.env
> ACL related properties should go under OOZIE-2897:
> mapreduce.job.acl-view-job
> mapreduce.job.acl-modfy-job



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

Re: Review Request 62936: OOZIE-2896 Ensure compatibility for existing LauncherMapper settings

2017-10-19 Thread András Piros via Review Board 

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/62936/
---

(Updated Oct. 19, 2017, 10:04 a.m.)


Review request for oozie, Attila Sasvari, Peter Cseh, and Peter Bacsko.


Changes
---

Updating more review comments.


Repository: oozie-git


Description
---

OOZIE-2896 Ensure compatibility for existing LauncherMapper settings


Diffs (updated)
-

  core/src/main/java/org/apache/oozie/action/hadoop/JavaActionExecutor.java 
76d0daa4f7528de6fb3c8993bae4dd28b6f7865b 
  
core/src/main/java/org/apache/oozie/action/hadoop/LauncherConfigurationInjector.java
 PRE-CREATION 
  core/src/main/java/org/apache/oozie/service/ConfigurationService.java 
a51933022c14ea4ce06a65cc3c123bed8a8f7f34 
  core/src/main/resources/oozie-default.xml 
9ba8fd4a0d9dd5f0bd6a64af3724c2f910d86457 
  core/src/test/java/org/apache/oozie/action/hadoop/TestJavaActionExecutor.java 
a7bd357f8938d148a150db43135a7a8fe7f0da4c 
  
core/src/test/java/org/apache/oozie/action/hadoop/TestLauncherConfigurationInjector.java
 PRE-CREATION 
  sharelib/oozie/src/main/java/org/apache/oozie/action/hadoop/LauncherAM.java 
ee1a32ac9b071009e1dbe78396520d778f58bddc 


Diff: https://reviews.apache.org/r/62936/diff/4/

Changes: https://reviews.apache.org/r/62936/diff/3-4/


Testing
---

JUnit tests:

* `TestJavaActionExecutor`
* `TestLauncherConfigurationFilter`

Functional tests: submit and run `examples/apps/java-main/workflow.xml` in 
three modes:

* only `oozie.launcher.` parameters are present: `oozie.launcher.` parameters 
are passed through
* `oozie.launcher.` and override (`yarn.`, `mapreduce.`, and `mapred.`) 
parameters are also present: `oozie.launcher.` parameters are passed through
* only override (`yarn.`, `mapreduce.`, and `mapred.`) parameters are present: 
override parameters are passed through


Thanks,

András Piros

[jira] [Updated] (OOZIE-2871) when Enable Kerberos, Oozie perform tasks throw “Client cannot authenticate via:[TOKEN, KERBEROS]”

2017-10-19 Thread Denes Bodo (JIRA) 

 [ 
https://issues.apache.org/jira/browse/OOZIE-2871?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Denes Bodo updated OOZIE-2871:
--
Attachment: secure_multicluster_distcp_workflow.xml

I've uploaded a workflow.xml file which can be used to reproduce the issue. 
Node and cluster names are masked.

In job.properties we set 'oozie.use.system.libpath=true'

> when Enable Kerberos,  Oozie perform tasks throw “Client cannot authenticate 
> via:[TOKEN, KERBEROS]”
> ---
>
> Key: OOZIE-2871
> URL: https://issues.apache.org/jira/browse/OOZIE-2871
> Project: Oozie
>  Issue Type: Bug
>  Components: security
>Affects Versions: 4.2.0
> Environment: Oozie version :4.2.0
> Hadoop version:2.7.2
> Both Oozie and Hadoop are enabled kerberos.
>Reporter: yangfang
>Priority: Critical
> Attachments: secure_multicluster_distcp_workflow.xml
>
>
> When  Oozie and Hadoop both enabled kerberos， I submitted a mapreduce   job 
> to oozie，then I got the error as below：
> 2017-04-27 13:37:12,677 WARN MapReduceActionExecutor: 523 - SERVER[zdh143] 
> USER[mr] GROUP[-] TOKEN[] APP[map-reduce-wf] 
> JOB[008-170427133546167-oozie-mr-W] 
> ACTION[008-170427133546167-oozie-mr-W@mr-node] Launcher exception: Failed 
> on local exception: java.io.IOException: 
> org.apache.hadoop.security.AccessControlException: Client cannot authenticate 
> via:[TOKEN, KERBEROS]; Host Details : local host is: "zdh142/10.43.183.142"; 
> destination host is: "zdh143":9000; 
> java.io.IOException: Failed on local exception: java.io.IOException: 
> org.apache.hadoop.security.AccessControlException: Client cannot authenticate 
> via:[TOKEN, KERBEROS]; Host Details : local host is: "zdh142/10.43.183.142"; 
> destination host is: "zdh143":9000; 
>   at org.apache.hadoop.net.NetUtils.wrapException(NetUtils.java:773)
>   at org.apache.hadoop.ipc.Client.call(Client.java:1479)
>   at org.apache.hadoop.ipc.Client.call(Client.java:1412)
>   at 
> org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:229)
>   at com.sun.proxy.$Proxy17.getFileInfo(Unknown Source)
>   at 
> org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolTranslatorPB.getFileInfo(ClientNamenodeProtocolTranslatorPB.java:771)
>   at sun.reflect.GeneratedMethodAccessor2.invoke(Unknown Source)
>   at 
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>   at java.lang.reflect.Method.invoke(Method.java:606)
>   at 
> org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:191)
>   at 
> org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:102)
>   at com.sun.proxy.$Proxy18.getFileInfo(Unknown Source)
>   at org.apache.hadoop.hdfs.DFSClient.getFileInfo(DFSClient.java:2108)
>   at 
> org.apache.hadoop.hdfs.DistributedFileSystem$22.doCall(DistributedFileSystem.java:1305)
>   at 
> org.apache.hadoop.hdfs.DistributedFileSystem$22.doCall(DistributedFileSystem.java:1301)
>   at 
> org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81)
>   at 
> org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1301)
>   at org.apache.hadoop.fs.FileSystem.exists(FileSystem.java:1426)
>   at 
> org.apache.hadoop.mapred.FileOutputFormat.checkOutputSpecs(FileOutputFormat.java:130)
>   at 
> org.apache.hadoop.mapreduce.JobSubmitter.checkSpecs(JobSubmitter.java:268)
>   at 
> org.apache.hadoop.mapreduce.JobSubmitter.submitJobInternal(JobSubmitter.java:139)
>   at org.apache.hadoop.mapreduce.Job$10.run(Job.java:1299)
>   at org.apache.hadoop.mapreduce.Job$10.run(Job.java:1296)
>   at java.security.AccessController.doPrivileged(Native Method)
>   at javax.security.auth.Subject.doAs(Subject.java:415)
>   at 
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1698)
>   at org.apache.hadoop.mapreduce.Job.submit(Job.java:1296)
>   at org.apache.hadoop.mapred.JobClient$1.run(JobClient.java:575)
>   at org.apache.hadoop.mapred.JobClient$1.run(JobClient.java:570)
>   at java.security.AccessController.doPrivileged(Native Method)
>   at javax.security.auth.Subject.doAs(Subject.java:415)
>   at 
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1698)
>   at 
> org.apache.hadoop.mapred.JobClient.submitJobInternal(JobClient.java:570)
>   at org.apache.hadoop.mapred.JobClient.submitJob(JobClient.java:561)
>   at 
> org.apache.oozie.action.hadoop.MapReduceMain.submitJob(MapReduceMain.java:102)
>   at 
> org.apache.oozie.action.hadoop.MapReduceMain.run(MapReduceMain.java:64)
>   at org.apache.oozi

[jira] [Commented] (OOZIE-2871) when Enable Kerberos, Oozie perform tasks throw “Client cannot authenticate via:[TOKEN, KERBEROS]”

2017-10-19 Thread Denes Bodo (JIRA) 

[ 
https://issues.apache.org/jira/browse/OOZIE-2871?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16210735#comment-16210735
 ] 

Denes Bodo commented on OOZIE-2871:
---

[~asasvari] A very similar issue can be reproducible between two secure 
clusters with the same realm. In that case using hadoop distcp command from 
command line works as expected, but when we ask Oozie it fails with the above 
error (and stack trace). Oozie version is 4.2.

> when Enable Kerberos,  Oozie perform tasks throw “Client cannot authenticate 
> via:[TOKEN, KERBEROS]”
> ---
>
> Key: OOZIE-2871
> URL: https://issues.apache.org/jira/browse/OOZIE-2871
> Project: Oozie
>  Issue Type: Bug
>  Components: security
>Affects Versions: 4.2.0
> Environment: Oozie version :4.2.0
> Hadoop version:2.7.2
> Both Oozie and Hadoop are enabled kerberos.
>Reporter: yangfang
>Priority: Critical
>
> When  Oozie and Hadoop both enabled kerberos， I submitted a mapreduce   job 
> to oozie，then I got the error as below：
> 2017-04-27 13:37:12,677 WARN MapReduceActionExecutor: 523 - SERVER[zdh143] 
> USER[mr] GROUP[-] TOKEN[] APP[map-reduce-wf] 
> JOB[008-170427133546167-oozie-mr-W] 
> ACTION[008-170427133546167-oozie-mr-W@mr-node] Launcher exception: Failed 
> on local exception: java.io.IOException: 
> org.apache.hadoop.security.AccessControlException: Client cannot authenticate 
> via:[TOKEN, KERBEROS]; Host Details : local host is: "zdh142/10.43.183.142"; 
> destination host is: "zdh143":9000; 
> java.io.IOException: Failed on local exception: java.io.IOException: 
> org.apache.hadoop.security.AccessControlException: Client cannot authenticate 
> via:[TOKEN, KERBEROS]; Host Details : local host is: "zdh142/10.43.183.142"; 
> destination host is: "zdh143":9000; 
>   at org.apache.hadoop.net.NetUtils.wrapException(NetUtils.java:773)
>   at org.apache.hadoop.ipc.Client.call(Client.java:1479)
>   at org.apache.hadoop.ipc.Client.call(Client.java:1412)
>   at 
> org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:229)
>   at com.sun.proxy.$Proxy17.getFileInfo(Unknown Source)
>   at 
> org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolTranslatorPB.getFileInfo(ClientNamenodeProtocolTranslatorPB.java:771)
>   at sun.reflect.GeneratedMethodAccessor2.invoke(Unknown Source)
>   at 
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>   at java.lang.reflect.Method.invoke(Method.java:606)
>   at 
> org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:191)
>   at 
> org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:102)
>   at com.sun.proxy.$Proxy18.getFileInfo(Unknown Source)
>   at org.apache.hadoop.hdfs.DFSClient.getFileInfo(DFSClient.java:2108)
>   at 
> org.apache.hadoop.hdfs.DistributedFileSystem$22.doCall(DistributedFileSystem.java:1305)
>   at 
> org.apache.hadoop.hdfs.DistributedFileSystem$22.doCall(DistributedFileSystem.java:1301)
>   at 
> org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81)
>   at 
> org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1301)
>   at org.apache.hadoop.fs.FileSystem.exists(FileSystem.java:1426)
>   at 
> org.apache.hadoop.mapred.FileOutputFormat.checkOutputSpecs(FileOutputFormat.java:130)
>   at 
> org.apache.hadoop.mapreduce.JobSubmitter.checkSpecs(JobSubmitter.java:268)
>   at 
> org.apache.hadoop.mapreduce.JobSubmitter.submitJobInternal(JobSubmitter.java:139)
>   at org.apache.hadoop.mapreduce.Job$10.run(Job.java:1299)
>   at org.apache.hadoop.mapreduce.Job$10.run(Job.java:1296)
>   at java.security.AccessController.doPrivileged(Native Method)
>   at javax.security.auth.Subject.doAs(Subject.java:415)
>   at 
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1698)
>   at org.apache.hadoop.mapreduce.Job.submit(Job.java:1296)
>   at org.apache.hadoop.mapred.JobClient$1.run(JobClient.java:575)
>   at org.apache.hadoop.mapred.JobClient$1.run(JobClient.java:570)
>   at java.security.AccessController.doPrivileged(Native Method)
>   at javax.security.auth.Subject.doAs(Subject.java:415)
>   at 
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1698)
>   at 
> org.apache.hadoop.mapred.JobClient.submitJobInternal(JobClient.java:570)
>   at org.apache.hadoop.mapred.JobClient.submitJob(JobClient.java:561)
>   at 
> org.apache.oozie.action.hadoop.MapReduceMain.submitJob(MapReduceMain.java:102)
>   at 
> org.apache.oozie.action.hadoop.MapReduceMain.run(MapRedu

[jira] [Commented] (OOZIE-3084) Add missing JVM properties to Jetty startup script

2017-10-19 Thread Andras Piros (JIRA) 

[ 
https://issues.apache.org/jira/browse/OOZIE-3084?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16210731#comment-16210731
 ] 

Andras Piros commented on OOZIE-3084:
-

[~asasvari] thanks for the review!

In fact I've checked that:
* no other Tomcat related options are needed to be put to Jetty startup scripts
* you're right, {{XLogService}} is started before (and should be started 
before, since we need appropriate logging) before {{ConfigurationService}}, 
that's why {{oozie.instance.id}} is not considered inside the log messages
* with my patch {{oozie.instance.id}} is filled, please see my previous comment

As for the testing, I've looked at {{TestXLog}}, but couldn't find a way to 
test any other thing that when a {{SERVER}} parameter is defined having the 
{{ConversionPattern}} containing {{oozie.instance.id}}, and that system 
property is set, log message would be rendered accordingly. This is the kind of 
test what is already tested, and / or regards Log4J functionality. ATM I cannot 
think of a test case where exactly the functionality covered by the patch 
(setting the system property from the startup scripts cause the log message to 
appear correctly) could be tested.

> Add missing JVM properties to Jetty startup script
> --
>
> Key: OOZIE-3084
> URL: https://issues.apache.org/jira/browse/OOZIE-3084
> Project: Oozie
>  Issue Type: Bug
>  Components: scripts
>Affects Versions: 5.0.0
>Reporter: Andras Piros
>Assignee: Andras Piros
> Fix For: 5.0.0
>
> Attachments: OOZIE-3084.001.patch, OOZIE-3084.001.patch, 
> OOZIE-3084.001.patch
>
>
> In OOZIE-2666, when Oozie has moved from Tomcat to Jetty, we also moved the 
> Tomcat related JVM options inside the startup script {{bin/oozied.sh}} to the 
> more specific {{bin/oozie-jetty-server.sh}}.
> It seems that not all the 
> [*{{catalina_opts}}*|https://github.com/apache/oozie/blob/branch-4.3/distro/src/main/bin/oozied.sh#L54-L74]
>  have been moved to 
> [*{{jetty_opts}}*|https://github.com/apache/oozie/blob/master/distro/src/main/bin/oozie-jetty-server.sh#L70-L86].
>  Such an example is {{oozie.instance.id}}, causing we don't have any server 
> name information within the Oozie server logs.
> So the task is to investigate startup scripts and put missing and needed 
> {{catalina_opts}} to {{jetty_opts}}.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)