[jira] [Comment Edited] (OOZIE-2715) Provide possibility to set default values for properties of a particular action type

[Peter Cseh (JIRA)]

[ 
https://issues.apache.org/jira/browse/OOZIE-2715?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16466449#comment-16466449
 ] 

Peter Cseh edited comment on OOZIE-2715 at 5/7/18 8:44 PM:
---

This is already possible via the 
{{oozie.service.HadoopAccessorService.action.configurations}} property.  Check 
it's description here:
https://oozie.apache.org/docs/5.0.0/oozie-default.xml


was (Author: gezapeti):
This is already possible via the 
{{oozie.service.HadoopAccessorService.action.configurations]} property.  Check 
it's description here:
https://oozie.apache.org/docs/5.0.0/oozie-default.xml

> Provide possibility to set default values for properties of a particular 
> action type
> 
>
> Key: OOZIE-2715
> URL: https://issues.apache.org/jira/browse/OOZIE-2715
> Project: Oozie
>  Issue Type: Wish
>  Components: action
>Affects Versions: 4.2.0
>Reporter: Istvan Vajnorak
>Priority: Major
>
> When it comes to retrying an action, one can set the default values for a 
> retry on the global level, and the override can happen then on the individual 
> step's level only.
> The proposal to be able to set retry and other properties on an action type 
> basis, could allow defaulting things and help not to do repetitive work on 
> each action instance in the workflow.
> Example:
> - One has 60 shell actions and 30 java actions in a workflow.
> - One sets " oozie.launcher.yarn.resourcemanager.am.max-attempts=1" globally
> - For shell steps they don't want this to get applied, so they have to 
> suppress the retry on those steps each time
> The proposal would allow an intermediate level of override similarly like the 
> hadoop conf allows with the *-default.xml, *-site.xml and code level 
> properties override.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (OOZIE-2715) Provide possibility to set default values for properties of a particular action type

[Peter Cseh (JIRA)]

[ 
https://issues.apache.org/jira/browse/OOZIE-2715?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16466449#comment-16466449
 ] 

Peter Cseh commented on OOZIE-2715:
---

This is already possible via the 
{{oozie.service.HadoopAccessorService.action.configurations]} property.  Check 
it's description here:
https://oozie.apache.org/docs/5.0.0/oozie-default.xml

> Provide possibility to set default values for properties of a particular 
> action type
> 
>
> Key: OOZIE-2715
> URL: https://issues.apache.org/jira/browse/OOZIE-2715
> Project: Oozie
>  Issue Type: Wish
>  Components: action
>Affects Versions: 4.2.0
>Reporter: Istvan Vajnorak
>Priority: Major
>
> When it comes to retrying an action, one can set the default values for a 
> retry on the global level, and the override can happen then on the individual 
> step's level only.
> The proposal to be able to set retry and other properties on an action type 
> basis, could allow defaulting things and help not to do repetitive work on 
> each action instance in the workflow.
> Example:
> - One has 60 shell actions and 30 java actions in a workflow.
> - One sets " oozie.launcher.yarn.resourcemanager.am.max-attempts=1" globally
> - For shell steps they don't want this to get applied, so they have to 
> suppress the retry on those steps each time
> The proposal would allow an intermediate level of override similarly like the 
> hadoop conf allows with the *-default.xml, *-site.xml and code level 
> properties override.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (OOZIE-3232) Oozie may waste up to 25% of the heap due to duplicate strings

[Andras Piros (JIRA)]

[ 
https://issues.apache.org/jira/browse/OOZIE-3232?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16466116#comment-16466116
 ] 

Andras Piros commented on OOZIE-3232:
-

[~mi...@cloudera.com] the failing test is unrelated.

> Oozie may waste up to 25% of the heap due to duplicate strings
> --
>
> Key: OOZIE-3232
> URL: https://issues.apache.org/jira/browse/OOZIE-3232
> Project: Oozie
>  Issue Type: Improvement
>  Components: core
>Reporter: Misha Dmitriev
>Assignee: Misha Dmitriev
>Priority: Major
> Attachments: OOZIE-3232-01.patch, jxray-analysis-dup-strings.png
>
>
> I've recently analyzed an Oozie heap dump obtained from a customer's 
> production job, using jxray (www.jxray.com). In this job, Oozie's heap 
> consumption is ~6GB, and it turns out that nearly 25% of it is wasted due to 
> a large number of duplicate strings. The screenshot below, taken from the 
> jxray report, illustrates the problem.
> !jxray-analysis-dup-strings.png|width=638,height=669!
> It turns out that a lot of duplicate strings come from the oozie's own code, 
> i.e. {{org.apache.oozie.*}}. In particular, the top data field wasting memory 
> is {{org.apache.oozie.StringBlob.string}} (wastes 2.6%, or ~160MB). From the 
> source code of {{StringBlob}} I see that it would be trivial to intern 
> (deduplicate) these strings by adding the call to {{String.intern()}} in the 
> constructor and a few other places. Similarly, various fields of 
> {{org.apache.oozie.WorkflowJobBean}} and {{WorkflowActionBean}} collectively 
> waste a lot of memory, and can be fixed in the same way.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (OOZIE-3232) Oozie may waste up to 25% of the heap due to duplicate strings

[Misha Dmitriev (JIRA)]

[ 
https://issues.apache.org/jira/browse/OOZIE-3232?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16466108#comment-16466108
 ] 

Misha Dmitriev commented on OOZIE-3232:
---

Thank you [~andras.piros] Can you please take a look at the above failed tests? 
I highly doubt that my changes could cause this.

> Oozie may waste up to 25% of the heap due to duplicate strings
> --
>
> Key: OOZIE-3232
> URL: https://issues.apache.org/jira/browse/OOZIE-3232
> Project: Oozie
>  Issue Type: Improvement
>  Components: core
>Reporter: Misha Dmitriev
>Assignee: Misha Dmitriev
>Priority: Major
> Attachments: OOZIE-3232-01.patch, jxray-analysis-dup-strings.png
>
>
> I've recently analyzed an Oozie heap dump obtained from a customer's 
> production job, using jxray (www.jxray.com). In this job, Oozie's heap 
> consumption is ~6GB, and it turns out that nearly 25% of it is wasted due to 
> a large number of duplicate strings. The screenshot below, taken from the 
> jxray report, illustrates the problem.
> !jxray-analysis-dup-strings.png|width=638,height=669!
> It turns out that a lot of duplicate strings come from the oozie's own code, 
> i.e. {{org.apache.oozie.*}}. In particular, the top data field wasting memory 
> is {{org.apache.oozie.StringBlob.string}} (wastes 2.6%, or ~160MB). From the 
> source code of {{StringBlob}} I see that it would be trivial to intern 
> (deduplicate) these strings by adding the call to {{String.intern()}} in the 
> constructor and a few other places. Similarly, various fields of 
> {{org.apache.oozie.WorkflowJobBean}} and {{WorkflowActionBean}} collectively 
> waste a lot of memory, and can be fixed in the same way.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (OOZIE-3235) Upgrade Old ActiveMQ dependency in Oozie

[Hadoop QA (JIRA)]

[ 
https://issues.apache.org/jira/browse/OOZIE-3235?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16465909#comment-16465909
 ] 

Hadoop QA commented on OOZIE-3235:
--


Testing JIRA OOZIE-3235

Cleaning local git workspace



{color:green}+1 PATCH_APPLIES{color}
{color:green}+1 CLEAN{color}
{color:red}-1 RAW_PATCH_ANALYSIS{color}
.{color:green}+1{color} the patch does not introduce any @author tags
.{color:green}+1{color} the patch does not introduce any tabs
.{color:green}+1{color} the patch does not introduce any trailing spaces
.{color:green}+1{color} the patch does not introduce any line longer than 
132
.{color:red}-1{color} the patch does not add/modify any testcase
{color:green}+1 RAT{color}
.{color:green}+1{color} the patch does not seem to introduce new RAT 
warnings
{color:green}+1 JAVADOC{color}
.{color:green}+1{color} the patch does not seem to introduce new Javadoc 
warnings
{color:green}+1 COMPILE{color}
.{color:green}+1{color} HEAD compiles
.{color:green}+1{color} patch compiles
.{color:green}+1{color} the patch does not seem to introduce new javac 
warnings
{color:green}+1{color} There are no new bugs found in total.
. {color:green}+1{color} There are no new bugs found in [examples].
. {color:green}+1{color} There are no new bugs found in [webapp].
. {color:green}+1{color} There are no new bugs found in [core].
. {color:green}+1{color} There are no new bugs found in [tools].
. {color:green}+1{color} There are no new bugs found in [server].
. {color:green}+1{color} There are no new bugs found in [docs].
. {color:green}+1{color} There are no new bugs found in [sharelib/hive2].
. {color:green}+1{color} There are no new bugs found in [sharelib/pig].
. {color:green}+1{color} There are no new bugs found in [sharelib/streaming].
. {color:green}+1{color} There are no new bugs found in [sharelib/hive].
. {color:green}+1{color} There are no new bugs found in [sharelib/hcatalog].
. {color:green}+1{color} There are no new bugs found in [sharelib/sqoop].
. {color:green}+1{color} There are no new bugs found in [sharelib/oozie].
. {color:green}+1{color} There are no new bugs found in [sharelib/distcp].
. {color:green}+1{color} There are no new bugs found in [sharelib/spark].
. {color:green}+1{color} There are no new bugs found in [client].
{color:green}+1 BACKWARDS_COMPATIBILITY{color}
.{color:green}+1{color} the patch does not change any JPA 
Entity/Colum/Basic/Lob/Transient annotations
.{color:green}+1{color} the patch does not modify JPA files
{color:green}+1 TESTS{color}
.Tests run: 2129
.{color:orange}Tests failed at first run:{color}
TestJavaActionExecutor#testCredentialsSkip
TestJMSAccessorService#testConnectionRetryExceptionListener
TestOozieDBCLI#testOozieDBCLI
.For the complete list of flaky tests, see TEST-SUMMARY-FULL files.
{color:green}+1 DISTRO{color}
.{color:green}+1{color} distro tarball builds with the patch 


{color:red}*-1 Overall result, please check the reported -1(s)*{color}


The full output of the test-patch run is available at

. https://builds.apache.org/job/PreCommit-OOZIE-Build/520/



> Upgrade Old ActiveMQ dependency in Oozie
> 
>
> Key: OOZIE-3235
> URL: https://issues.apache.org/jira/browse/OOZIE-3235
> Project: Oozie
>  Issue Type: Bug
>  Components: client
>Affects Versions: 5.1.0
>Reporter: Mate Juhasz
>Assignee: Mate Juhasz
>Priority: Minor
> Fix For: 5.1.0
>
> Attachments: OOZIE-3235-001.patch
>
>
> An old, and vulnerable version of ActiveMQ is used by Oozie's tests (5.13.3).
> It seems that the jars are even included in the oozie-client with "compile" 
> scope, but we only use them in unit tests.
> Oozie-client uses the JMS-API only from these jars, which could be included 
> separately (javax.jms).
> The version could be upgraded to 5.14.2 and the scope changed to "test" 
> instead of "compile".



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Failed: OOZIE-3235 PreCommit Build #520

[Apache Jenkins Server]

Jira: https://issues.apache.org/jira/browse/OOZIE-3235
Build: https://builds.apache.org/job/PreCommit-OOZIE-Build/520/

###
## LAST 100 LINES OF THE CONSOLE 
###
[...truncated 1.75 MB...]
[DEBUG] There are no new bugs found in [sharelib/streaming].
[TRACE] New XMLLib present, calling 'xmllint --xpath' to get bug instance counts
[DEBUG] There are no new bugs found in [sharelib/hive].
[TRACE] New XMLLib present, calling 'xmllint --xpath' to get bug instance counts
[DEBUG] There are no new bugs found in [sharelib/hcatalog].
[TRACE] New XMLLib present, calling 'xmllint --xpath' to get bug instance counts
[DEBUG] There are no new bugs found in [sharelib/sqoop].
[TRACE] New XMLLib present, calling 'xmllint --xpath' to get bug instance counts
[DEBUG] There are no new bugs found in [sharelib/oozie].
[TRACE] New XMLLib present, calling 'xmllint --xpath' to get bug instance counts
[DEBUG] There are no new bugs found in [sharelib/distcp].
[TRACE] New XMLLib present, calling 'xmllint --xpath' to get bug instance counts
[DEBUG] There are no new bugs found in [sharelib/spark].
[TRACE] New XMLLib present, calling 'xmllint --xpath' to get bug instance counts
[DEBUG] There are no new bugs found in [client].
[INFO] There are no new bugs found totally].
[TRACE] FindBugs diffs checked and reports created
[TRACE] Summary file size is 2177 bytes
[TRACE] Full summary file size is 1314 bytes
[TRACE] File 
[/home/jenkins/jenkins-slave/workspace/PreCommit-OOZIE-Build/test-patch/tmp/FINDBUGS_DIFF/diff/findbugs-diff-0.1.0-all.jar]
 removed
[TRACE] File 
[/home/jenkins/jenkins-slave/workspace/PreCommit-OOZIE-Build/test-patch/tmp/FINDBUGS_DIFF/diff/findbugs-diff-0.1.0-all.jar.md5sum]
 removed
  Running test-patch task BACKWARDS_COMPATIBILITY
  Running test-patch task TESTS
  Running test-patch task DISTRO


Testing JIRA OOZIE-3235

Cleaning local git workspace



+1 PATCH_APPLIES
+1 CLEAN
-1 RAW_PATCH_ANALYSIS
+1 the patch does not introduce any @author tags
+1 the patch does not introduce any tabs
+1 the patch does not introduce any trailing spaces
+1 the patch does not introduce any line longer than 132
-1 the patch does not add/modify any testcase
+1 RAT
+1 the patch does not seem to introduce new RAT warnings
+1 JAVADOC
+1 the patch does not seem to introduce new Javadoc warnings
+1 COMPILE
+1 HEAD compiles
+1 patch compiles
+1 the patch does not seem to introduce new javac warnings
+1 There are no new bugs found in total.
 +1 There are no new bugs found in [examples].
 +1 There are no new bugs found in [webapp].
 +1 There are no new bugs found in [core].
 +1 There are no new bugs found in [tools].
 +1 There are no new bugs found in [server].
 +1 There are no new bugs found in [docs].
 +1 There are no new bugs found in [sharelib/hive2].
 +1 There are no new bugs found in [sharelib/pig].
 +1 There are no new bugs found in [sharelib/streaming].
 +1 There are no new bugs found in [sharelib/hive].
 +1 There are no new bugs found in [sharelib/hcatalog].
 +1 There are no new bugs found in [sharelib/sqoop].
 +1 There are no new bugs found in [sharelib/oozie].
 +1 There are no new bugs found in [sharelib/distcp].
 +1 There are no new bugs found in [sharelib/spark].
 +1 There are no new bugs found in [client].
+1 BACKWARDS_COMPATIBILITY
+1 the patch does not change any JPA Entity/Colum/Basic/Lob/Transient 
annotations
+1 the patch does not modify JPA files
+1 TESTS
Tests run: 2129
Tests failed at first run:
TestJavaActionExecutor#testCredentialsSkip
TestJMSAccessorService#testConnectionRetryExceptionListener
TestOozieDBCLI#testOozieDBCLI
For the complete list of flaky tests, see TEST-SUMMARY-FULL files.
+1 DISTRO
+1 distro tarball builds with the patch 


-1 Overall result, please check the reported -1(s)


The full output of the test-patch run is available at

 https://builds.apache.org/job/PreCommit-OOZIE-Build/520/

Adding comment to JIRA
  % Total% Received % Xferd  Average Speed   TimeTime Time  Current
 Dload  Upload   Total   SpentLeft  Speed
  0 00 00 0  0  0 --:--:-- --:--:-- --:--:-- 0  
0 00 00 0  0  0 --:--:-- --:--:-- --:--:-- 
0{"self":"https://issues.apache.org/jira/rest/api/2/issue/13157539/comment/16465909","id":"16465909","author":{"self":"https://issues.apache.org/jira/rest/api/2/user?username=hadoopqa","name":"hadoopqa","key":"hadoopqa","emailAddress":"blackhole
 at hadoop dot apache dot 

[jira] [Commented] (OOZIE-2877) Oozie Git Action

[Peter Cseh (JIRA)]

[ 
https://issues.apache.org/jira/browse/OOZIE-2877?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16465839#comment-16465839
 ] 

Peter Cseh commented on OOZIE-2877:
---

I've taken an other quick look. I'm extremely pleased that a working example is 
part of the patch!

[~clayb], please respond on RB to my comments. :)

Some of them are nitpicks, but there are three bigger open questions:
 * credential files on  HDFS and how strict we should handle them
 * is estination-uri validated against the whitelisted namenodes? 
 * should we support job-tracker on new action types?

 

> Oozie Git Action
> 
>
> Key: OOZIE-2877
> URL: https://issues.apache.org/jira/browse/OOZIE-2877
> Project: Oozie
>  Issue Type: Sub-task
>  Components: action
>Reporter: Clay B.
>Assignee: Clay B.
>Priority: Major
>  Labels: action
> Fix For: trunk
>
> Attachments: 0001-OOZIE-2877-Oozie-Git-Action.patch, 
> 0002-OOZIE-2877-Oozie-Git-Action.patch, 
> 0003-OOZIE-2877-Oozie-Git-Action.patch, 
> 0004-OOZIE-2877-Oozie-Git-Action.patch, 
> 0005-OOZIE-2877-Oozie-Git-Action.patch, 
> 0006-OOZIE-2877-Oozie-Git-Action.patch, 
> 0007-OOZIE-2877-Oozie-Git-Action.patch, 
> 0008-OOZIE-2877-Oozie-Git-Action.patch, 
> 0009-OOZIE-2877-Oozie-Git-Action.patch, OOZIE-2877.010.patch, 
> OOZIE-2877.011.patch
>
>
> To aide in deploying ASCII artifacts to clusters, let's provide a tie-in for 
> a source-code management system. Git would be my preferred choice.
> Ideally, this could handle a user's key material e.g. for an ssh key to pull 
> down from a secured repository. This would free users from handling their own 
> key staging and clean-up on YARN nodes and only require them to store the key 
> secured in HDFS.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Re: Review Request 59620: This review board request is for an action to provide a Git action for Oozie

[Peter Cseh via Review Board]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/59620/#review202546
---




examples/src/main/apps/git/workflow.xml
Lines 24-25 (patched)


Let's check out apache/Oozie from github instead :)


- Peter Cseh


On May 4, 2018, 5:43 p.m., Clay B. wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/59620/
> ---
> 
> (Updated May 4, 2018, 5:43 p.m.)
> 
> 
> Review request for oozie and András Piros.
> 
> 
> Bugs: OOZIE-2877
> https://issues.apache.org/jira/browse/OOZIE-2877
> 
> 
> Repository: oozie-git
> 
> 
> Description
> ---
> 
> OOZIE-2877 - Oozie Git Action
> 
> 
> Diffs
> -
> 
>   client/src/main/java/org/apache/oozie/cli/OozieCLI.java 
> 4abc7502c0c9d8b59ded2baaed30c407ad073008 
>   client/src/main/resources/git-action-0.1.xsd PRE-CREATION 
>   core/src/main/java/org/apache/oozie/action/hadoop/GitActionExecutor.java 
> PRE-CREATION 
>   core/src/main/resources/oozie-default.xml 
> 6f8925828090cee29a818de30a7c31db0617d816 
>   docs/src/site/twiki/WorkflowFunctionalSpec.twiki 
> 76cbe21ecd79eae86885a8db77831bf2c40f2d4f 
>   examples/src/main/apps/git/job.properties PRE-CREATION 
>   examples/src/main/apps/git/workflow.xml PRE-CREATION 
>   pom.xml d9fe1b20f571b1a6afacb3ce53f4d409dd3d60b1 
>   sharelib/git/pom.xml PRE-CREATION 
>   sharelib/git/src/main/java/org/apache/oozie/action/hadoop/GitMain.java 
> PRE-CREATION 
>   
> sharelib/git/src/test/java/org/apache/oozie/action/hadoop/TestGitActionExecutor.java
>  PRE-CREATION 
>   
> sharelib/git/src/test/java/org/apache/oozie/action/hadoop/TestGitMainGetKey.java
>  PRE-CREATION 
>   
> sharelib/git/src/test/java/org/apache/oozie/action/hadoop/TestIntegrationGitActionExecutor.java
>  PRE-CREATION 
>   sharelib/pom.xml 6a0864d4e5b52160dc4e1a5cb9dfc34a037b67c8 
>   src/main/assemblies/sharelib.xml 07dc69c8276895b254e5af0cc021cce6ebad18f4 
>   webapp/pom.xml 797996912b6e6381b261a69f8eb1e012fe488fdf 
> 
> 
> Diff: https://reviews.apache.org/r/59620/diff/5/
> 
> 
> Testing
> ---
> 
> Tested using unit and integration tests. Still need to:
> * Test on a cluster
> * Test with an authenticated SSH hosted Git repo
> 
> Sumitted a request to the JGit community as their branch pulling code seems 
> to have an 
> [issue](https://dev.eclipse.org/mhonarc/lists/jgit-dev/msg03343.html).
> 
> Andras has tested against his patch 011 with results at 
> https://issues.apache.org/jira/browse/OOZIE-2877?focusedCommentId=16459523=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-16459523
> 
> 
> File Attachments
> 
> 
> 0001-OOZIE-2877-Oozie-Git-Action.patch
>   
> https://reviews.apache.org/media/uploaded/files/2017/05/29/24f90a78-3dc1-49fe-bf29-5927a3cd5e72__0001-OOZIE-2877-Oozie-Git-Action.patch
> Patch
>   
> https://reviews.apache.org/media/uploaded/files/2017/05/29/dd23dd72-67e0-456f-9b52-e566d8d17d16__0001-OOZIE-2877-Oozie-Git-Action.patch
> 
> 
> Thanks,
> 
> Clay B.
> 
>

Re: Review Request 59620: This review board request is for an action to provide a Git action for Oozie

[Peter Cseh via Review Board]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/59620/#review202545
---




core/src/main/java/org/apache/oozie/action/hadoop/GitActionExecutor.java
Lines 55-56 (patched)


oozie.oozie? seems redundant :)


- Peter Cseh


On May 4, 2018, 5:43 p.m., Clay B. wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/59620/
> ---
> 
> (Updated May 4, 2018, 5:43 p.m.)
> 
> 
> Review request for oozie and András Piros.
> 
> 
> Bugs: OOZIE-2877
> https://issues.apache.org/jira/browse/OOZIE-2877
> 
> 
> Repository: oozie-git
> 
> 
> Description
> ---
> 
> OOZIE-2877 - Oozie Git Action
> 
> 
> Diffs
> -
> 
>   client/src/main/java/org/apache/oozie/cli/OozieCLI.java 
> 4abc7502c0c9d8b59ded2baaed30c407ad073008 
>   client/src/main/resources/git-action-0.1.xsd PRE-CREATION 
>   core/src/main/java/org/apache/oozie/action/hadoop/GitActionExecutor.java 
> PRE-CREATION 
>   core/src/main/resources/oozie-default.xml 
> 6f8925828090cee29a818de30a7c31db0617d816 
>   docs/src/site/twiki/WorkflowFunctionalSpec.twiki 
> 76cbe21ecd79eae86885a8db77831bf2c40f2d4f 
>   examples/src/main/apps/git/job.properties PRE-CREATION 
>   examples/src/main/apps/git/workflow.xml PRE-CREATION 
>   pom.xml d9fe1b20f571b1a6afacb3ce53f4d409dd3d60b1 
>   sharelib/git/pom.xml PRE-CREATION 
>   sharelib/git/src/main/java/org/apache/oozie/action/hadoop/GitMain.java 
> PRE-CREATION 
>   
> sharelib/git/src/test/java/org/apache/oozie/action/hadoop/TestGitActionExecutor.java
>  PRE-CREATION 
>   
> sharelib/git/src/test/java/org/apache/oozie/action/hadoop/TestGitMainGetKey.java
>  PRE-CREATION 
>   
> sharelib/git/src/test/java/org/apache/oozie/action/hadoop/TestIntegrationGitActionExecutor.java
>  PRE-CREATION 
>   sharelib/pom.xml 6a0864d4e5b52160dc4e1a5cb9dfc34a037b67c8 
>   src/main/assemblies/sharelib.xml 07dc69c8276895b254e5af0cc021cce6ebad18f4 
>   webapp/pom.xml 797996912b6e6381b261a69f8eb1e012fe488fdf 
> 
> 
> Diff: https://reviews.apache.org/r/59620/diff/5/
> 
> 
> Testing
> ---
> 
> Tested using unit and integration tests. Still need to:
> * Test on a cluster
> * Test with an authenticated SSH hosted Git repo
> 
> Sumitted a request to the JGit community as their branch pulling code seems 
> to have an 
> [issue](https://dev.eclipse.org/mhonarc/lists/jgit-dev/msg03343.html).
> 
> Andras has tested against his patch 011 with results at 
> https://issues.apache.org/jira/browse/OOZIE-2877?focusedCommentId=16459523=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-16459523
> 
> 
> File Attachments
> 
> 
> 0001-OOZIE-2877-Oozie-Git-Action.patch
>   
> https://reviews.apache.org/media/uploaded/files/2017/05/29/24f90a78-3dc1-49fe-bf29-5927a3cd5e72__0001-OOZIE-2877-Oozie-Git-Action.patch
> Patch
>   
> https://reviews.apache.org/media/uploaded/files/2017/05/29/dd23dd72-67e0-456f-9b52-e566d8d17d16__0001-OOZIE-2877-Oozie-Git-Action.patch
> 
> 
> Thanks,
> 
> Clay B.
> 
>

[jira] [Commented] (OOZIE-3235) Upgrade Old ActiveMQ dependency in Oozie

[Hadoop QA (JIRA)]

[ 
https://issues.apache.org/jira/browse/OOZIE-3235?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16465824#comment-16465824
 ] 

Hadoop QA commented on OOZIE-3235:
--

PreCommit-OOZIE-Build started


> Upgrade Old ActiveMQ dependency in Oozie
> 
>
> Key: OOZIE-3235
> URL: https://issues.apache.org/jira/browse/OOZIE-3235
> Project: Oozie
>  Issue Type: Bug
>  Components: client
>Affects Versions: 5.1.0
>Reporter: Mate Juhasz
>Assignee: Mate Juhasz
>Priority: Minor
> Fix For: 5.1.0
>
> Attachments: OOZIE-3235-001.patch
>
>
> An old, and vulnerable version of ActiveMQ is used by Oozie's tests (5.13.3).
> It seems that the jars are even included in the oozie-client with "compile" 
> scope, but we only use them in unit tests.
> Oozie-client uses the JMS-API only from these jars, which could be included 
> separately (javax.jms).
> The version could be upgraded to 5.14.2 and the scope changed to "test" 
> instead of "compile".



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (OOZIE-3219) Cannot compile with hadoop 3.1.0

[Andras Piros (JIRA)]

[ 
https://issues.apache.org/jira/browse/OOZIE-3219?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16465821#comment-16465821
 ] 

Andras Piros commented on OOZIE-3219:
-

Failing test seems unrelated. [~gezapeti] can you pls. review? Thanks!

> Cannot compile with hadoop 3.1.0
> 
>
> Key: OOZIE-3219
> URL: https://issues.apache.org/jira/browse/OOZIE-3219
> Project: Oozie
>  Issue Type: Bug
>  Components: core
>Affects Versions: 5.0.0
> Environment: {code:java}
> Apache Maven 3.5.3 (3383c37e1f9e9b3bc3df5050c29c8aff9f295297; 
> 2018-02-24T19:49:05Z)
> Maven home: /opt/maven/apache-maven-3.5.3
> Java version: 1.8.0_161, vendor: Oracle Corporation
> Java home: /usr/lib/jvm/java-1.8.0-openjdk-1.8.0.161-0.b14.el7_4.x86_64/jre
> Default locale: en_US, platform encoding: UTF-8
> OS name: "linux", version: "3.10.0-693.21.1.el7.x86_64", arch: "amd64", 
> family: "unix"{code}
> latest 5.0 release bits
>Reporter: Artem Ervits
>Assignee: Artem Ervits
>Priority: Major
> Fix For: 5.1.0
>
> Attachments: OOZIE-3219.00.patch, OOZIE-3219.002.patch
>
>
> command failure on the following, same happens with 3.0.0. Issue does not 
> appear with < 3.0
> {code:java}
> mvn clean install -DskipTests -Dhadoop.version=3.1.0{code}
> {code:java}
> [ERROR] COMPILATION ERROR :
> [INFO] -
> [ERROR] 
> /tmp/oozie-5.0.0/core/src/main/java/org/apache/oozie/util/db/FailingConnectionWrapper.java:[24,37]
>  package org.apache.directory.api.util does not exist
> [ERROR] 
> /tmp/oozie-5.0.0/core/src/main/java/org/apache/oozie/util/db/FailingConnectionWrapper.java:[352,41]
>  cannot find symbol
> symbol: variable Strings
> location: class 
> org.apache.oozie.util.db.FailingConnectionWrapper.OozieDmlStatementPredicate
> [INFO] 2 errors
> [INFO] -
> [INFO] 
> 
> [INFO] Reactor Summary:
> [INFO]
> [INFO] Apache Oozie Main 5.0.0  SUCCESS [ 3.768 s]
> [INFO] Apache Oozie Client  SUCCESS [ 30.924 
> s]
> [INFO] Apache Oozie Share Lib Oozie ... SUCCESS [ 20.904 
> s]
> [INFO] Apache Oozie Share Lib HCatalog  SUCCESS [ 9.150 s]
> [INFO] Apache Oozie Share Lib Distcp .. SUCCESS [ 7.938 s]
> [INFO] Apache Oozie Core .. FAILURE [ 13.929 
> s]
> [INFO] Apache Oozie Share Lib Streaming ... SKIPPED
> [INFO] Apache Oozie Share Lib Pig . SKIPPED
> [INFO] Apache Oozie Share Lib Hive  SKIPPED
> [INFO] Apache Oozie Share Lib Hive 2 .. SKIPPED
> [INFO] Apache Oozie Share Lib Sqoop ... SKIPPED
> [INFO] Apache Oozie Examples .. SKIPPED
> [INFO] Apache Oozie Share Lib Spark ... SKIPPED
> [INFO] Apache Oozie Share Lib . SKIPPED
> [INFO] Apache Oozie Docs .. SKIPPED
> [INFO] Apache Oozie WebApp  SKIPPED
> [INFO] Apache Oozie Tools . SKIPPED
> [INFO] Apache Oozie MiniOozie . SKIPPED
> [INFO] Apache Oozie Server  SKIPPED
> [INFO] Apache Oozie Distro  SKIPPED
> [INFO] Apache Oozie ZooKeeper Security Tests 5.0.0  SKIPPED
> [INFO] 
> 
> [INFO] BUILD FAILURE
> [INFO] 
> 
> [INFO] Total time: 01:28 min
> [INFO] Finished at: 2018-04-13T14:03:17Z
> [INFO] 
> 
> [ERROR] Failed to execute goal 
> org.apache.maven.plugins:maven-compiler-plugin:3.7.0:compile 
> (default-compile) on project oozie-core: Compilation failure: Compilation 
> failure:
> [ERROR] 
> /tmp/oozie-5.0.0/core/src/main/java/org/apache/oozie/util/db/FailingConnectionWrapper.java:[24,37]
>  package org.apache.directory.api.util does not exist
> [ERROR] 
> /tmp/oozie-5.0.0/core/src/main/java/org/apache/oozie/util/db/FailingConnectionWrapper.java:[352,41]
>  cannot find symbol
> [ERROR] symbol: variable Strings
> [ERROR] location: class 
> org.apache.oozie.util.db.FailingConnectionWrapper.OozieDmlStatementPredicate
> [ERROR] -> [Help 1]
> [ERROR]
> [ERROR] To see the full stack trace of the errors, re-run Maven with the -e 
> switch.
> [ERROR] Re-run Maven using the -X switch to enable full debug logging.
> [ERROR]
> [ERROR] For more information about the errors and possible 

Failed: OOZIE-3219 PreCommit Build #518

[Apache Jenkins Server]

Jira: https://issues.apache.org/jira/browse/OOZIE-3219
Build: https://builds.apache.org/job/PreCommit-OOZIE-Build/518/

###
## LAST 100 LINES OF THE CONSOLE 
###
[...truncated 1.71 MB...]
[TRACE] New XMLLib present, calling 'xmllint --xpath' to get bug instance counts
[DEBUG] There are no new bugs found in [docs].
[TRACE] New XMLLib present, calling 'xmllint --xpath' to get bug instance counts
[DEBUG] There are no new bugs found in [server].
[INFO] There are no new bugs found totally].
[TRACE] FindBugs diffs checked and reports created
[TRACE] Summary file size is 2177 bytes
[TRACE] Full summary file size is 1314 bytes
[TRACE] File 
[/home/jenkins/jenkins-slave/workspace/PreCommit-OOZIE-Build/test-patch/tmp/FINDBUGS_DIFF/diff/findbugs-diff-0.1.0-all.jar]
 removed
[TRACE] File 
[/home/jenkins/jenkins-slave/workspace/PreCommit-OOZIE-Build/test-patch/tmp/FINDBUGS_DIFF/diff/findbugs-diff-0.1.0-all.jar.md5sum]
 removed
  Running test-patch task BACKWARDS_COMPATIBILITY
  Running test-patch task TESTS
  Running test-patch task DISTRO


Testing JIRA OOZIE-3219

Cleaning local git workspace



+1 PATCH_APPLIES
+1 CLEAN
-1 RAW_PATCH_ANALYSIS
+1 the patch does not introduce any @author tags
+1 the patch does not introduce any tabs
+1 the patch does not introduce any trailing spaces
+1 the patch does not introduce any line longer than 132
-1 the patch does not add/modify any testcase
+1 RAT
+1 the patch does not seem to introduce new RAT warnings
+1 JAVADOC
+1 the patch does not seem to introduce new Javadoc warnings
+1 COMPILE
+1 HEAD compiles
+1 patch compiles
+1 the patch does not seem to introduce new javac warnings
+1 There are no new bugs found in total.
 +1 There are no new bugs found in [webapp].
 +1 There are no new bugs found in [core].
 +1 There are no new bugs found in [tools].
 +1 There are no new bugs found in [sharelib/hive2].
 +1 There are no new bugs found in [sharelib/distcp].
 +1 There are no new bugs found in [sharelib/hcatalog].
 +1 There are no new bugs found in [sharelib/streaming].
 +1 There are no new bugs found in [sharelib/sqoop].
 +1 There are no new bugs found in [sharelib/oozie].
 +1 There are no new bugs found in [sharelib/pig].
 +1 There are no new bugs found in [sharelib/hive].
 +1 There are no new bugs found in [sharelib/spark].
 +1 There are no new bugs found in [client].
 +1 There are no new bugs found in [examples].
 +1 There are no new bugs found in [docs].
 +1 There are no new bugs found in [server].
+1 BACKWARDS_COMPATIBILITY
+1 the patch does not change any JPA Entity/Colum/Basic/Lob/Transient 
annotations
+1 the patch does not modify JPA files
-1 TESTS
Tests run: 2129
Tests failed: 1
Tests errors: 0

The patch failed the following testcases:

testHiveAction(org.apache.oozie.action.hadoop.TestHiveActionExecutor)

Tests failing with errors:


Tests failed at first run:
TestJavaActionExecutor#testCredentialsSkip
TestCoordActionsKillXCommand#testActionKillCommandActionNumbers
TestZKUUIDService#testResetSequence_withMultiThread
TestOozieDBCLI#testOozieDBCLI
For the complete list of flaky tests, see TEST-SUMMARY-FULL files.
+1 DISTRO
+1 distro tarball builds with the patch 


-1 Overall result, please check the reported -1(s)


The full output of the test-patch run is available at

 https://builds.apache.org/job/PreCommit-OOZIE-Build/518/

Adding comment to JIRA
  % Total% Received % Xferd  Average Speed   TimeTime Time  Current
 Dload  Upload   Total   SpentLeft  Speed
  0 00 00 0  0  0 --:--:-- --:--:-- --:--:-- 
0100  33510 0  100  3351  0   3590 --:--:-- --:--:-- --:--:--  
3587{"self":"https://issues.apache.org/jira/rest/api/2/issue/13152326/comment/16465820","id":"16465820","author":{"self":"https://issues.apache.org/jira/rest/api/2/user?username=hadoopqa","name":"hadoopqa","key":"hadoopqa","emailAddress":"blackhole
 at hadoop dot apache dot 
org","avatarUrls":{"48x48":"https://issues.apache.org/jira/secure/useravatar?ownerId=hadoopqa=10393","24x24":"https://issues.apache.org/jira/secure/useravatar?size=small=hadoopqa=10393","16x16":"https://issues.apache.org/jira/secure/useravatar?size=xsmall=hadoopqa=10393","32x32":"https://issues.apache.org/jira/secure/useravatar?size=medium=hadoopqa=10393"},"displayName":"Hadoop
 QA","active":true,"timeZone":"Etc/UTC"},"body":"\nTesting JIRA 
OOZIE-3219\n\nCleaning local git 
workspace\n\n\n\n{color:green}+1 
PATCH_APPLIES{color}\n{color:green}+1 CLEAN{color}\n{color:red}-1 
RAW_PATCH_ANALYSIS{color}\n.{color:green}+1{color} the patch does not 
introduce any @author tags\n.{color:green}+1{color} the patch does not 
introduce any tabs\n.

[jira] [Commented] (OOZIE-3219) Cannot compile with hadoop 3.1.0

[Hadoop QA (JIRA)]

[ 
https://issues.apache.org/jira/browse/OOZIE-3219?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16465820#comment-16465820
 ] 

Hadoop QA commented on OOZIE-3219:
--


Testing JIRA OOZIE-3219

Cleaning local git workspace



{color:green}+1 PATCH_APPLIES{color}
{color:green}+1 CLEAN{color}
{color:red}-1 RAW_PATCH_ANALYSIS{color}
.{color:green}+1{color} the patch does not introduce any @author tags
.{color:green}+1{color} the patch does not introduce any tabs
.{color:green}+1{color} the patch does not introduce any trailing spaces
.{color:green}+1{color} the patch does not introduce any line longer than 
132
.{color:red}-1{color} the patch does not add/modify any testcase
{color:green}+1 RAT{color}
.{color:green}+1{color} the patch does not seem to introduce new RAT 
warnings
{color:green}+1 JAVADOC{color}
.{color:green}+1{color} the patch does not seem to introduce new Javadoc 
warnings
{color:green}+1 COMPILE{color}
.{color:green}+1{color} HEAD compiles
.{color:green}+1{color} patch compiles
.{color:green}+1{color} the patch does not seem to introduce new javac 
warnings
{color:green}+1{color} There are no new bugs found in total.
. {color:green}+1{color} There are no new bugs found in [webapp].
. {color:green}+1{color} There are no new bugs found in [core].
. {color:green}+1{color} There are no new bugs found in [tools].
. {color:green}+1{color} There are no new bugs found in [sharelib/hive2].
. {color:green}+1{color} There are no new bugs found in [sharelib/distcp].
. {color:green}+1{color} There are no new bugs found in [sharelib/hcatalog].
. {color:green}+1{color} There are no new bugs found in [sharelib/streaming].
. {color:green}+1{color} There are no new bugs found in [sharelib/sqoop].
. {color:green}+1{color} There are no new bugs found in [sharelib/oozie].
. {color:green}+1{color} There are no new bugs found in [sharelib/pig].
. {color:green}+1{color} There are no new bugs found in [sharelib/hive].
. {color:green}+1{color} There are no new bugs found in [sharelib/spark].
. {color:green}+1{color} There are no new bugs found in [client].
. {color:green}+1{color} There are no new bugs found in [examples].
. {color:green}+1{color} There are no new bugs found in [docs].
. {color:green}+1{color} There are no new bugs found in [server].
{color:green}+1 BACKWARDS_COMPATIBILITY{color}
.{color:green}+1{color} the patch does not change any JPA 
Entity/Colum/Basic/Lob/Transient annotations
.{color:green}+1{color} the patch does not modify JPA files
{color:red}-1 TESTS{color}
.Tests run: 2129
.Tests failed: 1
.Tests errors: 0

.The patch failed the following testcases:

testHiveAction(org.apache.oozie.action.hadoop.TestHiveActionExecutor)

.Tests failing with errors:


.{color:orange}Tests failed at first run:{color}
TestJavaActionExecutor#testCredentialsSkip
TestCoordActionsKillXCommand#testActionKillCommandActionNumbers
TestZKUUIDService#testResetSequence_withMultiThread
TestOozieDBCLI#testOozieDBCLI
.For the complete list of flaky tests, see TEST-SUMMARY-FULL files.
{color:green}+1 DISTRO{color}
.{color:green}+1{color} distro tarball builds with the patch 


{color:red}*-1 Overall result, please check the reported -1(s)*{color}


The full output of the test-patch run is available at

. https://builds.apache.org/job/PreCommit-OOZIE-Build/518/



> Cannot compile with hadoop 3.1.0
> 
>
> Key: OOZIE-3219
> URL: https://issues.apache.org/jira/browse/OOZIE-3219
> Project: Oozie
>  Issue Type: Bug
>  Components: core
>Affects Versions: 5.0.0
> Environment: {code:java}
> Apache Maven 3.5.3 (3383c37e1f9e9b3bc3df5050c29c8aff9f295297; 
> 2018-02-24T19:49:05Z)
> Maven home: /opt/maven/apache-maven-3.5.3
> Java version: 1.8.0_161, vendor: Oracle Corporation
> Java home: /usr/lib/jvm/java-1.8.0-openjdk-1.8.0.161-0.b14.el7_4.x86_64/jre
> Default locale: en_US, platform encoding: UTF-8
> OS name: "linux", version: "3.10.0-693.21.1.el7.x86_64", arch: "amd64", 
> family: "unix"{code}
> latest 5.0 release bits
>Reporter: Artem Ervits
>Assignee: Artem Ervits
>Priority: Major
> Fix For: 5.1.0
>
> Attachments: OOZIE-3219.00.patch, OOZIE-3219.002.patch
>
>
> command failure on the following, same happens with 3.0.0. Issue does not 
> appear with < 3.0
> {code:java}
> mvn clean install -DskipTests -Dhadoop.version=3.1.0{code}
> {code:java}
> [ERROR] COMPILATION ERROR :
> [INFO] -
> [ERROR] 
> /tmp/oozie-5.0.0/core/src/main/java/org/apache/oozie/util/db/FailingConnectionWrapper.java:[24,37]
>  package org.apache.directory.api.util does not exist
> [ERROR] 
> 

[jira] [Commented] (OOZIE-3232) Oozie may waste up to 25% of the heap due to duplicate strings

[Andras Piros (JIRA)]

[ 
https://issues.apache.org/jira/browse/OOZIE-3232?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16465762#comment-16465762
 ] 

Andras Piros commented on OOZIE-3232:
-

Thanks for the contribution [~mi...@cloudera.com]! Looking into doing some 
performance testing.

> Oozie may waste up to 25% of the heap due to duplicate strings
> --
>
> Key: OOZIE-3232
> URL: https://issues.apache.org/jira/browse/OOZIE-3232
> Project: Oozie
>  Issue Type: Improvement
>  Components: core
>Reporter: Misha Dmitriev
>Assignee: Misha Dmitriev
>Priority: Major
> Attachments: OOZIE-3232-01.patch, jxray-analysis-dup-strings.png
>
>
> I've recently analyzed an Oozie heap dump obtained from a customer's 
> production job, using jxray (www.jxray.com). In this job, Oozie's heap 
> consumption is ~6GB, and it turns out that nearly 25% of it is wasted due to 
> a large number of duplicate strings. The screenshot below, taken from the 
> jxray report, illustrates the problem.
> !jxray-analysis-dup-strings.png|width=638,height=669!
> It turns out that a lot of duplicate strings come from the oozie's own code, 
> i.e. {{org.apache.oozie.*}}. In particular, the top data field wasting memory 
> is {{org.apache.oozie.StringBlob.string}} (wastes 2.6%, or ~160MB). >From the 
> source code of {{StringBlob}} I see that it would be trivial to intern 
> (deduplicate) these strings by adding the call to {{String.intern()}} in the 
> constructor and a few other places. Similarly, various fields of 
> {{org.apache.oozie.WorkflowJobBean}} and {{WorkflowActionBean}} collectively 
> waste a lot of memory, and can be fixed in the same way.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Failed: OOZIE-2629 PreCommit Build #519

[Apache Jenkins Server]

Jira: https://issues.apache.org/jira/browse/OOZIE-2629
Build: https://builds.apache.org/job/PreCommit-OOZIE-Build/519/

###
## LAST 100 LINES OF THE CONSOLE 
###
[...truncated 761.32 KB...]
[INFO] --- maven-compiler-plugin:3.7.0:testCompile (default-testCompile) @ 
oozie-zookeeper-security-tests ---
[INFO] Changes detected - recompiling the module!
[INFO] Compiling 2 source files to 
/home/jenkins/jenkins-slave/workspace/PreCommit-OOZIE-Build/zookeeper-security-tests/target/test-classes
[INFO] 
/home/jenkins/jenkins-slave/workspace/PreCommit-OOZIE-Build/zookeeper-security-tests/src/test/java/org/apache/oozie/util/TestZKUtilsWithSecurity.java:
 Some input files use or override a deprecated API.
[INFO] 
/home/jenkins/jenkins-slave/workspace/PreCommit-OOZIE-Build/zookeeper-security-tests/src/test/java/org/apache/oozie/util/TestZKUtilsWithSecurity.java:
 Recompile with -Xlint:deprecation for details.
[INFO] 
[INFO] >>> findbugs-maven-plugin:3.0.5:check (default-cli) > :findbugs @ 
oozie-zookeeper-security-tests >>>
[INFO] 
[INFO] --- findbugs-maven-plugin:3.0.5:findbugs (findbugs) @ 
oozie-zookeeper-security-tests ---
[INFO] 
[INFO] <<< findbugs-maven-plugin:3.0.5:check (default-cli) < :findbugs @ 
oozie-zookeeper-security-tests <<<
[INFO] 
[INFO] 
[INFO] --- findbugs-maven-plugin:3.0.5:check (default-cli) @ 
oozie-zookeeper-security-tests ---
[INFO] 
[INFO] Reactor Summary:
[INFO] 
[INFO] Apache Oozie Main .. SUCCESS [  4.791 s]
[INFO] Apache Oozie Client  SUCCESS [ 26.588 s]
[INFO] Apache Oozie Share Lib Oozie ... SUCCESS [ 17.776 s]
[INFO] Apache Oozie Share Lib HCatalog  SUCCESS [  8.476 s]
[INFO] Apache Oozie Share Lib Distcp .. SUCCESS [  7.019 s]
[INFO] Apache Oozie Core .. SUCCESS [01:26 min]
[INFO] Apache Oozie Share Lib Streaming ... SUCCESS [  7.698 s]
[INFO] Apache Oozie Share Lib Pig . SUCCESS [  7.886 s]
[INFO] Apache Oozie Share Lib Hive  SUCCESS [  8.517 s]
[INFO] Apache Oozie Share Lib Hive 2 .. SUCCESS [  9.558 s]
[INFO] Apache Oozie Share Lib Sqoop ... SUCCESS [  6.296 s]
[INFO] Apache Oozie Examples .. SUCCESS [  9.104 s]
[INFO] Apache Oozie Share Lib Spark ... SUCCESS [ 10.910 s]
[INFO] Apache Oozie Share Lib . SUCCESS [  0.021 s]
[INFO] Apache Oozie Docs .. SUCCESS [  3.070 s]
[INFO] Apache Oozie WebApp  SUCCESS [  8.854 s]
[INFO] Apache Oozie Tools . SUCCESS [ 13.812 s]
[INFO] Apache Oozie MiniOozie . SUCCESS [  1.900 s]
[INFO] Apache Oozie Server  SUCCESS [ 11.471 s]
[INFO] Apache Oozie Distro  SUCCESS [  2.449 s]
[INFO] Apache Oozie ZooKeeper Security Tests .. SUCCESS [  3.929 s]
[INFO] 
[INFO] BUILD SUCCESS
[INFO] 
[INFO] Total time: 04:18 min
[INFO] Finished at: 2018-05-07T10:36:57Z
[INFO] Final Memory: 169M/1260M
[INFO] 
[TRACE] FindBugs output in HEAD verified and saved
  Running test-patch task BACKWARDS_COMPATIBILITY
  Running test-patch task TESTS
  Running test-patch task DISTRO

Applying patch

Checking patch 
a/tools/src/main/java/org/apache/oozie/tools/OozieSharelibCLI.java => 
b/tools/src/main/java/org/apache/oozie/tools/OozieSharelibCLI.java...
error: a/tools/src/main/java/org/apache/oozie/tools/OozieSharelibCLI.java: No 
such file or directory
Checking patch 
tools/src/main/java/org/apache/oozie/tools/OozieSharelibCLI.java...
error: while searching for:
String hdfsUri = command.getCommandLine().getOptionValue(FS_OPT);
Path srcPath = new Path(srcFile.toString());

Services services = new Services();
services.getConf().set(Services.CONF_SERVICE_CLASSES,
"org.apache.oozie.service.LiteWorkflowAppService, 
org.apache.oozie.service.HadoopAccessorService");
services.getConf().set(Services.CONF_SERVICE_EXT_CLASSES, "");
services.init();
WorkflowAppService lwas = services.get(WorkflowAppService.class);
HadoopAccessorService has = 
services.get(HadoopAccessorService.class);
Path dstPath = lwas.getSystemLibPath();

if (sharelibAction.equals(CREATE_CMD) || 
sharelibAction.equals(UPGRADE_CMD)){

[jira] [Commented] (OOZIE-2629) Get OozieShareLibCLI to perform a final rename to destPath when creating a Sharelib

[Hadoop QA (JIRA)]

[ 
https://issues.apache.org/jira/browse/OOZIE-2629?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16465756#comment-16465756
 ] 

Hadoop QA commented on OOZIE-2629:
--


Testing JIRA OOZIE-2629

Cleaning local git workspace



{color:red}-1{color} Patch failed to apply to head of branch




> Get OozieShareLibCLI to perform a final rename to destPath when creating a 
> Sharelib
> ---
>
> Key: OOZIE-2629
> URL: https://issues.apache.org/jira/browse/OOZIE-2629
> Project: Oozie
>  Issue Type: Improvement
>Affects Versions: 4.3.0
>Reporter: Yishan Yang
>Priority: Minor
> Fix For: trunk
>
> Attachments: OOZIE-2629-1.patch, OOZIE-2629-2.patch, 
> OOZIE-2629-3.patch, OOZIE-2629-4.patch, OOZIE-2629-5.patch
>
>
> Change the OozieShareLibCLI to perform a final 'rename' to avoid a condition 
> where 'half' of the sharelib could exist during sharelib installation, if 
> Oozie attempts to run a workflow at the same time as the sharelib 
> installation. Oozie will only start if the sharelib exists, but this by 
> itself is not enough since the sharelib could be in an unusable state.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (OOZIE-2629) Get OozieShareLibCLI to perform a final rename to destPath when creating a Sharelib

[Hadoop QA (JIRA)]

[ 
https://issues.apache.org/jira/browse/OOZIE-2629?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16465749#comment-16465749
 ] 

Hadoop QA commented on OOZIE-2629:
--

PreCommit-OOZIE-Build started


> Get OozieShareLibCLI to perform a final rename to destPath when creating a 
> Sharelib
> ---
>
> Key: OOZIE-2629
> URL: https://issues.apache.org/jira/browse/OOZIE-2629
> Project: Oozie
>  Issue Type: Improvement
>Affects Versions: 4.3.0
>Reporter: Yishan Yang
>Priority: Minor
> Fix For: trunk
>
> Attachments: OOZIE-2629-1.patch, OOZIE-2629-2.patch, 
> OOZIE-2629-3.patch, OOZIE-2629-4.patch, OOZIE-2629-5.patch
>
>
> Change the OozieShareLibCLI to perform a final 'rename' to avoid a condition 
> where 'half' of the sharelib could exist during sharelib installation, if 
> Oozie attempts to run a workflow at the same time as the sharelib 
> installation. Oozie will only start if the sharelib exists, but this by 
> itself is not enough since the sharelib could be in an unusable state.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (OOZIE-3230) TestPigActionExecutor in oozie-sharelib fail with junit.framework.AssertionFailedError

[Andras Piros (JIRA)]

[ 
https://issues.apache.org/jira/browse/OOZIE-3230?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16465744#comment-16465744
 ] 

Andras Piros commented on OOZIE-3230:
-

[~alishap] did you set {{JAVA_HOME}} environment variable?

I cannot reproduce:
{noformat}
[INFO] ---
[INFO]  T E S T S
[INFO] ---
[INFO] Running org.apache.oozie.action.hadoop.TestPigActionExecutor
[INFO] Tests run: 10, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 280.274 
s - in org.apache.oozie.action.hadoop.TestPigActionExecutor
[INFO] 
[INFO] Results:
[INFO] 
[INFO] Tests run: 10, Failures: 0, Errors: 0, Skipped: 0
{noformat}

> TestPigActionExecutor in oozie-sharelib fail with 
> junit.framework.AssertionFailedError
> --
>
> Key: OOZIE-3230
> URL: https://issues.apache.org/jira/browse/OOZIE-3230
> Project: Oozie
>  Issue Type: Bug
> Environment: uname -a
> Linux hj-ibmibm9350 4.4.0-112-generic #135-Ubuntu SMP Fri Jan 19 11:48:36 UTC 
> 2018 x86_64 x86_64 x86_64 GNU/Linux
>Reporter: Alisha Prabhu
>Priority: Major
>
> Maven command used is :  mvn -Dtest=TestPigActionExecutor test
> {code:java}
> [ERROR] Failures:
> [ERROR] 
> TestPigActionExecutor.testExecutionStats:190->XTestCase.waitUntilYarnAppDoneAndAssertSuccess:1333
>  YARN App state for app application_1524809863775_0007 expected: 
> but was:
> [ERROR] 
> TestPigActionExecutor.testExecutionStatsWithMaxStatsSizeLimit:271->XTestCase.waitUntilYarnAppDoneAndAssertSuccess:1333
>  YARN App state for app application_1524809863775_0002 expected: 
> but was:
> [ERROR] 
> TestPigActionExecutor.testExecutionStatsWithRetrieveStatsFalse:292->XTestCase.waitUntilYarnAppDoneAndAssertSuccess:1333
>  YARN App state for app application_1524809863775_0005 expected: 
> but was:
> [ERROR] 
> TestPigActionExecutor.testExternalChildIds:242->XTestCase.waitUntilYarnAppDoneAndAssertSuccess:1333
>  YARN App state for app application_1524809863775_0006 expected: 
> but was:
> [ERROR] 
> TestPigActionExecutor.testPig:319->_testSubmit:153->XTestCase.waitUntilYarnAppDoneAndAssertSuccess:1333
>  YARN App state for app application_1524809863775_0001 expected: 
> but was:
> [ERROR] 
> TestPigActionExecutor.testPigError:325->_testSubmit:153->XTestCase.waitUntilYarnAppDoneAndAssertSuccess:1333
>  YARN App state for app application_1524809863775_0003 expected: 
> but was:
> [ERROR] 
> TestPigActionExecutor.testUdfPig:387->_testSubmit:153->XTestCase.waitUntilYarnAppDoneAndAssertSuccess:1333
>  YARN App state for app application_1524809863775_0008 expected: 
> but was:
> [INFO]
> [ERROR] Tests run: 10, Failures: 7, Errors: 0, Skipped: 0
> {code}



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (OOZIE-3217) Enable definition of admin users using oozie-site.xml

[Andras Piros (JIRA)]

[ 
https://issues.apache.org/jira/browse/OOZIE-3217?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16465735#comment-16465735
 ] 

Andras Piros commented on OOZIE-3217:
-

[~orova] failing tests seem unrelated.

Can you please fix following pre-commit build errors:
{quote}
-1 RAW_PATCH_ANALYSIS
. -1 the patch contains 3 line(s) longer than 132 characters
. -1 There are [3] new bugs found below threshold in [core] that must be fixed.
. You can find the FindBugs diff here (look for the red and orange ones): 
core/findbugs-new.html
. The most important FindBugs errors are:
. At AuthorizationService.java:[line 192]: Found reliance on default encoding 
in org.apache.oozie.service.AuthorizationService.loadAdminUsersFromFile(): new 
java.io.InputStreamReader(InputStream)
. At AuthorizationService.java:[line 192]: 
org.apache.oozie.service.AuthorizationService.loadAdminUsersFromFile() may fail 
to close stream
. At AuthorizationService.java:[line 189]: 
java/io/File.(Ljava/lang/String;Ljava/lang/String;)V reads a file whose 
location might be specified by user input
. At AuthorizationService.java:[line 187]
{quote}

> Enable definition of admin users using oozie-site.xml
> -
>
> Key: OOZIE-3217
> URL: https://issues.apache.org/jira/browse/OOZIE-3217
> Project: Oozie
>  Issue Type: Improvement
>Affects Versions: 5.0.0
>Reporter: Peter Orova
>Assignee: Peter Orova
>Priority: Minor
> Attachments: OOZIE-3217.001.patch, OOZIE-3217.002.patch, 
> OOZIE-3217.003.patch
>
>
> Currently the list of admin users is defined in the {{adminusers.txt}} file 
> hard coded to the Oozie config dir. For a more streamlined solution, we could 
> define the list of admin users via {{oozie-site.xml}} by introducing the 
> following configuration, which receives the comma separated values of the 
> users that are admins.
> {{oozie.service.AuthorizationService.admin.users}}



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (OOZIE-3234) Revise TestAuthorizationService

[Andras Piros (JIRA)]

[ 
https://issues.apache.org/jira/browse/OOZIE-3234?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16465730#comment-16465730
 ] 

Andras Piros commented on OOZIE-3234:
-

I like the idea [~orova]!

> Revise TestAuthorizationService
> ---
>
> Key: OOZIE-3234
> URL: https://issues.apache.org/jira/browse/OOZIE-3234
> Project: Oozie
>  Issue Type: Improvement
>  Components: tests
>Affects Versions: 5.0.0
>Reporter: Peter Orova
>Assignee: Peter Orova
>Priority: Minor
>
> The {{TestAuthorizationService}}'s maintainability is low due to
>  * the many magic boolean variables
>  * the init methods having 3+ parameters
> One way this could be addressed is to introduce a simplified version of 
> Builder pattern via a private nested class, to replace the functionality of 
> the {{init}} methods.
> This approach would allow for a much more readable, flexible and maintainable 
> code.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Comment Edited] (OOZIE-3227) Eliminate duplicated dependencies from distributed cache

[Andras Piros (JIRA)]

[ 
https://issues.apache.org/jira/browse/OOZIE-3227?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16465677#comment-16465677
 ] 

Andras Piros edited comment on OOZIE-3227 at 5/7/18 10:10 AM:
--

[~dionusos] I cannot make Oozie code compile w/ Hadoop 3, and hence, cannot 
reproduce. Waiting for OOZIE-3219 to be resolved.


was (Author: andras.piros):
[~dionusos] I cannot make Oozie code compile w/ Hadoop 3, and hence, cannot 
reproduce. If you have a patch for that, could you please update OOZIE-3219? 
Thanks!

> Eliminate duplicated dependencies from distributed cache
> 
>
> Key: OOZIE-3227
> URL: https://issues.apache.org/jira/browse/OOZIE-3227
> Project: Oozie
>  Issue Type: Sub-task
>  Components: core
>Reporter: Denes Bodo
>Assignee: Denes Bodo
>Priority: Major
>
> Using Hadoop 3 it is not allowed to have multiple dependencies with same file 
> names on the list of *mapreduce.job.cache.files*.
> The issue occurs when I have the same file name on multiple sharelib folders 
> and/or my application's lib folder. This can be avoided but not easy all the 
> time.
> I suggest to remove the duplicates from this list.
> A quick workaround for the source code in JavaActionExecutor is like:
> {code}
> removeDuplicatedDependencies(launcherJobConf, 
> "mapreduce.job.cache.files");
> removeDuplicatedDependencies(launcherJobConf, 
> "mapreduce.job.cache.archives");
> ..
> private void removeDuplicatedDependencies(JobConf conf, String key) {
> final Map nameToPath = new HashMap<>();
> StringBuilder uniqList = new StringBuilder();
> for(String dependency: conf.get(key).split(",")) {
> final String[] arr = dependency.split("/");
> final String dependencyName = arr[arr.length - 1];
> if(nameToPath.containsKey(dependencyName)) {
> LOG.warn(dependencyName + " [" + dependency + "] is already 
> defined in " + key + ". Skipping...");
> } else {
> nameToPath.put(dependencyName, dependency);
> uniqList.append(dependency).append(",");
> }
> }
> uniqList.setLength(uniqList.length() - 1);
> conf.set(key, uniqList.toString());
> }
> {code}
> Other way is to eliminate the deprecated 
> *org.apache.hadoop.filecache.DistributedCache*.
> I am going to have a deeper understanding how we should use distributed cache 
> and all the comments are welcome.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (OOZIE-3219) Cannot compile with hadoop 3.1.0

[Hadoop QA (JIRA)]

[ 
https://issues.apache.org/jira/browse/OOZIE-3219?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16465727#comment-16465727
 ] 

Hadoop QA commented on OOZIE-3219:
--

PreCommit-OOZIE-Build started


> Cannot compile with hadoop 3.1.0
> 
>
> Key: OOZIE-3219
> URL: https://issues.apache.org/jira/browse/OOZIE-3219
> Project: Oozie
>  Issue Type: Bug
>  Components: core
>Affects Versions: 5.0.0
> Environment: {code:java}
> Apache Maven 3.5.3 (3383c37e1f9e9b3bc3df5050c29c8aff9f295297; 
> 2018-02-24T19:49:05Z)
> Maven home: /opt/maven/apache-maven-3.5.3
> Java version: 1.8.0_161, vendor: Oracle Corporation
> Java home: /usr/lib/jvm/java-1.8.0-openjdk-1.8.0.161-0.b14.el7_4.x86_64/jre
> Default locale: en_US, platform encoding: UTF-8
> OS name: "linux", version: "3.10.0-693.21.1.el7.x86_64", arch: "amd64", 
> family: "unix"{code}
> latest 5.0 release bits
>Reporter: Artem Ervits
>Assignee: Artem Ervits
>Priority: Major
> Fix For: 5.1.0
>
> Attachments: OOZIE-3219.00.patch, OOZIE-3219.002.patch
>
>
> command failure on the following, same happens with 3.0.0. Issue does not 
> appear with < 3.0
> {code:java}
> mvn clean install -DskipTests -Dhadoop.version=3.1.0{code}
> {code:java}
> [ERROR] COMPILATION ERROR :
> [INFO] -
> [ERROR] 
> /tmp/oozie-5.0.0/core/src/main/java/org/apache/oozie/util/db/FailingConnectionWrapper.java:[24,37]
>  package org.apache.directory.api.util does not exist
> [ERROR] 
> /tmp/oozie-5.0.0/core/src/main/java/org/apache/oozie/util/db/FailingConnectionWrapper.java:[352,41]
>  cannot find symbol
> symbol: variable Strings
> location: class 
> org.apache.oozie.util.db.FailingConnectionWrapper.OozieDmlStatementPredicate
> [INFO] 2 errors
> [INFO] -
> [INFO] 
> 
> [INFO] Reactor Summary:
> [INFO]
> [INFO] Apache Oozie Main 5.0.0  SUCCESS [ 3.768 s]
> [INFO] Apache Oozie Client  SUCCESS [ 30.924 
> s]
> [INFO] Apache Oozie Share Lib Oozie ... SUCCESS [ 20.904 
> s]
> [INFO] Apache Oozie Share Lib HCatalog  SUCCESS [ 9.150 s]
> [INFO] Apache Oozie Share Lib Distcp .. SUCCESS [ 7.938 s]
> [INFO] Apache Oozie Core .. FAILURE [ 13.929 
> s]
> [INFO] Apache Oozie Share Lib Streaming ... SKIPPED
> [INFO] Apache Oozie Share Lib Pig . SKIPPED
> [INFO] Apache Oozie Share Lib Hive  SKIPPED
> [INFO] Apache Oozie Share Lib Hive 2 .. SKIPPED
> [INFO] Apache Oozie Share Lib Sqoop ... SKIPPED
> [INFO] Apache Oozie Examples .. SKIPPED
> [INFO] Apache Oozie Share Lib Spark ... SKIPPED
> [INFO] Apache Oozie Share Lib . SKIPPED
> [INFO] Apache Oozie Docs .. SKIPPED
> [INFO] Apache Oozie WebApp  SKIPPED
> [INFO] Apache Oozie Tools . SKIPPED
> [INFO] Apache Oozie MiniOozie . SKIPPED
> [INFO] Apache Oozie Server  SKIPPED
> [INFO] Apache Oozie Distro  SKIPPED
> [INFO] Apache Oozie ZooKeeper Security Tests 5.0.0  SKIPPED
> [INFO] 
> 
> [INFO] BUILD FAILURE
> [INFO] 
> 
> [INFO] Total time: 01:28 min
> [INFO] Finished at: 2018-04-13T14:03:17Z
> [INFO] 
> 
> [ERROR] Failed to execute goal 
> org.apache.maven.plugins:maven-compiler-plugin:3.7.0:compile 
> (default-compile) on project oozie-core: Compilation failure: Compilation 
> failure:
> [ERROR] 
> /tmp/oozie-5.0.0/core/src/main/java/org/apache/oozie/util/db/FailingConnectionWrapper.java:[24,37]
>  package org.apache.directory.api.util does not exist
> [ERROR] 
> /tmp/oozie-5.0.0/core/src/main/java/org/apache/oozie/util/db/FailingConnectionWrapper.java:[352,41]
>  cannot find symbol
> [ERROR] symbol: variable Strings
> [ERROR] location: class 
> org.apache.oozie.util.db.FailingConnectionWrapper.OozieDmlStatementPredicate
> [ERROR] -> [Help 1]
> [ERROR]
> [ERROR] To see the full stack trace of the errors, re-run Maven with the -e 
> switch.
> [ERROR] Re-run Maven using the -X switch to enable full debug logging.
> [ERROR]
> [ERROR] For more information about the errors and possible solutions, please 
> read the following articles:
> 

[jira] [Commented] (OOZIE-3219) Cannot compile with hadoop 3.1.0

[Andras Piros (JIRA)]

[ 
https://issues.apache.org/jira/browse/OOZIE-3219?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16465722#comment-16465722
 ] 

Andras Piros commented on OOZIE-3219:
-

[~gezapeti] [~dbist13] with the latest patch, Oozie can compile using 
{{hadoop.version=2.6.0}} and {{hadoop.version=3.1.0}}.

> Cannot compile with hadoop 3.1.0
> 
>
> Key: OOZIE-3219
> URL: https://issues.apache.org/jira/browse/OOZIE-3219
> Project: Oozie
>  Issue Type: Bug
>  Components: core
>Affects Versions: 5.0.0
> Environment: {code:java}
> Apache Maven 3.5.3 (3383c37e1f9e9b3bc3df5050c29c8aff9f295297; 
> 2018-02-24T19:49:05Z)
> Maven home: /opt/maven/apache-maven-3.5.3
> Java version: 1.8.0_161, vendor: Oracle Corporation
> Java home: /usr/lib/jvm/java-1.8.0-openjdk-1.8.0.161-0.b14.el7_4.x86_64/jre
> Default locale: en_US, platform encoding: UTF-8
> OS name: "linux", version: "3.10.0-693.21.1.el7.x86_64", arch: "amd64", 
> family: "unix"{code}
> latest 5.0 release bits
>Reporter: Artem Ervits
>Assignee: Artem Ervits
>Priority: Major
> Fix For: 5.1.0
>
> Attachments: OOZIE-3219.00.patch, OOZIE-3219.002.patch
>
>
> command failure on the following, same happens with 3.0.0. Issue does not 
> appear with < 3.0
> {code:java}
> mvn clean install -DskipTests -Dhadoop.version=3.1.0{code}
> {code:java}
> [ERROR] COMPILATION ERROR :
> [INFO] -
> [ERROR] 
> /tmp/oozie-5.0.0/core/src/main/java/org/apache/oozie/util/db/FailingConnectionWrapper.java:[24,37]
>  package org.apache.directory.api.util does not exist
> [ERROR] 
> /tmp/oozie-5.0.0/core/src/main/java/org/apache/oozie/util/db/FailingConnectionWrapper.java:[352,41]
>  cannot find symbol
> symbol: variable Strings
> location: class 
> org.apache.oozie.util.db.FailingConnectionWrapper.OozieDmlStatementPredicate
> [INFO] 2 errors
> [INFO] -
> [INFO] 
> 
> [INFO] Reactor Summary:
> [INFO]
> [INFO] Apache Oozie Main 5.0.0  SUCCESS [ 3.768 s]
> [INFO] Apache Oozie Client  SUCCESS [ 30.924 
> s]
> [INFO] Apache Oozie Share Lib Oozie ... SUCCESS [ 20.904 
> s]
> [INFO] Apache Oozie Share Lib HCatalog  SUCCESS [ 9.150 s]
> [INFO] Apache Oozie Share Lib Distcp .. SUCCESS [ 7.938 s]
> [INFO] Apache Oozie Core .. FAILURE [ 13.929 
> s]
> [INFO] Apache Oozie Share Lib Streaming ... SKIPPED
> [INFO] Apache Oozie Share Lib Pig . SKIPPED
> [INFO] Apache Oozie Share Lib Hive  SKIPPED
> [INFO] Apache Oozie Share Lib Hive 2 .. SKIPPED
> [INFO] Apache Oozie Share Lib Sqoop ... SKIPPED
> [INFO] Apache Oozie Examples .. SKIPPED
> [INFO] Apache Oozie Share Lib Spark ... SKIPPED
> [INFO] Apache Oozie Share Lib . SKIPPED
> [INFO] Apache Oozie Docs .. SKIPPED
> [INFO] Apache Oozie WebApp  SKIPPED
> [INFO] Apache Oozie Tools . SKIPPED
> [INFO] Apache Oozie MiniOozie . SKIPPED
> [INFO] Apache Oozie Server  SKIPPED
> [INFO] Apache Oozie Distro  SKIPPED
> [INFO] Apache Oozie ZooKeeper Security Tests 5.0.0  SKIPPED
> [INFO] 
> 
> [INFO] BUILD FAILURE
> [INFO] 
> 
> [INFO] Total time: 01:28 min
> [INFO] Finished at: 2018-04-13T14:03:17Z
> [INFO] 
> 
> [ERROR] Failed to execute goal 
> org.apache.maven.plugins:maven-compiler-plugin:3.7.0:compile 
> (default-compile) on project oozie-core: Compilation failure: Compilation 
> failure:
> [ERROR] 
> /tmp/oozie-5.0.0/core/src/main/java/org/apache/oozie/util/db/FailingConnectionWrapper.java:[24,37]
>  package org.apache.directory.api.util does not exist
> [ERROR] 
> /tmp/oozie-5.0.0/core/src/main/java/org/apache/oozie/util/db/FailingConnectionWrapper.java:[352,41]
>  cannot find symbol
> [ERROR] symbol: variable Strings
> [ERROR] location: class 
> org.apache.oozie.util.db.FailingConnectionWrapper.OozieDmlStatementPredicate
> [ERROR] -> [Help 1]
> [ERROR]
> [ERROR] To see the full stack trace of the errors, re-run Maven with the -e 
> switch.
> [ERROR] Re-run Maven using the -X switch to enable full debug logging.
> [ERROR]
> [ERROR] 

[jira] [Updated] (OOZIE-3219) Cannot compile with hadoop 3.1.0

[Andras Piros (JIRA)]

 [ 
https://issues.apache.org/jira/browse/OOZIE-3219?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Andras Piros updated OOZIE-3219:

Attachment: OOZIE-3219.002.patch

> Cannot compile with hadoop 3.1.0
> 
>
> Key: OOZIE-3219
> URL: https://issues.apache.org/jira/browse/OOZIE-3219
> Project: Oozie
>  Issue Type: Bug
>  Components: core
>Affects Versions: 5.0.0
> Environment: {code:java}
> Apache Maven 3.5.3 (3383c37e1f9e9b3bc3df5050c29c8aff9f295297; 
> 2018-02-24T19:49:05Z)
> Maven home: /opt/maven/apache-maven-3.5.3
> Java version: 1.8.0_161, vendor: Oracle Corporation
> Java home: /usr/lib/jvm/java-1.8.0-openjdk-1.8.0.161-0.b14.el7_4.x86_64/jre
> Default locale: en_US, platform encoding: UTF-8
> OS name: "linux", version: "3.10.0-693.21.1.el7.x86_64", arch: "amd64", 
> family: "unix"{code}
> latest 5.0 release bits
>Reporter: Artem Ervits
>Assignee: Artem Ervits
>Priority: Major
> Fix For: 5.1.0
>
> Attachments: OOZIE-3219.00.patch, OOZIE-3219.002.patch
>
>
> command failure on the following, same happens with 3.0.0. Issue does not 
> appear with < 3.0
> {code:java}
> mvn clean install -DskipTests -Dhadoop.version=3.1.0{code}
> {code:java}
> [ERROR] COMPILATION ERROR :
> [INFO] -
> [ERROR] 
> /tmp/oozie-5.0.0/core/src/main/java/org/apache/oozie/util/db/FailingConnectionWrapper.java:[24,37]
>  package org.apache.directory.api.util does not exist
> [ERROR] 
> /tmp/oozie-5.0.0/core/src/main/java/org/apache/oozie/util/db/FailingConnectionWrapper.java:[352,41]
>  cannot find symbol
> symbol: variable Strings
> location: class 
> org.apache.oozie.util.db.FailingConnectionWrapper.OozieDmlStatementPredicate
> [INFO] 2 errors
> [INFO] -
> [INFO] 
> 
> [INFO] Reactor Summary:
> [INFO]
> [INFO] Apache Oozie Main 5.0.0  SUCCESS [ 3.768 s]
> [INFO] Apache Oozie Client  SUCCESS [ 30.924 
> s]
> [INFO] Apache Oozie Share Lib Oozie ... SUCCESS [ 20.904 
> s]
> [INFO] Apache Oozie Share Lib HCatalog  SUCCESS [ 9.150 s]
> [INFO] Apache Oozie Share Lib Distcp .. SUCCESS [ 7.938 s]
> [INFO] Apache Oozie Core .. FAILURE [ 13.929 
> s]
> [INFO] Apache Oozie Share Lib Streaming ... SKIPPED
> [INFO] Apache Oozie Share Lib Pig . SKIPPED
> [INFO] Apache Oozie Share Lib Hive  SKIPPED
> [INFO] Apache Oozie Share Lib Hive 2 .. SKIPPED
> [INFO] Apache Oozie Share Lib Sqoop ... SKIPPED
> [INFO] Apache Oozie Examples .. SKIPPED
> [INFO] Apache Oozie Share Lib Spark ... SKIPPED
> [INFO] Apache Oozie Share Lib . SKIPPED
> [INFO] Apache Oozie Docs .. SKIPPED
> [INFO] Apache Oozie WebApp  SKIPPED
> [INFO] Apache Oozie Tools . SKIPPED
> [INFO] Apache Oozie MiniOozie . SKIPPED
> [INFO] Apache Oozie Server  SKIPPED
> [INFO] Apache Oozie Distro  SKIPPED
> [INFO] Apache Oozie ZooKeeper Security Tests 5.0.0  SKIPPED
> [INFO] 
> 
> [INFO] BUILD FAILURE
> [INFO] 
> 
> [INFO] Total time: 01:28 min
> [INFO] Finished at: 2018-04-13T14:03:17Z
> [INFO] 
> 
> [ERROR] Failed to execute goal 
> org.apache.maven.plugins:maven-compiler-plugin:3.7.0:compile 
> (default-compile) on project oozie-core: Compilation failure: Compilation 
> failure:
> [ERROR] 
> /tmp/oozie-5.0.0/core/src/main/java/org/apache/oozie/util/db/FailingConnectionWrapper.java:[24,37]
>  package org.apache.directory.api.util does not exist
> [ERROR] 
> /tmp/oozie-5.0.0/core/src/main/java/org/apache/oozie/util/db/FailingConnectionWrapper.java:[352,41]
>  cannot find symbol
> [ERROR] symbol: variable Strings
> [ERROR] location: class 
> org.apache.oozie.util.db.FailingConnectionWrapper.OozieDmlStatementPredicate
> [ERROR] -> [Help 1]
> [ERROR]
> [ERROR] To see the full stack trace of the errors, re-run Maven with the -e 
> switch.
> [ERROR] Re-run Maven using the -X switch to enable full debug logging.
> [ERROR]
> [ERROR] For more information about the errors and possible solutions, please 
> read the following articles:
> [ERROR] [Help 1] 
> 

[jira] [Updated] (OOZIE-3219) Cannot compile with hadoop 3.1.0

[Andras Piros (JIRA)]

 [ 
https://issues.apache.org/jira/browse/OOZIE-3219?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Andras Piros updated OOZIE-3219:

Attachment: (was: OOZIE-3219.002.patch)

> Cannot compile with hadoop 3.1.0
> 
>
> Key: OOZIE-3219
> URL: https://issues.apache.org/jira/browse/OOZIE-3219
> Project: Oozie
>  Issue Type: Bug
>  Components: core
>Affects Versions: 5.0.0
> Environment: {code:java}
> Apache Maven 3.5.3 (3383c37e1f9e9b3bc3df5050c29c8aff9f295297; 
> 2018-02-24T19:49:05Z)
> Maven home: /opt/maven/apache-maven-3.5.3
> Java version: 1.8.0_161, vendor: Oracle Corporation
> Java home: /usr/lib/jvm/java-1.8.0-openjdk-1.8.0.161-0.b14.el7_4.x86_64/jre
> Default locale: en_US, platform encoding: UTF-8
> OS name: "linux", version: "3.10.0-693.21.1.el7.x86_64", arch: "amd64", 
> family: "unix"{code}
> latest 5.0 release bits
>Reporter: Artem Ervits
>Assignee: Artem Ervits
>Priority: Major
> Fix For: 5.1.0
>
> Attachments: OOZIE-3219.00.patch
>
>
> command failure on the following, same happens with 3.0.0. Issue does not 
> appear with < 3.0
> {code:java}
> mvn clean install -DskipTests -Dhadoop.version=3.1.0{code}
> {code:java}
> [ERROR] COMPILATION ERROR :
> [INFO] -
> [ERROR] 
> /tmp/oozie-5.0.0/core/src/main/java/org/apache/oozie/util/db/FailingConnectionWrapper.java:[24,37]
>  package org.apache.directory.api.util does not exist
> [ERROR] 
> /tmp/oozie-5.0.0/core/src/main/java/org/apache/oozie/util/db/FailingConnectionWrapper.java:[352,41]
>  cannot find symbol
> symbol: variable Strings
> location: class 
> org.apache.oozie.util.db.FailingConnectionWrapper.OozieDmlStatementPredicate
> [INFO] 2 errors
> [INFO] -
> [INFO] 
> 
> [INFO] Reactor Summary:
> [INFO]
> [INFO] Apache Oozie Main 5.0.0  SUCCESS [ 3.768 s]
> [INFO] Apache Oozie Client  SUCCESS [ 30.924 
> s]
> [INFO] Apache Oozie Share Lib Oozie ... SUCCESS [ 20.904 
> s]
> [INFO] Apache Oozie Share Lib HCatalog  SUCCESS [ 9.150 s]
> [INFO] Apache Oozie Share Lib Distcp .. SUCCESS [ 7.938 s]
> [INFO] Apache Oozie Core .. FAILURE [ 13.929 
> s]
> [INFO] Apache Oozie Share Lib Streaming ... SKIPPED
> [INFO] Apache Oozie Share Lib Pig . SKIPPED
> [INFO] Apache Oozie Share Lib Hive  SKIPPED
> [INFO] Apache Oozie Share Lib Hive 2 .. SKIPPED
> [INFO] Apache Oozie Share Lib Sqoop ... SKIPPED
> [INFO] Apache Oozie Examples .. SKIPPED
> [INFO] Apache Oozie Share Lib Spark ... SKIPPED
> [INFO] Apache Oozie Share Lib . SKIPPED
> [INFO] Apache Oozie Docs .. SKIPPED
> [INFO] Apache Oozie WebApp  SKIPPED
> [INFO] Apache Oozie Tools . SKIPPED
> [INFO] Apache Oozie MiniOozie . SKIPPED
> [INFO] Apache Oozie Server  SKIPPED
> [INFO] Apache Oozie Distro  SKIPPED
> [INFO] Apache Oozie ZooKeeper Security Tests 5.0.0  SKIPPED
> [INFO] 
> 
> [INFO] BUILD FAILURE
> [INFO] 
> 
> [INFO] Total time: 01:28 min
> [INFO] Finished at: 2018-04-13T14:03:17Z
> [INFO] 
> 
> [ERROR] Failed to execute goal 
> org.apache.maven.plugins:maven-compiler-plugin:3.7.0:compile 
> (default-compile) on project oozie-core: Compilation failure: Compilation 
> failure:
> [ERROR] 
> /tmp/oozie-5.0.0/core/src/main/java/org/apache/oozie/util/db/FailingConnectionWrapper.java:[24,37]
>  package org.apache.directory.api.util does not exist
> [ERROR] 
> /tmp/oozie-5.0.0/core/src/main/java/org/apache/oozie/util/db/FailingConnectionWrapper.java:[352,41]
>  cannot find symbol
> [ERROR] symbol: variable Strings
> [ERROR] location: class 
> org.apache.oozie.util.db.FailingConnectionWrapper.OozieDmlStatementPredicate
> [ERROR] -> [Help 1]
> [ERROR]
> [ERROR] To see the full stack trace of the errors, re-run Maven with the -e 
> switch.
> [ERROR] Re-run Maven using the -X switch to enable full debug logging.
> [ERROR]
> [ERROR] For more information about the errors and possible solutions, please 
> read the following articles:
> [ERROR] [Help 1] 
> 

[jira] [Resolved] (OOZIE-3220) oozie sharelib pig fails when compiled with hadoop 3

[Andras Piros (JIRA)]

 [ 
https://issues.apache.org/jira/browse/OOZIE-3220?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Andras Piros resolved OOZIE-3220.
-
Resolution: Duplicate

Will solve in OOZIE-3219.

> oozie sharelib pig fails when compiled with hadoop 3
> 
>
> Key: OOZIE-3220
> URL: https://issues.apache.org/jira/browse/OOZIE-3220
> Project: Oozie
>  Issue Type: Bug
>Affects Versions: 5.0.0
> Environment: Oozie 5.0 release
> {code:java}
> Apache Maven 3.5.3 (3383c37e1f9e9b3bc3df5050c29c8aff9f295297; 
> 2018-02-24T19:49:05Z)
> Maven home: /opt/maven/apache-maven-3.5.3
> Java version: 1.8.0_161, vendor: Oracle Corporation
> Java home: /usr/lib/jvm/java-1.8.0-openjdk-1.8.0.161-0.b14.el7_4.x86_64/jre
> Default locale: en_US, platform encoding: UTF-8
> OS name: "linux", version: "3.10.0-693.21.1.el7.x86_64", arch: "amd64", 
> family: "unix"{code}
>Reporter: Artem Ervits
>Assignee: Artem Ervits
>Priority: Major
>
> {code:java}
> mvn clean install -DskipTests -Dhadoop.version=3.1.0{code}
> {code:java}
> [ERROR] COMPILATION ERROR :
> [INFO] -
> [ERROR] 
> /tmp/oozie-5.0.0/sharelib/pig/src/main/java/org/apache/oozie/action/hadoop/PigMain.java:[30,30]
>  package org.apache.hadoop.conf does not exist
> [ERROR] 
> /tmp/oozie-5.0.0/sharelib/pig/src/main/java/org/apache/oozie/action/hadoop/PigMain.java:[31,28]
>  cannot find symbol
> symbol: class Path
> location: package org.apache.hadoop.fs
> [ERROR] 
> /tmp/oozie-5.0.0/sharelib/pig/src/main/java/org/apache/oozie/action/hadoop/OoziePigStats.java:[142,37]
>  cannot access org.apache.hadoop.io.Writable
> class file for org.apache.hadoop.io.Writable not found
> [ERROR] 
> /tmp/oozie-5.0.0/sharelib/pig/src/main/java/org/apache/oozie/action/hadoop/PigMain.java:[87,9]
>  cannot find symbol
> symbol: class Configuration
> location: class org.apache.oozie.action.hadoop.PigMain
> [ERROR] 
> /tmp/oozie-5.0.0/sharelib/pig/src/main/java/org/apache/oozie/action/hadoop/PigMain.java:[87,40]
>  cannot find symbol
> symbol: class Configuration
> location: class org.apache.oozie.action.hadoop.PigMain
> [ERROR] 
> /tmp/oozie-5.0.0/sharelib/pig/src/main/java/org/apache/oozie/action/hadoop/PigMain.java:[98,36]
>  cannot find symbol
> symbol: class Path
> location: class org.apache.oozie.action.hadoop.PigMain
> [INFO] 6 errors
> [INFO] -
> [INFO] 
> 
> [INFO] Reactor Summary:
> [INFO]
> [INFO] Apache Oozie Main 5.0.0  SUCCESS [ 5.637 s]
> [INFO] Apache Oozie Client  SUCCESS [ 35.895 
> s]
> [INFO] Apache Oozie Share Lib Oozie ... SUCCESS [ 24.548 
> s]
> [INFO] Apache Oozie Share Lib HCatalog  SUCCESS [ 13.107 
> s]
> [INFO] Apache Oozie Share Lib Distcp .. SUCCESS [ 9.785 s]
> [INFO] Apache Oozie Core .. SUCCESS [03:03 
> min]
> [INFO] Apache Oozie Share Lib Streaming ... SUCCESS [ 14.031 
> s]
> [INFO] Apache Oozie Share Lib Pig . FAILURE [ 18.706 
> s]
> [INFO] Apache Oozie Share Lib Hive  SKIPPED
> [INFO] Apache Oozie Share Lib Hive 2 .. SKIPPED
> [INFO] Apache Oozie Share Lib Sqoop ... SKIPPED
> [INFO] Apache Oozie Examples .. SKIPPED
> [INFO] Apache Oozie Share Lib Spark ... SKIPPED
> [INFO] Apache Oozie Share Lib . SKIPPED
> [INFO] Apache Oozie Docs .. SKIPPED
> [INFO] Apache Oozie WebApp  SKIPPED
> [INFO] Apache Oozie Tools . SKIPPED
> [INFO] Apache Oozie MiniOozie . SKIPPED
> [INFO] Apache Oozie Server  SKIPPED
> [INFO] Apache Oozie Distro  SKIPPED
> [INFO] Apache Oozie ZooKeeper Security Tests 5.0.0  SKIPPED
> [INFO] 
> 
> [INFO] BUILD FAILURE
> [INFO] 
> 
> [INFO] Total time: 05:07 min
> [INFO] Finished at: 2018-04-13T16:03:09Z
> [INFO] 
> 
> [ERROR] Failed to execute goal 
> org.apache.maven.plugins:maven-compiler-plugin:3.7.0:compile 
> (default-compile) on project oozie-sharelib-pig: Compilation failure: 
> Compilation failure:
> [ERROR] 
> /tmp/oozie-5.0.0/sharelib/pig/src/main/java/org/apache/oozie/action/hadoop/PigMain.java:[30,30]
>  package 

[jira] [Updated] (OOZIE-3219) Cannot compile with hadoop 3.1.0

[Andras Piros (JIRA)]

 [ 
https://issues.apache.org/jira/browse/OOZIE-3219?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Andras Piros updated OOZIE-3219:

Attachment: OOZIE-3219.002.patch

> Cannot compile with hadoop 3.1.0
> 
>
> Key: OOZIE-3219
> URL: https://issues.apache.org/jira/browse/OOZIE-3219
> Project: Oozie
>  Issue Type: Bug
>  Components: core
>Affects Versions: 5.0.0
> Environment: {code:java}
> Apache Maven 3.5.3 (3383c37e1f9e9b3bc3df5050c29c8aff9f295297; 
> 2018-02-24T19:49:05Z)
> Maven home: /opt/maven/apache-maven-3.5.3
> Java version: 1.8.0_161, vendor: Oracle Corporation
> Java home: /usr/lib/jvm/java-1.8.0-openjdk-1.8.0.161-0.b14.el7_4.x86_64/jre
> Default locale: en_US, platform encoding: UTF-8
> OS name: "linux", version: "3.10.0-693.21.1.el7.x86_64", arch: "amd64", 
> family: "unix"{code}
> latest 5.0 release bits
>Reporter: Artem Ervits
>Assignee: Artem Ervits
>Priority: Major
> Fix For: 5.1.0
>
> Attachments: OOZIE-3219.00.patch
>
>
> command failure on the following, same happens with 3.0.0. Issue does not 
> appear with < 3.0
> {code:java}
> mvn clean install -DskipTests -Dhadoop.version=3.1.0{code}
> {code:java}
> [ERROR] COMPILATION ERROR :
> [INFO] -
> [ERROR] 
> /tmp/oozie-5.0.0/core/src/main/java/org/apache/oozie/util/db/FailingConnectionWrapper.java:[24,37]
>  package org.apache.directory.api.util does not exist
> [ERROR] 
> /tmp/oozie-5.0.0/core/src/main/java/org/apache/oozie/util/db/FailingConnectionWrapper.java:[352,41]
>  cannot find symbol
> symbol: variable Strings
> location: class 
> org.apache.oozie.util.db.FailingConnectionWrapper.OozieDmlStatementPredicate
> [INFO] 2 errors
> [INFO] -
> [INFO] 
> 
> [INFO] Reactor Summary:
> [INFO]
> [INFO] Apache Oozie Main 5.0.0  SUCCESS [ 3.768 s]
> [INFO] Apache Oozie Client  SUCCESS [ 30.924 
> s]
> [INFO] Apache Oozie Share Lib Oozie ... SUCCESS [ 20.904 
> s]
> [INFO] Apache Oozie Share Lib HCatalog  SUCCESS [ 9.150 s]
> [INFO] Apache Oozie Share Lib Distcp .. SUCCESS [ 7.938 s]
> [INFO] Apache Oozie Core .. FAILURE [ 13.929 
> s]
> [INFO] Apache Oozie Share Lib Streaming ... SKIPPED
> [INFO] Apache Oozie Share Lib Pig . SKIPPED
> [INFO] Apache Oozie Share Lib Hive  SKIPPED
> [INFO] Apache Oozie Share Lib Hive 2 .. SKIPPED
> [INFO] Apache Oozie Share Lib Sqoop ... SKIPPED
> [INFO] Apache Oozie Examples .. SKIPPED
> [INFO] Apache Oozie Share Lib Spark ... SKIPPED
> [INFO] Apache Oozie Share Lib . SKIPPED
> [INFO] Apache Oozie Docs .. SKIPPED
> [INFO] Apache Oozie WebApp  SKIPPED
> [INFO] Apache Oozie Tools . SKIPPED
> [INFO] Apache Oozie MiniOozie . SKIPPED
> [INFO] Apache Oozie Server  SKIPPED
> [INFO] Apache Oozie Distro  SKIPPED
> [INFO] Apache Oozie ZooKeeper Security Tests 5.0.0  SKIPPED
> [INFO] 
> 
> [INFO] BUILD FAILURE
> [INFO] 
> 
> [INFO] Total time: 01:28 min
> [INFO] Finished at: 2018-04-13T14:03:17Z
> [INFO] 
> 
> [ERROR] Failed to execute goal 
> org.apache.maven.plugins:maven-compiler-plugin:3.7.0:compile 
> (default-compile) on project oozie-core: Compilation failure: Compilation 
> failure:
> [ERROR] 
> /tmp/oozie-5.0.0/core/src/main/java/org/apache/oozie/util/db/FailingConnectionWrapper.java:[24,37]
>  package org.apache.directory.api.util does not exist
> [ERROR] 
> /tmp/oozie-5.0.0/core/src/main/java/org/apache/oozie/util/db/FailingConnectionWrapper.java:[352,41]
>  cannot find symbol
> [ERROR] symbol: variable Strings
> [ERROR] location: class 
> org.apache.oozie.util.db.FailingConnectionWrapper.OozieDmlStatementPredicate
> [ERROR] -> [Help 1]
> [ERROR]
> [ERROR] To see the full stack trace of the errors, re-run Maven with the -e 
> switch.
> [ERROR] Re-run Maven using the -X switch to enable full debug logging.
> [ERROR]
> [ERROR] For more information about the errors and possible solutions, please 
> read the following articles:
> [ERROR] [Help 1] 
> 

[jira] [Updated] (OOZIE-3219) Cannot compile with hadoop 3.1.0

[Andras Piros (JIRA)]

 [ 
https://issues.apache.org/jira/browse/OOZIE-3219?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Andras Piros updated OOZIE-3219:

Summary: Cannot compile with hadoop 3.1.0  (was: oozie core fails when 
compiled with hadoop 3.x)

> Cannot compile with hadoop 3.1.0
> 
>
> Key: OOZIE-3219
> URL: https://issues.apache.org/jira/browse/OOZIE-3219
> Project: Oozie
>  Issue Type: Bug
>  Components: core
>Affects Versions: 5.0.0
> Environment: {code:java}
> Apache Maven 3.5.3 (3383c37e1f9e9b3bc3df5050c29c8aff9f295297; 
> 2018-02-24T19:49:05Z)
> Maven home: /opt/maven/apache-maven-3.5.3
> Java version: 1.8.0_161, vendor: Oracle Corporation
> Java home: /usr/lib/jvm/java-1.8.0-openjdk-1.8.0.161-0.b14.el7_4.x86_64/jre
> Default locale: en_US, platform encoding: UTF-8
> OS name: "linux", version: "3.10.0-693.21.1.el7.x86_64", arch: "amd64", 
> family: "unix"{code}
> latest 5.0 release bits
>Reporter: Artem Ervits
>Assignee: Artem Ervits
>Priority: Major
> Fix For: 5.1.0
>
> Attachments: OOZIE-3219.00.patch
>
>
> command failure on the following, same happens with 3.0.0. Issue does not 
> appear with < 3.0
> {code:java}
> mvn clean install -DskipTests -Dhadoop.version=3.1.0{code}
> {code:java}
> [ERROR] COMPILATION ERROR :
> [INFO] -
> [ERROR] 
> /tmp/oozie-5.0.0/core/src/main/java/org/apache/oozie/util/db/FailingConnectionWrapper.java:[24,37]
>  package org.apache.directory.api.util does not exist
> [ERROR] 
> /tmp/oozie-5.0.0/core/src/main/java/org/apache/oozie/util/db/FailingConnectionWrapper.java:[352,41]
>  cannot find symbol
> symbol: variable Strings
> location: class 
> org.apache.oozie.util.db.FailingConnectionWrapper.OozieDmlStatementPredicate
> [INFO] 2 errors
> [INFO] -
> [INFO] 
> 
> [INFO] Reactor Summary:
> [INFO]
> [INFO] Apache Oozie Main 5.0.0  SUCCESS [ 3.768 s]
> [INFO] Apache Oozie Client  SUCCESS [ 30.924 
> s]
> [INFO] Apache Oozie Share Lib Oozie ... SUCCESS [ 20.904 
> s]
> [INFO] Apache Oozie Share Lib HCatalog  SUCCESS [ 9.150 s]
> [INFO] Apache Oozie Share Lib Distcp .. SUCCESS [ 7.938 s]
> [INFO] Apache Oozie Core .. FAILURE [ 13.929 
> s]
> [INFO] Apache Oozie Share Lib Streaming ... SKIPPED
> [INFO] Apache Oozie Share Lib Pig . SKIPPED
> [INFO] Apache Oozie Share Lib Hive  SKIPPED
> [INFO] Apache Oozie Share Lib Hive 2 .. SKIPPED
> [INFO] Apache Oozie Share Lib Sqoop ... SKIPPED
> [INFO] Apache Oozie Examples .. SKIPPED
> [INFO] Apache Oozie Share Lib Spark ... SKIPPED
> [INFO] Apache Oozie Share Lib . SKIPPED
> [INFO] Apache Oozie Docs .. SKIPPED
> [INFO] Apache Oozie WebApp  SKIPPED
> [INFO] Apache Oozie Tools . SKIPPED
> [INFO] Apache Oozie MiniOozie . SKIPPED
> [INFO] Apache Oozie Server  SKIPPED
> [INFO] Apache Oozie Distro  SKIPPED
> [INFO] Apache Oozie ZooKeeper Security Tests 5.0.0  SKIPPED
> [INFO] 
> 
> [INFO] BUILD FAILURE
> [INFO] 
> 
> [INFO] Total time: 01:28 min
> [INFO] Finished at: 2018-04-13T14:03:17Z
> [INFO] 
> 
> [ERROR] Failed to execute goal 
> org.apache.maven.plugins:maven-compiler-plugin:3.7.0:compile 
> (default-compile) on project oozie-core: Compilation failure: Compilation 
> failure:
> [ERROR] 
> /tmp/oozie-5.0.0/core/src/main/java/org/apache/oozie/util/db/FailingConnectionWrapper.java:[24,37]
>  package org.apache.directory.api.util does not exist
> [ERROR] 
> /tmp/oozie-5.0.0/core/src/main/java/org/apache/oozie/util/db/FailingConnectionWrapper.java:[352,41]
>  cannot find symbol
> [ERROR] symbol: variable Strings
> [ERROR] location: class 
> org.apache.oozie.util.db.FailingConnectionWrapper.OozieDmlStatementPredicate
> [ERROR] -> [Help 1]
> [ERROR]
> [ERROR] To see the full stack trace of the errors, re-run Maven with the -e 
> switch.
> [ERROR] Re-run Maven using the -X switch to enable full debug logging.
> [ERROR]
> [ERROR] For more information about the errors and possible solutions, please 
> read the 

[jira] [Commented] (OOZIE-3227) Eliminate duplicated dependencies from distributed cache

[Andras Piros (JIRA)]

[ 
https://issues.apache.org/jira/browse/OOZIE-3227?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16465677#comment-16465677
 ] 

Andras Piros commented on OOZIE-3227:
-

[~dionusos] I cannot make Oozie code compile w/ Hadoop 3, and hence, cannot 
reproduce. If you have a patch for that, could you please update OOZIE-3219? 
Thanks!

> Eliminate duplicated dependencies from distributed cache
> 
>
> Key: OOZIE-3227
> URL: https://issues.apache.org/jira/browse/OOZIE-3227
> Project: Oozie
>  Issue Type: Sub-task
>  Components: core
>Reporter: Denes Bodo
>Assignee: Denes Bodo
>Priority: Major
>
> Using Hadoop 3 it is not allowed to have multiple dependencies with same file 
> names on the list of *mapreduce.job.cache.files*.
> The issue occurs when I have the same file name on multiple sharelib folders 
> and/or my application's lib folder. This can be avoided but not easy all the 
> time.
> I suggest to remove the duplicates from this list.
> A quick workaround for the source code in JavaActionExecutor is like:
> {code}
> removeDuplicatedDependencies(launcherJobConf, 
> "mapreduce.job.cache.files");
> removeDuplicatedDependencies(launcherJobConf, 
> "mapreduce.job.cache.archives");
> ..
> private void removeDuplicatedDependencies(JobConf conf, String key) {
> final Map nameToPath = new HashMap<>();
> StringBuilder uniqList = new StringBuilder();
> for(String dependency: conf.get(key).split(",")) {
> final String[] arr = dependency.split("/");
> final String dependencyName = arr[arr.length - 1];
> if(nameToPath.containsKey(dependencyName)) {
> LOG.warn(dependencyName + " [" + dependency + "] is already 
> defined in " + key + ". Skipping...");
> } else {
> nameToPath.put(dependencyName, dependency);
> uniqList.append(dependency).append(",");
> }
> }
> uniqList.setLength(uniqList.length() - 1);
> conf.set(key, uniqList.toString());
> }
> {code}
> Other way is to eliminate the deprecated 
> *org.apache.hadoop.filecache.DistributedCache*.
> I am going to have a deeper understanding how we should use distributed cache 
> and all the comments are welcome.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (OOZIE-3235) Old ActiveMQ dependency in Oozie

[Mate Juhasz (JIRA)]

Mate Juhasz created OOZIE-3235:
--

 Summary: Old ActiveMQ dependency in Oozie
 Key: OOZIE-3235
 URL: https://issues.apache.org/jira/browse/OOZIE-3235
 Project: Oozie
  Issue Type: Bug
  Components: client
Affects Versions: 5.1.0
Reporter: Mate Juhasz
Assignee: Mate Juhasz
 Fix For: 5.1.0


An old, and vulnerable version of ActiveMQ is used by Oozie's tests (5.13.3).
It seems that the jars are even included in the oozie-client with "compile" 
scope, but we only use them in unit tests.

Oozie-client uses the JMS-API only from these jars, which could be included 
separately (javax.jms).

The version could be upgraded to 5.14.2 and the scope changed to "test" instead 
of "compile".



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (OOZIE-3235) Upgrade Old ActiveMQ dependency in Oozie

[Mate Juhasz (JIRA)]

 [ 
https://issues.apache.org/jira/browse/OOZIE-3235?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Mate Juhasz updated OOZIE-3235:
---
Summary: Upgrade Old ActiveMQ dependency in Oozie  (was: Old ActiveMQ 
dependency in Oozie)

> Upgrade Old ActiveMQ dependency in Oozie
> 
>
> Key: OOZIE-3235
> URL: https://issues.apache.org/jira/browse/OOZIE-3235
> Project: Oozie
>  Issue Type: Bug
>  Components: client
>Affects Versions: 5.1.0
>Reporter: Mate Juhasz
>Assignee: Mate Juhasz
>Priority: Minor
> Fix For: 5.1.0
>
>
> An old, and vulnerable version of ActiveMQ is used by Oozie's tests (5.13.3).
> It seems that the jars are even included in the oozie-client with "compile" 
> scope, but we only use them in unit tests.
> Oozie-client uses the JMS-API only from these jars, which could be included 
> separately (javax.jms).
> The version could be upgraded to 5.14.2 and the scope changed to "test" 
> instead of "compile".



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (OOZIE-2877) Oozie Git Action

[Andras Piros (JIRA)]

[ 
https://issues.apache.org/jira/browse/OOZIE-2877?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16465569#comment-16465569
 ] 

Andras Piros commented on OOZIE-2877:
-

[~clayb] thanks for contributing! I've closed some old JIRA comments, and 
updated the rest. Best is to always update both RB and JIRA w/ the same patch - 
the Hadoop QA has then a chance to run pre-commit Jenkins job.

[~gezapeti] would you mind making another review turn? Thanks!

> Oozie Git Action
> 
>
> Key: OOZIE-2877
> URL: https://issues.apache.org/jira/browse/OOZIE-2877
> Project: Oozie
>  Issue Type: Sub-task
>  Components: action
>Reporter: Clay B.
>Assignee: Clay B.
>Priority: Major
>  Labels: action
> Fix For: trunk
>
> Attachments: 0001-OOZIE-2877-Oozie-Git-Action.patch, 
> 0002-OOZIE-2877-Oozie-Git-Action.patch, 
> 0003-OOZIE-2877-Oozie-Git-Action.patch, 
> 0004-OOZIE-2877-Oozie-Git-Action.patch, 
> 0005-OOZIE-2877-Oozie-Git-Action.patch, 
> 0006-OOZIE-2877-Oozie-Git-Action.patch, 
> 0007-OOZIE-2877-Oozie-Git-Action.patch, 
> 0008-OOZIE-2877-Oozie-Git-Action.patch, 
> 0009-OOZIE-2877-Oozie-Git-Action.patch, OOZIE-2877.010.patch, 
> OOZIE-2877.011.patch
>
>
> To aide in deploying ASCII artifacts to clusters, let's provide a tie-in for 
> a source-code management system. Git would be my preferred choice.
> Ideally, this could handle a user's key material e.g. for an ssh key to pull 
> down from a secured repository. This would free users from handling their own 
> key staging and clean-up on YARN nodes and only require them to store the key 
> secured in HDFS.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Re: Review Request 59620: This review board request is for an action to provide a Git action for Oozie

[András Piros via Review Board]

> On May 30, 2017, 11:08 a.m., András Piros wrote:
> > sharelib/git/src/main/java/org/apache/oozie/action/hadoop/GitMain.java
> > Lines 84-88 (patched)
> > 
> >
> > Aren't there existing constants from Apache Hadoop project?
> 
> Clay B. wrote:
> I am not finding a clear authoritative sources in the 
> [https://github.com/apache/hadoop](Hadoop) project for `user.name` nor 
> `mapred.*.job.*`; would you have a suggestion? The 
> [JavaActionExecutor](https://github.com/apache/oozie/blob/83d4ddf45aa16649bd9fae367fa915379d5781cd/core/src/main/java/org/apache/oozie/action/hadoop/JavaActionExecutor.java#L114-L146)
>  defines these all bespoke too sadly.
> 
> I did find YARN has the resource manager in 
> [https://github.com/apache/hadoop/blob/18c494a00c8ead768f3a868b450dceea485559df/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/conf/YarnConfiguration.java#L139-L148](YarnConfiguration.java).
> 
> Would it make more sense to have an Oozie constants class for these? (If 
> so, it looks like this might be a JIRA in its own scope with how often 
> repeated some of these strings are.)

`"user.name"` -> `Hive2Credentials.USER_NAME`, `OozieClient.USER_NAME`

For the MR1 config keys there isn't a current one in the Hadoop repo, for the 
newer ones:
`"mapreduce.jobtracker.address"` -> 
[`MRConfig.MASTER_ADDRESS`](https://github.com/apache/hadoop/blob/f67237cbe7bc48a1b9088e990800b37529f1db2a/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/MRConfig.java#L65)

I think less dependencies are always better, I'm for introducing these 
non-Oozie constants in `GitMain`, and reusing Oozie constants from code inside 
Oozie.


> On May 30, 2017, 11:08 a.m., András Piros wrote:
> > sharelib/git/src/main/java/org/apache/oozie/action/hadoop/GitMain.java
> > Lines 199 (patched)
> > 
> >
> > Extract to another class.
> 
> Clay B. wrote:
> Please pardon my lack of creativity, however, are you thinking of a repo 
> class perhaps to encapsulate all the repository operations? Or a class for 
> some other reason?

I'm thinking of a `GitRepositoryOperations` class to encapsulate all the 
repository operation, having fields like `gitSrc`, `branch`, `outputDir`, 
`credentialFile` initialized in constructor.


> On May 30, 2017, 11:08 a.m., András Piros wrote:
> > sharelib/git/src/main/java/org/apache/oozie/action/hadoop/GitMain.java
> > Lines 254 (patched)
> > 
> >
> > Extract to another class.
> 
> Clay B. wrote:
> Similarly, would this fit better in a repo class (if split off 
> independently) or independent of the repo code and `GitMain`?

I'm thinking of a `GitRepositoryOperations` class to encapsulate all the 
repository operation, having fields like `gitSrc`, `branch`, `outputDir`, 
`credentialFile` initialized in constructor.


> On May 30, 2017, 11:08 a.m., András Piros wrote:
> > sharelib/git/src/test/java/org/apache/oozie/action/hadoop/TestIntegrationGitActionExecutor.java
> > Lines 60 (patched)
> > 
> >
> > Are system props reset after this test run?
> 
> Clay B. wrote:
> I am not sure what the entire chain of `ActionExecutorTestCase`'s methods 
> are doing. However, I am not explicitly clearing this to my knowledge; do you 
> expect issues? (For reference, it seems 
> [`TestJavaActionExecutor()`](https://github.com/apache/oozie/blob/5998c18fde1da769e91e3ef1bcca484723730c76/core/src/test/java/org/apache/oozie/action/hadoop/TestJavaActionExecutor.java#L90)
>  is not clearing these either?)

I'd rather not set any system properties here but go for 
[`Services.get().getConf().set()`](https://github.com/apache/oozie/blob/eb168360c550943828d9fe2c7bfdcd4f5e830003/core/src/test/java/org/apache/oozie/service/TestActionService.java#L52-L53),
 if that works.

If not, I'd still set this system property in `@Before void setUp()` and reset 
in `@After void tearDown()` as this one is used by other important pieces of 
code.


> On May 30, 2017, 11:08 a.m., András Piros wrote:
> > sharelib/git/src/test/java/org/apache/oozie/action/hadoop/TestIntegrationGitActionExecutor.java
> > Lines 80 (patched)
> > 
> >
> > Please cut this into independent test cases, and name these accordingly.
> 
> Clay B. wrote:
> Could you provide me more direction on how you would ike this method 
> separated? Since the assertions in this method depend on the run state of the 
> action under test, I would need to run the action many times (all with the 
> same configuration); to me that would be a single case usually?

The current unit 

[jira] Subscription: Oozie Patch Available

[jira]

Issue Subscription
Filter: Oozie Patch Available (106 issues)

Subscriber: ooziedaily

Key Summary
OOZIE-3232  Oozie may waste up to 25% of the heap due to duplicate strings
https://issues.apache.org/jira/browse/OOZIE-3232
OOZIE-3221  Rename DEFAULT_LAUNCHER_MAX_ATTEMPS
https://issues.apache.org/jira/browse/OOZIE-3221
OOZIE-3219  oozie core fails when compiled with hadoop 3.x
https://issues.apache.org/jira/browse/OOZIE-3219
OOZIE-3218  Oozie Sqoop action with command splits the select clause into 
multiple parts due to delimiter being space
https://issues.apache.org/jira/browse/OOZIE-3218
OOZIE-3217  Enable definition of admin users using oozie-site.xml
https://issues.apache.org/jira/browse/OOZIE-3217
OOZIE-3199  Let system property restriction configurable
https://issues.apache.org/jira/browse/OOZIE-3199
OOZIE-3196  Authorization: restrict world readability by user
https://issues.apache.org/jira/browse/OOZIE-3196
OOZIE-3194  Oozie should set proper permissions to sharelib after upload
https://issues.apache.org/jira/browse/OOZIE-3194
OOZIE-3193  Applications are not killed when submitted via subworkflow
https://issues.apache.org/jira/browse/OOZIE-3193
OOZIE-3186  Oozie is unable to use configuration linked using jceks://file/...
https://issues.apache.org/jira/browse/OOZIE-3186
OOZIE-3185  Conflicting JARs org.apache.derby exist in Oozie
https://issues.apache.org/jira/browse/OOZIE-3185
OOZIE-3179  Adding a configurable config-default.xml location to a workflow
https://issues.apache.org/jira/browse/OOZIE-3179
OOZIE-3170  Oozie Diagnostic Bundle tool fails with NPE due to missing service 
class
https://issues.apache.org/jira/browse/OOZIE-3170
OOZIE-3135  Configure log4j2 in SqoopMain
https://issues.apache.org/jira/browse/OOZIE-3135
OOZIE-3109  Escape log-streaming's HTML-specific characters
https://issues.apache.org/jira/browse/OOZIE-3109
OOZIE-3105  testJMXInstrumentation from the 
org.apache.oozie.util.TestMetricsInstrumentation class is flaky
https://issues.apache.org/jira/browse/OOZIE-3105
OOZIE-3094  fix for grammar mistake
https://issues.apache.org/jira/browse/OOZIE-3094
OOZIE-3091  Oozie Sqoop Avro Import fails with "java.lang.NoClassDefFoundError: 
org/apache/avro/mapred/AvroWrapper"
https://issues.apache.org/jira/browse/OOZIE-3091
OOZIE-3071  Oozie 4.3 Spark sharelib ueses a different version of commons-lang3 
than Spark 2.2.0
https://issues.apache.org/jira/browse/OOZIE-3071
OOZIE-3063  Sanitizing variables that are part of openjpa.ConnectionProperties
https://issues.apache.org/jira/browse/OOZIE-3063
OOZIE-3062  Set HADOOP_CONF_DIR for spark action
https://issues.apache.org/jira/browse/OOZIE-3062
OOZIE-3061  Kill only those child jobs which are not already killed
https://issues.apache.org/jira/browse/OOZIE-3061
OOZIE-3002  address findbugs errors in client lib
https://issues.apache.org/jira/browse/OOZIE-3002
OOZIE-2975  code clean up in pig sharelib, replace Exception with more 
explicit, add try with resources, StringBuilder instead of StringBuffer
https://issues.apache.org/jira/browse/OOZIE-2975
OOZIE-2956  Fix Findbugs warnings related to reliance on default encoding in 
oozie-core
https://issues.apache.org/jira/browse/OOZIE-2956
OOZIE-2955  Fix Findbugs warnings related to reliance on default encoding in 
oozie-client
https://issues.apache.org/jira/browse/OOZIE-2955
OOZIE-2954  Fix Checkstyle issues in oozie-client
https://issues.apache.org/jira/browse/OOZIE-2954
OOZIE-2953  Fix Checkstyle issues in oozie-tools
https://issues.apache.org/jira/browse/OOZIE-2953
OOZIE-2952  Fix Findbugs warnings in oozie-sharelib-oozie
https://issues.apache.org/jira/browse/OOZIE-2952
OOZIE-2949  Escape quotes whitespaces in Sqoop  field
https://issues.apache.org/jira/browse/OOZIE-2949
OOZIE-2942  [examples] Fix Findbugs warnings
https://issues.apache.org/jira/browse/OOZIE-2942
OOZIE-2927  Append new line character for Hive2 query using query tag
https://issues.apache.org/jira/browse/OOZIE-2927
OOZIE-2883  OOZIE throw the error "Missing 
[oozie.service.ProxyUserService.proxyuser.oozie.service.ProxyUserService.proxyuser.mr.groups]
 property"
https://issues.apache.org/jira/browse/OOZIE-2883
OOZIE-2877  Oozie Git Action
https://issues.apache.org/jira/browse/OOZIE-2877
OOZIE-2867  Timezone handling for Coordinators: emphasize "Continent/City" 
format
https://issues.apache.org/jira/browse/OOZIE-2867
OOZIE-2834  ParameterVerifier logging non-useful warning for workflow definition
https://issues.apache.org/jira/browse/OOZIE-2834
OOZIE-2833  when using uber mode the regex pattern used in the