[jira] [Commented] (OOZIE-2780) Upgrade minimum Hadoop version to 2.6.0
[
https://issues.apache.org/jira/browse/OOZIE-2780?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16070234#comment-16070234
]
Jan Hentschel commented on OOZIE-2780:
--
[~dbist13] I opened OOZIE-2976 for adding support for setting extended
attributes.
> Upgrade minimum Hadoop version to 2.6.0
> ---
>
> Key: OOZIE-2780
> URL: https://issues.apache.org/jira/browse/OOZIE-2780
> Project: Oozie
> Issue Type: Improvement
> Components: core
>Affects Versions: 4.3.0
>Reporter: Artem Ervits
>Assignee: Artem Ervits
>Priority: Minor
> Labels: newbie
> Fix For: 5.0.0
>
> Attachments: OOZIE-2780-0.patch, OOZIE-2780-1.patch,
> OOZIE-2780-2.patch
>
>
> I'd like to contribute an fs action enhancement to modify extended attributes
> to core but methods to change extended attributes is not available in
> hadoop-common < 2.6.0. Currently hadoop-common is at 2.4. I am contributing a
> patch to bump to 2.6 but so far I haven't been able to generate a clean test
> as tests in core module fail randomly. On the first run, I had a failure on
> {code} Failed tests:
> testCheckAfterActionDelete(org.apache.oozie.service.TestHAPartitionDependencyManagerService):
> Expected: but was: [000-170123095541655-oozie-aerv-C@1]{code}
> and on second run
> {code}Failed tests: testWaitFor(org.apache.oozie.test.TestXTestCase):
> expected:<1000.0> but was:<2036.0>{code}
> I can't tell whether it is due to my patch?
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (OOZIE-2780) Upgrade minimum Hadoop version to 2.6.0
[
https://issues.apache.org/jira/browse/OOZIE-2780?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16068412#comment-16068412
]
Artem Ervits commented on OOZIE-2780:
-
[~Jan Hentschel] sorry for delayed response, no I didn't create the jira for
it. Please go ahead and create one.
> Upgrade minimum Hadoop version to 2.6.0
> ---
>
> Key: OOZIE-2780
> URL: https://issues.apache.org/jira/browse/OOZIE-2780
> Project: Oozie
> Issue Type: Improvement
> Components: core
>Affects Versions: 4.3.0
>Reporter: Artem Ervits
>Assignee: Artem Ervits
>Priority: Minor
> Labels: newbie
> Fix For: 5.0.0
>
> Attachments: OOZIE-2780-0.patch, OOZIE-2780-1.patch,
> OOZIE-2780-2.patch
>
>
> I'd like to contribute an fs action enhancement to modify extended attributes
> to core but methods to change extended attributes is not available in
> hadoop-common < 2.6.0. Currently hadoop-common is at 2.4. I am contributing a
> patch to bump to 2.6 but so far I haven't been able to generate a clean test
> as tests in core module fail randomly. On the first run, I had a failure on
> {code} Failed tests:
> testCheckAfterActionDelete(org.apache.oozie.service.TestHAPartitionDependencyManagerService):
> Expected: but was: [000-170123095541655-oozie-aerv-C@1]{code}
> and on second run
> {code}Failed tests: testWaitFor(org.apache.oozie.test.TestXTestCase):
> expected:<1000.0> but was:<2036.0>{code}
> I can't tell whether it is due to my patch?
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (OOZIE-2780) Upgrade minimum Hadoop version to 2.6.0
[
https://issues.apache.org/jira/browse/OOZIE-2780?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16052821#comment-16052821
]
Jan Hentschel commented on OOZIE-2780:
--
[~dbist13] Did you already create a ticket for the extended attributes? I
probably have a patch to include that.
> Upgrade minimum Hadoop version to 2.6.0
> ---
>
> Key: OOZIE-2780
> URL: https://issues.apache.org/jira/browse/OOZIE-2780
> Project: Oozie
> Issue Type: Improvement
> Components: core
>Affects Versions: 4.3.0
>Reporter: Artem Ervits
>Assignee: Artem Ervits
>Priority: Minor
> Labels: newbie
> Fix For: 5.0.0
>
> Attachments: OOZIE-2780-0.patch, OOZIE-2780-1.patch,
> OOZIE-2780-2.patch
>
>
> I'd like to contribute an fs action enhancement to modify extended attributes
> to core but methods to change extended attributes is not available in
> hadoop-common < 2.6.0. Currently hadoop-common is at 2.4. I am contributing a
> patch to bump to 2.6 but so far I haven't been able to generate a clean test
> as tests in core module fail randomly. On the first run, I had a failure on
> {code} Failed tests:
> testCheckAfterActionDelete(org.apache.oozie.service.TestHAPartitionDependencyManagerService):
> Expected: but was: [000-170123095541655-oozie-aerv-C@1]{code}
> and on second run
> {code}Failed tests: testWaitFor(org.apache.oozie.test.TestXTestCase):
> expected:<1000.0> but was:<2036.0>{code}
> I can't tell whether it is due to my patch?
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (OOZIE-2780) Upgrade minimum Hadoop version to 2.6.0
[
https://issues.apache.org/jira/browse/OOZIE-2780?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16010218#comment-16010218
]
Peter Bacsko commented on OOZIE-2780:
-
Thanks [~dbist13] - I'm closing this JIRA then.
> Upgrade minimum Hadoop version to 2.6.0
> ---
>
> Key: OOZIE-2780
> URL: https://issues.apache.org/jira/browse/OOZIE-2780
> Project: Oozie
> Issue Type: Improvement
> Components: core
>Affects Versions: 4.3.0
>Reporter: Artem Ervits
>Assignee: Artem Ervits
>Priority: Minor
> Labels: newbie
> Fix For: 5.0.0
>
> Attachments: OOZIE-2780-0.patch, OOZIE-2780-1.patch,
> OOZIE-2780-2.patch
>
>
> I'd like to contribute an fs action enhancement to modify extended attributes
> to core but methods to change extended attributes is not available in
> hadoop-common < 2.6.0. Currently hadoop-common is at 2.4. I am contributing a
> patch to bump to 2.6 but so far I haven't been able to generate a clean test
> as tests in core module fail randomly. On the first run, I had a failure on
> {code} Failed tests:
> testCheckAfterActionDelete(org.apache.oozie.service.TestHAPartitionDependencyManagerService):
> Expected: but was: [000-170123095541655-oozie-aerv-C@1]{code}
> and on second run
> {code}Failed tests: testWaitFor(org.apache.oozie.test.TestXTestCase):
> expected:<1000.0> but was:<2036.0>{code}
> I can't tell whether it is due to my patch?
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Commented] (OOZIE-2780) Upgrade minimum Hadoop version to 2.6.0
[
https://issues.apache.org/jira/browse/OOZIE-2780?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16008954#comment-16008954
]
Artem Ervits commented on OOZIE-2780:
-
[~pbacsko] I ran the whole test class with both tests included and each test
separately. The cause is commons-io as before. The third test
TestJavaActionExecutor fails in all cases and looking into the issues there,
I'm suspecting logic for yarn.timeline-service.enabled changed between Hadoop
2.4 and 2.6. So to summarize, the two ZK tests pass with my patch provided in
https://issues.apache.org/jira/browse/OOZIE-2888 and the test for
TestJavaActionExecutor I'm still working through and trying to figure out the
logic.
> Upgrade minimum Hadoop version to 2.6.0
> ---
>
> Key: OOZIE-2780
> URL: https://issues.apache.org/jira/browse/OOZIE-2780
> Project: Oozie
> Issue Type: Improvement
> Components: core
>Affects Versions: 4.3.0
>Reporter: Artem Ervits
>Assignee: Artem Ervits
>Priority: Minor
> Labels: newbie
> Fix For: 5.0.0
>
> Attachments: OOZIE-2780-0.patch, OOZIE-2780-1.patch,
> OOZIE-2780-2.patch
>
>
> I'd like to contribute an fs action enhancement to modify extended attributes
> to core but methods to change extended attributes is not available in
> hadoop-common < 2.6.0. Currently hadoop-common is at 2.4. I am contributing a
> patch to bump to 2.6 but so far I haven't been able to generate a clean test
> as tests in core module fail randomly. On the first run, I had a failure on
> {code} Failed tests:
> testCheckAfterActionDelete(org.apache.oozie.service.TestHAPartitionDependencyManagerService):
> Expected: but was: [000-170123095541655-oozie-aerv-C@1]{code}
> and on second run
> {code}Failed tests: testWaitFor(org.apache.oozie.test.TestXTestCase):
> expected:<1000.0> but was:<2036.0>{code}
> I can't tell whether it is due to my patch?
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Commented] (OOZIE-2780) Upgrade minimum Hadoop version to 2.6.0
[
https://issues.apache.org/jira/browse/OOZIE-2780?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16008928#comment-16008928
]
Peter Bacsko commented on OOZIE-2780:
-
[~dbist] thanks for investigating the root cause.
ZK security tests are tricky, it could be that one test passes if it runs on it
own (I've seen this happening a while ago). So a failing test might affect the
next one, especially since ZK is not restarted between the tests if I'm not
mistaken.
> Upgrade minimum Hadoop version to 2.6.0
> ---
>
> Key: OOZIE-2780
> URL: https://issues.apache.org/jira/browse/OOZIE-2780
> Project: Oozie
> Issue Type: Improvement
> Components: core
>Affects Versions: 4.3.0
>Reporter: Artem Ervits
>Assignee: Artem Ervits
>Priority: Minor
> Labels: newbie
> Fix For: 5.0.0
>
> Attachments: OOZIE-2780-0.patch, OOZIE-2780-1.patch,
> OOZIE-2780-2.patch
>
>
> I'd like to contribute an fs action enhancement to modify extended attributes
> to core but methods to change extended attributes is not available in
> hadoop-common < 2.6.0. Currently hadoop-common is at 2.4. I am contributing a
> patch to bump to 2.6 but so far I haven't been able to generate a clean test
> as tests in core module fail randomly. On the first run, I had a failure on
> {code} Failed tests:
> testCheckAfterActionDelete(org.apache.oozie.service.TestHAPartitionDependencyManagerService):
> Expected: but was: [000-170123095541655-oozie-aerv-C@1]{code}
> and on second run
> {code}Failed tests: testWaitFor(org.apache.oozie.test.TestXTestCase):
> expected:<1000.0> but was:<2036.0>{code}
> I can't tell whether it is due to my patch?
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Commented] (OOZIE-2780) Upgrade minimum Hadoop version to 2.6.0
[
https://issues.apache.org/jira/browse/OOZIE-2780?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16008618#comment-16008618
]
Artem Ervits commented on OOZIE-2780:
-
also created https://issues.apache.org/jira/browse/OOZIE-2888 to track
org.apache.oozie.util.TestZKUtilsWithSecurity failures
> Upgrade minimum Hadoop version to 2.6.0
> ---
>
> Key: OOZIE-2780
> URL: https://issues.apache.org/jira/browse/OOZIE-2780
> Project: Oozie
> Issue Type: Improvement
> Components: core
>Affects Versions: 4.3.0
>Reporter: Artem Ervits
>Assignee: Artem Ervits
>Priority: Minor
> Labels: newbie
> Fix For: 5.0.0
>
> Attachments: OOZIE-2780-0.patch, OOZIE-2780-1.patch,
> OOZIE-2780-2.patch
>
>
> I'd like to contribute an fs action enhancement to modify extended attributes
> to core but methods to change extended attributes is not available in
> hadoop-common < 2.6.0. Currently hadoop-common is at 2.4. I am contributing a
> patch to bump to 2.6 but so far I haven't been able to generate a clean test
> as tests in core module fail randomly. On the first run, I had a failure on
> {code} Failed tests:
> testCheckAfterActionDelete(org.apache.oozie.service.TestHAPartitionDependencyManagerService):
> Expected: but was: [000-170123095541655-oozie-aerv-C@1]{code}
> and on second run
> {code}Failed tests: testWaitFor(org.apache.oozie.test.TestXTestCase):
> expected:<1000.0> but was:<2036.0>{code}
> I can't tell whether it is due to my patch?
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Commented] (OOZIE-2780) Upgrade minimum Hadoop version to 2.6.0
[
https://issues.apache.org/jira/browse/OOZIE-2780?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16008590#comment-16008590
]
Artem Ervits commented on OOZIE-2780:
-
[~pbacsko] for the one test that definitely fails with 2.6, I see the following
https://github.com/apache/oozie/blob/master/core/src/test/java/org/apache/oozie/action/hadoop/TestJavaActionExecutor.java#L2485-L2491,
if configuration is not set, property
{code}
yarn.timeline-service.enabled
{code}
as per
https://hadoop.apache.org/docs/current/hadoop-yarn/hadoop-yarn-site/TimelineServer.html
defaults to false, so doesn't the test
{code}
assertEquals("true",
launcherConf.get(JavaActionExecutor.HADOOP_YARN_TIMELINE_SERVICE_ENABLED));
{code}
need to expect false, rather than true?
then a few more tests for the property check the expected property in multiple
ways
{code}
assertEquals("true",
launcherConf.get(JavaActionExecutor.HADOOP_YARN_TIMELINE_SERVICE_ENABLED));
{code}
and
{code}
assertTrue(launcherConf.getBoolean(JavaActionExecutor.HADOOP_YARN_TIMELINE_SERVICE_ENABLED,
false));
{code}
[~rkanter] please advise? Again, this test passes if I exclude commons-io from
hadoop project or increase the commons-io version from 2.1 to 2.4 in main Oozie
pom.
> Upgrade minimum Hadoop version to 2.6.0
> ---
>
> Key: OOZIE-2780
> URL: https://issues.apache.org/jira/browse/OOZIE-2780
> Project: Oozie
> Issue Type: Improvement
> Components: core
>Affects Versions: 4.3.0
>Reporter: Artem Ervits
>Assignee: Artem Ervits
>Priority: Minor
> Labels: newbie
> Fix For: 5.0.0
>
> Attachments: OOZIE-2780-0.patch, OOZIE-2780-1.patch,
> OOZIE-2780-2.patch
>
>
> I'd like to contribute an fs action enhancement to modify extended attributes
> to core but methods to change extended attributes is not available in
> hadoop-common < 2.6.0. Currently hadoop-common is at 2.4. I am contributing a
> patch to bump to 2.6 but so far I haven't been able to generate a clean test
> as tests in core module fail randomly. On the first run, I had a failure on
> {code} Failed tests:
> testCheckAfterActionDelete(org.apache.oozie.service.TestHAPartitionDependencyManagerService):
> Expected: but was: [000-170123095541655-oozie-aerv-C@1]{code}
> and on second run
> {code}Failed tests: testWaitFor(org.apache.oozie.test.TestXTestCase):
> expected:<1000.0> but was:<2036.0>{code}
> I can't tell whether it is due to my patch?
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Commented] (OOZIE-2780) Upgrade minimum Hadoop version to 2.6.0
[
https://issues.apache.org/jira/browse/OOZIE-2780?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16008472#comment-16008472
]
Artem Ervits commented on OOZIE-2780:
-
[~pbacsko] the 2nd test also passes on my environment so I think it is not
related.
{code}
mvn clean
-Dtest=org.apache.oozie.util.TestZKUtilsWithSecurity#testCheckAndSetACLs test
-Dhadoop.version=2.6.0
{code}
{noformat}
---
T E S T S
---
Java HotSpot(TM) 64-Bit Server VM warning: ignoring option MaxPermSize=512m;
support was removed in 8.0
Concurrency config is parallel='classes', perCoreThreadCount=false,
threadCount=1, useUnlimitedThreads=false
objc[38079]: Class JavaLaunchHelper is implemented in both
/Library/Java/JavaVirtualMachines/jdk1.8.0_51.jdk/Contents/Home/jre/bin/java
(0x1081f84c0) and
/Library/Java/JavaVirtualMachines/jdk1.8.0_51.jdk/Contents/Home/jre/lib/libinstrument.dylib
(0x1207ee4e0). One of the two will be used. Which one is undefined.
Running org.apache.oozie.util.TestZKUtilsWithSecurity
Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 47.371 sec
Results :
Tests run: 1, Failures: 0, Errors: 0, Skipped: 0
[INFO]
[INFO] Reactor Summary:
[INFO]
[INFO] Apache Oozie Main .. SUCCESS [ 0.493 s]
[INFO] Apache Oozie Hadoop Utils hadoop-2-4.4.0-SNAPSHOT .. SUCCESS [ 1.906 s]
[INFO] Apache Oozie Hadoop Distcp hadoop-2-4.4.0-SNAPSHOT . SUCCESS [ 0.071 s]
[INFO] Apache Oozie Hadoop Auth hadoop-2-4.4.0-SNAPSHOT Test SUCCESS [ 0.267 s]
[INFO] Apache Oozie Hadoop Libs ... SUCCESS [ 0.014 s]
[INFO] Apache Oozie Client SUCCESS [ 2.770 s]
[INFO] Apache Oozie Share Lib Oozie ... SUCCESS [ 2.807 s]
[INFO] Apache Oozie Share Lib HCatalog SUCCESS [ 2.010 s]
[INFO] Apache Oozie Share Lib Distcp .. SUCCESS [ 0.659 s]
[INFO] Apache Oozie Core .. SUCCESS [ 20.873 s]
[INFO] Apache Oozie Share Lib Streaming ... SUCCESS [ 4.373 s]
[INFO] Apache Oozie Share Lib Pig . SUCCESS [ 3.171 s]
[INFO] Apache Oozie Share Lib Hive SUCCESS [ 3.496 s]
[INFO] Apache Oozie Share Lib Hive 2 .. SUCCESS [ 3.215 s]
[INFO] Apache Oozie Share Lib Sqoop ... SUCCESS [ 1.830 s]
[INFO] Apache Oozie Examples .. SUCCESS [ 11.999 s]
[INFO] Apache Oozie Share Lib Spark ... SUCCESS [01:14 min]
[INFO] Apache Oozie Share Lib . SUCCESS [ 0.012 s]
[INFO] Apache Oozie Docs .. SUCCESS [ 0.896 s]
[INFO] Apache Oozie WebApp SUCCESS [ 16.385 s]
[INFO] Apache Oozie Tools . SUCCESS [ 8.460 s]
[INFO] Apache Oozie MiniOozie . SUCCESS [ 5.051 s]
[INFO] Apache Oozie Server SUCCESS [ 13.079 s]
[INFO] Apache Oozie Distro SUCCESS [ 1.359 s]
[INFO] Apache Oozie ZooKeeper Security Tests .. SUCCESS [ 51.172 s]
[INFO]
[INFO] BUILD SUCCESS
[INFO]
[INFO] Total time: 03:51 min
[INFO] Finished at: 2017-05-12T13:51:59-04:00
[INFO] Final Memory: 279M/456M
{noformat}
> Upgrade minimum Hadoop version to 2.6.0
> ---
>
> Key: OOZIE-2780
> URL: https://issues.apache.org/jira/browse/OOZIE-2780
> Project: Oozie
> Issue Type: Improvement
> Components: core
>Affects Versions: 4.3.0
>Reporter: Artem Ervits
>Assignee: Artem Ervits
>Priority: Minor
> Labels: newbie
> Fix For: 5.0.0
>
> Attachments: OOZIE-2780-0.patch, OOZIE-2780-1.patch,
> OOZIE-2780-2.patch
>
>
> I'd like to contribute an fs action enhancement to modify extended attributes
> to core but methods to change extended attributes is not available in
> hadoop-common < 2.6.0. Currently hadoop-common is at 2.4. I am contributing a
> patch to bump to 2.6 but so far I haven't been able to generate a clean test
> as tests in core module fail randomly. On the first run, I had a failure on
> {code} Failed tests:
> testCheckAfterActionDelete(org.apache.oozie.service.TestHAPartitionDependencyManagerService):
> Expected: but was: [000-170123095541655-oozie-aerv-C@1]{code}
> and on second run
> {code}Failed tests: testWaitFor(org.apache.oozie.test.TestXTestCase):
> expected:<1000.0
[jira] [Commented] (OOZIE-2780) Upgrade minimum Hadoop version to 2.6.0
[
https://issues.apache.org/jira/browse/OOZIE-2780?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16008449#comment-16008449
]
Artem Ervits commented on OOZIE-2780:
-
[~pbacsko] thanks for catching this. I looked at it briefly, specifically
testNewUsingACLs(org.apache.oozie.util.TestZKUtilsWithSecurity):
org/apache/commons/io/Charsets
is due to commons-io version. Both hadoop-2.4 and hadoop 2.6 ship with v. 2.4
and Oozie ships with commons-io 2.1. I was able to reproduce the tests and if I
change commons-io to 2.4 tests pass.
{noformat}
mvn clean -Dtest=org.apache.oozie.util.TestZKUtilsWithSecurity#testNewUsingACLs
test -Dhadoop.version=2.4.0
---
T E S T S
---
Java HotSpot(TM) 64-Bit Server VM warning: ignoring option MaxPermSize=512m;
support was removed in 8.0
Concurrency config is parallel='classes', perCoreThreadCount=false,
threadCount=1, useUnlimitedThreads=false
objc[37421]: Class JavaLaunchHelper is implemented in both
/Library/Java/JavaVirtualMachines/jdk1.8.0_51.jdk/Contents/Home/jre/bin/java
(0x10f9dd4c0) and
/Library/Java/JavaVirtualMachines/jdk1.8.0_51.jdk/Contents/Home/jre/lib/libinstrument.dylib
(0x118ef04e0). One of the two will be used. Which one is undefined.
Running org.apache.oozie.util.TestZKUtilsWithSecurity
Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 48.103 sec
Results :
Tests run: 1, Failures: 0, Errors: 0, Skipped: 0
[INFO]
[INFO] Reactor Summary:
[INFO]
[INFO] Apache Oozie Main .. SUCCESS [ 0.561 s]
[INFO] Apache Oozie Hadoop Utils hadoop-2-4.4.0-SNAPSHOT .. SUCCESS [ 1.790 s]
[INFO] Apache Oozie Hadoop Distcp hadoop-2-4.4.0-SNAPSHOT . SUCCESS [ 0.074 s]
[INFO] Apache Oozie Hadoop Auth hadoop-2-4.4.0-SNAPSHOT Test SUCCESS [ 0.141 s]
[INFO] Apache Oozie Hadoop Libs ... SUCCESS [ 0.018 s]
[INFO] Apache Oozie Client SUCCESS [ 2.707 s]
[INFO] Apache Oozie Share Lib Oozie ... SUCCESS [ 2.717 s]
[INFO] Apache Oozie Share Lib HCatalog SUCCESS [ 2.200 s]
[INFO] Apache Oozie Share Lib Distcp .. SUCCESS [ 0.731 s]
[INFO] Apache Oozie Core .. SUCCESS [ 21.682 s]
[INFO] Apache Oozie Share Lib Streaming ... SUCCESS [ 4.871 s]
[INFO] Apache Oozie Share Lib Pig . SUCCESS [ 3.887 s]
[INFO] Apache Oozie Share Lib Hive SUCCESS [ 3.992 s]
[INFO] Apache Oozie Share Lib Hive 2 .. SUCCESS [ 3.684 s]
[INFO] Apache Oozie Share Lib Sqoop ... SUCCESS [ 2.624 s]
[INFO] Apache Oozie Examples .. SUCCESS [ 8.485 s]
[INFO] Apache Oozie Share Lib Spark ... SUCCESS [ 27.002 s]
[INFO] Apache Oozie Share Lib . SUCCESS [ 0.017 s]
[INFO] Apache Oozie Docs .. SUCCESS [ 0.342 s]
[INFO] Apache Oozie WebApp SUCCESS [ 1.809 s]
[INFO] Apache Oozie Tools . SUCCESS [ 3.444 s]
[INFO] Apache Oozie MiniOozie . SUCCESS [ 1.463 s]
[INFO] Apache Oozie Server SUCCESS [ 4.529 s]
[INFO] Apache Oozie Distro SUCCESS [ 1.836 s]
[INFO] Apache Oozie ZooKeeper Security Tests .. SUCCESS [ 52.087 s]
[INFO]
[INFO] BUILD SUCCESS
[INFO]
[INFO] Total time: 02:33 min
[INFO] Finished at: 2017-05-12T13:25:00-04:00
[INFO] Final Memory: 77M/403M
{noformat}
now changing commons-io to 2.4
{noformat}
mvn clean -Dtest=org.apache.oozie.util.TestZKUtilsWithSecurity#testNewUsingACLs
test -Dhadoop.version=2.6.0
---
T E S T S
---
Java HotSpot(TM) 64-Bit Server VM warning: ignoring option MaxPermSize=512m;
support was removed in 8.0
Concurrency config is parallel='classes', perCoreThreadCount=false,
threadCount=1, useUnlimitedThreads=false
objc[37788]: Class JavaLaunchHelper is implemented in both
/Library/Java/JavaVirtualMachines/jdk1.8.0_51.jdk/Contents/Home/jre/bin/java
(0x1007094c0) and
/Library/Java/JavaVirtualMachines/jdk1.8.0_51.jdk/Contents/Home/jre/lib/libinstrument.dylib
(0x109dd54e0). One of the two will be used. Which one is undefined.
Running org.apache.oozie.util.TestZKUtilsWithSecurity
Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 53.012 sec
Results :
Tests run: 1, Failures: 0, Errors: 0, Skipped: 0
[jira] [Commented] (OOZIE-2780) Upgrade minimum Hadoop version to 2.6.0
[
https://issues.apache.org/jira/browse/OOZIE-2780?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16008193#comment-16008193
]
Peter Bacsko commented on OOZIE-2780:
-
I found two more tests that fail consistently because of Hadoop 2.6.0 upgrade:
{noformat}
Running org.apache.oozie.util.TestZKUtilsWithSecurity
Tests run: 2, Failures: 0, Errors: 2, Skipped: 0, Time elapsed: 45.108 sec <<<
FAILURE!
Results :
Tests in error:
testNewUsingACLs(org.apache.oozie.util.TestZKUtilsWithSecurity):
org/apache/commons/io/Charsets
testCheckAndSetACLs(org.apache.oozie.util.TestZKUtilsWithSecurity): Another
CacheManager with same name 'apacheds' already exists in the same VM. Please
provide unique names for each CacheManager in the config or do one of
following:(..)
Tests run: 2, Failures: 0, Errors: 2, Skipped: 0
{noformat}
> Upgrade minimum Hadoop version to 2.6.0
> ---
>
> Key: OOZIE-2780
> URL: https://issues.apache.org/jira/browse/OOZIE-2780
> Project: Oozie
> Issue Type: Improvement
> Components: core
>Affects Versions: 4.3.0
>Reporter: Artem Ervits
>Assignee: Artem Ervits
>Priority: Minor
> Labels: newbie
> Fix For: 5.0.0
>
> Attachments: OOZIE-2780-0.patch, OOZIE-2780-1.patch,
> OOZIE-2780-2.patch
>
>
> I'd like to contribute an fs action enhancement to modify extended attributes
> to core but methods to change extended attributes is not available in
> hadoop-common < 2.6.0. Currently hadoop-common is at 2.4. I am contributing a
> patch to bump to 2.6 but so far I haven't been able to generate a clean test
> as tests in core module fail randomly. On the first run, I had a failure on
> {code} Failed tests:
> testCheckAfterActionDelete(org.apache.oozie.service.TestHAPartitionDependencyManagerService):
> Expected: but was: [000-170123095541655-oozie-aerv-C@1]{code}
> and on second run
> {code}Failed tests: testWaitFor(org.apache.oozie.test.TestXTestCase):
> expected:<1000.0> but was:<2036.0>{code}
> I can't tell whether it is due to my patch?
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Commented] (OOZIE-2780) Upgrade minimum Hadoop version to 2.6.0
[
https://issues.apache.org/jira/browse/OOZIE-2780?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16005153#comment-16005153
]
Robert Kanter commented on OOZIE-2780:
--
I filed OOZIE-2886.
> Upgrade minimum Hadoop version to 2.6.0
> ---
>
> Key: OOZIE-2780
> URL: https://issues.apache.org/jira/browse/OOZIE-2780
> Project: Oozie
> Issue Type: Improvement
> Components: core
>Affects Versions: 4.3.0
>Reporter: Artem Ervits
>Assignee: Artem Ervits
>Priority: Minor
> Labels: newbie
> Attachments: OOZIE-2780-0.patch, OOZIE-2780-1.patch,
> OOZIE-2780-2.patch
>
>
> I'd like to contribute an fs action enhancement to modify extended attributes
> to core but methods to change extended attributes is not available in
> hadoop-common < 2.6.0. Currently hadoop-common is at 2.4. I am contributing a
> patch to bump to 2.6 but so far I haven't been able to generate a clean test
> as tests in core module fail randomly. On the first run, I had a failure on
> {code} Failed tests:
> testCheckAfterActionDelete(org.apache.oozie.service.TestHAPartitionDependencyManagerService):
> Expected: but was: [000-170123095541655-oozie-aerv-C@1]{code}
> and on second run
> {code}Failed tests: testWaitFor(org.apache.oozie.test.TestXTestCase):
> expected:<1000.0> but was:<2036.0>{code}
> I can't tell whether it is due to my patch?
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Commented] (OOZIE-2780) Upgrade minimum Hadoop version to 2.6.0
[
https://issues.apache.org/jira/browse/OOZIE-2780?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16005136#comment-16005136
]
Robert Kanter commented on OOZIE-2780:
--
+1
I'll file another JIRA to fix the other hadoop jars issue - the sharelibs
shouldn't have any hadoop jars.
> Upgrade minimum Hadoop version to 2.6.0
> ---
>
> Key: OOZIE-2780
> URL: https://issues.apache.org/jira/browse/OOZIE-2780
> Project: Oozie
> Issue Type: Improvement
> Components: core
>Affects Versions: 4.3.0
>Reporter: Artem Ervits
>Assignee: Artem Ervits
>Priority: Minor
> Labels: newbie
> Attachments: OOZIE-2780-0.patch, OOZIE-2780-1.patch,
> OOZIE-2780-2.patch
>
>
> I'd like to contribute an fs action enhancement to modify extended attributes
> to core but methods to change extended attributes is not available in
> hadoop-common < 2.6.0. Currently hadoop-common is at 2.4. I am contributing a
> patch to bump to 2.6 but so far I haven't been able to generate a clean test
> as tests in core module fail randomly. On the first run, I had a failure on
> {code} Failed tests:
> testCheckAfterActionDelete(org.apache.oozie.service.TestHAPartitionDependencyManagerService):
> Expected: but was: [000-170123095541655-oozie-aerv-C@1]{code}
> and on second run
> {code}Failed tests: testWaitFor(org.apache.oozie.test.TestXTestCase):
> expected:<1000.0> but was:<2036.0>{code}
> I can't tell whether it is due to my patch?
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Commented] (OOZIE-2780) Upgrade minimum Hadoop version to 2.6.0
[
https://issues.apache.org/jira/browse/OOZIE-2780?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16004983#comment-16004983
]
Peter Cseh commented on OOZIE-2780:
---
I've tried it out with a current master: /bin/mkdistro.sh -DskipTests
-Dhadoop.version=2.7.0
{noformat}
./sharelib/target/oozie-sharelib-4.4.0-SNAPSHOT/share/lib/distcp/hadoop-distcp-2.7.0.jar
./sharelib/target/oozie-sharelib-4.4.0-SNAPSHOT/share/lib/hcatalog/parquet-hadoop-bundle-1.6.0.jar
./sharelib/target/oozie-sharelib-4.4.0-SNAPSHOT/share/lib/hive/hadoop-annotations-2.7.0.jar
./sharelib/target/oozie-sharelib-4.4.0-SNAPSHOT/share/lib/hive/hadoop-yarn-api-2.7.0.jar
./sharelib/target/oozie-sharelib-4.4.0-SNAPSHOT/share/lib/hive/hadoop-yarn-common-2.7.0.jar
./sharelib/target/oozie-sharelib-4.4.0-SNAPSHOT/share/lib/hive/hadoop-yarn-server-applicationhistoryservice-2.6.0.jar
./sharelib/target/oozie-sharelib-4.4.0-SNAPSHOT/share/lib/hive/hadoop-yarn-server-common-2.7.0.jar
./sharelib/target/oozie-sharelib-4.4.0-SNAPSHOT/share/lib/hive/hadoop-yarn-server-resourcemanager-2.7.0.jar
./sharelib/target/oozie-sharelib-4.4.0-SNAPSHOT/share/lib/hive/hadoop-yarn-server-web-proxy-2.6.0.jar
./sharelib/target/oozie-sharelib-4.4.0-SNAPSHOT/share/lib/hive/parquet-hadoop-bundle-1.6.0.jar
./sharelib/target/oozie-sharelib-4.4.0-SNAPSHOT/share/lib/hive2/hadoop-annotations-2.7.0.jar
./sharelib/target/oozie-sharelib-4.4.0-SNAPSHOT/share/lib/hive2/hadoop-yarn-api-2.7.0.jar
./sharelib/target/oozie-sharelib-4.4.0-SNAPSHOT/share/lib/hive2/hadoop-yarn-common-2.7.0.jar
./sharelib/target/oozie-sharelib-4.4.0-SNAPSHOT/share/lib/hive2/hadoop-yarn-server-applicationhistoryservice-2.6.0.jar
./sharelib/target/oozie-sharelib-4.4.0-SNAPSHOT/share/lib/hive2/hadoop-yarn-server-common-2.7.0.jar
./sharelib/target/oozie-sharelib-4.4.0-SNAPSHOT/share/lib/hive2/hadoop-yarn-server-resourcemanager-2.7.0.jar
./sharelib/target/oozie-sharelib-4.4.0-SNAPSHOT/share/lib/hive2/hadoop-yarn-server-web-proxy-2.6.0.jar
./sharelib/target/oozie-sharelib-4.4.0-SNAPSHOT/share/lib/hive2/parquet-hadoop-bundle-1.6.0.jar
{noformat}
Without defining a hadoop version it's 2.4.0 mixing with 2.6.0:
{noformat}
./sharelib/target/oozie-sharelib-4.4.0-SNAPSHOT/share/lib/hive2/hadoop-annotations-2.4.0.jar
./sharelib/target/oozie-sharelib-4.4.0-SNAPSHOT/share/lib/hive2/hadoop-yarn-api-2.4.0.jar
./sharelib/target/oozie-sharelib-4.4.0-SNAPSHOT/share/lib/hive2/hadoop-yarn-common-2.4.0.jar
./sharelib/target/oozie-sharelib-4.4.0-SNAPSHOT/share/lib/hive2/hadoop-yarn-server-applicationhistoryservice-2.6.0.jar
./sharelib/target/oozie-sharelib-4.4.0-SNAPSHOT/share/lib/hive2/hadoop-yarn-server-common-2.4.0.jar
./sharelib/target/oozie-sharelib-4.4.0-SNAPSHOT/share/lib/hive2/hadoop-yarn-server-resourcemanager-2.4.0.jar
./sharelib/target/oozie-sharelib-4.4.0-SNAPSHOT/share/lib/hive2/hadoop-yarn-server-web-proxy-2.6.0.jar
{noformat}
I think we can go ahead and commit this. At least it will synchronize the
hadoop versions we're using by default.
So +1 (pending [~rkanter])
> Upgrade minimum Hadoop version to 2.6.0
> ---
>
> Key: OOZIE-2780
> URL: https://issues.apache.org/jira/browse/OOZIE-2780
> Project: Oozie
> Issue Type: Improvement
> Components: core
>Affects Versions: 4.3.0
>Reporter: Artem Ervits
>Assignee: Artem Ervits
>Priority: Minor
> Labels: newbie
> Attachments: OOZIE-2780-0.patch, OOZIE-2780-1.patch,
> OOZIE-2780-2.patch
>
>
> I'd like to contribute an fs action enhancement to modify extended attributes
> to core but methods to change extended attributes is not available in
> hadoop-common < 2.6.0. Currently hadoop-common is at 2.4. I am contributing a
> patch to bump to 2.6 but so far I haven't been able to generate a clean test
> as tests in core module fail randomly. On the first run, I had a failure on
> {code} Failed tests:
> testCheckAfterActionDelete(org.apache.oozie.service.TestHAPartitionDependencyManagerService):
> Expected: but was: [000-170123095541655-oozie-aerv-C@1]{code}
> and on second run
> {code}Failed tests: testWaitFor(org.apache.oozie.test.TestXTestCase):
> expected:<1000.0> but was:<2036.0>{code}
> I can't tell whether it is due to my patch?
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Commented] (OOZIE-2780) Upgrade minimum Hadoop version to 2.6.0
[
https://issues.apache.org/jira/browse/OOZIE-2780?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16004712#comment-16004712
]
Artem Ervits commented on OOZIE-2780:
-
[~gezapeti] & [~rkanter] could use your advice on next steps then? The patch
still adds some improvement and we can tackle the hive issues in separate Jira.
Please advise.
> Upgrade minimum Hadoop version to 2.6.0
> ---
>
> Key: OOZIE-2780
> URL: https://issues.apache.org/jira/browse/OOZIE-2780
> Project: Oozie
> Issue Type: Improvement
> Components: core
>Affects Versions: 4.3.0
>Reporter: Artem Ervits
>Assignee: Artem Ervits
>Priority: Minor
> Labels: newbie
> Attachments: OOZIE-2780-0.patch, OOZIE-2780-1.patch,
> OOZIE-2780-2.patch
>
>
> I'd like to contribute an fs action enhancement to modify extended attributes
> to core but methods to change extended attributes is not available in
> hadoop-common < 2.6.0. Currently hadoop-common is at 2.4. I am contributing a
> patch to bump to 2.6 but so far I haven't been able to generate a clean test
> as tests in core module fail randomly. On the first run, I had a failure on
> {code} Failed tests:
> testCheckAfterActionDelete(org.apache.oozie.service.TestHAPartitionDependencyManagerService):
> Expected: but was: [000-170123095541655-oozie-aerv-C@1]{code}
> and on second run
> {code}Failed tests: testWaitFor(org.apache.oozie.test.TestXTestCase):
> expected:<1000.0> but was:<2036.0>{code}
> I can't tell whether it is due to my patch?
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Commented] (OOZIE-2780) Upgrade minimum Hadoop version to 2.6.0
[
https://issues.apache.org/jira/browse/OOZIE-2780?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16004136#comment-16004136
]
Peter Cseh commented on OOZIE-2780:
---
so I've run a depenency:tree -Dhadoop.version=2.7.0
When the hadoop component is referenced in our poms, the versions are correct.
Where the dependencies are only transitive, they'll stay at 2.6.0, mostly via
hive-something:
{noformat}
+- org.apache.hive.shims:hive-shims-0.23:jar:1.2.0:runtime
[INFO] | | | | \-
org.apache.hadoop:hadoop-yarn-server-resourcemanager:jar:2.6.0:runtime
[INFO] | | | | +- org.apache.hadoop:hadoop-annotations:jar:2.6.0:runtime
[INFO] | | | | +- org.apache.hadoop:hadoop-yarn-common:jar:2.6.0:runtime
[INFO] | | | | +- org.apache.hadoop:hadoop-yarn-api:jar:2.6.0:runtime
{noformat}
I don't think we can do too many things about this.
> Upgrade minimum Hadoop version to 2.6.0
> ---
>
> Key: OOZIE-2780
> URL: https://issues.apache.org/jira/browse/OOZIE-2780
> Project: Oozie
> Issue Type: Improvement
> Components: core
>Affects Versions: 4.3.0
>Reporter: Artem Ervits
>Assignee: Artem Ervits
>Priority: Minor
> Labels: newbie
> Attachments: OOZIE-2780-0.patch, OOZIE-2780-1.patch,
> OOZIE-2780-2.patch
>
>
> I'd like to contribute an fs action enhancement to modify extended attributes
> to core but methods to change extended attributes is not available in
> hadoop-common < 2.6.0. Currently hadoop-common is at 2.4. I am contributing a
> patch to bump to 2.6 but so far I haven't been able to generate a clean test
> as tests in core module fail randomly. On the first run, I had a failure on
> {code} Failed tests:
> testCheckAfterActionDelete(org.apache.oozie.service.TestHAPartitionDependencyManagerService):
> Expected: but was: [000-170123095541655-oozie-aerv-C@1]{code}
> and on second run
> {code}Failed tests: testWaitFor(org.apache.oozie.test.TestXTestCase):
> expected:<1000.0> but was:<2036.0>{code}
> I can't tell whether it is due to my patch?
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Commented] (OOZIE-2780) Upgrade minimum Hadoop version to 2.6.0
[
https://issues.apache.org/jira/browse/OOZIE-2780?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16003791#comment-16003791
]
Artem Ervits commented on OOZIE-2780:
-
[~rkanter] I'm still chasing this, ran the build in debug mode and found the
following
{code}
Overriding previous definition of property "hadoop.version"
Setting project property: hadoop.version -> 2.6.0
{code}
can't tell from the output where it's happening yet.
I am referring to hard-coded part here
https://github.com/apache/oozie/blob/f5554dd315c730286edbb333b4008e48620e2acc/pom.xml#L261-L265
I fixed it as part of my patch by referring to hadoop.version. Also curiously,
found https://issues.apache.org/jira/browse/OOZIE-2884 as it is causing me to
go on wild goose chase and will be addressing both Jiras. I am able to
reproduce your output.
{noformat}
find . -name hadoop*.jar | grep 2.6.0
./distro/target/oozie-4.4.0-SNAPSHOT-distro/oozie-4.4.0-SNAPSHOT/embedded-oozie-server/webapp/WEB-INF/lib/hadoop-annotations-2.6.0.jar
./distro/target/oozie-4.4.0-SNAPSHOT-distro/oozie-4.4.0-SNAPSHOT/embedded-oozie-server/webapp/WEB-INF/lib/hadoop-yarn-api-2.6.0.jar
./distro/target/oozie-4.4.0-SNAPSHOT-distro/oozie-4.4.0-SNAPSHOT/embedded-oozie-server/webapp/WEB-INF/lib/hadoop-yarn-common-2.6.0.jar
./distro/target/oozie-4.4.0-SNAPSHOT-distro/oozie-4.4.0-SNAPSHOT/embedded-oozie-server/webapp/WEB-INF/lib/hadoop-yarn-server-applicationhistoryservice-2.6.0.jar
./distro/target/oozie-4.4.0-SNAPSHOT-distro/oozie-4.4.0-SNAPSHOT/embedded-oozie-server/webapp/WEB-INF/lib/hadoop-yarn-server-common-2.6.0.jar
./distro/target/oozie-4.4.0-SNAPSHOT-distro/oozie-4.4.0-SNAPSHOT/embedded-oozie-server/webapp/WEB-INF/lib/hadoop-yarn-server-resourcemanager-2.6.0.jar
./distro/target/oozie-4.4.0-SNAPSHOT-distro/oozie-4.4.0-SNAPSHOT/embedded-oozie-server/webapp/WEB-INF/lib/hadoop-yarn-server-web-proxy-2.6.0.jar
./server/target/dependency/hadoop-yarn-api-2.6.0.jar
./server/target/dependency/hadoop-yarn-common-2.6.0.jar
./server/target/dependency/hadoop-yarn-server-applicationhistoryservice-2.6.0.jar
./server/target/dependency/hadoop-yarn-server-common-2.6.0.jar
./server/target/dependency/hadoop-yarn-server-resourcemanager-2.6.0.jar
./server/target/dependency/hadoop-yarn-server-web-proxy-2.6.0.jar
./sharelib/hive/target/partial-sharelib/share/lib/hive/hadoop-yarn-server-applicationhistoryservice-2.6.0.jar
./sharelib/hive/target/partial-sharelib/share/lib/hive/hadoop-yarn-server-web-proxy-2.6.0.jar
./sharelib/hive2/target/partial-sharelib/share/lib/hive2/hadoop-yarn-server-applicationhistoryservice-2.6.0.jar
./sharelib/hive2/target/partial-sharelib/share/lib/hive2/hadoop-yarn-server-web-proxy-2.6.0.jar
./sharelib/target/oozie-sharelib-4.4.0-SNAPSHOT/share/lib/hive/hadoop-yarn-server-applicationhistoryservice-2.6.0.jar
./sharelib/target/oozie-sharelib-4.4.0-SNAPSHOT/share/lib/hive/hadoop-yarn-server-web-proxy-2.6.0.jar
./sharelib/target/oozie-sharelib-4.4.0-SNAPSHOT/share/lib/hive2/hadoop-yarn-server-applicationhistoryservice-2.6.0.jar
./sharelib/target/oozie-sharelib-4.4.0-SNAPSHOT/share/lib/hive2/hadoop-yarn-server-web-proxy-2.6.0.jar
./webapp/target/oozie-webapp-4.4.0-SNAPSHOT/WEB-INF/lib/hadoop-annotations-2.6.0.jar
./webapp/target/oozie-webapp-4.4.0-SNAPSHOT/WEB-INF/lib/hadoop-yarn-api-2.6.0.jar
./webapp/target/oozie-webapp-4.4.0-SNAPSHOT/WEB-INF/lib/hadoop-yarn-common-2.6.0.jar
./webapp/target/oozie-webapp-4.4.0-SNAPSHOT/WEB-INF/lib/hadoop-yarn-server-applicationhistoryservice-2.6.0.jar
./webapp/target/oozie-webapp-4.4.0-SNAPSHOT/WEB-INF/lib/hadoop-yarn-server-common-2.6.0.jar
./webapp/target/oozie-webapp-4.4.0-SNAPSHOT/WEB-INF/lib/hadoop-yarn-server-resourcemanager-2.6.0.jar
./webapp/target/oozie-webapp-4.4.0-SNAPSHOT/WEB-INF/lib/hadoop-yarn-server-web-proxy-2.6.0.jar
{noformat}
> Upgrade minimum Hadoop version to 2.6.0
> ---
>
> Key: OOZIE-2780
> URL: https://issues.apache.org/jira/browse/OOZIE-2780
> Project: Oozie
> Issue Type: Improvement
> Components: core
>Affects Versions: 4.3.0
>Reporter: Artem Ervits
>Assignee: Artem Ervits
>Priority: Minor
> Labels: newbie
> Attachments: OOZIE-2780-0.patch, OOZIE-2780-1.patch,
> OOZIE-2780-2.patch
>
>
> I'd like to contribute an fs action enhancement to modify extended attributes
> to core but methods to change extended attributes is not available in
> hadoop-common < 2.6.0. Currently hadoop-common is at 2.4. I am contributing a
> patch to bump to 2.6 but so far I haven't been able to generate a clean test
> as tests in core module fail randomly. On the first run, I had a failure on
> {code} Failed tests:
> testCheckAfterActionDelete(org.apache.oozie.service.TestHAPartitionDependencyManagerService):
> Expected: but was: [000-170123095541655-oozie-aerv-C@1]{code}
> and on seco
[jira] [Commented] (OOZIE-2780) Upgrade minimum Hadoop version to 2.6.0
[
https://issues.apache.org/jira/browse/OOZIE-2780?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16001713#comment-16001713
]
Robert Kanter commented on OOZIE-2780:
--
Thanks [~dbist13] for finding those. Now it works correctly and only pulls in
Hadoop 2.6.0 jars. However, if I try to build with a newer Hadoop, there's
also Hadoop 2.6.0 jars mixed in. For instance:
{noformat}
mvn clean package assembly:single -DskipTests -Dhadoop.version=2.7.0
{noformat}
{noformat}
>> [95] 15:48 : oozie-git (master) :: find . -name hadoop*.jar | grep 2.7.0
./client/target/oozie-client-4.4.0-SNAPSHOT-client/oozie-client-4.4.0-SNAPSHOT/lib/hadoop-auth-2.7.0.jar
./server/target/dependency/hadoop-annotations-2.7.0.jar
./server/target/dependency/hadoop-auth-2.7.0.jar
./server/target/dependency/hadoop-common-2.7.0.jar
./sharelib/distcp/target/partial-sharelib/share/lib/distcp/hadoop-distcp-2.7.0.jar
./sharelib/hive/target/partial-sharelib/share/lib/hive/hadoop-annotations-2.7.0.jar
./sharelib/hive/target/partial-sharelib/share/lib/hive/hadoop-yarn-api-2.7.0.jar
./sharelib/hive/target/partial-sharelib/share/lib/hive/hadoop-yarn-common-2.7.0.jar
./sharelib/hive/target/partial-sharelib/share/lib/hive/hadoop-yarn-server-common-2.7.0.jar
./sharelib/hive/target/partial-sharelib/share/lib/hive/hadoop-yarn-server-resourcemanager-2.7.0.jar
./sharelib/hive2/target/partial-sharelib/share/lib/hive2/hadoop-annotations-2.7.0.jar
./sharelib/hive2/target/partial-sharelib/share/lib/hive2/hadoop-yarn-api-2.7.0.jar
./sharelib/hive2/target/partial-sharelib/share/lib/hive2/hadoop-yarn-common-2.7.0.jar
./sharelib/hive2/target/partial-sharelib/share/lib/hive2/hadoop-yarn-server-common-2.7.0.jar
./sharelib/hive2/target/partial-sharelib/share/lib/hive2/hadoop-yarn-server-resourcemanager-2.7.0.jar
./sharelib/streaming/target/partial-sharelib/share/lib/mapreduce-streaming/hadoop-streaming-2.7.0.jar
./sharelib/target/oozie-sharelib-4.4.0-SNAPSHOT/share/lib/distcp/hadoop-distcp-2.7.0.jar
./sharelib/target/oozie-sharelib-4.4.0-SNAPSHOT/share/lib/hive/hadoop-annotations-2.7.0.jar
./sharelib/target/oozie-sharelib-4.4.0-SNAPSHOT/share/lib/hive/hadoop-yarn-api-2.7.0.jar
./sharelib/target/oozie-sharelib-4.4.0-SNAPSHOT/share/lib/hive/hadoop-yarn-common-2.7.0.jar
./sharelib/target/oozie-sharelib-4.4.0-SNAPSHOT/share/lib/hive/hadoop-yarn-server-common-2.7.0.jar
./sharelib/target/oozie-sharelib-4.4.0-SNAPSHOT/share/lib/hive/hadoop-yarn-server-resourcemanager-2.7.0.jar
./sharelib/target/oozie-sharelib-4.4.0-SNAPSHOT/share/lib/hive2/hadoop-annotations-2.7.0.jar
./sharelib/target/oozie-sharelib-4.4.0-SNAPSHOT/share/lib/hive2/hadoop-yarn-api-2.7.0.jar
./sharelib/target/oozie-sharelib-4.4.0-SNAPSHOT/share/lib/hive2/hadoop-yarn-common-2.7.0.jar
./sharelib/target/oozie-sharelib-4.4.0-SNAPSHOT/share/lib/hive2/hadoop-yarn-server-common-2.7.0.jar
./sharelib/target/oozie-sharelib-4.4.0-SNAPSHOT/share/lib/hive2/hadoop-yarn-server-resourcemanager-2.7.0.jar
./sharelib/target/oozie-sharelib-4.4.0-SNAPSHOT/share/lib/mapreduce-streaming/hadoop-streaming-2.7.0.jar
{noformat}
{noformat}
>> [96] 15:48 : oozie-git (master) :: find . -name hadoop*.jar | grep 2.6.0
./distro/target/oozie-4.4.0-SNAPSHOT-distro/oozie-4.4.0-SNAPSHOT/embedded-oozie-server/webapp/WEB-INF/lib/hadoop-annotations-2.6.0.jar
./distro/target/oozie-4.4.0-SNAPSHOT-distro/oozie-4.4.0-SNAPSHOT/embedded-oozie-server/webapp/WEB-INF/lib/hadoop-yarn-api-2.6.0.jar
./distro/target/oozie-4.4.0-SNAPSHOT-distro/oozie-4.4.0-SNAPSHOT/embedded-oozie-server/webapp/WEB-INF/lib/hadoop-yarn-common-2.6.0.jar
./distro/target/oozie-4.4.0-SNAPSHOT-distro/oozie-4.4.0-SNAPSHOT/embedded-oozie-server/webapp/WEB-INF/lib/hadoop-yarn-server-applicationhistoryservice-2.6.0.jar
./distro/target/oozie-4.4.0-SNAPSHOT-distro/oozie-4.4.0-SNAPSHOT/embedded-oozie-server/webapp/WEB-INF/lib/hadoop-yarn-server-common-2.6.0.jar
./distro/target/oozie-4.4.0-SNAPSHOT-distro/oozie-4.4.0-SNAPSHOT/embedded-oozie-server/webapp/WEB-INF/lib/hadoop-yarn-server-resourcemanager-2.6.0.jar
./distro/target/oozie-4.4.0-SNAPSHOT-distro/oozie-4.4.0-SNAPSHOT/embedded-oozie-server/webapp/WEB-INF/lib/hadoop-yarn-server-web-proxy-2.6.0.jar
./server/target/dependency/hadoop-yarn-api-2.6.0.jar
./server/target/dependency/hadoop-yarn-common-2.6.0.jar
./server/target/dependency/hadoop-yarn-server-applicationhistoryservice-2.6.0.jar
./server/target/dependency/hadoop-yarn-server-common-2.6.0.jar
./server/target/dependency/hadoop-yarn-server-resourcemanager-2.6.0.jar
./server/target/dependency/hadoop-yarn-server-web-proxy-2.6.0.jar
./sharelib/hive/target/partial-sharelib/share/lib/hive/hadoop-yarn-server-applicationhistoryservice-2.6.0.jar
./sharelib/hive/target/partial-sharelib/share/lib/hive/hadoop-yarn-server-web-proxy-2.6.0.jar
./sharelib/hive2/target/partial-sharelib/share/lib/hive2/hadoop-yarn-server-applicationhistoryservice-2.6.0.jar
./sharelib/hive2/target/partial-sharelib/share/lib/hive2/had
[jira] [Commented] (OOZIE-2780) Upgrade minimum Hadoop version to 2.6.0
[
https://issues.apache.org/jira/browse/OOZIE-2780?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15999312#comment-15999312
]
Hadoop QA commented on OOZIE-2780:
--
Testing JIRA OOZIE-2780
Cleaning local git workspace
{color:green}+1 PATCH_APPLIES{color}
{color:green}+1 CLEAN{color}
{color:red}-1 RAW_PATCH_ANALYSIS{color}
.{color:green}+1{color} the patch does not introduce any @author tags
.{color:green}+1{color} the patch does not introduce any tabs
.{color:green}+1{color} the patch does not introduce any trailing spaces
.{color:green}+1{color} the patch does not introduce any line longer than
132
.{color:red}-1{color} the patch does not add/modify any testcase
{color:green}+1 RAT{color}
.{color:green}+1{color} the patch does not seem to introduce new RAT
warnings
{color:green}+1 JAVADOC{color}
.{color:green}+1{color} the patch does not seem to introduce new Javadoc
warnings
{color:green}+1 COMPILE{color}
.{color:green}+1{color} HEAD compiles
.{color:green}+1{color} patch compiles
.{color:green}+1{color} the patch does not seem to introduce new javac
warnings
{color:green}+1{color} There are no new bugs found in total.
. {color:green}+1{color} There are no new bugs found in [server].
. {color:green}+1{color} There are no new bugs found in [client].
. {color:green}+1{color} There are no new bugs found in [core].
. {color:green}+1{color} There are no new bugs found in [docs].
. {color:green}+1{color} There are no new bugs found in
[hadooplibs/hadoop-utils-2].
. {color:green}+1{color} There are no new bugs found in [tools].
. {color:green}+1{color} There are no new bugs found in [examples].
. {color:green}+1{color} There are no new bugs found in [sharelib/streaming].
. {color:green}+1{color} There are no new bugs found in [sharelib/sqoop].
. {color:green}+1{color} There are no new bugs found in [sharelib/distcp].
. {color:green}+1{color} There are no new bugs found in [sharelib/oozie].
. {color:green}+1{color} There are no new bugs found in [sharelib/hcatalog].
. {color:green}+1{color} There are no new bugs found in [sharelib/hive].
. {color:green}+1{color} There are no new bugs found in [sharelib/hive2].
. {color:green}+1{color} There are no new bugs found in [sharelib/pig].
. {color:green}+1{color} There are no new bugs found in [sharelib/spark].
{color:green}+1 BACKWARDS_COMPATIBILITY{color}
.{color:green}+1{color} the patch does not change any JPA
Entity/Colum/Basic/Lob/Transient annotations
.{color:green}+1{color} the patch does not modify JPA files
{color:red}-1 TESTS{color}
.Tests run: 1934
.Tests failed: 1
.Tests errors: 2
.The patch failed the following testcases:
.
testUpdateConfForTimeLineServiceEnabled(org.apache.oozie.action.hadoop.TestJavaActionExecutor)
.Tests failing with errors:
. testNewUsingACLs(org.apache.oozie.util.TestZKUtilsWithSecurity)
. testCheckAndSetACLs(org.apache.oozie.util.TestZKUtilsWithSecurity)
{color:green}+1 DISTRO{color}
.{color:green}+1{color} distro tarball builds with the patch
{color:red}*-1 Overall result, please check the reported -1(s)*{color}
The full output of the test-patch run is available at
. https://builds.apache.org/job/oozie-trunk-precommit-build/3807/
> Upgrade minimum Hadoop version to 2.6.0
> ---
>
> Key: OOZIE-2780
> URL: https://issues.apache.org/jira/browse/OOZIE-2780
> Project: Oozie
> Issue Type: Improvement
> Components: core
>Affects Versions: 4.3.0
>Reporter: Artem Ervits
>Assignee: Artem Ervits
>Priority: Minor
> Labels: newbie
> Attachments: OOZIE-2780-0.patch, OOZIE-2780-1.patch,
> OOZIE-2780-2.patch
>
>
> I'd like to contribute an fs action enhancement to modify extended attributes
> to core but methods to change extended attributes is not available in
> hadoop-common < 2.6.0. Currently hadoop-common is at 2.4. I am contributing a
> patch to bump to 2.6 but so far I haven't been able to generate a clean test
> as tests in core module fail randomly. On the first run, I had a failure on
> {code} Failed tests:
> testCheckAfterActionDelete(org.apache.oozie.service.TestHAPartitionDependencyManagerService):
> Expected: but was: [000-170123095541655-oozie-aerv-C@1]{code}
> and on second run
> {code}Failed tests: testWaitFor(org.apache.oozie.test.TestXTestCase):
> expected:<1000.0> but was:<2036.0>{code}
> I can't tell whether it is due to my patch?
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Commented] (OOZIE-2780) Upgrade minimum Hadoop version to 2.6.0
[
https://issues.apache.org/jira/browse/OOZIE-2780?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15999055#comment-15999055
]
Robert Kanter commented on OOZIE-2780:
--
Hmmm, it looks like it's pulling in a mix of Hadoop 2.4.0 and 2.6.0 jars:
{noformat}
>> [25] 15:22 : oozie-git (master) :: find . -name hadoop*.jar | grep 2.4.0
./client/target/oozie-client-4.4.0-SNAPSHOT-client/oozie-client-4.4.0-SNAPSHOT/lib/hadoop-auth-2.4.0.jar
./server/target/dependency/hadoop-annotations-2.4.0.jar
./server/target/dependency/hadoop-auth-2.4.0.jar
./server/target/dependency/hadoop-common-2.4.0.jar
./sharelib/distcp/target/partial-sharelib/share/lib/distcp/hadoop-distcp-2.4.0.jar
./sharelib/hive/target/partial-sharelib/share/lib/hive/hadoop-annotations-2.4.0.jar
./sharelib/hive/target/partial-sharelib/share/lib/hive/hadoop-yarn-api-2.4.0.jar
./sharelib/hive/target/partial-sharelib/share/lib/hive/hadoop-yarn-common-2.4.0.jar
./sharelib/hive/target/partial-sharelib/share/lib/hive/hadoop-yarn-server-common-2.4.0.jar
./sharelib/hive/target/partial-sharelib/share/lib/hive/hadoop-yarn-server-resourcemanager-2.4.0.jar
./sharelib/hive2/target/partial-sharelib/share/lib/hive2/hadoop-annotations-2.4.0.jar
./sharelib/hive2/target/partial-sharelib/share/lib/hive2/hadoop-yarn-api-2.4.0.jar
./sharelib/hive2/target/partial-sharelib/share/lib/hive2/hadoop-yarn-common-2.4.0.jar
./sharelib/hive2/target/partial-sharelib/share/lib/hive2/hadoop-yarn-server-common-2.4.0.jar
./sharelib/hive2/target/partial-sharelib/share/lib/hive2/hadoop-yarn-server-resourcemanager-2.4.0.jar
./sharelib/streaming/target/partial-sharelib/share/lib/mapreduce-streaming/hadoop-streaming-2.4.0.jar
./sharelib/target/oozie-sharelib-4.4.0-SNAPSHOT/share/lib/distcp/hadoop-distcp-2.4.0.jar
./sharelib/target/oozie-sharelib-4.4.0-SNAPSHOT/share/lib/hive/hadoop-annotations-2.4.0.jar
./sharelib/target/oozie-sharelib-4.4.0-SNAPSHOT/share/lib/hive/hadoop-yarn-api-2.4.0.jar
./sharelib/target/oozie-sharelib-4.4.0-SNAPSHOT/share/lib/hive/hadoop-yarn-common-2.4.0.jar
./sharelib/target/oozie-sharelib-4.4.0-SNAPSHOT/share/lib/hive/hadoop-yarn-server-common-2.4.0.jar
./sharelib/target/oozie-sharelib-4.4.0-SNAPSHOT/share/lib/hive/hadoop-yarn-server-resourcemanager-2.4.0.jar
./sharelib/target/oozie-sharelib-4.4.0-SNAPSHOT/share/lib/hive2/hadoop-annotations-2.4.0.jar
./sharelib/target/oozie-sharelib-4.4.0-SNAPSHOT/share/lib/hive2/hadoop-yarn-api-2.4.0.jar
./sharelib/target/oozie-sharelib-4.4.0-SNAPSHOT/share/lib/hive2/hadoop-yarn-common-2.4.0.jar
./sharelib/target/oozie-sharelib-4.4.0-SNAPSHOT/share/lib/hive2/hadoop-yarn-server-common-2.4.0.jar
./sharelib/target/oozie-sharelib-4.4.0-SNAPSHOT/share/lib/hive2/hadoop-yarn-server-resourcemanager-2.4.0.jar
./sharelib/target/oozie-sharelib-4.4.0-SNAPSHOT/share/lib/mapreduce-streaming/hadoop-streaming-2.4.0.jar
{noformat}
{noformat}
>> [26] 15:22 : oozie-git (master) :: find . -name hadoop*.jar | grep 2.6.0
./distro/target/oozie-4.4.0-SNAPSHOT-distro/oozie-4.4.0-SNAPSHOT/embedded-oozie-server/webapp/WEB-INF/lib/hadoop-annotations-2.6.0.jar
./distro/target/oozie-4.4.0-SNAPSHOT-distro/oozie-4.4.0-SNAPSHOT/embedded-oozie-server/webapp/WEB-INF/lib/hadoop-yarn-api-2.6.0.jar
./distro/target/oozie-4.4.0-SNAPSHOT-distro/oozie-4.4.0-SNAPSHOT/embedded-oozie-server/webapp/WEB-INF/lib/hadoop-yarn-common-2.6.0.jar
./distro/target/oozie-4.4.0-SNAPSHOT-distro/oozie-4.4.0-SNAPSHOT/embedded-oozie-server/webapp/WEB-INF/lib/hadoop-yarn-server-applicationhistoryservice-2.6.0.jar
./distro/target/oozie-4.4.0-SNAPSHOT-distro/oozie-4.4.0-SNAPSHOT/embedded-oozie-server/webapp/WEB-INF/lib/hadoop-yarn-server-common-2.6.0.jar
./distro/target/oozie-4.4.0-SNAPSHOT-distro/oozie-4.4.0-SNAPSHOT/embedded-oozie-server/webapp/WEB-INF/lib/hadoop-yarn-server-resourcemanager-2.6.0.jar
./distro/target/oozie-4.4.0-SNAPSHOT-distro/oozie-4.4.0-SNAPSHOT/embedded-oozie-server/webapp/WEB-INF/lib/hadoop-yarn-server-web-proxy-2.6.0.jar
./server/target/dependency/hadoop-yarn-api-2.6.0.jar
./server/target/dependency/hadoop-yarn-common-2.6.0.jar
./server/target/dependency/hadoop-yarn-server-applicationhistoryservice-2.6.0.jar
./server/target/dependency/hadoop-yarn-server-common-2.6.0.jar
./server/target/dependency/hadoop-yarn-server-resourcemanager-2.6.0.jar
./server/target/dependency/hadoop-yarn-server-web-proxy-2.6.0.jar
./sharelib/hive/target/partial-sharelib/share/lib/hive/hadoop-yarn-server-applicationhistoryservice-2.6.0.jar
./sharelib/hive/target/partial-sharelib/share/lib/hive/hadoop-yarn-server-web-proxy-2.6.0.jar
./sharelib/hive2/target/partial-sharelib/share/lib/hive2/hadoop-yarn-server-applicationhistoryservice-2.6.0.jar
./sharelib/hive2/target/partial-sharelib/share/lib/hive2/hadoop-yarn-server-web-proxy-2.6.0.jar
./sharelib/target/oozie-sharelib-4.4.0-SNAPSHOT/share/lib/hive/hadoop-yarn-server-applicationhistoryservice-2.6.0.jar
./sharelib/target/oozie-sharelib-4.4.0-SNAPSHOT/share/lib/hive/hadoo
