Build failed in Jenkins: Hadoop-Common-trunk #1527
See https://builds.apache.org/job/Hadoop-Common-trunk/1527/changes Changes: [yzhang] HDFS-8596. TestDistributedFileSystem et al tests are broken in branch-2 due to incorrect setting of datanode attribute. Contributed by Yongjun Zhang. [arp] HDFS-8595. TestCommitBlockSynchronization fails in branch-2.7. (Patch applies to all branches). (Contributed by Arpit Agarwal) -- [...truncated 5173 lines...] Tests run: 16, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 1.053 sec - in org.apache.hadoop.util.TestFindClass Running org.apache.hadoop.util.TestLightWeightGSet Tests run: 2, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.206 sec - in org.apache.hadoop.util.TestLightWeightGSet Running org.apache.hadoop.util.TestPureJavaCrc32 Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.703 sec - in org.apache.hadoop.util.TestPureJavaCrc32 Running org.apache.hadoop.util.TestStringUtils Tests run: 14, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.461 sec - in org.apache.hadoop.util.TestStringUtils Running org.apache.hadoop.util.TestProtoUtil Tests run: 2, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.482 sec - in org.apache.hadoop.util.TestProtoUtil Running org.apache.hadoop.util.TestSignalLogger Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.279 sec - in org.apache.hadoop.util.TestSignalLogger Running org.apache.hadoop.util.TestDiskChecker Tests run: 15, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 1.874 sec - in org.apache.hadoop.util.TestDiskChecker Running org.apache.hadoop.util.TestShutdownThreadsHelper Tests run: 2, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 3.324 sec - in org.apache.hadoop.util.TestShutdownThreadsHelper Running org.apache.hadoop.util.TestCacheableIPList Tests run: 4, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 1.421 sec - in org.apache.hadoop.util.TestCacheableIPList Running org.apache.hadoop.util.TestLineReader Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.349 sec - in org.apache.hadoop.util.TestLineReader Running org.apache.hadoop.util.TestIdentityHashStore Tests run: 3, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.28 sec - in org.apache.hadoop.util.TestIdentityHashStore Running org.apache.hadoop.util.TestClasspath Tests run: 7, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.676 sec - in org.apache.hadoop.util.TestClasspath Running org.apache.hadoop.util.TestApplicationClassLoader Tests run: 4, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.329 sec - in org.apache.hadoop.util.TestApplicationClassLoader Running org.apache.hadoop.util.TestShell Tests run: 4, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 5.573 sec - in org.apache.hadoop.util.TestShell Running org.apache.hadoop.util.TestShutdownHookManager Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.436 sec - in org.apache.hadoop.util.TestShutdownHookManager Running org.apache.hadoop.util.TestConfTest Tests run: 14, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.613 sec - in org.apache.hadoop.util.TestConfTest Running org.apache.hadoop.util.TestHttpExceptionUtils Tests run: 7, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.963 sec - in org.apache.hadoop.util.TestHttpExceptionUtils Running org.apache.hadoop.util.TestJarFinder Tests run: 4, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.863 sec - in org.apache.hadoop.util.TestJarFinder Running org.apache.hadoop.util.hash.TestHash Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.464 sec - in org.apache.hadoop.util.hash.TestHash Running org.apache.hadoop.util.TestLightWeightCache Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 10.486 sec - in org.apache.hadoop.util.TestLightWeightCache Running org.apache.hadoop.util.TestNativeCodeLoader Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.174 sec - in org.apache.hadoop.util.TestNativeCodeLoader Running org.apache.hadoop.util.TestReflectionUtils Tests run: 5, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.698 sec - in org.apache.hadoop.util.TestReflectionUtils Running org.apache.hadoop.crypto.TestCryptoStreams Tests run: 14, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 39.851 sec - in org.apache.hadoop.crypto.TestCryptoStreams Running org.apache.hadoop.crypto.TestCryptoStreamsForLocalFS Tests run: 14, Failures: 0, Errors: 0, Skipped: 5, Time elapsed: 17.942 sec - in org.apache.hadoop.crypto.TestCryptoStreamsForLocalFS Running org.apache.hadoop.crypto.TestOpensslCipher Tests run: 3, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.3 sec - in org.apache.hadoop.crypto.TestOpensslCipher Running org.apache.hadoop.crypto.TestCryptoStreamsWithOpensslAesCtrCryptoCodec Tests run: 14, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 27.592 sec - in org.apache.hadoop.crypto.TestCryptoStreamsWithOpensslAesCtrCryptoCodec Running org.apache.hadoop.crypto.TestCryptoStreamsNormal Tests run:
Build failed in Jenkins: Hadoop-common-trunk-Java8 #229
See https://builds.apache.org/job/Hadoop-common-trunk-Java8/229/changes Changes: [yzhang] HDFS-8596. TestDistributedFileSystem et al tests are broken in branch-2 due to incorrect setting of datanode attribute. Contributed by Yongjun Zhang. [arp] HDFS-8595. TestCommitBlockSynchronization fails in branch-2.7. (Patch applies to all branches). (Contributed by Arpit Agarwal) -- [...truncated 5535 lines...] Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.707 sec - in org.apache.hadoop.security.TestAuthenticationFilter Java HotSpot(TM) 64-Bit Server VM warning: ignoring option MaxPermSize=768m; support was removed in 8.0 Running org.apache.hadoop.security.TestGroupFallback Tests run: 4, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.798 sec - in org.apache.hadoop.security.TestGroupFallback Java HotSpot(TM) 64-Bit Server VM warning: ignoring option MaxPermSize=768m; support was removed in 8.0 Running org.apache.hadoop.security.TestProxyUserFromEnv Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.684 sec - in org.apache.hadoop.security.TestProxyUserFromEnv Java HotSpot(TM) 64-Bit Server VM warning: ignoring option MaxPermSize=768m; support was removed in 8.0 Running org.apache.hadoop.security.TestUGILoginFromKeytab Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 6.404 sec - in org.apache.hadoop.security.TestUGILoginFromKeytab Java HotSpot(TM) 64-Bit Server VM warning: ignoring option MaxPermSize=768m; support was removed in 8.0 Running org.apache.hadoop.security.TestLdapGroupsMappingWithPosixGroup Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.672 sec - in org.apache.hadoop.security.TestLdapGroupsMappingWithPosixGroup Java HotSpot(TM) 64-Bit Server VM warning: ignoring option MaxPermSize=768m; support was removed in 8.0 Running org.apache.hadoop.security.TestShellBasedIdMapping Tests run: 7, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 2.862 sec - in org.apache.hadoop.security.TestShellBasedIdMapping Java HotSpot(TM) 64-Bit Server VM warning: ignoring option MaxPermSize=768m; support was removed in 8.0 Running org.apache.hadoop.security.TestDoAsEffectiveUser Tests run: 9, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 1.333 sec - in org.apache.hadoop.security.TestDoAsEffectiveUser Java HotSpot(TM) 64-Bit Server VM warning: ignoring option MaxPermSize=768m; support was removed in 8.0 Running org.apache.hadoop.security.TestUGIWithExternalKdc Tests run: 1, Failures: 0, Errors: 0, Skipped: 1, Time elapsed: 0.076 sec - in org.apache.hadoop.security.TestUGIWithExternalKdc Java HotSpot(TM) 64-Bit Server VM warning: ignoring option MaxPermSize=768m; support was removed in 8.0 Running org.apache.hadoop.security.TestJNIGroupsMapping Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.469 sec - in org.apache.hadoop.security.TestJNIGroupsMapping Java HotSpot(TM) 64-Bit Server VM warning: ignoring option MaxPermSize=768m; support was removed in 8.0 Running org.apache.hadoop.security.TestGroupsCaching Tests run: 10, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 1.643 sec - in org.apache.hadoop.security.TestGroupsCaching Java HotSpot(TM) 64-Bit Server VM warning: ignoring option MaxPermSize=768m; support was removed in 8.0 Running org.apache.hadoop.security.alias.TestCredentialProviderFactory Tests run: 6, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 1.179 sec - in org.apache.hadoop.security.alias.TestCredentialProviderFactory Java HotSpot(TM) 64-Bit Server VM warning: ignoring option MaxPermSize=768m; support was removed in 8.0 Running org.apache.hadoop.security.alias.TestCredentialProvider Tests run: 2, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.1 sec - in org.apache.hadoop.security.alias.TestCredentialProvider Java HotSpot(TM) 64-Bit Server VM warning: ignoring option MaxPermSize=768m; support was removed in 8.0 Running org.apache.hadoop.security.alias.TestCredShell Tests run: 9, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 1.332 sec - in org.apache.hadoop.security.alias.TestCredShell Java HotSpot(TM) 64-Bit Server VM warning: ignoring option MaxPermSize=768m; support was removed in 8.0 Running org.apache.hadoop.security.ssl.TestReloadingX509TrustManager Tests run: 5, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 2.61 sec - in org.apache.hadoop.security.ssl.TestReloadingX509TrustManager Java HotSpot(TM) 64-Bit Server VM warning: ignoring option MaxPermSize=768m; support was removed in 8.0 Running org.apache.hadoop.security.ssl.TestSSLFactory Tests run: 15, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 3.389 sec - in org.apache.hadoop.security.ssl.TestSSLFactory Java HotSpot(TM) 64-Bit Server VM warning: ignoring option MaxPermSize=768m; support was removed in 8.0 Running org.apache.hadoop.security.TestLdapGroupsMapping Tests run: 5, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 1.246 sec - in
Re: set up jenkins test for branch-2
On Jun 14, 2015, at 1:00 PM, Yongjun Zhang yzh...@cloudera.com wrote: From time to time we saw changes that work fine in trunk but not branch-2, and we don't catch the issue in a timely manner. The difference between trunk and branch-2 is sufficient to justify periodic jenkins test and even pre-commit test for branch-2. What is your expected outcome? The current nightly builds have been failing pretty consistently for a while now. Adding another one isn't going to make the situation any better since everyone pretty much ignores them. As I said at the HDFS BOF: we desperately need a moratorium on adding new features and need to work on stabilizing both branch-2 and trunk. We seem to be in a quantity over quality mode and it's slowly killing the project. We haven't had a stable release since 2.4.1, which is approaching the 1 year mark. (Yes, I know. Various vendors have shipped releases since 2.4.1 as part of their distributions. Before saying anything, perhaps you should spend some time with your support folks….) As Sean pointed out, precommit for branch-2 has been working for a while now if one provides enough hints to test-patch as part of the patch name.
Re: Protocol Buffers version
I can't answer the original question but can point out the protostuff ( https://github.com/protostuff/protostuff) folks have been responsive and friendly in the past when we (HBase) were curious about swapping in their stuff. Two significant benefits of protostuff, IMHO, is ASL 2 licensing and everything is implemented in Java including the compiler. On Mon, Jun 15, 2015 at 8:49 AM, Sean Busbey bus...@cloudera.com wrote: Anyone have a read on how the protobuf folks would feel about that? Apache has a history of not accepting projects that are non-amicable forks. On Mon, Jun 15, 2015 at 9:24 AM, Allen Wittenauer a...@altiscale.com wrote: On Jun 12, 2015, at 1:03 PM, Alan Burlison alan.burli...@oracle.com wrote: On 14/05/2015 18:41, Chris Nauroth wrote: As a reminder though, the community probably would want to see a strong justification for the upgrade in terms of features or performance or something else. Right now, I'm not seeing a significant benefit for us based on my reading of their release notes. I think it's worthwhile to figure this out first. Otherwise, there is a risk that any testing work turns out to be a wasted effort. One reason at least: PB 2.5.0 has no support for Solaris SPARC. 2.6.1 does. That's a pretty good reason. Some of us had a discussion at Summit about effectively forking protobuf and making it an Apache TLP. This would give us a chance to get out from under Google's blind spot, guarantee better compatibility across the ecosystem, etc, etc. It is sounding more and more like that's really what needs to happen. -- Sean -- Best regards, - Andy Problems worthy of attack prove their worth by hitting back. - Piet Hein (via Tom White)
Re: Protocol Buffers version
On Jun 12, 2015, at 1:03 PM, Alan Burlison alan.burli...@oracle.com wrote: On 14/05/2015 18:41, Chris Nauroth wrote: As a reminder though, the community probably would want to see a strong justification for the upgrade in terms of features or performance or something else. Right now, I'm not seeing a significant benefit for us based on my reading of their release notes. I think it's worthwhile to figure this out first. Otherwise, there is a risk that any testing work turns out to be a wasted effort. One reason at least: PB 2.5.0 has no support for Solaris SPARC. 2.6.1 does. That's a pretty good reason. Some of us had a discussion at Summit about effectively forking protobuf and making it an Apache TLP. This would give us a chance to get out from under Google's blind spot, guarantee better compatibility across the ecosystem, etc, etc. It is sounding more and more like that's really what needs to happen.
Re: Protocol Buffers version
Anyone have a read on how the protobuf folks would feel about that? Apache has a history of not accepting projects that are non-amicable forks. On Mon, Jun 15, 2015 at 9:24 AM, Allen Wittenauer a...@altiscale.com wrote: On Jun 12, 2015, at 1:03 PM, Alan Burlison alan.burli...@oracle.com wrote: On 14/05/2015 18:41, Chris Nauroth wrote: As a reminder though, the community probably would want to see a strong justification for the upgrade in terms of features or performance or something else. Right now, I'm not seeing a significant benefit for us based on my reading of their release notes. I think it's worthwhile to figure this out first. Otherwise, there is a risk that any testing work turns out to be a wasted effort. One reason at least: PB 2.5.0 has no support for Solaris SPARC. 2.6.1 does. That's a pretty good reason. Some of us had a discussion at Summit about effectively forking protobuf and making it an Apache TLP. This would give us a chance to get out from under Google's blind spot, guarantee better compatibility across the ecosystem, etc, etc. It is sounding more and more like that's really what needs to happen. -- Sean
Re: [DISCUSS] project for pre-commit patch testing (was Re: upstream jenkins build broken?)
+1 (Have been talking to Sean in private on the subject -- seems appropriate to voice some public support) I'd be interested in this for Accumulo and Slider. For Accumulo, we've come a far way without a pre-commit build, primarily due to a CTR process. We have seen the repeated questions of how do I run the tests which a more automated workflow would help with, IMO. I think Slider could benefit with the same reasons. I'd also be giddy to see the recent improvements in Hadoop trickle down into the other projects that Allen already mentioned. Take this as record that I'd be happy to try to help out where possible. Sean Busbey wrote: thank you for making a more digestible version Allen. :) If you're interested in soliciting feedback from other projects, I created ASF short links to this thread in common-dev and hbase: * http://s.apache.org/yetus-discuss-hadoop * http://s.apache.org/yetus-discuss-hbase While I agree that it's important to get feedback from ASF projects that might find this useful, I can say that recently I've been involved in the non-ASF project YCSB and both the pretest and better shell stuff would be immensely useful over there. On Mon, Jun 15, 2015 at 10:36 PM, Allen Wittenauera...@altiscale.com wrote: I'm clearly +1 on this idea. As part of the rewrite in Hadoop of test-patch, it was amazing to see how far and wide this bit of code as spread. So I see consolidating everyone's efforts as a huge win for a large number of projects. (esp considering how many I saw suffering from a variety of identified bugs! ) But…. I think it's important for people involved in those other projects to speak up and voice an opinion as to whether this is useful. To summarize: In the short term, a single location to get/use a precommit patch tester rather than everyone building/supporting their own in their spare time. FWIW, we've already got the code base modified to be pluggable. We've written some basic/simple plugins that support Hadoop, HBase, Tajo, Tez, Pig, and Flink. For HBase and Flink, this does include their custom checks. Adding support for other project shouldn't be hard. Simple projects take almost no time after seeing the basic pattern. I think it's worthwhile highlighting that means support for both JIRA and GitHub as well as Ant and Maven from the same code base. Longer term: Well, we clearly have ideas of things that we want to do. Adding more features to test-patch (review board? gradle?) is obvious. But what about teasing apart and generalizing some of the other shell bits from projects? A common library for building CLI tools to fault injection to release documentation creation tools to … I'd even like to see us get as advanced as a run this program to auto-generate daemon stop/start bits. I had a few chats with people about this idea at Hadoop Summit. What's truly exciting are the ideas that people had once they realized what kinds of problems we're trying to solve. It's always amazing the problems that projects have that could be solved by these types of solutions. Let's stop hiding our cool toys in this area. So, what feedback and ideas do you have in this area? Are you a yay or a nay? On Jun 15, 2015, at 4:47 PM, Sean Busbeybus...@cloudera.com wrote: Oof. I had meant to push on this again but life got in the way and now the June board meeting is upon us. Sorry everyone. In the event that this ends up contentious, hopefully one of the copied communities can give us a branch to work in. I know everyone is busy, so here's the short version of this email: I'd like to move some of the code currently in Hadoop (test-patch) into a new TLP focused on QA tooling. I'm not sure what the best format for priming this conversation is. ORC filled in the incubator project proposal template, but I'm not sure how much that confused the issue. So to start, I'll just write what I'm hoping we can accomplish in general terms here. All software development projects that are community based (that is, accepting outside contributions) face a common QA problem for vetting in-coming contributions. Hadoop is fortunate enough to be sufficiently popular that the weight of the problem drove tool development (i.e. test-patch). That tool is generalizable enough that a bunch of other TLPs have adopted their own forks. Unfortunately, in most projects this kind of QA work is an enabler rather than a primary concern, so often the tooling is worked on ad-hoc and little shared improvements happen across projects. Since the tooling itself is never a primary concern, any made is rarely reused outside of ASF projects. Over the last couple months a few of us have been working on generalizing the tooling present in the Hadoop code base (because it was the most mature out of all those in the various projects) and it's reached a point where we think we can start bringing on other downstream
Re: [DISCUSS] project for pre-commit patch testing (was Re: upstream jenkins build broken?)
+1 ZooKeeper is another project that has expressed interest in improving its pre-commit process lately. I understand Allen has had some success applying this to the ZooKeeper build too, with some small caveats around quirks in the build.xml that I think we can resolve. I'm interested in defining how the release model works for a project like this. The current model of forking and checking it in directly to multiple projects leads to the fragmentation and bugs described earlier in the thread. Another possible model is something more dynamic, like a bootstrap script capable of checking out a release from a git tag before launching pre-commit. I'm interested to hear from various projects on how they'd like to integrate. --Chris Nauroth On 6/15/15, 8:57 PM, Josh Elser els...@apache.org wrote: +1 (Have been talking to Sean in private on the subject -- seems appropriate to voice some public support) I'd be interested in this for Accumulo and Slider. For Accumulo, we've come a far way without a pre-commit build, primarily due to a CTR process. We have seen the repeated questions of how do I run the tests which a more automated workflow would help with, IMO. I think Slider could benefit with the same reasons. I'd also be giddy to see the recent improvements in Hadoop trickle down into the other projects that Allen already mentioned. Take this as record that I'd be happy to try to help out where possible. Sean Busbey wrote: thank you for making a more digestible version Allen. :) If you're interested in soliciting feedback from other projects, I created ASF short links to this thread in common-dev and hbase: * http://s.apache.org/yetus-discuss-hadoop * http://s.apache.org/yetus-discuss-hbase While I agree that it's important to get feedback from ASF projects that might find this useful, I can say that recently I've been involved in the non-ASF project YCSB and both the pretest and better shell stuff would be immensely useful over there. On Mon, Jun 15, 2015 at 10:36 PM, Allen Wittenauera...@altiscale.com wrote: I'm clearly +1 on this idea. As part of the rewrite in Hadoop of test-patch, it was amazing to see how far and wide this bit of code as spread. So I see consolidating everyone's efforts as a huge win for a large number of projects. (esp considering how many I saw suffering from a variety of identified bugs! ) But…. I think it's important for people involved in those other projects to speak up and voice an opinion as to whether this is useful. To summarize: In the short term, a single location to get/use a precommit patch tester rather than everyone building/supporting their own in their spare time. FWIW, we've already got the code base modified to be pluggable. We've written some basic/simple plugins that support Hadoop, HBase, Tajo, Tez, Pig, and Flink. For HBase and Flink, this does include their custom checks. Adding support for other project shouldn't be hard. Simple projects take almost no time after seeing the basic pattern. I think it's worthwhile highlighting that means support for both JIRA and GitHub as well as Ant and Maven from the same code base. Longer term: Well, we clearly have ideas of things that we want to do. Adding more features to test-patch (review board? gradle?) is obvious. But what about teasing apart and generalizing some of the other shell bits from projects? A common library for building CLI tools to fault injection to release documentation creation tools to … I'd even like to see us get as advanced as a run this program to auto-generate daemon stop/start bits. I had a few chats with people about this idea at Hadoop Summit. What's truly exciting are the ideas that people had once they realized what kinds of problems we're trying to solve. It's always amazing the problems that projects have that could be solved by these types of solutions. Let's stop hiding our cool toys in this area. So, what feedback and ideas do you have in this area? Are you a yay or a nay? On Jun 15, 2015, at 4:47 PM, Sean Busbeybus...@cloudera.com wrote: Oof. I had meant to push on this again but life got in the way and now the June board meeting is upon us. Sorry everyone. In the event that this ends up contentious, hopefully one of the copied communities can give us a branch to work in. I know everyone is busy, so here's the short version of this email: I'd like to move some of the code currently in Hadoop (test-patch) into a new TLP focused on QA tooling. I'm not sure what the best format for priming this conversation is. ORC filled in the incubator project proposal template, but I'm not sure how much that confused the issue. So to start, I'll just write what I'm hoping we can accomplish in general terms here. All software development projects that are community based (that is, accepting outside contributions) face a
Re: [DISCUSS] project for pre-commit patch testing (was Re: upstream jenkins build broken?)
I'm clearly +1 on this idea. As part of the rewrite in Hadoop of test-patch, it was amazing to see how far and wide this bit of code as spread. So I see consolidating everyone's efforts as a huge win for a large number of projects. (esp considering how many I saw suffering from a variety of identified bugs! ) But…. I think it's important for people involved in those other projects to speak up and voice an opinion as to whether this is useful. To summarize: In the short term, a single location to get/use a precommit patch tester rather than everyone building/supporting their own in their spare time. FWIW, we've already got the code base modified to be pluggable. We've written some basic/simple plugins that support Hadoop, HBase, Tajo, Tez, Pig, and Flink. For HBase and Flink, this does include their custom checks. Adding support for other project shouldn't be hard. Simple projects take almost no time after seeing the basic pattern. I think it's worthwhile highlighting that means support for both JIRA and GitHub as well as Ant and Maven from the same code base. Longer term: Well, we clearly have ideas of things that we want to do. Adding more features to test-patch (review board? gradle?) is obvious. But what about teasing apart and generalizing some of the other shell bits from projects? A common library for building CLI tools to fault injection to release documentation creation tools to … I'd even like to see us get as advanced as a run this program to auto-generate daemon stop/start bits. I had a few chats with people about this idea at Hadoop Summit. What's truly exciting are the ideas that people had once they realized what kinds of problems we're trying to solve. It's always amazing the problems that projects have that could be solved by these types of solutions. Let's stop hiding our cool toys in this area. So, what feedback and ideas do you have in this area? Are you a yay or a nay? On Jun 15, 2015, at 4:47 PM, Sean Busbey bus...@cloudera.com wrote: Oof. I had meant to push on this again but life got in the way and now the June board meeting is upon us. Sorry everyone. In the event that this ends up contentious, hopefully one of the copied communities can give us a branch to work in. I know everyone is busy, so here's the short version of this email: I'd like to move some of the code currently in Hadoop (test-patch) into a new TLP focused on QA tooling. I'm not sure what the best format for priming this conversation is. ORC filled in the incubator project proposal template, but I'm not sure how much that confused the issue. So to start, I'll just write what I'm hoping we can accomplish in general terms here. All software development projects that are community based (that is, accepting outside contributions) face a common QA problem for vetting in-coming contributions. Hadoop is fortunate enough to be sufficiently popular that the weight of the problem drove tool development (i.e. test-patch). That tool is generalizable enough that a bunch of other TLPs have adopted their own forks. Unfortunately, in most projects this kind of QA work is an enabler rather than a primary concern, so often the tooling is worked on ad-hoc and little shared improvements happen across projects. Since the tooling itself is never a primary concern, any made is rarely reused outside of ASF projects. Over the last couple months a few of us have been working on generalizing the tooling present in the Hadoop code base (because it was the most mature out of all those in the various projects) and it's reached a point where we think we can start bringing on other downstream users. This means we need to start establishing things like a release cadence and to grow the new contributors we have to handle more project responsibility. Personally, I think that means it's time to move out from under Hadoop to drive things as our own community. Eventually, I hope the community can help draw in a group of folks traditionally underrepresented in ASF projects, namely QA and operations folks. I think test-patch by itself has enough scope to justify a project. Having a solid set of build tools that are customizable to fit the norms of different software communities is a bunch of work. Making it work well in both the context of automated test systems like Jenkins and for individual developers is even more work. We could easily also take over maintenance of things like shelldocs, since test-patch is the primary consumer of that currently but it's generally useful tooling. In addition to test-patch, I think the proposed project has some future growth potential. Given some adoption of test-patch to prove utility, the project could build on the ties it makes to start building tools to help projects do their own longer-run testing. Note that I'm talking about the tools to build
Re: [DISCUSS] project for pre-commit patch testing (was Re: upstream jenkins build broken?)
thank you for making a more digestible version Allen. :) If you're interested in soliciting feedback from other projects, I created ASF short links to this thread in common-dev and hbase: * http://s.apache.org/yetus-discuss-hadoop * http://s.apache.org/yetus-discuss-hbase While I agree that it's important to get feedback from ASF projects that might find this useful, I can say that recently I've been involved in the non-ASF project YCSB and both the pretest and better shell stuff would be immensely useful over there. On Mon, Jun 15, 2015 at 10:36 PM, Allen Wittenauer a...@altiscale.com wrote: I'm clearly +1 on this idea. As part of the rewrite in Hadoop of test-patch, it was amazing to see how far and wide this bit of code as spread. So I see consolidating everyone's efforts as a huge win for a large number of projects. (esp considering how many I saw suffering from a variety of identified bugs! ) But…. I think it's important for people involved in those other projects to speak up and voice an opinion as to whether this is useful. To summarize: In the short term, a single location to get/use a precommit patch tester rather than everyone building/supporting their own in their spare time. FWIW, we've already got the code base modified to be pluggable. We've written some basic/simple plugins that support Hadoop, HBase, Tajo, Tez, Pig, and Flink. For HBase and Flink, this does include their custom checks. Adding support for other project shouldn't be hard. Simple projects take almost no time after seeing the basic pattern. I think it's worthwhile highlighting that means support for both JIRA and GitHub as well as Ant and Maven from the same code base. Longer term: Well, we clearly have ideas of things that we want to do. Adding more features to test-patch (review board? gradle?) is obvious. But what about teasing apart and generalizing some of the other shell bits from projects? A common library for building CLI tools to fault injection to release documentation creation tools to … I'd even like to see us get as advanced as a run this program to auto-generate daemon stop/start bits. I had a few chats with people about this idea at Hadoop Summit. What's truly exciting are the ideas that people had once they realized what kinds of problems we're trying to solve. It's always amazing the problems that projects have that could be solved by these types of solutions. Let's stop hiding our cool toys in this area. So, what feedback and ideas do you have in this area? Are you a yay or a nay? On Jun 15, 2015, at 4:47 PM, Sean Busbey bus...@cloudera.com wrote: Oof. I had meant to push on this again but life got in the way and now the June board meeting is upon us. Sorry everyone. In the event that this ends up contentious, hopefully one of the copied communities can give us a branch to work in. I know everyone is busy, so here's the short version of this email: I'd like to move some of the code currently in Hadoop (test-patch) into a new TLP focused on QA tooling. I'm not sure what the best format for priming this conversation is. ORC filled in the incubator project proposal template, but I'm not sure how much that confused the issue. So to start, I'll just write what I'm hoping we can accomplish in general terms here. All software development projects that are community based (that is, accepting outside contributions) face a common QA problem for vetting in-coming contributions. Hadoop is fortunate enough to be sufficiently popular that the weight of the problem drove tool development (i.e. test-patch). That tool is generalizable enough that a bunch of other TLPs have adopted their own forks. Unfortunately, in most projects this kind of QA work is an enabler rather than a primary concern, so often the tooling is worked on ad-hoc and little shared improvements happen across projects. Since the tooling itself is never a primary concern, any made is rarely reused outside of ASF projects. Over the last couple months a few of us have been working on generalizing the tooling present in the Hadoop code base (because it was the most mature out of all those in the various projects) and it's reached a point where we think we can start bringing on other downstream users. This means we need to start establishing things like a release cadence and to grow the new contributors we have to handle more project responsibility. Personally, I think that means it's time to move out from under Hadoop to drive things as our own community. Eventually, I hope the community can help draw in a group of folks traditionally underrepresented in ASF projects, namely QA and operations folks. I think test-patch by itself has enough scope to justify a project. Having a solid set of build tools that are customizable to fit the norms of
Re: set up jenkins test for branch-2
Thanks Sean and Allen! I was not aware of that there is already a way to trigger branch-2 test. Good to know. There are multiple solutions here: 1. When posting patches, we can post two versions of patches, one for trunk, and one for branch-2, so to trigger two pre-commit jenkins test for both branches. This would help catching issue before committing. However, it's going to be more workload on the testing machines, so alternatively, we can probably defer the branch-2 testing until committing to trunk and before branch-2 testing. 2. Only post patch for trunk, we cherry-pick to branch-2 when committing. We can set up periodic jenkins test (such as nightly) for branch-2 to catch problem periodically. But that's basically being ignored by us as Allen pointed out. The goal in my mind is to try our best to have both commits to trunk and branch-2 to pass jenkins test, thus help on quality. Thanks. --Yongjun On Mon, Jun 15, 2015 at 7:21 AM, Allen Wittenauer a...@altiscale.com wrote: On Jun 14, 2015, at 1:00 PM, Yongjun Zhang yzh...@cloudera.com wrote: From time to time we saw changes that work fine in trunk but not branch-2, and we don't catch the issue in a timely manner. The difference between trunk and branch-2 is sufficient to justify periodic jenkins test and even pre-commit test for branch-2. What is your expected outcome? The current nightly builds have been failing pretty consistently for a while now. Adding another one isn't going to make the situation any better since everyone pretty much ignores them. As I said at the HDFS BOF: we desperately need a moratorium on adding new features and need to work on stabilizing both branch-2 and trunk. We seem to be in a quantity over quality mode and it's slowly killing the project. We haven't had a stable release since 2.4.1, which is approaching the 1 year mark. (Yes, I know. Various vendors have shipped releases since 2.4.1 as part of their distributions. Before saying anything, perhaps you should spend some time with your support folks….) As Sean pointed out, precommit for branch-2 has been working for a while now if one provides enough hints to test-patch as part of the patch name.
[jira] [Created] (HADOOP-12090) minikdc-related unit tests fail consistently on some platforms
Sangjin Lee created HADOOP-12090:
Summary: minikdc-related unit tests fail consistently on some
platforms
Key: HADOOP-12090
URL: https://issues.apache.org/jira/browse/HADOOP-12090
Project: Hadoop Common
Issue Type: Bug
Components: kms
Affects Versions: 2.7.0
Reporter: Sangjin Lee
Assignee: Sangjin Lee
On some platforms all unit tests that use minikdc fail consistently. Those
tests include TestKMS, TestSaslDataTransfer, TestTimelineAuthenticationFilter,
etc.
Typical failures on the unit tests:
{noformat}
java.lang.AssertionError:
org.apache.hadoop.security.authentication.client.AuthenticationException:
GSSException: No valid credentials provided (Mechanism level: Cannot get a KDC
reply)
at org.junit.Assert.fail(Assert.java:88)
at
org.apache.hadoop.crypto.key.kms.server.TestKMS$8$4.run(TestKMS.java:1154)
at
org.apache.hadoop.crypto.key.kms.server.TestKMS$8$4.run(TestKMS.java:1145)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:415)
at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1645)
at
org.apache.hadoop.crypto.key.kms.server.TestKMS.doAs(TestKMS.java:261)
at
org.apache.hadoop.crypto.key.kms.server.TestKMS.access$100(TestKMS.java:76)
{noformat}
The errors that cause this failure on the KDC server on the minikdc are a
NullPointerException:
{noformat}
org.apache.mina.filter.codec.ProtocolDecoderException:
java.lang.NullPointerException: message (Hexdump: ...)
at
org.apache.mina.filter.codec.ProtocolCodecFilter.messageReceived(ProtocolCodecFilter.java:234)
at
org.apache.mina.core.filterchain.DefaultIoFilterChain.callNextMessageReceived(DefaultIoFilterChain.java:434)
at
org.apache.mina.core.filterchain.DefaultIoFilterChain.access$1200(DefaultIoFilterChain.java:48)
at
org.apache.mina.core.filterchain.DefaultIoFilterChain$EntryImpl$1.messageReceived(DefaultIoFilterChain.java:802)
at
org.apache.mina.core.filterchain.IoFilterAdapter.messageReceived(IoFilterAdapter.java:120)
at
org.apache.mina.core.filterchain.DefaultIoFilterChain.callNextMessageReceived(DefaultIoFilterChain.java:434)
at
org.apache.mina.core.filterchain.DefaultIoFilterChain.fireMessageReceived(DefaultIoFilterChain.java:426)
at
org.apache.mina.core.polling.AbstractPollingIoProcessor.read(AbstractPollingIoProcessor.java:604)
at
org.apache.mina.core.polling.AbstractPollingIoProcessor.process(AbstractPollingIoProcessor.java:564)
at
org.apache.mina.core.polling.AbstractPollingIoProcessor.process(AbstractPollingIoProcessor.java:553)
at
org.apache.mina.core.polling.AbstractPollingIoProcessor.access$400(AbstractPollingIoProcessor.java:57)
at
org.apache.mina.core.polling.AbstractPollingIoProcessor$Processor.run(AbstractPollingIoProcessor.java:892)
at
org.apache.mina.util.NamePreservingRunnable.run(NamePreservingRunnable.java:65)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)
Caused by: java.lang.NullPointerException: message
at
org.apache.mina.filter.codec.AbstractProtocolDecoderOutput.write(AbstractProtocolDecoderOutput.java:44)
at
org.apache.directory.server.kerberos.protocol.codec.MinaKerberosDecoder.decode(MinaKerberosDecoder.java:65)
at
org.apache.mina.filter.codec.ProtocolCodecFilter.messageReceived(ProtocolCodecFilter.java:224)
... 15 more
{noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
Re: set up jenkins test for branch-2
On Mon, Jun 15, 2015 at 1:39 PM, Yongjun Zhang yzh...@cloudera.com wrote: Thanks Sean and Allen! I was not aware of that there is already a way to trigger branch-2 test. Good to know. There are multiple solutions here: 1. When posting patches, we can post two versions of patches, one for trunk, and one for branch-2, so to trigger two pre-commit jenkins test for both branches. This would help catching issue before committing. However, it's going to be more workload on the testing machines, so alternatively, we can probably defer the branch-2 testing until committing to trunk and before branch-2 testing. 2. Only post patch for trunk, we cherry-pick to branch-2 when committing. We can set up periodic jenkins test (such as nightly) for branch-2 to catch problem periodically. But that's basically being ignored by us as Allen pointed out. I wouldn't worry about the load on the test machines. If you do, a third option is for committers to use test-patch on their local machine after making the branch-2 version of the patch. -- Sean
[jira] [Created] (HADOOP-12089) StorageException complaining no lease ID when updating FolderLastModifiedTime in WASB
Duo Xu created HADOOP-12089:
---
Summary: StorageException complaining no lease ID when updating
FolderLastModifiedTime in WASB
Key: HADOOP-12089
URL: https://issues.apache.org/jira/browse/HADOOP-12089
Project: Hadoop Common
Issue Type: Bug
Components: tools
Affects Versions: 2.7.0
Reporter: Duo Xu
Assignee: Duo Xu
This is a similar issue to HADOOP-11523. HADOOP-11523 happens when HBase is
doing distributed log splitting. This JIRA happens when HBase is deleting old
WALs and trying to update /hbase/oldWALs folder.
The fix is the same as HADOOP-11523.
{code}
2015-06-10 08:11:40,636 WARN
org.apache.hadoop.hbase.master.cleaner.CleanerChore: Error while deleting:
wasb://dthbasecus...@dthbasestoragecus1.blob.core.windows.net/hbase/oldWALs/workernode10.dthbasecus1.g1.internal.cloudapp.net%2C60020%2C1433908062461.1433921692855
org.apache.hadoop.fs.azure.AzureException:
com.microsoft.azure.storage.StorageException: There is currently a lease on the
blob and no lease ID was specified in the request.
at
org.apache.hadoop.fs.azure.AzureNativeFileSystemStore.updateFolderLastModifiedTime(AzureNativeFileSystemStore.java:2602)
at
org.apache.hadoop.fs.azure.AzureNativeFileSystemStore.updateFolderLastModifiedTime(AzureNativeFileSystemStore.java:2613)
at
org.apache.hadoop.fs.azure.NativeAzureFileSystem.delete(NativeAzureFileSystem.java:1505)
at
org.apache.hadoop.fs.azure.NativeAzureFileSystem.delete(NativeAzureFileSystem.java:1437)
at
org.apache.hadoop.hbase.master.cleaner.CleanerChore.checkAndDeleteFiles(CleanerChore.java:256)
at
org.apache.hadoop.hbase.master.cleaner.CleanerChore.checkAndDeleteEntries(CleanerChore.java:157)
at
org.apache.hadoop.hbase.master.cleaner.CleanerChore.chore(CleanerChore.java:124)
at org.apache.hadoop.hbase.Chore.run(Chore.java:80)
at java.lang.Thread.run(Thread.java:745)
Caused by: com.microsoft.azure.storage.StorageException: There is currently a
lease on the blob and no lease ID was specified in the request.
at
com.microsoft.azure.storage.StorageException.translateException(StorageException.java:162)
at
com.microsoft.azure.storage.core.StorageRequest.materializeException(StorageRequest.java:307)
at
com.microsoft.azure.storage.core.ExecutionEngine.executeWithRetry(ExecutionEngine.java:177)
at
com.microsoft.azure.storage.blob.CloudBlob.uploadProperties(CloudBlob.java:2991)
at
org.apache.hadoop.fs.azure.StorageInterfaceImpl$CloudBlobWrapperImpl.uploadProperties(StorageInterfaceImpl.java:372)
at
org.apache.hadoop.fs.azure.AzureNativeFileSystemStore.updateFolderLastModifiedTime(AzureNativeFileSystemStore.java:2597)
... 8 more
{code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
Re: Protocol Buffers version
On Mon, Jun 15, 2015 at 8:57 AM, Andrew Purtell apurt...@apache.org wrote: I can't answer the original question but can point out the protostuff ( https://github.com/protostuff/protostuff) folks have been responsive and friendly in the past when we (HBase) were curious about swapping in their stuff. Two significant benefits of protostuff, IMHO, is ASL 2 licensing and everything is implemented in Java including the compiler. Big +1 to protostuff from community, licensing and implementation perspectives. Thanks, Roman.
Re: Protocol Buffers version
On Mon, Jun 15, 2015 at 7:24 AM, Allen Wittenauer a...@altiscale.com wrote: On Jun 12, 2015, at 1:03 PM, Alan Burlison alan.burli...@oracle.com wrote: On 14/05/2015 18:41, Chris Nauroth wrote: As a reminder though, the community probably would want to see a strong justification for the upgrade in terms of features or performance or something else. Right now, I'm not seeing a significant benefit for us based on my reading of their release notes. I think it's worthwhile to figure this out first. Otherwise, there is a risk that any testing work turns out to be a wasted effort. One reason at least: PB 2.5.0 has no support for Solaris SPARC. 2.6.1 does. That's a pretty good reason. Some of us had a discussion at Summit about effectively forking protobuf and making it an Apache TLP. This would give us a chance to get out from under Google's blind spot, guarantee better compatibility across the ecosystem, etc, etc. It is sounding more and more like that's really what needs to happen. I agree that it would be nice if the protobuf project avoided making backwards-incompatible API changes within a minor release. But in practice, we have had the same issues with Jackson, Guava, jets3t, and other dependencies. Nearly every important Hadoop dependency has made backwards-incompatible API changes within a minor release of the dependency... and that's one reason we are using such old versions of everything. I don't think PB deserves to be singled out as much as it has been. I think the work going on now to implement CLASSPATH isolation in Hadoop will really be beneficial here because we will be able to upgrade without worrying about these problems. cheers, Colin
Re: What is the limit to the number of properties set in the configuration object
Much like zombo.com, the only limit is yourself. But huge Configuration objects are going to be really inefficient, so I would look elsewhere for storing lots of data. best, Colin On Fri, Jun 12, 2015 at 7:30 PM, Sitaraman Vilayannur vrsitaramanietfli...@gmail.com wrote: Thanks Allen, what is the total size limit? Sitaraman On Fri, Jun 12, 2015 at 10:53 PM, Allen Wittenauer a...@altiscale.com wrote: On Jun 12, 2015, at 12:37 AM, Sitaraman Vilayannur vrsitaramanietfli...@gmail.com wrote: Hi, What is the limit on the number of properties that can be set using set(String s1, String s2) on the Configuration object for hadoop? Is this limit configurable if so what is the maximum that can be set? It's a total size of the conf limit, not a number of limit. In general, you shouldn't pack it full of stuff as calling Configuration is expensive. Use a side-input/distributed cache file for mass quantities of bits.
Re: 2.7.1 status
We are down to one blocker and a few critical tickets. I’ll try to push out an RC in a day or two. Thanks +Vinod On Jun 1, 2015, at 10:45 AM, Vinod Kumar Vavilapalli vino...@hortonworks.com wrote: Tx for the move on that JIRA, folks. https://issues.apache.org/jira/issues/?filter=12331550 still shows 4 blockers / 4 criticals. I'm going to start pushing them in/out. Thanks +Vinod On May 27, 2015, at 3:20 PM, Chris Nauroth cnaur...@hortonworks.com wrote: Thanks, Larry. I have marked HADOOP-11934 as a blocker for 2.7.1. I have reviewed and +1'd it. I can commit it after we get feedback from Jenkins. --Chris Nauroth On 5/26/15, 12:41 PM, larry mccay lmc...@apache.org wrote: Hi Vinod - I think that https://issues.apache.org/jira/browse/HADOOP-11934 should also be added to the blocker list. This is a critical bug in our ability to protect the LDAP connection password in LdapGroupsMapper. thanks! --larry On Tue, May 26, 2015 at 3:32 PM, Vinod Kumar Vavilapalli vino...@hortonworks.com wrote: Tx for reporting this, Elliot. Made it a blocker, not with a deeper understanding of the problem. Can you please chime in with your opinion and perhaps code reviews? Thanks +Vinod On May 26, 2015, at 10:48 AM, Elliott Clark ecl...@apache.org wrote: HADOOP-12001 should probably be added to the blocker list since it's a regression that can keep ldap from working.
Re: CMake and CMAKE_LD_FLAGS
Hi Alan, I think you are right that CMAKE_LD_FLAGS has never done anything, and setting it was always a mistake. The uses of CMAKE_LD_FLAGS in JNIFlags.cmake are harmless, since the -m32 option is not needed for the linker. However, it's more concerning that hadoop-mapreduce-client-nativetask seems to be using this macro to set -no-undefined -version-info 0:1:0 and some other stuff. Can you create a JIRA to change these uses over to the correct CMake directive? If you create a JIRA that makes only that change, it should be pretty quick to review and commit. best, Colin On Thu, Jun 4, 2015 at 7:41 AM, Alan Burlison alan.burli...@oracle.com wrote: On 03/06/2015 21:23, Alan Burlison wrote: hadoop-common-project/hadoop-common/src/JNIFlags.cmake sets the CMAKE_LD_FLAGS variable to the flags required for the linker. However, despite the name, CMAKE_LD_FLAGS is *not* a valid standard CMake variable Confirmed, I've put nonsense flags in CMAKE_LD_FLAGS that should produce an invalid linker command-line and they don't appear in the generated makefiles and the build works fine. If I put the same nonsense flags in the proper variables then the build blows up as I'd expect. -- Alan Burlison --
Re: Maven always detects changes - Is this a Docker 'feature'?
Hi Darrell, Sorry, I'm not familiar with this feature of Maven. Perhaps try asking on the Apache Maven mailing list? best, Colin On Fri, May 22, 2015 at 8:34 AM, Darrell Taylor darrell.tay...@gmail.com wrote: Hi, Is it normal behaviour for maven to detect changes when I run tests with no changes? e.g. $ mvn test -Dtest=TestDFSShell -nsu -o ... [INFO] --- maven-compiler-plugin:3.1:compile (default-compile) @ hadoop-hdfs --- [INFO] Changes detected - recompiling the module! [INFO] Compiling 576 source files to /home/darrell/hadoop/hadoop-hdfs-project/hadoop-hdfs/target/classes ... Then I run the same command again without touching anything else and it compiles everything again. It's getting rather tedious. I am running this from inside the docker container. Any help appreciated. Thanks Darrell.
Re: 2.7.1 status
Hi Vinod, Thank you for working to release 2.7! YARN-3798 looks a critical issue for branch-2.7. I'll mark it as a blocker of 2.7.1 if possible. Thanks, - Tsuyoshi On Tue, Jun 16, 2015 at 6:58 AM, Vinod Kumar Vavilapalli vino...@hortonworks.com wrote: We are down to one blocker and a few critical tickets. I’ll try to push out an RC in a day or two. Thanks +Vinod On Jun 1, 2015, at 10:45 AM, Vinod Kumar Vavilapalli vino...@hortonworks.com wrote: Tx for the move on that JIRA, folks. https://issues.apache.org/jira/issues/?filter=12331550 still shows 4 blockers / 4 criticals. I'm going to start pushing them in/out. Thanks +Vinod On May 27, 2015, at 3:20 PM, Chris Nauroth cnaur...@hortonworks.com wrote: Thanks, Larry. I have marked HADOOP-11934 as a blocker for 2.7.1. I have reviewed and +1'd it. I can commit it after we get feedback from Jenkins. --Chris Nauroth On 5/26/15, 12:41 PM, larry mccay lmc...@apache.org wrote: Hi Vinod - I think that https://issues.apache.org/jira/browse/HADOOP-11934 should also be added to the blocker list. This is a critical bug in our ability to protect the LDAP connection password in LdapGroupsMapper. thanks! --larry On Tue, May 26, 2015 at 3:32 PM, Vinod Kumar Vavilapalli vino...@hortonworks.com wrote: Tx for reporting this, Elliot. Made it a blocker, not with a deeper understanding of the problem. Can you please chime in with your opinion and perhaps code reviews? Thanks +Vinod On May 26, 2015, at 10:48 AM, Elliott Clark ecl...@apache.org wrote: HADOOP-12001 should probably be added to the blocker list since it's a regression that can keep ldap from working.
Re: [DISCUSS] project for pre-commit patch testing (was Re: upstream jenkins build broken?)
Oof. I had meant to push on this again but life got in the way and now the June board meeting is upon us. Sorry everyone. In the event that this ends up contentious, hopefully one of the copied communities can give us a branch to work in. I know everyone is busy, so here's the short version of this email: I'd like to move some of the code currently in Hadoop (test-patch) into a new TLP focused on QA tooling. I'm not sure what the best format for priming this conversation is. ORC filled in the incubator project proposal template, but I'm not sure how much that confused the issue. So to start, I'll just write what I'm hoping we can accomplish in general terms here. All software development projects that are community based (that is, accepting outside contributions) face a common QA problem for vetting in-coming contributions. Hadoop is fortunate enough to be sufficiently popular that the weight of the problem drove tool development (i.e. test-patch). That tool is generalizable enough that a bunch of other TLPs have adopted their own forks. Unfortunately, in most projects this kind of QA work is an enabler rather than a primary concern, so often the tooling is worked on ad-hoc and little shared improvements happen across projects. Since the tooling itself is never a primary concern, any made is rarely reused outside of ASF projects. Over the last couple months a few of us have been working on generalizing the tooling present in the Hadoop code base (because it was the most mature out of all those in the various projects) and it's reached a point where we think we can start bringing on other downstream users. This means we need to start establishing things like a release cadence and to grow the new contributors we have to handle more project responsibility. Personally, I think that means it's time to move out from under Hadoop to drive things as our own community. Eventually, I hope the community can help draw in a group of folks traditionally underrepresented in ASF projects, namely QA and operations folks. I think test-patch by itself has enough scope to justify a project. Having a solid set of build tools that are customizable to fit the norms of different software communities is a bunch of work. Making it work well in both the context of automated test systems like Jenkins and for individual developers is even more work. We could easily also take over maintenance of things like shelldocs, since test-patch is the primary consumer of that currently but it's generally useful tooling. In addition to test-patch, I think the proposed project has some future growth potential. Given some adoption of test-patch to prove utility, the project could build on the ties it makes to start building tools to help projects do their own longer-run testing. Note that I'm talking about the tools to build QA processes and not a particular set of tested components. Specifically, I think the ChaosMonkey work that's in HBase should be generalizable as a fault injection framework (either based on that code or something like it). Doing this for arbitrary software is obviously very difficult, and a part of easing that will be to make (and then favor) tooling to allow projects to have operational glue that looks the same. Namely, the shell work that's been done in hadoop-functions.sh would be a great foundational layer that could bring good daemon handling practices to a whole slew of software projects. In the event that these frameworks and tools get adopted by parts of the Hadoop ecosystem, that could make the job of i.e. Bigtop substantially easier. I've reached out to a few folks who have been involved in the current test-patch work or expressed interest in helping out on getting it used in other projects. Right now, the proposed PMC would be (alphabetical by last name): * Andrew Bayer (ASF member, incubator pmc, bigtop pmc, flume pmc, jclouds pmc, sqoop pmc, all around Jenkins expert) * Sean Busbey (ASF member, accumulo pmc, hbase pmc) * Nick Dimiduk (hbase pmc, phoenix pmc) * Chris Nauroth (ASF member, incubator pmc, hadoop pmc) * Andrew Purtell (ASF member, incubator pmc, bigtop pmc, hbase pmc, phoenix pmc) * Allen Wittenauer (hadoop committer) That PMC gives us several members and a bunch of folks familiar with the ASF. Combined with the code already existing in Apache spaces, I think that gives us sufficient justification for a direct board proposal. The planned project name is Apache Yetus. It's an archaic genus of sea snail and most of our project will be focused on shell scripts. N.b.: this does not mean that the Hadoop community would _have_ to rely on the new TLP, but I hope that once we have a release that can be evaluated there'd be enough benefit to strongly encourage it. This has mostly been focused on scope and community issues, and I'd love to talk through any feedback on that. Additionally, are there any other points folks want to make sure are covered before we have a resolution? On Sat, Jun 6, 2015
