Hi Folks! Work in a feature branch is now being tracked by HADOOP-12111.
On Thu, Jun 18, 2015 at 10:07 PM, Sean Busbey <bus...@cloudera.com> wrote: > It looks like we have consensus. > > I'll start drafting up a proposal for the next board meeting (July 15th). > Once we work out the name I'll submit a PODLINGNAMESEARCH jira to track > that we did due diligence on whatever we pick. > > In the mean time, Hadoop PMC would y'all be willing to host us in a branch > so that we can start prepping things now? We would want branch commit > rights for the proposed new PMC. > > > -Sean > > > On Mon, Jun 15, 2015 at 6:47 PM, Sean Busbey <bus...@cloudera.com> wrote: > >> Oof. I had meant to push on this again but life got in the way and now >> the June board meeting is upon us. Sorry everyone. In the event that this >> ends up contentious, hopefully one of the copied communities can give us a >> branch to work in. >> >> I know everyone is busy, so here's the short version of this email: I'd >> like to move some of the code currently in Hadoop (test-patch) into a new >> TLP focused on QA tooling. I'm not sure what the best format for priming >> this conversation is. ORC filled in the incubator project proposal >> template, but I'm not sure how much that confused the issue. So to start, >> I'll just write what I'm hoping we can accomplish in general terms here. >> >> All software development projects that are community based (that is, >> accepting outside contributions) face a common QA problem for vetting >> in-coming contributions. Hadoop is fortunate enough to be sufficiently >> popular that the weight of the problem drove tool development (i.e. >> test-patch). That tool is generalizable enough that a bunch of other TLPs >> have adopted their own forks. Unfortunately, in most projects this kind of >> QA work is an enabler rather than a primary concern, so often the tooling >> is worked on ad-hoc and little shared improvements happen across projects. >> Since >> the tooling itself is never a primary concern, any made is rarely reused >> outside of ASF projects. >> >> Over the last couple months a few of us have been working on generalizing >> the tooling present in the Hadoop code base (because it was the most mature >> out of all those in the various projects) and it's reached a point where we >> think we can start bringing on other downstream users. This means we need >> to start establishing things like a release cadence and to grow the new >> contributors we have to handle more project responsibility. Personally, I >> think that means it's time to move out from under Hadoop to drive things as >> our own community. Eventually, I hope the community can help draw in a >> group of folks traditionally underrepresented in ASF projects, namely QA >> and operations folks. >> >> I think test-patch by itself has enough scope to justify a project. >> Having a solid set of build tools that are customizable to fit the norms of >> different software communities is a bunch of work. Making it work well in >> both the context of automated test systems like Jenkins and for individual >> developers is even more work. We could easily also take over maintenance of >> things like shelldocs, since test-patch is the primary consumer of that >> currently but it's generally useful tooling. >> >> In addition to test-patch, I think the proposed project has some future >> growth potential. Given some adoption of test-patch to prove utility, the >> project could build on the ties it makes to start building tools to help >> projects do their own longer-run testing. Note that I'm talking about the >> tools to build QA processes and not a particular set of tested components. >> Specifically, I think the ChaosMonkey work that's in HBase should be >> generalizable as a fault injection framework (either based on that code or >> something like it). Doing this for arbitrary software is obviously very >> difficult, and a part of easing that will be to make (and then favor) >> tooling to allow projects to have operational glue that looks the same. >> Namely, the shell work that's been done in hadoop-functions.sh would be a >> great foundational layer that could bring good daemon handling practices to >> a whole slew of software projects. In the event that these frameworks and >> tools get adopted by parts of the Hadoop ecosystem, that could make the job >> of i.e. Bigtop substantially easier. >> >> I've reached out to a few folks who have been involved in the current >> test-patch work or expressed interest in helping out on getting it used in >> other projects. Right now, the proposed PMC would be (alphabetical by last >> name): >> >> * Andrew Bayer (ASF member, incubator pmc, bigtop pmc, flume pmc, jclouds >> pmc, sqoop pmc, all around Jenkins expert) >> * Sean Busbey (ASF member, accumulo pmc, hbase pmc) >> * Nick Dimiduk (hbase pmc, phoenix pmc) >> * Chris Nauroth (ASF member, incubator pmc, hadoop pmc) >> * Andrew Purtell (ASF member, incubator pmc, bigtop pmc, hbase pmc, >> phoenix pmc) >> * Allen Wittenauer (hadoop committer) >> >> That PMC gives us several members and a bunch of folks familiar with the >> ASF. Combined with the code already existing in Apache spaces, I think that >> gives us sufficient justification for a direct board proposal. >> >> The planned project name is "Apache Yetus". It's an archaic genus of sea >> snail and most of our project will be focused on shell scripts. >> >> N.b.: this does not mean that the Hadoop community would _have_ to rely >> on the new TLP, but I hope that once we have a release that can be >> evaluated there'd be enough benefit to strongly encourage it. >> >> This has mostly been focused on scope and community issues, and I'd love >> to talk through any feedback on that. Additionally, are there any other >> points folks want to make sure are covered before we have a resolution? >> >> On Sat, Jun 6, 2015 at 10:43 PM, Sean Busbey <bus...@cloudera.com> wrote: >> >>> Sorry for the resend. I figured this deserves a [DISCUSS] flag. >>> >>> >>> >>> On Sat, Jun 6, 2015 at 10:39 PM, Sean Busbey <bus...@cloudera.com> >>> wrote: >>> >>>> Hi Folks! >>>> >>>> After working on test-patch with other folks for the last few months, I >>>> think we've reached the point where we can make the fastest progress >>>> towards the goal of a general use pre-commit patch tester by spinning >>>> things into a project focused on just that. I think we have a mature enough >>>> code base and a sufficient fledgling community, so I'm going to put >>>> together a tlp proposal. >>>> >>>> Thanks for the feedback thus far from use within Hadoop. I hope we can >>>> continue to make things more useful. >>>> >>>> -Sean >>>> >>>> On Wed, Mar 11, 2015 at 5:16 PM, Sean Busbey <bus...@cloudera.com> >>>> wrote: >>>> >>>>> HBase's dev-support folder is where the scripts and support files >>>>> live. We've only recently started adding anything to the maven builds >>>>> that's specific to jenkins[1]; so far it's diagnostic stuff, but that's >>>>> where I'd add in more if we ran into the same permissions problems y'all >>>>> are having. >>>>> >>>>> There's also our precommit job itself, though it isn't large[2]. >>>>> AFAIK, we don't properly back this up anywhere, we just notify each other >>>>> of changes on a particular mail thread[3]. >>>>> >>>>> [1]: https://github.com/apache/hbase/blob/master/pom.xml#L1687 >>>>> [2]: https://builds.apache.org/job/PreCommit-HBASE-Build/ (they're >>>>> all read because I just finished fixing "mvn site" running out of permgen) >>>>> [3]: http://s.apache.org/NT0 >>>>> >>>>> >>>>> On Wed, Mar 11, 2015 at 4:51 PM, Chris Nauroth < >>>>> cnaur...@hortonworks.com> wrote: >>>>> >>>>>> Sure, thanks Sean! Do we just look in the dev-support folder in the >>>>>> HBase >>>>>> repo? Is there any additional context we need to be aware of? >>>>>> >>>>>> Chris Nauroth >>>>>> Hortonworks >>>>>> http://hortonworks.com/ >>>>>> >>>>>> >>>>>> >>>>>> >>>>>> >>>>>> >>>>>> On 3/11/15, 2:44 PM, "Sean Busbey" <bus...@cloudera.com> wrote: >>>>>> >>>>>> >+dev@hbase >>>>>> > >>>>>> >HBase has recently been cleaning up our precommit jenkins jobs to >>>>>> make >>>>>> >them >>>>>> >more robust. From what I can tell our stuff started off as an earlier >>>>>> >version of what Hadoop uses for testing. >>>>>> > >>>>>> >Folks on either side open to an experiment of combining our precommit >>>>>> >check >>>>>> >tooling? In principle we should be looking for the same kinds of >>>>>> things. >>>>>> > >>>>>> >Naturally we'll still need different jenkins jobs to handle different >>>>>> >resource needs and we'd need to figure out where stuff eventually >>>>>> lives, >>>>>> >but that could come later. >>>>>> > >>>>>> >On Wed, Mar 11, 2015 at 4:34 PM, Chris Nauroth < >>>>>> cnaur...@hortonworks.com> >>>>>> >wrote: >>>>>> > >>>>>> >> The only thing I'm aware of is the failOnError option: >>>>>> >> >>>>>> >> >>>>>> >> >>>>>> http://maven.apache.org/plugins/maven-clean-plugin/examples/ignoring-erro >>>>>> >>rs >>>>>> >> .html >>>>>> >> >>>>>> >> >>>>>> >> I prefer that we don't disable this, because ignoring different >>>>>> kinds of >>>>>> >> failures could leave our build directories in an indeterminate >>>>>> state. >>>>>> >>For >>>>>> >> example, we could end up with an old class file on the classpath >>>>>> for >>>>>> >>test >>>>>> >> runs that was supposedly deleted. >>>>>> >> >>>>>> >> I think it's worth exploring Eddy's suggestion to try simulating >>>>>> failure >>>>>> >> by placing a file where the code expects to see a directory. That >>>>>> might >>>>>> >> even let us enable some of these tests that are skipped on Windows, >>>>>> >> because Windows allows access for the owner even after permissions >>>>>> have >>>>>> >> been stripped. >>>>>> >> >>>>>> >> Chris Nauroth >>>>>> >> Hortonworks >>>>>> >> http://hortonworks.com/ >>>>>> >> >>>>>> >> >>>>>> >> >>>>>> >> >>>>>> >> >>>>>> >> >>>>>> >> On 3/11/15, 2:10 PM, "Colin McCabe" <cmcc...@alumni.cmu.edu> >>>>>> wrote: >>>>>> >> >>>>>> >> >Is there a maven plugin or setting we can use to simply remove >>>>>> >> >directories that have no executable permissions on them? Clearly >>>>>> we >>>>>> >> >have the permission to do this from a technical point of view >>>>>> (since >>>>>> >> >we created the directories as the jenkins user), it's simply that >>>>>> the >>>>>> >> >code refuses to do it. >>>>>> >> > >>>>>> >> >Otherwise I guess we can just fix those tests... >>>>>> >> > >>>>>> >> >Colin >>>>>> >> > >>>>>> >> >On Tue, Mar 10, 2015 at 2:43 PM, Lei Xu <l...@cloudera.com> wrote: >>>>>> >> >> Thanks a lot for looking into HDFS-7722, Chris. >>>>>> >> >> >>>>>> >> >> In HDFS-7722: >>>>>> >> >> TestDataNodeVolumeFailureXXX tests reset data dir permissions in >>>>>> >> >>TearDown(). >>>>>> >> >> TestDataNodeHotSwapVolumes reset permissions in a finally >>>>>> clause. >>>>>> >> >> >>>>>> >> >> Also I ran mvn test several times on my machine and all tests >>>>>> passed. >>>>>> >> >> >>>>>> >> >> However, since in DiskChecker#checkDirAccess(): >>>>>> >> >> >>>>>> >> >> private static void checkDirAccess(File dir) throws >>>>>> >>DiskErrorException { >>>>>> >> >> if (!dir.isDirectory()) { >>>>>> >> >> throw new DiskErrorException("Not a directory: " >>>>>> >> >> + dir.toString()); >>>>>> >> >> } >>>>>> >> >> >>>>>> >> >> checkAccessByFileMethods(dir); >>>>>> >> >> } >>>>>> >> >> >>>>>> >> >> One potentially safer alternative is replacing data dir with a >>>>>> >>regular >>>>>> >> >> file to stimulate disk failures. >>>>>> >> >> >>>>>> >> >> On Tue, Mar 10, 2015 at 2:19 PM, Chris Nauroth >>>>>> >> >><cnaur...@hortonworks.com> wrote: >>>>>> >> >>> TestDataNodeHotSwapVolumes, TestDataNodeVolumeFailure, >>>>>> >> >>> TestDataNodeVolumeFailureReporting, and >>>>>> >> >>> TestDataNodeVolumeFailureToleration all remove executable >>>>>> >>permissions >>>>>> >> >>>from >>>>>> >> >>> directories like the one Colin mentioned to simulate disk >>>>>> failures >>>>>> >>at >>>>>> >> >>>data >>>>>> >> >>> nodes. I reviewed the code for all of those, and they all >>>>>> appear >>>>>> >>to be >>>>>> >> >>> doing the necessary work to restore executable permissions at >>>>>> the >>>>>> >>end >>>>>> >> >>>of >>>>>> >> >>> the test. The only recent uncommitted patch I¹ve seen that >>>>>> makes >>>>>> >> >>>changes >>>>>> >> >>> in these test suites is HDFS-7722. That patch still looks fine >>>>>> >> >>>though. I >>>>>> >> >>> don¹t know if there are other uncommitted patches that changed >>>>>> these >>>>>> >> >>>test >>>>>> >> >>> suites. >>>>>> >> >>> >>>>>> >> >>> I suppose it¹s also possible that the JUnit process >>>>>> unexpectedly >>>>>> >>died >>>>>> >> >>> after removing executable permissions but before restoring >>>>>> them. >>>>>> >>That >>>>>> >> >>> always would have been a weakness of these test suites, >>>>>> regardless >>>>>> >>of >>>>>> >> >>>any >>>>>> >> >>> recent changes. >>>>>> >> >>> >>>>>> >> >>> Chris Nauroth >>>>>> >> >>> Hortonworks >>>>>> >> >>> http://hortonworks.com/ >>>>>> >> >>> >>>>>> >> >>> >>>>>> >> >>> >>>>>> >> >>> >>>>>> >> >>> >>>>>> >> >>> >>>>>> >> >>> On 3/10/15, 1:47 PM, "Aaron T. Myers" <a...@cloudera.com> >>>>>> wrote: >>>>>> >> >>> >>>>>> >> >>>>Hey Colin, >>>>>> >> >>>> >>>>>> >> >>>>I asked Andrew Bayer, who works with Apache Infra, what's >>>>>> going on >>>>>> >>with >>>>>> >> >>>>these boxes. He took a look and concluded that some perms are >>>>>> being >>>>>> >> >>>>set in >>>>>> >> >>>>those directories by our unit tests which are precluding those >>>>>> files >>>>>> >> >>>>from >>>>>> >> >>>>getting deleted. He's going to clean up the boxes for us, but >>>>>> we >>>>>> >>should >>>>>> >> >>>>expect this to keep happening until we can fix the test in >>>>>> question >>>>>> >>to >>>>>> >> >>>>properly clean up after itself. >>>>>> >> >>>> >>>>>> >> >>>>To help narrow down which commit it was that started this, >>>>>> Andrew >>>>>> >>sent >>>>>> >> >>>>me >>>>>> >> >>>>this info: >>>>>> >> >>>> >>>>>> >> >>>>"/home/jenkins/jenkins-slave/workspace/PreCommit-HDFS- >>>>>> >> >>>>>> >>>>>> >>>>>>Build/hadoop-hdfs-project/hadoop-hdfs/target/test/data/dfs/data/data3 >>>>>> >>>>>>/ >>>>>> >> >>>>has >>>>>> >> >>>>500 perms, so I'm guessing that's the problem. Been that way >>>>>> since >>>>>> >>9:32 >>>>>> >> >>>>UTC >>>>>> >> >>>>on March 5th." >>>>>> >> >>>> >>>>>> >> >>>>-- >>>>>> >> >>>>Aaron T. Myers >>>>>> >> >>>>Software Engineer, Cloudera >>>>>> >> >>>> >>>>>> >> >>>>On Tue, Mar 10, 2015 at 1:24 PM, Colin P. McCabe >>>>>> >><cmcc...@apache.org> >>>>>> >> >>>>wrote: >>>>>> >> >>>> >>>>>> >> >>>>> Hi all, >>>>>> >> >>>>> >>>>>> >> >>>>> A very quick (and not thorough) survey shows that I can't >>>>>> find any >>>>>> >> >>>>> jenkins jobs that succeeded from the last 24 hours. Most of >>>>>> them >>>>>> >> >>>>>seem >>>>>> >> >>>>> to be failing with some variant of this message: >>>>>> >> >>>>> >>>>>> >> >>>>> [ERROR] Failed to execute goal >>>>>> >> >>>>> org.apache.maven.plugins:maven-clean-plugin:2.5:clean >>>>>> >>(default-clean) >>>>>> >> >>>>> on project hadoop-hdfs: Failed to clean project: Failed to >>>>>> delete >>>>>> >> >>>>> >>>>>> >> >>>>> >>>>>> >> >>>>>> >>>>>> >>>>>>>/home/jenkins/jenkins-slave/workspace/PreCommit-HDFS-Build/hadoop-hd >>>>>> >>>>>>>fs >>>>>> >> >>>>>-pr >>>>>> >> >>>>>oject/hadoop-hdfs/target/test/data/dfs/data/data3 >>>>>> >> >>>>> -> [Help 1] >>>>>> >> >>>>> >>>>>> >> >>>>> Any ideas how this happened? Bad disk, unit test setting >>>>>> wrong >>>>>> >> >>>>> permissions? >>>>>> >> >>>>> >>>>>> >> >>>>> Colin >>>>>> >> >>>>> >>>>>> >> >>> >>>>>> >> >> >>>>>> >> >> >>>>>> >> >> >>>>>> >> >> -- >>>>>> >> >> Lei (Eddy) Xu >>>>>> >> >> Software Engineer, Cloudera >>>>>> >> >>>>>> >> >>>>>> > >>>>>> > >>>>>> >-- >>>>>> >Sean >>>>>> >>>>>> >>>>> >>>>> >>>>> -- >>>>> Sean >>>>> >>>> >>>> >>>> >>>> -- >>>> Sean >>>> >>> >>> >>> >>> -- >>> Sean >>> >> >> >> >> -- >> Sean >> > > > > -- > Sean > -- Sean
