+1 for making this a separate project. We've always struggled with a lot of forks of the test-patch code and perhaps this project can help create something that works well for multiple projects.
Bypassing the incubator seems kind of weird (I didn't know that was an option) but I will let other people with more experience in the ASF comment on that. You mentioned that "most of our project will be focused on shell scripts" I guess based on the existing test-patch code. Allen did a lot of good work in this area recently. I am curious if you evaluated languages such as Python or Node.js for this use-case. Shell scripts can get a little... tricky beyond a certain size. On the other hand, if we are standardizing on shell, which shell and which version? Perhaps bash 3.5+? Also, what will be the mechanism for customizing this for each project? Ideally the customizations needed would be small so we could share the most code. cheers, Colin On Tue, Jun 16, 2015 at 7:55 PM, Sean Busbey <bus...@cloudera.com> wrote: > I'm going to try responding to several things at once here, so apologies if > I miss anyone and sorry for the long email. :) > > > On Tue, Jun 16, 2015 at 3:44 PM, Steve Loughran <ste...@hortonworks.com> > wrote: > >> I think it's good to have a general build/test process projects can share, >> so +1 to pulling it out. You should get help from others. >> >> regarding incubation, it is a lot of work, especially for something that's >> more of an in-house tool than an artifact to release and redistribute. >> >> You can't just use apache labs or the build project's repo to work on this? >> >> if you do want to incubate, we may want to nominate the hadoop project as >> the monitoring PMC, rather than incubator@. >> >> -steve >> >> > Important note: we're proposing a board resolution that would directly pull > this code base out into a new TLP; there'd be no incubator, we'd just > continue building community and start making releases. > > The proposed PMC believes the tooling we're talking about has direct > applicability to projects well outside of the ASF. Lot's of other open > source projects run on community contributions and have a general need for > better QA tools. Given that problem set and the presence of a community > working to solve it, there's no reason this needs to be treated as an > in-house build project. We certainly want to be useful to ASF projects and > getting them on-board given our current optimization for ASF infra will > certainly be easier, but we're not limited to that (and our current > prerequisites, a CI tool and jira or github, are pretty broadly available). > > > On Tue, Jun 16, 2015 at 10:13 AM, Nick Dimiduk <ndimi...@apache.org> wrote: > >> >> Since we're tossing out names, how about Apache Bootstrap? It's a >> meta-project to help other projects get off the ground, after all. >> > > > There's already a web development framework named Bootstrap[1]. It's also > used by several ASF projects, so I think it best to avoid the confusion. > > The name is, of course, up to the proposed PMC. As a bit of background, the > current name Yetus fulfills Allen's desire to have something shell related > and my desire to have a project that starts with Y (there are currently no > ASF projects that start with Y). The universe of names that fill in these > two is very small, AFAICT. I did a brief suitability search and didn't find > any blockers. > > > On Tue, Jun 16, 2015 at 11:59 AM, Allen Wittenauer <a...@altiscale.com> > wrote: > >> >> Since a couple of people have brought it up: >> >> I think the release question is probably one of the big question >> marks. Other than tar balls, how does something like this actually get >> used downstream? >> >> For test-patch, in particular, I have a few thoughts on this: >> >> Short term: >> >> * Projects that want to move RIGHT NOW would modify their Jenkins >> jobs to checkout from the Yetus repo (preferably at a well known tag or >> branch) in one directory and their project repo in another directory. Then >> it’s just a matter of passing the correct flags to test-patch. This is >> pretty much how I’ve been personally running test-patch for about 6 months >> now. Under Jenkins, we’ve seen this work with NiFi (incubating) already. >> >> * Create a stub version of test-patch that projects could check >> into their repo, replacing the existing test-patch. This stub version >> would git clone from either ASF or github and then execute test-patch >> accordingly on demand. With the correct smarts, it could make sure it has >> a cached version to prevent continual clones. >> >> Longer term: >> >> * I’ve been toying with the idea of (ab)using Java repos and >> packaging as a transportation layer, either in addition or in combination >> with something like a maven plugin. Something like this would clearly be >> better for offline usage and/or to lower the network traffic. >> > > It's important that the project follow ASF guidelines on publishing > releases[2]. So long as we publish releases to the distribution directory I > think we'd be fine having folks work off of the corresponding tag. I'm not > sure there's much reason to do that, however. A Jenkins job can just as > easily grab a release tarball as a git tag and we're not talking about a > large amount of stuff. The kind of build setup that Chris N mentioned is > also totally doable now that there's a build description DSL for Jenkins[3]. > > For individual developers, I don't see any reason we can't package things > up as a tool, similar to how findbugs or shellcheck work. We can make OS > packages (or homebrew for OS X) if we want to make stand alone installation > on developer machines real easy. Those same packages could be installed on > the ASF build machines, provided some ASF project wanted to make use of > Yetus. > > Having releases will incur some turn around time for when folks want to see > fixes, but that's a trade off around release cadence we can work out longer > term. > > I would like to have one or two projects that can work off of the bleeding > edge repo, but we'd have to get that to mesh with foundation policy. My gut > tells me we should be able to come up with an agreement that makes such a > project "part of the development community" but the specifics will have to > be worked out. > > > On Tue, Jun 16, 2015 at 4:41 AM, Jonathan Hsieh <j...@cloudera.com> wrote: > >> How would we have to add testing to our pre commit testing? >> >> Each project has likely customized their own core commit scripts (multiple >> Jvms versions, checkstyle, javadoc exceptions etc.). We should probably >> soliciit interest from other projects who already have fancy precommit >> tests beyond HBase/Hadoop too. >> > > I'm not sure if Allen's response above answered this for you. The current > state of test-patch has a plugin system for adding new tests. It is also > customizable to handle project idiosyncrasies (atleast the ones we've seen > so far, like unspecified module dependencies, multiple projects per repo, > or use of ant). Releases will naturally include docs on how to leverage > those to customize what a particular project needs tested pre-commit. > > Given the nature of the work Yetus is hoping to enable, I think it's safe > to assume the project community is going to be doing a fair bit of outreach > to help show projects outside of HBase/Hadoop how things can work for them. > > > > [1]: http://getbootstrap.com/ > [2]: http://www.apache.org/dev/release.html > [3]: https://wiki.jenkins-ci.org/display/JENKINS/Job+DSL+Plugin > > -- > Sean
