+1 for a separate project and going directly to TLP if possible (as Hadoop itself did when split out of Nutch)
+1 for having language discussions once it's a TLP :-) Cheers, Nigel > On Jun 22, 2015, at 1:55 PM, Andrew Purtell <apurt...@apache.org> wrote: > >> On Mon, Jun 22, 2015 at 1:03 PM, Nick Dimiduk <ndimi...@gmail.com> wrote: >> >> On Mon, Jun 22, 2015 at 12:43 PM, Colin P. McCabe <cmcc...@apache.org> >> wrote: >> >>> You mentioned that "most of our project will be focused on shell >>> scripts" I guess based on the existing test-patch code. Allen did a >>> lot of good work in this area recently. I am curious if you evaluated >>> languages such as Python or Node.js for this use-case. Shell scripts >>> can get a little... tricky beyond a certain size. On the other hand, >>> if we are standardizing on shell, which shell and which version? >>> Perhaps bash 3.5+? >> >> I'll also add that shell is not helpful for a cross-platform set of >> tooling. I recently added a daemon to Apache Phoenix; an explicit >> requirement was Windows support. I ended up implementing a solution in >> python because that environment is platform-agnostic and still systems-y >> enough. I think this is something this project should seriously consider. > > In my opinion, historically, test-patch hasn't needed to be cross platform > because the only first class development environment for Hadoop has been > Linux. Growing beyond this could absolutely be one focus of Yetus should > that be a consensus goal of the community. The seed of the project, though, > is today's test-patch, which is implemented in bash. That's where we are > today. Language "discussions" (smile) can and should be forward looking. > > >> On Mon, Jun 22, 2015 at 1:03 PM, Nick Dimiduk <ndimi...@gmail.com> wrote: >> >> On Mon, Jun 22, 2015 at 12:43 PM, Colin P. McCabe <cmcc...@apache.org> >> wrote: >> >>> You mentioned that "most of our project will be focused on shell >>> scripts" I guess based on the existing test-patch code. Allen did a >>> lot of good work in this area recently. I am curious if you evaluated >>> languages such as Python or Node.js for this use-case. Shell scripts >>> can get a little... tricky beyond a certain size. On the other hand, >>> if we are standardizing on shell, which shell and which version? >>> Perhaps bash 3.5+? >> >> I'll also add that shell is not helpful for a cross-platform set of >> tooling. I recently added a daemon to Apache Phoenix; an explicit >> requirement was Windows support. I ended up implementing a solution in >> python because that environment is platform-agnostic and still systems-y >> enough. I think this is something this project should seriously consider. >> >> -n >> >> On Tue, Jun 16, 2015 at 7:55 PM, Sean Busbey <bus...@cloudera.com> wrote: >>>> I'm going to try responding to several things at once here, so >> apologies >>> if >>>> I miss anyone and sorry for the long email. :) >>>> >>>> >>>> On Tue, Jun 16, 2015 at 3:44 PM, Steve Loughran < >> ste...@hortonworks.com> >>>> wrote: >>>> >>>>> I think it's good to have a general build/test process projects can >>> share, >>>>> so +1 to pulling it out. You should get help from others. >>>>> >>>>> regarding incubation, it is a lot of work, especially for something >>> that's >>>>> more of an in-house tool than an artifact to release and redistribute. >>>>> >>>>> You can't just use apache labs or the build project's repo to work on >>> this? >>>>> >>>>> if you do want to incubate, we may want to nominate the hadoop project >>> as >>>>> the monitoring PMC, rather than incubator@. >>>>> >>>>> -steve >>>> Important note: we're proposing a board resolution that would directly >>> pull >>>> this code base out into a new TLP; there'd be no incubator, we'd just >>>> continue building community and start making releases. >>>> >>>> The proposed PMC believes the tooling we're talking about has direct >>>> applicability to projects well outside of the ASF. Lot's of other open >>>> source projects run on community contributions and have a general need >>> for >>>> better QA tools. Given that problem set and the presence of a community >>>> working to solve it, there's no reason this needs to be treated as an >>>> in-house build project. We certainly want to be useful to ASF projects >>> and >>>> getting them on-board given our current optimization for ASF infra will >>>> certainly be easier, but we're not limited to that (and our current >>>> prerequisites, a CI tool and jira or github, are pretty broadly >>> available). >>>> >>>> >>>>> On Tue, Jun 16, 2015 at 10:13 AM, Nick Dimiduk <ndimi...@apache.org> >>>> wrote: >>>> >>>>> >>>>> Since we're tossing out names, how about Apache Bootstrap? It's a >>>>> meta-project to help other projects get off the ground, after all. >>>> >>>> >>>> There's already a web development framework named Bootstrap[1]. It's >> also >>>> used by several ASF projects, so I think it best to avoid the >> confusion. >>>> >>>> The name is, of course, up to the proposed PMC. As a bit of background, >>> the >>>> current name Yetus fulfills Allen's desire to have something shell >>> related >>>> and my desire to have a project that starts with Y (there are currently >>> no >>>> ASF projects that start with Y). The universe of names that fill in >> these >>>> two is very small, AFAICT. I did a brief suitability search and didn't >>> find >>>> any blockers. >>>> >>>> >>>> On Tue, Jun 16, 2015 at 11:59 AM, Allen Wittenauer <a...@altiscale.com> >>>> wrote: >>>> >>>>> >>>>> Since a couple of people have brought it up: >>>>> >>>>> I think the release question is probably one of the big >> question >>>>> marks. Other than tar balls, how does something like this actually >> get >>>>> used downstream? >>>>> >>>>> For test-patch, in particular, I have a few thoughts on this: >>>>> >>>>> Short term: >>>>> >>>>> * Projects that want to move RIGHT NOW would modify their >>> Jenkins >>>>> jobs to checkout from the Yetus repo (preferably at a well known tag >> or >>>>> branch) in one directory and their project repo in another directory. >>> Then >>>>> it’s just a matter of passing the correct flags to test-patch. This >> is >>>>> pretty much how I’ve been personally running test-patch for about 6 >>> months >>>>> now. Under Jenkins, we’ve seen this work with NiFi (incubating) >> already. >>>>> >>>>> * Create a stub version of test-patch that projects could >> check >>>>> into their repo, replacing the existing test-patch. This stub version >>>>> would git clone from either ASF or github and then execute test-patch >>>>> accordingly on demand. With the correct smarts, it could make sure it >>> has >>>>> a cached version to prevent continual clones. >>>>> >>>>> Longer term: >>>>> >>>>> * I’ve been toying with the idea of (ab)using Java repos and >>>>> packaging as a transportation layer, either in addition or in >>> combination >>>>> with something like a maven plugin. Something like this would clearly >>> be >>>>> better for offline usage and/or to lower the network traffic. >>>> >>>> It's important that the project follow ASF guidelines on publishing >>>> releases[2]. So long as we publish releases to the distribution >>> directory I >>>> think we'd be fine having folks work off of the corresponding tag. I'm >>> not >>>> sure there's much reason to do that, however. A Jenkins job can just as >>>> easily grab a release tarball as a git tag and we're not talking about >> a >>>> large amount of stuff. The kind of build setup that Chris N mentioned >> is >>>> also totally doable now that there's a build description DSL for >>> Jenkins[3]. >>>> >>>> For individual developers, I don't see any reason we can't package >> things >>>> up as a tool, similar to how findbugs or shellcheck work. We can make >> OS >>>> packages (or homebrew for OS X) if we want to make stand alone >>> installation >>>> on developer machines real easy. Those same packages could be installed >>> on >>>> the ASF build machines, provided some ASF project wanted to make use of >>>> Yetus. >>>> >>>> Having releases will incur some turn around time for when folks want to >>> see >>>> fixes, but that's a trade off around release cadence we can work out >>> longer >>>> term. >>>> >>>> I would like to have one or two projects that can work off of the >>> bleeding >>>> edge repo, but we'd have to get that to mesh with foundation policy. My >>> gut >>>> tells me we should be able to come up with an agreement that makes >> such a >>>> project "part of the development community" but the specifics will have >>> to >>>> be worked out. >>>> >>>> >>>>> On Tue, Jun 16, 2015 at 4:41 AM, Jonathan Hsieh <j...@cloudera.com> >>>> wrote: >>>> >>>>> How would we have to add testing to our pre commit testing? >>>>> >>>>> Each project has likely customized their own core commit scripts >>> (multiple >>>>> Jvms versions, checkstyle, javadoc exceptions etc.). We should >> probably >>>>> soliciit interest from other projects who already have fancy precommit >>>>> tests beyond HBase/Hadoop too. >>>> >>>> I'm not sure if Allen's response above answered this for you. The >> current >>>> state of test-patch has a plugin system for adding new tests. It is >> also >>>> customizable to handle project idiosyncrasies (atleast the ones we've >>> seen >>>> so far, like unspecified module dependencies, multiple projects per >> repo, >>>> or use of ant). Releases will naturally include docs on how to leverage >>>> those to customize what a particular project needs tested pre-commit. >>>> >>>> Given the nature of the work Yetus is hoping to enable, I think it's >> safe >>>> to assume the project community is going to be doing a fair bit of >>> outreach >>>> to help show projects outside of HBase/Hadoop how things can work for >>> them. >>>> >>>> >>>> >>>> [1]: http://getbootstrap.com/ >>>> [2]: http://www.apache.org/dev/release.html >>>> [3]: https://wiki.jenkins-ci.org/display/JENKINS/Job+DSL+Plugin >>>> >>>> -- >>>> Sean > > > > -- > Best regards, > > - Andy > > Problems worthy of attack prove their worth by hitting back. - Piet Hein > (via Tom White)
