On Mon, Jun 22, 2015 at 1:03 PM, Nick Dimiduk <ndimi...@gmail.com> wrote:
> On Mon, Jun 22, 2015 at 12:43 PM, Colin P. McCabe <cmcc...@apache.org> > wrote: > > > You mentioned that "most of our project will be focused on shell > > scripts" I guess based on the existing test-patch code. Allen did a > > lot of good work in this area recently. I am curious if you evaluated > > languages such as Python or Node.js for this use-case. Shell scripts > > can get a little... tricky beyond a certain size. On the other hand, > > if we are standardizing on shell, which shell and which version? > > Perhaps bash 3.5+? > > > > I'll also add that shell is not helpful for a cross-platform set of > tooling. I recently added a daemon to Apache Phoenix; an explicit > requirement was Windows support. I ended up implementing a solution in > python because that environment is platform-agnostic and still systems-y > enough. I think this is something this project should seriously consider. > In my opinion, historically, test-patch hasn't needed to be cross platform because the only first class development environment for Hadoop has been Linux. Growing beyond this could absolutely be one focus of Yetus should that be a consensus goal of the community. The seed of the project, though, is today's test-patch, which is implemented in bash. That's where we are today. Language "discussions" (smile) can and should be forward looking. On Mon, Jun 22, 2015 at 1:03 PM, Nick Dimiduk <ndimi...@gmail.com> wrote: > On Mon, Jun 22, 2015 at 12:43 PM, Colin P. McCabe <cmcc...@apache.org> > wrote: > > > You mentioned that "most of our project will be focused on shell > > scripts" I guess based on the existing test-patch code. Allen did a > > lot of good work in this area recently. I am curious if you evaluated > > languages such as Python or Node.js for this use-case. Shell scripts > > can get a little... tricky beyond a certain size. On the other hand, > > if we are standardizing on shell, which shell and which version? > > Perhaps bash 3.5+? > > > > I'll also add that shell is not helpful for a cross-platform set of > tooling. I recently added a daemon to Apache Phoenix; an explicit > requirement was Windows support. I ended up implementing a solution in > python because that environment is platform-agnostic and still systems-y > enough. I think this is something this project should seriously consider. > > -n > > On Tue, Jun 16, 2015 at 7:55 PM, Sean Busbey <bus...@cloudera.com> wrote: > > > I'm going to try responding to several things at once here, so > apologies > > if > > > I miss anyone and sorry for the long email. :) > > > > > > > > > On Tue, Jun 16, 2015 at 3:44 PM, Steve Loughran < > ste...@hortonworks.com> > > > wrote: > > > > > >> I think it's good to have a general build/test process projects can > > share, > > >> so +1 to pulling it out. You should get help from others. > > >> > > >> regarding incubation, it is a lot of work, especially for something > > that's > > >> more of an in-house tool than an artifact to release and redistribute. > > >> > > >> You can't just use apache labs or the build project's repo to work on > > this? > > >> > > >> if you do want to incubate, we may want to nominate the hadoop project > > as > > >> the monitoring PMC, rather than incubator@. > > >> > > >> -steve > > >> > > >> > > > Important note: we're proposing a board resolution that would directly > > pull > > > this code base out into a new TLP; there'd be no incubator, we'd just > > > continue building community and start making releases. > > > > > > The proposed PMC believes the tooling we're talking about has direct > > > applicability to projects well outside of the ASF. Lot's of other open > > > source projects run on community contributions and have a general need > > for > > > better QA tools. Given that problem set and the presence of a community > > > working to solve it, there's no reason this needs to be treated as an > > > in-house build project. We certainly want to be useful to ASF projects > > and > > > getting them on-board given our current optimization for ASF infra will > > > certainly be easier, but we're not limited to that (and our current > > > prerequisites, a CI tool and jira or github, are pretty broadly > > available). > > > > > > > > > On Tue, Jun 16, 2015 at 10:13 AM, Nick Dimiduk <ndimi...@apache.org> > > wrote: > > > > > >> > > >> Since we're tossing out names, how about Apache Bootstrap? It's a > > >> meta-project to help other projects get off the ground, after all. > > >> > > > > > > > > > There's already a web development framework named Bootstrap[1]. It's > also > > > used by several ASF projects, so I think it best to avoid the > confusion. > > > > > > The name is, of course, up to the proposed PMC. As a bit of background, > > the > > > current name Yetus fulfills Allen's desire to have something shell > > related > > > and my desire to have a project that starts with Y (there are currently > > no > > > ASF projects that start with Y). The universe of names that fill in > these > > > two is very small, AFAICT. I did a brief suitability search and didn't > > find > > > any blockers. > > > > > > > > > On Tue, Jun 16, 2015 at 11:59 AM, Allen Wittenauer <a...@altiscale.com> > > > wrote: > > > > > >> > > >> Since a couple of people have brought it up: > > >> > > >> I think the release question is probably one of the big > question > > >> marks. Other than tar balls, how does something like this actually > get > > >> used downstream? > > >> > > >> For test-patch, in particular, I have a few thoughts on this: > > >> > > >> Short term: > > >> > > >> * Projects that want to move RIGHT NOW would modify their > > Jenkins > > >> jobs to checkout from the Yetus repo (preferably at a well known tag > or > > >> branch) in one directory and their project repo in another directory. > > Then > > >> it’s just a matter of passing the correct flags to test-patch. This > is > > >> pretty much how I’ve been personally running test-patch for about 6 > > months > > >> now. Under Jenkins, we’ve seen this work with NiFi (incubating) > already. > > >> > > >> * Create a stub version of test-patch that projects could > check > > >> into their repo, replacing the existing test-patch. This stub version > > >> would git clone from either ASF or github and then execute test-patch > > >> accordingly on demand. With the correct smarts, it could make sure it > > has > > >> a cached version to prevent continual clones. > > >> > > >> Longer term: > > >> > > >> * I’ve been toying with the idea of (ab)using Java repos and > > >> packaging as a transportation layer, either in addition or in > > combination > > >> with something like a maven plugin. Something like this would clearly > > be > > >> better for offline usage and/or to lower the network traffic. > > >> > > > > > > It's important that the project follow ASF guidelines on publishing > > > releases[2]. So long as we publish releases to the distribution > > directory I > > > think we'd be fine having folks work off of the corresponding tag. I'm > > not > > > sure there's much reason to do that, however. A Jenkins job can just as > > > easily grab a release tarball as a git tag and we're not talking about > a > > > large amount of stuff. The kind of build setup that Chris N mentioned > is > > > also totally doable now that there's a build description DSL for > > Jenkins[3]. > > > > > > For individual developers, I don't see any reason we can't package > things > > > up as a tool, similar to how findbugs or shellcheck work. We can make > OS > > > packages (or homebrew for OS X) if we want to make stand alone > > installation > > > on developer machines real easy. Those same packages could be installed > > on > > > the ASF build machines, provided some ASF project wanted to make use of > > > Yetus. > > > > > > Having releases will incur some turn around time for when folks want to > > see > > > fixes, but that's a trade off around release cadence we can work out > > longer > > > term. > > > > > > I would like to have one or two projects that can work off of the > > bleeding > > > edge repo, but we'd have to get that to mesh with foundation policy. My > > gut > > > tells me we should be able to come up with an agreement that makes > such a > > > project "part of the development community" but the specifics will have > > to > > > be worked out. > > > > > > > > > On Tue, Jun 16, 2015 at 4:41 AM, Jonathan Hsieh <j...@cloudera.com> > > wrote: > > > > > >> How would we have to add testing to our pre commit testing? > > >> > > >> Each project has likely customized their own core commit scripts > > (multiple > > >> Jvms versions, checkstyle, javadoc exceptions etc.). We should > probably > > >> soliciit interest from other projects who already have fancy precommit > > >> tests beyond HBase/Hadoop too. > > >> > > > > > > I'm not sure if Allen's response above answered this for you. The > current > > > state of test-patch has a plugin system for adding new tests. It is > also > > > customizable to handle project idiosyncrasies (atleast the ones we've > > seen > > > so far, like unspecified module dependencies, multiple projects per > repo, > > > or use of ant). Releases will naturally include docs on how to leverage > > > those to customize what a particular project needs tested pre-commit. > > > > > > Given the nature of the work Yetus is hoping to enable, I think it's > safe > > > to assume the project community is going to be doing a fair bit of > > outreach > > > to help show projects outside of HBase/Hadoop how things can work for > > them. > > > > > > > > > > > > [1]: http://getbootstrap.com/ > > > [2]: http://www.apache.org/dev/release.html > > > [3]: https://wiki.jenkins-ci.org/display/JENKINS/Job+DSL+Plugin > > > > > > -- > > > Sean > > > -- Best regards, - Andy Problems worthy of attack prove their worth by hitting back. - Piet Hein (via Tom White)
