Hi Ted. Incubator has a number of benefits for new projects; helping them learn the ropes of Apache, setup infrastructure, build community, create their own Apache blessed release, etc... That's a big part of the responsibility that IPMC members take on when a project enters incubation, something we'd (ZKPMC) have to do ourselves in the approach you outline. Would you be able to commit that sort of time?
wrt examples: MRUnit is an incubator project, it was a contrib of Hadoop. Hadoop recently worked to shed itself of all (most?) contribs. This is similar no? Apache Felix is another example (in the extreme, it has 20 subs) http://felix.apache.org/site/index.html Doug has always told me that you want to be guided by community more than anything. Keep two projects together if they are the same community, otw separating them is ok (he strongly favors incubator over sub). Patrick On Mon, Dec 5, 2011 at 12:36 PM, Ted Dunning <ted.dunn...@gmail.com> wrote: > -1 > > I don't think that a full incubator process is needed or even useful here. > Incubator is ONE process in Apache process for this, but a contrib > artifact is another way to do it. > > This isn't a sub-project. This is additional ZK software. It just isn't > the core stuff. This is more akin to Solr being added to Lucene than it > is like Mahout spinning out of Lucene ultimately as a top level project. > > > On Mon, Dec 5, 2011 at 11:58 AM, Mahadev Konar <maha...@hortonworks.com>wrote: > >> +1... >> >> mahadev >> >> On Mon, Dec 5, 2011 at 11:42 AM, Benjamin Reed <br...@apache.org> wrote: >> > +1 >> > >> > On Mon, Dec 5, 2011 at 11:11 AM, Patrick Hunt <ph...@apache.org> wrote: >> >> Incubator is the Apache recommended way to handle this. We could be >> >> the sponsor, with the intent that once Curator graduates from the >> >> incubator we would accept it >> >> >> http://incubator.apache.org/incubation/Roles_and_Responsibilities.html#Sponsor >> >> anyone could join the curator effort, including existing ZK >> >> committers/contributors. >> >> >> >> if curator felt it should be a tlp (at time of graduation) I think >> >> that would be fine too. >> >> >> >> I would be happy to Champion/Mentor this effort. >> >> >> >> Patrick >> >> >> >> On Mon, Dec 5, 2011 at 11:06 AM, Ted Dunning <ted.dunn...@gmail.com> >> wrote: >> >>> Ben, >> >>> >> >>> I think that you are right on target with this. We need to have a >> wider >> >>> community of developers in order to have the bandwidth to handle >> recipes >> >>> and the recipes really do need separate release cycles. Independent >> >>> releases should be not much of an issue given how compatible ZK >> versions >> >>> tend to be. >> >>> >> >>> On Mon, Dec 5, 2011 at 10:59 AM, Benjamin Reed <br...@apache.org> >> wrote: >> >>> >> >>>> i also agreed that it would be great to build a community around >> >>>> curator especially if it could pull in other zk recipes. i've felt for >> >>>> a while that the core developers don't have the bandwidth to really >> >>>> keep track of recipe development and recipes are a very important >> >>>> aspect of zk. so, having a self sustaining community of developers >> >>>> focused on that would be awesome. being able to have a separate >> >>>> release cycle is also important i think. (i may be reading too much >> >>>> into the proposal though :) >> >>>> >> >>>> ben >> >>>> >> >>>> On Mon, Dec 5, 2011 at 7:14 AM, Flavio Junqueira <f...@yahoo-inc.com> >> >>>> wrote: >> >>>> > I suppose you're just trying to sense if others think that it would >> be a >> >>>> > good idea to have it as a subproject. If so, I'm in favor of >> putting a >> >>>> > proposal together, +1. >> >>>> > >> >>>> > -Flavio >> >>>> > >> >>>> > On Dec 2, 2011, at 8:45 PM, Patrick Hunt wrote: >> >>>> > >> >>>> >> Somehow this got off list, Jordan and I just noticed, summary so >> far: >> >>>> >> >> >>>> >> ---------- Forwarded message ---------- >> >>>> >> From: Patrick Hunt <ph...@apache.org> >> >>>> >> Date: Fri, Dec 2, 2011 at 9:56 AM >> >>>> >> Subject: Re: Proposal: create a separate ZooKeeper release >> artifact for >> >>>> >> recipes >> >>>> >> To: Jordan Zimmerman <jzimmer...@netflix.com> >> >>>> >> >> >>>> >> >> >>>> >> Did you mean to reply just to me? I think this is a great idea btw, >> >>>> >> just need to work out the mechanics of getting it integrated and >> get >> >>>> >> the others on board. Would it help to talk f2f? >> >>>> >> >> >>>> >> On Thu, Dec 1, 2011 at 6:20 PM, Jordan Zimmerman < >> >>>> jzimmer...@netflix.com> >> >>>> >> wrote: >> >>>> >>> >> >>>> >>> From talking to folks here (and my own feeling), I'd like to have >> >>>> Curator >> >>>> >>> get as wide use as possible. Some sort of sanction from the >> ZooKeeper >> >>>> >>> team >> >>>> >>> would do that. I don't see a lot of downside to getting a >> community >> >>>> >>> around >> >>>> >>> it either. Sure, it's nice to be the only guy I need to please, >> but the >> >>>> >>> benefits of a community outweigh that. >> >>>> >>> >> >>>> >>> I think the beauty of folding it in to ZooKeeper is that the >> structure >> >>>> is >> >>>> >>> already there: Jira, website, etc. I (and probably one other from >> >>>> >>> Netflix) >> >>>> >>> could take on the role of managing the Curator portion so that it >> >>>> doesn't >> >>>> >>> burden you folks. I'm doing that internally at Netflix anyway. >> Frankly, >> >>>> >>> it >> >>>> >>> sounds like fun (famous last words). >> >>>> >>> >> >>>> >>> -Jordan >> >>>> >>> >> >>>> >>> On 12/1/11 5:36 PM, "Patrick Hunt" <ph...@apache.org> wrote: >> >>>> >>> >> >>>> >>>> On Thu, Dec 1, 2011 at 2:05 PM, Jordan Zimmerman >> >>>> >>>> <jzimmer...@netflix.com> >> >>>> >>>> wrote: >> >>>> >>>>> >> >>>> >>>>> The feeling here is that it would be nice to push Curator into >> ZK. >> >>>> >>>>> However, we'd still like to contribute to it. If you don't want >> >>>> >>>>> separate >> >>>> >>>>> contributor lists, I could be a ZK contributor but explicitly >> only >> >>>> work >> >>>> >>>>> on >> >>>> >>>>> Curator. Of course, we're open to your suggestions on this as >> well. >> >>>> >>>> >> >>>> >>>> >> >>>> >>>> Don't get me wrong, I don't mind having one list of >> >>>> >>>> contributors/committers for ZK/Curator, I'm personally fine with >> >>>> >>>> having one. If you notice in my original proposal I lean towards >> that >> >>>> >>>> direction (single set of committers with multiple artifacts) >> >>>> >>>> >> >>>> >>>> I see a benefit to having something like Curator available to >> users, >> >>>> >>>> open sourcing it on GH accomplishes this. Bringing something to >> Apache >> >>>> >>>> means building a community around it though, not just making the >> code >> >>>> >>>> open source. This is analogous to when ZK itself was on >> sourceforge, >> >>>> >>>> eventually moving to Apache. >> >>>> >>>> >> >>>> >>>> Here's the issue: building community adds overhead, which you >> might >> >>>> >>>> not be willing to sign up for. I'm not sure we (zk community) >> can take >> >>>> >>>> on this additional cost alone. Hence my question -- I _wouldn't_ >> want >> >>>> >>>> to you leave, that's the point. We'd need some set of core >> "Curator >> >>>> >>>> contributors" who would continue to work on Curator, answer >> questions, >> >>>> >>>> shepherd new contributor/committers/etc... If that doesn't >> happen the >> >>>> >>>> code will get imported, see some use, but never really reach >> it's full >> >>>> >>>> potential. >> >>>> >>>> >> >>>> >>>> Hope that helps. LMK. >> >>>> >>>> >> >>>> >>>> Patrick >> >>>> >>>> >> >>>> >>>>> >> >>>> >>>>> On 12/1/11 2:01 PM, "Patrick Hunt" <ph...@apache.org> wrote: >> >>>> >>>>> >> >>>> >>>>>> On Thu, Dec 1, 2011 at 1:26 PM, Jordan Zimmerman >> >>>> >>>>>> <jzimmer...@netflix.com> >> >>>> >>>>>> wrote: >> >>>> >>>>>>> >> >>>> >>>>>>> I've spoken about this with the appropriate folks here at >> Netflix >> >>>> and >> >>>> >>>>>>> we're interested in pursuing this. What are the next steps? >> >>>> >>>>>> >> >>>> >>>>>> >> >>>> >>>>>> Hi Jordan, what are your intentions. Do you want to build a >> >>>> community >> >>>> >>>>>> around "Apache Curator" or are you just looking to push the >> source >> >>>> >>>>>> into ZK and move on to other things? >> >>>> >>>>>> >> >>>> >>>>>> Patrick >> >>>> >>>>>> >> >>>> >>>>>> >> >>>> >>>>>>> >> >>>> >>>>>>> >> >>>> >>>>>>> On 11/30/11 5:42 PM, "Ted Dunning" <ted.dunn...@gmail.com> >> wrote: >> >>>> >>>>>>> >> >>>> >>>>>>>> Separator committer lists are generally frowned on by Apache >> (ref >> >>>> >>>>>>>> Lucene/Solr and also Mahout). >> >>>> >>>>>>>> >> >>>> >>>>>>>> They also are essentially unnecessary. It isn't that big a >> deal >> >>>> to >> >>>> >>>>>>>> have a >> >>>> >>>>>>>> single committer list and depend on reversion to enforce >> community >> >>>> >>>>>>>> standards. Core ZK is review-then-commit and should >> continue to >> >>>> be. >> >>>> >>>>>>>> Add-on modules that are relatively independent of the core >> can >> >>>> quite >> >>>> >>>>>>>> reasonably be commit-then-review or whatever level of care is >> >>>> >>>>>>>> implied >> >>>> >>>>>>>> by >> >>>> >>>>>>>> module's content. The review can include a determination of >> who >> >>>> >>>>>>>> should >> >>>> >>>>>>>> actually do the commit. >> >>>> >>>>>>>> >> >>>> >>>>>>>> Moreover, projects can easily have more than one artifact. >> Mahout >> >>>> >>>>>>>> has >> >>>> >>>>>>>> three independent artifacts (mahout-math, mahout-collections >> and >> >>>> the >> >>>> >>>>>>>> core >> >>>> >>>>>>>> stuff with examples and integration code). That works well. >> >>>> There >> >>>> >>>>>>>> are >> >>>> >>>>>>>> different sets of committers who are typically the ones who >> work >> >>>> on >> >>>> >>>>>>>> different components. The overall level of diligence >> required in >> >>>> >>>>>>>> Mahout >> >>>> >>>>>>>> is >> >>>> >>>>>>>> much lower than for Zookeeper as you might expect, but the >> >>>> principle >> >>>> >>>>>>>> that >> >>>> >>>>>>>> multiple artifacts are fine is still the same. Likewise, >> the idea >> >>>> >>>>>>>> of >> >>>> >>>>>>>> committers with specialized expertise also applies here. >> >>>> >>>>>>>> >> >>>> >>>>>>>> There is a tradition in Zookeeper of having a very high bar >> for >> >>>> >>>>>>>> committership, but experience in some other projects >> indicates >> >>>> that >> >>>> >>>>>>>> this >> >>>> >>>>>>>> doesn't actually help quality or security all that much and >> it can >> >>>> >>>>>>>> be >> >>>> >>>>>>>> argued to decrease community involvement a bit. >> >>>> >>>>>>>> >> >>>> >>>>>>>> >> >>>> >>>>>>>> On Wed, Nov 30, 2011 at 5:11 PM, Patrick Hunt < >> ph...@apache.org> >> >>>> >>>>>>>> wrote: >> >>>> >>>>>>>> >> >>>> >>>>>>>>> Curator has been getting a lot of interest and was just >> >>>> officially >> >>>> >>>>>>>>> announced by Netflix: >> >>>> >>>>>>>>> >> >>>> >>>>>>>>> >> >>>> >>>>>>>>> >> >>>> >>>>>>>>> >> >>>> http://techblog.netflix.com/2011/11/introducing-curator-netflix-zooke >> >>>> >>>>>>>>> ep >> >>>> >>>>>>>>> er >> >>>> >>>>>>>>> .html >> >>>> >>>>>>>>> >> >>>> >>>>>>>>> I chatted briefly with @adrianco @chad_walters @randgalt >> (Jordan >> >>>> >>>>>>>>> Zimmerman) on twitter and the idea of moving Curator into >> >>>> >>>>>>>>> Apache/ZooKeeper came up: >> >>>> >>>>>>>>> https://twitter.com/#!/search/phunt%20curator >> >>>> >>>>>>>>> >> >>>> >>>>>>>>> I wanted to restart this thread as a way of discussing this >> idea >> >>>> in >> >>>> >>>>>>>>> more depth, and to gauge the interest of both communities. I >> >>>> >>>>>>>>> personally think this would be a great thing: both for >> users and >> >>>> >>>>>>>>> for >> >>>> >>>>>>>>> the development community itself. Thoughts? >> >>>> >>>>>>>>> >> >>>> >>>>>>>>> Re the mechanics. See my original post below. Seems to me >> we (ZK >> >>>> >>>>>>>>> PMC) >> >>>> >>>>>>>>> could sponsor Curator as an incubator project with the >> intent to >> >>>> >>>>>>>>> bring >> >>>> >>>>>>>>> it to ZK as either a subproject (it's own committer list) >> or as a >> >>>> >>>>>>>>> separate release artifact of the ZK TLP itself. Or just >> shortcut >> >>>> >>>>>>>>> the >> >>>> >>>>>>>>> incubator process altogether. My preference would be to go >> >>>> >>>>>>>>> incubator >> >>>> >>>>>>>>> first. At some point we would deprecate the existing ZK >> recipes >> >>>> >>>>>>>>> (ala >> >>>> >>>>>>>>> Curator had suitable replacements). >> >>>> >>>>>>>>> >> >>>> >>>>>>>>> Patrick >> >>>> >>>>>>>>> >> >>>> >>>>>>>>> On Fri, Jul 22, 2011 at 10:59 AM, Benjamin Reed < >> >>>> br...@apache.org> >> >>>> >>>>>>>>> wrote: >> >>>> >>>>>>>>>> >> >>>> >>>>>>>>>> while i agree with the sentiment of not fragmenting the >> >>>> zookeeper >> >>>> >>>>>>>>>> community and recipe committers also moving into core >> >>>> >>>>>>>>> >> >>>> >>>>>>>>> development, i >> >>>> >>>>>>>>>> >> >>>> >>>>>>>>>> also think it would be good if a strong community of >> developers >> >>>> >>>>>>>>> >> >>>> >>>>>>>>> grew >> >>>> >>>>>>>>>> >> >>>> >>>>>>>>>> up around zookeeper recipes. to do that they need a sense >> of >> >>>> >>>>>>>>> >> >>>> >>>>>>>>> identity, >> >>>> >>>>>>>>>> >> >>>> >>>>>>>>>> and i'm not sure if they can get that with this proposal. >> >>>> >>>>>>>>>> >> >>>> >>>>>>>>>> on the other hand the proposal does address all of the >> other >> >>>> >>>>>>>>> >> >>>> >>>>>>>>> issues >> >>>> >>>>>>>>> i >> >>>> >>>>>>>>>> >> >>>> >>>>>>>>>> have with recipe maintenance. the key is to grow a >> committer >> >>>> base. >> >>>> >>>>>>>>>> >> >>>> >>>>>>>>>> ben >> >>>> >>>>>>>>>> >> >>>> >>>>>>>>>> On Wed, Jul 13, 2011 at 11:04 AM, Patrick Hunt < >> >>>> ph...@apache.org> >> >>>> >>>>>>>>> >> >>>> >>>>>>>>> wrote: >> >>>> >>>>>>>>>>> >> >>>> >>>>>>>>>>> During the recent developer meetup we discussed how to >> more >> >>>> >>>>>>>>>>> effectively grow the recipes. >> >>>> >>>>>>>>>>> >> >>>> >>>>>>>>> >> >>>> >>>>>>>>> >> >>>> >>>>>>>>> >> >>>> https://cwiki.apache.org/confluence/display/ZOOKEEPER/June2011Develop >> >>>> >>>>>>>>> er >> >>>> >>>>>>>>> Me >> >>>> >>>>>>>>> eting >> >>>> >>>>>>>>>>> >> >>>> >>>>>>>>>>> >> >>>> >>>>>>>>>>> The concerns seemed to be around managing changes, >> community >> >>>> >>>>>>>>> >> >>>> >>>>>>>>> growth, >> >>>> >>>>>>>>>>> >> >>>> >>>>>>>>>>> and releases. We discussed a number of approaches: >> >>>> >>>>>>>>>>> * move recipes into apache-extras >> >>>> >>>>>>>>>>> * or github >> >>>> >>>>>>>>>>> * or incubator >> >>>> >>>>>>>>>>> * or sub-project >> >>>> >>>>>>>>>>> >> >>>> >>>>>>>>>>> each of these has it's good and bad points. One >> alternative >> >>>> that >> >>>> >>>>>>>>> >> >>>> >>>>>>>>> I >> >>>> >>>>>>>>> put >> >>>> >>>>>>>>>>> >> >>>> >>>>>>>>>>> forth was to create a separate release artifact for >> recipes as >> >>>> >>>>>>>>> >> >>>> >>>>>>>>> part >> >>>> >>>>>>>>> of >> >>>> >>>>>>>>>>> >> >>>> >>>>>>>>>>> the ZooKeeper project itself. What would this look like? >> >>>> >>>>>>>>>>> >> >>>> >>>>>>>>>>> * zookeeper/trunk/src/recipes would move to >> >>>> >>>>>>>>>>> zookeeper/recipes/{trunk,branches,tags,...} >> >>>> >>>>>>>>>>> * there would be distinct/separate releases of zookeeper >> and >> >>>> >>>>>>>>>>> zookeeper-recipes artifacts >> >>>> >>>>>>>>>>> ** these releases would not necessarily be synchronized, >> >>>> although >> >>>> >>>>>>>>> >> >>>> >>>>>>>>> we'd >> >>>> >>>>>>>>>>> >> >>>> >>>>>>>>>>> want to ensure that particular versions of each release >> work >> >>>> >>>>>>>>> >> >>>> >>>>>>>>> together >> >>>> >>>>>>>>>>> >> >>>> >>>>>>>>>>> ** these releases would go through our standard release >> >>>> process - >> >>>> >>>>>>>>> >> >>>> >>>>>>>>> ie >> >>>> >>>>>>>>>>> >> >>>> >>>>>>>>>>> creation of release candidate by a committer and approval >> by >> >>>> the >> >>>> >>>>>>>>> >> >>>> >>>>>>>>> PMC >> >>>> >>>>>>>>>>> >> >>>> >>>>>>>>>>> ** recipes would no longer be included in "zookeeper" >> (core) >> >>>> >>>>>>>>> >> >>>> >>>>>>>>> releases >> >>>> >>>>>>>>>>> >> >>>> >>>>>>>>>>> * continue to use the ZOOKEEPER jira, with "recipes" >> component >> >>>> >>>>>>>>>>> * space in the web and wiki sites for recipes specific >> pages >> >>>> >>>>>>>>>>> * recipes continue to use the std zookeeper mailing lists >> >>>> >>>>>>>>>>> * commits to recipes would continue to use our regular >> commit >> >>>> >>>>>>>>> >> >>>> >>>>>>>>> process >> >>>> >>>>>>>>>>> >> >>>> >>>>>>>>>>> (jira, rtc, etc...) >> >>>> >>>>>>>>>>> * we would accept new "ZooKeeper committers" with the >> intent >> >>>> that >> >>>> >>>>>>>>> >> >>>> >>>>>>>>> they >> >>>> >>>>>>>>>>> >> >>>> >>>>>>>>>>> focus on their area of expertise, whether this be core >> >>>> (zookeeper >> >>>> >>>>>>>>>>> trunk) or recipes. >> >>>> >>>>>>>>>>> >> >>>> >>>>>>>>>>> This last item is the more interesting I think, the other >> >>>> issues >> >>>> >>>>>>>>> >> >>>> >>>>>>>>> are >> >>>> >>>>>>>>>>> >> >>>> >>>>>>>>>>> all pretty straightforward. Although moving to sub (or out >> >>>> >>>>>>>>> >> >>>> >>>>>>>>> entirely) >> >>>> >>>>>>>>>>> >> >>>> >>>>>>>>>>> would allow us to have a separate commit list it would >> fragment >> >>>> >>>>>>>>> >> >>>> >>>>>>>>> the >> >>>> >>>>>>>>>>> >> >>>> >>>>>>>>>>> ZooKeeper community. I also believe that over time >> committers >> >>>> to >> >>>> >>>>>>>>>>> recipes would more likely get experience with core and >> start >> >>>> >>>>>>>>>>> contributing there as well. The only issue really, if you >> want >> >>>> to >> >>>> >>>>>>>>> >> >>>> >>>>>>>>> call >> >>>> >>>>>>>>>>> >> >>>> >>>>>>>>>>> it that, is that we'd have committers with "guardrails" >> >>>> implied. >> >>>> >>>>>>>>> >> >>>> >>>>>>>>> ie >> >>>> >>>>>>>>>>> >> >>>> >>>>>>>>>>> someone with more expertise in recipes than core. However >> there >> >>>> >>>>>>>>> >> >>>> >>>>>>>>> are >> >>>> >>>>>>>>>>> >> >>>> >>>>>>>>>>> many other Apache projects that take this approach, and >> do it >> >>>> >>>>>>>>>>> successfully. Regardless RTC allows for review both >> before and >> >>>> >>>>>>>>> >> >>>> >>>>>>>>> after >> >>>> >>>>>>>>> a >> >>>> >>>>>>>>>>> >> >>>> >>>>>>>>>>> change is committed. >> >>>> >>>>>>>>>>> >> >>>> >>>>>>>>>>> What do you think? >> >>>> >>>>>>>>>>> >> >>>> >>>>>>>>>>> Patrick >> >>>> >>>>>>>>>>> >> >>>> >>>>>>>>> >> >>>> >>>>>>> >> >>>> >>>>>> >> >>>> >>>>> >> >>>> >>>> >> >>>> >>> >> >>>> > >> >>>> > flavio >> >>>> > junqueira >> >>>> > >> >>>> > research scientist >> >>>> > >> >>>> > f...@yahoo-inc.com >> >>>> > direct +34 93-183-8828 >> >>>> > >> >>>> > avinguda diagonal 177, 8th floor, barcelona, 08018, es >> >>>> > phone (408) 349 3300 fax (408) 349 3301 >> >>>> > >> >>>> >>
