Ted, can you talk more about this? How do you see this working? FYI - I had a nice lunch with Patrick today. Personally (and Netflixily) I'm not very interested in Incubator. The recipes are useless without ZooKeeper itself so I can't imagine recipes ever becoming a TLP. Patrick and I talked about going the sub-project route (though he prefers Incubator). My desires are:
* Get widespread use/visibility for Curator * Get additional committers to Curator * Help ZK users by making quality recipe implementations readily available -JZ On 12/5/11 12:36 PM, "Ted Dunning" <ted.dunn...@gmail.com> wrote: >-1 > >I don't think that a full incubator process is needed or even useful here. > Incubator is ONE process in Apache process for this, but a contrib >artifact is another way to do it. > >This isn't a sub-project. This is additional ZK software. It just isn't >the core stuff. This is more akin to Solr being added to Lucene than it >is like Mahout spinning out of Lucene ultimately as a top level project. > > >On Mon, Dec 5, 2011 at 11:58 AM, Mahadev Konar ><maha...@hortonworks.com>wrote: > >> +1... >> >> mahadev >> >> On Mon, Dec 5, 2011 at 11:42 AM, Benjamin Reed <br...@apache.org> wrote: >> > +1 >> > >> > On Mon, Dec 5, 2011 at 11:11 AM, Patrick Hunt <ph...@apache.org> >>wrote: >> >> Incubator is the Apache recommended way to handle this. We could be >> >> the sponsor, with the intent that once Curator graduates from the >> >> incubator we would accept it >> >> >> >>http://incubator.apache.org/incubation/Roles_and_Responsibilities.html#Sp >>onsor >> >> anyone could join the curator effort, including existing ZK >> >> committers/contributors. >> >> >> >> if curator felt it should be a tlp (at time of graduation) I think >> >> that would be fine too. >> >> >> >> I would be happy to Champion/Mentor this effort. >> >> >> >> Patrick >> >> >> >> On Mon, Dec 5, 2011 at 11:06 AM, Ted Dunning <ted.dunn...@gmail.com> >> wrote: >> >>> Ben, >> >>> >> >>> I think that you are right on target with this. We need to have a >> wider >> >>> community of developers in order to have the bandwidth to handle >> recipes >> >>> and the recipes really do need separate release cycles. Independent >> >>> releases should be not much of an issue given how compatible ZK >> versions >> >>> tend to be. >> >>> >> >>> On Mon, Dec 5, 2011 at 10:59 AM, Benjamin Reed <br...@apache.org> >> wrote: >> >>> >> >>>> i also agreed that it would be great to build a community around >> >>>> curator especially if it could pull in other zk recipes. i've felt >>for >> >>>> a while that the core developers don't have the bandwidth to really >> >>>> keep track of recipe development and recipes are a very important >> >>>> aspect of zk. so, having a self sustaining community of developers >> >>>> focused on that would be awesome. being able to have a separate >> >>>> release cycle is also important i think. (i may be reading too much >> >>>> into the proposal though :) >> >>>> >> >>>> ben >> >>>> >> >>>> On Mon, Dec 5, 2011 at 7:14 AM, Flavio Junqueira >><f...@yahoo-inc.com> >> >>>> wrote: >> >>>> > I suppose you're just trying to sense if others think that it >>would >> be a >> >>>> > good idea to have it as a subproject. If so, I'm in favor of >> putting a >> >>>> > proposal together, +1. >> >>>> > >> >>>> > -Flavio >> >>>> > >> >>>> > On Dec 2, 2011, at 8:45 PM, Patrick Hunt wrote: >> >>>> > >> >>>> >> Somehow this got off list, Jordan and I just noticed, summary so >> far: >> >>>> >> >> >>>> >> ---------- Forwarded message ---------- >> >>>> >> From: Patrick Hunt <ph...@apache.org> >> >>>> >> Date: Fri, Dec 2, 2011 at 9:56 AM >> >>>> >> Subject: Re: Proposal: create a separate ZooKeeper release >> artifact for >> >>>> >> recipes >> >>>> >> To: Jordan Zimmerman <jzimmer...@netflix.com> >> >>>> >> >> >>>> >> >> >>>> >> Did you mean to reply just to me? I think this is a great idea >>btw, >> >>>> >> just need to work out the mechanics of getting it integrated and >> get >> >>>> >> the others on board. Would it help to talk f2f? >> >>>> >> >> >>>> >> On Thu, Dec 1, 2011 at 6:20 PM, Jordan Zimmerman < >> >>>> jzimmer...@netflix.com> >> >>>> >> wrote: >> >>>> >>> >> >>>> >>> From talking to folks here (and my own feeling), I'd like to >>have >> >>>> Curator >> >>>> >>> get as wide use as possible. Some sort of sanction from the >> ZooKeeper >> >>>> >>> team >> >>>> >>> would do that. I don't see a lot of downside to getting a >> community >> >>>> >>> around >> >>>> >>> it either. Sure, it's nice to be the only guy I need to please, >> but the >> >>>> >>> benefits of a community outweigh that. >> >>>> >>> >> >>>> >>> I think the beauty of folding it in to ZooKeeper is that the >> structure >> >>>> is >> >>>> >>> already there: Jira, website, etc. I (and probably one other >>from >> >>>> >>> Netflix) >> >>>> >>> could take on the role of managing the Curator portion so that >>it >> >>>> doesn't >> >>>> >>> burden you folks. I'm doing that internally at Netflix anyway. >> Frankly, >> >>>> >>> it >> >>>> >>> sounds like fun (famous last words). >> >>>> >>> >> >>>> >>> -Jordan >> >>>> >>> >> >>>> >>> On 12/1/11 5:36 PM, "Patrick Hunt" <ph...@apache.org> wrote: >> >>>> >>> >> >>>> >>>> On Thu, Dec 1, 2011 at 2:05 PM, Jordan Zimmerman >> >>>> >>>> <jzimmer...@netflix.com> >> >>>> >>>> wrote: >> >>>> >>>>> >> >>>> >>>>> The feeling here is that it would be nice to push Curator >>into >> ZK. >> >>>> >>>>> However, we'd still like to contribute to it. If you don't >>want >> >>>> >>>>> separate >> >>>> >>>>> contributor lists, I could be a ZK contributor but explicitly >> only >> >>>> work >> >>>> >>>>> on >> >>>> >>>>> Curator. Of course, we're open to your suggestions on this as >> well. >> >>>> >>>> >> >>>> >>>> >> >>>> >>>> Don't get me wrong, I don't mind having one list of >> >>>> >>>> contributors/committers for ZK/Curator, I'm personally fine >>with >> >>>> >>>> having one. If you notice in my original proposal I lean >>towards >> that >> >>>> >>>> direction (single set of committers with multiple artifacts) >> >>>> >>>> >> >>>> >>>> I see a benefit to having something like Curator available to >> users, >> >>>> >>>> open sourcing it on GH accomplishes this. Bringing something >>to >> Apache >> >>>> >>>> means building a community around it though, not just making >>the >> code >> >>>> >>>> open source. This is analogous to when ZK itself was on >> sourceforge, >> >>>> >>>> eventually moving to Apache. >> >>>> >>>> >> >>>> >>>> Here's the issue: building community adds overhead, which you >> might >> >>>> >>>> not be willing to sign up for. I'm not sure we (zk community) >> can take >> >>>> >>>> on this additional cost alone. Hence my question -- I >>_wouldn't_ >> want >> >>>> >>>> to you leave, that's the point. We'd need some set of core >> "Curator >> >>>> >>>> contributors" who would continue to work on Curator, answer >> questions, >> >>>> >>>> shepherd new contributor/committers/etc... If that doesn't >> happen the >> >>>> >>>> code will get imported, see some use, but never really reach >> it's full >> >>>> >>>> potential. >> >>>> >>>> >> >>>> >>>> Hope that helps. LMK. >> >>>> >>>> >> >>>> >>>> Patrick >> >>>> >>>> >> >>>> >>>>> >> >>>> >>>>> On 12/1/11 2:01 PM, "Patrick Hunt" <ph...@apache.org> wrote: >> >>>> >>>>> >> >>>> >>>>>> On Thu, Dec 1, 2011 at 1:26 PM, Jordan Zimmerman >> >>>> >>>>>> <jzimmer...@netflix.com> >> >>>> >>>>>> wrote: >> >>>> >>>>>>> >> >>>> >>>>>>> I've spoken about this with the appropriate folks here at >> Netflix >> >>>> and >> >>>> >>>>>>> we're interested in pursuing this. What are the next steps? >> >>>> >>>>>> >> >>>> >>>>>> >> >>>> >>>>>> Hi Jordan, what are your intentions. Do you want to build a >> >>>> community >> >>>> >>>>>> around "Apache Curator" or are you just looking to push the >> source >> >>>> >>>>>> into ZK and move on to other things? >> >>>> >>>>>> >> >>>> >>>>>> Patrick >> >>>> >>>>>> >> >>>> >>>>>> >> >>>> >>>>>>> >> >>>> >>>>>>> >> >>>> >>>>>>> On 11/30/11 5:42 PM, "Ted Dunning" <ted.dunn...@gmail.com> >> wrote: >> >>>> >>>>>>> >> >>>> >>>>>>>> Separator committer lists are generally frowned on by >>Apache >> (ref >> >>>> >>>>>>>> Lucene/Solr and also Mahout). >> >>>> >>>>>>>> >> >>>> >>>>>>>> They also are essentially unnecessary. It isn't that big >>a >> deal >> >>>> to >> >>>> >>>>>>>> have a >> >>>> >>>>>>>> single committer list and depend on reversion to enforce >> community >> >>>> >>>>>>>> standards. Core ZK is review-then-commit and should >> continue to >> >>>> be. >> >>>> >>>>>>>> Add-on modules that are relatively independent of the core >> can >> >>>> quite >> >>>> >>>>>>>> reasonably be commit-then-review or whatever level of >>care is >> >>>> >>>>>>>> implied >> >>>> >>>>>>>> by >> >>>> >>>>>>>> module's content. The review can include a determination >>of >> who >> >>>> >>>>>>>> should >> >>>> >>>>>>>> actually do the commit. >> >>>> >>>>>>>> >> >>>> >>>>>>>> Moreover, projects can easily have more than one artifact. >> Mahout >> >>>> >>>>>>>> has >> >>>> >>>>>>>> three independent artifacts (mahout-math, >>mahout-collections >> and >> >>>> the >> >>>> >>>>>>>> core >> >>>> >>>>>>>> stuff with examples and integration code). That works >>well. >> >>>> There >> >>>> >>>>>>>> are >> >>>> >>>>>>>> different sets of committers who are typically the ones >>who >> work >> >>>> on >> >>>> >>>>>>>> different components. The overall level of diligence >> required in >> >>>> >>>>>>>> Mahout >> >>>> >>>>>>>> is >> >>>> >>>>>>>> much lower than for Zookeeper as you might expect, but the >> >>>> principle >> >>>> >>>>>>>> that >> >>>> >>>>>>>> multiple artifacts are fine is still the same. Likewise, >> the idea >> >>>> >>>>>>>> of >> >>>> >>>>>>>> committers with specialized expertise also applies here. >> >>>> >>>>>>>> >> >>>> >>>>>>>> There is a tradition in Zookeeper of having a very high >>bar >> for >> >>>> >>>>>>>> committership, but experience in some other projects >> indicates >> >>>> that >> >>>> >>>>>>>> this >> >>>> >>>>>>>> doesn't actually help quality or security all that much >>and >> it can >> >>>> >>>>>>>> be >> >>>> >>>>>>>> argued to decrease community involvement a bit. >> >>>> >>>>>>>> >> >>>> >>>>>>>> >> >>>> >>>>>>>> On Wed, Nov 30, 2011 at 5:11 PM, Patrick Hunt < >> ph...@apache.org> >> >>>> >>>>>>>> wrote: >> >>>> >>>>>>>> >> >>>> >>>>>>>>> Curator has been getting a lot of interest and was just >> >>>> officially >> >>>> >>>>>>>>> announced by Netflix: >> >>>> >>>>>>>>> >> >>>> >>>>>>>>> >> >>>> >>>>>>>>> >> >>>> >>>>>>>>> >> >>>> >>http://techblog.netflix.com/2011/11/introducing-curator-netflix-zooke >> >>>> >>>>>>>>> ep >> >>>> >>>>>>>>> er >> >>>> >>>>>>>>> .html >> >>>> >>>>>>>>> >> >>>> >>>>>>>>> I chatted briefly with @adrianco @chad_walters @randgalt >> (Jordan >> >>>> >>>>>>>>> Zimmerman) on twitter and the idea of moving Curator into >> >>>> >>>>>>>>> Apache/ZooKeeper came up: >> >>>> >>>>>>>>> https://twitter.com/#!/search/phunt%20curator >> >>>> >>>>>>>>> >> >>>> >>>>>>>>> I wanted to restart this thread as a way of discussing >>this >> idea >> >>>> in >> >>>> >>>>>>>>> more depth, and to gauge the interest of both >>communities. I >> >>>> >>>>>>>>> personally think this would be a great thing: both for >> users and >> >>>> >>>>>>>>> for >> >>>> >>>>>>>>> the development community itself. Thoughts? >> >>>> >>>>>>>>> >> >>>> >>>>>>>>> Re the mechanics. See my original post below. Seems to me >> we (ZK >> >>>> >>>>>>>>> PMC) >> >>>> >>>>>>>>> could sponsor Curator as an incubator project with the >> intent to >> >>>> >>>>>>>>> bring >> >>>> >>>>>>>>> it to ZK as either a subproject (it's own committer list) >> or as a >> >>>> >>>>>>>>> separate release artifact of the ZK TLP itself. Or just >> shortcut >> >>>> >>>>>>>>> the >> >>>> >>>>>>>>> incubator process altogether. My preference would be to >>go >> >>>> >>>>>>>>> incubator >> >>>> >>>>>>>>> first. At some point we would deprecate the existing ZK >> recipes >> >>>> >>>>>>>>> (ala >> >>>> >>>>>>>>> Curator had suitable replacements). >> >>>> >>>>>>>>> >> >>>> >>>>>>>>> Patrick >> >>>> >>>>>>>>> >> >>>> >>>>>>>>> On Fri, Jul 22, 2011 at 10:59 AM, Benjamin Reed < >> >>>> br...@apache.org> >> >>>> >>>>>>>>> wrote: >> >>>> >>>>>>>>>> >> >>>> >>>>>>>>>> while i agree with the sentiment of not fragmenting the >> >>>> zookeeper >> >>>> >>>>>>>>>> community and recipe committers also moving into core >> >>>> >>>>>>>>> >> >>>> >>>>>>>>> development, i >> >>>> >>>>>>>>>> >> >>>> >>>>>>>>>> also think it would be good if a strong community of >> developers >> >>>> >>>>>>>>> >> >>>> >>>>>>>>> grew >> >>>> >>>>>>>>>> >> >>>> >>>>>>>>>> up around zookeeper recipes. to do that they need a >>sense >> of >> >>>> >>>>>>>>> >> >>>> >>>>>>>>> identity, >> >>>> >>>>>>>>>> >> >>>> >>>>>>>>>> and i'm not sure if they can get that with this >>proposal. >> >>>> >>>>>>>>>> >> >>>> >>>>>>>>>> on the other hand the proposal does address all of the >> other >> >>>> >>>>>>>>> >> >>>> >>>>>>>>> issues >> >>>> >>>>>>>>> i >> >>>> >>>>>>>>>> >> >>>> >>>>>>>>>> have with recipe maintenance. the key is to grow a >> committer >> >>>> base. >> >>>> >>>>>>>>>> >> >>>> >>>>>>>>>> ben >> >>>> >>>>>>>>>> >> >>>> >>>>>>>>>> On Wed, Jul 13, 2011 at 11:04 AM, Patrick Hunt < >> >>>> ph...@apache.org> >> >>>> >>>>>>>>> >> >>>> >>>>>>>>> wrote: >> >>>> >>>>>>>>>>> >> >>>> >>>>>>>>>>> During the recent developer meetup we discussed how to >> more >> >>>> >>>>>>>>>>> effectively grow the recipes. >> >>>> >>>>>>>>>>> >> >>>> >>>>>>>>> >> >>>> >>>>>>>>> >> >>>> >>>>>>>>> >> >>>> >>https://cwiki.apache.org/confluence/display/ZOOKEEPER/June2011Develop >> >>>> >>>>>>>>> er >> >>>> >>>>>>>>> Me >> >>>> >>>>>>>>> eting >> >>>> >>>>>>>>>>> >> >>>> >>>>>>>>>>> >> >>>> >>>>>>>>>>> The concerns seemed to be around managing changes, >> community >> >>>> >>>>>>>>> >> >>>> >>>>>>>>> growth, >> >>>> >>>>>>>>>>> >> >>>> >>>>>>>>>>> and releases. We discussed a number of approaches: >> >>>> >>>>>>>>>>> * move recipes into apache-extras >> >>>> >>>>>>>>>>> * or github >> >>>> >>>>>>>>>>> * or incubator >> >>>> >>>>>>>>>>> * or sub-project >> >>>> >>>>>>>>>>> >> >>>> >>>>>>>>>>> each of these has it's good and bad points. One >> alternative >> >>>> that >> >>>> >>>>>>>>> >> >>>> >>>>>>>>> I >> >>>> >>>>>>>>> put >> >>>> >>>>>>>>>>> >> >>>> >>>>>>>>>>> forth was to create a separate release artifact for >> recipes as >> >>>> >>>>>>>>> >> >>>> >>>>>>>>> part >> >>>> >>>>>>>>> of >> >>>> >>>>>>>>>>> >> >>>> >>>>>>>>>>> the ZooKeeper project itself. What would this look >>like? >> >>>> >>>>>>>>>>> >> >>>> >>>>>>>>>>> * zookeeper/trunk/src/recipes would move to >> >>>> >>>>>>>>>>> zookeeper/recipes/{trunk,branches,tags,...} >> >>>> >>>>>>>>>>> * there would be distinct/separate releases of >>zookeeper >> and >> >>>> >>>>>>>>>>> zookeeper-recipes artifacts >> >>>> >>>>>>>>>>> ** these releases would not necessarily be >>synchronized, >> >>>> although >> >>>> >>>>>>>>> >> >>>> >>>>>>>>> we'd >> >>>> >>>>>>>>>>> >> >>>> >>>>>>>>>>> want to ensure that particular versions of each release >> work >> >>>> >>>>>>>>> >> >>>> >>>>>>>>> together >> >>>> >>>>>>>>>>> >> >>>> >>>>>>>>>>> ** these releases would go through our standard release >> >>>> process - >> >>>> >>>>>>>>> >> >>>> >>>>>>>>> ie >> >>>> >>>>>>>>>>> >> >>>> >>>>>>>>>>> creation of release candidate by a committer and >>approval >> by >> >>>> the >> >>>> >>>>>>>>> >> >>>> >>>>>>>>> PMC >> >>>> >>>>>>>>>>> >> >>>> >>>>>>>>>>> ** recipes would no longer be included in "zookeeper" >> (core) >> >>>> >>>>>>>>> >> >>>> >>>>>>>>> releases >> >>>> >>>>>>>>>>> >> >>>> >>>>>>>>>>> * continue to use the ZOOKEEPER jira, with "recipes" >> component >> >>>> >>>>>>>>>>> * space in the web and wiki sites for recipes specific >> pages >> >>>> >>>>>>>>>>> * recipes continue to use the std zookeeper mailing >>lists >> >>>> >>>>>>>>>>> * commits to recipes would continue to use our regular >> commit >> >>>> >>>>>>>>> >> >>>> >>>>>>>>> process >> >>>> >>>>>>>>>>> >> >>>> >>>>>>>>>>> (jira, rtc, etc...) >> >>>> >>>>>>>>>>> * we would accept new "ZooKeeper committers" with the >> intent >> >>>> that >> >>>> >>>>>>>>> >> >>>> >>>>>>>>> they >> >>>> >>>>>>>>>>> >> >>>> >>>>>>>>>>> focus on their area of expertise, whether this be core >> >>>> (zookeeper >> >>>> >>>>>>>>>>> trunk) or recipes. >> >>>> >>>>>>>>>>> >> >>>> >>>>>>>>>>> This last item is the more interesting I think, the >>other >> >>>> issues >> >>>> >>>>>>>>> >> >>>> >>>>>>>>> are >> >>>> >>>>>>>>>>> >> >>>> >>>>>>>>>>> all pretty straightforward. Although moving to sub (or >>out >> >>>> >>>>>>>>> >> >>>> >>>>>>>>> entirely) >> >>>> >>>>>>>>>>> >> >>>> >>>>>>>>>>> would allow us to have a separate commit list it would >> fragment >> >>>> >>>>>>>>> >> >>>> >>>>>>>>> the >> >>>> >>>>>>>>>>> >> >>>> >>>>>>>>>>> ZooKeeper community. I also believe that over time >> committers >> >>>> to >> >>>> >>>>>>>>>>> recipes would more likely get experience with core and >> start >> >>>> >>>>>>>>>>> contributing there as well. The only issue really, if >>you >> want >> >>>> to >> >>>> >>>>>>>>> >> >>>> >>>>>>>>> call >> >>>> >>>>>>>>>>> >> >>>> >>>>>>>>>>> it that, is that we'd have committers with "guardrails" >> >>>> implied. >> >>>> >>>>>>>>> >> >>>> >>>>>>>>> ie >> >>>> >>>>>>>>>>> >> >>>> >>>>>>>>>>> someone with more expertise in recipes than core. >>However >> there >> >>>> >>>>>>>>> >> >>>> >>>>>>>>> are >> >>>> >>>>>>>>>>> >> >>>> >>>>>>>>>>> many other Apache projects that take this approach, and >> do it >> >>>> >>>>>>>>>>> successfully. Regardless RTC allows for review both >> before and >> >>>> >>>>>>>>> >> >>>> >>>>>>>>> after >> >>>> >>>>>>>>> a >> >>>> >>>>>>>>>>> >> >>>> >>>>>>>>>>> change is committed. >> >>>> >>>>>>>>>>> >> >>>> >>>>>>>>>>> What do you think? >> >>>> >>>>>>>>>>> >> >>>> >>>>>>>>>>> Patrick >> >>>> >>>>>>>>>>> >> >>>> >>>>>>>>> >> >>>> >>>>>>> >> >>>> >>>>>> >> >>>> >>>>> >> >>>> >>>> >> >>>> >>> >> >>>> > >> >>>> > flavio >> >>>> > junqueira >> >>>> > >> >>>> > research scientist >> >>>> > >> >>>> > f...@yahoo-inc.com >> >>>> > direct +34 93-183-8828 >> >>>> > >> >>>> > avinguda diagonal 177, 8th floor, barcelona, 08018, es >> >>>> > phone (408) 349 3300 fax (408) 349 3301 >> >>>> > >> >>>> >>
