Might I suggest that this sounds like good info to document on the wiki for committers getting started. I'd add it but I'm not in the allow-list. :)
</JamesM> On 1/17/14 4:00 AM, "Garren Smith" <gar...@apache.org> wrote: >I'm claiming 2nd person added! > >On 17 Jan 2014, at 1:28 PM, Noah Slater <nsla...@apache.org> wrote: > >> Psst. A little birdy tells me that if you ask nicely, the infra folks >> will add you to the Apache GitHub org too, so you can show off your >> Apache affiliation. I was the first person added. Because I may have >> been the first to ask. ;) >> >> On 17 January 2014 11:56, Noah Slater <nsla...@apache.org> wrote: >>> Awesome, thanks Paul. >>> >>> Note to all devs: if you want your contributions to CouchDB to show up >>> on your GitHub profile, you have to star each of the repositories. >>> (That's just how GitHub mechanics work for repo mirrors.) >>> >>> You can find them all here: >>> >>> https://github.com/apache >>> >>> On 17 January 2014 00:00, Paul Davis <paul.joseph.da...@gmail.com> >>>wrote: >>>> New repos are up: https://git-wip-us.apache.org/repos/asf?s=couchdb >>>> >>>> I'm gonna go through and initialize them with history from master or >>>> one of the bigcouch and rcouch branches as appropriate. >>>> >>>> On Thu, Jan 16, 2014 at 2:12 PM, Paul Davis >>>><paul.joseph.da...@gmail.com> wrote: >>>>> Infrastructure ticket opened: >>>>>https://issues.apache.org/jira/browse/INFRA-7203 >>>>> >>>>> On Thu, Jan 16, 2014 at 1:42 PM, Jan Lehnardt <j...@apache.org> wrote: >>>>>> >>>>>> On 16 Jan 2014, at 20:42 , Paul Davis <paul.joseph.da...@gmail.com> >>>>>>wrote: >>>>>> >>>>>>> It doesn't appear that this is objectionable to anyone. Does anyone >>>>>>> have an objection to us having infra/me create these repos to use >>>>>>>for >>>>>>> the bigcouch/rcouch merge work? This won't affect master or >>>>>>>releases >>>>>>> until those merges finish. >>>>>> >>>>>> no objections. >>>>>> >>>>>> Jan >>>>>> -- >>>>>> >>>>>>> >>>>>>> On Tue, Jan 14, 2014 at 11:02 PM, Paul J Davis >>>>>>> <paul.joseph.da...@gmail.com> wrote: >>>>>>>> >>>>>>>> >>>>>>>>> On Jan 14, 2014, at 8:37 PM, Benoit Chesneau >>>>>>>>><bchesn...@gmail.com> wrote: >>>>>>>>> >>>>>>>>> On Wed, Jan 15, 2014 at 12:22 AM, Paul Davis >>>>>>>>><paul.joseph.da...@gmail.com>wrote: >>>>>>>>> >>>>>>>>>> I've recently been having discussions about how to handle the >>>>>>>>>> repository configuration for various bits of CouchDB >>>>>>>>>>post-merge. The >>>>>>>>>> work that Benoit has been doing on the rcouch merge branch have >>>>>>>>>>also >>>>>>>>>> touched on this topic as well. >>>>>>>>>> >>>>>>>>>> The background for those unfamiliar is that the standard >>>>>>>>>>operating >>>>>>>>>> procedure for Erlang is to have a single Erlang application per >>>>>>>>>> repository and then rely on rebar to fetch each dependency. >>>>>>>>>> Traditionally in CouchDB land we've always just included the >>>>>>>>>>source to >>>>>>>>>> all applications in a single monolithic repository and >>>>>>>>>>periodically >>>>>>>>>> reimport changes from upstream dependencies. >>>>>>>>>> >>>>>>>>>> Recently rcouch changed from the monolithic repository to use >>>>>>>>>>external >>>>>>>>>> repositories for some dependencies. Originally the BigCouch >>>>>>>>>>used an >>>>>>>>>> even more federated scheme that had each Erlang application in >>>>>>>>>>an >>>>>>>>>> external repository (and the core couch Erlang application was >>>>>>>>>>in the >>>>>>>>>> root repository). When Bob Newson and I did the initial hacking >>>>>>>>>>on the >>>>>>>>>> BigCouch merge we pulled those external dependencies into the >>>>>>>>>>root >>>>>>>>>> repository reverting back to the large monolithic approach. >>>>>>>>>> >>>>>>>>>> After trying to deal with the merge and contemplating how >>>>>>>>>>various >>>>>>>>>> Erlang release things might work it's become fairly apparent >>>>>>>>>>that the >>>>>>>>>> monolithic approach is a bit constrictive. For instance, part of >>>>>>>>>> rebar's versioning abilities lets you tag repositories to >>>>>>>>>>generate >>>>>>>>>> versions rather than manually updating versions in source files. >>>>>>>>>> Another thing I've found on other projects is that having each >>>>>>>>>> application in a separate repository requires developers to >>>>>>>>>>think a >>>>>>>>>> bit more detailed about the public internal interfaces used >>>>>>>>>>through >>>>>>>>>> out the system. We've done some work to this extent already with >>>>>>>>>> separating source directories but forcing commits to multiple >>>>>>>>>> repositories shoots up a big red flag that maybe there's a high >>>>>>>>>>level >>>>>>>>>> of coupling between two bits of code. >>>>>>>>>> >>>>>>>>>> Other benefits of having the multiple repository setup is that >>>>>>>>>>its >>>>>>>>>> possible that this lends itself to being integrated with the >>>>>>>>>>proposed >>>>>>>>>> plugin system. It'd be fairly trivial to have a script that >>>>>>>>>>went and >>>>>>>>>> fetched plugins that aren't developed at Apache (as a >>>>>>>>>>./configure time >>>>>>>>>> switch type of thing). Having a system like this would also >>>>>>>>>>allow us >>>>>>>>>> to have groups focused on particular bits of development not >>>>>>>>>>have to >>>>>>>>>> concern themselves with the unrelated parts of the system. >>>>>>>>>> >>>>>>>>>> Given all that, I'd like to propose that we move to having a >>>>>>>>>> repository for each application/dependency that we use to build >>>>>>>>>> CouchDB. Each repository would be hosted on ASF infra and >>>>>>>>>>mirrored to >>>>>>>>>> GitHub as expected. This means that we could have the root >>>>>>>>>>repository >>>>>>>>>> be a simple repo that contains packaging/release/build stuff >>>>>>>>>>that >>>>>>>>>> would enable lots of the ideas offered on configurable types of >>>>>>>>>> release generation. I've included an initial list of >>>>>>>>>>repositories at >>>>>>>>>> the end of this email. Its basically just the apps that have >>>>>>>>>>been >>>>>>>>>> split out in either rcouch or bigcouch plus a few other bits >>>>>>>>>>from >>>>>>>>>> CouchDB master. >>>>>>>>>> >>>>>>>>>> I would also point out that even though our main repo would >>>>>>>>>>need to >>>>>>>>>> fetch other dependencies from the internet to build the final >>>>>>>>>>output, >>>>>>>>>> we fully intend that our release tarballs would *not* have this >>>>>>>>>> requirement. Ie, when we go to cut a release part of the >>>>>>>>>>process the >>>>>>>>>> RM would run would be to pull all of those dependencies before >>>>>>>>>> creating a tarball that would be wholly self contained. Given an >>>>>>>>>> apache-couchdb-x.y.z.tar.gz release file, there won't be a >>>>>>>>>>requirement >>>>>>>>>> to have access to the ASF git repos. >>>>>>>>>> >>>>>>>>>> I'm not entirely sure how controversial this is for anyone. For >>>>>>>>>>the >>>>>>>>>> most part the reactions I remember hearing were more concerned >>>>>>>>>>on >>>>>>>>>> whether the infrastructure team would allow us to use this sort >>>>>>>>>>of >>>>>>>>>> configuration. I looked yesterday and asked and apparently its >>>>>>>>>> something we can request but as always we'll want to verify >>>>>>>>>>again if >>>>>>>>>> we have consensus to move in this direction. >>>>>>>>>> >>>>>>>>>> Anyone have comments or flames? Right now I'm just interested in >>>>>>>>>> feeling out what sort of (lack of?) consensus there is on such a >>>>>>>>>> change. If there's general consensus I'd think we'd do a vote >>>>>>>>>>in a >>>>>>>>>> couple weeks and if that passes then start on down this road >>>>>>>>>>for the >>>>>>>>>> two merge projects and then it would become part of master once >>>>>>>>>>those >>>>>>>>>> land (as opposed to doing this to master and then attempting to >>>>>>>>>>merge >>>>>>>>>> rcouch/bigcouch onto that somehow). >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> This is a quick pass at listing what extra repositories I'd have >>>>>>>>>> created. Some of these applications only exist in the bigcouch >>>>>>>>>>and/or >>>>>>>>>> rcouch branches so that's where the unfamiliar application >>>>>>>>>>names are >>>>>>>>>> from. I'd also point out that the documentation and fauxton >>>>>>>>>>things are >>>>>>>>>> just on a whim in that we could decouple that development from >>>>>>>>>>the >>>>>>>>>> erlang development. I can see arguments for an against those. >>>>>>>>>>I'm much >>>>>>>>>> less concerned on that aspect than the Erlang parts that are >>>>>>>>>>directly >>>>>>>>>> affected by rebar/Erlang conventions. >>>>>>>>>> >>>>>>>>>> chttpd >>>>>>>>>> config >>>>>>>>>> couch >>>>>>>>>> couch_collate >>>>>>>>>> couch_dbupdates >>>>>>>>>> couch_httpd >>>>>>>>>> couch_index >>>>>>>>>> couch_mrview >>>>>>>>>> couch_plugins >>>>>>>>>> couch_replicator >>>>>>>>>> documentation >>>>>>>>>> ddoc_cache >>>>>>>>>> ets_lru >>>>>>>>>> fabric >>>>>>>>>> fauxton >>>>>>>>>> ibrowse >>>>>>>>>> jiffy >>>>>>>>>> mem3 >>>>>>>>>> mochiweb >>>>>>>>>> oauth >>>>>>>>>> rebar >>>>>>>>>> rexi >>>>>>>>>> snappy >>>>>>>>>> twig >>>>>>>>> >>>>>>>>> >>>>>>>>> I also contemplated this and and I am generally +1 on this. And >>>>>>>>>definitely >>>>>>>>> +1 to mirror them on the apache git if possible. I have a >>>>>>>>>couple of >>>>>>>>> comments though. >>>>>>>>> >>>>>>>>> Initially I also had everything separated in its own source >>>>>>>>>repository. 1 >>>>>>>>> year ago I merged back as one core repo the couchdb erlang >>>>>>>>>applications and >>>>>>>>> put all the dependencies in the refuge repository or in the >>>>>>>>>refuge CDN for >>>>>>>>> the spidermonkey and ICU sources. >>>>>>>>> >>>>>>>>> I merged back as one core repo the couchdb erlang applications >>>>>>>>>because they >>>>>>>>> were a little too much dependant. Especially couch_httpd, >>>>>>>>>couch_index and >>>>>>>>> couch_mrview. These applications are not yet enough by >>>>>>>>>themselves. >>>>>>>>> >>>>>>>>> Imo if we split everything in their own apps, then we should >>>>>>>>>make sure >>>>>>>>> that couch_httpd can be used without couch_index and >>>>>>>>>couch_mrview (which >>>>>>>>> means that "all_docs" is available in couch_httpd). Also we >>>>>>>>>should be able >>>>>>>>> to just launch couch without any of the above. And probably >>>>>>>>>without the >>>>>>>>> need of an ini. The couch_query_server module thing is an >>>>>>>>>interesting case. >>>>>>>>> bigcouch is also introducing `ddoc_cache` which I am not sure >>>>>>>>>why it is >>>>>>>>> provided as a standalone app. Does it means it can be replaced >>>>>>>>>by another >>>>>>>>> application eventually? Why not having it simply in the couch >>>>>>>>>application? >>>>>>>>> Does it needs to be updated separately? >>>>>>>>> >>>>>>>>> Also all our base applications should also be named spaced >>>>>>>>>correctly so >>>>>>>>> they will be strictly identified as erlang modules: "config" is >>>>>>>>>too >>>>>>>>> generic, "ddoc_cache" too. Others are probably OK. >>>>>>>>> >>>>>>>>> There are probably other things that we could provide as apps: >>>>>>>>> >>>>>>>>> - couch_daemon, >>>>>>>>> - couch_js >>>>>>>>> - couch_external >>>>>>>>> - couch_stats >>>>>>>>> - couch_compaction_daemon >>>>>>>>> - couch_httpd_proxy >>>>>>>>> >>>>>>>>> Anyway again i'm +1 for this move, I really think it's a good >>>>>>>>>idea. >>>>>>>>> >>>>>>>>> - benoit >>>>>>>> >>>>>>>> I agree on most of this. Roughly I see three general points. >>>>>>>> >>>>>>>> First, deciding on whether some things are external deps is >>>>>>>>definitely up for discussion. Whether couch_mrview is a different >>>>>>>>app/repo is not necessarily clear cut. Personally I think I over >>>>>>>>engineered couch_index which blurs the lines a bit. If I could >>>>>>>>wave a wand I'd have just couch_mrview and it'd be separate. More >>>>>>>>importantly I think the separate repos makes these things more >>>>>>>>apparent. The fact were discussing this sort of architecture thing >>>>>>>>is suggestive that it's forcing us to think a bit harder. >>>>>>>> >>>>>>>> Second is the aspect of composability. For instance the mrview >>>>>>>>thing to me is obviously a different repo precisely so a user >>>>>>>>could import couch (_core?) directly without requiring the spider >>>>>>>>monkey dependency. The monolithic repo doesn't allow this without >>>>>>>>some very non-standard tooling. >>>>>>>> >>>>>>>> Thirdly, app naming is always a contention. The config name was >>>>>>>>actually a hot code upgrade concern. We couldn't reuse >>>>>>>>couch_config directly at the time. And Adam was also hopeful we >>>>>>>>could the it into a useful non-specific config app. >>>>>>>> >>>>>>>> Fourthly, and related to secondly, we'll also want to look at >>>>>>>>splitting other apps out as necessary. The ones you listed I think >>>>>>>>aren't controversial it's just that no one has done it yet. My >>>>>>>>list was purely what existed so far without attempting to carve >>>>>>>>things up more. I definitely agree we should carve more in just >>>>>>>>wanted to cover consensus that carving is the right direction. >>>>>>>> >>>>>>>> Fifthly, I'm done typing on my phone. I'll fill in more thoughts >>>>>>>>tomorrow. >>>>>>>> >>>>>> >>> >>> >>> >>> -- >>> Noah Slater >>> https://twitter.com/nslater >> >> >> >> -- >> Noah Slater >> https://twitter.com/nslater >
