Re: [DISCUSS] CouchDB Roadmap
Nice summary Jan thanks. Just to also comment on Mango aggregations. This is a bit of a stumbling block for FDB. Some of the work I did recently was to move the Mango selector matching from the co-ordinator down to the actual node that has the document. The next step would be to then be to do in-memory aggregations on the node and send the result back to the co-ordinator. Unfortunately FDB doesn't support doing that. So any mango aggregation work that happens now we need to make sure it is possible in FDB as well. On Fri, Jan 25, 2019 at 1:57 PM Jan Lehnardt wrote: > Hi everyone, > > this is one of the threads we promised. This one covers the roadmap > discussion. > > This thread aims to answer: > > - which version would the FDB work land in? > - what happens with non-FDB CouchDB until then? > - specifically, how are the current roadmap items affected? > - how are the two different versions being managed in the future. > > Out of scope: > > - whether how how we name things publicly. > > The below contains my current thinking for an ideal way to handle this and > it is presented as a strawperson argument to be picked apart and refined, > designed to solicit input form everybody. > > > Please comment on the details of The Proposal (below) as well as the > categorisation and comments in The Tickets (also below). There are a lot of > details in here, feel free to comment on just one aspect of this rather > than feeling compelled to make a comment on everything. > > * * * > > > Naming prelude: to make this thread easier to follow, I’ll call the > current `master` version of CouchDB “Erlang CouchDB” (despite there being > non-Erlang bits in there) and the version that would be based on > FoundationDB “FBD CouchDB”. > > Note: these aren’t suggestions for marketing names or anything, just > shorthand for this particular discussion. > > * * * > > > ## Guiding Principles: > > - Getting to a position where FDB CouchDB is production ready is going > take a while. In the meantime, Erlang CouchDB will continue to receive > feature-, bugfix-, and security-updates. This is *at least* until an FDB > CouchDB verion is available, plus a grace period for a transition that is > to be discussed (gut feel, this is anything between 1, 2 and 3 years). > > - As a result of a multi-year dev-lifespan of Erlang CouchDB, and > extrapolating from experience in existing CouchDB production timespans, we > can easily imagine folks running Erlang CouchDB in production for *at > least* the next 10 years (At Neighbourhoodie, we are still supporting > CouchDB 1.2.1 customers e.g.) > > - A direct consequence of this, and in light of major feature work *at > some point* transitioning to FDB CouchDB, we should take this opportunity > and rally to make Erlang CouchDB “The Best Erlang CouchDB we can make”™. > That means we’ll re-evaluate the current roadmap against what’s essential > to make that happen, while leaving things that might be easier to do on FDB > CouchDB for the new setup. > > > * * * > > > ## The Proposal > > In very broad strokes, I imagine a CouchDB 3.x series that is “The Best > Erlang CouchDB we can make”™ that has all the big ticket items we planned > on doing modulo some that we leave for FDB CouchDB because they become > possible or significantly easier. The main reason for calling this 3.x > would be a set of deprecations as well as the addition of the _access field > to docs (where we would want a forward compatible version of the 2.x series > as well, so upgrades become possible) and the imposing of additional limits > to document, attachment and HTTP request sizes to be within recommended > best practices and to be forward compatible with FDB CouchDB, but which can > be brought back to 2.x settings for folks who need it. > > FDB CouchDB would then become the 4.x series. Even though we make major > changes under the hood, we would want for most CouchDB 2.x/3.x users to be > able to upgrade as seamlessly as possible, so we’ll aim for a high degree > of API compatibility like we’ve done in the 1.x -> 2.x transition and which > worked very well for us. > > * * * > > > ## The Annotated Roadmap > > The rest of this email is a list of all tickets that currently have > `roadmap` label with a comment on how to proceed with it plus a few bits > and bobs that are relevant in the context of an FDB future. > > Broadly, there are these categories: > > 1. we should add this to “The Best Erlang CouchDB we can make”™, category > YES > > 2. this is a ticket that can be worked on independently of either CouchDB > variant (say improvements to Mango, which can happen at any time), category > INDIE > > 3. we should hold off until FDB CouchDB to tackle this feature, category > WAIT > > > ## The Tickets > > ### Category: YES > > Build and ship view/secondary index warmer > https://github.com/apache/couchdb/issues/1825 > > Comment: n/a > > > Update SpiderMonkey version (was: move to Chakra Core) >
Re: [DISCUSS] CouchDB Roadmap
Hi, An excellent summary, I agree on almost all points. Aggregate functions in Mango will need a plan that we can carry forward to fdb couchdb. As noted in my opening email, the current aggregate functionality (“reduce”) in couchdb is the main piece that we don’t have a solution to with fdb (because we do not intimately control the b+tree updates). The “cluster aware” client ticket I think we should close as “will not do”. I don’t think we should remove that level of abstraction. An extra round of applause for redesigning the couchdb security system, though the three bullet points in 1504 add up to much less than a redesign. B. > On 25 Jan 2019, at 11:56, Jan Lehnardt wrote: > > Hi everyone, > > this is one of the threads we promised. This one covers the roadmap > discussion. > > This thread aims to answer: > > - which version would the FDB work land in? > - what happens with non-FDB CouchDB until then? > - specifically, how are the current roadmap items affected? > - how are the two different versions being managed in the future. > > Out of scope: > > - whether how how we name things publicly. > > The below contains my current thinking for an ideal way to handle this and it > is presented as a strawperson argument to be picked apart and refined, > designed to solicit input form everybody. > > > Please comment on the details of The Proposal (below) as well as the > categorisation and comments in The Tickets (also below). There are a lot of > details in here, feel free to comment on just one aspect of this rather than > feeling compelled to make a comment on everything. > > * * * > > > Naming prelude: to make this thread easier to follow, I’ll call the current > `master` version of CouchDB “Erlang CouchDB” (despite there being non-Erlang > bits in there) and the version that would be based on FoundationDB “FBD > CouchDB”. > > Note: these aren’t suggestions for marketing names or anything, just > shorthand for this particular discussion. > > * * * > > > ## Guiding Principles: > > - Getting to a position where FDB CouchDB is production ready is going take a > while. In the meantime, Erlang CouchDB will continue to receive feature-, > bugfix-, and security-updates. This is *at least* until an FDB CouchDB verion > is available, plus a grace period for a transition that is to be discussed > (gut feel, this is anything between 1, 2 and 3 years). > > - As a result of a multi-year dev-lifespan of Erlang CouchDB, and > extrapolating from experience in existing CouchDB production timespans, we > can easily imagine folks running Erlang CouchDB in production for *at least* > the next 10 years (At Neighbourhoodie, we are still supporting CouchDB 1.2.1 > customers e.g.) > > - A direct consequence of this, and in light of major feature work *at some > point* transitioning to FDB CouchDB, we should take this opportunity and > rally to make Erlang CouchDB “The Best Erlang CouchDB we can make”™. That > means we’ll re-evaluate the current roadmap against what’s essential to make > that happen, while leaving things that might be easier to do on FDB CouchDB > for the new setup. > > > * * * > > > ## The Proposal > > In very broad strokes, I imagine a CouchDB 3.x series that is “The Best > Erlang CouchDB we can make”™ that has all the big ticket items we planned on > doing modulo some that we leave for FDB CouchDB because they become possible > or significantly easier. The main reason for calling this 3.x would be a set > of deprecations as well as the addition of the _access field to docs (where > we would want a forward compatible version of the 2.x series as well, so > upgrades become possible) and the imposing of additional limits to document, > attachment and HTTP request sizes to be within recommended best practices and > to be forward compatible with FDB CouchDB, but which can be brought back to > 2.x settings for folks who need it. > > FDB CouchDB would then become the 4.x series. Even though we make major > changes under the hood, we would want for most CouchDB 2.x/3.x users to be > able to upgrade as seamlessly as possible, so we’ll aim for a high degree of > API compatibility like we’ve done in the 1.x -> 2.x transition and which > worked very well for us. > > * * * > > > ## The Annotated Roadmap > > The rest of this email is a list of all tickets that currently have `roadmap` > label with a comment on how to proceed with it plus a few bits and bobs that > are relevant in the context of an FDB future. > > Broadly, there are these categories: > > 1. we should add this to “The Best Erlang CouchDB we can make”™, category YES > > 2. this is a ticket that can be worked on independently of either CouchDB > variant (say improvements to Mango, which can happen at any time), category > INDIE > > 3. we should hold off until FDB CouchDB to tackle this feature, category WAIT > > > ## The Tickets > > ### Category: YES > > Build and ship
[DISCUSS] CouchDB Roadmap
Hi everyone, this is one of the threads we promised. This one covers the roadmap discussion. This thread aims to answer: - which version would the FDB work land in? - what happens with non-FDB CouchDB until then? - specifically, how are the current roadmap items affected? - how are the two different versions being managed in the future. Out of scope: - whether how how we name things publicly. The below contains my current thinking for an ideal way to handle this and it is presented as a strawperson argument to be picked apart and refined, designed to solicit input form everybody. Please comment on the details of The Proposal (below) as well as the categorisation and comments in The Tickets (also below). There are a lot of details in here, feel free to comment on just one aspect of this rather than feeling compelled to make a comment on everything. * * * Naming prelude: to make this thread easier to follow, I’ll call the current `master` version of CouchDB “Erlang CouchDB” (despite there being non-Erlang bits in there) and the version that would be based on FoundationDB “FBD CouchDB”. Note: these aren’t suggestions for marketing names or anything, just shorthand for this particular discussion. * * * ## Guiding Principles: - Getting to a position where FDB CouchDB is production ready is going take a while. In the meantime, Erlang CouchDB will continue to receive feature-, bugfix-, and security-updates. This is *at least* until an FDB CouchDB verion is available, plus a grace period for a transition that is to be discussed (gut feel, this is anything between 1, 2 and 3 years). - As a result of a multi-year dev-lifespan of Erlang CouchDB, and extrapolating from experience in existing CouchDB production timespans, we can easily imagine folks running Erlang CouchDB in production for *at least* the next 10 years (At Neighbourhoodie, we are still supporting CouchDB 1.2.1 customers e.g.) - A direct consequence of this, and in light of major feature work *at some point* transitioning to FDB CouchDB, we should take this opportunity and rally to make Erlang CouchDB “The Best Erlang CouchDB we can make”™. That means we’ll re-evaluate the current roadmap against what’s essential to make that happen, while leaving things that might be easier to do on FDB CouchDB for the new setup. * * * ## The Proposal In very broad strokes, I imagine a CouchDB 3.x series that is “The Best Erlang CouchDB we can make”™ that has all the big ticket items we planned on doing modulo some that we leave for FDB CouchDB because they become possible or significantly easier. The main reason for calling this 3.x would be a set of deprecations as well as the addition of the _access field to docs (where we would want a forward compatible version of the 2.x series as well, so upgrades become possible) and the imposing of additional limits to document, attachment and HTTP request sizes to be within recommended best practices and to be forward compatible with FDB CouchDB, but which can be brought back to 2.x settings for folks who need it. FDB CouchDB would then become the 4.x series. Even though we make major changes under the hood, we would want for most CouchDB 2.x/3.x users to be able to upgrade as seamlessly as possible, so we’ll aim for a high degree of API compatibility like we’ve done in the 1.x -> 2.x transition and which worked very well for us. * * * ## The Annotated Roadmap The rest of this email is a list of all tickets that currently have `roadmap` label with a comment on how to proceed with it plus a few bits and bobs that are relevant in the context of an FDB future. Broadly, there are these categories: 1. we should add this to “The Best Erlang CouchDB we can make”™, category YES 2. this is a ticket that can be worked on independently of either CouchDB variant (say improvements to Mango, which can happen at any time), category INDIE 3. we should hold off until FDB CouchDB to tackle this feature, category WAIT ## The Tickets ### Category: YES Build and ship view/secondary index warmer https://github.com/apache/couchdb/issues/1825 Comment: n/a Update SpiderMonkey version (was: move to Chakra Core) https://github.com/apache/couchdb/issues/1875 Comment: we should get on this, really. Migrate to rebar3: https://github.com/apache/couchdb/issues/1428 Comment: n/a Improve plugin architecture / Erlang release building https://github.com/apache/couchdb/issues/1515 Comment: I believe there is a separate proposal for this coming. 1.x/2.x Deprecations https://github.com/apache/couchdb/issues/1534 Comment: to make upgrading to FDB CouchDB easier, we should be actively addressing deprecations as well as system limitations. Retire the node-local interface (port 5986) https://github.com/apache/couchdb/issues/1523 Comment: Joan is already working on this. Automate incrementally growing a cluster https://github.com/apache/couchdb/issues/1517 Comment: Nick
