Nice summary Jan thanks. Just to also comment on Mango aggregations. This is a bit of a stumbling block for FDB. Some of the work I did recently was to move the Mango selector matching from the co-ordinator down to the actual node that has the document. The next step would be to then be to do in-memory aggregations on the node and send the result back to the co-ordinator. Unfortunately FDB doesn't support doing that. So any mango aggregation work that happens now we need to make sure it is possible in FDB as well.
On Fri, Jan 25, 2019 at 1:57 PM Jan Lehnardt <j...@apache.org> wrote: > Hi everyone, > > this is one of the threads we promised. This one covers the roadmap > discussion. > > This thread aims to answer: > > - which version would the FDB work land in? > - what happens with non-FDB CouchDB until then? > - specifically, how are the current roadmap items affected? > - how are the two different versions being managed in the future. > > Out of scope: > > - whether how how we name things publicly. > > The below contains my current thinking for an ideal way to handle this and > it is presented as a strawperson argument to be picked apart and refined, > designed to solicit input form everybody. > > > Please comment on the details of The Proposal (below) as well as the > categorisation and comments in The Tickets (also below). There are a lot of > details in here, feel free to comment on just one aspect of this rather > than feeling compelled to make a comment on everything. > > * * * > > > Naming prelude: to make this thread easier to follow, I’ll call the > current `master` version of CouchDB “Erlang CouchDB” (despite there being > non-Erlang bits in there) and the version that would be based on > FoundationDB “FBD CouchDB”. > > Note: these aren’t suggestions for marketing names or anything, just > shorthand for this particular discussion. > > * * * > > > ## Guiding Principles: > > - Getting to a position where FDB CouchDB is production ready is going > take a while. In the meantime, Erlang CouchDB will continue to receive > feature-, bugfix-, and security-updates. This is *at least* until an FDB > CouchDB verion is available, plus a grace period for a transition that is > to be discussed (gut feel, this is anything between 1, 2 and 3 years). > > - As a result of a multi-year dev-lifespan of Erlang CouchDB, and > extrapolating from experience in existing CouchDB production timespans, we > can easily imagine folks running Erlang CouchDB in production for *at > least* the next 10 years (At Neighbourhoodie, we are still supporting > CouchDB 1.2.1 customers e.g.) > > - A direct consequence of this, and in light of major feature work *at > some point* transitioning to FDB CouchDB, we should take this opportunity > and rally to make Erlang CouchDB “The Best Erlang CouchDB we can make”™. > That means we’ll re-evaluate the current roadmap against what’s essential > to make that happen, while leaving things that might be easier to do on FDB > CouchDB for the new setup. > > > * * * > > > ## The Proposal > > In very broad strokes, I imagine a CouchDB 3.x series that is “The Best > Erlang CouchDB we can make”™ that has all the big ticket items we planned > on doing modulo some that we leave for FDB CouchDB because they become > possible or significantly easier. The main reason for calling this 3.x > would be a set of deprecations as well as the addition of the _access field > to docs (where we would want a forward compatible version of the 2.x series > as well, so upgrades become possible) and the imposing of additional limits > to document, attachment and HTTP request sizes to be within recommended > best practices and to be forward compatible with FDB CouchDB, but which can > be brought back to 2.x settings for folks who need it. > > FDB CouchDB would then become the 4.x series. Even though we make major > changes under the hood, we would want for most CouchDB 2.x/3.x users to be > able to upgrade as seamlessly as possible, so we’ll aim for a high degree > of API compatibility like we’ve done in the 1.x -> 2.x transition and which > worked very well for us. > > * * * > > > ## The Annotated Roadmap > > The rest of this email is a list of all tickets that currently have > `roadmap` label with a comment on how to proceed with it plus a few bits > and bobs that are relevant in the context of an FDB future. > > Broadly, there are these categories: > > 1. we should add this to “The Best Erlang CouchDB we can make”™, category > YES > > 2. this is a ticket that can be worked on independently of either CouchDB > variant (say improvements to Mango, which can happen at any time), category > INDIE > > 3. we should hold off until FDB CouchDB to tackle this feature, category > WAIT > > > ## The Tickets > > ### Category: YES > > Build and ship view/secondary index warmer > https://github.com/apache/couchdb/issues/1825 > > Comment: n/a > > > Update SpiderMonkey version (was: move to Chakra Core) > https://github.com/apache/couchdb/issues/1875 > > Comment: we should get on this, really. > > > Migrate to rebar3: https://github.com/apache/couchdb/issues/1428 > > Comment: n/a > > > Improve plugin architecture / Erlang release building > https://github.com/apache/couchdb/issues/1515 > > Comment: I believe there is a separate proposal for this coming. > > > 1.x/2.x Deprecations https://github.com/apache/couchdb/issues/1534 > > Comment: to make upgrading to FDB CouchDB easier, we should be actively > addressing deprecations as well as system limitations. > > > Retire the node-local interface (port 5986) > https://github.com/apache/couchdb/issues/1523 > > Comment: Joan is already working on this. > > > Automate incrementally growing a cluster > https://github.com/apache/couchdb/issues/1517 > > Comment: Nick is already working on this, see other threads on dev@ > > > Better support for db-per-user > https://github.com/apache/couchdb/issues/1524 > > Comment: I’m working on this > > > > ### Category: INDIE > > Add aggregation functions to Mango: > https://github.com/apache/couchdb/issues/1254 > > Comment: this is an important feature that we should have sooner than > later, but Mango improvements can happen (mostly) independently of Erlang > vs. FDB CouchDB and can appear in either or both versions. > > > Improve CouchDB "plugin" architecture for geo/search > https://github.com/apache/couchdb/issues/1269 > > Comment: this only affects Erlang CouchDB for now, a similar discussion > should happen for FDB CouchDB, but that’s not on the roadmap just yet. > > > Simpler API: https://github.com/apache/couchdb/issues/1496 > > Comment: FDB CouchDB will re-use chttpd to make sure to be as close to API > compatible as possible. Any new/versioned APIs would be treated similarly. > > > Skip deletes / tombstone eviction > https://github.com/apache/couchdb/issues/1505 > > Comments: this needs review in light of clustered purge being available > now. > > > Groping a few things for efficiency: > > - Support server side / auto conflict resolution > https://github.com/apache/couchdb/issues/1506 > - First class conflict handling > https://github.com/apache/couchdb/issues/1507 > - Selective database syncing https://github.com/apache/couchdb/issues/1508 > - Support JOINs in Mango https://github.com/apache/couchdb/issues/1510 > - Support arbitrary result sorting in views and/or mango > https://github.com/apache/couchdb/issues/1511 > - Mango: Add Bitwise Operators > https://github.com/apache/couchdb/issues/1512 > - NoJS Mode https://github.com/apache/couchdb/issues/1513 > - Query Server Protocol v2 > https://github.com/apache/couchdb/issues/1514https://github.com/apache/couchdb/issues/1514 > - Support a Windows CI environment > https://github.com/apache/couchdb/issues/1535 > - Support CouchDB as hex / Erlang dep > https://github.com/apache/couchdb/issues/1536 > - Additional Mango-based update handler / VDU functionality > https://github.com/apache/couchdb/issues/1554 > - Mobile First replication protocol > https://github.com/apache/couchdb/issues/1503 > - Redesign CouchDB security system > https://github.com/apache/couchdb/issues/1504 > - Support GraphQL https://github.com/apache/couchdb/issues/1499 > - Synthetic load test suite https://github.com/apache/couchdb/issues/1521 > - Streaming API for attachment data > https://github.com/apache/couchdb/issues/1540 > - Improve config subsystem https://github.com/apache/couchdb/issues/1595 > - Externalise Erlang Term things (_revs, replication _ids) > https://github.com/apache/couchdb/issues/1530 > > Comment: n/a > > > Cluster-aware clients / API https://github.com/apache/couchdb/issues/1519 > > Comment: I don’t know if this would even be possible in FDB CouchDB, but > it would be a nice addition to Erlang CouchDB, but I don’t think it’s a > critical feature. > > > Schema extraction https://github.com/apache/couchdb/issues/1525 > > Comment: might be easier to do in FDB CouchDB > > > Database corruption detection & repair > https://github.com/apache/couchdb/issues/1526 > > Comment: only applies to Erlang CouchDB really, commercial solutions exist. > > > Do not shipping records over the inter-node wire > https://github.com/apache/couchdb/issues/1538 > > Comment: FDB CouchDB would not have this particular issue > > > Multi-tenancy support https://github.com/apache/couchdb/issues/1539 > > Comment: n/a > > > RFC: Sub-Document Operations https://github.com/apache/couchdb/issues/1559 > > Comment: Erlang CouchDB version is in the works, FDB CouchDB version would > be even more efficient internally due to per-field storage. > > > Improve time series data storage options > https://github.com/apache/couchdb/issues/1531 > > Comment: this would look different in either variant, but can happen > independently > > > ### Category: WAIT > > Support renaming a database https://github.com/apache/couchdb/issues/1502 > > Comment: this will become a lot easier in FDB-land > > > HTTP/2 Support https://github.com/apache/couchdb/issues/1497 > > Comment: FDB CouchDB will re-use chttpd to make sure to be as close to API > compatible as possible, which won’t get us towards HTTP/2 just yet. Any > discussion about updating the HTTP layer would happen after the FDB CouchDB > transition. Plus HAProxy now ships decent HTTP/2 support, so at least > cluster clients can get the benefits today already. > > > Support "very wide” clusters https://github.com/apache/couchdb/issues/1527 > > Comment: this is one of the things we want FDB for in the first place, > afaict. > > > Server-side mass update function > https://github.com/apache/couchdb/issues/1532 > > Comment: This becomes a lot easier in FDB CouchDB, maybe we just wait for > that. > > > Auto-record data on document create / update > https://github.com/apache/couchdb/issues/1533 > > Comment: this is extremely gnarly in an eventually consistent scenario and > in a permanently consistent FDB CouchDB world becomes much easier to do. > > * * * > > > Best > Jan > -- > -- > Professional Support for Apache CouchDB: > https://neighbourhood.ie/couchdb-support/ > >
