Re: [DISCUSS] Rebase CouchDB on top of FoundationDB

2019-02-01 Thread Robert Newson
Hi, Thanks for the links, that's the detail we need. While we can't exclude the possibility of it, FoundationDB has an extraordinary amount of testing. It's far more likely that there will be bugs in the layer of code that we write to talk to it, which of course we can also then fix and releas

Re: [DISCUSS] Rebase CouchDB on top of FoundationDB

2019-02-01 Thread Eli Stevens (Gmail)
To the best of my recollection, it was these issues: https://github.com/apache/couchdb/issues/745 (leading to https://github.com/apache/couchdb/pull/1200 and then https://github.com/apache/couchdb/pull/1253) https://github.com/apache/couchdb/issues/1093 (also points back at 745?) I could have swo

Re: [DISCUSS] Rebase CouchDB on top of FoundationDB

2019-02-01 Thread Robert Newson
Hi, Avoiding unintended regressions is a high priority for the team and we will need to lean on the community here to keep us honest. I'd appreciate a summary of the capability regressions that affected you. The CouchDB PMC is interacting with the FoundationDB team now (you can see the beginni

Re: [DISCUSS] Rebase CouchDB on top of FoundationDB

2019-02-01 Thread Eli Stevens (Gmail)
A couple other topics that I think would make sense to discuss: - How to manage edge-case performance or capability regressions resulting from the switch. My former team couldn't use 2.x in production until 2.2 due to a handful of these kinds of issues. What's going to happen when users blocked du

Re: [DISCUSS] Rebase CouchDB on top of FoundationDB

2019-01-31 Thread Adam Kocoloski
> On Jan 31, 2019, at 12:27 PM, nicholas a. evans > wrote: > >> I called out the problems with reduce functionality in the first post of >> this thread specifically to shake out people's concerns there, so thank you >> for voicing yours. The current approach to reduce only works because we >

Re: [DISCUSS] Rebase CouchDB on top of FoundationDB

2019-01-31 Thread nicholas a. evans
Thanks Robert, (comments inline, below) On Thu, Jan 31, 2019 at 9:10 AM Robert Newson wrote: > I think it would significantly complicate this work to make it a per-database > decision. Thanks, and that makes perfect sense. That question was really just for my curiosity. > I called out the prob

Re: [DISCUSS] Rebase CouchDB on top of FoundationDB

2019-01-31 Thread Robert Newson
Hi Nick, I don't think anyone responded to your points yet. I think it would significantly complicate this work to make it a per-database decision. I think it has to be a wholesale cutover to a new backend with appropriate warnings in release notes and guidance on migration. This is why the pl

Re: [DISCUSS] Rebase CouchDB on top of FoundationDB

2019-01-27 Thread nicholas a. evans
Thanks Jan, On Sun, Jan 27, 2019, 3:43 AM Jan Lehnardt The FDB proposal is starting at a higher level than the pluggable storage > engines. This isn't just about storage, but also about having a new > abstraction over the distributed systems aspects of CouchDB. > Right, FoundationDB would *also*

Re: [DISCUSS] Rebase CouchDB on top of FoundationDB

2019-01-27 Thread Jan Lehnardt
of Couchdb. Mango is quite non-deterministic >>> when it comes to performance (defining the right indexes is cumbersome >>> compared to using views, and its difficult to know if a query will be >>> completed by doing in-memory filtering). And people keep reporting a numbe

Re: [DISCUSS] Rebase CouchDB on top of FoundationDB

2019-01-26 Thread nicholas a. evans
ery will be > > completed by doing in-memory filtering). And people keep reporting a number > > of troubling bugs. So moving people to Mango is not only about updating > > applications, this is also losing quite substantial features. > > > > All in all, my point is that with that the chang

Re: [DISCUSS] Rebase CouchDB on top of FoundationDB

2019-01-26 Thread Jan Lehnardt
re folks like us >> for who the API of the view/reduce pipeline is central. So that hopefully >> this can be taken into account as the merits of this proposal are being >> reviewed. >> >> >> De : Dave Cottlehuber >> Envoyé : samedi 26

Re: [DISCUSS] Rebase CouchDB on top of FoundationDB

2019-01-26 Thread Robert Newson
folks like us > for who the API of the view/reduce pipeline is central. So that hopefully > this can be taken into account as the merits of this proposal are being > reviewed. > > ____________________ > De : Dave Cottlehuber > Envoyé : samedi 26 janvier 20

RE: [DISCUSS] Rebase CouchDB on top of FoundationDB

2019-01-26 Thread Reddy B .
De : Dave Cottlehuber Envoyé : samedi 26 janvier 2019 15:31:24 À : dev@couchdb.apache.org Objet : Re: [DISCUSS] Rebase CouchDB on top of FoundationDB On Fri, 25 Jan 2019, at 09:58, Robert Samuel Newson wrote: > Thanks for sharing this Bob, and also thanks everybody who shared

Re: [DISCUSS] Rebase CouchDB on top of FoundationDB

2019-01-26 Thread Dave Cottlehuber
On Fri, 25 Jan 2019, at 09:58, Robert Samuel Newson wrote: > Thanks for sharing this Bob, and also thanks everybody who shared their thoughts too. I'm super excited, partly because we get to keep all our Couchy goodness, and also that FDB brings some really interesting operational capabilities to

Re: [DISCUSS] Rebase CouchDB on top of FoundationDB

2019-01-25 Thread Robert Samuel Newson
Hi, This isn’t the thread (yet!) to get into this level of detail just yet, but I do have some thoughts. The two uses of sha256 here seem inappropriate to me. Users will typically choose short, readable names for both user_name and db_name, and this would force long, random looking strings on

Re: [DISCUSS] Rebase CouchDB on top of FoundationDB

2019-01-24 Thread Michael Fair
> > As to your wishlist - let's break that off to a different discussion, > Thanks Joan! Totally agreed, I didn't mean those to be Roadmap proposals, merely 'justifications', in a sense, of things I saw likely becoming easier for me to "get into" as a consequence of the FDB change over; which hel

Re: [DISCUSS] Rebase CouchDB on top of FoundationDB

2019-01-24 Thread Ilya Khlopotov
First I apologize if you receive it twice (slightly different versions as well). It looks like my email is miss-configured since reply to dev@couchdb.apache.org from mail client didn't go through. > This eliminates the "length" of the path string concern and keeps every > document field a strai

Re: [DISCUSS] Rebase CouchDB on top of FoundationDB

2019-01-24 Thread Robert Samuel Newson
Hi, In theory, yes. But using a SHA256 key would be bad in practice for at least the same reasons a fully random doc id is bad in couchdb (and why our default "sequential" algorithm works the way it does). Specifically, no two related keys would likely land anywhere near each other in the keysp

Re: [DISCUSS] Rebase CouchDB on top of FoundationDB

2019-01-24 Thread Joan Touzet
Hi Michael, Thanks so much for a great series of emails. With my PMC hat on: > Is this actually worthy of a 3.0 moniker; it seems like it could be > (breaking changes and dropping 1.X compatibility)? This is part of the to-come Branding discussion, but yes, there will be a clear delineation betw

Re: [DISCUSS] Rebase CouchDB on top of FoundationDB

2019-01-24 Thread Jan Lehnardt
Heya Mike, this is excellent input and exactly the type of stuff we want to nail down in subsequent discussions. Best Jan — > On 24. Jan 2019, at 13:46, Michael Fair wrote: > > On Thu, Jan 24, 2019 at 2:11 AM Robert Samuel Newson > wrote: > >> >> We’d expand each document into a series of

Re: [DISCUSS] Rebase CouchDB on top of FoundationDB

2019-01-24 Thread Jan Lehnardt
do we differentiate between "CouchDB Classic" >>>> and "New CouchDB" in a succinct and clear way? >>>> >>>> * FoundationDB - all the non-technical aspects. Review of _their_ >>>> project governance, cross-project p

Re: [DISCUSS] Rebase CouchDB on top of FoundationDB

2019-01-24 Thread Michael Fair
On Thu, Jan 24, 2019 at 2:11 AM Robert Samuel Newson wrote: > > We’d expand each document into a series of key-value pairs, where the key > is the full path into the object and the value is the scalar value. E.g, > > {“foo”: 12, “bar”, {“baz”: 13}} > > Would be > > foo => 12 > bar.baz => 13 I r

Re: [DISCUSS] Rebase CouchDB on top of FoundationDB

2019-01-24 Thread Michael Fair
ical. IBM doesn't get to just > >> throw code over the wall at us. Similarly, should we > >>choose to work on proposed features, or stuff from > >>the roadmap, we need to be able to cooperate. No > >>

Re: [DISCUSS] Rebase CouchDB on top of FoundationDB

2019-01-24 Thread Robert Samuel Newson
Hi, Absolutely agree. While strictly orthogonal, document level access has been a wishlist item for a lot of users for a very long time. The main problem is how it interacts with (i.e, breaks) replication. That said, Jan is working on per-document level access now (pre-FoundationDB) based on a

Re: [DISCUSS] Rebase CouchDB on top of FoundationDB

2019-01-24 Thread Robert Samuel Newson
topics by PMC members over the >> coming days (but not all at once, so everyone has time to reflect and >> respond.) >> >> My initial take on the proposal: it's GOOD that we're finally >> addressing some of the problems that 2.x brought to the table, and if >&g

Re: [DISCUSS] Rebase CouchDB on top of FoundationDB

2019-01-24 Thread Robert Samuel Newson
Hi, We’d expand each document into a series of key-value pairs, where the key is the full path into the object and the value is the scalar value. E.g, {“foo”: 12, “bar”, {“baz”: 13}} Would be foo => 12 bar.baz => 13 With some suitable prefix to denote database name and document id. If the 10k

Re: [DISCUSS] Rebase CouchDB on top of FoundationDB

2019-01-24 Thread Jan Lehnardt
Hi Will, your opinion is as valuable as anyone else’s here, and with such a large proposal, we need to hear everyone’s voice. I’m currently the (unofficial) champion of this proposal and I have some wip code that just last weekend got to a stage that showed that with some more work, this could

Re: [DISCUSS] Rebase CouchDB on top of FoundationDB

2019-01-24 Thread Jan Lehnardt
Hi ermouth, thanks for chiming in! More details in the technical discussion, but as a preview: the current idea is use FDB key/values (and their limitations) as JSON field key/values. So you can make documents > 10/100k, but there would be a limit for each key in a doc that is 10/100. But ther

Re: [DISCUSS] Rebase CouchDB on top of FoundationDB

2019-01-24 Thread Jan Lehnardt
tually be many, many threads I expect, >>including everyone's favourite on release mgmt :P >> >> New threads will be started on these topics by PMC members over the >> coming days (but not all at once, so everyone has time to reflect and >> respond.) &g

Re: [DISCUSS] Rebase CouchDB on top of FoundationDB

2019-01-23 Thread William Edney
As someone who is a user of CouchDB and who's normally a lurker here (which maybe means my opinion doesn't mean much), I would humbly submit that a higher priority (for me at least) is the lack of per-user document access as detailed here: https://github.com/apache/couchdb/issues/1524 I'm not sur

Re: [DISCUSS] Rebase CouchDB on top of FoundationDB

2019-01-23 Thread ermouth
Hi, Robert > as something we don’t know how to carry over yet Is there any workaround for FDB limitations for key size (10kb) and value size (100kb)? ermouth

Re: [DISCUSS] Rebase CouchDB on top of FoundationDB

2019-01-23 Thread Michael Fair
me of the problems that 2.x brought to the table, and if > this is the best way to do so, then so be it. I want to know more > about the technical details, and I want to see a more formal RFC before > voting on it, though. > > -Joan 'And now for something completely different...'

Re: [DISCUSS] Rebase CouchDB on top of FoundationDB

2019-01-23 Thread Joan Touzet
'And now for something completely different...' Touzet [*] http://communitymgt.wikia.com/wiki/Cookie_Licking - Original Message - > From: "Jan Lehnardt" > To: "CouchDB Developers" > Sent: Wednesday, January 23, 2019 8:33:30 AM > Subject: Re: [DISCUSS] Rebase Couc

Re: [DISCUSS] Rebase CouchDB on top of FoundationDB

2019-01-23 Thread Robert Newson
Hi Eli, I agree with Jan that those are great topics and we owe them a response. In my opening post I called out custom reduces as something we don’t know how to carry over yet. That doesn’t mean they are being removed per se. That decision would happen on the mailing list and only after the d

Re: [DISCUSS] Rebase CouchDB on top of FoundationDB

2019-01-23 Thread Jan Lehnardt
Hi Eli, Thanks for chiming in. These are all good topics and are in some form or another already on our list to be discussed. Re query servers: it is for now really just custom reduces and arbitrary startkey/endkey ranges. JS views aren't going anywhere. Cheers Jan — > On 23. Jan 2019, at 18

Re: [DISCUSS] Rebase CouchDB on top of FoundationDB

2019-01-23 Thread Eli Stevens (Gmail)
I'd like to request that there be threads where it's appropriate to discuss: - Managing the refactoring/merge process to avoid the previous situation where 1.x was mostly dead, but 2.x wasn't going to land for a few years. - Other features to deprecate at the same time as losing JS reduce (I assum

Re: [DISCUSS] Rebase CouchDB on top of FoundationDB

2019-01-23 Thread Jan Lehnardt
Hi Bob, this is all very exciting! First up, full disclosure, the CouchDB PMC has had about two weeks to think about this already, so if any of the following doesn’t sound like a knee-jerk reaction, that’s why. I’m personally tentatively optimistic about this proposal and I’m willing to work