Hi Nick,
Thanks for writing in.
The FDB proposal is starting at a higher level than the pluggable storage
engines. This isn't just about storage, but also about having a new abstraction
over the distributed systems aspects of CouchDB.
As for reduce: CouchDB will *not* lose reduce. Details are TBD, so let's wait
to discuss them for when the technical proposal for that part is out, please.
So far, all IBM has mentioned is that in their preliminary exploration of this,
they couldn't find a trivial way to support *efficient querying* for *custom
reduce functions* (anything that isn't _sum/_count/_stats).
Cheers
Jan
—
> On 27. Jan 2019, at 08:02, nicholas a. evans <nicholas.ev...@gmail.com> wrote:
>
> This seems like an exciting development. Could it build on the recent
> pluggable storage engine work?
>
> Speaking as a user, losing efficient reduce results queries would be a
> *very* big deal for me. But I'm naively guessing a workaround might be
> to create a tree just for reductions, updated in the same transaction
> as the map emits? This could be opt-in (or opt-out), only if you
> actually needed it. E.g. perhaps use tuples => values analogous to:
>
> ("db", "_design/foo", "_view/bar", "_btree_reduce") => {"val": ...,
> childkeys: ["a", "f", "k", "p", "u", "z"]}
> ("db", "_design/foo", "_view/bar", "_btree_reduce", 0) => {"val":
> ..., childkeys: ["a", "b", "c", "d", "e"]}
> ("db", "_design/foo", "_view/bar", "_btree_reduce", 0, 0) => {"val":
> ..., childkeys: ["a", "aa", "ab", "ac", "ad", "ae"]}
> ("db", "_design/foo", "_view/bar", "_btree_reduce", 0, 1) => {"val":
> ..., childkeys: ["b", "ba", "bb", "bc", "bd", "be"]}
> etc...
>
> Since 90% of my reduce queries use group=exact or group_level (some
> views are *always* queried with group=exact), it might also make sense
> to (opt-in) store reductions per view key at every group_level (or
> only for group=exact). Generation could piggyback on the btree
> reductions (i.e. when there are tens of thousands of emits per key).
> For a little extra disk space, this would allow *never* running reduce
> fns during update={false,lazy} queries (only during updates).
>
> ("db", "_design/foo", "_view/bar", "_group_reduce", "a") => {"val": ...}
> ("db", "_design/foo", "_view/bar", "_group_reduce", "a", 0) => {"val": ...}
> ("db", "_design/foo", "_view/bar", "_group_reduce", "a", 0, "foo") =>
> {"val": ...}
> etc...
>
> It seems to me, a similar approach could maybe be used to (finally!?)
> create an opt-in view changes feed? I would love a view changes feed
> that enabled a quick and easy "chained" map reduce pipeline, or
> "replicating" group-reduced results into a new DB (like Cloudant's
> deprecated dbcopy feature), or something like RethinkDB's streaming
> results.
>
> ("db", "_design/foo", "_view/bar", "_seq", 1) => {"add": ["key1",
> "key2", "key3"]}
> ("db", "_design/foo", "_view/bar", "_seq", 2) => {"add": ["key4"],
> "del": ["key2"]}
> etc...
>
> --
> Nick Evans
>
>> On Sat, Jan 26, 2019 at 1:25 PM Robert Newson <rnew...@apache.org> wrote:
>>
>> Hi,
>>
>> It’s only arbitrary start/end key of the reduce values we have no solution
>> for yet . For map-only, we can and would supply arbitrary start/end key.
>>
>> More explicitly, it’s only _efficient_ reduce results that are lost, because
>> we can’t store the intermediate reduce values on inner btree nodes. We could
>> calculate the reduce value dynamically by reading the entire range of
>> selected keys and calculate the reduce value each time.
>>
>> Finally, this is a known gap in our investigation. It doesn’t mean there
>> isn’t an answer to be discovered.
>>
>> B
>>
>>> On 26 Jan 2019, at 17:20, Reddy B. <redd...@live.fr> wrote:
>>>
>>> Hello,
>>>
>>> Just to add the modest perspective of a user. I appreciate the benefits of
>>> taking advantage of the infrastructure provided by FDB both from a quality
>>> perspective and from a maintenance and ease of expansion perspective.
>>>
>>> However, this development makes me really worried of being burned as a user
>>> so to speak. Losing arbitrary reduce functions would be a big concern. But
>>> losing arbitrary startkey and endkey would be an even bigger concern.
>>>
>>> This is not about the inconvenience of updating our codebase, this is about
>>> losing the ability to do quite significant things, losing expressiveness so
>>> to speak. We make extensive use of startkeys/endkeys for things ranging
>>> from geoqueries using a simple application-based geohash implementation,
>>> all the way to matching the documents belonging to a tenant using complex
>>> keys. So if this is indeed the feature we'd be losing, this is quite a big
>>> deal in my opinion. I think all our data access layers would need to be
>>> rewritten but I do not even know how.
>>>
>>> For I do not know if we are representative, but we intentionally stay away
>>> from Mango to leverage the performance benefits of using precomputed views
>>> which see as a key feature of Couchdb. Mango is quite non-deterministic
>>> when it comes to performance (defining the right indexes is cumbersome
>>> compared to using views, and its difficult to know if a query will be
>>> completed by doing in-memory filtering). And people keep reporting a number
>>> of troubling bugs. So moving people to Mango is not only about updating
>>> applications, this is also losing quite substantial features.
>>>
>>> All in all, my point is that with that the changes I hear, I feel like a
>>> lot of the technical assumptions we made when we settled on Couchdb would
>>> no longer hold. There are patterns we wouldn't be able to use, and I don't
>>> even think that it would still be possible to develop real world
>>> applications solely relying of the view/reduce pipeline only if we do not
>>> have the level of expressiveness provided by custom reduce and arbitrary
>>> startkeys/endkeys. Without these two structures, we will burned in certain
>>> situations.
>>>
>>> Just wanted to voice this concern to highlight that there are folks like us
>>> for who the API of the view/reduce pipeline is central. So that hopefully
>>> this can be taken into account as the merits of this proposal are being
>>> reviewed.
>>>
>>> ________________________________
>>> De : Dave Cottlehuber <d...@skunkwerks.at>
>>> Envoyé : samedi 26 janvier 2019 15:31:24
>>> À : dev@couchdb.apache.org
>>> Objet : Re: [DISCUSS] Rebase CouchDB on top of FoundationDB
>>>
>>>> On Fri, 25 Jan 2019, at 09:58, Robert Samuel Newson wrote:
>>>>
>>>
>>> Thanks for sharing this Bob, and also thanks everybody who shared their
>>> thoughts too.
>>>
>>> I'm super excited, partly because we get to keep all our Couchy
>>> goodness, and also that FDB brings some really interesting operational
>>> capabilities to the table that normally you spend a decade trying to
>>> build from scratch. The level of testing that has gone into FDB is
>>> astounding[1].
>>>
>>> Things like seamless data migration, expanding storage and rebalancing
>>> shards and nodes, as anybody who's dealt with large or long-lived
>>> couchdb clusters knows are Hard Problems today.
>>>
>>> There's clearly a lot of work to be done -- it's early days -- and it
>>> changes a lot of non-visible things like packaging, dependencies,
>>> cross-platform support, and a markedly different operations model -- but
>>> I'm most excited about the opportunities here at the storage layer for
>>> us.
>>>
>>> Regarding handling larger k/v items than what fdb can handle, is covered
>>> in the forums already[2] and is similar to how we'd query multiple docs
>>> from a couchdb view today using an array-based complex/compound key:
>>>
>>> [0, ..] would give you all the docs in that view under key 0
>>>
>>> except that in FDB that query would happen for a single couchdb doc, and
>>> returning a range query to achieve that. Similar to multiple docs, there
>>> are some traps around managing that in an atomic fashion at the higher
>>> layer.
>>>
>>> I'm sure there are many more things like this we'll need to wrap our
>>> heads around!
>>>
>>> Especial thanks to the dual-hat-wearing IBM folk who have engaged with
>>> the community so early in the process -- basically at the napkin
>>> stage[3].
>>>
>>> [1]: https://www.youtube.com/watch?v=4fFDFbi3toc
>>> [2]:
>>> https://forums.foundationdb.org/t/intent-roadmap-to-handle-larger-value-sizes/126
>>> [3]: https://www.computerhistory.org/atchm/the-two-napkin-protocol/ the
>>> famous napkin where BGP, the modern internet's backbone routing
>>> protocol, was described.
>>>
>>> A+
>>> Dave
>>
