Hey everyone! I've gotten enough of a FoundationDB layer prototype implemented [1] to start sharing publicly. This is emphatically no where near useful to non-CouchDB-developers. The motivation for this work was to try and get enough of a basic prototype written so that we can all start fleshing out the various RFCs with actual implementations to compare and contrast and so on.
To be clear, I've made a lot of intentionally "bad" choices while writing this to both limit the scope of what I was trying to accomplish and also to make super clear that I don't expect any of this code to be "final" in any way whatsoever. This work is purely so that everyone has an initial code base that can be "turned on" so to speak. To that end, here's a non-exhaustive list of some of the silliness I've done: 1. All document bodies must fit inside a single value 2. All requests must fit within the single fdb transaction limits 3. I'm using binary_to_term for things like the revision tree 4. The revision tree has to fit in a single value 5. There's basically 0 supported query string parameters at this point 6. Nothing outside super basic db/doc ops is implemented (i.e., no views) However, what it does do is start! And it talks to FoundationDB! So at least that bit seems to be reliable (only tested on OS X via FoundationDB binary installers so super duper caveat on that "reliable"). There's a small test script [2] that shows what it's currently capable of. A quick glance at that should give a pretty good idea of how little is actually implemented in this prototype. There's also a list of notes I've been keeping as I've been hacking on this that also tries to gather a bunch of questions that'll need to be answered [3] as we continue to work on this. To that end, I have learned a couple lessons from working with FoundationDB from this work that I'd like to share. First is that while we can cache a bunch of stuff, we have to be able to ensure that the cache is invalidated properly when various bits of metadata change. There's a feature on FoundationDB master [1] for this specific issue. I've faked the same behavior using an arbitrary key but the `fabric2_fdb:is_current/1` function I think is a good implementation of this done correctly. Secondly, I spent a lot of time trying to figure out how to use FoundationDB's Directory and Subspace layers inside the CouchDB layer. After barking up that tree for a long time I've basically decided that the best answer is probably "don't". I do open a single directory at the root, but that's merely in order to play nice with any other layers that use the directory layer. Inside the "CouchDB directory" its all strictly Tuple Layer direct code. The Subspace Layer seems to be basically useless in Erlang. First, its a very thin wrapper over the Tuple Layer that basically just holds onto a prefix that's prepended onto the tuple layer operations. In other languages the Subspace Layer has a lot of syntactical sugar that makes them useful. Erlang doesn't support any of that so it ends up being more of a burden to use that rather than just using the Tuple Layer directly. Dropping the use of directories and subspaces has greatly simplified the implementation thus far. In terms of code layout, nearly all of the new implementation is in `src/fabric/src/fabric2*` modules. There's also a few changes to chttpd obviously to call the new code as well as commenting out parts of features so I didn't have to follow all the various call stacks updating huge swathes of semi-unrelated code. I'd be super interested to hear feed back and see people start running with this in whatever direction catches their fancy. Hopefully this proves useful for people to start writing implementations of the various RFCs so we can make progress on those fronts. [1] https://github.com/apache/couchdb/compare/prototype/fdb-layer [2] https://github.com/apache/couchdb/blob/prototype/fdb-layer/fdb-test.py [3] https://github.com/apache/couchdb/blob/prototype/fdb-layer/FDB_NOTES.md [4] https://forums.foundationdb.org/t/a-new-tool-for-managing-layer-metadata/1191
