All, Just to be clear before starting: this is a proposal from Cloudant not just myself :D
## What? Introduce a user-specified shard key per document and a way for the user to scope queries to a single shard using this key, thereby reducing query latency by allowing the database to consult only the shard required to answer the user's query. Documents can share a shard key and all documents with a given shard key are written to the same shard. The shard key therefore overrides using the document ID as the key to shard by. ## What problem does this proposal address? To improve performance and throughput of queries generally, and specifically to significantly decrease the effects of data size on query latency and cluster load leading to better data scaling characteristics of the database. CouchDB's current queries -- whether to views, search indexes or mango -- perform badly when the shard count (Q) for a database is high. This is because the coordinator node can't know in advance which shards hold results for the query, so must make a scatter-gather request to all shards, even those that in the end return no results. In turn this generates more load on all servers in the cluster because of many possibly redundant file reads. We did some hacky performance testing of this idea by modifying the current view API to allow one to specify a specific view shard in the query. The results were very promising, as they showed that queries started scaling with similar characteristics to lookups, as you'd hope for, in that Q ceased having such a large effect and load on the cluster decreased. ## API There are a few general considerations which act as constraints on the API. To get these out in the open: • The combined <shardkey>:<dockey> must be a valid document id. This enforces uniqueness of shardkey:docid across the database and invariance between document updates. • Aside from calculation of shard location, we treat the combined <shardkey>:<dockey> as we do any other document _id. • All existing tooling should continue to work as-is (e.g., changes feed doesn't have to change to specify a shardkey:dockey pair and the replicator will "just work"). • For Cloudant, it's useful to be able to differentiate queries by path for our internal accounting. For this we have a documentid becoming a composite thing: `<shardkey>:<documentkey>` • shardkey can be any valid characters for a document ID, but must not start with an _ or contain a : or /. • documentkey can be any valid characters for a document ID. Further : characters are treated as part of the key; meaningless. From this, our proposed API is, which should also help clarify a bit how things work: • Enable shard keys using database create: `PUT /db?use_shard_key=true`. Default false. • For databases where this is enabled, every document needs a shard key. • Upload a document: `PUT /db/<shardkey>:<dockey>` • In these databases, documents need user-specified document IDs as the shard key needs to be set by the user. • Therefore reject POSTing documents to /db for simplicity (rather than having validation of an _id field). • Other things that create documents will need the validation of document ID too: • Copy a document. Same as now but reject when destination document id does not have a shard key. • Calls like `POST /db/_bulk_docs`, `POST /db/_design/mydoc/_update/myupdatefunction` and others that accept documents via POST bodies will need to inspect the body for shard keys. • Get a document: `GET /db/<shardkey>:<dockey>`. • Special documents: • Design documents and local documents have no shardkey in their docID. This is enforced. • Reject documents with IDs of the form `<shardkey>:_design/foo`. • Query requests introduce a shardkey into the path, following an explicit `_shardkey` path part: • Mango: `POST /db/_shardkey/<shardkey>/_find`. • Views • `GET /db/_shardkey/<shardkey>/_design/mydoc/_view/myview?include_docs=true` • `POST /db/_shardkey/<shardkey>/_design/mydoc/_view/myview?include_docs=true` • Query results are restricted to documents with the shard key specified. Which makes things harder but leaves the door open for future things like shard-splitting without changing result sets. And it seems like what one would expect! • The above is a white list of APIs, so a few notes on some cases I've left out with reasons: • _all_docs (can see uses, but would rather keep API small to start with). Note that Mango depends on the internal _all_docs API. • _changes (use-cases better served by explicit shard API). ## What are we NOT addressing here? This proposal only provides indirect shard addressing -- it's specifically not covering things discussed previously [1] per-shard changes feeds where one needs to directly address each shard. The shard key only controls the bucketing of documents into shards and doesn't say anything about, for example, whether two shard keys end up on the same shard -- that's an internal detail. I'd really appreciate thoughts and comments before Cloudant get started on implementation of this. Mike. [1]: https://issues.apache.org/jira/browse/COUCHDB-2791