Thanks, Jan! > Let me know if you want to discuss any API or implementation specifics.
Yeah definitely. I'll start another thread to discuss API specifics there. Cheers, -Nick On Wed, Jan 23, 2019 at 6:23 AM Jan Lehnardt <j...@apache.org> wrote: > Heya Nick, > > sorry for not replying earlier, I had only sent a quick celebratory note > on IRC. > > I’d absolutely love having this feature. Let me know if you want to > discuss any API or implementation specifics. > > Best > Jan > — > > > On 9. Jan 2019, at 18:14, Nick Vatamaniuc <vatam...@gmail.com> wrote: > > > > Since CouchDB 2.0 clustered databases have had a fixed Q value defined at > > creation. This often requires users to predict database usage ahead of > time > > which can be hard to do. A too low of a value might result in large > shards, > > slower performance, and needing more disk space to do compactions. > > > > > > > > It would be nice to start with a low Q initially, for example Q=1 and as > > usage grows to be able to split some shards that grow too big. Especially > > after the partitioned query work, ( > > https://github.com/apache/couchdb/pull/1789) there will be a higher > chance > > of having uneven sized shards and so it will be beneficial to split the > > larger ones to even out the size distribution across the cluster. > > > > > > > > This proposal is basically to introduce such a feature to Apache CouchDB > > 2.x. > > > > > > > > From the user's perspective, there would be a new HTTP API endpoint. A > POST > > request to it with a node and a shard path would start a shard split job. > > Users would be able to monitor the state of this job and see when it > > completed. In the future this opens the possibility of writing an > > auto-splitting service that splits shards automatically when they reach a > > particular size or based on other parameters. > > > > > > > > Paul Davis and I have been experimenting over the last few months to see > if > > it is possible to do this. That progress so far is here: > > > > > > > > https://github.com/cloudant/couchdb/commits/shard-splitting > > > > > > > > Most of the bits are in mem3_shard_* and couch_db_split modules. > > > > > > > > There is an initial bulk copy of data from the source shard to the target > > shards. So a shard in the 00-ff range would be split into two shards with > > ranges 00-7f and 80-ff. While copying, each document ID is hashed and > > depending which side of the range it falls, it would end up either in the > > 00-7f shard or the 80-ff one. Then, when that is done, indices are > rebuilt > > for each shard range. Finally, the cluster-wide shard map is updated and > > the source shard is deleted. > > > > > > > > There are other details such as the internal replicator needing to know > how > > to replicate to a target that was split, and handling uneven shard copies > > in fabric coordinators. The HTTP API also would need to be figured out > and > > implemented and many other bits. > > > > > > > > What does the community at large think about this? If we like it, I can > > move that work to an ASF CouchDB branch and open a PR to finalize the > > design and continue the discussion there. > > -- > Professional Support for Apache CouchDB: > https://neighbourhood.ie/couchdb-support/ > >
