Re: Transparently rebalancing a Solr cluster without splitting or moving shards

2014-07-08 Thread Damien Dykman
Thanks for your suggestions and recommendations. If I understand correctly, the MIGRATE command does shard splitting (around the range of the split.key) and merging behind the scene. Though, it's a bit difficult to properly monitor the actual migration, set the proper timeouts, know when to

Transparently rebalancing a Solr cluster without splitting or moving shards

2014-07-07 Thread Damien Dykman
I have a cluster of N boxes/nodes and I'd like to add M boxes/nodes and rebalance data accordingly. Lets add the following constraints: - 1. boxes have different characteristics (RAM, CPU, disks) - 2. different number of shards per box/node (lets pretend we have found the sweet spot for

Re: Transparently rebalancing a Solr cluster without splitting or moving shards

2014-07-07 Thread Shawn Heisey
On 7/7/2014 12:41 PM, Damien Dykman wrote: I have a cluster of N boxes/nodes and I'd like to add M boxes/nodes and rebalance data accordingly. Lets add the following constraints: - 1. boxes have different characteristics (RAM, CPU, disks) - 2. different number of shards per box/node

Re: Transparently rebalancing a Solr cluster without splitting or moving shards

2014-07-07 Thread Damien Dykman
Thanks Shawn, clean way to do it, indeed. And going your route, one could even copy the existing shards into the new collection and then delete the data which is getting reindexed on the new nodes. That would spare reindexing everything. But in my case, I add boxes after a noticeable

Re: Transparently rebalancing a Solr cluster without splitting or moving shards

2014-07-07 Thread Shawn Heisey
Thanks Shawn, clean way to do it, indeed. And going your route, one could even copy the existing shards into the new collection and then delete the data which is getting reindexed on the new nodes. That would spare reindexing everything. But in my case, I add boxes after a noticeable