I worked on a handful of large clusters (> 200 nodes) using vnodes, and
there were some serious issues with both performance and availability.  We
had to put in a LOT of work to fix the problems.

I agree with Jeff - it's way better to manage multiple clusters than a
really large one.


On Fri, Jul 10, 2020 at 2:49 PM Jeff Jirsa <jji...@gmail.com> wrote:

> 1000 instances are fine if you're not using vnodes.
>
> I'm not sure what the limit is if you're using vnodes.
>
> If you might get to 1000, shard early before you get there. Running 8x100
> host clusters will be easier than one 800 host cluster.
>
>
> On Fri, Jul 10, 2020 at 2:19 PM Isaac Reath (BLOOMBERG/ 919 3RD A) <
> ire...@bloomberg.net> wrote:
>
>> Hi All,
>>
>> I’m currently dealing with a use case that is running on around 200
>> nodes, due to growth of their product as well as onboarding additional data
>> sources, we are looking at having to expand that to around 700 nodes, and
>> potentially beyond to 1000+. To that end I have a couple of questions:
>>
>> 1) For those who have experienced managing clusters at that scale, what
>> types of operational challenges have you run into that you might not see
>> when operating 100 node clusters? A couple that come to mind are version
>> (especially major version) upgrades become a lot more risky as it no longer
>> becomes feasible to do a blue / green style deployment of the database and
>> backup & restore operations seem far more error prone as well for the same
>> reason (having to do an in-place restore instead of being able to spin up a
>> new cluster to restore to).
>>
>> 2) Is there a cluster size beyond which sharding across multiple clusters
>> becomes the recommended approach?
>>
>> Thanks,
>> Isaac
>>
>>

Reply via email to