You should look into routing, instead of different sized clusters. Shard size of 5-10GB is good, more can lead to increased recovery times. Cluster management is easy with suitable configuration management - puppet, chef etc.
Regards, Mark Walkom Infrastructure Engineer Campaign Monitor email: [email protected] web: www.campaignmonitor.com On 20 March 2014 09:16, Brad Jordan <[email protected]> wrote: > Hey all, > > I have 30 machines to build my ES cluster. I am going to need to build > 2000 indexes with 14400 shards total. Should I build two clusters or one? > > I have a multi customer data set. 65% of the customers have very little > data, "small customers". The other 35% have 100x more data, > "big customers". My thought was to build one cluster where all the > "small customers" share a set of 60 "monthly" indexes each with 100 shards. > I would build a second cluster where each customer gets their own set of 60 > "monthly" indexes each with 6 shards. This configuration is so that I can > keep shard sizes below 5gb across both clusters. The small cluster is > routed by customerId. The big cluster uses the default routing strategy. > > questions: > 1) Should I make two clusters or just one? > 2) Do I need to keep shard sizes below 5gb? > 3) Is management of one cluster with 2000 indexes and 14400 shards more > difficult than 2 clusters where "small" cluster has 60 indexes and 600 > shards and "big" cluster has 1900 indexes and 13000 shards? > > Thanks, > Brad > > -- > You received this message because you are subscribed to the Google Groups > "elasticsearch" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to [email protected]. > To view this discussion on the web visit > https://groups.google.com/d/msgid/elasticsearch/9e14329b-ebb3-4948-9fd3-eaf10cce7477%40googlegroups.com<https://groups.google.com/d/msgid/elasticsearch/9e14329b-ebb3-4948-9fd3-eaf10cce7477%40googlegroups.com?utm_medium=email&utm_source=footer> > . > For more options, visit https://groups.google.com/d/optout. > -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/CAEM624Z8HwRh8j-BJj2pBKea1H55CRZFVL668w1NNzQn%2Bgrh6A%40mail.gmail.com. For more options, visit https://groups.google.com/d/optout.
