In testing and replicating the issue, this slow down has been seen occurring with empty indices.
The running cluster is at present ~100 GB across 2,200 Indices with a total of 13,500 shards and ~430,000,000 documents. We have 7GB RAM and 5GB heap on the data nodes - haven't looked overly carefully but don't think the heap is maxing out on any of the nodes when this occurs. Thanks, Paul. On Tuesday, May 13, 2014 11:02:32 AM UTC+1, Mark Walkom wrote: > > Sounds like the inevitable "add more nodes" situation. > > How much RAM on each node, how big is your data set? > > Regards, > Mark Walkom > > Infrastructure Engineer > Campaign Monitor > email: ma...@campaignmonitor.com <javascript:> > web: www.campaignmonitor.com > > > On 13 May 2014 19:59, Paul <codi...@gmail.com <javascript:>> wrote: > >> We are seeing a slow down in shard initialization speed as the number of >> shards/indices grows in our cluster. >> >> With 0-100's of indices/shards existing in the cluster a new bulk >> creation of indices up the 100's at a time is fine, we see them pass >> through the states and get a green cluster in a reasonable amount of time. >> >> As the total cluster size grows to 1000+ indices (3000+ shards) we begin >> to notice that the first rounds of initialization take longer to process, >> it seems to speed up after the first few batches, but this slow down leads >> to "failed to process cluster event (create-index [index_1112], cause >> [auto(bulk api)]) within 30s" type messages in the Master logs - the >> indices are eventually created. >> >> >> Has anyone else experienced this? (did you find the cause / way to fix?) >> >> Is this somewhat expected behaviour? - are we approaching something >> incorrectly? (there are 3 data nodes involved, with 3 shards per index) >> >> -- >> You received this message because you are subscribed to the Google Groups >> "elasticsearch" group. >> To unsubscribe from this group and stop receiving emails from it, send an >> email to elasticsearc...@googlegroups.com <javascript:>. >> To view this discussion on the web visit >> https://groups.google.com/d/msgid/elasticsearch/f34157df-b34e-4d69-a8bd-d8cffb2e5667%40googlegroups.com<https://groups.google.com/d/msgid/elasticsearch/f34157df-b34e-4d69-a8bd-d8cffb2e5667%40googlegroups.com?utm_medium=email&utm_source=footer> >> . >> For more options, visit https://groups.google.com/d/optout. >> > > -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/8bca4439-5c70-48b6-b5bd-45631e0a5fb2%40googlegroups.com. For more options, visit https://groups.google.com/d/optout.
