This looks very interesting, thanks.
On Tuesday, May 13, 2014 11:38:27 AM UTC+1, Mark Harwood wrote: > > This API should give an indication on any backlog in processing the > cluster state: > http://www.elasticsearch.org/guide/en/elasticsearch/reference/current/cluster-pending.html > > > > On Tuesday, May 13, 2014 11:29:20 AM UTC+1, Paul wrote: >> >> Ok, do you know if there are clear indicators when limits are being >> reached? >> >> We don't see errors in the logs (apart from the 30s timeout) but if there >> are system or ES provided metrics that we can track to know when we need to >> scale it would be really useful. >> >> >> Thanks, >> >> Paul. >> >> >> >> On Tuesday, May 13, 2014 11:24:06 AM UTC+1, Mark Walkom wrote: >>> >>> Empty or not, there is still metadata that ES needs to maintain in the >>> cluster state. So the more indexes you have open the bigger that is and the >>> more resources required to track it. >>> >>> Regards, >>> Mark Walkom >>> >>> Infrastructure Engineer >>> Campaign Monitor >>> email: ma...@campaignmonitor.com >>> web: www.campaignmonitor.com >>> >>> >>> On 13 May 2014 20:16, Paul <codi...@gmail.com> wrote: >>> >>>> In testing and replicating the issue, this slow down has been seen >>>> occurring with empty indices. >>>> >>>> The running cluster is at present ~100 GB across 2,200 Indices with a >>>> total of 13,500 shards and ~430,000,000 documents. >>>> >>>> We have 7GB RAM and 5GB heap on the data nodes - haven't looked overly >>>> carefully but don't think the heap is maxing out on any of the nodes when >>>> this occurs. >>>> >>>> >>>> Thanks, >>>> >>>> Paul. >>>> >>>> >>>> On Tuesday, May 13, 2014 11:02:32 AM UTC+1, Mark Walkom wrote: >>>>> >>>>> Sounds like the inevitable "add more nodes" situation. >>>>> >>>>> How much RAM on each node, how big is your data set? >>>>> >>>>> Regards, >>>>> Mark Walkom >>>>> >>>>> Infrastructure Engineer >>>>> Campaign Monitor >>>>> email: ma...@campaignmonitor.com >>>>> web: www.campaignmonitor.com >>>>> >>>>> >>>>> On 13 May 2014 19:59, Paul <codi...@gmail.com> wrote: >>>>> >>>>>> We are seeing a slow down in shard initialization speed as the number >>>>>> of shards/indices grows in our cluster. >>>>>> >>>>>> With 0-100's of indices/shards existing in the cluster a new bulk >>>>>> creation of indices up the 100's at a time is fine, we see them pass >>>>>> through the states and get a green cluster in a reasonable amount of >>>>>> time. >>>>>> >>>>>> As the total cluster size grows to 1000+ indices (3000+ shards) we >>>>>> begin to notice that the first rounds of initialization take longer to >>>>>> process, it seems to speed up after the first few batches, but this slow >>>>>> down leads to "failed to process cluster event (create-index >>>>>> [index_1112], cause [auto(bulk api)]) within 30s" type messages in the >>>>>> Master logs - the indices are eventually created. >>>>>> >>>>>> >>>>>> Has anyone else experienced this? (did you find the cause / way to >>>>>> fix?) >>>>>> >>>>>> Is this somewhat expected behaviour? - are we approaching something >>>>>> incorrectly? (there are 3 data nodes involved, with 3 shards per index) >>>>>> >>>>>> -- >>>>>> You received this message because you are subscribed to the Google >>>>>> Groups "elasticsearch" group. >>>>>> To unsubscribe from this group and stop receiving emails from it, >>>>>> send an email to elasticsearc...@googlegroups.com. >>>>>> To view this discussion on the web visit https://groups.google.com/d/ >>>>>> msgid/elasticsearch/f34157df-b34e-4d69-a8bd-d8cffb2e5667% >>>>>> 40googlegroups.com<https://groups.google.com/d/msgid/elasticsearch/f34157df-b34e-4d69-a8bd-d8cffb2e5667%40googlegroups.com?utm_medium=email&utm_source=footer> >>>>>> . >>>>>> For more options, visit https://groups.google.com/d/optout. >>>>>> >>>>> >>>>> -- >>>> You received this message because you are subscribed to the Google >>>> Groups "elasticsearch" group. >>>> To unsubscribe from this group and stop receiving emails from it, send >>>> an email to elasticsearc...@googlegroups.com. >>>> To view this discussion on the web visit >>>> https://groups.google.com/d/msgid/elasticsearch/8bca4439-5c70-48b6-b5bd-45631e0a5fb2%40googlegroups.com<https://groups.google.com/d/msgid/elasticsearch/8bca4439-5c70-48b6-b5bd-45631e0a5fb2%40googlegroups.com?utm_medium=email&utm_source=footer> >>>> . >>>> For more options, visit https://groups.google.com/d/optout. >>>> >>> >>> -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/dbdfe1ea-7b1b-4e65-bcdb-251aceab1fe0%40googlegroups.com. For more options, visit https://groups.google.com/d/optout.
