Great, thank you. We are creating another cluster with more disk space to avoid this situations. By any chance do you have the link to the issue?
2015-01-15 13:26 GMT-03:00 Kimbro Staken <ksta...@kstaken.com>: > I've experienced what you're describing. I called it a "shard relocation > storm" and it's really tough to get under control. I opened a ticket on the > issue and a fix was supposedly included in 1.4.2. What version are you > running? > > If you want to truly manually manage this situation you could set > cluster.routing.allocation.disk.threshold_enabled to false but that will > likely cause other issues. I ended up just setting > cluster.routing.allocation.disk.watermark.high to a really low value and > actively managed shard allocations to prevent nodes from getting anywhere > near that value. This is tricky as the way ES allocates shards it can > easily run nodes out of disk if you're regularly creating new indices and > those grow rapidly. > > Kimbro > > On Thu, Jan 15, 2015 at 6:14 AM, Matías Waisgold <mwaisg...@gmail.com> > wrote: > >> Yes, I've seen that but the problem is that when the threshold is reached >> it removes all shards from the server instead of just removing 1 and >> balance. And when that happens the cluster starts to move shards over >> everywhere and it never stops. >> >> Another problem we are having is that in the file storage we see data >> from shards that are not assigned to itself so it can´t allocate anything >> in this dirty state. >> >> 2015-01-15 0:09 GMT-03:00 Mark Walkom <markwal...@gmail.com>: >> >>> You could do this, but it's a lot of manual overhead to have to deal >>> with. >>> However ES does have some disk space awareness during allocation, take a >>> look at >>> http://www.elasticsearch.org/guide/en/elasticsearch/reference/master/index-modules-allocation.html#disk >>> >>> On 15 January 2015 at 10:57, Matías Waisgold <mwaisg...@gmail.com> >>> wrote: >>> >>>> Hi is there any setting that I can put to ES that it automatically >>>> assigns shards that are unassigned but never ever rebalance the cluster? >>>> I´ve found several issues when rebalancing and prefer to do it manually. >>>> If I set cluster.routing.allocation.enable to "none" nothing happens. >>>> If I set it to "all" then it starts rebalancing. >>>> >>>> Is it ok to combine cluster.routing.allocation.allow_rebalance to >>>> "none" and cluster.routing.allocation.enable to "all". >>>> >>>> The issue is mainly because we are running low on disk and when that >>>> happens elasticsearch removes all shards from an instance, that doesn´t >>>> care about cluster.routing.allocation.cluster_concurrent_rebalance and >>>> starts moving shards like crazy around the entire cluster, filling the >>>> storage on other instances in the way that it will never stop balancing. >>>> >>>> Kind regards >>>> >>>> -- >>>> You received this message because you are subscribed to the Google >>>> Groups "elasticsearch" group. >>>> To unsubscribe from this group and stop receiving emails from it, send >>>> an email to elasticsearch+unsubscr...@googlegroups.com. >>>> To view this discussion on the web visit >>>> https://groups.google.com/d/msgid/elasticsearch/666a4d70-2497-4a2b-8c5e-774c7d0617b7%40googlegroups.com >>>> <https://groups.google.com/d/msgid/elasticsearch/666a4d70-2497-4a2b-8c5e-774c7d0617b7%40googlegroups.com?utm_medium=email&utm_source=footer> >>>> . >>>> For more options, visit https://groups.google.com/d/optout. >>>> >>> >>> -- >>> You received this message because you are subscribed to a topic in the >>> Google Groups "elasticsearch" group. >>> To unsubscribe from this topic, visit >>> https://groups.google.com/d/topic/elasticsearch/CHqlig1M-T0/unsubscribe. >>> To unsubscribe from this group and all its topics, send an email to >>> elasticsearch+unsubscr...@googlegroups.com. >>> To view this discussion on the web visit >>> https://groups.google.com/d/msgid/elasticsearch/CAEYi1X8KXKpmnAPWvr8a_Mgny75KkkKxRFP_bJVhQL20bhR0UQ%40mail.gmail.com >>> <https://groups.google.com/d/msgid/elasticsearch/CAEYi1X8KXKpmnAPWvr8a_Mgny75KkkKxRFP_bJVhQL20bhR0UQ%40mail.gmail.com?utm_medium=email&utm_source=footer> >>> . >>> >>> For more options, visit https://groups.google.com/d/optout. >>> >> >> -- >> You received this message because you are subscribed to the Google Groups >> "elasticsearch" group. >> To unsubscribe from this group and stop receiving emails from it, send an >> email to elasticsearch+unsubscr...@googlegroups.com. >> To view this discussion on the web visit >> https://groups.google.com/d/msgid/elasticsearch/CAMaTqYqFmk8t7couOmYEyPYNZPKepT8nKVrCM6fvSPW0CUjMwA%40mail.gmail.com >> <https://groups.google.com/d/msgid/elasticsearch/CAMaTqYqFmk8t7couOmYEyPYNZPKepT8nKVrCM6fvSPW0CUjMwA%40mail.gmail.com?utm_medium=email&utm_source=footer> >> . >> >> For more options, visit https://groups.google.com/d/optout. >> > > -- > You received this message because you are subscribed to a topic in the > Google Groups "elasticsearch" group. > To unsubscribe from this topic, visit > https://groups.google.com/d/topic/elasticsearch/CHqlig1M-T0/unsubscribe. > To unsubscribe from this group and all its topics, send an email to > elasticsearch+unsubscr...@googlegroups.com. > To view this discussion on the web visit > https://groups.google.com/d/msgid/elasticsearch/CAA0DmXaW8AdZJhGPGTRqD%3DYCSQ%2B2JdM-oGGpxkRgi0BZLOw2rg%40mail.gmail.com > <https://groups.google.com/d/msgid/elasticsearch/CAA0DmXaW8AdZJhGPGTRqD%3DYCSQ%2B2JdM-oGGpxkRgi0BZLOw2rg%40mail.gmail.com?utm_medium=email&utm_source=footer> > . > > For more options, visit https://groups.google.com/d/optout. > -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/CAMaTqYq%2B_S1_CaPJ-_HN8F%3DizX9VjKmF_W5dhDFpAmS6kxw0WA%40mail.gmail.com. For more options, visit https://groups.google.com/d/optout.
