So is this still happening with 1.4.2? Here's the ticket. Looks like the fix was supposed to be in 1.4.1
https://github.com/elasticsearch/elasticsearch/issues/8538 On Thu, Jan 15, 2015 at 10:55 AM, Matías Waisgold <mwaisg...@gmail.com> wrote: > Great, thank you. We are creating another cluster with more disk space to > avoid this situations. > By any chance do you have the link to the issue? > > 2015-01-15 13:26 GMT-03:00 Kimbro Staken <ksta...@kstaken.com>: > >> I've experienced what you're describing. I called it a "shard relocation >> storm" and it's really tough to get under control. I opened a ticket on the >> issue and a fix was supposedly included in 1.4.2. What version are you >> running? >> >> If you want to truly manually manage this situation you could set >> cluster.routing.allocation.disk.threshold_enabled to false but that will >> likely cause other issues. I ended up just setting >> cluster.routing.allocation.disk.watermark.high to a really low value and >> actively managed shard allocations to prevent nodes from getting anywhere >> near that value. This is tricky as the way ES allocates shards it can >> easily run nodes out of disk if you're regularly creating new indices and >> those grow rapidly. >> >> Kimbro >> >> On Thu, Jan 15, 2015 at 6:14 AM, Matías Waisgold <mwaisg...@gmail.com> >> wrote: >> >>> Yes, I've seen that but the problem is that when the threshold is >>> reached it removes all shards from the server instead of just removing 1 >>> and balance. And when that happens the cluster starts to move shards over >>> everywhere and it never stops. >>> >>> Another problem we are having is that in the file storage we see data >>> from shards that are not assigned to itself so it can´t allocate anything >>> in this dirty state. >>> >>> 2015-01-15 0:09 GMT-03:00 Mark Walkom <markwal...@gmail.com>: >>> >>>> You could do this, but it's a lot of manual overhead to have to deal >>>> with. >>>> However ES does have some disk space awareness during allocation, take >>>> a look at >>>> http://www.elasticsearch.org/guide/en/elasticsearch/reference/master/index-modules-allocation.html#disk >>>> >>>> On 15 January 2015 at 10:57, Matías Waisgold <mwaisg...@gmail.com> >>>> wrote: >>>> >>>>> Hi is there any setting that I can put to ES that it automatically >>>>> assigns shards that are unassigned but never ever rebalance the cluster? >>>>> I´ve found several issues when rebalancing and prefer to do it >>>>> manually. >>>>> If I set cluster.routing.allocation.enable to "none" nothing happens. >>>>> If I set it to "all" then it starts rebalancing. >>>>> >>>>> Is it ok to combine cluster.routing.allocation.allow_rebalance to >>>>> "none" and cluster.routing.allocation.enable to "all". >>>>> >>>>> The issue is mainly because we are running low on disk and when that >>>>> happens elasticsearch removes all shards from an instance, that doesn´t >>>>> care about cluster.routing.allocation.cluster_concurrent_rebalance and >>>>> starts moving shards like crazy around the entire cluster, filling the >>>>> storage on other instances in the way that it will never stop balancing. >>>>> >>>>> Kind regards >>>>> >>>>> -- >>>>> You received this message because you are subscribed to the Google >>>>> Groups "elasticsearch" group. >>>>> To unsubscribe from this group and stop receiving emails from it, send >>>>> an email to elasticsearch+unsubscr...@googlegroups.com. >>>>> To view this discussion on the web visit >>>>> https://groups.google.com/d/msgid/elasticsearch/666a4d70-2497-4a2b-8c5e-774c7d0617b7%40googlegroups.com >>>>> <https://groups.google.com/d/msgid/elasticsearch/666a4d70-2497-4a2b-8c5e-774c7d0617b7%40googlegroups.com?utm_medium=email&utm_source=footer> >>>>> . >>>>> For more options, visit https://groups.google.com/d/optout. >>>>> >>>> >>>> -- >>>> You received this message because you are subscribed to a topic in the >>>> Google Groups "elasticsearch" group. >>>> To unsubscribe from this topic, visit >>>> https://groups.google.com/d/topic/elasticsearch/CHqlig1M-T0/unsubscribe >>>> . >>>> To unsubscribe from this group and all its topics, send an email to >>>> elasticsearch+unsubscr...@googlegroups.com. >>>> To view this discussion on the web visit >>>> https://groups.google.com/d/msgid/elasticsearch/CAEYi1X8KXKpmnAPWvr8a_Mgny75KkkKxRFP_bJVhQL20bhR0UQ%40mail.gmail.com >>>> <https://groups.google.com/d/msgid/elasticsearch/CAEYi1X8KXKpmnAPWvr8a_Mgny75KkkKxRFP_bJVhQL20bhR0UQ%40mail.gmail.com?utm_medium=email&utm_source=footer> >>>> . >>>> >>>> For more options, visit https://groups.google.com/d/optout. >>>> >>> >>> -- >>> You received this message because you are subscribed to the Google >>> Groups "elasticsearch" group. >>> To unsubscribe from this group and stop receiving emails from it, send >>> an email to elasticsearch+unsubscr...@googlegroups.com. >>> To view this discussion on the web visit >>> https://groups.google.com/d/msgid/elasticsearch/CAMaTqYqFmk8t7couOmYEyPYNZPKepT8nKVrCM6fvSPW0CUjMwA%40mail.gmail.com >>> <https://groups.google.com/d/msgid/elasticsearch/CAMaTqYqFmk8t7couOmYEyPYNZPKepT8nKVrCM6fvSPW0CUjMwA%40mail.gmail.com?utm_medium=email&utm_source=footer> >>> . >>> >>> For more options, visit https://groups.google.com/d/optout. >>> >> >> -- >> You received this message because you are subscribed to a topic in the >> Google Groups "elasticsearch" group. >> To unsubscribe from this topic, visit >> https://groups.google.com/d/topic/elasticsearch/CHqlig1M-T0/unsubscribe. >> To unsubscribe from this group and all its topics, send an email to >> elasticsearch+unsubscr...@googlegroups.com. >> To view this discussion on the web visit >> https://groups.google.com/d/msgid/elasticsearch/CAA0DmXaW8AdZJhGPGTRqD%3DYCSQ%2B2JdM-oGGpxkRgi0BZLOw2rg%40mail.gmail.com >> <https://groups.google.com/d/msgid/elasticsearch/CAA0DmXaW8AdZJhGPGTRqD%3DYCSQ%2B2JdM-oGGpxkRgi0BZLOw2rg%40mail.gmail.com?utm_medium=email&utm_source=footer> >> . >> >> For more options, visit https://groups.google.com/d/optout. >> > > -- > You received this message because you are subscribed to the Google Groups > "elasticsearch" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to elasticsearch+unsubscr...@googlegroups.com. > To view this discussion on the web visit > https://groups.google.com/d/msgid/elasticsearch/CAMaTqYq%2B_S1_CaPJ-_HN8F%3DizX9VjKmF_W5dhDFpAmS6kxw0WA%40mail.gmail.com > <https://groups.google.com/d/msgid/elasticsearch/CAMaTqYq%2B_S1_CaPJ-_HN8F%3DizX9VjKmF_W5dhDFpAmS6kxw0WA%40mail.gmail.com?utm_medium=email&utm_source=footer> > . > > For more options, visit https://groups.google.com/d/optout. > -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/CAA0DmXar33evahvzhzbQ2zgO_kv%3Du4YtMaUMfZcj9MNJWC%3D_MA%40mail.gmail.com. For more options, visit https://groups.google.com/d/optout.
