Re: cluster.routing.allocation.enable behavior (sticky shard allocation not working as expected)
On Tuesday, July 29, 2014 3:27:13 PM UTC-4, Ivan Brusic wrote: Have you changed your gateway settings? http://www.elasticsearch.org/guide/en/elasticsearch/reference/current/modules-gateway.html#recover-after It still remains a bit of black magic to me. Sometimes it works, sometimes it does not. Ivan, I have read over that documentation several times and I don't understand how it would help me. By which I mean I honestly don't understand how it would help me - not that I am unwilling to try it. Those settings read like they control when recovery would begin, but my problem isn't that recovery is starting when I don't want it to, it's that when I start it (by setting shard allocation about to all) shards that I'd think would just stick with the node they were previously on, get assigned to other nodes, then ultimately get rebalanced back to the original node. At this point I am finding that for quick restarts, just doing them with no additional prep work allows me to recover in ~30m, vs ~24h. So for now I'm just going to do that. Whatever I am doing wrong here just isn't at all clear to me. Thanks for your advice. If I have misunderstood the settings you pointed me at and you think you can help me understand, I'd be grateful for more information. Andy Cheers, Ivan On Mon, Jul 28, 2014 at 1:52 PM, Andrew Davidoff davi...@qedmf.net javascript: wrote: I guess I'll ask about this once more for now. This happened again today. I set allocation to new_primaries, restarted a node, set allocation back to all, and the cluster is assigning across all nodes the shards that were on the restarted node, and when it's done, which will probably take a day, it'll likely rebalance by moving them back to the original node. I have to assume I'm doing something wrong here. Am I? Thanks for any advice. Andy -- You received this message because you are subscribed to the Google Groups elasticsearch group. To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearc...@googlegroups.com javascript:. To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/fc6ffe88-1a01-452e-b971-2b8fc222cba3%40googlegroups.com https://groups.google.com/d/msgid/elasticsearch/fc6ffe88-1a01-452e-b971-2b8fc222cba3%40googlegroups.com?utm_medium=emailutm_source=footer . For more options, visit https://groups.google.com/d/optout. -- You received this message because you are subscribed to the Google Groups elasticsearch group. To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/b8cf076f-2494-4b0e-b6c3-5a21d1c9f9a1%40googlegroups.com. For more options, visit https://groups.google.com/d/optout.
Re: cluster.routing.allocation.enable behavior (sticky shard allocation not working as expected)
The idea is that the cluster should be delayed when a cluster rebalance occurs, but even with these settings, I often find that shards are moved immediately. Are you using the default stores throttling settings? I found them to be quite low. Cheers, Ivan On Wed, Jul 30, 2014 at 6:02 AM, Andrew Davidoff david...@qedmf.net wrote: On Tuesday, July 29, 2014 3:27:13 PM UTC-4, Ivan Brusic wrote: Have you changed your gateway settings? http://www. elasticsearch.org/guide/en/elasticsearch/reference/ current/modules-gateway.html#recover-after It still remains a bit of black magic to me. Sometimes it works, sometimes it does not. Ivan, I have read over that documentation several times and I don't understand how it would help me. By which I mean I honestly don't understand how it would help me - not that I am unwilling to try it. Those settings read like they control when recovery would begin, but my problem isn't that recovery is starting when I don't want it to, it's that when I start it (by setting shard allocation about to all) shards that I'd think would just stick with the node they were previously on, get assigned to other nodes, then ultimately get rebalanced back to the original node. At this point I am finding that for quick restarts, just doing them with no additional prep work allows me to recover in ~30m, vs ~24h. So for now I'm just going to do that. Whatever I am doing wrong here just isn't at all clear to me. Thanks for your advice. If I have misunderstood the settings you pointed me at and you think you can help me understand, I'd be grateful for more information. Andy Cheers, Ivan On Mon, Jul 28, 2014 at 1:52 PM, Andrew Davidoff davi...@qedmf.net wrote: I guess I'll ask about this once more for now. This happened again today. I set allocation to new_primaries, restarted a node, set allocation back to all, and the cluster is assigning across all nodes the shards that were on the restarted node, and when it's done, which will probably take a day, it'll likely rebalance by moving them back to the original node. I have to assume I'm doing something wrong here. Am I? Thanks for any advice. Andy -- You received this message because you are subscribed to the Google Groups elasticsearch group. To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearc...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/ msgid/elasticsearch/fc6ffe88-1a01-452e-b971-2b8fc222cba3% 40googlegroups.com https://groups.google.com/d/msgid/elasticsearch/fc6ffe88-1a01-452e-b971-2b8fc222cba3%40googlegroups.com?utm_medium=emailutm_source=footer . For more options, visit https://groups.google.com/d/optout. -- You received this message because you are subscribed to the Google Groups elasticsearch group. To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/b8cf076f-2494-4b0e-b6c3-5a21d1c9f9a1%40googlegroups.com https://groups.google.com/d/msgid/elasticsearch/b8cf076f-2494-4b0e-b6c3-5a21d1c9f9a1%40googlegroups.com?utm_medium=emailutm_source=footer . For more options, visit https://groups.google.com/d/optout. -- You received this message because you are subscribed to the Google Groups elasticsearch group. To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/CALY%3DcQBnsTUiXYZVyUgnuU6k7nN8NJiwh2UK_JPTOhEVa8BCRA%40mail.gmail.com. For more options, visit https://groups.google.com/d/optout.
Re: cluster.routing.allocation.enable behavior (sticky shard allocation not working as expected)
I've seen this as well Ivan, and have also had a few people on IRC comment on the same thing - shards that are local are not simply being initialised, but being reallocated elsewhere. Regards, Mark Walkom Infrastructure Engineer Campaign Monitor email: ma...@campaignmonitor.com web: www.campaignmonitor.com On 31 July 2014 03:54, Ivan Brusic i...@brusic.com wrote: The idea is that the cluster should be delayed when a cluster rebalance occurs, but even with these settings, I often find that shards are moved immediately. Are you using the default stores throttling settings? I found them to be quite low. Cheers, Ivan On Wed, Jul 30, 2014 at 6:02 AM, Andrew Davidoff david...@qedmf.net wrote: On Tuesday, July 29, 2014 3:27:13 PM UTC-4, Ivan Brusic wrote: Have you changed your gateway settings? http://www. elasticsearch.org/guide/en/elasticsearch/reference/ current/modules-gateway.html#recover-after It still remains a bit of black magic to me. Sometimes it works, sometimes it does not. Ivan, I have read over that documentation several times and I don't understand how it would help me. By which I mean I honestly don't understand how it would help me - not that I am unwilling to try it. Those settings read like they control when recovery would begin, but my problem isn't that recovery is starting when I don't want it to, it's that when I start it (by setting shard allocation about to all) shards that I'd think would just stick with the node they were previously on, get assigned to other nodes, then ultimately get rebalanced back to the original node. At this point I am finding that for quick restarts, just doing them with no additional prep work allows me to recover in ~30m, vs ~24h. So for now I'm just going to do that. Whatever I am doing wrong here just isn't at all clear to me. Thanks for your advice. If I have misunderstood the settings you pointed me at and you think you can help me understand, I'd be grateful for more information. Andy Cheers, Ivan On Mon, Jul 28, 2014 at 1:52 PM, Andrew Davidoff davi...@qedmf.net wrote: I guess I'll ask about this once more for now. This happened again today. I set allocation to new_primaries, restarted a node, set allocation back to all, and the cluster is assigning across all nodes the shards that were on the restarted node, and when it's done, which will probably take a day, it'll likely rebalance by moving them back to the original node. I have to assume I'm doing something wrong here. Am I? Thanks for any advice. Andy -- You received this message because you are subscribed to the Google Groups elasticsearch group. To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearc...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/ msgid/elasticsearch/fc6ffe88-1a01-452e-b971-2b8fc222cba3% 40googlegroups.com https://groups.google.com/d/msgid/elasticsearch/fc6ffe88-1a01-452e-b971-2b8fc222cba3%40googlegroups.com?utm_medium=emailutm_source=footer . For more options, visit https://groups.google.com/d/optout. -- You received this message because you are subscribed to the Google Groups elasticsearch group. To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/b8cf076f-2494-4b0e-b6c3-5a21d1c9f9a1%40googlegroups.com https://groups.google.com/d/msgid/elasticsearch/b8cf076f-2494-4b0e-b6c3-5a21d1c9f9a1%40googlegroups.com?utm_medium=emailutm_source=footer . For more options, visit https://groups.google.com/d/optout. -- You received this message because you are subscribed to the Google Groups elasticsearch group. To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/CALY%3DcQBnsTUiXYZVyUgnuU6k7nN8NJiwh2UK_JPTOhEVa8BCRA%40mail.gmail.com https://groups.google.com/d/msgid/elasticsearch/CALY%3DcQBnsTUiXYZVyUgnuU6k7nN8NJiwh2UK_JPTOhEVa8BCRA%40mail.gmail.com?utm_medium=emailutm_source=footer . For more options, visit https://groups.google.com/d/optout. -- You received this message because you are subscribed to the Google Groups elasticsearch group. To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/CAEM624ZRRCJNbVBcaaJ_0dsSGou_xhx5g8wDGvdeZKV6XQ3qsA%40mail.gmail.com. For more options, visit https://groups.google.com/d/optout.
Re: cluster.routing.allocation.enable behavior (sticky shard allocation not working as expected)
Have you changed your gateway settings? http://www.elasticsearch.org/guide/en/elasticsearch/reference/current/modules-gateway.html#recover-after It still remains a bit of black magic to me. Sometimes it works, sometimes it does not. Cheers, Ivan On Mon, Jul 28, 2014 at 1:52 PM, Andrew Davidoff david...@qedmf.net wrote: I guess I'll ask about this once more for now. This happened again today. I set allocation to new_primaries, restarted a node, set allocation back to all, and the cluster is assigning across all nodes the shards that were on the restarted node, and when it's done, which will probably take a day, it'll likely rebalance by moving them back to the original node. I have to assume I'm doing something wrong here. Am I? Thanks for any advice. Andy -- You received this message because you are subscribed to the Google Groups elasticsearch group. To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/fc6ffe88-1a01-452e-b971-2b8fc222cba3%40googlegroups.com https://groups.google.com/d/msgid/elasticsearch/fc6ffe88-1a01-452e-b971-2b8fc222cba3%40googlegroups.com?utm_medium=emailutm_source=footer . For more options, visit https://groups.google.com/d/optout. -- You received this message because you are subscribed to the Google Groups elasticsearch group. To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/CALY%3DcQAewGBdDpXMBX1%2BZOMmsu%3D5JE1E7jJc3XFm1cNqHHYQGw%40mail.gmail.com. For more options, visit https://groups.google.com/d/optout.
Re: cluster.routing.allocation.enable behavior (sticky shard allocation not working as expected)
I guess I'll ask about this once more for now. This happened again today. I set allocation to new_primaries, restarted a node, set allocation back to all, and the cluster is assigning across all nodes the shards that were on the restarted node, and when it's done, which will probably take a day, it'll likely rebalance by moving them back to the original node. I have to assume I'm doing something wrong here. Am I? Thanks for any advice. Andy -- You received this message because you are subscribed to the Google Groups elasticsearch group. To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/fc6ffe88-1a01-452e-b971-2b8fc222cba3%40googlegroups.com. For more options, visit https://groups.google.com/d/optout.
RE: cluster.routing.allocation.enable behavior (sticky shard allocation not working as expected)
Andrew, Have you found a solution (or explaination) to your issue ? We are using elasticsearch 1.1.1, what about you ? -- Grégoire -- You received this message because you are subscribed to the Google Groups elasticsearch group. To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/abb533aca6be4983a05269a35b72f7ba%40FRDCPWEXCH002.criteois.lan. For more options, visit https://groups.google.com/d/optout.
Re: cluster.routing.allocation.enable behavior (sticky shard allocation not working as expected)
On Mon, Jul 7, 2014 at 4:16 AM, Grégoire Seux g.s...@criteo.com wrote: Andrew, Have you found a solution (or explaination) to your issue ? We are using elasticsearch 1.1.1, what about you ? Hi, I haven't learned anything new. To be clear about my problem, I am aware that I must re-enable routing after having disabled it. My issue is that I expect all the UNASSIGNED shards to go back to the same node, but some do not, only to get rebalanced back there later. I am running elasticsearch 1.2.1. Andy -- You received this message because you are subscribed to the Google Groups elasticsearch group. To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/CAJLXCZSM9MKC0MN_aaivrFwmE6U-%3Dmkqobun946XcnPk2BCcHw%40mail.gmail.com. For more options, visit https://groups.google.com/d/optout.
Re: cluster.routing.allocation.enable behavior (sticky shard allocation not working as expected)
On Wed, Jul 02, 2014 at 05:43:26AM -0700, Andrew Davidoff wrote:
How can I avoid this, and make shards on a restarted node come back on the
same node?
Hello,
I have exactly the same issue.
My objective is to make a rolling restart script which wait for green
cluster state before restarting a node.
I use:
curl -XPUT -s $host:$port/_cluster/settings -d
'{transient:{cluster.routing.allocation.enable: new_primaries}}'
to allow the cluster to work (and be able to create indices) during
restart.
But same issue: node is back up but nothing happen until I enable all
allocation again
I have gone through elasticsearch documentation related to recovery,
gateway, cluster settings without finding any parameters to activate or
configure this initial recovery of local indices.
--
Grégoire
--
You received this message because you are subscribed to the Google Groups
elasticsearch group.
To unsubscribe from this group and stop receiving emails from it, send an email
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/20140702142754.GA2140%40criteo-scalasto.criteo.prod.
For more options, visit https://groups.google.com/d/optout.
