Re: cluster.routing.allocation.enable behavior (sticky shard allocation not working as expected)

2014-07-30 Thread Mark Walkom 
I've seen this as well Ivan, and have also had a few people on IRC comment
on the same thing - shards that are local are not simply being initialised,
but being reallocated elsewhere.

Regards,
Mark Walkom

Infrastructure Engineer
Campaign Monitor
email: ma...@campaignmonitor.com
web: www.campaignmonitor.com


On 31 July 2014 03:54, Ivan Brusic  wrote:

> The idea is that the cluster should be delayed when a cluster rebalance
> occurs, but even with these settings, I often find that shards are moved
> immediately.
>
> Are you using the default stores throttling settings? I found them to be
> quite low.
>
> Cheers,
>
> Ivan
>
>
> On Wed, Jul 30, 2014 at 6:02 AM, Andrew Davidoff 
> wrote:
>
>>
>>
>> On Tuesday, July 29, 2014 3:27:13 PM UTC-4, Ivan Brusic wrote:
>>>
>>> Have you changed your gateway settings? http://www.
>>> elasticsearch.org/guide/en/elasticsearch/reference/
>>> current/modules-gateway.html#recover-after
>>>
>>> It still remains a bit of black magic to me. Sometimes it works,
>>> sometimes it does not.
>>>
>>
>> Ivan,
>>
>> I have read over that documentation several times and I don't understand
>> how it would help me. By which I mean I honestly don't understand how it
>> would help me - not that I am unwilling to try it. Those settings read like
>> they control when recovery would begin, but my problem isn't that recovery
>> is starting when I don't want it to, it's that when I start it (by setting
>> shard allocation about to "all") shards that I'd think would just stick
>> with the node they were previously on, get assigned to other nodes, then
>> ultimately get rebalanced back to the original node.
>>
>> At this point I am finding that for quick restarts, just doing them with
>> no additional prep work allows me to recover in ~30m, vs ~24h. So for now
>> I'm just going to do that. Whatever I am doing wrong here just isn't at all
>> clear to me.
>>
>> Thanks for your advice. If I have misunderstood the settings you pointed
>> me at and you think you can help me understand, I'd be grateful for more
>> information.
>>
>> Andy
>>
>>
>>
>>> Cheers,
>>>
>>> Ivan
>>>
>>>
>>> On Mon, Jul 28, 2014 at 1:52 PM, Andrew Davidoff 
>>> wrote:
>>>
  I guess I'll ask about this once more for now. This happened again
 today. I set allocation to new_primaries, restarted a node, set allocation
 back to all, and the cluster is assigning across all nodes the shards that
 were on the restarted node, and when it's done, which will probably take a
 day, it'll likely rebalance by moving them back to the original node. I
 have to assume I'm doing something wrong here. Am I?

 Thanks for any advice.
 Andy

 --
 You received this message because you are subscribed to the Google
 Groups "elasticsearch" group.
 To unsubscribe from this group and stop receiving emails from it, send
 an email to elasticsearc...@googlegroups.com.
 To view this discussion on the web visit https://groups.google.com/d/
 msgid/elasticsearch/fc6ffe88-1a01-452e-b971-2b8fc222cba3%
 40googlegroups.com
 
 .

 For more options, visit https://groups.google.com/d/optout.

>>>
>>>  --
>> You received this message because you are subscribed to the Google Groups
>> "elasticsearch" group.
>> To unsubscribe from this group and stop receiving emails from it, send an
>> email to elasticsearch+unsubscr...@googlegroups.com.
>> To view this discussion on the web visit
>> https://groups.google.com/d/msgid/elasticsearch/b8cf076f-2494-4b0e-b6c3-5a21d1c9f9a1%40googlegroups.com
>> 
>> .
>>
>> For more options, visit https://groups.google.com/d/optout.
>>
>
>  --
> You received this message because you are subscribed to the Google Groups
> "elasticsearch" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to elasticsearch+unsubscr...@googlegroups.com.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/elasticsearch/CALY%3DcQBnsTUiXYZVyUgnuU6k7nN8NJiwh2UK_JPTOhEVa8BCRA%40mail.gmail.com
> 
> .
>
> For more options, visit https://groups.google.com/d/optout.
>

-- 
You received this message because you are subscribed to the Google Groups 
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/elasticsearch/CAEM624ZRRCJNbVBcaaJ_0dsSGou_xhx5g8wDGvdeZKV6XQ3qsA%40mail.gmail.com.
For more options, visit https://groups.go

Re: cluster.routing.allocation.enable behavior (sticky shard allocation not working as expected)

2014-07-30 Thread Ivan Brusic 
The idea is that the cluster should be delayed when a cluster rebalance
occurs, but even with these settings, I often find that shards are moved
immediately.

Are you using the default stores throttling settings? I found them to be
quite low.

Cheers,

Ivan


On Wed, Jul 30, 2014 at 6:02 AM, Andrew Davidoff  wrote:

>
>
> On Tuesday, July 29, 2014 3:27:13 PM UTC-4, Ivan Brusic wrote:
>>
>> Have you changed your gateway settings? http://www.
>> elasticsearch.org/guide/en/elasticsearch/reference/
>> current/modules-gateway.html#recover-after
>>
>> It still remains a bit of black magic to me. Sometimes it works,
>> sometimes it does not.
>>
>
> Ivan,
>
> I have read over that documentation several times and I don't understand
> how it would help me. By which I mean I honestly don't understand how it
> would help me - not that I am unwilling to try it. Those settings read like
> they control when recovery would begin, but my problem isn't that recovery
> is starting when I don't want it to, it's that when I start it (by setting
> shard allocation about to "all") shards that I'd think would just stick
> with the node they were previously on, get assigned to other nodes, then
> ultimately get rebalanced back to the original node.
>
> At this point I am finding that for quick restarts, just doing them with
> no additional prep work allows me to recover in ~30m, vs ~24h. So for now
> I'm just going to do that. Whatever I am doing wrong here just isn't at all
> clear to me.
>
> Thanks for your advice. If I have misunderstood the settings you pointed
> me at and you think you can help me understand, I'd be grateful for more
> information.
>
> Andy
>
>
>
>> Cheers,
>>
>> Ivan
>>
>>
>> On Mon, Jul 28, 2014 at 1:52 PM, Andrew Davidoff 
>> wrote:
>>
>>> I guess I'll ask about this once more for now. This happened again
>>> today. I set allocation to new_primaries, restarted a node, set allocation
>>> back to all, and the cluster is assigning across all nodes the shards that
>>> were on the restarted node, and when it's done, which will probably take a
>>> day, it'll likely rebalance by moving them back to the original node. I
>>> have to assume I'm doing something wrong here. Am I?
>>>
>>> Thanks for any advice.
>>> Andy
>>>
>>> --
>>> You received this message because you are subscribed to the Google
>>> Groups "elasticsearch" group.
>>> To unsubscribe from this group and stop receiving emails from it, send
>>> an email to elasticsearc...@googlegroups.com.
>>> To view this discussion on the web visit https://groups.google.com/d/
>>> msgid/elasticsearch/fc6ffe88-1a01-452e-b971-2b8fc222cba3%
>>> 40googlegroups.com
>>> 
>>> .
>>>
>>> For more options, visit https://groups.google.com/d/optout.
>>>
>>
>>  --
> You received this message because you are subscribed to the Google Groups
> "elasticsearch" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to elasticsearch+unsubscr...@googlegroups.com.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/elasticsearch/b8cf076f-2494-4b0e-b6c3-5a21d1c9f9a1%40googlegroups.com
> 
> .
>
> For more options, visit https://groups.google.com/d/optout.
>

-- 
You received this message because you are subscribed to the Google Groups 
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/elasticsearch/CALY%3DcQBnsTUiXYZVyUgnuU6k7nN8NJiwh2UK_JPTOhEVa8BCRA%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

Re: cluster.routing.allocation.enable behavior (sticky shard allocation not working as expected)

2014-07-30 Thread Andrew Davidoff 


On Tuesday, July 29, 2014 3:27:13 PM UTC-4, Ivan Brusic wrote:
>
> Have you changed your gateway settings? 
> http://www.elasticsearch.org/guide/en/elasticsearch/reference/current/modules-gateway.html#recover-after
>
> It still remains a bit of black magic to me. Sometimes it works, sometimes 
> it does not.
>

Ivan,

I have read over that documentation several times and I don't understand 
how it would help me. By which I mean I honestly don't understand how it 
would help me - not that I am unwilling to try it. Those settings read like 
they control when recovery would begin, but my problem isn't that recovery 
is starting when I don't want it to, it's that when I start it (by setting 
shard allocation about to "all") shards that I'd think would just stick 
with the node they were previously on, get assigned to other nodes, then 
ultimately get rebalanced back to the original node.

At this point I am finding that for quick restarts, just doing them with no 
additional prep work allows me to recover in ~30m, vs ~24h. So for now I'm 
just going to do that. Whatever I am doing wrong here just isn't at all 
clear to me.

Thanks for your advice. If I have misunderstood the settings you pointed me 
at and you think you can help me understand, I'd be grateful for more 
information.

Andy



> Cheers,
>
> Ivan
>
>
> On Mon, Jul 28, 2014 at 1:52 PM, Andrew Davidoff  > wrote:
>
>> I guess I'll ask about this once more for now. This happened again today. 
>> I set allocation to new_primaries, restarted a node, set allocation back to 
>> all, and the cluster is assigning across all nodes the shards that were on 
>> the restarted node, and when it's done, which will probably take a day, 
>> it'll likely rebalance by moving them back to the original node. I have to 
>> assume I'm doing something wrong here. Am I?
>>
>> Thanks for any advice.
>> Andy
>>
>> -- 
>> You received this message because you are subscribed to the Google Groups 
>> "elasticsearch" group.
>> To unsubscribe from this group and stop receiving emails from it, send an 
>> email to elasticsearc...@googlegroups.com .
>> To view this discussion on the web visit 
>> https://groups.google.com/d/msgid/elasticsearch/fc6ffe88-1a01-452e-b971-2b8fc222cba3%40googlegroups.com
>>  
>> 
>> .
>>
>> For more options, visit https://groups.google.com/d/optout.
>>
>
>

-- 
You received this message because you are subscribed to the Google Groups 
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/elasticsearch/b8cf076f-2494-4b0e-b6c3-5a21d1c9f9a1%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: cluster.routing.allocation.enable behavior (sticky shard allocation not working as expected)

2014-07-29 Thread Ivan Brusic 
Have you changed your gateway settings?
http://www.elasticsearch.org/guide/en/elasticsearch/reference/current/modules-gateway.html#recover-after

It still remains a bit of black magic to me. Sometimes it works, sometimes
it does not.

Cheers,

Ivan


On Mon, Jul 28, 2014 at 1:52 PM, Andrew Davidoff  wrote:

> I guess I'll ask about this once more for now. This happened again today.
> I set allocation to new_primaries, restarted a node, set allocation back to
> all, and the cluster is assigning across all nodes the shards that were on
> the restarted node, and when it's done, which will probably take a day,
> it'll likely rebalance by moving them back to the original node. I have to
> assume I'm doing something wrong here. Am I?
>
> Thanks for any advice.
> Andy
>
> --
> You received this message because you are subscribed to the Google Groups
> "elasticsearch" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to elasticsearch+unsubscr...@googlegroups.com.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/elasticsearch/fc6ffe88-1a01-452e-b971-2b8fc222cba3%40googlegroups.com
> 
> .
>
> For more options, visit https://groups.google.com/d/optout.
>

-- 
You received this message because you are subscribed to the Google Groups 
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/elasticsearch/CALY%3DcQAewGBdDpXMBX1%2BZOMmsu%3D5JE1E7jJc3XFm1cNqHHYQGw%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

Re: cluster.routing.allocation.enable behavior (sticky shard allocation not working as expected)

2014-07-28 Thread Andrew Davidoff 
I guess I'll ask about this once more for now. This happened again today. I 
set allocation to new_primaries, restarted a node, set allocation back to 
all, and the cluster is assigning across all nodes the shards that were on 
the restarted node, and when it's done, which will probably take a day, 
it'll likely rebalance by moving them back to the original node. I have to 
assume I'm doing something wrong here. Am I?

Thanks for any advice.
Andy

-- 
You received this message because you are subscribed to the Google Groups 
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/elasticsearch/fc6ffe88-1a01-452e-b971-2b8fc222cba3%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: cluster.routing.allocation.enable behavior (sticky shard allocation not working as expected)

2014-07-07 Thread Andrew Davidoff 
On Mon, Jul 7, 2014 at 4:16 AM, Grégoire Seux  wrote:
> Andrew,
>
> Have you found a solution (or explaination) to your issue ?
> We are using elasticsearch 1.1.1, what about you ?

Hi,

I haven't learned anything new. To be clear about my problem, I am
aware that I must re-enable routing after having disabled it. My issue
is that I expect all the UNASSIGNED shards to go back to the same
node, but some do not, only to get rebalanced back there later. I am
running elasticsearch 1.2.1.

Andy

-- 
You received this message because you are subscribed to the Google Groups 
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/elasticsearch/CAJLXCZSM9MKC0MN_aaivrFwmE6U-%3Dmkqobun946XcnPk2BCcHw%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

RE: cluster.routing.allocation.enable behavior (sticky shard allocation not working as expected)

2014-07-07 Thread Grégoire Seux 
Andrew,

Have you found a solution (or explaination) to your issue ?
We are using elasticsearch 1.1.1, what about you ?

-- 
Grégoire

-- 
You received this message because you are subscribed to the Google Groups 
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/elasticsearch/abb533aca6be4983a05269a35b72f7ba%40FRDCPWEXCH002.criteois.lan.
For more options, visit https://groups.google.com/d/optout.

Re: cluster.routing.allocation.enable behavior (sticky shard allocation not working as expected)

2014-07-02 Thread Gregoire Seux 
On Wed, Jul 02, 2014 at 05:43:26AM -0700, Andrew Davidoff wrote:

> How can I avoid this, and make shards on a restarted node come back on the 
> same node?

Hello,

I have exactly the same issue.
My objective is to make a rolling restart script which wait for green
cluster state before restarting a node.
I use:
> curl -XPUT -s $host:$port/_cluster/settings -d 
> '{"transient":{"cluster.routing.allocation.enable": "new_primaries"}}'

to allow the cluster to work (and be able to create indices) during
restart.

But same issue: node is back up but nothing happen until I enable all
allocation again

I have gone through elasticsearch documentation related to recovery,
gateway, cluster settings without finding any parameters to activate or
configure this initial recovery of local indices.

-- 
Grégoire

-- 
You received this message because you are subscribed to the Google Groups 
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/elasticsearch/20140702142754.GA2140%40criteo-scalasto.criteo.prod.
For more options, visit https://groups.google.com/d/optout.