Re: 3 Node Cluster With Nodes Out of Sync

[Yosi Haran]

Thanks! I'll check out the link you sent and see if it'll help...

We're are running version 1.0.0:

"version": {
"number": "1.0.0",
"build_hash": "a46900e9c72c0a623d71b54016357d5f94c8ea32",
"build_timestamp": "2014-02-12T16:18:34Z",
"build_snapshot": false,
"lucene_version": "4.6"
  }



On Tuesday, November 25, 2014 6:41:36 PM UTC+2, Itamar Syn-Hershko wrote:
>
> minimum_master_nodes still doesn't protect you from all possible failure 
> scenarios, see http://aphyr.com/posts/317-call-me-maybe-elasticsearch
>
> What version are you running?
>
> --
>
> Itamar Syn-Hershko
> http://code972.com | @synhershko 
> Freelance Developer & Consultant
> Author of RavenDB in Action 
>
> On Tue, Nov 25, 2014 at 6:37 PM, Yosi Haran  > wrote:
>
>> Hi Guys,
>>
>> We are running a 3 node cluster, and each node returns a different number 
>> of documents when issued a direct HTTP "_count" call.
>>
>> The cluster holds about 150K documents and the differences range from 
>> 30~50 documents, but are still troubling.
>>
>> This shouldn't be a "split brain" problem, since we have set:
>> discovery.zen.minimum_master_nodes: 2
>> We also have a "client" node, but since client nodes are eligible to be 
>> master, I understand that they shouldn't affect the master election process.
>>
>> Any Ideas about why and how this is happening?
>>
>> Thanks!
>>
>>  -- 
>> You received this message because you are subscribed to the Google Groups 
>> "elasticsearch" group.
>> To unsubscribe from this group and stop receiving emails from it, send an 
>> email to elasticsearc...@googlegroups.com .
>> To view this discussion on the web visit 
>> https://groups.google.com/d/msgid/elasticsearch/8ed828c8-fb62-413a-9fe0-91806fcf34e6%40googlegroups.com
>>  
>> 
>> .
>> For more options, visit https://groups.google.com/d/optout.
>>
>
>

-- 
You received this message because you are subscribed to the Google Groups 
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/elasticsearch/ec97ea83-bfef-4831-8af0-9a7a86170124%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: 3 Node Cluster With Nodes Out of Sync

[Yosi Haran]

Thanks! I'll check out the link and let you know if it helped.

We are running version 1.0.0

On Tuesday, November 25, 2014 6:41:36 PM UTC+2, Itamar Syn-Hershko wrote:
>
> minimum_master_nodes still doesn't protect you from all possible failure 
> scenarios, see http://aphyr.com/posts/317-call-me-maybe-elasticsearch
>
> What version are you running?
>
> --
>
> Itamar Syn-Hershko
> http://code972.com | @synhershko 
> Freelance Developer & Consultant
> Author of RavenDB in Action 
>
> On Tue, Nov 25, 2014 at 6:37 PM, Yosi Haran  > wrote:
>
>> Hi Guys,
>>
>> We are running a 3 node cluster, and each node returns a different number 
>> of documents when issued a direct HTTP "_count" call.
>>
>> The cluster holds about 150K documents and the differences range from 
>> 30~50 documents, but are still troubling.
>>
>> This shouldn't be a "split brain" problem, since we have set:
>> discovery.zen.minimum_master_nodes: 2
>> We also have a "client" node, but since client nodes are eligible to be 
>> master, I understand that they shouldn't affect the master election process.
>>
>> Any Ideas about why and how this is happening?
>>
>> Thanks!
>>
>>  -- 
>> You received this message because you are subscribed to the Google Groups 
>> "elasticsearch" group.
>> To unsubscribe from this group and stop receiving emails from it, send an 
>> email to elasticsearc...@googlegroups.com .
>> To view this discussion on the web visit 
>> https://groups.google.com/d/msgid/elasticsearch/8ed828c8-fb62-413a-9fe0-91806fcf34e6%40googlegroups.com
>>  
>> 
>> .
>> For more options, visit https://groups.google.com/d/optout.
>>
>
>

-- 
You received this message because you are subscribed to the Google Groups 
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/elasticsearch/5addc70b-b76c-46d5-b122-cf730827d677%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: 3 Node Cluster With Nodes Out of Sync

[Itamar Syn-Hershko]

If this is replicas only, you should be able to set replica count to 0 and
then after a while back to 2 again

If this is sharded, then no, you'll have to reindex from scratch.

--

Itamar Syn-Hershko
http://code972.com | @synhershko 
Freelance Developer & Consultant
Author of RavenDB in Action 

On Wed, Nov 26, 2014 at 10:26 AM, Yosi Haran  wrote:

> Alright, we'll try upgrading. Thanks :)
>
> Meanwhile, any advice on how to fix an inconsistency once it is found? Is
> there an API to forcefully "sync" nodes, or at-least reindex from a
> specific node?
>
> On Tuesday, November 25, 2014 8:44:44 PM UTC+2, Itamar Syn-Hershko wrote:
>>
>> I suggest you upgrade to 1.4 and try again - see
>> http://www.elasticsearch.org/guide/en/elasticsearch/
>> resiliency/current/index.html
>>
>> --
>>
>> Itamar Syn-Hershko
>> http://code972.com | @synhershko 
>> Freelance Developer & Consultant
>> Author of RavenDB in Action 
>>
>> On Tue, Nov 25, 2014 at 7:29 PM, Yosi Haran  wrote:
>>
>>> 1.0.0
>>>
>>> On Tuesday, November 25, 2014 6:41:36 PM UTC+2, Itamar Syn-Hershko wrote:

 minimum_master_nodes still doesn't protect you from all possible
 failure scenarios, see http://aphyr.com/posts/317-call-me-maybe-
 elasticsearch

 What version are you running?

 --

 Itamar Syn-Hershko
 http://code972.com | @synhershko 
 Freelance Developer & Consultant
 Author of RavenDB in Action 

 On Tue, Nov 25, 2014 at 6:37 PM, Yosi Haran  wrote:

> Hi Guys,
>
> We are running a 3 node cluster, and each node returns a different
> number of documents when issued a direct HTTP "_count" call.
>
> The cluster holds about 150K documents and the differences range from
> 30~50 documents, but are still troubling.
>
> This shouldn't be a "split brain" problem, since we have set:
> discovery.zen.minimum_master_nodes: 2
> We also have a "client" node, but since client nodes are eligible to
> be master, I understand that they shouldn't affect the master election
> process.
>
> Any Ideas about why and how this is happening?
>
> Thanks!
>
>  --
> You received this message because you are subscribed to the Google
> Groups "elasticsearch" group.
> To unsubscribe from this group and stop receiving emails from it, send
> an email to elasticsearc...@googlegroups.com.
> To view this discussion on the web visit https://groups.google.com/d/
> msgid/elasticsearch/8ed828c8-fb62-413a-9fe0-91806fcf34e6%40goo
> glegroups.com
> 
> .
> For more options, visit https://groups.google.com/d/optout.
>

  --
>>> You received this message because you are subscribed to the Google
>>> Groups "elasticsearch" group.
>>> To unsubscribe from this group and stop receiving emails from it, send
>>> an email to elasticsearc...@googlegroups.com.
>>> To view this discussion on the web visit https://groups.google.com/d/
>>> msgid/elasticsearch/fa91da1d-2127-4f70-96e0-15125a5af3bc%
>>> 40googlegroups.com
>>> 
>>> .
>>>
>>> For more options, visit https://groups.google.com/d/optout.
>>>
>>
>>  --
> You received this message because you are subscribed to the Google Groups
> "elasticsearch" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to elasticsearch+unsubscr...@googlegroups.com.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/elasticsearch/b53c3df3-1d7f-4381-884a-713f605a7fba%40googlegroups.com
> 
> .
>
> For more options, visit https://groups.google.com/d/optout.
>

-- 
You received this message because you are subscribed to the Google Groups 
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/elasticsearch/CAHTr4ZuJuoVgzzc3bgguk5a9WmFkEaEPEv7DnbET-iQVREfriw%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

Re: 3 Node Cluster With Nodes Out of Sync

[Yosi Haran]

Alright, we'll try upgrading. Thanks :)

Meanwhile, any advice on how to fix an inconsistency once it is found? Is 
there an API to forcefully "sync" nodes, or at-least reindex from a 
specific node?

On Tuesday, November 25, 2014 8:44:44 PM UTC+2, Itamar Syn-Hershko wrote:
>
> I suggest you upgrade to 1.4 and try again - see 
> http://www.elasticsearch.org/guide/en/elasticsearch/resiliency/current/index.html
>
> --
>
> Itamar Syn-Hershko
> http://code972.com | @synhershko 
> Freelance Developer & Consultant
> Author of RavenDB in Action 
>
> On Tue, Nov 25, 2014 at 7:29 PM, Yosi Haran  > wrote:
>
>> 1.0.0
>>
>> On Tuesday, November 25, 2014 6:41:36 PM UTC+2, Itamar Syn-Hershko wrote:
>>>
>>> minimum_master_nodes still doesn't protect you from all possible failure 
>>> scenarios, see http://aphyr.com/posts/317-call-me-maybe-elasticsearch
>>>
>>> What version are you running?
>>>
>>> --
>>>
>>> Itamar Syn-Hershko
>>> http://code972.com | @synhershko 
>>> Freelance Developer & Consultant
>>> Author of RavenDB in Action 
>>>
>>> On Tue, Nov 25, 2014 at 6:37 PM, Yosi Haran  wrote:
>>>
 Hi Guys,

 We are running a 3 node cluster, and each node returns a different 
 number of documents when issued a direct HTTP "_count" call.

 The cluster holds about 150K documents and the differences range from 
 30~50 documents, but are still troubling.

 This shouldn't be a "split brain" problem, since we have set:
 discovery.zen.minimum_master_nodes: 2
 We also have a "client" node, but since client nodes are eligible to be 
 master, I understand that they shouldn't affect the master election 
 process.

 Any Ideas about why and how this is happening?

 Thanks!

  -- 
 You received this message because you are subscribed to the Google 
 Groups "elasticsearch" group.
 To unsubscribe from this group and stop receiving emails from it, send 
 an email to elasticsearc...@googlegroups.com.
 To view this discussion on the web visit https://groups.google.com/d/
 msgid/elasticsearch/8ed828c8-fb62-413a-9fe0-91806fcf34e6%
 40googlegroups.com 
 
 .
 For more options, visit https://groups.google.com/d/optout.

>>>
>>>  -- 
>> You received this message because you are subscribed to the Google Groups 
>> "elasticsearch" group.
>> To unsubscribe from this group and stop receiving emails from it, send an 
>> email to elasticsearc...@googlegroups.com .
>> To view this discussion on the web visit 
>> https://groups.google.com/d/msgid/elasticsearch/fa91da1d-2127-4f70-96e0-15125a5af3bc%40googlegroups.com
>>  
>> 
>> .
>>
>> For more options, visit https://groups.google.com/d/optout.
>>
>
>

-- 
You received this message because you are subscribed to the Google Groups 
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/elasticsearch/b53c3df3-1d7f-4381-884a-713f605a7fba%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: 3 Node Cluster With Nodes Out of Sync

[Itamar Syn-Hershko]

I suggest you upgrade to 1.4 and try again - see
http://www.elasticsearch.org/guide/en/elasticsearch/resiliency/current/index.html

--

Itamar Syn-Hershko
http://code972.com | @synhershko 
Freelance Developer & Consultant
Author of RavenDB in Action 

On Tue, Nov 25, 2014 at 7:29 PM, Yosi Haran  wrote:

> 1.0.0
>
> On Tuesday, November 25, 2014 6:41:36 PM UTC+2, Itamar Syn-Hershko wrote:
>>
>> minimum_master_nodes still doesn't protect you from all possible failure
>> scenarios, see http://aphyr.com/posts/317-call-me-maybe-elasticsearch
>>
>> What version are you running?
>>
>> --
>>
>> Itamar Syn-Hershko
>> http://code972.com | @synhershko 
>> Freelance Developer & Consultant
>> Author of RavenDB in Action 
>>
>> On Tue, Nov 25, 2014 at 6:37 PM, Yosi Haran  wrote:
>>
>>> Hi Guys,
>>>
>>> We are running a 3 node cluster, and each node returns a different
>>> number of documents when issued a direct HTTP "_count" call.
>>>
>>> The cluster holds about 150K documents and the differences range from
>>> 30~50 documents, but are still troubling.
>>>
>>> This shouldn't be a "split brain" problem, since we have set:
>>> discovery.zen.minimum_master_nodes: 2
>>> We also have a "client" node, but since client nodes are eligible to be
>>> master, I understand that they shouldn't affect the master election process.
>>>
>>> Any Ideas about why and how this is happening?
>>>
>>> Thanks!
>>>
>>>  --
>>> You received this message because you are subscribed to the Google
>>> Groups "elasticsearch" group.
>>> To unsubscribe from this group and stop receiving emails from it, send
>>> an email to elasticsearc...@googlegroups.com.
>>> To view this discussion on the web visit https://groups.google.com/d/
>>> msgid/elasticsearch/8ed828c8-fb62-413a-9fe0-91806fcf34e6%
>>> 40googlegroups.com
>>> 
>>> .
>>> For more options, visit https://groups.google.com/d/optout.
>>>
>>
>>  --
> You received this message because you are subscribed to the Google Groups
> "elasticsearch" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to elasticsearch+unsubscr...@googlegroups.com.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/elasticsearch/fa91da1d-2127-4f70-96e0-15125a5af3bc%40googlegroups.com
> 
> .
>
> For more options, visit https://groups.google.com/d/optout.
>

-- 
You received this message because you are subscribed to the Google Groups 
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/elasticsearch/CAHTr4Ztx1Ze2dUNJno7MkD76PoDbuMp64o11sHsqKFMU1VeXaA%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

Re: 3 Node Cluster With Nodes Out of Sync

[Yosi Haran]

1.0.0

On Tuesday, November 25, 2014 6:41:36 PM UTC+2, Itamar Syn-Hershko wrote:
>
> minimum_master_nodes still doesn't protect you from all possible failure 
> scenarios, see http://aphyr.com/posts/317-call-me-maybe-elasticsearch
>
> What version are you running?
>
> --
>
> Itamar Syn-Hershko
> http://code972.com | @synhershko 
> Freelance Developer & Consultant
> Author of RavenDB in Action 
>
> On Tue, Nov 25, 2014 at 6:37 PM, Yosi Haran  > wrote:
>
>> Hi Guys,
>>
>> We are running a 3 node cluster, and each node returns a different number 
>> of documents when issued a direct HTTP "_count" call.
>>
>> The cluster holds about 150K documents and the differences range from 
>> 30~50 documents, but are still troubling.
>>
>> This shouldn't be a "split brain" problem, since we have set:
>> discovery.zen.minimum_master_nodes: 2
>> We also have a "client" node, but since client nodes are eligible to be 
>> master, I understand that they shouldn't affect the master election process.
>>
>> Any Ideas about why and how this is happening?
>>
>> Thanks!
>>
>>  -- 
>> You received this message because you are subscribed to the Google Groups 
>> "elasticsearch" group.
>> To unsubscribe from this group and stop receiving emails from it, send an 
>> email to elasticsearc...@googlegroups.com .
>> To view this discussion on the web visit 
>> https://groups.google.com/d/msgid/elasticsearch/8ed828c8-fb62-413a-9fe0-91806fcf34e6%40googlegroups.com
>>  
>> 
>> .
>> For more options, visit https://groups.google.com/d/optout.
>>
>
>

-- 
You received this message because you are subscribed to the Google Groups 
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/elasticsearch/fa91da1d-2127-4f70-96e0-15125a5af3bc%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: 3 Node Cluster With Nodes Out of Sync

[Itamar Syn-Hershko]

minimum_master_nodes still doesn't protect you from all possible failure
scenarios, see http://aphyr.com/posts/317-call-me-maybe-elasticsearch

What version are you running?

--

Itamar Syn-Hershko
http://code972.com | @synhershko 
Freelance Developer & Consultant
Author of RavenDB in Action 

On Tue, Nov 25, 2014 at 6:37 PM, Yosi Haran  wrote:

> Hi Guys,
>
> We are running a 3 node cluster, and each node returns a different number
> of documents when issued a direct HTTP "_count" call.
>
> The cluster holds about 150K documents and the differences range from
> 30~50 documents, but are still troubling.
>
> This shouldn't be a "split brain" problem, since we have set:
> discovery.zen.minimum_master_nodes: 2
> We also have a "client" node, but since client nodes are eligible to be
> master, I understand that they shouldn't affect the master election process.
>
> Any Ideas about why and how this is happening?
>
> Thanks!
>
>  --
> You received this message because you are subscribed to the Google Groups
> "elasticsearch" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to elasticsearch+unsubscr...@googlegroups.com.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/elasticsearch/8ed828c8-fb62-413a-9fe0-91806fcf34e6%40googlegroups.com
> 
> .
> For more options, visit https://groups.google.com/d/optout.
>

-- 
You received this message because you are subscribed to the Google Groups 
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/elasticsearch/CAHTr4Zu6r2z39iT%2B%3Dy7b6Brez%2BhLz8davSRaq0UmMviCCqV_sQ%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.