[
https://issues.apache.org/jira/browse/CURATOR-504?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Yuri Tceretian updated CURATOR-504:
-----------------------------------
Description:
We use LeaderLatch in a lot of places in our system and when ZooKeeper ensemble
is unstable and clients are reconnecting to logs are full of messages like the
following:
{{
[2017-08-31
19:18:34,562][ERROR][org.apache.curator.framework.recipes.leader.LeaderLatch]
Can't find our node. Resetting. Index: -1 {}
}}
According to the
[implementation|https://github.com/apache/curator/blob/4251fe328908e5fca37af034fabc190aa452c73f/curator-recipes/src/main/java/org/apache/curator/framework/recipes/leader/LeaderLatch.java#L529-L536],
this can happen in two cases:
* When internal state `ourPath` is null
* When the list of latches does not have the expected one.
I believe we hit the first condition because of races that occur after client
reconnects to ZooKeeper.
* Client reconnects to ZooKeeper and LeaderLatch gets the event and calls
reset method which set the internal state (`ourPath`) to null, removes old
latch and creates a new one. This happens in thread
"Curator-ConnectionStateManager-0".
* Almost simultaneously, LeaderLatch gets another even NodeDeleted
([here|https://github.com/apache/curator/blob/4251fe328908e5fca37af034fabc190aa452c73f/curator-recipes/src/main/java/org/apache/curator/framework/recipes/leader/LeaderLatch.java#L543-L554])
and tries to re-read the list of latches and check leadership. This happens in
the thread "main-EventThread".
Therefore, sometimes there is a situation when method `checkLeadership` is
called when `ourPath` is null.
Below is an approximate diagram of what happens:
!51868597-65791000-231c-11e9-9bfa-1def62bc3ea1.png|width=1261,height=150!
was:
We use LeaderLatch in a lot of places in our system and when ZooKeeper ensemble
is unstable and clients are reconnecting to logs are full of messages like the
following:
{{
[2017-08-31
19:18:34,562][ERROR][org.apache.curator.framework.recipes.leader.LeaderLatch]
Can't find our node. Resetting. Index: -1 {}
}}
According to the implementation, this can happen in two cases:
[https://github.com/apache/curator/blob/4251fe328908e5fca37af034fabc190aa452c73f/curator-recipes/src/main/java/org/apache/curator/framework/recipes/leader/LeaderLatch.java#L529-L536]
* When internal state `ourPath` is null
* When the list of latches does not have the expected one.
I believe we hit the first condition because of races that occur after client
reconnects to ZooKeeper.
* Client reconnects to ZooKeeper and LeaderLatch gets the event and calls
reset method which set the internal state (`ourPath`) to null, removes old
latch and creates a new one. This happens in thread
"Curator-ConnectionStateManager-0".
* Almost simultaneously, LeaderLatch gets another even NodeDeleted
([here|https://github.com/apache/curator/blob/4251fe328908e5fca37af034fabc190aa452c73f/curator-recipes/src/main/java/org/apache/curator/framework/recipes/leader/LeaderLatch.java#L543-L554])
and tries to re-read the list of latches and check leadership. This happens in
the thread "main-EventThread".
Therefore, sometimes there is a situation when method `checkLeadership` is
called when `ourPath` is null.
Below is approximate diagram of what happens:
!51868597-65791000-231c-11e9-9bfa-1def62bc3ea1.png!
> Race conditions in LeaderLatch after reconnecting to ensemble
> -------------------------------------------------------------
>
> Key: CURATOR-504
> URL: https://issues.apache.org/jira/browse/CURATOR-504
> Project: Apache Curator
> Issue Type: Bug
> Affects Versions: 4.1.0
> Reporter: Yuri Tceretian
> Assignee: Jordan Zimmerman
> Priority: Minor
> Attachments: 51868597-65791000-231c-11e9-9bfa-1def62bc3ea1.png
>
>
> We use LeaderLatch in a lot of places in our system and when ZooKeeper
> ensemble is unstable and clients are reconnecting to logs are full of
> messages like the following:
> {{
> [2017-08-31
> 19:18:34,562][ERROR][org.apache.curator.framework.recipes.leader.LeaderLatch]
> Can't find our node. Resetting. Index: -1 {}
> }}
> According to the
> [implementation|https://github.com/apache/curator/blob/4251fe328908e5fca37af034fabc190aa452c73f/curator-recipes/src/main/java/org/apache/curator/framework/recipes/leader/LeaderLatch.java#L529-L536],
> this can happen in two cases:
> * When internal state `ourPath` is null
> * When the list of latches does not have the expected one.
> I believe we hit the first condition because of races that occur after client
> reconnects to ZooKeeper.
> * Client reconnects to ZooKeeper and LeaderLatch gets the event and calls
> reset method which set the internal state (`ourPath`) to null, removes old
> latch and creates a new one. This happens in thread
> "Curator-ConnectionStateManager-0".
> * Almost simultaneously, LeaderLatch gets another even NodeDeleted
> ([here|https://github.com/apache/curator/blob/4251fe328908e5fca37af034fabc190aa452c73f/curator-recipes/src/main/java/org/apache/curator/framework/recipes/leader/LeaderLatch.java#L543-L554])
> and tries to re-read the list of latches and check leadership. This happens
> in the thread "main-EventThread".
> Therefore, sometimes there is a situation when method `checkLeadership` is
> called when `ourPath` is null.
> Below is an approximate diagram of what happens:
> !51868597-65791000-231c-11e9-9bfa-1def62bc3ea1.png|width=1261,height=150!
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
