[jira] [Commented] (CURATOR-460) Timed tolerance for connection suspended leads to simultaneous leaders

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/CURATOR-460?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16717838#comment-16717838
 ] 

ASF GitHub Bot commented on CURATOR-460:


Github user Randgalt commented on the issue:

https://github.com/apache/curator/pull/243
  
Please read this tech note and see if it applies: 
https://issues.apache.org/jira/browse/CURATOR-460


> Timed tolerance for connection suspended leads to simultaneous leaders
> --
>
> Key: CURATOR-460
> URL: https://issues.apache.org/jira/browse/CURATOR-460
> Project: Apache Curator
>  Issue Type: Bug
>  Components: Recipes
>Affects Versions: 3.3.0, 4.0.0, 4.0.1
>Reporter: Antonio Rafael Rodrigues
>Assignee: Cameron McKenzie
>Priority: Major
> Fix For: 4.1.0
>
> Attachments: ConnectionStateManager.patch, 
> Console-leaderSelector1.log, Console-leaderSelector2.log, 
> CuratorLeaderSelectorPOC.java, LogApp1.log, LogApp2.log
>
>
> Starting from Curator 3, after losing the connection to zookeeper, curator 
> emits a SUSPENDED event and then starts an internal timer, if the time of the 
> negotiatied session timeout get hit and curator didn't reconnect to zookeeper 
> it emits a LOST event.
> For example :
>  Given negotiated session timeout = 40
> ||Time (seconds)||Event||
> |0|SUSPENDED (curator has been disconnected from zookeeper)|
> |40|LOST (curator is still disconnected, it must have been lost the sesion as 
> it is disconnected for 40sec)|
> Given this scenario we could ,theoretically, ignore the SUSPENDED event and 
> consider the leadership as lost just if the ConnectionStateListener receives 
> the LOST event.
> But this feature seems to have introduced a bug (from my point of view)
> *Case of failure*
> ||Time (seconds)||Event||
> |0|Something went wrong with the connected zookeeper (in my case, the network 
> inteface of zookeeper's server has gone down). Curator stops hearing heart 
> beats from zookeeper, but doesn't lose the connection (from some reason that 
> I don't know, if the network interface of the server goes down, Curator 
> doesn't lose the connection)|
> |~26.666|SUSPENDED (after 26 seconds without hearing heartbeats, curator 
> emits a SUSPENDED event) 26 is from "readTimeout = sessionTimeout * 2 / 3" 
> from the class ClientCnxn.java from zookeeper client. At this point, curator 
> starts counting 40 sec.|
> |26.666 to 40|During this period, Curator is trying to connect to other 
> zookeeper instances but the other instances, in my example case, are also 
> uncheachable.|
> |40|Session has expired and the other instance has taken leadership ( in my 
> example, the other instance can connect to zookeeper )|
> |66.666|LOST ( after 40sec from SUSPENDED, curator finally sends LOST)|
> As you can see, if we are ignoring the SUSPENDED event, the second 
> application instance acquires the leadership ~26 seconds before the first 
> instance notice that it lost the leadership.
>  I understand it seems to be a very rare case, but either way I think it 
> should be addressed.
> *Reproduce it*
> I have came up with a way to reproduce this easily. I know this is not a 
> recommended production setup, but it reproduces the problem anyway.
> 1) On a virtual machine, with 2 network interfaces (eth0[192.168.0.101], 
> eth1[192.168.0.102]) , I installed one zookeeper instance.
>  2) I setup application1 with the leadership receipe, with 40sec of 
> negotiated timeout, pointing just to 192.168.0.101  .Now it is the leader
>  3) I setup application2 with the leadership receipe, with 40sec of 
> negotiated timeout, pointing just to 192.168.0.102  .Now it it is NOT the 
> leader
>  4) On the server I turn the eth0[192.168.0.101] interface down  [ ifconfig 
> eth0 down ]
>  5) After 26 seconds, application1 says : 
>            :ClientCnxn$SendThread@] - Client session timed out, have not 
> heard from server in 26679ms for sessionid
>             ConnectionStateManager@237] - State change: SUSPENDED
>         NOTE: I'm ignoring the SUSPENDED event
> 6) After 40 seconds, application2 takes leadership
>  7) After 66 seconds, application1 says :
>              ConnectionStateManager@237] - State change: LOST
>          
>          NOTE: Just at this point, I consider that the application1 has lost 
> leadership
> Then, for 26 seconds, we had 2 leaders.
> If you confirm it as a bug, I think I could help.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (CURATOR-460) Timed tolerance for connection suspended leads to simultaneous leaders

[Cameron McKenzie (JIRA)]

[ 
https://issues.apache.org/jira/browse/CURATOR-460?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16440224#comment-16440224
 ] 

Cameron McKenzie commented on CURATOR-460:
--

The new tech note looks good. Thanks for writing that up. I fixed up one minor 
typo.

> Timed tolerance for connection suspended leads to simultaneous leaders
> --
>
> Key: CURATOR-460
> URL: https://issues.apache.org/jira/browse/CURATOR-460
> Project: Apache Curator
>  Issue Type: Bug
>  Components: Recipes
>Affects Versions: 3.3.0, 4.0.0, 4.0.1
>Reporter: Antonio Rafael Rodrigues
>Assignee: Cameron McKenzie
>Priority: Major
> Fix For: 4.0.2
>
> Attachments: ConnectionStateManager.patch, 
> Console-leaderSelector1.log, Console-leaderSelector2.log, 
> CuratorLeaderSelectorPOC.java, LogApp1.log, LogApp2.log
>
>
> Starting from Curator 3, after losing the connection to zookeeper, curator 
> emits a SUSPENDED event and then starts an internal timer, if the time of the 
> negotiatied session timeout get hit and curator didn't reconnect to zookeeper 
> it emits a LOST event.
> For example :
>  Given negotiated session timeout = 40
> ||Time (seconds)||Event||
> |0|SUSPENDED (curator has been disconnected from zookeeper)|
> |40|LOST (curator is still disconnected, it must have been lost the sesion as 
> it is disconnected for 40sec)|
> Given this scenario we could ,theoretically, ignore the SUSPENDED event and 
> consider the leadership as lost just if the ConnectionStateListener receives 
> the LOST event.
> But this feature seems to have introduced a bug (from my point of view)
> *Case of failure*
> ||Time (seconds)||Event||
> |0|Something went wrong with the connected zookeeper (in my case, the network 
> inteface of zookeeper's server has gone down). Curator stops hearing heart 
> beats from zookeeper, but doesn't lose the connection (from some reason that 
> I don't know, if the network interface of the server goes down, Curator 
> doesn't lose the connection)|
> |~26.666|SUSPENDED (after 26 seconds without hearing heartbeats, curator 
> emits a SUSPENDED event) 26 is from "readTimeout = sessionTimeout * 2 / 3" 
> from the class ClientCnxn.java from zookeeper client. At this point, curator 
> starts counting 40 sec.|
> |26.666 to 40|During this period, Curator is trying to connect to other 
> zookeeper instances but the other instances, in my example case, are also 
> uncheachable.|
> |40|Session has expired and the other instance has taken leadership ( in my 
> example, the other instance can connect to zookeeper )|
> |66.666|LOST ( after 40sec from SUSPENDED, curator finally sends LOST)|
> As you can see, if we are ignoring the SUSPENDED event, the second 
> application instance acquires the leadership ~26 seconds before the first 
> instance notice that it lost the leadership.
>  I understand it seems to be a very rare case, but either way I think it 
> should be addressed.
> *Reproduce it*
> I have came up with a way to reproduce this easily. I know this is not a 
> recommended production setup, but it reproduces the problem anyway.
> 1) On a virtual machine, with 2 network interfaces (eth0[192.168.0.101], 
> eth1[192.168.0.102]) , I installed one zookeeper instance.
>  2) I setup application1 with the leadership receipe, with 40sec of 
> negotiated timeout, pointing just to 192.168.0.101  .Now it is the leader
>  3) I setup application2 with the leadership receipe, with 40sec of 
> negotiated timeout, pointing just to 192.168.0.102  .Now it it is NOT the 
> leader
>  4) On the server I turn the eth0[192.168.0.101] interface down  [ ifconfig 
> eth0 down ]
>  5) After 26 seconds, application1 says : 
>            :ClientCnxn$SendThread@] - Client session timed out, have not 
> heard from server in 26679ms for sessionid
>             ConnectionStateManager@237] - State change: SUSPENDED
>         NOTE: I'm ignoring the SUSPENDED event
> 6) After 40 seconds, application2 takes leadership
>  7) After 66 seconds, application1 says :
>              ConnectionStateManager@237] - State change: LOST
>          
>          NOTE: Just at this point, I consider that the application1 has lost 
> leadership
> Then, for 26 seconds, we had 2 leaders.
> If you confirm it as a bug, I think I could help.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (CURATOR-460) Timed tolerance for connection suspended leads to simultaneous leaders

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/CURATOR-460?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16438557#comment-16438557
 ] 

ASF GitHub Bot commented on CURATOR-460:


Github user Randgalt commented on the issue:

https://github.com/apache/curator/pull/262
  
The new Tech Note: https://cwiki.apache.org/confluence/display/CURATOR/TN14 
- I'd appreciate reviews.


> Timed tolerance for connection suspended leads to simultaneous leaders
> --
>
> Key: CURATOR-460
> URL: https://issues.apache.org/jira/browse/CURATOR-460
> Project: Apache Curator
>  Issue Type: Bug
>  Components: Recipes
>Affects Versions: 3.3.0, 4.0.0, 4.0.1
>Reporter: Antonio Rafael Rodrigues
>Assignee: Cameron McKenzie
>Priority: Major
> Fix For: 4.0.2
>
> Attachments: ConnectionStateManager.patch, 
> Console-leaderSelector1.log, Console-leaderSelector2.log, 
> CuratorLeaderSelectorPOC.java, LogApp1.log, LogApp2.log
>
>
> Starting from Curator 3, after losing the connection to zookeeper, curator 
> emits a SUSPENDED event and then starts an internal timer, if the time of the 
> negotiatied session timeout get hit and curator didn't reconnect to zookeeper 
> it emits a LOST event.
> For example :
>  Given negotiated session timeout = 40
> ||Time (seconds)||Event||
> |0|SUSPENDED (curator has been disconnected from zookeeper)|
> |40|LOST (curator is still disconnected, it must have been lost the sesion as 
> it is disconnected for 40sec)|
> Given this scenario we could ,theoretically, ignore the SUSPENDED event and 
> consider the leadership as lost just if the ConnectionStateListener receives 
> the LOST event.
> But this feature seems to have introduced a bug (from my point of view)
> *Case of failure*
> ||Time (seconds)||Event||
> |0|Something went wrong with the connected zookeeper (in my case, the network 
> inteface of zookeeper's server has gone down). Curator stops hearing heart 
> beats from zookeeper, but doesn't lose the connection (from some reason that 
> I don't know, if the network interface of the server goes down, Curator 
> doesn't lose the connection)|
> |~26.666|SUSPENDED (after 26 seconds without hearing heartbeats, curator 
> emits a SUSPENDED event) 26 is from "readTimeout = sessionTimeout * 2 / 3" 
> from the class ClientCnxn.java from zookeeper client. At this point, curator 
> starts counting 40 sec.|
> |26.666 to 40|During this period, Curator is trying to connect to other 
> zookeeper instances but the other instances, in my example case, are also 
> uncheachable.|
> |40|Session has expired and the other instance has taken leadership ( in my 
> example, the other instance can connect to zookeeper )|
> |66.666|LOST ( after 40sec from SUSPENDED, curator finally sends LOST)|
> As you can see, if we are ignoring the SUSPENDED event, the second 
> application instance acquires the leadership ~26 seconds before the first 
> instance notice that it lost the leadership.
>  I understand it seems to be a very rare case, but either way I think it 
> should be addressed.
> *Reproduce it*
> I have came up with a way to reproduce this easily. I know this is not a 
> recommended production setup, but it reproduces the problem anyway.
> 1) On a virtual machine, with 2 network interfaces (eth0[192.168.0.101], 
> eth1[192.168.0.102]) , I installed one zookeeper instance.
>  2) I setup application1 with the leadership receipe, with 40sec of 
> negotiated timeout, pointing just to 192.168.0.101  .Now it is the leader
>  3) I setup application2 with the leadership receipe, with 40sec of 
> negotiated timeout, pointing just to 192.168.0.102  .Now it it is NOT the 
> leader
>  4) On the server I turn the eth0[192.168.0.101] interface down  [ ifconfig 
> eth0 down ]
>  5) After 26 seconds, application1 says : 
>            :ClientCnxn$SendThread@] - Client session timed out, have not 
> heard from server in 26679ms for sessionid
>             ConnectionStateManager@237] - State change: SUSPENDED
>         NOTE: I'm ignoring the SUSPENDED event
> 6) After 40 seconds, application2 takes leadership
>  7) After 66 seconds, application1 says :
>              ConnectionStateManager@237] - State change: LOST
>          
>          NOTE: Just at this point, I consider that the application1 has lost 
> leadership
> Then, for 26 seconds, we had 2 leaders.
> If you confirm it as a bug, I think I could help.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (CURATOR-460) Timed tolerance for connection suspended leads to simultaneous leaders

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/CURATOR-460?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16434823#comment-16434823
 ] 

ASF GitHub Bot commented on CURATOR-460:


Github user asfgit closed the pull request at:

https://github.com/apache/curator/pull/262


> Timed tolerance for connection suspended leads to simultaneous leaders
> --
>
> Key: CURATOR-460
> URL: https://issues.apache.org/jira/browse/CURATOR-460
> Project: Apache Curator
>  Issue Type: Bug
>  Components: Recipes
>Affects Versions: 3.3.0, 4.0.0, 4.0.1
>Reporter: Antonio Rafael Rodrigues
>Assignee: Cameron McKenzie
>Priority: Major
> Fix For: 4.0.2
>
> Attachments: ConnectionStateManager.patch, 
> Console-leaderSelector1.log, Console-leaderSelector2.log, 
> CuratorLeaderSelectorPOC.java, LogApp1.log, LogApp2.log
>
>
> Starting from Curator 3, after losing the connection to zookeeper, curator 
> emits a SUSPENDED event and then starts an internal timer, if the time of the 
> negotiatied session timeout get hit and curator didn't reconnect to zookeeper 
> it emits a LOST event.
> For example :
>  Given negotiated session timeout = 40
> ||Time (seconds)||Event||
> |0|SUSPENDED (curator has been disconnected from zookeeper)|
> |40|LOST (curator is still disconnected, it must have been lost the sesion as 
> it is disconnected for 40sec)|
> Given this scenario we could ,theoretically, ignore the SUSPENDED event and 
> consider the leadership as lost just if the ConnectionStateListener receives 
> the LOST event.
> But this feature seems to have introduced a bug (from my point of view)
> *Case of failure*
> ||Time (seconds)||Event||
> |0|Something went wrong with the connected zookeeper (in my case, the network 
> inteface of zookeeper's server has gone down). Curator stops hearing heart 
> beats from zookeeper, but doesn't lose the connection (from some reason that 
> I don't know, if the network interface of the server goes down, Curator 
> doesn't lose the connection)|
> |~26.666|SUSPENDED (after 26 seconds without hearing heartbeats, curator 
> emits a SUSPENDED event) 26 is from "readTimeout = sessionTimeout * 2 / 3" 
> from the class ClientCnxn.java from zookeeper client. At this point, curator 
> starts counting 40 sec.|
> |26.666 to 40|During this period, Curator is trying to connect to other 
> zookeeper instances but the other instances, in my example case, are also 
> uncheachable.|
> |40|Session has expired and the other instance has taken leadership ( in my 
> example, the other instance can connect to zookeeper )|
> |66.666|LOST ( after 40sec from SUSPENDED, curator finally sends LOST)|
> As you can see, if we are ignoring the SUSPENDED event, the second 
> application instance acquires the leadership ~26 seconds before the first 
> instance notice that it lost the leadership.
>  I understand it seems to be a very rare case, but either way I think it 
> should be addressed.
> *Reproduce it*
> I have came up with a way to reproduce this easily. I know this is not a 
> recommended production setup, but it reproduces the problem anyway.
> 1) On a virtual machine, with 2 network interfaces (eth0[192.168.0.101], 
> eth1[192.168.0.102]) , I installed one zookeeper instance.
>  2) I setup application1 with the leadership receipe, with 40sec of 
> negotiated timeout, pointing just to 192.168.0.101  .Now it is the leader
>  3) I setup application2 with the leadership receipe, with 40sec of 
> negotiated timeout, pointing just to 192.168.0.102  .Now it it is NOT the 
> leader
>  4) On the server I turn the eth0[192.168.0.101] interface down  [ ifconfig 
> eth0 down ]
>  5) After 26 seconds, application1 says : 
>            :ClientCnxn$SendThread@] - Client session timed out, have not 
> heard from server in 26679ms for sessionid
>             ConnectionStateManager@237] - State change: SUSPENDED
>         NOTE: I'm ignoring the SUSPENDED event
> 6) After 40 seconds, application2 takes leadership
>  7) After 66 seconds, application1 says :
>              ConnectionStateManager@237] - State change: LOST
>          
>          NOTE: Just at this point, I consider that the application1 has lost 
> leadership
> Then, for 26 seconds, we had 2 leaders.
> If you confirm it as a bug, I think I could help.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (CURATOR-460) Timed tolerance for connection suspended leads to simultaneous leaders

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/CURATOR-460?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16434816#comment-16434816
 ] 

ASF GitHub Bot commented on CURATOR-460:


Github user cammckenzie commented on the issue:

https://github.com/apache/curator/pull/262
  
Ok, will merge this now.

Thanks @arrodrigues !


> Timed tolerance for connection suspended leads to simultaneous leaders
> --
>
> Key: CURATOR-460
> URL: https://issues.apache.org/jira/browse/CURATOR-460
> Project: Apache Curator
>  Issue Type: Bug
>  Components: Recipes
>Affects Versions: 3.3.0, 4.0.0, 4.0.1
>Reporter: Antonio Rafael Rodrigues
>Assignee: Cameron McKenzie
>Priority: Major
> Attachments: ConnectionStateManager.patch, 
> Console-leaderSelector1.log, Console-leaderSelector2.log, 
> CuratorLeaderSelectorPOC.java, LogApp1.log, LogApp2.log
>
>
> Starting from Curator 3, after losing the connection to zookeeper, curator 
> emits a SUSPENDED event and then starts an internal timer, if the time of the 
> negotiatied session timeout get hit and curator didn't reconnect to zookeeper 
> it emits a LOST event.
> For example :
>  Given negotiated session timeout = 40
> ||Time (seconds)||Event||
> |0|SUSPENDED (curator has been disconnected from zookeeper)|
> |40|LOST (curator is still disconnected, it must have been lost the sesion as 
> it is disconnected for 40sec)|
> Given this scenario we could ,theoretically, ignore the SUSPENDED event and 
> consider the leadership as lost just if the ConnectionStateListener receives 
> the LOST event.
> But this feature seems to have introduced a bug (from my point of view)
> *Case of failure*
> ||Time (seconds)||Event||
> |0|Something went wrong with the connected zookeeper (in my case, the network 
> inteface of zookeeper's server has gone down). Curator stops hearing heart 
> beats from zookeeper, but doesn't lose the connection (from some reason that 
> I don't know, if the network interface of the server goes down, Curator 
> doesn't lose the connection)|
> |~26.666|SUSPENDED (after 26 seconds without hearing heartbeats, curator 
> emits a SUSPENDED event) 26 is from "readTimeout = sessionTimeout * 2 / 3" 
> from the class ClientCnxn.java from zookeeper client. At this point, curator 
> starts counting 40 sec.|
> |26.666 to 40|During this period, Curator is trying to connect to other 
> zookeeper instances but the other instances, in my example case, are also 
> uncheachable.|
> |40|Session has expired and the other instance has taken leadership ( in my 
> example, the other instance can connect to zookeeper )|
> |66.666|LOST ( after 40sec from SUSPENDED, curator finally sends LOST)|
> As you can see, if we are ignoring the SUSPENDED event, the second 
> application instance acquires the leadership ~26 seconds before the first 
> instance notice that it lost the leadership.
>  I understand it seems to be a very rare case, but either way I think it 
> should be addressed.
> *Reproduce it*
> I have came up with a way to reproduce this easily. I know this is not a 
> recommended production setup, but it reproduces the problem anyway.
> 1) On a virtual machine, with 2 network interfaces (eth0[192.168.0.101], 
> eth1[192.168.0.102]) , I installed one zookeeper instance.
>  2) I setup application1 with the leadership receipe, with 40sec of 
> negotiated timeout, pointing just to 192.168.0.101  .Now it is the leader
>  3) I setup application2 with the leadership receipe, with 40sec of 
> negotiated timeout, pointing just to 192.168.0.102  .Now it it is NOT the 
> leader
>  4) On the server I turn the eth0[192.168.0.101] interface down  [ ifconfig 
> eth0 down ]
>  5) After 26 seconds, application1 says : 
>            :ClientCnxn$SendThread@] - Client session timed out, have not 
> heard from server in 26679ms for sessionid
>             ConnectionStateManager@237] - State change: SUSPENDED
>         NOTE: I'm ignoring the SUSPENDED event
> 6) After 40 seconds, application2 takes leadership
>  7) After 66 seconds, application1 says :
>              ConnectionStateManager@237] - State change: LOST
>          
>          NOTE: Just at this point, I consider that the application1 has lost 
> leadership
> Then, for 26 seconds, we had 2 leaders.
> If you confirm it as a bug, I think I could help.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (CURATOR-460) Timed tolerance for connection suspended leads to simultaneous leaders

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/CURATOR-460?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16433873#comment-16433873
 ] 

ASF GitHub Bot commented on CURATOR-460:


Github user Randgalt commented on the issue:

https://github.com/apache/curator/pull/262
  
Yeah, I agree @cammckenzie. Let's merge what we have here and I'll write up 
a Tech Note.


> Timed tolerance for connection suspended leads to simultaneous leaders
> --
>
> Key: CURATOR-460
> URL: https://issues.apache.org/jira/browse/CURATOR-460
> Project: Apache Curator
>  Issue Type: Bug
>  Components: Recipes
>Affects Versions: 3.3.0, 4.0.0, 4.0.1
>Reporter: Antonio Rafael Rodrigues
>Assignee: Cameron McKenzie
>Priority: Major
> Attachments: ConnectionStateManager.patch, 
> Console-leaderSelector1.log, Console-leaderSelector2.log, 
> CuratorLeaderSelectorPOC.java, LogApp1.log, LogApp2.log
>
>
> Starting from Curator 3, after losing the connection to zookeeper, curator 
> emits a SUSPENDED event and then starts an internal timer, if the time of the 
> negotiatied session timeout get hit and curator didn't reconnect to zookeeper 
> it emits a LOST event.
> For example :
>  Given negotiated session timeout = 40
> ||Time (seconds)||Event||
> |0|SUSPENDED (curator has been disconnected from zookeeper)|
> |40|LOST (curator is still disconnected, it must have been lost the sesion as 
> it is disconnected for 40sec)|
> Given this scenario we could ,theoretically, ignore the SUSPENDED event and 
> consider the leadership as lost just if the ConnectionStateListener receives 
> the LOST event.
> But this feature seems to have introduced a bug (from my point of view)
> *Case of failure*
> ||Time (seconds)||Event||
> |0|Something went wrong with the connected zookeeper (in my case, the network 
> inteface of zookeeper's server has gone down). Curator stops hearing heart 
> beats from zookeeper, but doesn't lose the connection (from some reason that 
> I don't know, if the network interface of the server goes down, Curator 
> doesn't lose the connection)|
> |~26.666|SUSPENDED (after 26 seconds without hearing heartbeats, curator 
> emits a SUSPENDED event) 26 is from "readTimeout = sessionTimeout * 2 / 3" 
> from the class ClientCnxn.java from zookeeper client. At this point, curator 
> starts counting 40 sec.|
> |26.666 to 40|During this period, Curator is trying to connect to other 
> zookeeper instances but the other instances, in my example case, are also 
> uncheachable.|
> |40|Session has expired and the other instance has taken leadership ( in my 
> example, the other instance can connect to zookeeper )|
> |66.666|LOST ( after 40sec from SUSPENDED, curator finally sends LOST)|
> As you can see, if we are ignoring the SUSPENDED event, the second 
> application instance acquires the leadership ~26 seconds before the first 
> instance notice that it lost the leadership.
>  I understand it seems to be a very rare case, but either way I think it 
> should be addressed.
> *Reproduce it*
> I have came up with a way to reproduce this easily. I know this is not a 
> recommended production setup, but it reproduces the problem anyway.
> 1) On a virtual machine, with 2 network interfaces (eth0[192.168.0.101], 
> eth1[192.168.0.102]) , I installed one zookeeper instance.
>  2) I setup application1 with the leadership receipe, with 40sec of 
> negotiated timeout, pointing just to 192.168.0.101  .Now it is the leader
>  3) I setup application2 with the leadership receipe, with 40sec of 
> negotiated timeout, pointing just to 192.168.0.102  .Now it it is NOT the 
> leader
>  4) On the server I turn the eth0[192.168.0.101] interface down  [ ifconfig 
> eth0 down ]
>  5) After 26 seconds, application1 says : 
>            :ClientCnxn$SendThread@] - Client session timed out, have not 
> heard from server in 26679ms for sessionid
>             ConnectionStateManager@237] - State change: SUSPENDED
>         NOTE: I'm ignoring the SUSPENDED event
> 6) After 40 seconds, application2 takes leadership
>  7) After 66 seconds, application1 says :
>              ConnectionStateManager@237] - State change: LOST
>          
>          NOTE: Just at this point, I consider that the application1 has lost 
> leadership
> Then, for 26 seconds, we had 2 leaders.
> If you confirm it as a bug, I think I could help.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (CURATOR-460) Timed tolerance for connection suspended leads to simultaneous leaders

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/CURATOR-460?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16433391#comment-16433391
 ] 

ASF GitHub Bot commented on CURATOR-460:


Github user cammckenzie commented on the issue:

https://github.com/apache/curator/pull/262
  
To Curator, don't they all look the same? Ultimately, Curator will just get 
a Disconnected event?

Given that we can't tell the difference between them, I think that we just 
have to document this as a limitation (tech note and in the relevant classes?) 
and maybe send out an email on the users email list.

I think that we should ultimately just leave the other handling as is. 
Different % values can be passed to StandardConnectionHandlingPolicy to achieve 
the same thing as an AggressiveConnectionHandlingPolicy just by passing in a 
value of 33 yes?

This PR is still relevant though and should be merged.


> Timed tolerance for connection suspended leads to simultaneous leaders
> --
>
> Key: CURATOR-460
> URL: https://issues.apache.org/jira/browse/CURATOR-460
> Project: Apache Curator
>  Issue Type: Bug
>  Components: Recipes
>Affects Versions: 3.3.0, 4.0.0, 4.0.1
>Reporter: Antonio Rafael Rodrigues
>Assignee: Cameron McKenzie
>Priority: Major
> Attachments: ConnectionStateManager.patch, 
> Console-leaderSelector1.log, Console-leaderSelector2.log, 
> CuratorLeaderSelectorPOC.java, LogApp1.log, LogApp2.log
>
>
> Starting from Curator 3, after losing the connection to zookeeper, curator 
> emits a SUSPENDED event and then starts an internal timer, if the time of the 
> negotiatied session timeout get hit and curator didn't reconnect to zookeeper 
> it emits a LOST event.
> For example :
>  Given negotiated session timeout = 40
> ||Time (seconds)||Event||
> |0|SUSPENDED (curator has been disconnected from zookeeper)|
> |40|LOST (curator is still disconnected, it must have been lost the sesion as 
> it is disconnected for 40sec)|
> Given this scenario we could ,theoretically, ignore the SUSPENDED event and 
> consider the leadership as lost just if the ConnectionStateListener receives 
> the LOST event.
> But this feature seems to have introduced a bug (from my point of view)
> *Case of failure*
> ||Time (seconds)||Event||
> |0|Something went wrong with the connected zookeeper (in my case, the network 
> inteface of zookeeper's server has gone down). Curator stops hearing heart 
> beats from zookeeper, but doesn't lose the connection (from some reason that 
> I don't know, if the network interface of the server goes down, Curator 
> doesn't lose the connection)|
> |~26.666|SUSPENDED (after 26 seconds without hearing heartbeats, curator 
> emits a SUSPENDED event) 26 is from "readTimeout = sessionTimeout * 2 / 3" 
> from the class ClientCnxn.java from zookeeper client. At this point, curator 
> starts counting 40 sec.|
> |26.666 to 40|During this period, Curator is trying to connect to other 
> zookeeper instances but the other instances, in my example case, are also 
> uncheachable.|
> |40|Session has expired and the other instance has taken leadership ( in my 
> example, the other instance can connect to zookeeper )|
> |66.666|LOST ( after 40sec from SUSPENDED, curator finally sends LOST)|
> As you can see, if we are ignoring the SUSPENDED event, the second 
> application instance acquires the leadership ~26 seconds before the first 
> instance notice that it lost the leadership.
>  I understand it seems to be a very rare case, but either way I think it 
> should be addressed.
> *Reproduce it*
> I have came up with a way to reproduce this easily. I know this is not a 
> recommended production setup, but it reproduces the problem anyway.
> 1) On a virtual machine, with 2 network interfaces (eth0[192.168.0.101], 
> eth1[192.168.0.102]) , I installed one zookeeper instance.
>  2) I setup application1 with the leadership receipe, with 40sec of 
> negotiated timeout, pointing just to 192.168.0.101  .Now it is the leader
>  3) I setup application2 with the leadership receipe, with 40sec of 
> negotiated timeout, pointing just to 192.168.0.102  .Now it it is NOT the 
> leader
>  4) On the server I turn the eth0[192.168.0.101] interface down  [ ifconfig 
> eth0 down ]
>  5) After 26 seconds, application1 says : 
>            :ClientCnxn$SendThread@] - Client session timed out, have not 
> heard from server in 26679ms for sessionid
>             ConnectionStateManager@237] - State change: SUSPENDED
>         NOTE: I'm ignoring the SUSPENDED event
> 6) After 40 seconds, application2 takes leadership
>  7) After 66 seconds, application1 says :
>              ConnectionStateManager@237] - State change: LOST
>          
>          NOTE: Just at this point, I consider that the 

[jira] [Commented] (CURATOR-460) Timed tolerance for connection suspended leads to simultaneous leaders

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/CURATOR-460?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16433285#comment-16433285
 ] 

ASF GitHub Bot commented on CURATOR-460:


Github user Randgalt commented on the issue:

https://github.com/apache/curator/pull/262
  
> -If the connection between client and server is lost a disconnected event 
is received essentially immediately.
> -If a heart beat is missed it takes 2/3 of the session timeout.

Thinking more about this... There are three scenarios:

1. The internal ZK client detects a missed heartbeat, closes the connection 
and sends `Event.KeeperState.Disconnected` (note: this is done by 
ClientCnxn.java in the SendThread class's run() method - the huge catch 
handler).
2. The server detects a missed heartbeat and closes the connection which 
causes the client to get a closed connection and most likely a 
`SocketException` (which is handled in that same catch clause)
3. The connection fails for other reasons (TCP errors, server crashes, 
etc.). 

Unfortunately, cases 2 and 3 look _exactly the same_ to the client. In case 
2, the client should ideally act as if 2/3 of the session have elapsed. In case 
3, the client should act as if none of the session has elapsed. Worse still, if 
the entire ZK cluster is down, it could conceivably come back up and clients 
won't lose their sessions at all because "time" in ZK is based on the leader's 
start time (I still think there's value in killing the session on the client 
side anyway as clients could be left hanging interminably). 

So, even if we can know the client side heartbeat miss (case 1), we have no 
way of mitigating cases 2/3. I'm not sure what to do. Maybe leave 
`StandardConnectionHandlingPolicy` as is and add a new, optional policy, 
`AggressiveConnectionHandlingPolicy` that always assumes connection loss is due 
to missed heartbeats? Or, possibly, just do nothing and leave things as they 
are?

Thoughts?



> Timed tolerance for connection suspended leads to simultaneous leaders
> --
>
> Key: CURATOR-460
> URL: https://issues.apache.org/jira/browse/CURATOR-460
> Project: Apache Curator
>  Issue Type: Bug
>  Components: Recipes
>Affects Versions: 3.3.0, 4.0.0, 4.0.1
>Reporter: Antonio Rafael Rodrigues
>Assignee: Cameron McKenzie
>Priority: Major
> Attachments: ConnectionStateManager.patch, 
> Console-leaderSelector1.log, Console-leaderSelector2.log, 
> CuratorLeaderSelectorPOC.java, LogApp1.log, LogApp2.log
>
>
> Starting from Curator 3, after losing the connection to zookeeper, curator 
> emits a SUSPENDED event and then starts an internal timer, if the time of the 
> negotiatied session timeout get hit and curator didn't reconnect to zookeeper 
> it emits a LOST event.
> For example :
>  Given negotiated session timeout = 40
> ||Time (seconds)||Event||
> |0|SUSPENDED (curator has been disconnected from zookeeper)|
> |40|LOST (curator is still disconnected, it must have been lost the sesion as 
> it is disconnected for 40sec)|
> Given this scenario we could ,theoretically, ignore the SUSPENDED event and 
> consider the leadership as lost just if the ConnectionStateListener receives 
> the LOST event.
> But this feature seems to have introduced a bug (from my point of view)
> *Case of failure*
> ||Time (seconds)||Event||
> |0|Something went wrong with the connected zookeeper (in my case, the network 
> inteface of zookeeper's server has gone down). Curator stops hearing heart 
> beats from zookeeper, but doesn't lose the connection (from some reason that 
> I don't know, if the network interface of the server goes down, Curator 
> doesn't lose the connection)|
> |~26.666|SUSPENDED (after 26 seconds without hearing heartbeats, curator 
> emits a SUSPENDED event) 26 is from "readTimeout = sessionTimeout * 2 / 3" 
> from the class ClientCnxn.java from zookeeper client. At this point, curator 
> starts counting 40 sec.|
> |26.666 to 40|During this period, Curator is trying to connect to other 
> zookeeper instances but the other instances, in my example case, are also 
> uncheachable.|
> |40|Session has expired and the other instance has taken leadership ( in my 
> example, the other instance can connect to zookeeper )|
> |66.666|LOST ( after 40sec from SUSPENDED, curator finally sends LOST)|
> As you can see, if we are ignoring the SUSPENDED event, the second 
> application instance acquires the leadership ~26 seconds before the first 
> instance notice that it lost the leadership.
>  I understand it seems to be a very rare case, but either way I think it 
> should be addressed.
> *Reproduce it*
> I have came up with a way to reproduce this easily. I know this is not a 
> recommended production setup, but 

[jira] [Commented] (CURATOR-460) Timed tolerance for connection suspended leads to simultaneous leaders

[Cameron McKenzie (JIRA)]

[ 
https://issues.apache.org/jira/browse/CURATOR-460?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16433253#comment-16433253
 ] 

Cameron McKenzie commented on CURATOR-460:
--

Agreed, [~arrodrigues], not sure where the best place to document it is though. 
I don't think that logging a warning is the best option as that will just cause 
a headache for Ops people.

Maybe we just document it as a technote and in the ConnectionHandlingPolicy / 
StandardConnectionHandlingPolicy.

> Timed tolerance for connection suspended leads to simultaneous leaders
> --
>
> Key: CURATOR-460
> URL: https://issues.apache.org/jira/browse/CURATOR-460
> Project: Apache Curator
>  Issue Type: Bug
>  Components: Recipes
>Affects Versions: 3.3.0, 4.0.0, 4.0.1
>Reporter: Antonio Rafael Rodrigues
>Assignee: Cameron McKenzie
>Priority: Major
> Attachments: ConnectionStateManager.patch, 
> Console-leaderSelector1.log, Console-leaderSelector2.log, 
> CuratorLeaderSelectorPOC.java, LogApp1.log, LogApp2.log
>
>
> Starting from Curator 3, after losing the connection to zookeeper, curator 
> emits a SUSPENDED event and then starts an internal timer, if the time of the 
> negotiatied session timeout get hit and curator didn't reconnect to zookeeper 
> it emits a LOST event.
> For example :
>  Given negotiated session timeout = 40
> ||Time (seconds)||Event||
> |0|SUSPENDED (curator has been disconnected from zookeeper)|
> |40|LOST (curator is still disconnected, it must have been lost the sesion as 
> it is disconnected for 40sec)|
> Given this scenario we could ,theoretically, ignore the SUSPENDED event and 
> consider the leadership as lost just if the ConnectionStateListener receives 
> the LOST event.
> But this feature seems to have introduced a bug (from my point of view)
> *Case of failure*
> ||Time (seconds)||Event||
> |0|Something went wrong with the connected zookeeper (in my case, the network 
> inteface of zookeeper's server has gone down). Curator stops hearing heart 
> beats from zookeeper, but doesn't lose the connection (from some reason that 
> I don't know, if the network interface of the server goes down, Curator 
> doesn't lose the connection)|
> |~26.666|SUSPENDED (after 26 seconds without hearing heartbeats, curator 
> emits a SUSPENDED event) 26 is from "readTimeout = sessionTimeout * 2 / 3" 
> from the class ClientCnxn.java from zookeeper client. At this point, curator 
> starts counting 40 sec.|
> |26.666 to 40|During this period, Curator is trying to connect to other 
> zookeeper instances but the other instances, in my example case, are also 
> uncheachable.|
> |40|Session has expired and the other instance has taken leadership ( in my 
> example, the other instance can connect to zookeeper )|
> |66.666|LOST ( after 40sec from SUSPENDED, curator finally sends LOST)|
> As you can see, if we are ignoring the SUSPENDED event, the second 
> application instance acquires the leadership ~26 seconds before the first 
> instance notice that it lost the leadership.
>  I understand it seems to be a very rare case, but either way I think it 
> should be addressed.
> *Reproduce it*
> I have came up with a way to reproduce this easily. I know this is not a 
> recommended production setup, but it reproduces the problem anyway.
> 1) On a virtual machine, with 2 network interfaces (eth0[192.168.0.101], 
> eth1[192.168.0.102]) , I installed one zookeeper instance.
>  2) I setup application1 with the leadership receipe, with 40sec of 
> negotiated timeout, pointing just to 192.168.0.101  .Now it is the leader
>  3) I setup application2 with the leadership receipe, with 40sec of 
> negotiated timeout, pointing just to 192.168.0.102  .Now it it is NOT the 
> leader
>  4) On the server I turn the eth0[192.168.0.101] interface down  [ ifconfig 
> eth0 down ]
>  5) After 26 seconds, application1 says : 
>            :ClientCnxn$SendThread@] - Client session timed out, have not 
> heard from server in 26679ms for sessionid
>             ConnectionStateManager@237] - State change: SUSPENDED
>         NOTE: I'm ignoring the SUSPENDED event
> 6) After 40 seconds, application2 takes leadership
>  7) After 66 seconds, application1 says :
>              ConnectionStateManager@237] - State change: LOST
>          
>          NOTE: Just at this point, I consider that the application1 has lost 
> leadership
> Then, for 26 seconds, we had 2 leaders.
> If you confirm it as a bug, I think I could help.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (CURATOR-460) Timed tolerance for connection suspended leads to simultaneous leaders

[Antonio Rafael Rodrigues (JIRA)]

[ 
https://issues.apache.org/jira/browse/CURATOR-460?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16433235#comment-16433235
 ] 

Antonio Rafael Rodrigues commented on CURATOR-460:
--

[~cammckenzie] - You totally understood  my point. 
But, in my opinion, even though the "missing heart beats"  is rare, Curator 
should consider it. Curator should at least warn the user about this 
possibility in some way (log warning, documentation), this behavior should not 
be silent.

As a user, I expect that if I say that I want 100% of tolerance of 
disconnection, I would have it, not 133%.

> Timed tolerance for connection suspended leads to simultaneous leaders
> --
>
> Key: CURATOR-460
> URL: https://issues.apache.org/jira/browse/CURATOR-460
> Project: Apache Curator
>  Issue Type: Bug
>  Components: Recipes
>Affects Versions: 3.3.0, 4.0.0, 4.0.1
>Reporter: Antonio Rafael Rodrigues
>Assignee: Cameron McKenzie
>Priority: Major
> Attachments: ConnectionStateManager.patch, 
> Console-leaderSelector1.log, Console-leaderSelector2.log, 
> CuratorLeaderSelectorPOC.java, LogApp1.log, LogApp2.log
>
>
> Starting from Curator 3, after losing the connection to zookeeper, curator 
> emits a SUSPENDED event and then starts an internal timer, if the time of the 
> negotiatied session timeout get hit and curator didn't reconnect to zookeeper 
> it emits a LOST event.
> For example :
>  Given negotiated session timeout = 40
> ||Time (seconds)||Event||
> |0|SUSPENDED (curator has been disconnected from zookeeper)|
> |40|LOST (curator is still disconnected, it must have been lost the sesion as 
> it is disconnected for 40sec)|
> Given this scenario we could ,theoretically, ignore the SUSPENDED event and 
> consider the leadership as lost just if the ConnectionStateListener receives 
> the LOST event.
> But this feature seems to have introduced a bug (from my point of view)
> *Case of failure*
> ||Time (seconds)||Event||
> |0|Something went wrong with the connected zookeeper (in my case, the network 
> inteface of zookeeper's server has gone down). Curator stops hearing heart 
> beats from zookeeper, but doesn't lose the connection (from some reason that 
> I don't know, if the network interface of the server goes down, Curator 
> doesn't lose the connection)|
> |~26.666|SUSPENDED (after 26 seconds without hearing heartbeats, curator 
> emits a SUSPENDED event) 26 is from "readTimeout = sessionTimeout * 2 / 3" 
> from the class ClientCnxn.java from zookeeper client. At this point, curator 
> starts counting 40 sec.|
> |26.666 to 40|During this period, Curator is trying to connect to other 
> zookeeper instances but the other instances, in my example case, are also 
> uncheachable.|
> |40|Session has expired and the other instance has taken leadership ( in my 
> example, the other instance can connect to zookeeper )|
> |66.666|LOST ( after 40sec from SUSPENDED, curator finally sends LOST)|
> As you can see, if we are ignoring the SUSPENDED event, the second 
> application instance acquires the leadership ~26 seconds before the first 
> instance notice that it lost the leadership.
>  I understand it seems to be a very rare case, but either way I think it 
> should be addressed.
> *Reproduce it*
> I have came up with a way to reproduce this easily. I know this is not a 
> recommended production setup, but it reproduces the problem anyway.
> 1) On a virtual machine, with 2 network interfaces (eth0[192.168.0.101], 
> eth1[192.168.0.102]) , I installed one zookeeper instance.
>  2) I setup application1 with the leadership receipe, with 40sec of 
> negotiated timeout, pointing just to 192.168.0.101  .Now it is the leader
>  3) I setup application2 with the leadership receipe, with 40sec of 
> negotiated timeout, pointing just to 192.168.0.102  .Now it it is NOT the 
> leader
>  4) On the server I turn the eth0[192.168.0.101] interface down  [ ifconfig 
> eth0 down ]
>  5) After 26 seconds, application1 says : 
>            :ClientCnxn$SendThread@] - Client session timed out, have not 
> heard from server in 26679ms for sessionid
>             ConnectionStateManager@237] - State change: SUSPENDED
>         NOTE: I'm ignoring the SUSPENDED event
> 6) After 40 seconds, application2 takes leadership
>  7) After 66 seconds, application1 says :
>              ConnectionStateManager@237] - State change: LOST
>          
>          NOTE: Just at this point, I consider that the application1 has lost 
> leadership
> Then, for 26 seconds, we had 2 leaders.
> If you confirm it as a bug, I think I could help.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (CURATOR-460) Timed tolerance for connection suspended leads to simultaneous leaders

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/CURATOR-460?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16433068#comment-16433068
 ] 

ASF GitHub Bot commented on CURATOR-460:


Github user cammckenzie commented on the issue:

https://github.com/apache/curator/pull/262
  
My issue is the difference in timing in these events:
-If the connection between client and server is lost a disconnected event 
is received essentially immediately.
-If a heart beat is missed it takes 2/3 of the session timeout.

So, we can't reliably do anything with timers because we can't tell the 
difference between these two events. In the first case, ideally, we would 
inject a LOST event after the negotiated session timeout MS. In the second 
case, ideally, we would inject a LOST event after 1/3 of the negotiated session 
timeout MS.

For a short session timeout the difference is minimal, but for a longer 
session timeout, it will be significant. I guess all we can do is the approach 
that we already have where the client of the Curator library can decide what % 
of the session timeout they want to use before a session timeout is injected. I 
would think that in general, connection loss is more likely than missed heart 
beats, so maybe we should just leave the default % of session timeout as it is 
at 100% as this will be correct for the most likely case.


> Timed tolerance for connection suspended leads to simultaneous leaders
> --
>
> Key: CURATOR-460
> URL: https://issues.apache.org/jira/browse/CURATOR-460
> Project: Apache Curator
>  Issue Type: Bug
>  Components: Recipes
>Affects Versions: 3.3.0, 4.0.0, 4.0.1
>Reporter: Antonio Rafael Rodrigues
>Assignee: Cameron McKenzie
>Priority: Major
> Attachments: ConnectionStateManager.patch, 
> Console-leaderSelector1.log, Console-leaderSelector2.log, 
> CuratorLeaderSelectorPOC.java, LogApp1.log, LogApp2.log
>
>
> Starting from Curator 3, after losing the connection to zookeeper, curator 
> emits a SUSPENDED event and then starts an internal timer, if the time of the 
> negotiatied session timeout get hit and curator didn't reconnect to zookeeper 
> it emits a LOST event.
> For example :
>  Given negotiated session timeout = 40
> ||Time (seconds)||Event||
> |0|SUSPENDED (curator has been disconnected from zookeeper)|
> |40|LOST (curator is still disconnected, it must have been lost the sesion as 
> it is disconnected for 40sec)|
> Given this scenario we could ,theoretically, ignore the SUSPENDED event and 
> consider the leadership as lost just if the ConnectionStateListener receives 
> the LOST event.
> But this feature seems to have introduced a bug (from my point of view)
> *Case of failure*
> ||Time (seconds)||Event||
> |0|Something went wrong with the connected zookeeper (in my case, the network 
> inteface of zookeeper's server has gone down). Curator stops hearing heart 
> beats from zookeeper, but doesn't lose the connection (from some reason that 
> I don't know, if the network interface of the server goes down, Curator 
> doesn't lose the connection)|
> |~26.666|SUSPENDED (after 26 seconds without hearing heartbeats, curator 
> emits a SUSPENDED event) 26 is from "readTimeout = sessionTimeout * 2 / 3" 
> from the class ClientCnxn.java from zookeeper client. At this point, curator 
> starts counting 40 sec.|
> |26.666 to 40|During this period, Curator is trying to connect to other 
> zookeeper instances but the other instances, in my example case, are also 
> uncheachable.|
> |40|Session has expired and the other instance has taken leadership ( in my 
> example, the other instance can connect to zookeeper )|
> |66.666|LOST ( after 40sec from SUSPENDED, curator finally sends LOST)|
> As you can see, if we are ignoring the SUSPENDED event, the second 
> application instance acquires the leadership ~26 seconds before the first 
> instance notice that it lost the leadership.
>  I understand it seems to be a very rare case, but either way I think it 
> should be addressed.
> *Reproduce it*
> I have came up with a way to reproduce this easily. I know this is not a 
> recommended production setup, but it reproduces the problem anyway.
> 1) On a virtual machine, with 2 network interfaces (eth0[192.168.0.101], 
> eth1[192.168.0.102]) , I installed one zookeeper instance.
>  2) I setup application1 with the leadership receipe, with 40sec of 
> negotiated timeout, pointing just to 192.168.0.101  .Now it is the leader
>  3) I setup application2 with the leadership receipe, with 40sec of 
> negotiated timeout, pointing just to 192.168.0.102  .Now it it is NOT the 
> leader
>  4) On the server I turn the eth0[192.168.0.101] interface down  [ ifconfig 
> eth0 down ]
>  5) After 26 seconds, application1 says : 
>          

[jira] [Commented] (CURATOR-460) Timed tolerance for connection suspended leads to simultaneous leaders

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/CURATOR-460?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16431974#comment-16431974
 ] 

ASF GitHub Bot commented on CURATOR-460:


Github user Randgalt commented on the issue:

https://github.com/apache/curator/pull/262
  
The ZK client code sends Disconnected when the Server disconnects or the 
server misses a heartbeat. So, that should be sufficient right?


> Timed tolerance for connection suspended leads to simultaneous leaders
> --
>
> Key: CURATOR-460
> URL: https://issues.apache.org/jira/browse/CURATOR-460
> Project: Apache Curator
>  Issue Type: Bug
>  Components: Recipes
>Affects Versions: 3.3.0, 4.0.0, 4.0.1
>Reporter: Antonio Rafael Rodrigues
>Assignee: Cameron McKenzie
>Priority: Major
> Attachments: ConnectionStateManager.patch, 
> Console-leaderSelector1.log, Console-leaderSelector2.log, 
> CuratorLeaderSelectorPOC.java, LogApp1.log, LogApp2.log
>
>
> Starting from Curator 3, after losing the connection to zookeeper, curator 
> emits a SUSPENDED event and then starts an internal timer, if the time of the 
> negotiatied session timeout get hit and curator didn't reconnect to zookeeper 
> it emits a LOST event.
> For example :
>  Given negotiated session timeout = 40
> ||Time (seconds)||Event||
> |0|SUSPENDED (curator has been disconnected from zookeeper)|
> |40|LOST (curator is still disconnected, it must have been lost the sesion as 
> it is disconnected for 40sec)|
> Given this scenario we could ,theoretically, ignore the SUSPENDED event and 
> consider the leadership as lost just if the ConnectionStateListener receives 
> the LOST event.
> But this feature seems to have introduced a bug (from my point of view)
> *Case of failure*
> ||Time (seconds)||Event||
> |0|Something went wrong with the connected zookeeper (in my case, the network 
> inteface of zookeeper's server has gone down). Curator stops hearing heart 
> beats from zookeeper, but doesn't lose the connection (from some reason that 
> I don't know, if the network interface of the server goes down, Curator 
> doesn't lose the connection)|
> |~26.666|SUSPENDED (after 26 seconds without hearing heartbeats, curator 
> emits a SUSPENDED event) 26 is from "readTimeout = sessionTimeout * 2 / 3" 
> from the class ClientCnxn.java from zookeeper client. At this point, curator 
> starts counting 40 sec.|
> |26.666 to 40|During this period, Curator is trying to connect to other 
> zookeeper instances but the other instances, in my example case, are also 
> uncheachable.|
> |40|Session has expired and the other instance has taken leadership ( in my 
> example, the other instance can connect to zookeeper )|
> |66.666|LOST ( after 40sec from SUSPENDED, curator finally sends LOST)|
> As you can see, if we are ignoring the SUSPENDED event, the second 
> application instance acquires the leadership ~26 seconds before the first 
> instance notice that it lost the leadership.
>  I understand it seems to be a very rare case, but either way I think it 
> should be addressed.
> *Reproduce it*
> I have came up with a way to reproduce this easily. I know this is not a 
> recommended production setup, but it reproduces the problem anyway.
> 1) On a virtual machine, with 2 network interfaces (eth0[192.168.0.101], 
> eth1[192.168.0.102]) , I installed one zookeeper instance.
>  2) I setup application1 with the leadership receipe, with 40sec of 
> negotiated timeout, pointing just to 192.168.0.101  .Now it is the leader
>  3) I setup application2 with the leadership receipe, with 40sec of 
> negotiated timeout, pointing just to 192.168.0.102  .Now it it is NOT the 
> leader
>  4) On the server I turn the eth0[192.168.0.101] interface down  [ ifconfig 
> eth0 down ]
>  5) After 26 seconds, application1 says : 
>            :ClientCnxn$SendThread@] - Client session timed out, have not 
> heard from server in 26679ms for sessionid
>             ConnectionStateManager@237] - State change: SUSPENDED
>         NOTE: I'm ignoring the SUSPENDED event
> 6) After 40 seconds, application2 takes leadership
>  7) After 66 seconds, application1 says :
>              ConnectionStateManager@237] - State change: LOST
>          
>          NOTE: Just at this point, I consider that the application1 has lost 
> leadership
> Then, for 26 seconds, we had 2 leaders.
> If you confirm it as a bug, I think I could help.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (CURATOR-460) Timed tolerance for connection suspended leads to simultaneous leaders

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/CURATOR-460?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16431402#comment-16431402
 ] 

ASF GitHub Bot commented on CURATOR-460:


Github user cammckenzie commented on the issue:

https://github.com/apache/curator/pull/262
  
I'm conflicted about the LOST injection stuff. I guess all we can do is err 
on the side of caution and leave it as is. That will cover what is presumably 
the most common case, where the connection to ZK is lost rather than the case 
this JIRA covers where the connection is still up but there's no heart beat.

While we will never be able to exactly match the timing on the server side, 
I think that we should strive to have a reasonable approximation of it. Not 
being able to tell when the last heart beat was successfully sent to the server 
is problematic.


> Timed tolerance for connection suspended leads to simultaneous leaders
> --
>
> Key: CURATOR-460
> URL: https://issues.apache.org/jira/browse/CURATOR-460
> Project: Apache Curator
>  Issue Type: Bug
>  Components: Recipes
>Affects Versions: 3.3.0, 4.0.0, 4.0.1
>Reporter: Antonio Rafael Rodrigues
>Assignee: Cameron McKenzie
>Priority: Major
> Attachments: ConnectionStateManager.patch, 
> Console-leaderSelector1.log, Console-leaderSelector2.log, 
> CuratorLeaderSelectorPOC.java, LogApp1.log, LogApp2.log
>
>
> Starting from Curator 3, after losing the connection to zookeeper, curator 
> emits a SUSPENDED event and then starts an internal timer, if the time of the 
> negotiatied session timeout get hit and curator didn't reconnect to zookeeper 
> it emits a LOST event.
> For example :
>  Given negotiated session timeout = 40
> ||Time (seconds)||Event||
> |0|SUSPENDED (curator has been disconnected from zookeeper)|
> |40|LOST (curator is still disconnected, it must have been lost the sesion as 
> it is disconnected for 40sec)|
> Given this scenario we could ,theoretically, ignore the SUSPENDED event and 
> consider the leadership as lost just if the ConnectionStateListener receives 
> the LOST event.
> But this feature seems to have introduced a bug (from my point of view)
> *Case of failure*
> ||Time (seconds)||Event||
> |0|Something went wrong with the connected zookeeper (in my case, the network 
> inteface of zookeeper's server has gone down). Curator stops hearing heart 
> beats from zookeeper, but doesn't lose the connection (from some reason that 
> I don't know, if the network interface of the server goes down, Curator 
> doesn't lose the connection)|
> |~26.666|SUSPENDED (after 26 seconds without hearing heartbeats, curator 
> emits a SUSPENDED event) 26 is from "readTimeout = sessionTimeout * 2 / 3" 
> from the class ClientCnxn.java from zookeeper client. At this point, curator 
> starts counting 40 sec.|
> |26.666 to 40|During this period, Curator is trying to connect to other 
> zookeeper instances but the other instances, in my example case, are also 
> uncheachable.|
> |40|Session has expired and the other instance has taken leadership ( in my 
> example, the other instance can connect to zookeeper )|
> |66.666|LOST ( after 40sec from SUSPENDED, curator finally sends LOST)|
> As you can see, if we are ignoring the SUSPENDED event, the second 
> application instance acquires the leadership ~26 seconds before the first 
> instance notice that it lost the leadership.
>  I understand it seems to be a very rare case, but either way I think it 
> should be addressed.
> *Reproduce it*
> I have came up with a way to reproduce this easily. I know this is not a 
> recommended production setup, but it reproduces the problem anyway.
> 1) On a virtual machine, with 2 network interfaces (eth0[192.168.0.101], 
> eth1[192.168.0.102]) , I installed one zookeeper instance.
>  2) I setup application1 with the leadership receipe, with 40sec of 
> negotiated timeout, pointing just to 192.168.0.101  .Now it is the leader
>  3) I setup application2 with the leadership receipe, with 40sec of 
> negotiated timeout, pointing just to 192.168.0.102  .Now it it is NOT the 
> leader
>  4) On the server I turn the eth0[192.168.0.101] interface down  [ ifconfig 
> eth0 down ]
>  5) After 26 seconds, application1 says : 
>            :ClientCnxn$SendThread@] - Client session timed out, have not 
> heard from server in 26679ms for sessionid
>             ConnectionStateManager@237] - State change: SUSPENDED
>         NOTE: I'm ignoring the SUSPENDED event
> 6) After 40 seconds, application2 takes leadership
>  7) After 66 seconds, application1 says :
>              ConnectionStateManager@237] - State change: LOST
>          
>          NOTE: Just at this point, I consider that the application1 has lost 
> leadership
> Then, for 26 seconds, we had 2 

[jira] [Commented] (CURATOR-460) Timed tolerance for connection suspended leads to simultaneous leaders

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/CURATOR-460?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16429942#comment-16429942
 ] 

ASF GitHub Bot commented on CURATOR-460:


Github user Randgalt commented on the issue:

https://github.com/apache/curator/pull/262
  
I'd be -1 on more hacking into ZooKeeper. Trying to get all these timings 
to be accurate is a lost cause anyway - we can never simulate in the client the 
exact timing of the server. I suggest adding a configurable (via CLI) "slosh" 
value or something like that.


> Timed tolerance for connection suspended leads to simultaneous leaders
> --
>
> Key: CURATOR-460
> URL: https://issues.apache.org/jira/browse/CURATOR-460
> Project: Apache Curator
>  Issue Type: Bug
>  Components: Recipes
>Affects Versions: 3.3.0, 4.0.0, 4.0.1
>Reporter: Antonio Rafael Rodrigues
>Assignee: Cameron McKenzie
>Priority: Major
> Attachments: ConnectionStateManager.patch, 
> Console-leaderSelector1.log, Console-leaderSelector2.log, 
> CuratorLeaderSelectorPOC.java, LogApp1.log, LogApp2.log
>
>
> Starting from Curator 3, after losing the connection to zookeeper, curator 
> emits a SUSPENDED event and then starts an internal timer, if the time of the 
> negotiatied session timeout get hit and curator didn't reconnect to zookeeper 
> it emits a LOST event.
> For example :
>  Given negotiated session timeout = 40
> ||Time (seconds)||Event||
> |0|SUSPENDED (curator has been disconnected from zookeeper)|
> |40|LOST (curator is still disconnected, it must have been lost the sesion as 
> it is disconnected for 40sec)|
> Given this scenario we could ,theoretically, ignore the SUSPENDED event and 
> consider the leadership as lost just if the ConnectionStateListener receives 
> the LOST event.
> But this feature seems to have introduced a bug (from my point of view)
> *Case of failure*
> ||Time (seconds)||Event||
> |0|Something went wrong with the connected zookeeper (in my case, the network 
> inteface of zookeeper's server has gone down). Curator stops hearing heart 
> beats from zookeeper, but doesn't lose the connection (from some reason that 
> I don't know, if the network interface of the server goes down, Curator 
> doesn't lose the connection)|
> |~26.666|SUSPENDED (after 26 seconds without hearing heartbeats, curator 
> emits a SUSPENDED event) 26 is from "readTimeout = sessionTimeout * 2 / 3" 
> from the class ClientCnxn.java from zookeeper client. At this point, curator 
> starts counting 40 sec.|
> |26.666 to 40|During this period, Curator is trying to connect to other 
> zookeeper instances but the other instances, in my example case, are also 
> uncheachable.|
> |40|Session has expired and the other instance has taken leadership ( in my 
> example, the other instance can connect to zookeeper )|
> |66.666|LOST ( after 40sec from SUSPENDED, curator finally sends LOST)|
> As you can see, if we are ignoring the SUSPENDED event, the second 
> application instance acquires the leadership ~26 seconds before the first 
> instance notice that it lost the leadership.
>  I understand it seems to be a very rare case, but either way I think it 
> should be addressed.
> *Reproduce it*
> I have came up with a way to reproduce this easily. I know this is not a 
> recommended production setup, but it reproduces the problem anyway.
> 1) On a virtual machine, with 2 network interfaces (eth0[192.168.0.101], 
> eth1[192.168.0.102]) , I installed one zookeeper instance.
>  2) I setup application1 with the leadership receipe, with 40sec of 
> negotiated timeout, pointing just to 192.168.0.101  .Now it is the leader
>  3) I setup application2 with the leadership receipe, with 40sec of 
> negotiated timeout, pointing just to 192.168.0.102  .Now it it is NOT the 
> leader
>  4) On the server I turn the eth0[192.168.0.101] interface down  [ ifconfig 
> eth0 down ]
>  5) After 26 seconds, application1 says : 
>            :ClientCnxn$SendThread@] - Client session timed out, have not 
> heard from server in 26679ms for sessionid
>             ConnectionStateManager@237] - State change: SUSPENDED
>         NOTE: I'm ignoring the SUSPENDED event
> 6) After 40 seconds, application2 takes leadership
>  7) After 66 seconds, application1 says :
>              ConnectionStateManager@237] - State change: LOST
>          
>          NOTE: Just at this point, I consider that the application1 has lost 
> leadership
> Then, for 26 seconds, we had 2 leaders.
> If you confirm it as a bug, I think I could help.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (CURATOR-460) Timed tolerance for connection suspended leads to simultaneous leaders

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/CURATOR-460?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16429910#comment-16429910
 ] 

ASF GitHub Bot commented on CURATOR-460:


Github user cammckenzie commented on the issue:

https://github.com/apache/curator/pull/262
  
That's a good point @arrodrigues 

I don't think we can tell the difference between Disconnected event from ZK 
due to loss of the underlying socket and a Disconected event due to no 
heartbeats from the server for 2/3 of session timeout, which, from memory is 
what you were describing at the start of this conversation long ago.

With a bit of hacking, I believe that could extended the Zookeeper class to 
allow Curator to get access to the ClientCnxn object and from that, the 
underlying socket. I don't know if this is the right approach though, it will 
certainly couple Curator more tightly to ZK.

Thoughts @Randgalt ?


> Timed tolerance for connection suspended leads to simultaneous leaders
> --
>
> Key: CURATOR-460
> URL: https://issues.apache.org/jira/browse/CURATOR-460
> Project: Apache Curator
>  Issue Type: Bug
>  Components: Recipes
>Affects Versions: 3.3.0, 4.0.0, 4.0.1
>Reporter: Antonio Rafael Rodrigues
>Assignee: Cameron McKenzie
>Priority: Major
> Attachments: ConnectionStateManager.patch, 
> Console-leaderSelector1.log, Console-leaderSelector2.log, 
> CuratorLeaderSelectorPOC.java, LogApp1.log, LogApp2.log
>
>
> Starting from Curator 3, after losing the connection to zookeeper, curator 
> emits a SUSPENDED event and then starts an internal timer, if the time of the 
> negotiatied session timeout get hit and curator didn't reconnect to zookeeper 
> it emits a LOST event.
> For example :
>  Given negotiated session timeout = 40
> ||Time (seconds)||Event||
> |0|SUSPENDED (curator has been disconnected from zookeeper)|
> |40|LOST (curator is still disconnected, it must have been lost the sesion as 
> it is disconnected for 40sec)|
> Given this scenario we could ,theoretically, ignore the SUSPENDED event and 
> consider the leadership as lost just if the ConnectionStateListener receives 
> the LOST event.
> But this feature seems to have introduced a bug (from my point of view)
> *Case of failure*
> ||Time (seconds)||Event||
> |0|Something went wrong with the connected zookeeper (in my case, the network 
> inteface of zookeeper's server has gone down). Curator stops hearing heart 
> beats from zookeeper, but doesn't lose the connection (from some reason that 
> I don't know, if the network interface of the server goes down, Curator 
> doesn't lose the connection)|
> |~26.666|SUSPENDED (after 26 seconds without hearing heartbeats, curator 
> emits a SUSPENDED event) 26 is from "readTimeout = sessionTimeout * 2 / 3" 
> from the class ClientCnxn.java from zookeeper client. At this point, curator 
> starts counting 40 sec.|
> |26.666 to 40|During this period, Curator is trying to connect to other 
> zookeeper instances but the other instances, in my example case, are also 
> uncheachable.|
> |40|Session has expired and the other instance has taken leadership ( in my 
> example, the other instance can connect to zookeeper )|
> |66.666|LOST ( after 40sec from SUSPENDED, curator finally sends LOST)|
> As you can see, if we are ignoring the SUSPENDED event, the second 
> application instance acquires the leadership ~26 seconds before the first 
> instance notice that it lost the leadership.
>  I understand it seems to be a very rare case, but either way I think it 
> should be addressed.
> *Reproduce it*
> I have came up with a way to reproduce this easily. I know this is not a 
> recommended production setup, but it reproduces the problem anyway.
> 1) On a virtual machine, with 2 network interfaces (eth0[192.168.0.101], 
> eth1[192.168.0.102]) , I installed one zookeeper instance.
>  2) I setup application1 with the leadership receipe, with 40sec of 
> negotiated timeout, pointing just to 192.168.0.101  .Now it is the leader
>  3) I setup application2 with the leadership receipe, with 40sec of 
> negotiated timeout, pointing just to 192.168.0.102  .Now it it is NOT the 
> leader
>  4) On the server I turn the eth0[192.168.0.101] interface down  [ ifconfig 
> eth0 down ]
>  5) After 26 seconds, application1 says : 
>            :ClientCnxn$SendThread@] - Client session timed out, have not 
> heard from server in 26679ms for sessionid
>             ConnectionStateManager@237] - State change: SUSPENDED
>         NOTE: I'm ignoring the SUSPENDED event
> 6) After 40 seconds, application2 takes leadership
>  7) After 66 seconds, application1 says :
>              ConnectionStateManager@237] - State change: LOST
>          
>          NOTE: Just at this point, I consider 

[jira] [Commented] (CURATOR-460) Timed tolerance for connection suspended leads to simultaneous leaders

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/CURATOR-460?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16429204#comment-16429204
 ] 

ASF GitHub Bot commented on CURATOR-460:


Github user arrodrigues commented on the issue:

https://github.com/apache/curator/pull/262
  
@cammckenzie It is on 2/3 of session timeout. 
I'd like to remember this will happen just in the scenario that I described 
in the issue description, in many other network issues the suspended event 
would came immediatly.
Beyond the 33%, shouldn't we consider adding an margin of error, like using 
30%, 25% ? Because besides the 33% of session timeout, we could also have some 
delay to execute the leadership relinquish. Furthermore, I think ZK's ticktime, 
that generally is 2sec, also can potentially adds more delay to the formula. 



> Timed tolerance for connection suspended leads to simultaneous leaders
> --
>
> Key: CURATOR-460
> URL: https://issues.apache.org/jira/browse/CURATOR-460
> Project: Apache Curator
>  Issue Type: Bug
>  Components: Recipes
>Affects Versions: 3.3.0, 4.0.0, 4.0.1
>Reporter: Antonio Rafael Rodrigues
>Assignee: Cameron McKenzie
>Priority: Major
> Attachments: ConnectionStateManager.patch, 
> Console-leaderSelector1.log, Console-leaderSelector2.log, 
> CuratorLeaderSelectorPOC.java, LogApp1.log, LogApp2.log
>
>
> Starting from Curator 3, after losing the connection to zookeeper, curator 
> emits a SUSPENDED event and then starts an internal timer, if the time of the 
> negotiatied session timeout get hit and curator didn't reconnect to zookeeper 
> it emits a LOST event.
> For example :
>  Given negotiated session timeout = 40
> ||Time (seconds)||Event||
> |0|SUSPENDED (curator has been disconnected from zookeeper)|
> |40|LOST (curator is still disconnected, it must have been lost the sesion as 
> it is disconnected for 40sec)|
> Given this scenario we could ,theoretically, ignore the SUSPENDED event and 
> consider the leadership as lost just if the ConnectionStateListener receives 
> the LOST event.
> But this feature seems to have introduced a bug (from my point of view)
> *Case of failure*
> ||Time (seconds)||Event||
> |0|Something went wrong with the connected zookeeper (in my case, the network 
> inteface of zookeeper's server has gone down). Curator stops hearing heart 
> beats from zookeeper, but doesn't lose the connection (from some reason that 
> I don't know, if the network interface of the server goes down, Curator 
> doesn't lose the connection)|
> |~26.666|SUSPENDED (after 26 seconds without hearing heartbeats, curator 
> emits a SUSPENDED event) 26 is from "readTimeout = sessionTimeout * 2 / 3" 
> from the class ClientCnxn.java from zookeeper client. At this point, curator 
> starts counting 40 sec.|
> |26.666 to 40|During this period, Curator is trying to connect to other 
> zookeeper instances but the other instances, in my example case, are also 
> uncheachable.|
> |40|Session has expired and the other instance has taken leadership ( in my 
> example, the other instance can connect to zookeeper )|
> |66.666|LOST ( after 40sec from SUSPENDED, curator finally sends LOST)|
> As you can see, if we are ignoring the SUSPENDED event, the second 
> application instance acquires the leadership ~26 seconds before the first 
> instance notice that it lost the leadership.
>  I understand it seems to be a very rare case, but either way I think it 
> should be addressed.
> *Reproduce it*
> I have came up with a way to reproduce this easily. I know this is not a 
> recommended production setup, but it reproduces the problem anyway.
> 1) On a virtual machine, with 2 network interfaces (eth0[192.168.0.101], 
> eth1[192.168.0.102]) , I installed one zookeeper instance.
>  2) I setup application1 with the leadership receipe, with 40sec of 
> negotiated timeout, pointing just to 192.168.0.101  .Now it is the leader
>  3) I setup application2 with the leadership receipe, with 40sec of 
> negotiated timeout, pointing just to 192.168.0.102  .Now it it is NOT the 
> leader
>  4) On the server I turn the eth0[192.168.0.101] interface down  [ ifconfig 
> eth0 down ]
>  5) After 26 seconds, application1 says : 
>            :ClientCnxn$SendThread@] - Client session timed out, have not 
> heard from server in 26679ms for sessionid
>             ConnectionStateManager@237] - State change: SUSPENDED
>         NOTE: I'm ignoring the SUSPENDED event
> 6) After 40 seconds, application2 takes leadership
>  7) After 66 seconds, application1 says :
>              ConnectionStateManager@237] - State change: LOST
>          
>          NOTE: Just at this point, I consider that the application1 has lost 
> leadership
> Then, for 26 seconds, we had 2 leaders.
> If you confirm 

[jira] [Commented] (CURATOR-460) Timed tolerance for connection suspended leads to simultaneous leaders

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/CURATOR-460?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16427987#comment-16427987
 ] 

ASF GitHub Bot commented on CURATOR-460:


Github user cammckenzie commented on the issue:

https://github.com/apache/curator/pull/262
  
The changes look good to me, thanks @arrodrigues 

@Randgalt , I think the approach you're suggesting for setting up the 
default session timeout stuff looks OK. I thought that the suspended event came 
at 1/3 of the session timeout, which is why I was suggesting 66%. I will need 
to double check. If it's 2/3 of session timeout, then yes, 33%


> Timed tolerance for connection suspended leads to simultaneous leaders
> --
>
> Key: CURATOR-460
> URL: https://issues.apache.org/jira/browse/CURATOR-460
> Project: Apache Curator
>  Issue Type: Bug
>  Components: Recipes
>Affects Versions: 3.3.0, 4.0.0, 4.0.1
>Reporter: Antonio Rafael Rodrigues
>Assignee: Cameron McKenzie
>Priority: Major
> Attachments: ConnectionStateManager.patch, 
> Console-leaderSelector1.log, Console-leaderSelector2.log, 
> CuratorLeaderSelectorPOC.java, LogApp1.log, LogApp2.log
>
>
> Starting from Curator 3, after losing the connection to zookeeper, curator 
> emits a SUSPENDED event and then starts an internal timer, if the time of the 
> negotiatied session timeout get hit and curator didn't reconnect to zookeeper 
> it emits a LOST event.
> For example :
>  Given negotiated session timeout = 40
> ||Time (seconds)||Event||
> |0|SUSPENDED (curator has been disconnected from zookeeper)|
> |40|LOST (curator is still disconnected, it must have been lost the sesion as 
> it is disconnected for 40sec)|
> Given this scenario we could ,theoretically, ignore the SUSPENDED event and 
> consider the leadership as lost just if the ConnectionStateListener receives 
> the LOST event.
> But this feature seems to have introduced a bug (from my point of view)
> *Case of failure*
> ||Time (seconds)||Event||
> |0|Something went wrong with the connected zookeeper (in my case, the network 
> inteface of zookeeper's server has gone down). Curator stops hearing heart 
> beats from zookeeper, but doesn't lose the connection (from some reason that 
> I don't know, if the network interface of the server goes down, Curator 
> doesn't lose the connection)|
> |~26.666|SUSPENDED (after 26 seconds without hearing heartbeats, curator 
> emits a SUSPENDED event) 26 is from "readTimeout = sessionTimeout * 2 / 3" 
> from the class ClientCnxn.java from zookeeper client. At this point, curator 
> starts counting 40 sec.|
> |26.666 to 40|During this period, Curator is trying to connect to other 
> zookeeper instances but the other instances, in my example case, are also 
> uncheachable.|
> |40|Session has expired and the other instance has taken leadership ( in my 
> example, the other instance can connect to zookeeper )|
> |66.666|LOST ( after 40sec from SUSPENDED, curator finally sends LOST)|
> As you can see, if we are ignoring the SUSPENDED event, the second 
> application instance acquires the leadership ~26 seconds before the first 
> instance notice that it lost the leadership.
>  I understand it seems to be a very rare case, but either way I think it 
> should be addressed.
> *Reproduce it*
> I have came up with a way to reproduce this easily. I know this is not a 
> recommended production setup, but it reproduces the problem anyway.
> 1) On a virtual machine, with 2 network interfaces (eth0[192.168.0.101], 
> eth1[192.168.0.102]) , I installed one zookeeper instance.
>  2) I setup application1 with the leadership receipe, with 40sec of 
> negotiated timeout, pointing just to 192.168.0.101  .Now it is the leader
>  3) I setup application2 with the leadership receipe, with 40sec of 
> negotiated timeout, pointing just to 192.168.0.102  .Now it it is NOT the 
> leader
>  4) On the server I turn the eth0[192.168.0.101] interface down  [ ifconfig 
> eth0 down ]
>  5) After 26 seconds, application1 says : 
>            :ClientCnxn$SendThread@] - Client session timed out, have not 
> heard from server in 26679ms for sessionid
>             ConnectionStateManager@237] - State change: SUSPENDED
>         NOTE: I'm ignoring the SUSPENDED event
> 6) After 40 seconds, application2 takes leadership
>  7) After 66 seconds, application1 says :
>              ConnectionStateManager@237] - State change: LOST
>          
>          NOTE: Just at this point, I consider that the application1 has lost 
> leadership
> Then, for 26 seconds, we had 2 leaders.
> If you confirm it as a bug, I think I could help.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (CURATOR-460) Timed tolerance for connection suspended leads to simultaneous leaders

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/CURATOR-460?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16427894#comment-16427894
 ] 

ASF GitHub Bot commented on CURATOR-460:


Github user arrodrigues commented on a diff in the pull request:

https://github.com/apache/curator/pull/262#discussion_r179654000
  
--- Diff: 
curator-framework/src/main/java/org/apache/curator/framework/state/ConnectionStateManager.java
 ---
@@ -340,4 +338,12 @@ private void setCurrentConnectionState(ConnectionState 
newConnectionState)
 currentConnectionState = newConnectionState;
 startOfSuspendedEpoch = (currentConnectionState == 
ConnectionState.SUSPENDED) ? System.currentTimeMillis() : 0;
 }
+
+private synchronized int getUseSessionTimeoutMs() {
--- End diff --

I've made it volatile now.


> Timed tolerance for connection suspended leads to simultaneous leaders
> --
>
> Key: CURATOR-460
> URL: https://issues.apache.org/jira/browse/CURATOR-460
> Project: Apache Curator
>  Issue Type: Bug
>  Components: Recipes
>Affects Versions: 3.3.0, 4.0.0, 4.0.1
>Reporter: Antonio Rafael Rodrigues
>Assignee: Cameron McKenzie
>Priority: Major
> Attachments: ConnectionStateManager.patch, 
> Console-leaderSelector1.log, Console-leaderSelector2.log, 
> CuratorLeaderSelectorPOC.java, LogApp1.log, LogApp2.log
>
>
> Starting from Curator 3, after losing the connection to zookeeper, curator 
> emits a SUSPENDED event and then starts an internal timer, if the time of the 
> negotiatied session timeout get hit and curator didn't reconnect to zookeeper 
> it emits a LOST event.
> For example :
>  Given negotiated session timeout = 40
> ||Time (seconds)||Event||
> |0|SUSPENDED (curator has been disconnected from zookeeper)|
> |40|LOST (curator is still disconnected, it must have been lost the sesion as 
> it is disconnected for 40sec)|
> Given this scenario we could ,theoretically, ignore the SUSPENDED event and 
> consider the leadership as lost just if the ConnectionStateListener receives 
> the LOST event.
> But this feature seems to have introduced a bug (from my point of view)
> *Case of failure*
> ||Time (seconds)||Event||
> |0|Something went wrong with the connected zookeeper (in my case, the network 
> inteface of zookeeper's server has gone down). Curator stops hearing heart 
> beats from zookeeper, but doesn't lose the connection (from some reason that 
> I don't know, if the network interface of the server goes down, Curator 
> doesn't lose the connection)|
> |~26.666|SUSPENDED (after 26 seconds without hearing heartbeats, curator 
> emits a SUSPENDED event) 26 is from "readTimeout = sessionTimeout * 2 / 3" 
> from the class ClientCnxn.java from zookeeper client. At this point, curator 
> starts counting 40 sec.|
> |26.666 to 40|During this period, Curator is trying to connect to other 
> zookeeper instances but the other instances, in my example case, are also 
> uncheachable.|
> |40|Session has expired and the other instance has taken leadership ( in my 
> example, the other instance can connect to zookeeper )|
> |66.666|LOST ( after 40sec from SUSPENDED, curator finally sends LOST)|
> As you can see, if we are ignoring the SUSPENDED event, the second 
> application instance acquires the leadership ~26 seconds before the first 
> instance notice that it lost the leadership.
>  I understand it seems to be a very rare case, but either way I think it 
> should be addressed.
> *Reproduce it*
> I have came up with a way to reproduce this easily. I know this is not a 
> recommended production setup, but it reproduces the problem anyway.
> 1) On a virtual machine, with 2 network interfaces (eth0[192.168.0.101], 
> eth1[192.168.0.102]) , I installed one zookeeper instance.
>  2) I setup application1 with the leadership receipe, with 40sec of 
> negotiated timeout, pointing just to 192.168.0.101  .Now it is the leader
>  3) I setup application2 with the leadership receipe, with 40sec of 
> negotiated timeout, pointing just to 192.168.0.102  .Now it it is NOT the 
> leader
>  4) On the server I turn the eth0[192.168.0.101] interface down  [ ifconfig 
> eth0 down ]
>  5) After 26 seconds, application1 says : 
>            :ClientCnxn$SendThread@] - Client session timed out, have not 
> heard from server in 26679ms for sessionid
>             ConnectionStateManager@237] - State change: SUSPENDED
>         NOTE: I'm ignoring the SUSPENDED event
> 6) After 40 seconds, application2 takes leadership
>  7) After 66 seconds, application1 says :
>              ConnectionStateManager@237] - State change: LOST
>          
>          NOTE: Just at this point, I consider that the application1 has lost 
> leadership
> Then, for 26 seconds, we had 2 

[jira] [Commented] (CURATOR-460) Timed tolerance for connection suspended leads to simultaneous leaders

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/CURATOR-460?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16427888#comment-16427888
 ] 

ASF GitHub Bot commented on CURATOR-460:


Github user arrodrigues commented on a diff in the pull request:

https://github.com/apache/curator/pull/262#discussion_r179653281
  
--- Diff: 
curator-framework/src/main/java/org/apache/curator/framework/state/ConnectionStateManager.java
 ---
@@ -340,4 +338,12 @@ private void setCurrentConnectionState(ConnectionState 
newConnectionState)
 currentConnectionState = newConnectionState;
 startOfSuspendedEpoch = (currentConnectionState == 
ConnectionState.SUSPENDED) ? System.currentTimeMillis() : 0;
 }
+
+private synchronized int getUseSessionTimeoutMs() {
--- End diff --

Perfect.


> Timed tolerance for connection suspended leads to simultaneous leaders
> --
>
> Key: CURATOR-460
> URL: https://issues.apache.org/jira/browse/CURATOR-460
> Project: Apache Curator
>  Issue Type: Bug
>  Components: Recipes
>Affects Versions: 3.3.0, 4.0.0, 4.0.1
>Reporter: Antonio Rafael Rodrigues
>Assignee: Cameron McKenzie
>Priority: Major
> Attachments: ConnectionStateManager.patch, 
> Console-leaderSelector1.log, Console-leaderSelector2.log, 
> CuratorLeaderSelectorPOC.java, LogApp1.log, LogApp2.log
>
>
> Starting from Curator 3, after losing the connection to zookeeper, curator 
> emits a SUSPENDED event and then starts an internal timer, if the time of the 
> negotiatied session timeout get hit and curator didn't reconnect to zookeeper 
> it emits a LOST event.
> For example :
>  Given negotiated session timeout = 40
> ||Time (seconds)||Event||
> |0|SUSPENDED (curator has been disconnected from zookeeper)|
> |40|LOST (curator is still disconnected, it must have been lost the sesion as 
> it is disconnected for 40sec)|
> Given this scenario we could ,theoretically, ignore the SUSPENDED event and 
> consider the leadership as lost just if the ConnectionStateListener receives 
> the LOST event.
> But this feature seems to have introduced a bug (from my point of view)
> *Case of failure*
> ||Time (seconds)||Event||
> |0|Something went wrong with the connected zookeeper (in my case, the network 
> inteface of zookeeper's server has gone down). Curator stops hearing heart 
> beats from zookeeper, but doesn't lose the connection (from some reason that 
> I don't know, if the network interface of the server goes down, Curator 
> doesn't lose the connection)|
> |~26.666|SUSPENDED (after 26 seconds without hearing heartbeats, curator 
> emits a SUSPENDED event) 26 is from "readTimeout = sessionTimeout * 2 / 3" 
> from the class ClientCnxn.java from zookeeper client. At this point, curator 
> starts counting 40 sec.|
> |26.666 to 40|During this period, Curator is trying to connect to other 
> zookeeper instances but the other instances, in my example case, are also 
> uncheachable.|
> |40|Session has expired and the other instance has taken leadership ( in my 
> example, the other instance can connect to zookeeper )|
> |66.666|LOST ( after 40sec from SUSPENDED, curator finally sends LOST)|
> As you can see, if we are ignoring the SUSPENDED event, the second 
> application instance acquires the leadership ~26 seconds before the first 
> instance notice that it lost the leadership.
>  I understand it seems to be a very rare case, but either way I think it 
> should be addressed.
> *Reproduce it*
> I have came up with a way to reproduce this easily. I know this is not a 
> recommended production setup, but it reproduces the problem anyway.
> 1) On a virtual machine, with 2 network interfaces (eth0[192.168.0.101], 
> eth1[192.168.0.102]) , I installed one zookeeper instance.
>  2) I setup application1 with the leadership receipe, with 40sec of 
> negotiated timeout, pointing just to 192.168.0.101  .Now it is the leader
>  3) I setup application2 with the leadership receipe, with 40sec of 
> negotiated timeout, pointing just to 192.168.0.102  .Now it it is NOT the 
> leader
>  4) On the server I turn the eth0[192.168.0.101] interface down  [ ifconfig 
> eth0 down ]
>  5) After 26 seconds, application1 says : 
>            :ClientCnxn$SendThread@] - Client session timed out, have not 
> heard from server in 26679ms for sessionid
>             ConnectionStateManager@237] - State change: SUSPENDED
>         NOTE: I'm ignoring the SUSPENDED event
> 6) After 40 seconds, application2 takes leadership
>  7) After 66 seconds, application1 says :
>              ConnectionStateManager@237] - State change: LOST
>          
>          NOTE: Just at this point, I consider that the application1 has lost 
> leadership
> Then, for 26 seconds, we had 2 leaders.
> If you 

[jira] [Commented] (CURATOR-460) Timed tolerance for connection suspended leads to simultaneous leaders

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/CURATOR-460?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16427881#comment-16427881
 ] 

ASF GitHub Bot commented on CURATOR-460:


Github user arrodrigues commented on a diff in the pull request:

https://github.com/apache/curator/pull/262#discussion_r179652355
  
--- Diff: 
curator-framework/src/main/java/org/apache/curator/framework/state/ConnectionStateManager.java
 ---
@@ -340,4 +338,12 @@ private void setCurrentConnectionState(ConnectionState 
newConnectionState)
 currentConnectionState = newConnectionState;
 startOfSuspendedEpoch = (currentConnectionState == 
ConnectionState.SUSPENDED) ? System.currentTimeMillis() : 0;
 }
+
+private synchronized int getUseSessionTimeoutMs() {
--- End diff --

Yes. I had this discussion with @cammckenzie some comments ago. Every 
access to startOfSuspendedEpoch is synchronized on "this" except this point, we 
could make it volatile or sync this point more, so we choose to sync it. In the 
first version I synchronized just the line accessing startOfSuspendedEpoch, but 
after adding getUseSessionTimeoutMs I tought it'd be more readable  sync the 
whole method.
Do you prefer to make it volatile?


> Timed tolerance for connection suspended leads to simultaneous leaders
> --
>
> Key: CURATOR-460
> URL: https://issues.apache.org/jira/browse/CURATOR-460
> Project: Apache Curator
>  Issue Type: Bug
>  Components: Recipes
>Affects Versions: 3.3.0, 4.0.0, 4.0.1
>Reporter: Antonio Rafael Rodrigues
>Assignee: Cameron McKenzie
>Priority: Major
> Attachments: ConnectionStateManager.patch, 
> Console-leaderSelector1.log, Console-leaderSelector2.log, 
> CuratorLeaderSelectorPOC.java, LogApp1.log, LogApp2.log
>
>
> Starting from Curator 3, after losing the connection to zookeeper, curator 
> emits a SUSPENDED event and then starts an internal timer, if the time of the 
> negotiatied session timeout get hit and curator didn't reconnect to zookeeper 
> it emits a LOST event.
> For example :
>  Given negotiated session timeout = 40
> ||Time (seconds)||Event||
> |0|SUSPENDED (curator has been disconnected from zookeeper)|
> |40|LOST (curator is still disconnected, it must have been lost the sesion as 
> it is disconnected for 40sec)|
> Given this scenario we could ,theoretically, ignore the SUSPENDED event and 
> consider the leadership as lost just if the ConnectionStateListener receives 
> the LOST event.
> But this feature seems to have introduced a bug (from my point of view)
> *Case of failure*
> ||Time (seconds)||Event||
> |0|Something went wrong with the connected zookeeper (in my case, the network 
> inteface of zookeeper's server has gone down). Curator stops hearing heart 
> beats from zookeeper, but doesn't lose the connection (from some reason that 
> I don't know, if the network interface of the server goes down, Curator 
> doesn't lose the connection)|
> |~26.666|SUSPENDED (after 26 seconds without hearing heartbeats, curator 
> emits a SUSPENDED event) 26 is from "readTimeout = sessionTimeout * 2 / 3" 
> from the class ClientCnxn.java from zookeeper client. At this point, curator 
> starts counting 40 sec.|
> |26.666 to 40|During this period, Curator is trying to connect to other 
> zookeeper instances but the other instances, in my example case, are also 
> uncheachable.|
> |40|Session has expired and the other instance has taken leadership ( in my 
> example, the other instance can connect to zookeeper )|
> |66.666|LOST ( after 40sec from SUSPENDED, curator finally sends LOST)|
> As you can see, if we are ignoring the SUSPENDED event, the second 
> application instance acquires the leadership ~26 seconds before the first 
> instance notice that it lost the leadership.
>  I understand it seems to be a very rare case, but either way I think it 
> should be addressed.
> *Reproduce it*
> I have came up with a way to reproduce this easily. I know this is not a 
> recommended production setup, but it reproduces the problem anyway.
> 1) On a virtual machine, with 2 network interfaces (eth0[192.168.0.101], 
> eth1[192.168.0.102]) , I installed one zookeeper instance.
>  2) I setup application1 with the leadership receipe, with 40sec of 
> negotiated timeout, pointing just to 192.168.0.101  .Now it is the leader
>  3) I setup application2 with the leadership receipe, with 40sec of 
> negotiated timeout, pointing just to 192.168.0.102  .Now it it is NOT the 
> leader
>  4) On the server I turn the eth0[192.168.0.101] interface down  [ ifconfig 
> eth0 down ]
>  5) After 26 seconds, application1 says : 
>            :ClientCnxn$SendThread@] - Client session timed out, have not 
> heard from server in 26679ms for sessionid
>  

[jira] [Commented] (CURATOR-460) Timed tolerance for connection suspended leads to simultaneous leaders

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/CURATOR-460?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16427875#comment-16427875
 ] 

ASF GitHub Bot commented on CURATOR-460:


Github user Randgalt commented on a diff in the pull request:

https://github.com/apache/curator/pull/262#discussion_r179651175
  
--- Diff: 
curator-framework/src/main/java/org/apache/curator/framework/state/ConnectionStateManager.java
 ---
@@ -340,4 +338,12 @@ private void setCurrentConnectionState(ConnectionState 
newConnectionState)
 currentConnectionState = newConnectionState;
 startOfSuspendedEpoch = (currentConnectionState == 
ConnectionState.SUSPENDED) ? System.currentTimeMillis() : 0;
 }
+
+private synchronized int getUseSessionTimeoutMs() {
--- End diff --

Looks like the comment on `startOfSuspendedEpoch` "guarded by sync" is a 
lie. `getUseSessionTimeoutMs()` doesn't need to be synchronized if we make 
`startOfSuspendedEpoch` volatile.


> Timed tolerance for connection suspended leads to simultaneous leaders
> --
>
> Key: CURATOR-460
> URL: https://issues.apache.org/jira/browse/CURATOR-460
> Project: Apache Curator
>  Issue Type: Bug
>  Components: Recipes
>Affects Versions: 3.3.0, 4.0.0, 4.0.1
>Reporter: Antonio Rafael Rodrigues
>Assignee: Cameron McKenzie
>Priority: Major
> Attachments: ConnectionStateManager.patch, 
> Console-leaderSelector1.log, Console-leaderSelector2.log, 
> CuratorLeaderSelectorPOC.java, LogApp1.log, LogApp2.log
>
>
> Starting from Curator 3, after losing the connection to zookeeper, curator 
> emits a SUSPENDED event and then starts an internal timer, if the time of the 
> negotiatied session timeout get hit and curator didn't reconnect to zookeeper 
> it emits a LOST event.
> For example :
>  Given negotiated session timeout = 40
> ||Time (seconds)||Event||
> |0|SUSPENDED (curator has been disconnected from zookeeper)|
> |40|LOST (curator is still disconnected, it must have been lost the sesion as 
> it is disconnected for 40sec)|
> Given this scenario we could ,theoretically, ignore the SUSPENDED event and 
> consider the leadership as lost just if the ConnectionStateListener receives 
> the LOST event.
> But this feature seems to have introduced a bug (from my point of view)
> *Case of failure*
> ||Time (seconds)||Event||
> |0|Something went wrong with the connected zookeeper (in my case, the network 
> inteface of zookeeper's server has gone down). Curator stops hearing heart 
> beats from zookeeper, but doesn't lose the connection (from some reason that 
> I don't know, if the network interface of the server goes down, Curator 
> doesn't lose the connection)|
> |~26.666|SUSPENDED (after 26 seconds without hearing heartbeats, curator 
> emits a SUSPENDED event) 26 is from "readTimeout = sessionTimeout * 2 / 3" 
> from the class ClientCnxn.java from zookeeper client. At this point, curator 
> starts counting 40 sec.|
> |26.666 to 40|During this period, Curator is trying to connect to other 
> zookeeper instances but the other instances, in my example case, are also 
> uncheachable.|
> |40|Session has expired and the other instance has taken leadership ( in my 
> example, the other instance can connect to zookeeper )|
> |66.666|LOST ( after 40sec from SUSPENDED, curator finally sends LOST)|
> As you can see, if we are ignoring the SUSPENDED event, the second 
> application instance acquires the leadership ~26 seconds before the first 
> instance notice that it lost the leadership.
>  I understand it seems to be a very rare case, but either way I think it 
> should be addressed.
> *Reproduce it*
> I have came up with a way to reproduce this easily. I know this is not a 
> recommended production setup, but it reproduces the problem anyway.
> 1) On a virtual machine, with 2 network interfaces (eth0[192.168.0.101], 
> eth1[192.168.0.102]) , I installed one zookeeper instance.
>  2) I setup application1 with the leadership receipe, with 40sec of 
> negotiated timeout, pointing just to 192.168.0.101  .Now it is the leader
>  3) I setup application2 with the leadership receipe, with 40sec of 
> negotiated timeout, pointing just to 192.168.0.102  .Now it it is NOT the 
> leader
>  4) On the server I turn the eth0[192.168.0.101] interface down  [ ifconfig 
> eth0 down ]
>  5) After 26 seconds, application1 says : 
>            :ClientCnxn$SendThread@] - Client session timed out, have not 
> heard from server in 26679ms for sessionid
>             ConnectionStateManager@237] - State change: SUSPENDED
>         NOTE: I'm ignoring the SUSPENDED event
> 6) After 40 seconds, application2 takes leadership
>  7) After 66 seconds, application1 says :
>              ConnectionStateManager@237] - State 

[jira] [Commented] (CURATOR-460) Timed tolerance for connection suspended leads to simultaneous leaders

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/CURATOR-460?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16427872#comment-16427872
 ] 

ASF GitHub Bot commented on CURATOR-460:


Github user arrodrigues commented on a diff in the pull request:

https://github.com/apache/curator/pull/262#discussion_r179650717
  
--- Diff: 
curator-framework/src/main/java/org/apache/curator/framework/state/ConnectionStateManager.java
 ---
@@ -251,9 +251,15 @@ private void processEvents()
 {
 try
 {
+
+long localStartOfSuspendedEpoch;
+synchronized (this) {
+localStartOfSuspendedEpoch = 
this.startOfSuspendedEpoch;
+}
 int lastNegotiatedSessionTimeoutMs = 
client.getZookeeperClient().getLastNegotiatedSessionTimeoutMs();
 int useSessionTimeoutMs = (lastNegotiatedSessionTimeoutMs 
> 0) ? lastNegotiatedSessionTimeoutMs : sessionTimeoutMs;
-long elapsedMs = startOfSuspendedEpoch == 0 ? 
useSessionTimeoutMs / 2 : System.currentTimeMillis() - startOfSuspendedEpoch;
+useSessionTimeoutMs = sessionExpirationPercent > 0 && 
localStartOfSuspendedEpoch != 0 ? (useSessionTimeoutMs * 
sessionExpirationPercent) / 100 : useSessionTimeoutMs;
--- End diff --

I think you mean useSessionTimeoutMs calc. I created a method for that. I'm 
not very happy with the name I gave to it "getUseSessionTimeoutMs", suggestions 
are certainly welcome.


> Timed tolerance for connection suspended leads to simultaneous leaders
> --
>
> Key: CURATOR-460
> URL: https://issues.apache.org/jira/browse/CURATOR-460
> Project: Apache Curator
>  Issue Type: Bug
>  Components: Recipes
>Affects Versions: 3.3.0, 4.0.0, 4.0.1
>Reporter: Antonio Rafael Rodrigues
>Assignee: Cameron McKenzie
>Priority: Major
> Attachments: ConnectionStateManager.patch, 
> Console-leaderSelector1.log, Console-leaderSelector2.log, 
> CuratorLeaderSelectorPOC.java, LogApp1.log, LogApp2.log
>
>
> Starting from Curator 3, after losing the connection to zookeeper, curator 
> emits a SUSPENDED event and then starts an internal timer, if the time of the 
> negotiatied session timeout get hit and curator didn't reconnect to zookeeper 
> it emits a LOST event.
> For example :
>  Given negotiated session timeout = 40
> ||Time (seconds)||Event||
> |0|SUSPENDED (curator has been disconnected from zookeeper)|
> |40|LOST (curator is still disconnected, it must have been lost the sesion as 
> it is disconnected for 40sec)|
> Given this scenario we could ,theoretically, ignore the SUSPENDED event and 
> consider the leadership as lost just if the ConnectionStateListener receives 
> the LOST event.
> But this feature seems to have introduced a bug (from my point of view)
> *Case of failure*
> ||Time (seconds)||Event||
> |0|Something went wrong with the connected zookeeper (in my case, the network 
> inteface of zookeeper's server has gone down). Curator stops hearing heart 
> beats from zookeeper, but doesn't lose the connection (from some reason that 
> I don't know, if the network interface of the server goes down, Curator 
> doesn't lose the connection)|
> |~26.666|SUSPENDED (after 26 seconds without hearing heartbeats, curator 
> emits a SUSPENDED event) 26 is from "readTimeout = sessionTimeout * 2 / 3" 
> from the class ClientCnxn.java from zookeeper client. At this point, curator 
> starts counting 40 sec.|
> |26.666 to 40|During this period, Curator is trying to connect to other 
> zookeeper instances but the other instances, in my example case, are also 
> uncheachable.|
> |40|Session has expired and the other instance has taken leadership ( in my 
> example, the other instance can connect to zookeeper )|
> |66.666|LOST ( after 40sec from SUSPENDED, curator finally sends LOST)|
> As you can see, if we are ignoring the SUSPENDED event, the second 
> application instance acquires the leadership ~26 seconds before the first 
> instance notice that it lost the leadership.
>  I understand it seems to be a very rare case, but either way I think it 
> should be addressed.
> *Reproduce it*
> I have came up with a way to reproduce this easily. I know this is not a 
> recommended production setup, but it reproduces the problem anyway.
> 1) On a virtual machine, with 2 network interfaces (eth0[192.168.0.101], 
> eth1[192.168.0.102]) , I installed one zookeeper instance.
>  2) I setup application1 with the leadership receipe, with 40sec of 
> negotiated timeout, pointing just to 192.168.0.101  .Now it is the leader
>  3) I setup application2 with the leadership receipe, with 40sec of 
> negotiated timeout, pointing just to 192.168.0.102  .Now it 

[jira] [Commented] (CURATOR-460) Timed tolerance for connection suspended leads to simultaneous leaders

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/CURATOR-460?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16426654#comment-16426654
 ] 

ASF GitHub Bot commented on CURATOR-460:


Github user Randgalt commented on the issue:

https://github.com/apache/curator/pull/262
  
I'm thinking we change `StandardConnectionHandlingPolicy`. It should be 33 
though right? By the time we get the Disconnected 2/3 of a session have 
elapsed. 

However, this could affect existing clients badly. So, probably the best 
thing is to make the value configurable via command line with the default being 
33. So, something like:

```
public class StandardConnectionHandlingPolicy implements 
ConnectionHandlingPolicy 
{
private static final int default_expiration_percent = 
Integer.getInteger("curator-connection-expiration-percent", 33);

private final int expirationPercent;

public StandardConnectionHandlingPolicy()
{
this(default_expiration_percent);
}
}
```


> Timed tolerance for connection suspended leads to simultaneous leaders
> --
>
> Key: CURATOR-460
> URL: https://issues.apache.org/jira/browse/CURATOR-460
> Project: Apache Curator
>  Issue Type: Bug
>  Components: Recipes
>Affects Versions: 3.3.0, 4.0.0, 4.0.1
>Reporter: Antonio Rafael Rodrigues
>Assignee: Cameron McKenzie
>Priority: Major
> Attachments: ConnectionStateManager.patch, 
> Console-leaderSelector1.log, Console-leaderSelector2.log, 
> CuratorLeaderSelectorPOC.java, LogApp1.log, LogApp2.log
>
>
> Starting from Curator 3, after losing the connection to zookeeper, curator 
> emits a SUSPENDED event and then starts an internal timer, if the time of the 
> negotiatied session timeout get hit and curator didn't reconnect to zookeeper 
> it emits a LOST event.
> For example :
>  Given negotiated session timeout = 40
> ||Time (seconds)||Event||
> |0|SUSPENDED (curator has been disconnected from zookeeper)|
> |40|LOST (curator is still disconnected, it must have been lost the sesion as 
> it is disconnected for 40sec)|
> Given this scenario we could ,theoretically, ignore the SUSPENDED event and 
> consider the leadership as lost just if the ConnectionStateListener receives 
> the LOST event.
> But this feature seems to have introduced a bug (from my point of view)
> *Case of failure*
> ||Time (seconds)||Event||
> |0|Something went wrong with the connected zookeeper (in my case, the network 
> inteface of zookeeper's server has gone down). Curator stops hearing heart 
> beats from zookeeper, but doesn't lose the connection (from some reason that 
> I don't know, if the network interface of the server goes down, Curator 
> doesn't lose the connection)|
> |~26.666|SUSPENDED (after 26 seconds without hearing heartbeats, curator 
> emits a SUSPENDED event) 26 is from "readTimeout = sessionTimeout * 2 / 3" 
> from the class ClientCnxn.java from zookeeper client. At this point, curator 
> starts counting 40 sec.|
> |26.666 to 40|During this period, Curator is trying to connect to other 
> zookeeper instances but the other instances, in my example case, are also 
> uncheachable.|
> |40|Session has expired and the other instance has taken leadership ( in my 
> example, the other instance can connect to zookeeper )|
> |66.666|LOST ( after 40sec from SUSPENDED, curator finally sends LOST)|
> As you can see, if we are ignoring the SUSPENDED event, the second 
> application instance acquires the leadership ~26 seconds before the first 
> instance notice that it lost the leadership.
>  I understand it seems to be a very rare case, but either way I think it 
> should be addressed.
> *Reproduce it*
> I have came up with a way to reproduce this easily. I know this is not a 
> recommended production setup, but it reproduces the problem anyway.
> 1) On a virtual machine, with 2 network interfaces (eth0[192.168.0.101], 
> eth1[192.168.0.102]) , I installed one zookeeper instance.
>  2) I setup application1 with the leadership receipe, with 40sec of 
> negotiated timeout, pointing just to 192.168.0.101  .Now it is the leader
>  3) I setup application2 with the leadership receipe, with 40sec of 
> negotiated timeout, pointing just to 192.168.0.102  .Now it it is NOT the 
> leader
>  4) On the server I turn the eth0[192.168.0.101] interface down  [ ifconfig 
> eth0 down ]
>  5) After 26 seconds, application1 says : 
>            :ClientCnxn$SendThread@] - Client session timed out, have not 
> heard from server in 26679ms for sessionid
>             ConnectionStateManager@237] - State change: SUSPENDED
>         NOTE: I'm ignoring the SUSPENDED event
> 6) After 40 seconds, application2 takes leadership
>  7) After 66 seconds, application1 says :
>           

[jira] [Commented] (CURATOR-460) Timed tolerance for connection suspended leads to simultaneous leaders

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/CURATOR-460?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16426645#comment-16426645
 ] 

ASF GitHub Bot commented on CURATOR-460:


Github user Randgalt commented on a diff in the pull request:

https://github.com/apache/curator/pull/262#discussion_r179396466
  
--- Diff: 
curator-framework/src/main/java/org/apache/curator/framework/state/ConnectionStateManager.java
 ---
@@ -251,9 +251,15 @@ private void processEvents()
 {
 try
 {
+
+long localStartOfSuspendedEpoch;
+synchronized (this) {
+localStartOfSuspendedEpoch = 
this.startOfSuspendedEpoch;
+}
 int lastNegotiatedSessionTimeoutMs = 
client.getZookeeperClient().getLastNegotiatedSessionTimeoutMs();
 int useSessionTimeoutMs = (lastNegotiatedSessionTimeoutMs 
> 0) ? lastNegotiatedSessionTimeoutMs : sessionTimeoutMs;
-long elapsedMs = startOfSuspendedEpoch == 0 ? 
useSessionTimeoutMs / 2 : System.currentTimeMillis() - startOfSuspendedEpoch;
+useSessionTimeoutMs = sessionExpirationPercent > 0 && 
localStartOfSuspendedEpoch != 0 ? (useSessionTimeoutMs * 
sessionExpirationPercent) / 100 : useSessionTimeoutMs;
--- End diff --

The `sessionExpirationPercent` calc is the same as in the other spot in the 
code right? We can abstract a method that does this.


> Timed tolerance for connection suspended leads to simultaneous leaders
> --
>
> Key: CURATOR-460
> URL: https://issues.apache.org/jira/browse/CURATOR-460
> Project: Apache Curator
>  Issue Type: Bug
>  Components: Recipes
>Affects Versions: 3.3.0, 4.0.0, 4.0.1
>Reporter: Antonio Rafael Rodrigues
>Assignee: Cameron McKenzie
>Priority: Major
> Attachments: ConnectionStateManager.patch, 
> Console-leaderSelector1.log, Console-leaderSelector2.log, 
> CuratorLeaderSelectorPOC.java, LogApp1.log, LogApp2.log
>
>
> Starting from Curator 3, after losing the connection to zookeeper, curator 
> emits a SUSPENDED event and then starts an internal timer, if the time of the 
> negotiatied session timeout get hit and curator didn't reconnect to zookeeper 
> it emits a LOST event.
> For example :
>  Given negotiated session timeout = 40
> ||Time (seconds)||Event||
> |0|SUSPENDED (curator has been disconnected from zookeeper)|
> |40|LOST (curator is still disconnected, it must have been lost the sesion as 
> it is disconnected for 40sec)|
> Given this scenario we could ,theoretically, ignore the SUSPENDED event and 
> consider the leadership as lost just if the ConnectionStateListener receives 
> the LOST event.
> But this feature seems to have introduced a bug (from my point of view)
> *Case of failure*
> ||Time (seconds)||Event||
> |0|Something went wrong with the connected zookeeper (in my case, the network 
> inteface of zookeeper's server has gone down). Curator stops hearing heart 
> beats from zookeeper, but doesn't lose the connection (from some reason that 
> I don't know, if the network interface of the server goes down, Curator 
> doesn't lose the connection)|
> |~26.666|SUSPENDED (after 26 seconds without hearing heartbeats, curator 
> emits a SUSPENDED event) 26 is from "readTimeout = sessionTimeout * 2 / 3" 
> from the class ClientCnxn.java from zookeeper client. At this point, curator 
> starts counting 40 sec.|
> |26.666 to 40|During this period, Curator is trying to connect to other 
> zookeeper instances but the other instances, in my example case, are also 
> uncheachable.|
> |40|Session has expired and the other instance has taken leadership ( in my 
> example, the other instance can connect to zookeeper )|
> |66.666|LOST ( after 40sec from SUSPENDED, curator finally sends LOST)|
> As you can see, if we are ignoring the SUSPENDED event, the second 
> application instance acquires the leadership ~26 seconds before the first 
> instance notice that it lost the leadership.
>  I understand it seems to be a very rare case, but either way I think it 
> should be addressed.
> *Reproduce it*
> I have came up with a way to reproduce this easily. I know this is not a 
> recommended production setup, but it reproduces the problem anyway.
> 1) On a virtual machine, with 2 network interfaces (eth0[192.168.0.101], 
> eth1[192.168.0.102]) , I installed one zookeeper instance.
>  2) I setup application1 with the leadership receipe, with 40sec of 
> negotiated timeout, pointing just to 192.168.0.101  .Now it is the leader
>  3) I setup application2 with the leadership receipe, with 40sec of 
> negotiated timeout, pointing just to 192.168.0.102  .Now it it is NOT the 
> leader
>  4) On the server I turn the 

[jira] [Commented] (CURATOR-460) Timed tolerance for connection suspended leads to simultaneous leaders

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/CURATOR-460?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16426441#comment-16426441
 ] 

ASF GitHub Bot commented on CURATOR-460:


Github user cammckenzie commented on the issue:

https://github.com/apache/curator/pull/262
  
Thanks for the updates, I think this looks good.

I still think we need to look at the default behaviour of the 
ConnectionStateListener though. Currently, a LOST event will only be injected 
after 4/3 of the session timeout (+ some small error). I think that this 
default behaviour should be changed so that the LOST event is injected at 
session timeout (+ some small error). This could be achieved by either changing 
the StandardConnectionHandlingPolicy default constructor to use 66 instead of 
100 for the %, or by changing the ConnectionStateListener logic. I think it's 
probably simpler to change via the StandardConnectionHandlingPolicy, because 
the ConnectionStateListener logic interacts with the % defined in the 
StandardConnectionHandlingPolicy and has more potential to break existing use 
cases.

What are your thoughts @Randgalt ?


> Timed tolerance for connection suspended leads to simultaneous leaders
> --
>
> Key: CURATOR-460
> URL: https://issues.apache.org/jira/browse/CURATOR-460
> Project: Apache Curator
>  Issue Type: Bug
>  Components: Recipes
>Affects Versions: 3.3.0, 4.0.0, 4.0.1
>Reporter: Antonio Rafael Rodrigues
>Assignee: Cameron McKenzie
>Priority: Major
> Attachments: ConnectionStateManager.patch, 
> Console-leaderSelector1.log, Console-leaderSelector2.log, 
> CuratorLeaderSelectorPOC.java, LogApp1.log, LogApp2.log
>
>
> Starting from Curator 3, after losing the connection to zookeeper, curator 
> emits a SUSPENDED event and then starts an internal timer, if the time of the 
> negotiatied session timeout get hit and curator didn't reconnect to zookeeper 
> it emits a LOST event.
> For example :
>  Given negotiated session timeout = 40
> ||Time (seconds)||Event||
> |0|SUSPENDED (curator has been disconnected from zookeeper)|
> |40|LOST (curator is still disconnected, it must have been lost the sesion as 
> it is disconnected for 40sec)|
> Given this scenario we could ,theoretically, ignore the SUSPENDED event and 
> consider the leadership as lost just if the ConnectionStateListener receives 
> the LOST event.
> But this feature seems to have introduced a bug (from my point of view)
> *Case of failure*
> ||Time (seconds)||Event||
> |0|Something went wrong with the connected zookeeper (in my case, the network 
> inteface of zookeeper's server has gone down). Curator stops hearing heart 
> beats from zookeeper, but doesn't lose the connection (from some reason that 
> I don't know, if the network interface of the server goes down, Curator 
> doesn't lose the connection)|
> |~26.666|SUSPENDED (after 26 seconds without hearing heartbeats, curator 
> emits a SUSPENDED event) 26 is from "readTimeout = sessionTimeout * 2 / 3" 
> from the class ClientCnxn.java from zookeeper client. At this point, curator 
> starts counting 40 sec.|
> |26.666 to 40|During this period, Curator is trying to connect to other 
> zookeeper instances but the other instances, in my example case, are also 
> uncheachable.|
> |40|Session has expired and the other instance has taken leadership ( in my 
> example, the other instance can connect to zookeeper )|
> |66.666|LOST ( after 40sec from SUSPENDED, curator finally sends LOST)|
> As you can see, if we are ignoring the SUSPENDED event, the second 
> application instance acquires the leadership ~26 seconds before the first 
> instance notice that it lost the leadership.
>  I understand it seems to be a very rare case, but either way I think it 
> should be addressed.
> *Reproduce it*
> I have came up with a way to reproduce this easily. I know this is not a 
> recommended production setup, but it reproduces the problem anyway.
> 1) On a virtual machine, with 2 network interfaces (eth0[192.168.0.101], 
> eth1[192.168.0.102]) , I installed one zookeeper instance.
>  2) I setup application1 with the leadership receipe, with 40sec of 
> negotiated timeout, pointing just to 192.168.0.101  .Now it is the leader
>  3) I setup application2 with the leadership receipe, with 40sec of 
> negotiated timeout, pointing just to 192.168.0.102  .Now it it is NOT the 
> leader
>  4) On the server I turn the eth0[192.168.0.101] interface down  [ ifconfig 
> eth0 down ]
>  5) After 26 seconds, application1 says : 
>            :ClientCnxn$SendThread@] - Client session timed out, have not 
> heard from server in 26679ms for sessionid
>             ConnectionStateManager@237] - State change: SUSPENDED
>         NOTE: I'm ignoring the SUSPENDED event
> 6) 

[jira] [Commented] (CURATOR-460) Timed tolerance for connection suspended leads to simultaneous leaders

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/CURATOR-460?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16426355#comment-16426355
 ] 

ASF GitHub Bot commented on CURATOR-460:


Github user arrodrigues commented on a diff in the pull request:

https://github.com/apache/curator/pull/262#discussion_r179319231
  
--- Diff: 
curator-recipes/src/test/java/org/apache/curator/framework/recipes/leader/TestLeaderLatch.java
 ---
@@ -222,6 +223,35 @@ public void notLeader()
 next.add(states.poll(timing.forSessionSleep().milliseconds(), 
TimeUnit.MILLISECONDS));
 next.add(states.poll(timing.forSessionSleep().milliseconds(), 
TimeUnit.MILLISECONDS));
 
Assert.assertTrue(next.equals(Arrays.asList(ConnectionState.LOST.name(), 
"false")) || next.equals(Arrays.asList("false", ConnectionState.LOST.name())), 
next.toString());
+
+latch.close();
+client.close();
+
+timing.sleepABit();
+states.clear();
+server = new TestingServer();
+client = CuratorFrameworkFactory.builder()
+.connectString(server.getConnectString())
+.connectionTimeoutMs(1000)
+.sessionTimeoutMs(timing.session())
+.retryPolicy(new RetryOneTime(1))
+.connectionStateErrorPolicy(new 
SessionConnectionStateErrorPolicy())
+.connectionHandlingPolicy(new 
StandardConnectionHandlingPolicy(30))
+.build();
+
client.getConnectionStateListenable().addListener(stateListener);
+client.start();
+latch = new LeaderLatch(client, "/test");
+latch.addListener(listener);
+latch.start();
+
Assert.assertEquals(states.poll(timing.forWaiting().milliseconds(), 
TimeUnit.MILLISECONDS), ConnectionState.CONNECTED.name());
+
Assert.assertEquals(states.poll(timing.forWaiting().milliseconds(), 
TimeUnit.MILLISECONDS), "true");
+server.close();
+
Assert.assertEquals(states.poll(timing.forWaiting().milliseconds(), 
TimeUnit.MILLISECONDS), ConnectionState.SUSPENDED.name());
+next = Lists.newArrayList();
+
+next.add(states.poll(timing.session() / 3, 
TimeUnit.MILLISECONDS));
+next.add(states.poll(timing.forSleepingABit().milliseconds(), 
TimeUnit.MILLISECONDS));
+
Assert.assertTrue(next.equals(Arrays.asList(ConnectionState.LOST.name(), 
"false")) || next.equals(Arrays.asList("false", ConnectionState.LOST.name())), 
next.toString());
--- End diff --

OK. I'm going to adjust that.


> Timed tolerance for connection suspended leads to simultaneous leaders
> --
>
> Key: CURATOR-460
> URL: https://issues.apache.org/jira/browse/CURATOR-460
> Project: Apache Curator
>  Issue Type: Bug
>  Components: Recipes
>Affects Versions: 3.3.0, 4.0.0, 4.0.1
>Reporter: Antonio Rafael Rodrigues
>Assignee: Cameron McKenzie
>Priority: Major
> Attachments: ConnectionStateManager.patch, 
> Console-leaderSelector1.log, Console-leaderSelector2.log, 
> CuratorLeaderSelectorPOC.java, LogApp1.log, LogApp2.log
>
>
> Starting from Curator 3, after losing the connection to zookeeper, curator 
> emits a SUSPENDED event and then starts an internal timer, if the time of the 
> negotiatied session timeout get hit and curator didn't reconnect to zookeeper 
> it emits a LOST event.
> For example :
>  Given negotiated session timeout = 40
> ||Time (seconds)||Event||
> |0|SUSPENDED (curator has been disconnected from zookeeper)|
> |40|LOST (curator is still disconnected, it must have been lost the sesion as 
> it is disconnected for 40sec)|
> Given this scenario we could ,theoretically, ignore the SUSPENDED event and 
> consider the leadership as lost just if the ConnectionStateListener receives 
> the LOST event.
> But this feature seems to have introduced a bug (from my point of view)
> *Case of failure*
> ||Time (seconds)||Event||
> |0|Something went wrong with the connected zookeeper (in my case, the network 
> inteface of zookeeper's server has gone down). Curator stops hearing heart 
> beats from zookeeper, but doesn't lose the connection (from some reason that 
> I don't know, if the network interface of the server goes down, Curator 
> doesn't lose the connection)|
> |~26.666|SUSPENDED (after 26 seconds without hearing heartbeats, curator 
> emits a SUSPENDED event) 26 is from "readTimeout = sessionTimeout * 2 / 3" 
> from the class ClientCnxn.java from zookeeper client. At this point, curator 
> starts counting 40 sec.|
> |26.666 to 40|During 

[jira] [Commented] (CURATOR-460) Timed tolerance for connection suspended leads to simultaneous leaders

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/CURATOR-460?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16426354#comment-16426354
 ] 

ASF GitHub Bot commented on CURATOR-460:


Github user arrodrigues commented on a diff in the pull request:

https://github.com/apache/curator/pull/262#discussion_r179319128
  
--- Diff: 
curator-framework/src/main/java/org/apache/curator/framework/state/ConnectionStateManager.java
 ---
@@ -253,6 +253,7 @@ private void processEvents()
 {
 int lastNegotiatedSessionTimeoutMs = 
client.getZookeeperClient().getLastNegotiatedSessionTimeoutMs();
 int useSessionTimeoutMs = (lastNegotiatedSessionTimeoutMs 
> 0) ? lastNegotiatedSessionTimeoutMs : sessionTimeoutMs;
+useSessionTimeoutMs = sessionExpirationPercent > 0 && 
startOfSuspendedEpoch != 0 ? (useSessionTimeoutMs * sessionExpirationPercent) / 
100 : useSessionTimeoutMs;
--- End diff --

Therefore, I'm going to synchronize the access at that point.


> Timed tolerance for connection suspended leads to simultaneous leaders
> --
>
> Key: CURATOR-460
> URL: https://issues.apache.org/jira/browse/CURATOR-460
> Project: Apache Curator
>  Issue Type: Bug
>  Components: Recipes
>Affects Versions: 3.3.0, 4.0.0, 4.0.1
>Reporter: Antonio Rafael Rodrigues
>Assignee: Cameron McKenzie
>Priority: Major
> Attachments: ConnectionStateManager.patch, 
> Console-leaderSelector1.log, Console-leaderSelector2.log, 
> CuratorLeaderSelectorPOC.java, LogApp1.log, LogApp2.log
>
>
> Starting from Curator 3, after losing the connection to zookeeper, curator 
> emits a SUSPENDED event and then starts an internal timer, if the time of the 
> negotiatied session timeout get hit and curator didn't reconnect to zookeeper 
> it emits a LOST event.
> For example :
>  Given negotiated session timeout = 40
> ||Time (seconds)||Event||
> |0|SUSPENDED (curator has been disconnected from zookeeper)|
> |40|LOST (curator is still disconnected, it must have been lost the sesion as 
> it is disconnected for 40sec)|
> Given this scenario we could ,theoretically, ignore the SUSPENDED event and 
> consider the leadership as lost just if the ConnectionStateListener receives 
> the LOST event.
> But this feature seems to have introduced a bug (from my point of view)
> *Case of failure*
> ||Time (seconds)||Event||
> |0|Something went wrong with the connected zookeeper (in my case, the network 
> inteface of zookeeper's server has gone down). Curator stops hearing heart 
> beats from zookeeper, but doesn't lose the connection (from some reason that 
> I don't know, if the network interface of the server goes down, Curator 
> doesn't lose the connection)|
> |~26.666|SUSPENDED (after 26 seconds without hearing heartbeats, curator 
> emits a SUSPENDED event) 26 is from "readTimeout = sessionTimeout * 2 / 3" 
> from the class ClientCnxn.java from zookeeper client. At this point, curator 
> starts counting 40 sec.|
> |26.666 to 40|During this period, Curator is trying to connect to other 
> zookeeper instances but the other instances, in my example case, are also 
> uncheachable.|
> |40|Session has expired and the other instance has taken leadership ( in my 
> example, the other instance can connect to zookeeper )|
> |66.666|LOST ( after 40sec from SUSPENDED, curator finally sends LOST)|
> As you can see, if we are ignoring the SUSPENDED event, the second 
> application instance acquires the leadership ~26 seconds before the first 
> instance notice that it lost the leadership.
>  I understand it seems to be a very rare case, but either way I think it 
> should be addressed.
> *Reproduce it*
> I have came up with a way to reproduce this easily. I know this is not a 
> recommended production setup, but it reproduces the problem anyway.
> 1) On a virtual machine, with 2 network interfaces (eth0[192.168.0.101], 
> eth1[192.168.0.102]) , I installed one zookeeper instance.
>  2) I setup application1 with the leadership receipe, with 40sec of 
> negotiated timeout, pointing just to 192.168.0.101  .Now it is the leader
>  3) I setup application2 with the leadership receipe, with 40sec of 
> negotiated timeout, pointing just to 192.168.0.102  .Now it it is NOT the 
> leader
>  4) On the server I turn the eth0[192.168.0.101] interface down  [ ifconfig 
> eth0 down ]
>  5) After 26 seconds, application1 says : 
>            :ClientCnxn$SendThread@] - Client session timed out, have not 
> heard from server in 26679ms for sessionid
>             ConnectionStateManager@237] - State change: SUSPENDED
>         NOTE: I'm ignoring the SUSPENDED event
> 6) After 40 seconds, application2 takes leadership
>  7) After 66 seconds, application1 says :
>              

[jira] [Commented] (CURATOR-460) Timed tolerance for connection suspended leads to simultaneous leaders

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/CURATOR-460?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16424962#comment-16424962
 ] 

ASF GitHub Bot commented on CURATOR-460:


Github user cammckenzie commented on a diff in the pull request:

https://github.com/apache/curator/pull/262#discussion_r179022253
  
--- Diff: 
curator-recipes/src/test/java/org/apache/curator/framework/recipes/leader/TestLeaderLatch.java
 ---
@@ -222,6 +223,35 @@ public void notLeader()
 next.add(states.poll(timing.forSessionSleep().milliseconds(), 
TimeUnit.MILLISECONDS));
 next.add(states.poll(timing.forSessionSleep().milliseconds(), 
TimeUnit.MILLISECONDS));
 
Assert.assertTrue(next.equals(Arrays.asList(ConnectionState.LOST.name(), 
"false")) || next.equals(Arrays.asList("false", ConnectionState.LOST.name())), 
next.toString());
+
+latch.close();
+client.close();
+
+timing.sleepABit();
+states.clear();
+server = new TestingServer();
+client = CuratorFrameworkFactory.builder()
+.connectString(server.getConnectString())
+.connectionTimeoutMs(1000)
+.sessionTimeoutMs(timing.session())
+.retryPolicy(new RetryOneTime(1))
+.connectionStateErrorPolicy(new 
SessionConnectionStateErrorPolicy())
+.connectionHandlingPolicy(new 
StandardConnectionHandlingPolicy(30))
+.build();
+
client.getConnectionStateListenable().addListener(stateListener);
+client.start();
+latch = new LeaderLatch(client, "/test");
+latch.addListener(listener);
+latch.start();
+
Assert.assertEquals(states.poll(timing.forWaiting().milliseconds(), 
TimeUnit.MILLISECONDS), ConnectionState.CONNECTED.name());
+
Assert.assertEquals(states.poll(timing.forWaiting().milliseconds(), 
TimeUnit.MILLISECONDS), "true");
+server.close();
+
Assert.assertEquals(states.poll(timing.forWaiting().milliseconds(), 
TimeUnit.MILLISECONDS), ConnectionState.SUSPENDED.name());
+next = Lists.newArrayList();
+
+next.add(states.poll(timing.session() / 3, 
TimeUnit.MILLISECONDS));
+next.add(states.poll(timing.forSleepingABit().milliseconds(), 
TimeUnit.MILLISECONDS));
+
Assert.assertTrue(next.equals(Arrays.asList(ConnectionState.LOST.name(), 
"false")) || next.equals(Arrays.asList("false", ConnectionState.LOST.name())), 
next.toString());
--- End diff --

I think that the existing test case is trying to test connection / session 
loss cases in the context of the LeaderLatch recipe (thus the checks for "true" 
and "false"), whereas for this new test case, we don't really care about the 
fact that a LeaderLatch is running, rather that the ConnectionStateManager is 
doing the right thing. So, I'd prefer to see a standalone test in a 
TestConnectionStateManager class.


> Timed tolerance for connection suspended leads to simultaneous leaders
> --
>
> Key: CURATOR-460
> URL: https://issues.apache.org/jira/browse/CURATOR-460
> Project: Apache Curator
>  Issue Type: Bug
>  Components: Recipes
>Affects Versions: 3.3.0, 4.0.0, 4.0.1
>Reporter: Antonio Rafael Rodrigues
>Assignee: Cameron McKenzie
>Priority: Major
> Attachments: ConnectionStateManager.patch, 
> Console-leaderSelector1.log, Console-leaderSelector2.log, 
> CuratorLeaderSelectorPOC.java, LogApp1.log, LogApp2.log
>
>
> Starting from Curator 3, after losing the connection to zookeeper, curator 
> emits a SUSPENDED event and then starts an internal timer, if the time of the 
> negotiatied session timeout get hit and curator didn't reconnect to zookeeper 
> it emits a LOST event.
> For example :
>  Given negotiated session timeout = 40
> ||Time (seconds)||Event||
> |0|SUSPENDED (curator has been disconnected from zookeeper)|
> |40|LOST (curator is still disconnected, it must have been lost the sesion as 
> it is disconnected for 40sec)|
> Given this scenario we could ,theoretically, ignore the SUSPENDED event and 
> consider the leadership as lost just if the ConnectionStateListener receives 
> the LOST event.
> But this feature seems to have introduced a bug (from my point of view)
> *Case of failure*
> ||Time (seconds)||Event||
> |0|Something went wrong with the connected zookeeper (in my case, the network 
> inteface of zookeeper's server has gone down). Curator stops hearing heart 
> beats from zookeeper, but doesn't lose the connection (from some reason that 
> I 

[jira] [Commented] (CURATOR-460) Timed tolerance for connection suspended leads to simultaneous leaders

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/CURATOR-460?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16424960#comment-16424960
 ] 

ASF GitHub Bot commented on CURATOR-460:


Github user cammckenzie commented on a diff in the pull request:

https://github.com/apache/curator/pull/262#discussion_r179022050
  
--- Diff: 
curator-framework/src/main/java/org/apache/curator/framework/state/ConnectionStateManager.java
 ---
@@ -253,6 +253,7 @@ private void processEvents()
 {
 int lastNegotiatedSessionTimeoutMs = 
client.getZookeeperClient().getLastNegotiatedSessionTimeoutMs();
 int useSessionTimeoutMs = (lastNegotiatedSessionTimeoutMs 
> 0) ? lastNegotiatedSessionTimeoutMs : sessionTimeoutMs;
+useSessionTimeoutMs = sessionExpirationPercent > 0 && 
startOfSuspendedEpoch != 0 ? (useSessionTimeoutMs * sessionExpirationPercent) / 
100 : useSessionTimeoutMs;
--- End diff --

Yeah, you're probably right. If we change to volatile there will be quite a 
few more changes.


> Timed tolerance for connection suspended leads to simultaneous leaders
> --
>
> Key: CURATOR-460
> URL: https://issues.apache.org/jira/browse/CURATOR-460
> Project: Apache Curator
>  Issue Type: Bug
>  Components: Recipes
>Affects Versions: 3.3.0, 4.0.0, 4.0.1
>Reporter: Antonio Rafael Rodrigues
>Assignee: Cameron McKenzie
>Priority: Major
> Attachments: ConnectionStateManager.patch, 
> Console-leaderSelector1.log, Console-leaderSelector2.log, 
> CuratorLeaderSelectorPOC.java, LogApp1.log, LogApp2.log
>
>
> Starting from Curator 3, after losing the connection to zookeeper, curator 
> emits a SUSPENDED event and then starts an internal timer, if the time of the 
> negotiatied session timeout get hit and curator didn't reconnect to zookeeper 
> it emits a LOST event.
> For example :
>  Given negotiated session timeout = 40
> ||Time (seconds)||Event||
> |0|SUSPENDED (curator has been disconnected from zookeeper)|
> |40|LOST (curator is still disconnected, it must have been lost the sesion as 
> it is disconnected for 40sec)|
> Given this scenario we could ,theoretically, ignore the SUSPENDED event and 
> consider the leadership as lost just if the ConnectionStateListener receives 
> the LOST event.
> But this feature seems to have introduced a bug (from my point of view)
> *Case of failure*
> ||Time (seconds)||Event||
> |0|Something went wrong with the connected zookeeper (in my case, the network 
> inteface of zookeeper's server has gone down). Curator stops hearing heart 
> beats from zookeeper, but doesn't lose the connection (from some reason that 
> I don't know, if the network interface of the server goes down, Curator 
> doesn't lose the connection)|
> |~26.666|SUSPENDED (after 26 seconds without hearing heartbeats, curator 
> emits a SUSPENDED event) 26 is from "readTimeout = sessionTimeout * 2 / 3" 
> from the class ClientCnxn.java from zookeeper client. At this point, curator 
> starts counting 40 sec.|
> |26.666 to 40|During this period, Curator is trying to connect to other 
> zookeeper instances but the other instances, in my example case, are also 
> uncheachable.|
> |40|Session has expired and the other instance has taken leadership ( in my 
> example, the other instance can connect to zookeeper )|
> |66.666|LOST ( after 40sec from SUSPENDED, curator finally sends LOST)|
> As you can see, if we are ignoring the SUSPENDED event, the second 
> application instance acquires the leadership ~26 seconds before the first 
> instance notice that it lost the leadership.
>  I understand it seems to be a very rare case, but either way I think it 
> should be addressed.
> *Reproduce it*
> I have came up with a way to reproduce this easily. I know this is not a 
> recommended production setup, but it reproduces the problem anyway.
> 1) On a virtual machine, with 2 network interfaces (eth0[192.168.0.101], 
> eth1[192.168.0.102]) , I installed one zookeeper instance.
>  2) I setup application1 with the leadership receipe, with 40sec of 
> negotiated timeout, pointing just to 192.168.0.101  .Now it is the leader
>  3) I setup application2 with the leadership receipe, with 40sec of 
> negotiated timeout, pointing just to 192.168.0.102  .Now it it is NOT the 
> leader
>  4) On the server I turn the eth0[192.168.0.101] interface down  [ ifconfig 
> eth0 down ]
>  5) After 26 seconds, application1 says : 
>            :ClientCnxn$SendThread@] - Client session timed out, have not 
> heard from server in 26679ms for sessionid
>             ConnectionStateManager@237] - State change: SUSPENDED
>         NOTE: I'm ignoring the SUSPENDED event
> 6) After 40 seconds, application2 takes leadership
>  7) After 66 seconds, 

[jira] [Commented] (CURATOR-460) Timed tolerance for connection suspended leads to simultaneous leaders

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/CURATOR-460?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16424930#comment-16424930
 ] 

ASF GitHub Bot commented on CURATOR-460:


Github user arrodrigues commented on a diff in the pull request:

https://github.com/apache/curator/pull/262#discussion_r179016314
  
--- Diff: 
curator-recipes/src/test/java/org/apache/curator/framework/recipes/leader/TestLeaderLatch.java
 ---
@@ -222,6 +223,35 @@ public void notLeader()
 next.add(states.poll(timing.forSessionSleep().milliseconds(), 
TimeUnit.MILLISECONDS));
 next.add(states.poll(timing.forSessionSleep().milliseconds(), 
TimeUnit.MILLISECONDS));
 
Assert.assertTrue(next.equals(Arrays.asList(ConnectionState.LOST.name(), 
"false")) || next.equals(Arrays.asList("false", ConnectionState.LOST.name())), 
next.toString());
+
+latch.close();
+client.close();
+
+timing.sleepABit();
+states.clear();
+server = new TestingServer();
+client = CuratorFrameworkFactory.builder()
+.connectString(server.getConnectString())
+.connectionTimeoutMs(1000)
+.sessionTimeoutMs(timing.session())
+.retryPolicy(new RetryOneTime(1))
+.connectionStateErrorPolicy(new 
SessionConnectionStateErrorPolicy())
+.connectionHandlingPolicy(new 
StandardConnectionHandlingPolicy(30))
+.build();
+
client.getConnectionStateListenable().addListener(stateListener);
+client.start();
+latch = new LeaderLatch(client, "/test");
+latch.addListener(listener);
+latch.start();
+
Assert.assertEquals(states.poll(timing.forWaiting().milliseconds(), 
TimeUnit.MILLISECONDS), ConnectionState.CONNECTED.name());
+
Assert.assertEquals(states.poll(timing.forWaiting().milliseconds(), 
TimeUnit.MILLISECONDS), "true");
+server.close();
+
Assert.assertEquals(states.poll(timing.forWaiting().milliseconds(), 
TimeUnit.MILLISECONDS), ConnectionState.SUSPENDED.name());
+next = Lists.newArrayList();
+
+next.add(states.poll(timing.session() / 3, 
TimeUnit.MILLISECONDS));
+next.add(states.poll(timing.forSleepingABit().milliseconds(), 
TimeUnit.MILLISECONDS));
+
Assert.assertTrue(next.equals(Arrays.asList(ConnectionState.LOST.name(), 
"false")) || next.equals(Arrays.asList("false", ConnectionState.LOST.name())), 
next.toString());
--- End diff --

I agree with your point. I am also not confortable with the place I put 
this test, but just to explain myself:

I've chosen to improve this already existing test because its name is 
"testErrorPolicies" and the fix was exactly on the errorPolicies behaviour. 
Moreover, the already existing test, tests the injection of LOST after 100% of 
the session timeout has elapsed, I've just copy-pasted some lines, and set to 
inject lost after 30% of sessionTimeout.
In my opinion, if we move the lines I added from here to another test 
class, we should also move the whole already existing testErrorPolicies method.

What do you think?


> Timed tolerance for connection suspended leads to simultaneous leaders
> --
>
> Key: CURATOR-460
> URL: https://issues.apache.org/jira/browse/CURATOR-460
> Project: Apache Curator
>  Issue Type: Bug
>  Components: Recipes
>Affects Versions: 3.3.0, 4.0.0, 4.0.1
>Reporter: Antonio Rafael Rodrigues
>Assignee: Cameron McKenzie
>Priority: Major
> Attachments: ConnectionStateManager.patch, 
> Console-leaderSelector1.log, Console-leaderSelector2.log, 
> CuratorLeaderSelectorPOC.java, LogApp1.log, LogApp2.log
>
>
> Starting from Curator 3, after losing the connection to zookeeper, curator 
> emits a SUSPENDED event and then starts an internal timer, if the time of the 
> negotiatied session timeout get hit and curator didn't reconnect to zookeeper 
> it emits a LOST event.
> For example :
>  Given negotiated session timeout = 40
> ||Time (seconds)||Event||
> |0|SUSPENDED (curator has been disconnected from zookeeper)|
> |40|LOST (curator is still disconnected, it must have been lost the sesion as 
> it is disconnected for 40sec)|
> Given this scenario we could ,theoretically, ignore the SUSPENDED event and 
> consider the leadership as lost just if the ConnectionStateListener receives 
> the LOST event.
> But this feature seems to have introduced a bug (from my point of view)
> *Case of failure*
> ||Time (seconds)||Event||
> 

[jira] [Commented] (CURATOR-460) Timed tolerance for connection suspended leads to simultaneous leaders

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/CURATOR-460?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16424878#comment-16424878
 ] 

ASF GitHub Bot commented on CURATOR-460:


Github user arrodrigues commented on a diff in the pull request:

https://github.com/apache/curator/pull/262#discussion_r179010977
  
--- Diff: 
curator-framework/src/main/java/org/apache/curator/framework/state/ConnectionStateManager.java
 ---
@@ -253,6 +253,7 @@ private void processEvents()
 {
 int lastNegotiatedSessionTimeoutMs = 
client.getZookeeperClient().getLastNegotiatedSessionTimeoutMs();
 int useSessionTimeoutMs = (lastNegotiatedSessionTimeoutMs 
> 0) ? lastNegotiatedSessionTimeoutMs : sessionTimeoutMs;
+useSessionTimeoutMs = sessionExpirationPercent > 0 && 
startOfSuspendedEpoch != 0 ? (useSessionTimeoutMs * sessionExpirationPercent) / 
100 : useSessionTimeoutMs;
--- End diff --

I agree that make startOfSuspendedEpoch vollatile would be cleaner, but 
given that all other points accessing startOfSuspendedEpoch are already 
synchronized, I think that synchronize also this point would be more consistent 
with the rest of the code. Just an oppinion.


> Timed tolerance for connection suspended leads to simultaneous leaders
> --
>
> Key: CURATOR-460
> URL: https://issues.apache.org/jira/browse/CURATOR-460
> Project: Apache Curator
>  Issue Type: Bug
>  Components: Recipes
>Affects Versions: 3.3.0, 4.0.0, 4.0.1
>Reporter: Antonio Rafael Rodrigues
>Assignee: Cameron McKenzie
>Priority: Major
> Attachments: ConnectionStateManager.patch, 
> Console-leaderSelector1.log, Console-leaderSelector2.log, 
> CuratorLeaderSelectorPOC.java, LogApp1.log, LogApp2.log
>
>
> Starting from Curator 3, after losing the connection to zookeeper, curator 
> emits a SUSPENDED event and then starts an internal timer, if the time of the 
> negotiatied session timeout get hit and curator didn't reconnect to zookeeper 
> it emits a LOST event.
> For example :
>  Given negotiated session timeout = 40
> ||Time (seconds)||Event||
> |0|SUSPENDED (curator has been disconnected from zookeeper)|
> |40|LOST (curator is still disconnected, it must have been lost the sesion as 
> it is disconnected for 40sec)|
> Given this scenario we could ,theoretically, ignore the SUSPENDED event and 
> consider the leadership as lost just if the ConnectionStateListener receives 
> the LOST event.
> But this feature seems to have introduced a bug (from my point of view)
> *Case of failure*
> ||Time (seconds)||Event||
> |0|Something went wrong with the connected zookeeper (in my case, the network 
> inteface of zookeeper's server has gone down). Curator stops hearing heart 
> beats from zookeeper, but doesn't lose the connection (from some reason that 
> I don't know, if the network interface of the server goes down, Curator 
> doesn't lose the connection)|
> |~26.666|SUSPENDED (after 26 seconds without hearing heartbeats, curator 
> emits a SUSPENDED event) 26 is from "readTimeout = sessionTimeout * 2 / 3" 
> from the class ClientCnxn.java from zookeeper client. At this point, curator 
> starts counting 40 sec.|
> |26.666 to 40|During this period, Curator is trying to connect to other 
> zookeeper instances but the other instances, in my example case, are also 
> uncheachable.|
> |40|Session has expired and the other instance has taken leadership ( in my 
> example, the other instance can connect to zookeeper )|
> |66.666|LOST ( after 40sec from SUSPENDED, curator finally sends LOST)|
> As you can see, if we are ignoring the SUSPENDED event, the second 
> application instance acquires the leadership ~26 seconds before the first 
> instance notice that it lost the leadership.
>  I understand it seems to be a very rare case, but either way I think it 
> should be addressed.
> *Reproduce it*
> I have came up with a way to reproduce this easily. I know this is not a 
> recommended production setup, but it reproduces the problem anyway.
> 1) On a virtual machine, with 2 network interfaces (eth0[192.168.0.101], 
> eth1[192.168.0.102]) , I installed one zookeeper instance.
>  2) I setup application1 with the leadership receipe, with 40sec of 
> negotiated timeout, pointing just to 192.168.0.101  .Now it is the leader
>  3) I setup application2 with the leadership receipe, with 40sec of 
> negotiated timeout, pointing just to 192.168.0.102  .Now it it is NOT the 
> leader
>  4) On the server I turn the eth0[192.168.0.101] interface down  [ ifconfig 
> eth0 down ]
>  5) After 26 seconds, application1 says : 
>            :ClientCnxn$SendThread@] - Client session timed out, have not 
> heard from server in 26679ms for sessionid
>             

[jira] [Commented] (CURATOR-460) Timed tolerance for connection suspended leads to simultaneous leaders

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/CURATOR-460?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16424838#comment-16424838
 ] 

ASF GitHub Bot commented on CURATOR-460:


Github user cammckenzie commented on a diff in the pull request:

https://github.com/apache/curator/pull/262#discussion_r179001383
  
--- Diff: 
curator-framework/src/main/java/org/apache/curator/framework/state/ConnectionStateManager.java
 ---
@@ -253,6 +253,7 @@ private void processEvents()
 {
 int lastNegotiatedSessionTimeoutMs = 
client.getZookeeperClient().getLastNegotiatedSessionTimeoutMs();
 int useSessionTimeoutMs = (lastNegotiatedSessionTimeoutMs 
> 0) ? lastNegotiatedSessionTimeoutMs : sessionTimeoutMs;
+useSessionTimeoutMs = sessionExpirationPercent > 0 && 
startOfSuspendedEpoch != 0 ? (useSessionTimeoutMs * sessionExpirationPercent) / 
100 : useSessionTimeoutMs;
--- End diff --

I think you're right about synchronization of access to the 
startOfSuspendedEpoch here. It can be modified via the addStateChange and 
setToSuspended methods (which already synchronize access to it). I wonder if 
it's cleaner just to make startOfSuspendedEpoch volatile?


> Timed tolerance for connection suspended leads to simultaneous leaders
> --
>
> Key: CURATOR-460
> URL: https://issues.apache.org/jira/browse/CURATOR-460
> Project: Apache Curator
>  Issue Type: Bug
>  Components: Recipes
>Affects Versions: 3.3.0, 4.0.0, 4.0.1
>Reporter: Antonio Rafael Rodrigues
>Assignee: Cameron McKenzie
>Priority: Major
> Attachments: ConnectionStateManager.patch, 
> Console-leaderSelector1.log, Console-leaderSelector2.log, 
> CuratorLeaderSelectorPOC.java, LogApp1.log, LogApp2.log
>
>
> Starting from Curator 3, after losing the connection to zookeeper, curator 
> emits a SUSPENDED event and then starts an internal timer, if the time of the 
> negotiatied session timeout get hit and curator didn't reconnect to zookeeper 
> it emits a LOST event.
> For example :
>  Given negotiated session timeout = 40
> ||Time (seconds)||Event||
> |0|SUSPENDED (curator has been disconnected from zookeeper)|
> |40|LOST (curator is still disconnected, it must have been lost the sesion as 
> it is disconnected for 40sec)|
> Given this scenario we could ,theoretically, ignore the SUSPENDED event and 
> consider the leadership as lost just if the ConnectionStateListener receives 
> the LOST event.
> But this feature seems to have introduced a bug (from my point of view)
> *Case of failure*
> ||Time (seconds)||Event||
> |0|Something went wrong with the connected zookeeper (in my case, the network 
> inteface of zookeeper's server has gone down). Curator stops hearing heart 
> beats from zookeeper, but doesn't lose the connection (from some reason that 
> I don't know, if the network interface of the server goes down, Curator 
> doesn't lose the connection)|
> |~26.666|SUSPENDED (after 26 seconds without hearing heartbeats, curator 
> emits a SUSPENDED event) 26 is from "readTimeout = sessionTimeout * 2 / 3" 
> from the class ClientCnxn.java from zookeeper client. At this point, curator 
> starts counting 40 sec.|
> |26.666 to 40|During this period, Curator is trying to connect to other 
> zookeeper instances but the other instances, in my example case, are also 
> uncheachable.|
> |40|Session has expired and the other instance has taken leadership ( in my 
> example, the other instance can connect to zookeeper )|
> |66.666|LOST ( after 40sec from SUSPENDED, curator finally sends LOST)|
> As you can see, if we are ignoring the SUSPENDED event, the second 
> application instance acquires the leadership ~26 seconds before the first 
> instance notice that it lost the leadership.
>  I understand it seems to be a very rare case, but either way I think it 
> should be addressed.
> *Reproduce it*
> I have came up with a way to reproduce this easily. I know this is not a 
> recommended production setup, but it reproduces the problem anyway.
> 1) On a virtual machine, with 2 network interfaces (eth0[192.168.0.101], 
> eth1[192.168.0.102]) , I installed one zookeeper instance.
>  2) I setup application1 with the leadership receipe, with 40sec of 
> negotiated timeout, pointing just to 192.168.0.101  .Now it is the leader
>  3) I setup application2 with the leadership receipe, with 40sec of 
> negotiated timeout, pointing just to 192.168.0.102  .Now it it is NOT the 
> leader
>  4) On the server I turn the eth0[192.168.0.101] interface down  [ ifconfig 
> eth0 down ]
>  5) After 26 seconds, application1 says : 
>            :ClientCnxn$SendThread@] - Client session timed out, have not 
> heard from server in 26679ms for sessionid
>             

[jira] [Commented] (CURATOR-460) Timed tolerance for connection suspended leads to simultaneous leaders

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/CURATOR-460?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16424837#comment-16424837
 ] 

ASF GitHub Bot commented on CURATOR-460:


Github user cammckenzie commented on a diff in the pull request:

https://github.com/apache/curator/pull/262#discussion_r179001451
  
--- Diff: 
curator-recipes/src/test/java/org/apache/curator/framework/recipes/leader/TestLeaderLatch.java
 ---
@@ -222,6 +223,35 @@ public void notLeader()
 next.add(states.poll(timing.forSessionSleep().milliseconds(), 
TimeUnit.MILLISECONDS));
 next.add(states.poll(timing.forSessionSleep().milliseconds(), 
TimeUnit.MILLISECONDS));
 
Assert.assertTrue(next.equals(Arrays.asList(ConnectionState.LOST.name(), 
"false")) || next.equals(Arrays.asList("false", ConnectionState.LOST.name())), 
next.toString());
+
+latch.close();
+client.close();
+
+timing.sleepABit();
+states.clear();
+server = new TestingServer();
+client = CuratorFrameworkFactory.builder()
+.connectString(server.getConnectString())
+.connectionTimeoutMs(1000)
+.sessionTimeoutMs(timing.session())
+.retryPolicy(new RetryOneTime(1))
+.connectionStateErrorPolicy(new 
SessionConnectionStateErrorPolicy())
+.connectionHandlingPolicy(new 
StandardConnectionHandlingPolicy(30))
+.build();
+
client.getConnectionStateListenable().addListener(stateListener);
+client.start();
+latch = new LeaderLatch(client, "/test");
+latch.addListener(listener);
+latch.start();
+
Assert.assertEquals(states.poll(timing.forWaiting().milliseconds(), 
TimeUnit.MILLISECONDS), ConnectionState.CONNECTED.name());
+
Assert.assertEquals(states.poll(timing.forWaiting().milliseconds(), 
TimeUnit.MILLISECONDS), "true");
+server.close();
+
Assert.assertEquals(states.poll(timing.forWaiting().milliseconds(), 
TimeUnit.MILLISECONDS), ConnectionState.SUSPENDED.name());
+next = Lists.newArrayList();
+
+next.add(states.poll(timing.session() / 3, 
TimeUnit.MILLISECONDS));
+next.add(states.poll(timing.forSleepingABit().milliseconds(), 
TimeUnit.MILLISECONDS));
+
Assert.assertTrue(next.equals(Arrays.asList(ConnectionState.LOST.name(), 
"false")) || next.equals(Arrays.asList("false", ConnectionState.LOST.name())), 
next.toString());
--- End diff --

The issue isn't really the LeaderLatch specifically is it? I think it's 
probably cleaner to create a test case under curator-framework for the 
ConnectionStateManager and then have a test explicitly for injected LOST 
events, rather than have something in the LeaderLatch tests.

Because, this test doesn't really cover the use case that was discussed 
originally (2 clients being leader at the same time).


> Timed tolerance for connection suspended leads to simultaneous leaders
> --
>
> Key: CURATOR-460
> URL: https://issues.apache.org/jira/browse/CURATOR-460
> Project: Apache Curator
>  Issue Type: Bug
>  Components: Recipes
>Affects Versions: 3.3.0, 4.0.0, 4.0.1
>Reporter: Antonio Rafael Rodrigues
>Assignee: Cameron McKenzie
>Priority: Major
> Attachments: ConnectionStateManager.patch, 
> Console-leaderSelector1.log, Console-leaderSelector2.log, 
> CuratorLeaderSelectorPOC.java, LogApp1.log, LogApp2.log
>
>
> Starting from Curator 3, after losing the connection to zookeeper, curator 
> emits a SUSPENDED event and then starts an internal timer, if the time of the 
> negotiatied session timeout get hit and curator didn't reconnect to zookeeper 
> it emits a LOST event.
> For example :
>  Given negotiated session timeout = 40
> ||Time (seconds)||Event||
> |0|SUSPENDED (curator has been disconnected from zookeeper)|
> |40|LOST (curator is still disconnected, it must have been lost the sesion as 
> it is disconnected for 40sec)|
> Given this scenario we could ,theoretically, ignore the SUSPENDED event and 
> consider the leadership as lost just if the ConnectionStateListener receives 
> the LOST event.
> But this feature seems to have introduced a bug (from my point of view)
> *Case of failure*
> ||Time (seconds)||Event||
> |0|Something went wrong with the connected zookeeper (in my case, the network 
> inteface of zookeeper's server has gone down). Curator stops hearing heart 
> beats from zookeeper, but doesn't lose the connection (from some reason that 
> I 

[jira] [Commented] (CURATOR-460) Timed tolerance for connection suspended leads to simultaneous leaders

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/CURATOR-460?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16423402#comment-16423402
 ] 

ASF GitHub Bot commented on CURATOR-460:


GitHub user arrodrigues opened a pull request:

https://github.com/apache/curator/pull/262

[CURATOR-460] Timed tolerance for connection suspended leads to 
simultaneous leaders



You can merge this pull request into a Git repository by running:

$ git pull https://github.com/arrodrigues/curator CURATOR-460

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/curator/pull/262.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #262


commit 72b7b8709c822b9170d8d7beb2814c4c36b89e58
Author: javando 
Date:   2018-04-03T02:27:13Z

CURATOR-460 Adjust to obey sessionExpirationPercent value




> Timed tolerance for connection suspended leads to simultaneous leaders
> --
>
> Key: CURATOR-460
> URL: https://issues.apache.org/jira/browse/CURATOR-460
> Project: Apache Curator
>  Issue Type: Bug
>  Components: Recipes
>Affects Versions: 3.3.0, 4.0.0, 4.0.1
>Reporter: Antonio Rafael Rodrigues
>Assignee: Cameron McKenzie
>Priority: Major
> Attachments: ConnectionStateManager.patch, 
> Console-leaderSelector1.log, Console-leaderSelector2.log, 
> CuratorLeaderSelectorPOC.java, LogApp1.log, LogApp2.log
>
>
> Starting from Curator 3, after losing the connection to zookeeper, curator 
> emits a SUSPENDED event and then starts an internal timer, if the time of the 
> negotiatied session timeout get hit and curator didn't reconnect to zookeeper 
> it emits a LOST event.
> For example :
>  Given negotiated session timeout = 40
> ||Time (seconds)||Event||
> |0|SUSPENDED (curator has been disconnected from zookeeper)|
> |40|LOST (curator is still disconnected, it must have been lost the sesion as 
> it is disconnected for 40sec)|
> Given this scenario we could ,theoretically, ignore the SUSPENDED event and 
> consider the leadership as lost just if the ConnectionStateListener receives 
> the LOST event.
> But this feature seems to have introduced a bug (from my point of view)
> *Case of failure*
> ||Time (seconds)||Event||
> |0|Something went wrong with the connected zookeeper (in my case, the network 
> inteface of zookeeper's server has gone down). Curator stops hearing heart 
> beats from zookeeper, but doesn't lose the connection (from some reason that 
> I don't know, if the network interface of the server goes down, Curator 
> doesn't lose the connection)|
> |~26.666|SUSPENDED (after 26 seconds without hearing heartbeats, curator 
> emits a SUSPENDED event) 26 is from "readTimeout = sessionTimeout * 2 / 3" 
> from the class ClientCnxn.java from zookeeper client. At this point, curator 
> starts counting 40 sec.|
> |26.666 to 40|During this period, Curator is trying to connect to other 
> zookeeper instances but the other instances, in my example case, are also 
> uncheachable.|
> |40|Session has expired and the other instance has taken leadership ( in my 
> example, the other instance can connect to zookeeper )|
> |66.666|LOST ( after 40sec from SUSPENDED, curator finally sends LOST)|
> As you can see, if we are ignoring the SUSPENDED event, the second 
> application instance acquires the leadership ~26 seconds before the first 
> instance notice that it lost the leadership.
>  I understand it seems to be a very rare case, but either way I think it 
> should be addressed.
> *Reproduce it*
> I have came up with a way to reproduce this easily. I know this is not a 
> recommended production setup, but it reproduces the problem anyway.
> 1) On a virtual machine, with 2 network interfaces (eth0[192.168.0.101], 
> eth1[192.168.0.102]) , I installed one zookeeper instance.
>  2) I setup application1 with the leadership receipe, with 40sec of 
> negotiated timeout, pointing just to 192.168.0.101  .Now it is the leader
>  3) I setup application2 with the leadership receipe, with 40sec of 
> negotiated timeout, pointing just to 192.168.0.102  .Now it it is NOT the 
> leader
>  4) On the server I turn the eth0[192.168.0.101] interface down  [ ifconfig 
> eth0 down ]
>  5) After 26 seconds, application1 says : 
>            :ClientCnxn$SendThread@] - Client session timed out, have not 
> heard from server in 26679ms for sessionid
>             ConnectionStateManager@237] - State change: SUSPENDED
>         NOTE: I'm ignoring the SUSPENDED event
> 6) After 40 seconds, application2 takes leadership
>  7) After 66 seconds, application1 says :
>              ConnectionStateManager@237] - State change: LOST
>          
>         

[jira] [Commented] (CURATOR-460) Timed tolerance for connection suspended leads to simultaneous leaders

[Cameron McKenzie (JIRA)]

[ 
https://issues.apache.org/jira/browse/CURATOR-460?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16421899#comment-16421899
 ] 

Cameron McKenzie commented on CURATOR-460:
--

hey [~arrodrigues], I think creating a PR is probably the cleanest way forward.

I can't access code at the moment, but should be able to look at stuff in the 
next couple of days.

cheers

> Timed tolerance for connection suspended leads to simultaneous leaders
> --
>
> Key: CURATOR-460
> URL: https://issues.apache.org/jira/browse/CURATOR-460
> Project: Apache Curator
>  Issue Type: Bug
>  Components: Recipes
>Affects Versions: 3.3.0, 4.0.0, 4.0.1
>Reporter: Antonio Rafael Rodrigues
>Assignee: Cameron McKenzie
>Priority: Major
> Attachments: ConnectionStateManager.patch, 
> Console-leaderSelector1.log, Console-leaderSelector2.log, 
> CuratorLeaderSelectorPOC.java, LogApp1.log, LogApp2.log
>
>
> Starting from Curator 3, after losing the connection to zookeeper, curator 
> emits a SUSPENDED event and then starts an internal timer, if the time of the 
> negotiatied session timeout get hit and curator didn't reconnect to zookeeper 
> it emits a LOST event.
> For example :
>  Given negotiated session timeout = 40
> ||Time (seconds)||Event||
> |0|SUSPENDED (curator has been disconnected from zookeeper)|
> |40|LOST (curator is still disconnected, it must have been lost the sesion as 
> it is disconnected for 40sec)|
> Given this scenario we could ,theoretically, ignore the SUSPENDED event and 
> consider the leadership as lost just if the ConnectionStateListener receives 
> the LOST event.
> But this feature seems to have introduced a bug (from my point of view)
> *Case of failure*
> ||Time (seconds)||Event||
> |0|Something went wrong with the connected zookeeper (in my case, the network 
> inteface of zookeeper's server has gone down). Curator stops hearing heart 
> beats from zookeeper, but doesn't lose the connection (from some reason that 
> I don't know, if the network interface of the server goes down, Curator 
> doesn't lose the connection)|
> |~26.666|SUSPENDED (after 26 seconds without hearing heartbeats, curator 
> emits a SUSPENDED event) 26 is from "readTimeout = sessionTimeout * 2 / 3" 
> from the class ClientCnxn.java from zookeeper client. At this point, curator 
> starts counting 40 sec.|
> |26.666 to 40|During this period, Curator is trying to connect to other 
> zookeeper instances but the other instances, in my example case, are also 
> uncheachable.|
> |40|Session has expired and the other instance has taken leadership ( in my 
> example, the other instance can connect to zookeeper )|
> |66.666|LOST ( after 40sec from SUSPENDED, curator finally sends LOST)|
> As you can see, if we are ignoring the SUSPENDED event, the second 
> application instance acquires the leadership ~26 seconds before the first 
> instance notice that it lost the leadership.
>  I understand it seems to be a very rare case, but either way I think it 
> should be addressed.
> *Reproduce it*
> I have came up with a way to reproduce this easily. I know this is not a 
> recommended production setup, but it reproduces the problem anyway.
> 1) On a virtual machine, with 2 network interfaces (eth0[192.168.0.101], 
> eth1[192.168.0.102]) , I installed one zookeeper instance.
>  2) I setup application1 with the leadership receipe, with 40sec of 
> negotiated timeout, pointing just to 192.168.0.101  .Now it is the leader
>  3) I setup application2 with the leadership receipe, with 40sec of 
> negotiated timeout, pointing just to 192.168.0.102  .Now it it is NOT the 
> leader
>  4) On the server I turn the eth0[192.168.0.101] interface down  [ ifconfig 
> eth0 down ]
>  5) After 26 seconds, application1 says : 
>            :ClientCnxn$SendThread@] - Client session timed out, have not 
> heard from server in 26679ms for sessionid
>             ConnectionStateManager@237] - State change: SUSPENDED
>         NOTE: I'm ignoring the SUSPENDED event
> 6) After 40 seconds, application2 takes leadership
>  7) After 66 seconds, application1 says :
>              ConnectionStateManager@237] - State change: LOST
>          
>          NOTE: Just at this point, I consider that the application1 has lost 
> leadership
> Then, for 26 seconds, we had 2 leaders.
> If you confirm it as a bug, I think I could help.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (CURATOR-460) Timed tolerance for connection suspended leads to simultaneous leaders

[Antonio Rafael Rodrigues (JIRA)]

[ 
https://issues.apache.org/jira/browse/CURATOR-460?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16421369#comment-16421369
 ] 

Antonio Rafael Rodrigues commented on CURATOR-460:
--

1 - I managed to create a test that tests the patch I attached here. I'd like 
to know if I should open a pull request just for it, or If I should post here 
to then Cameron can incorporate it on his final solution.
2 - For the innitially reported issue, I didn't find a way to simulate loss of 
heartbeats. I thought about creating a sleep on the ConnectionStateListener 
declared in TestLeaderLatch#testErrorPolicies so that we'd get an delay on 
receiving the SUSPENDED, but I thought it'd be pointless. 
3 - About the StandardConnectionHandlingPolicy's default value, perhaps we 
should consider removing the no-args constructor.

> Timed tolerance for connection suspended leads to simultaneous leaders
> --
>
> Key: CURATOR-460
> URL: https://issues.apache.org/jira/browse/CURATOR-460
> Project: Apache Curator
>  Issue Type: Bug
>  Components: Recipes
>Affects Versions: 3.3.0, 4.0.0, 4.0.1
>Reporter: Antonio Rafael Rodrigues
>Assignee: Cameron McKenzie
>Priority: Major
> Attachments: ConnectionStateManager.patch, 
> Console-leaderSelector1.log, Console-leaderSelector2.log, 
> CuratorLeaderSelectorPOC.java, LogApp1.log, LogApp2.log
>
>
> Starting from Curator 3, after losing the connection to zookeeper, curator 
> emits a SUSPENDED event and then starts an internal timer, if the time of the 
> negotiatied session timeout get hit and curator didn't reconnect to zookeeper 
> it emits a LOST event.
> For example :
>  Given negotiated session timeout = 40
> ||Time (seconds)||Event||
> |0|SUSPENDED (curator has been disconnected from zookeeper)|
> |40|LOST (curator is still disconnected, it must have been lost the sesion as 
> it is disconnected for 40sec)|
> Given this scenario we could ,theoretically, ignore the SUSPENDED event and 
> consider the leadership as lost just if the ConnectionStateListener receives 
> the LOST event.
> But this feature seems to have introduced a bug (from my point of view)
> *Case of failure*
> ||Time (seconds)||Event||
> |0|Something went wrong with the connected zookeeper (in my case, the network 
> inteface of zookeeper's server has gone down). Curator stops hearing heart 
> beats from zookeeper, but doesn't lose the connection (from some reason that 
> I don't know, if the network interface of the server goes down, Curator 
> doesn't lose the connection)|
> |~26.666|SUSPENDED (after 26 seconds without hearing heartbeats, curator 
> emits a SUSPENDED event) 26 is from "readTimeout = sessionTimeout * 2 / 3" 
> from the class ClientCnxn.java from zookeeper client. At this point, curator 
> starts counting 40 sec.|
> |26.666 to 40|During this period, Curator is trying to connect to other 
> zookeeper instances but the other instances, in my example case, are also 
> uncheachable.|
> |40|Session has expired and the other instance has taken leadership ( in my 
> example, the other instance can connect to zookeeper )|
> |66.666|LOST ( after 40sec from SUSPENDED, curator finally sends LOST)|
> As you can see, if we are ignoring the SUSPENDED event, the second 
> application instance acquires the leadership ~26 seconds before the first 
> instance notice that it lost the leadership.
>  I understand it seems to be a very rare case, but either way I think it 
> should be addressed.
> *Reproduce it*
> I have came up with a way to reproduce this easily. I know this is not a 
> recommended production setup, but it reproduces the problem anyway.
> 1) On a virtual machine, with 2 network interfaces (eth0[192.168.0.101], 
> eth1[192.168.0.102]) , I installed one zookeeper instance.
>  2) I setup application1 with the leadership receipe, with 40sec of 
> negotiated timeout, pointing just to 192.168.0.101  .Now it is the leader
>  3) I setup application2 with the leadership receipe, with 40sec of 
> negotiated timeout, pointing just to 192.168.0.102  .Now it it is NOT the 
> leader
>  4) On the server I turn the eth0[192.168.0.101] interface down  [ ifconfig 
> eth0 down ]
>  5) After 26 seconds, application1 says : 
>            :ClientCnxn$SendThread@] - Client session timed out, have not 
> heard from server in 26679ms for sessionid
>             ConnectionStateManager@237] - State change: SUSPENDED
>         NOTE: I'm ignoring the SUSPENDED event
> 6) After 40 seconds, application2 takes leadership
>  7) After 66 seconds, application1 says :
>              ConnectionStateManager@237] - State change: LOST
>          
>          NOTE: Just at this point, I consider that the application1 has lost 
> leadership
> Then, for 26 seconds, we had 2 leaders.
> If you 

[jira] [Commented] (CURATOR-460) Timed tolerance for connection suspended leads to simultaneous leaders

[Jordan Zimmerman (JIRA)]

[ 
https://issues.apache.org/jira/browse/CURATOR-460?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16420560#comment-16420560
 ] 

Jordan Zimmerman commented on CURATOR-460:
--

I don't know... it's all related. We could just do everything here. We should 
also consider changing {{StandardConnectionHandlingPolicy}} so that it's 
default value is 33. But, I worry how that will affect existing users. Thoughts 
([~cammckenzie]).

> Timed tolerance for connection suspended leads to simultaneous leaders
> --
>
> Key: CURATOR-460
> URL: https://issues.apache.org/jira/browse/CURATOR-460
> Project: Apache Curator
>  Issue Type: Bug
>  Components: Recipes
>Affects Versions: 3.3.0, 4.0.0, 4.0.1
>Reporter: Antonio Rafael Rodrigues
>Assignee: Cameron McKenzie
>Priority: Major
> Attachments: ConnectionStateManager.patch, 
> Console-leaderSelector1.log, Console-leaderSelector2.log, 
> CuratorLeaderSelectorPOC.java, LogApp1.log, LogApp2.log
>
>
> Starting from Curator 3, after losing the connection to zookeeper, curator 
> emits a SUSPENDED event and then starts an internal timer, if the time of the 
> negotiatied session timeout get hit and curator didn't reconnect to zookeeper 
> it emits a LOST event.
> For example :
>  Given negotiated session timeout = 40
> ||Time (seconds)||Event||
> |0|SUSPENDED (curator has been disconnected from zookeeper)|
> |40|LOST (curator is still disconnected, it must have been lost the sesion as 
> it is disconnected for 40sec)|
> Given this scenario we could ,theoretically, ignore the SUSPENDED event and 
> consider the leadership as lost just if the ConnectionStateListener receives 
> the LOST event.
> But this feature seems to have introduced a bug (from my point of view)
> *Case of failure*
> ||Time (seconds)||Event||
> |0|Something went wrong with the connected zookeeper (in my case, the network 
> inteface of zookeeper's server has gone down). Curator stops hearing heart 
> beats from zookeeper, but doesn't lose the connection (from some reason that 
> I don't know, if the network interface of the server goes down, Curator 
> doesn't lose the connection)|
> |~26.666|SUSPENDED (after 26 seconds without hearing heartbeats, curator 
> emits a SUSPENDED event) 26 is from "readTimeout = sessionTimeout * 2 / 3" 
> from the class ClientCnxn.java from zookeeper client. At this point, curator 
> starts counting 40 sec.|
> |26.666 to 40|During this period, Curator is trying to connect to other 
> zookeeper instances but the other instances, in my example case, are also 
> uncheachable.|
> |40|Session has expired and the other instance has taken leadership ( in my 
> example, the other instance can connect to zookeeper )|
> |66.666|LOST ( after 40sec from SUSPENDED, curator finally sends LOST)|
> As you can see, if we are ignoring the SUSPENDED event, the second 
> application instance acquires the leadership ~26 seconds before the first 
> instance notice that it lost the leadership.
>  I understand it seems to be a very rare case, but either way I think it 
> should be addressed.
> *Reproduce it*
> I have came up with a way to reproduce this easily. I know this is not a 
> recommended production setup, but it reproduces the problem anyway.
> 1) On a virtual machine, with 2 network interfaces (eth0[192.168.0.101], 
> eth1[192.168.0.102]) , I installed one zookeeper instance.
>  2) I setup application1 with the leadership receipe, with 40sec of 
> negotiated timeout, pointing just to 192.168.0.101  .Now it is the leader
>  3) I setup application2 with the leadership receipe, with 40sec of 
> negotiated timeout, pointing just to 192.168.0.102  .Now it it is NOT the 
> leader
>  4) On the server I turn the eth0[192.168.0.101] interface down  [ ifconfig 
> eth0 down ]
>  5) After 26 seconds, application1 says : 
>            :ClientCnxn$SendThread@] - Client session timed out, have not 
> heard from server in 26679ms for sessionid
>             ConnectionStateManager@237] - State change: SUSPENDED
>         NOTE: I'm ignoring the SUSPENDED event
> 6) After 40 seconds, application2 takes leadership
>  7) After 66 seconds, application1 says :
>              ConnectionStateManager@237] - State change: LOST
>          
>          NOTE: Just at this point, I consider that the application1 has lost 
> leadership
> Then, for 26 seconds, we had 2 leaders.
> If you confirm it as a bug, I think I could help.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (CURATOR-460) Timed tolerance for connection suspended leads to simultaneous leaders

[Antonio Rafael Rodrigues (JIRA)]

[ 
https://issues.apache.org/jira/browse/CURATOR-460?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16420490#comment-16420490
 ] 

Antonio Rafael Rodrigues commented on CURATOR-460:
--

[~randgalt] - My goal with the patch was just to add information to the 
discussion, not to merge it.

Moreover, I think that the fact of the application not obeying the given 
expirationPercent is a different issue, therefore I should open a new issue 
reporting that.

Do you agree?

> Timed tolerance for connection suspended leads to simultaneous leaders
> --
>
> Key: CURATOR-460
> URL: https://issues.apache.org/jira/browse/CURATOR-460
> Project: Apache Curator
>  Issue Type: Bug
>  Components: Recipes
>Affects Versions: 3.3.0, 4.0.0, 4.0.1
>Reporter: Antonio Rafael Rodrigues
>Assignee: Cameron McKenzie
>Priority: Major
> Attachments: ConnectionStateManager.patch, 
> Console-leaderSelector1.log, Console-leaderSelector2.log, 
> CuratorLeaderSelectorPOC.java, LogApp1.log, LogApp2.log
>
>
> Starting from Curator 3, after losing the connection to zookeeper, curator 
> emits a SUSPENDED event and then starts an internal timer, if the time of the 
> negotiatied session timeout get hit and curator didn't reconnect to zookeeper 
> it emits a LOST event.
> For example :
>  Given negotiated session timeout = 40
> ||Time (seconds)||Event||
> |0|SUSPENDED (curator has been disconnected from zookeeper)|
> |40|LOST (curator is still disconnected, it must have been lost the sesion as 
> it is disconnected for 40sec)|
> Given this scenario we could ,theoretically, ignore the SUSPENDED event and 
> consider the leadership as lost just if the ConnectionStateListener receives 
> the LOST event.
> But this feature seems to have introduced a bug (from my point of view)
> *Case of failure*
> ||Time (seconds)||Event||
> |0|Something went wrong with the connected zookeeper (in my case, the network 
> inteface of zookeeper's server has gone down). Curator stops hearing heart 
> beats from zookeeper, but doesn't lose the connection (from some reason that 
> I don't know, if the network interface of the server goes down, Curator 
> doesn't lose the connection)|
> |~26.666|SUSPENDED (after 26 seconds without hearing heartbeats, curator 
> emits a SUSPENDED event) 26 is from "readTimeout = sessionTimeout * 2 / 3" 
> from the class ClientCnxn.java from zookeeper client. At this point, curator 
> starts counting 40 sec.|
> |26.666 to 40|During this period, Curator is trying to connect to other 
> zookeeper instances but the other instances, in my example case, are also 
> uncheachable.|
> |40|Session has expired and the other instance has taken leadership ( in my 
> example, the other instance can connect to zookeeper )|
> |66.666|LOST ( after 40sec from SUSPENDED, curator finally sends LOST)|
> As you can see, if we are ignoring the SUSPENDED event, the second 
> application instance acquires the leadership ~26 seconds before the first 
> instance notice that it lost the leadership.
>  I understand it seems to be a very rare case, but either way I think it 
> should be addressed.
> *Reproduce it*
> I have came up with a way to reproduce this easily. I know this is not a 
> recommended production setup, but it reproduces the problem anyway.
> 1) On a virtual machine, with 2 network interfaces (eth0[192.168.0.101], 
> eth1[192.168.0.102]) , I installed one zookeeper instance.
>  2) I setup application1 with the leadership receipe, with 40sec of 
> negotiated timeout, pointing just to 192.168.0.101  .Now it is the leader
>  3) I setup application2 with the leadership receipe, with 40sec of 
> negotiated timeout, pointing just to 192.168.0.102  .Now it it is NOT the 
> leader
>  4) On the server I turn the eth0[192.168.0.101] interface down  [ ifconfig 
> eth0 down ]
>  5) After 26 seconds, application1 says : 
>            :ClientCnxn$SendThread@] - Client session timed out, have not 
> heard from server in 26679ms for sessionid
>             ConnectionStateManager@237] - State change: SUSPENDED
>         NOTE: I'm ignoring the SUSPENDED event
> 6) After 40 seconds, application2 takes leadership
>  7) After 66 seconds, application1 says :
>              ConnectionStateManager@237] - State change: LOST
>          
>          NOTE: Just at this point, I consider that the application1 has lost 
> leadership
> Then, for 26 seconds, we had 2 leaders.
> If you confirm it as a bug, I think I could help.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (CURATOR-460) Timed tolerance for connection suspended leads to simultaneous leaders

[Jordan Zimmerman (JIRA)]

[ 
https://issues.apache.org/jira/browse/CURATOR-460?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16420282#comment-16420282
 ] 

Jordan Zimmerman commented on CURATOR-460:
--

[~arrodrigues] we don't accept patches. See here: 
[https://cwiki.apache.org/confluence/display/CURATOR/Submitting+Pull+Requests]

> Timed tolerance for connection suspended leads to simultaneous leaders
> --
>
> Key: CURATOR-460
> URL: https://issues.apache.org/jira/browse/CURATOR-460
> Project: Apache Curator
>  Issue Type: Bug
>  Components: Recipes
>Affects Versions: 3.3.0, 4.0.0, 4.0.1
>Reporter: Antonio Rafael Rodrigues
>Assignee: Cameron McKenzie
>Priority: Major
> Attachments: ConnectionStateManager.patch, 
> Console-leaderSelector1.log, Console-leaderSelector2.log, 
> CuratorLeaderSelectorPOC.java, LogApp1.log, LogApp2.log
>
>
> Starting from Curator 3, after losing the connection to zookeeper, curator 
> emits a SUSPENDED event and then starts an internal timer, if the time of the 
> negotiatied session timeout get hit and curator didn't reconnect to zookeeper 
> it emits a LOST event.
> For example :
>  Given negotiated session timeout = 40
> ||Time (seconds)||Event||
> |0|SUSPENDED (curator has been disconnected from zookeeper)|
> |40|LOST (curator is still disconnected, it must have been lost the sesion as 
> it is disconnected for 40sec)|
> Given this scenario we could ,theoretically, ignore the SUSPENDED event and 
> consider the leadership as lost just if the ConnectionStateListener receives 
> the LOST event.
> But this feature seems to have introduced a bug (from my point of view)
> *Case of failure*
> ||Time (seconds)||Event||
> |0|Something went wrong with the connected zookeeper (in my case, the network 
> inteface of zookeeper's server has gone down). Curator stops hearing heart 
> beats from zookeeper, but doesn't lose the connection (from some reason that 
> I don't know, if the network interface of the server goes down, Curator 
> doesn't lose the connection)|
> |~26.666|SUSPENDED (after 26 seconds without hearing heartbeats, curator 
> emits a SUSPENDED event) 26 is from "readTimeout = sessionTimeout * 2 / 3" 
> from the class ClientCnxn.java from zookeeper client. At this point, curator 
> starts counting 40 sec.|
> |26.666 to 40|During this period, Curator is trying to connect to other 
> zookeeper instances but the other instances, in my example case, are also 
> uncheachable.|
> |40|Session has expired and the other instance has taken leadership ( in my 
> example, the other instance can connect to zookeeper )|
> |66.666|LOST ( after 40sec from SUSPENDED, curator finally sends LOST)|
> As you can see, if we are ignoring the SUSPENDED event, the second 
> application instance acquires the leadership ~26 seconds before the first 
> instance notice that it lost the leadership.
>  I understand it seems to be a very rare case, but either way I think it 
> should be addressed.
> *Reproduce it*
> I have came up with a way to reproduce this easily. I know this is not a 
> recommended production setup, but it reproduces the problem anyway.
> 1) On a virtual machine, with 2 network interfaces (eth0[192.168.0.101], 
> eth1[192.168.0.102]) , I installed one zookeeper instance.
>  2) I setup application1 with the leadership receipe, with 40sec of 
> negotiated timeout, pointing just to 192.168.0.101  .Now it is the leader
>  3) I setup application2 with the leadership receipe, with 40sec of 
> negotiated timeout, pointing just to 192.168.0.102  .Now it it is NOT the 
> leader
>  4) On the server I turn the eth0[192.168.0.101] interface down  [ ifconfig 
> eth0 down ]
>  5) After 26 seconds, application1 says : 
>            :ClientCnxn$SendThread@] - Client session timed out, have not 
> heard from server in 26679ms for sessionid
>             ConnectionStateManager@237] - State change: SUSPENDED
>         NOTE: I'm ignoring the SUSPENDED event
> 6) After 40 seconds, application2 takes leadership
>  7) After 66 seconds, application1 says :
>              ConnectionStateManager@237] - State change: LOST
>          
>          NOTE: Just at this point, I consider that the application1 has lost 
> leadership
> Then, for 26 seconds, we had 2 leaders.
> If you confirm it as a bug, I think I could help.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (CURATOR-460) Timed tolerance for connection suspended leads to simultaneous leaders

[Antonio Rafael Rodrigues (JIRA)]

[ 
https://issues.apache.org/jira/browse/CURATOR-460?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16420118#comment-16420118
 ] 

Antonio Rafael Rodrigues commented on CURATOR-460:
--

 

I've made a change on the code and it worked as expected. Now it respects the 
{{new StandardConnectionHandlingPolicy(33). }}  I've uploaded a patch.

My test printed:
{code:java}
leaderSelector1 LOST leadership 2018-03-30T03:21:53.676Z
leaderSelector2 HAS  leadership 2018-03-30T03:21:52.077Z{code}
The moment of expiration injection:
{code:java}
2018-03-30 00:21:53,666 [myid:] - WARN  
[Curator-ConnectionStateManager-0:ConnectionStateManager@310] - Session timeout 
has elapsed while SUSPENDED. Injecting a session expiration. Elapsed ms: 13215. 
Adjusted session timeout ms: 13200
{code}
The second instance still gets the leadership almost 2 seconds before the first 
one loses it, but I suspect it's due to ZK's tick time. My Zk has 2sec of 
tickTime

[^ConnectionStateManager.patch]

 

> Timed tolerance for connection suspended leads to simultaneous leaders
> --
>
> Key: CURATOR-460
> URL: https://issues.apache.org/jira/browse/CURATOR-460
> Project: Apache Curator
>  Issue Type: Bug
>  Components: Recipes
>Affects Versions: 3.3.0, 4.0.0, 4.0.1
>Reporter: Antonio Rafael Rodrigues
>Assignee: Cameron McKenzie
>Priority: Major
> Attachments: ConnectionStateManager.patch, 
> Console-leaderSelector1.log, Console-leaderSelector2.log, 
> CuratorLeaderSelectorPOC.java, LogApp1.log, LogApp2.log
>
>
> Starting from Curator 3, after losing the connection to zookeeper, curator 
> emits a SUSPENDED event and then starts an internal timer, if the time of the 
> negotiatied session timeout get hit and curator didn't reconnect to zookeeper 
> it emits a LOST event.
> For example :
>  Given negotiated session timeout = 40
> ||Time (seconds)||Event||
> |0|SUSPENDED (curator has been disconnected from zookeeper)|
> |40|LOST (curator is still disconnected, it must have been lost the sesion as 
> it is disconnected for 40sec)|
> Given this scenario we could ,theoretically, ignore the SUSPENDED event and 
> consider the leadership as lost just if the ConnectionStateListener receives 
> the LOST event.
> But this feature seems to have introduced a bug (from my point of view)
> *Case of failure*
> ||Time (seconds)||Event||
> |0|Something went wrong with the connected zookeeper (in my case, the network 
> inteface of zookeeper's server has gone down). Curator stops hearing heart 
> beats from zookeeper, but doesn't lose the connection (from some reason that 
> I don't know, if the network interface of the server goes down, Curator 
> doesn't lose the connection)|
> |~26.666|SUSPENDED (after 26 seconds without hearing heartbeats, curator 
> emits a SUSPENDED event) 26 is from "readTimeout = sessionTimeout * 2 / 3" 
> from the class ClientCnxn.java from zookeeper client. At this point, curator 
> starts counting 40 sec.|
> |26.666 to 40|During this period, Curator is trying to connect to other 
> zookeeper instances but the other instances, in my example case, are also 
> uncheachable.|
> |40|Session has expired and the other instance has taken leadership ( in my 
> example, the other instance can connect to zookeeper )|
> |66.666|LOST ( after 40sec from SUSPENDED, curator finally sends LOST)|
> As you can see, if we are ignoring the SUSPENDED event, the second 
> application instance acquires the leadership ~26 seconds before the first 
> instance notice that it lost the leadership.
>  I understand it seems to be a very rare case, but either way I think it 
> should be addressed.
> *Reproduce it*
> I have came up with a way to reproduce this easily. I know this is not a 
> recommended production setup, but it reproduces the problem anyway.
> 1) On a virtual machine, with 2 network interfaces (eth0[192.168.0.101], 
> eth1[192.168.0.102]) , I installed one zookeeper instance.
>  2) I setup application1 with the leadership receipe, with 40sec of 
> negotiated timeout, pointing just to 192.168.0.101  .Now it is the leader
>  3) I setup application2 with the leadership receipe, with 40sec of 
> negotiated timeout, pointing just to 192.168.0.102  .Now it it is NOT the 
> leader
>  4) On the server I turn the eth0[192.168.0.101] interface down  [ ifconfig 
> eth0 down ]
>  5) After 26 seconds, application1 says : 
>            :ClientCnxn$SendThread@] - Client session timed out, have not 
> heard from server in 26679ms for sessionid
>             ConnectionStateManager@237] - State change: SUSPENDED
>         NOTE: I'm ignoring the SUSPENDED event
> 6) After 40 seconds, application2 takes leadership
>  7) After 66 seconds, application1 says :
>              ConnectionStateManager@237] - State change: 

[jira] [Commented] (CURATOR-460) Timed tolerance for connection suspended leads to simultaneous leaders

[Antonio Rafael Rodrigues (JIRA)]

[ 
https://issues.apache.org/jira/browse/CURATOR-460?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16420045#comment-16420045
 ] 

Antonio Rafael Rodrigues commented on CURATOR-460:
--

[~randgalt] - I did the test and I haven't gotten the expected results. I'm 
uploading the log.

It seems that even if I use {{.connectionHandlingPolicy(new 
StandardConnectionHandlingPolicy(33))}} Curator still waits 40secs after 
SUSPENDED.

See:
{code:java}
Session timeout has elapsed while SUSPENDED. Injecting a session expiration. 
Elapsed ms: 40010. Adjusted session timeout ms: 13200{code}
[^LogApp1.log]

[^LogApp2.log]

 

[~cammckenzie] - I'll try to build the tests, but I don't know if I'll have 
enough time. But anyway you don't need a ZKcluster for the test, just one 
instance in a virtual machine with 2 network interfaces.

 

> Timed tolerance for connection suspended leads to simultaneous leaders
> --
>
> Key: CURATOR-460
> URL: https://issues.apache.org/jira/browse/CURATOR-460
> Project: Apache Curator
>  Issue Type: Bug
>  Components: Recipes
>Affects Versions: 3.3.0, 4.0.0, 4.0.1
>Reporter: Antonio Rafael Rodrigues
>Assignee: Cameron McKenzie
>Priority: Major
> Attachments: Console-leaderSelector1.log, 
> Console-leaderSelector2.log, CuratorLeaderSelectorPOC.java, LogApp1.log, 
> LogApp2.log
>
>
> Starting from Curator 3, after losing the connection to zookeeper, curator 
> emits a SUSPENDED event and then starts an internal timer, if the time of the 
> negotiatied session timeout get hit and curator didn't reconnect to zookeeper 
> it emits a LOST event.
> For example :
>  Given negotiated session timeout = 40
> ||Time (seconds)||Event||
> |0|SUSPENDED (curator has been disconnected from zookeeper)|
> |40|LOST (curator is still disconnected, it must have been lost the sesion as 
> it is disconnected for 40sec)|
> Given this scenario we could ,theoretically, ignore the SUSPENDED event and 
> consider the leadership as lost just if the ConnectionStateListener receives 
> the LOST event.
> But this feature seems to have introduced a bug (from my point of view)
> *Case of failure*
> ||Time (seconds)||Event||
> |0|Something went wrong with the connected zookeeper (in my case, the network 
> inteface of zookeeper's server has gone down). Curator stops hearing heart 
> beats from zookeeper, but doesn't lose the connection (from some reason that 
> I don't know, if the network interface of the server goes down, Curator 
> doesn't lose the connection)|
> |~26.666|SUSPENDED (after 26 seconds without hearing heartbeats, curator 
> emits a SUSPENDED event) 26 is from "readTimeout = sessionTimeout * 2 / 3" 
> from the class ClientCnxn.java from zookeeper client. At this point, curator 
> starts counting 40 sec.|
> |26.666 to 40|During this period, Curator is trying to connect to other 
> zookeeper instances but the other instances, in my example case, are also 
> uncheachable.|
> |40|Session has expired and the other instance has taken leadership ( in my 
> example, the other instance can connect to zookeeper )|
> |66.666|LOST ( after 40sec from SUSPENDED, curator finally sends LOST)|
> As you can see, if we are ignoring the SUSPENDED event, the second 
> application instance acquires the leadership ~26 seconds before the first 
> instance notice that it lost the leadership.
>  I understand it seems to be a very rare case, but either way I think it 
> should be addressed.
> *Reproduce it*
> I have came up with a way to reproduce this easily. I know this is not a 
> recommended production setup, but it reproduces the problem anyway.
> 1) On a virtual machine, with 2 network interfaces (eth0[192.168.0.101], 
> eth1[192.168.0.102]) , I installed one zookeeper instance.
>  2) I setup application1 with the leadership receipe, with 40sec of 
> negotiated timeout, pointing just to 192.168.0.101  .Now it is the leader
>  3) I setup application2 with the leadership receipe, with 40sec of 
> negotiated timeout, pointing just to 192.168.0.102  .Now it it is NOT the 
> leader
>  4) On the server I turn the eth0[192.168.0.101] interface down  [ ifconfig 
> eth0 down ]
>  5) After 26 seconds, application1 says : 
>            :ClientCnxn$SendThread@] - Client session timed out, have not 
> heard from server in 26679ms for sessionid
>             ConnectionStateManager@237] - State change: SUSPENDED
>         NOTE: I'm ignoring the SUSPENDED event
> 6) After 40 seconds, application2 takes leadership
>  7) After 66 seconds, application1 says :
>              ConnectionStateManager@237] - State change: LOST
>          
>          NOTE: Just at this point, I consider that the application1 has lost 
> leadership
> Then, for 26 seconds, we had 2 leaders.
> If you confirm it as a bug, I 

[jira] [Commented] (CURATOR-460) Timed tolerance for connection suspended leads to simultaneous leaders

[Jordan Zimmerman (JIRA)]

[ 
https://issues.apache.org/jira/browse/CURATOR-460?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16418645#comment-16418645
 ] 

Jordan Zimmerman commented on CURATOR-460:
--

[~arrodrigues] - can you alter your test by adding this line to the 
CuratorFrameworkFactory:

{{.connectionHandlingPolicy(new StandardConnectionHandlingPolicy(33))}}

This should cause Curator to inject the session expiration at 1/3 of a session 
and this would conform to the case where the ZooKeeper server drops the 
connection at a missed heartbeat (2/3 of a session) and then removes the 
session when the full session timeout has elapsed.

> Timed tolerance for connection suspended leads to simultaneous leaders
> --
>
> Key: CURATOR-460
> URL: https://issues.apache.org/jira/browse/CURATOR-460
> Project: Apache Curator
>  Issue Type: Bug
>  Components: Recipes
>Affects Versions: 3.3.0, 4.0.0, 4.0.1
>Reporter: Antonio Rafael Rodrigues
>Assignee: Cameron McKenzie
>Priority: Major
> Attachments: Console-leaderSelector1.log, 
> Console-leaderSelector2.log, CuratorLeaderSelectorPOC.java
>
>
> Starting from Curator 3, after losing the connection to zookeeper, curator 
> emits a SUSPENDED event and then starts an internal timer, if the time of the 
> negotiatied session timeout get hit and curator didn't reconnect to zookeeper 
> it emits a LOST event.
> For example :
>  Given negotiated session timeout = 40
> ||Time (seconds)||Event||
> |0|SUSPENDED (curator has been disconnected from zookeeper)|
> |40|LOST (curator is still disconnected, it must have been lost the sesion as 
> it is disconnected for 40sec)|
> Given this scenario we could ,theoretically, ignore the SUSPENDED event and 
> consider the leadership as lost just if the ConnectionStateListener receives 
> the LOST event.
> But this feature seems to have introduced a bug (from my point of view)
> *Case of failure*
> ||Time (seconds)||Event||
> |0|Something went wrong with the connected zookeeper (in my case, the network 
> inteface of zookeeper's server has gone down). Curator stops hearing heart 
> beats from zookeeper, but doesn't lose the connection (from some reason that 
> I don't know, if the network interface of the server goes down, Curator 
> doesn't lose the connection)|
> |~26.666|SUSPENDED (after 26 seconds without hearing heartbeats, curator 
> emits a SUSPENDED event) 26 is from "readTimeout = sessionTimeout * 2 / 3" 
> from the class ClientCnxn.java from zookeeper client. At this point, curator 
> starts counting 40 sec.|
> |26.666 to 40|During this period, Curator is trying to connect to other 
> zookeeper instances but the other instances, in my example case, are also 
> uncheachable.|
> |40|Session has expired and the other instance has taken leadership ( in my 
> example, the other instance can connect to zookeeper )|
> |66.666|LOST ( after 40sec from SUSPENDED, curator finally sends LOST)|
> As you can see, if we are ignoring the SUSPENDED event, the second 
> application instance acquires the leadership ~26 seconds before the first 
> instance notice that it lost the leadership.
>  I understand it seems to be a very rare case, but either way I think it 
> should be addressed.
> *Reproduce it*
> I have came up with a way to reproduce this easily. I know this is not a 
> recommended production setup, but it reproduces the problem anyway.
> 1) On a virtual machine, with 2 network interfaces (eth0[192.168.0.101], 
> eth1[192.168.0.102]) , I installed one zookeeper instance.
>  2) I setup application1 with the leadership receipe, with 40sec of 
> negotiated timeout, pointing just to 192.168.0.101  .Now it is the leader
>  3) I setup application2 with the leadership receipe, with 40sec of 
> negotiated timeout, pointing just to 192.168.0.102  .Now it it is NOT the 
> leader
>  4) On the server I turn the eth0[192.168.0.101] interface down  [ ifconfig 
> eth0 down ]
>  5) After 26 seconds, application1 says : 
>            :ClientCnxn$SendThread@] - Client session timed out, have not 
> heard from server in 26679ms for sessionid
>             ConnectionStateManager@237] - State change: SUSPENDED
>         NOTE: I'm ignoring the SUSPENDED event
> 6) After 40 seconds, application2 takes leadership
>  7) After 66 seconds, application1 says :
>              ConnectionStateManager@237] - State change: LOST
>          
>          NOTE: Just at this point, I consider that the application1 has lost 
> leadership
> Then, for 26 seconds, we had 2 leaders.
> If you confirm it as a bug, I think I could help.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (CURATOR-460) Timed tolerance for connection suspended leads to simultaneous leaders

[Jordan Zimmerman (JIRA)]

[ 
https://issues.apache.org/jira/browse/CURATOR-460?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16418621#comment-16418621
 ] 

Jordan Zimmerman commented on CURATOR-460:
--

It's worth pointing out that Curator has always recommended that users treat 
SUSPENDED as a complete connection loss. E.g. from the Curator website 
[http://curator.apache.org/errors.html]

 
|SUSPENDED|There has been a loss of connection. Leaders, locks, etc. should 
suspend until the connection is re-established.|

 

> Timed tolerance for connection suspended leads to simultaneous leaders
> --
>
> Key: CURATOR-460
> URL: https://issues.apache.org/jira/browse/CURATOR-460
> Project: Apache Curator
>  Issue Type: Bug
>  Components: Recipes
>Affects Versions: 3.3.0, 4.0.0, 4.0.1
>Reporter: Antonio Rafael Rodrigues
>Assignee: Cameron McKenzie
>Priority: Major
> Attachments: Console-leaderSelector1.log, 
> Console-leaderSelector2.log, CuratorLeaderSelectorPOC.java
>
>
> Starting from Curator 3, after losing the connection to zookeeper, curator 
> emits a SUSPENDED event and then starts an internal timer, if the time of the 
> negotiatied session timeout get hit and curator didn't reconnect to zookeeper 
> it emits a LOST event.
> For example :
>  Given negotiated session timeout = 40
> ||Time (seconds)||Event||
> |0|SUSPENDED (curator has been disconnected from zookeeper)|
> |40|LOST (curator is still disconnected, it must have been lost the sesion as 
> it is disconnected for 40sec)|
> Given this scenario we could ,theoretically, ignore the SUSPENDED event and 
> consider the leadership as lost just if the ConnectionStateListener receives 
> the LOST event.
> But this feature seems to have introduced a bug (from my point of view)
> *Case of failure*
> ||Time (seconds)||Event||
> |0|Something went wrong with the connected zookeeper (in my case, the network 
> inteface of zookeeper's server has gone down). Curator stops hearing heart 
> beats from zookeeper, but doesn't lose the connection (from some reason that 
> I don't know, if the network interface of the server goes down, Curator 
> doesn't lose the connection)|
> |~26.666|SUSPENDED (after 26 seconds without hearing heartbeats, curator 
> emits a SUSPENDED event) 26 is from "readTimeout = sessionTimeout * 2 / 3" 
> from the class ClientCnxn.java from zookeeper client. At this point, curator 
> starts counting 40 sec.|
> |26.666 to 40|During this period, Curator is trying to connect to other 
> zookeeper instances but the other instances, in my example case, are also 
> uncheachable.|
> |40|Session has expired and the other instance has taken leadership ( in my 
> example, the other instance can connect to zookeeper )|
> |66.666|LOST ( after 40sec from SUSPENDED, curator finally sends LOST)|
> As you can see, if we are ignoring the SUSPENDED event, the second 
> application instance acquires the leadership ~26 seconds before the first 
> instance notice that it lost the leadership.
>  I understand it seems to be a very rare case, but either way I think it 
> should be addressed.
> *Reproduce it*
> I have came up with a way to reproduce this easily. I know this is not a 
> recommended production setup, but it reproduces the problem anyway.
> 1) On a virtual machine, with 2 network interfaces (eth0[192.168.0.101], 
> eth1[192.168.0.102]) , I installed one zookeeper instance.
>  2) I setup application1 with the leadership receipe, with 40sec of 
> negotiated timeout, pointing just to 192.168.0.101  .Now it is the leader
>  3) I setup application2 with the leadership receipe, with 40sec of 
> negotiated timeout, pointing just to 192.168.0.102  .Now it it is NOT the 
> leader
>  4) On the server I turn the eth0[192.168.0.101] interface down  [ ifconfig 
> eth0 down ]
>  5) After 26 seconds, application1 says : 
>            :ClientCnxn$SendThread@] - Client session timed out, have not 
> heard from server in 26679ms for sessionid
>             ConnectionStateManager@237] - State change: SUSPENDED
>         NOTE: I'm ignoring the SUSPENDED event
> 6) After 40 seconds, application2 takes leadership
>  7) After 66 seconds, application1 says :
>              ConnectionStateManager@237] - State change: LOST
>          
>          NOTE: Just at this point, I consider that the application1 has lost 
> leadership
> Then, for 26 seconds, we had 2 leaders.
> If you confirm it as a bug, I think I could help.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (CURATOR-460) Timed tolerance for connection suspended leads to simultaneous leaders

[Cameron McKenzie (JIRA)]

[ 
https://issues.apache.org/jira/browse/CURATOR-460?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16418383#comment-16418383
 ] 

Cameron McKenzie commented on CURATOR-460:
--

Thanks [~arrodrigues], do you think it would be possible to reproduce the 
problem in the form of a Curator unit test?

I don't have access to a full ZK cluster at the moment, so it will be difficult 
for me to run your tests.

> Timed tolerance for connection suspended leads to simultaneous leaders
> --
>
> Key: CURATOR-460
> URL: https://issues.apache.org/jira/browse/CURATOR-460
> Project: Apache Curator
>  Issue Type: Bug
>  Components: Recipes
>Affects Versions: 3.3.0, 4.0.0, 4.0.1
>Reporter: Antonio Rafael Rodrigues
>Assignee: Cameron McKenzie
>Priority: Major
> Attachments: Console-leaderSelector1.log, 
> Console-leaderSelector2.log, CuratorLeaderSelectorPOC.java
>
>
> Starting from Curator 3, after losing the connection to zookeeper, curator 
> emits a SUSPENDED event and then starts an internal timer, if the time of the 
> negotiatied session timeout get hit and curator didn't reconnect to zookeeper 
> it emits a LOST event.
> For example :
>  Given negotiated session timeout = 40
> ||Time (seconds)||Event||
> |0|SUSPENDED (curator has been disconnected from zookeeper)|
> |40|LOST (curator is still disconnected, it must have been lost the sesion as 
> it is disconnected for 40sec)|
> Given this scenario we could ,theoretically, ignore the SUSPENDED event and 
> consider the leadership as lost just if the ConnectionStateListener receives 
> the LOST event.
> But this feature seems to have introduced a bug (from my point of view)
> *Case of failure*
> ||Time (seconds)||Event||
> |0|Something went wrong with the connected zookeeper (in my case, the network 
> inteface of zookeeper's server has gone down). Curator stops hearing heart 
> beats from zookeeper, but doesn't lose the connection (from some reason that 
> I don't know, if the network interface of the server goes down, Curator 
> doesn't lose the connection)|
> |~26.666|SUSPENDED (after 26 seconds without hearing heartbeats, curator 
> emits a SUSPENDED event) 26 is from "readTimeout = sessionTimeout * 2 / 3" 
> from the class ClientCnxn.java from zookeeper client. At this point, curator 
> starts counting 40 sec.|
> |26.666 to 40|During this period, Curator is trying to connect to other 
> zookeeper instances but the other instances, in my example case, are also 
> uncheachable.|
> |40|Session has expired and the other instance has taken leadership ( in my 
> example, the other instance can connect to zookeeper )|
> |66.666|LOST ( after 40sec from SUSPENDED, curator finally sends LOST)|
> As you can see, if we are ignoring the SUSPENDED event, the second 
> application instance acquires the leadership ~26 seconds before the first 
> instance notice that it lost the leadership.
>  I understand it seems to be a very rare case, but either way I think it 
> should be addressed.
> *Reproduce it*
> I have came up with a way to reproduce this easily. I know this is not a 
> recommended production setup, but it reproduces the problem anyway.
> 1) On a virtual machine, with 2 network interfaces (eth0[192.168.0.101], 
> eth1[192.168.0.102]) , I installed one zookeeper instance.
>  2) I setup application1 with the leadership receipe, with 40sec of 
> negotiated timeout, pointing just to 192.168.0.101  .Now it is the leader
>  3) I setup application2 with the leadership receipe, with 40sec of 
> negotiated timeout, pointing just to 192.168.0.102  .Now it it is NOT the 
> leader
>  4) On the server I turn the eth0[192.168.0.101] interface down  [ ifconfig 
> eth0 down ]
>  5) After 26 seconds, application1 says : 
>            :ClientCnxn$SendThread@] - Client session timed out, have not 
> heard from server in 26679ms for sessionid
>             ConnectionStateManager@237] - State change: SUSPENDED
>         NOTE: I'm ignoring the SUSPENDED event
> 6) After 40 seconds, application2 takes leadership
>  7) After 66 seconds, application1 says :
>              ConnectionStateManager@237] - State change: LOST
>          
>          NOTE: Just at this point, I consider that the application1 has lost 
> leadership
> Then, for 26 seconds, we had 2 leaders.
> If you confirm it as a bug, I think I could help.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (CURATOR-460) Timed tolerance for connection suspended leads to simultaneous leaders

[Antonio Rafael Rodrigues (JIRA)]

[ 
https://issues.apache.org/jira/browse/CURATOR-460?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16418368#comment-16418368
 ] 

Antonio Rafael Rodrigues commented on CURATOR-460:
--

I'm using the Leader Election recipe. In my previous tests I was  implementing 
ConnectionStateListener by myself and ignoring the SUSPENDED event, but to be 
more in line with the 3.0 way I've built another test using 
SessionConnectionStateErrorPolicy and I got results very close to the previous 
ones.

I've uploaded a class CuratorLeaderSelectorPOC that has two tests 
(leaderSelector1,leaderSelector2), one to start each application's instance.

I did the following:
1. run test leaderSelector1
wait for the Message "leadeSelector1 HAS leadership" on the console

2. run test leaderSelector2
wait for the start

3. On the VM server running zookeeper, I executed:
{noformat}
ifconfig eth0 down{noformat}
NOTE: eth0 => 192.168.0.104

On the console of loaderSelector1 I got:
{code:java}
...
Started leadeSelector1
..
leadeSelector1 HAS leadership 2018-03-29T02:56:06.402Z
...
leadeSelector1 LOST leadership 2018-03-29T02:57:26.508Z{code}

On the console of loaderSelector2 I got:
{code:java}
...
Started leadeSelector2
...
leadeSelector2 HAS leadership 2018-03-29T02:56:57.445Z{code}

I tried to pass a value to new StandardConnectionHandlingPolicy(), like 30, but 
it didn't make effect, I still don't know why.

Attachments:

[^CuratorLeaderSelectorPOC.java]

[^Console-leaderSelector1.log]

[^Console-leaderSelector2.log]

> Timed tolerance for connection suspended leads to simultaneous leaders
> --
>
> Key: CURATOR-460
> URL: https://issues.apache.org/jira/browse/CURATOR-460
> Project: Apache Curator
>  Issue Type: Bug
>  Components: Recipes
>Affects Versions: 3.3.0, 4.0.0, 4.0.1
>Reporter: Antonio Rafael Rodrigues
>Assignee: Cameron McKenzie
>Priority: Major
> Attachments: Console-leaderSelector1.log, 
> Console-leaderSelector2.log, CuratorLeaderSelectorPOC.java
>
>
> Starting from Curator 3, after losing the connection to zookeeper, curator 
> emits a SUSPENDED event and then starts an internal timer, if the time of the 
> negotiatied session timeout get hit and curator didn't reconnect to zookeeper 
> it emits a LOST event.
> For example :
>  Given negotiated session timeout = 40
> ||Time (seconds)||Event||
> |0|SUSPENDED (curator has been disconnected from zookeeper)|
> |40|LOST (curator is still disconnected, it must have been lost the sesion as 
> it is disconnected for 40sec)|
> Given this scenario we could ,theoretically, ignore the SUSPENDED event and 
> consider the leadership as lost just if the ConnectionStateListener receives 
> the LOST event.
> But this feature seems to have introduced a bug (from my point of view)
> *Case of failure*
> ||Time (seconds)||Event||
> |0|Something went wrong with the connected zookeeper (in my case, the network 
> inteface of zookeeper's server has gone down). Curator stops hearing heart 
> beats from zookeeper, but doesn't lose the connection (from some reason that 
> I don't know, if the network interface of the server goes down, Curator 
> doesn't lose the connection)|
> |~26.666|SUSPENDED (after 26 seconds without hearing heartbeats, curator 
> emits a SUSPENDED event) 26 is from "readTimeout = sessionTimeout * 2 / 3" 
> from the class ClientCnxn.java from zookeeper client. At this point, curator 
> starts counting 40 sec.|
> |26.666 to 40|During this period, Curator is trying to connect to other 
> zookeeper instances but the other instances, in my example case, are also 
> uncheachable.|
> |40|Session has expired and the other instance has taken leadership ( in my 
> example, the other instance can connect to zookeeper )|
> |66.666|LOST ( after 40sec from SUSPENDED, curator finally sends LOST)|
> As you can see, if we are ignoring the SUSPENDED event, the second 
> application instance acquires the leadership ~26 seconds before the first 
> instance notice that it lost the leadership.
>  I understand it seems to be a very rare case, but either way I think it 
> should be addressed.
> *Reproduce it*
> I have came up with a way to reproduce this easily. I know this is not a 
> recommended production setup, but it reproduces the problem anyway.
> 1) On a virtual machine, with 2 network interfaces (eth0[192.168.0.101], 
> eth1[192.168.0.102]) , I installed one zookeeper instance.
>  2) I setup application1 with the leadership receipe, with 40sec of 
> negotiated timeout, pointing just to 192.168.0.101  .Now it is the leader
>  3) I setup application2 with the leadership receipe, with 40sec of 
> negotiated timeout, pointing just to 192.168.0.102  .Now it it is NOT the 
> leader
>  4) On the server I turn the eth0[192.168.0.101] interface down  

[jira] [Commented] (CURATOR-460) Timed tolerance for connection suspended leads to simultaneous leaders

[Cameron McKenzie (JIRA)]

[ 
https://issues.apache.org/jira/browse/CURATOR-460?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16416601#comment-16416601
 ] 

Cameron McKenzie commented on CURATOR-460:
--

I'm referring to timeAtSuspendedEvent as the time that Curator considers itself 
suspended, as we can't access this information from ZK (as you've pointed out).

I think that the default handling for Curator should be changed to make the 
LOST event occur at the negotiated session timeout number of seconds since last 
ping / interaction with the server, as this is how it's calculated on the 
Zookeeper server side.

While we can work around it with configuration etc. I think that the default 
should be correct. I don't see any reason why the current behaviour should be 
preserved because I think it's incorrect.

Which leader recipe were you using for your tests? The 
StandardConnectionStateErrorPolicy treats both SUSPENDED and LOST as the same 
thing. So, you must have been using either a customer error policy or the 
SessionConnectionStateErrorPolicy?

> Timed tolerance for connection suspended leads to simultaneous leaders
> --
>
> Key: CURATOR-460
> URL: https://issues.apache.org/jira/browse/CURATOR-460
> Project: Apache Curator
>  Issue Type: Bug
>  Components: Recipes
>Affects Versions: 3.3.0, 4.0.0, 4.0.1
>Reporter: Antonio Rafael Rodrigues
>Assignee: Cameron McKenzie
>Priority: Major
>
> Starting from Curator 3, after losing the connection to zookeeper, curator 
> emits a SUSPENDED event and then starts an internal timer, if the time of the 
> negotiatied session timeout get hit and curator didn't reconnect to zookeeper 
> it emits a LOST event.
> For example :
>  Given negotiated session timeout = 40
> ||Time (seconds)||Event||
> |0|SUSPENDED (curator has been disconnected from zookeeper)|
> |40|LOST (curator is still disconnected, it must have been lost the sesion as 
> it is disconnected for 40sec)|
> Given this scenario we could ,theoretically, ignore the SUSPENDED event and 
> consider the leadership as lost just if the ConnectionStateListener receives 
> the LOST event.
> But this feature seems to have introduced a bug (from my point of view)
> *Case of failure*
> ||Time (seconds)||Event||
> |0|Something went wrong with the connected zookeeper (in my case, the network 
> inteface of zookeeper's server has gone down). Curator stops hearing heart 
> beats from zookeeper, but doesn't lose the connection (from some reason that 
> I don't know, if the network interface of the server goes down, Curator 
> doesn't lose the connection)|
> |~26.666|SUSPENDED (after 26 seconds without hearing heartbeats, curator 
> emits a SUSPENDED event) 26 is from "readTimeout = sessionTimeout * 2 / 3" 
> from the class ClientCnxn.java from zookeeper client. At this point, curator 
> starts counting 40 sec.|
> |26.666 to 40|During this period, Curator is trying to connect to other 
> zookeeper instances but the other instances, in my example case, are also 
> uncheachable.|
> |40|Session has expired and the other instance has taken leadership ( in my 
> example, the other instance can connect to zookeeper )|
> |66.666|LOST ( after 40sec from SUSPENDED, curator finally sends LOST)|
> As you can see, if we are ignoring the SUSPENDED event, the second 
> application instance acquires the leadership ~26 seconds before the first 
> instance notice that it lost the leadership.
>  I understand it seems to be a very rare case, but either way I think it 
> should be addressed.
> *Reproduce it*
> I have came up with a way to reproduce this easily. I know this is not a 
> recommended production setup, but it reproduces the problem anyway.
> 1) On a virtual machine, with 2 network interfaces (eth0[192.168.0.101], 
> eth1[192.168.0.102]) , I installed one zookeeper instance.
>  2) I setup application1 with the leadership receipe, with 40sec of 
> negotiated timeout, pointing just to 192.168.0.101  .Now it is the leader
>  3) I setup application2 with the leadership receipe, with 40sec of 
> negotiated timeout, pointing just to 192.168.0.102  .Now it it is NOT the 
> leader
>  4) On the server I turn the eth0[192.168.0.101] interface down  [ ifconfig 
> eth0 down ]
>  5) After 26 seconds, application1 says : 
>            :ClientCnxn$SendThread@] - Client session timed out, have not 
> heard from server in 26679ms for sessionid
>             ConnectionStateManager@237] - State change: SUSPENDED
>         NOTE: I'm ignoring the SUSPENDED event
> 6) After 40 seconds, application2 takes leadership
>  7) After 66 seconds, application1 says :
>              ConnectionStateManager@237] - State change: LOST
>          
>          NOTE: Just at this point, I consider that the application1 has lost 
> leadership
> Then, for 26 

[jira] [Commented] (CURATOR-460) Timed tolerance for connection suspended leads to simultaneous leaders

[Antonio Rafael Rodrigues (JIRA)]

[ 
https://issues.apache.org/jira/browse/CURATOR-460?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16416588#comment-16416588
 ] 

Antonio Rafael Rodrigues commented on CURATOR-460:
--

1) I think the timeAtSuspendedEvent you are talking about is the one hold by 
clientCnxnSocket#getIdleRecv() in the class ClientCnxn.java of ZK code, that 
call returns the time of last interaction with the server, but as far as I 
understand the code, we don't have access to the clientCnxnSocket from Curator. 
So we don't know for sure the elapsed time.

Even if Curator had access to getIdleRecv() I don't think if would be very 
useful, as after Zookeeper realizes that it has passed 2/3 of the 
sessionTimeout without heard from server, it enqueues the error and resets the 
time counter, by calling clientCnxnSocket.updateLastSendAndHeard()

As you said, there is also the possibility of start the Curator's timer already 
in 2/3 of sessionTimeout. By the way, it already can be achieved by using 
SessionConnectionStateErrorPolicy and setting 
StandardConnectionHandlingPolicy.setSimulatedSessionExpirationPercent(*35*)

2)There is already a way to treat SUSPENDED or LOST as the same thing, just use 
StandardConnectionStateErrorPolicy. 

To sum up:

1) I can be wrong, but looking at the code, there is no way to know the real 
elapsed time without changing zookeeper ( I hope someone check that )

2) We can start the counter already in 2/3 of session timeout, even though 
there is already a way to do it by configuration.

 

> Timed tolerance for connection suspended leads to simultaneous leaders
> --
>
> Key: CURATOR-460
> URL: https://issues.apache.org/jira/browse/CURATOR-460
> Project: Apache Curator
>  Issue Type: Bug
>  Components: Recipes
>Affects Versions: 3.3.0, 4.0.0, 4.0.1
>Reporter: Antonio Rafael Rodrigues
>Assignee: Cameron McKenzie
>Priority: Major
>
> Starting from Curator 3, after losing the connection to zookeeper, curator 
> emits a SUSPENDED event and then starts an internal timer, if the time of the 
> negotiatied session timeout get hit and curator didn't reconnect to zookeeper 
> it emits a LOST event.
> For example :
>  Given negotiated session timeout = 40
> ||Time (seconds)||Event||
> |0|SUSPENDED (curator has been disconnected from zookeeper)|
> |40|LOST (curator is still disconnected, it must have been lost the sesion as 
> it is disconnected for 40sec)|
> Given this scenario we could ,theoretically, ignore the SUSPENDED event and 
> consider the leadership as lost just if the ConnectionStateListener receives 
> the LOST event.
> But this feature seems to have introduced a bug (from my point of view)
> *Case of failure*
> ||Time (seconds)||Event||
> |0|Something went wrong with the connected zookeeper (in my case, the network 
> inteface of zookeeper's server has gone down). Curator stops hearing heart 
> beats from zookeeper, but doesn't lose the connection (from some reason that 
> I don't know, if the network interface of the server goes down, Curator 
> doesn't lose the connection)|
> |~26.666|SUSPENDED (after 26 seconds without hearing heartbeats, curator 
> emits a SUSPENDED event) 26 is from "readTimeout = sessionTimeout * 2 / 3" 
> from the class ClientCnxn.java from zookeeper client. At this point, curator 
> starts counting 40 sec.|
> |26.666 to 40|During this period, Curator is trying to connect to other 
> zookeeper instances but the other instances, in my example case, are also 
> uncheachable.|
> |40|Session has expired and the other instance has taken leadership ( in my 
> example, the other instance can connect to zookeeper )|
> |66.666|LOST ( after 40sec from SUSPENDED, curator finally sends LOST)|
> As you can see, if we are ignoring the SUSPENDED event, the second 
> application instance acquires the leadership ~26 seconds before the first 
> instance notice that it lost the leadership.
>  I understand it seems to be a very rare case, but either way I think it 
> should be addressed.
> *Reproduce it*
> I have came up with a way to reproduce this easily. I know this is not a 
> recommended production setup, but it reproduces the problem anyway.
> 1) On a virtual machine, with 2 network interfaces (eth0[192.168.0.101], 
> eth1[192.168.0.102]) , I installed one zookeeper instance.
>  2) I setup application1 with the leadership receipe, with 40sec of 
> negotiated timeout, pointing just to 192.168.0.101  .Now it is the leader
>  3) I setup application2 with the leadership receipe, with 40sec of 
> negotiated timeout, pointing just to 192.168.0.102  .Now it it is NOT the 
> leader
>  4) On the server I turn the eth0[192.168.0.101] interface down  [ ifconfig 
> eth0 down ]
>  5) After 26 seconds, application1 says : 
>            

[jira] [Commented] (CURATOR-460) Timed tolerance for connection suspended leads to simultaneous leaders

[Cameron McKenzie (JIRA)]

[ 
https://issues.apache.org/jira/browse/CURATOR-460?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16416362#comment-16416362
 ] 

Cameron McKenzie commented on CURATOR-460:
--

My reading of the ZK code is that the session expiration works by maintaining a 
collection of sessions. Periodically this collection gets polled for all of the 
sessions that have not been accessed for > session timeout and these are 
expired. Whenever there's interaction with the session the timestamp of last 
interaction gets updated within this collection, thus keeping it alive.

So, the session expiration count really begins as soon as the last command has 
been received by ZK, rather than at a missed heartbeat. So, I think that 
Curator should inject the LOST event at:

timeAtSuspendedEvent (which should be 2/3 of session timeout) + 1/3 of session 
timeout

I'm not familiar with the ZK codebase though, so I could be reading things 
incorrectly. Certainly the logs provided by [~arrodrigues] seem to indicate 
this is how it works.

Given that Curator is inferring ZK state while disconnected though, there is 
always going to be a window of error. Perhaps the leadership recipes should 
have the option of treating SUSPENDED and LOST in the same fashion for mission 
critical stuff where Curator's inference is not acceptable?

> Timed tolerance for connection suspended leads to simultaneous leaders
> --
>
> Key: CURATOR-460
> URL: https://issues.apache.org/jira/browse/CURATOR-460
> Project: Apache Curator
>  Issue Type: Bug
>  Components: Recipes
>Affects Versions: 3.3.0, 4.0.0, 4.0.1
>Reporter: Antonio Rafael Rodrigues
>Assignee: Cameron McKenzie
>Priority: Major
>
> Starting from Curator 3, after losing the connection to zookeeper, curator 
> emits a SUSPENDED event and then starts an internal timer, if the time of the 
> negotiatied session timeout get hit and curator didn't reconnect to zookeeper 
> it emits a LOST event.
> For example :
>  Given negotiated session timeout = 40
> ||Time (seconds)||Event||
> |0|SUSPENDED (curator has been disconnected from zookeeper)|
> |40|LOST (curator is still disconnected, it must have been lost the sesion as 
> it is disconnected for 40sec)|
> Given this scenario we could ,theoretically, ignore the SUSPENDED event and 
> consider the leadership as lost just if the ConnectionStateListener receives 
> the LOST event.
> But this feature seems to have introduced a bug (from my point of view)
> *Case of failure*
> ||Time (seconds)||Event||
> |0|Something went wrong with the connected zookeeper (in my case, the network 
> inteface of zookeeper's server has gone down). Curator stops hearing heart 
> beats from zookeeper, but doesn't lose the connection (from some reason that 
> I don't know, if the network interface of the server goes down, Curator 
> doesn't lose the connection)|
> |~26.666|SUSPENDED (after 26 seconds without hearing heartbeats, curator 
> emits a SUSPENDED event) 26 is from "readTimeout = sessionTimeout * 2 / 3" 
> from the class ClientCnxn.java from zookeeper client. At this point, curator 
> starts counting 40 sec.|
> |26.666 to 40|During this period, Curator is trying to connect to other 
> zookeeper instances but the other instances, in my example case, are also 
> uncheachable.|
> |40|Session has expired and the other instance has taken leadership ( in my 
> example, the other instance can connect to zookeeper )|
> |66.666|LOST ( after 40sec from SUSPENDED, curator finally sends LOST)|
> As you can see, if we are ignoring the SUSPENDED event, the second 
> application instance acquires the leadership ~26 seconds before the first 
> instance notice that it lost the leadership.
>  I understand it seems to be a very rare case, but either way I think it 
> should be addressed.
> *Reproduce it*
> I have came up with a way to reproduce this easily. I know this is not a 
> recommended production setup, but it reproduces the problem anyway.
> 1) On a virtual machine, with 2 network interfaces (eth0[192.168.0.101], 
> eth1[192.168.0.102]) , I installed one zookeeper instance.
>  2) I setup application1 with the leadership receipe, with 40sec of 
> negotiated timeout, pointing just to 192.168.0.101  .Now it is the leader
>  3) I setup application2 with the leadership receipe, with 40sec of 
> negotiated timeout, pointing just to 192.168.0.102  .Now it it is NOT the 
> leader
>  4) On the server I turn the eth0[192.168.0.101] interface down  [ ifconfig 
> eth0 down ]
>  5) After 26 seconds, application1 says : 
>            :ClientCnxn$SendThread@] - Client session timed out, have not 
> heard from server in 26679ms for sessionid
>             ConnectionStateManager@237] - State change: SUSPENDED
>         NOTE: I'm ignoring the SUSPENDED event
> 6) 

[jira] [Commented] (CURATOR-460) Timed tolerance for connection suspended leads to simultaneous leaders

[Jordan Zimmerman (JIRA)]

[ 
https://issues.apache.org/jira/browse/CURATOR-460?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16415190#comment-16415190
 ] 

Jordan Zimmerman commented on CURATOR-460:
--

If needed, we could initialize the counter with 2/3 of a session or whatever 
fraction of a session is implied by the ZooKeeper code.

> Timed tolerance for connection suspended leads to simultaneous leaders
> --
>
> Key: CURATOR-460
> URL: https://issues.apache.org/jira/browse/CURATOR-460
> Project: Apache Curator
>  Issue Type: Bug
>  Components: Recipes
>Affects Versions: 3.3.0, 4.0.0, 4.0.1
>Reporter: Antonio Rafael Rodrigues
>Assignee: Cameron McKenzie
>Priority: Major
>
> Starting from Curator 3, after losing the connection to zookeeper, curator 
> emits a SUSPENDED event and then starts an internal timer, if the time of the 
> negotiatied session timeout get hit and curator didn't reconnect to zookeeper 
> it emits a LOST event.
> For example :
>  Given negotiated session timeout = 40
> ||Time (seconds)||Event||
> |0|SUSPENDED (curator has been disconnected from zookeeper)|
> |40|LOST (curator is still disconnected, it must have been lost the sesion as 
> it is disconnected for 40sec)|
> Given this scenario we could ,theoretically, ignore the SUSPENDED event and 
> consider the leadership as lost just if the ConnectionStateListener receives 
> the LOST event.
> But this feature seems to have introduced a bug (from my point of view)
> *Case of failure*
> ||Time (seconds)||Event||
> |0|Something went wrong with the connected zookeeper (in my case, the network 
> inteface of zookeeper's server has gone down). Curator stops hearing heart 
> beats from zookeeper, but doesn't lose the connection (from some reason that 
> I don't know, if the network interface of the server goes down, Curator 
> doesn't lose the connection)|
> |~26.666|SUSPENDED (after 26 seconds without hearing heartbeats, curator 
> emits a SUSPENDED event) 26 is from "readTimeout = sessionTimeout * 2 / 3" 
> from the class ClientCnxn.java from zookeeper client. At this point, curator 
> starts counting 40 sec.|
> |26.666 to 40|During this period, Curator is trying to connect to other 
> zookeeper instances but the other instances, in my example case, are also 
> uncheachable.|
> |40|Session has expired and the other instance has taken leadership ( in my 
> example, the other instance can connect to zookeeper )|
> |66.666|LOST ( after 40sec from SUSPENDED, curator finally sends LOST)|
> As you can see, if we are ignoring the SUSPENDED event, the second 
> application instance acquires the leadership ~26 seconds before the first 
> instance notice that it lost the leadership.
>  I understand it seems to be a very rare case, but either way I think it 
> should be addressed.
> *Reproduce it*
> I have came up with a way to reproduce this easily. I know this is not a 
> recommended production setup, but it reproduces the problem anyway.
> 1) On a virtual machine, with 2 network interfaces (eth0[192.168.0.101], 
> eth1[192.168.0.102]) , I installed one zookeeper instance.
>  2) I setup application1 with the leadership receipe, with 40sec of 
> negotiated timeout, pointing just to 192.168.0.101  .Now it is the leader
>  3) I setup application2 with the leadership receipe, with 40sec of 
> negotiated timeout, pointing just to 192.168.0.102  .Now it it is NOT the 
> leader
>  4) On the server I turn the eth0[192.168.0.101] interface down  [ ifconfig 
> eth0 down ]
>  5) After 26 seconds, application1 says : 
>            :ClientCnxn$SendThread@] - Client session timed out, have not 
> heard from server in 26679ms for sessionid
>             ConnectionStateManager@237] - State change: SUSPENDED
>         NOTE: I'm ignoring the SUSPENDED event
> 6) After 40 seconds, application2 takes leadership
>  7) After 66 seconds, application1 says :
>              ConnectionStateManager@237] - State change: LOST
>          
>          NOTE: Just at this point, I consider that the application1 has lost 
> leadership
> Then, for 26 seconds, we had 2 leaders.
> If you confirm it as a bug, I think I could help.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (CURATOR-460) Timed tolerance for connection suspended leads to simultaneous leaders

[Antonio Rafael Rodrigues (JIRA)]

[ 
https://issues.apache.org/jira/browse/CURATOR-460?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16414925#comment-16414925
 ] 

Antonio Rafael Rodrigues commented on CURATOR-460:
--

I'm not very experienced on Curator's or Zookeeper's code, but at first glance, 
it seems to me that it is not possible to know the already elapsed time from 
Curator code. I think it would be possible if Zookeeper publishes the elapsed 
time inside the enqueued  WatchedEvent that put curator on SUSPENDED, but it 
doesn't.

> Timed tolerance for connection suspended leads to simultaneous leaders
> --
>
> Key: CURATOR-460
> URL: https://issues.apache.org/jira/browse/CURATOR-460
> Project: Apache Curator
>  Issue Type: Bug
>  Components: Recipes
>Affects Versions: 3.3.0, 4.0.0, 4.0.1
>Reporter: Antonio Rafael Rodrigues
>Assignee: Cameron McKenzie
>Priority: Major
>
> Starting from Curator 3, after losing the connection to zookeeper, curator 
> emits a SUSPENDED event and then starts an internal timer, if the time of the 
> negotiatied session timeout get hit and curator didn't reconnect to zookeeper 
> it emits a LOST event.
> For example :
>  Given negotiated session timeout = 40
> ||Time (seconds)||Event||
> |0|SUSPENDED (curator has been disconnected from zookeeper)|
> |40|LOST (curator is still disconnected, it must have been lost the sesion as 
> it is disconnected for 40sec)|
> Given this scenario we could ,theoretically, ignore the SUSPENDED event and 
> consider the leadership as lost just if the ConnectionStateListener receives 
> the LOST event.
> But this feature seems to have introduced a bug (from my point of view)
> *Case of failure*
> ||Time (seconds)||Event||
> |0|Something went wrong with the connected zookeeper (in my case, the network 
> inteface of zookeeper's server has gone down). Curator stops hearing heart 
> beats from zookeeper, but doesn't lose the connection (from some reason that 
> I don't know, if the network interface of the server goes down, Curator 
> doesn't lose the connection)|
> |~26.666|SUSPENDED (after 26 seconds without hearing heartbeats, curator 
> emits a SUSPENDED event) 26 is from "readTimeout = sessionTimeout * 2 / 3" 
> from the class ClientCnxn.java from zookeeper client. At this point, curator 
> starts counting 40 sec.|
> |26.666 to 40|During this period, Curator is trying to connect to other 
> zookeeper instances but the other instances, in my example case, are also 
> uncheachable.|
> |40|Session has expired and the other instance has taken leadership ( in my 
> example, the other instance can connect to zookeeper )|
> |66.666|LOST ( after 40sec from SUSPENDED, curator finally sends LOST)|
> As you can see, if we are ignoring the SUSPENDED event, the second 
> application instance acquires the leadership ~26 seconds before the first 
> instance notice that it lost the leadership.
>  I understand it seems to be a very rare case, but either way I think it 
> should be addressed.
> *Reproduce it*
> I have came up with a way to reproduce this easily. I know this is not a 
> recommended production setup, but it reproduces the problem anyway.
> 1) On a virtual machine, with 2 network interfaces (eth0[192.168.0.101], 
> eth1[192.168.0.102]) , I installed one zookeeper instance.
>  2) I setup application1 with the leadership receipe, with 40sec of 
> negotiated timeout, pointing just to 192.168.0.101  .Now it is the leader
>  3) I setup application2 with the leadership receipe, with 40sec of 
> negotiated timeout, pointing just to 192.168.0.102  .Now it it is NOT the 
> leader
>  4) On the server I turn the eth0[192.168.0.101] interface down  [ ifconfig 
> eth0 down ]
>  5) After 26 seconds, application1 says : 
>            :ClientCnxn$SendThread@] - Client session timed out, have not 
> heard from server in 26679ms for sessionid
>             ConnectionStateManager@237] - State change: SUSPENDED
>         NOTE: I'm ignoring the SUSPENDED event
> 6) After 40 seconds, application2 takes leadership
>  7) After 66 seconds, application1 says :
>              ConnectionStateManager@237] - State change: LOST
>          
>          NOTE: Just at this point, I consider that the application1 has lost 
> leadership
> Then, for 26 seconds, we had 2 leaders.
> If you confirm it as a bug, I think I could help.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (CURATOR-460) Timed tolerance for connection suspended leads to simultaneous leaders

[Cameron McKenzie (JIRA)]

[ 
https://issues.apache.org/jira/browse/CURATOR-460?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16413586#comment-16413586
 ] 

Cameron McKenzie commented on CURATOR-460:
--

I'll try and have a look in the next couple of days.

> Timed tolerance for connection suspended leads to simultaneous leaders
> --
>
> Key: CURATOR-460
> URL: https://issues.apache.org/jira/browse/CURATOR-460
> Project: Apache Curator
>  Issue Type: Bug
>  Components: Recipes
>Affects Versions: 3.3.0, 4.0.0, 4.0.1
>Reporter: Antonio Rafael Rodrigues
>Priority: Major
>
> Starting from Curator 3, after losing the connection to zookeeper, curator 
> emits a SUSPENDED event and then starts an internal timer, if the time of the 
> negotiatied session timeout get hit and curator didn't reconnect to zookeeper 
> it emits a LOST event.
> For example :
>  Given negotiated session timeout = 40
> ||Time (seconds)||Event||
> |0|SUSPENDED (curator has been disconnected from zookeeper)|
> |40|LOST (curator is still disconnected, it must have been lost the sesion as 
> it is disconnected for 40sec)|
> Given this scenario we could ,theoretically, ignore the SUSPENDED event and 
> consider the leadership as lost just if the ConnectionStateListener receives 
> the LOST event.
> But this feature seems to have introduced a bug (from my point of view)
> *Case of failure*
> ||Time (seconds)||Event||
> |0|Something went wrong with the connected zookeeper (in my case, the network 
> inteface of zookeeper's server has gone down). Curator stops hearing heart 
> beats from zookeeper, but doesn't lose the connection (from some reason that 
> I don't know, if the network interface of the server goes down, Curator 
> doesn't lose the connection)|
> |~26.666|SUSPENDED (after 26 seconds without hearing heartbeats, curator 
> emits a SUSPENDED event) 26 is from "readTimeout = sessionTimeout * 2 / 3" 
> from the class ClientCnxn.java from zookeeper client. At this point, curator 
> starts counting 40 sec.|
> |26.666 to 40|During this period, Curator is trying to connect to other 
> zookeeper instances but the other instances, in my example case, are also 
> uncheachable.|
> |40|Session has expired and the other instance has taken leadership ( in my 
> example, the other instance can connect to zookeeper )|
> |66.666|LOST ( after 40sec from SUSPENDED, curator finally sends LOST)|
> As you can see, if we are ignoring the SUSPENDED event, the second 
> application instance acquires the leadership ~26 seconds before the first 
> instance notice that it lost the leadership.
>  I understand it seems to be a very rare case, but either way I think it 
> should be addressed.
> *Reproduce it*
> I have came up with a way to reproduce this easily. I know this is not a 
> recommended production setup, but it reproduces the problem anyway.
> 1) On a virtual machine, with 2 network interfaces (eth0[192.168.0.101], 
> eth1[192.168.0.102]) , I installed one zookeeper instance.
>  2) I setup application1 with the leadership receipe, with 40sec of 
> negotiated timeout, pointing just to 192.168.0.101  .Now it is the leader
>  3) I setup application2 with the leadership receipe, with 40sec of 
> negotiated timeout, pointing just to 192.168.0.102  .Now it it is NOT the 
> leader
>  4) On the server I turn the eth0[192.168.0.101] interface down  [ ifconfig 
> eth0 down ]
>  5) After 26 seconds, application1 says : 
>            :ClientCnxn$SendThread@] - Client session timed out, have not 
> heard from server in 26679ms for sessionid
>             ConnectionStateManager@237] - State change: SUSPENDED
>         NOTE: I'm ignoring the SUSPENDED event
> 6) After 40 seconds, application2 takes leadership
>  7) After 66 seconds, application1 says :
>              ConnectionStateManager@237] - State change: LOST
>          
>          NOTE: Just at this point, I consider that the application1 has lost 
> leadership
> Then, for 26 seconds, we had 2 leaders.
> If you confirm it as a bug, I think I could help.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (CURATOR-460) Timed tolerance for connection suspended leads to simultaneous leaders

[Jordan Zimmerman (JIRA)]

[ 
https://issues.apache.org/jira/browse/CURATOR-460?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16413543#comment-16413543
 ] 

Jordan Zimmerman commented on CURATOR-460:
--

Maybe I misunderstood how ZK managed session expiration when I wrote this (not 
unlikely). I thought that session expiration occurs after the first lost 
heartbeat. We'll need to dig into the ZK source code. If need be, we can alter 
ConnectionStateManager#setCurrentConnectionState so that 
{{startOfSuspendedEpoch}} takes the already lapsed heartbeat into account.

> Timed tolerance for connection suspended leads to simultaneous leaders
> --
>
> Key: CURATOR-460
> URL: https://issues.apache.org/jira/browse/CURATOR-460
> Project: Apache Curator
>  Issue Type: Bug
>  Components: Recipes
>Affects Versions: 3.3.0, 4.0.0, 4.0.1
>Reporter: Antonio Rafael Rodrigues
>Priority: Major
>
> Starting from Curator 3, after losing the connection to zookeeper, curator 
> emits a SUSPENDED event and then starts an internal timer, if the time of the 
> negotiatied session timeout get hit and curator didn't reconnect to zookeeper 
> it emits a LOST event.
> For example :
>  Given negotiated session timeout = 40
> ||Time (seconds)||Event||
> |0|SUSPENDED (curator has been disconnected from zookeeper)|
> |40|LOST (curator is still disconnected, it must have been lost the sesion as 
> it is disconnected for 40sec)|
> Given this scenario we could ,theoretically, ignore the SUSPENDED event and 
> consider the leadership as lost just if the ConnectionStateListener receives 
> the LOST event.
> But this feature seems to have introduced a bug (from my point of view)
> *Case of failure*
> ||Time (seconds)||Event||
> |0|Something went wrong with the connected zookeeper (in my case, the network 
> inteface of zookeeper's server has gone down). Curator stops hearing heart 
> beats from zookeeper, but doesn't lose the connection (from some reason that 
> I don't know, if the network interface of the server goes down, Curator 
> doesn't lose the connection)|
> |~26.666|SUSPENDED (after 26 seconds without hearing heartbeats, curator 
> emits a SUSPENDED event) 26 is from "readTimeout = sessionTimeout * 2 / 3" 
> from the class ClientCnxn.java from zookeeper client. At this point, curator 
> starts counting 40 sec.|
> |26.666 to 40|During this period, Curator is trying to connect to other 
> zookeeper instances but the other instances, in my example case, are also 
> uncheachable.|
> |40|Session has expired and the other instance has taken leadership ( in my 
> example, the other instance can connect to zookeeper )|
> |66.666|LOST ( after 40sec from SUSPENDED, curator finally sends LOST)|
> As you can see, if we are ignoring the SUSPENDED event, the second 
> application instance acquires the leadership ~26 seconds before the first 
> instance notice that it lost the leadership.
>  I understand it seems to be a very rare case, but either way I think it 
> should be addressed.
> *Reproduce it*
> I have came up with a way to reproduce this easily. I know this is not a 
> recommended production setup, but it reproduces the problem anyway.
> 1) On a virtual machine, with 2 network interfaces (eth0[192.168.0.101], 
> eth1[192.168.0.102]) , I installed one zookeeper instance.
>  2) I setup application1 with the leadership receipe, with 40sec of 
> negotiated timeout, pointing just to 192.168.0.101  .Now it is the leader
>  3) I setup application2 with the leadership receipe, with 40sec of 
> negotiated timeout, pointing just to 192.168.0.102  .Now it it is NOT the 
> leader
>  4) On the server I turn the eth0[192.168.0.101] interface down  [ ifconfig 
> eth0 down ]
>  5) After 26 seconds, application1 says : 
>            :ClientCnxn$SendThread@] - Client session timed out, have not 
> heard from server in 26679ms for sessionid
>             ConnectionStateManager@237] - State change: SUSPENDED
>         NOTE: I'm ignoring the SUSPENDED event
> 6) After 40 seconds, application2 takes leadership
>  7) After 66 seconds, application1 says :
>              ConnectionStateManager@237] - State change: LOST
>          
>          NOTE: Just at this point, I consider that the application1 has lost 
> leadership
> Then, for 26 seconds, we had 2 leaders.
> If you confirm it as a bug, I think I could help.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (CURATOR-460) Timed tolerance for connection suspended leads to simultaneous leaders

[Cameron McKenzie (JIRA)]

[ 
https://issues.apache.org/jira/browse/CURATOR-460?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16413197#comment-16413197
 ] 

Cameron McKenzie commented on CURATOR-460:
--

This seems like a problem with Curator's timers then doesn't it? It should be 
40 seconds from the initial connection loss that the LOST event is generated, 
not 40 seconds after the SUSPENDED event is generated, which is what you seem 
to be seeing.

> Timed tolerance for connection suspended leads to simultaneous leaders
> --
>
> Key: CURATOR-460
> URL: https://issues.apache.org/jira/browse/CURATOR-460
> Project: Apache Curator
>  Issue Type: Bug
>  Components: Recipes
>Affects Versions: 3.3.0, 4.0.0, 4.0.1
>Reporter: Antonio Rafael Rodrigues
>Priority: Major
>
> Starting from Curator 3, after losing the connection to zookeeper, curator 
> emits a SUSPENDED event and then starts an internal timer, if the time of the 
> negotiatied session timeout get hit and curator didn't reconnect to zookeeper 
> it emits a LOST event.
> For example :
>  Given negotiated session timeout = 40
> ||Time (seconds)||Event||
> |0|SUSPENDED (curator has been disconnected from zookeeper)|
> |40|LOST (curator is still disconnected, it must have been lost the sesion as 
> it is disconnected for 40sec)|
> Given this scenario we could ,theoretically, ignore the SUSPENDED event and 
> consider the leadership as lost just if the ConnectionStateListener receives 
> the LOST event.
> But this feature seems to have introduced a bug (from my point of view)
> *Case of failure*
> ||Time (seconds)||Event||
> |0|Something went wrong with the connected zookeeper (in my case, the network 
> inteface of zookeeper's server has gone down). Curator stops hearing heart 
> beats from zookeeper, but doesn't lose the connection (from some reason that 
> I don't know, if the network interface of the server goes down, Curator 
> doesn't lose the connection)|
> |~26.666|SUSPENDED (after 26 seconds without hearing heartbeats, curator 
> emits a SUSPENDED event) 26 is from "readTimeout = sessionTimeout * 2 / 3" 
> from the class ClientCnxn.java from zookeeper client. At this point, curator 
> starts counting 40 sec.|
> |26.666 to 40|During this period, Curator is trying to connect to other 
> zookeeper instances but the other instances, in my example case, are also 
> uncheachable.|
> |40|Session has expired and the other instance has taken leadership ( in my 
> example, the other instance can connect to zookeeper )|
> |66.666|LOST ( after 40sec from SUSPENDED, curator finally sends LOST)|
> As you can see, if we are ignoring the SUSPENDED event, the second 
> application instance acquires the leadership ~26 seconds before the first 
> instance notice that it lost the leadership.
>  I understand it seems to be a very rare case, but either way I think it 
> should be addressed.
> *Reproduce it*
> I have came up with a way to reproduce this easily. I know this is not a 
> recommended production setup, but it reproduces the problem anyway.
> 1) On a virtual machine, with 2 network interfaces (eth0[192.168.0.101], 
> eth1[192.168.0.102]) , I installed one zookeeper instance.
>  2) I setup application1 with the leadership receipe, with 40sec of 
> negotiated timeout, pointing just to 192.168.0.101  .Now it is the leader
>  3) I setup application2 with the leadership receipe, with 40sec of 
> negotiated timeout, pointing just to 192.168.0.102  .Now it it is NOT the 
> leader
>  4) On the server I turn the eth0[192.168.0.101] interface down  [ ifconfig 
> eth0 down ]
>  5) After 26 seconds, application1 says : 
>            :ClientCnxn$SendThread@] - Client session timed out, have not 
> heard from server in 26679ms for sessionid
>             ConnectionStateManager@237] - State change: SUSPENDED
>         NOTE: I'm ignoring the SUSPENDED event
> 6) After 40 seconds, application2 takes leadership
>  7) After 66 seconds, application1 says :
>              ConnectionStateManager@237] - State change: LOST
>          
>          NOTE: Just at this point, I consider that the application1 has lost 
> leadership
> Then, for 26 seconds, we had 2 leaders.
> If you confirm it as a bug, I think I could help.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)