[
https://issues.apache.org/jira/browse/CURATOR-638?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17536891#comment-17536891
]
Matt Martin commented on CURATOR-638:
-------------------------------------
The issue here appears to be caused or exacerbated by how zookeeper
configuration events are handled. When the curator is created against a list of
nodes, the underlying zookeeper client is provided the same list of nodes.
After connection, the {{EnsembleTracker}} receives a configuration event with
the same list of nodes, but it resolves those names to IPs and sets that as the
connection string.
When the zookeeper node the client connected with goes away, the curator
connection state becomes SUSPENDED before connecting to another node and
changing to RECONNECTED. The original node comes back up on another IP and the
other node in the cluster also cycles to a new IP before the node we
reconnected to goes down.
At this point zookeeper tries to connect to the IP address for the original
node, either from the connection string update or something cached internally.
That connection fails, since that IP isn't a zookeeper node. The same is then
tried for the original IP of the second node the client had connected with,
which is also not a zookeeper node anymore. Followed by an attempt at the IP
for the third node, to which the client had never connected and something
(curator or zookeeper) had earlier resolved to a now useless IP.
After all this, the client is unable to connect to any zookeeper nodes even
though all the requested hosts exist at different IPs. Curator closes the
client and creates a new one, but the connection string was changed to IP
addresses before any of the nodes changed and that client will never be able to
connect to zookeeper.
Why does curator resolve the hostnames to IPs that will never update?
Is that action on the connection string why zookeeper starts using old IPs
after nodes go down, even though it originally used the hostnames?
> Curator disconnect from zookeeper when IPs change
> -------------------------------------------------
>
> Key: CURATOR-638
> URL: https://issues.apache.org/jira/browse/CURATOR-638
> Project: Apache Curator
> Issue Type: Bug
> Components: Client, Recipes
> Affects Versions: 5.2.1
> Environment: Docker or Kubernetes, docker example provided
> Reporter: Francis Simon
> Priority: Blocker
> Attachments: zkissue.zip
>
>
> Blocking usage of Zookeeper in production. Tried testing a few versions all
> had the issue. Effects any recipes that use ephemeral nodes. Example
> attached.
> We use multiple Apache Curator recipes in our system which is running in
> Docker and Kubernetes. The behavior I am seeing is that curator appears to
> resolve to the IP address of the containers rather than being tied to DNS
> names. I have seen old tickets on this, but the behavior is reproducible on
> the latest code release.
> We are running zookeeper in containers on kubernetes. In kubernetes many
> things could cause a container to move hosts, the pod disruption budget
> ensures that a quorum is always present. But with this bug if all nodes
> move for any reason and get new IP addresses clients will disconnect when
> they shouldn't. Disconnecting has the bad side effect that all ephemeral
> nodes are lost. This effects for us coordination, distributed locking and
> service discovery. Causes production downtime so marked as a Blocker.
> I have a simple sample which just uses the service discovery recipe to
> register a bunch of services in zookeeper. I run the example in docker
> compose. It is 100% reproducible.
>
> {code:java}
> # Standup zookeeper and wait for it to be healthy
> docker-compose up -d zookeeper1 zookeeper2 zookeeper3
> # Stand up a server and make sure it is connected and working as expected
> docker-compose up -d server1
> # Take down a single zookeeper node and stand up another agent.
> # The agent will grab the old zookeepers IP address
> docker-compose rm -s zookeeper1
> docker-compose up -d server2
> # Bring the zookeeper node back up.
> # Wait for it be healthy
> docker-compose up -d zookeeper1
> # Then take down the next zookeeper node and stand up another agent.
> # The agent will grab the old zookeepers IP address
> docker-compose rm -s zookeeper2
> docker-compose up -d server3
> # Bring the zookeeper node back up.
> # Wait for it be healthy
> docker-compose up -d zookeeper2
> # Then take down the next zookeeper node and stand up another agent.
> # The agent will grab the old zookeepers IP address
> docker-compose rm -s zookeeper3
> docker-compose up -d server4
> # Bring the zookeeper node back up.
> # Wait for it be healthy
> docker-compose up -d zookeeper3{code}
>
> At the time of taking down the 3rd zookeeper node, the first server1 that was
> stood up will now receive a disconnected status because the IP of all three
> nodes has no changes form the original IP addresses.
>
> {code:java}
> server1_1 | Query instances for servicetest
> server1_1 | Exception in thread "main" java.lang.RuntimeException:
> org.apache.zookeeper.KeeperException$ConnectionLossException: KeeperErrorCode
> = ConnectionLoss for /myservices/test/62e23a0b-dfdb-46f5-966f-8dc7a4978c70
> server1_1 | at
> org.apache.curator.shaded.com.google.common.base.Throwables.propagate(Throwables.java:241)
> server1_1 | at
> org.apache.curator.utils.ExceptionAccumulator.propagate(ExceptionAccumulator.java:38)
> server1_1 | at
> org.apache.curator.x.discovery.details.ServiceDiscoveryImpl.close(ServiceDiscoveryImpl.java:171)
> server1_1 | at
> org.apache.curator.shaded.com.google.common.io.Closeables.close(Closeables.java:78)
> server1_1 | at
> org.apache.curator.utils.CloseableUtils.closeQuietly(CloseableUtils.java:59)
> server1_1 | at zkissue.App.main(App.java:72)
> server1_1 | Caused by:
> org.apache.zookeeper.KeeperException$ConnectionLossException: KeeperErrorCode
> = ConnectionLoss for /myservices/test/62e23a0b-dfdb-46f5-966f-8dc7a4978c70
> server1_1 | at
> org.apache.zookeeper.KeeperException.create(KeeperException.java:102)
> server1_1 | at
> org.apache.zookeeper.KeeperException.create(KeeperException.java:54)
> server1_1 | at
> org.apache.zookeeper.ZooKeeper.delete(ZooKeeper.java:2001)
> server1_1 | at
> org.apache.curator.framework.imps.DeleteBuilderImpl$5.call(DeleteBuilderImpl.java:274)
> server1_1 | at
> org.apache.curator.framework.imps.DeleteBuilderImpl$5.call(DeleteBuilderImpl.java:268)
> server1_1 | at
> org.apache.curator.RetryLoop.callWithRetry(RetryLoop.java:93)
> server1_1 | at
> org.apache.curator.framework.imps.DeleteBuilderImpl.pathInForeground(DeleteBuilderImpl.java:265)
> server1_1 | at
> org.apache.curator.framework.imps.DeleteBuilderImpl.forPath(DeleteBuilderImpl.java:249)
> server1_1 | at
> org.apache.curator.framework.imps.DeleteBuilderImpl.forPath(DeleteBuilderImpl.java:34)
> server1_1 | at
> org.apache.curator.x.discovery.details.ServiceDiscoveryImpl.internalUnregisterService(ServiceDiscoveryImpl.java:520)
> server1_1 | at
> org.apache.curator.x.discovery.details.ServiceDiscoveryImpl.close(ServiceDiscoveryImpl.java:157)
> server1_1 | ... 3 more
> {code}
>
> This causes it to disconnect and lose its discovery state which can be seen
> from the other services.
> {code:java}
> server2_1 | Query instances for servicetest
> server2_1 | test
> server2_1 | service description: http://server-4:57456
> server2_1 | service description: http://server-3:37740
> server2_1 | service description: http://server-2:40219{code}
>
> Should mention that the Zookeeper cluster is always happy and healthy. This
> is a client side issue.
>
>
--
This message was sent by Atlassian Jira
(v8.20.7#820007)
