Hi there,
We were recently running into an issue in cloud env. Specifically, we have a kafka cluster setup in cloud, and it turns out to be a failure when the zookeeper node is replaced, namely the ip changed while the hostname remains same. We have patched PR150 (https://github.com/apache/zookeeper/pull/150/files) but it seems still not works. While the new zk node ip can be re-resolved and zk client in kafka can establish connection to zk node, but the client seems not able to read content from the connection. Anyone encounters similar issue? Pasted the log from kafka server. [2018-05-11 03:47:46,806] WARN SASL configuration failed: javax.security.auth.login.LoginException: No JAAS configuration section named 'Client' was found in specified JAAS configuration file: '/usr/share/kafka/config/kafka_server_jaas. conf'. Will continue connection to Zookeeper server without SASL authentication, if Zookeeper server allows it. (org.apache.zookeeper.ClientCnxn) [2018-05-11 03:47:46,806] ERROR [ZooKeeperClient] Auth failed. (kafka.zookeeper.ZooKeeperClient) [2018-05-11 03:47:46,806] ERROR [ZooKeeperClient] Auth failed. (kafka.zookeeper.ZooKeeperClient) [2018-05-11 03:47:46,806] INFO Opening socket connection to server host.name/10.147.164.83:2181 (org.apache.zookeeper.ClientCnxn) [2018-05-11 03:47:49,815] WARN Session 0x1634d42d88a001e for server null, unexpected error, closing socket connection and attempting reconnect (org.apache.zookeeper.ClientCnxn) java.net.NoRouteToHostException: No route to host at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:717) at org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:361) at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1141) [2018-05-11 03:47:50,667] INFO Resolving again hostname: host.name (org.apache.zookeeper.client.StaticHostProvider) [2018-05-11 03:47:50,667] WARN Cannot re-resolve server: host.name/10.147.164.83:2181 (org.apache.zookeeper.client.StaticHostProvider) java.net.UnknownHostException: host.name at java.net.InetAddress.getAllByName0(InetAddress.java:1280) at java.net.InetAddress.getAllByName(InetAddress.java:1192) at java.net.InetAddress.getAllByName(InetAddress.java:1126) at org.apache.zookeeper.client.StaticHostProvider.next(StaticHostProvider.java:142) at org.apache.zookeeper.ClientCnxn$SendThread.startConnect(ClientCnxn.java:997) at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1060) [2018-05-11 03:47:51,668] WARN SASL configuration failed: javax.security.auth.login.LoginException: No JAAS configuration section named 'Client' was found in specified JAAS configuration file: '/usr/share/kafka/config/kafka_server_jaas. conf'. Will continue connection to Zookeeper server without SASL authentication, if Zookeeper server allows it. (org.apache.zookeeper.ClientCnxn) [2018-05-11 03:47:51,668] ERROR [ZooKeeperClient] Auth failed. (kafka.zookeeper.ZooKeeperClient) [2018-05-11 03:47:51,668] ERROR [ZooKeeperClient] Auth failed. (kafka.zookeeper.ZooKeeperClient) [2018-05-11 03:47:51,668] INFO Opening socket connection to server host.name/10.147.164.83:2181 (org.apache.zookeeper.ClientCnxn) [2018-05-11 03:47:52,822] WARN Session 0x1634d42d88a001e for server null, unexpected error, closing socket connection and attempting reconnect (org.apache.zookeeper.ClientCnxn) java.net.NoRouteToHostException: No route to host at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:717) at org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:361) at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1141) [2018-05-11 03:47:53,427] INFO Resolving again hostname: host.name (org.apache.zookeeper.client.StaticHostProvider) [2018-05-11 03:47:54,429] WARN SASL configuration failed: javax.security.auth.login.LoginException: No JAAS configuration section named 'Client' was found in specified JAAS configuration file: '/usr/share/kafka/config/kafka_server_jaas. conf'. Will continue connection to Zookeeper server without SASL authentication, if Zookeeper server allows it. (org.apache.zookeeper.ClientCnxn) [2018-05-11 03:47:54,429] ERROR [ZooKeeperClient] Auth failed. (kafka.zookeeper.ZooKeeperClient) [2018-05-11 03:47:54,429] ERROR [ZooKeeperClient] Auth failed. (kafka.zookeeper.ZooKeeperClient) [2018-05-11 03:47:54,429] INFO Opening socket connection to server host.name/10.147.161.203:2181 (org.apache.zookeeper.ClientCnxn) [2018-05-11 03:47:55,383] INFO Socket connection established to host.name/10.147.161.203:2181, initiating session (org.apache.zookeeper.ClientCnxn) [2018-05-11 03:47:55,386] INFO Unable to read additional data from server sessionid 0x1634d42d88a001e, likely server has closed socket, closing socket connection and attempting reconnect (org.apache.zookeeper.ClientCnxn) this ends up with continuous log flushing, the log is repeated in a sequence like above highlighted ones. Anyone can share thoughts regarding this issue? Thanks, Jeff
