Are you sure you’re getting the same IP address? Regarding zookeeper connection being closed, is kubernetes doing a soft shutdown of your container? If so, zookeeper is asked politely to stop. – Best regards, Radek Gruchalski radek@gruchalski.commailto:ra...@gruchalski.com de.linkedin.com/in/radgruchalski +4917685656526
Confidentiality: This communication is intended for the above-named person and may be confidential and/or legally privileged. If it has come to you in error you must take no action based on it, nor must you copy or show it to anyone; please delete/destroy and inform the sender immediately. On May 10, 2016 at 5:47:24 PM, Paolo Patierno (ppatie...@live.com) wrote: Hi all, experiencing with Kafka on Kubernetes I have the following error on Kafka server reconnection ... A cluster with one zookeeper and two kafka server ... I turn off the zookeeper pod but kubernetes restart it and guaratees the same IP address for it but the kafka server starts to retry connection failing with following trace : [2016-05-10 15:40:55,046] WARN Session 0x1549b308dd20002 for server 10.0.0.184/10.0.0.184:2181, unexpected error, closing socket connection and attempting reconnect (org.apache.zookeeper.ClientCnxn) java.io.IOException: Connection reset by peer at sun.nio.ch.FileDispatcherImpl.read0(Native Method) at sun.nio.ch.SocketDispatcher.read(SocketDispatcher.java:39) at sun.nio.ch.IOUtil.readIntoNativeBuffer(IOUtil.java:223) at sun.nio.ch.IOUtil.read(IOUtil.java:192) at sun.nio.ch.SocketChannelImpl.read(SocketChannelImpl.java:380) at org.apache.zookeeper.ClientCnxnSocketNIO.doIO(ClientCnxnSocketNIO.java:68) at org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:366) at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1081) [2016-05-10 15:40:55,149] INFO zookeeper state changed (Disconnected) (org.I0Itec.zkclient.ZkClient) [2016-05-10 15:40:57,093] INFO Opening socket connection to server 10.0.0.184/10.0.0.184:2181. Will not attempt to authenticate using SASL (unknown error) (org.apache.zookeeper.ClientCnxn) [2016-05-10 15:40:57,093] INFO Socket connection established to 10.0.0.184/10.0.0.184:2181, initiating session (org.apache.zookeeper.ClientCnxn) [2016-05-10 15:40:57,158] INFO Unable to read additional data from server sessionid 0x1549b308dd20002, likely server has closed socket, closing socket connection and attempting reconnect (org.apache.zookeeper.ClientCnxn) [2016-05-10 15:40:58,936] INFO Opening socket connection to server 10.0.0.184/10.0.0.184:2181. Will not attempt to authenticate using SASL (unknown error) (org.apache.zookeeper.ClientCnxn) [2016-05-10 15:40:58,936] INFO Socket connection established to 10.0.0.184/10.0.0.184:2181, initiating session (org.apache.zookeeper.ClientCnxn) [2016-05-10 15:40:58,937] INFO Unable to read additional data from server sessionid 0x1549b308dd20002, likely server has closed socket, closing socket connection and attempting reconnect (org.apache.zookeeper.ClientCnxn) [2016-05-10 15:41:00,845] INFO Opening socket connection to server 10.0.0.184/10.0.0.184:2181. Will not attempt to authenticate using SASL (unknown error) (org.apache.zookeeper.ClientCnxn) [2016-05-10 15:41:00,845] INFO Socket connection established to 10.0.0.184/10.0.0.184:2181, initiating session (org.apache.zookeeper.ClientCnxn) [2016-05-10 15:41:00,846] INFO Unable to read additional data from server sessionid 0x1549b308dd20002, likely server has closed socket, closing socket connection and attempting reconnect (org.apache.zookeeper.ClientCnxn) [2016-05-10 15:41:02,071] INFO Opening socket connection to server 10.0.0.184/10.0.0.184:2181. Will not attempt to authenticate using SASL (unknown error) (org.apache.zookeeper.ClientCnxn) [2016-05-10 15:41:02,071] INFO Socket connection established to 10.0.0.184/10.0.0.184:2181, initiating session (org.apache.zookeeper.ClientCnxn) [2016-05-10 15:41:02,072] INFO Unable to read additional data from server sessionid 0x1549b308dd20002, likely server has closed socket, closing socket connection and attempting reconnect (org.apache.zookeeper.ClientCnxn) [2016-05-10 15:41:03,336] INFO Opening socket connection to server 10.0.0.184/10.0.0.184:2181. Will not attempt to authenticate using SASL (unknown error) (org.apache.zookeeper.ClientCnxn) [2016-05-10 15:41:03,336] INFO Socket connection established to 10.0.0.184/10.0.0.184:2181, initiating session (org.apache.zookeeper.ClientCnxn) [2016-05-10 15:41:03,337] INFO Unable to read additional data from server sessionid 0x1549b308dd20002, likely server has closed socket, closing socket connection and attempting reconnect (org.apache.zookeeper.ClientCnxn) [2016-05-10 15:41:05,121] INFO Opening socket connection to server 10.0.0.184/10.0.0.184:2181. Will not attempt to authenticate using SASL (unknown error) (org.apache.zookeeper.ClientCnxn) [2016-05-10 15:41:05,121] INFO Socket connection established to 10.0.0.184/10.0.0.184:2181, initiating session (org.apache.zookeeper.ClientCnxn) [2016-05-10 15:41:05,122] INFO Unable to read additional data from server sessionid 0x1549b308dd20002, likely server has closed socket, closing socket connection and attempting reconnect (org.apache.zookeeper.ClientCnxn) You can see when the first zookeeper dies and connection is lost ... and all the retries by kafka server in order to connect to the new one (same IP, same port). Why the zookeeper server closes the connection (I can see FIN ACK frames on Wireshark) Thanks, Paolo. Paolo PatiernoSenior Software Engineer (IoT) @ Red Hat Microsoft MVP on Windows Embedded & IoTMicrosoft Azure Advisor Twitter : @ppatierno Linkedin : paolopatierno Blog : DevExperience
