[
https://issues.apache.org/jira/browse/KAFKA-7488?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Luigi Tagliamonte updated KAFKA-7488:
-------------------------------------
Description:
This issue seems related to https://issues.apache.org/jira/browse/KAFKA-2729
that has been resolved here https://issues.apache.org/jira/browse/KAFKA-2729
The issue still exists in Kafka 1.1
Cluster details:
* 3 Kafka nodes cluster running 1.1
* 3 Zookeeper node cluster running 3.4.10
Today meanwhile I was replacing a zookeeper server
([10.48.208.70|http://10.48.208.70/]) the leader among the brokers experienced
this issue:
{code:java}
[2018-10-05 21:03:02,799] INFO [GroupMetadataManager brokerId=1] Removed 0
expired offsets in 0 milliseconds.
(kafka.coordinator.group.GroupMetadataManager)
[2018-10-05 21:08:20,060] INFO Unable to read additional data from server
sessionid 0x34663b434985000e, likely server has closed socket, closing socket
connection and attempting reconnect (org.apache.zookeeper.ClientCnxn)
[2018-10-05 21:08:21,001] INFO Opening socket connection to server
10.48.208.70/10.48.208.70:2181. Will not attempt to authenticate using SASL
(unknown error) (org.apache.zookeeper.ClientCnxn)
[2018-10-05 21:08:21,003] WARN Session 0x34663b434985000e for server null,
unexpected error, closing socket connection and attempting reconnect
(org.apache.zookeeper.ClientCnxn)
java.net.ConnectException: Connection refused
at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:717)
at
org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:361)
at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1141)
[2018-10-05 21:08:21,797] INFO Opening socket connection to server
10.48.210.44/10.48.210.44:2181. Will not attempt to authenticate using SASL
(unknown error) (org.apache.zookeeper.ClientCnxn)
[2018-10-05 21:08:21,799] INFO Socket connection established to
10.48.210.44/10.48.210.44:2181, initiating session
(org.apache.zookeeper.ClientCnxn)
[2018-10-05 21:08:21,802] INFO Session establishment complete on server
10.48.210.44/10.48.210.44:2181, sessionid = 0x34663b434985000e, negotiated
timeout = 6000 (org.apache.zookeeper.ClientCnxn)
[2018-10-05 21:08:28,015] INFO Creating /controller (is it secure? false)
(kafka.zk.KafkaZkClient)
[2018-10-05 21:08:28,015] INFO Creating /controller (is it secure? false)
(kafka.zk.KafkaZkClient)
[2018-10-05 21:08:28,025] ERROR Error while creating ephemeral at /controller,
node already exists and owner '3703712903740981258' does not match current
session '3775770497779040270' (kafka.zk.KafkaZkClient$CheckedEphemeral)
[2018-10-05 21:08:28,025] ERROR Error while creating ephemeral at /controller,
node already exists and owner '3703712903740981258' does not match current
session '3775770497779040270' (kafka.zk.KafkaZkClient$CheckedEphemeral)
[2018-10-05 21:08:28,025] INFO Result of znode creation at /controller is:
NODEEXISTS (kafka.zk.KafkaZkClient)
[2018-10-05 21:08:28,025] INFO Result of znode creation at /controller is:
NODEEXISTS (kafka.zk.KafkaZkClient)
[2018-10-05 21:08:42,561] INFO [Partition <PT_NAME>-store-changelog-7 broker=1]
Shrinking ISR from 2,1,3 to 1 (kafka.cluster.Partition)
[2018-10-05 21:08:42,561] INFO [Partition <PT_NAME>-store-changelog-7 broker=1]
Shrinking ISR from 2,1,3 to 1 (kafka.cluster.Partition)
[2018-10-05 21:08:42,569] INFO [Partition <PT_NAME>-store-changelog-7 broker=1]
Cached zkVersion [11] not equal to that in zookeeper, skip updating ISR
(kafka.cluster.Partition)
[2018-10-05 21:08:42,569] INFO [Partition <PT_NAME>-store-changelog-7 broker=1]
Cached zkVersion [11] not equal to that in zookeeper, skip updating ISR
(kafka.cluster.Partition)
[2018-10-05 21:08:42,569] INFO [Partition <PT_NAME>bycontact_0-19 broker=1]
Shrinking ISR from 2,1,3 to 1 (kafka.cluster.Partition)
[2018-10-05 21:08:42,569] INFO [Partition <PT_NAME>bycontact_0-19 broker=1]
Shrinking ISR from 2,1,3 to 1 (kafka.cluster.Partition)
[2018-10-05 21:08:42,574] INFO [Partition <PT_NAME>bycontact_0-19 broker=1]
Cached zkVersion [44] not equal to that in zookeeper, skip updating ISR
(kafka.cluster.Partition)
[2018-10-05 21:08:42,574] INFO [Partition <PT_NAME>bycontact_0-19 broker=1]
Cached zkVersion [44] not equal to that in zookeeper, skip updating ISR
(kafka.cluster.Partition){code}
The only way in order to recover was to restart the broker.
was:
This issue seems related to https://issues.apache.org/jira/browse/KAFKA-2729
that has been resolved here https://issues.apache.org/jira/browse/KAFKA-2729
The issue still exists in Kafka 1.1
Cluster details:
* 3 Kafka nodes cluster running 1.1
* 3 Zookeeper node cluster running 3.4.10
Today meanwhile I was replacing a zookeeper server the leader among the brokers
experienced this issue:
{code:java}
[2018-10-05 21:03:02,799] INFO [GroupMetadataManager brokerId=1] Removed 0
expired offsets in 0 milliseconds.
(kafka.coordinator.group.GroupMetadataManager)
[2018-10-05 21:08:20,060] INFO Unable to read additional data from server
sessionid 0x34663b434985000e, likely server has closed socket, closing socket
connection and attempting reconnect (org.apache.zookeeper.ClientCnxn)
[2018-10-05 21:08:21,001] INFO Opening socket connection to server
10.48.208.70/10.48.208.70:2181. Will not attempt to authenticate using SASL
(unknown error) (org.apache.zookeeper.ClientCnxn)
[2018-10-05 21:08:21,003] WARN Session 0x34663b434985000e for server null,
unexpected error, closing socket connection and attempting reconnect
(org.apache.zookeeper.ClientCnxn)
java.net.ConnectException: Connection refused
at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:717)
at
org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:361)
at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1141)
[2018-10-05 21:08:21,797] INFO Opening socket connection to server
10.48.210.44/10.48.210.44:2181. Will not attempt to authenticate using SASL
(unknown error) (org.apache.zookeeper.ClientCnxn)
[2018-10-05 21:08:21,799] INFO Socket connection established to
10.48.210.44/10.48.210.44:2181, initiating session
(org.apache.zookeeper.ClientCnxn)
[2018-10-05 21:08:21,802] INFO Session establishment complete on server
10.48.210.44/10.48.210.44:2181, sessionid = 0x34663b434985000e, negotiated
timeout = 6000 (org.apache.zookeeper.ClientCnxn)
[2018-10-05 21:08:28,015] INFO Creating /controller (is it secure? false)
(kafka.zk.KafkaZkClient)
[2018-10-05 21:08:28,015] INFO Creating /controller (is it secure? false)
(kafka.zk.KafkaZkClient)
[2018-10-05 21:08:28,025] ERROR Error while creating ephemeral at /controller,
node already exists and owner '3703712903740981258' does not match current
session '3775770497779040270' (kafka.zk.KafkaZkClient$CheckedEphemeral)
[2018-10-05 21:08:28,025] ERROR Error while creating ephemeral at /controller,
node already exists and owner '3703712903740981258' does not match current
session '3775770497779040270' (kafka.zk.KafkaZkClient$CheckedEphemeral)
[2018-10-05 21:08:28,025] INFO Result of znode creation at /controller is:
NODEEXISTS (kafka.zk.KafkaZkClient)
[2018-10-05 21:08:28,025] INFO Result of znode creation at /controller is:
NODEEXISTS (kafka.zk.KafkaZkClient)
[2018-10-05 21:08:42,561] INFO [Partition <PT_NAME>-store-changelog-7 broker=1]
Shrinking ISR from 2,1,3 to 1 (kafka.cluster.Partition)
[2018-10-05 21:08:42,561] INFO [Partition <PT_NAME>-store-changelog-7 broker=1]
Shrinking ISR from 2,1,3 to 1 (kafka.cluster.Partition)
[2018-10-05 21:08:42,569] INFO [Partition <PT_NAME>-store-changelog-7 broker=1]
Cached zkVersion [11] not equal to that in zookeeper, skip updating ISR
(kafka.cluster.Partition)
[2018-10-05 21:08:42,569] INFO [Partition <PT_NAME>-store-changelog-7 broker=1]
Cached zkVersion [11] not equal to that in zookeeper, skip updating ISR
(kafka.cluster.Partition)
[2018-10-05 21:08:42,569] INFO [Partition <PT_NAME>bycontact_0-19 broker=1]
Shrinking ISR from 2,1,3 to 1 (kafka.cluster.Partition)
[2018-10-05 21:08:42,569] INFO [Partition <PT_NAME>bycontact_0-19 broker=1]
Shrinking ISR from 2,1,3 to 1 (kafka.cluster.Partition)
[2018-10-05 21:08:42,574] INFO [Partition <PT_NAME>bycontact_0-19 broker=1]
Cached zkVersion [44] not equal to that in zookeeper, skip updating ISR
(kafka.cluster.Partition)
[2018-10-05 21:08:42,574] INFO [Partition <PT_NAME>bycontact_0-19 broker=1]
Cached zkVersion [44] not equal to that in zookeeper, skip updating ISR
(kafka.cluster.Partition){code}
The only way in order to recover was to restart the broker.
> Controller not recovering after disconnection to zookeeper
> ----------------------------------------------------------
>
> Key: KAFKA-7488
> URL: https://issues.apache.org/jira/browse/KAFKA-7488
> Project: Kafka
> Issue Type: Bug
> Components: controller
> Affects Versions: 1.1.0
> Reporter: Luigi Tagliamonte
> Priority: Major
>
> This issue seems related to https://issues.apache.org/jira/browse/KAFKA-2729
> that has been resolved here https://issues.apache.org/jira/browse/KAFKA-2729
> The issue still exists in Kafka 1.1
> Cluster details:
> * 3 Kafka nodes cluster running 1.1
> * 3 Zookeeper node cluster running 3.4.10
> Today meanwhile I was replacing a zookeeper server
> ([10.48.208.70|http://10.48.208.70/]) the leader among the brokers
> experienced this issue:
> {code:java}
> [2018-10-05 21:03:02,799] INFO [GroupMetadataManager brokerId=1] Removed 0
> expired offsets in 0 milliseconds.
> (kafka.coordinator.group.GroupMetadataManager)
> [2018-10-05 21:08:20,060] INFO Unable to read additional data from server
> sessionid 0x34663b434985000e, likely server has closed socket, closing socket
> connection and attempting reconnect (org.apache.zookeeper.ClientCnxn)
> [2018-10-05 21:08:21,001] INFO Opening socket connection to server
> 10.48.208.70/10.48.208.70:2181. Will not attempt to authenticate using SASL
> (unknown error) (org.apache.zookeeper.ClientCnxn)
> [2018-10-05 21:08:21,003] WARN Session 0x34663b434985000e for server null,
> unexpected error, closing socket connection and attempting reconnect
> (org.apache.zookeeper.ClientCnxn)
> java.net.ConnectException: Connection refused
> at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
> at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:717)
> at
> org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:361)
> at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1141)
> [2018-10-05 21:08:21,797] INFO Opening socket connection to server
> 10.48.210.44/10.48.210.44:2181. Will not attempt to authenticate using SASL
> (unknown error) (org.apache.zookeeper.ClientCnxn)
> [2018-10-05 21:08:21,799] INFO Socket connection established to
> 10.48.210.44/10.48.210.44:2181, initiating session
> (org.apache.zookeeper.ClientCnxn)
> [2018-10-05 21:08:21,802] INFO Session establishment complete on server
> 10.48.210.44/10.48.210.44:2181, sessionid = 0x34663b434985000e, negotiated
> timeout = 6000 (org.apache.zookeeper.ClientCnxn)
> [2018-10-05 21:08:28,015] INFO Creating /controller (is it secure? false)
> (kafka.zk.KafkaZkClient)
> [2018-10-05 21:08:28,015] INFO Creating /controller (is it secure? false)
> (kafka.zk.KafkaZkClient)
> [2018-10-05 21:08:28,025] ERROR Error while creating ephemeral at
> /controller, node already exists and owner '3703712903740981258' does not
> match current session '3775770497779040270'
> (kafka.zk.KafkaZkClient$CheckedEphemeral)
> [2018-10-05 21:08:28,025] ERROR Error while creating ephemeral at
> /controller, node already exists and owner '3703712903740981258' does not
> match current session '3775770497779040270'
> (kafka.zk.KafkaZkClient$CheckedEphemeral)
> [2018-10-05 21:08:28,025] INFO Result of znode creation at /controller is:
> NODEEXISTS (kafka.zk.KafkaZkClient)
> [2018-10-05 21:08:28,025] INFO Result of znode creation at /controller is:
> NODEEXISTS (kafka.zk.KafkaZkClient)
> [2018-10-05 21:08:42,561] INFO [Partition <PT_NAME>-store-changelog-7
> broker=1] Shrinking ISR from 2,1,3 to 1 (kafka.cluster.Partition)
> [2018-10-05 21:08:42,561] INFO [Partition <PT_NAME>-store-changelog-7
> broker=1] Shrinking ISR from 2,1,3 to 1 (kafka.cluster.Partition)
> [2018-10-05 21:08:42,569] INFO [Partition <PT_NAME>-store-changelog-7
> broker=1] Cached zkVersion [11] not equal to that in zookeeper, skip updating
> ISR (kafka.cluster.Partition)
> [2018-10-05 21:08:42,569] INFO [Partition <PT_NAME>-store-changelog-7
> broker=1] Cached zkVersion [11] not equal to that in zookeeper, skip updating
> ISR (kafka.cluster.Partition)
> [2018-10-05 21:08:42,569] INFO [Partition <PT_NAME>bycontact_0-19 broker=1]
> Shrinking ISR from 2,1,3 to 1 (kafka.cluster.Partition)
> [2018-10-05 21:08:42,569] INFO [Partition <PT_NAME>bycontact_0-19 broker=1]
> Shrinking ISR from 2,1,3 to 1 (kafka.cluster.Partition)
> [2018-10-05 21:08:42,574] INFO [Partition <PT_NAME>bycontact_0-19 broker=1]
> Cached zkVersion [44] not equal to that in zookeeper, skip updating ISR
> (kafka.cluster.Partition)
> [2018-10-05 21:08:42,574] INFO [Partition <PT_NAME>bycontact_0-19 broker=1]
> Cached zkVersion [44] not equal to that in zookeeper, skip updating ISR
> (kafka.cluster.Partition){code}
> The only way in order to recover was to restart the broker.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
