[
https://issues.apache.org/jira/browse/KAFKA-7888?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16758638#comment-16758638
]
Jun Rao commented on KAFKA-7888:
--------------------------------
[~kemalerden], from the log, the "Cached zkVersion" log started around 23:53:35.
{code:java}
server_b13.log.2019-01-26-22:[2019-01-26 23:53:35,041] INFO [Partition
ucTrade-6 broker=13] Cached zkVersion [21] not equal to that in zookeeper, skip
updating ISR (kafka.cluster.Partition)
{code}
The controller log showed that broker 13 was never able to re-register itself
in ZK after 23:53:16.
{code:java}
controller_b14.log.2019-01-26-23:[2019-01-26 23:53:16,267] INFO [Controller
id=14] Newly added brokers: , deleted brokers: 13, all live brokers: 14,15
(kafka.controller.KafkaController)
controller_b14.log.2019-01-26-23:[2019-01-26 23:53:42,281] INFO [Controller
id=14] Newly added brokers: , deleted brokers: 15, all live brokers: 14
(kafka.controller.KafkaController)
controller_b14.log.2019-01-26-23:[2019-01-26 23:53:46,809] INFO [Controller
id=14] Newly added brokers: 15, deleted brokers: , all live brokers: 14,15
(kafka.controller.KafkaController)
{code}
>From broker 13's log, it failed to re-register itself in ZK around 23:53:11.
{code:java}
server_b13.log.2019-01-26-22:[2019-01-26 23:53:11,841] ERROR Error while
creating ephemeral at /brokers/ids/13, node already exists and owner
'937991457960493056' does not match current session '1010049473220837376'
(kafka.zk.KafkaZkClient$CheckedEphemeral)
server_b13.log.2019-01-26-22:[2019-01-26 23:53:11,841] INFO Result of znode
creation at /brokers/ids/13 is: NODEEXISTS (kafka.zk.KafkaZkClient)
{code}
We recently fixed KAFKA-7165 which could lead to the above. Perhaps you could
try 2.2.0 when it's released.
> kafka cluster not recovering - Shrinking ISR from 14,13 to 13
> (kafka.cluster.Partition) continously
> ---------------------------------------------------------------------------------------------------
>
> Key: KAFKA-7888
> URL: https://issues.apache.org/jira/browse/KAFKA-7888
> Project: Kafka
> Issue Type: Bug
> Components: controller, replication, zkclient
> Affects Versions: 2.1.0
> Environment: using kafka_2.12-2.1.0
> 3 ZKs 3 Broker cluster, using 3 boxes (1 ZK and 1 broker on each box),
> default.replication factor: 2,
> offset replication factor was 1 when the error happened, increased to 2 after
> seeing this error by reassigning-partitions.
> compression: default (producer) on broker but sending gzip from producers.
> linux (redhat) etx4 kafka logs on single local disk
> Reporter: Kemal ERDEN
> Priority: Major
> Attachments: combined.log, producer.log
>
>
> we're seeing the following repeating logs on our kafka cluster from time to
> time which seems to cause messages expiring on Producers and the cluster
> going into a non-recoverable state. The only fix seems to be to restart
> brokers.
> {{Shrinking ISR from 14,13 to 13 (kafka.cluster.Partition)}}
> {{Cached zkVersion [21] not equal to that in zookeeper, skip updating ISR
> (kafka.cluster.Partition)}}
> and later on the following log is repeated:
> {{Got user-level KeeperException when processing sessionid:0xe046aa4f8e60000
> type:setData cxid:0x2df zxid:0xa000001fd txntype:-1 reqpath:n/a Error
> Path:/brokers/topics/ucTrade/partitions/6/state Error:KeeperErrorCode =
> BadVersion for /brokers/topics/ucTrade/partitions/6/state}}
> We haven't interfered with any of the brokers/zookeepers whilst this happened.
> I've attached a combined log which represents a combination of controller,
> server and state change logs from each broker (ids 13,14 and 15, log files
> have the suffix b13, b14, b15 respectively)
> We have increased the heaps from 1g to 6g for the brokers and from 512m to 4g
> for the zookeepers since this happened but not sure if it is relevant. the ZK
> logs are unfortunately overwritten so can't provide those.
> We produce varying message sizes but some messages are relatively large (6mb)
> but we use compression on the producers (set to gzip).
> I've attached some logs from one of our producers as well.
> producer.properties that we've changed:
> spring.kafka.producer.key-serializer=org.apache.kafka.common.serialization.StringSerializer
> spring.kafka.producer.compression-type=gzip
> spring.kafka.producer.retries=5
> spring.kafka.producer.acks=-1
> spring.kafka.producer.batch-size=1048576
> spring.kafka.producer.properties.linger.ms=200
> spring.kafka.producer.properties.request.timeout.ms=600000
> spring.kafka.producer.properties.max.block.ms=240000
> spring.kafka.producer.properties.max.request.size=104857600
>
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
