[
https://issues.apache.org/jira/browse/KAFKA-1120?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16142304#comment-16142304
]
James Cheng commented on KAFKA-1120:
------------------------------------
Onur, while you are figuring out a fix, can you recommend anything we can do to
avoid triggering the scenario? We have high partition counts, and so are
encountering long ControlledShutdown events, which means we are hitting this
quite often.
It seems like the bug gets triggered when the broker gets back into zookeeper
before the controlled shutdown is finished being processed. So, I either make
ControlledShutdown get processed faster, or I make it so that the broker gets
into zookeeper slower. Would that be right?
I can make ControlledShutdown get processed faster by reducing partition count,
for example.
I can make the broker get into zookeeper slower either by making sure it takes
longer to shutdown (increasing controlled.shutdown.retry.backoff.ms?) or
delaying startup ("sleep 60 && ./bin/kafka-server-start.sh")
> Controller could miss a broker state change
> --------------------------------------------
>
> Key: KAFKA-1120
> URL: https://issues.apache.org/jira/browse/KAFKA-1120
> Project: Kafka
> Issue Type: Bug
> Components: core
> Affects Versions: 0.8.1
> Reporter: Jun Rao
> Labels: reliability
> Fix For: 1.0.0
>
>
> When the controller is in the middle of processing a task (e.g., preferred
> leader election, broker change), it holds a controller lock. During this
> time, a broker could have de-registered and re-registered itself in ZK. After
> the controller finishes processing the current task, it will start processing
> the logic in the broker change listener. However, it will see no broker
> change and therefore won't do anything to the restarted broker. This broker
> will be in a weird state since the controller doesn't inform it to become the
> leader of any partition. Yet, the cached metadata in other brokers could
> still list that broker as the leader for some partitions. Client requests
> routed to that broker will then get a TopicOrPartitionNotExistException. This
> broker will continue to be in this bad state until it's restarted again.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
