[
https://issues.apache.org/jira/browse/KAFKA-3949?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Guozhang Wang updated KAFKA-3949:
---------------------------------
Labels: new-consumer-threading-should-fix (was: )
> Consumer topic subscription change may be ignored if a rebalance is in
> progress
> -------------------------------------------------------------------------------
>
> Key: KAFKA-3949
> URL: https://issues.apache.org/jira/browse/KAFKA-3949
> Project: Kafka
> Issue Type: Bug
> Components: consumer
> Affects Versions: 0.9.0.1, 0.10.0.0
> Reporter: Jason Gustafson
> Assignee: Jason Gustafson
> Priority: Major
> Labels: new-consumer-threading-should-fix
> Fix For: 0.10.1.0
>
>
> The consumer's regex subscription works by matching all topics fetched from a
> metadata update against the provided pattern. When a new topic is created or
> an old topic is deleted, we update the list of subscribed topics and request
> a rebalance by setting the {{needsPartitionAssignment}} flag inside
> {{SubscriptionState}}. On the next call to {{poll()}}, the consumer will
> observe the flag and begin the rebalance by sending a JoinGroup. The problem
> is that it does not account for the fact that a rebalance could already be in
> progress at the time the metadata is updated. This causes the following
> sequence:
> 1. Rebalance begins (needsPartitionAssignment is set True)
> 2. Metadata max age expires and and update is triggered
> 3. Update returns and causes a topic subscription change
> (needsPartitionAssignment set again to True).
> 4. Rebalance completes (needsPartitionAssignment is set False)
> In this situation, we will not request a new rebalance which will prevent us
> from receiving an assignment from any topics added to the consumer's
> subscription when the metadata was updated. This state will persist until
> another event causes the group to rebalance.
> A related problem may occur if a rebalance is interrupted with the wakeup()
> API, and the user calls subscribe(topics) with a change to the subscription
> set.
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
