[ 
https://issues.apache.org/jira/browse/KAFKA-3949?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Guozhang Wang updated KAFKA-3949:
---------------------------------
    Labels: new-consumer-threading-should-fix  (was: )

> Consumer topic subscription change may be ignored if a rebalance is in 
> progress
> -------------------------------------------------------------------------------
>
>                 Key: KAFKA-3949
>                 URL: https://issues.apache.org/jira/browse/KAFKA-3949
>             Project: Kafka
>          Issue Type: Bug
>          Components: consumer
>    Affects Versions: 0.9.0.1, 0.10.0.0
>            Reporter: Jason Gustafson
>            Assignee: Jason Gustafson
>            Priority: Major
>              Labels: new-consumer-threading-should-fix
>             Fix For: 0.10.1.0
>
>
> The consumer's regex subscription works by matching all topics fetched from a 
> metadata update against the provided pattern. When a new topic is created or 
> an old topic is deleted, we update the list of subscribed topics and request 
> a rebalance by setting the {{needsPartitionAssignment}} flag inside 
> {{SubscriptionState}}. On the next call to {{poll()}}, the consumer will 
> observe the flag and begin the rebalance by sending a JoinGroup. The problem 
> is that it does not account for the fact that a rebalance could already be in 
> progress at the time the metadata is updated. This causes the following 
> sequence:
> 1. Rebalance begins (needsPartitionAssignment is set True)
> 2. Metadata max age expires and and update is triggered
> 3. Update returns and causes a topic subscription change 
> (needsPartitionAssignment set again to True).
> 4. Rebalance completes (needsPartitionAssignment is set False)
> In this situation, we will not request a new rebalance which will prevent us 
> from receiving an assignment from any topics added to the consumer's 
> subscription when the metadata was updated. This state will persist until 
> another event causes the group to rebalance.
> A related problem may occur if a rebalance is interrupted with the wakeup() 
> API, and the user calls subscribe(topics) with a change to the subscription 
> set. 



--
This message was sent by Atlassian Jira
(v8.20.1#820001)

Reply via email to