[ https://issues.apache.org/jira/browse/KAFKA-10056?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17119015#comment-17119015 ]
Hai Lin commented on KAFKA-10056: --------------------------------- Thanks for the patch [~rsivaram], are we going to back port this to 2.4? I was running 2.4.1 when I ran into this issue. > Consumer metadata may use outdated groupSubscription that doesn't contain > newly subscribed topics > ------------------------------------------------------------------------------------------------- > > Key: KAFKA-10056 > URL: https://issues.apache.org/jira/browse/KAFKA-10056 > Project: Kafka > Issue Type: Bug > Components: consumer > Affects Versions: 2.5.0 > Reporter: Rajini Sivaram > Assignee: Rajini Sivaram > Priority: Major > Fix For: 2.6.0, 2.5.1 > > > From [~hai_lin] in KAFKA-9181: > I did notice some issue after this patch, here is what I observe. > Consumer metadata might skip first metadata update, cause grouopSubscription > is not reset. In my case, the consumer coordinator thread hijack the update > by calling newMetadataRequestAndVersion with outdated groupSubscription > before joinPrepare() happen. The groupSubscription will get reset later and > it will eventually get update later, and this won't be an issue for initial > consumer subscribe(since the groupSubscription is empty anyway), but it might > happen the following subscribe when groupSubscription is not empty. This will > create a discrepancy between subscription and groupSubscription, if any new > metadata request happened in between, metadataTopics will return outdated > group information. > > h4. The happy path > * Consumer call subscribe > Update {{needUpdated}}, bump up > {{requestVersion}} and update {{subscription}} in {{SubscriptionState}} > > {{prepareJoin()}} was call in first {{poll()}} to reset {{groupSubscription}} > -> next time when metadata update was call and {{metadataTopics()}} returns > {{subscription}} since {{groupSubscription}} is empty -> update call issue to > broker to fetch partition information for new topic > h4. In our case > * Consumer call subscribe > Update {{needUpdated}}, bump up > {{requestVersion}} and update {{subscription}}(not {{groupSubscription}}) in > {{SubscriptionState}} > Consumer Coordinator heartbeat thread call metadata > request and {{SubscriptionState}} gave away the current requestVersion and > outdated {{groupSubscription}} > making request for metadata update with > outdated subscription -> request comes back to client and since > {{requestVersion}} is up to latest, it reset {{needUpdated}} flag -> > {{joinPrepare()}} called and reset {{groupSubscription}} > no new metadata > update request follow cause {{needUpdated}} was reset -> metadata request > will happen when {{metadata.max.age}} reaches. -- This message was sent by Atlassian Jira (v8.3.4#803005)