Sean Quah created KAFKA-19431:
---------------------------------
Summary: Stronger assignment consistency with subscription for
consumer groups
Key: KAFKA-19431
URL: https://issues.apache.org/jira/browse/KAFKA-19431
Project: Kafka
Issue Type: Improvement
Components: group-coordinator
Reporter: Sean Quah
Assignee: Sean Quah
Currently, consumer group assignments are eventually consistent with
subscriptions: when a member has unrevoked partitions, it is not allowed to
reconcile with the latest target assignment. If a member with unrevoked
partitions shrinks its subscription, it may observe assignments from the broker
containing topics it is no longer subscribed to.
If we wanted to, we could tighten this up at the cost of extra CPU time. Note
that it's not feasible to close the gap for regex subscriptions, since there
will always be a window when the regex is not yet resolved and we cannot tell
whether a topic is part of the subscription.
One way to do this would be to update {{CurrentAssignmentBuilder}} and
* Add a {{MetadataImage}} and map of resolved regexes
* Define the set of subscribed topic uuids as the union of the topic name
subscription and resolved regex topic names, like how
{{TargetAssignmentBuilder}} does it.
** When the regex is unresolved, we can’t know which topics are part of the
subscription. We treat all topics as part of the subscription, so that we do
not filter out any topics.
* Update the loop over topics in {{computeNextAssignment}} to treat the
assigned partitions as an empty set when the topic is not part of the
subscription.
* Do not advance the member epoch past the target assignment epoch when
exiting the {{UNREVOKED_PARTITIONS}} state.
* Define a {{computeCurrentAssignment}} method that drops any unsubscribed
topics from the member’s current assignment and transitions the member to
{{UNREVOKED_PARTITIONS}} if any topics were dropped.
* Use computeCurrentAssignment on the other {{UNREVOKED_PARTITIONS}} path.
Additionally, if we ever end up with asynchronous assignors (such as
client-computed ones),
* We add a new flag to {{maybeReconcile}} called
{{forceSubscriptionConsistency}}. When the flag is set, we run the
{{CurrentAssignmentBuilder}} even when reconciled to the target assignment,
since the target assignment can lag behind the group epoch.
* Use {{computeCurrentAssignment}} on all {{CurrentAssignmentBuilder}} paths
that do not use computeNextAssignment.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
