Lucas Brutschy created KAFKA-16198:
--------------------------------------
Summary: Reconciliation may lose partitions when topic metadata is
delayed
Key: KAFKA-16198
URL: https://issues.apache.org/jira/browse/KAFKA-16198
Project: Kafka
Issue Type: Bug
Components: clients, consumer
Reporter: Lucas Brutschy
Assignee: Lucas Brutschy
Fix For: 3.8.0
The current reconciliation code in `AsyncKafkaConsumer`s `MembershipManager`
may lose part of the server-provided assignment when metadata is delayed. The
reason is incorrect handling of partially resolved topic names, as in this
example:
* We get assigned {{T1-1}} and {{T2-1}}
* We reconcile {{{}T1-1{}}}, {{T2-1}} remains in {{assignmentUnresolved}}
since the topic id {{T2}} is not known yet
* We get new cluster metadata, which includes {{{}T2{}}}, so {{T2-1}} is moved
to {{assignmentReadyToReconcile}}
* We call {{reconcile}} -- {{T2-1}} is now treated as the full assignment, so
{{T1-1}} is being revoked
* We end up with assignment {{T2-1, which is inconsistent with the broker-side
target assignment.}}
Generally, this seems to be a problem around semantics of the internal
collections `assignmentUnresolved` and `assignmentReadyToReconcile`. Absence of
a topic in `assignmentReadyToReconcile` may mean either revocation of the topic
partition(s), or unavailability of a topic name for the topic.
Internal state with simpler and correct invariant could be achieved by using a
single collection `currentTargetAssignment` which is based on topic IDs and
always corresponds to the latest assignment received from the broker. During
every attempted reconciliation, all topic IDs will be resolved from the local
cache, which should not introduce a lot of overhead. `assignmentUnresolved` and
`assignmentReadyToReconcile` are removed.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
