Hi All, I recently discovered the race condition where Kafka clients may request metadata from brokers who have not yet received a snapshot of the cluster metadata. At scale with ZK managed clusters we see this happen frequently when brokers are restarted, possibly due to the admin client's preference for the least loaded node. This results in unexpected behavior for some metadata requests on the Admin api. For example describeTopics will fail with an UnknownTopicOrPartition exception for topics which do exist in the cluster.
I also learned that consumers and producers ignore uninitialized metadata by detecting if the set of brokers in a metadata response is empty: https://github.com/apache/kafka/blob/trunk/clients/src/main/java/org/apache/kafka/clients/NetworkClient.java#L1187-L1191 The only Admin API that has this protection seems to be listConsumerGroups which throws a StaleMetadataException for the empty brokers condition: https://github.com/apache/kafka/blob/trunk/clients/src/main/java/org/apache/kafka/clients/admin/KafkaAdminClient.java#L3369-L3371 I'm interested in making it clearer to users of the Admin api when they should retry a metadata request because it was handled by a broker with uninitialized metadata. I think the proper fix here would be to have brokers respond with a UninitializedMetadataException when handling metadata requests if they haven't yet received a metadata snapshot. That is a big change that would need to be handled appropriately in all clients. A more immediate fix would be to change the KafkaAdminClient to *always* detect the empty brokers condition when getting a MetadataResponse and throw a StaleMetadataException or some other RetriableException. Some questions I have: 1. Does the likelihood of a broker responding with stale metadata decrease significantly or entirely when using KRAFT? I can understand not fixing this if that is the case. I tried but could not reproduce this behavior using kafka integration tests for both ZK and KRAFT. 2. Do we want to go for the proper fix or the more immediate one? 3. Would the immediate fix mentioned above, patching the KafkaAdminClient, require a KIP or should I just PR that? 4. Is StaleMetadataException the exception we want to use for the unitialized metadata case? From the docs for both StaleMetadataException and InvalidMetadataException it seems more geared toward old data, not uninitialized data. Thanks for your time and I hope this is an appropriate discussion! Chris Wildman
