Lianet Magrans created KAFKA-18690:
--------------------------------------
Summary: New consumer doesn't updateLatestMetadata for topics
matching new regex
Key: KAFKA-18690
URL: https://issues.apache.org/jira/browse/KAFKA-18690
Project: Kafka
Issue Type: Bug
Components: consumer
Reporter: Lianet Magrans
Assignee: Sean Quah
Fix For: 4.0.0
When the consumer subscribes to the new RE2J regex resolved on the broker, it's
not keeping the leader and epoch info from the metadata response, so not able
to properly update the fetch positions.
After getting the assignment with the topics matching the regex, and receiving
the topics in a metadata response, the consumer is not retaining topics that
are assigned with Re2Jpattern so fails with
Leader for partition {} is unknown for fetching offset {}
This seems to be the reason behind it, not retaining topics assigned with RE2J
[https://github.com/apache/kafka/blob/d19b60521001697bc9d518cfc3d585e7b78d15a3/clients/src/main/java/org/apache/kafka/clients/consumer/internals/ConsumerMetadata.java#L97
|https://github.com/apache/kafka/blob/d19b60521001697bc9d518cfc3d585e7b78d15a3/clients/src/main/java/org/apache/kafka/clients/consumer/internals/ConsumerMetadata.java#L97]
So when handling the metadata response that includes the matching topics it
will never make it to update the leader/epoch info.
[https://github.com/apache/kafka/blob/d19b60521001697bc9d518cfc3d585e7b78d15a3/clients/src/main/java/org/apache/kafka/clients/Metadata.java#L521]
To validate with should end with a test similar to
[https://github.com/apache/kafka/blob/d19b60521001697bc9d518cfc3d585e7b78d15a3/core/src/test/scala/integration/kafka/api/PlaintextConsumerSubscriptionTest.scala#L251]
but including consumption.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
