[
https://issues.apache.org/jira/browse/KAFKA-15615?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Kirk True updated KAFKA-15615:
------------------------------
Description:
[During a review of the new
fetcher|https://github.com/apache/kafka/pull/14406#discussion_r1333393941],
[~junrao] found what appears to be an opportunity for optimization.
When a fetch response receives an error about partition leadership, fencing,
etc. a metadata refresh is triggered. However, it takes time for that refresh
to occur, and in the interim, it appears that the consumer will blindly attempt
to fetch data for the partition again, in kind of a "definition of insanity"
type of way. Ideally, the consumer would have a way to temporarily ignore those
partitions, in a way somewhat like the "pausing" approach so that they are
skipped until the metadata refresh response is fully processed.
This affects both the existing KafkaConsumer and the new PrototypeAsyncConsumer.
> Improve handling of fetching during metadata updates
> ----------------------------------------------------
>
> Key: KAFKA-15615
> URL: https://issues.apache.org/jira/browse/KAFKA-15615
> Project: Kafka
> Issue Type: Sub-task
> Reporter: Kirk True
> Assignee: Kirk True
> Priority: Major
>
> [During a review of the new
> fetcher|https://github.com/apache/kafka/pull/14406#discussion_r1333393941],
> [~junrao] found what appears to be an opportunity for optimization.
> When a fetch response receives an error about partition leadership, fencing,
> etc. a metadata refresh is triggered. However, it takes time for that refresh
> to occur, and in the interim, it appears that the consumer will blindly
> attempt to fetch data for the partition again, in kind of a "definition of
> insanity" type of way. Ideally, the consumer would have a way to temporarily
> ignore those partitions, in a way somewhat like the "pausing" approach so
> that they are skipped until the metadata refresh response is fully processed.
> This affects both the existing KafkaConsumer and the new
> PrototypeAsyncConsumer.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
