[
https://issues.apache.org/jira/browse/KAFKA-4631?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15823429#comment-15823429
]
Ewen Cheslack-Postava commented on KAFKA-4631:
----------------------------------------------
Would this be a constant timeout or yet another config? And especially if the
former, tied to a fixed # of retries? Or maybe retries up until the first
"normal" metadata refresh?
I kind of like this idea, but also feel like it's just sort of papering over
bigger underlying issues (i.e. the fact that you can't know when topic creation
metadata updates are actually complete). Those could be limitations we simply
want to accept (validating the entire cluster has updated metadata might be
expensive) or something we want to address in a better way (e.g. ensure that if
a metadata request shows a partition isn't known, immediately do a retry but
make sure it goes to the controller).
> Refresh consumer metadata more frequently for unknown subscribed topics
> -----------------------------------------------------------------------
>
> Key: KAFKA-4631
> URL: https://issues.apache.org/jira/browse/KAFKA-4631
> Project: Kafka
> Issue Type: Bug
> Reporter: Jason Gustafson
> Assignee: Matthias J. Sax
> Fix For: 0.10.2.0
>
>
> By default, the consumer refreshes metadata every 5 minutes. In testing, it
> can often happen that a topic is created at about the same time that the
> consumer is started. In the worst case, creation finishes after the consumer
> fetches metadata, and the test must wait 5 minutes for the consumer to
> refresh metadata in order to discover the topic. To address this problem,
> users can decrease the metadata refresh interval, but this means more
> frequent refreshes even after all topics are known. An improvement would be
> to internally let the consumer fetch metadata more frequently when the
> consumer encounters unknown topics. Perhaps every 5-10 seconds would be
> reasonable, for example.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
