[
https://issues.apache.org/jira/browse/KAFKA-2948?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15045372#comment-15045372
]
Jiangjie Qin commented on KAFKA-2948:
-------------------------------------
[~rsivaram] As you pointed out we never remove the topics set in producer
metadata. I am not sure if removing the topic from the set when we see
UNKNOWN_TOPIC_OR_PARTITION error code is the right way to fix this, because
UNKNOWN_TOPIC_OR_PARTITION can also occur in other cases such as partition
reassignment, the producer is supposed to retry in this case.
Maybe a TTL is a better solution here. e.g. If the producer hasn't sent data to
a particular topic since last metadata refresh, we can remove the topic from
metadata topic set on next metadata refresh.
> Kafka producer does not cope well with topic deletions
> ------------------------------------------------------
>
> Key: KAFKA-2948
> URL: https://issues.apache.org/jira/browse/KAFKA-2948
> Project: Kafka
> Issue Type: Bug
> Components: producer
> Affects Versions: 0.9.0.0
> Reporter: Rajini Sivaram
> Assignee: Rajini Sivaram
>
> Kafka producer gets metadata for topics when send is invoked and thereafter
> it attempts to keep the metadata up-to-date without any explicit requests
> from the client. This works well in static environments, but when topics are
> added or deleted, list of topics in Metadata grows but never shrinks. Apart
> from being a memory leak, this results in constant requests for metadata for
> deleted topics.
> We are running into this issue with the Confluent REST server where topic
> deletion from tests are filling up logs with warnings about unknown topics.
> Auto-create is turned off in our Kafka cluster.
> I am happy to provide a fix, but am not sure what the right fix is. Does it
> make sense to remove topics from the metadata list when
> UNKNOWN_TOPIC_OR_PARTITION response is received if there are no outstanding
> sends? It doesn't look very straightforward to do this, so any alternative
> suggestions are welcome.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
