[ https://issues.apache.org/jira/browse/KAFKA-2948?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15047349#comment-15047349 ]
Rajini Sivaram commented on KAFKA-2948: --------------------------------------- [~mgharat] [~becket_qin] The code that I am testing at the moment uses the current config `metadata.max.age.ms`. If no messages are sent to a topic for this interval, then the topic is removed from the metadata set. Subsequent send will add it back to the set. I am also marking the topic for delete if a send fails because no metadata was available for a topic, to limit the number of retries for deleted topics. Will submit a PR later today for review. > Kafka producer does not cope well with topic deletions > ------------------------------------------------------ > > Key: KAFKA-2948 > URL: https://issues.apache.org/jira/browse/KAFKA-2948 > Project: Kafka > Issue Type: Bug > Components: producer > Affects Versions: 0.9.0.0 > Reporter: Rajini Sivaram > Assignee: Rajini Sivaram > > Kafka producer gets metadata for topics when send is invoked and thereafter > it attempts to keep the metadata up-to-date without any explicit requests > from the client. This works well in static environments, but when topics are > added or deleted, list of topics in Metadata grows but never shrinks. Apart > from being a memory leak, this results in constant requests for metadata for > deleted topics. > We are running into this issue with the Confluent REST server where topic > deletion from tests are filling up logs with warnings about unknown topics. > Auto-create is turned off in our Kafka cluster. > I am happy to provide a fix, but am not sure what the right fix is. Does it > make sense to remove topics from the metadata list when > UNKNOWN_TOPIC_OR_PARTITION response is received if there are no outstanding > sends? It doesn't look very straightforward to do this, so any alternative > suggestions are welcome. -- This message was sent by Atlassian JIRA (v6.3.4#6332)