Hello Kafka team,

We are using 0.9.0.1 (latest stable) of Kafka server and client libraries. We use Java client for communicating with the Kafka installation. Our simple application uses a single instance of KafkaProducer (since the javadoc of that class states it's thread safe and recommended to be shared across the threads) for the lifetime of our application runtime. We seem to be running into a potential issue in the Kafka producer and the way it expects metadata for topics which it had communicated before but are no longer around.

The use case we have, where we run into this issue is as follows:

1. Our application is sent the name of the topic to which the application sends a message using the KafkaProducer 2. The topics is short lived and after a while the topic is deleted via Kafka tools externally 3. Our application continues to run and the next time it receives a request to send a message to _some other topic_, it ends up running into an issue where it endlessly floods the logs with messages:

10:17:53,245 WARN [NetworkClient] - Error while fetching metadata with correlation id 122 : {name-of-the-topic-that-got-deleted=UNKNOWN_TOPIC_OR_PARTITION} 10:17:53,347 WARN [NetworkClient] - Error while fetching metadata with correlation id 123 : {name-of-the-topic-that-got-deleted=UNKNOWN_TOPIC_OR_PARTITION} 10:17:53,449 WARN [NetworkClient] - Error while fetching metadata with correlation id 124 : {name-of-the-topic-that-got-deleted=UNKNOWN_TOPIC_OR_PARTITION}

It appears that the KafkaProducer wants to get hold of the metadata for this topic which we deleted externally and which we have no intention to communicate to anymore. These logs never stop, till we bring down the application or close that producer instance.

This looks like an issue to me since the producer should either be aware that the topic got deleted and no longer request for metadata (unless there is an explicit call to send some message to it from the user application) or it should just ignore the fact that the metadata for this topic isn't there and move on without logging these logs (unless, again, there is an explicit call to send some message to the deleted topic, from the user application).

Looking at the code in the Kafka, it appears that the "topics" set gets added with the topic name of the topic to which a communication is established by the KafkaProducer. Once added, that topic name, never gets removed from that set for the lifetime of that producer, even in cases like these where the topic is deleted and never again used with that producer.

Do you think this is a bug in the Kafka code? I have a simple application which reproduces this easily on a 0.9.0.1 setup here https://gist.github.com/jaikiran/45e9ce510c259267b28821b84105a25a.

Let me know if you need more details about this.

-Jaikiran

Reply via email to