Hello Kafka team,
We are using 0.9.0.1 (latest stable) of Kafka server and client
libraries. We use Java client for communicating with the Kafka
installation. Our simple application uses a single instance of
KafkaProducer (since the javadoc of that class states it's thread safe
and recommended to be shared across the threads) for the lifetime of our
application runtime. We seem to be running into a potential issue in the
Kafka producer and the way it expects metadata for topics which it had
communicated before but are no longer around.
The use case we have, where we run into this issue is as follows:
1. Our application is sent the name of the topic to which the
application sends a message using the KafkaProducer
2. The topics is short lived and after a while the topic is deleted via
Kafka tools externally
3. Our application continues to run and the next time it receives a
request to send a message to _some other topic_, it ends up running into
an issue where it endlessly floods the logs with messages:
10:17:53,245 WARN [NetworkClient] - Error while fetching metadata with
correlation id 122 :
{name-of-the-topic-that-got-deleted=UNKNOWN_TOPIC_OR_PARTITION}
10:17:53,347 WARN [NetworkClient] - Error while fetching metadata with
correlation id 123 :
{name-of-the-topic-that-got-deleted=UNKNOWN_TOPIC_OR_PARTITION}
10:17:53,449 WARN [NetworkClient] - Error while fetching metadata with
correlation id 124 :
{name-of-the-topic-that-got-deleted=UNKNOWN_TOPIC_OR_PARTITION}
It appears that the KafkaProducer wants to get hold of the metadata for
this topic which we deleted externally and which we have no intention to
communicate to anymore. These logs never stop, till we bring down the
application or close that producer instance.
This looks like an issue to me since the producer should either be aware
that the topic got deleted and no longer request for metadata (unless
there is an explicit call to send some message to it from the user
application) or it should just ignore the fact that the metadata for
this topic isn't there and move on without logging these logs (unless,
again, there is an explicit call to send some message to the deleted
topic, from the user application).
Looking at the code in the Kafka, it appears that the "topics" set gets
added with the topic name of the topic to which a communication is
established by the KafkaProducer. Once added, that topic name, never
gets removed from that set for the lifetime of that producer, even in
cases like these where the topic is deleted and never again used with
that producer.
Do you think this is a bug in the Kafka code? I have a simple
application which reproduces this easily on a 0.9.0.1 setup here
https://gist.github.com/jaikiran/45e9ce510c259267b28821b84105a25a.
Let me know if you need more details about this.
-Jaikiran