There is an open JIRA for this issue (
https://issues.apache.org/jira/browse/KAFKA-3065). The PR is quite old and
needs rebasing. I will take a look at it today.
On Sun, May 15, 2016 at 6:14 AM, Jaikiran Pai <jai.forums2...@gmail.com>
wrote:
> Hello Kafka team,
>
>
> We are using 0.9.0.1 (latest stable) of Kafka server and client libraries.
> We use Java client for communicating with the Kafka installation. Our
> simple application uses a single instance of KafkaProducer (since the
> javadoc of that class states it's thread safe and recommended to be shared
> across the threads) for the lifetime of our application runtime. We seem to
> be running into a potential issue in the Kafka producer and the way it
> expects metadata for topics which it had communicated before but are no
> longer around.
>
> The use case we have, where we run into this issue is as follows:
>
> 1. Our application is sent the name of the topic to which the application
> sends a message using the KafkaProducer
> 2. The topics is short lived and after a while the topic is deleted via
> Kafka tools externally
> 3. Our application continues to run and the next time it receives a
> request to send a message to _some other topic_, it ends up running into an
> issue where it endlessly floods the logs with messages:
>
> 10:17:53,245 WARN [NetworkClient] - Error while fetching metadata with
> correlation id 122 :
> {name-of-the-topic-that-got-deleted=UNKNOWN_TOPIC_OR_PARTITION}
> 10:17:53,347 WARN [NetworkClient] - Error while fetching metadata with
> correlation id 123 :
> {name-of-the-topic-that-got-deleted=UNKNOWN_TOPIC_OR_PARTITION}
> 10:17:53,449 WARN [NetworkClient] - Error while fetching metadata with
> correlation id 124 :
> {name-of-the-topic-that-got-deleted=UNKNOWN_TOPIC_OR_PARTITION}
>
> It appears that the KafkaProducer wants to get hold of the metadata for
> this topic which we deleted externally and which we have no intention to
> communicate to anymore. These logs never stop, till we bring down the
> application or close that producer instance.
>
> This looks like an issue to me since the producer should either be aware
> that the topic got deleted and no longer request for metadata (unless there
> is an explicit call to send some message to it from the user application)
> or it should just ignore the fact that the metadata for this topic isn't
> there and move on without logging these logs (unless, again, there is an
> explicit call to send some message to the deleted topic, from the user
> application).
>
> Looking at the code in the Kafka, it appears that the "topics" set gets
> added with the topic name of the topic to which a communication is
> established by the KafkaProducer. Once added, that topic name, never gets
> removed from that set for the lifetime of that producer, even in cases like
> these where the topic is deleted and never again used with that producer.
>
> Do you think this is a bug in the Kafka code? I have a simple application
> which reproduces this easily on a 0.9.0.1 setup here
> https://gist.github.com/jaikiran/45e9ce510c259267b28821b84105a25a.
>
> Let me know if you need more details about this.
>
> -Jaikiran
>
--
Regards,
Rajini
