Sorry, that was the wrong JIRA. https://issues.apache.org/jira/browse/KAFKA-2948 is the one which addresses this issue.
On Mon, May 16, 2016 at 7:52 AM, Rajini Sivaram < rajinisiva...@googlemail.com> wrote: > There is an open JIRA for this issue ( > https://issues.apache.org/jira/browse/KAFKA-3065). The PR is quite old > and needs rebasing. I will take a look at it today. > > On Sun, May 15, 2016 at 6:14 AM, Jaikiran Pai <jai.forums2...@gmail.com> > wrote: > >> Hello Kafka team, >> >> >> We are using 0.9.0.1 (latest stable) of Kafka server and client >> libraries. We use Java client for communicating with the Kafka >> installation. Our simple application uses a single instance of >> KafkaProducer (since the javadoc of that class states it's thread safe and >> recommended to be shared across the threads) for the lifetime of our >> application runtime. We seem to be running into a potential issue in the >> Kafka producer and the way it expects metadata for topics which it had >> communicated before but are no longer around. >> >> The use case we have, where we run into this issue is as follows: >> >> 1. Our application is sent the name of the topic to which the application >> sends a message using the KafkaProducer >> 2. The topics is short lived and after a while the topic is deleted via >> Kafka tools externally >> 3. Our application continues to run and the next time it receives a >> request to send a message to _some other topic_, it ends up running into an >> issue where it endlessly floods the logs with messages: >> >> 10:17:53,245 WARN [NetworkClient] - Error while fetching metadata with >> correlation id 122 : >> {name-of-the-topic-that-got-deleted=UNKNOWN_TOPIC_OR_PARTITION} >> 10:17:53,347 WARN [NetworkClient] - Error while fetching metadata with >> correlation id 123 : >> {name-of-the-topic-that-got-deleted=UNKNOWN_TOPIC_OR_PARTITION} >> 10:17:53,449 WARN [NetworkClient] - Error while fetching metadata with >> correlation id 124 : >> {name-of-the-topic-that-got-deleted=UNKNOWN_TOPIC_OR_PARTITION} >> >> It appears that the KafkaProducer wants to get hold of the metadata for >> this topic which we deleted externally and which we have no intention to >> communicate to anymore. These logs never stop, till we bring down the >> application or close that producer instance. >> >> This looks like an issue to me since the producer should either be aware >> that the topic got deleted and no longer request for metadata (unless there >> is an explicit call to send some message to it from the user application) >> or it should just ignore the fact that the metadata for this topic isn't >> there and move on without logging these logs (unless, again, there is an >> explicit call to send some message to the deleted topic, from the user >> application). >> >> Looking at the code in the Kafka, it appears that the "topics" set gets >> added with the topic name of the topic to which a communication is >> established by the KafkaProducer. Once added, that topic name, never gets >> removed from that set for the lifetime of that producer, even in cases like >> these where the topic is deleted and never again used with that producer. >> >> Do you think this is a bug in the Kafka code? I have a simple application >> which reproduces this easily on a 0.9.0.1 setup here >> https://gist.github.com/jaikiran/45e9ce510c259267b28821b84105a25a. >> >> Let me know if you need more details about this. >> >> -Jaikiran >> > > > > -- > Regards, > > Rajini > -- Regards, Rajini
