On Monday 16 May 2016 04:52 PM, Rajini Sivaram wrote:
Hi Jaikiran,
1) If you delete a topic with no outstanding sends, you will see one WARN
log entry and the topic is immediately removed from the metadata. If you
delete a topic with outstanding sends, you will see a sequence of WARN logs
until the outstanding send requests expire
That's reasonable.
2) The 0.10.0.0 release is imminent and I think it would be too late to
include this.
Fair enough.
Thank you for the quick and clear response.
-Jaikiran
On Mon, May 16, 2016 at 11:59 AM, Jaikiran Pai <jai.forums2...@gmail.com>
wrote:
Thank you for looking into this Rajini.
A few questions related to this:
1) Looking at that patch, it looks like that WARN message will still be
logged for a "few" times till that topic is removed from the Set maintained
in the Metadata. Is my understanding correct? If yes, would it be possible
to log that message at DEBUG (or lower level) if the producer send is
initiated for a different topic and not for the topic which isn't existent?
That was the application where this gets logged doesn't really have to be
bothered by it. It does make sense to error out or WARN about the missing
topic only if the send was issued against that topic.
2) I see that the JIRA https://issues.apache.org/jira/browse/KAFKA-2948
is marked for 0.10.1.0. I haven't been following the Kafka release plans.
Does this mean that there won't be a 0.9.x release any more? I see a
0.10.0.0 release candidate being voted up. Is 0.10.0.0 the next planned
release? If yes, is there any chance this one can make into that release?
Or will we have to wait for 0.10.1.0 for this to be available in a release?
If that's the case, our application might have to change the log level
setting in our logging configuration for NetworkClient to use ERROR level
since this message completely messes up the entire application logs. I had
a quick look at the NetworkClient code on 0.9.0 branch
https://github.com/apache/kafka/blob/0.9.0/clients/src/main/java/org/apache/kafka/clients/NetworkClient.java
and I think we should be fine if we bumped up the log level of that class
in our logging config to be ERROR, since it doesn't log anything real
useful below that level from that class.
-Jaikiran
On Monday 16 May 2016 02:30 PM, Rajini Sivaram wrote:
Sorry, that was the wrong JIRA.
https://issues.apache.org/jira/browse/KAFKA-2948 is the one which
addresses
this issue.
On Mon, May 16, 2016 at 7:52 AM, Rajini Sivaram <
rajinisiva...@googlemail.com> wrote:
There is an open JIRA for this issue (
https://issues.apache.org/jira/browse/KAFKA-3065). The PR is quite old
and needs rebasing. I will take a look at it today.
On Sun, May 15, 2016 at 6:14 AM, Jaikiran Pai <jai.forums2...@gmail.com>
wrote:
Hello Kafka team,
We are using 0.9.0.1 (latest stable) of Kafka server and client
libraries. We use Java client for communicating with the Kafka
installation. Our simple application uses a single instance of
KafkaProducer (since the javadoc of that class states it's thread safe
and
recommended to be shared across the threads) for the lifetime of our
application runtime. We seem to be running into a potential issue in the
Kafka producer and the way it expects metadata for topics which it had
communicated before but are no longer around.
The use case we have, where we run into this issue is as follows:
1. Our application is sent the name of the topic to which the
application
sends a message using the KafkaProducer
2. The topics is short lived and after a while the topic is deleted via
Kafka tools externally
3. Our application continues to run and the next time it receives a
request to send a message to _some other topic_, it ends up running
into an
issue where it endlessly floods the logs with messages:
10:17:53,245 WARN [NetworkClient] - Error while fetching metadata with
correlation id 122 :
{name-of-the-topic-that-got-deleted=UNKNOWN_TOPIC_OR_PARTITION}
10:17:53,347 WARN [NetworkClient] - Error while fetching metadata with
correlation id 123 :
{name-of-the-topic-that-got-deleted=UNKNOWN_TOPIC_OR_PARTITION}
10:17:53,449 WARN [NetworkClient] - Error while fetching metadata with
correlation id 124 :
{name-of-the-topic-that-got-deleted=UNKNOWN_TOPIC_OR_PARTITION}
It appears that the KafkaProducer wants to get hold of the metadata for
this topic which we deleted externally and which we have no intention to
communicate to anymore. These logs never stop, till we bring down the
application or close that producer instance.
This looks like an issue to me since the producer should either be aware
that the topic got deleted and no longer request for metadata (unless
there
is an explicit call to send some message to it from the user
application)
or it should just ignore the fact that the metadata for this topic isn't
there and move on without logging these logs (unless, again, there is an
explicit call to send some message to the deleted topic, from the user
application).
Looking at the code in the Kafka, it appears that the "topics" set gets
added with the topic name of the topic to which a communication is
established by the KafkaProducer. Once added, that topic name, never
gets
removed from that set for the lifetime of that producer, even in cases
like
these where the topic is deleted and never again used with that
producer.
Do you think this is a bug in the Kafka code? I have a simple
application
which reproduces this easily on a 0.9.0.1 setup here
https://gist.github.com/jaikiran/45e9ce510c259267b28821b84105a25a.
Let me know if you need more details about this.
-Jaikiran
--
Regards,
Rajini
