[
https://issues.apache.org/jira/browse/KAFKA-4385?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15641989#comment-15641989
]
Jiangjie Qin commented on KAFKA-4385:
-------------------------------------
[~Jun Yao] Is this really a problem? The loop in
{{KafkaProducer.waitOnMetadata()}} is to ensure the metadata of a particular
topic is available. It is different from the while loop in
{{Metadata.awaitMetadataUpdate()}} which only ensure the metadata is refreshed.
The reason we need to send multiple MetadataRequest when producing to a new
topic is that in Kafka the topic creation is asynchronous (we all agree it is
confusing). If the producer relies on the auto topic creation when producing to
a new topic, likely the first metadata response will not have the metadata of
the new topic included. That is why we need to refresh the metadata again in
the {{KafkaProducer.waitOnMetadata()}}. If the topic creation is synchronous,
then we may not need that loop.
> producer is sending too many unnecessary meta data request if the meta data
> for a topic is not available and "auto.create.topics.enable" =false
> -----------------------------------------------------------------------------------------------------------------------------------------------
>
> Key: KAFKA-4385
> URL: https://issues.apache.org/jira/browse/KAFKA-4385
> Project: Kafka
> Issue Type: Bug
> Reporter: Jun Yao
>
> All current kafka-client producer implementation (<= 0.10.1.0),
> When sending a msg to a topic, it will first check if meta data for this
> topic is available or not,
> when not available, it will set "metadata.requestUpdate()" and wait for meta
> data from brokers,
> The thing is inside "org.apache.kafka.clients.Metadata.awaitUpdate()", it's
> already doing a "while (this.version <= lastVersion)" loop waiting for new
> version response,
> So the loop inside
> "org.apache.kafka.clients.producer.KafkaProducer.waitOnMetadata() is not
> needed,
> When "auto.create.topics.enable" is false, sending msgs to a non-exist topic
> will trigger too many meta requests, everytime a metadata response is
> returned, because it does not contain the metadata for the topic, it's going
> to try again until TimeoutException is thrown;
> This is a waste and sometimes causes too much overhead when unexpected msgs
> are arrived.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
