[
https://issues.apache.org/jira/browse/KAFKA-4385?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15641173#comment-15641173
]
ASF GitHub Bot commented on KAFKA-4385:
---------------------------------------
GitHub user yaojuncn opened a pull request:
https://github.com/apache/kafka/pull/2107
KAFKA-4385-avoid-unnecessary-meta-request-overhead
add a producer config: "metadata.fetch.max.count"
default to Integer.MAX_VALUE, then everything is the same as before;
when specified, the metadata request will be limited by both
"metadata.fetch.timeout.ms" (time limit) and "metadata.fetch.max.count"
(request count)
when "auto.create.topics.enable=false", "metadata.fetch.max.count" can be
set as 1, so that it could avoid lots of unncessary duplicate metadata
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/yaojuncn/kafka
KAFKA-4385-avoid-unnecessary-meta-request-overhead
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/kafka/pull/2107.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #2107
----
commit 20311836762f02a29b880b7edee76567b1d60c7c
Author: yaojuncn <[email protected]>
Date: 2016-11-06T05:12:22Z
KAFKA-4385-avoid-unnecessary-meta-request-overhead
add a producer config: "metadata.fetch.max.count"
default to Integer.MAX_VALUE, then everything is the same as before;
when specified, the metadata request will be limited by both
"metadata.fetch.timeout.ms" (time limit) and "metadata.fetch.max.count"
(request count)
when "auto.create.topics.enable=false", "metadata.fetch.max.count" can be
set as 1, so that it could avoid lots of unncessary duplicate metadata
----
> producer is sending too many unnecessary meta data request if the meta data
> for a topic is not available and "auto.create.topics.enable" =false
> -----------------------------------------------------------------------------------------------------------------------------------------------
>
> Key: KAFKA-4385
> URL: https://issues.apache.org/jira/browse/KAFKA-4385
> Project: Kafka
> Issue Type: Bug
> Reporter: Jun Yao
>
> All current kafka-client producer implementation (<= 0.10.1.0),
> When sending a msg to a topic, it will first check if meta data for this
> topic is available or not,
> when not available, it will set "metadata.requestUpdate()" and wait for meta
> data from brokers,
> The thing is inside "org.apache.kafka.clients.Metadata.awaitUpdate()", it's
> already doing a "while (this.version <= lastVersion)" loop waiting for new
> version response,
> So the loop inside
> "org.apache.kafka.clients.producer.KafkaProducer.waitOnMetadata() is not
> needed,
> When "auto.create.topics.enable" is false, sending msgs to a non-exist topic
> will trigger too many meta requests, everytime a metadata response is
> returned, because it does not contain the metadata for the topic, it's going
> to try again until TimeoutException is thrown;
> This is a waste and sometimes causes too much overhead when unexpected msgs
> are arrived.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
