[
https://issues.apache.org/jira/browse/KAFKA-3236?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15176728#comment-15176728
]
Thomas Graves commented on KAFKA-3236:
--------------------------------------
Yeah it would be possible to do it outside in our code its just not as nice.
>From my understanding the metadata gather thread is ok to block. it needs the
>metadata to do anything with that topic. Note this is one of our users I'm
>representing here so I'm just going by what they are saying. I think its ok
>to block here because the send() thread can still go ahead and send to the
>topics it already has metadata on.
Note that also send() only blocks waiting on the metadata if its not already
cached, in waitOnMetadata:
while(metadata.fetch().partitionsForTopic(topic) == null) {
so by our metadata thread making sure it is there first I don't think send()
will block waiting on metadata at all. Then I think the metadata gets updated
when needed by internal kafka threads and doesn't block the sender.
We do also expect the buffer full to be much more frequent give the metadata
should be cached anyway.
> Honor Producer Configuration "block.on.buffer.full"
> ---------------------------------------------------
>
> Key: KAFKA-3236
> URL: https://issues.apache.org/jira/browse/KAFKA-3236
> Project: Kafka
> Issue Type: Improvement
> Components: producer
> Affects Versions: 0.9.0.0
> Reporter: Thomas Graves
> Assignee: Thomas Graves
>
> In Kafka-0.9, "max.block.ms" is used to control how long the following
> methods will block.
> KafkaProducer.send() when
> * Buffer is full
> * Metadata is unavailable
> KafkaProducer.partitionsFor() when
> * Metadata is unavailable
> However when "block.on.buffer.full" is set to false, "max.block.ms" is in
> effect whenever a buffer is requested/allocated from the Producer BufferPool.
> Instead it should throw a BufferExhaustedException without waiting for
> "max.block.ms"
> This is particulary useful if a producer application does not wish to block
> at all on KafkaProducer.send() . We avoid waiting on KafkaProducer.send()
> when metadata is unavailable by invoking send() only if the producer instance
> has fetched the metadata for the topic in a different thread using the same
> producer instance. However "max.block.ms" is still required to specify a
> timeout for bootstrapping the metadata fetch.
> We should resolve this limitation by decoupling "max.block.ms" and
> "block.on.buffer.full".
> * "max.block.ms" will be used exclusively for fetching metadata when
> "block.on.buffer.full" = false (in pure non-blocking mode )
> * "max.block.ms" will be applicable to both fetching metadata as well as
> buffer allocation when "block.on.buffer.full = true
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
