[
https://issues.apache.org/jira/browse/KAFKA-3539?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16800714#comment-16800714
]
radai rosenblatt commented on KAFKA-3539:
-----------------------------------------
IIUC, the root of the problem is the kafka producer stores compressed batches
of msgs, in a map keyed by the partition these msgs are intended for.
since without metadata there's no knowing the layout of a topic the producer
cant tell where to "place" a msg, which is why it blocks on no metadata.
one possible solution would be to have an "unknown" msg bucket (with some
finite capacity) where msgs of unknown destination go. the biggest issue with
this is that those msgs cannot be compressed (as kafka compresses batches, not
individual msgs, and there's no guarantee that everything in the unknown bucket
will go into the same batch).
once metadata is obtained the "unknown bucket" would need to be iterated over,
and the msgs deposited (and compressed) into the correct queues. this would
need to happen when metadata arrives and before any new msgs are allowed into
the producer (to not violate order)
> KafkaProducer.send() may block even though it returns the Future
> ----------------------------------------------------------------
>
> Key: KAFKA-3539
> URL: https://issues.apache.org/jira/browse/KAFKA-3539
> Project: Kafka
> Issue Type: Bug
> Components: producer
> Reporter: Oleg Zhurakousky
> Priority: Critical
>
> You can get more details from the [email protected] by searching on the
> thread with the subject "KafkaProducer block on send".
> The bottom line is that method that returns Future must never block, since it
> essentially violates the Future contract as it was specifically designed to
> return immediately passing control back to the user to check for completion,
> cancel etc.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)