[
https://issues.apache.org/jira/browse/KAFKA-3539?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17343043#comment-17343043
]
Ewen Cheslack-Postava commented on KAFKA-3539:
----------------------------------------------
[~mjsax] They aren't really duplicates. It's possible to also just have races
between whatever creates a topic and a process that tries to produce to it. I
think 3450 implicitly assumes a simple sequential model where create should
complete, and only then processes are deployed. In practice, there are lots of
convergence based systems that can do without this with the right preflight
checks (topic creation may happen in parallel to deploying an app that uses it,
but does not have permissions to create topics). They may be related, but at
least the high level description of 3450 wrt topic auto creation doesn't quite
align with this one.
[~moses.nakamura] this is a very tricky thing to get right. At first glance we
should make anything blocking (or with simple timeout) just be tied to the
producer constructor, but at that point we don't know what info to get. Adding
a queue can help, but adds overhead and really just delays the issue.
Configurable timeouts allow you to set your tolerance for blocking operations
(you're going to spend at least some time in the send() call anyway).
Alternatively, we could not wait on metadata at all and only check if it's
available, request it if it is not, and then bail with a RetriableException.
Pretty much any solution proposed so far is going to need a KIP (it's at a
minimum a behavioral change, for your option 1 it's an API addition). A good
starting point if you want to address this is to enumerate the variety of
options and their pros/cons/compatibility issues.
> KafkaProducer.send() may block even though it returns the Future
> ----------------------------------------------------------------
>
> Key: KAFKA-3539
> URL: https://issues.apache.org/jira/browse/KAFKA-3539
> Project: Kafka
> Issue Type: Bug
> Components: producer
> Reporter: Oleg Zhurakousky
> Priority: Critical
> Labels: needs-discussion, needs-kip
>
> You can get more details from the us...@kafka.apache.org by searching on the
> thread with the subject "KafkaProducer block on send".
> The bottom line is that method that returns Future must never block, since it
> essentially violates the Future contract as it was specifically designed to
> return immediately passing control back to the user to check for completion,
> cancel etc.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
