[
https://issues.apache.org/jira/browse/KAFKA-2168?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14562061#comment-14562061
]
Neha Narkhede commented on KAFKA-2168:
--------------------------------------
There are tradeoffs to having multiple threads per consumer instance vs having
a consumer instance per thread. The consumer code is simpler in the latter
design, the throughput is better but the # of TCP connections are fewer in the
former design. Some of the concerns [~ewencp] brings up above can be mitigated
if there is a separate consumer instance per user thread and others can be
mitigated by the user picking the right timeout on poll() that they are
comfortable blocking on. All of this would mean explicitly stating that the
consumer APIs are not threadsafe and that the user should create multiple
consumer instances across threads instead of sharing one. We still need to make
sure close() can be called from a separate thread as [~ewencp] correctly points
out, though the change isn't large if we go down this route.
It seems like it is simpler to stick to the original intention of the design
and not share consumer instances across threads?
> New consumer poll() can block other calls like position(), commit(), and
> close() indefinitely
> ---------------------------------------------------------------------------------------------
>
> Key: KAFKA-2168
> URL: https://issues.apache.org/jira/browse/KAFKA-2168
> Project: Kafka
> Issue Type: Bug
> Components: clients, consumer
> Reporter: Ewen Cheslack-Postava
> Assignee: Jason Gustafson
>
> The new consumer is currently using very coarse-grained synchronization. For
> most methods this isn't a problem since they finish quickly once the lock is
> acquired, but poll() might run for a long time (and commonly will since
> polling with long timeouts is a normal use case). This means any operations
> invoked from another thread may block until the poll() call completes.
> Some example use cases where this can be a problem:
> * A shutdown hook is registered to trigger shutdown and invokes close(). It
> gets invoked from another thread and blocks indefinitely.
> * User wants to manage offset commit themselves in a background thread. If
> the commit policy is not purely time based, it's not currently possibly to
> make sure the call to commit() will be processed promptly.
> Two possible solutions to this:
> 1. Make sure a lock is not held during the actual select call. Since we have
> multiple layers (KafkaConsumer -> NetworkClient -> Selector -> nio Selector)
> this is probably hard to make work cleanly since locking is currently only
> performed at the KafkaConsumer level and we'd want it unlocked around a
> single line of code in Selector.
> 2. Wake up the selector before synchronizing for certain operations. This
> would require some additional coordination to make sure the caller of
> wakeup() is able to acquire the lock promptly (instead of, e.g., the poll()
> thread being woken up and then promptly reacquiring the lock with a
> subsequent long poll() call).
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
