[
https://issues.apache.org/jira/browse/KAFKA-2217?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14559646#comment-14559646
]
Jason Gustafson commented on KAFKA-2217:
----------------------------------------
KAFKA-2168 was the main reason. Committing offsets or getting metadata from an
external thread can be blocked by long polls with the current synchronization
policy in KafkaConsumer. So I was looking into finer-grained synchronization,
but this is difficult at present due to all of state in the underlying objects.
This was a first attempt to try to make some of these objects safer. I can
understand the concern about the extra allocations though. I've been doing some
performance testing to see if it introduces noticeable GC overhead, and so far,
it hasn't.
> Refactor Client Selectable Interface for Better Concurrency Options
> -------------------------------------------------------------------
>
> Key: KAFKA-2217
> URL: https://issues.apache.org/jira/browse/KAFKA-2217
> Project: Kafka
> Issue Type: Improvement
> Reporter: Jason Gustafson
> Assignee: Jason Gustafson
> Attachments: KAFKA-2217.patch, KAFKA-2217_2015-05-25_10:45:30.patch,
> KAFKA-2217_2015-05-26_09:37:29.patch
>
>
> The current Selectable interface makes thread-safe usage without external
> locking unlikely. In particular, the interface requires implementations to
> store internal lists containing the results from an invocation of poll. This
> makes dealing with issues such as KAFKA-2168 more difficult since it adds
> state which must be synchronized. Here are the offending methods:
> {code:java}
> interface Selectable {
> void poll(long timeout);
> List<NetworkSend> completedSends();
> List<NetworkReceive> completedReceives();
> List<Integer> disconnected();
> List<Integer> connected();
> // rest excluded
> }
> {code}
> The user is required to invoke poll, then extract the results from the
> corresponding methods. In order to avoid missing events, the caller must hold
> an external lock while they access the results of the poll.
> Instead, we can return the results directly from poll call using a container
> class. For example:
> {code:java}
> class PollResult {
> List<NetworkSend> completedSends;
> List<NetworkReceive> completedReceives;
> List<Integer> disconnected;
> List<Integer> connected;
> }
> interface Selectable {
> PollResult poll(long timeout);
> }
> {code}
> This should get us closer to a thread-safe NetworkClient, which would enable
> a more granular approach to synchronizing the KafkaConsumer.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
