[
https://issues.apache.org/jira/browse/KAFKA-2123?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14522016#comment-14522016
]
Ewen Cheslack-Postava commented on KAFKA-2123:
----------------------------------------------
This version is now ready for review. I ended up going with the simpler
interface discussed on the mailing list. I ended up running into too many
synchronization and proper layering of code dealing with the Future
implementation because of the way the synchronization is currently handled for
the consumer. I'll file a separate issue for those problems and think about if
there's a path to eventually getting the interface with the Future. In the mean
time, the worst case is that we end up settling on this interface, which was
the other candidate discussed.
Some notes:
* Added a commit.retries setting. This is specific to commit offset requests
which is why it has the commit. prefix instead of just being retries. Also, the
default setting is currently -1, which gives infinite retries (in both sync and
async modes). I think that's actually a bad idea since a default that can block
indefinitely seems like it's always a bad idea to me, but we'd need to discuss
the alternatives.
* Added queueing of offset commit requests. I debated how best to handle this.
At first I was thinking we might be able to do something smarter to combine
requests, but in the face of errors (especially partial errors that are
isolated to one topic partition), it gets difficult to figure out how to handle
callbacks. Simple queuing seems like the right solution to me, and for the vast
majority of use cases it either has no impact (you used the sync mode) or has
little impact (you used async to commit all offsets automatically). Only
unusual cases where you're submitting the offsets map and doing partial commits
might care about smarter behavior.
> Make new consumer offset commit API use callback + future
> ---------------------------------------------------------
>
> Key: KAFKA-2123
> URL: https://issues.apache.org/jira/browse/KAFKA-2123
> Project: Kafka
> Issue Type: Improvement
> Components: clients, consumer
> Reporter: Ewen Cheslack-Postava
> Assignee: Ewen Cheslack-Postava
> Fix For: 0.8.3
>
> Attachments: KAFKA-2123.patch, KAFKA-2123_2015-04-30_11:23:05.patch
>
>
> The current version of the offset commit API in the new consumer is
> void commit(offsets, commit type)
> where the commit type is either sync or async. This means you need to use
> sync if you ever want confirmation that the commit succeeded. Some
> applications will want to use asynchronous offset commit, but be able to tell
> when the commit completes.
> This is basically the same problem that had to be fixed going from old
> consumer -> new consumer and I'd suggest the same fix using a callback +
> future combination. The new API would be
> Future<Void> commit(Map<TopicPartition, Long> offsets, ConsumerCommitCallback
> callback);
> where ConsumerCommitCallback contains a single method:
> public void onCompletion(Exception exception);
> We can provide shorthand variants of commit() for eliding the different
> arguments.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
