[
https://issues.apache.org/jira/browse/KAFKA-364?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13397486#comment-13397486
]
Ross Black commented on KAFKA-364:
----------------------------------
I send batched messages with compression, and use the offsets retrieved by the
consumer to get exactly-once semantics (by persisting consumer state with the
offsets). When using the message set iterator, for a e.g. batch of 5 messages
the offset returned for messages 1-4 is the start of the *current* batch, and
the offset for message 5 is the start of the *next* batch. My code has to wait
for the offset to change from the previous message before it persists (so that
my consumer state is only persisted when a batch has been completed). To me,
this feels awkward in that it is not very explicit in the API (you have to know
about internals to understand the processing required). I think it could be
useful to expose a flag that indicated batch-end, or to directly expose message
batches (similar to the way shallowIterator does?).
Thanks,
Ross
> Consumer re-design
> ------------------
>
> Key: KAFKA-364
> URL: https://issues.apache.org/jira/browse/KAFKA-364
> Project: Kafka
> Issue Type: New Feature
> Reporter: Neha Narkhede
> Assignee: Neha Narkhede
>
> We've received quite a lot of feedback on the consumer side features over the
> past few months. Some of them are improvements to the current consumer design
> and some are simply new feature/API requests. I have attempted to write up
> the requirements that I've heard on this wiki -
> https://cwiki.apache.org/confluence/display/KAFKA/Consumer+Client+Re-Design
> This would involve some significant changes to the consumer APIs, so we would
> like to collect feedback on the proposal from our community. Since the list
> of changes is not small, we would like to understand if some features are
> preferred over others, and more importantly, if some features are not
> required at all.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
