[ https://issues.apache.org/jira/browse/KAFKA-10009?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17173671#comment-17173671 ]
Werner Daehn commented on KAFKA-10009: -------------------------------------- What would be needed is either * endOffsets() returns the last existing offset. A bit dangerous as it might return offset 100, then a log compaction happens and then we start reading. * poll() telling that it retrieved the last record. That should be doable. Then we can call poll(100ms) in a loop until it tells us no-more-data via another getter. If I am interested in the current records only I stop polling now, all others will simply continue calling poll(). And the best of it, no side effects and backward compatibility. The one thing I don't know is if poll even has a chance to get that information yet or if the broker must be changed as well. I have seen many similar questions and no real solution, so this is a popular request. > Add method for getting last record offset in kafka partition > ------------------------------------------------------------ > > Key: KAFKA-10009 > URL: https://issues.apache.org/jira/browse/KAFKA-10009 > Project: Kafka > Issue Type: New Feature > Components: clients, consumer > Reporter: Yuriy Badalyantc > Priority: Major > > As far as I understand, at the current moment, there is no reliable way for > getting offset of the last record in the partition using java client. There > is {{endOffsets}} method in the consumer. And usually {{endOffsets - 1}} > works fine. But in the case of transactional producer, topic may contain > offsets without a record. And {{endOffsets - 1}} will point to the offset > without record. > This feature will help in situations when consumer application wants to > consume the whole topic. Checking of beginning and last record offset will > give lower and upper bounds for consuming. Of course, it is doable with the > current consumer implementation, but I need to check {{position}} after each > poll. > Also, I believe that this feature may help with monitoring and operations. -- This message was sent by Atlassian Jira (v8.3.4#803005)