Will Funnell created KAFKA-2500:
-----------------------------------
Summary: Make logEndOffset available in the 0.8.3 Consumer
Key: KAFKA-2500
URL: https://issues.apache.org/jira/browse/KAFKA-2500
Project: Kafka
Issue Type: Improvement
Components: consumer
Affects Versions: 0.8.3
Reporter: Will Funnell
Assignee: Neha Narkhede
Priority: Critical
Fix For: 0.8.3
Originally created in the old consumer here:
https://issues.apache.org/jira/browse/KAFKA-1977
The requirement is to create a snapshot from the Kafka topic but NOT do
continual reads after that point. For example you might be creating a backup of
the data to a file.
This ticket covers the addition of the functionality to the new consumer.
In order to achieve that, a recommended solution by Joel Koshy and Jay Kreps
was to expose the high watermark, as maxEndOffset, from the FetchResponse
object through to each MessageAndMetadata object in order to be aware when the
consumer has reached the end of each partition.
The submitted patch achieves this by adding the maxEndOffset to the
PartitionTopicInfo, which is updated when a new message arrives in the
ConsumerFetcherThread and then exposed in MessageAndMetadata.
See here for discussion:
http://search-hadoop.com/m/4TaT4TpJy71
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
