Tejas Patil created KAFKA-1000:
----------------------------------
Summary: Inbuilt consumer offset management
Key: KAFKA-1000
URL: https://issues.apache.org/jira/browse/KAFKA-1000
Project: Kafka
Issue Type: New Feature
Components: consumer
Affects Versions: 0.8.1
Reporter: Tejas Patil
Assignee: Tejas Patil
Priority: Minor
This Jitra corresponds to the phase #2 of Offset Management [0]. After
KAFKA-657 [1], we have a protocol for consumers to commit and fetch offsets
from brokers. Currently, consumers are not using this API and directly talking
with Zookeeper.
This Jira will involve following:
1. Consumers should now use the offset commit API
2. Add a special topic in kafka for storing offsets
3. Pluggable offset commit logic so that users can pick the existing ZK based
storage or inbuilt storage for offsets.
4. Leader brokers would now maintain an additional hash table of offsets for
the group-topic-partitions that they lead
5. Consumers send their offset fetch requests to brokers. Receiving broker may
or may not be the leader for the requested group-topic-partitions so would have
to forward the request to respective leader brokers.
Joel and I have been working on this and I would share the initial patch and a
writeup covering the nitty-gritty of the implementation.
[0] : https://cwiki.apache.org/confluence/display/KAFKA/Offset+Management
[1] : https://issues.apache.org/jira/browse/KAFKA-657
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
