[
https://issues.apache.org/jira/browse/KAFKA-238?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Neha Narkhede updated KAFKA-238:
--------------------------------
Attachment: kafka-238-v1.patch
This is a draft patch that includes the new getTopicMetadata API on the server.
Changes include -
1. Kafka server always requires a zookeeper connection
2. Additional configs on the server are
2.1 auto.create.topics which controls auto creation of topics in the
getTopicMetadata API. Default is true
2.2 default.replication.factor which controls the replication factor if the
topic is auto created
3. getTopicMetadata request takes in a list of topics. One question is if no
list is specified, would returning metadata for all topics make sense ?
3.1 topic metadata request has the following format -
number of topics (4 bytes)
list of topics (2 bytes + topic.length per topic) detailedMetadata (2 bytes)
timestamp (8 bytes) (optional if detailedMetadata = 1 )
count (4 bytes) (optional, if detailedMetadata = 1)
3.2 topic metadata response has the following format -
topic
partition-metadata
3.3. partition-metadata has the following format -
partition-id
leader (optional)
replicas
in-sync replicas
log metadata (optional)
3.4 log metadata has the following format -
number of log segments
total size of this log in bytes
log segment metadata (optional)
3.5 log segment metadata has the following format -
beginning offset
last modified timestamp
size of segment in bytes
4. Test suite cleanup to include zookeeper dependency for the
KafkaServerHarness
5. Added unit test TestTopicMetadataTest with mock log manager. This tests the
new request format, auto creation of topic and metadata response
> add a getTopicMetaData method in broker and expose it to producer
> ------------------------------------------------------------------
>
> Key: KAFKA-238
> URL: https://issues.apache.org/jira/browse/KAFKA-238
> Project: Kafka
> Issue Type: Sub-task
> Components: core
> Reporter: Jun Rao
> Assignee: Neha Narkhede
> Attachments: kafka-238-v1.patch
>
>
> We need a way to propagate the leader and the partition information to the
> producer so that it can do load balancing and semantic partitioning. One way
> to do that is to have the producer get the information from ZK directly. This
> means that the producer needs to maintain a ZK session and has to subscribe
> to watchers, which can be complicated. An alternative approach is to have the
> following api on the broker.
> TopicMetaData getTopicMetaData(String: topic)
> TopicMetaData {
> Array[PartitionMetaData]: partitionsMetaData
> }
> PartitionMetaData {
> Int: partitionId
> String: leaderHostname
> Int: leaderPort
> }
> Using this api, the producer can get the metadata about a topic during
> initial startup or leadership change of a partition.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
