[
https://issues.apache.org/jira/browse/KAFKA-9166?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Satish Duggana updated KAFKA-9166:
--
Description:
Brief description of the ask is mentioned in KIP-500's
[BrokerMetadataManagement|https://cwiki.apache.org/confluence/display/KAFKA/KIP-500%3A+Replace+ZooKeeper+with+a+Self-Managed+Metadata+Quorum#KIP-500:ReplaceZooKeeperwithaSelf-ManagedMetadataQuorum-BrokerMetadataManagement]
Instead of the controller pushing out updates to the other brokers, those
brokers will fetch updates from the active controller via the new MetadataFetch
API.
A MetadataFetch is similar to a fetch request. Just like with a fetch request,
the broker will track the offset of the last updates it fetched, and only
request newer updates from the active controller.
The broker will persist the metadata it fetched to disk. This will allow the
broker to start up very quickly, even if there are hundreds of thousands or
even millions of partitions. (Note that since this persistence is an
optimization, we can leave it out of the first version, if it makes development
easier.)
Most of the time, the broker should only need to fetch the deltas, not the full
state. However, if the broker is too far behind the active controller, or if
the broker has no cached metadata at all, the controller will send a full
metadata image rather than a series of deltas.
was:
Instead of the controller pushing out updates to the other brokers, those
brokers will fetch updates from the active controller via the new MetadataFetch
API.
A MetadataFetch is similar to a fetch request. Just like with a fetch request,
the broker will track the offset of the last updates it fetched, and only
request newer updates from the active controller.
The broker will persist the metadata it fetched to disk. This will allow the
broker to start up very quickly, even if there are hundreds of thousands or
even millions of partitions. (Note that since this persistence is an
optimization, we can leave it out of the first version, if it makes development
easier.)
Most of the time, the broker should only need to fetch the deltas, not the full
state. However, if the broker is too far behind the active controller, or if
the broker has no cached metadata at all, the controller will send a full
metadata image rather than a series of deltas.
> Implement MetadataFetch API
> ---
>
> Key: KAFKA-9166
> URL: https://issues.apache.org/jira/browse/KAFKA-9166
> Project: Kafka
> Issue Type: Sub-task
> Components: controller
>Reporter: Viktor Somogyi-Vass
>Priority: Major
>
> Brief description of the ask is mentioned in KIP-500's
> [BrokerMetadataManagement|https://cwiki.apache.org/confluence/display/KAFKA/KIP-500%3A+Replace+ZooKeeper+with+a+Self-Managed+Metadata+Quorum#KIP-500:ReplaceZooKeeperwithaSelf-ManagedMetadataQuorum-BrokerMetadataManagement]
> Instead of the controller pushing out updates to the other brokers, those
> brokers will fetch updates from the active controller via the new
> MetadataFetch API.
> A MetadataFetch is similar to a fetch request. Just like with a fetch
> request, the broker will track the offset of the last updates it fetched, and
> only request newer updates from the active controller.
> The broker will persist the metadata it fetched to disk. This will allow the
> broker to start up very quickly, even if there are hundreds of thousands or
> even millions of partitions. (Note that since this persistence is an
> optimization, we can leave it out of the first version, if it makes
> development easier.)
> Most of the time, the broker should only need to fetch the deltas, not the
> full state. However, if the broker is too far behind the active controller,
> or if the broker has no cached metadata at all, the controller will send a
> full metadata image rather than a series of deltas.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
