cmccabe opened a new pull request #10931: URL: https://github.com/apache/kafka/pull/10931
This PR implements broker-side KRaft snapshots, including both saving and loading. The code for triggering a periodic broker-side snapshot will come in a follow-on PR. Loading should work with just this PR. It also implements reloading broker snapshots after initialization. In order to facilitate snapshots, this PR introduces the concept of MetadataImage and MetadataDelta. MetadataImage represents the metadata state retained in memory. It is basically a generalization of MetadataCache that includes a few things that MetadataCache does not (such as features and client quotas.) KRaftMetadataCache is now an accessor for the data stored in this object. Similarly, MetadataImage replaces CacheConfigRespository and ClientQuotaCache. It also subsumes kafka.server.metadata.MetadataImage and related classes. MetadataDelta represents a change to a MetadataImage. When a KRaft snapshot is loaded, we will accumulate all the changes into a MetadataDelta first, prior to applying it. If we must reload a snapshot because we fell too far behind while consuming metadata, the resulting MetadataDelta will contain all the changes needed to catch us up. During normal operation, MetadataDelta is also used to accumulate the changes of each incoming batch of metadata records. These incremental deltas should be relatively small. I have removed the logic for updating the various manager objects from BrokerMetadataListener and placed it into BrokerMetadataPublisher. This makes it easier to unit test BrokerMetadataListener. This is a work in progress ... I need to add some more tests -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
