GitHub user benstopford opened a pull request:
https://github.com/apache/kafka/pull/2743
KIP-101: Alter Replication Protocol to use Leader Epoch rather than High
Watermark for Truncation
This PR describes the addition of Partition Level Leader Epochs to messages
in Kafka as a mechanism for fixing some known issues in the replication
protocol. Full details can be found here:
[KIP-101
Reference](https://cwiki.apache.org/confluence/display/KAFKA/KIP-101+-+Alter+Replication+Protocol+to+use+Leader+Epoch+rather+than+High+Watermark+for+Truncation)
*The key elements are*:
- Epochs are stamped on messages as they enter the leader.
- Epochs are tracked in both leader and follower in a new checkpoint file.
- A new API allows followers to retrieve the leader's latest offset for a
particular epoch.
- The logic for truncating the log, when a replica becomes a follower, has
been moved from Partition into the ReplicaFetcherThread
- When partitions are added to the ReplicaFetcherThread they are added in
an initialising state. Initialising partitions request leader epochs and then
truncate their logs appropriately.
This test provides a good overview of the workflow
`EpochDrivenReplicationProtocolAcceptanceTest.shouldFollowLeaderEpochBasicWorkflow()`
The corrupted log use case is covered by the test
`EpochDrivenReplicationProtocolAcceptanceTest.offsetsShouldNotGoBackwards()`
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/benstopford/kafka kip-101-v2
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/kafka/pull/2743.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #2743
----
commit 768dfeb9c5ef8b78b48fd898311d2cb459e564fd
Author: Ben Stopford <[email protected]>
Date: 2017-03-27T16:16:16Z
KIP-101: Push after merge.
----
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [email protected] or file a JIRA ticket
with INFRA.
---
