jsancio commented on a change in pull request #10899:
URL: https://github.com/apache/kafka/pull/10899#discussion_r655536129
##########
File path:
clients/src/main/java/org/apache/kafka/common/record/MemoryRecords.java
##########
@@ -664,4 +666,55 @@ private static void writeLeaderChangeMessage(ByteBuffer
buffer,
builder.close();
}
+ public static MemoryRecords withSnapshotHeaderRecord(
+ long initialOffset,
+ long timestamp,
+ int leaderEpoch,
+ ByteBuffer buffer,
+ MetadataSnapshotHeaderRecord snapshotHeaderRecord) {
Review comment:
We have the following document: https://kafka.apache.org/coding-guide
##########
File path:
raft/src/main/java/org/apache/kafka/raft/internals/BatchAccumulator.java
##########
@@ -233,6 +235,68 @@ public void appendLeaderChangeMessage(LeaderChangeMessage
leaderChangeMessage, l
}
}
+
+ public void appendSnapshotHeaderMessage(MetadataSnapshotHeaderRecord
snapshotHeaderRecord) {
+ appendLock.lock();
+ try {
+ // Ideally there should be nothing in the batch here.
+ // TODO verify this
Review comment:
It is a valid/correct check but it doesn't cover all of the cases. For
example, in theory `SnapshotWriter` could do the following:
```
accumulator.appendSnapshotHeaderMessage(...);
snapshot.appendBatches(accumulator.drain());
accumulator.appendSnapshotHeaderMessage(...);
```
This is okay from the `BatchAccumulator`'s point of view because the
accumulator doesn't have enough information. I am also not sure if we want to
encode the invariant of `SnapshotHeaderMessage` and `SnaspshotFooterMessage`
into the `BatchAccumulator`.
I think only the `SnapshotWriter` has enough information to make these
invariant checks. What do you think?
##########
File path: raft/src/main/java/org/apache/kafka/snapshot/SnapshotWriter.java
##########
@@ -135,9 +210,12 @@ public void append(List<T> records) {
/**
* Freezes the snapshot by flushing all pending writes and marking it as
immutable.
+ *
+ * Also adds a {@link MetadataSnapshotFooterRecord} to the end of the
snapshot
*/
public void freeze() {
appendBatches(accumulator.drain());
+ finalizeSnapshotWithFooter();
Review comment:
When `freeze` is called it is possible that the `accumulator` has
batches. When freezing the `SnapshotWriter`, it needs to:
1. Add the footer batch to the accumulator
2. Drain any pending batches in the accumulator including the control batch
appended in 1.
3. Append those batches to the underlying `RawSnapshotWriter`.
##########
File path:
raft/src/main/java/org/apache/kafka/raft/internals/BatchAccumulator.java
##########
@@ -233,6 +235,68 @@ public void appendLeaderChangeMessage(LeaderChangeMessage
leaderChangeMessage, l
}
}
+
+ public void appendSnapshotHeaderMessage(MetadataSnapshotHeaderRecord
snapshotHeaderRecord) {
+ appendLock.lock();
+ try {
+ // Ideally there should be nothing in the batch here.
+ // TODO verify this
Review comment:
I think that having tests that check this behavior for `SnaphsotWriter`
is the best we can do.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
