zhuxiangyi commented on code in PR #8139:
URL: https://github.com/apache/paimon/pull/8139#discussion_r3468268328
##########
paimon-core/src/main/java/org/apache/paimon/operation/FileStoreCommitImpl.java:
##########
@@ -1164,6 +1165,150 @@ public boolean replaceManifestList(
return commitSnapshotImpl(newSnapshot, emptyList());
}
+ @Override
+ public boolean restoreAsLatest(Snapshot targetSnapshot) {
+ Snapshot latest =
+ checkNotNull(
+ snapshotManager.latestSnapshot(),
+ "Latest snapshot is null, can not restore.");
+
+ Map<FileEntry.Identifier, ManifestEntry> latestEntries = new
HashMap<>();
+ FileEntry.mergeEntries(
+ manifestFile,
+ manifestList.readDataManifests(latest),
+ latestEntries,
+ options.scanManifestParallelism());
+
+ latestEntries.entrySet().removeIf(entry -> entry.getValue().kind() !=
FileKind.ADD);
+
+ Map<FileEntry.Identifier, ManifestEntry> targetEntries = new
HashMap<>();
+ FileEntry.mergeEntries(
+ manifestFile,
+ manifestList.readDataManifests(targetSnapshot),
+ targetEntries,
+ options.scanManifestParallelism());
+ targetEntries.entrySet().removeIf(entry -> entry.getValue().kind() !=
FileKind.ADD);
+
+ List<ManifestEntry> deltaFiles = new ArrayList<>();
Review Comment:
You're right. A deletion-vector-only delete/update doesn't rewrite the data
file, so the identifier-based diff left an empty data delta with
`deltaRecordCount` 0, and streaming overwrite readers (which read the DELTA
data manifests and don't load DV indexes) could skip the restore.
Fixed by comparing the deletion vectors between the previous latest and the
target (via the index-manifest `dvRanges`) and re-emitting each data file whose
DV changed as DELETE(latest) + ADD(target). This makes the restore transition
visible to streaming overwrite readers. The physical row count is unchanged, so
the pair nets to zero in the delta record count, matching the unchanged
`totalRecordCount`.
Note this makes restore behave like a regular overwrite for DV tables;
applying the DV itself during streaming-read-overwrite is a separate,
pre-existing limitation of that read path and is out of scope here.
Added a `FileStoreCommitTest` covering a DV-only restore transition.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
