JingsongLi commented on code in PR #10582:
URL: https://github.com/apache/paimon/pull/10582#discussion_r4294949524
##########
paimon-core/src/main/java/org/apache/paimon/operation/FileStoreCommitImpl.java:
##########
@@ -728,8 +729,16 @@ public void abort(List<CommitMessage> commitMessages) {
dataFilesToDelete.addAll(commitMessage.compactIncrement().compactAfter());
dataFilesToDelete.addAll(commitMessage.compactIncrement().changelogFiles());
+ // A level upgrade may reuse a main file while adding new
auxiliary files.
+ Set<Path> compactInputPaths =
+ commitMessage.compactIncrement().compactBefore().stream()
+ .flatMap(file ->
file.collectFiles(dataPathFactory).stream())
+ .collect(Collectors.toSet());
+
for (DataFileMeta file : dataFilesToDelete) {
-
file.collectFiles(dataPathFactory).forEach(fileIO::deleteQuietly);
+ file.collectFiles(dataPathFactory).stream()
+ .filter(path -> !compactInputPaths.contains(path))
Review Comment:
[P2] Keep cleaning newly written files that also appear in compactBefore
`compactBefore` also contains inputs created by the current uncommitted
write, so protecting all of its paths filters files out of
`newFilesIncrement().newFiles()` cleanup. I reproduced this with the production
TestFileStore writer: write one row, `compact(false)`, write the same key
again, then `compact(true)` and `prepareCommit(true, 0)`. The generated message
has 2 fresh files, the same 2 files in compactBefore, and 1 rewritten
compactAfter file (no reused output path). After abort the new output is
deleted but both fresh input files remain, with no snapshot ever committed; the
exact base abort removes both inputs for the same writer sequence.
MergeTreeWriter deliberately includes those inputs in both increments, and
prepareCommit has already drained its later close-time cleanup state. Please
keep abort-owned new files eligible for deletion even when they are compaction
inputs, and protect only inputs owned by an existing snapshot. Add this real
flush → compact → prepare �
� abort case alongside the reused-file regression.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
