taoran92 opened a new issue, #10447: URL: https://github.com/apache/paimon/issues/10447
We discovered this issue in an internal production workload at our company using Flink to write to a Paimon append-only table. ### Search before asking - [x] I searched in the [issues](https://github.com/apache/paimon/issues) and found no report of the same problem. ### Paimon version master. The affected implementation is present at commit `61c8956729ab4b89fe53488e035c2d14dd7eeb57` (the base of the proposed fix). ### Compute Engine Flink. The problem is in the shared `paimon-core` append-only writer; the reproduction below describes the writer-level sequence. ### Minimal reproduce step 1. Start with five small data files, `A`, `B`, `C`, `D`, and `E`, in the same append-only bucket. 2. Compact `A`, `B`, and `C` into an intermediate file `X`. Ensure `X` is still small enough to be selected for another compaction. 3. Before calling `prepareCommit`, add `D` and `E` to the writer's compaction candidates and compact `X`, `D`, and `E` into `Y`. 4. Call `prepareCommit` and inspect its compaction increment, then convert it to manifest entries and merge them with the original files. ```text First compaction: A + B + C -> X Second compaction: X + D + E -> Y (no prepareCommit between the two) Actual increment: compactBefore = [A, B, C, X, D, E] compactAfter = [X, Y] Expected increment: compactBefore = [A, B, C, D, E] compactAfter = [Y] ``` At the Java writer level, each round can be completed using `flush(false, false)` followed by `sync()`, with `addNewFiles(...)` supplying the second round's inputs. Only call `prepareCommit(true)` after the second round. ### What doesn't meet your expectations? The intermediate file `X` is retained in both sides of the compaction increment even though it was produced and consumed before any commit was prepared. Its physical data file and associated extra files are also retained. `ManifestEntryChanges.collect` emits all `compactBefore` files as DELETE entries before emitting `compactAfter` files as ADD entries. This produces `DELETE X` followed by `ADD X`, although `X` was never part of the previously committed state. In `FileEntry.mergeEntries`, the DELETE is retained in the map, and the subsequent ADD hits the existing-entry check: ```text Trying to add file ... which is already in the the map: ... ``` This explains how repeated compaction can fail manifest merging. The message above is from the source-code check, rather than a quoted production log. Only the original inputs and final output should appear in this increment. Intermediate files produced and consumed before `prepareCommit`, including their extra files, should be cleaned up. If `prepareCommit` runs between the two compactions, however, `X` has already been handed to a committer and must remain available, even if that commit has not finished. The second increment must then delete `X` through the normal commit path. ### Anything else? The affected `AppendOnlyWriter.trySyncLatestCompaction` simply appends every compaction result to two lists: ```java // Results from successive rounds accumulate without eliminating intermediate files. compactBefore.addAll(result.before()); compactAfter.addAll(result.after()); ``` [Affected writer implementation](https://github.com/apache/paimon/blob/61c8956729ab4b89fe53488e035c2d14dd7eeb57/paimon-core/src/main/java/org/apache/paimon/append/AppendOnlyWriter.java) A local fix and regression test have been prepared. The test covers repeated compaction both with and without an intervening `prepareCommit`, checks intermediate data and extra-file cleanup, and checks that manifest merging leaves only the final output. ### Are you willing to submit a PR? - [x] I'm willing to submit a PR! -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
