tzphh opened a new issue, #10458: URL: https://github.com/apache/paimon/issues/10458
### Search before asking - [x] I searched in the [issues](https://github.com/apache/paimon/issues) and found nothing similar. ### Paimon version 2.2-SNAPSHOT; upstream master inspected at commit `9c3c5187a4bd045d31b1ed9fb0decd58e6d63e8e` (2026-10-09). The two planning classes were verified byte-for-byte against this commit, then compiled locally and exercised with synthetic file metadata. This is not a full build or an end-to-end test of that master revision. ### Compute Engine Java API / paimon-core compaction planning, JDK 17. The issue was investigated in the context of Flink batch compaction, but the reproduction below does not require a Flink cluster. ### Minimal reproduce step Use a Data Evolution table's normal-file planning path with these inputs: - One partition, 12 normal Parquet files, each 200 MiB. - Contiguous, non-overlapping row-ID ranges: file i has firstRowId = i * 100 and rowCount = 100. - `target-file-size = 512 MB`, `compaction.min.file-num = 5`, `source.split.open-file-cost = 4 MB`. - No BLOB/VECTOR files, no deletion-vector materialization, large-file splitting disabled. A minimal metadata-only reproduction can be added to the existing `CompactCandidateRangeCollectorTest` (its current imports already cover this method): ```java @Test void testContiguousFilesExceedTargetBelowMinFileCount() { long mib = 1024L * 1024L; CompactCandidateRangeCollector collector = new CompactCandidateRangeCollector( 12, 512 * mib, 1024 * mib, 4 * mib, 5, Long.MAX_VALUE); for (int i = 0; i < 12; i++) { collector.add(0, NORMAL_FILE, i * 100L, 100L, 200 * mib); } List<String> ranges = new ArrayList<>(); collector.finish((start, end, count) -> ranges.add(start + "-" + end + ":" + count)); assertThat(ranges).isNotEmpty(); // Fails: ranges is empty. } ``` Run the test from the repository root: ```bash mvn -pl paimon-core -am -Dtest=CompactCandidateRangeCollectorTest \ -Dsurefire.failIfNoSpecifiedTests=false test ``` A separate synthetic-metadata harness also called `CompactPlanner.compactPlan` directly, bypassing the candidate collector, and then via the collector-selected ranges: | Input | Target / min files | Candidate ranges | Direct planner tasks | Tasks after candidate filtering | | --- | --- | ---: | ---: | ---: | | 12 x 200 MiB | 512 MiB / 5 | 0 | 0 | 0 | | 12 x 200 MiB | 512 MiB / 2 | 4 | 4 | 4 | | 12 x 200 MiB | 1024 MiB / 5 | 2 | 2 | 2 | | 10 x 10 MiB | 512 MiB / 5 | 1 | 1 | 1 | | 5 x 128 MiB | 512 MiB / 5 | 1 | 1 | 1 | MiB means 1024 * 1024 bytes. These results test selection/planning only; no Parquet files are read or rewritten. ### What doesn't meet your expectations? Expected: compaction should be able to make progress on these contiguous normal files. Accumulating enough bytes for compaction should not repeatedly discard the group before it can satisfy the file-count condition. Actual: every three files reach 600 MiB, exceeding the 512 MiB target. The group is flushed/drained, but rejected because it contains only 3 files (< 5). It is then cleared instead of carried forward. This repeats four times, so all 12 files produce zero candidates and zero tasks. Adding more files of the same size does not resolve this pattern. This can make a compaction invocation finish without merging any of these files. There is no exception in the reproduction. ### Anything else? Both selection stages contain this interaction: 1. [`CompactCandidateRangeCollector`](https://github.com/apache/paimon/blob/9c3c5187a4bd045d31b1ed9fb0decd58e6d63e8e/paimon-core/src/main/java/org/apache/paimon/append/dataevolution/CompactCandidateRangeCollector.java#L540-L565) flushes the bin when its normal weight exceeds the target. [`CandidateBin.emit`](https://github.com/apache/paimon/blob/9c3c5187a4bd045d31b1ed9fb0decd58e6d63e8e/paimon-core/src/main/java/org/apache/paimon/append/dataevolution/CompactCandidateRangeCollector.java#L635-L650) still requires the minimum normal-file count. 2. [`CompactPlanner`](https://github.com/apache/paimon/blob/9c3c5187a4bd045d31b1ed9fb0decd58e6d63e8e/paimon-core/src/main/java/org/apache/paimon/append/dataevolution/DataEvolutionCompactCoordinator.java#L373-L398) drains on the size threshold; [`triggerTask`](https://github.com/apache/paimon/blob/9c3c5187a4bd045d31b1ed9fb0decd58e6d63e8e/paimon-core/src/main/java/org/apache/paimon/append/dataevolution/DataEvolutionCompactCoordinator.java#L452-L466) then applies the file-count check. Fixing only the final planner would leave the candidate prefilter blocking these files. One possible approach is to let a bin with multiple normal files trigger on accumulated size as well as file count, while retaining partition/row-ID boundaries and avoiding unnecessary single-file rewrites. The large-file splitting support in #9709 does not cover this case: each 200 MiB file is smaller than the 512 MiB target. Enabling it with ratio 1 or 2 still produced zero candidates/tasks in the same harness. This report concerns core Data Evolution file selection, separately from the compact_database routing issue #10102. ### Are you willing to submit a PR? - [ ] I'm willing to submit a PR! -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
