Stephen0421 commented on code in PR #10749:
URL: https://github.com/apache/paimon/pull/10749#discussion_r4360856739
##########
paimon-core/src/main/java/org/apache/paimon/append/dataevolution/DataEvolutionCompactCoordinator.java:
##########
@@ -530,6 +563,13 @@ private List<DataEvolutionCompactTask> triggerTask(
return tasks;
}
+ private boolean vectorStoreFilesCompactable(List<DataFileMeta>
vectorStoreFiles) {
+ return vectorStoreFiles.size() > 1
+ && vectorStoreFiles.size() >= compactMinFileNum
+ && DataEvolutionVectorCompactTask.coversRangeForEveryField(
+ vectorStoreFiles, schemaFetcher,
currentVectorFieldIds);
Review Comment:
Thanks for catching this! Fixed in d948fe053d. Vector compaction now
converges, and a rolled output layout is no longer replanned.
**Root cause**: `vectorStoreFilesCompactable` only checked the file count
and field coverage, so an already compacted layout with at least
`compaction.min.file-num` rolled files was selected again on every replan.
**Fix** (two layers, both keep the all-files-per-range constraint):
1. **Select by overlap and target size**
(`DataEvolutionCompactCoordinator`): a vector group is rewritten only if some
rows are stored in more than one vector file (partial column updates), or more
than one file is below `vector.target-file-size`. Rolled output has no overlap
and at most one small trailing file, so later invocations skip it. If a group
is selected, the task still takes every vector file of its range.
2. **Track completed vector outputs within an invocation**
(`CompactProcedure`), like `completedNormalFiles`: a group made up only of this
invocation's vector outputs is not planned again. The size rule alone is not
enough here, because several formats roll by an estimated size (e.g. Parquet's
buffered data size, ORC/Vortex/Mosaic estimates), so a rolled file's final size
can end up below the target. If a new file lands in the range, the whole group
is still compacted together.
The Flink source plans once against a fixed snapshot and never replans its
own outputs, so it is unaffected.
**Tests**:
- `VectorStoreTableTest#testCompactVectorFilesRollsByVectorTargetSize`: now
asserts that planning the compacted snapshot returns no task. The input now
includes a column update, because two non-overlapping files above the target
are correctly skipped.
- `DataEvolutionCompactCoordinatorTest`: new cases for skipping rolled
output, merging only when overlapping or with at least two small files, and
skipping completed vector outputs unless a new file joins the range.
- New bounded Spark procedure test `data evolution vector compaction does
not recompact rolled output`: the commit callback fails if the procedure
commits more than once. It fails on the previous code, because the procedure
tried a second commit that rewrote its own output, and it passes with this fix,
with one commit and the expected three rolled vector files.
I also updated `data-evolution-maintenance.mdx` to describe when a vector
group is skipped.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
