JingsongLi commented on code in PR #10749:
URL: https://github.com/apache/paimon/pull/10749#discussion_r4357374770
##########
paimon-core/src/main/java/org/apache/paimon/append/dataevolution/DataEvolutionCompactCoordinator.java:
##########
@@ -530,6 +563,13 @@ private List<DataEvolutionCompactTask> triggerTask(
return tasks;
}
+ private boolean vectorStoreFilesCompactable(List<DataFileMeta>
vectorStoreFiles) {
+ return vectorStoreFiles.size() > 1
+ && vectorStoreFiles.size() >= compactMinFileNum
+ && DataEvolutionVectorCompactTask.coversRangeForEveryField(
+ vectorStoreFiles, schemaFetcher,
currentVectorFieldIds);
Review Comment:
**[P1] Make vector compaction converge after rolling its output**
This predicate also accepts the output of a successful compaction whenever
the writer rolls into at least `compaction.min.file-num` vector files. With
`vector-compaction.enabled=true`, Spark's `compact` procedure can therefore
keep rewriting these files and committing new snapshots indefinitely:
`DataEvolutionRewriteExecutor` replans the committed snapshot after each
commit, `CompactProcedure` creates a new coordinator, and the completed-file
protection excludes vector outputs.
I reproduced the repeated planning by extending
`VectorStoreTableTest#testCompactVectorFilesRollsByVectorTargetSize`:
- Write two batches of 1,250 rows, with `compaction.min.file-num=2` and
`vector.target-file-size=1 b`.
- The first compaction produces one normal file covering `[0,2499]` and
three vector files covering `[0,999]`, `[1000,1999]`, and `[2000,2499]`.
- A second compaction replaces all three vector files with new filenames but
identical ranges. Planning the resulting snapshot still returns another
`VECTOR` task.
The core reproduction and Spark control flow establish the loop; I did not
run an end-to-end Spark hang test. This is not limited to the small test
target: it also occurs when an ordinary compaction range needs at least the
minimum number of rolled vector outputs.
Please make selection account for target size and overlap so that already
compacted, non-overlapping output layouts are skipped when no useful merge
remains, while preserving the all-files-per-range correctness constraint.
Tracking completed vector outputs within an invocation can additionally prevent
repeated work in the same call. Please also add a regression assertion that
subsequent planning produces no vector task for this layout, or a bounded Spark
procedure test that verifies termination.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
