Stephen0421 opened a new pull request, #10749:
URL: https://github.com/apache/paimon/pull/10749
### Purpose
Data evolution compaction could not compact dedicated vector-store files
(`*.vector.<format>`). The `compactVector` path in
`DataEvolutionCompactCoordinator` was a TODO, and
`DataEvolutionNormalCompactTask` threw `UnsupportedOperationException`. Partial
vector column updates therefore pile up small vector files that are never
merged.
This PR adds vector-store file compaction, modeled on blob compaction:
- Add the option `vector-compaction.enabled` (default `false`). The Flink
`DataEvolutionTableCompactSource` and the Spark `CompactProcedure` now read it
instead of hard-coding `false`.
- Add `DataEvolutionVectorCompactTask` (`TaskType.VECTOR`). It reads its
input vector files through the data evolution read path, so each vector field
keeps the value from the file with the highest sequence number. It writes the
current vector fields with the configured vector format and rolls output by
`vector.target-file-size`. Row IDs are preserved, so deletion vectors on the
normal files stay valid.
- Planning in `DataEvolutionCompactCoordinator`:
- A task always takes **all** vector-store files of its row range.
Compacted files inherit the max sequence number of their inputs, so a file left
out could start or stop winning rows against them.
- Vector files spanning several normal files are merged only when those
normal files are merged in the same batch. Otherwise, each normal file's vector
files are compacted on their own. Vector task ranges are added to the normal
task's protected ranges, so a normal output never cuts through a vector output.
- A group is skipped if a current vector field covers only part of its row
range. Otherwise the rewrite would store NULL for rows that never had that
field.
- Fields dropped from the table are ignored in that coverage check, and
the rewrite does not keep them, matching normal compaction.
- With `global-index.column-update-action=IGNORE`, compacted vector files
keep per-column `writeColsSequences`, so compaction does not look like a column
update to the global index. The shared logic moved from
`DataEvolutionNormalCompactTask` to `DataEvolutionCompactTask`.
- `DataEvolutionCompactTaskSerializer` adds type code 3 but stays at version
4. The new type has no extra payload, and bumping the version would stop every
existing job from restoring in-flight compaction tasks.
Not included: materializing deletion vectors into vector-store files.
### Tests
- `DataEvolutionCompactCoordinatorTest`:
- Planning vector tasks within one normal file and across normal files
merged in the same batch.
- Skipping a field that covers only part of the range, and ignoring
dropped fields.
- Serializer round trip for `VECTOR` tasks, and rejection of non-vector
inputs.
- `VectorStoreTableTest`:
- Partial column updates keep the latest value per column after compaction
(json and parquet), including projections and after a column rename.
- A column that is missing for part of the range is skipped.
- A dropped column is not kept.
- Output rolls by `vector.target-file-size`.
- Per-column sequences are kept under
`global-index.column-update-action=IGNORE`.
- Regression: `DataEvolutionNormalCompactTaskTest`, `BlobTableTest`,
`DataEvolutionTableTest`, `DataEvolutionDeletionVectorTest`,
`MultipleBlobTableTest` and `CompactCandidateRangeCollectorTest` pass.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
