JingsongLi opened a new pull request, #10667: URL: https://github.com/apache/paimon/pull/10667
### Purpose Enable batch incremental clustering for ordinary-column Data Evolution tables through the shared incremental-clustering planner and the existing Spark/Flink distributed global-sort and local-sort pipelines. Dedicated BLOB/VECTOR files remain unsupported. Selection keeps complete row-ID groups together. Readers merge current column values and apply deletion vectors before rewriting; commits assign fresh row IDs, reject overlapping concurrent changes, and atomically invalidate affected global indexes, including indexes committed during an optimistic retry. ### Tests - Core: clustering selection, partial updates, deletion-vector materialization, output rolling, fresh row IDs, independent appends, stale UPDATE/DELETE/index rejection, and real optimistic retries with concurrent BTree index creation (DV enabled and disabled). - Spark: seven focused procedure tests covering distributed global/local sorting, minor/full selection, historical partitions with idle filtering, updates/deletes, and fully deleted partitions. - Flink: parameterized global/local clustering tests with interleaved input and cross-file range assertions, partition-scoped index invalidation, fully deleted partitions, and existing ordinary clustering regressions. - Latest focused Maven verification without `fast-build`: 55 core tests and 5 Flink tests passed; related schema/planner/compaction suites and seven Spark tests also passed earlier in this task. Formatting, static checks, and `git diff --check` passed. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
