JingsongLi opened a new pull request, #10687: URL: https://github.com/apache/paimon/pull/10687
Incremental clustering scans the selected partitions twice: once to choose clustering inputs and again to find accumulated Data Evolution column versions outside that selection. Make `IncrementalClusterManager.plan` produce both task sets from the same snapshot scan and have Flink and Spark consume that plan. Keep the two execution strategies: clustering sorts and assigns fresh RowIDs, while normal compaction preserves RowIDs, levels, deletion vectors, and unaffected index coverage. Normal tasks still exclude complete groups selected for clustering. Spark applies its partition-idle filter to both task sets. Existing history-partition discovery and its separate scans remain unchanged. Validation: - Passed 143 core cases covering one selected-partition scan, mixed levels without duplicate inputs, normal-only plans, empty and pinned snapshots, partition filtering, history partitions, RowIDs, deletion vectors, and concurrent reassignment/retry. - Passed 12 Flink 1.20 integration cases. The enhanced mixed-maintenance case includes a partition with only normal compaction and checks RowIDs, levels, deletion vectors, indexes, and query results; regressions also cover local/global sort and ordinary DV clustering. - Passed seven Spark 3.5 integration cases. The enhanced local/global-sort cases cover idle filtering for normal-only and mixed plans, followed by successful compaction with RowID and deletion-vector assertions; regressions cover ordinary clustering, distributed sorting, and minor/full strategies. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
