JingsongLi commented on PR #10391: URL: https://github.com/apache/paimon/pull/10391#issuecomment-6081966383
**[P2] Preserve dedicated Vector file identity in the real copy path before checking overlapping normal files** The previous concurrent row-ID allocation finding is fixed in `65c3b7b341006210ce23ce3d464fe02bb48d3e68`: the core competing-append and actual commit-retry tests pass. I found a different regression in an actual Spark 3.5.8 `CALL sys.copy`. `ListDataFilesOperator.pickDataFiles` chooses each target filename with `newPath()`, so a source `.vector.parquet` file becomes an ordinary `data-…parquet` filename. The new `CopiedDataFiles.checkNoColumnUpdates` then classifies both that dedicated file and its normal anchor as normal files. Since they correctly share a row-ID range, it rejects a healthy source as a column update. Reproduction on JDK 8: - Create a source schema `id INT, embedding VECTOR(2, FLOAT)`, with `row-tracking.enabled=true`, `data-evolution.enabled=true`, `file.format=parquet`, and `vector.file.format=parquet`. - Write and commit two rows `(1, [1.0, 0.0])`, `(2, [2.0, 0.0])` once. There are no updates, copy-on-write files, or schema changes. - Execute `CALL sys.copy(source_table => 'review_vector_src', target_table => 'review_vector_dst')`. Current code fails with `Cannot copy data files data-…parquet and data-…parquet … they hold the same rows, one updating columns of the other. Compact the source table first`. Reverting only the new `CopiedDataFiles.adaptToTarget` call makes the same real procedure succeed, and reading the target returns exactly `(1, [1.0, 0.0]), (2, [2.0, 0.0])`. Please preserve the dedicated file identity through target naming/metadata adaptation, so the overlap check is applied only to genuinely overlapping normal column files. Add a Vector case to the actual Spark procedure suite. `CopiedDataFilesTest.prepareCopy` preserves source filenames, unlike the production operator, so its dedicated-file coverage does not exercise this boundary. Validation: 197 core tests passed with standard Maven checks (one existing disabled Blob rename test); the existing 11 Spark copy cases passed in the reproduction run, and the additional real Vector case failed. The one-case control passed after reverting the new call. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
