zhuxiangyi commented on PR #10391: URL: https://github.com/apache/paimon/pull/10391#issuecomment-6187520188
@JingsongLi Thanks, reproduced: on 65c3b7b341 your steps fail with the same error. **Fix** (2fc3953975) - Cause: `ListDataFilesOperator.pickDataFiles` named every target file with `newPath()`, dropping the kind a dedicated file carries in its name. - It now uses `CopyFilesUtil.newDataFilePath`: a vector-store file keeps `.vector.<format>`, a blob file keeps its `.blob` / `.video` extension. - Blob files had the same problem, with a worse effect: on master a copied `.blob` file is renamed to `.parquet`, so reading the target fails right after the copy. **Tests** - A Vector case in `CopyFilesProcedureTest`, with your schema and writes. It fails before the fix with your message; after it, the target has the source rows and `_ROW_ID`s. - Cases for a blob table and a partitioned vector table. - `prepareCopy` is unchanged: the naming lives in the Spark module, which core tests cannot reach. `CopyFilesUtilTest` tests the naming instead. **Another problem** (18746b4c4f) This is a regression of the approach in 65c3b7b (the commit assigns the row ids). The commit gives vector-store files row ids from one counter for the whole commit, which does not skip normal files without a vector-store file: - a vector column added later: the copy succeeds but silently attaches vectors to other rows; - only the vector column updated (or a blob column): the commit fails with `IllegalStateException: This is a bug…`. The new `checkDedicatedFilesFollowTheirRows` refuses such copies before the commit, asking to rewrite those rows in the source with `INSERT OVERWRITE` first (verified to work). I refuse rather than support them because supporting them means changing how `RowTrackingCommitUtils` assigns row ids, which every write goes through. I can do that in a separate PR if you prefer. **Validation** (JDK 8, Spark 3) - Spark copy cases 17/17 (11 existing + 6 new) - `CopiedDataFilesTest` 17/17, `CopyFilesUtilTest` 4/4 - spotless and checkstyle clean; rebased on the latest master, so the commit of the previous round is now 0949d5a1d0 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
