cshuo opened a new pull request, #19903: URL: https://github.com/apache/hudi/pull/19903
### Describe the issue this Pull Request addresses Closes #19857. Flink CDC base-file reads do not consistently apply VECTOR decoding or preserve VECTOR schema metadata. Parquet stores VECTOR values as binary data, while Lance needs the requested VECTOR schema to retain element type and dimension information. ### Summary and Changelog - Reuse the existing Parquet VECTOR conversion in `HoodieCdcSplitReaderFunction` and `MergeOnReadInputFormat`, and pass the requested Hudi schema to Lance readers. - Lazily initialize the schema and conversion metadata needed by each file format and reuse them across base files. - Reject VECTOR columns with `DATA_BEFORE` or `DATA_BEFORE_AFTER` logging during table creation, write-client schema validation, and `commitTableChange`. Explain why image logging would require unsupported nested VECTOR fields and recommend `OP_KEY_ONLY`. - Add real Parquet/Lance file tests for both Flink CDC readers and COW/MOR tables, covering base-file inference, native `AS_IS + OP_KEY_ONLY`, nullable vectors, projection order, ordinary arrays, CDC row kinds, and Lance dimension validation. Validation: 105 targeted tests passed with JDK 17 and the `flink2.2` profile, including `TestHoodieCDCUtils`, `TestHoodieTableMetaClient#testCdc*`, `TestBaseHoodieWriteClient#validate*`, `TestCdcVectorRead`, `TestHoodieCdcSplitReaderFunction`, `TestCdcIterators`, and `TestCdcImageManager`. `git diff --check` also passed. ### Impact Flink CDC can return VECTOR values as Flink arrays through the corrected base-file paths. VECTOR tables using CDC must use `OP_KEY_ONLY`; image logging is rejected at the validation entry points above. Existing VECTOR encodings and ordinary ARRAY handling are preserved. ### Risk Level Low. The reader changes reuse existing conversion utilities and stay within the base-file read paths. Targeted tests cover both formats and both CDC reader implementations, as well as supported and rejected logging configurations. The unchanged `LOG_FILE` inference path still lacks dedicated VECTOR coverage and is a follow-up testing opportunity. Inline CDC logs are outside this change's test scope. ### Documentation Update Added method documentation explaining the image-logging restriction and why `OP_KEY_ONLY` remains supported. The validation error includes the configuration needed to select `OP_KEY_ONLY`. ### Contributor's checklist - [ ] Read through [contributor's guide](https://hudi.apache.org/contribute/how-to-contribute) - [x] Enough context is provided in the sections above - [x] Adequate tests were added if applicable -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
