cshuo opened a new pull request, #19831: URL: https://github.com/apache/hudi/pull/19831
### Describe the issue this Pull Request addresses Closes #19820. Flink can identify VECTOR columns through the existing Hoodie schema, but the Lance writer previously converted only from Flink `RowType`. This lost the distinction between VECTOR and regular ARRAY columns, so Flink could not write the Lance fixed-size-list representation used by the Spark path. ### Summary and Changelog - Pass the Hoodie write schema through the Flink Lance writer and use it to distinguish VECTOR columns from regular ARRAY columns. - Encode top-level FLOAT and DOUBLE VECTOR columns as Arrow `FixedSizeList` values, validate their configured dimensions, and reject unsupported element types with a clear error. - Persist `hoodie.vector.columns` in Lance schema metadata so VECTOR identity is available to other engines. - Add focused schema-conversion, row-writer, and Lance file-writer coverage for VECTOR values, nulls, dimensions, metadata, and unsupported types. ### Impact Flink can now write top-level FLOAT and DOUBLE VECTOR columns to Lance base files using the existing vector-column configuration. Regular ARRAY columns retain their existing Lance representation. No new configuration is introduced. The writer performs a constant-time dimension check per non-null VECTOR value and does not add a separate element pre-scan. ### Risk Level medium. This changes the Lance on-disk encoding selected for configured VECTOR columns and threads the Hoodie schema through the internal Flink Lance writer. The encoding follows the existing Spark Lance representation. The targeted Flink Lance test suite passed 13 tests with no failures, errors, or skips. ### Documentation Update None. ### Contributor's checklist - [ ] Read through [contributor's guide](https://hudi.apache.org/contribute/how-to-contribute) - [ ] Enough context is provided in the sections above - [ ] Adequate tests were added if applicable -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
