cshuo opened a new pull request, #19842:
URL: https://github.com/apache/hudi/pull/19842
### Describe the issue this Pull Request addresses
Closes #19821.
Following the Lance VECTOR writer support merged in #19831, Flink also needs
to read these columns as ArrayData and preserve their VECTOR schema identity.
The existing reader only handles variable-length Arrow lists.
### Summary and Changelog
- Read Arrow FixedSizeList values as Flink arrays while retaining ordinary
List support.
- Restore VECTOR schema identity from the Lance footer's vector-column
metadata and Arrow element type and dimension.
- Validate requested vector dimensions and element types against the file
schema.
- Add real Lance file round-trip coverage for FLOAT/DOUBLE vectors, null
vectors, ordinary arrays, reordered projections, schema restoration, and
incompatible dimensions.
- Add COW/MOR upsert-read coverage, including completed MOR compaction.
### Impact
Enables Flink to read top-level FLOAT and DOUBLE VECTOR columns from
Hudi-written Lance files. No configuration or file-format changes. INT8
vectors, nested vectors, and Lance schema evolution remain out of scope.
Reading materializes the elements as ArrayData using the existing array
conversion path.
### Risk Level
Medium. The change affects Lance array decoding and schema reconstruction.
Regression coverage includes ordinary arrays and the existing writer tests.
Validation on Flink 2.1: 21 tests passed, with no failures, errors, or
skipped tests; the Maven reactor and Checkstyle passed.
```bash
mvn -Pflink2.1 -pl hudi-flink-datasource/hudi-flink -am \
'-Dtest=TestHoodieFlinkLanceArrowUtils,TestLanceRowDataWriter,TestHoodieRowDataLanceReader,ITTestVectorDataSource#testLanceVectorUpsertRead'
\
-Dsurefire.failIfNoSpecifiedTests=false -DskipITs -DskipSparkTests
-DskipScalaTests test
```
### Documentation Update
The Flink VECTOR/Lance support documentation should describe FLOAT/DOUBLE
vector reads. No documentation changes are included in this PR.
### Contributor's checklist
- [ ] Read through [contributor's
guide](https://hudi.apache.org/contribute/how-to-contribute)
- [x] Enough context is provided in the sections above
- [x] Adequate tests were added if applicable
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
