cshuo opened a new pull request, #19866: URL: https://github.com/apache/hudi/pull/19866
### Describe the issue this Pull Request addresses Closes #19864. A Flink job using `hoodie.table.base.file.format=LANCE` fails when initializing the writer with `ClassNotFoundException: org.apache.arrow.memory.BufferAllocator`. The bundle includes Hudi's Lance readers and writers but omits their runtime dependencies from the shade allowlist. ### Summary and Changelog - Include `lance-core`, the six Arrow vector/format/memory/C Data artifacts, `jar-jni`, FlatBuffers, and Netty buffer/common needed for Lance file I/O. Exclude the unused Arrow dataset and Lance namespace client dependency chains. - Relocate Netty, including Arrow's buffer patch, FlatBuffers, and the Java JNI loader into the Hudi namespace. Preserve Arrow and Lance package names required by native bindings. - Add a Lance `ARRAY<FLOAT>` insertion smoke test to the existing Flink bundle validation flow. ### Impact Flink deployments can use Lance base-file I/O with the Hudi bundle without manually installing Arrow/Lance JARs. Including native binaries increases the bundle size. No configuration or public API changes. ### Risk Level Medium: additional libraries and relocations affect runtime class loading and JNI initialization. Validation performed: - Built the Flink 2.1 and 2.2 bundles and verified Lance array writes and read-back values on local Flink 2.1.1 and 2.2.1 standalone clusters using `sql-client.sh -j`. - Ran 15 isolated dependency ablations: removing the dataset/namespace chains still passed; removing any of the remaining 11 artifacts failed. - Verified native file round trips covering fixed-size vectors, nulls, metadata, and multiple batches. Repeated with older Netty/FlatBuffers ahead of the bundle on the classpath and confirmed the relocated classes were loaded from the bundle. - Checked shell syntax and diff whitespace. Maven unit tests were skipped for the packaging builds. ### Documentation Update None. ### Contributor's checklist - [ ] Read through [contributor's guide](https://hudi.apache.org/contribute/how-to-contribute) - [x] Enough context is provided in the sections above - [x] Adequate tests were added if applicable -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
