cshuo opened a new issue, #19864:
URL: https://github.com/apache/hudi/issues/19864
### Describe the problem
The Hudi Flink bundle does not package the runtime dependencies required by
the Lance file format. A Flink job that deploys only
`hudi-flink2.2-bundle-1.3.0-SNAPSHOT.jar` fails as soon as it initializes the
Lance writer.
This was reproduced with a local Flink 2.2.1 standalone cluster and a Hudi
table configured with `hoodie.table.base.file.format=LANCE`.
The first missing class is from Arrow:
```text
java.lang.NoClassDefFoundError: org/apache/arrow/memory/BufferAllocator
at
org.apache.hudi.io.memory.HoodieArrowAllocator.getRootAllocator(HoodieArrowAllocator.java:67)
at
org.apache.hudi.io.memory.HoodieArrowAllocator.newChildAllocator(HoodieArrowAllocator.java:79)
at
org.apache.hudi.io.lance.HoodieBaseLanceWriter.<init>(HoodieBaseLanceWriter.java:92)
at
org.apache.hudi.io.storage.row.HoodieRowDataLanceWriter.<init>(HoodieRowDataLanceWriter.java:75)
Caused by: java.lang.ClassNotFoundException:
org.apache.arrow.memory.BufferAllocator
```
Inspection of the generated bundle confirms that it contains the Hudi Lance
reader/writer classes, but not the corresponding `org.apache.arrow` or
`org.lance` runtime classes. The Flink bundle shade configuration uses an
artifact allowlist that currently omits these dependencies.
The job can proceed only after manually adding the Lance and Arrow runtime
jars, including `lance-core`, the Arrow vector/memory/C-data/dataset artifacts,
and their required JNI/runtime dependencies, to Flink's `lib` directory.
### Expected behavior
The Hudi Flink bundle should contain the dependencies required to use the
Lance base-file format, so that Lance read and write paths work when the bundle
is deployed without manually installing additional Lance/Arrow jars.
### Environment
- Flink: 2.2.1
- Hudi: current `1.3.0-SNAPSHOT`
- Deployment: local standalone cluster
- Artifact: `hudi-flink2.2-bundle-1.3.0-SNAPSHOT.jar`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
