yuqi1129 opened a new issue, #12682:
URL: https://github.com/apache/gravitino/issues/12682
### Version
main branch
### Describe what's wrong
The Paimon catalog fails when its warehouse uses the `s3a://` scheme because
`catalog-lakehouse-paimon` does not include `hadoop-aws` in its runtime
dependencies.
Paimon falls back to `HadoopFileIO` for an `s3a://` warehouse. Hadoop then
attempts to load
`org.apache.hadoop.fs.s3a.S3AFileSystem`, but that class is unavailable in
the Paimon catalog
runtime classpath.
### Error message and/or stacktrace
```text
java.lang.ClassNotFoundException:
Class org.apache.hadoop.fs.s3a.S3AFileSystem not found
at org.apache.hadoop.conf.Configuration.getClass(Configuration.java:2693)
at org.apache.hadoop.fs.FileSystem.getFileSystemClass(FileSystem.java:3573)
at
org.apache.paimon.fs.hadoop.HadoopFileIO.createFileSystem(HadoopFileIO.java:199)
at
org.apache.paimon.hive.HiveCatalog.createHiveCatalog(HiveCatalog.java:1685)
```
### How to reproduce
Create a Paimon catalog with a Hive backend and an S3A warehouse:
```java
Map<String, String> properties = new HashMap<>();
properties.put("catalog-backend", "hive");
properties.put("warehouse", "s3a://test-bucket/warehouse");
properties.put("uri", "thrift://localhost:9083");
Catalog catalog =
metalake.createCatalog(
"paimon_catalog",
Catalog.Type.RELATIONAL,
"lakehouse-paimon",
"Paimon catalog",
properties);
catalog.asSchemas().createSchema("test_schema", null,
Collections.emptyMap());
```
Schema creation fails with `S3AFileSystem` not found.
### Additional context
`catalog-lakehouse-paimon` includes Hadoop Common and HDFS dependencies but
not `hadoop-aws`.
A Java reproduction test confirmed the exception. Adding the following
runtime dependency removes
the `ClassNotFoundException` and successfully initializes the Hadoop S3A
filesystem:
```kotlin
runtimeOnly(libs.hadoop3.aws)
```
The fix should also add an integration test covering an `s3a://` Paimon
warehouse so the production
runtime dependency is verified.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
