dengc367 opened a new issue, #14351:
URL: https://github.com/apache/gravitino/issues/14351
### Version
main branch
### Describe what's wrong
Opening a Paimon catalog whose warehouse is on Aliyun OSS through the Flink
connector fails with
`Aliyun OSS endpoint should not be null or empty`, even though
`fs.oss.endpoint`,
`fs.oss.accessKeyId` and `fs.oss.accessKeySecret` are set on the Gravitino
catalog.
`GravitinoPaimonCatalog#open()` moves every `fs.*` option into the Hadoop
`Configuration` before
creating the inner Paimon catalog (#12121). That is correct for
`HadoopFileIO` and `S3FileIO`,
which read the Hadoop configuration. Paimon's `OSSFileIO`, the default
`FileIO` for OSS
warehouses, instead builds its Hadoop configuration only from
`CatalogContext#options()` and
never reads `CatalogContext#hadoopConf()`. After the move the `fs.oss.*`
entries are no longer in
the options, so `OSSFileIO` sees an empty endpoint.
The same warehouse works when the Paimon catalog is configured directly in
Flink, which is what
makes the connector look broken rather than misconfigured.
### Error message and/or stacktrace
```
java.lang.IllegalArgumentException: Aliyun OSS endpoint should not be null
or empty.
at org.apache.paimon.fs.FileIO.getOptions(FileIO.java)
at org.apache.paimon.oss.OSSFileIO.<init>(OSSFileIO.java)
```
### How to reproduce
1. Create a Paimon catalog in Gravitino with
`warehouse=oss://<bucket>/<path>` and the OSS
endpoint/key options, using the Flink connector on Flink
1.19/1.20/2.1/2.2.
2. Open the catalog from Flink (`USE CATALOG <catalog>;` or any DDL against
it).
3. The catalog fails with the error above.
### Additional context
Only the `fs.oss.*` options are affected. The fix is to copy them back into
the inner catalog's
option map after the Hadoop move, leaving every other filesystem on the
Hadoop configuration.
I have the patch and a test ready and will open a PR.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]