dengc367 commented on code in PR #14352:
URL: https://github.com/apache/gravitino/pull/14352#discussion_r4419938494
##########
flink-connector/flink-common/src/main/java/org/apache/gravitino/flink/connector/paimon/GravitinoPaimonCatalog.java:
##########
@@ -197,6 +199,24 @@ private static void movePaimonStorageOptionsToConf(
paimonOptions, hadoopConf, PaimonConstants.OSS_SECRET_KEY,
PaimonConstants.OSS_SECRET_KEY);
}
+ /**
+ * Paimon's OSSFileIO (paimon-oss 1.x) builds its Hadoop configuration only
from the catalog
+ * options and never reads the Hadoop configuration, so after the move above
the OSS endpoint and
+ * keys would be invisible to it ("Aliyun OSS endpoint should not be null or
empty"). Hand those
+ * entries back to the inner catalog's options; they stay out of this
connector's own option map
+ * and other filesystems keep using the Hadoop configuration.
+ */
+ @VisibleForTesting
+ static void copyOssOptionsBackToPaimonOptions(
Review Comment:
Thanks @diqiu50, all three points are valid. Updated in bc76f48:
- `copyOssOptionsBackToPaimonOptions()` now returns early unless the
warehouse scheme is `oss://`, so `fs.oss.*` entries that come from
`core-site.xml` under `HADOOP_CONF_DIR` stay in the Hadoop configuration for
HDFS/S3/local warehouses and are not copied into Paimon options.
- Dropped the `containsKey` guard — you're right that it was always true:
`extractHadoopConfiguration` strips every `fs.*`/`dfs.*`/`hadoop.*` key before
this point, and `movePaimonStorageOptionsToConf` removes whatever credential
vending injected afterwards.
- Fixed the `mutableOptions` comment and the `createInnerCatalog` `@param`
javadoc, which no longer claim the inner catalog options are sanitized for OSS
warehouses.
- Added `testOpenKeepsOssOptionsOutOfPaimonOptionsForNonOssWarehouse`: with
`warehouse=s3://...`, `fs.oss.*` stays out of the captured Paimon options while
still reaching the Hadoop configuration.
`TestGravitinoPaimonCatalog` passes on Flink 1.19/1.20/2.1/2.2 (16 tests
each).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]