pzhdfy commented on code in PR #69493:
URL: https://github.com/apache/doris/pull/69493#discussion_r4360683823
##########
fe/fe-core/src/main/java/org/apache/doris/datasource/property/metastore/AbstractPaimonProperties.java:
##########
@@ -123,6 +124,34 @@ protected void appendUserHadoopConfig(Configuration conf)
{
normalizeS3Config().forEach(conf::set);
}
+ /**
+ * Mirrors the S3 credentials Doris assembled into the Hadoop configuration
+ * onto the catalog options, so the pinned Paimon 2.1 SDK sees them.
+ *
+ * <p>Doris stores a catalog's storage credentials (fs.s3a.access.key,
fs.s3a.secret.key,
+ * fs.s3a.endpoint, ...) only in the Hadoop Configuration handed to
+ * CatalogContext.create(options, conf) — the paimon 1.4 FileIO consulted
exactly
+ * that conf. The pinned 2.1 S3FileIO instead reads its credentials
exclusively from
+ * CatalogContext.options() (S3FileIO#configure ->
loadHadoopConfigFromContext copies
+ * the s3./s3a./fs.s3a.-prefixed option entries and never looks at the
conf), so
+ * without this mirror every s3-warehouse catalog initializes the SDK with
no
+ * credentials and the first file operation fails on the ambient AWS
provider chain
+ * ("Unable to load credentials from system settings"). Mirroring the
whole fs.s3a.*
+ * set keeps user extras (path style, connection limits) flowing as well;
entries
+ * outside the prefix are not the SDK's concern.
+ */
+ protected void mirrorHadoopS3OptionsToCatalogOptions(Configuration conf) {
+ if (conf == null || catalogOptions == null) {
+ return;
+ }
+ conf.forEach(entry -> {
+ String key = entry.getKey();
+ if (key.startsWith(HADOOP_S3A_PREFIX) &&
StringUtils.isNotBlank(entry.getValue())) {
+ catalogOptions.set(key, entry.getValue());
Review Comment:
Confirmed and fixed in af5c7cc: the Hadoop `fs.s3a.*` mirror now fills only
options the catalog has not already set, so an explicit assumed-role
credentials provider (set via a `paimon.fs.s3a.*` property) keeps precedence
over the SimpleAWSCredentialsProvider the generated Hadoop configuration
carries; absent defaults are still filled. Covered by
`testHadoopS3MirrorDoesNotOverwriteExplicitProvider` (13/13 properties UTs
green). This affects the filesystem, HMS and JDBC paimon catalog variants
alike.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
