XJDKC commented on code in PR #724:
URL: https://github.com/apache/polaris/pull/724#discussion_r1917111359
##########
service/common/src/main/java/org/apache/polaris/service/catalog/BasePolarisCatalog.java:
##########
@@ -1627,6 +1627,12 @@ private FileIO refreshIOWithCredentials(
// the credentials should always override table-level properties, since
// storage configuration will be found at whatever entity defines it
tableProperties.putAll(credentialsMap);
+
+ // Populate the internal properties to FileIO in case the FileIO
implementation needs them
+ Map<String, String> internalProperties =
+
storageInfoEntity.map(PolarisEntity::getInternalPropertiesAsMap).orElse(Map.of());
Review Comment:
> there is no "Polaris FileIO"
The `Polaris FileIO` here means the FileIO used by Polaris to access the
storage, for aws s3, it's `S3FileIO`, for Azure Storage, it's `ADLSFileIO`. It
could also be a custom FileIO since we can inject FileIOFactory
> Is rather to have have properly typed configurations rather than two
generic property bags
But in the current Iceberg SDK and Polaris pattern, customers can add custom
properties to the `properties` map (the public map, not the
`internalProperties`) in PolarisCatalogEntity. For example, if customers want
to pass certain properties to Polaris’s FileIO, they can use property names
with the prefix `table-defaults.`. However, Polaris itself lacks a mechanism to
pass additional internal properties to FileIO.
> this change is meant to deal with credentials
This change is not meant to deal with credentials, **credentials are already
passed to FileIO without this change** (Code Pointer:
[BasePolarisCatalog.java#L1589-L1619](https://github.com/apache/polaris/blob/734e6979ccd010109323544c9f6dc9f0dcaecb8b/service/common/src/main/java/org/apache/polaris/service/catalog/BasePolarisCatalog.java#L1589-L1619
)). This change is trying to pass some existing storage config to FileIO.
Also this change doesn't add any new property in the `internalProperties` map,
for now, `internalProperties` map contains the following info:
```json
{
"storage_configuratio_info": "{
"@type" : "AwsStorageConfigurationInfo",
"storageType" : "S3",
"allowedLocations" : [ "s3://externally-owned-bucket",
"s3://my-bucket/path/to/data" ],
"roleARN" : "arn:aws:iam::012345678901:role/jdoe",
"fileIoImplClassName" : "org.apache.iceberg.aws.s3.S3FileIO",
"externalId" : "externalId"
}"
}
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
