dimas-b commented on code in PR #724:
URL: https://github.com/apache/polaris/pull/724#discussion_r1917396779
##########
service/common/src/main/java/org/apache/polaris/service/catalog/BasePolarisCatalog.java:
##########
@@ -1627,6 +1627,12 @@ private FileIO refreshIOWithCredentials(
// the credentials should always override table-level properties, since
// storage configuration will be found at whatever entity defines it
tableProperties.putAll(credentialsMap);
+
+ // Populate the internal properties to FileIO in case the FileIO
implementation needs them
+ Map<String, String> internalProperties =
+
storageInfoEntity.map(PolarisEntity::getInternalPropertiesAsMap).orElse(Map.of());
Review Comment:
> The Polaris FileIO here means the FileIO used by Polaris to access the
storage [...]
That is the situation now, but it does not have to be in the future. Suppose
Polaris code evolves and the way it accesses storage changes, yet existing
deployments will maintain property bags designed for use with Iceberg FileIO.
This will stifle Polaris Server evolution, IMHO.
From my POV, as far as Polaris backend is concerned it is preferable to
define clear interfaces (by extension clear meaning of properties) that are
under Polaris control so that when the backend evolves, those properties can
still be supported explicitly.
More specifically, I think in this case it might be worth adding a
translation layer from a fixed well-defined set of Polaris properties (set by
users) to Iceberg FileIO properties (translated by Polaris code). Does this
sound reasonable?
If the extra properties are meant for consumption by custom code extending
Polaris Backend services, that can still be supported, but it should be clear
that the interpretation of those properties is the responsibility of the
extension, not of Polaris basic code.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
