leekeiabstraction commented on code in PR #2569:
URL: https://github.com/apache/fluss/pull/2569#discussion_r2784129548
##########
website/docs/quickstart/lakehouse.md:
##########
@@ -276,12 +358,14 @@ services:
cp /tmp/opt/*.jar /opt/flink/opt/ 2>/dev/null || true;
/docker-entrypoint.sh jobmanager"
environment:
+ - AWS_ACCESS_KEY_ID=rustfsadmin
Review Comment:
Unfortunately not. `CatalogUtil.loadCatalog()` instantiate `HadoopCatalog`
by using hadoop configuration obtained via Flink environment and then call
`HadoopCatalog.initialize()` on it using properties from CREATE CATALOG DDL;
See code below on how only the Hadoop config is used to create the file system
via `Util.getFs()`, the properties which were derived from DDL is not used to
create the file system.
```java
public HadoopCatalog(Configuration conf, String warehouseLocation) {
this.setConf(conf);
...
}
public void initialize(String name, Map<String, String> properties) {
this.catalogProperties = ImmutableMap.copyOf(properties);
String inputWarehouseLocation = (String)properties.get("warehouse");
Preconditions.checkArgument(!Strings.isNullOrEmpty(inputWarehouseLocation),
"Cannot initialize HadoopCatalog because warehousePath must not be null or
empty");
this.catalogName = name;
this.warehouseLocation =
LocationUtil.stripTrailingSlash(inputWarehouseLocation);
this.fs = Util.getFs(new Path(this.warehouseLocation), this.conf);
String fileIOImpl = (String)properties.getOrDefault("io-impl",
"org.apache.iceberg.hadoop.HadoopFileIO");
this.fileIO = CatalogUtil.loadFileIO(fileIOImpl, properties,
this.conf);
this.lockManager = LockManagers.from(properties);
this.closeableGroup = new CloseableGroup();
this.closeableGroup.addCloseable(this.lockManager);
this.closeableGroup.addCloseable(this.metricsReporter());
this.closeableGroup.setSuppressCloseFailure(true);
this.suppressPermissionError =
Boolean.parseBoolean((String)properties.get("suppress-permission-error"));
}
```
I have found the experience of configuring Paimon/Iceberg quite inconsistent
(it wasn't just the credentials config, the path and endpoint as well) not
straightforward. Perhaps we should consider improving this from the fluss side
e.g. consolidating the format of configuration key, ensuring that
configurations are threaded through properly from Fluss configurations as
Iceberg / Paimon handles Hadoop configurations and catalog properties
differently.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
