danhuawang opened a new issue, #11687:
URL: https://github.com/apache/gravitino/issues/11687
### What would you like to be improved?
When an Iceberg catalog uses a **JDBC metadata backend** and is accessed by
an engine that loads a *native* Iceberg `JdbcCatalog` on the client side (the
Flink/Spark `lakehouse-iceberg` connectors), the JDBC user/password can only
reach the client through credential vending. The catalog properties `jdbc-user`
/ `jdbc-password` are marked `hidden` (`IcebergCatalogPropertiesMetadata`), so
they are never sent to the client directly; the `jdbc-user-password` credential
provider is the only delivery channel.
The server auto-derives this provider in
`IcebergCatalog.addCatalogSpecificCredentialProviders` when
`catalog-backend=jdbc` and JDBC credentials are present. However,
`BaseCatalog.propertiesWithCredentialProviders` short-circuits the entire
derivation as soon as the user has set `credential-providers` explicitly:
```java
// core/src/main/java/org/apache/gravitino/connector/BaseCatalog.java
if
(StringUtils.isNotBlank(props.get(CredentialConstants.CREDENTIAL_PROVIDERS))) {
return props; // auto-derivation (including JDBC) is skipped entirely
}
```
As a result, a common and reasonable configuration silently breaks JDBC
access. For an S3-backed, JDBC-metadata Iceberg catalog, a user naturally sets:
```
credential-providers = s3-token
```
This opts into S3 token vending but, as a side effect, suppresses the
functionally-required `jdbc-user-password` provider. The Flink connector then
opens the native `JdbcCatalog` with no password and fails:
```
org.apache.iceberg.jdbc.UncheckedSQLException: Failed to connect:
jdbc:postgresql://.../gravitinoirc3
Caused by: org.postgresql.util.PSQLException: The server requested
SCRAM-based authentication,
but no password was provided.
```
The coupling is the core problem: storage credential vending (`s3-token`,
`oss-token`, ...) is genuinely *optional* and a legitimate user choice, but
`jdbc-user-password` for a JDBC backend is *required for the catalog to open at
all*. They should not share a single all-or-nothing switch.
Note the failure is engine-specific. Trino (which talks to the Iceberg REST
service, where the server holds the JDBC credentials) works fine with the same
catalog; only engines that connect to the JDBC backend directly are affected.
This asymmetry makes the behavior surprising.
### How should we improve?
Stop treating an explicit `credential-providers` as a reason to skip *all*
auto-derivation. Instead, always union the catalog-specific providers that are
required for the backend to function (notably `jdbc-user-password`) with the
explicitly configured list, deduplicating the result.
Sketch for `BaseCatalog.propertiesWithCredentialProviders`:
```java
public Map<String, String> propertiesWithCredentialProviders() {
Map<String, String> props = Maps.newHashMap(entity().getProperties());
List<String> providers = new ArrayList<>();
String explicit = props.get(CredentialConstants.CREDENTIAL_PROVIDERS);
if (StringUtils.isNotBlank(explicit)) {
providers.addAll(Splitter.on(',').trimResults().omitEmptyStrings().splitToList(explicit));
}
// Always ensure backend-required (e.g. JDBC) providers are present.
addCatalogSpecificCredentialProviders(props, providers);
if (!providers.isEmpty()) {
props.put(
CredentialConstants.CREDENTIAL_PROVIDERS,
providers.stream().distinct().collect(Collectors.joining(",")));
}
return props;
}
```
Open design decision to settle in the PR:
- **Minimal (preferred):** force-add only the backend-required provider
(`jdbc-user-password`), and keep "explicit wins" for *storage* providers so
users can still opt out of S3/OSS/Azure/GCS vending. This preserves existing
behavior for storage while fixing the broken JDBC case.
- **Broad:** union storage providers too. Simpler mental model, but removes
the ability to disable auto-detected storage vending, which some users may rely
on.
I lean toward the minimal option. A unit test should cover:
`catalog-backend=jdbc` + `credential-providers=s3-token` yields a provider set
containing both `s3-token` and `jdbc-user-password`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
