markhoerth opened a new issue, #12689:
URL: https://github.com/apache/gravitino/issues/12689
### Version
main branch
### Describe what's wrong
### Version
trino-irc-1.3 / branch-1.3
### Describe what's wrong
On a Trino cluster with workers, no Gravitino-registered catalog can be
queried when the Gravitino client uses oauth2. Catalogs register on the
coordinator and metadata reads work, so the cluster looks healthy. Any
query that dispatches a task to a worker fails.
On a freshly started worker:
Initialization of the GravitinoConnector failed Invalid Gravitino client
configuration for authType 'oauth2': OAuth credential is required.
Please set gravitino.client.oauth2.credential
at GravitinoAuthProvider.buildOAuthProvider(GravitinoAuthProvider.java:318)
at GravitinoAuthProvider.build(GravitinoAuthProvider.java:149)
at CatalogConnectorManager.config(CatalogConnectorManager.java:127)
at GravitinoConnectorFactory.create(GravitinoConnectorFactory.java:132)
`toCatalogConfig` omits `gravitino.client.oauth2.credential` from the
generated CREATE CATALOG because it is flagged security sensitive. The
worker receives `path`, `scope` and `serverUri` but no credential, so it
cannot build a client.
On a worker that has been up longer the same failure surfaces as an NPE
at `CatalogConnectorManager.retrieveMetalake` line 258, because
`createCatalogConnectorContext` calls
`metalakes.computeIfAbsent(catalog.getMetalake(), this::retrieveMetalake)`
unconditionally and the client is null. Same root cause, two messages.
Scope is every provider, not one adapter. Reproduced with a
lakehouse-iceberg catalog and an elasticsearch catalog. It does not
self-recover and it survives worker restarts.
The same redaction breaks IRC routing: a routed catalog fails its first
`fetchConfig` with NotAuthorizedException for the same reason. One
omission, two failures.
There is no configuration workaround.
`trino.bypass.gravitino.client.oauth2.credential` does not reach the
Gravitino client config, and it stops the catalog registering at all
until removed.
### Note on the fix
Keeping the secret out of a logged CREATE statement is the right goal,
and the environment-variable mechanism is the right answer: it is how
Trino already expects secrets to be supplied, `${ENV:...}` is native
config syntax, and on Kubernetes the value comes from a Secret and never
lands in a file, a statement, or query history.
Two properties per secret are required — the literal set to the
`${ENV:...}` placeholder plus the
`gravitino.dynamic-catalog.environment-variable.` mapping.
What needs to ship with it:
- [ ] The mechanism documented on the connector page. Nothing in the
installation doc or the PR mentions it today.
- [ ] An error that names the missing property and the environment
variable. The current message says the OAuth credential is required,
which points at the entry catalog where the credential is in fact
present.
- [ ] A release note stating that multi-node oauth2 deployments must
configure the mapping on the coordinator and every worker before
upgrading. Registration still succeeds and metadata still lists after
an upgrade, so the cluster looks fine and the failure only appears
when a query reaches a worker.
- [ ] Confirmation of whether other sensitive properties are affected,
`trino.jdbc.password` in particular.
### Not claimed
This was reproduced on the current build. No controlled comparison was
run against the previous one, so this is not being claimed as a
regression.
### Error message and/or stacktrace
na
### How to reproduce
na
### Additional context
na
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
