markhoerth opened a new issue, #12687:
URL: https://github.com/apache/gravitino/issues/12687
### What would you like to be improved?
### What this covers
Six items found while building and testing the Spark connector from
the trino-irc-1.3 branch against a customer cluster running Spark
3.5.6 / Scala 2.12.18 with Iceberg runtime 1.8.0. The connector works
there: it registers the metalake's catalogs, routes Iceberg through
the Iceberg REST service, and reads S3 data using credentials vended
by Gravitino at query time with no static S3 keys in the session.
These are the changes needed before it ships. Items 1 through 3 were
discussed and agreed in principle; this issue records them so they can
be tracked to completion.
### Items
- [ ] **1. Keep the IRC credential separately configurable.**
`IcebergRestOAuthConfig` reuses the Gravitino client's OAuth2
credential for the routed Iceberg REST catalog. Shared configuration
is the right default and should stay. What is needed is the ability
to override it, because the Iceberg REST service is a separate
endpoint on a separate port and a deployment may use a different
client for data-plane access than for metadata access. The Trino
connector already keeps the two as separate properties, and the
environment-variable example for the credential fix uses two
variables for exactly this reason. Both engines should default to
reuse and both should allow a distinct IRC credential. Do not remove
the separate Trino properties to match Spark.
- [ ] **2. Partial explicit auth must not silently produce a broken
config.** `hasExplicitAuthentication` matches on any one legacy key,
so setting only the credential disables reuse entirely and the
catalog ends up with no `rest.auth.type`, no scope and no
`oauth2-server-uri`. Either fill in the remainder from the shared
config or fail with an error naming what is missing.
- [ ] **3. Spark and Trino need the same behavior when no IRC endpoint
is discovered.** Spark falls back silently to hive/jdbc translation.
Trino as of 087a95bff throws. Same branch, same feature, opposite
default.
- [ ] **4. Spark discovery runs once at catalog init and is never
re-checked.** A server that is briefly unreachable when a session
starts leaves that session unrouted for its entire life, with no
error. Discovery needs to retry, and the unrouted state needs to be
visible to the user.
- [ ] **5. Document the OAuth reuse path.** The Spark docs cover Basic
auth only. They do not mention OAuth reuse or
`spark.sql.gravitino.iceberg.reuseOAuth2`, which is the property most
users will need.
- [ ] **6. Spark needs the same escape hatch as Trino, and both need
release notes.** With rest-routing enabled by default, a customer on
a released 1.3 server has every non-REST Iceberg catalog change
behavior. Trino has `gravitino.iceberg.rest-routing-enabled`; Spark
needs the equivalent. Both are behavior changes for existing
deployments and belong in the 1.3.1 release notes.
### Environment note for the docs
Spark images that ship only the version 1 AWS SDK for Hadoop S3A need
`iceberg-aws-bundle` on the classpath before `S3FileIO` can
initialize. The Storage section mentions this; it is worth calling out
explicitly, because it fails with a `NoClassDefFoundError` that does
not obviously point at a missing bundle.
### Branch note
The Spark work is on `trino-irc-1.3` while the Trino connector
deployed at the customer came off `trino-irc-1.3-es`. The branches
should be aligned so there is one thing to ship.
### How should we improve?
_No response_
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
