diqiu50 commented on code in PR #12710: URL: https://github.com/apache/gravitino/pull/12710#discussion_r3910329228
########## docs/spark-connector/spark-catalog-iceberg.md: ########## @@ -148,13 +148,111 @@ Gravitino catalog property names with the prefix `spark.bypass.` are passed to S Iceberg catalog property `cache-enabled` is setting to `false` internally and not allowed to change. ::: +## Routing Through the Gravitino Iceberg REST Server + +If the Gravitino server exposes an [Iceberg REST catalog](../iceberg-rest-service.md) (IRC) endpoint for the +current metalake, the Spark connector automatically routes `hive` and `jdbc` backed Iceberg catalogs through +that endpoint instead of talking to the Hive metastore or JDBC database directly. This has no effect on +catalogs whose `catalog-backend` is already `rest` or `custom`. + +Routing through the IRC server is the only way to receive short-lived, per-table **vended credentials** +that Iceberg's native REST protocol refreshes automatically. The non-REST path can still inject a single +vended credential fetched once at catalog initialization (see +[Credential vending](../security/credential-vending.md)), but it is not refreshed per table access. + +REST routing is enabled by default. The endpoint is discovered once per Spark application — for the life +of the Gravitino Spark plugin, not per `SparkSession` — and is not re-checked afterward. + +- If no discoverable endpoint is found (for example, the `iceberg-rest` auxiliary service is disabled or + not configured with `catalog-config-provider=dynamic-config-provider`) and routing was left at its + default, the connector falls back to the native Hive/JDBC backend and logs a warning. Set + `spark.sql.gravitino.iceberg.rest-routing-enabled=true` explicitly to require Iceberg REST routing and + fail catalog initialization instead. +- A catalog whose warehouse uses a scheme with a native Iceberg FileIO (`s3://`, `gs://`, `abfs://`, etc.) + must have [credential vending](../security/credential-vending.md) configured (`credential-providers`) + before it can be routed: routing replaces any static storage credentials for that FileIO with vended + ones, and without credential vending the catalog would lose storage access. Catalog initialization + fails with an actionable error if this is not configured. Set `rest-routing-enabled=false` for that + catalog to keep using the legacy Hive/JDBC backend instead. + +To force a specific endpoint instead of relying on auto-discovery, set: + +```properties +spark.sql.gravitino.iceberg.rest-uri http://<gravitino-host>:9001/iceberg +``` + +To retain the legacy Hive/JDBC translation and skip endpoint discovery, disable routing explicitly: + +```properties +spark.sql.gravitino.iceberg.rest-routing-enabled false +``` + +If Gravitino requires authentication on the IRC endpoint, pass the Iceberg REST client's own auth +properties using the `spark.sql.gravitino.iceberg.rest.` prefix. For example, for Basic authentication: + +```properties +spark.sql.gravitino.iceberg.rest.rest.auth.type basic +spark.sql.gravitino.iceberg.rest.rest.auth.basic.username <username> +spark.sql.gravitino.iceberg.rest.rest.auth.basic.password <password> +``` + +See [Connect Spark to Iceberg REST](../iceberg-rest-engine/spark.md) for the full set of supported +`rest.auth.*` properties and how to configure them when connecting directly to the IRC endpoint. + +When the Gravitino client uses OAuth2, the connector reuses its OAuth2 client configuration for IRC by +default. This avoids duplicating configuration when both endpoints accept the same client identity. The +equivalent explicit setting is: + +```properties +spark.sql.gravitino.iceberg.reuseOAuth2 true Review Comment: Matches the existing properties-block style used elsewhere in this doc -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
