FANNG1 opened a new pull request, #11280: URL: https://github.com/apache/gravitino/pull/11280
### What changes were proposed in this pull request? Add a design document `docs/unified-engine-access.md` proposing a provider-level `engine-access-mode` configuration for Gravitino connectors. The core idea: users only configure the Gravitino server address and metalake; the engine connector automatically registers engine-native catalogs (e.g. Iceberg REST via `SparkCatalog`, Lance REST via `LanceNamespaceSparkCatalog`) by translating existing Gravitino catalog properties, without requiring separate engine-side configuration. Key points in the design: - New config: `spark.sql.gravitino.<provider>.engine-access-mode = auto | gravitino | native` - `auto` selects `gravitino` if a Gravitino connector exists for the provider, otherwise `native` - Native connector config is derived from existing catalog properties (no new properties added) - Iceberg defaults to `gravitino` (preserving current behavior); Lance defaults to `native` - `GravitinoDriverPlugin` calls `listCatalogsInfo()` at startup and registers catalogs accordingly - Credential vending reuses Gravitino's existing credential infrastructure - Design is extensible to Flink, Trino, Doris, and Daft Open questions documented for community discussion: 1. Dedicated `lakehouse-lance` provider vs. `lakehouse-generic + format=lance` 2. Whether native access should be restricted to REST backends 3. DDL operations in native mode and metadata consistency 4. Full load vs. lazy load of catalogs at startup ### Why are the changes needed? In mixed Iceberg-and-Lance query scenarios, users currently must maintain multiple sets of engine-side catalog configurations that mirror Gravitino catalog definitions. Any change to a Gravitino catalog requires synchronized updates across Spark, Flink, Trino, and other engines. This proposal eliminates that duplication while preserving backward compatibility for existing Gravitino connector users. Fix: N/A (design proposal for community discussion) ### Does this PR introduce _any_ user-facing change? This PR only adds a documentation file. No code changes are included. The proposed configuration keys and behaviors described in the document are subject to community discussion. ### How was this patch tested? Documentation-only change; no code testing required. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
