FANNG1 opened a new issue, #11064: URL: https://github.com/apache/gravitino/issues/11064
### What would you like to be improved? Gravitino should provide a better integration model with Gravitino Iceberg REST and Lance REST for Spark clients. Today, when Spark queries Iceberg and Lance tables through Gravitino, the client still needs complex per-catalog configuration. In practice, users need to configure Iceberg REST and Lance REST catalogs separately in Spark, and each catalog requires its own settings such as catalog type, URI, extensions, and sometimes storage-related properties. This creates several problems: 1. The client configuration is too complex for federated query scenarios. 2. Spark clients need to maintain catalog-by-catalog configuration outside Gravitino. 3. When a new catalog is added in Gravitino, the Spark client configuration also needs to be updated and refreshed. 4. The client-side operational burden grows as the number of catalogs increases. 5. This weakens the value of Gravitino as a unified metadata and access layer. For example, to query Iceberg and Lance together from Spark, the client may need configuration like this: ```text spark.plugins=org.apache.gravitino.spark.connector.plugin.GravitinoSparkPlugin spark.sql.extensions=org.lance.spark.extensions.LanceSparkSessionExtensions spark.sql.gravitino.uri=http://127.0.0.1:8090 spark.sql.gravitino.metalake=test spark.sql.gravitino.enableIcebergSupport=true spark.sql.catalog.iceberg_rest=org.apache.iceberg.spark.SparkCatalog spark.sql.catalog.iceberg_rest.type=rest spark.sql.catalog.iceberg_rest.uri=http://127.0.0.1:9001/iceberg/ spark.sql.catalog.iceberg_rest.header.X-Iceberg-Access-Delegation=vended-credentials spark.sql.catalog.lance=org.lance.spark.LanceNamespaceSparkCatalog spark.sql.catalog.lance.impl=rest spark.sql.catalog.lance.uri=http://127.0.0.1:9101/lance spark.sql.catalog.lance.parent=lance_catalog spark.sql.catalog.lance.storage.endpoint=http://127.0.0.1:9000 spark.sql.catalog.lance.storage.aws_allow_http=true spark.sql.catalog.lance.storage.access_key_id=minioadmin spark.sql.catalog.lance.storage.secret_access_key=minioadmin spark.sql.catalog.lance.storage.region=us-east-1 ``` This is hard to maintain on the client side. If a new Iceberg REST catalog or Lance REST catalog is added in Gravitino, the Spark client may also need new catalog entries or updated configuration before it can use that catalog. Ideally, users should only need to configure the Gravitino server address in Spark, and then be able to run federated query and analysis across Iceberg and Lance tables managed by Gravitino, without manually defining each underlying REST catalog on the client side. Example of the expected experience: ```text spark.sql.gravitino.uri=http://127.0.0.1:8090 spark.sql.gravitino.metalake=test ``` Then Spark can directly discover and use Iceberg REST and Lance REST catalogs managed by Gravitino for unified query and analysis. ### How should we improve? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
