FANNG1 opened a new issue, #11063: URL: https://github.com/apache/gravitino/issues/11063
### What would you like to be improved? When Spark queries Lance tables through Gravitino Lance REST, the Spark side still needs to provide the underlying storage configuration explicitly, such as: ```text spark.sql.catalog.lance.storage.endpoint=http://127.0.0.1:9000 spark.sql.catalog.lance.storage.aws_allow_http=true spark.sql.catalog.lance.storage.access_key_id=minioadmin spark.sql.catalog.lance.storage.secret_access_key=minioadmin spark.sql.catalog.lance.storage.region=us-east-1 ``` This makes the integration much more complicated than expected. In practice, users already register the Lance catalog in Gravitino, and Gravitino already knows the table location, such as `s3://contacts/raw/lance`. However, Spark still has to repeat the storage endpoint and credentials to make Lance queries work. This leads to several problems: 1. Spark configuration is too verbose for a simple federated query scenario. 2. Storage credentials are duplicated between Gravitino and Spark. 3. Users need to understand internal storage details even though they are already managed by Gravitino. 4. The integration experience is inconsistent with the goal of using Gravitino as the metadata and access entry point. For Spark + Gravitino + Lance integration, it would be better if users only needed to configure the Gravitino/Lance REST endpoint and catalog identity, without passing storage credentials again on the Spark side. ### How should we improve? Gravitino should manage and provide the Lance storage configuration for Spark queries, instead of requiring Spark to pass the storage parameters manually. Possible directions: 1. Allow the Lance catalog properties in Gravitino to include the required storage configuration, such as endpoint, region, path-style access, and credential provider information. 2. Make Lance REST read and use these managed properties when serving Spark requests. 3. If credentials need to be delegated, provide a Gravitino-managed mechanism similar in spirit to the Iceberg REST flow, so Spark does not need direct static storage credentials. 4. Keep the Spark-side configuration minimal, ideally only requiring: - Lance Spark extension - Lance catalog type/URI - Gravitino/Lance REST endpoint - metalake/catalog identification if needed Expected result: Users can query Lance tables from Spark with much simpler configuration, while Gravitino remains the single place to manage storage-related settings for Lance catalogs. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
