jarredhj0214 opened a new issue, #58444:
URL: https://github.com/apache/spark/issues/58444
### Problem
We are using Spark 3.3.2 with Apache Gravitino and observed that all
top-level V2 catalogs need to be registered eagerly through Spark configuration:
```properties
spark.sql.catalog.<catalog_name>=<catalog_class>
```
This appears to be a general limitation of Spark's V2 catalog lookup model
rather than a version-specific bug. I also checked the current catalog loading
path in master, and the same static-registration model still applies.
When Spark resolves a multipart identifier such as:
```sql
SELECT * FROM catalog_a.db.table
```
`CatalogManager.catalog("catalog_a")` eventually calls
`Catalogs.load("catalog_a", conf)`. If `spark.sql.catalog.catalog_a` is not
configured, Spark throws `CatalogNotFoundException` immediately.
Spark supports lazy instantiation for configured catalogs, but it does not
currently support lazy discovery for catalog names managed by an external
catalog service.
### Motivation
External catalog services, such as Apache Gravitino or similar enterprise
catalog services, may manage a large number of catalogs. Since Spark requires
each top-level catalog name to be configured before analysis, connectors have
to eagerly materialize all visible catalog names into Spark configuration
during application startup.
This has several drawbacks:
- Spark application startup becomes slower as the number of catalogs grows.
- Catalogs created after Spark application startup are not naturally
discoverable.
- Connectors have to preload catalog names only to satisfy Spark's static
registration model.
- The actual catalog implementation may still be initialized lazily, but the
top-level catalog name must be eagerly registered.
For example, an external catalog service connector may currently need to
list visible catalogs at driver initialization time and register entries like:
```properties
spark.sql.catalog.catalog_a=...
spark.sql.catalog.catalog_b=...
spark.sql.catalog.catalog_c=...
```
even if the Spark application only accesses one of them.
### Proposal
Introduce an optional fallback resolver for unknown V2 catalog names.
One possible API shape could be:
```scala
trait CatalogResolver {
def resolveCatalog(name: String, conf: SQLConf): Option[CatalogPlugin]
}
```
Resolvers could be configured statically, for example:
```properties
spark.sql.catalog.fallbackResolvers=com.example.ExternalCatalogResolver
```
Then `CatalogManager.catalog(name)` could follow this order:
1. Return the cached catalog if it has already been loaded.
2. Try the existing `spark.sql.catalog.<name>` based loading path.
3. If the catalog is not configured, invoke fallback resolvers in configured
order.
4. If a resolver returns a `CatalogPlugin`, cache it in `CatalogManager`.
5. If no resolver resolves the catalog, throw the existing
`CatalogNotFoundException`.
The default behavior would remain unchanged when no fallback resolver is
configured.
### Example
With a fallback resolver configured, a user could query:
```sql
SELECT * FROM iceberg_prod.db.table
```
without predefining:
```properties
spark.sql.catalog.iceberg_prod=...
```
The resolver would receive `iceberg_prod`, query an external catalog
service, construct or load the proper `CatalogPlugin`, initialize it, and
return it to Spark.
### Compatibility
This should be fully backward compatible:
- No behavior changes unless fallback resolvers are explicitly configured.
- Existing `spark.sql.catalog.<name>` configuration keeps priority.
- Existing `CatalogNotFoundException` behavior remains when no resolver
matches.
- Resolved catalogs can follow the same caching and lifecycle behavior as
other V2 catalogs.
### Open Questions
- Should the resolver return an initialized `CatalogPlugin`, or return
catalog class/options and let Spark initialize it through the existing
`Catalogs.load` path?
- Should resolver configuration live in `SQLConf` or `SparkConf`?
- Should multiple resolvers be supported, with the first successful resolver
winning?
- Should Spark expose refresh/invalidation for dynamically resolved
catalogs, or should the initial proposal only cover discovery?
- Should this be limited to SQL analysis on the driver, or should there be
explicit constraints to avoid catalog resolution from executor-side paths?
### Summary
Spark already supports lazy instantiation of configured V2 catalogs. This
proposal is to add optional lazy discovery for unknown V2 catalog names, which
would help integrations with external catalog services avoid eagerly
registering every visible catalog at application startup.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
