yadavay-amzn opened a new pull request, #57468: URL: https://github.com/apache/spark/pull/57468
### What changes were proposed in this pull request? Follow-up to #56627 ([SPARK-57518](https://issues.apache.org/jira/browse/SPARK-57518)), addressing review feedback. `SparkGetSchemasOperation` special-cased the session catalog: when the current catalog was `spark_catalog`, it listed databases via the V1 `SessionCatalog.listDatabases`, and only used the DSv2 `SupportsNamespaces.listNamespaces()` path for other catalogs. That assumed a `spark_catalog` override (e.g. a custom `CatalogExtension`) delegates its namespace listing to the built-in session catalog, which is not guaranteed. This change lists schemas uniformly through the current catalog's `SupportsNamespaces.listNamespaces()` for all catalogs, and drops the `spark_catalog` special-case. The built-in `V2SessionCatalog` also implements `SupportsNamespaces`, so the default session catalog keeps working, and an overridden `spark_catalog` now correctly reports its own namespaces. The `global_temp` database is retained as a special case for the session catalog only, since it is a Spark pseudo-namespace rather than a real catalog namespace. ### Why are the changes needed? The previous approach could report the wrong schemas for a `spark_catalog` override that does not delegate namespace listing to the built-in session catalog. ### Does this PR introduce _any_ user-facing change? Behavior change on the DSv2 metadata path only: schema-name matching now uses JDBC pattern semantics (`%` and `_`), consistent with `DatabaseMetaData.getSchemas`. Hive-style glob (`*`) is not a JDBC wildcard and is no longer matched on this path. ### Performance consideration `SupportsNamespaces.listNamespaces()` has no schema-pattern argument (unlike V1 `listDatabases(pattern)`), so the schema pattern is applied client-side. For a catalog with many namespaces this lists all top-level namespaces and then filters, rather than pushing the pattern down. The current DSv2 API does not support pushing the pattern down; if this becomes a concern, a DSv2 API enhancement to accept a pattern would be a separate improvement. Raising it here for discussion. ### How was this patch tested? `SparkMetadataOperationSuite`: verifies `spark_catalog` lists its namespaces via `SupportsNamespaces` and that `global_temp` still appears. Existing getSchemas coverage passes. `SparkGetTablesOperation`/`SparkGetColumnsOperation` do not have this special-case and are unchanged. ### Was this patch authored or co-authored using generative AI tooling? Authored with assistance by Claude Opus 4.8. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
