yyanyy opened a new pull request, #58462: URL: https://github.com/apache/spark/pull/58462
### What changes were proposed in this pull request? This is tracked by [SPARK-59163](https://issues.apache.org/jira/browse/SPARK-59163). It follows #57585 and completes the table-state option reuse requested in [this review comment](https://github.com/apache/spark/pull/57585#discussion_r3819899362). The earlier PR introduced the projection and has already landed; this PR is its subsequent internal cleanup. `RelationResolution` already computed the projected table-state options for `TableCacheKey`, but table loading projected the full option map again. The projected map passed to `CatalogV2Util.lookupCachedRelation` was also projected a second time inside that method. This PR: - as a minor refactoring, adds consistently named `*WithStateOptions` entry points and makes the existing full-option helpers project once before delegating to them; - reuses `TableCacheKey.stateOptions` for shared relation cache lookup in both persistent relation and `V2TableReference` resolution, without repeating the projection for table loading; - reuses the single projection in `V2TableRefreshUtil` for refresh deduplication, shared cache lookup, and catalog loading. ### Why are the changes needed? The previous flow repeatedly normalized the catalog's state-option key set and materialized a full projection during an uncached relation resolution or execution refresh. The new flow computes that projection once and reuses the same map for Spark's table pin, catalog load, and shared cache lookup. ### Does this PR introduce _any_ user-facing change? No. ### How was this patch tested? The existing `CatalogV2UtilSuite` and `DataSourceV2OptionSuite` cover table-state option filtering, persistent relation resolution, `V2TableReference` resolution, shared relation cache matching, and execution refresh. No call-count assertions were added for this implementation-only cleanup. ```bash XDG_RUNTIME_DIR=/tmp/yan-sbt-runtime \ JAVA_HOME=/usr/lib/jvm/java-17-openjdk-amd64 \ DEFAULT_ARTIFACT_REPOSITORY=https://maven-proxy.cloud.databricks.com \ MAVEN_MIRROR_URL=https://maven-proxy.cloud.databricks.com \ build/sbt \ 'catalyst/testOnly org.apache.spark.sql.connector.catalog.CatalogV2UtilSuite' \ 'sql/testOnly org.apache.spark.sql.connector.DataSourceV2OptionSuite' ``` `CatalogV2UtilSuite` ran 19/19 tests and `DataSourceV2OptionSuite` ran 56/56 tests, with no failures, cancellations, ignores, or pending tests. No test-case filters were used within the suites; the run used SBT's incremental compilation cache. `DataSourceV2OptionSuite` emitted its existing non-failing possible thread-leak warning. ```bash XDG_RUNTIME_DIR=/tmp/yan-sbt-runtime \ JAVA_HOME=/usr/lib/jvm/java-17-openjdk-amd64 \ DEFAULT_ARTIFACT_REPOSITORY=https://maven-proxy.cloud.databricks.com \ MAVEN_MIRROR_URL=https://maven-proxy.cloud.databricks.com \ build/sbt \ 'catalyst / Compile / scalastyle' \ 'catalyst / Test / scalastyle' \ 'sql / Compile / scalastyle' \ 'sql / Test / scalastyle' \ 'catalyst / Compile / checkstyle' ``` The four scalastyle tasks processed 718 Catalyst main files, 440 Catalyst test files, 826 SQL core main files, and 1,106 SQL core test files with no errors or warnings. Catalyst main checkstyle also completed with no issues. ### Was this patch authored or co-authored using generative AI tooling? Generated-by: OpenAI Codex -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
