szehon-ho opened a new pull request, #58370: URL: https://github.com/apache/spark/pull/58370
### What changes were proposed in this pull request? This PR restores support for nested `NamedReference`s returned by runtime-filtering scans. It removes the top-level-only validation added in #58296 from the public `SupportsRuntimeFiltering` and `SupportsRuntimeV2Filtering` interfaces and from the internal `SupportsRuntimeCatalystFiltering` interface. The test scans now preserve nested partition source paths, and end-to-end dynamic partition pruning tests cover both `SupportsRuntimeV2Filtering` and `SupportsRuntimeCatalystFiltering`. The tests verify that a scan advertising `derives.toStr` receives a predicate containing that nested access and prunes the expected partition. ### Why are the changes needed? The validation in #58296 was prompted by this review comment: > `filterAttributes()` requires top-level read-schema attributes. Please make > `runtimeFilterAttrs` reject nested references even when the parent is a struct and change this > assertion to expect that error; accepting `s.tz` here widens it to `s` and makes filters over > every field eligible. https://github.com/apache/spark/pull/58296#discussion_r3859456719 The concern is valid: Spark currently stores the resolved references in an `AttributeSet`, so a nested reference such as `derives.toStr` contributes its root attribute, `derives`, to the eligibility set. That representation cannot distinguish `derives.toStr` from a sibling field. However, rejecting nested references is a breaking change for existing data sources. Iceberg's `SparkBatchQueryScan.filterAttributes()` collects the source IDs of partition fields and converts each source column's full path to a `NamedReference`: https://github.com/apache/iceberg/blob/ef8a69dc3dab5858d612770062bbf5a6043170f0/spark/v4.0/spark/src/main/java/org/apache/iceberg/spark/source/SparkBatchQueryScan.java#L105-L123 For a partition transform such as `truncate(2, derives.toStr)`, Iceberg therefore advertises `derives.toStr`. Before #58296, Spark resolved that nested reference and used its root attribute for DPP eligibility, but the runtime predicate delivered to Iceberg retained the nested access. Iceberg then projected the predicate through every partition spec and evaluated the projected expression to prune scan tasks: https://github.com/apache/iceberg/blob/ef8a69dc3dab5858d612770062bbf5a6043170f0/spark/v4.0/spark/src/main/java/org/apache/iceberg/spark/source/SparkBatchQueryScan.java#L127-L146 This flow worked end to end. The new validation instead rejected Iceberg's declaration while Spark was building the scan relation. This PR restores compatibility. A follow-up should make runtime-filter eligibility path-aware so Spark can distinguish `derives.toStr` from sibling fields without rejecting nested declarations. ### Does this PR introduce _any_ user-facing change? Yes. Data sources may again advertise nested runtime-filter attributes. This restores the behavior before #58296: connectors such as Iceberg no longer fail during planning for these scans and can receive the nested runtime predicate for partition pruning. ### How was this patch tested? Added end-to-end DPP regression coverage for both the public V2 filtering interface and the internal Catalyst filtering interface: ``` MAVEN_MIRROR_URL=https://maven-proxy.cloud.databricks.com \ build/sbt -batch -java-home /usr/lib/jvm/java-17-openjdk-amd64 \ 'sql/testOnly org.apache.spark.sql.connector.DataSourceV2SQLSuiteV2Filter -- -z "nested partition source column receives a DPP runtime filter"' \ 'sql/testOnly org.apache.spark.sql.connector.DataSourceV2CatalystRuntimeFilterSuite -- -z "DPP filter on a nested partition source"' ``` Both tests passed. ### Was this patch authored or co-authored using generative AI tooling? Generated-by: Codex with GPT-5 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
