szehon-ho opened a new pull request, #58412: URL: https://github.com/apache/spark/pull/58412
### What changes were proposed in this pull request? This is a targeted backport of #58370 to `branch-4.3`. It: - allows nested references from ordinary `filterAttributes()` and preserves nested access in the pushed filter; - keeps `fullyPushedFilterAttributes()` limited to top-level attributes, where Spark can safely remove post-scan evaluation; - replaces the prior internal failure with the structured `DATA_SOURCE_INVALID_RUNTIME_FILTER_ATTRIBUTE` error condition; - fixes nested runtime group filtering for row-level operations; and - adds coverage for Catalyst, V1, V2, DPP, dotted-name collisions, invalid attributes, and group- and delta-based row-level operations. This is a net backport. The 4.4 commit was applied onto code that already contained the intermediate #59012 fixture refactoring. This PR adapts the final behavior to the 4.3 codebase without pulling in those unrelated refactors. ### Why are the changes needed? `branch-4.3` already contains the unreleased `SupportsRuntimeCatalystFiltering` API introduced by `87064f614d8`, but it does not contain the final correction from #58370. Without this backport, connectors cannot report nested runtime-filter attributes correctly, and nested row-level runtime filtering can resolve the wrong expression shape. Because Spark 4.3 is unreleased, its API contract should match the corrected behavior before release. Method signatures and binary compatibility are unchanged. Spark continues to track eligibility by root attribute. Making that tracking fully path-aware is separate follow-up work in [SPARK-59095](https://issues.apache.org/jira/browse/SPARK-59095). ### Does this PR introduce _any_ user-facing change? Yes. Runtime-filtering scans may report nested references from ordinary `filterAttributes()`. Nested references from `fullyPushedFilterAttributes()` remain unsupported and now produce a structured analysis error. For `SupportsRuntimeCatalystFiltering`, this corrects behavior on the unreleased Spark 4.3 branch. There is no API signature change. ### How was this patch tested? - `SPARK_GENERATE_GOLDEN_FILES=1 build/sbt 'core/testOnly *SparkThrowableSuite -- -t "Error conditions are correctly formatted"'` - `build/sbt 'core/testOnly org.apache.spark.SparkThrowableSuite'` - `build/sbt 'sql/testOnly org.apache.spark.sql.connector.DataSourceV2CatalystRuntimeFilterSuite'` - Focused nested DPP tests in `DataSourceV2SQLSuiteV1Filter` and `DataSourceV2SQLSuiteV2Filter` - `GroupBasedRowLevelOperationCatalystRuntimeFilterSuite` - `DeltaBasedRowLevelOperationCatalystRuntimeFilterSuite` - `catalyst/scalastyle` - `sql/scalastyle` - `catalyst/checkstyle` ### Was this patch authored or co-authored using generative AI tooling? Generated-by: Codex with GPT-5 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
