szehon-ho opened a new pull request, #58503: URL: https://github.com/apache/spark/pull/58503
### What changes were proposed in this pull request? This follow-up addresses the upstream review findings from #58412 before its Spark 4.3 backport proceeds. It: - restricts the in-memory V1, V2, Catalyst, and row-level runtime-filtering fixtures to identity partition transforms while preserving the real partition-key ordinal; - requires every `fullyPushedFilterAttributes()` entry to be an exact entry in `filterAttributes()`; - reports invalid runtime-filter declarations consistently from DPP, row-level group filtering, and iterative pushdown paths; - clarifies that Spark currently tracks runtime-filter eligibility by root attribute; and - adds regressions for transformed partition sources and invalid fully-pushed declarations. ### Why are the changes needed? The test fixtures could advertise a transformed source column and then compare its source value directly with a transformed partition key. That could incorrectly remove a matching partition before the residual predicate was evaluated. Also, a scan could declare a top-level struct as fully pushed while only declaring one nested field as filterable. Because Spark tracks eligibility by root attribute, this could remove a required post-scan predicate. Invalid connector declarations could additionally surface different errors depending on the planning path. ### Does this PR introduce _any_ user-facing change? Yes, on the unreleased `master` branch. Invalid Catalyst runtime-filter scan declarations are now rejected consistently during planning. Correct connector declarations and released Spark behavior are unchanged. ### How was this patch tested? Tests were added for Catalyst runtime filtering, V1/V2 DPP, and both group-based and delta-based row-level operations. - `build/sbt catalyst/Test/compile sql/Test/compile` - `SPARK_GENERATE_GOLDEN_FILES=1 build/sbt 'core/testOnly *SparkThrowableSuite -- -t "Error conditions are correctly formatted"'` - `SPARK_LOCAL_IP=localhost build/sbt 'sql/testOnly *DataSourceV2CatalystRuntimeFilterSuite'` - Focused transformed-partition DPP tests in `DataSourceV2SQLSuiteV1Filter` and `DataSourceV2SQLSuiteV2Filter` - Focused non-identity partition transform tests in `GroupBasedRowLevelOperationCatalystRuntimeFilterSuite` and `DeltaBasedRowLevelOperationCatalystRuntimeFilterSuite` - `build/sbt catalyst/scalastyle sql/scalastyle catalyst/checkstyle` - `git diff --check` ### Was this patch authored or co-authored using generative AI tooling? Generated-by: Codex with GPT-5 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
