szehon-ho opened a new pull request, #58752:
URL: https://github.com/apache/spark/pull/58752
### What changes were proposed in this pull request?
This PR computes runtime-filter eligibility once before dispatching to the
SupportsRuntimeV2Filtering or SupportsRuntimeCatalystFiltering path.
Both paths then use the same filtered collection. It also updates the method
documentation to describe the enforced behavior instead of requiring callers to
satisfy a precondition.
### Why are the changes needed?
Runtime filter eligibility is independent of the connector interface.
Applying the shared check before interface dispatch removes duplicated
screening and avoids relying on callers to exclude filters that Spark cannot
safely push.
### Does this PR introduce _any_ user-facing change?
No.
### How was this patch tested?
No new tests were added because existing tests cover non-deterministic
runtime filters for both connector paths.
The following focused tests passed:
SPARK_LOCAL_IP=127.0.0.1 build/sbt 'sql/testOnly
org.apache.spark.sql.connector.DataSourceV2SQLSuiteV2Filter -- -z "SPARK-58207:
non-deterministic scalar subquery filters are not pushed into runtimeFilters"'
SPARK_LOCAL_IP=127.0.0.1 build/sbt 'sql/testOnly
org.apache.spark.sql.connector.DataSourceV2CatalystRuntimeFilterSuite -- -z
"non-deterministic predicate on fully pushed attributes"'
Also ran dev/lint-scala.
### Was this patch authored or co-authored using generative AI tooling?
Generated-by: OpenAI Codex (GPT-5)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
