Szehon Ho created SPARK-59095:
---------------------------------
Summary: Make runtime filter eligibility path-aware
Key: SPARK-59095
URL: https://issues.apache.org/jira/browse/SPARK-59095
Project: Spark
Issue Type: Improvement
Components: SQL
Affects Versions: 5.0.0
Reporter: Szehon Ho
Spark currently tracks runtime-filter eligibility in an AttributeSet, which
reduces nested references to their root attributes. This is safe for ordinary
filterAttributes() because Spark retains post-scan evaluation, but it is not
safe for fullyPushedFilterAttributes(), where Spark removes the post-scan
predicate. As a result, connectors cannot soundly declare one fully pushed
nested path: declaring the root struct would also make predicates on sibling
fields appear fully pushed. Make eligibility path-aware so Spark can
distinguish nested paths and safely support fully pushed nested runtime-filter
attributes.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
