sunchao commented on code in PR #58370:
URL: https://github.com/apache/spark/pull/58370#discussion_r3882905270
##########
sql/catalyst/src/main/scala/org/apache/spark/sql/execution/datasources/v2/DataSourceV2Relation.scala:
##########
@@ -223,21 +223,11 @@ case class DataSourceV2ScanRelation(
case s: SupportsRuntimeCatalystFiltering =>
s.fullyPushedFilterAttributes()
case _ => Array.empty[NamedReference]
}
- resolveTopLevelFilterAttrs(filterAttrs)
+ resolveFilterAttrs(filterAttrs)
Review Comment:
[P1] Preserve nested paths for fully pushed filters
This also permits nested references from fullyPushedFilterAttributes(), but
AttributeSet keeps only their root AttributeReference. Declaring only s.part
therefore makes the scalar-subquery predicate on s.other in `SELECT * FROM fact
WHERE s.other = (SELECT max(v) FROM dim)` pass the fully-pushed subset check in
DataSourceV2Strategy. Spark removes that equality from postScanFilters even
though a scan partitioned only by s.part cannot evaluate the sibling field from
its partition key.
I reproduced this with the existing in-memory Catalyst fixture, PARTITIONED
BY (s.part), and TBLPROPERTIES('fully-pushed-filter-attributes'='s.part').
Given (id, s.part, s.other) values (1, 1, 10), (2, 1, 20), (3, 2, 30), and a
dimension value of 10, the query should return only id 1. The PR CI build
returns ids 1, 2, and 3 with AQE both on and off. Only the IS NOT NULL residual
remains; the equality is no longer enforced. The matching-field and
no-fully-pushed controls return the correct rows.
Please retain the nested-reference restriction for fully pushed declarations
until eligibility preserves complete field paths, and cover sibling-field
predicates before allowing Spark to remove their residual evaluation.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
