yadavay-amzn commented on code in PR #56244:
URL: https://github.com/apache/spark/pull/56244#discussion_r3503431104
##########
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/optimizer/Optimizer.scala:
##########
@@ -141,6 +141,7 @@ abstract class Optimizer(catalogManager: CatalogManager)
BooleanSimplification,
SimplifyConditionals,
PushFoldableIntoBranches,
+ DecomposeStructComparison,
Review Comment:
@cloud-fan thanks, agreed the placement was the core issue. Reworked to do
the decomposition at the pushdown/translation layer instead of as a logical
rule:
- Removed the DecomposeStructComparison logical rule entirely, along with
its exact-NULL-equivalence machinery (IsNotNull guard /
filterEquivalentToConjunction / decomposeEqualTo / decomposeEqualNullSafe).
- Added DataSourceStrategy.expandStructPredicatesForPushdown, wired into
FileSourceStrategy (V1) and PushDownUtils.pushFilters (V2). It adds field-level
equality predicates as pushdown candidates; the original struct predicate is
left untouched and retained post-scan, so the pushed field predicates only need
to be a sound over-approximation, not NULL-exact. This also fixes the
capability regression you raised: the whole-struct predicate is still present
for a source that can push it natively.
- Soundness: for a struct literal with a NULL field we do not push s.fi =
null (that would drop valid rows); only non-null fields are pushed, still a
sound superset. Both = and <=> are handled.
- Renamed the confs to spark.sql.sources.structPredicateDecompose.{enabled
(default true), maxFields (default 100)} to reflect the sources/pushdown
placement.
- New StructPredicatePushdownSuite asserts the field predicates reach
PushedFilters, the original filter is retained post-scan, null-valued literal
fields are not pushed, maxFields is bounded, conf on/off, and rule-on ==
rule-off parity across whole-null / all-null-fields / non-null rows.
Does this placement look right to you? Happy to adjust (e.g. default OFF, or
scoping by source capability) if you would prefer.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
