szehon-ho commented on code in PR #58948:
URL: https://github.com/apache/spark/pull/58948#discussion_r4079051733
##########
sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/v2/PushDownUtils.scala:
##########
@@ -105,14 +116,22 @@ object PushDownUtils extends Logging {
val translatedFilters = mutable.ArrayBuffer.empty[Predicate]
val untranslatableExprs = mutable.ArrayBuffer.empty[Expression]
+ def translateFilter(expression: Expression): Option[Predicate] = {
+ DataSourceV2Strategy.translateFilterV2WithMapping(
+ expression, Some(translatedFilterToExpr))
+ }
+
for (filterExpr <- deterministicFilters) {
- val translated =
- DataSourceV2Strategy.translateFilterV2WithMapping(
- filterExpr, Some(translatedFilterToExpr))
- if (translated.isEmpty) {
- untranslatableExprs += filterExpr
- } else {
- translatedFilters += translated.get
+ translateFilter(filterExpr) match {
+ case Some(filter) =>
+ translatedFilters += filter
+ case None =>
+ untranslatableExprs += filterExpr
+ extractPushablePredicate(
+ filterExpr,
+ DataSourceV2Strategy.translateFilterV2)
+ .flatMap(translateFilter)
+ .foreach(translatedFilters += _)
Review Comment:
Could we avoid retaining the synthesized predicate as an additional
post-scan filter when the original expression is already retained? This applies
to both filter APIs.
For example, let `P = id = 1 OR (id = 2 AND U)` and let the extracted
predicate be `Q = id = 1 OR id = 2`, where `U` cannot be translated. If the
source rejects `Q` and returns it for reevaluation, residual reconstruction
produces `Q AND P`. Since `P` already implies `Q`, that extra condition is
redundant.
This can affect statistics: `FilterEstimation` multiplies the selectivities
of the two sides of `AND`, while estimating `OR` does not update column
statistics. With 10 non-null distinct ID values, no histogram, and `U`
unsupported by the estimator, the estimated selectivity is 19% for `P` but
3.61% for `Q AND P`. Cost-based join reordering runs after scan pushdown and
before the later boolean cleanup, so the redundant filter can affect join
planning.
Could we track the predicates synthesized by partial extraction and omit
their returned residuals when the original predicate is still enforced? A
regression test with a source that rejects `Q` and reports row/column
statistics would cover the residual shape and cardinality estimate. This
concern is based on code inspection; I have not run a regression test.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
