cloud-fan commented on code in PR #56575:
URL: https://github.com/apache/spark/pull/56575#discussion_r3502621789
##########
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/util/V2ExpressionBuilder.scala:
##########
@@ -94,6 +94,8 @@ class V2ExpressionBuilder(e: Expression, isPredicate: Boolean
= false) extends L
private def generateExpression(
expr: Expression, isPredicate: Boolean = false): Option[V2Expression] =
expr match {
case literal: Literal => Some(translateLiteral(literal))
+ // DelegateExpression is a Spark-internal wrapper; push down its real
definition instead.
+ case d: DelegateExpression => generateExpression(d.definition, isPredicate)
Review Comment:
Fixed. `rebuildExpressionFromFilter` now consults the mapping with an exact
`translatedFilterToExpr.get(predicate)` lookup *before* structurally descending
into `V2And`/`V2Or`/`V2Not`.
For an ordinary compound predicate this lookup misses (only its leaves are
mapped at translation time), so it falls through to the structural cases
exactly as before -- behavior unchanged. But a `DelegateExpression` whose
`definition` is a compound (e.g. `And(a > 1, b < 2)`) is translated as a single
leaf, so the whole `V2And` maps back to the delegate; checking the map first
restores the original delegate directly instead of recursing into its
synthetic, unmapped children and throwing `Failed to rebuild Expression for
filter`. This is granularity-correct at every level of descent, so a delegate
nested inside an ordinary compound is covered too.
Test: `DataSourceV2StrategySuite."SPARK-57512: a compound-definition
DelegateExpression round-trips through filter rebuild"` translates a
compound-definition delegate to a structural `V2And` and asserts the rebuild
restores it via the exact map entry (the rejecting-source path).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
