nsivabalan commented on code in PR #20457:
URL: https://github.com/apache/hudi/pull/20457#discussion_r4460923055
##########
hudi-spark-datasource/hudi-spark-common/src/main/scala/org/apache/spark/sql/BaseHoodieCatalystExpressionUtils.scala:
##########
@@ -81,11 +81,13 @@ abstract class BaseHoodieCatalystExpressionUtils extends
HoodieCatalystExpressio
// Math Expressions
// Binary
- case Add(OrderPreservingTransformation(attrRef), _, _) => Some(attrRef)
- case Add(_, OrderPreservingTransformation(attrRef), _) => Some(attrRef)
- case Multiply(OrderPreservingTransformation(attrRef), _, _) =>
Some(attrRef)
- case Multiply(_, OrderPreservingTransformation(attrRef), _) =>
Some(attrRef)
- case Divide(OrderPreservingTransformation(attrRef), _, _) =>
Some(attrRef)
+ // NOTE: The other operand must not reference the column, or it could
reverse the
+ // ordering (for ex, A + A * -2 = -A)
+ case Add(OrderPreservingTransformation(attrRef), other, _) if
other.references.isEmpty => Some(attrRef)
Review Comment:
`other.references.isEmpty` still admits a non-deterministic operand: `A +
rand() < 5` has no references, but data skipping re-evaluates `rand()` against
the min/max bounds, so it can prune a file with matching rows. `other.foldable`
would require an actual constant, which is what the NOTE describes.
Non-blocking.
##########
hudi-spark-datasource/hudi-spark-common/src/main/scala/org/apache/spark/sql/BaseHoodieCatalystExpressionUtils.scala:
##########
@@ -81,11 +81,13 @@ abstract class BaseHoodieCatalystExpressionUtils extends
HoodieCatalystExpressio
// Math Expressions
// Binary
- case Add(OrderPreservingTransformation(attrRef), _, _) => Some(attrRef)
- case Add(_, OrderPreservingTransformation(attrRef), _) => Some(attrRef)
- case Multiply(OrderPreservingTransformation(attrRef), _, _) =>
Some(attrRef)
- case Multiply(_, OrderPreservingTransformation(attrRef), _) =>
Some(attrRef)
- case Divide(OrderPreservingTransformation(attrRef), _, _) =>
Some(attrRef)
+ // NOTE: The other operand must not reference the column, or it could
reverse the
+ // ordering (for ex, A + A * -2 = -A)
+ case Add(OrderPreservingTransformation(attrRef), other, _) if
other.references.isEmpty => Some(attrRef)
+ case Add(other, OrderPreservingTransformation(attrRef), _) if
other.references.isEmpty => Some(attrRef)
+ case Multiply(OrderPreservingTransformation(attrRef), factor, _) if
isPositiveNumericLiteral(factor) => Some(attrRef)
+ case Multiply(factor, OrderPreservingTransformation(attrRef), _) if
isPositiveNumericLiteral(factor) => Some(attrRef)
+ case Divide(OrderPreservingTransformation(attrRef), divisor, _) if
isPositiveNumericLiteral(divisor) => Some(attrRef)
case BitwiseOr(OrderPreservingTransformation(attrRef), _) =>
Some(attrRef)
Review Comment:
Pre-existing, not introduced here: `BitwiseOr` does not preserve ordering
either, e.g. `3 | 4 = 7` but `4 | 4 = 4`, so `A | 4 = 4` can prune a file with
min 3 / max 4 although `A = 4` matches. #20460 only covers overflow in `Add`,
`Multiply` and `ShiftLeft`. Could these two arms be dropped here, or tracked in
#20460 / #20449?
##########
hudi-spark-datasource/hudi-spark-common/src/main/scala/org/apache/spark/sql/BaseHoodieCatalystExpressionUtils.scala:
##########
@@ -111,5 +113,21 @@ abstract class BaseHoodieCatalystExpressionUtils extends
HoodieCatalystExpressio
}
}
}
+
+ // Negative factors reverse the ordering and zero collapses it.
Non-literal factors are not
+ // matched since the optimizer folds constant factors to literals before
data skipping runs
+ private def isPositiveNumericLiteral(expr: Expression): Boolean = expr
match {
+ case Literal(value, _) => value match {
+ case b: Byte => b > 0
+ case s: Short => s > 0
+ case i: Int => i > 0
+ case l: Long => l > 0
+ case f: Float => f > 0
+ case d: Double => d > 0
Review Comment:
A positive infinity literal passes `d > 0`. For a file with `A` in `[0, 1]`,
`0 * Infinity` is `NaN`, which Spark sorts above every number, so the bound
from the min ends up above the bound from the max and the ordering breaks.
Could this also require the factor to be finite (same for `Float`)? Edge case,
non-blocking.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
