voonhous commented on code in PR #20742:
URL: https://github.com/apache/hudi/pull/20742#discussion_r4623621004
##########
hudi-spark-datasource/hudi-spark-common/src/main/scala/org/apache/spark/sql/hudi/dataskipping/rules/ConstantOperandRule.scala:
##########
@@ -53,11 +57,11 @@ import scala.util.Try
object ConstantOperandRule extends RangeRule {
override def steps(term: Expression): Seq[Step] = term match {
- case DateAdd(start, _) => Seq(Step(term, start, Increasing, fileGuard =
None, partial = false))
- case DateSub(start, _) => Seq(Step(term, start, Increasing, fileGuard =
None, partial = false))
+ case DateAdd(start, days) if days.foldable => Seq(Step(term, start,
Increasing, fileGuard = None, partial = false))
Review Comment:
**minor:** Not introduced here, but `date_add`/`date_sub` with a constant
can still wrap: Spark's `nullSafeEval` is a bare int add/subtract in 3.5 and
4.x, even under ANSI, so `date_add(d, 2147483647) < DATE'2024-01-01'` over d in
[1969-12-31, 1970-01-02] prunes the matching file (reproduced locally in
`testNonOrderPreservingTransformationsSource`).
Contrived, edge case that is probably very rare, so not blocking. Could we
add a file guard like #20532's `addFits` to these two arms, or at least list
them in the overflow NOTE and track it under #20449?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]