voonhous commented on code in PR #20742:
URL: https://github.com/apache/hudi/pull/20742#discussion_r4623621015
##########
hudi-spark-datasource/hudi-spark-common/src/main/scala/org/apache/spark/sql/hudi/dataskipping/TermRange.scala:
##########
@@ -143,7 +143,7 @@ object TermRange {
/**
* Re-binds EVERY reference to the column, including ones in operands off
the rule chain (for ex,
- * the days operand of `date_add(d, dayofmonth(d))`), matching the pre-stage
translator. A [[Step]]
+ * the length operand of `left(s, length(s))`), matching the pre-stage
translator. A [[Step]]
Review Comment:
**minor:** After this PR, `PrefixRule`'s `Left(str, _)` is the only arm
still taking a column-dependent operand, and `left(s, length(s))` is the
identity, so it does not show the hazard; `left(s, length(s) - 1)` does ('ab'
maps to 'a', the larger 'b' to ''). It is dead post-optimizer today but goes
live once #19479 matches `Substring`. Not blocking. Could we add `if
len.foldable` to that arm here and use the reversing form as the example?
##########
hudi-spark-datasource/hudi-spark-common/src/main/scala/org/apache/spark/sql/hudi/dataskipping/rules/ConstantOperandRule.scala:
##########
@@ -53,11 +57,11 @@ import scala.util.Try
object ConstantOperandRule extends RangeRule {
override def steps(term: Expression): Seq[Step] = term match {
- case DateAdd(start, _) => Seq(Step(term, start, Increasing, fileGuard =
None, partial = false))
- case DateSub(start, _) => Seq(Step(term, start, Increasing, fileGuard =
None, partial = false))
+ case DateAdd(start, days) if days.foldable => Seq(Step(term, start,
Increasing, fileGuard = None, partial = false))
Review Comment:
**minor:** Not introduced here, but `date_add`/`date_sub` with a constant
can still wrap: Spark's `nullSafeEval` is a bare int add/subtract in 3.5 and
4.x, even under ANSI, so `date_add(d, 2147483647) < DATE'2024-01-01'` over d in
[1969-12-31, 1970-01-02] prunes the matching file (reproduced locally in
`testNonOrderPreservingTransformationsSource`). Contrived, so not blocking.
Could we add a file guard like #20532's `addFits` to these two arms, or at
least list them in the overflow NOTE and track it under #20449?
##########
hudi-spark-datasource/hudi-spark-common/src/main/scala/org/apache/spark/sql/hudi/dataskipping/rules/ConstantOperandRule.scala:
##########
@@ -101,18 +105,12 @@ object ConstantOperandRule extends RangeRule {
case Divide(dividend, divisor, _) if isPositiveNumericLiteral(divisor) =>
Seq(Step(term, dividend, Increasing, fileGuard = None, partial =
hasDecimalResult(term)))
- // NOTE: BitwiseOr is not monotonic (for ex, 1 < 2 but 1 | 2 = 3 > 2 | 2 =
2), #20449
- case BitwiseOr(left, right) =>
- Seq(
- Step(term, left, Increasing, fileGuard = None, partial = false),
- Step(term, right, Increasing, fileGuard = None, partial = false))
-
// NOTE: Spark only shifts int and bigint values, by an int amount
case ShiftLeft(value, amount) =>
IntegralLimits.of(term).filter(t => t.dataType == IntegerType ||
t.dataType == LongType).toSeq.flatMap { t =>
integralConstant(amount).map(n => guardedStep(term, value,
t.shiftLeftFits(n)))
}
- case ShiftRight(value, _) => Seq(Step(term, value, Increasing, fileGuard =
None, partial = false))
+ case ShiftRight(value, amount) if amount.foldable => Seq(Step(term, value,
Increasing, fileGuard = None, partial = false))
Review Comment:
**nit:** The golden inputs (`DataSkippingGoldenInputs.scala:80`) have
`ShiftLeft(a, a)` and `ShiftRight(Literal(8), a)` but no `ShiftRight(a, a)`, so
none of the 51 changed golden lines pins this new guard; only the matcher and
SQL tests do. Feel free to ignore, but could we add `ShiftRight(a, a)` to an
appended golden list so the golden file records it too?
##########
hudi-spark-datasource/hudi-spark/src/test/scala/org/apache/spark/sql/hudi/feature/TestDataSkippingQuery.scala:
##########
@@ -361,6 +361,57 @@ class TestDataSkippingQuery extends HoodieSparkSqlTestBase
{
}
}
+ test("Data skipping keeps files for terms that are not order-preserving in
the column") {
+ Seq(false, true).foreach { partitionStats =>
+ withTempDir { tmp =>
+ val tableName = generateTableName
+ withSQLConf(
+ "hoodie.metadata.enable" -> "true",
+ "hoodie.metadata.index.column.stats.enable" -> "true",
+ "hoodie.metadata.index.partition.stats.enable" ->
partitionStats.toString,
+ "hoodie.metadata.index.column.stats.column.list" -> "a,d",
+ "hoodie.enable.data.skipping" -> "false",
+ "hoodie.fileIndex.dataSkippingFailureMode" -> "strict"
+ ) {
+ spark.sql(
+ s"""
+ |create table $tableName (id int, a bigint, d date, ts long, dt
string) using hudi
+ | tblproperties (primaryKey = 'id', orderingFields = 'ts', type
= 'cow')
+ | partitioned by (dt)
+ | location '${tmp.getCanonicalPath}'
+ |""".stripMargin)
+ // One file per partition, so that each file's min and max come from
two rows
+ spark.sql(s"insert into $tableName values (1, 1, date'2024-01-31',
1, 'par1'), " +
+ "(2, 2, date'2024-02-01', 1, 'par1')")
+ spark.sql(s"insert into $tableName values (3, 64, date'2024-05-10',
1, 'par2'), " +
+ "(4, 65, date'2024-05-11', 1, 'par2')")
+ assert(spark.sql(s"select input_file_name() from
$tableName").collect().map(_.getString(0)).toSet.size == 2)
+ // Each term maps the min of its file above the max: 1 | 2 = 3 > 2 |
2 = 2, a bigint shifts by
+ // the low 6 bits of the amount (64 >> 64 = 64 > 65 >> 65 = 32), and
dayofmonth drops from 31 to
+ // 1 at the end of January
+ val queries = Seq(
+ "(a | 2) = 3" -> 1,
+ "(2 | a) = 3" -> 1,
+ "shiftright(a, a) = 64" -> 3,
+ "date_add(d, dayofmonth(d)) = date'2024-03-02'" -> 1,
+ "date_sub(d, -dayofmonth(d)) = date'2024-03-02'" -> 1,
+ "datediff(d, trunc(d, 'MM')) = 30" -> 1)
+ def checkQueries(): Unit = queries.foreach { case (filter, id) =>
+ withClue(s"$filter: ") {
+ checkAnswer(s"select id from $tableName where $filter")(Seq(id))
+ }
+ }
+ checkQueries()
+ withSQLConf("hoodie.enable.data.skipping" -> "true") {
+ spark.sql(s"alter table $tableName set tblproperties
('hoodie.enable.data.skipping' = 'true')")
Review Comment:
**nit:** This is the fifth copy of the skipping off/on toggle in this class
(also :190, :246, :305, :353). Feel free to ignore, but could we pull it into a
private `checkWithAndWithoutDataSkipping(tableName)(check)` helper and use it
here? Moving the older copies can wait.
##########
hudi-spark-datasource/hudi-spark/src/test/scala/org/apache/spark/sql/hudi/feature/TestDataSkippingQuery.scala:
##########
@@ -361,6 +361,57 @@ class TestDataSkippingQuery extends HoodieSparkSqlTestBase
{
}
}
+ test("Data skipping keeps files for terms that are not order-preserving in
the column") {
+ Seq(false, true).foreach { partitionStats =>
Review Comment:
**nit:** With partition stats on,
`PartitionStatsIndexSupport.filterNonSqlExpressions` drops the `date_add`,
`date_sub` and `shiftright(a, a)` filters before translation (it matches
`DateAdd`, `DateSub` and the `Cast` inside `shiftright`), so only the two `|`
filters and `datediff` reach partition stats here. Feel free to ignore, but
could a comment above this loop say which filters exercise that path?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]