nsivabalan commented on issue #20296: URL: https://github.com/apache/hudi/issues/20296#issuecomment-6381016266
Thanks @voonhous, agreed this is one class of bug, and the oracle-based property test is the right fix. Please go ahead and pick it up. We reproduced your list on a real table (Spark 3.5.3, 1.3.0-SNAPSHOT, single-file COW table, data skipping on vs. off, ANSI off). All of them return wrong results: multiply/divide by a negative, `BitwiseOr`, `Add`/`ShiftLeft` overflow, `date_format` with a non-sortable format, and `to_unix_timestamp(str, fmt)`. A few more gaps the test should cover: 1. **`LIKE 'abc%'` on a plain column**, with no transform. The prefix match is rewritten as `min <= 'abc' <= max`, so a file with min=`abc1`, max=`abc9` is pruned (2 rows → 0). Filed separately as #20430, since it affects every prefix query. 2. **Transforms that return NULL for real values.** `log10`/`ln` on a negative min, `to_date` on a string that can't be parsed, and decimal-narrowing casts all produce a NULL bound, and the file is dropped. 3. **DST.** `date_format(ts, 'yyyy-MM-dd HH:mm')` (a big-endian format), `cast(ts as string)` and `from_utc_timestamp` all break across a fall-back transition (`America/New_York`, 2026-11-01). Please add session timezone as a test dimension alongside ANSI mode. 4. **Casts.** `isCastPreservingOrdering` only excludes string ↔ numeric. `cast(bigint as int)` wraps around, and `cast(str as date)` / `to_date(str)` on unpadded strings like `2024-2-1` reorders values. | Predicate | rows (skipping off → on) | |---|---| | `s like 'abc%'` | 2 → 0 | | `i * -1 = -2` / `i / -1 = -2` | 1 → 0 | | `(b \| 2) = 3` | 1 → 0 | | `a + 1 < 0` / `shiftleft(a, 1) < 0` (overflow) | 1 → 0 | | `log10(n) < 0.5` / `ln(n) < 1.0` (min = -1) | 1 → 0 | | `date_format(dt, 'dd-MM-yyyy') = '15-03-2026'` | 1 → 0 | | `date_format(ts, 'yyyy-MM-dd HH:mm') > '2026-11-01 01:45'` (NY) | 1 → 0 | | `cast(ts as string) > '2026-11-01 01:45'` (NY) | 1 → 0 | | `from_utc_timestamp(ts, 'America/New_York') > ...` | 1 → 0 | | `to_date(sd) = date'2024-02-01'` / `cast(sd as date) = ...` | 1 → 0 | | `to_unix_timestamp(sd, 'yyyy-M-d') = ...` | 1 → 0 | | `to_date(sd2) >= date'2024-01-01'` (max string can't be parsed) | 1 → 0 | | `cast(l as int) < 0` | 1 → 0 | | `cast(dec as decimal(3,1)) >= 1.0` | 1 → 0 | One correction to the overall picture: `AllowedTransformationExpression` requires exactly one column reference, so `col + otherCol` is not matched. Also, `to_timestamp(str, fmt)` never reached the matcher in our run (Spark replaces it before Hudi sees it), so the `ParseToTimestamp` branch may be dead code on 3.5. For the harness, it would be good to go through `getCandidateFiles` rather than only `translateIntoColumnStatsIndexFilterExpr`, so the missing-stats handling from #20328 is covered too. The partition-stats path should be included as well. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
