voonhous opened a new issue, #20568:
URL: https://github.com/apache/hudi/issues/20568
### Describe the problem
Data skipping on an `IN` list of more than 10 values over a DATE or
TIMESTAMP column fails to compile against the stats table. Spark's `OptimizeIn`
turns such a list into an `InSet`
(`spark.sql.optimizer.inSetConversionThreshold`), whose values are Catalyst
internal ones. `Verdict` makes literals from them with `Literal(value)`
(`Verdict.scala:43`, `:74`), so a DATE value becomes an INT and a TIMESTAMP
value a BIGINT, which do not compare with the DATE / TIMESTAMP stats columns:
```
AnalysisException: [DATATYPE_MISMATCH.BINARY_OP_DIFF_TYPES] Cannot resolve
"(d_minValue <= 19611)" due to data type mismatch: the left and right operands
of the binary operator have incompatible types ("DATE" and "INT").
```
With `hoodie.fileIndex.dataSkippingFailureMode=strict` the query fails. With
the default `fallback`, it logs the error and loses all pruning.
`golden-index-filters.txt` pins the output: `d INSET (0:date)` compiles to
`((d_minValue <= 0:int) AND (d_maxValue >= 0:int))`.
### Repro
```sql
SET hoodie.metadata.index.column.stats.enable=true;
CREATE TABLE t (ts STRING, d DATE, inst TIMESTAMP, uuid STRING, rider
STRING, fare DOUBLE, city STRING) USING HUDI
PARTITIONED BY (city)
TBLPROPERTIES (primaryKey = 'uuid', preCombineField = 'ts');
INSERT INTO t VALUES
('2023-09-19 23:40:49',DATE'2023-09-19',TIMESTAMP'2023-09-19
23:40:49','u1','rider-A',19.10,'san_francisco'),
('2023-09-21 16:34:26',DATE'2023-09-21',TIMESTAMP'2023-09-21
16:34:26','u2','rider-E',93.50,'san_francisco'),
('2023-09-23 09:42:17',DATE'2023-09-23',TIMESTAMP'2023-09-23
09:42:17','u3','rider-F',34.15,'sao_paulo'),
('2023-09-19 02:38:07',DATE'2023-09-19',TIMESTAMP'2023-09-19
02:38:07','u4','rider-I',41.06,'chennai');
SELECT rider FROM t WHERE d IN (DATE'2023-09-11', DATE'2023-09-12',
DATE'2023-09-13', DATE'2023-09-14', DATE'2023-09-15', DATE'2023-09-16',
DATE'2023-09-17', DATE'2023-09-18', DATE'2023-09-19', DATE'2023-09-20',
DATE'2023-09-21');
```
Master `7888bf909694`, Spark 3.5, `hoodie.enable.data.skipping=true`:
| filter | `fallback` | `strict` |
| --- | --- | --- |
| `d IN` 11 dates | correct rows, 3 of 3 files scanned | `AnalysisException`
|
| `inst IN` 11 timestamps | correct rows, 3 of 3 files scanned |
`AnalysisException` |
| `d IN` 2 dates (control, stays `In`) | 1 file | 1 file |
### Suggested fix
Build the literals with the term's type, for ex `Literal.create(value,
term.dataType)`, keeping the UTF8String handling.
Related: #20540 keeps `InSet` out of the expression index because of this.
#20537 removes that gate, so it needs this fix first.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]