voonhous opened a new issue, #20567:
URL: https://github.com/apache/hudi/issues/20567
### Describe the problem
A bloom filter expression index over a DATE result drops files holding
matching rows, so the query returns wrong results. The reader and the writer
build different keys for the same value:
- Writer: `row.getAs(columnToIndex).toString()`
(`SparkMetadataWriterUtils.java:243`), the external value, for ex `2023-09-22`.
- Reader: `lit.value.toString` (`RecordLevelIndexSupport.scala:183`), the
Catalyst internal value, for ex `19622` (epoch days).
The probe never matches, so every file is pruned. String results are not
affected. A TIMESTAMP result goes through the same two lines (epoch micros
against `java.sql.Timestamp.toString`), but only DATE was run.
### Repro
```sql
SET
hoodie.write.lock.provider=org.apache.hudi.client.transaction.lock.InProcessLockProvider;
CREATE TABLE t (ts STRING, d DATE, uuid STRING, rider STRING, fare DOUBLE,
city STRING) USING HUDI
PARTITIONED BY (city)
TBLPROPERTIES (primaryKey = 'uuid', preCombineField = 'ts');
INSERT INTO t VALUES
('2023-09-19
23:40:49',DATE'2023-09-19','u1','rider-A',19.10,'san_francisco'),
('2023-09-21
16:34:26',DATE'2023-09-21','u2','rider-E',93.50,'san_francisco'),
('2023-09-23 09:42:17',DATE'2023-09-23','u3','rider-F',34.15,'sao_paulo'),
('2023-09-19 02:38:07',DATE'2023-09-19','u4','rider-I',41.06,'chennai');
CREATE INDEX d_next ON t USING bloom_filters(d) options(expr='date_add',
days='1');
```
Master `7888bf909694`, Spark 3.5:
| filter | data skipping off | data skipping on |
| --- | --- | --- |
| `date_add(d, 1) = DATE'2023-09-22'` | `rider-E` | no rows, 0 files |
| `date_add(d, 1) IN (DATE'2023-09-22', DATE'2023-09-24')` | `rider-E`,
`rider-F` | no rows, 0 files |
### Suggested fix
Build the probe key from the literal's external value, so that it has the
writer's string form. Until then, a bloom filter expression index could skip
terms that are not string or integral.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]