voonhous opened a new issue, #20567:
URL: https://github.com/apache/hudi/issues/20567

   ### Describe the problem
   
   A bloom filter expression index over a DATE result drops files holding 
matching rows, so the query returns wrong results. The reader and the writer 
build different keys for the same value:
   
   - Writer: `row.getAs(columnToIndex).toString()` 
(`SparkMetadataWriterUtils.java:243`), the external value, for ex `2023-09-22`.
   - Reader: `lit.value.toString` (`RecordLevelIndexSupport.scala:183`), the 
Catalyst internal value, for ex `19622` (epoch days).
   
   The probe never matches, so every file is pruned. String results are not 
affected. A TIMESTAMP result goes through the same two lines (epoch micros 
against `java.sql.Timestamp.toString`), but only DATE was run.
   
   ### Repro
   
   ```sql
   SET 
hoodie.write.lock.provider=org.apache.hudi.client.transaction.lock.InProcessLockProvider;
   
   CREATE TABLE t (ts STRING, d DATE, uuid STRING, rider STRING, fare DOUBLE, 
city STRING) USING HUDI
   PARTITIONED BY (city)
   TBLPROPERTIES (primaryKey = 'uuid', preCombineField = 'ts');
   
   INSERT INTO t VALUES
   ('2023-09-19 
23:40:49',DATE'2023-09-19','u1','rider-A',19.10,'san_francisco'),
   ('2023-09-21 
16:34:26',DATE'2023-09-21','u2','rider-E',93.50,'san_francisco'),
   ('2023-09-23 09:42:17',DATE'2023-09-23','u3','rider-F',34.15,'sao_paulo'),
   ('2023-09-19 02:38:07',DATE'2023-09-19','u4','rider-I',41.06,'chennai');
   
   CREATE INDEX d_next ON t USING bloom_filters(d) options(expr='date_add', 
days='1');
   ```
   
   Master `7888bf909694`, Spark 3.5:
   
   | filter | data skipping off | data skipping on |
   | --- | --- | --- |
   | `date_add(d, 1) = DATE'2023-09-22'` | `rider-E` | no rows, 0 files |
   | `date_add(d, 1) IN (DATE'2023-09-22', DATE'2023-09-24')` | `rider-E`, 
`rider-F` | no rows, 0 files |
   
   ### Suggested fix
   
   Build the probe key from the literal's external value, so that it has the 
writer's string form. Until then, a bloom filter expression index could skip 
terms that are not string or integral.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]

Reply via email to