voonhous opened a new issue, #20569:
URL: https://github.com/apache/hudi/issues/20569

   ### Describe the problem
   
   On an expression index with index version V1 (table version below 9), a 
filter on a function whose result type differs from the column's type fails the 
query. `ExpressionIndexSupport.transpose` decodes the stored min / max with the 
source column's type (`ExpressionIndexSupport.scala:269`, then 
`extractWrapperValueV1`), but the values were written with the function's type. 
With `ts TIMESTAMP`, `hour(ts)` stores INT stats, and `deserialize` does 
`value.asInstanceOf[Long]` for `TimestampType` 
(`ColumnStatsIndexSupport.scala:510`).
   
   The failure comes from `prunePartitions`, so 
`hoodie.fileIndex.dataSkippingFailureMode=fallback` does not catch it. V2 is 
not affected: it decodes from the stored value metadata.
   
   ### Repro
   
   ```sql
   SET hoodie.write.table.version=8;
   SET hoodie.write.auto.upgrade=false;
   SET hoodie.metadata.index.column.stats.enable=false;
   SET 
hoodie.write.lock.provider=org.apache.hudi.client.transaction.lock.InProcessLockProvider;
   
   CREATE TABLE t (ts TIMESTAMP, uuid STRING, rider STRING, fare DOUBLE, city 
STRING) USING HUDI
   PARTITIONED BY (city)
   TBLPROPERTIES (primaryKey = 'uuid', preCombineField = 'ts', 
'hoodie.write.table.version' = '8');
   
   INSERT INTO t VALUES
   (TIMESTAMP'2023-09-19 23:40:49','u1','rider-A',19.10,'san_francisco'),
   (TIMESTAMP'2023-09-21 16:34:26','u2','rider-E',93.50,'san_francisco'),
   (TIMESTAMP'2023-09-23 09:42:17','u3','rider-F',34.15,'sao_paulo'),
   (TIMESTAMP'2023-09-19 02:38:07','u4','rider-I',41.06,'chennai');
   
   CREATE INDEX ts_hour ON t USING column_stats(ts) options(expr='hour');
   CREATE INDEX ts_year ON t USING column_stats(ts) options(expr='year');
   ```
   
   Master `7888bf909694`, Spark 3.5, `hoodie.enable.data.skipping=true`, both 
indexes at V1:
   
   | filter | result |
   | --- | --- |
   | `hour(ts) = 16` | fails: `HoodieException: Error occurs when executing 
mapToPair` |
   | `hour(ts) IN (16, 19)` | fails, same error |
   | `year(ts) = 2023` | fails, same error |
   | `year(ts) IN (2023, 2024)` | runs, index not read |
   
   The last row only runs because the `IN` branch of `filterQueryWithRecordKey` 
does not unwrap the cast Spark adds inside `year`. #20540 makes that filter 
read the index, so it fails the same way there.
   
   ### Suggested fix
   
   Decode V1 values with the type of the indexed expression instead of the 
source column's type. #20540 already passes that type to `transpose` as 
`statsType`.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]

Reply via email to