voonhous opened a new issue, #20569: URL: https://github.com/apache/hudi/issues/20569
### Describe the problem On an expression index with index version V1 (table version below 9), a filter on a function whose result type differs from the column's type fails the query. `ExpressionIndexSupport.transpose` decodes the stored min / max with the source column's type (`ExpressionIndexSupport.scala:269`, then `extractWrapperValueV1`), but the values were written with the function's type. With `ts TIMESTAMP`, `hour(ts)` stores INT stats, and `deserialize` does `value.asInstanceOf[Long]` for `TimestampType` (`ColumnStatsIndexSupport.scala:510`). The failure comes from `prunePartitions`, so `hoodie.fileIndex.dataSkippingFailureMode=fallback` does not catch it. V2 is not affected: it decodes from the stored value metadata. ### Repro ```sql SET hoodie.write.table.version=8; SET hoodie.write.auto.upgrade=false; SET hoodie.metadata.index.column.stats.enable=false; SET hoodie.write.lock.provider=org.apache.hudi.client.transaction.lock.InProcessLockProvider; CREATE TABLE t (ts TIMESTAMP, uuid STRING, rider STRING, fare DOUBLE, city STRING) USING HUDI PARTITIONED BY (city) TBLPROPERTIES (primaryKey = 'uuid', preCombineField = 'ts', 'hoodie.write.table.version' = '8'); INSERT INTO t VALUES (TIMESTAMP'2023-09-19 23:40:49','u1','rider-A',19.10,'san_francisco'), (TIMESTAMP'2023-09-21 16:34:26','u2','rider-E',93.50,'san_francisco'), (TIMESTAMP'2023-09-23 09:42:17','u3','rider-F',34.15,'sao_paulo'), (TIMESTAMP'2023-09-19 02:38:07','u4','rider-I',41.06,'chennai'); CREATE INDEX ts_hour ON t USING column_stats(ts) options(expr='hour'); CREATE INDEX ts_year ON t USING column_stats(ts) options(expr='year'); ``` Master `7888bf909694`, Spark 3.5, `hoodie.enable.data.skipping=true`, both indexes at V1: | filter | result | | --- | --- | | `hour(ts) = 16` | fails: `HoodieException: Error occurs when executing mapToPair` | | `hour(ts) IN (16, 19)` | fails, same error | | `year(ts) = 2023` | fails, same error | | `year(ts) IN (2023, 2024)` | runs, index not read | The last row only runs because the `IN` branch of `filterQueryWithRecordKey` does not unwrap the cast Spark adds inside `year`. #20540 makes that filter read the index, so it fails the same way there. ### Suggested fix Decode V1 values with the type of the indexed expression instead of the source column's type. #20540 already passes that type to `transpose` as `statsType`. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
