Indian-boult opened a new pull request, #20740: URL: https://github.com/apache/hudi/pull/20740
### Describe the issue this Pull Request addresses Closes #20569 On a table of version 8, an expression index stores V1 values, which record only their physical type (a DATE is an INT). `ExpressionIndexSupport#transpose` decoded them with the type of the source column, but the writer stored the values of the indexed expression. An index whose function changes the type fails the query, for ex: - `hour(ts)` or `year(ts)` on a TIMESTAMP column stores INT values, and decoding them as TIMESTAMP throws `ClassCastException: class java.lang.Integer cannot be cast to class java.lang.Long` at `ColumnStatsIndexSupport.deserialize`. - `date_format(ts, 'yyyy-MM-dd')` stores STRING values: `java.lang.String cannot be cast to class java.lang.Long`. - `to_date(day)` and `to_timestamp(day)` on a STRING column store DATE and TIMESTAMP values. They come back as INT and BIGINT and fail when the transposed DataFrame is built: `The value (19619) of the type (java.lang.Integer) cannot be converted to the DATE type`. The error is thrown from `HoodieFileIndex#prunePartitionsAndGetFileSlices` (expression index partition pruning), so the query fails with `hoodie.fileIndex.dataSkippingFailureMode=fallback` (the default) as well as `strict`. A function that returns a STRING or a number over a STRING or numeric column, such as `upper(rider)`, was not affected: those values pass through the decoder unchanged. Table version 9 (V2) is not affected: V2 values are decoded from their value metadata. ### Summary and Changelog A V1 expression index on a function that changes the type is read again: the query returns the right rows and prunes files, as on table version 9. - `ExpressionIndexSupport#transpose`: decode V1 min / max values with `statsType`, the type of the query term that matched the index, instead of the source column's type. #20540 already passes `statsType` to `transpose` and types the transposed schema with it, so the decoded values now match that schema. The unused table schema field map is removed. V2 decoding is unchanged. - Other V1 decoders: `ColumnStatsIndexSupport#transpose` decodes column stats with the column's own type, which is right there. No other code decodes expression index values. - Tests: `TestExpressionIndex#Test Expression Index Of Table Version 8 Reads Stats Of The Expression's Type` uses the table and filters of #20569 (one base file per partition), plus a STRING column. It checks that the table is at version 8 and the five indexes are at V1, then runs `hour(ts) = 16`, `hour(ts) IN (16, 19)`, `year(ts) = 2023`, `year(ts) IN (2023, 2024)`, `year(ts) = 2024`, `date_format(ts, 'yyyy-MM-dd') = '2023-09-23'`, `to_date(day) = DATE'2023-09-21'` and `to_timestamp(day) = TIMESTAMP'2023-09-21 00:00:00'`, and asserts the rows and the number of files scanned. On master it fails at the first query with the `ClassCastException` above. ### Impact Queries with data skipping on a table of version 8 no longer fail when they filter on an expression index whose function changes the type, for ex `hour`, `year` or `date_format` on a TIMESTAMP column, or `to_date` or `to_timestamp` on a STRING column. No public API or config change. ### Risk Level low. One decode call changes, on the V1 path of the expression index reader only. For the functions that were not affected, the decoded values are the same as before. A temporary probe ran 10 filters on the same table at versions 8 and 9: with the fix, version 8 returns the same rows and scans the same number of files as version 9, and version 9 results are unchanged. ### Documentation Update none ### Contributor's checklist - [x] Read through [contributor's guide](https://hudi.apache.org/contribute/how-to-contribute) - [x] Enough context is provided in the sections above - [x] Adequate tests were added if applicable -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
