Indian-boult opened a new pull request, #20740:
URL: https://github.com/apache/hudi/pull/20740

   
   ### Describe the issue this Pull Request addresses
   
   Closes #20569
   
   On a table of version 8, an expression index stores V1 values, which record 
only their physical type (a DATE is an INT). `ExpressionIndexSupport#transpose` 
decoded them with the type of the source column, but the writer stored the 
values of the indexed expression. An index whose function changes the type 
fails the query, for ex:
   
   - `hour(ts)` or `year(ts)` on a TIMESTAMP column stores INT values, and 
decoding them as TIMESTAMP throws `ClassCastException: class java.lang.Integer 
cannot be cast to class java.lang.Long` at 
`ColumnStatsIndexSupport.deserialize`.
   - `date_format(ts, 'yyyy-MM-dd')` stores STRING values: `java.lang.String 
cannot be cast to class java.lang.Long`.
   - `to_date(day)` and `to_timestamp(day)` on a STRING column store DATE and 
TIMESTAMP values. They come back as INT and BIGINT and fail when the transposed 
DataFrame is built: `The value (19619) of the type (java.lang.Integer) cannot 
be converted to the DATE type`.
   
   The error is thrown from `HoodieFileIndex#prunePartitionsAndGetFileSlices` 
(expression index partition pruning), so the query fails with 
`hoodie.fileIndex.dataSkippingFailureMode=fallback` (the default) as well as 
`strict`. A function that returns a STRING or a number over a STRING or numeric 
column, such as `upper(rider)`, was not affected: those values pass through the 
decoder unchanged. Table version 9 (V2) is not affected: V2 values are decoded 
from their value metadata.
   
   ### Summary and Changelog
   
   A V1 expression index on a function that changes the type is read again: the 
query returns the right rows and prunes files, as on table version 9.
   
   - `ExpressionIndexSupport#transpose`: decode V1 min / max values with 
`statsType`, the type of the query term that matched the index, instead of the 
source column's type. #20540 already passes `statsType` to `transpose` and 
types the transposed schema with it, so the decoded values now match that 
schema. The unused table schema field map is removed. V2 decoding is unchanged.
   - Other V1 decoders: `ColumnStatsIndexSupport#transpose` decodes column 
stats with the column's own type, which is right there. No other code decodes 
expression index values.
   - Tests: `TestExpressionIndex#Test Expression Index Of Table Version 8 Reads 
Stats Of The Expression's Type` uses the table and filters of #20569 (one base 
file per partition), plus a STRING column. It checks that the table is at 
version 8 and the five indexes are at V1, then runs `hour(ts) = 16`, `hour(ts) 
IN (16, 19)`, `year(ts) = 2023`, `year(ts) IN (2023, 2024)`, `year(ts) = 2024`, 
`date_format(ts, 'yyyy-MM-dd') = '2023-09-23'`, `to_date(day) = 
DATE'2023-09-21'` and `to_timestamp(day) = TIMESTAMP'2023-09-21 00:00:00'`, and 
asserts the rows and the number of files scanned. On master it fails at the 
first query with the `ClassCastException` above.
   
   ### Impact
   
   Queries with data skipping on a table of version 8 no longer fail when they 
filter on an expression index whose function changes the type, for ex `hour`, 
`year` or `date_format` on a TIMESTAMP column, or `to_date` or `to_timestamp` 
on a STRING column. No public API or config change.
   
   ### Risk Level
   
   low. One decode call changes, on the V1 path of the expression index reader 
only. For the functions that were not affected, the decoded values are the same 
as before. A temporary probe ran 10 filters on the same table at versions 8 and 
9: with the fix, version 8 returns the same rows and scans the same number of 
files as version 9, and version 9 results are unchanged.
   
   ### Documentation Update
   
   none
   
   ### Contributor's checklist
   
   - [x] Read through [contributor's 
guide](https://hudi.apache.org/contribute/how-to-contribute)
   - [x] Enough context is provided in the sections above
   - [x] Adequate tests were added if applicable
   
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]

  • [PR] fix(spark): decode V1 expr... Indian-boult (via GitHub) via commits

Reply via email to