wombatu-kun opened a new pull request, #20366: URL: https://github.com/apache/hudi/pull/20366
### Describe the issue this Pull Request addresses Closes #20365 Spark log compaction failed on table versions 8 and 9 whenever it rewrote a delete with a string or decimal ordering value. The file group reader had already converted those values to Spark engine types, and `HoodieDeleteBlock` rejects them with `UnsupportedOperationException` from `wrapValueIntoAvro`. ### Summary and Changelog - New `RecordContext.convertOrderingValueToJava` turns an engine-typed ordering value into a Java type that a delete block can store. It is the identity by default. - `BaseSparkInternalRecordContext` overrides it: `HoodieUTF8String` becomes `String` and Spark `Decimal` becomes `BigDecimal`, element by element when there are several ordering fields. - `FileGroupReaderBasedInlineLogAppendHandle` converts the ordering value of every delete before it rebuilds the record for the compacted log file. - `TestMORDataSource` runs log compaction over a delete block with string and decimal ordering, for both record types, at versions 8 and 9. After compaction, an older update must leave the key deleted and a newer one must bring it back. - `TestBaseSparkInternalRowReaderContext` covers the conversion itself. ### Impact Log compaction of Spark MOR tables at versions 8 and 9 works again for string and decimal ordering fields, and the compacted deletes keep their ordering values. Version 9 is the default of 1.2.x. The fix applies with the Avro record merger and the Spark one, because log compaction always reads through the Spark context. Other ordering types already reached the delete block as Java values and are written as before. Version 10 tables use the native log handle, which is unchanged. Flink pipelines only log-compact the metadata table, which reads through the Avro context, so no Flink override is added. #20341 fixes the same failure for deletes that win Spark precombine. ### Risk Level low The only change is the type of the ordering value that log compaction writes into inline delete blocks. Verified: - On master, every row of the new test fails with `Unsupported type of the value (class org.apache.hudi.Spark3HoodieUTF8String)` or `(class org.apache.spark.sql.types.Decimal)`. - Removing either the Spark override or the handle change brings that failure back. - With `hoodie.log.compaction.inline=true`, the same failure reaches a plain datasource write. ### Documentation Update none ### Contributor's checklist - [x] Read through [contributor's guide](https://hudi.apache.org/contribute/how-to-contribute) - [x] Enough context is provided in the sections above - [x] Adequate tests were added if applicable -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
