wombatu-kun commented on code in PR #20366:
URL: https://github.com/apache/hudi/pull/20366#discussion_r4303633226
##########
hudi-client/hudi-client-common/src/main/java/org/apache/hudi/io/FileGroupReaderBasedNativeLogAppendHandle.java:
##########
@@ -97,8 +98,13 @@ public void doAppend() {
.withSortOutput(hoodieTable.requireSortedRecords())
// instead of using config.enableOptimizedLogBlocksScan(), we set to
true as log compaction blocks only supported in scanV2
.build()) {
+ // Log compaction reads deletes through the engine reader context. A
table that writes them through another
+ // record context, e.g. the Avro one for the Avro record type, can only
store Java ordering values.
+ boolean javaOrderingForDeletes =
readerContext.getRecordContext().getEngineRecordType()
+ != hoodieTable.getRecordContextForWrite().getEngineRecordType();
recordItr = new
CloseableMappingIterator<>(fileGroupReader.getLogRecordsOnly(), record -> {
- HoodieRecord<T> hoodieRecord =
readerContext.getRecordContext().constructHoodieRecord(record);
+ HoodieRecord<T> hoodieRecord =
readerContext.getRecordContext().constructHoodieRecord(
+ javaOrderingForDeletes ? withJavaOrderingValue(record) : record);
Review Comment:
Done bbc84e2cb9af: the native handle is back to master, and
`HoodieNativeLogFormatWriter` now stores the Java copy of an empty record that
another record context built, the value the inline `writeDelete()` already
stores through `getOrderingValueAsJava()`. Log compaction keeps the Spark
reader because the Avro one breaks decimal ordering: with your change, a
decimal-ordered key whose update and older delete meet in log compaction, now
part of `testLogCompactionRewritesDeletesWithEngineOrdering`, fails at version
9 with `ClassCastException: GenericData$Fixed cannot be cast to BigDecimal` in
`shouldKeepNewerRecord`, since delete blocks yield `BigDecimal` and data blocks
raw `GenericData$Fixed`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
