cshuo commented on code in PR #20366:
URL: https://github.com/apache/hudi/pull/20366#discussion_r4303123521
##########
hudi-client/hudi-client-common/src/main/java/org/apache/hudi/io/FileGroupReaderBasedNativeLogAppendHandle.java:
##########
@@ -97,8 +98,13 @@ public void doAppend() {
.withSortOutput(hoodieTable.requireSortedRecords())
// instead of using config.enableOptimizedLogBlocksScan(), we set to
true as log compaction blocks only supported in scanV2
.build()) {
+ // Log compaction reads deletes through the engine reader context. A
table that writes them through another
+ // record context, e.g. the Avro one for the Avro record type, can only
store Java ordering values.
+ boolean javaOrderingForDeletes =
readerContext.getRecordContext().getEngineRecordType()
+ != hoodieTable.getRecordContextForWrite().getEngineRecordType();
recordItr = new
CloseableMappingIterator<>(fileGroupReader.getLogRecordsOnly(), record -> {
- HoodieRecord<T> hoodieRecord =
readerContext.getRecordContext().constructHoodieRecord(record);
+ HoodieRecord<T> hoodieRecord =
readerContext.getRecordContext().constructHoodieRecord(
+ javaOrderingForDeletes ? withJavaOrderingValue(record) : record);
Review Comment:
**Align the log-compaction reader with the write record type**
The changes to this handle should be unnecessary if log compaction selects a
reader record context that matches its writer record context. In the
non-metadata `LOG_COMPACT` branch of `HoodieCompactor`, could we use:
```java
readerContextFactory = context.getReaderContextFactoryForWrite(
metaClient, config.getRecordMerger().getRecordType(), config.getProps());
```
This selects an Avro reader for Avro writes and a Spark reader for Spark
writes, eliminating the cross-context ordering conversion here. The existing
metadata-table reader setup can remain unchanged.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
