cshuo commented on code in PR #20654:
URL: https://github.com/apache/hudi/pull/20654#discussion_r4559507873
##########
hudi-flink-datasource/hudi-flink/src/main/java/org/apache/hudi/table/format/FlinkRowDataReaderContext.java:
##########
@@ -113,8 +113,11 @@ public ClosableIterator<RowData> getFileRecordIterator(
.getReaderFactory(HoodieRecord.HoodieRecordType.FLINK)
.getFileReader(tableConfig, filePath, format, Option.empty());
try {
+ // Native delete logs may omit business key columns, so query predicates
can discard tombstones needed for merging.
+ List<ExpressionPredicates.Predicate> predicates =
FSUtils.isNativeDeleteLogFile(filePath.getName())
Review Comment:
1. **Merge semantics:** Agreed. We already restrict MOR pushdown to
predicates on primary-key columns or `_hoodie_record_key`. Even with this
safeguard, applying those predicates to native delete files is difficult
because we store the encoded record key rather than the business-key columns.
Translating predicates requires preserving type and composite-key encoding
semantics. This PR therefore skips delete-log pushdown while retaining the
existing MOR-safe filtering for base files and data logs.
2. For **Spark:** : Spark avoids this specific missing-column issue:
`HoodieSparkParquetReader` builds `ParquetFilters` from the actual file schema,
so predicates referencing absent business-key columns are not pushed down.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]