danny0405 commented on code in PR #20654:
URL: https://github.com/apache/hudi/pull/20654#discussion_r4561149881
##########
hudi-flink-datasource/hudi-flink/src/main/java/org/apache/hudi/table/format/RecordIterators.java:
##########
@@ -157,6 +155,21 @@ public static ClosableIterator<RowData>
getParquetRecordIterator(
}
}
+ static FilterPredicate createFilterPredicate(MessageType fileSchema,
List<Predicate> predicates, FilterPredicate configuredFilter) {
+ FilterPredicate filterPredicate = configuredFilter;
Review Comment:
[P2] Validate the configured predicate against the physical schema too
The new guard only covers `predicates`; `configuredFilter` is copied
straight into the result. If the Hadoop configuration contains
`ParquetInputFormat.FILTER_PREDICATE = eq(binaryColumn("uuid"), "k")`,
`getFilterPredicate(conf)` passes it here even for a native delete file whose
schema contains only `_hoodie_record_key` and ordering fields. With an empty
query-predicate list this method still returns that filter, and Parquet can
discard every delete row group because `uuid` is absent, leaving the original
tombstone-loss issue reachable through this supported configuration path.
Could we apply the same physical-column validation to the configured
predicate (conservatively omit the whole expression when a referenced column is
absent), or explicitly prevent configured predicates from reaching native
delete scans? Please add a regression that sets `FILTER_PREDICATE` in the
configuration and verifies the tombstone survives. I reproduced the
predicate-retention/row-group-drop behavior with an isolated probe of this
helper's logic against Parquet 1.15.2; I have not run an end-to-end Flink test
for this configuration.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]