cshuo commented on code in PR #20481:
URL: https://github.com/apache/hudi/pull/20481#discussion_r4434115489
##########
hudi-flink-datasource/hudi-flink/src/main/java/org/apache/hudi/table/HoodieTableSource.java:
##########
@@ -399,7 +399,10 @@ public Result applyFilters(List<ResolvedExpression>
filters) {
Tuple2<List<ResolvedExpression>, List<ResolvedExpression>> splitFilters =
splitExprByPartitionCall(simpleFilters, this.partitionKeys, this.tableRowType);
this.predicates = ExpressionPredicates.fromExpression(splitFilters.f0);
this.columnStatsProbe = ColumnStatsProbe.newInstance(splitFilters.f0);
- this.partitionPruner = createPartitionPruner(splitFilters.f1,
columnStatsProbe);
+ // Partition stats only reflect the latest table state and can exclude
values in historical incremental reads.
+ boolean incrementalRead = conf.get(FlinkOptions.READ_AS_STREAMING)
Review Comment:
Thanks for pointing this out. For the table source,
`HoodieTableFactory.setupReadOptions` calls
`OptionsResolver.isIncrementalQuery` and sets `QUERY_TYPE` to `incremental`
when either boundary is configured, before `applyFilters` runs.
The historical-read regression test also covers this configuration: it sets
`read.start-commit` and `read.end-commit` without explicitly setting
`query.type`, for both batch and streaming reads.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
