shangxinli commented on code in PR #20481:
URL: https://github.com/apache/hudi/pull/20481#discussion_r4433997215
##########
hudi-flink-datasource/hudi-flink/src/main/java/org/apache/hudi/table/HoodieTableSource.java:
##########
@@ -399,7 +399,10 @@ public Result applyFilters(List<ResolvedExpression>
filters) {
Tuple2<List<ResolvedExpression>, List<ResolvedExpression>> splitFilters =
splitExprByPartitionCall(simpleFilters, this.partitionKeys, this.tableRowType);
this.predicates = ExpressionPredicates.fromExpression(splitFilters.f0);
this.columnStatsProbe = ColumnStatsProbe.newInstance(splitFilters.f0);
- this.partitionPruner = createPartitionPruner(splitFilters.f1,
columnStatsProbe);
+ // Partition stats only reflect the latest table state and can exclude
values in historical incremental reads.
+ boolean incrementalRead = conf.get(FlinkOptions.READ_AS_STREAMING)
Review Comment:
this is narrower than OptionsResolver.isIncrementalQuery, which also treats
READ_START_COMMIT/READ_END_COMMIT being set as incremental regardless of
QUERY_TYPE. a batch read with those set but QUERY_TYPE left at the snapshot
default still hits the stale-partition-stats bug this PR is fixing.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
