Guosmilesmile commented on code in PR #17946:
URL: https://github.com/apache/iceberg/pull/17946#discussion_r3950604496
##########
flink/v2.3/flink/src/main/java/org/apache/iceberg/flink/source/IcebergTableSource.java:
##########
@@ -166,18 +199,35 @@ public void applyLimit(long newLimit) {
@Override
public Result applyFilters(List<ResolvedExpression> flinkFilters) {
List<ResolvedExpression> acceptedFilters = Lists.newArrayList();
+ List<ResolvedExpression> remainingFilters = Lists.newArrayList();
List<Expression> expressions = Lists.newArrayList();
+ Table table = null;
+
for (ResolvedExpression resolvedExpression : flinkFilters) {
Optional<Expression> icebergExpression =
FlinkFilters.convert(resolvedExpression);
- if (icebergExpression.isPresent()) {
- expressions.add(icebergExpression.get());
- acceptedFilters.add(resolvedExpression);
+ if (icebergExpression.isEmpty()) {
+ remainingFilters.add(resolvedExpression);
+ continue;
+ }
+
+ Expression expression = icebergExpression.get();
+ expressions.add(expression);
+ acceptedFilters.add(resolvedExpression);
+
+ if (table == null) {
+ table = loadTable();
+ }
+
+ if (ExpressionUtil.selectsPartitions(expression, table, caseSensitive)) {
+ LOG.info("Evaluating {} entirely on the Iceberg side", expression);
+ } else {
+ remainingFilters.add(resolvedExpression);
}
}
this.filters = expressions;
- return Result.of(acceptedFilters, flinkFilters);
+ return Result.of(acceptedFilters, remainingFilters);
Review Comment:
I’m planning to keep this as is for now and see what everyone thinks about
the overall direction. If the PR is considered large, we can split it up later.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
