aokolnychyi commented on a change in pull request #1094:
URL: https://github.com/apache/iceberg/pull/1094#discussion_r435592491
##########
File path: core/src/main/java/org/apache/iceberg/AllDataFilesTable.java
##########
@@ -85,19 +86,23 @@ public Schema schema() {
private AllDataFilesTableScan(
TableOperations ops, Table table, Long snapshotId, Schema schema,
Expression rowFilter,
- boolean caseSensitive, boolean colStats, Collection<String>
selectedColumns, Schema fileSchema,
+ boolean ignoreResiduals, boolean caseSensitive, boolean colStats,
+ Collection<String> selectedColumns, Schema fileSchema,
ImmutableMap<String, String> options) {
- super(ops, table, snapshotId, schema, rowFilter, caseSensitive,
colStats, selectedColumns, options);
+ super(
+ ops, table, snapshotId, schema, rowFilter, ignoreResiduals,
+ caseSensitive, colStats, selectedColumns, options);
Review comment:
I don't have a strong opinion here. We have to check whether we can hit
this filtering problem during our actions that rewrite metadata in the future.
In metadata tables, we have either static tasks or we read Avro right now. So,
we cannot really skip parts of a file like in Parquet, but it is still
important to think this through. What if something changes and we start
filtering out records on our side before passing records to query engines or
write some metadata as Parquet? Static metadata tables have `alwaysTrue` as
their data filter. Others, on the other hand, do have a valid data filter
passed further that can cause issues.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
