rdblue commented on code in PR #6775:
URL: https://github.com/apache/iceberg/pull/6775#discussion_r1111320283
##########
python/pyiceberg/io/pyarrow.py:
##########
@@ -546,11 +568,45 @@ def project_table(
id for id in projected_schema.field_ids if not
isinstance(projected_schema.find_type(id), (MapType, ListType))
}.union(extract_field_ids(bound_row_filter))
+ tasks_data_files: List[FileScanTask] = []
+ tasks_positional_deletes: List[FileScanTask] = []
+ for task in tasks:
Review Comment:
Scan planning should use an index of delete files to find delete files for
each file task. Then the file task has a data file to read and deletes to
apply. That process should handle sequence number filtering so that we don't
apply deletes to files that have newer sequence numbers.
At this point, all we should need to do is to read the delete files, filter
the deletes by the data file path, then merge the resulting delete vectors
together. Then do the mask-based filter that you have above.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
