[
https://issues.apache.org/jira/browse/IMPALA-14587?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=18075878#comment-18075878
]
ASF subversion and git services commented on IMPALA-14587:
----------------------------------------------------------
Commit 7ceefb1ed84c2d3e69cdf5688d294adcb67ee28a in impala's branch
refs/heads/master from Peter Rozsa
[ https://gitbox.apache.org/repos/asf?p=impala.git;h=7ceefb1ed ]
IMPALA-14588: Implement Iceberg deletion vector reading/writing
This is the second part of a multi-part implementation adding support
for Iceberg deletion vectors stored in Puffin files. The first part
is added as IMPALA-14755, this patch covers IMPALA-14587 and
IMPALA-14588. This commit wires the Puffin reader/writer infrastructure
from part 1 into the query execution pipeline and catalog layer,
enabling DELETE on Iceberg V3 tables using deletion vectors.
Puffin file writing is partition-scoped: the delete sink creates one
Puffin file per output partition, and each blob inside that file is a
serialised RoaringBitmap64 covering exactly the deleted row positions
of one data file in that partition. When a data file already has a
deletion vector from a previous DELETE, the existing bitmap is OR-ed
with the new one before the merged blob is written, so each Puffin file
always holds the complete, up-to-date set of deleted positions for its
partition.
DELETE on a V3 table is rejected at analysis time if the table has any
existing V2 position- or equality-delete files. The table must first be
compacted with OPTIMIZE TABLE to remove those files before DELETE can be
used on them.
Testing:
- iceberg-v3-delete.test and iceberg-v3-delete-partition-sort.test
added.
- Manually validated that deletion vectors written by Spark can be read
from Impala and deletion vectors written by Impala can be read from
Spark.
Generated-by: Claude Sonnet 4.6
Change-Id: I5613c31a7aa46b94b7c70386c939c08cc68632cd
Reviewed-on: http://gerrit.cloudera.org:8080/24143
Reviewed-by: Zoltan Borok-Nagy <[email protected]>
Tested-by: Impala Public Jenkins <[email protected]>
> Add read support for Iceberg V3 deletion vectors
> ------------------------------------------------
>
> Key: IMPALA-14587
> URL: https://issues.apache.org/jira/browse/IMPALA-14587
> Project: IMPALA
> Issue Type: Sub-task
> Reporter: Zoltán Borók-Nagy
> Priority: Major
> Labels: impala-iceberg
>
> Add read support for tables that have Iceberg V3 deletion vectors internally.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
