[
https://issues.apache.org/jira/browse/IMPALA-15197?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=18097706#comment-18097706
]
ASF subversion and git services commented on IMPALA-15197:
----------------------------------------------------------
Commit bd13998f7da0b86403c033235c7c1ae9ce55c11b in impala's branch
refs/heads/master from Peter Rozsa
[ https://gitbox.apache.org/repos/asf?p=impala.git;h=bd13998f7 ]
IMPALA-15197: Shuffle Iceberg position deletes by file path for V2 tables
An UPDATE/DELETE with a join on an unpartitioned Iceberg V2 table could
silently produce a duplicate row instead of failing with "Duplicated row in
DELETE sink." when the join matched a target row more than once.
IcebergBufferedDeleteSink detects duplicate position deletes per fragment
instance, but unpartitioned V2 tables did not shuffle by data file path, so
duplicate deletes for the same file could land on different instances and go
undetected. This shuffles by INPUT__FILE__NAME for all unpartitioned Iceberg
tables.
Testing:
- Updated iceberg-v2-delete and iceberg-v2-update planner tests.
Change-Id: I2391cae4aa73eace9b902b31af55f3c461c20057
Assisted-by: Claude Opus 4.8 (OpenCode)
Reviewed-on: http://gerrit.cloudera.org:8080/24610
Reviewed-by: Impala Public Jenkins <[email protected]>
Tested-by: Impala Public Jenkins <[email protected]>
> Iceberg V2 UPDATE with join can miss duplicate-row detection in distributed
> execution
> -------------------------------------------------------------------------------------
>
> Key: IMPALA-15197
> URL: https://issues.apache.org/jira/browse/IMPALA-15197
> Project: IMPALA
> Issue Type: Bug
> Reporter: Peter Rozsa
> Assignee: Peter Rozsa
> Priority: Major
> Labels: impala-iceberg
>
> UPDATE ... FROM on an unpartitioned Iceberg V2 table that joins against a
> reference table may not detect when the same target row matches multiple
> reference rows. In that case the UPDATE should fail with Duplicated row in
> DELETE sink. but instead succeeds silently and produces a duplicate row in
> the table.
>
> {code:java}
> create table tgt (id int, val bigint)
> stored by iceberg tblproperties ('format-version'='2');
> insert into tgt values (0,0),(1,10),(2,20),(3,30),(4,40),(5,50),(6,60),(7,70);
> compute stats tgt;
> -- Reference table with no stats so the planner puts it on the probe side
> create table ref_pq (i int, bi bigint) stored as parquet;
> insert into ref_pq values
> (0,100),(1,101),(2,102),(3,103),(4,104),(5,105),(6,106),(7,107);
> insert into ref_pq values
> (0,200),(1,201),(2,202),(3,203),(4,204),(5,205),(6,206),(7,207);
> -- Each id in ref_pq appears twice -> multiple matches per target row
> update tgt set val = bi + 1 from tgt, ref_pq where id = i;
> select count(*) total, count(distinct id) distinct_ids from tgt; {code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
