[
https://issues.apache.org/jira/browse/IMPALA-14921?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=18076696#comment-18076696
]
ASF subversion and git services commented on IMPALA-14921:
----------------------------------------------------------
Commit 10772982f204c06deb68a44ffcd66eebd86ba615 in impala's branch
refs/heads/master from Peter Rozsa
[ https://gitbox.apache.org/repos/asf?p=impala.git;h=10772982f ]
IMPALA-14921: Push Iceberg deletion vectors to remote-FS scan ranges
Iceberg data files on object stores go through TFileSplitGeneratorSpec
rather than transformBlocksToScanRanges, so
IcebergScanNode.decorateScanRange() was never called for them. This
left TScanRange.iceberg_deletion_vector unpopulated after backend spec
expansion, causing IcebergDeleteBuilder to skip DV loading and return
deleted rows to the query.
Fix by introducing a decorateSplitSpec() hook in HdfsScanNode (mirroring
the existing decorateScanRange() hook) called from
generateScanRangeSpecs() before the spec is added to split_specs.
Scheduler::GenerateScanRanges propagates the binary blob to every
TScanRange it generates from the spec.
Testing:
- manually validated that an Iceberg table written to S3 with deletes
reads back correctly.
Change-Id: I41d7ba5b6e2033a9c2bcc700c6e4efb817368bbb
Reviewed-on: http://gerrit.cloudera.org:8080/24236
Reviewed-by: Impala Public Jenkins <[email protected]>
Tested-by: Impala Public Jenkins <[email protected]>
> Iceberg test_v3_delete failures
> -------------------------------
>
> Key: IMPALA-14921
> URL: https://issues.apache.org/jira/browse/IMPALA-14921
> Project: IMPALA
> Issue Type: Bug
> Reporter: Quanlong Huang
> Assignee: Peter Rozsa
> Priority: Critical
> Labels: broken-build
>
> Saw these tests failed in a downstream build:
> {noformat}
> query_test.test_iceberg.TestIcebergV3Table.test_v3_delete["table_format:
> parquet/none" | "protocol: hs2" | "exec_option: {'abort_on_error': 1,
> 'batch_size': 0, 'disable_codegen': False, 'disable_codegen_rows_threshold':
> 0, 'exec_single_node_rows_threshold': 0, 'num_nodes': 0, 'test_replan': 1}"]
> query_test.test_iceberg.TestIcebergV3Table.test_v3_delete_on_existing_dv_table["table_format:
> parquet/none" | "protocol: hs2" | "exec_option: {'abort_on_error': 1,
> 'batch_size': 0, 'disable_codegen': False, 'disable_codegen_rows_threshold':
> 0, 'exec_single_node_rows_threshold': 0, 'num_nodes': 0, 'test_replan': 1}"]
> query_test.test_iceberg.TestIcebergV3Table.test_v3_delete_on_v2_equality_delete_table["table_format:
> parquet/none" | "protocol: hs2" | "exec_option: {'abort_on_error': 1,
> 'batch_size': 0, 'disable_codegen': False, 'disable_codegen_rows_threshold':
> 0, 'exec_single_node_rows_threshold': 0, 'num_nodes': 0, 'test_replan':
> 1}"]{noformat}
> test_v3_delete
> {noformat}
> SELECT * FROM iceberg_v3_deletion_vectors;
> -- 2026-04-23 18:44:21,806 INFO MainThread:
> 744acc0c4cd7ecdc:ca2ac51500000000: query started
> -- 2026-04-23 18:44:21,851 INFO MainThread:
> 744acc0c4cd7ecdc:ca2ac51500000000: getting log for operation
> -- 2026-04-23 18:44:21,852 INFO MainThread:
> 744acc0c4cd7ecdc:ca2ac51500000000: getting runtime profile operation
> -- 2026-04-23 18:44:21,855 INFO MainThread:
> 744acc0c4cd7ecdc:ca2ac51500000000: closing query for operation
> -- 2026-04-23 18:44:21,855 ERROR MainThread: Comparing QueryTestResults
> (expected vs actual):
> 1 == 1
> 3 != 2
> 5 != 3
> Number of rows returned (expected vs actual): 3 != 5{noformat}
> test_v3_delete_on_existing_dv_table
> {noformat}
> SELECT * FROM iceberg_v3_deletion_vectors ORDER BY i;
> -- 2026-04-23 18:44:45,644 INFO MainThread:
> b4491db3f045cee0:864f6eee00000000: query started
> -- 2026-04-23 18:44:45,646 INFO MainThread:
> b4491db3f045cee0:864f6eee00000000: getting log for operation
> -- 2026-04-23 18:44:45,646 INFO MainThread:
> b4491db3f045cee0:864f6eee00000000: getting runtime profile operation
> -- 2026-04-23 18:44:45,650 INFO MainThread:
> b4491db3f045cee0:864f6eee00000000: closing query for operation
> -- 2026-04-23 18:44:45,650 ERROR MainThread: Comparing QueryTestResults
> (expected vs actual):
> 1 == 1
> 3 != 2
> 5 != 3
> Number of rows returned (expected vs actual): 3 != 5{noformat}
> test_v3_delete_on_v2_equality_delete_table
> {noformat}
> SELECT * FROM iceberg_v2_delete_equality ORDER BY id;
> -- 2026-04-23 18:49:39,253 INFO MainThread:
> 4a47e112a2c6e5ff:604f43ea00000000: query started
> -- 2026-04-23 18:49:39,256 INFO MainThread:
> 4a47e112a2c6e5ff:604f43ea00000000: getting log for operation
> -- 2026-04-23 18:49:39,256 INFO MainThread:
> 4a47e112a2c6e5ff:604f43ea00000000: getting runtime profile operation
> -- 2026-04-23 18:49:39,261 INFO MainThread:
> 4a47e112a2c6e5ff:604f43ea00000000: closing query for operation
> -- 2026-04-23 18:49:39,261 ERROR MainThread: Comparing QueryTestResults
> (expected vs actual):
> 2,'test_2_updated' != 1,'test_1_base'
> Number of rows returned (expected vs actual): 1 != 2{noformat}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
