[ https://issues.apache.org/jira/browse/IMPALA-11388?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17798769#comment-17798769 ]
ASF subversion and git services commented on IMPALA-11388: ---------------------------------------------------------- Commit 012996a06b9978200bf061a9d26a3b248bcb605a in impala's branch refs/heads/master from Gabor Kaszab [ https://gitbox.apache.org/repos/asf?p=impala.git;h=012996a06 ] IMPALA-12597: Basic Equality delete read support for Iceberg tables In general, applying equality deletes is similar to how position deletes are applied to data files: using a LEFT ANTI JOIN where the SCAN for the data rows is on the left side while the SCAN for the delete rows is on the right side of the JOIN. The difference is the virtual columns and the conjuncts being used. For equality deletes the data sequence number of a delete file has to be greater than the data sequence number of the data file being investigated. This information is added as a virtual column to the scans and a conjunct is created in the JOIN node to check the relation. The equality delete fields from the delete files are checked agains the respective columns of the data SCANS. This patch makes it possible for Impala to read Iceberg tables with basic equality delete files. The Iceberg spec gives great flexibility for engines for writing equality deletes, however in practice Flink, one of the engines that write EQ-deletes supports only a subset of the use cases. This patch focuses on reading the EQ-deletes written by Flink. The restrictions are the following: - All equality delete files in a table should have the same equality field ID list. - For partitioned Iceberg tables it is expected that the partition values are also written into the equality delete files. - Tables with equality deletes shouldn't have partition or schema evolution. - Floating point equality columns aren't supported. - If a malformed equality delete file doesn't have some of the equality field IDs then Parquet reader will fill those missing fields with NULLs. As a side effect this will drop the rows from the result where the corresponding data columns have a null value. See IMPALA-11388 epic Jira for more details. Testing: - Checked if the existing functional_parquet.iceberg_v2_delete_equality table can be read successfully. - Added new test tables so that E2E tests can validate correctness. Change-Id: I2053e6f321c69f1c82059a84a5d99aeaa9814cad Reviewed-on: http://gerrit.cloudera.org:8080/20753 Reviewed-by: Impala Public Jenkins <impala-public-jenk...@cloudera.com> Tested-by: Impala Public Jenkins <impala-public-jenk...@cloudera.com> > Add support for equality-based deletes > -------------------------------------- > > Key: IMPALA-11388 > URL: https://issues.apache.org/jira/browse/IMPALA-11388 > Project: IMPALA > Issue Type: Epic > Components: Frontend > Reporter: Zoltán Borók-Nagy > Assignee: Gabor Kaszab > Priority: Major > Labels: impala-iceberg > > Iceberg V2 adds support for row-level modifications. > One way to implement this is via equality based delete files: > https://iceberg.apache.org/spec/#equality-delete-files > https://iceberg.apache.org/spec/#scan-planning > We could implement this via doing ANTI HASH JOIN between data and delete > files. Similarly to what we do for Hive full ACID tables: > https://github.com/apache/impala/blob/f5fc08573352d0a1943296209791a4db17268086/fe/src/main/java/org/apache/impala/planner/SingleNodePlanner.java#L1729-L1735 > The complexity comes when different delete files use different set of > columns. In that case we will need multiple ANTI HASH JOINs on top of each > other. -- This message was sent by Atlassian Jira (v8.20.10#820010) --------------------------------------------------------------------- To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org For additional commands, e-mail: issues-all-h...@impala.apache.org