[
https://issues.apache.org/jira/browse/IMPALA-12597?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17798768#comment-17798768
]
ASF subversion and git services commented on IMPALA-12597:
----------------------------------------------------------
Commit 012996a06b9978200bf061a9d26a3b248bcb605a in impala's branch
refs/heads/master from Gabor Kaszab
[ https://gitbox.apache.org/repos/asf?p=impala.git;h=012996a06 ]
IMPALA-12597: Basic Equality delete read support for Iceberg tables
In general, applying equality deletes is similar to how position
deletes are applied to data files: using a LEFT ANTI JOIN where the
SCAN for the data rows is on the left side while the SCAN for the
delete rows is on the right side of the JOIN. The difference is the
virtual columns and the conjuncts being used.
For equality deletes the data sequence number of a delete file has to
be greater than the data sequence number of the data file being
investigated. This information is added as a virtual column to the
scans and a conjunct is created in the JOIN node to check the relation.
The equality delete fields from the delete files are checked agains the
respective columns of the data SCANS.
This patch makes it possible for Impala to read Iceberg tables with
basic equality delete files. The Iceberg spec gives great flexibility
for engines for writing equality deletes, however in practice Flink,
one of the engines that write EQ-deletes supports only a subset of the
use cases. This patch focuses on reading the EQ-deletes written by
Flink.
The restrictions are the following:
- All equality delete files in a table should have the same equality
field ID list.
- For partitioned Iceberg tables it is expected that the partition
values are also written into the equality delete files.
- Tables with equality deletes shouldn't have partition or schema
evolution.
- Floating point equality columns aren't supported.
- If a malformed equality delete file doesn't have some of the equality
field IDs then Parquet reader will fill those missing fields with
NULLs. As a side effect this will drop the rows from the result where
the corresponding data columns have a null value.
See IMPALA-11388 epic Jira for more details.
Testing:
- Checked if the existing functional_parquet.iceberg_v2_delete_equality
table can be read successfully.
- Added new test tables so that E2E tests can validate correctness.
Change-Id: I2053e6f321c69f1c82059a84a5d99aeaa9814cad
Reviewed-on: http://gerrit.cloudera.org:8080/20753
Reviewed-by: Impala Public Jenkins <impala-public-jenk...@cloudera.com>
Tested-by: Impala Public Jenkins <impala-public-jenk...@cloudera.com>
> Basic equality delete support
> -----------------------------
>
> Key: IMPALA-12597
> URL: https://issues.apache.org/jira/browse/IMPALA-12597
> Project: IMPALA
> Issue Type: Sub-task
> Components: Backend, Frontend
> Reporter: Gabor Kaszab
> Assignee: Gabor Kaszab
> Priority: Major
> Labels: impala-iceberg
>
> To split up the Equality-delete read support task, let's deliver a patch for
> some initial support first. The idea here is that apparently Flink (one of
> the engines that can write equality delete files) can write only a subset of
> the possible equality delete use cases that are allowed by the Iceberg spec.
> So as a first step let's deliver the functionality that is required to read
> the EQ-deletes written by Flink. The use case: when Flink writes EQ-deletes
> is for tables in upsert mode (primary key is a must in this case) in order to
> guarantee the uniqueness of the primary key fields, for each insert (that is
> in fact an upsert) Flink writes one delete file to remove the previous row
> with the given PK (even if there hasn't been any) and then writes data files
> with the new data.
> How we can narrow down the functionality to be implemented on Impala side:
> * The set of PK columns is not alterable, so we don't have to implement when
> different EQ-delete files have different equality field ID lists.
> * Flink's ALTER TABLE for Iceberg tables doesn't allow partition and schema
> evolution. We can reject queries on eq-delete tables where there was
> partition or schema evolution.
> * As eq-deletes are written to NOT NULL PK's we could omit the case where
> there are NULLs in the eq-delete file. (Update, this seemed easy to solve, so
> will be part of this patch)
> * For partitioned tables Flink requires the partition columns to be part of
> the PK. As a result each EQ-delete file will have the partition values too so
> no need to add extra logic to check if the partition spec ID and the
> partition values match between the data and delete files.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org
For additional commands, e-mail: issues-all-h...@impala.apache.org
