Hello Gabor Kaszab, wangsheng, Csaba Ringhofer, Impala Public Jenkins,
I'd like you to reexamine a change. Please visit
http://gerrit.cloudera.org:8080/16939
to look at the new patch set (#4).
Change subject: IMPALA-10432: INSERT INTO Iceberg tables with partition
transforms
......................................................................
IMPALA-10432: INSERT INTO Iceberg tables with partition transforms
INSERT INTO Iceberg tables that use partition transforms. Partition
transforms are functions that calculate partition data from row data.
There are the following partition transforms in Iceberg:
https://iceberg.apache.org/spec/#partition-transforms
* IDENTITY
* BUCKET
* TRUNCATE
* YEAR
* MONTH
* DAY
* HOUR
INSERT INTO identity-partitioned Iceberg tables are already supported.
This patch adds support for the rest of the transforms.
We create the partitioning expressions in InsertStmt. Based on these
expressions data are automatically shuffled and sorted by the backend
executors before rows are given to the table sink operators. The table
sink operator writes the partitions one-by-one and creates a
human-readable partition path for them.
In the end, we will convert the partition path to partition data and
create Iceberg DataFiles with information about the files written.
Testing:
* added planner test
* added e2e tests
Change-Id: I3edf02048cea78703837b248c55219c22d512b78
---
M fe/src/main/java/org/apache/impala/analysis/IcebergPartitionField.java
M fe/src/main/java/org/apache/impala/analysis/IcebergPartitionTransform.java
M fe/src/main/java/org/apache/impala/analysis/InsertStmt.java
M fe/src/main/java/org/apache/impala/planner/IcebergScanNode.java
M fe/src/main/java/org/apache/impala/service/IcebergCatalogOpExecutor.java
M fe/src/main/java/org/apache/impala/util/IcebergUtil.java
M testdata/datasets/functional/functional_schema_template.sql
M testdata/datasets/functional/schema_constraints.csv
M testdata/workloads/functional-planner/queries/PlannerTest/insert.test
M testdata/workloads/functional-query/queries/QueryTest/iceberg-negative.test
A
testdata/workloads/functional-query/queries/QueryTest/iceberg-partition-transform-insert.test
M tests/query_test/test_iceberg.py
12 files changed, 1,167 insertions(+), 43 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/39/16939/4
--
To view, visit http://gerrit.cloudera.org:8080/16939
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: I3edf02048cea78703837b248c55219c22d512b78
Gerrit-Change-Number: 16939
Gerrit-PatchSet: 4
Gerrit-Owner: Zoltan Borok-Nagy <borokna...@cloudera.com>
Gerrit-Reviewer: Csaba Ringhofer <csringho...@cloudera.com>
Gerrit-Reviewer: Gabor Kaszab <gaborkas...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <impala-public-jenk...@cloudera.com>
Gerrit-Reviewer: Zoltan Borok-Nagy <borokna...@cloudera.com>
Gerrit-Reviewer: wangsheng <sky...@163.com>
