ji chen has uploaded a new patch set (#3). (
http://gerrit.cloudera.org:8080/23469 )
Change subject: IMPALA-14092: Support querying of paimon data table via JNI
......................................................................
IMPALA-14092: Support querying of paimon data table via JNI
This patch mainly implement the querying of paimon data table
through JNI based scanner.
Features implemented:
- support partition pruning.
- support predicate pushdown.
- support column pruning.
We implemented this by treating the paimon table as normal
unpartitioned table. When querying paimon table:
- We pushdown available filter expressions to paimon native
predicates to paimon to decide paimon splits need to be scanned,
and then transfer splits to BE do the jni-based scan operation.
- We also collect the required columns that need to be scanned,
and pass the columns to BE for column pruning.This is implemented
by passing the field ids of the columns to BE, instead of column
position to support schema evolution.
Current supported impala data types for query includes:
- BOOLEAN
- TINYINT
- SMALLINT
- INTEGER
- BIGINT
- FLOAT
- DOUBLE
- STRING
- DECIMAL(P,S)
- TIMESTAMP
- CHAR(N)
- VARCHAR(N)
- BINARY
- DATE
In addition, to avoid potential risk of un-supported operation against
paimon table,this patch also prohibit all unsupported DML/DDL
operations againt paimon table.
TODO:
- Patches pending submission:
- Support tpcds/tpch data-loading
for paimon data table.
- Virtual Column query support for querying
paimon data table.
- Query support with time travel.
- Query support for paimon meta tables.
- WIP:
- Snapshot incremental read.
- Complex type query support.
- Native paimon table scanner, instead of
jni based.
Testing:
- Create table tests in functional_schema_template.sql
- Add TestPaimonScannerWithLimit in test_scanners.py
- Add unit test for AnalyzeDDLTest.java.
- Add unit test for AnalyzerTest.java.
- Add test_paimon_negative and test_paimon_query in test_paimon.py.
- Already passed the tpcds/tpch test for paimon table, due to the
testing table data is currently generated by spark, and it is
not supported by impala now, we have to do this since hive
doesn't support generating paimon table for dynamic-partitioned
tables. we plan to submit a separate patch for tpcds/tpch data
loading and associated tpcds/tpch query tests.
Change-Id: I0ecab952622e99109927056461d3d5d12f4b295f
---
M be/CMakeLists.txt
M be/src/exec/CMakeLists.txt
M be/src/exec/exec-node.cc
A be/src/exec/paimon/CMakeLists.txt
A be/src/exec/paimon/paimon-jni-row-reader.cc
A be/src/exec/paimon/paimon-jni-row-reader.h
A be/src/exec/paimon/paimon-jni-scan-node.cc
A be/src/exec/paimon/paimon-jni-scan-node.h
A be/src/exec/paimon/paimon-jni-scanner.cc
A be/src/exec/paimon/paimon-jni-scanner.h
A be/src/exec/paimon/paimon-scan-plan-node.cc
A be/src/exec/paimon/paimon-scan-plan-node.h
M be/src/runtime/descriptors.cc
M be/src/runtime/descriptors.h
M be/src/scheduling/scheduler.cc
M be/src/service/impalad-main.cc
M common/thrift/CatalogObjects.thrift
M common/thrift/PlanNodes.thrift
M fe/src/main/java/org/apache/impala/analysis/AlterTableOrViewRenameStmt.java
M fe/src/main/java/org/apache/impala/analysis/AlterTableStmt.java
M fe/src/main/java/org/apache/impala/analysis/ComputeStatsStmt.java
M fe/src/main/java/org/apache/impala/analysis/CreateTableLikeFileStmt.java
M fe/src/main/java/org/apache/impala/analysis/CreateTableLikeStmt.java
M fe/src/main/java/org/apache/impala/analysis/DropStatsStmt.java
M fe/src/main/java/org/apache/impala/analysis/InsertStmt.java
M fe/src/main/java/org/apache/impala/analysis/LikePredicate.java
M fe/src/main/java/org/apache/impala/analysis/paimon/PaimonAnalyzer.java
M fe/src/main/java/org/apache/impala/catalog/Column.java
M fe/src/main/java/org/apache/impala/catalog/Table.java
M fe/src/main/java/org/apache/impala/catalog/local/LocalPaimonTable.java
M fe/src/main/java/org/apache/impala/catalog/local/LocalTable.java
M fe/src/main/java/org/apache/impala/catalog/paimon/FePaimonTable.java
A fe/src/main/java/org/apache/impala/catalog/paimon/PaimonColumn.java
A fe/src/main/java/org/apache/impala/catalog/paimon/PaimonHiveTypeUtils.java
R fe/src/main/java/org/apache/impala/catalog/paimon/PaimonImpalaTypeUtils.java
M fe/src/main/java/org/apache/impala/catalog/paimon/PaimonTable.java
M fe/src/main/java/org/apache/impala/catalog/paimon/PaimonUtil.java
A fe/src/main/java/org/apache/impala/planner/PaimonScanNode.java
A fe/src/main/java/org/apache/impala/planner/PaimonScanPlanner.java
M fe/src/main/java/org/apache/impala/planner/SingleNodePlanner.java
A fe/src/main/java/org/apache/impala/planner/paimon/ImpalaExprConverter.java
A
fe/src/main/java/org/apache/impala/planner/paimon/PaimonPartitionPredicateVisitor.java
A fe/src/main/java/org/apache/impala/planner/paimon/PaimonSplit.java
A fe/src/main/java/org/apache/impala/planner/paimon/PredicateExtractor.java
A fe/src/main/java/org/apache/impala/util/paimon/PaimonInspectorUtil.java
A fe/src/main/java/org/apache/impala/util/paimon/PaimonJniScanner.java
M fe/src/test/java/org/apache/impala/analysis/AnalyzeDDLTest.java
M fe/src/test/java/org/apache/impala/analysis/AnalyzerTest.java
M fe/src/test/java/org/apache/impala/catalog/paimon/ImpalaTypeUtilsTest.java
A
testdata/data/paimon_test/paimon_catalog/warehouse/functional.db/alltypes_paimon/bucket-0/data-275038ac-cf80-4caa-b9c3-43854884f257-0.parquet
A
testdata/data/paimon_test/paimon_catalog/warehouse/functional.db/alltypes_paimon/bucket-0/data-4724d1a4-637d-4275-8439-d4edd1a4583e-0.parquet
A
testdata/data/paimon_test/paimon_catalog/warehouse/functional.db/alltypes_paimon/bucket-0/data-7201ded8-6883-42ef-a39f-a99fc8a83938-0.parquet
A
testdata/data/paimon_test/paimon_catalog/warehouse/functional.db/alltypes_paimon/bucket-0/data-7e80aa3a-5863-4cb5-bdb4-84d0d9002dd0-0.parquet
A
testdata/data/paimon_test/paimon_catalog/warehouse/functional.db/alltypes_paimon/bucket-0/data-81c20e92-de0c-4ea1-bf5e-989ecb844645-0.parquet
A
testdata/data/paimon_test/paimon_catalog/warehouse/functional.db/alltypes_paimon/bucket-0/data-9a6dbc25-fd98-471d-adca-02aa6c8f2751-0.parquet
A
testdata/data/paimon_test/paimon_catalog/warehouse/functional.db/alltypes_paimon/bucket-0/data-9b24cb51-8c3a-448b-a0f3-605ed71e7caa-0.parquet
A
testdata/data/paimon_test/paimon_catalog/warehouse/functional.db/alltypes_paimon/bucket-0/data-c079a770-9eb6-4b91-b3c8-f6c84fbaf7ce-0.parquet
A
testdata/data/paimon_test/paimon_catalog/warehouse/functional.db/alltypes_paimon/bucket-0/data-d943b655-eed4-4ee0-9a2e-024ff4596832-0.parquet
A
testdata/data/paimon_test/paimon_catalog/warehouse/functional.db/alltypes_paimon/bucket-0/data-dffe00a3-de35-4a74-96df-cd0fb7d08070-0.parquet
A
testdata/data/paimon_test/paimon_catalog/warehouse/functional.db/alltypes_paimon/bucket-0/data-e6940374-4033-48c7-b6d9-ebca0e7d0a4d-0.parquet
A
testdata/data/paimon_test/paimon_catalog/warehouse/functional.db/alltypes_paimon/bucket-0/data-eef44c7a-9856-4817-8f1f-a2a7dbc331c5-0.parquet
A
testdata/data/paimon_test/paimon_catalog/warehouse/functional.db/alltypes_paimon/manifest/manifest-fab63fbc-acd6-424a-ac4e-2a848ea51f52-0
A
testdata/data/paimon_test/paimon_catalog/warehouse/functional.db/alltypes_paimon/manifest/manifest-list-fa3d4f50-7f45-4d37-b5e7-22e6cf3f2cb0-0
A
testdata/data/paimon_test/paimon_catalog/warehouse/functional.db/alltypes_paimon/manifest/manifest-list-fa3d4f50-7f45-4d37-b5e7-22e6cf3f2cb0-1
A
testdata/data/paimon_test/paimon_catalog/warehouse/functional.db/alltypes_paimon/schema/schema-0
A
testdata/data/paimon_test/paimon_catalog/warehouse/functional.db/alltypes_paimon/snapshot/EARLIEST
A
testdata/data/paimon_test/paimon_catalog/warehouse/functional.db/alltypes_paimon/snapshot/LATEST
A
testdata/data/paimon_test/paimon_catalog/warehouse/functional.db/alltypes_paimon/snapshot/snapshot-1
A
testdata/data/paimon_test/paimon_catalog/warehouse/functional.db/alltypes_structs_paimon/bucket-0/data-04a2d395-e55d-4720-bdf9-d0aea3177a01-0.parquet
A
testdata/data/paimon_test/paimon_catalog/warehouse/functional.db/alltypes_structs_paimon/bucket-0/data-0ff3cbcd-bd56-43ef-8b00-543a83cc5e6f-0.parquet
A
testdata/data/paimon_test/paimon_catalog/warehouse/functional.db/alltypes_structs_paimon/bucket-0/data-15fe0349-ce6f-4228-9031-f1d556dfc808-0.parquet
A
testdata/data/paimon_test/paimon_catalog/warehouse/functional.db/alltypes_structs_paimon/bucket-0/data-43259d5c-33d4-4fa2-bed0-40e8966ed9e0-0.parquet
A
testdata/data/paimon_test/paimon_catalog/warehouse/functional.db/alltypes_structs_paimon/bucket-0/data-4e2d864e-0aa9-4b5e-8b6d-4261540072f3-0.parquet
A
testdata/data/paimon_test/paimon_catalog/warehouse/functional.db/alltypes_structs_paimon/bucket-0/data-602c116d-3e51-40b6-914d-950170ece098-0.parquet
A
testdata/data/paimon_test/paimon_catalog/warehouse/functional.db/alltypes_structs_paimon/bucket-0/data-6a5bf48f-e00f-45dc-8d44-62582cf6d881-0.parquet
A
testdata/data/paimon_test/paimon_catalog/warehouse/functional.db/alltypes_structs_paimon/bucket-0/data-72d3fa1b-ff44-4088-a2d6-22902d35e795-0.parquet
A
testdata/data/paimon_test/paimon_catalog/warehouse/functional.db/alltypes_structs_paimon/bucket-0/data-7f917551-249f-4a13-9ec9-43be271f7b21-0.parquet
A
testdata/data/paimon_test/paimon_catalog/warehouse/functional.db/alltypes_structs_paimon/bucket-0/data-8b7a002b-cf8d-4d17-ba96-0389d2dfb456-0.parquet
A
testdata/data/paimon_test/paimon_catalog/warehouse/functional.db/alltypes_structs_paimon/bucket-0/data-b5785d5b-c764-429f-82c5-aa76bbf15a63-0.parquet
A
testdata/data/paimon_test/paimon_catalog/warehouse/functional.db/alltypes_structs_paimon/bucket-0/data-f71174f8-719a-4ed5-84d0-244a6c26604b-0.parquet
A
testdata/data/paimon_test/paimon_catalog/warehouse/functional.db/alltypes_structs_paimon/manifest/manifest-85ea4264-6480-4517-9d44-2a5ba022a922-0
A
testdata/data/paimon_test/paimon_catalog/warehouse/functional.db/alltypes_structs_paimon/manifest/manifest-list-67f31438-ac2d-4d03-9bd9-4e1d55120c5c-0
A
testdata/data/paimon_test/paimon_catalog/warehouse/functional.db/alltypes_structs_paimon/manifest/manifest-list-67f31438-ac2d-4d03-9bd9-4e1d55120c5c-1
A
testdata/data/paimon_test/paimon_catalog/warehouse/functional.db/alltypes_structs_paimon/schema/schema-0
A
testdata/data/paimon_test/paimon_catalog/warehouse/functional.db/alltypes_structs_paimon/snapshot/EARLIEST
A
testdata/data/paimon_test/paimon_catalog/warehouse/functional.db/alltypes_structs_paimon/snapshot/LATEST
A
testdata/data/paimon_test/paimon_catalog/warehouse/functional.db/alltypes_structs_paimon/snapshot/snapshot-1
A
testdata/data/paimon_test/paimon_catalog/warehouse/functional.db/paimon_decimal_tbl/bucket-0/data-58e01102-7ce1-4410-8022-50afc44aa215-0.parquet
A
testdata/data/paimon_test/paimon_catalog/warehouse/functional.db/paimon_decimal_tbl/bucket-0/data-6d2bcd70-7bfc-4e04-8a37-52bb441bccc1-0.parquet
A
testdata/data/paimon_test/paimon_catalog/warehouse/functional.db/paimon_decimal_tbl/bucket-0/data-7f39c604-2f10-4bea-bd61-32787ba71065-0.parquet
A
testdata/data/paimon_test/paimon_catalog/warehouse/functional.db/paimon_decimal_tbl/bucket-0/data-8def1f02-46fd-4114-8c89-cbae117a1b61-0.parquet
A
testdata/data/paimon_test/paimon_catalog/warehouse/functional.db/paimon_decimal_tbl/bucket-0/data-a100aae7-ed6e-4f04-8648-be059c9e2942-0.parquet
A
testdata/data/paimon_test/paimon_catalog/warehouse/functional.db/paimon_decimal_tbl/bucket-0/data-bddf22a5-302c-4e14-bf86-dd897d868b42-0.parquet
A
testdata/data/paimon_test/paimon_catalog/warehouse/functional.db/paimon_decimal_tbl/bucket-0/data-cf7233f7-0e36-49e0-bd10-7b1f94ca1408-0.parquet
A
testdata/data/paimon_test/paimon_catalog/warehouse/functional.db/paimon_decimal_tbl/bucket-0/data-ddc2526d-a35f-4c0f-91e8-def352b6432e-0.parquet
A
testdata/data/paimon_test/paimon_catalog/warehouse/functional.db/paimon_decimal_tbl/manifest/manifest-1e64bd23-89a3-40ab-8f11-b11467e340fe-0
A
testdata/data/paimon_test/paimon_catalog/warehouse/functional.db/paimon_decimal_tbl/manifest/manifest-8c8eef70-e1cc-4019-bd2b-1df0c773321a-0
A
testdata/data/paimon_test/paimon_catalog/warehouse/functional.db/paimon_decimal_tbl/manifest/manifest-list-b15e96d1-cea5-44c2-a5b4-dc8c7041463f-0
A
testdata/data/paimon_test/paimon_catalog/warehouse/functional.db/paimon_decimal_tbl/manifest/manifest-list-b15e96d1-cea5-44c2-a5b4-dc8c7041463f-1
A
testdata/data/paimon_test/paimon_catalog/warehouse/functional.db/paimon_decimal_tbl/manifest/manifest-list-b9930407-daa0-43cc-b4fa-107aef9a5ffc-0
A
testdata/data/paimon_test/paimon_catalog/warehouse/functional.db/paimon_decimal_tbl/manifest/manifest-list-b9930407-daa0-43cc-b4fa-107aef9a5ffc-1
A
testdata/data/paimon_test/paimon_catalog/warehouse/functional.db/paimon_decimal_tbl/schema/schema-0
A
testdata/data/paimon_test/paimon_catalog/warehouse/functional.db/paimon_decimal_tbl/snapshot/EARLIEST
A
testdata/data/paimon_test/paimon_catalog/warehouse/functional.db/paimon_decimal_tbl/snapshot/LATEST
A
testdata/data/paimon_test/paimon_catalog/warehouse/functional.db/paimon_decimal_tbl/snapshot/snapshot-1
A
testdata/data/paimon_test/paimon_catalog/warehouse/functional.db/paimon_decimal_tbl/snapshot/snapshot-2
A
testdata/data/paimon_test/paimon_catalog/warehouse/functional.db/paimon_primitive_alltypes/bucket-0/data-3728bff4-abe9-4ab9-8837-081b21a80e70-0.parquet
A
testdata/data/paimon_test/paimon_catalog/warehouse/functional.db/paimon_primitive_alltypes/bucket-0/data-493438a3-8481-4c98-8125-856ff5b42a12-0.parquet
A
testdata/data/paimon_test/paimon_catalog/warehouse/functional.db/paimon_primitive_alltypes/bucket-0/data-59e8fabf-ccba-47d6-a88a-0571d4e0e7a0-0.parquet
A
testdata/data/paimon_test/paimon_catalog/warehouse/functional.db/paimon_primitive_alltypes/bucket-0/data-6db45077-3cc7-46cf-b0c8-f7b5f42c19f7-0.parquet
A
testdata/data/paimon_test/paimon_catalog/warehouse/functional.db/paimon_primitive_alltypes/bucket-0/data-a73a0979-5cb2-4876-b7eb-1a8d8dcd90f7-0.parquet
A
testdata/data/paimon_test/paimon_catalog/warehouse/functional.db/paimon_primitive_alltypes/bucket-0/data-af00a36c-e45c-48c3-8004-9055399d7147-0.parquet
A
testdata/data/paimon_test/paimon_catalog/warehouse/functional.db/paimon_primitive_alltypes/bucket-0/data-b9ea2c17-a695-4a31-98da-ac1da8ca242f-0.parquet
A
testdata/data/paimon_test/paimon_catalog/warehouse/functional.db/paimon_primitive_alltypes/manifest/manifest-32d94d25-0a6f-499e-9953-e4d2dd8de146-0
A
testdata/data/paimon_test/paimon_catalog/warehouse/functional.db/paimon_primitive_alltypes/manifest/manifest-65f28661-1546-4a1a-930e-f31118e55bd9-0
A
testdata/data/paimon_test/paimon_catalog/warehouse/functional.db/paimon_primitive_alltypes/manifest/manifest-list-5db2c8c6-e253-4fba-b0c2-71dc19d66678-0
A
testdata/data/paimon_test/paimon_catalog/warehouse/functional.db/paimon_primitive_alltypes/manifest/manifest-list-5db2c8c6-e253-4fba-b0c2-71dc19d66678-1
A
testdata/data/paimon_test/paimon_catalog/warehouse/functional.db/paimon_primitive_alltypes/manifest/manifest-list-c6343add-3b4b-4a00-a470-de159bd83b3d-0
A
testdata/data/paimon_test/paimon_catalog/warehouse/functional.db/paimon_primitive_alltypes/manifest/manifest-list-c6343add-3b4b-4a00-a470-de159bd83b3d-1
A
testdata/data/paimon_test/paimon_catalog/warehouse/functional.db/paimon_primitive_alltypes/schema/schema-0
A
testdata/data/paimon_test/paimon_catalog/warehouse/functional.db/paimon_primitive_alltypes/snapshot/EARLIEST
A
testdata/data/paimon_test/paimon_catalog/warehouse/functional.db/paimon_primitive_alltypes/snapshot/LATEST
A
testdata/data/paimon_test/paimon_catalog/warehouse/functional.db/paimon_primitive_alltypes/snapshot/snapshot-1
A
testdata/data/paimon_test/paimon_catalog/warehouse/functional.db/paimon_primitive_alltypes/snapshot/snapshot-2
M testdata/datasets/functional/functional_schema_template.sql
M testdata/datasets/functional/schema_constraints.csv
A testdata/workloads/functional-query/queries/QueryTest/paimon-negative.test
A testdata/workloads/functional-query/queries/QueryTest/paimon-query.test
M tests/query_test/test_paimon.py
M tests/query_test/test_scanners.py
130 files changed, 4,787 insertions(+), 80 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/69/23469/3
--
To view, visit http://gerrit.cloudera.org:8080/23469
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: I0ecab952622e99109927056461d3d5d12f4b295f
Gerrit-Change-Number: 23469
Gerrit-PatchSet: 3
Gerrit-Owner: ji chen <[email protected]>
Gerrit-Reviewer: Impala Public Jenkins <[email protected]>
