ji chen has uploaded a new patch set (#2). (
http://gerrit.cloudera.org:8080/23469 )
Change subject: IMPALA-14092: Support querying of paimon data table via JNI.
......................................................................
IMPALA-14092: Support querying of paimon data table via JNI.
This patch mainly implement the querying of paimon data table
through JNI based scanner.
Features implemented:
- support partition pruning.
- support predicate pushdown.
- support column pruning.
We implemented this by treating the paimon table as normal
unpartitioned table. When querying paimon table:
- We pushdown available filter expressions to paimon native
predicates to paimon to decide paimon splits need to be scanned,
and then transfer splits to BE do the jni-based scan operation.
- We also collect the required columns that need to be scanned,
and pass the columns to BE for column pruning.This is implemented
by passing the field ids of the columns to BE, instead of column
position to support schema evolution.
Current supported impala data types for query includes:
- BOOLEAN
- TINYINT
- SMALLINT
- INTEGER
- BIGINT
- FLOAT
- DOUBLE
- STRING
- DECIMAL(P,S)
- TIMESTAMP
- CHAR(N)
- VARCHAR(N)
- BINARY
- DATE
For the table column, the datatype of which is unsupported,
the query result for the column will be NULL.
In addition, to avoid potential risk of un-supported operation againt paimon
table,this patch also prohibit all unsupported DML/DDL operation againt
paimon table.
TODO:
- Patches pending submission:
- Support spark based tpcds/tpch data-loading
for paimon data table.
- Virtual Column query support for querying
paimon data table.
- Query support with time travel.
- Query support for paimon meta tables.
- WIP:
- Complex type query support.
- Native paimon table scanner, instead of
jni based.
Testing:
- Create table tests in functional_schema_template.sql
- Add TestPaimonScannerWithLimit in test_scanners.py
- Add unit test for AnalyzeDDLTest.java.
- Add unit test for AnalyzerTest.java.
- Add test_paimon_negative and test_paimon_query in test_paimon.py.
- Already passed the tpcds/tpch test for paimon table, due to the
testing table data is currently generated by spark, hive has issue
while generating data for dynamic-partitioned tables.
will submit a separate patch to support using spark for tpcds/tpch
data-loading.
Change-Id: I0ecab952622e99109927056461d3d5d12f4b295f
---
M be/CMakeLists.txt
M be/src/exec/CMakeLists.txt
M be/src/exec/exec-node.cc
A be/src/exec/paimon/CMakeLists.txt
A be/src/exec/paimon/paimon-jni-row-reader.cc
A be/src/exec/paimon/paimon-jni-row-reader.h
A be/src/exec/paimon/paimon-jni-scan-node.cc
A be/src/exec/paimon/paimon-jni-scan-node.h
A be/src/exec/paimon/paimon-jni-scanner.cc
A be/src/exec/paimon/paimon-jni-scanner.h
A be/src/exec/paimon/paimon-scan-plan-node.cc
A be/src/exec/paimon/paimon-scan-plan-node.h
M be/src/runtime/descriptors.cc
M be/src/runtime/descriptors.h
M be/src/scheduling/scheduler.cc
M be/src/service/impalad-main.cc
M common/thrift/CatalogObjects.thrift
M common/thrift/PlanNodes.thrift
M fe/src/main/java/org/apache/impala/analysis/AlterTableOrViewRenameStmt.java
M fe/src/main/java/org/apache/impala/analysis/AlterTableStmt.java
M fe/src/main/java/org/apache/impala/analysis/ComputeStatsStmt.java
M fe/src/main/java/org/apache/impala/analysis/CreateTableLikeFileStmt.java
M fe/src/main/java/org/apache/impala/analysis/CreateTableLikeStmt.java
M fe/src/main/java/org/apache/impala/analysis/DropStatsStmt.java
M fe/src/main/java/org/apache/impala/analysis/InsertStmt.java
M fe/src/main/java/org/apache/impala/analysis/LikePredicate.java
M fe/src/main/java/org/apache/impala/analysis/paimon/PaimonAnalyzer.java
M fe/src/main/java/org/apache/impala/catalog/Column.java
M fe/src/main/java/org/apache/impala/catalog/Table.java
M fe/src/main/java/org/apache/impala/catalog/local/LocalPaimonTable.java
M fe/src/main/java/org/apache/impala/catalog/local/LocalTable.java
M fe/src/main/java/org/apache/impala/catalog/paimon/FePaimonTable.java
A fe/src/main/java/org/apache/impala/catalog/paimon/PaimonColumn.java
A fe/src/main/java/org/apache/impala/catalog/paimon/PaimonHiveTypeUtils.java
R fe/src/main/java/org/apache/impala/catalog/paimon/PaimonImpalaTypeUtils.java
M fe/src/main/java/org/apache/impala/catalog/paimon/PaimonTable.java
M fe/src/main/java/org/apache/impala/catalog/paimon/PaimonUtil.java
A fe/src/main/java/org/apache/impala/planner/PaimonScanNode.java
A fe/src/main/java/org/apache/impala/planner/PaimonScanPlanner.java
M fe/src/main/java/org/apache/impala/planner/SingleNodePlanner.java
A fe/src/main/java/org/apache/impala/planner/paimon/ImpalaExprConverter.java
A
fe/src/main/java/org/apache/impala/planner/paimon/PaimonPartitionPredicateVisitor.java
A fe/src/main/java/org/apache/impala/planner/paimon/PaimonSplit.java
A fe/src/main/java/org/apache/impala/planner/paimon/PredicateExtractor.java
A fe/src/main/java/org/apache/impala/util/paimon/PaimonInspectorUtil.java
A fe/src/main/java/org/apache/impala/util/paimon/PaimonJniScanner.java
M fe/src/test/java/org/apache/impala/analysis/AnalyzeDDLTest.java
M fe/src/test/java/org/apache/impala/analysis/AnalyzerTest.java
M fe/src/test/java/org/apache/impala/catalog/paimon/ImpalaTypeUtilsTest.java
A
testdata/data/paimon_test/paimon_catalog/warehouse/functional.db/alltypes_paimon/bucket-0/data-275038ac-cf80-4caa-b9c3-43854884f257-0.parquet
A
testdata/data/paimon_test/paimon_catalog/warehouse/functional.db/alltypes_paimon/bucket-0/data-4724d1a4-637d-4275-8439-d4edd1a4583e-0.parquet
A
testdata/data/paimon_test/paimon_catalog/warehouse/functional.db/alltypes_paimon/bucket-0/data-7201ded8-6883-42ef-a39f-a99fc8a83938-0.parquet
A
testdata/data/paimon_test/paimon_catalog/warehouse/functional.db/alltypes_paimon/bucket-0/data-7e80aa3a-5863-4cb5-bdb4-84d0d9002dd0-0.parquet
A
testdata/data/paimon_test/paimon_catalog/warehouse/functional.db/alltypes_paimon/bucket-0/data-81c20e92-de0c-4ea1-bf5e-989ecb844645-0.parquet
A
testdata/data/paimon_test/paimon_catalog/warehouse/functional.db/alltypes_paimon/bucket-0/data-9a6dbc25-fd98-471d-adca-02aa6c8f2751-0.parquet
A
testdata/data/paimon_test/paimon_catalog/warehouse/functional.db/alltypes_paimon/bucket-0/data-9b24cb51-8c3a-448b-a0f3-605ed71e7caa-0.parquet
A
testdata/data/paimon_test/paimon_catalog/warehouse/functional.db/alltypes_paimon/bucket-0/data-c079a770-9eb6-4b91-b3c8-f6c84fbaf7ce-0.parquet
A
testdata/data/paimon_test/paimon_catalog/warehouse/functional.db/alltypes_paimon/bucket-0/data-d943b655-eed4-4ee0-9a2e-024ff4596832-0.parquet
A
testdata/data/paimon_test/paimon_catalog/warehouse/functional.db/alltypes_paimon/bucket-0/data-dffe00a3-de35-4a74-96df-cd0fb7d08070-0.parquet
A
testdata/data/paimon_test/paimon_catalog/warehouse/functional.db/alltypes_paimon/bucket-0/data-e6940374-4033-48c7-b6d9-ebca0e7d0a4d-0.parquet
A
testdata/data/paimon_test/paimon_catalog/warehouse/functional.db/alltypes_paimon/bucket-0/data-eef44c7a-9856-4817-8f1f-a2a7dbc331c5-0.parquet
A
testdata/data/paimon_test/paimon_catalog/warehouse/functional.db/alltypes_paimon/manifest/manifest-fab63fbc-acd6-424a-ac4e-2a848ea51f52-0
A
testdata/data/paimon_test/paimon_catalog/warehouse/functional.db/alltypes_paimon/manifest/manifest-list-fa3d4f50-7f45-4d37-b5e7-22e6cf3f2cb0-0
A
testdata/data/paimon_test/paimon_catalog/warehouse/functional.db/alltypes_paimon/manifest/manifest-list-fa3d4f50-7f45-4d37-b5e7-22e6cf3f2cb0-1
A
testdata/data/paimon_test/paimon_catalog/warehouse/functional.db/alltypes_paimon/schema/schema-0
A
testdata/data/paimon_test/paimon_catalog/warehouse/functional.db/alltypes_paimon/snapshot/EARLIEST
A
testdata/data/paimon_test/paimon_catalog/warehouse/functional.db/alltypes_paimon/snapshot/LATEST
A
testdata/data/paimon_test/paimon_catalog/warehouse/functional.db/alltypes_paimon/snapshot/snapshot-1
A
testdata/data/paimon_test/paimon_catalog/warehouse/functional.db/alltypes_structs_paimon/bucket-0/data-04a2d395-e55d-4720-bdf9-d0aea3177a01-0.parquet
A
testdata/data/paimon_test/paimon_catalog/warehouse/functional.db/alltypes_structs_paimon/bucket-0/data-0ff3cbcd-bd56-43ef-8b00-543a83cc5e6f-0.parquet
A
testdata/data/paimon_test/paimon_catalog/warehouse/functional.db/alltypes_structs_paimon/bucket-0/data-15fe0349-ce6f-4228-9031-f1d556dfc808-0.parquet
A
testdata/data/paimon_test/paimon_catalog/warehouse/functional.db/alltypes_structs_paimon/bucket-0/data-43259d5c-33d4-4fa2-bed0-40e8966ed9e0-0.parquet
A
testdata/data/paimon_test/paimon_catalog/warehouse/functional.db/alltypes_structs_paimon/bucket-0/data-4e2d864e-0aa9-4b5e-8b6d-4261540072f3-0.parquet
A
testdata/data/paimon_test/paimon_catalog/warehouse/functional.db/alltypes_structs_paimon/bucket-0/data-602c116d-3e51-40b6-914d-950170ece098-0.parquet
A
testdata/data/paimon_test/paimon_catalog/warehouse/functional.db/alltypes_structs_paimon/bucket-0/data-6a5bf48f-e00f-45dc-8d44-62582cf6d881-0.parquet
A
testdata/data/paimon_test/paimon_catalog/warehouse/functional.db/alltypes_structs_paimon/bucket-0/data-72d3fa1b-ff44-4088-a2d6-22902d35e795-0.parquet
A
testdata/data/paimon_test/paimon_catalog/warehouse/functional.db/alltypes_structs_paimon/bucket-0/data-7f917551-249f-4a13-9ec9-43be271f7b21-0.parquet
A
testdata/data/paimon_test/paimon_catalog/warehouse/functional.db/alltypes_structs_paimon/bucket-0/data-8b7a002b-cf8d-4d17-ba96-0389d2dfb456-0.parquet
A
testdata/data/paimon_test/paimon_catalog/warehouse/functional.db/alltypes_structs_paimon/bucket-0/data-b5785d5b-c764-429f-82c5-aa76bbf15a63-0.parquet
A
testdata/data/paimon_test/paimon_catalog/warehouse/functional.db/alltypes_structs_paimon/bucket-0/data-f71174f8-719a-4ed5-84d0-244a6c26604b-0.parquet
A
testdata/data/paimon_test/paimon_catalog/warehouse/functional.db/alltypes_structs_paimon/manifest/manifest-85ea4264-6480-4517-9d44-2a5ba022a922-0
A
testdata/data/paimon_test/paimon_catalog/warehouse/functional.db/alltypes_structs_paimon/manifest/manifest-list-67f31438-ac2d-4d03-9bd9-4e1d55120c5c-0
A
testdata/data/paimon_test/paimon_catalog/warehouse/functional.db/alltypes_structs_paimon/manifest/manifest-list-67f31438-ac2d-4d03-9bd9-4e1d55120c5c-1
A
testdata/data/paimon_test/paimon_catalog/warehouse/functional.db/alltypes_structs_paimon/schema/schema-0
A
testdata/data/paimon_test/paimon_catalog/warehouse/functional.db/alltypes_structs_paimon/snapshot/EARLIEST
A
testdata/data/paimon_test/paimon_catalog/warehouse/functional.db/alltypes_structs_paimon/snapshot/LATEST
A
testdata/data/paimon_test/paimon_catalog/warehouse/functional.db/alltypes_structs_paimon/snapshot/snapshot-1
A
testdata/data/paimon_test/paimon_catalog/warehouse/functional.db/paimon_decimal_tbl/bucket-0/data-58e01102-7ce1-4410-8022-50afc44aa215-0.parquet
A
testdata/data/paimon_test/paimon_catalog/warehouse/functional.db/paimon_decimal_tbl/bucket-0/data-6d2bcd70-7bfc-4e04-8a37-52bb441bccc1-0.parquet
A
testdata/data/paimon_test/paimon_catalog/warehouse/functional.db/paimon_decimal_tbl/bucket-0/data-7f39c604-2f10-4bea-bd61-32787ba71065-0.parquet
A
testdata/data/paimon_test/paimon_catalog/warehouse/functional.db/paimon_decimal_tbl/bucket-0/data-8def1f02-46fd-4114-8c89-cbae117a1b61-0.parquet
A
testdata/data/paimon_test/paimon_catalog/warehouse/functional.db/paimon_decimal_tbl/bucket-0/data-a100aae7-ed6e-4f04-8648-be059c9e2942-0.parquet
A
testdata/data/paimon_test/paimon_catalog/warehouse/functional.db/paimon_decimal_tbl/bucket-0/data-bddf22a5-302c-4e14-bf86-dd897d868b42-0.parquet
A
testdata/data/paimon_test/paimon_catalog/warehouse/functional.db/paimon_decimal_tbl/bucket-0/data-cf7233f7-0e36-49e0-bd10-7b1f94ca1408-0.parquet
A
testdata/data/paimon_test/paimon_catalog/warehouse/functional.db/paimon_decimal_tbl/bucket-0/data-ddc2526d-a35f-4c0f-91e8-def352b6432e-0.parquet
A
testdata/data/paimon_test/paimon_catalog/warehouse/functional.db/paimon_decimal_tbl/manifest/manifest-1e64bd23-89a3-40ab-8f11-b11467e340fe-0
A
testdata/data/paimon_test/paimon_catalog/warehouse/functional.db/paimon_decimal_tbl/manifest/manifest-8c8eef70-e1cc-4019-bd2b-1df0c773321a-0
A
testdata/data/paimon_test/paimon_catalog/warehouse/functional.db/paimon_decimal_tbl/manifest/manifest-list-b15e96d1-cea5-44c2-a5b4-dc8c7041463f-0
A
testdata/data/paimon_test/paimon_catalog/warehouse/functional.db/paimon_decimal_tbl/manifest/manifest-list-b15e96d1-cea5-44c2-a5b4-dc8c7041463f-1
A
testdata/data/paimon_test/paimon_catalog/warehouse/functional.db/paimon_decimal_tbl/manifest/manifest-list-b9930407-daa0-43cc-b4fa-107aef9a5ffc-0
A
testdata/data/paimon_test/paimon_catalog/warehouse/functional.db/paimon_decimal_tbl/manifest/manifest-list-b9930407-daa0-43cc-b4fa-107aef9a5ffc-1
A
testdata/data/paimon_test/paimon_catalog/warehouse/functional.db/paimon_decimal_tbl/schema/schema-0
A
testdata/data/paimon_test/paimon_catalog/warehouse/functional.db/paimon_decimal_tbl/snapshot/EARLIEST
A
testdata/data/paimon_test/paimon_catalog/warehouse/functional.db/paimon_decimal_tbl/snapshot/LATEST
A
testdata/data/paimon_test/paimon_catalog/warehouse/functional.db/paimon_decimal_tbl/snapshot/snapshot-1
A
testdata/data/paimon_test/paimon_catalog/warehouse/functional.db/paimon_decimal_tbl/snapshot/snapshot-2
A
testdata/data/paimon_test/paimon_catalog/warehouse/functional.db/paimon_primitive_alltypes/bucket-0/data-3728bff4-abe9-4ab9-8837-081b21a80e70-0.parquet
A
testdata/data/paimon_test/paimon_catalog/warehouse/functional.db/paimon_primitive_alltypes/bucket-0/data-493438a3-8481-4c98-8125-856ff5b42a12-0.parquet
A
testdata/data/paimon_test/paimon_catalog/warehouse/functional.db/paimon_primitive_alltypes/bucket-0/data-59e8fabf-ccba-47d6-a88a-0571d4e0e7a0-0.parquet
A
testdata/data/paimon_test/paimon_catalog/warehouse/functional.db/paimon_primitive_alltypes/bucket-0/data-6db45077-3cc7-46cf-b0c8-f7b5f42c19f7-0.parquet
A
testdata/data/paimon_test/paimon_catalog/warehouse/functional.db/paimon_primitive_alltypes/bucket-0/data-a73a0979-5cb2-4876-b7eb-1a8d8dcd90f7-0.parquet
A
testdata/data/paimon_test/paimon_catalog/warehouse/functional.db/paimon_primitive_alltypes/bucket-0/data-af00a36c-e45c-48c3-8004-9055399d7147-0.parquet
A
testdata/data/paimon_test/paimon_catalog/warehouse/functional.db/paimon_primitive_alltypes/bucket-0/data-b9ea2c17-a695-4a31-98da-ac1da8ca242f-0.parquet
A
testdata/data/paimon_test/paimon_catalog/warehouse/functional.db/paimon_primitive_alltypes/manifest/manifest-32d94d25-0a6f-499e-9953-e4d2dd8de146-0
A
testdata/data/paimon_test/paimon_catalog/warehouse/functional.db/paimon_primitive_alltypes/manifest/manifest-65f28661-1546-4a1a-930e-f31118e55bd9-0
A
testdata/data/paimon_test/paimon_catalog/warehouse/functional.db/paimon_primitive_alltypes/manifest/manifest-list-5db2c8c6-e253-4fba-b0c2-71dc19d66678-0
A
testdata/data/paimon_test/paimon_catalog/warehouse/functional.db/paimon_primitive_alltypes/manifest/manifest-list-5db2c8c6-e253-4fba-b0c2-71dc19d66678-1
A
testdata/data/paimon_test/paimon_catalog/warehouse/functional.db/paimon_primitive_alltypes/manifest/manifest-list-c6343add-3b4b-4a00-a470-de159bd83b3d-0
A
testdata/data/paimon_test/paimon_catalog/warehouse/functional.db/paimon_primitive_alltypes/manifest/manifest-list-c6343add-3b4b-4a00-a470-de159bd83b3d-1
A
testdata/data/paimon_test/paimon_catalog/warehouse/functional.db/paimon_primitive_alltypes/schema/schema-0
A
testdata/data/paimon_test/paimon_catalog/warehouse/functional.db/paimon_primitive_alltypes/snapshot/EARLIEST
A
testdata/data/paimon_test/paimon_catalog/warehouse/functional.db/paimon_primitive_alltypes/snapshot/LATEST
A
testdata/data/paimon_test/paimon_catalog/warehouse/functional.db/paimon_primitive_alltypes/snapshot/snapshot-1
A
testdata/data/paimon_test/paimon_catalog/warehouse/functional.db/paimon_primitive_alltypes/snapshot/snapshot-2
M testdata/datasets/functional/functional_schema_template.sql
M testdata/datasets/functional/schema_constraints.csv
A testdata/workloads/functional-query/queries/QueryTest/paimon-negative.test
A testdata/workloads/functional-query/queries/QueryTest/paimon-query.test
M tests/query_test/test_paimon.py
M tests/query_test/test_scanners.py
130 files changed, 4,787 insertions(+), 80 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/69/23469/2
--
To view, visit http://gerrit.cloudera.org:8080/23469
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: I0ecab952622e99109927056461d3d5d12f4b295f
Gerrit-Change-Number: 23469
Gerrit-PatchSet: 2
Gerrit-Owner: ji chen <[email protected]>
Gerrit-Reviewer: Impala Public Jenkins <[email protected]>
