Tamas Mate has uploaded a new patch set (#3). ( http://gerrit.cloudera.org:8080/20010 )
Change subject: IMPALA-11996: Scanner change for Iceberg metadtata querying ...................................................................... IMPALA-11996: Scanner change for Iceberg metadtata querying This commit adds a scan node for querying Iceberg metadata tables. The method resembles to HBase table scans, the scan is executed with the help of the Iceberg API and through JNI. There is only one fragment created to query the Iceberg Metadata table which is supposed to be executed on the coordinator node that already has the metadata ready. This change will not cover struct column types these slots are se to NULL, it will be done in IMPALA-12205. Testing: - Added e2e tests for querying metadata tables - Updated planner tests Change-Id: I0e943cecd77f5ef7af7cd07e2b596f2c5b4331e7 --- M be/CMakeLists.txt M be/src/exec/CMakeLists.txt M be/src/exec/exec-node.cc A be/src/exec/iceberg-metadata/CMakeLists.txt A be/src/exec/iceberg-metadata/iceberg-metadata-scan-node.cc A be/src/exec/iceberg-metadata/iceberg-metadata-scan-node.h A be/src/exec/iceberg-metadata/iceberg-metadata-table-scanner.cc A be/src/exec/iceberg-metadata/iceberg-metadata-table-scanner.h M be/src/scheduling/scheduler.cc M be/src/service/frontend.cc M be/src/service/frontend.h M be/src/util/jni-util.cc M be/src/util/jni-util.h M common/thrift/PlanNodes.thrift M fe/src/main/java/org/apache/impala/analysis/IcebergMetadataTableRef.java M fe/src/main/java/org/apache/impala/analysis/Path.java M fe/src/main/java/org/apache/impala/catalog/iceberg/IcebergMetadataTable.java M fe/src/main/java/org/apache/impala/planner/DistributedPlanner.java M fe/src/main/java/org/apache/impala/planner/IcebergMetadataScanNode.java M fe/src/main/java/org/apache/impala/planner/IcebergScanPlanner.java M fe/src/main/java/org/apache/impala/planner/SingleNodePlanner.java M fe/src/main/java/org/apache/impala/service/Frontend.java M fe/src/main/java/org/apache/impala/service/JniFrontend.java A testdata/data/iceberg_test/iceberg_test_metadata/data/3d481ed88b2941f0-ea33816200000000_1109948289_data.0.parq A testdata/data/iceberg_test/iceberg_test_metadata/data/944a2355e618932f-18f086b600000000_1283312202_data.0.parq A testdata/data/iceberg_test/iceberg_test_metadata/data/d64ba652b36f2d76-d075207600000000_883987622_data.0.parq A testdata/data/iceberg_test/iceberg_test_metadata/data/delete-1f43b217940cc094-fedf515600000000_248998721_data.0.parq A testdata/data/iceberg_test/iceberg_test_metadata/data/delete-bc4b163507d1da6f-a691cf6100000002_511900219_data.0.parq A testdata/data/iceberg_test/iceberg_test_metadata/metadata/38e5a1bd-5b7f-4eae-9362-16a2de3c575d-m0.avro A testdata/data/iceberg_test/iceberg_test_metadata/metadata/7c952273-c5d6-43d2-bb1b-72fbe1555f25-m0.avro A testdata/data/iceberg_test/iceberg_test_metadata/metadata/88a39285-529f-41a4-bd69-6d2560fac64e-m0.avro A testdata/data/iceberg_test/iceberg_test_metadata/metadata/ef78f8eb-b2b5-41df-a2a0-7944ac3c38da-m0.avro A testdata/data/iceberg_test/iceberg_test_metadata/metadata/snap-7858675898458780516-1-ef78f8eb-b2b5-41df-a2a0-7944ac3c38da.avro A testdata/data/iceberg_test/iceberg_test_metadata/metadata/snap-8283026816932323050-1-38e5a1bd-5b7f-4eae-9362-16a2de3c575d.avro A testdata/data/iceberg_test/iceberg_test_metadata/metadata/snap-8491702501245661704-1-88a39285-529f-41a4-bd69-6d2560fac64e.avro A testdata/data/iceberg_test/iceberg_test_metadata/metadata/snap-9046920472784493998-1-7c952273-c5d6-43d2-bb1b-72fbe1555f25.avro A testdata/data/iceberg_test/iceberg_test_metadata/metadata/v1.metadata.json A testdata/data/iceberg_test/iceberg_test_metadata/metadata/v2.metadata.json A testdata/data/iceberg_test/iceberg_test_metadata/metadata/v3.metadata.json A testdata/data/iceberg_test/iceberg_test_metadata/metadata/v4.metadata.json A testdata/data/iceberg_test/iceberg_test_metadata/metadata/v5.metadata.json A testdata/data/iceberg_test/iceberg_test_metadata/metadata/version-hint.txt M testdata/datasets/functional/functional_schema_template.sql M testdata/datasets/functional/schema_constraints.csv M testdata/workloads/functional-planner/queries/PlannerTest/iceberg-metadata-table-scan.test M testdata/workloads/functional-query/queries/QueryTest/iceberg-metadata-tables.test 46 files changed, 1,762 insertions(+), 156 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/10/20010/3 -- To view, visit http://gerrit.cloudera.org:8080/20010 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: I0e943cecd77f5ef7af7cd07e2b596f2c5b4331e7 Gerrit-Change-Number: 20010 Gerrit-PatchSet: 3 Gerrit-Owner: Tamas Mate <tma...@apache.org> Gerrit-Reviewer: Anonymous Coward <lipeng...@apache.org> Gerrit-Reviewer: Gabor Kaszab <gaborkas...@cloudera.com> Gerrit-Reviewer: Gergely Fürnstáhl <g.furnst...@gmail.com> Gerrit-Reviewer: Impala Public Jenkins <impala-public-jenk...@cloudera.com> Gerrit-Reviewer: Tamas Mate <tma...@apache.org> Gerrit-Reviewer: Zoltan Borok-Nagy <borokna...@cloudera.com>
