[Impala-ASF-CR] IMPALA-11996: Scanner change for Iceberg metadtata querying
Gabor Kaszab has posted comments on this change. (
http://gerrit.cloudera.org:8080/20010 )
Change subject: IMPALA-11996: Scanner change for Iceberg metadtata querying
..
Patch Set 5:
(64 comments)
Thanks for the patch, Tamas! Does really seem like a lot of work.
I took a first look and I think I found some mem leaks around
IcebergMetadataScanNode.
I still have to digest the code in IcebergMetadataTableScanner, though.
Honestly, for me it seems pretty ugly to have JNI call within c++ for literally
everything. I naively thought that we could somehow let the Java part do the
Java stuff and the C++ part only meant to ask for the next set of results in
some format, like thrift.
Even if that's not possible, I think we can give some subtask to the Java part,
like "please create me the object for the metadata table" and then we can hide
the majority of the java class/variable/method/type references in the c++ code.
Can't we somehow keep the Java references minimal and let's say maintain the
iterator that traverses the results, but then ask the Java part to get us the
actual results giving it the iterator? Could results be passed in thrift or
some buffer format between the 2 words? Once we got them, we could move the
values into the row_batch.
I'm curious what others think about this, though.
http://gerrit.cloudera.org:8080/#/c/20010/5//COMMIT_MSG
Commit Message:
http://gerrit.cloudera.org:8080/#/c/20010/5//COMMIT_MSG@7
PS5, Line 7: metadtata
typo
http://gerrit.cloudera.org:8080/#/c/20010/5//COMMIT_MSG@17
PS5, Line 17: se
typo
http://gerrit.cloudera.org:8080/#/c/20010/5//COMMIT_MSG@17
PS5, Line 17: struct column types
it's not just struct but nested types in general
http://gerrit.cloudera.org:8080/#/c/20010/5/be/src/exec/iceberg-metadata/iceberg-metadata-scan-node.h
File be/src/exec/iceberg-metadata/iceberg-metadata-scan-node.h:
http://gerrit.cloudera.org:8080/#/c/20010/5/be/src/exec/iceberg-metadata/iceberg-metadata-scan-node.h@20
PS5, Line 20: #include "exec/iceberg-metadata/iceberg-metadata-table-scanner.h"
Would it help to remove this include if we had a forward declaration of
IcebergMetadataTableScanner in this header file?
http://gerrit.cloudera.org:8080/#/c/20010/5/be/src/exec/iceberg-metadata/iceberg-metadata-scan-node.h@22
PS5, Line 22: #include "runtime/runtime-state.h"
: #include "util/jni-util.h"
same as above
http://gerrit.cloudera.org:8080/#/c/20010/5/be/src/exec/iceberg-metadata/iceberg-metadata-scan-node.h@41
PS5, Line 41: /// ScanNode ancestor -> ExecNode
I don't think this comment is neccessary
http://gerrit.cloudera.org:8080/#/c/20010/5/be/src/exec/iceberg-metadata/iceberg-metadata-scan-node.h@42
PS5, Line 42: class IcebergMetadataScanNode : public ScanNode {
Don't you need a virtual destructor for this class?
http://gerrit.cloudera.org:8080/#/c/20010/5/be/src/exec/iceberg-metadata/iceberg-metadata-scan-node.h@49
PS5, Line 49: Iceberg TableScan
What is an Iceberg 'TableScan'? I haven't found any reference in the cc file.
http://gerrit.cloudera.org:8080/#/c/20010/5/be/src/exec/iceberg-metadata/iceberg-metadata-scan-node.h@52
PS5, Line 52: /// Get next rowbatch from the table scanner
this comment doesn't add much
http://gerrit.cloudera.org:8080/#/c/20010/5/be/src/exec/iceberg-metadata/iceberg-metadata-scan-node.h@55
PS5, Line 55: /// Close the Iceberg TableScan
This comment doesn't add much
http://gerrit.cloudera.org:8080/#/c/20010/5/be/src/exec/iceberg-metadata/iceberg-metadata-scan-node.h@58
PS5, Line 58: Status GetCatalogTable(JNIEnv* env, jobject* jtable);
private?
http://gerrit.cloudera.org:8080/#/c/20010/5/be/src/exec/iceberg-metadata/iceberg-metadata-scan-node.h@60
PS5, Line 60: protected:
Are there any derived classes from this one? I haven't found any. What's the
reason having protected members?
http://gerrit.cloudera.org:8080/#/c/20010/5/be/src/exec/iceberg-metadata/iceberg-metadata-scan-node.h@61
PS5, Line 61: tuple_desc_
nit: I think we use ' char around variable names in comments. like 'tuple_desc_'
http://gerrit.cloudera.org:8080/#/c/20010/5/be/src/exec/iceberg-metadata/iceberg-metadata-scan-node.h@67
PS5, Line 67: metadtata
typo
http://gerrit.cloudera.org:8080/#/c/20010/5/be/src/exec/iceberg-metadata/iceberg-metadata-scan-node.h@69
PS5, Line 69: const string* metadata_table_name_;
does this have to be a pointer? isn't regular string enough?
http://gerrit.cloudera.org:8080/#/c/20010/5/be/src/exec/iceberg-metadata/iceberg-metadata-scan-node.h@73
PS5, Line 73: scoped_ptr
unique_ptr ?
http://gerrit.cloudera.org:8080/#/c/20010/5/be/src/exec/iceberg-metadata/iceberg-metadata-scan-node.cc
File be/src/exec/iceberg-metadata/iceberg-metadata-scan-node.cc:
http://gerrit.cloudera.org:8080/#/c/20010/5/be/src/exec/iceberg-metadata/iceberg-metadata-scan-node.cc@36
PS5, Line 36: table_name_(new
TTableName(pnode.tnode_->iceberg_scan_metadata_node.tabl
[Impala-ASF-CR] IMPALA-11996: Scanner change for Iceberg metadtata querying
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/20010 ) Change subject: IMPALA-11996: Scanner change for Iceberg metadtata querying .. Patch Set 5: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/13803/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/20010 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I0e943cecd77f5ef7af7cd07e2b596f2c5b4331e7 Gerrit-Change-Number: 20010 Gerrit-PatchSet: 5 Gerrit-Owner: Tamas Mate Gerrit-Reviewer: Anonymous Coward Gerrit-Reviewer: Gabor Kaszab Gerrit-Reviewer: Gergely Fürnstáhl Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tamas Mate Gerrit-Reviewer: Zoltan Borok-Nagy Gerrit-Comment-Date: Tue, 22 Aug 2023 14:43:07 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-11996: Scanner change for Iceberg metadtata querying
Tamas Mate has uploaded a new patch set (#5). ( http://gerrit.cloudera.org:8080/20010 ) Change subject: IMPALA-11996: Scanner change for Iceberg metadtata querying .. IMPALA-11996: Scanner change for Iceberg metadtata querying This commit adds a scan node for querying Iceberg metadata tables. The method resembles to HBase table scans, the scan is executed with the help of the Iceberg API and through JNI. There is only one fragment created to query the Iceberg Metadata table which is supposed to be executed on the coordinator node that already has the metadata ready. This change will not cover struct column types these slots are se to NULL, it will be done in IMPALA-12205. Testing: - Added e2e tests for querying metadata tables - Updated planner tests Change-Id: I0e943cecd77f5ef7af7cd07e2b596f2c5b4331e7 --- M be/CMakeLists.txt M be/src/exec/CMakeLists.txt M be/src/exec/exec-node.cc A be/src/exec/iceberg-metadata/CMakeLists.txt A be/src/exec/iceberg-metadata/iceberg-metadata-scan-node.cc A be/src/exec/iceberg-metadata/iceberg-metadata-scan-node.h A be/src/exec/iceberg-metadata/iceberg-metadata-table-scanner.cc A be/src/exec/iceberg-metadata/iceberg-metadata-table-scanner.h M be/src/scheduling/scheduler.cc M be/src/service/frontend.cc M be/src/service/frontend.h M be/src/util/jni-util.cc M be/src/util/jni-util.h M common/thrift/PlanNodes.thrift M fe/src/main/java/org/apache/impala/analysis/IcebergMetadataTableRef.java M fe/src/main/java/org/apache/impala/analysis/Path.java M fe/src/main/java/org/apache/impala/catalog/iceberg/IcebergMetadataTable.java M fe/src/main/java/org/apache/impala/planner/DistributedPlanner.java M fe/src/main/java/org/apache/impala/planner/IcebergMetadataScanNode.java M fe/src/main/java/org/apache/impala/planner/IcebergScanPlanner.java M fe/src/main/java/org/apache/impala/planner/SingleNodePlanner.java M fe/src/main/java/org/apache/impala/service/Frontend.java M fe/src/main/java/org/apache/impala/service/JniFrontend.java A testdata/data/iceberg_test/iceberg_test_metadata/data/3d481ed88b2941f0-ea338162_1109948289_data.0.parq A testdata/data/iceberg_test/iceberg_test_metadata/data/944a2355e618932f-18f086b6_1283312202_data.0.parq A testdata/data/iceberg_test/iceberg_test_metadata/data/d64ba652b36f2d76-d0752076_883987622_data.0.parq A testdata/data/iceberg_test/iceberg_test_metadata/data/delete-1f43b217940cc094-fedf5156_248998721_data.0.parq A testdata/data/iceberg_test/iceberg_test_metadata/data/delete-bc4b163507d1da6f-a691cf610002_511900219_data.0.parq A testdata/data/iceberg_test/iceberg_test_metadata/metadata/38e5a1bd-5b7f-4eae-9362-16a2de3c575d-m0.avro A testdata/data/iceberg_test/iceberg_test_metadata/metadata/7c952273-c5d6-43d2-bb1b-72fbe1555f25-m0.avro A testdata/data/iceberg_test/iceberg_test_metadata/metadata/88a39285-529f-41a4-bd69-6d2560fac64e-m0.avro A testdata/data/iceberg_test/iceberg_test_metadata/metadata/ef78f8eb-b2b5-41df-a2a0-7944ac3c38da-m0.avro A testdata/data/iceberg_test/iceberg_test_metadata/metadata/snap-7858675898458780516-1-ef78f8eb-b2b5-41df-a2a0-7944ac3c38da.avro A testdata/data/iceberg_test/iceberg_test_metadata/metadata/snap-8283026816932323050-1-38e5a1bd-5b7f-4eae-9362-16a2de3c575d.avro A testdata/data/iceberg_test/iceberg_test_metadata/metadata/snap-8491702501245661704-1-88a39285-529f-41a4-bd69-6d2560fac64e.avro A testdata/data/iceberg_test/iceberg_test_metadata/metadata/snap-9046920472784493998-1-7c952273-c5d6-43d2-bb1b-72fbe1555f25.avro A testdata/data/iceberg_test/iceberg_test_metadata/metadata/v1.metadata.json A testdata/data/iceberg_test/iceberg_test_metadata/metadata/v2.metadata.json A testdata/data/iceberg_test/iceberg_test_metadata/metadata/v3.metadata.json A testdata/data/iceberg_test/iceberg_test_metadata/metadata/v4.metadata.json A testdata/data/iceberg_test/iceberg_test_metadata/metadata/v5.metadata.json A testdata/data/iceberg_test/iceberg_test_metadata/metadata/version-hint.txt M testdata/datasets/functional/functional_schema_template.sql M testdata/datasets/functional/schema_constraints.csv M testdata/workloads/functional-planner/queries/PlannerTest/iceberg-metadata-table-scan.test M testdata/workloads/functional-query/queries/QueryTest/iceberg-metadata-tables.test 46 files changed, 1,765 insertions(+), 156 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/10/20010/5 -- To view, visit http://gerrit.cloudera.org:8080/20010 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: I0e943cecd77f5ef7af7cd07e2b596f2c5b4331e7 Gerrit-Change-Number: 20010 Gerrit-PatchSet: 5 Gerrit-Owner: Tamas Mate Gerrit-Reviewer: Anonymous Coward Gerrit-Reviewer: Gabor Kaszab Gerrit-Reviewer: Gergely Fürnstáhl Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tamas Mate Gerrit-Reviewer: Zolta
[Impala-ASF-CR] IMPALA-11996: Scanner change for Iceberg metadtata querying
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/20010 ) Change subject: IMPALA-11996: Scanner change for Iceberg metadtata querying .. Patch Set 4: Build Failed https://jenkins.impala.io/job/gerrit-code-review-checks/13802/ : Initial code review checks failed. See linked job for details on the failure. -- To view, visit http://gerrit.cloudera.org:8080/20010 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I0e943cecd77f5ef7af7cd07e2b596f2c5b4331e7 Gerrit-Change-Number: 20010 Gerrit-PatchSet: 4 Gerrit-Owner: Tamas Mate Gerrit-Reviewer: Anonymous Coward Gerrit-Reviewer: Gabor Kaszab Gerrit-Reviewer: Gergely Fürnstáhl Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tamas Mate Gerrit-Reviewer: Zoltan Borok-Nagy Gerrit-Comment-Date: Tue, 22 Aug 2023 13:29:21 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-11996: Scanner change for Iceberg metadtata querying
Tamas Mate has uploaded a new patch set (#4). ( http://gerrit.cloudera.org:8080/20010 ) Change subject: IMPALA-11996: Scanner change for Iceberg metadtata querying .. IMPALA-11996: Scanner change for Iceberg metadtata querying This commit adds a scan node for querying Iceberg metadata tables. The method resembles to HBase table scans, the scan is executed with the help of the Iceberg API and through JNI. There is only one fragment created to query the Iceberg Metadata table which is supposed to be executed on the coordinator node that already has the metadata ready. This change will not cover struct column types these slots are se to NULL, it will be done in IMPALA-12205. Testing: - Added e2e tests for querying metadata tables - Updated planner tests Change-Id: I0e943cecd77f5ef7af7cd07e2b596f2c5b4331e7 --- M be/CMakeLists.txt M be/src/exec/CMakeLists.txt M be/src/exec/exec-node.cc A be/src/exec/iceberg-metadata/CMakeLists.txt A be/src/exec/iceberg-metadata/iceberg-metadata-scan-node.cc A be/src/exec/iceberg-metadata/iceberg-metadata-scan-node.h A be/src/exec/iceberg-metadata/iceberg-metadata-table-scanner.cc A be/src/exec/iceberg-metadata/iceberg-metadata-table-scanner.h M be/src/scheduling/scheduler.cc M be/src/service/frontend.cc M be/src/service/frontend.h M be/src/util/jni-util.cc M be/src/util/jni-util.h M common/thrift/PlanNodes.thrift M fe/src/main/java/org/apache/impala/analysis/IcebergMetadataTableRef.java M fe/src/main/java/org/apache/impala/analysis/Path.java M fe/src/main/java/org/apache/impala/catalog/iceberg/IcebergMetadataTable.java M fe/src/main/java/org/apache/impala/planner/DistributedPlanner.java M fe/src/main/java/org/apache/impala/planner/IcebergMetadataScanNode.java M fe/src/main/java/org/apache/impala/planner/IcebergScanPlanner.java M fe/src/main/java/org/apache/impala/planner/SingleNodePlanner.java M fe/src/main/java/org/apache/impala/service/Frontend.java M fe/src/main/java/org/apache/impala/service/JniFrontend.java A testdata/data/iceberg_test/iceberg_test_metadata/data/3d481ed88b2941f0-ea338162_1109948289_data.0.parq A testdata/data/iceberg_test/iceberg_test_metadata/data/944a2355e618932f-18f086b6_1283312202_data.0.parq A testdata/data/iceberg_test/iceberg_test_metadata/data/d64ba652b36f2d76-d0752076_883987622_data.0.parq A testdata/data/iceberg_test/iceberg_test_metadata/data/delete-1f43b217940cc094-fedf5156_248998721_data.0.parq A testdata/data/iceberg_test/iceberg_test_metadata/data/delete-bc4b163507d1da6f-a691cf610002_511900219_data.0.parq A testdata/data/iceberg_test/iceberg_test_metadata/metadata/38e5a1bd-5b7f-4eae-9362-16a2de3c575d-m0.avro A testdata/data/iceberg_test/iceberg_test_metadata/metadata/7c952273-c5d6-43d2-bb1b-72fbe1555f25-m0.avro A testdata/data/iceberg_test/iceberg_test_metadata/metadata/88a39285-529f-41a4-bd69-6d2560fac64e-m0.avro A testdata/data/iceberg_test/iceberg_test_metadata/metadata/ef78f8eb-b2b5-41df-a2a0-7944ac3c38da-m0.avro A testdata/data/iceberg_test/iceberg_test_metadata/metadata/snap-7858675898458780516-1-ef78f8eb-b2b5-41df-a2a0-7944ac3c38da.avro A testdata/data/iceberg_test/iceberg_test_metadata/metadata/snap-8283026816932323050-1-38e5a1bd-5b7f-4eae-9362-16a2de3c575d.avro A testdata/data/iceberg_test/iceberg_test_metadata/metadata/snap-8491702501245661704-1-88a39285-529f-41a4-bd69-6d2560fac64e.avro A testdata/data/iceberg_test/iceberg_test_metadata/metadata/snap-9046920472784493998-1-7c952273-c5d6-43d2-bb1b-72fbe1555f25.avro A testdata/data/iceberg_test/iceberg_test_metadata/metadata/v1.metadata.json A testdata/data/iceberg_test/iceberg_test_metadata/metadata/v2.metadata.json A testdata/data/iceberg_test/iceberg_test_metadata/metadata/v3.metadata.json A testdata/data/iceberg_test/iceberg_test_metadata/metadata/v4.metadata.json A testdata/data/iceberg_test/iceberg_test_metadata/metadata/v5.metadata.json A testdata/data/iceberg_test/iceberg_test_metadata/metadata/version-hint.txt M testdata/datasets/functional/functional_schema_template.sql M testdata/datasets/functional/schema_constraints.csv M testdata/workloads/functional-planner/queries/PlannerTest/iceberg-metadata-table-scan.test M testdata/workloads/functional-query/queries/QueryTest/iceberg-metadata-tables.test 46 files changed, 1,767 insertions(+), 156 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/10/20010/4 -- To view, visit http://gerrit.cloudera.org:8080/20010 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: I0e943cecd77f5ef7af7cd07e2b596f2c5b4331e7 Gerrit-Change-Number: 20010 Gerrit-PatchSet: 4 Gerrit-Owner: Tamas Mate Gerrit-Reviewer: Anonymous Coward Gerrit-Reviewer: Gabor Kaszab Gerrit-Reviewer: Gergely Fürnstáhl Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tamas Mate Gerrit-Reviewer: Zolta
[Impala-ASF-CR] IMPALA-11996: Scanner change for Iceberg metadtata querying
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/20010 ) Change subject: IMPALA-11996: Scanner change for Iceberg metadtata querying .. Patch Set 3: Build Failed https://jenkins.impala.io/job/gerrit-code-review-checks/13791/ : Initial code review checks failed. See linked job for details on the failure. -- To view, visit http://gerrit.cloudera.org:8080/20010 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I0e943cecd77f5ef7af7cd07e2b596f2c5b4331e7 Gerrit-Change-Number: 20010 Gerrit-PatchSet: 3 Gerrit-Owner: Tamas Mate Gerrit-Reviewer: Anonymous Coward Gerrit-Reviewer: Gabor Kaszab Gerrit-Reviewer: Gergely Fürnstáhl Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tamas Mate Gerrit-Reviewer: Zoltan Borok-Nagy Gerrit-Comment-Date: Mon, 21 Aug 2023 15:08:01 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-11996: Scanner change for Iceberg metadtata querying
Impala Public Jenkins has posted comments on this change. (
http://gerrit.cloudera.org:8080/20010 )
Change subject: IMPALA-11996: Scanner change for Iceberg metadtata querying
..
Patch Set 3:
(11 comments)
http://gerrit.cloudera.org:8080/#/c/20010/3/be/src/exec/iceberg-metadata/iceberg-metadata-table-scanner.h
File be/src/exec/iceberg-metadata/iceberg-metadata-table-scanner.h:
http://gerrit.cloudera.org:8080/#/c/20010/3/be/src/exec/iceberg-metadata/iceberg-metadata-table-scanner.h@115
PS3, Line 115:
line has trailing whitespace
http://gerrit.cloudera.org:8080/#/c/20010/3/be/src/exec/iceberg-metadata/iceberg-metadata-table-scanner.cc
File be/src/exec/iceberg-metadata/iceberg-metadata-table-scanner.cc:
http://gerrit.cloudera.org:8080/#/c/20010/3/be/src/exec/iceberg-metadata/iceberg-metadata-table-scanner.cc@39
PS3, Line 39:
iceberg_api_scan_timer_(ADD_TIMER(scan_node_->runtime_profile(),
line has trailing whitespace
http://gerrit.cloudera.org:8080/#/c/20010/3/be/src/exec/iceberg-metadata/iceberg-metadata-table-scanner.cc@186
PS3, Line 186: jobject column_field = env->CallObjectMethod(columns,
list_get_, slot_desc->col_pos());
line too long (91 > 90)
http://gerrit.cloudera.org:8080/#/c/20010/3/be/src/exec/iceberg-metadata/iceberg-metadata-table-scanner.cc@245
PS3, Line 245: RETURN_IF_ERROR(MaterializeNextRow(env, struct_like_row,
tuple, state, row_batch->tuple_data_pool()));
line too long (108 > 90)
http://gerrit.cloudera.org:8080/#/c/20010/3/be/src/exec/iceberg-metadata/iceberg-metadata-table-scanner.cc@249
PS3, Line 249: if
(ExecNode::EvalConjuncts(scan_node_->conjunct_evals().data(),
scan_node_->conjunct_evals().size(), tuple_row)) {
line too long (121 > 90)
http://gerrit.cloudera.org:8080/#/c/20010/3/be/src/exec/iceberg-metadata/iceberg-metadata-table-scanner.cc@255
PS3, Line 255: Tuple::ClearNullBits(tuple,
tuple_desc_->null_bytes_offset(), tuple_desc_->num_null_bytes());
line too long (101 > 90)
http://gerrit.cloudera.org:8080/#/c/20010/3/be/src/exec/iceberg-metadata/iceberg-metadata-table-scanner.cc@263
PS3, Line 263: Status IcebergMetadataTableScanner::MaterializeNextRow(JNIEnv*
env, jobject struct_like_row,
line too long (92 > 90)
http://gerrit.cloudera.org:8080/#/c/20010/3/be/src/exec/iceberg-metadata/iceberg-metadata-table-scanner.cc@280
PS3, Line 280: RETURN_IF_ERROR(ReadStringValue(env, slot_desc,
struct_like_row, tuple, tuple_data_pool));
line too long (98 > 90)
http://gerrit.cloudera.org:8080/#/c/20010/3/be/src/service/frontend.cc
File be/src/service/frontend.cc:
http://gerrit.cloudera.org:8080/#/c/20010/3/be/src/service/frontend.cc@110
PS3, Line 110: {"getCatalogTable",
"([B)Lorg/apache/impala/catalog/FeTable;", &get_catalog_table_id_},
line too long (91 > 90)
http://gerrit.cloudera.org:8080/#/c/20010/3/fe/src/main/java/org/apache/impala/planner/IcebergMetadataScanNode.java
File fe/src/main/java/org/apache/impala/planner/IcebergMetadataScanNode.java:
http://gerrit.cloudera.org:8080/#/c/20010/3/fe/src/main/java/org/apache/impala/planner/IcebergMetadataScanNode.java@40
PS3, Line 40: private final static Logger LOG =
LoggerFactory.getLogger(IcebergMetadataScanNode.class);
line too long (91 > 90)
http://gerrit.cloudera.org:8080/#/c/20010/3/fe/src/main/java/org/apache/impala/planner/IcebergMetadataScanNode.java@45
PS3, Line 45: protected IcebergMetadataScanNode(PlanNodeId id, List
conjuncts, TableRef tblRef) {
line too long (91 > 90)
--
To view, visit http://gerrit.cloudera.org:8080/20010
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I0e943cecd77f5ef7af7cd07e2b596f2c5b4331e7
Gerrit-Change-Number: 20010
Gerrit-PatchSet: 3
Gerrit-Owner: Tamas Mate
Gerrit-Reviewer: Anonymous Coward
Gerrit-Reviewer: Gabor Kaszab
Gerrit-Reviewer: Gergely Fürnstáhl
Gerrit-Reviewer: Impala Public Jenkins
Gerrit-Reviewer: Tamas Mate
Gerrit-Reviewer: Zoltan Borok-Nagy
Gerrit-Comment-Date: Mon, 21 Aug 2023 14:43:25 +
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-11996: Scanner change for Iceberg metadtata querying
Tamas Mate has uploaded a new patch set (#3). ( http://gerrit.cloudera.org:8080/20010 ) Change subject: IMPALA-11996: Scanner change for Iceberg metadtata querying .. IMPALA-11996: Scanner change for Iceberg metadtata querying This commit adds a scan node for querying Iceberg metadata tables. The method resembles to HBase table scans, the scan is executed with the help of the Iceberg API and through JNI. There is only one fragment created to query the Iceberg Metadata table which is supposed to be executed on the coordinator node that already has the metadata ready. This change will not cover struct column types these slots are se to NULL, it will be done in IMPALA-12205. Testing: - Added e2e tests for querying metadata tables - Updated planner tests Change-Id: I0e943cecd77f5ef7af7cd07e2b596f2c5b4331e7 --- M be/CMakeLists.txt M be/src/exec/CMakeLists.txt M be/src/exec/exec-node.cc A be/src/exec/iceberg-metadata/CMakeLists.txt A be/src/exec/iceberg-metadata/iceberg-metadata-scan-node.cc A be/src/exec/iceberg-metadata/iceberg-metadata-scan-node.h A be/src/exec/iceberg-metadata/iceberg-metadata-table-scanner.cc A be/src/exec/iceberg-metadata/iceberg-metadata-table-scanner.h M be/src/scheduling/scheduler.cc M be/src/service/frontend.cc M be/src/service/frontend.h M be/src/util/jni-util.cc M be/src/util/jni-util.h M common/thrift/PlanNodes.thrift M fe/src/main/java/org/apache/impala/analysis/IcebergMetadataTableRef.java M fe/src/main/java/org/apache/impala/analysis/Path.java M fe/src/main/java/org/apache/impala/catalog/iceberg/IcebergMetadataTable.java M fe/src/main/java/org/apache/impala/planner/DistributedPlanner.java M fe/src/main/java/org/apache/impala/planner/IcebergMetadataScanNode.java M fe/src/main/java/org/apache/impala/planner/IcebergScanPlanner.java M fe/src/main/java/org/apache/impala/planner/SingleNodePlanner.java M fe/src/main/java/org/apache/impala/service/Frontend.java M fe/src/main/java/org/apache/impala/service/JniFrontend.java A testdata/data/iceberg_test/iceberg_test_metadata/data/3d481ed88b2941f0-ea338162_1109948289_data.0.parq A testdata/data/iceberg_test/iceberg_test_metadata/data/944a2355e618932f-18f086b6_1283312202_data.0.parq A testdata/data/iceberg_test/iceberg_test_metadata/data/d64ba652b36f2d76-d0752076_883987622_data.0.parq A testdata/data/iceberg_test/iceberg_test_metadata/data/delete-1f43b217940cc094-fedf5156_248998721_data.0.parq A testdata/data/iceberg_test/iceberg_test_metadata/data/delete-bc4b163507d1da6f-a691cf610002_511900219_data.0.parq A testdata/data/iceberg_test/iceberg_test_metadata/metadata/38e5a1bd-5b7f-4eae-9362-16a2de3c575d-m0.avro A testdata/data/iceberg_test/iceberg_test_metadata/metadata/7c952273-c5d6-43d2-bb1b-72fbe1555f25-m0.avro A testdata/data/iceberg_test/iceberg_test_metadata/metadata/88a39285-529f-41a4-bd69-6d2560fac64e-m0.avro A testdata/data/iceberg_test/iceberg_test_metadata/metadata/ef78f8eb-b2b5-41df-a2a0-7944ac3c38da-m0.avro A testdata/data/iceberg_test/iceberg_test_metadata/metadata/snap-7858675898458780516-1-ef78f8eb-b2b5-41df-a2a0-7944ac3c38da.avro A testdata/data/iceberg_test/iceberg_test_metadata/metadata/snap-8283026816932323050-1-38e5a1bd-5b7f-4eae-9362-16a2de3c575d.avro A testdata/data/iceberg_test/iceberg_test_metadata/metadata/snap-8491702501245661704-1-88a39285-529f-41a4-bd69-6d2560fac64e.avro A testdata/data/iceberg_test/iceberg_test_metadata/metadata/snap-9046920472784493998-1-7c952273-c5d6-43d2-bb1b-72fbe1555f25.avro A testdata/data/iceberg_test/iceberg_test_metadata/metadata/v1.metadata.json A testdata/data/iceberg_test/iceberg_test_metadata/metadata/v2.metadata.json A testdata/data/iceberg_test/iceberg_test_metadata/metadata/v3.metadata.json A testdata/data/iceberg_test/iceberg_test_metadata/metadata/v4.metadata.json A testdata/data/iceberg_test/iceberg_test_metadata/metadata/v5.metadata.json A testdata/data/iceberg_test/iceberg_test_metadata/metadata/version-hint.txt M testdata/datasets/functional/functional_schema_template.sql M testdata/datasets/functional/schema_constraints.csv M testdata/workloads/functional-planner/queries/PlannerTest/iceberg-metadata-table-scan.test M testdata/workloads/functional-query/queries/QueryTest/iceberg-metadata-tables.test 46 files changed, 1,762 insertions(+), 156 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/10/20010/3 -- To view, visit http://gerrit.cloudera.org:8080/20010 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: I0e943cecd77f5ef7af7cd07e2b596f2c5b4331e7 Gerrit-Change-Number: 20010 Gerrit-PatchSet: 3 Gerrit-Owner: Tamas Mate Gerrit-Reviewer: Anonymous Coward Gerrit-Reviewer: Gabor Kaszab Gerrit-Reviewer: Gergely Fürnstáhl Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tamas Mate Gerrit-Reviewer: Zolta
