[Impala-ASF-CR] IMPALA-3880: Add list of all tables queried to runtime profile
Peter Rozsa has uploaded this change for review. ( http://gerrit.cloudera.org:8080/19401 Change subject: IMPALA-3880: Add list of all tables queried to runtime profile .. IMPALA-3880: Add list of all tables queried to runtime profile This change adds a new info string to the frontend runtime profile which contains the referenced tables by the query in a comma-separated format. Change-Id: Ib474a5c6522032679701103aa225a18edca62f5a --- M fe/src/main/java/org/apache/impala/service/Frontend.java M tests/shell/test_shell_commandline.py 2 files changed, 29 insertions(+), 1 deletion(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/01/19401/1 -- To view, visit http://gerrit.cloudera.org:8080/19401 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newchange Gerrit-Change-Id: Ib474a5c6522032679701103aa225a18edca62f5a Gerrit-Change-Number: 19401 Gerrit-PatchSet: 1 Gerrit-Owner: Peter Rozsa
[Impala-ASF-CR] IMPALA-11827: do not cache admission control service's IP address in impalad
Impala Public Jenkins has submitted this change and it was merged. (
http://gerrit.cloudera.org:8080/19403 )
Change subject: IMPALA-11827: do not cache admission control service's IP
address in impalad
..
IMPALA-11827: do not cache admission control service's IP address in impalad
The Impalad's ExecEnv caches resolved IP address for admission control
service. The Impala server runs a heart beat thread for admission
control service and it relies on the cached resolved address in ExecEnv.
This breaks when IP address for admission control services changes.
This patch removes the 'admission_service_address_' member and its
getter in the ExecEnv class and adds a GetAdmissionServiceAddress()
interface to dynamically get the resolved address, if global admission
control service is enabled ('admission_service_host' flag is set).
Testing: Manually tested by restarting global admission controller
service in a Kubernetes setup where it gets new IP address on startup.
Coordinator's admission heart beat thread was able to connect to the
admission control service using the new IP address. Also made sure that
the coordinator and admissiond communication was working as expected
upon admissiond restarts.
Change-Id: I09b4c52644f9e1b3c1f0e68c9900464d722517af
Reviewed-on: http://gerrit.cloudera.org:8080/19403
Reviewed-by: Impala Public Jenkins
Tested-by: Impala Public Jenkins
---
M be/src/runtime/exec-env.cc
M be/src/runtime/exec-env.h
M be/src/scheduling/admission-control-service.cc
M be/src/service/impala-server.cc
4 files changed, 21 insertions(+), 18 deletions(-)
Approvals:
Impala Public Jenkins: Looks good to me, approved; Verified
--
To view, visit http://gerrit.cloudera.org:8080/19403
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: merged
Gerrit-Change-Id: I09b4c52644f9e1b3c1f0e68c9900464d722517af
Gerrit-Change-Number: 19403
Gerrit-PatchSet: 4
Gerrit-Owner: Abhishek Rawat
Gerrit-Reviewer: Andrew Sherman
Gerrit-Reviewer: Impala Public Jenkins
Gerrit-Reviewer: Wenzhe Zhou
[Impala-ASF-CR] IMPALA-11827: do not cache admission control service's IP address in impalad
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/19403 ) Change subject: IMPALA-11827: do not cache admission control service's IP address in impalad .. Patch Set 3: Verified+1 -- To view, visit http://gerrit.cloudera.org:8080/19403 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I09b4c52644f9e1b3c1f0e68c9900464d722517af Gerrit-Change-Number: 19403 Gerrit-PatchSet: 3 Gerrit-Owner: Abhishek Rawat Gerrit-Reviewer: Andrew Sherman Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Wenzhe Zhou Gerrit-Comment-Date: Fri, 06 Jan 2023 06:53:52 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-11662: Improve 'refresh iceberg tbl on oss' performance
lipeng...@sensorsdata.cn has posted comments on this change. (
http://gerrit.cloudera.org:8080/19379 )
Change subject: IMPALA-11662: Improve 'refresh iceberg_tbl_on_oss' performance
..
Patch Set 6:
(6 comments)
Thanks for comments!
http://gerrit.cloudera.org:8080/#/c/19379/5/fe/src/main/java/org/apache/impala/catalog/FileMetadataLoader.java
File fe/src/main/java/org/apache/impala/catalog/FileMetadataLoader.java:
http://gerrit.cloudera.org:8080/#/c/19379/5/fe/src/main/java/org/apache/impala/catalog/FileMetadataLoader.java@212
PS5, Line 212: edFds_.add
> I think fd cannot be null here, see my comment in IcebergFileMetadataLoader
Done
http://gerrit.cloudera.org:8080/#/c/19379/5/fe/src/main/java/org/apache/impala/catalog/IcebergFileMetadataLoader.java
File fe/src/main/java/org/apache/impala/catalog/IcebergFileMetadataLoader.java:
http://gerrit.cloudera.org:8080/#/c/19379/5/fe/src/main/java/org/apache/impala/catalog/IcebergFileMetadataLoader.java@17
PS5, Line 17:
> nit: please add empty line above this
Done
http://gerrit.cloudera.org:8080/#/c/19379/5/fe/src/main/java/org/apache/impala/catalog/IcebergFileMetadataLoader.java@57
PS5, Line 57: IcebergFileMetadataLoader.class);
> nit: please add comment why we need this
Done
http://gerrit.cloudera.org:8080/#/c/19379/5/fe/src/main/java/org/apache/impala/catalog/IcebergFileMetadataLoader.java@89
PS5, Line 89: if (relUri.isAbsolute() ||
relUri.getPath().startsWith(Path.SEPARATOR)) {
: if (canDataOutsideOfTableLocation_) {
: absPath = fileStatus.getPath().toString();
: } else {
> skippedFiles means the following based on its comment: "Number of files ski
Yes, we should!
http://gerrit.cloudera.org:8080/#/c/19379/5/fe/src/main/java/org/apache/impala/catalog/IcebergFileMetadataLoader.java@124
PS5, Line 124:
> This comment mentions "Avoid the cost of directory listing", then there is
Done
http://gerrit.cloudera.org:8080/#/c/19379/5/fe/src/main/java/org/apache/impala/catalog/ParallelFileMetadataLoader.java
File fe/src/main/java/org/apache/impala/catalog/ParallelFileMetadataLoader.java:
http://gerrit.cloudera.org:8080/#/c/19379/5/fe/src/main/java/org/apache/impala/catalog/ParallelFileMetadataLoader.java@113
PS5, Line 113: if (format.equals(HdfsFileFormat.ICEBERG)) {
> I see, sorry, I should have taken a closer look at IcebergFileMetadataLoade
Done
--
To view, visit http://gerrit.cloudera.org:8080/19379
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: If2ee8b6b7559e6590698b46ef1d574e55ed52f9a
Gerrit-Change-Number: 19379
Gerrit-PatchSet: 6
Gerrit-Owner: Anonymous Coward
Gerrit-Reviewer: Anonymous Coward
Gerrit-Reviewer: Gergely Fürnstáhl
Gerrit-Reviewer: Impala Public Jenkins
Gerrit-Reviewer: Michael Smith
Gerrit-Reviewer: Tamas Mate
Gerrit-Reviewer: Zoltan Borok-Nagy
Gerrit-Comment-Date: Fri, 06 Jan 2023 06:24:14 +
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-11786: Preserve memory for codegen cache
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/19377 ) Change subject: IMPALA-11786: Preserve memory for codegen cache .. Patch Set 5: Verified+1 -- To view, visit http://gerrit.cloudera.org:8080/19377 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Iebdc04ba1b91578d74684209a11c815225b8505a Gerrit-Change-Number: 19377 Gerrit-PatchSet: 5 Gerrit-Owner: Yida Wu Gerrit-Reviewer: Abhishek Rawat Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Michael Smith Gerrit-Reviewer: Qifan Chen Gerrit-Reviewer: Yida Wu Gerrit-Comment-Date: Fri, 06 Jan 2023 06:28:06 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-11786: Preserve memory for codegen cache
Impala Public Jenkins has submitted this change and it was merged. ( http://gerrit.cloudera.org:8080/19377 ) Change subject: IMPALA-11786: Preserve memory for codegen cache .. IMPALA-11786: Preserve memory for codegen cache IMPALA-11470 adds support for codegen cache, however the admission controller is not aware of the memory usage of the codegen cache, while the codegen cache is actually using the memory quota from the query memory. It could result in query failures when running heavy workloads and admission controller has fully admitted queries. This patch subtracts the codegen cache capacity from the admission memory limit during initialization, therefore preserving the memory consumption of codegen cache from the beginning, and treating it as a separate memory independent to the query memory reservation. Also reduces the max codegen cache memory from 20 percent to 10 percent, and changes some failed testcases due to the reduction of the admit memory limit. Tests: Passed exhaustive tests. Change-Id: Iebdc04ba1b91578d74684209a11c815225b8505a Reviewed-on: http://gerrit.cloudera.org:8080/19377 Reviewed-by: Impala Public Jenkins Tested-by: Impala Public Jenkins --- M be/src/runtime/exec-env.cc M tests/custom_cluster/test_admission_controller.py M tests/custom_cluster/test_executor_groups.py M tests/custom_cluster/test_jvm_mem_tracking.py 4 files changed, 40 insertions(+), 25 deletions(-) Approvals: Impala Public Jenkins: Looks good to me, approved; Verified -- To view, visit http://gerrit.cloudera.org:8080/19377 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: merged Gerrit-Change-Id: Iebdc04ba1b91578d74684209a11c815225b8505a Gerrit-Change-Number: 19377 Gerrit-PatchSet: 6 Gerrit-Owner: Yida Wu Gerrit-Reviewer: Abhishek Rawat Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Michael Smith Gerrit-Reviewer: Qifan Chen Gerrit-Reviewer: Yida Wu
[Impala-ASF-CR] IMPALA-11662: Improve 'refresh iceberg tbl on oss' performance
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/19379 ) Change subject: IMPALA-11662: Improve 'refresh iceberg_tbl_on_oss' performance .. Patch Set 6: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/12117/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/19379 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: If2ee8b6b7559e6590698b46ef1d574e55ed52f9a Gerrit-Change-Number: 19379 Gerrit-PatchSet: 6 Gerrit-Owner: Anonymous Coward Gerrit-Reviewer: Anonymous Coward Gerrit-Reviewer: Gergely Fürnstáhl Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Michael Smith Gerrit-Reviewer: Tamas Mate Gerrit-Reviewer: Zoltan Borok-Nagy Gerrit-Comment-Date: Fri, 06 Jan 2023 06:17:54 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-11662: Improve 'refresh iceberg tbl on oss' performance
lipeng...@sensorsdata.cn has uploaded a new patch set (#6). ( http://gerrit.cloudera.org:8080/19379 ) Change subject: IMPALA-11662: Improve 'refresh iceberg_tbl_on_oss' performance .. IMPALA-11662: Improve 'refresh iceberg_tbl_on_oss' performance Iceberg provides rich metadata, the cost of directory listing on OSS service e.g. S3A is higher than the cost on HDFS, we could create the file descriptors from Iceberg metadata instead of using org.apache.hadoop.fs.FileSystem#listFiles. The only thing missing there is the last_modification_time of the files. But since Iceberg files are immutable, we could just come up with a special timestamp for these files. Test: * existing tests * test on COS with my local test environment Change-Id: If2ee8b6b7559e6590698b46ef1d574e55ed52f9a --- M fe/src/main/java/org/apache/impala/catalog/FeIcebergTable.java M fe/src/main/java/org/apache/impala/catalog/FileMetadataLoader.java M fe/src/main/java/org/apache/impala/catalog/HdfsPartition.java M fe/src/main/java/org/apache/impala/catalog/HdfsTable.java A fe/src/main/java/org/apache/impala/catalog/IcebergFileMetadataLoader.java M fe/src/main/java/org/apache/impala/catalog/IcebergTable.java M fe/src/main/java/org/apache/impala/catalog/ParallelFileMetadataLoader.java 7 files changed, 322 insertions(+), 83 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/79/19379/6 -- To view, visit http://gerrit.cloudera.org:8080/19379 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: If2ee8b6b7559e6590698b46ef1d574e55ed52f9a Gerrit-Change-Number: 19379 Gerrit-PatchSet: 6 Gerrit-Owner: Anonymous Coward Gerrit-Reviewer: Anonymous Coward Gerrit-Reviewer: Gergely Fürnstáhl Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Michael Smith Gerrit-Reviewer: Tamas Mate Gerrit-Reviewer: Zoltan Borok-Nagy
[Impala-ASF-CR] IMPALA-11827: do not cache admission control service's IP address in impalad
Wenzhe Zhou has posted comments on this change. ( http://gerrit.cloudera.org:8080/19403 ) Change subject: IMPALA-11827: do not cache admission control service's IP address in impalad .. Patch Set 2: Code-Review+1 -- To view, visit http://gerrit.cloudera.org:8080/19403 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I09b4c52644f9e1b3c1f0e68c9900464d722517af Gerrit-Change-Number: 19403 Gerrit-PatchSet: 2 Gerrit-Owner: Abhishek Rawat Gerrit-Reviewer: Andrew Sherman Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Wenzhe Zhou Gerrit-Comment-Date: Fri, 06 Jan 2023 01:40:47 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-11827: do not cache admission control service's IP address in impalad
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/19403 ) Change subject: IMPALA-11827: do not cache admission control service's IP address in impalad .. Patch Set 3: Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/8952/ DRY_RUN=false -- To view, visit http://gerrit.cloudera.org:8080/19403 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I09b4c52644f9e1b3c1f0e68c9900464d722517af Gerrit-Change-Number: 19403 Gerrit-PatchSet: 3 Gerrit-Owner: Abhishek Rawat Gerrit-Reviewer: Andrew Sherman Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Wenzhe Zhou Gerrit-Comment-Date: Fri, 06 Jan 2023 01:41:53 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-11827: do not cache admission control service's IP address in impalad
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/19403 ) Change subject: IMPALA-11827: do not cache admission control service's IP address in impalad .. Patch Set 3: Code-Review+2 -- To view, visit http://gerrit.cloudera.org:8080/19403 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I09b4c52644f9e1b3c1f0e68c9900464d722517af Gerrit-Change-Number: 19403 Gerrit-PatchSet: 3 Gerrit-Owner: Abhishek Rawat Gerrit-Reviewer: Andrew Sherman Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Wenzhe Zhou Gerrit-Comment-Date: Fri, 06 Jan 2023 01:41:52 + Gerrit-HasComments: No
[Impala-ASF-CR] WIP IMPALA-11809: Support non unique primary key for Kudu
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/19383 ) Change subject: WIP IMPALA-11809: Support non unique primary key for Kudu .. Patch Set 5: Build Failed https://jenkins.impala.io/job/gerrit-code-review-checks/12116/ : Initial code review checks failed. See linked job for details on the failure. -- To view, visit http://gerrit.cloudera.org:8080/19383 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I4d7882bf3d01a3492cc9827c072d1f3200d9eebd Gerrit-Change-Number: 19383 Gerrit-PatchSet: 5 Gerrit-Owner: Wenzhe Zhou Gerrit-Reviewer: Abhishek Chennaka Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Kurt Deschler Gerrit-Reviewer: Marton Greber Gerrit-Comment-Date: Fri, 06 Jan 2023 01:41:34 + Gerrit-HasComments: No
[Impala-ASF-CR] WIP IMPALA-11809: Support non unique primary key for Kudu
Wenzhe Zhou has uploaded a new patch set (#5). ( http://gerrit.cloudera.org:8080/19383 ) Change subject: WIP IMPALA-11809: Support non unique primary key for Kudu .. WIP IMPALA-11809: Support non unique primary key for Kudu Kudu engine adds support for non unique primary key. It adds a column 'auto_increment_id' automatically in a table which has non unique primary key. The non unique primary key and 'auto_increment_id' form unique composite primary key. This patch integrated new version of Kudu which support non unique primary key, added syntactic support for creating table with non unique primary key. Example: CREATE TABLE tbl (i INT NON UNIQUE PRIMARY KEY, s STRING) PARTITION BY HASH (i) partitions 3 STORED as KUDU; CREATE TABLE tbl (i INT, s STRING, NON UNIQUE PRIMARY KEY(i)) PARTITION BY HASH (i) partitions 3 STORED as KUDU; CREATE TABLE tbl NON UNIQUE PRIMARY KEY(id) PARTITION BY HASH (id) partitions 3 STORED as KUDU AS SELECT id, string_col FROM functional.alltypes WHERE id = 10; Kudu assign values for column 'auto_increment_id' automatically when inserting rows so insertion statements don't need to specify values for column 'auto_increment_id'. SELECT statement does not show 'auto_increment_id' column unless the column is explicitly specified in select list. UPSERT operation is not supported now for Kudu table with auto incrementing column due to limitation in Kudu engine. Testing: - Integrated new version of Kudu built on local machine, ran manual test in impala-shell with queries to create tables with non unique primary key, and tested insert/update/delete operations for Kudu tables with non unique primary key. - TODO build toolchian with new version of Kudu, including the commits for KUDU-1945. Run core test. Change-Id: I4d7882bf3d01a3492cc9827c072d1f3200d9eebd --- M common/thrift/CatalogObjects.thrift M common/thrift/JniCatalog.thrift M fe/src/main/cup/sql-parser.cup M fe/src/main/java/org/apache/impala/analysis/AlterTableAddColsStmt.java M fe/src/main/java/org/apache/impala/analysis/ColumnDef.java M fe/src/main/java/org/apache/impala/analysis/CreateTableAsSelectStmt.java M fe/src/main/java/org/apache/impala/analysis/CreateTableLikeFileStmt.java M fe/src/main/java/org/apache/impala/analysis/CreateTableStmt.java M fe/src/main/java/org/apache/impala/analysis/InsertStmt.java M fe/src/main/java/org/apache/impala/analysis/SelectStmt.java M fe/src/main/java/org/apache/impala/analysis/TableDef.java M fe/src/main/java/org/apache/impala/analysis/ToSqlUtils.java M fe/src/main/java/org/apache/impala/catalog/Db.java M fe/src/main/java/org/apache/impala/catalog/FeDb.java M fe/src/main/java/org/apache/impala/catalog/FeKuduTable.java M fe/src/main/java/org/apache/impala/catalog/KuduColumn.java M fe/src/main/java/org/apache/impala/catalog/KuduTable.java M fe/src/main/java/org/apache/impala/catalog/local/LocalDb.java M fe/src/main/java/org/apache/impala/catalog/local/LocalKuduTable.java M fe/src/main/java/org/apache/impala/service/KuduCatalogOpExecutor.java M fe/src/main/java/org/apache/impala/util/KuduUtil.java M fe/src/main/jflex/sql-scanner.flex M fe/src/test/java/org/apache/impala/analysis/AnalyzeDDLTest.java M fe/src/test/java/org/apache/impala/analysis/AnalyzeKuduDDLTest.java M fe/src/test/java/org/apache/impala/analysis/ParserTest.java M testdata/workloads/functional-query/queries/QueryTest/kudu-scan-node.test M testdata/workloads/functional-query/queries/QueryTest/kudu_alter.test M testdata/workloads/functional-query/queries/QueryTest/kudu_create.test M testdata/workloads/functional-query/queries/QueryTest/kudu_insert.test M testdata/workloads/functional-query/queries/QueryTest/kudu_update.test M testdata/workloads/functional-query/queries/QueryTest/kudu_upsert.test 31 files changed, 735 insertions(+), 101 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/83/19383/5 -- To view, visit http://gerrit.cloudera.org:8080/19383 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: I4d7882bf3d01a3492cc9827c072d1f3200d9eebd Gerrit-Change-Number: 19383 Gerrit-PatchSet: 5 Gerrit-Owner: Wenzhe Zhou Gerrit-Reviewer: Abhishek Chennaka Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Kurt Deschler Gerrit-Reviewer: Marton Greber
[Impala-ASF-CR] IMPALA-11786: Preserve memory for codegen cache
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/19377 ) Change subject: IMPALA-11786: Preserve memory for codegen cache .. Patch Set 5: Code-Review+2 -- To view, visit http://gerrit.cloudera.org:8080/19377 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Iebdc04ba1b91578d74684209a11c815225b8505a Gerrit-Change-Number: 19377 Gerrit-PatchSet: 5 Gerrit-Owner: Yida Wu Gerrit-Reviewer: Abhishek Rawat Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Michael Smith Gerrit-Reviewer: Qifan Chen Gerrit-Reviewer: Yida Wu Gerrit-Comment-Date: Fri, 06 Jan 2023 01:21:47 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-11786: Preserve memory for codegen cache
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/19377 ) Change subject: IMPALA-11786: Preserve memory for codegen cache .. Patch Set 5: Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/8951/ DRY_RUN=false -- To view, visit http://gerrit.cloudera.org:8080/19377 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Iebdc04ba1b91578d74684209a11c815225b8505a Gerrit-Change-Number: 19377 Gerrit-PatchSet: 5 Gerrit-Owner: Yida Wu Gerrit-Reviewer: Abhishek Rawat Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Michael Smith Gerrit-Reviewer: Qifan Chen Gerrit-Reviewer: Yida Wu Gerrit-Comment-Date: Fri, 06 Jan 2023 01:21:48 + Gerrit-HasComments: No
[Impala-ASF-CR](asf-site) IMPALA-11825: Doc update 4.2.0 breaks web search results
Quanlong Huang has posted comments on this change. ( http://gerrit.cloudera.org:8080/19402 ) Change subject: IMPALA-11825: Doc update 4.2.0 breaks web search results .. Patch Set 1: Instead of deep copying the whole folder, can we just create a symlink of "html -> asf-site-html" ? It works in my local env: git fetch http://gerrit.cloudera.org:8080/Impala-ASF refs/changes/02/19402/1 && git checkout FETCH_HEAD cd docs/build rm -rf html ln -s asf-site-html html Open html/index.html in the browser. It seems other links also work. -- To view, visit http://gerrit.cloudera.org:8080/19402 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: asf-site Gerrit-MessageType: comment Gerrit-Change-Id: Iab1d5fe5e3be85f26aeeb24be9e0018935d2e424 Gerrit-Change-Number: 19402 Gerrit-PatchSet: 1 Gerrit-Owner: Daniel Becker Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Quanlong Huang Gerrit-Reviewer: Tamas Mate Gerrit-Reviewer: Zoltan Borok-Nagy Gerrit-Comment-Date: Fri, 06 Jan 2023 00:52:31 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-11808: Add support for reload event in catalogD
Aman Sinha has submitted this change and it was merged. ( http://gerrit.cloudera.org:8080/19378 ) Change subject: IMPALA-11808: Add support for reload event in catalogD .. IMPALA-11808: Add support for reload event in catalogD This patch supports a new event called reload event in catalogD. This event is used to update table/file metadata for a refresh or invalidate command in other replicas of catalogDs when one of the replicas issued refresh/invalidate command. HIVE-26838 is the Hive jira that adds support for this reload event. This feature is disabled by default using a config enable_reload_events. To use this feature set this config to true and impala will be able to fire reload events. The processing of reload events in the event processor is always enabled. There is an end-to-end test added for this feature which currently checks firing/creation of the reload event and self-event check in the event processor. TODO: end-to-end test should also test this reload event in the event processor. There is also a follow up jira IMPALA-11822 to track the optimization patch for this feature. Change-Id: Ic62d58837d356dc2113f3c0904228ac9de484136 Reviewed-on: http://gerrit.cloudera.org:8080/19378 Tested-by: Aman Sinha Reviewed-by: Andrew Sherman Reviewed-by: Aman Sinha --- M be/src/catalog/catalog-server.cc M be/src/util/backend-gflag-util.cc M common/thrift/BackendGflags.thrift M fe/src/compat-apache-hive-3/java/org/apache/impala/compat/MetastoreShim.java M fe/src/compat-hive-3/java/org/apache/impala/compat/MetastoreShim.java M fe/src/main/java/org/apache/impala/catalog/events/MetastoreEvents.java M fe/src/main/java/org/apache/impala/service/BackendConfig.java M fe/src/main/java/org/apache/impala/service/CatalogOpExecutor.java M tests/custom_cluster/test_events_custom_configs.py 9 files changed, 337 insertions(+), 2 deletions(-) Approvals: Aman Sinha: Looks good to me, approved; Verified Andrew Sherman: Looks good to me, but someone else must approve -- To view, visit http://gerrit.cloudera.org:8080/19378 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: merged Gerrit-Change-Id: Ic62d58837d356dc2113f3c0904228ac9de484136 Gerrit-Change-Number: 19378 Gerrit-PatchSet: 17 Gerrit-Owner: Sai Hemanth Gantasala Gerrit-Reviewer: Abhishek Rawat Gerrit-Reviewer: Aman Sinha Gerrit-Reviewer: Andrew Sherman Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Quanlong Huang Gerrit-Reviewer: Sai Hemanth Gantasala
[Impala-ASF-CR] IMPALA-11808: Add support for reload event in catalogD
Aman Sinha has posted comments on this change. ( http://gerrit.cloudera.org:8080/19378 ) Change subject: IMPALA-11808: Add support for reload event in catalogD .. Patch Set 16: Code-Review+2 -- To view, visit http://gerrit.cloudera.org:8080/19378 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ic62d58837d356dc2113f3c0904228ac9de484136 Gerrit-Change-Number: 19378 Gerrit-PatchSet: 16 Gerrit-Owner: Sai Hemanth Gantasala Gerrit-Reviewer: Abhishek Rawat Gerrit-Reviewer: Aman Sinha Gerrit-Reviewer: Andrew Sherman Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Quanlong Huang Gerrit-Reviewer: Sai Hemanth Gantasala Gerrit-Comment-Date: Fri, 06 Jan 2023 00:38:30 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-11827: do not cache admission control service's IP address in impalad
Andrew Sherman has posted comments on this change. ( http://gerrit.cloudera.org:8080/19403 ) Change subject: IMPALA-11827: do not cache admission control service's IP address in impalad .. Patch Set 2: Code-Review+2 LGTM -- To view, visit http://gerrit.cloudera.org:8080/19403 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I09b4c52644f9e1b3c1f0e68c9900464d722517af Gerrit-Change-Number: 19403 Gerrit-PatchSet: 2 Gerrit-Owner: Abhishek Rawat Gerrit-Reviewer: Andrew Sherman Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Wenzhe Zhou Gerrit-Comment-Date: Fri, 06 Jan 2023 00:37:46 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-11808: Add support for reload event in catalogD
Andrew Sherman has posted comments on this change. ( http://gerrit.cloudera.org:8080/19378 ) Change subject: IMPALA-11808: Add support for reload event in catalogD .. Patch Set 16: Code-Review+1 -- To view, visit http://gerrit.cloudera.org:8080/19378 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ic62d58837d356dc2113f3c0904228ac9de484136 Gerrit-Change-Number: 19378 Gerrit-PatchSet: 16 Gerrit-Owner: Sai Hemanth Gantasala Gerrit-Reviewer: Abhishek Rawat Gerrit-Reviewer: Aman Sinha Gerrit-Reviewer: Andrew Sherman Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Quanlong Huang Gerrit-Reviewer: Sai Hemanth Gantasala Gerrit-Comment-Date: Fri, 06 Jan 2023 00:29:22 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-11827: do not cache admission control service's IP address in impalad
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/19403 ) Change subject: IMPALA-11827: do not cache admission control service's IP address in impalad .. Patch Set 2: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/12115/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/19403 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I09b4c52644f9e1b3c1f0e68c9900464d722517af Gerrit-Change-Number: 19403 Gerrit-PatchSet: 2 Gerrit-Owner: Abhishek Rawat Gerrit-Reviewer: Andrew Sherman Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Wenzhe Zhou Gerrit-Comment-Date: Fri, 06 Jan 2023 00:13:02 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-11827: do not cache admission control service's IP address in impalad
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/19403 ) Change subject: IMPALA-11827: do not cache admission control service's IP address in impalad .. Patch Set 1: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/12114/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/19403 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I09b4c52644f9e1b3c1f0e68c9900464d722517af Gerrit-Change-Number: 19403 Gerrit-PatchSet: 1 Gerrit-Owner: Abhishek Rawat Gerrit-Reviewer: Andrew Sherman Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Wenzhe Zhou Gerrit-Comment-Date: Fri, 06 Jan 2023 00:09:46 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-11801: Use thrift 0.11.0 in FE when compiling with Apache Hive3
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/19400 ) Change subject: IMPALA-11801: Use thrift 0.11.0 in FE when compiling with Apache Hive3 .. Patch Set 2: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/12113/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/19400 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Idcdae97920241429c6dd0bd2b4ac1debfbde9d23 Gerrit-Change-Number: 19400 Gerrit-PatchSet: 2 Gerrit-Owner: Quanlong Huang Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Fucun Chu Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Quanlong Huang Gerrit-Comment-Date: Fri, 06 Jan 2023 00:04:36 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-11827: do not cache admission control service's IP address in impalad
Abhishek Rawat has uploaded a new patch set (#2). (
http://gerrit.cloudera.org:8080/19403 )
Change subject: IMPALA-11827: do not cache admission control service's IP
address in impalad
..
IMPALA-11827: do not cache admission control service's IP address in impalad
The Impalad's ExecEnv caches resolved IP address for admission control
service. The Impala server runs a heart beat thread for admission
control service and it relies on the cached resolved address in ExecEnv.
This breaks when IP address for admission control services changes.
This patch removes the 'admission_service_address_' member and its
getter in the ExecEnv class and adds a GetAdmissionServiceAddress()
interface to dynamically get the resolved address, if global admission
control service is enabled ('admission_service_host' flag is set).
Testing: Manually tested by restarting global admission controller
service in a Kubernetes setup where it gets new IP address on startup.
Coordinator's admission heart beat thread was able to connect to the
admission control service using the new IP address. Also made sure that
the coordinator and admissiond communication was working as expected
upon admissiond restarts.
Change-Id: I09b4c52644f9e1b3c1f0e68c9900464d722517af
---
M be/src/runtime/exec-env.cc
M be/src/runtime/exec-env.h
M be/src/scheduling/admission-control-service.cc
M be/src/service/impala-server.cc
4 files changed, 21 insertions(+), 18 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/03/19403/2
--
To view, visit http://gerrit.cloudera.org:8080/19403
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: I09b4c52644f9e1b3c1f0e68c9900464d722517af
Gerrit-Change-Number: 19403
Gerrit-PatchSet: 2
Gerrit-Owner: Abhishek Rawat
Gerrit-Reviewer: Andrew Sherman
Gerrit-Reviewer: Impala Public Jenkins
Gerrit-Reviewer: Wenzhe Zhou
[Impala-ASF-CR] IMPALA-11827: do not cache admission control service's IP address in impalad
Impala Public Jenkins has posted comments on this change. (
http://gerrit.cloudera.org:8080/19403 )
Change subject: IMPALA-11827: do not cache admission control service's IP
address in impalad
..
Patch Set 1:
(1 comment)
http://gerrit.cloudera.org:8080/#/c/19403/1/be/src/runtime/exec-env.cc
File be/src/runtime/exec-env.cc:
http://gerrit.cloudera.org:8080/#/c/19403/1/be/src/runtime/exec-env.cc@621
PS1, Line 621: Status ExecEnv::GetAdmissionServiceAddress(NetworkAddressPB&
admission_service_address) const {
line too long (95 > 90)
--
To view, visit http://gerrit.cloudera.org:8080/19403
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I09b4c52644f9e1b3c1f0e68c9900464d722517af
Gerrit-Change-Number: 19403
Gerrit-PatchSet: 1
Gerrit-Owner: Abhishek Rawat
Gerrit-Reviewer: Andrew Sherman
Gerrit-Reviewer: Impala Public Jenkins
Gerrit-Reviewer: Wenzhe Zhou
Gerrit-Comment-Date: Thu, 05 Jan 2023 23:50:05 +
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-11827: do not cache admission control service's IP address in impalad
Abhishek Rawat has uploaded this change for review. (
http://gerrit.cloudera.org:8080/19403
Change subject: IMPALA-11827: do not cache admission control service's IP
address in impalad
..
IMPALA-11827: do not cache admission control service's IP address in impalad
The Impalad's ExecEnv caches resolved IP address for admission control
service. The Impala server runs a heart beat thread for admission
control service and it relies on the cached resolved address in ExecEnv.
This breaks when IP address for admission control services changes.
This patch removes the 'admission_service_address_' member and its
getter in the ExecEnv class and adds a GetAdmissionServiceAddress()
interface to dynamically get the resolved address, if global admission
control service is enabled ('admission_service_host' flag is set).
Testing: Manually tested by restarting global admission controller
service in a Kubernetes setup where it gets new IP address on startup.
Coordinator's admission heart beat thread was able to connect to the
admission control service using the new IP address. Also made sure that
the coordinator and admissiond communication was working as expected
upon admissiond restarts.
Change-Id: I09b4c52644f9e1b3c1f0e68c9900464d722517af
---
M be/src/runtime/exec-env.cc
M be/src/runtime/exec-env.h
M be/src/scheduling/admission-control-service.cc
M be/src/service/impala-server.cc
4 files changed, 20 insertions(+), 18 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/03/19403/1
--
To view, visit http://gerrit.cloudera.org:8080/19403
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newchange
Gerrit-Change-Id: I09b4c52644f9e1b3c1f0e68c9900464d722517af
Gerrit-Change-Number: 19403
Gerrit-PatchSet: 1
Gerrit-Owner: Abhishek Rawat
Gerrit-Reviewer: Andrew Sherman
Gerrit-Reviewer: Impala Public Jenkins
Gerrit-Reviewer: Wenzhe Zhou
[Impala-ASF-CR] IMPALA-11801: Use thrift 0.11.0 in FE when compiling with Apache Hive3
Quanlong Huang has posted comments on this change. ( http://gerrit.cloudera.org:8080/19400 ) Change subject: IMPALA-11801: Use thrift 0.11.0 in FE when compiling with Apache Hive3 .. Patch Set 2: (1 comment) http://gerrit.cloudera.org:8080/#/c/19400/1/bin/impala-config.sh File bin/impala-config.sh: http://gerrit.cloudera.org:8080/#/c/19400/1/bin/impala-config.sh@183 PS1, Line 183: ell/ext-py and > USE_APACHE_HIVE is only set in line 325 (which led to the build failure) Oops, sorry that I just verified the script with manually setting USE_APACHE_HIVE=true.. -- To view, visit http://gerrit.cloudera.org:8080/19400 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Idcdae97920241429c6dd0bd2b4ac1debfbde9d23 Gerrit-Change-Number: 19400 Gerrit-PatchSet: 2 Gerrit-Owner: Quanlong Huang Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Fucun Chu Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Quanlong Huang Gerrit-Comment-Date: Thu, 05 Jan 2023 23:48:27 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-11801: Use thrift 0.11.0 in FE when compiling with Apache Hive3
Hello Fucun Chu, Csaba Ringhofer, Impala Public Jenkins, I'd like you to reexamine a change. Please visit http://gerrit.cloudera.org:8080/19400 to look at the new patch set (#2). Change subject: IMPALA-11801: Use thrift 0.11.0 in FE when compiling with Apache Hive3 .. IMPALA-11801: Use thrift 0.11.0 in FE when compiling with Apache Hive3 HMS client of Apache Hive3 can't run on thrift versions >= 0.14. Impala fails to start when compiling with Apache Hive3. This patch switches back to the old thrift version (0.11.0) used in FE when compiling on Apache Hive3. Tests: - Compiled and launched Impala cluster on Apache Hive3. - Verified some simple queries. - Enabling all the tests will be continued in IMPALA-10871. Change-Id: Idcdae97920241429c6dd0bd2b4ac1debfbde9d23 --- M bin/impala-config.sh 1 file changed, 12 insertions(+), 3 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/00/19400/2 -- To view, visit http://gerrit.cloudera.org:8080/19400 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: Idcdae97920241429c6dd0bd2b4ac1debfbde9d23 Gerrit-Change-Number: 19400 Gerrit-PatchSet: 2 Gerrit-Owner: Quanlong Huang Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Fucun Chu Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Quanlong Huang
[Impala-ASF-CR] IMPALA-11808: Add support for reload event in catalogD
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/19378 ) Change subject: IMPALA-11808: Add support for reload event in catalogD .. Patch Set 16: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/12112/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/19378 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ic62d58837d356dc2113f3c0904228ac9de484136 Gerrit-Change-Number: 19378 Gerrit-PatchSet: 16 Gerrit-Owner: Sai Hemanth Gantasala Gerrit-Reviewer: Abhishek Rawat Gerrit-Reviewer: Aman Sinha Gerrit-Reviewer: Andrew Sherman Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Quanlong Huang Gerrit-Reviewer: Sai Hemanth Gantasala Gerrit-Comment-Date: Thu, 05 Jan 2023 23:37:18 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-11808: Add support for reload event in catalogD
Aman Sinha has posted comments on this change. ( http://gerrit.cloudera.org:8080/19378 ) Change subject: IMPALA-11808: Add support for reload event in catalogD .. Patch Set 16: Verified+1 Code-Review+1 Latest patchset LGTM. Carry forward previous +1 from Quanlong and Csaba and adding mine. Will bump to +2 after Andrew has another look. Also carry forward previous Verified +1 since the latest changes are not affecting the code functionality. -- To view, visit http://gerrit.cloudera.org:8080/19378 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ic62d58837d356dc2113f3c0904228ac9de484136 Gerrit-Change-Number: 19378 Gerrit-PatchSet: 16 Gerrit-Owner: Sai Hemanth Gantasala Gerrit-Reviewer: Abhishek Rawat Gerrit-Reviewer: Aman Sinha Gerrit-Reviewer: Andrew Sherman Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Quanlong Huang Gerrit-Reviewer: Sai Hemanth Gantasala Gerrit-Comment-Date: Thu, 05 Jan 2023 23:26:59 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-11808: Add support for reload event in catalogD
Sai Hemanth Gantasala has posted comments on this change. ( http://gerrit.cloudera.org:8080/19378 ) Change subject: IMPALA-11808: Add support for reload event in catalogD .. Patch Set 16: > Patch Set 15: > > (1 comment) Clarified the comment -- To view, visit http://gerrit.cloudera.org:8080/19378 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ic62d58837d356dc2113f3c0904228ac9de484136 Gerrit-Change-Number: 19378 Gerrit-PatchSet: 16 Gerrit-Owner: Sai Hemanth Gantasala Gerrit-Reviewer: Abhishek Rawat Gerrit-Reviewer: Aman Sinha Gerrit-Reviewer: Andrew Sherman Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Quanlong Huang Gerrit-Reviewer: Sai Hemanth Gantasala Gerrit-Comment-Date: Thu, 05 Jan 2023 23:16:18 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-11808: Add support for reload event in catalogD
Hello Quanlong Huang, Aman Sinha, Andrew Sherman, Abhishek Rawat, Csaba Ringhofer, Impala Public Jenkins, I'd like you to reexamine a change. Please visit http://gerrit.cloudera.org:8080/19378 to look at the new patch set (#16). Change subject: IMPALA-11808: Add support for reload event in catalogD .. IMPALA-11808: Add support for reload event in catalogD This patch supports a new event called reload event in catalogD. This event is used to update table/file metadata for a refresh or invalidate command in other replicas of catalogDs when one of the replicas issued refresh/invalidate command. HIVE-26838 is the Hive jira that adds support for this reload event. This feature is disabled by default using a config enable_reload_events. To use this feature set this config to true and impala will be able to fire reload events. The processing of reload events in the event processor is always enabled. There is an end-to-end test added for this feature which currently checks firing/creation of the reload event and self-event check in the event processor. TODO: end-to-end test should also test this reload event in the event processor. There is also a follow up jira IMPALA-11822 to track the optimization patch for this feature. Change-Id: Ic62d58837d356dc2113f3c0904228ac9de484136 --- M be/src/catalog/catalog-server.cc M be/src/util/backend-gflag-util.cc M common/thrift/BackendGflags.thrift M fe/src/compat-apache-hive-3/java/org/apache/impala/compat/MetastoreShim.java M fe/src/compat-hive-3/java/org/apache/impala/compat/MetastoreShim.java M fe/src/main/java/org/apache/impala/catalog/events/MetastoreEvents.java M fe/src/main/java/org/apache/impala/service/BackendConfig.java M fe/src/main/java/org/apache/impala/service/CatalogOpExecutor.java M tests/custom_cluster/test_events_custom_configs.py 9 files changed, 337 insertions(+), 2 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/78/19378/16 -- To view, visit http://gerrit.cloudera.org:8080/19378 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: Ic62d58837d356dc2113f3c0904228ac9de484136 Gerrit-Change-Number: 19378 Gerrit-PatchSet: 16 Gerrit-Owner: Sai Hemanth Gantasala Gerrit-Reviewer: Abhishek Rawat Gerrit-Reviewer: Aman Sinha Gerrit-Reviewer: Andrew Sherman Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Quanlong Huang Gerrit-Reviewer: Sai Hemanth Gantasala
[Impala-ASF-CR] IMPALA-11375 Impala shell outputs details of each RPC
Jason Fehr has posted comments on this change. ( http://gerrit.cloudera.org:8080/19388 ) Change subject: IMPALA-11375 Impala shell outputs details of each RPC .. Patch Set 12: Code-Review+1 carry forward +1 for very minor code modification -- To view, visit http://gerrit.cloudera.org:8080/19388 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I36f8dbc96726aa2a573133acbe8a558299381f8b Gerrit-Change-Number: 19388 Gerrit-PatchSet: 12 Gerrit-Owner: Jason Fehr Gerrit-Reviewer: Abhishek Rawat Gerrit-Reviewer: Andrew Sherman Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Jason Fehr Gerrit-Comment-Date: Thu, 05 Jan 2023 22:35:24 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-11808: Add support for reload event in catalogD
Sai Hemanth Gantasala has posted comments on this change. ( http://gerrit.cloudera.org:8080/19378 ) Change subject: IMPALA-11808: Add support for reload event in catalogD .. Patch Set 15: (1 comment) http://gerrit.cloudera.org:8080/#/c/19378/13/be/src/catalog/catalog-server.cc File be/src/catalog/catalog-server.cc: http://gerrit.cloudera.org:8080/#/c/19378/13/be/src/catalog/catalog-server.cc@128 PS13, Line 128: "to fire a refresh/invalidate table event to the HMS and other event processors " > OK still confused. "to fire a refresh/invalidate table event to the HMS and any other event processors (such as other Impala catalogds) so they can process this event." -- To view, visit http://gerrit.cloudera.org:8080/19378 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ic62d58837d356dc2113f3c0904228ac9de484136 Gerrit-Change-Number: 19378 Gerrit-PatchSet: 15 Gerrit-Owner: Sai Hemanth Gantasala Gerrit-Reviewer: Abhishek Rawat Gerrit-Reviewer: Aman Sinha Gerrit-Reviewer: Andrew Sherman Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Quanlong Huang Gerrit-Reviewer: Sai Hemanth Gantasala Gerrit-Comment-Date: Thu, 05 Jan 2023 22:25:30 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-11808: Add support for reload event in catalogD
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/19378 ) Change subject: IMPALA-11808: Add support for reload event in catalogD .. Patch Set 15: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/12111/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/19378 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ic62d58837d356dc2113f3c0904228ac9de484136 Gerrit-Change-Number: 19378 Gerrit-PatchSet: 15 Gerrit-Owner: Sai Hemanth Gantasala Gerrit-Reviewer: Abhishek Rawat Gerrit-Reviewer: Aman Sinha Gerrit-Reviewer: Andrew Sherman Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Quanlong Huang Gerrit-Reviewer: Sai Hemanth Gantasala Gerrit-Comment-Date: Thu, 05 Jan 2023 22:20:00 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-11808: Add support for reload event in catalogD
Andrew Sherman has posted comments on this change. ( http://gerrit.cloudera.org:8080/19378 ) Change subject: IMPALA-11808: Add support for reload event in catalogD .. Patch Set 13: (3 comments) http://gerrit.cloudera.org:8080/#/c/19378/13/be/src/catalog/catalog-server.cc File be/src/catalog/catalog-server.cc: http://gerrit.cloudera.org:8080/#/c/19378/13/be/src/catalog/catalog-server.cc@128 PS13, Line 128: "to fire a refresh/invalidate table event to the HMS and other event processors " > Let's say there are two replicas of impala talking to HMS, One impala servi OK still confused. Is it "to fire a refresh/invalidate table event to the HMS which in turn generates events so that other event processors (such as other Impala catalogds) can process this event." or "to fire a refresh/invalidate table event to the HMS and any other event processors (such as other Impala catalogds) so they can process this event." or something else :-) http://gerrit.cloudera.org:8080/#/c/19378/13/fe/src/compat-apache-hive-3/java/org/apache/impala/compat/MetastoreShim.java File fe/src/compat-apache-hive-3/java/org/apache/impala/compat/MetastoreShim.java: http://gerrit.cloudera.org:8080/#/c/19378/13/fe/src/compat-apache-hive-3/java/org/apache/impala/compat/MetastoreShim.java@429 PS13, Line 429:* CDP Hive-3 only function. > Just following some other APIs comments in the same class. Refer L#552 L#56 OK http://gerrit.cloudera.org:8080/#/c/19378/13/fe/src/compat-apache-hive-3/java/org/apache/impala/compat/MetastoreShim.java@439 PS13, Line 439:* CDP Hive-3 only function. > Just following some other APIs comments in the same class. Refer L#552 L#56 OK -- To view, visit http://gerrit.cloudera.org:8080/19378 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ic62d58837d356dc2113f3c0904228ac9de484136 Gerrit-Change-Number: 19378 Gerrit-PatchSet: 13 Gerrit-Owner: Sai Hemanth Gantasala Gerrit-Reviewer: Abhishek Rawat Gerrit-Reviewer: Aman Sinha Gerrit-Reviewer: Andrew Sherman Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Quanlong Huang Gerrit-Reviewer: Sai Hemanth Gantasala Gerrit-Comment-Date: Thu, 05 Jan 2023 22:13:09 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-11808: Add support for reload event in catalogD
Sai Hemanth Gantasala has posted comments on this change. ( http://gerrit.cloudera.org:8080/19378 ) Change subject: IMPALA-11808: Add support for reload event in catalogD .. Patch Set 15: (7 comments) http://gerrit.cloudera.org:8080/#/c/19378/13//COMMIT_MSG Commit Message: http://gerrit.cloudera.org:8080/#/c/19378/13//COMMIT_MSG@14 PS13, Line 14: feature is disabled by default using a config > Nit: "by" Ack http://gerrit.cloudera.org:8080/#/c/19378/13/be/src/catalog/catalog-server.cc File be/src/catalog/catalog-server.cc: http://gerrit.cloudera.org:8080/#/c/19378/13/be/src/catalog/catalog-server.cc@128 PS13, Line 128: "to fire a refresh/invalidate table event to the HMS and other event processors " > I don't understand this sentence. Let's say there are two replicas of impala talking to HMS, One impala service would fire the event and the other impala service (event processor) would process this event. Does it make sense? http://gerrit.cloudera.org:8080/#/c/19378/13/fe/src/compat-apache-hive-3/java/org/apache/impala/compat/MetastoreShim.java File fe/src/compat-apache-hive-3/java/org/apache/impala/compat/MetastoreShim.java: http://gerrit.cloudera.org:8080/#/c/19378/13/fe/src/compat-apache-hive-3/java/org/apache/impala/compat/MetastoreShim.java@429 PS13, Line 429:* CDP Hive-3 only function. > Should we mention CDP in upstream code? Just following some other APIs comments in the same class. Refer L#552 L#560 L#568 L#577 http://gerrit.cloudera.org:8080/#/c/19378/13/fe/src/compat-apache-hive-3/java/org/apache/impala/compat/MetastoreShim.java@439 PS13, Line 439:* CDP Hive-3 only function. > Should we mention CDP in upstream code? Just following some other APIs comments in the same class. Refer L#552 L#560 L#568 L#577 http://gerrit.cloudera.org:8080/#/c/19378/13/fe/src/compat-hive-3/java/org/apache/impala/compat/MetastoreShim.java File fe/src/compat-hive-3/java/org/apache/impala/compat/MetastoreShim.java: http://gerrit.cloudera.org:8080/#/c/19378/13/fe/src/compat-hive-3/java/org/apache/impala/compat/MetastoreShim.java@505 PS13, Line 505:* Fires a reload event to HMS notification log. In Hive-3 the relaod event > Nit: "a reload", "the reload event" Ack http://gerrit.cloudera.org:8080/#/c/19378/13/tests/custom_cluster/test_events_custom_configs.py File tests/custom_cluster/test_events_custom_configs.py: http://gerrit.cloudera.org:8080/#/c/19378/13/tests/custom_cluster/test_events_custom_configs.py@246 PS13, Line 246: > Some spaces here that could be deleted? Ack http://gerrit.cloudera.org:8080/#/c/19378/13/tests/custom_cluster/test_events_custom_configs.py@252 PS13, Line 252: # Check if there is a reload event fired after refresh query. > Nit: "a reload event" Ack -- To view, visit http://gerrit.cloudera.org:8080/19378 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ic62d58837d356dc2113f3c0904228ac9de484136 Gerrit-Change-Number: 19378 Gerrit-PatchSet: 15 Gerrit-Owner: Sai Hemanth Gantasala Gerrit-Reviewer: Abhishek Rawat Gerrit-Reviewer: Aman Sinha Gerrit-Reviewer: Andrew Sherman Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Quanlong Huang Gerrit-Reviewer: Sai Hemanth Gantasala Gerrit-Comment-Date: Thu, 05 Jan 2023 22:00:03 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-11808: Add support for reload event in catalogD
Hello Quanlong Huang, Aman Sinha, Andrew Sherman, Abhishek Rawat, Csaba Ringhofer, Impala Public Jenkins, I'd like you to reexamine a change. Please visit http://gerrit.cloudera.org:8080/19378 to look at the new patch set (#15). Change subject: IMPALA-11808: Add support for reload event in catalogD .. IMPALA-11808: Add support for reload event in catalogD This patch supports a new event called reload event in catalogD. This event is used to update table/file metadata for a refresh or invalidate command in other replicas of catalogDs when one of the replicas issued refresh/invalidate command. HIVE-26838 is the Hive jira that adds support for this reload event. This feature is disabled by default using a config enable_reload_events. To use this feature set this config to true and impala will be able to fire reload events. The processing of reload events in the event processor is always enabled. There is an end-to-end test added for this feature which currently checks firing/creation of the reload event and self-event check in the event processor. TODO: end-to-end test should also test this reload event in the event processor. There is also a follow up jira IMPALA-11822 to track the optimization patch for this feature. Change-Id: Ic62d58837d356dc2113f3c0904228ac9de484136 --- M be/src/catalog/catalog-server.cc M be/src/util/backend-gflag-util.cc M common/thrift/BackendGflags.thrift M fe/src/compat-apache-hive-3/java/org/apache/impala/compat/MetastoreShim.java M fe/src/compat-hive-3/java/org/apache/impala/compat/MetastoreShim.java M fe/src/main/java/org/apache/impala/catalog/events/MetastoreEvents.java M fe/src/main/java/org/apache/impala/service/BackendConfig.java M fe/src/main/java/org/apache/impala/service/CatalogOpExecutor.java M tests/custom_cluster/test_events_custom_configs.py 9 files changed, 336 insertions(+), 2 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/78/19378/15 -- To view, visit http://gerrit.cloudera.org:8080/19378 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: Ic62d58837d356dc2113f3c0904228ac9de484136 Gerrit-Change-Number: 19378 Gerrit-PatchSet: 15 Gerrit-Owner: Sai Hemanth Gantasala Gerrit-Reviewer: Abhishek Rawat Gerrit-Reviewer: Aman Sinha Gerrit-Reviewer: Andrew Sherman Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Quanlong Huang Gerrit-Reviewer: Sai Hemanth Gantasala
[Impala-ASF-CR] IMPALA-11808: Add support for reload event in catalogD
Sai Hemanth Gantasala has posted comments on this change. ( http://gerrit.cloudera.org:8080/19378 ) Change subject: IMPALA-11808: Add support for reload event in catalogD .. Patch Set 14: (2 comments) http://gerrit.cloudera.org:8080/#/c/19378/13//COMMIT_MSG Commit Message: http://gerrit.cloudera.org:8080/#/c/19378/13//COMMIT_MSG@10 PS13, Line 10: to upd > nit: to update .. Ack http://gerrit.cloudera.org:8080/#/c/19378/13//COMMIT_MSG@11 PS13, Line 11: en one > nit: one of the .. Ack -- To view, visit http://gerrit.cloudera.org:8080/19378 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ic62d58837d356dc2113f3c0904228ac9de484136 Gerrit-Change-Number: 19378 Gerrit-PatchSet: 14 Gerrit-Owner: Sai Hemanth Gantasala Gerrit-Reviewer: Abhishek Rawat Gerrit-Reviewer: Aman Sinha Gerrit-Reviewer: Andrew Sherman Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Quanlong Huang Gerrit-Reviewer: Sai Hemanth Gantasala Gerrit-Comment-Date: Thu, 05 Jan 2023 21:44:19 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-11808: Add support for reload event in catalogD
Hello Quanlong Huang, Aman Sinha, Andrew Sherman, Abhishek Rawat, Csaba Ringhofer, Impala Public Jenkins, I'd like you to reexamine a change. Please visit http://gerrit.cloudera.org:8080/19378 to look at the new patch set (#14). Change subject: IMPALA-11808: Add support for reload event in catalogD .. IMPALA-11808: Add support for reload event in catalogD This patch supports a new event called reload event in catalogD. This event is used to update table/file metadata for a refresh or invalidate command in other replicas of catalogDs when one of the replicas issued refresh/invalidate command. HIVE-26838 is the Hive jira that adds support for this reload event. This feature is disabled dy default using a config enable_reload_events. To use this feature set this config to true and impala will be able to fire reload events. The processing of reload events in the event processor is always enabled. There is an end-to-end test added for this feature which currently checks firing/creation of the reload event and self-event check in the event processor. TODO: end-to-end test should also test this reload event in the event processor. There is also a follow up jira IMPALA-11822 to track the optimization patch for this feature. Change-Id: Ic62d58837d356dc2113f3c0904228ac9de484136 --- M be/src/catalog/catalog-server.cc M be/src/util/backend-gflag-util.cc M common/thrift/BackendGflags.thrift M fe/src/compat-apache-hive-3/java/org/apache/impala/compat/MetastoreShim.java M fe/src/compat-hive-3/java/org/apache/impala/compat/MetastoreShim.java M fe/src/main/java/org/apache/impala/catalog/events/MetastoreEvents.java M fe/src/main/java/org/apache/impala/service/BackendConfig.java M fe/src/main/java/org/apache/impala/service/CatalogOpExecutor.java M tests/custom_cluster/test_events_custom_configs.py 9 files changed, 336 insertions(+), 2 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/78/19378/14 -- To view, visit http://gerrit.cloudera.org:8080/19378 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: Ic62d58837d356dc2113f3c0904228ac9de484136 Gerrit-Change-Number: 19378 Gerrit-PatchSet: 14 Gerrit-Owner: Sai Hemanth Gantasala Gerrit-Reviewer: Abhishek Rawat Gerrit-Reviewer: Aman Sinha Gerrit-Reviewer: Andrew Sherman Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Quanlong Huang Gerrit-Reviewer: Sai Hemanth Gantasala
[Impala-ASF-CR] IMPALA-11808: Add support for reload event in catalogD
Andrew Sherman has posted comments on this change. ( http://gerrit.cloudera.org:8080/19378 ) Change subject: IMPALA-11808: Add support for reload event in catalogD .. Patch Set 13: (7 comments) Some spelling nits http://gerrit.cloudera.org:8080/#/c/19378/13//COMMIT_MSG Commit Message: http://gerrit.cloudera.org:8080/#/c/19378/13//COMMIT_MSG@14 PS13, Line 14: feature is disabled dy default using a config Nit: "by" http://gerrit.cloudera.org:8080/#/c/19378/13/be/src/catalog/catalog-server.cc File be/src/catalog/catalog-server.cc: http://gerrit.cloudera.org:8080/#/c/19378/13/be/src/catalog/catalog-server.cc@128 PS13, Line 128: "to fire a refresh/invalidate table event to the HMS and other event processors " I don't understand this sentence. Should "and" be "so"? http://gerrit.cloudera.org:8080/#/c/19378/13/fe/src/compat-apache-hive-3/java/org/apache/impala/compat/MetastoreShim.java File fe/src/compat-apache-hive-3/java/org/apache/impala/compat/MetastoreShim.java: http://gerrit.cloudera.org:8080/#/c/19378/13/fe/src/compat-apache-hive-3/java/org/apache/impala/compat/MetastoreShim.java@429 PS13, Line 429:* CDP Hive-3 only function. Should we mention CDP in upstream code? http://gerrit.cloudera.org:8080/#/c/19378/13/fe/src/compat-apache-hive-3/java/org/apache/impala/compat/MetastoreShim.java@439 PS13, Line 439:* CDP Hive-3 only function. Should we mention CDP in upstream code? http://gerrit.cloudera.org:8080/#/c/19378/13/fe/src/compat-hive-3/java/org/apache/impala/compat/MetastoreShim.java File fe/src/compat-hive-3/java/org/apache/impala/compat/MetastoreShim.java: http://gerrit.cloudera.org:8080/#/c/19378/13/fe/src/compat-hive-3/java/org/apache/impala/compat/MetastoreShim.java@505 PS13, Line 505:* Fires an reload event to HMS notification log. In Hive-3 the relaod event Nit: "a reload", "the reload event" http://gerrit.cloudera.org:8080/#/c/19378/13/tests/custom_cluster/test_events_custom_configs.py File tests/custom_cluster/test_events_custom_configs.py: http://gerrit.cloudera.org:8080/#/c/19378/13/tests/custom_cluster/test_events_custom_configs.py@246 PS13, Line 246: Some spaces here that could be deleted? http://gerrit.cloudera.org:8080/#/c/19378/13/tests/custom_cluster/test_events_custom_configs.py@252 PS13, Line 252: # Check if there is an reload event fired after refresh query. Nit: "a reload event" -- To view, visit http://gerrit.cloudera.org:8080/19378 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ic62d58837d356dc2113f3c0904228ac9de484136 Gerrit-Change-Number: 19378 Gerrit-PatchSet: 13 Gerrit-Owner: Sai Hemanth Gantasala Gerrit-Reviewer: Abhishek Rawat Gerrit-Reviewer: Aman Sinha Gerrit-Reviewer: Andrew Sherman Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Quanlong Huang Gerrit-Reviewer: Sai Hemanth Gantasala Gerrit-Comment-Date: Thu, 05 Jan 2023 21:42:37 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-11808: Add support for reload event in catalogD
Aman Sinha has posted comments on this change. ( http://gerrit.cloudera.org:8080/19378 ) Change subject: IMPALA-11808: Add support for reload event in catalogD .. Patch Set 13: (2 comments) http://gerrit.cloudera.org:8080/#/c/19378/13//COMMIT_MSG Commit Message: http://gerrit.cloudera.org:8080/#/c/19378/13//COMMIT_MSG@10 PS13, Line 10: update nit: to update .. http://gerrit.cloudera.org:8080/#/c/19378/13//COMMIT_MSG@11 PS13, Line 11: one the nit: one of the .. -- To view, visit http://gerrit.cloudera.org:8080/19378 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ic62d58837d356dc2113f3c0904228ac9de484136 Gerrit-Change-Number: 19378 Gerrit-PatchSet: 13 Gerrit-Owner: Sai Hemanth Gantasala Gerrit-Reviewer: Abhishek Rawat Gerrit-Reviewer: Aman Sinha Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Quanlong Huang Gerrit-Reviewer: Sai Hemanth Gantasala Gerrit-Comment-Date: Thu, 05 Jan 2023 21:41:27 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-11808: Add support for reload event in catalogD
Sai Hemanth Gantasala has posted comments on this change. ( http://gerrit.cloudera.org:8080/19378 ) Change subject: IMPALA-11808: Add support for reload event in catalogD .. Patch Set 13: (3 comments) http://gerrit.cloudera.org:8080/#/c/19378/12//COMMIT_MSG Commit Message: http://gerrit.cloudera.org:8080/#/c/19378/12//COMMIT_MSG@13 PS12, Line 13: adds > nit: adds Ack http://gerrit.cloudera.org:8080/#/c/19378/12//COMMIT_MSG@14 PS12, Line 14: disable > nit: disabled Ack http://gerrit.cloudera.org:8080/#/c/19378/12//COMMIT_MSG@17 PS12, Line 17: of rel > nit: 'of' reload events.. Ack -- To view, visit http://gerrit.cloudera.org:8080/19378 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ic62d58837d356dc2113f3c0904228ac9de484136 Gerrit-Change-Number: 19378 Gerrit-PatchSet: 13 Gerrit-Owner: Sai Hemanth Gantasala Gerrit-Reviewer: Abhishek Rawat Gerrit-Reviewer: Aman Sinha Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Quanlong Huang Gerrit-Reviewer: Sai Hemanth Gantasala Gerrit-Comment-Date: Thu, 05 Jan 2023 21:33:08 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-11808: Add support for reload event in catalogD
Hello Quanlong Huang, Aman Sinha, Abhishek Rawat, Csaba Ringhofer, Impala Public Jenkins, I'd like you to reexamine a change. Please visit http://gerrit.cloudera.org:8080/19378 to look at the new patch set (#13). Change subject: IMPALA-11808: Add support for reload event in catalogD .. IMPALA-11808: Add support for reload event in catalogD This patch supports a new event called reload event in catalogD. This event is used update table/file metadata for a refresh or invalidate command in other replicas of catalogDs when one the replicas issued refresh/invalidate command. HIVE-26838 is the Hive jira that adds support for this reload event. This feature is disabled dy default using a config enable_reload_events. To use this feature set this config to true and impala will be able to fire reload events. The processing of reload events in the event processor is always enabled. There is an end-to-end test added for this feature which currently checks firing/creation of the reload event and self-event check in the event processor. TODO: end-to-end test should also test this reload event in the event processor. There is also a follow up jira IMPALA-11822 to track the optimization patch for this feature. Change-Id: Ic62d58837d356dc2113f3c0904228ac9de484136 --- M be/src/catalog/catalog-server.cc M be/src/util/backend-gflag-util.cc M common/thrift/BackendGflags.thrift M fe/src/compat-apache-hive-3/java/org/apache/impala/compat/MetastoreShim.java M fe/src/compat-hive-3/java/org/apache/impala/compat/MetastoreShim.java M fe/src/main/java/org/apache/impala/catalog/events/MetastoreEvents.java M fe/src/main/java/org/apache/impala/service/BackendConfig.java M fe/src/main/java/org/apache/impala/service/CatalogOpExecutor.java M tests/custom_cluster/test_events_custom_configs.py 9 files changed, 336 insertions(+), 2 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/78/19378/13 -- To view, visit http://gerrit.cloudera.org:8080/19378 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: Ic62d58837d356dc2113f3c0904228ac9de484136 Gerrit-Change-Number: 19378 Gerrit-PatchSet: 13 Gerrit-Owner: Sai Hemanth Gantasala Gerrit-Reviewer: Abhishek Rawat Gerrit-Reviewer: Aman Sinha Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Quanlong Huang Gerrit-Reviewer: Sai Hemanth Gantasala
[Impala-ASF-CR] IMPALA-11375 Impala shell outputs details of each RPC
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/19388 ) Change subject: IMPALA-11375 Impala shell outputs details of each RPC .. Patch Set 12: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/12110/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/19388 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I36f8dbc96726aa2a573133acbe8a558299381f8b Gerrit-Change-Number: 19388 Gerrit-PatchSet: 12 Gerrit-Owner: Jason Fehr Gerrit-Reviewer: Abhishek Rawat Gerrit-Reviewer: Andrew Sherman Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Jason Fehr Gerrit-Comment-Date: Thu, 05 Jan 2023 21:35:55 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-11808: Add support for reload event in catalogD
Aman Sinha has posted comments on this change. ( http://gerrit.cloudera.org:8080/19378 ) Change subject: IMPALA-11808: Add support for reload event in catalogD .. Patch Set 12: (3 comments) http://gerrit.cloudera.org:8080/#/c/19378/12//COMMIT_MSG Commit Message: http://gerrit.cloudera.org:8080/#/c/19378/12//COMMIT_MSG@13 PS12, Line 13: addes nit: adds http://gerrit.cloudera.org:8080/#/c/19378/12//COMMIT_MSG@14 PS12, Line 14: disable nit: disabled http://gerrit.cloudera.org:8080/#/c/19378/12//COMMIT_MSG@17 PS12, Line 17: reload nit: 'of' reload events.. -- To view, visit http://gerrit.cloudera.org:8080/19378 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ic62d58837d356dc2113f3c0904228ac9de484136 Gerrit-Change-Number: 19378 Gerrit-PatchSet: 12 Gerrit-Owner: Sai Hemanth Gantasala Gerrit-Reviewer: Abhishek Rawat Gerrit-Reviewer: Aman Sinha Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Quanlong Huang Gerrit-Reviewer: Sai Hemanth Gantasala Gerrit-Comment-Date: Thu, 05 Jan 2023 21:16:53 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-11375 Impala shell outputs details of each RPC
Jason Fehr has uploaded a new patch set (#12). ( http://gerrit.cloudera.org:8080/19388 ) Change subject: IMPALA-11375 Impala shell outputs details of each RPC .. IMPALA-11375 Impala shell outputs details of each RPC When the Impala shell is using the hs2 protocol, it makes multiple RPCs to the Impala daemon. These calls pass Thrift objects back and forth. This change adds the '--show_rpc' which outputs the details of the RPCs to stdout and the '--rpc_file' flag which outputs the RPC details to the specified file path. RPC details include: - operation name - request attempt count - Impala session/query ids (if applicable) - call duration - call status (success/failure) - request Thrift objects - response Thrift objects Certain information is not included in the RPC details: - Thrift object attributes named 'secret' or 'password' are redacted. - Thrift objects with a type of TRowSet or TGetRuntimeProfileResp are not include as the information contained within them is already available in the standard output from the Impala shell. Testing: - Added new tests in the end-to-end test suite. Change-Id: I36f8dbc96726aa2a573133acbe8a558299381f8b --- M bin/impala-shell.sh M shell/impala_client.py M shell/impala_shell.py M shell/impala_shell_config_defaults.py M shell/make_shell_tarball.sh M shell/option_parser.py M shell/packaging/make_python_package.sh A shell/thrift_printer.py M tests/shell/test_shell_commandline.py 9 files changed, 556 insertions(+), 147 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/88/19388/12 -- To view, visit http://gerrit.cloudera.org:8080/19388 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: I36f8dbc96726aa2a573133acbe8a558299381f8b Gerrit-Change-Number: 19388 Gerrit-PatchSet: 12 Gerrit-Owner: Jason Fehr Gerrit-Reviewer: Abhishek Rawat Gerrit-Reviewer: Andrew Sherman Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Jason Fehr
[Impala-ASF-CR] IMPALA-11375 Impala shell outputs details of each RPC
Jason Fehr has posted comments on this change. ( http://gerrit.cloudera.org:8080/19388 ) Change subject: IMPALA-11375 Impala shell outputs details of each RPC .. Patch Set 11: (2 comments) addressed comments http://gerrit.cloudera.org:8080/#/c/19388/11//COMMIT_MSG Commit Message: http://gerrit.cloudera.org:8080/#/c/19388/11//COMMIT_MSG@2 PS11, Line 2: Author: jasonmfehr > Nit: Author still shows gmail address I was going to fix this on future gerrit changes only, but since there is another code change needed on this PR I am going to amend the commit and fix the email. http://gerrit.cloudera.org:8080/#/c/19388/11/shell/option_parser.py File shell/option_parser.py: http://gerrit.cloudera.org:8080/#/c/19388/11/shell/option_parser.py@216 PS11, Line 216: "file. Ignored if protocol is beeswax.") > Nit: delete double space Done -- To view, visit http://gerrit.cloudera.org:8080/19388 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I36f8dbc96726aa2a573133acbe8a558299381f8b Gerrit-Change-Number: 19388 Gerrit-PatchSet: 11 Gerrit-Owner: Jason Fehr Gerrit-Reviewer: Abhishek Rawat Gerrit-Reviewer: Andrew Sherman Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Jason Fehr Gerrit-Comment-Date: Thu, 05 Jan 2023 21:15:50 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-11375 Impala shell outputs details of each RPC
Andrew Sherman has posted comments on this change. ( http://gerrit.cloudera.org:8080/19388 ) Change subject: IMPALA-11375 Impala shell outputs details of each RPC .. Patch Set 11: Code-Review+1 (3 comments) LGTM http://gerrit.cloudera.org:8080/#/c/19388/11//COMMIT_MSG Commit Message: http://gerrit.cloudera.org:8080/#/c/19388/11//COMMIT_MSG@2 PS11, Line 2: Author: jasonmfehr Nit: Author still shows gmail address http://gerrit.cloudera.org:8080/#/c/19388/9/shell/impala_client.py File shell/impala_client.py: http://gerrit.cloudera.org:8080/#/c/19388/9/shell/impala_client.py@776 PS9, Line 776: key_value_pairs = set_query_options.items() > That's actually the formatting used elsewhere, thus I did not change it. ok http://gerrit.cloudera.org:8080/#/c/19388/11/shell/option_parser.py File shell/option_parser.py: http://gerrit.cloudera.org:8080/#/c/19388/11/shell/option_parser.py@216 PS11, Line 216: "file. Ignored if protocol is beeswax.") Nit: delete double space -- To view, visit http://gerrit.cloudera.org:8080/19388 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I36f8dbc96726aa2a573133acbe8a558299381f8b Gerrit-Change-Number: 19388 Gerrit-PatchSet: 11 Gerrit-Owner: Jason Fehr Gerrit-Reviewer: Abhishek Rawat Gerrit-Reviewer: Andrew Sherman Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Jason Fehr Gerrit-Comment-Date: Thu, 05 Jan 2023 20:16:05 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-11375 Impala shell outputs details of each RPC
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/19388 ) Change subject: IMPALA-11375 Impala shell outputs details of each RPC .. Patch Set 11: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/12109/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/19388 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I36f8dbc96726aa2a573133acbe8a558299381f8b Gerrit-Change-Number: 19388 Gerrit-PatchSet: 11 Gerrit-Owner: Jason Fehr Gerrit-Reviewer: Abhishek Rawat Gerrit-Reviewer: Andrew Sherman Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Jason Fehr Gerrit-Comment-Date: Thu, 05 Jan 2023 18:20:47 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-11375 Impala shell outputs details of each RPC
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/19388 ) Change subject: IMPALA-11375 Impala shell outputs details of each RPC .. Patch Set 10: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/12108/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/19388 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I36f8dbc96726aa2a573133acbe8a558299381f8b Gerrit-Change-Number: 19388 Gerrit-PatchSet: 10 Gerrit-Owner: Jason Fehr Gerrit-Reviewer: Abhishek Rawat Gerrit-Reviewer: Andrew Sherman Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Jason Fehr Gerrit-Comment-Date: Thu, 05 Jan 2023 18:11:45 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-11375 Impala shell outputs details of each RPC
Jason Fehr has uploaded a new patch set (#11). ( http://gerrit.cloudera.org:8080/19388 ) Change subject: IMPALA-11375 Impala shell outputs details of each RPC .. IMPALA-11375 Impala shell outputs details of each RPC When the Impala shell is using the hs2 protocol, it makes multiple RPCs to the Impala daemon. These calls pass Thrift objects back and forth. This change adds the '--show_rpc' which outputs the details of the RPCs to stdout and the '--rpc_file' flag which outputs the RPC details to the specified file path. RPC details include: - operation name - request attempt count - Impala session/query ids (if applicable) - call duration - call status (success/failure) - request Thrift objects - response Thrift objects Certain information is not included in the RPC details: - Thrift object attributes named 'secret' or 'password' are redacted. - Thrift objects with a type of TRowSet or TGetRuntimeProfileResp are not include as the information contained within them is already available in the standard output from the Impala shell. Testing: - Added new tests in the end-to-end test suite. Change-Id: I36f8dbc96726aa2a573133acbe8a558299381f8b --- M bin/impala-shell.sh M shell/impala_client.py M shell/impala_shell.py M shell/impala_shell_config_defaults.py M shell/make_shell_tarball.sh M shell/option_parser.py M shell/packaging/make_python_package.sh A shell/thrift_printer.py M tests/shell/test_shell_commandline.py 9 files changed, 556 insertions(+), 147 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/88/19388/11 -- To view, visit http://gerrit.cloudera.org:8080/19388 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: I36f8dbc96726aa2a573133acbe8a558299381f8b Gerrit-Change-Number: 19388 Gerrit-PatchSet: 11 Gerrit-Owner: Jason Fehr Gerrit-Reviewer: Abhishek Rawat Gerrit-Reviewer: Andrew Sherman Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Jason Fehr
[Impala-ASF-CR] IMPALA-11375 Impala shell outputs details of each RPC
Jason Fehr has posted comments on this change. (
http://gerrit.cloudera.org:8080/19388 )
Change subject: IMPALA-11375 Impala shell outputs details of each RPC
..
Patch Set 9:
(22 comments)
comments addressed
http://gerrit.cloudera.org:8080/#/c/19388/9//COMMIT_MSG
Commit Message:
http://gerrit.cloudera.org:8080/#/c/19388/9//COMMIT_MSG@2
PS9, Line 2: Author: jasonmfehr
> If you want you can use a cloudera address
Next change will use my cloudera email address.
http://gerrit.cloudera.org:8080/#/c/19388/9//COMMIT_MSG@29
PS9, Line 29: already available in the standars output from the Impala shell.
> Nit: "standard"
Done
http://gerrit.cloudera.org:8080/#/c/19388/9/shell/impala_client.py
File shell/impala_client.py:
http://gerrit.cloudera.org:8080/#/c/19388/9/shell/impala_client.py@701
PS9, Line 701: resp = self._do_hs2_rpc(
> Nit: I think this style looks neater
Done
http://gerrit.cloudera.org:8080/#/c/19388/9/shell/impala_client.py@776
PS9, Line 776: # PingImpalaHS2Service rpc is idempotent and so safe to
retry.
> Nit: I expected a blank line before this
That's actually the formatting used elsewhere, thus I did not change it.
http://gerrit.cloudera.org:8080/#/c/19388/9/shell/impala_client.py@829
PS9, Line 829: self._clear_current_query_id()
> This method seems unusual in that it doesn't use try...finally for the curr
It was a miss on my part because this method is different. Adding the
try...finally
http://gerrit.cloudera.org:8080/#/c/19388/9/shell/impala_client.py@837
PS9, Line 837: self._current_query_id = self.get_query_id_str(query_handle)
> We always calculate the _current_query_id even when show_rpc or rpc_file ar
Done
http://gerrit.cloudera.org:8080/#/c/19388/9/shell/impala_client.py@1216
PS9, Line 1216: f = open(self.rpc_file, "a")
> It looks like we always open the file for append, so if I specify an existi
The file is opened in append mode. That way, it behaves more like a log file.
It's the safer option because existing data does not get deleted. If a user
accidentally specifies the wrong output file, their data won't be erased.
I added a sentence to the help text of the '--rpc_file' command line option
explaining that it appends to existing files.
http://gerrit.cloudera.org:8080/#/c/19388/9/shell/impala_client.py@1218
PS9, Line 1218: f.close()
> Better maybe:
Done
http://gerrit.cloudera.org:8080/#/c/19388/9/shell/impala_client.py@1246
PS9, Line 1246: f = open(self.rpc_file, "a")
> Use "with" as noted above?
Done
http://gerrit.cloudera.org:8080/#/c/19388/9/shell/option_parser.py
File shell/option_parser.py:
http://gerrit.cloudera.org:8080/#/c/19388/9/shell/option_parser.py@209
PS9, Line 209: parser.add_option("--show_rpc", dest="show_rpc",
> Reading this I thought that I always needed to set show_rpc, and that I set
I like this idea. Changing '--show_rpc' to '--rpc_stdout' and making changes
to the help text to make it clearer.
http://gerrit.cloudera.org:8080/#/c/19388/9/shell/option_parser.py@211
PS9, Line 211: help="Always show hs2 rpc details. Ignored
if protocol is beeswax")
> Nit: use 1 space?
Done
http://gerrit.cloudera.org:8080/#/c/19388/9/shell/option_parser.py@213
PS9, Line 213: help="If set, hs2 rpc call details are
writte to the given file. "
> Nit: "written"
Done
http://gerrit.cloudera.org:8080/#/c/19388/9/shell/thrift_printer.py
File shell/thrift_printer.py:
http://gerrit.cloudera.org:8080/#/c/19388/9/shell/thrift_printer.py@27
PS9, Line 27: to stdout, stderrr, or any other file handles."""
> Nit: "stderr"
Done
http://gerrit.cloudera.org:8080/#/c/19388/9/shell/thrift_printer.py@30
PS9, Line 30: # redacted_fields - list of names of object attributes who's
> Nit: "whose"
Done
http://gerrit.cloudera.org:8080/#/c/19388/9/shell/thrift_printer.py@33
PS9, Line 33: # base indentation where all other
indetations
> Nit: "indentations"
Done
http://gerrit.cloudera.org:8080/#/c/19388/9/shell/thrift_printer.py@36
PS9, Line 36: # will not be be printed out, useful to
ignore
> Nit: "will not be"
Done
http://gerrit.cloudera.org:8080/#/c/19388/9/shell/thrift_printer.py@56
PS9, Line 56: be anything that has a write(string) method on it.
> Nit: just "has a write(string) method" may be clearer?
Done
http://gerrit.cloudera.org:8080/#/c/19388/9/shell/thrift_printer.py@64
PS9, Line 64: # thrfit_obj - the object to print out, its attributes will
> Nit: "thrift_obj"
Done
http://gerrit.cloudera.org:8080/#/c/19388/9/shell/thrift_printer.py@69
PS9, Line 69: # write(str) method on it
> Nit: just "has a write(string) method" may be clearer?
Done
http://gerrit.cloudera.org:8080/#/c/19388/9/shell/thrift_printer.py@96
PS9, Line 96: # TODO -- print out attributes in alphabetical order
> Looks like this is done
[Impala-ASF-CR] IMPALA-11375 Impala shell outputs details of each RPC
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/19388 ) Change subject: IMPALA-11375 Impala shell outputs details of each RPC .. Patch Set 10: (2 comments) http://gerrit.cloudera.org:8080/#/c/19388/10/shell/impala_client.py File shell/impala_client.py: http://gerrit.cloudera.org:8080/#/c/19388/10/shell/impala_client.py@1258 PS10, Line 1258: flake8: W293 blank line contains whitespace http://gerrit.cloudera.org:8080/#/c/19388/10/shell/impala_client.py@1258 PS10, Line 1258: line has trailing whitespace -- To view, visit http://gerrit.cloudera.org:8080/19388 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I36f8dbc96726aa2a573133acbe8a558299381f8b Gerrit-Change-Number: 19388 Gerrit-PatchSet: 10 Gerrit-Owner: Jason Fehr Gerrit-Reviewer: Abhishek Rawat Gerrit-Reviewer: Andrew Sherman Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Jason Fehr Gerrit-Comment-Date: Thu, 05 Jan 2023 17:52:55 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-11375 Impala shell outputs details of each RPC
Jason Fehr has uploaded a new patch set (#10). ( http://gerrit.cloudera.org:8080/19388 ) Change subject: IMPALA-11375 Impala shell outputs details of each RPC .. IMPALA-11375 Impala shell outputs details of each RPC When the Impala shell is using the hs2 protocol, it makes multiple RPCs to the Impala daemon. These calls pass Thrift objects back and forth. This change adds the '--show_rpc' which outputs the details of the RPCs to stdout and the '--rpc_file' flag which outputs the RPC details to the specified file path. RPC details include: - operation name - request attempt count - Impala session/query ids (if applicable) - call duration - call status (success/failure) - request Thrift objects - response Thrift objects Certain information is not included in the RPC details: - Thrift object attributes named 'secret' or 'password' are redacted. - Thrift objects with a type of TRowSet or TGetRuntimeProfileResp are not include as the information contained within them is already available in the standard output from the Impala shell. Testing: - Added new tests in the end-to-end test suite. Change-Id: I36f8dbc96726aa2a573133acbe8a558299381f8b --- M bin/impala-shell.sh M shell/impala_client.py M shell/impala_shell.py M shell/impala_shell_config_defaults.py M shell/make_shell_tarball.sh M shell/option_parser.py M shell/packaging/make_python_package.sh A shell/thrift_printer.py M tests/shell/test_shell_commandline.py 9 files changed, 556 insertions(+), 147 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/88/19388/10 -- To view, visit http://gerrit.cloudera.org:8080/19388 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: I36f8dbc96726aa2a573133acbe8a558299381f8b Gerrit-Change-Number: 19388 Gerrit-PatchSet: 10 Gerrit-Owner: Jason Fehr Gerrit-Reviewer: Abhishek Rawat Gerrit-Reviewer: Andrew Sherman Gerrit-Reviewer: Impala Public Jenkins
[Impala-ASF-CR] IMPALA-9551: Allow mixed complex types in select list
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/19322 ) Change subject: IMPALA-9551: Allow mixed complex types in select list .. Patch Set 7: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/12107/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/19322 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I476d98884b5fd192dfcd4feeec7947526aebe993 Gerrit-Change-Number: 19322 Gerrit-PatchSet: 7 Gerrit-Owner: Daniel Becker Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Daniel Becker Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Michael Smith Gerrit-Reviewer: Peter Rozsa Gerrit-Comment-Date: Thu, 05 Jan 2023 17:08:08 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-10893: Use old schema during iceberg time travel.
Zoltan Borok-Nagy has posted comments on this change. (
http://gerrit.cloudera.org:8080/19380 )
Change subject: IMPALA-10893: Use old schema during iceberg time travel.
..
Patch Set 2:
(5 comments)
Left a few nit comments, otherwise the change looks great!
http://gerrit.cloudera.org:8080/#/c/19380/2/fe/src/main/java/org/apache/impala/analysis/Analyzer.java
File fe/src/main/java/org/apache/impala/analysis/Analyzer.java:
http://gerrit.cloudera.org:8080/#/c/19380/2/fe/src/main/java/org/apache/impala/analysis/Analyzer.java@1309
PS2, Line 1309: timeTravelTable.readSchema();
Could it be part of the constructor of IcebergTimeTravelTable?
http://gerrit.cloudera.org:8080/#/c/19380/2/fe/src/main/java/org/apache/impala/analysis/DescribeTableStmt.java
File fe/src/main/java/org/apache/impala/analysis/DescribeTableStmt.java:
http://gerrit.cloudera.org:8080/#/c/19380/2/fe/src/main/java/org/apache/impala/analysis/DescribeTableStmt.java@95
PS2, Line 95: , null)
Maybe we could overload Analyzer.resolvePath() so we wouldn't need to change
unrelated code paths.
http://gerrit.cloudera.org:8080/#/c/19380/2/fe/src/main/java/org/apache/impala/analysis/TableRef.java
File fe/src/main/java/org/apache/impala/analysis/TableRef.java:
http://gerrit.cloudera.org:8080/#/c/19380/2/fe/src/main/java/org/apache/impala/analysis/TableRef.java@471
PS2, Line 471: so we
: // cannot check it yet.
"so we cannot check if the table supports time travel"?
http://gerrit.cloudera.org:8080/#/c/19380/2/tests/authorization/test_ranger.py
File tests/authorization/test_ranger.py:
http://gerrit.cloudera.org:8080/#/c/19380/2/tests/authorization/test_ranger.py@1844
PS2, Line 1844:
nit: +2 indent
http://gerrit.cloudera.org:8080/#/c/19380/2/tests/query_test/test_iceberg.py
File tests/query_test/test_iceberg.py:
http://gerrit.cloudera.org:8080/#/c/19380/2/tests/query_test/test_iceberg.py@456
PS2, Line 456: # FIXME use new ts??
Please resolve this. Does the following query work?
SELECT * FROM {tbl} FOR SYSTEM_TIME AS OF
MINUS
SELECT *, NULL FROM {tbl} FOR SYSTEM_TIME AS OF
Or since the second subquery has NULL the whole query returns every row? Then
maybe we could
SELECT i FROM {tbl} FOR SYSTEM_TIME AS OF
MINUS
SELECT i FROM {tbl} FOR SYSTEM_TIME AS OF
--
To view, visit http://gerrit.cloudera.org:8080/19380
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I7cbef6e20bbb567e517744fb1f34d880970399ab
Gerrit-Change-Number: 19380
Gerrit-PatchSet: 2
Gerrit-Owner: Andrew Sherman
Gerrit-Reviewer: Gabor Kaszab
Gerrit-Reviewer: Gergely Fürnstáhl
Gerrit-Reviewer: Impala Public Jenkins
Gerrit-Reviewer: Tamas Mate
Gerrit-Reviewer: Zoltan Borok-Nagy
Gerrit-Comment-Date: Thu, 05 Jan 2023 16:57:08 +
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-9551: Allow mixed complex types in select list
Daniel Becker has uploaded a new patch set (#7). ( http://gerrit.cloudera.org:8080/19322 ) Change subject: IMPALA-9551: Allow mixed complex types in select list .. IMPALA-9551: Allow mixed complex types in select list Currently collections and structs are supported in the select list, also when they are nested (structs in structs and collections in collections), but mixing different kinds of complex types, i.e. having structs in collections or vice versa, is not supported. This patch adds support for mixed complex types in the select list. Testing: - Created a new test table, 'collection_struct_mix', that contains mixed complex types. - Added tests in mixed-collections-and-structs.test that test having mixed complex types in the select list. These tests are called from test_nested_types.py::TestMixedCollectionsAndStructsInSelectList. - Ran existing tests that test collections and structs in the select list; test queries that expected a failure in case of mixed complex types have been moved to mixed-collections-and-structs.test and now expect success. Change-Id: I476d98884b5fd192dfcd4feeec7947526aebe993 --- M be/src/exec/unnest-node.cc M be/src/exprs/slot-ref.h M be/src/runtime/complex-value-writer.h M be/src/runtime/complex-value-writer.inline.h M be/src/runtime/descriptors.cc M be/src/runtime/raw-value.cc M be/src/service/hs2-util.cc M be/src/service/query-result-set.cc M fe/src/main/java/org/apache/impala/analysis/Analyzer.java M fe/src/main/java/org/apache/impala/analysis/SlotDescriptor.java M fe/src/main/java/org/apache/impala/analysis/SlotRef.java M fe/src/main/java/org/apache/impala/analysis/TupleDescriptor.java M fe/src/main/java/org/apache/impala/planner/SingleNodePlanner.java M fe/src/test/java/org/apache/impala/analysis/AnalyzeStmtsTest.java M testdata/datasets/functional/functional_schema_template.sql M testdata/datasets/functional/schema_constraints.csv A testdata/workloads/functional-query/queries/QueryTest/mixed-collections-and-structs.test M testdata/workloads/functional-query/queries/QueryTest/struct-in-select-list.test M tests/query_test/test_nested_types.py 19 files changed, 590 insertions(+), 132 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/22/19322/7 -- To view, visit http://gerrit.cloudera.org:8080/19322 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: I476d98884b5fd192dfcd4feeec7947526aebe993 Gerrit-Change-Number: 19322 Gerrit-PatchSet: 7 Gerrit-Owner: Daniel Becker Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Daniel Becker Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Michael Smith Gerrit-Reviewer: Peter Rozsa
[Impala-ASF-CR] IMPALA-11808: Add support for reload event in catalogD
Hello Quanlong Huang, Abhishek Rawat, Csaba Ringhofer, Impala Public Jenkins, I'd like you to reexamine a change. Please visit http://gerrit.cloudera.org:8080/19378 to look at the new patch set (#12). Change subject: IMPALA-11808: Add support for reload event in catalogD .. IMPALA-11808: Add support for reload event in catalogD This patch supports a new event called reload event in catalogD. This event is used update table/file metadata for a refresh or invalidate command in other replicas of catalogDs when one the replicas issued refresh/invalidate command. HIVE-26838 is the Hive jira that addes support for this reload event. This feature is disable dy default using a config enable_reload_events. To use this feature set this config to true and impala will be able to fire reload events. The processing reload events in the event processor is always enabled. There is an end-to-end test added for this feature which currently checks firing/creation of the reload event and self-event check in the event processor. TODO: end-to-end test should also test this reload event in the event processor. There is also a follow up jira IMPALA-11822 to track the optimization patch for this feature. Change-Id: Ic62d58837d356dc2113f3c0904228ac9de484136 --- M be/src/catalog/catalog-server.cc M be/src/util/backend-gflag-util.cc M common/thrift/BackendGflags.thrift M fe/src/compat-apache-hive-3/java/org/apache/impala/compat/MetastoreShim.java M fe/src/compat-hive-3/java/org/apache/impala/compat/MetastoreShim.java M fe/src/main/java/org/apache/impala/catalog/events/MetastoreEvents.java M fe/src/main/java/org/apache/impala/service/BackendConfig.java M fe/src/main/java/org/apache/impala/service/CatalogOpExecutor.java M tests/custom_cluster/test_events_custom_configs.py 9 files changed, 336 insertions(+), 2 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/78/19378/12 -- To view, visit http://gerrit.cloudera.org:8080/19378 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: Ic62d58837d356dc2113f3c0904228ac9de484136 Gerrit-Change-Number: 19378 Gerrit-PatchSet: 12 Gerrit-Owner: Sai Hemanth Gantasala Gerrit-Reviewer: Abhishek Rawat Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Quanlong Huang Gerrit-Reviewer: Sai Hemanth Gantasala
[Impala-ASF-CR] IMPALA-11808: Add support for reload event in catalogD
Sai Hemanth Gantasala has posted comments on this change. ( http://gerrit.cloudera.org:8080/19378 ) Change subject: IMPALA-11808: Add support for reload event in catalogD .. Patch Set 12: (4 comments) http://gerrit.cloudera.org:8080/#/c/19378/10//COMMIT_MSG Commit Message: http://gerrit.cloudera.org:8080/#/c/19378/10//COMMIT_MSG@7 PS10, Line 7: reload even > Is there a jira about adding this new event in HMS? It could be mentioned i Ack http://gerrit.cloudera.org:8080/#/c/19378/10//COMMIT_MSG@9 PS10, Line 9: This patch supports a new event called reload event in catalogD. > Can you add a few more info? Ack http://gerrit.cloudera.org:8080/#/c/19378/10//COMMIT_MSG@12 PS10, Line 12: invalidat > typo Ack http://gerrit.cloudera.org:8080/#/c/19378/10//COMMIT_MSG@13 PS10, Line 13: Hive jira that addes support for this reload event. This > A bit of info could be added about testing - currently only event creating Ack -- To view, visit http://gerrit.cloudera.org:8080/19378 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ic62d58837d356dc2113f3c0904228ac9de484136 Gerrit-Change-Number: 19378 Gerrit-PatchSet: 12 Gerrit-Owner: Sai Hemanth Gantasala Gerrit-Reviewer: Abhishek Rawat Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Quanlong Huang Gerrit-Reviewer: Sai Hemanth Gantasala Gerrit-Comment-Date: Thu, 05 Jan 2023 16:41:23 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-11620: Enable setting 'write.format.default'
Zoltan Borok-Nagy has posted comments on this change. ( http://gerrit.cloudera.org:8080/19221 ) Change subject: IMPALA-11620: Enable setting 'write.format.default' .. Patch Set 3: Code-Review+1 I can upgrade my vote to +2 once the open comments are resolved. -- To view, visit http://gerrit.cloudera.org:8080/19221 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I22d0a8a18fce99015fcfe1fd15cb4a4d4c2deaec Gerrit-Change-Number: 19221 Gerrit-PatchSet: 3 Gerrit-Owner: Noemi Pap-Takacs Gerrit-Reviewer: Daniel Becker Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Noemi Pap-Takacs Gerrit-Reviewer: Tamas Mate Gerrit-Reviewer: Zoltan Borok-Nagy Gerrit-Comment-Date: Thu, 05 Jan 2023 15:36:42 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-11808: Add support for reload event in catalogD
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/19378 ) Change subject: IMPALA-11808: Add support for reload event in catalogD .. Patch Set 11: Verified+1 -- To view, visit http://gerrit.cloudera.org:8080/19378 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ic62d58837d356dc2113f3c0904228ac9de484136 Gerrit-Change-Number: 19378 Gerrit-PatchSet: 11 Gerrit-Owner: Sai Hemanth Gantasala Gerrit-Reviewer: Abhishek Rawat Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Quanlong Huang Gerrit-Reviewer: Sai Hemanth Gantasala Gerrit-Comment-Date: Thu, 05 Jan 2023 15:38:15 + Gerrit-HasComments: No
[Impala-ASF-CR](asf-site) IMPALA-11825: Doc update 4.2.0 breaks web search results
Zoltan Borok-Nagy has posted comments on this change. ( http://gerrit.cloudera.org:8080/19402 ) Change subject: IMPALA-11825: Doc update 4.2.0 breaks web search results .. Patch Set 1: Code-Review+1 -- To view, visit http://gerrit.cloudera.org:8080/19402 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: asf-site Gerrit-MessageType: comment Gerrit-Change-Id: Iab1d5fe5e3be85f26aeeb24be9e0018935d2e424 Gerrit-Change-Number: 19402 Gerrit-PatchSet: 1 Gerrit-Owner: Daniel Becker Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Quanlong Huang Gerrit-Reviewer: Tamas Mate Gerrit-Reviewer: Zoltan Borok-Nagy Gerrit-Comment-Date: Thu, 05 Jan 2023 15:30:19 + Gerrit-HasComments: No
[Impala-ASF-CR](asf-site) IMPALA-11825: Doc update 4.2.0 breaks web search results
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/19402 ) Change subject: IMPALA-11825: Doc update 4.2.0 breaks web search results .. Patch Set 1: Verified-1 Build Failed https://jenkins.impala.io/job/gerrit-docs-auto-test/687/ : Doc tests failed. See linked job for details on the failure. -- To view, visit http://gerrit.cloudera.org:8080/19402 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: asf-site Gerrit-MessageType: comment Gerrit-Change-Id: Iab1d5fe5e3be85f26aeeb24be9e0018935d2e424 Gerrit-Change-Number: 19402 Gerrit-PatchSet: 1 Gerrit-Owner: Daniel Becker Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Quanlong Huang Gerrit-Reviewer: Tamas Mate Gerrit-Comment-Date: Thu, 05 Jan 2023 15:07:05 + Gerrit-HasComments: No
[Impala-ASF-CR](asf-site) IMPALA-11825: Doc update 4.2.0 breaks web search results
Daniel Becker has uploaded this change for review. ( http://gerrit.cloudera.org:8080/19402 Change subject: IMPALA-11825: Doc update 4.2.0 breaks web search results .. IMPALA-11825: Doc update 4.2.0 breaks web search results We recently updated the upstream Impala docs in https://gerrit.cloudera.org/#/c/19367/. The doc build system introduced a new directory with CSS formatted html but also renamed the old 'html' directory to 'plain-html'. The problem is that changing the directory name from 'html' to 'plain-html' breaks web search results as they still have the old links. This change is a quick and dirty fix. It duplicates the 'asf-site-html' directory as 'html' so the old web links continue to work. We should come up with a clean solution in a follow-up change. Change-Id: Iab1d5fe5e3be85f26aeeb24be9e0018935d2e424 --- A docs/build/html/css/commonltr.css A docs/build/html/css/commonrtl.css A docs/build/html/css/dita-ot-doc.css A docs/build/html/images/impala_arch.jpeg A docs/build/html/index.html A docs/build/html/shared/ImpalaVariables.html A docs/build/html/shared/impala_common.html A docs/build/html/topics/impala_abort_on_error.html A docs/build/html/topics/impala_adls.html A docs/build/html/topics/impala_admin.html A docs/build/html/topics/impala_admission.html A docs/build/html/topics/impala_admission_config.html A docs/build/html/topics/impala_aggregate_functions.html A docs/build/html/topics/impala_aliases.html A docs/build/html/topics/impala_allow_erasure_coded_files.html A docs/build/html/topics/impala_allow_unsupported_formats.html A docs/build/html/topics/impala_alter_database.html A docs/build/html/topics/impala_alter_table.html A docs/build/html/topics/impala_alter_view.html A docs/build/html/topics/impala_analytic_functions.html A docs/build/html/topics/impala_appx_count_distinct.html A docs/build/html/topics/impala_appx_median.html A docs/build/html/topics/impala_array.html A docs/build/html/topics/impala_auditing.html A docs/build/html/topics/impala_authentication.html A docs/build/html/topics/impala_authorization.html A docs/build/html/topics/impala_avg.html A docs/build/html/topics/impala_avro.html A docs/build/html/topics/impala_batch_size.html A docs/build/html/topics/impala_bigint.html A docs/build/html/topics/impala_bit_functions.html A docs/build/html/topics/impala_boolean.html A docs/build/html/topics/impala_breakpad.html A docs/build/html/topics/impala_broadcast_bytes_limit.html A docs/build/html/topics/impala_buffer_pool_limit.html A docs/build/html/topics/impala_char.html A docs/build/html/topics/impala_client.html A docs/build/html/topics/impala_comment.html A docs/build/html/topics/impala_comments.html A docs/build/html/topics/impala_complex_types.html A docs/build/html/topics/impala_components.html A docs/build/html/topics/impala_compression_codec.html A docs/build/html/topics/impala_compute_stats.html A docs/build/html/topics/impala_compute_stats_min_sample_size.html A docs/build/html/topics/impala_concepts.html A docs/build/html/topics/impala_conditional_functions.html A docs/build/html/topics/impala_config.html A docs/build/html/topics/impala_config_options.html A docs/build/html/topics/impala_config_performance.html A docs/build/html/topics/impala_connecting.html A docs/build/html/topics/impala_conversion_functions.html A docs/build/html/topics/impala_count.html A docs/build/html/topics/impala_create_database.html A docs/build/html/topics/impala_create_function.html A docs/build/html/topics/impala_create_role.html A docs/build/html/topics/impala_create_table.html A docs/build/html/topics/impala_create_view.html A docs/build/html/topics/impala_custom_timezones.html A docs/build/html/topics/impala_data_cache.html A docs/build/html/topics/impala_databases.html A docs/build/html/topics/impala_datatypes.html A docs/build/html/topics/impala_date.html A docs/build/html/topics/impala_datetime_functions.html A docs/build/html/topics/impala_ddl.html A docs/build/html/topics/impala_debug_action.html A docs/build/html/topics/impala_decimal.html A docs/build/html/topics/impala_decimal_v2.html A docs/build/html/topics/impala_dedicated_coordinator.html A docs/build/html/topics/impala_default_file_format.html A docs/build/html/topics/impala_default_hints_insert_statement.html A docs/build/html/topics/impala_default_join_distribution_mode.html A docs/build/html/topics/impala_default_spillable_buffer_size.html A docs/build/html/topics/impala_default_transactional_type.html A docs/build/html/topics/impala_delegation.html A docs/build/html/topics/impala_delete.html A docs/build/html/topics/impala_delete_stats_in_truncate.html A docs/build/html/topics/impala_describe.html A docs/build/html/topics/impala_development.html A docs/build/html/topics/impala_disable_codegen.html A docs/build/html/topics/impala_disable_codegen_rows_threshold.html A docs/build/html/topics/impala_disable_hbase_num_rows_estimate.html A
[Impala-ASF-CR](asf-site) IMPALA-11825: Doc update 4.2.0 breaks web search results
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/19402 ) Change subject: IMPALA-11825: Doc update 4.2.0 breaks web search results .. Patch Set 1: Build Started https://jenkins.impala.io/job/gerrit-docs-auto-test/687/ Testing docs change - this change appears to modify docs/ and no code. This is experimental - please report any issues to tarmstr...@cloudera.com or on this JIRA: IMPALA-7317 -- To view, visit http://gerrit.cloudera.org:8080/19402 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: asf-site Gerrit-MessageType: comment Gerrit-Change-Id: Iab1d5fe5e3be85f26aeeb24be9e0018935d2e424 Gerrit-Change-Number: 19402 Gerrit-PatchSet: 1 Gerrit-Owner: Daniel Becker Gerrit-Reviewer: Impala Public Jenkins Gerrit-Comment-Date: Thu, 05 Jan 2023 15:02:42 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-11801: Use thrift 0.11.0 in FE when compiling with Apache Hive3
Csaba Ringhofer has posted comments on this change. ( http://gerrit.cloudera.org:8080/19400 ) Change subject: IMPALA-11801: Use thrift 0.11.0 in FE when compiling with Apache Hive3 .. Patch Set 1: (1 comment) http://gerrit.cloudera.org:8080/#/c/19400/1/bin/impala-config.sh File bin/impala-config.sh: http://gerrit.cloudera.org:8080/#/c/19400/1/bin/impala-config.sh@183 PS1, Line 183: USE_APACHE_HIVE USE_APACHE_HIVE is only set in line 325 (which led to the build failure) -- To view, visit http://gerrit.cloudera.org:8080/19400 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Idcdae97920241429c6dd0bd2b4ac1debfbde9d23 Gerrit-Change-Number: 19400 Gerrit-PatchSet: 1 Gerrit-Owner: Quanlong Huang Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Fucun Chu Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Quanlong Huang Gerrit-Comment-Date: Thu, 05 Jan 2023 11:32:31 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-11565: Support IF NOT EXISTS in alter table add columns for kudu/iceberg table
Quanlong Huang has posted comments on this change. ( http://gerrit.cloudera.org:8080/18953 ) Change subject: IMPALA-11565: Support IF NOT EXISTS in alter table add columns for kudu/iceberg table .. Patch Set 13: Code-Review+1 (2 comments) LGTM http://gerrit.cloudera.org:8080/#/c/18953/13/common/thrift/JniCatalog.thrift File common/thrift/JniCatalog.thrift: http://gerrit.cloudera.org:8080/#/c/18953/13/common/thrift/JniCatalog.thrift@225 PS13, Line 225: 1: optional list columns If no new columns are added, i.e. all columns exist, I think we don't need to trigger catalogd RPCs since there are nothing catalogd needs to do. However, this is an optimization. We can do it in a separate JIRA. http://gerrit.cloudera.org:8080/#/c/18953/13/fe/src/main/java/org/apache/impala/service/CatalogOpExecutor.java File fe/src/main/java/org/apache/impala/service/CatalogOpExecutor.java: http://gerrit.cloudera.org:8080/#/c/18953/13/fe/src/main/java/org/apache/impala/service/CatalogOpExecutor.java@1036 PS13, Line 1036: In AlterTableAddColsStmt, there is remove for columns. : // May cause columns to be empty. nit: columns could be ignored/cleared in AlterTableAddColsStmt -- To view, visit http://gerrit.cloudera.org:8080/18953 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I82590e5372e881f2e81d4ed3dd0d32a2d3ddb517 Gerrit-Change-Number: 18953 Gerrit-PatchSet: 13 Gerrit-Owner: Baike Xia Gerrit-Reviewer: Anonymous Coward Gerrit-Reviewer: Baike Xia Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Jian Zhang Gerrit-Reviewer: Quanlong Huang Gerrit-Comment-Date: Thu, 05 Jan 2023 11:29:49 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-11626, IMPALA-11808: Bump GBN to 36109364 for new HMS events
Impala Public Jenkins has submitted this change and it was merged. ( http://gerrit.cloudera.org:8080/19399 ) Change subject: IMPALA-11626, IMPALA-11808: Bump GBN to 36109364 for new HMS events .. IMPALA-11626, IMPALA-11808: Bump GBN to 36109364 for new HMS events This bumps the dependent CDP version to 7.2.16.0-272 for HIVE-24329 and HIVE-26838. Tests: - Passed exhaustive tests Change-Id: I71dd1bf7b6d744f23008d8cefeeff7adcc8afa41 Reviewed-on: http://gerrit.cloudera.org:8080/19399 Reviewed-by: Impala Public Jenkins Tested-by: Impala Public Jenkins --- M bin/impala-config.sh 1 file changed, 11 insertions(+), 11 deletions(-) Approvals: Impala Public Jenkins: Looks good to me, approved; Verified -- To view, visit http://gerrit.cloudera.org:8080/19399 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: merged Gerrit-Change-Id: I71dd1bf7b6d744f23008d8cefeeff7adcc8afa41 Gerrit-Change-Number: 19399 Gerrit-PatchSet: 5 Gerrit-Owner: Quanlong Huang Gerrit-Reviewer: Aman Sinha Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Sai Hemanth Gantasala
[Impala-ASF-CR] IMPALA-11626, IMPALA-11808: Bump GBN to 36109364 for new HMS events
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/19399 ) Change subject: IMPALA-11626, IMPALA-11808: Bump GBN to 36109364 for new HMS events .. Patch Set 4: Verified+1 -- To view, visit http://gerrit.cloudera.org:8080/19399 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I71dd1bf7b6d744f23008d8cefeeff7adcc8afa41 Gerrit-Change-Number: 19399 Gerrit-PatchSet: 4 Gerrit-Owner: Quanlong Huang Gerrit-Reviewer: Aman Sinha Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Sai Hemanth Gantasala Gerrit-Comment-Date: Thu, 05 Jan 2023 11:24:05 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-7969: Always admit trivial queries immediately
Csaba Ringhofer has posted comments on this change. ( http://gerrit.cloudera.org:8080/19214 ) Change subject: IMPALA-7969: Always admit trivial queries immediately .. Patch Set 3: Code-Review+1 (1 comment) http://gerrit.cloudera.org:8080/#/c/19214/3/fe/src/main/java/org/apache/impala/planner/TrivialQueryChecker.java File fe/src/main/java/org/apache/impala/planner/TrivialQueryChecker.java: http://gerrit.cloudera.org:8080/#/c/19214/3/fe/src/main/java/org/apache/impala/planner/TrivialQueryChecker.java@66 PS3, Line 66: HasFunctionSleep This recursive function could be simplifed by using expr.collectAll() + a Predicate. There are several predicates like this, e.g. https://github.com/apache/impala/blob/5b349cb5492827e7584412de3922d263ac441d26/fe/src/main/java/org/apache/impala/analysis/Expr.java#L104 -- To view, visit http://gerrit.cloudera.org:8080/19214 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I2a729764e3055d7eb11900c96c82ff53eb261f91 Gerrit-Change-Number: 19214 Gerrit-PatchSet: 3 Gerrit-Owner: Yida Wu Gerrit-Reviewer: Abhishek Rawat Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Yida Wu Gerrit-Comment-Date: Thu, 05 Jan 2023 10:49:36 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-11808: Add support for reload event in catalogD
Csaba Ringhofer has posted comments on this change. ( http://gerrit.cloudera.org:8080/19378 ) Change subject: IMPALA-11808: Add support for reload event in catalogD .. Patch Set 11: Code-Review+1 Started the tests without giving +2 to see if there is any error. -- To view, visit http://gerrit.cloudera.org:8080/19378 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ic62d58837d356dc2113f3c0904228ac9de484136 Gerrit-Change-Number: 19378 Gerrit-PatchSet: 11 Gerrit-Owner: Sai Hemanth Gantasala Gerrit-Reviewer: Abhishek Rawat Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Quanlong Huang Gerrit-Reviewer: Sai Hemanth Gantasala Gerrit-Comment-Date: Thu, 05 Jan 2023 10:32:32 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-11808: Add support for reload event in catalogD
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/19378 ) Change subject: IMPALA-11808: Add support for reload event in catalogD .. Patch Set 11: Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/8950/ DRY_RUN=false -- To view, visit http://gerrit.cloudera.org:8080/19378 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ic62d58837d356dc2113f3c0904228ac9de484136 Gerrit-Change-Number: 19378 Gerrit-PatchSet: 11 Gerrit-Owner: Sai Hemanth Gantasala Gerrit-Reviewer: Abhishek Rawat Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Quanlong Huang Gerrit-Reviewer: Sai Hemanth Gantasala Gerrit-Comment-Date: Thu, 05 Jan 2023 10:31:42 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-11808: Add support for reload event in catalogD
Csaba Ringhofer has posted comments on this change. (
http://gerrit.cloudera.org:8080/19378 )
Change subject: IMPALA-11808: Add support for reload event in catalogD
..
Patch Set 10: Code-Review+1
(5 comments)
The code looks good to me, some comments for the commit message
http://gerrit.cloudera.org:8080/#/c/19378/10//COMMIT_MSG
Commit Message:
http://gerrit.cloudera.org:8080/#/c/19378/10//COMMIT_MSG@7
PS10, Line 7: reload even
Is there a jira about adding this new event in HMS? It could be mentioned in
the commit message
http://gerrit.cloudera.org:8080/#/c/19378/10//COMMIT_MSG@9
PS10, Line 9: This patch supports a new event called reload event in catalogD.
Can you add a few more info?
- both sending and processing of the event is implemented in this patch
- the flag could be mentioned and that it is disabled by default
http://gerrit.cloudera.org:8080/#/c/19378/10//COMMIT_MSG@12
PS10, Line 12: invlidate
typo
http://gerrit.cloudera.org:8080/#/c/19378/10//COMMIT_MSG@13
PS10, Line 13:
A bit of info could be added about testing - currently only event creating and
self event checking is implemented. It would be great to have tests for the
event processing too - a TODO could be added about this.
http://gerrit.cloudera.org:8080/#/c/19378/10/fe/src/compat-hive-3/java/org/apache/impala/compat/MetastoreShim.java
File fe/src/compat-hive-3/java/org/apache/impala/compat/MetastoreShim.java:
http://gerrit.cloudera.org:8080/#/c/19378/10/fe/src/compat-hive-3/java/org/apache/impala/compat/MetastoreShim.java@549
PS10, Line 549: throws MetastoreNotificationException{
nit: extra space
--
To view, visit http://gerrit.cloudera.org:8080/19378
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Ic62d58837d356dc2113f3c0904228ac9de484136
Gerrit-Change-Number: 19378
Gerrit-PatchSet: 10
Gerrit-Owner: Sai Hemanth Gantasala
Gerrit-Reviewer: Abhishek Rawat
Gerrit-Reviewer: Csaba Ringhofer
Gerrit-Reviewer: Impala Public Jenkins
Gerrit-Reviewer: Quanlong Huang
Gerrit-Reviewer: Sai Hemanth Gantasala
Gerrit-Comment-Date: Thu, 05 Jan 2023 10:11:07 +
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-11620: Enable setting 'write.format.default'
Daniel Becker has posted comments on this change. ( http://gerrit.cloudera.org:8080/19221 ) Change subject: IMPALA-11620: Enable setting 'write.format.default' .. Patch Set 3: (2 comments) http://gerrit.cloudera.org:8080/#/c/19221/3//COMMIT_MSG Commit Message: http://gerrit.cloudera.org:8080/#/c/19221/3//COMMIT_MSG@10 PS3, Line 10: then Nit: than http://gerrit.cloudera.org:8080/#/c/19221/3//COMMIT_MSG@23 PS3, Line 23: tableproperty Nit: table property -- To view, visit http://gerrit.cloudera.org:8080/19221 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I22d0a8a18fce99015fcfe1fd15cb4a4d4c2deaec Gerrit-Change-Number: 19221 Gerrit-PatchSet: 3 Gerrit-Owner: Noemi Pap-Takacs Gerrit-Reviewer: Daniel Becker Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Noemi Pap-Takacs Gerrit-Reviewer: Tamas Mate Gerrit-Reviewer: Zoltan Borok-Nagy Gerrit-Comment-Date: Thu, 05 Jan 2023 10:03:27 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-11620: Enable setting 'write.format.default'
Tamas Mate has posted comments on this change. ( http://gerrit.cloudera.org:8080/19221 ) Change subject: IMPALA-11620: Enable setting 'write.format.default' .. Patch Set 3: Code-Review+1 Thank you for this fix Noemi, LGTM! -- To view, visit http://gerrit.cloudera.org:8080/19221 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I22d0a8a18fce99015fcfe1fd15cb4a4d4c2deaec Gerrit-Change-Number: 19221 Gerrit-PatchSet: 3 Gerrit-Owner: Noemi Pap-Takacs Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Noemi Pap-Takacs Gerrit-Reviewer: Tamas Mate Gerrit-Reviewer: Zoltan Borok-Nagy Gerrit-Comment-Date: Thu, 05 Jan 2023 08:04:13 + Gerrit-HasComments: No
