[Impala-ASF-CR] IMPALA-12486: Add catalog metrics for metadata loading
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/20673 ) Change subject: IMPALA-12486: Add catalog metrics for metadata loading .. Patch Set 1: Build Failed https://jenkins.impala.io/job/gerrit-code-review-checks/14354/ : Initial code review checks failed. See linked job for details on the failure. -- To view, visit http://gerrit.cloudera.org:8080/20673 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Icef7b123bdcb0f5b8572635eeaacd8294990f9ba Gerrit-Change-Number: 20673 Gerrit-PatchSet: 1 Gerrit-Owner: Quanlong Huang Gerrit-Reviewer: Impala Public Jenkins Gerrit-Comment-Date: Tue, 07 Nov 2023 06:30:40 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-12544: Replace scan progress with query progress as progress reporting for the shell
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/20672 ) Change subject: IMPALA-12544: Replace scan progress with query progress as progress reporting for the shell .. Patch Set 2: Build Failed https://jenkins.impala.io/job/gerrit-code-review-checks/14353/ : Initial code review checks failed. See linked job for details on the failure. -- To view, visit http://gerrit.cloudera.org:8080/20672 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I11a704885505442b7499a026fcee3b86696cd064 Gerrit-Change-Number: 20672 Gerrit-PatchSet: 2 Gerrit-Owner: Zihao Ye Gerrit-Reviewer: Impala Public Jenkins Gerrit-Comment-Date: Tue, 07 Nov 2023 06:26:13 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-12513: Allow to reset metadata when the CatalogD becomes active
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/20614 ) Change subject: IMPALA-12513: Allow to reset metadata when the CatalogD becomes active .. Patch Set 8: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/14352/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/20614 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I2b54f36f96e7901499105e51790d8e2300d7fea9 Gerrit-Change-Number: 20614 Gerrit-PatchSet: 8 Gerrit-Owner: ttz <2433038...@qq.com> Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Quanlong Huang Gerrit-Reviewer: Wenzhe Zhou Gerrit-Reviewer: ttz <2433038...@qq.com> Gerrit-Comment-Date: Tue, 07 Nov 2023 06:15:48 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-12486: Add catalog metrics for metadata loading
Quanlong Huang has uploaded this change for review. ( http://gerrit.cloudera.org:8080/20673 Change subject: IMPALA-12486: Add catalog metrics for metadata loading .. IMPALA-12486: Add catalog metrics for metadata loading This patch adds the following catalog metrics which indicate the load on HDFS for loading file metadata: - catalog.num-file-metadata-loading-threads: The total size of all thread pools used in loading file metadata. - catalog.num-file-metadata-loading-tasks: The total number of unfinished file metadata loading tasks. Each task corresponds to a partition. - catalog.num-tables-loading-file-metadata: The total number of tables that are loading file metadata. Also adds some metrics for metadata loading on all tables. Note that metadata loading of an HDFS table consists of loading HMS metadata and HDFS file metadata, etc. - catalog.num-tables-loading-metadata: The total number of tables that are loading metadata. - catalog.num-tables-async-loading-metadata: The total number of tables that are loading metadata asynchorously. E.g. the initial metadata loading triggered by the first access on a table. - catalog.num-tables-waiting-for-async-loading: The total number of tables that are waiting for asynchorous loading. If this number raises, consider bumping --num_metadata_loading_threads. Three metrics about the catalog cache are also added: - catalog.num-databases - catalog.num-tables - catalog.num-functions Note that the first two are also shown in WebUI of coordinators and we plan to deprecate them. Tests - Launch catalogd locally with load_catalog_in_background=true and verified the metrics. - Add e2e tests in tests/webserver/test_web_pages.py Change-Id: Icef7b123bdcb0f5b8572635eeaacd8294990f9ba --- M be/src/catalog/catalog-server.cc M be/src/catalog/catalog-server.h M common/thrift/JniCatalog.thrift M common/thrift/metrics.json M fe/src/main/java/org/apache/impala/catalog/CatalogObjectCache.java M fe/src/main/java/org/apache/impala/catalog/CatalogServiceCatalog.java M fe/src/main/java/org/apache/impala/catalog/DataSourceTable.java M fe/src/main/java/org/apache/impala/catalog/Db.java M fe/src/main/java/org/apache/impala/catalog/FileMetadataLoader.java M fe/src/main/java/org/apache/impala/catalog/HBaseTable.java M fe/src/main/java/org/apache/impala/catalog/HdfsTable.java M fe/src/main/java/org/apache/impala/catalog/IcebergFileMetadataLoader.java M fe/src/main/java/org/apache/impala/catalog/KuduTable.java M fe/src/main/java/org/apache/impala/catalog/ParallelFileMetadataLoader.java M fe/src/main/java/org/apache/impala/catalog/Table.java M fe/src/main/java/org/apache/impala/catalog/TableLoadingMgr.java M fe/src/main/java/org/apache/impala/catalog/View.java M fe/src/main/java/org/apache/impala/service/JniCatalog.java M tests/webserver/test_web_pages.py 19 files changed, 279 insertions(+), 13 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/73/20673/1 -- To view, visit http://gerrit.cloudera.org:8080/20673 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newchange Gerrit-Change-Id: Icef7b123bdcb0f5b8572635eeaacd8294990f9ba Gerrit-Change-Number: 20673 Gerrit-PatchSet: 1 Gerrit-Owner: Quanlong Huang
[Impala-ASF-CR] IMPALA-12544: Replace scan progress with query progress as progress reporting for the shell
Hello Impala Public Jenkins, I'd like you to reexamine a change. Please visit http://gerrit.cloudera.org:8080/20672 to look at the new patch set (#2). Change subject: IMPALA-12544: Replace scan progress with query progress as progress reporting for the shell .. IMPALA-12544: Replace scan progress with query progress as progress reporting for the shell This patch modifies the query progress bar in impala-shell by replacing the scan progress with the query progress. The query progress is calculated using the number of completed fragment instances divided by the total number of fragment instances. This approach provides a more accurate reflection of the actual completion progress of the query compared to the scan progress, which is calculated based on the completed ScanRanges divided by the total ScanRanges. Especially for computation-intensive queries that involve complex aggregations or sorting, such as tpcds query78, there is often additional time required for computation even after the scanning is complete. In such cases, displaying 100% scan progress would be inaccurate. Change-Id: I11a704885505442b7499a026fcee3b86696cd064 --- M shell/impala_shell.py 1 file changed, 5 insertions(+), 4 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/72/20672/2 -- To view, visit http://gerrit.cloudera.org:8080/20672 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: I11a704885505442b7499a026fcee3b86696cd064 Gerrit-Change-Number: 20672 Gerrit-PatchSet: 2 Gerrit-Owner: Zihao Ye Gerrit-Reviewer: Impala Public Jenkins
[Impala-ASF-CR] IMPALA-12544: Replace scan progress with query progress as progress reporting for the shell
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/20672 ) Change subject: IMPALA-12544: Replace scan progress with query progress as progress reporting for the shell .. Patch Set 2: (1 comment) http://gerrit.cloudera.org:8080/#/c/20672/2/shell/impala_shell.py File shell/impala_shell.py: http://gerrit.cloudera.org:8080/#/c/20672/2/shell/impala_shell.py@1313 PS2, Line 1313: / flake8: W504 line break after binary operator -- To view, visit http://gerrit.cloudera.org:8080/20672 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I11a704885505442b7499a026fcee3b86696cd064 Gerrit-Change-Number: 20672 Gerrit-PatchSet: 2 Gerrit-Owner: Zihao Ye Gerrit-Reviewer: Impala Public Jenkins Gerrit-Comment-Date: Tue, 07 Nov 2023 06:03:12 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-12513: Allow to reset metadata when the CatalogD becomes active
ttz has posted comments on this change. ( http://gerrit.cloudera.org:8080/20614 ) Change subject: IMPALA-12513: Allow to reset metadata when the CatalogD becomes active .. Patch Set 8: (4 comments) Thanks Wenzhe! http://gerrit.cloudera.org:8080/#/c/20614/7//COMMIT_MSG Commit Message: http://gerrit.cloudera.org:8080/#/c/20614/7//COMMIT_MSG@11 PS7, Line 11: become > nit: becomes Done http://gerrit.cloudera.org:8080/#/c/20614/7//COMMIT_MSG@14 PS7, Line 14: become > nit: becomes Done http://gerrit.cloudera.org:8080/#/c/20614/7/be/src/catalog/catalog-server.cc File be/src/catalog/catalog-server.cc: http://gerrit.cloudera.org:8080/#/c/20614/7/be/src/catalog/catalog-server.cc@548 PS7, Line 548: FLAGS > upper case: FLAGS_ Done http://gerrit.cloudera.org:8080/#/c/20614/7/tests/custom_cluster/test_catalogd_ha.py File tests/custom_cluster/test_catalogd_ha.py: http://gerrit.cloudera.org:8080/#/c/20614/7/tests/custom_cluster/test_catalogd_ha.py@395 PS7, Line 395: statestored_args="--enable_catalogd_ha=true " > set statestored_args="--use_subscriber_id_as_catalogd_priority=true", other Done -- To view, visit http://gerrit.cloudera.org:8080/20614 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I2b54f36f96e7901499105e51790d8e2300d7fea9 Gerrit-Change-Number: 20614 Gerrit-PatchSet: 8 Gerrit-Owner: ttz <2433038...@qq.com> Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Quanlong Huang Gerrit-Reviewer: Wenzhe Zhou Gerrit-Reviewer: ttz <2433038...@qq.com> Gerrit-Comment-Date: Tue, 07 Nov 2023 05:50:11 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-12513: Allow to reset metadata when the CatalogD becomes active
Hello Quanlong Huang, Wenzhe Zhou, Impala Public Jenkins, I'd like you to reexamine a change. Please visit http://gerrit.cloudera.org:8080/20614 to look at the new patch set (#8). Change subject: IMPALA-12513: Allow to reset metadata when the CatalogD becomes active .. IMPALA-12513: Allow to reset metadata when the CatalogD becomes active When switching active catalogd, the loaded metadata in the standby catalogd may not be the latest. A backend flag should be provided to control whether to reset metadata when the catalogd become active. Adds the following startup flags for catalogd: 'catalogd_ha_reset_metadata_on_failover'. Default is false. If true, reset all metadata when the catalogd becomes active. Testing: - Added a test case to start both catalogds with flag 'catalogd_ha_reset_metadata_on_failover' as true. - Passed core tests Change-Id: I2b54f36f96e7901499105e51790d8e2300d7fea9 --- M be/src/catalog/catalog-server.cc M tests/custom_cluster/test_catalogd_ha.py 2 files changed, 53 insertions(+), 0 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/14/20614/8 -- To view, visit http://gerrit.cloudera.org:8080/20614 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: I2b54f36f96e7901499105e51790d8e2300d7fea9 Gerrit-Change-Number: 20614 Gerrit-PatchSet: 8 Gerrit-Owner: ttz <2433038...@qq.com> Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Quanlong Huang Gerrit-Reviewer: Wenzhe Zhou Gerrit-Reviewer: ttz <2433038...@qq.com>
[Impala-ASF-CR] IMPALA-12511: Bump thrift version to fix THRIFT-5670
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/20660 ) Change subject: IMPALA-12511: Bump thrift version to fix THRIFT-5670 .. Patch Set 4: Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/9893/ DRY_RUN=false -- To view, visit http://gerrit.cloudera.org:8080/20660 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I02f396fd0c27cde03883c06d6376feaf07ca8c13 Gerrit-Change-Number: 20660 Gerrit-PatchSet: 4 Gerrit-Owner: Quanlong Huang Gerrit-Reviewer: Daniel Becker Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Michael Smith Gerrit-Reviewer: Quanlong Huang Gerrit-Reviewer: Wenzhe Zhou Gerrit-Comment-Date: Tue, 07 Nov 2023 05:35:20 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-12511: Bump thrift version to fix THRIFT-5670
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/20660 ) Change subject: IMPALA-12511: Bump thrift version to fix THRIFT-5670 .. Patch Set 4: Code-Review+2 -- To view, visit http://gerrit.cloudera.org:8080/20660 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I02f396fd0c27cde03883c06d6376feaf07ca8c13 Gerrit-Change-Number: 20660 Gerrit-PatchSet: 4 Gerrit-Owner: Quanlong Huang Gerrit-Reviewer: Daniel Becker Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Michael Smith Gerrit-Reviewer: Quanlong Huang Gerrit-Reviewer: Wenzhe Zhou Gerrit-Comment-Date: Tue, 07 Nov 2023 05:35:19 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-12513: Allow to reset metadata when the CatalogD become active
Wenzhe Zhou has posted comments on this change. ( http://gerrit.cloudera.org:8080/20614 ) Change subject: IMPALA-12513: Allow to reset metadata when the CatalogD become active .. Patch Set 7: (4 comments) http://gerrit.cloudera.org:8080/#/c/20614/7//COMMIT_MSG Commit Message: http://gerrit.cloudera.org:8080/#/c/20614/7//COMMIT_MSG@11 PS7, Line 11: become nit: becomes http://gerrit.cloudera.org:8080/#/c/20614/7//COMMIT_MSG@14 PS7, Line 14: become nit: becomes http://gerrit.cloudera.org:8080/#/c/20614/7/be/src/catalog/catalog-server.cc File be/src/catalog/catalog-server.cc: http://gerrit.cloudera.org:8080/#/c/20614/7/be/src/catalog/catalog-server.cc@548 PS7, Line 548: Flags upper case: FLAGS_ http://gerrit.cloudera.org:8080/#/c/20614/7/tests/custom_cluster/test_catalogd_ha.py File tests/custom_cluster/test_catalogd_ha.py: http://gerrit.cloudera.org:8080/#/c/20614/7/tests/custom_cluster/test_catalogd_ha.py@395 PS7, Line 395: statestored_args="--enable_catalogd_ha=true ", set statestored_args="--use_subscriber_id_as_catalogd_priority=true", otherwise the statestore elect active catalogd randomly. -- To view, visit http://gerrit.cloudera.org:8080/20614 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I2b54f36f96e7901499105e51790d8e2300d7fea9 Gerrit-Change-Number: 20614 Gerrit-PatchSet: 7 Gerrit-Owner: ttz <2433038...@qq.com> Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Quanlong Huang Gerrit-Reviewer: Wenzhe Zhou Gerrit-Reviewer: ttz <2433038...@qq.com> Gerrit-Comment-Date: Tue, 07 Nov 2023 05:23:46 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-12513: Allow to reset metadata when the CatalogD become active
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/20614 ) Change subject: IMPALA-12513: Allow to reset metadata when the CatalogD become active .. Patch Set 7: Build Failed https://jenkins.impala.io/job/gerrit-code-review-checks/14351/ : Initial code review checks failed. See linked job for details on the failure. -- To view, visit http://gerrit.cloudera.org:8080/20614 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I2b54f36f96e7901499105e51790d8e2300d7fea9 Gerrit-Change-Number: 20614 Gerrit-PatchSet: 7 Gerrit-Owner: ttz <2433038...@qq.com> Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Quanlong Huang Gerrit-Reviewer: Wenzhe Zhou Gerrit-Reviewer: ttz <2433038...@qq.com> Gerrit-Comment-Date: Tue, 07 Nov 2023 04:42:51 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-12448: Avoid getting stuck when refreshing a non-existent partition
Hello Quanlong Huang, Impala Public Jenkins, I'd like you to reexamine a change. Please visit http://gerrit.cloudera.org:8080/20490 to look at the new patch set (#10). Change subject: IMPALA-12448: Avoid getting stuck when refreshing a non-existent partition .. IMPALA-12448: Avoid getting stuck when refreshing a non-existent partition In some cases, we shouldn't wait for the version specified in 'result.version' because the table version has not changed after resetting metadata, such as refreshing a non-existent partition. If not, the query may be stuck for a long time. This patch determines whether to wait by comparing the table version before and after resetting. Tests: * added e2e tests Change-Id: Iace7cdadda300b03896f92415822266354421887 --- M be/src/catalog/catalog-server.cc M be/src/util/backend-gflag-util.cc M common/thrift/BackendGflags.thrift M fe/src/main/java/org/apache/impala/catalog/TopicUpdateLog.java M fe/src/main/java/org/apache/impala/service/CatalogOpExecutor.java M tests/metadata/test_refresh_partition.py 6 files changed, 64 insertions(+), 4 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/90/20490/10 -- To view, visit http://gerrit.cloudera.org:8080/20490 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: Iace7cdadda300b03896f92415822266354421887 Gerrit-Change-Number: 20490 Gerrit-PatchSet: 10 Gerrit-Owner: ttz <2433038...@qq.com> Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Quanlong Huang Gerrit-Reviewer: ttz <2433038...@qq.com>
[Impala-ASF-CR] IMPALA-12542: Wait for query cancellation in webserver test
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/20668 ) Change subject: IMPALA-12542: Wait for query cancellation in webserver test .. Patch Set 3: Verified+1 -- To view, visit http://gerrit.cloudera.org:8080/20668 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I1c3bc83955be025e83a426377221767694baa444 Gerrit-Change-Number: 20668 Gerrit-PatchSet: 3 Gerrit-Owner: Michael Smith Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Michael Smith Gerrit-Reviewer: Riza Suminto Gerrit-Comment-Date: Tue, 07 Nov 2023 04:36:39 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-12513: Allow to reset metadata when the CatalogD become active
Hello Quanlong Huang, Wenzhe Zhou, Impala Public Jenkins, I'd like you to reexamine a change. Please visit http://gerrit.cloudera.org:8080/20614 to look at the new patch set (#7). Change subject: IMPALA-12513: Allow to reset metadata when the CatalogD become active .. IMPALA-12513: Allow to reset metadata when the CatalogD become active When switching active catalogd, the loaded metadata in the standby catalogd may not be the latest. A backend flag should be provided to control whether to reset metadata when the catalogd become active. Adds the following startup flags for catalogd: 'catalogd_ha_reset_metadata_on_failover'. Default is false. If true, reset all metadata when the catalogd become active. Testing: - Added a test case to start both catalogds with flag 'catalogd_ha_reset_metadata_on_failover' as true. - Passed core tests Change-Id: I2b54f36f96e7901499105e51790d8e2300d7fea9 --- M be/src/catalog/catalog-server.cc M tests/custom_cluster/test_catalogd_ha.py 2 files changed, 52 insertions(+), 0 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/14/20614/7 -- To view, visit http://gerrit.cloudera.org:8080/20614 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: I2b54f36f96e7901499105e51790d8e2300d7fea9 Gerrit-Change-Number: 20614 Gerrit-PatchSet: 7 Gerrit-Owner: ttz <2433038...@qq.com> Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Quanlong Huang Gerrit-Reviewer: Wenzhe Zhou Gerrit-Reviewer: ttz <2433038...@qq.com>
[Impala-ASF-CR] IMPALA-12513: Allow to reset metadata when the CatalogD become active
Hello Quanlong Huang, Wenzhe Zhou, Impala Public Jenkins, I'd like you to reexamine a change. Please visit http://gerrit.cloudera.org:8080/20614 to look at the new patch set (#6). Change subject: IMPALA-12513: Allow to reset metadata when the CatalogD become active .. IMPALA-12513: Allow to reset metadata when the CatalogD become active When switching active catalogd, the loaded metadata in the standby catalogd may not be the latest. A backend flag should be provided to control whether to reset metadata when the catalogd become active. Adds the following startup flags for catalogd: 'catalogd_ha_reset_metadata_on_failover'. Default is false. If true, reset all metadata when the catalogd become active. Testing: - Added a test case to start both catalogds with flag 'catalogd_ha_reset_metadata_on_failover' as true. - Passed core tests Change-Id: I2b54f36f96e7901499105e51790d8e2300d7fea9 --- M be/src/catalog/catalog-server.cc M tests/custom_cluster/test_catalogd_ha.py 2 files changed, 52 insertions(+), 0 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/14/20614/6 -- To view, visit http://gerrit.cloudera.org:8080/20614 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: I2b54f36f96e7901499105e51790d8e2300d7fea9 Gerrit-Change-Number: 20614 Gerrit-PatchSet: 6 Gerrit-Owner: ttz <2433038...@qq.com> Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Quanlong Huang Gerrit-Reviewer: Wenzhe Zhou Gerrit-Reviewer: ttz <2433038...@qq.com>
[Impala-ASF-CR] [WIP] IMPALA-3268: Add support for SHOW VIEWS statement
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/20669 ) Change subject: [WIP] IMPALA-3268: Add support for SHOW VIEWS statement .. Patch Set 1: Verified+1 -- To view, visit http://gerrit.cloudera.org:8080/20669 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I321fc5350392a815949a4e7d2a64d60466689788 Gerrit-Change-Number: 20669 Gerrit-PatchSet: 1 Gerrit-Owner: Fang-Yu Rao Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Michael Smith Gerrit-Comment-Date: Tue, 07 Nov 2023 04:10:04 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-12544: Replace scan progress with query progress as progress reporting for the shell
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/20672 ) Change subject: IMPALA-12544: Replace scan progress with query progress as progress reporting for the shell .. Patch Set 1: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/14350/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/20672 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I11a704885505442b7499a026fcee3b86696cd064 Gerrit-Change-Number: 20672 Gerrit-PatchSet: 1 Gerrit-Owner: Zihao Ye Gerrit-Reviewer: Impala Public Jenkins Gerrit-Comment-Date: Tue, 07 Nov 2023 04:07:10 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-12544: Replace scan progress with query progress as progress reporting for the shell
Zihao Ye has uploaded this change for review. ( http://gerrit.cloudera.org:8080/20672 Change subject: IMPALA-12544: Replace scan progress with query progress as progress reporting for the shell .. IMPALA-12544: Replace scan progress with query progress as progress reporting for the shell This patch modifies the query progress bar in impala-shell by replacing the scan progress with the query progress. The query progress is calculated using the number of completed fragment instances divided by the total number of fragment instances. This approach provides a more accurate reflection of the actual completion progress of the query compared to the scan progress, which is calculated based on the completed ScanRanges divided by the total ScanRanges. Especially for computation-intensive queries that involve complex aggregations or sorting, such as tpcds query78, there is often additional time required for computation even after the scanning is complete. In such cases, displaying 100% scan progress would be inaccurate. Change-Id: I11a704885505442b7499a026fcee3b86696cd064 --- M shell/impala_shell.py 1 file changed, 5 insertions(+), 4 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/72/20672/1 -- To view, visit http://gerrit.cloudera.org:8080/20672 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newchange Gerrit-Change-Id: I11a704885505442b7499a026fcee3b86696cd064 Gerrit-Change-Number: 20672 Gerrit-PatchSet: 1 Gerrit-Owner: Zihao Ye
[Impala-ASF-CR] IMPALA-12544: Replace scan progress with query progress as progress reporting for the shell
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/20672 ) Change subject: IMPALA-12544: Replace scan progress with query progress as progress reporting for the shell .. Patch Set 1: (1 comment) http://gerrit.cloudera.org:8080/#/c/20672/1/shell/impala_shell.py File shell/impala_shell.py: http://gerrit.cloudera.org:8080/#/c/20672/1/shell/impala_shell.py@1313 PS1, Line 1313: / flake8: W504 line break after binary operator -- To view, visit http://gerrit.cloudera.org:8080/20672 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I11a704885505442b7499a026fcee3b86696cd064 Gerrit-Change-Number: 20672 Gerrit-PatchSet: 1 Gerrit-Owner: Zihao Ye Gerrit-Reviewer: Impala Public Jenkins Gerrit-Comment-Date: Tue, 07 Nov 2023 03:44:57 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-12513: Reset metadata when the CatalogD become active
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/20614 ) Change subject: IMPALA-12513: Reset metadata when the CatalogD become active .. Patch Set 5: Build Failed https://jenkins.impala.io/job/gerrit-code-review-checks/14349/ : Initial code review checks failed. See linked job for details on the failure. -- To view, visit http://gerrit.cloudera.org:8080/20614 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I2b54f36f96e7901499105e51790d8e2300d7fea9 Gerrit-Change-Number: 20614 Gerrit-PatchSet: 5 Gerrit-Owner: ttz <2433038...@qq.com> Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Quanlong Huang Gerrit-Reviewer: Wenzhe Zhou Gerrit-Reviewer: ttz <2433038...@qq.com> Gerrit-Comment-Date: Tue, 07 Nov 2023 03:43:10 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-12513: Reset metadata when the CatalogD become active
Hello Quanlong Huang, Wenzhe Zhou, Impala Public Jenkins, I'd like you to reexamine a change. Please visit http://gerrit.cloudera.org:8080/20614 to look at the new patch set (#5). Change subject: IMPALA-12513: Reset metadata when the CatalogD become active .. IMPALA-12513: Reset metadata when the CatalogD become active When switching active CatalogD, the loaded metadata in the candidate CatalogD may not be the latest. We should reset the metadata to ensure that there are no metadata inconsistency exceptions after failover. Change-Id: I2b54f36f96e7901499105e51790d8e2300d7fea9 --- M be/src/catalog/catalog-server.cc M tests/custom_cluster/test_catalogd_ha.py 2 files changed, 52 insertions(+), 0 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/14/20614/5 -- To view, visit http://gerrit.cloudera.org:8080/20614 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: I2b54f36f96e7901499105e51790d8e2300d7fea9 Gerrit-Change-Number: 20614 Gerrit-PatchSet: 5 Gerrit-Owner: ttz <2433038...@qq.com> Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Quanlong Huang Gerrit-Reviewer: Wenzhe Zhou Gerrit-Reviewer: ttz <2433038...@qq.com>
[Impala-ASF-CR] IMPALA-12448: Avoid getting stuck when refreshing a non-existent partition
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/20490 ) Change subject: IMPALA-12448: Avoid getting stuck when refreshing a non-existent partition .. Patch Set 9: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/14348/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/20490 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Iace7cdadda300b03896f92415822266354421887 Gerrit-Change-Number: 20490 Gerrit-PatchSet: 9 Gerrit-Owner: ttz <2433038...@qq.com> Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Quanlong Huang Gerrit-Reviewer: ttz <2433038...@qq.com> Gerrit-Comment-Date: Tue, 07 Nov 2023 02:54:08 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-12448: Avoid getting stuck when refreshing a non-existent partition
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/20490 ) Change subject: IMPALA-12448: Avoid getting stuck when refreshing a non-existent partition .. Patch Set 8: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/14347/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/20490 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Iace7cdadda300b03896f92415822266354421887 Gerrit-Change-Number: 20490 Gerrit-PatchSet: 8 Gerrit-Owner: ttz <2433038...@qq.com> Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Quanlong Huang Gerrit-Reviewer: ttz <2433038...@qq.com> Gerrit-Comment-Date: Tue, 07 Nov 2023 02:46:23 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-12448: Avoid getting stuck when refreshing a non-existent partition
Hello Quanlong Huang, Impala Public Jenkins, I'd like you to reexamine a change. Please visit http://gerrit.cloudera.org:8080/20490 to look at the new patch set (#9). Change subject: IMPALA-12448: Avoid getting stuck when refreshing a non-existent partition .. IMPALA-12448: Avoid getting stuck when refreshing a non-existent partition In some cases, we shouldn't wait for the version specified in 'result.version' because the table version has not changed after resetting metadata, such as refreshing a non-existent partition. If not, the query may be stuck for a long time. This patch determines whether to wait by comparing the table version before and after resetting. Change-Id: Iace7cdadda300b03896f92415822266354421887 --- M be/src/catalog/catalog-server.cc M be/src/util/backend-gflag-util.cc M common/thrift/BackendGflags.thrift M fe/src/main/java/org/apache/impala/catalog/TopicUpdateLog.java M fe/src/main/java/org/apache/impala/service/CatalogOpExecutor.java M tests/metadata/test_refresh_partition.py 6 files changed, 64 insertions(+), 4 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/90/20490/9 -- To view, visit http://gerrit.cloudera.org:8080/20490 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: Iace7cdadda300b03896f92415822266354421887 Gerrit-Change-Number: 20490 Gerrit-PatchSet: 9 Gerrit-Owner: ttz <2433038...@qq.com> Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Quanlong Huang Gerrit-Reviewer: ttz <2433038...@qq.com>
[Impala-ASF-CR] IMPALA-12448: Avoid getting stuck when refreshing a non-existent partition
Hello Quanlong Huang, Impala Public Jenkins, I'd like you to reexamine a change. Please visit http://gerrit.cloudera.org:8080/20490 to look at the new patch set (#8). Change subject: IMPALA-12448: Avoid getting stuck when refreshing a non-existent partition .. IMPALA-12448: Avoid getting stuck when refreshing a non-existent partition In some cases, we shouldn't wait for the version specified in 'result.version' because the table version has not changed after resetting metadata, such as refreshing a non-existent partition. If not, the query may be stuck for a long time. This patch determines whether to wait by comparing the table version before and after resetting. Change-Id: Iace7cdadda300b03896f92415822266354421887 --- M be/src/catalog/catalog-server.cc M be/src/util/backend-gflag-util.cc M common/thrift/BackendGflags.thrift M fe/src/main/java/org/apache/impala/catalog/TopicUpdateLog.java M fe/src/main/java/org/apache/impala/service/CatalogOpExecutor.java M tests/metadata/test_refresh_partition.py 6 files changed, 65 insertions(+), 4 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/90/20490/8 -- To view, visit http://gerrit.cloudera.org:8080/20490 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: Iace7cdadda300b03896f92415822266354421887 Gerrit-Change-Number: 20490 Gerrit-PatchSet: 8 Gerrit-Owner: ttz <2433038...@qq.com> Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Quanlong Huang Gerrit-Reviewer: ttz <2433038...@qq.com>
[Impala-ASF-CR] IMPALA-12448: Avoid getting stuck when refreshing a non-existent partition
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/20490 ) Change subject: IMPALA-12448: Avoid getting stuck when refreshing a non-existent partition .. Patch Set 8: (1 comment) http://gerrit.cloudera.org:8080/#/c/20490/8/tests/metadata/test_refresh_partition.py File tests/metadata/test_refresh_partition.py: http://gerrit.cloudera.org:8080/#/c/20490/8/tests/metadata/test_refresh_partition.py@24 PS8, Line 24: from time import sleep flake8: F401 'time.sleep' imported but unused -- To view, visit http://gerrit.cloudera.org:8080/20490 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Iace7cdadda300b03896f92415822266354421887 Gerrit-Change-Number: 20490 Gerrit-PatchSet: 8 Gerrit-Owner: ttz <2433038...@qq.com> Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Quanlong Huang Gerrit-Reviewer: ttz <2433038...@qq.com> Gerrit-Comment-Date: Tue, 07 Nov 2023 02:20:04 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-12516: Set HDFS limit based on memlock
Michael Smith has submitted this change and it was merged. ( http://gerrit.cloudera.org:8080/20623 ) Change subject: IMPALA-12516: Set HDFS limit based on memlock .. IMPALA-12516: Set HDFS limit based on memlock With RHEL 8 on AWS Graviton instances, dfs.datanode.max.locked.memory=64000 is insufficient to run query_test/test_hdfs_caching.py::TestHdfsCaching::test_table_is_cached. Sets dfs.datanode.max.locked.memory based on 'ulimit -l', and sets memlock to 64MB in bootstrap_system.sh to match modern defaults and provide space for future HDFS caching tests. New setting can be seen in admin output like node-1 will use ports DATANODE_PORT=31002, DATANODE_HTTP_PORT=31012, DATANODE_IPC_PORT=31022, DATANODE_HTTPS_PORT=31032, DATANODE_CLIENT_PORT=31042, NODEMANAGER_PORT=31102, NODEMANAGER_LOCALIZER_PORT=31122, NODEMANAGER_WEBUI_PORT=31142, KUDU_TS_RPC_PORT=31202, and KUDU_TS_WEBUI_PORT=31302; DATANODE_LOCKED_MEM=65536000 Change-Id: I7722ddd0c7fbd9bbd1979503952b7522b808194a Reviewed-on: http://gerrit.cloudera.org:8080/20623 Tested-by: Impala Public Jenkins Reviewed-by: Joe McDonnell --- M bin/bootstrap_system.sh M testdata/cluster/admin M testdata/cluster/node_templates/common/etc/hadoop/conf/hdfs-site.xml.tmpl 3 files changed, 19 insertions(+), 4 deletions(-) Approvals: Impala Public Jenkins: Verified Joe McDonnell: Looks good to me, approved -- To view, visit http://gerrit.cloudera.org:8080/20623 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: merged Gerrit-Change-Id: I7722ddd0c7fbd9bbd1979503952b7522b808194a Gerrit-Change-Number: 20623 Gerrit-PatchSet: 12 Gerrit-Owner: Michael Smith Gerrit-Reviewer: David Rorke Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Jason Fehr Gerrit-Reviewer: Joe McDonnell Gerrit-Reviewer: Laszlo Gaal Gerrit-Reviewer: Michael Smith
[Impala-ASF-CR] IMPALA-12516: Set HDFS limit based on memlock
Joe McDonnell has posted comments on this change. ( http://gerrit.cloudera.org:8080/20623 ) Change subject: IMPALA-12516: Set HDFS limit based on memlock .. Patch Set 11: Code-Review+2 This looks good to me, I'm glad that we're setting it to ~64MB, as that will be useful for other future tests. -- To view, visit http://gerrit.cloudera.org:8080/20623 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I7722ddd0c7fbd9bbd1979503952b7522b808194a Gerrit-Change-Number: 20623 Gerrit-PatchSet: 11 Gerrit-Owner: Michael Smith Gerrit-Reviewer: David Rorke Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Jason Fehr Gerrit-Reviewer: Joe McDonnell Gerrit-Reviewer: Laszlo Gaal Gerrit-Reviewer: Michael Smith Gerrit-Comment-Date: Tue, 07 Nov 2023 00:48:14 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-12527: Fix Iceberg metadata table test S3 paths
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/20659 ) Change subject: IMPALA-12527: Fix Iceberg metadata table test S3 paths .. Patch Set 2: Verified-1 Build failed: https://jenkins.impala.io/job/gerrit-verify-dryrun/9890/ -- To view, visit http://gerrit.cloudera.org:8080/20659 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I04ee4798c643f7cce59efa80b3327f7f37c80562 Gerrit-Change-Number: 20659 Gerrit-PatchSet: 2 Gerrit-Owner: Tamas Mate Gerrit-Reviewer: Gabor Kaszab Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Zoltan Borok-Nagy Gerrit-Comment-Date: Tue, 07 Nov 2023 00:35:54 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-10976: Sync db/table to latest HMS event for all DDL/DMLs
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/20367 ) Change subject: IMPALA-10976: Sync db/table to latest HMS event for all DDL/DMLs .. Patch Set 7: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/14346/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/20367 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ia250d0a943838086c187e5cb7c60035e5a564bbf Gerrit-Change-Number: 20367 Gerrit-PatchSet: 7 Gerrit-Owner: Sai Hemanth Gantasala Gerrit-Reviewer: Anonymous Coward Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Quanlong Huang Gerrit-Reviewer: Sai Hemanth Gantasala Gerrit-Comment-Date: Tue, 07 Nov 2023 00:17:49 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-12516: Set HDFS limit based on memlock
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/20623 ) Change subject: IMPALA-12516: Set HDFS limit based on memlock .. Patch Set 11: Verified+1 -- To view, visit http://gerrit.cloudera.org:8080/20623 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I7722ddd0c7fbd9bbd1979503952b7522b808194a Gerrit-Change-Number: 20623 Gerrit-PatchSet: 11 Gerrit-Owner: Michael Smith Gerrit-Reviewer: David Rorke Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Jason Fehr Gerrit-Reviewer: Joe McDonnell Gerrit-Reviewer: Laszlo Gaal Gerrit-Reviewer: Michael Smith Gerrit-Comment-Date: Tue, 07 Nov 2023 00:04:14 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-12542: Wait for query cancellation in webserver test
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/20668 ) Change subject: IMPALA-12542: Wait for query cancellation in webserver test .. Patch Set 3: Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/9892/ DRY_RUN=true -- To view, visit http://gerrit.cloudera.org:8080/20668 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I1c3bc83955be025e83a426377221767694baa444 Gerrit-Change-Number: 20668 Gerrit-PatchSet: 3 Gerrit-Owner: Michael Smith Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Michael Smith Gerrit-Reviewer: Riza Suminto Gerrit-Comment-Date: Tue, 07 Nov 2023 00:04:16 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-10976: Sync db/table to latest HMS event for all DDL/DMLs
Sai Hemanth Gantasala has posted comments on this change. (
http://gerrit.cloudera.org:8080/20367 )
Change subject: IMPALA-10976: Sync db/table to latest HMS event for all DDL/DMLs
..
Patch Set 7:
(4 comments)
http://gerrit.cloudera.org:8080/#/c/20367/6//COMMIT_MSG
Commit Message:
http://gerrit.cloudera.org:8080/#/c/20367/6//COMMIT_MSG@7
PS6, Line 7: IMPALA-10976: Sync db/table to latest HMS event
: for all DDL/DMLs
:
> nit: I think we can make it in one line and shorter, e.g.
Ack
http://gerrit.cloudera.org:8080/#/c/20367/4/fe/src/main/java/org/apache/impala/service/CatalogOpExecutor.java
File fe/src/main/java/org/apache/impala/service/CatalogOpExecutor.java:
http://gerrit.cloudera.org:8080/#/c/20367/4/fe/src/main/java/org/apache/impala/service/CatalogOpExecutor.java@7040
PS4, Line 7040: try {
:
HdfsCachingUtil.removePartitionCacheDirective(part.getParameters());
: } catch (ImpalaException e1) {
: String msg = String.format(
: "Partition %s.%s(%s): State: Leaked
caching directive. " +
: "Action: Manually uncache directory
%s via hdfs " +
: "cacheAdmin.", part.getDbName(),
part.getTableName(),
: part.getValues(),
part.getSd().getLocation());
: LOG.error(msg, e);
:
> Agree that we can keep invalidate as-is. For REFRESH, I think at least we s
Ack
http://gerrit.cloudera.org:8080/#/c/20367/6/fe/src/main/java/org/apache/impala/service/CatalogOpExecutor.java
File fe/src/main/java/org/apache/impala/service/CatalogOpExecutor.java:
http://gerrit.cloudera.org:8080/#/c/20367/6/fe/src/main/java/org/apache/impala/service/CatalogOpExecutor.java@1145
PS6, Line 1145: if (refreshedTable != null) {
> I think we should skip this as well when enableSyncToLatestEventOnDdls is o
Ack
http://gerrit.cloudera.org:8080/#/c/20367/6/fe/src/main/java/org/apache/impala/service/CatalogOpExecutor.java@1185
PS6, Line 1185: if (refreshedTable != null) {
> Same issue as L1145, this might reset the catalog version to an older one w
Ack
--
To view, visit http://gerrit.cloudera.org:8080/20367
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Ia250d0a943838086c187e5cb7c60035e5a564bbf
Gerrit-Change-Number: 20367
Gerrit-PatchSet: 7
Gerrit-Owner: Sai Hemanth Gantasala
Gerrit-Reviewer: Anonymous Coward
Gerrit-Reviewer: Csaba Ringhofer
Gerrit-Reviewer: Impala Public Jenkins
Gerrit-Reviewer: Quanlong Huang
Gerrit-Reviewer: Sai Hemanth Gantasala
Gerrit-Comment-Date: Mon, 06 Nov 2023 23:50:13 +
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-10976: Sync db/table to latest HMS event for all DDL/DMLs
Hello Quanlong Huang, k.venureddy2...@gmail.com, Csaba Ringhofer, Impala Public Jenkins, I'd like you to reexamine a change. Please visit http://gerrit.cloudera.org:8080/20367 to look at the new patch set (#7). Change subject: IMPALA-10976: Sync db/table to latest HMS event for all DDL/DMLs .. IMPALA-10976: Sync db/table to latest HMS event for all DDL/DMLs The idea is that when any DDL/DML operation is performed by Impala, it also syncs the db/table to its latest event ID as per HMS. This way updates to a db/table's are applied in the same order as they appear in the Notification log table in HMS which ensures consistency. Currently catalogD applies any updates received from Impala clients in-place. Instead it should perform an HMS operation first and then replay all the HMS events since the last synced event id. Implementation: when the enable_sync_to_latest_event_on_ddls flag is set to true, we do the DDL/DML operation first, i.e., perform HMS operation and then sync the db/table in the catalogD's cache to the latest event in HMS for the corresponding db/table. Currently we fetch all events greater than the db/table's lastSyncEventId and filter them in the events processor to sync only the current db/table events. Once HIVE-27499 is implemented, we can directly fetch the events only for the respective db/table and process them. Currently, there is no efficient way to identify if there are pending events for a db/table. Set 'enable_sync_to_latest_event_on_ddls' to true and 'invalidate_hms_cache_on_ddls' to false to use this feature. Note: We don't modify the cache using MetastoreEventsProcessor for alter table rename operation as this is a complex operation regarding cache modification. We modify cache using above process for 'refresh ' command only. This proceedure is not valid for the following commands 'refresh functions ', 'refresh ', 'refresh authorization', 'invalidate metadata '. Testing: 1) Added few tests in the MetaStoreEventProcessorForTest to verify this feature that simulates the metadata sync between HMS and Impala. 2) Added few tests in the CatalogHmsSyncToLatestEventIdTest class to the metadata sync between HMS end point, Catalog Metastore Server and Impala. The HMS end point serves as common interface to metadata changes outside the current Impala service such as Hive, Spark or other Impala service. Also verified the table lastSyncEventId is updated after the events are sync and confirmed that metastore event processor ignored these synced events. Change-Id: Ia250d0a943838086c187e5cb7c60035e5a564bbf --- M fe/src/main/java/org/apache/impala/catalog/events/MetastoreEventsProcessor.java M fe/src/main/java/org/apache/impala/service/BackendConfig.java M fe/src/main/java/org/apache/impala/service/CatalogOpExecutor.java M fe/src/test/java/org/apache/impala/catalog/events/MetastoreEventsProcessorTest.java M fe/src/test/java/org/apache/impala/catalog/metastore/CatalogHmsSyncToLatestEventIdTest.java 5 files changed, 667 insertions(+), 139 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/67/20367/7 -- To view, visit http://gerrit.cloudera.org:8080/20367 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: Ia250d0a943838086c187e5cb7c60035e5a564bbf Gerrit-Change-Number: 20367 Gerrit-PatchSet: 7 Gerrit-Owner: Sai Hemanth Gantasala Gerrit-Reviewer: Anonymous Coward Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Quanlong Huang Gerrit-Reviewer: Sai Hemanth Gantasala
[Impala-ASF-CR] IMPALA-12511: Bump thrift version to fix THRIFT-5670
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/20660 ) Change subject: IMPALA-12511: Bump thrift version to fix THRIFT-5670 .. Patch Set 3: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/14345/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/20660 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I02f396fd0c27cde03883c06d6376feaf07ca8c13 Gerrit-Change-Number: 20660 Gerrit-PatchSet: 3 Gerrit-Owner: Quanlong Huang Gerrit-Reviewer: Daniel Becker Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Michael Smith Gerrit-Reviewer: Quanlong Huang Gerrit-Reviewer: Wenzhe Zhou Gerrit-Comment-Date: Mon, 06 Nov 2023 23:40:45 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-12390 (part 4): Enable unnecessary-value-param
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/20494 ) Change subject: IMPALA-12390 (part 4): Enable unnecessary-value-param .. Patch Set 10: Verified+1 -- To view, visit http://gerrit.cloudera.org:8080/20494 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I8aa5d98596d82f615a0a728e0235e7dd9d8b5003 Gerrit-Change-Number: 20494 Gerrit-PatchSet: 10 Gerrit-Owner: Michael Smith Gerrit-Reviewer: Anonymous Coward Gerrit-Reviewer: Daniel Becker Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Joe McDonnell Gerrit-Reviewer: Michael Smith Gerrit-Comment-Date: Mon, 06 Nov 2023 23:36:47 + Gerrit-HasComments: No
[Impala-ASF-CR] [WIP] IMPALA-3268: Add support for SHOW VIEWS statement
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/20669 ) Change subject: [WIP] IMPALA-3268: Add support for SHOW VIEWS statement .. Patch Set 1: Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/9891/ DRY_RUN=true -- To view, visit http://gerrit.cloudera.org:8080/20669 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I321fc5350392a815949a4e7d2a64d60466689788 Gerrit-Change-Number: 20669 Gerrit-PatchSet: 1 Gerrit-Owner: Fang-Yu Rao Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Michael Smith Gerrit-Comment-Date: Mon, 06 Nov 2023 23:36:49 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-12511: Bump thrift version to fix THRIFT-5670
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/20660 ) Change subject: IMPALA-12511: Bump thrift version to fix THRIFT-5670 .. Patch Set 2: Verified-1 Build failed: https://jenkins.impala.io/job/gerrit-verify-dryrun/9886/ -- To view, visit http://gerrit.cloudera.org:8080/20660 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I02f396fd0c27cde03883c06d6376feaf07ca8c13 Gerrit-Change-Number: 20660 Gerrit-PatchSet: 2 Gerrit-Owner: Quanlong Huang Gerrit-Reviewer: Daniel Becker Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Michael Smith Gerrit-Reviewer: Quanlong Huang Gerrit-Reviewer: Wenzhe Zhou Gerrit-Comment-Date: Mon, 06 Nov 2023 23:29:56 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-12542: Wait for query cancellation in webserver test
Riza Suminto has posted comments on this change. (
http://gerrit.cloudera.org:8080/20668 )
Change subject: IMPALA-12542: Wait for query cancellation in webserver test
..
Patch Set 3:
(1 comment)
http://gerrit.cloudera.org:8080/#/c/20668/2/tests/webserver/test_web_pages.py
File tests/webserver/test_web_pages.py:
http://gerrit.cloudera.org:8080/#/c/20668/2/tests/webserver/test_web_pages.py@962
PS2, Line 962: response_json = self.try_until("query cancellation",
self.get_queries,
: lambda resp: resp['num_in_flight_queries'] == 0
> The timeout is 10s. You think that's insufficient?
I'd maybe go with 30s with 0.5s interval to avoid flakiness. But 10 might be
enough since this run serially.
--
To view, visit http://gerrit.cloudera.org:8080/20668
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I1c3bc83955be025e83a426377221767694baa444
Gerrit-Change-Number: 20668
Gerrit-PatchSet: 3
Gerrit-Owner: Michael Smith
Gerrit-Reviewer: Impala Public Jenkins
Gerrit-Reviewer: Michael Smith
Gerrit-Reviewer: Riza Suminto
Gerrit-Comment-Date: Mon, 06 Nov 2023 23:16:41 +
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-12511: Bump thrift version to fix THRIFT-5670
Michael Smith has posted comments on this change. ( http://gerrit.cloudera.org:8080/20660 ) Change subject: IMPALA-12511: Bump thrift version to fix THRIFT-5670 .. Patch Set 3: Code-Review+2 -- To view, visit http://gerrit.cloudera.org:8080/20660 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I02f396fd0c27cde03883c06d6376feaf07ca8c13 Gerrit-Change-Number: 20660 Gerrit-PatchSet: 3 Gerrit-Owner: Quanlong Huang Gerrit-Reviewer: Daniel Becker Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Michael Smith Gerrit-Reviewer: Quanlong Huang Gerrit-Reviewer: Wenzhe Zhou Gerrit-Comment-Date: Mon, 06 Nov 2023 23:14:43 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-12511: Bump thrift version to fix THRIFT-5670
Quanlong Huang has posted comments on this change. (
http://gerrit.cloudera.org:8080/20660 )
Change subject: IMPALA-12511: Bump thrift version to fix THRIFT-5670
..
Patch Set 3:
(1 comment)
http://gerrit.cloudera.org:8080/#/c/20660/2/bin/impala-config.sh
File bin/impala-config.sh:
http://gerrit.cloudera.org:8080/#/c/20660/2/bin/impala-config.sh@94
PS2, Line 94:
${IMPALA_TOOLCHAIN_COMMIT_HASH-c22e6eb3b95979bdd7a2ec8f86abb751b439f7a4}
> This hash should be updated. Unless we still need to merge into native-tool
Oops, done.
--
To view, visit http://gerrit.cloudera.org:8080/20660
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I02f396fd0c27cde03883c06d6376feaf07ca8c13
Gerrit-Change-Number: 20660
Gerrit-PatchSet: 3
Gerrit-Owner: Quanlong Huang
Gerrit-Reviewer: Daniel Becker
Gerrit-Reviewer: Impala Public Jenkins
Gerrit-Reviewer: Michael Smith
Gerrit-Reviewer: Quanlong Huang
Gerrit-Reviewer: Wenzhe Zhou
Gerrit-Comment-Date: Mon, 06 Nov 2023 23:13:57 +
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-12511: Bump thrift version to fix THRIFT-5670
Hello Daniel Becker, Wenzhe Zhou, Michael Smith, Impala Public Jenkins, I'd like you to reexamine a change. Please visit http://gerrit.cloudera.org:8080/20660 to look at the new patch set (#3). Change subject: IMPALA-12511: Bump thrift version to fix THRIFT-5670 .. IMPALA-12511: Bump thrift version to fix THRIFT-5670 This bumps the cpp thrift version to fix THRIFT-5670 which causes a confusing timeout log. The issue only happens in thrift-0.16+ so we don't need it in thrift-0.11.0-p5. Tests: - Built Impala locally with the fix and verified the logs. Change-Id: I02f396fd0c27cde03883c06d6376feaf07ca8c13 --- M bin/impala-config.sh 1 file changed, 6 insertions(+), 6 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/60/20660/3 -- To view, visit http://gerrit.cloudera.org:8080/20660 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: I02f396fd0c27cde03883c06d6376feaf07ca8c13 Gerrit-Change-Number: 20660 Gerrit-PatchSet: 3 Gerrit-Owner: Quanlong Huang Gerrit-Reviewer: Daniel Becker Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Michael Smith Gerrit-Reviewer: Wenzhe Zhou
[Impala-ASF-CR] [WIP] IMPALA-3268: Add support for SHOW VIEWS statement
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/20669 ) Change subject: [WIP] IMPALA-3268: Add support for SHOW VIEWS statement .. Patch Set 1: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/14344/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/20669 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I321fc5350392a815949a4e7d2a64d60466689788 Gerrit-Change-Number: 20669 Gerrit-PatchSet: 1 Gerrit-Owner: Fang-Yu Rao Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Michael Smith Gerrit-Comment-Date: Mon, 06 Nov 2023 22:43:27 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-12542: Wait for query cancellation in webserver test
Michael Smith has posted comments on this change. ( http://gerrit.cloudera.org:8080/20668 ) Change subject: IMPALA-12542: Wait for query cancellation in webserver test .. Patch Set 3: Code-Review+2 Carrying +2 for minor commit title update. -- To view, visit http://gerrit.cloudera.org:8080/20668 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I1c3bc83955be025e83a426377221767694baa444 Gerrit-Change-Number: 20668 Gerrit-PatchSet: 3 Gerrit-Owner: Michael Smith Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Michael Smith Gerrit-Reviewer: Riza Suminto Gerrit-Comment-Date: Mon, 06 Nov 2023 22:35:46 + Gerrit-HasComments: No
[Impala-ASF-CR] [WIP] IMPALA-3268: Add support for SHOW VIEWS statement
Michael Smith has posted comments on this change. ( http://gerrit.cloudera.org:8080/20669 ) Change subject: [WIP] IMPALA-3268: Add support for SHOW VIEWS statement .. Patch Set 1: (1 comment) http://gerrit.cloudera.org:8080/#/c/20669/1//COMMIT_MSG Commit Message: http://gerrit.cloudera.org:8080/#/c/20669/1//COMMIT_MSG@7 PS1, Line 7: [WIP] IMPALA-3268: Add support for SHOW VIEWS statement I'd love an example of what SHOW VIEWS looks like. -- To view, visit http://gerrit.cloudera.org:8080/20669 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I321fc5350392a815949a4e7d2a64d60466689788 Gerrit-Change-Number: 20669 Gerrit-PatchSet: 1 Gerrit-Owner: Fang-Yu Rao Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Michael Smith Gerrit-Comment-Date: Mon, 06 Nov 2023 22:35:07 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-12542: Wait for query cancellation in webserver test
Hello Riza Suminto, Impala Public Jenkins, I'd like you to reexamine a change. Please visit http://gerrit.cloudera.org:8080/20668 to look at the new patch set (#3). Change subject: IMPALA-12542: Wait for query cancellation in webserver test .. IMPALA-12542: Wait for query cancellation in webserver test Waits for query cancellation in web cancellation tests as cancellation may not be finalized when the API returns success in slow environments (such as ASAN). Testing: passed 100 iterations with ASAN. Change-Id: I1c3bc83955be025e83a426377221767694baa444 --- M tests/webserver/test_web_pages.py 1 file changed, 4 insertions(+), 1 deletion(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/68/20668/3 -- To view, visit http://gerrit.cloudera.org:8080/20668 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: I1c3bc83955be025e83a426377221767694baa444 Gerrit-Change-Number: 20668 Gerrit-PatchSet: 3 Gerrit-Owner: Michael Smith Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Michael Smith Gerrit-Reviewer: Riza Suminto
[Impala-ASF-CR] IMPALA-12542: Wait for query cancellation
Michael Smith has posted comments on this change. (
http://gerrit.cloudera.org:8080/20668 )
Change subject: IMPALA-12542: Wait for query cancellation
..
Patch Set 2:
(1 comment)
http://gerrit.cloudera.org:8080/#/c/20668/2/tests/webserver/test_web_pages.py
File tests/webserver/test_web_pages.py:
http://gerrit.cloudera.org:8080/#/c/20668/2/tests/webserver/test_web_pages.py@962
PS2, Line 962: response_json = self.try_until("query cancellation",
self.get_queries,
: lambda resp: resp['num_in_flight_queries'] == 0
> nit: might want to set larger timeout & interval, just in case.
The timeout is 10s. You think that's insufficient?
--
To view, visit http://gerrit.cloudera.org:8080/20668
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I1c3bc83955be025e83a426377221767694baa444
Gerrit-Change-Number: 20668
Gerrit-PatchSet: 2
Gerrit-Owner: Michael Smith
Gerrit-Reviewer: Impala Public Jenkins
Gerrit-Reviewer: Michael Smith
Gerrit-Reviewer: Riza Suminto
Gerrit-Comment-Date: Mon, 06 Nov 2023 22:33:04 +
Gerrit-HasComments: Yes
[Impala-ASF-CR] [WIP] IMPALA-3268: Add support for SHOW VIEWS statement
Fang-Yu Rao has uploaded this change for review. ( http://gerrit.cloudera.org:8080/20669 Change subject: [WIP] IMPALA-3268: Add support for SHOW VIEWS statement .. [WIP] IMPALA-3268: Add support for SHOW VIEWS statement To-do's: - Update code comments. - Add end-to-end tests. Testing: - Briefly verified that SHOW VIEWS statement works whether Impala daemon's startup flag '--use_local_catalog' is true. Change-Id: I321fc5350392a815949a4e7d2a64d60466689788 --- M be/src/service/client-request-state.cc M be/src/service/frontend.cc M be/src/service/frontend.h M common/thrift/Frontend.thrift M fe/src/main/cup/sql-parser.cup M fe/src/main/java/org/apache/impala/analysis/AnalysisContext.java A fe/src/main/java/org/apache/impala/analysis/ShowTablesOrViewsStmt.java M fe/src/main/java/org/apache/impala/analysis/ShowTablesStmt.java A fe/src/main/java/org/apache/impala/analysis/ShowViewsStmt.java M fe/src/main/java/org/apache/impala/catalog/Catalog.java M fe/src/main/java/org/apache/impala/catalog/Db.java M fe/src/main/java/org/apache/impala/catalog/FeCatalog.java M fe/src/main/java/org/apache/impala/catalog/FeDb.java M fe/src/main/java/org/apache/impala/catalog/local/LocalCatalog.java M fe/src/main/java/org/apache/impala/catalog/local/LocalDb.java M fe/src/main/java/org/apache/impala/service/Frontend.java M fe/src/main/java/org/apache/impala/service/JniCatalog.java M fe/src/main/java/org/apache/impala/service/JniFrontend.java M fe/src/main/jflex/sql-scanner.flex 19 files changed, 284 insertions(+), 96 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/69/20669/1 -- To view, visit http://gerrit.cloudera.org:8080/20669 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newchange Gerrit-Change-Id: I321fc5350392a815949a4e7d2a64d60466689788 Gerrit-Change-Number: 20669 Gerrit-PatchSet: 1 Gerrit-Owner: Fang-Yu Rao
[Impala-ASF-CR] IMPALA-12542: Wait for query cancellation
Riza Suminto has posted comments on this change. (
http://gerrit.cloudera.org:8080/20668 )
Change subject: IMPALA-12542: Wait for query cancellation
..
Patch Set 2: Code-Review+2
(1 comment)
Thanks for looking into this!
http://gerrit.cloudera.org:8080/#/c/20668/2/tests/webserver/test_web_pages.py
File tests/webserver/test_web_pages.py:
http://gerrit.cloudera.org:8080/#/c/20668/2/tests/webserver/test_web_pages.py@962
PS2, Line 962: response_json = self.try_until("query cancellation",
self.get_queries,
: lambda resp: resp['num_in_flight_queries'] == 0
nit: might want to set larger timeout & interval, just in case.
--
To view, visit http://gerrit.cloudera.org:8080/20668
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I1c3bc83955be025e83a426377221767694baa444
Gerrit-Change-Number: 20668
Gerrit-PatchSet: 2
Gerrit-Owner: Michael Smith
Gerrit-Reviewer: Impala Public Jenkins
Gerrit-Reviewer: Michael Smith
Gerrit-Reviewer: Riza Suminto
Gerrit-Comment-Date: Mon, 06 Nov 2023 22:06:39 +
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-12542: Wait for query cancellation
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/20668 ) Change subject: IMPALA-12542: Wait for query cancellation .. Patch Set 1: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/14343/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/20668 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I1c3bc83955be025e83a426377221767694baa444 Gerrit-Change-Number: 20668 Gerrit-PatchSet: 1 Gerrit-Owner: Michael Smith Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Michael Smith Gerrit-Reviewer: Riza Suminto Gerrit-Comment-Date: Mon, 06 Nov 2023 21:50:19 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-12545: Use 'unique database' in test restart services.py::TestRestart
Impala Public Jenkins has submitted this change and it was merged. ( http://gerrit.cloudera.org:8080/20662 ) Change subject: IMPALA-12545: Use 'unique_database' in test_restart_services.py::TestRestart .. IMPALA-12545: Use 'unique_database' in test_restart_services.py::TestRestart Some tests in 'test_restart_services.py::TestRestart' create tables but don't use a 'unique_database' for it. The table and column names are sometimes the same. The tests are run serially but if a table is not deleted successfully it may interfere with other tests. This change introduces 'unique_database' in tests that create tables. After this, explicitly deleting the tables is no longer necessary as the framework takes care of it. Testing: - all affected tests pass Change-Id: Id4c2cfb669d5ff9e4d8110a0035bae1147e2db5a Reviewed-on: http://gerrit.cloudera.org:8080/20662 Reviewed-by: Impala Public Jenkins Tested-by: Impala Public Jenkins --- M tests/custom_cluster/test_restart_services.py 1 file changed, 46 insertions(+), 45 deletions(-) Approvals: Impala Public Jenkins: Looks good to me, approved; Verified -- To view, visit http://gerrit.cloudera.org:8080/20662 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: merged Gerrit-Change-Id: Id4c2cfb669d5ff9e4d8110a0035bae1147e2db5a Gerrit-Change-Number: 20662 Gerrit-PatchSet: 3 Gerrit-Owner: Daniel Becker Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Noemi Pap-Takacs Gerrit-Reviewer: Riza Suminto
[Impala-ASF-CR] IMPALA-12545: Use 'unique database' in test restart services.py::TestRestart
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/20662 ) Change subject: IMPALA-12545: Use 'unique_database' in test_restart_services.py::TestRestart .. Patch Set 2: Verified+1 -- To view, visit http://gerrit.cloudera.org:8080/20662 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Id4c2cfb669d5ff9e4d8110a0035bae1147e2db5a Gerrit-Change-Number: 20662 Gerrit-PatchSet: 2 Gerrit-Owner: Daniel Becker Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Noemi Pap-Takacs Gerrit-Reviewer: Riza Suminto Gerrit-Comment-Date: Mon, 06 Nov 2023 21:41:27 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-12542: Wait for query cancellation
Hello Impala Public Jenkins, I'd like you to reexamine a change. Please visit http://gerrit.cloudera.org:8080/20668 to look at the new patch set (#2). Change subject: IMPALA-12542: Wait for query cancellation .. IMPALA-12542: Wait for query cancellation Waits for query cancellation in web cancellation tests as cancellation may not be finalized when the API returns success in slow environments (such as ASAN). Testing: passed 100 iterations with ASAN. Change-Id: I1c3bc83955be025e83a426377221767694baa444 --- M tests/webserver/test_web_pages.py 1 file changed, 4 insertions(+), 1 deletion(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/68/20668/2 -- To view, visit http://gerrit.cloudera.org:8080/20668 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: I1c3bc83955be025e83a426377221767694baa444 Gerrit-Change-Number: 20668 Gerrit-PatchSet: 2 Gerrit-Owner: Michael Smith Gerrit-Reviewer: Impala Public Jenkins
[Impala-ASF-CR] IMPALA-12542: Wait for query cancellation
Michael Smith has uploaded this change for review. ( http://gerrit.cloudera.org:8080/20668 Change subject: IMPALA-12542: Wait for query cancellation .. IMPALA-12542: Wait for query cancellation Waits for query cancellation in web cancellation tests as cancellation may not be finalized when the API returns success in slow environments (such as ASAN). Change-Id: I1c3bc83955be025e83a426377221767694baa444 --- M tests/webserver/test_web_pages.py 1 file changed, 4 insertions(+), 1 deletion(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/68/20668/1 -- To view, visit http://gerrit.cloudera.org:8080/20668 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newchange Gerrit-Change-Id: I1c3bc83955be025e83a426377221767694baa444 Gerrit-Change-Number: 20668 Gerrit-PatchSet: 1 Gerrit-Owner: Michael Smith
[Impala-ASF-CR] IMPALA-12527: Fix Iceberg metadata table test S3 paths
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/20659 ) Change subject: IMPALA-12527: Fix Iceberg metadata table test S3 paths .. Patch Set 2: Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/9890/ DRY_RUN=true -- To view, visit http://gerrit.cloudera.org:8080/20659 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I04ee4798c643f7cce59efa80b3327f7f37c80562 Gerrit-Change-Number: 20659 Gerrit-PatchSet: 2 Gerrit-Owner: Tamas Mate Gerrit-Reviewer: Gabor Kaszab Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Zoltan Borok-Nagy Gerrit-Comment-Date: Mon, 06 Nov 2023 19:59:02 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-12516: Set HDFS limit based on memlock
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/20623 ) Change subject: IMPALA-12516: Set HDFS limit based on memlock .. Patch Set 11: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/14342/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/20623 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I7722ddd0c7fbd9bbd1979503952b7522b808194a Gerrit-Change-Number: 20623 Gerrit-PatchSet: 11 Gerrit-Owner: Michael Smith Gerrit-Reviewer: David Rorke Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Jason Fehr Gerrit-Reviewer: Joe McDonnell Gerrit-Reviewer: Laszlo Gaal Gerrit-Reviewer: Michael Smith Gerrit-Comment-Date: Mon, 06 Nov 2023 19:58:55 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-12516: Set HDFS limit based on memlock
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/20623 ) Change subject: IMPALA-12516: Set HDFS limit based on memlock .. Patch Set 11: Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/9889/ DRY_RUN=true -- To view, visit http://gerrit.cloudera.org:8080/20623 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I7722ddd0c7fbd9bbd1979503952b7522b808194a Gerrit-Change-Number: 20623 Gerrit-PatchSet: 11 Gerrit-Owner: Michael Smith Gerrit-Reviewer: David Rorke Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Jason Fehr Gerrit-Reviewer: Joe McDonnell Gerrit-Reviewer: Laszlo Gaal Gerrit-Reviewer: Michael Smith Gerrit-Comment-Date: Mon, 06 Nov 2023 19:32:04 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-12516: Set HDFS limit based on memlock
Hello Laszlo Gaal, Jason Fehr, David Rorke, Joe McDonnell, Impala Public Jenkins, I'd like you to reexamine a change. Please visit http://gerrit.cloudera.org:8080/20623 to look at the new patch set (#11). Change subject: IMPALA-12516: Set HDFS limit based on memlock .. IMPALA-12516: Set HDFS limit based on memlock With RHEL 8 on AWS Graviton instances, dfs.datanode.max.locked.memory=64000 is insufficient to run query_test/test_hdfs_caching.py::TestHdfsCaching::test_table_is_cached. Sets dfs.datanode.max.locked.memory based on 'ulimit -l', and sets memlock to 64MB in bootstrap_system.sh to match modern defaults and provide space for future HDFS caching tests. New setting can be seen in admin output like node-1 will use ports DATANODE_PORT=31002, DATANODE_HTTP_PORT=31012, DATANODE_IPC_PORT=31022, DATANODE_HTTPS_PORT=31032, DATANODE_CLIENT_PORT=31042, NODEMANAGER_PORT=31102, NODEMANAGER_LOCALIZER_PORT=31122, NODEMANAGER_WEBUI_PORT=31142, KUDU_TS_RPC_PORT=31202, and KUDU_TS_WEBUI_PORT=31302; DATANODE_LOCKED_MEM=65536000 Change-Id: I7722ddd0c7fbd9bbd1979503952b7522b808194a --- M bin/bootstrap_system.sh M testdata/cluster/admin M testdata/cluster/node_templates/common/etc/hadoop/conf/hdfs-site.xml.tmpl 3 files changed, 19 insertions(+), 4 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/23/20623/11 -- To view, visit http://gerrit.cloudera.org:8080/20623 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: I7722ddd0c7fbd9bbd1979503952b7522b808194a Gerrit-Change-Number: 20623 Gerrit-PatchSet: 11 Gerrit-Owner: Michael Smith Gerrit-Reviewer: David Rorke Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Jason Fehr Gerrit-Reviewer: Joe McDonnell Gerrit-Reviewer: Laszlo Gaal Gerrit-Reviewer: Michael Smith
[Impala-ASF-CR] IMPALA-12516: Set HDFS limit based on memlock
Michael Smith has posted comments on this change. ( http://gerrit.cloudera.org:8080/20623 ) Change subject: IMPALA-12516: Set HDFS limit based on memlock .. Patch Set 10: (2 comments) http://gerrit.cloudera.org:8080/#/c/20623/10/bin/bootstrap_system.sh File bin/bootstrap_system.sh: http://gerrit.cloudera.org:8080/#/c/20623/10/bin/bootstrap_system.sh@478 PS10, Line 478: sudo tee > ah... so you needed the " | sudo tee ... " form to make the change stick? My issue was actually that I was connecting over SSH, and sshd was configured with UsePAM=no. Enabling that fixed it. This setting does require a new session as far as I can tell. The `sudo tee` is just a nice way to write with sudo. I'm just following the pattern for others, although I didn't add `-` for append. http://gerrit.cloudera.org:8080/#/c/20623/10/testdata/cluster/admin File testdata/cluster/admin: http://gerrit.cloudera.org:8080/#/c/20623/10/testdata/cluster/admin@253 PS10, Line 253: let > nit: maybe use the $((...)) for for consistency with the following lines? Done -- To view, visit http://gerrit.cloudera.org:8080/20623 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I7722ddd0c7fbd9bbd1979503952b7522b808194a Gerrit-Change-Number: 20623 Gerrit-PatchSet: 10 Gerrit-Owner: Michael Smith Gerrit-Reviewer: David Rorke Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Jason Fehr Gerrit-Reviewer: Joe McDonnell Gerrit-Reviewer: Laszlo Gaal Gerrit-Reviewer: Michael Smith Gerrit-Comment-Date: Mon, 06 Nov 2023 19:31:38 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-12516: Set HDFS limit based on memlock
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/20623 ) Change subject: IMPALA-12516: Set HDFS limit based on memlock .. Patch Set 10: Verified-1 Build failed: https://jenkins.impala.io/job/gerrit-verify-dryrun/9887/ -- To view, visit http://gerrit.cloudera.org:8080/20623 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I7722ddd0c7fbd9bbd1979503952b7522b808194a Gerrit-Change-Number: 20623 Gerrit-PatchSet: 10 Gerrit-Owner: Michael Smith Gerrit-Reviewer: David Rorke Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Jason Fehr Gerrit-Reviewer: Joe McDonnell Gerrit-Reviewer: Laszlo Gaal Gerrit-Reviewer: Michael Smith Gerrit-Comment-Date: Mon, 06 Nov 2023 19:28:37 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-12516: Set HDFS limit based on memlock
Michael Smith has removed a vote on this change. Change subject: IMPALA-12516: Set HDFS limit based on memlock .. Removed Verified-1 by Impala Public Jenkins -- To view, visit http://gerrit.cloudera.org:8080/20623 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: deleteVote Gerrit-Change-Id: I7722ddd0c7fbd9bbd1979503952b7522b808194a Gerrit-Change-Number: 20623 Gerrit-PatchSet: 10 Gerrit-Owner: Michael Smith Gerrit-Reviewer: David Rorke Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Jason Fehr Gerrit-Reviewer: Joe McDonnell Gerrit-Reviewer: Laszlo Gaal Gerrit-Reviewer: Michael Smith
[Impala-ASF-CR] IMPALA-12511: Bump thrift version to fix THRIFT-5670
Wenzhe Zhou has removed a vote on this change. Change subject: IMPALA-12511: Bump thrift version to fix THRIFT-5670 .. Removed Code-Review+2 by Impala Public Jenkins -- To view, visit http://gerrit.cloudera.org:8080/20660 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: deleteVote Gerrit-Change-Id: I02f396fd0c27cde03883c06d6376feaf07ca8c13 Gerrit-Change-Number: 20660 Gerrit-PatchSet: 2 Gerrit-Owner: Quanlong Huang Gerrit-Reviewer: Daniel Becker Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Michael Smith Gerrit-Reviewer: Wenzhe Zhou
[Impala-ASF-CR] IMPALA-11762: [DOCS] Reserved words documentation lags behind the code Crosschecked keywordMap from: https://github.com/apache/impala/blob/master/fe/src/main/jflex/sql-scanner.flex wit
Laszlo Gaal has posted comments on this change. ( http://gerrit.cloudera.org:8080/20605 ) Change subject: IMPALA-11762: [DOCS] Reserved words documentation lags behind the code Crosschecked keywordMap from: https://github.com/apache/impala/blob/master/fe/src/main/jflex/sql-scanner.flex with upstream docs: https://impala.apache.org/docs/build/html/topics/impal .. Patch Set 1: > Patch Set 1: > > (1 comment) > > Thank you for taking care of this! @riddhij16, are you planning to revisit this change? Would be great to see you reach "first commit" status with Impala :) -- To view, visit http://gerrit.cloudera.org:8080/20605 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I0ae58a4730c2e3d8d82cccdff23c1fff36117522 Gerrit-Change-Number: 20605 Gerrit-PatchSet: 1 Gerrit-Owner: Anonymous Coward Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Laszlo Gaal Gerrit-Comment-Date: Mon, 06 Nov 2023 19:11:55 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-12511: Bump thrift version to fix THRIFT-5670
Michael Smith has posted comments on this change. (
http://gerrit.cloudera.org:8080/20660 )
Change subject: IMPALA-12511: Bump thrift version to fix THRIFT-5670
..
Patch Set 2:
(1 comment)
http://gerrit.cloudera.org:8080/#/c/20660/2/bin/impala-config.sh
File bin/impala-config.sh:
http://gerrit.cloudera.org:8080/#/c/20660/2/bin/impala-config.sh@94
PS2, Line 94:
${IMPALA_TOOLCHAIN_COMMIT_HASH-4b0b632be50a166c29002cb5d51b2991267c7ab1}
This hash should be updated. Unless we still need to merge into
native-toolchain?
--
To view, visit http://gerrit.cloudera.org:8080/20660
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I02f396fd0c27cde03883c06d6376feaf07ca8c13
Gerrit-Change-Number: 20660
Gerrit-PatchSet: 2
Gerrit-Owner: Quanlong Huang
Gerrit-Reviewer: Daniel Becker
Gerrit-Reviewer: Impala Public Jenkins
Gerrit-Reviewer: Michael Smith
Gerrit-Reviewer: Wenzhe Zhou
Gerrit-Comment-Date: Mon, 06 Nov 2023 19:11:43 +
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-12516: Set HDFS limit based on memlock
Laszlo Gaal has posted comments on this change. ( http://gerrit.cloudera.org:8080/20623 ) Change subject: IMPALA-12516: Set HDFS limit based on memlock .. Patch Set 10: Code-Review+1 (2 comments) http://gerrit.cloudera.org:8080/#/c/20623/10/bin/bootstrap_system.sh File bin/bootstrap_system.sh: http://gerrit.cloudera.org:8080/#/c/20623/10/bin/bootstrap_system.sh@478 PS10, Line 478: sudo tee ah... so you needed the " | sudo tee ... " form to make the change stick? http://gerrit.cloudera.org:8080/#/c/20623/10/testdata/cluster/admin File testdata/cluster/admin: http://gerrit.cloudera.org:8080/#/c/20623/10/testdata/cluster/admin@253 PS10, Line 253: let nit: maybe use the $((...)) for for consistency with the following lines? -- To view, visit http://gerrit.cloudera.org:8080/20623 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I7722ddd0c7fbd9bbd1979503952b7522b808194a Gerrit-Change-Number: 20623 Gerrit-PatchSet: 10 Gerrit-Owner: Michael Smith Gerrit-Reviewer: David Rorke Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Jason Fehr Gerrit-Reviewer: Joe McDonnell Gerrit-Reviewer: Laszlo Gaal Gerrit-Reviewer: Michael Smith Gerrit-Comment-Date: Mon, 06 Nov 2023 19:09:48 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-12491: [DOCS] Add a note on the cache item
Michael Smith has posted comments on this change. ( http://gerrit.cloudera.org:8080/20611 ) Change subject: IMPALA-12491: [DOCS] Add a note on the cache item .. Patch Set 1: Code-Review+2 Makes sense to me. -- To view, visit http://gerrit.cloudera.org:8080/20611 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I508ce667181d635c17373c7336ea9f83984d7641 Gerrit-Change-Number: 20611 Gerrit-PatchSet: 1 Gerrit-Owner: Shajini Thayasingh Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Joe McDonnell Gerrit-Reviewer: Michael Smith Gerrit-Reviewer: Quanlong Huang Gerrit-Comment-Date: Mon, 06 Nov 2023 18:57:09 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-12491: [DOCS] Add a note on the cache item
Michael Smith has submitted this change and it was merged. ( http://gerrit.cloudera.org:8080/20611 ) Change subject: IMPALA-12491: [DOCS] Add a note on the cache item .. IMPALA-12491: [DOCS] Add a note on the cache item Described how the scan request will access the cache when there is no change in the mtime in the file metadata. Change-Id: I508ce667181d635c17373c7336ea9f83984d7641 Reviewed-on: http://gerrit.cloudera.org:8080/20611 Tested-by: Impala Public Jenkins Reviewed-by: Quanlong Huang Reviewed-by: Michael Smith --- M docs/topics/impala_data_cache.xml 1 file changed, 4 insertions(+), 0 deletions(-) Approvals: Impala Public Jenkins: Verified Quanlong Huang: Looks good to me, but someone else must approve Michael Smith: Looks good to me, approved -- To view, visit http://gerrit.cloudera.org:8080/20611 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: merged Gerrit-Change-Id: I508ce667181d635c17373c7336ea9f83984d7641 Gerrit-Change-Number: 20611 Gerrit-PatchSet: 2 Gerrit-Owner: Shajini Thayasingh Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Joe McDonnell Gerrit-Reviewer: Michael Smith Gerrit-Reviewer: Quanlong Huang
[Impala-ASF-CR] IMPALA-12390 (part 4): Enable unnecessary-value-param
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/20494 ) Change subject: IMPALA-12390 (part 4): Enable unnecessary-value-param .. Patch Set 10: Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/9888/ DRY_RUN=true -- To view, visit http://gerrit.cloudera.org:8080/20494 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I8aa5d98596d82f615a0a728e0235e7dd9d8b5003 Gerrit-Change-Number: 20494 Gerrit-PatchSet: 10 Gerrit-Owner: Michael Smith Gerrit-Reviewer: Anonymous Coward Gerrit-Reviewer: Daniel Becker Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Joe McDonnell Gerrit-Reviewer: Michael Smith Gerrit-Comment-Date: Mon, 06 Nov 2023 19:00:42 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-?????: First commit for experimental Calcite planner
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/20664 ) Change subject: IMPALA-?: First commit for experimental Calcite planner .. Patch Set 6: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/14341/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/20664 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I850679424d002c8acccb7421792ad9c498f3c8ab Gerrit-Change-Number: 20664 Gerrit-PatchSet: 6 Gerrit-Owner: Steve Carlin Gerrit-Reviewer: Impala Public Jenkins Gerrit-Comment-Date: Mon, 06 Nov 2023 18:58:17 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-12516: Set HDFS limit based on memlock
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/20623 ) Change subject: IMPALA-12516: Set HDFS limit based on memlock .. Patch Set 10: Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/9887/ DRY_RUN=true -- To view, visit http://gerrit.cloudera.org:8080/20623 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I7722ddd0c7fbd9bbd1979503952b7522b808194a Gerrit-Change-Number: 20623 Gerrit-PatchSet: 10 Gerrit-Owner: Michael Smith Gerrit-Reviewer: David Rorke Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Jason Fehr Gerrit-Reviewer: Joe McDonnell Gerrit-Reviewer: Laszlo Gaal Gerrit-Reviewer: Michael Smith Gerrit-Comment-Date: Mon, 06 Nov 2023 18:52:32 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-?????: First commit for experimental Calcite planner
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/20664 ) Change subject: IMPALA-?: First commit for experimental Calcite planner .. Patch Set 5: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/14340/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/20664 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I850679424d002c8acccb7421792ad9c498f3c8ab Gerrit-Change-Number: 20664 Gerrit-PatchSet: 5 Gerrit-Owner: Steve Carlin Gerrit-Reviewer: Impala Public Jenkins Gerrit-Comment-Date: Mon, 06 Nov 2023 18:50:43 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-?????: First commit for experimental Calcite planner
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/20664 ) Change subject: IMPALA-?: First commit for experimental Calcite planner .. Patch Set 4: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/14339/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/20664 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I850679424d002c8acccb7421792ad9c498f3c8ab Gerrit-Change-Number: 20664 Gerrit-PatchSet: 4 Gerrit-Owner: Steve Carlin Gerrit-Reviewer: Impala Public Jenkins Gerrit-Comment-Date: Mon, 06 Nov 2023 18:42:43 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-?????: First commit for experimental Calcite planner
Hello Impala Public Jenkins, I'd like you to reexamine a change. Please visit http://gerrit.cloudera.org:8080/20664 to look at the new patch set (#6). Change subject: IMPALA-?: First commit for experimental Calcite planner .. IMPALA-?: First commit for experimental Calcite planner This commit is the first commit of hopefully many to come using Calcite to compile and generate the TExecRequest structure used by the backend to run the query. This cut only works for a basic select from query for both qualified and unqualified tables. The entry point is the CalciteJniFrontend module which is a child class of JniFrontend. This is loaded using the config parameter "jni_frontend_class" as can be seen in the test_experimental_planner.py file. The CalciteJniFrontend drives the compilation via several steps, which are: parsing the query, loading metadata tables, validating the query, converting the query from the AST to a logical plan, creating the optimized plan, creating the physical plan (PlanNode), and creating the final distributed plan and TExecRequest. These steps are all called from CalciteJniFrontend. One of the goals here is to be non-intrusive as possible to the current planner. Most changes have been made in the experimental-planner directory, though some minor changes were needed in the main fe module. The ExecRequestCreator does require a refactoring of SingleNodePlanner, but to keep this non-intrusive, some code copying was done. So this does need to be refactored in a later commit and Jira IMPALA-? was created to keep track of this task. Change-Id: I850679424d002c8acccb7421792ad9c498f3c8ab --- M bin/impala-config.sh M fe/pom.xml M fe/src/main/java/org/apache/impala/analysis/TableName.java M fe/src/main/java/org/apache/impala/planner/PlannerContext.java M fe/src/main/java/org/apache/impala/service/JniFrontend.java A java/experimental-planner/pom.xml A java/experimental-planner/src/main/java/org/apache/impala/calcite/node/ConvertToImpalaRelRules.java A java/experimental-planner/src/main/java/org/apache/impala/calcite/node/CreateExprVisitor.java A java/experimental-planner/src/main/java/org/apache/impala/calcite/node/NodeWithExprs.java A java/experimental-planner/src/main/java/org/apache/impala/calcite/node/OptimizedHdfsScanNode.java A java/experimental-planner/src/main/java/org/apache/impala/calcite/node/OptimizedHdfsScanRel.java A java/experimental-planner/src/main/java/org/apache/impala/calcite/node/OptimizedPlanRel.java A java/experimental-planner/src/main/java/org/apache/impala/calcite/node/OptimizedProjectRelBase.java A java/experimental-planner/src/main/java/org/apache/impala/calcite/schema/CalciteDb.java A java/experimental-planner/src/main/java/org/apache/impala/calcite/schema/CalciteTable.java A java/experimental-planner/src/main/java/org/apache/impala/calcite/service/CalciteJniFrontend.java A java/experimental-planner/src/main/java/org/apache/impala/calcite/service/CalciteMetadataHandler.java A java/experimental-planner/src/main/java/org/apache/impala/calcite/service/CalciteOptimizer.java A java/experimental-planner/src/main/java/org/apache/impala/calcite/service/CalcitePhysPlanCreator.java A java/experimental-planner/src/main/java/org/apache/impala/calcite/service/CalciteQueryParser.java A java/experimental-planner/src/main/java/org/apache/impala/calcite/service/CalciteRelNodeConverter.java A java/experimental-planner/src/main/java/org/apache/impala/calcite/service/CalciteValidator.java A java/experimental-planner/src/main/java/org/apache/impala/calcite/service/CompilerStep.java A java/experimental-planner/src/main/java/org/apache/impala/calcite/service/ExecRequestCreator.java A java/experimental-planner/src/main/java/org/apache/impala/calcite/type/ImpalaTypeConverter.java A java/experimental-planner/src/main/java/org/apache/impala/calcite/type/ImpalaTypeSystemImpl.java M java/pom.xml A testdata/workloads/functional-query/queries/QueryTest/calcite.test A tests/custom_cluster/test_experimental_planner.py 29 files changed, 2,426 insertions(+), 3 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/64/20664/6 -- To view, visit http://gerrit.cloudera.org:8080/20664 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: I850679424d002c8acccb7421792ad9c498f3c8ab Gerrit-Change-Number: 20664 Gerrit-PatchSet: 6 Gerrit-Owner: Steve Carlin Gerrit-Reviewer: Impala Public Jenkins
[Impala-ASF-CR] IMPALA-12511: Bump thrift version to fix THRIFT-5670
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/20660 ) Change subject: IMPALA-12511: Bump thrift version to fix THRIFT-5670 .. Patch Set 2: Code-Review+2 -- To view, visit http://gerrit.cloudera.org:8080/20660 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I02f396fd0c27cde03883c06d6376feaf07ca8c13 Gerrit-Change-Number: 20660 Gerrit-PatchSet: 2 Gerrit-Owner: Quanlong Huang Gerrit-Reviewer: Daniel Becker Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Wenzhe Zhou Gerrit-Comment-Date: Mon, 06 Nov 2023 18:27:16 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-12511: Bump thrift version to fix THRIFT-5670
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/20660 ) Change subject: IMPALA-12511: Bump thrift version to fix THRIFT-5670 .. Patch Set 2: Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/9886/ DRY_RUN=false -- To view, visit http://gerrit.cloudera.org:8080/20660 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I02f396fd0c27cde03883c06d6376feaf07ca8c13 Gerrit-Change-Number: 20660 Gerrit-PatchSet: 2 Gerrit-Owner: Quanlong Huang Gerrit-Reviewer: Daniel Becker Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Wenzhe Zhou Gerrit-Comment-Date: Mon, 06 Nov 2023 18:27:17 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-?????: First commit for experimental Calcite planner
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/20664 ) Change subject: IMPALA-?: First commit for experimental Calcite planner .. Patch Set 3: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/14338/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/20664 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I850679424d002c8acccb7421792ad9c498f3c8ab Gerrit-Change-Number: 20664 Gerrit-PatchSet: 3 Gerrit-Owner: Steve Carlin Gerrit-Reviewer: Impala Public Jenkins Gerrit-Comment-Date: Mon, 06 Nov 2023 18:27:02 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-?????: First commit for experimental Calcite planner
Hello Impala Public Jenkins, I'd like you to reexamine a change. Please visit http://gerrit.cloudera.org:8080/20664 to look at the new patch set (#5). Change subject: IMPALA-?: First commit for experimental Calcite planner .. IMPALA-?: First commit for experimental Calcite planner This commit is the first commit of hopefully many to come using Calcite to compile and generate the TExecRequest structure used by the backend to run the query. This cut only works for a basic select from query for both qualified and unqualified tables. The entry point is the CalciteJniFrontend module which is a child class of JniFrontend. This is loaded using the config parameter "jni_frontend_class" as can be seen in the test_experimental_planner.py file. The CalciteJniFrontend drives the compilation via several steps, which are: parsing the query, loading metadata tables, validating the query, converting the query from the AST to a logical plan, creating the optimized plan, creating the physical plan (PlanNode), and creating the final distributed plan and TExecRequest. These steps are all called from CalciteJniFrontend. One of the goals here is to be non-intrusive as possible to the current planner. Most changes have been made in the experimental-planner directory, though some minor changes were needed in the main fe module. The ExecRequestCreator does require a refactoring of SingleNodePlanner, but to keep this non-intrusive, some code copying was done. So this does need to be refactored in a later commit and Jira IMPALA-? was created to keep track of this task. Change-Id: I850679424d002c8acccb7421792ad9c498f3c8ab --- M bin/impala-config.sh M fe/pom.xml M fe/src/main/java/org/apache/impala/analysis/TableName.java M fe/src/main/java/org/apache/impala/planner/PlannerContext.java M fe/src/main/java/org/apache/impala/service/JniFrontend.java A java/experimental-planner/pom.xml A java/experimental-planner/src/main/java/org/apache/impala/calcite/node/ConvertToImpalaRelRules.java A java/experimental-planner/src/main/java/org/apache/impala/calcite/node/CreateExprVisitor.java A java/experimental-planner/src/main/java/org/apache/impala/calcite/node/NodeWithExprs.java A java/experimental-planner/src/main/java/org/apache/impala/calcite/node/OptimizedHdfsScanNode.java A java/experimental-planner/src/main/java/org/apache/impala/calcite/node/OptimizedHdfsScanRel.java A java/experimental-planner/src/main/java/org/apache/impala/calcite/node/OptimizedPlanRel.java A java/experimental-planner/src/main/java/org/apache/impala/calcite/node/OptimizedProjectRelBase.java A java/experimental-planner/src/main/java/org/apache/impala/calcite/schema/CalciteDb.java A java/experimental-planner/src/main/java/org/apache/impala/calcite/schema/CalciteTable.java A java/experimental-planner/src/main/java/org/apache/impala/calcite/service/CalciteJniFrontend.java A java/experimental-planner/src/main/java/org/apache/impala/calcite/service/CalciteMetadataHandler.java A java/experimental-planner/src/main/java/org/apache/impala/calcite/service/CalciteOptimizer.java A java/experimental-planner/src/main/java/org/apache/impala/calcite/service/CalcitePhysPlanCreator.java A java/experimental-planner/src/main/java/org/apache/impala/calcite/service/CalciteQueryParser.java A java/experimental-planner/src/main/java/org/apache/impala/calcite/service/CalciteRelNodeConverter.java A java/experimental-planner/src/main/java/org/apache/impala/calcite/service/CalciteValidator.java A java/experimental-planner/src/main/java/org/apache/impala/calcite/service/CompilerStep.java A java/experimental-planner/src/main/java/org/apache/impala/calcite/service/ExecRequestCreator.java A java/experimental-planner/src/main/java/org/apache/impala/calcite/type/ImpalaTypeConverter.java A java/experimental-planner/src/main/java/org/apache/impala/calcite/type/ImpalaTypeSystemImpl.java M java/pom.xml A testdata/workloads/functional-query/queries/QueryTest/calcite.test A tests/custom_cluster/test_experimental_planner.py 29 files changed, 2,426 insertions(+), 3 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/64/20664/5 -- To view, visit http://gerrit.cloudera.org:8080/20664 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: I850679424d002c8acccb7421792ad9c498f3c8ab Gerrit-Change-Number: 20664 Gerrit-PatchSet: 5 Gerrit-Owner: Steve Carlin Gerrit-Reviewer: Impala Public Jenkins
[Impala-ASF-CR] IMPALA-?????: First commit for experimental Calcite planner
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/20664 ) Change subject: IMPALA-?: First commit for experimental Calcite planner .. Patch Set 5: (1 comment) http://gerrit.cloudera.org:8080/#/c/20664/5/tests/custom_cluster/test_experimental_planner.py File tests/custom_cluster/test_experimental_planner.py: http://gerrit.cloudera.org:8080/#/c/20664/5/tests/custom_cluster/test_experimental_planner.py@41 PS5, Line 41: + flake8: W504 line break after binary operator -- To view, visit http://gerrit.cloudera.org:8080/20664 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I850679424d002c8acccb7421792ad9c498f3c8ab Gerrit-Change-Number: 20664 Gerrit-PatchSet: 5 Gerrit-Owner: Steve Carlin Gerrit-Reviewer: Impala Public Jenkins Gerrit-Comment-Date: Mon, 06 Nov 2023 18:23:03 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-?????: First commit for experimental Calcite planner
Hello Impala Public Jenkins, I'd like you to reexamine a change. Please visit http://gerrit.cloudera.org:8080/20664 to look at the new patch set (#4). Change subject: IMPALA-?: First commit for experimental Calcite planner .. IMPALA-?: First commit for experimental Calcite planner This commit is the first commit of hopefully many to come using Calcite to compile and generate the TExecRequest structure used by the backend to run the query. This cut only works for a basic select from query for both qualified and unqualified tables. The entry point is the CalciteJniFrontend module which is a child class of JniFrontend. This is loaded using the config parameter "jni_frontend_class" as can be seen in the test_experimental_planner.py file. The CalciteJniFrontend drives the compilation via several steps, which are: parsing the query, loading metadata tables, validating the query, converting the query from the AST to a logical plan, creating the optimized plan, creating the physical plan (PlanNode), and creating the final distributed plan and TExecRequest. These steps are all called from CalciteJniFrontend. One of the goals here is to be non-intrusive as possible to the current planner. Most changes have been made in the experimental-planner directory, though some minor changes were needed in the main fe module. The ExecRequestCreator does require a refactoring of SingleNodePlanner, but to keep this non-intrusive, some code copying was done. So this does need to be refactored in a later commit and Jira IMPALA-? was created to keep track of this task. Change-Id: I850679424d002c8acccb7421792ad9c498f3c8ab --- M bin/impala-config.sh M fe/pom.xml M fe/src/main/java/org/apache/impala/analysis/TableName.java M fe/src/main/java/org/apache/impala/planner/PlannerContext.java M fe/src/main/java/org/apache/impala/service/JniFrontend.java A java/experimental-planner/pom.xml A java/experimental-planner/src/main/java/org/apache/impala/calcite/node/ConvertToImpalaRelRules.java A java/experimental-planner/src/main/java/org/apache/impala/calcite/node/CreateExprVisitor.java A java/experimental-planner/src/main/java/org/apache/impala/calcite/node/NodeWithExprs.java A java/experimental-planner/src/main/java/org/apache/impala/calcite/node/OptimizedHdfsScanNode.java A java/experimental-planner/src/main/java/org/apache/impala/calcite/node/OptimizedHdfsScanRel.java A java/experimental-planner/src/main/java/org/apache/impala/calcite/node/OptimizedPlanRel.java A java/experimental-planner/src/main/java/org/apache/impala/calcite/node/OptimizedProjectRelBase.java A java/experimental-planner/src/main/java/org/apache/impala/calcite/schema/CalciteDb.java A java/experimental-planner/src/main/java/org/apache/impala/calcite/schema/CalciteTable.java A java/experimental-planner/src/main/java/org/apache/impala/calcite/service/CalciteJniFrontend.java A java/experimental-planner/src/main/java/org/apache/impala/calcite/service/CalciteMetadataHandler.java A java/experimental-planner/src/main/java/org/apache/impala/calcite/service/CalciteOptimizer.java A java/experimental-planner/src/main/java/org/apache/impala/calcite/service/CalcitePhysPlanCreator.java A java/experimental-planner/src/main/java/org/apache/impala/calcite/service/CalciteQueryParser.java A java/experimental-planner/src/main/java/org/apache/impala/calcite/service/CalciteRelNodeConverter.java A java/experimental-planner/src/main/java/org/apache/impala/calcite/service/CalciteValidator.java A java/experimental-planner/src/main/java/org/apache/impala/calcite/service/CompilerStep.java A java/experimental-planner/src/main/java/org/apache/impala/calcite/service/ExecRequestCreator.java A java/experimental-planner/src/main/java/org/apache/impala/calcite/type/ImpalaTypeConverter.java A java/experimental-planner/src/main/java/org/apache/impala/calcite/type/ImpalaTypeSystemImpl.java M java/pom.xml A testdata/workloads/functional-query/queries/QueryTest/calcite.test A tests/custom_cluster/test_experimental_planner.py 29 files changed, 2,427 insertions(+), 3 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/64/20664/4 -- To view, visit http://gerrit.cloudera.org:8080/20664 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: I850679424d002c8acccb7421792ad9c498f3c8ab Gerrit-Change-Number: 20664 Gerrit-PatchSet: 4 Gerrit-Owner: Steve Carlin Gerrit-Reviewer: Impala Public Jenkins
[Impala-ASF-CR] IMPALA-?????: First commit for experimental Calcite planner
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/20664 ) Change subject: IMPALA-?: First commit for experimental Calcite planner .. Patch Set 4: (2 comments) http://gerrit.cloudera.org:8080/#/c/20664/4/tests/custom_cluster/test_experimental_planner.py File tests/custom_cluster/test_experimental_planner.py: http://gerrit.cloudera.org:8080/#/c/20664/4/tests/custom_cluster/test_experimental_planner.py@42 PS4, Line 42: flake8: E251 unexpected spaces around keyword / parameter equals http://gerrit.cloudera.org:8080/#/c/20664/4/tests/custom_cluster/test_experimental_planner.py@43 PS4, Line 43: " flake8: E131 continuation line unaligned for hanging indent -- To view, visit http://gerrit.cloudera.org:8080/20664 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I850679424d002c8acccb7421792ad9c498f3c8ab Gerrit-Change-Number: 20664 Gerrit-PatchSet: 4 Gerrit-Owner: Steve Carlin Gerrit-Reviewer: Impala Public Jenkins Gerrit-Comment-Date: Mon, 06 Nov 2023 18:15:30 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-?????: First commit for experimental Calcite planner
Hello Impala Public Jenkins, I'd like you to reexamine a change. Please visit http://gerrit.cloudera.org:8080/20664 to look at the new patch set (#3). Change subject: IMPALA-?: First commit for experimental Calcite planner .. IMPALA-?: First commit for experimental Calcite planner This commit is the first commit of hopefully many to come using Calcite to compile and generate the TExecRequest structure used by the backend to run the query. This cut only works for a basic select from query for both qualified and unqualified tables. The entry point is the CalciteJniFrontend module which is a child class of JniFrontend. This is loaded using the config parameter "jni_frontend_class" as can be seen in the test_experimental_planner.py file. The CalciteJniFrontend drives the compilation via several steps, which are: parsing the query, loading metadata tables, validating the query, converting the query from the AST to a logical plan, creating the optimized plan, creating the physical plan (PlanNode), and creating the final distributed plan and TExecRequest. These steps are all called from CalciteJniFrontend. One of the goals here is to be non-intrusive as possible to the current planner. Most changes have been made in the experimental-planner directory, though some minor changes were needed in the main fe module. The ExecRequestCreator does require a refactoring of SingleNodePlanner, but to keep this non-intrusive, some code copying was done. So this does need to be refactored in a later commit and Jira IMPALA-? was created to keep track of this task. Change-Id: I850679424d002c8acccb7421792ad9c498f3c8ab --- M bin/impala-config.sh M fe/pom.xml M fe/src/main/java/org/apache/impala/analysis/TableName.java M fe/src/main/java/org/apache/impala/planner/PlannerContext.java M fe/src/main/java/org/apache/impala/service/JniFrontend.java A java/experimental-planner/pom.xml A java/experimental-planner/src/main/java/org/apache/impala/calcite/node/ConvertToImpalaRelRules.java A java/experimental-planner/src/main/java/org/apache/impala/calcite/node/CreateExprVisitor.java A java/experimental-planner/src/main/java/org/apache/impala/calcite/node/NodeWithExprs.java A java/experimental-planner/src/main/java/org/apache/impala/calcite/node/OptimizedHdfsScanNode.java A java/experimental-planner/src/main/java/org/apache/impala/calcite/node/OptimizedHdfsScanRel.java A java/experimental-planner/src/main/java/org/apache/impala/calcite/node/OptimizedPlanRel.java A java/experimental-planner/src/main/java/org/apache/impala/calcite/node/OptimizedProjectRelBase.java A java/experimental-planner/src/main/java/org/apache/impala/calcite/schema/CalciteDb.java A java/experimental-planner/src/main/java/org/apache/impala/calcite/schema/CalciteTable.java A java/experimental-planner/src/main/java/org/apache/impala/calcite/service/CalciteJniFrontend.java A java/experimental-planner/src/main/java/org/apache/impala/calcite/service/CalciteMetadataHandler.java A java/experimental-planner/src/main/java/org/apache/impala/calcite/service/CalciteOptimizer.java A java/experimental-planner/src/main/java/org/apache/impala/calcite/service/CalcitePhysPlanCreator.java A java/experimental-planner/src/main/java/org/apache/impala/calcite/service/CalciteQueryParser.java A java/experimental-planner/src/main/java/org/apache/impala/calcite/service/CalciteRelNodeConverter.java A java/experimental-planner/src/main/java/org/apache/impala/calcite/service/CalciteValidator.java A java/experimental-planner/src/main/java/org/apache/impala/calcite/service/CompilerStep.java A java/experimental-planner/src/main/java/org/apache/impala/calcite/service/ExecRequestCreator.java A java/experimental-planner/src/main/java/org/apache/impala/calcite/type/ImpalaTypeConverter.java A java/experimental-planner/src/main/java/org/apache/impala/calcite/type/ImpalaTypeSystemImpl.java M java/pom.xml A testdata/workloads/functional-query/queries/QueryTest/calcite.test A tests/custom_cluster/test_experimental_planner.py 29 files changed, 2,427 insertions(+), 3 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/64/20664/3 -- To view, visit http://gerrit.cloudera.org:8080/20664 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: I850679424d002c8acccb7421792ad9c498f3c8ab Gerrit-Change-Number: 20664 Gerrit-PatchSet: 3 Gerrit-Owner: Steve Carlin Gerrit-Reviewer: Impala Public Jenkins
[Impala-ASF-CR] IMPALA-?????: First commit for experimental Calcite planner
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/20664 ) Change subject: IMPALA-?: First commit for experimental Calcite planner .. Patch Set 3: (3 comments) http://gerrit.cloudera.org:8080/#/c/20664/3/tests/custom_cluster/test_experimental_planner.py File tests/custom_cluster/test_experimental_planner.py: http://gerrit.cloudera.org:8080/#/c/20664/3/tests/custom_cluster/test_experimental_planner.py@42 PS3, Line 42: flake8: E251 unexpected spaces around keyword / parameter equals http://gerrit.cloudera.org:8080/#/c/20664/3/tests/custom_cluster/test_experimental_planner.py@42 PS3, Line 42: \ flake8: E502 the backslash is redundant between brackets http://gerrit.cloudera.org:8080/#/c/20664/3/tests/custom_cluster/test_experimental_planner.py@43 PS3, Line 43: " flake8: E131 continuation line unaligned for hanging indent -- To view, visit http://gerrit.cloudera.org:8080/20664 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I850679424d002c8acccb7421792ad9c498f3c8ab Gerrit-Change-Number: 20664 Gerrit-PatchSet: 3 Gerrit-Owner: Steve Carlin Gerrit-Reviewer: Impala Public Jenkins Gerrit-Comment-Date: Mon, 06 Nov 2023 17:59:22 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-?????: First commit for experimental Calcite planner
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/20664 ) Change subject: IMPALA-?: First commit for experimental Calcite planner .. Patch Set 2: Build Failed https://jenkins.impala.io/job/gerrit-code-review-checks/14337/ : Initial code review checks failed. See linked job for details on the failure. -- To view, visit http://gerrit.cloudera.org:8080/20664 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I850679424d002c8acccb7421792ad9c498f3c8ab Gerrit-Change-Number: 20664 Gerrit-PatchSet: 2 Gerrit-Owner: Steve Carlin Gerrit-Reviewer: Impala Public Jenkins Gerrit-Comment-Date: Mon, 06 Nov 2023 17:56:10 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-?????: First commit for experimental Calcite planner
Hello Impala Public Jenkins, I'd like you to reexamine a change. Please visit http://gerrit.cloudera.org:8080/20664 to look at the new patch set (#2). Change subject: IMPALA-?: First commit for experimental Calcite planner .. IMPALA-?: First commit for experimental Calcite planner This commit is the first commit of hopefully many to come using Calcite to compile and generate the TExecRequest structure used by the backend to run the query. This cut only works for a basic select from query for both qualified and unqualified tables. The entry point is the CalciteJniFrontend module which is a child class of JniFrontend. This is loaded using the config parameter "jni_frontend_class" as can be seen in the test_experimental_planner.py file. The CalciteJniFrontend drives the compilation via several steps, which are: parsing the query, loading metadata tables, validating the query, converting the query from the AST to a logical plan, creating the optimized plan, creating the physical plan (PlanNode), and creating the final distributed plan and TExecRequest. These steps are all called from CalciteJniFrontend. One of the goals here is to be non-intrusive as possible to the current planner. Most changes have been made in the experimental-planner directory, though some minor changes were needed in the main fe module. The ExecRequestCreator does require a refactoring of SingleNodePlanner, but to keep this non-intrusive, some code copying was done. So this does need to be refactored in a later commit and Jira IMPALA-? was created to keep track of this task. Change-Id: I850679424d002c8acccb7421792ad9c498f3c8ab --- M bin/impala-config.sh M fe/pom.xml M fe/src/main/java/org/apache/impala/analysis/TableName.java M fe/src/main/java/org/apache/impala/planner/PlannerContext.java M fe/src/main/java/org/apache/impala/service/JniFrontend.java A java/experimental-planner/pom.xml A java/experimental-planner/src/main/java/org/apache/impala/calcite/node/ConvertToImpalaRelRules.java A java/experimental-planner/src/main/java/org/apache/impala/calcite/node/CreateExprVisitor.java A java/experimental-planner/src/main/java/org/apache/impala/calcite/node/NodeWithExprs.java A java/experimental-planner/src/main/java/org/apache/impala/calcite/node/OptimizedHdfsScanNode.java A java/experimental-planner/src/main/java/org/apache/impala/calcite/node/OptimizedHdfsScanRel.java A java/experimental-planner/src/main/java/org/apache/impala/calcite/node/OptimizedPlanRel.java A java/experimental-planner/src/main/java/org/apache/impala/calcite/node/OptimizedProjectRelBase.java A java/experimental-planner/src/main/java/org/apache/impala/calcite/schema/CalciteDb.java A java/experimental-planner/src/main/java/org/apache/impala/calcite/schema/CalciteTable.java A java/experimental-planner/src/main/java/org/apache/impala/calcite/service/CalciteJniFrontend.java A java/experimental-planner/src/main/java/org/apache/impala/calcite/service/CalciteMetadataHandler.java A java/experimental-planner/src/main/java/org/apache/impala/calcite/service/CalciteOptimizer.java A java/experimental-planner/src/main/java/org/apache/impala/calcite/service/CalcitePhysPlanCreator.java A java/experimental-planner/src/main/java/org/apache/impala/calcite/service/CalciteQueryParser.java A java/experimental-planner/src/main/java/org/apache/impala/calcite/service/CalciteRelNodeConverter.java A java/experimental-planner/src/main/java/org/apache/impala/calcite/service/CalciteValidator.java A java/experimental-planner/src/main/java/org/apache/impala/calcite/service/CompilerStep.java A java/experimental-planner/src/main/java/org/apache/impala/calcite/service/ExecRequestCreator.java A java/experimental-planner/src/main/java/org/apache/impala/calcite/type/ImpalaTypeConverter.java A java/experimental-planner/src/main/java/org/apache/impala/calcite/type/ImpalaTypeSystemImpl.java M java/pom.xml A testdata/workloads/functional-query/queries/QueryTest/calcite.test A tests/custom_cluster/test_experimental_planner.py 29 files changed, 2,426 insertions(+), 3 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/64/20664/2 -- To view, visit http://gerrit.cloudera.org:8080/20664 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: I850679424d002c8acccb7421792ad9c498f3c8ab Gerrit-Change-Number: 20664 Gerrit-PatchSet: 2 Gerrit-Owner: Steve Carlin Gerrit-Reviewer: Impala Public Jenkins
[Impala-ASF-CR] IMPALA-?????: First commit for experimental Calcite planner
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/20664 ) Change subject: IMPALA-?: First commit for experimental Calcite planner .. Patch Set 2: (1 comment) http://gerrit.cloudera.org:8080/#/c/20664/2/tests/custom_cluster/test_experimental_planner.py File tests/custom_cluster/test_experimental_planner.py: http://gerrit.cloudera.org:8080/#/c/20664/2/tests/custom_cluster/test_experimental_planner.py@42 PS2, Line 42: " flake8: E501 line too long (92 > 90 characters) -- To view, visit http://gerrit.cloudera.org:8080/20664 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I850679424d002c8acccb7421792ad9c498f3c8ab Gerrit-Change-Number: 20664 Gerrit-PatchSet: 2 Gerrit-Owner: Steve Carlin Gerrit-Reviewer: Impala Public Jenkins Gerrit-Comment-Date: Mon, 06 Nov 2023 17:37:49 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-12545: Use 'unique database' in test restart services.py::TestRestart
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/20662 ) Change subject: IMPALA-12545: Use 'unique_database' in test_restart_services.py::TestRestart .. Patch Set 2: Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/9885/ DRY_RUN=false -- To view, visit http://gerrit.cloudera.org:8080/20662 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Id4c2cfb669d5ff9e4d8110a0035bae1147e2db5a Gerrit-Change-Number: 20662 Gerrit-PatchSet: 2 Gerrit-Owner: Daniel Becker Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Noemi Pap-Takacs Gerrit-Reviewer: Riza Suminto Gerrit-Comment-Date: Mon, 06 Nov 2023 17:15:31 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-12545: Use 'unique database' in test restart services.py::TestRestart
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/20662 ) Change subject: IMPALA-12545: Use 'unique_database' in test_restart_services.py::TestRestart .. Patch Set 2: Code-Review+2 -- To view, visit http://gerrit.cloudera.org:8080/20662 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Id4c2cfb669d5ff9e4d8110a0035bae1147e2db5a Gerrit-Change-Number: 20662 Gerrit-PatchSet: 2 Gerrit-Owner: Daniel Becker Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Noemi Pap-Takacs Gerrit-Reviewer: Riza Suminto Gerrit-Comment-Date: Mon, 06 Nov 2023 17:15:31 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-?????: First commit for experimental Calcite planner
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/20664 ) Change subject: IMPALA-?: First commit for experimental Calcite planner .. Patch Set 1: Build Failed https://jenkins.impala.io/job/gerrit-code-review-checks/14336/ : Initial code review checks failed. See linked job for details on the failure. -- To view, visit http://gerrit.cloudera.org:8080/20664 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I850679424d002c8acccb7421792ad9c498f3c8ab Gerrit-Change-Number: 20664 Gerrit-PatchSet: 1 Gerrit-Owner: Steve Carlin Gerrit-Reviewer: Impala Public Jenkins Gerrit-Comment-Date: Mon, 06 Nov 2023 16:47:54 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-12545: Use 'unique database' in test restart services.py::TestRestart
Riza Suminto has posted comments on this change. ( http://gerrit.cloudera.org:8080/20662 ) Change subject: IMPALA-12545: Use 'unique_database' in test_restart_services.py::TestRestart .. Patch Set 1: Code-Review+2 Looks good to me. Thanks for taking care of this! -- To view, visit http://gerrit.cloudera.org:8080/20662 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Id4c2cfb669d5ff9e4d8110a0035bae1147e2db5a Gerrit-Change-Number: 20662 Gerrit-PatchSet: 1 Gerrit-Owner: Daniel Becker Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Noemi Pap-Takacs Gerrit-Reviewer: Riza Suminto Gerrit-Comment-Date: Mon, 06 Nov 2023 16:41:42 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-12511: Bump thrift version to fix THRIFT-5670
Wenzhe Zhou has posted comments on this change. ( http://gerrit.cloudera.org:8080/20660 ) Change subject: IMPALA-12511: Bump thrift version to fix THRIFT-5670 .. Patch Set 1: Code-Review+2 -- To view, visit http://gerrit.cloudera.org:8080/20660 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I02f396fd0c27cde03883c06d6376feaf07ca8c13 Gerrit-Change-Number: 20660 Gerrit-PatchSet: 1 Gerrit-Owner: Quanlong Huang Gerrit-Reviewer: Daniel Becker Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Wenzhe Zhou Gerrit-Comment-Date: Mon, 06 Nov 2023 16:40:11 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-?????: First commit for experimental Calcite planner
Steve Carlin has uploaded this change for review. ( http://gerrit.cloudera.org:8080/20664 Change subject: IMPALA-?: First commit for experimental Calcite planner .. IMPALA-?: First commit for experimental Calcite planner This commit is the first commit of hopefully many to come using Calcite to compile and generate the TExecRequest structure used by the backend to run the query. This cut only works for a basic select from query for both qualified and unqualified tables. The entry point is the CalciteJniFrontend module which is a child class of JniFrontend. This is loaded using the config parameter "jni_frontend_class" as can be seen in the test_experimental_planner.py file. The CalciteJniFrontend drives the compilation via several steps, which are: parsing the query, loading metadata tables, validating the query, converting the query from the AST to a logical plan, creating the optimized plan, creating the physical plan (PlanNode), and creating the final distributed plan and TExecRequest. These steps are all called from CalciteJniFrontend. One of the goals here is to be non-intrusive as possible to the current planner. Most changes have been made in the experimental-planner directory, though some minor changes were needed in the main fe module. The ExecRequestCreator does require a refactoring of SingleNodePlanner, but to keep this non-intrusive, some code copying was done. So this does need to be refactored in a later commit and Jira IMPALA-? was created to keep track of this task. Change-Id: I850679424d002c8acccb7421792ad9c498f3c8ab --- M bin/impala-config.sh M fe/pom.xml M fe/src/main/java/org/apache/impala/analysis/TableName.java M fe/src/main/java/org/apache/impala/planner/PlannerContext.java M fe/src/main/java/org/apache/impala/service/JniFrontend.java A java/experimental-planner/pom.xml A java/experimental-planner/src/main/java/org/apache/impala/calcite/node/ConvertToImpalaRelRules.java A java/experimental-planner/src/main/java/org/apache/impala/calcite/node/CreateExprVisitor.java A java/experimental-planner/src/main/java/org/apache/impala/calcite/node/NodeWithExprs.java A java/experimental-planner/src/main/java/org/apache/impala/calcite/node/OptimizedHdfsScanNode.java A java/experimental-planner/src/main/java/org/apache/impala/calcite/node/OptimizedHdfsScanRel.java A java/experimental-planner/src/main/java/org/apache/impala/calcite/node/OptimizedPlanRel.java A java/experimental-planner/src/main/java/org/apache/impala/calcite/node/OptimizedProjectRelBase.java A java/experimental-planner/src/main/java/org/apache/impala/calcite/schema/CalciteDb.java A java/experimental-planner/src/main/java/org/apache/impala/calcite/schema/CalciteTable.java A java/experimental-planner/src/main/java/org/apache/impala/calcite/service/CalciteJniFrontend.java A java/experimental-planner/src/main/java/org/apache/impala/calcite/service/CalciteMetadataHandler.java A java/experimental-planner/src/main/java/org/apache/impala/calcite/service/CalciteOptimizer.java A java/experimental-planner/src/main/java/org/apache/impala/calcite/service/CalcitePhysPlanCreator.java A java/experimental-planner/src/main/java/org/apache/impala/calcite/service/CalciteQueryParser.java A java/experimental-planner/src/main/java/org/apache/impala/calcite/service/CalciteRelNodeConverter.java A java/experimental-planner/src/main/java/org/apache/impala/calcite/service/CalciteValidator.java A java/experimental-planner/src/main/java/org/apache/impala/calcite/service/CompilerStep.java A java/experimental-planner/src/main/java/org/apache/impala/calcite/service/ExecRequestCreator.java A java/experimental-planner/src/main/java/org/apache/impala/calcite/type/ImpalaTypeConverter.java A java/experimental-planner/src/main/java/org/apache/impala/calcite/type/ImpalaTypeSystemImpl.java M java/pom.xml A testdata/workloads/functional-query/queries/QueryTest/calcite.test A tests/custom_cluster/test_experimental_planner.py 29 files changed, 2,415 insertions(+), 3 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/64/20664/1 -- To view, visit http://gerrit.cloudera.org:8080/20664 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newchange Gerrit-Change-Id: I850679424d002c8acccb7421792ad9c498f3c8ab Gerrit-Change-Number: 20664 Gerrit-PatchSet: 1 Gerrit-Owner: Steve Carlin
[Impala-ASF-CR] IMPALA-?????: First commit for experimental Calcite planner
Impala Public Jenkins has posted comments on this change. (
http://gerrit.cloudera.org:8080/20664 )
Change subject: IMPALA-?: First commit for experimental Calcite planner
..
Patch Set 1:
(44 comments)
http://gerrit.cloudera.org:8080/#/c/20664/1/java/experimental-planner/src/main/java/org/apache/impala/calcite/node/OptimizedHdfsScanRel.java
File
java/experimental-planner/src/main/java/org/apache/impala/calcite/node/OptimizedHdfsScanRel.java:
http://gerrit.cloudera.org:8080/#/c/20664/1/java/experimental-planner/src/main/java/org/apache/impala/calcite/node/OptimizedHdfsScanRel.java@58
PS1, Line 58: protected static final Logger LOG =
LoggerFactory.getLogger(OptimizedHdfsScanRel.class.getName());
line too long (100 > 90)
http://gerrit.cloudera.org:8080/#/c/20664/1/java/experimental-planner/src/main/java/org/apache/impala/calcite/node/OptimizedHdfsScanRel.java@127
PS1, Line 127: SlotRef slotref = new
SlotRef(Path.createRawPath(baseTblRef.getUniqueAlias(), fieldName));
line too long (96 > 90)
http://gerrit.cloudera.org:8080/#/c/20664/1/java/experimental-planner/src/main/java/org/apache/impala/calcite/node/OptimizedHdfsScanRel.java@132
PS1, Line 132: + "is a complex type (array/map/struct) column. This
is not currently supported."));
line too long (96 > 90)
http://gerrit.cloudera.org:8080/#/c/20664/1/java/experimental-planner/src/main/java/org/apache/impala/calcite/node/OptimizedProjectRelBase.java
File
java/experimental-planner/src/main/java/org/apache/impala/calcite/node/OptimizedProjectRelBase.java:
http://gerrit.cloudera.org:8080/#/c/20664/1/java/experimental-planner/src/main/java/org/apache/impala/calcite/node/OptimizedProjectRelBase.java@47
PS1, Line 47: protected static final Logger LOG =
LoggerFactory.getLogger(OptimizedProjectRelBase.class);
line too long (93 > 90)
http://gerrit.cloudera.org:8080/#/c/20664/1/java/experimental-planner/src/main/java/org/apache/impala/calcite/node/OptimizedProjectRelBase.java@52
PS1, Line 52: super(project.getCluster(), project.getTraitSet(),
project.getInputs(), project.getRowType());
line too long (98 > 90)
http://gerrit.cloudera.org:8080/#/c/20664/1/java/experimental-planner/src/main/java/org/apache/impala/calcite/schema/CalciteTable.java
File
java/experimental-planner/src/main/java/org/apache/impala/calcite/schema/CalciteTable.java:
http://gerrit.cloudera.org:8080/#/c/20664/1/java/experimental-planner/src/main/java/org/apache/impala/calcite/schema/CalciteTable.java@34
PS1, Line 34: protected static final Logger LOG =
LoggerFactory.getLogger(CalciteTable.class.getName());
line too long (92 > 90)
http://gerrit.cloudera.org:8080/#/c/20664/1/java/experimental-planner/src/main/java/org/apache/impala/calcite/service/CalciteJniFrontend.java
File
java/experimental-planner/src/main/java/org/apache/impala/calcite/service/CalciteJniFrontend.java:
http://gerrit.cloudera.org:8080/#/c/20664/1/java/experimental-planner/src/main/java/org/apache/impala/calcite/service/CalciteJniFrontend.java@47
PS1, Line 47: * to walk through all the steps of compiling the query (e.g.
parsing, validating, etc... )
line too long (91 > 90)
http://gerrit.cloudera.org:8080/#/c/20664/1/java/experimental-planner/src/main/java/org/apache/impala/calcite/service/CalciteJniFrontend.java@52
PS1, Line 52: protected static final Logger LOG =
LoggerFactory.getLogger(CalciteJniFrontend.class.getName());
line too long (98 > 90)
http://gerrit.cloudera.org:8080/#/c/20664/1/java/experimental-planner/src/main/java/org/apache/impala/calcite/service/CalciteJniFrontend.java@124
PS1, Line 124: // Overly simple check. This needs to be adjusted since not
all queries begin with 'select',
line too long (97 > 90)
http://gerrit.cloudera.org:8080/#/c/20664/1/java/experimental-planner/src/main/java/org/apache/impala/calcite/service/CalciteJniFrontend.java@129
PS1, Line 129: public byte[] runThroughOriginalPlanner(byte[]
thriftQueryContext) throws ImpalaException {
line too long (93 > 90)
http://gerrit.cloudera.org:8080/#/c/20664/1/java/experimental-planner/src/main/java/org/apache/impala/calcite/service/CalciteMetadataHandler.java
File
java/experimental-planner/src/main/java/org/apache/impala/calcite/service/CalciteMetadataHandler.java:
http://gerrit.cloudera.org:8080/#/c/20664/1/java/experimental-planner/src/main/java/org/apache/impala/calcite/service/CalciteMetadataHandler.java@82
PS1, Line 82: private CalciteSchema createCalciteSchema(FeCatalog catalog,
Set tableNames) {
line too long (91 > 90)
http://gerrit.cloudera.org:8080/#/c/20664/1/java/experimental-planner/src/main/java/org/apache/impala/calcite/service/CalcitePhysPlanCreator.java
File
java/experimental-planner/src/main/java/org/apache/impala/calcite/service/CalcitePhysPlanCreator.java:
[Impala-ASF-CR] IMPALA-12470 (PART-2): delete temporary file in /tmp after class loaded
Wenzhe Zhou has submitted this change and it was merged. ( http://gerrit.cloudera.org:8080/20654 ) Change subject: IMPALA-12470 (PART-2): delete temporary file in /tmp after class loaded .. IMPALA-12470 (PART-2): delete temporary file in /tmp after class loaded This patch fixes the bug added in the previous patch for IMPALA-12470. It adds the prefix "file://" to the unix standard path string to create the corresponding valid hadoop.fs.Path object. For example: "/tmp" is converted to "file:///tmp". Testing: 1. Deleted all the jar files in the /tmp directory. 2. Ran the local jdbc ext data sources tests: - impala-py.test tests/query_test/test_ext_data_sources.py - impala-py.test tests/custom_cluster/test_ext_data_sources.py 3. Upon completion of the tests successfully, Verified that there were no .jar files in the /tmp directory. Change-Id: Iab7cc66383bc62f209987dd3fb42fc3fc6604726 Reviewed-on: http://gerrit.cloudera.org:8080/20654 Reviewed-by: Wenzhe Zhou Tested-by: Wenzhe Zhou --- M fe/src/main/java/org/apache/impala/planner/DataSourceScanNode.java M java/ext-data-source/jdbc/src/main/java/org/apache/impala/extdatasource/jdbc/dao/GenericJdbcDatabaseAccessor.java 2 files changed, 2 insertions(+), 2 deletions(-) Approvals: Wenzhe Zhou: Looks good to me, approved; Verified -- To view, visit http://gerrit.cloudera.org:8080/20654 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: merged Gerrit-Change-Id: Iab7cc66383bc62f209987dd3fb42fc3fc6604726 Gerrit-Change-Number: 20654 Gerrit-PatchSet: 4 Gerrit-Owner: Anonymous Coward Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Wenzhe Zhou
[Impala-ASF-CR] IMPALA-12535: Fix misleading metric keys for the threadz page
Daniel Becker has posted comments on this change. ( http://gerrit.cloudera.org:8080/20658 ) Change subject: IMPALA-12535: Fix misleading metric keys for the threadz page .. Patch Set 1: Code-Review+1 Do we test these keys somewhere? -- To view, visit http://gerrit.cloudera.org:8080/20658 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I15a8cf0a318bc7122d1f5df29f18d8e467249ef7 Gerrit-Change-Number: 20658 Gerrit-PatchSet: 1 Gerrit-Owner: Quanlong Huang Gerrit-Reviewer: Daniel Becker Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Yida Wu Gerrit-Comment-Date: Mon, 06 Nov 2023 16:11:33 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-12545: Use 'unique database' in test restart services.py::TestRestart
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/20662 ) Change subject: IMPALA-12545: Use 'unique_database' in test_restart_services.py::TestRestart .. Patch Set 1: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/14335/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/20662 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Id4c2cfb669d5ff9e4d8110a0035bae1147e2db5a Gerrit-Change-Number: 20662 Gerrit-PatchSet: 1 Gerrit-Owner: Daniel Becker Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Noemi Pap-Takacs Gerrit-Reviewer: Riza Suminto Gerrit-Comment-Date: Mon, 06 Nov 2023 16:06:42 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-12511: Bump thrift version to fix THRIFT-5670
Daniel Becker has posted comments on this change. ( http://gerrit.cloudera.org:8080/20660 ) Change subject: IMPALA-12511: Bump thrift version to fix THRIFT-5670 .. Patch Set 1: Code-Review+1 -- To view, visit http://gerrit.cloudera.org:8080/20660 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I02f396fd0c27cde03883c06d6376feaf07ca8c13 Gerrit-Change-Number: 20660 Gerrit-PatchSet: 1 Gerrit-Owner: Quanlong Huang Gerrit-Reviewer: Daniel Becker Gerrit-Reviewer: Impala Public Jenkins Gerrit-Comment-Date: Mon, 06 Nov 2023 15:54:48 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-12545: Use 'unique database' in test restart services.py::TestRestart
Daniel Becker has uploaded this change for review. ( http://gerrit.cloudera.org:8080/20662 Change subject: IMPALA-12545: Use 'unique_database' in test_restart_services.py::TestRestart .. IMPALA-12545: Use 'unique_database' in test_restart_services.py::TestRestart Some tests in 'test_restart_services.py::TestRestart' create tables but don't use a 'unique_database' for it. The table and column names are sometimes the same. The tests are run serially but if a table is not deleted successfully it may interfere with other tests. This change introduces 'unique_database' in tests that create tables. After this, explicitly deleting the tables is no longer necessary as the framework takes care of it. Testing: - all affected tests pass Change-Id: Id4c2cfb669d5ff9e4d8110a0035bae1147e2db5a --- M tests/custom_cluster/test_restart_services.py 1 file changed, 46 insertions(+), 45 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/62/20662/1 -- To view, visit http://gerrit.cloudera.org:8080/20662 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newchange Gerrit-Change-Id: Id4c2cfb669d5ff9e4d8110a0035bae1147e2db5a Gerrit-Change-Number: 20662 Gerrit-PatchSet: 1 Gerrit-Owner: Daniel Becker Gerrit-Reviewer: Noemi Pap-Takacs Gerrit-Reviewer: Riza Suminto
[Impala-ASF-CR] IMPALA-12470 (PART-2): delete temporary file in /tmp after class loaded
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/20654 ) Change subject: IMPALA-12470 (PART-2): delete temporary file in /tmp after class loaded .. Patch Set 3: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/14334/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/20654 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Iab7cc66383bc62f209987dd3fb42fc3fc6604726 Gerrit-Change-Number: 20654 Gerrit-PatchSet: 3 Gerrit-Owner: Anonymous Coward Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Wenzhe Zhou Gerrit-Comment-Date: Mon, 06 Nov 2023 15:07:50 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-12470 (PART-2): delete temporary file in /tmp after class loaded
gsi...@cloudera.com has uploaded a new patch set (#3). ( http://gerrit.cloudera.org:8080/20654 ) Change subject: IMPALA-12470 (PART-2): delete temporary file in /tmp after class loaded .. IMPALA-12470 (PART-2): delete temporary file in /tmp after class loaded This patch fixes the bug added in the previous patch for IMPALA-12470. It adds the prefix "file://" to the unix standard path string to create the corresponding valid hadoop.fs.Path object. For example: "/tmp" is converted to "file:///tmp". Testing: 1. Deleted all the jar files in the /tmp directory. 2. Ran the local jdbc ext data sources tests: - impala-py.test tests/query_test/test_ext_data_sources.py - impala-py.test tests/custom_cluster/test_ext_data_sources.py 3. Upon completion of the tests successfully, Verified that there were no .jar files in the /tmp directory. Change-Id: Iab7cc66383bc62f209987dd3fb42fc3fc6604726 --- M fe/src/main/java/org/apache/impala/planner/DataSourceScanNode.java M java/ext-data-source/jdbc/src/main/java/org/apache/impala/extdatasource/jdbc/dao/GenericJdbcDatabaseAccessor.java 2 files changed, 2 insertions(+), 2 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/54/20654/3 -- To view, visit http://gerrit.cloudera.org:8080/20654 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: Iab7cc66383bc62f209987dd3fb42fc3fc6604726 Gerrit-Change-Number: 20654 Gerrit-PatchSet: 3 Gerrit-Owner: Anonymous Coward Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Wenzhe Zhou
