[Impala-ASF-CR] IMPALA-12399: Pass eventTypeSkipList with OPEN TXN in NotificationEventRequest to avoid receiving OPEN TXN events from HMS
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/20427 ) Change subject: IMPALA-12399: Pass eventTypeSkipList with OPEN_TXN in NotificationEventRequest to avoid receiving OPEN_TXN events from HMS .. Patch Set 3: Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/9644/ DRY_RUN=true -- To view, visit http://gerrit.cloudera.org:8080/20427 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Id2f2e1040abce0b00a8a07bfe8b46afcd98290bf Gerrit-Change-Number: 20427 Gerrit-PatchSet: 3 Gerrit-Owner: Anonymous Coward Gerrit-Reviewer: Impala Public Jenkins Gerrit-Comment-Date: Tue, 29 Aug 2023 03:17:07 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-11535: Skip older events in the event processor based on the latestRefreshEventID
Quanlong Huang has posted comments on this change. (
http://gerrit.cloudera.org:8080/20022 )
Change subject: IMPALA-11535: Skip older events in the event processor based on
the latestRefreshEventID
..
Patch Set 14:
(1 comment)
http://gerrit.cloudera.org:8080/#/c/20022/14/tests/custom_cluster/test_events_custom_configs.py
File tests/custom_cluster/test_events_custom_configs.py:
http://gerrit.cloudera.org:8080/#/c/20022/14/tests/custom_cluster/test_events_custom_configs.py@337
PS14, Line 337: EventProcessorUtils.wait_for_event_processing(self)
Can we also verify no table/partitions are refreshed here? We can extract the
following pattern into a method to simplify the code:
def verify_skipping_hive_stmt_events(stmt):
self.run_stmt_in_hive(stmt)
self.client.execute("refresh ...")
get event metrics
EventProcessorUtils.wait_for_event_processing(self)
get event metrics and compare them
'stmt' can be multiple statments like 3 INSERTs.
--
To view, visit http://gerrit.cloudera.org:8080/20022
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Ic0dc5c7396d80616680d8a5805ce80db293b72e1
Gerrit-Change-Number: 20022
Gerrit-PatchSet: 14
Gerrit-Owner: Sai Hemanth Gantasala
Gerrit-Reviewer: Csaba Ringhofer
Gerrit-Reviewer: Impala Public Jenkins
Gerrit-Reviewer: Quanlong Huang
Gerrit-Reviewer: Sai Hemanth Gantasala
Gerrit-Reviewer: Zoltan Borok-Nagy
Gerrit-Comment-Date: Tue, 29 Aug 2023 02:29:58 +
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-12231: Bump GBN to get HMS thrift API changes
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/20420 ) Change subject: IMPALA-12231: Bump GBN to get HMS thrift API changes .. Patch Set 4: Build Failed https://jenkins.impala.io/job/gerrit-code-review-checks/13866/ : Initial code review checks failed. See linked job for details on the failure. -- To view, visit http://gerrit.cloudera.org:8080/20420 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I117873b628aed3e24280f9fcd79643f918c8d5f3 Gerrit-Change-Number: 20420 Gerrit-PatchSet: 4 Gerrit-Owner: Sai Hemanth Gantasala Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Quanlong Huang Gerrit-Reviewer: Sai Hemanth Gantasala Gerrit-Comment-Date: Tue, 29 Aug 2023 02:03:28 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-12231: Bump GBN to get HMS thrift API changes
Hello Quanlong Huang, Impala Public Jenkins, I'd like you to reexamine a change. Please visit http://gerrit.cloudera.org:8080/20420 to look at the new patch set (#4). Change subject: IMPALA-12231: Bump GBN to get HMS thrift API changes .. IMPALA-12231: Bump GBN to get HMS thrift API changes We need a couple of hive changes HIVE-27319 and HIVE-27337 for catalogD to work with latest HMS server to fix IMPALA-11768 and IMPALA-11939 respectively. Bump CDP_BUILD_NUMBER (GBN) to 44206393 Bump various CDP versiona numbers to be based on 7.2.18.0-273 TESTING: Exhaustive tests ran clean Added a couple of tests for IMPALA-11939 and IMPALA-11768 Change-Id: I117873b628aed3e24280f9fcd79643f918c8d5f3 --- M bin/impala-config.sh M java/shaded-deps/hive-exec/pom.xml M tests/custom_cluster/test_events_custom_configs.py 3 files changed, 43 insertions(+), 11 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/20/20420/4 -- To view, visit http://gerrit.cloudera.org:8080/20420 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: I117873b628aed3e24280f9fcd79643f918c8d5f3 Gerrit-Change-Number: 20420 Gerrit-PatchSet: 4 Gerrit-Owner: Sai Hemanth Gantasala Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Quanlong Huang Gerrit-Reviewer: Sai Hemanth Gantasala
[Impala-ASF-CR] IMPALA-12231: Bump GBN to get HMS thrift API changes
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/20420 ) Change subject: IMPALA-12231: Bump GBN to get HMS thrift API changes .. Patch Set 3: Build Failed https://jenkins.impala.io/job/gerrit-code-review-checks/13865/ : Initial code review checks failed. See linked job for details on the failure. -- To view, visit http://gerrit.cloudera.org:8080/20420 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I117873b628aed3e24280f9fcd79643f918c8d5f3 Gerrit-Change-Number: 20420 Gerrit-PatchSet: 3 Gerrit-Owner: Sai Hemanth Gantasala Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Quanlong Huang Gerrit-Reviewer: Sai Hemanth Gantasala Gerrit-Comment-Date: Tue, 29 Aug 2023 01:14:50 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-12399: Pass eventTypeSkipList with OPEN TXN in NotificationEventRequest to avoid receiving OPEN TXN events from HMS
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/20427 ) Change subject: IMPALA-12399: Pass eventTypeSkipList with OPEN_TXN in NotificationEventRequest to avoid receiving OPEN_TXN events from HMS .. Patch Set 3: Verified-1 Build failed: https://jenkins.impala.io/job/gerrit-verify-dryrun/9643/ -- To view, visit http://gerrit.cloudera.org:8080/20427 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Id2f2e1040abce0b00a8a07bfe8b46afcd98290bf Gerrit-Change-Number: 20427 Gerrit-PatchSet: 3 Gerrit-Owner: Anonymous Coward Gerrit-Reviewer: Impala Public Jenkins Gerrit-Comment-Date: Tue, 29 Aug 2023 00:59:26 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-12231: Bump GBN to get HMS thrift API changes
Hello Quanlong Huang, Impala Public Jenkins, I'd like you to reexamine a change. Please visit http://gerrit.cloudera.org:8080/20420 to look at the new patch set (#3). Change subject: IMPALA-12231: Bump GBN to get HMS thrift API changes .. IMPALA-12231: Bump GBN to get HMS thrift API changes We need a couple of hive changes HIVE-27319 and HIVE-27337 for catalogD to work with latest HMS server to fix IMPALA-11768 and IMPALA-11939 respectively. Bump CDP_BUILD_NUMBER (GBN) to 44206393 Bump various CDP versiona numbers to be based on 7.2.18.0-273 TESTING: Exhaustive tests ran clean Added a couple of tests for IMPALA-11939 and IMPALA-11768 Change-Id: I117873b628aed3e24280f9fcd79643f918c8d5f3 --- M bin/impala-config.sh M java/shaded-deps/hive-exec/pom.xml M tests/custom_cluster/test_events_custom_configs.py 3 files changed, 39 insertions(+), 11 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/20/20420/3 -- To view, visit http://gerrit.cloudera.org:8080/20420 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: I117873b628aed3e24280f9fcd79643f918c8d5f3 Gerrit-Change-Number: 20420 Gerrit-PatchSet: 3 Gerrit-Owner: Sai Hemanth Gantasala Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Quanlong Huang Gerrit-Reviewer: Sai Hemanth Gantasala
[Impala-ASF-CR] IMPALA-12409: Don't allow EXTERNAL Iceberg tables to point another Iceberg table in Hive catalog
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/20429 ) Change subject: IMPALA-12409: Don't allow EXTERNAL Iceberg tables to point another Iceberg table in Hive catalog .. Patch Set 1: Verified+1 -- To view, visit http://gerrit.cloudera.org:8080/20429 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ifb0d7f0e7ec40fba356bd58b43f68d070432de71 Gerrit-Change-Number: 20429 Gerrit-PatchSet: 1 Gerrit-Owner: Zoltan Borok-Nagy Gerrit-Reviewer: Impala Public Jenkins Gerrit-Comment-Date: Mon, 28 Aug 2023 21:50:22 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-12399: Pass eventTypeSkipList with OPEN TXN in NotificationEventRequest to avoid receiving OPEN TXN events from HMS
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/20427 ) Change subject: IMPALA-12399: Pass eventTypeSkipList with OPEN_TXN in NotificationEventRequest to avoid receiving OPEN_TXN events from HMS .. Patch Set 3: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/13864/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/20427 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Id2f2e1040abce0b00a8a07bfe8b46afcd98290bf Gerrit-Change-Number: 20427 Gerrit-PatchSet: 3 Gerrit-Owner: Anonymous Coward Gerrit-Reviewer: Impala Public Jenkins Gerrit-Comment-Date: Mon, 28 Aug 2023 21:01:27 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-12399: Pass eventTypeSkipList with OPEN TXN in NotificationEventRequest to avoid receiving OPEN TXN events from HMS
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/20427 ) Change subject: IMPALA-12399: Pass eventTypeSkipList with OPEN_TXN in NotificationEventRequest to avoid receiving OPEN_TXN events from HMS .. Patch Set 3: Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/9643/ DRY_RUN=true -- To view, visit http://gerrit.cloudera.org:8080/20427 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Id2f2e1040abce0b00a8a07bfe8b46afcd98290bf Gerrit-Change-Number: 20427 Gerrit-PatchSet: 3 Gerrit-Owner: Anonymous Coward Gerrit-Reviewer: Impala Public Jenkins Gerrit-Comment-Date: Mon, 28 Aug 2023 20:41:43 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-12399: Pass eventTypeSkipList with OPEN TXN in NotificationEventRequest to avoid receiving OPEN TXN events from HMS
k.venureddy2...@gmail.com has uploaded a new patch set (#3). ( http://gerrit.cloudera.org:8080/20427 ) Change subject: IMPALA-12399: Pass eventTypeSkipList with OPEN_TXN in NotificationEventRequest to avoid receiving OPEN_TXN events from HMS .. IMPALA-12399: Pass eventTypeSkipList with OPEN_TXN in NotificationEventRequest to avoid receiving OPEN_TXN events from HMS Notification events like OPEN_TXN are ignored on catalogd MetastoreEventsProcessor. So, we can pass eventTypeSkipList with OPEN_TXN in NotificationEventRequest while invoking get_next_notification() to avoid reading such notification messages from metastore and then ignoring on catalogd. OPEN_TXN event being more frequent(received even upon describe table operation from beeline), we can significantly reduce unwanted processing on both hive metastore and catalogd. Catalogd reads events in batches of EVENTS_BATCH_SIZE_PER_RPC, skipping such unnecessary events can help catchup the events faster. Testing: - Manually tested Change-Id: Id2f2e1040abce0b00a8a07bfe8b46afcd98290bf --- M fe/src/compat-hive-3/java/org/apache/impala/compat/MetastoreShim.java M fe/src/main/java/org/apache/impala/catalog/events/MetastoreEventsProcessor.java 2 files changed, 6 insertions(+), 0 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/27/20427/3 -- To view, visit http://gerrit.cloudera.org:8080/20427 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: Id2f2e1040abce0b00a8a07bfe8b46afcd98290bf Gerrit-Change-Number: 20427 Gerrit-PatchSet: 3 Gerrit-Owner: Anonymous Coward Gerrit-Reviewer: Impala Public Jenkins
[Impala-ASF-CR] IMPALA-12395: Override scan cardinality for optimized count star
Impala Public Jenkins has submitted this change and it was merged. ( http://gerrit.cloudera.org:8080/20406 ) Change subject: IMPALA-12395: Override scan cardinality for optimized count star .. IMPALA-12395: Override scan cardinality for optimized count star The cardinality estimate in HdfsScanNode.java for count queries does not account for the fact that the count optimization only scans metadata and not the actual columns. Optimized count star scan will return only 1 row per parquet row group. This patch override the scan cardinality with total number of files, which is the closest estimate to number of row group. Similar override already exist in IcebergScanNode.java. Testing: - Add count query testcases in test_query_cpu_count_divisor_default - Pass core tests Change-Id: Id5ce967657208057d50bd80adadac29ebb51cbc5 Reviewed-on: http://gerrit.cloudera.org:8080/20406 Reviewed-by: Impala Public Jenkins Tested-by: Impala Public Jenkins --- M fe/src/main/java/org/apache/impala/planner/HdfsScanNode.java M testdata/workloads/functional-planner/queries/PlannerTest/resource-requirements.test M tests/custom_cluster/test_executor_groups.py M tests/custom_cluster/test_query_retries.py 4 files changed, 73 insertions(+), 23 deletions(-) Approvals: Impala Public Jenkins: Looks good to me, approved; Verified -- To view, visit http://gerrit.cloudera.org:8080/20406 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: merged Gerrit-Change-Id: Id5ce967657208057d50bd80adadac29ebb51cbc5 Gerrit-Change-Number: 20406 Gerrit-PatchSet: 3 Gerrit-Owner: Riza Suminto Gerrit-Reviewer: Abhishek Rawat Gerrit-Reviewer: David Rorke Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Riza Suminto Gerrit-Reviewer: Wenzhe Zhou Gerrit-Reviewer: Zoltan Borok-Nagy
[Impala-ASF-CR] IMPALA-12395: Override scan cardinality for optimized count star
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/20406 ) Change subject: IMPALA-12395: Override scan cardinality for optimized count star .. Patch Set 2: Verified+1 -- To view, visit http://gerrit.cloudera.org:8080/20406 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Id5ce967657208057d50bd80adadac29ebb51cbc5 Gerrit-Change-Number: 20406 Gerrit-PatchSet: 2 Gerrit-Owner: Riza Suminto Gerrit-Reviewer: Abhishek Rawat Gerrit-Reviewer: David Rorke Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Riza Suminto Gerrit-Reviewer: Wenzhe Zhou Gerrit-Reviewer: Zoltan Borok-Nagy Gerrit-Comment-Date: Mon, 28 Aug 2023 20:32:11 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-12357: Skip scheduling bloom filter from full-build scan
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/20366 ) Change subject: IMPALA-12357: Skip scheduling bloom filter from full-build scan .. Patch Set 6: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/13862/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/20366 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I494533bc06da84e606cbd1ae161908089a5e Gerrit-Change-Number: 20366 Gerrit-PatchSet: 6 Gerrit-Owner: Riza Suminto Gerrit-Reviewer: Aman Sinha Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Riza Suminto Gerrit-Comment-Date: Mon, 28 Aug 2023 19:38:15 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-12382: Unregister executor when it's shutdown gracefully
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/20402 ) Change subject: IMPALA-12382: Unregister executor when it's shutdown gracefully .. Patch Set 3: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/13863/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/20402 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I6607285adedef1a0bdd87c444fe9d357259c0a50 Gerrit-Change-Number: 20402 Gerrit-PatchSet: 3 Gerrit-Owner: Wenzhe Zhou Gerrit-Reviewer: Impala Public Jenkins Gerrit-Comment-Date: Mon, 28 Aug 2023 19:51:55 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-12357: Skip scheduling bloom filter from full-build scan
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/20366 ) Change subject: IMPALA-12357: Skip scheduling bloom filter from full-build scan .. Patch Set 5: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/13861/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/20366 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I494533bc06da84e606cbd1ae161908089a5e Gerrit-Change-Number: 20366 Gerrit-PatchSet: 5 Gerrit-Owner: Riza Suminto Gerrit-Reviewer: Aman Sinha Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Riza Suminto Gerrit-Comment-Date: Mon, 28 Aug 2023 19:35:35 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-12382: Unregister executor when it's shutdown gracefully
Wenzhe Zhou has abandoned this change. ( http://gerrit.cloudera.org:8080/20402 ) Change subject: IMPALA-12382: Unregister executor when it's shutdown gracefully .. Abandoned We already have mechanism to avoid scheduling new task on the executors which are shutting down by marking the executor in "quiescing" state. -- To view, visit http://gerrit.cloudera.org:8080/20402 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: abandon Gerrit-Change-Id: I6607285adedef1a0bdd87c444fe9d357259c0a50 Gerrit-Change-Number: 20402 Gerrit-PatchSet: 3 Gerrit-Owner: Wenzhe Zhou
[Impala-ASF-CR] IMPALA-12382: Unregister executor when it's shutdown gracefully
Wenzhe Zhou has uploaded this change for review. ( http://gerrit.cloudera.org:8080/20402 Change subject: IMPALA-12382: Unregister executor when it's shutdown gracefully .. IMPALA-12382: Unregister executor when it's shutdown gracefully This patch adds new API UnregisterSubscriber() in StatestoreService. Executor will call this API to unregister itself with statestore when it's shutdown gracefully. This makes the executor been removed from cluster immediately and reduces the chance for coordinator to schedule fragments on the gracefully showndown executors. Testing: - TODO add new end-to-end unit-test in TestGracefulShutdown. - Passed core tests. Change-Id: I6607285adedef1a0bdd87c444fe9d357259c0a50 --- M be/src/rpc/thrift-server-test.cc M be/src/runtime/exec-env.cc M be/src/runtime/exec-env.h M be/src/service/impala-server.cc M be/src/statestore/statestore-service-client-wrapper.h M be/src/statestore/statestore-subscriber.cc M be/src/statestore/statestore-subscriber.h M be/src/statestore/statestore.cc M be/src/statestore/statestore.h M common/thrift/StatestoreService.thrift M tests/custom_cluster/test_restart_services.py 11 files changed, 166 insertions(+), 4 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/02/20402/3 -- To view, visit http://gerrit.cloudera.org:8080/20402 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newchange Gerrit-Change-Id: I6607285adedef1a0bdd87c444fe9d357259c0a50 Gerrit-Change-Number: 20402 Gerrit-PatchSet: 3 Gerrit-Owner: Wenzhe Zhou
[Impala-ASF-CR] IMPALA-12357: Skip scheduling bloom filter from full-build scan
Riza Suminto has posted comments on this change. ( http://gerrit.cloudera.org:8080/20366 ) Change subject: IMPALA-12357: Skip scheduling bloom filter from full-build scan .. Patch Set 6: (3 comments) http://gerrit.cloudera.org:8080/#/c/20366/4//COMMIT_MSG Commit Message: http://gerrit.cloudera.org:8080/#/c/20366/4//COMMIT_MSG@29 PS4, Line 29: thus reducing the bloom filter building, : aggregation, and evalua > This applies to the remote bloom filters right ? For local filters that are This patch does not differentiate between the locality of the filter target. I look again and found that it is still beneficial to prune LOCAL filter as well. For example, all pruned filter from new testcases in bloom-filter-assignment.test are LOCAL filter. The pruning is also correct since the JOIN node has BROADCAST build. http://gerrit.cloudera.org:8080/#/c/20366/4/fe/src/main/java/org/apache/impala/planner/RuntimeFilterGenerator.java File fe/src/main/java/org/apache/impala/planner/RuntimeFilterGenerator.java: http://gerrit.cloudera.org:8080/#/c/20366/4/fe/src/main/java/org/apache/impala/planner/RuntimeFilterGenerator.java@252 PS4, Line 252: private int level_ = 1; > Is 'rank' the right term here ? Normally, rank implies some type of compar Replaced with level. http://gerrit.cloudera.org:8080/#/c/20366/4/fe/src/main/java/org/apache/impala/planner/RuntimeFilterGenerator.java@844 PS4, Line 844: = true; > For testing purposes or for a backdoor in a real deployment, would it be us Added max_filter_error_rate_from_full_scan, mainly to allow disabling this feature. There are also existing max_filter_error_rate flag and RUNTIME_FILTER_ERROR_RATE query option to tune runtime filter accuracy. The difference is those configs are preliminary used to calculate ideal bloom filter size before RUNTIME_FILTER_MAX_SIZE and RUNTIME_FILTER_MIN_SIZE limits are applied. -- To view, visit http://gerrit.cloudera.org:8080/20366 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I494533bc06da84e606cbd1ae161908089a5e Gerrit-Change-Number: 20366 Gerrit-PatchSet: 6 Gerrit-Owner: Riza Suminto Gerrit-Reviewer: Aman Sinha Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Riza Suminto Gerrit-Comment-Date: Mon, 28 Aug 2023 19:22:50 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-12357: Skip scheduling bloom filter from full-build scan
Hello Aman Sinha, Csaba Ringhofer, Impala Public Jenkins, I'd like you to reexamine a change. Please visit http://gerrit.cloudera.org:8080/20366 to look at the new patch set (#6). Change subject: IMPALA-12357: Skip scheduling bloom filter from full-build scan .. IMPALA-12357: Skip scheduling bloom filter from full-build scan PK-FK join between a dimension table and a fact table is common occurrences in a query. Such join often does not involve any predicate filter in the dimension table. Thus, bloom filter value from this kind of dimension table scan (PK) will most likely to have all values from the fact table column (FK). It is ineffective to generate this filter because it is unlikely to reject any rows, especially if the bloom filter size is large and has high false positive probability (fpp) estimate. This patch skip scheduling bloom filter from join node that has this characteristics: 1. Build side is full table scan 2. The build scan does not have any predicate filter nor consume any runtime filter 3. The planned bloom filter has fpp estimate higher than 0.9 PK-FK relationship is rarely defined and enforced at table schema definition. Therefore, the third criteria replace the PK-FK relationship characteristic checks. It is also a narrower criteria that only target the large bloom filters, thus reducing the bloom filter building, aggregation, and evaluation overhead. Non-bloom filters remain unchanged as they are relatively lighter to build and evaluate than bloom filter. Testing: - Add testcase in testBloomFilterAssignment - Pass core tests - Ran TPC-DS 3TB with following query options: * RUNTIME_FILTER_MIN_SIZE=8192 * RUNTIME_FILTER_MAX_SIZE=2097152 * MAX_NUM_RUNTIME_FILTERS=50 * RUNTIME_FILTER_WAIT_TIME_MS=1 22 out of 103 queries show reduction in number of runtime bloom filters without any notable performance regression. Change-Id: I494533bc06da84e606cbd1ae161908089a5e --- M be/src/service/fe-support.cc M be/src/util/backend-gflag-util.cc M common/thrift/BackendGflags.thrift M fe/src/main/java/org/apache/impala/common/TreeNode.java M fe/src/main/java/org/apache/impala/planner/RuntimeFilterGenerator.java M fe/src/main/java/org/apache/impala/service/BackendConfig.java M fe/src/main/java/org/apache/impala/service/FeSupport.java M testdata/workloads/functional-planner/queries/PlannerTest/bloom-filter-assignment.test 8 files changed, 446 insertions(+), 37 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/66/20366/6 -- To view, visit http://gerrit.cloudera.org:8080/20366 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: I494533bc06da84e606cbd1ae161908089a5e Gerrit-Change-Number: 20366 Gerrit-PatchSet: 6 Gerrit-Owner: Riza Suminto Gerrit-Reviewer: Aman Sinha Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Riza Suminto
[Impala-ASF-CR] IMPALA-12357: Skip scheduling bloom filter from full-build scan
Hello Aman Sinha, Csaba Ringhofer, Impala Public Jenkins, I'd like you to reexamine a change. Please visit http://gerrit.cloudera.org:8080/20366 to look at the new patch set (#5). Change subject: IMPALA-12357: Skip scheduling bloom filter from full-build scan .. IMPALA-12357: Skip scheduling bloom filter from full-build scan PK-FK join between a dimension table and a fact table is common occurrences in a query. Such join often does not involve any predicate filter in the dimension table. Thus, bloom filter value from this kind of dimension table scan (PK) will most likely to have all values from the fact table column (FK). It is ineffective to generate this filter because it is unlikely to reject any rows, especially if the bloom filter size is large and has high false positive probability (fpp) estimate. This patch skip scheduling bloom filter from join node that has this characteristics: 1. Build side is full table scan 2. The build scan does not have any predicate filter nor consume any runtime filter 3. The planned bloom filter has fpp estimate higher than 0.9 PK-FK relationship is rarely defined and enforced at table schema definition. Therefore, the third criteria replace the PK-FK relationship characteristic checks. It is also a narrower criteria that only target the large bloom filters, thus reducing the bloom filter building, aggregation, and evaluation overhead. Non-bloom filters remain unchanged as they are relatively lighter to build and evaluate than bloom filter. Testing: - Add testcase in testBloomFilterAssignment - Pass core tests - Ran TPC-DS 3TB with following query options: * RUNTIME_FILTER_MIN_SIZE=8192 * RUNTIME_FILTER_MAX_SIZE=2097152 * MAX_NUM_RUNTIME_FILTERS=50 * RUNTIME_FILTER_WAIT_TIME_MS=1 22 out of 103 queries show reduction in number of runtime bloom filters without any notable performance regression. Change-Id: I494533bc06da84e606cbd1ae161908089a5e --- M be/src/service/fe-support.cc M be/src/util/backend-gflag-util.cc M common/thrift/BackendGflags.thrift M fe/src/main/java/org/apache/impala/common/TreeNode.java M fe/src/main/java/org/apache/impala/planner/RuntimeFilterGenerator.java M fe/src/main/java/org/apache/impala/service/BackendConfig.java M fe/src/main/java/org/apache/impala/service/FeSupport.java M testdata/workloads/functional-planner/queries/PlannerTest/bloom-filter-assignment.test 8 files changed, 446 insertions(+), 37 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/66/20366/5 -- To view, visit http://gerrit.cloudera.org:8080/20366 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: I494533bc06da84e606cbd1ae161908089a5e Gerrit-Change-Number: 20366 Gerrit-PatchSet: 5 Gerrit-Owner: Riza Suminto Gerrit-Reviewer: Aman Sinha Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Riza Suminto
[Impala-ASF-CR] IMPALA-10856: Show client hosts and connections in the web UI
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/20358 ) Change subject: IMPALA-10856: Show client hosts and connections in the web UI .. Patch Set 10: Verified+1 -- To view, visit http://gerrit.cloudera.org:8080/20358 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ie89015b00e1b97a1836eeca205b2c80b32300227 Gerrit-Change-Number: 20358 Gerrit-PatchSet: 10 Gerrit-Owner: ttz <2433038...@qq.com> Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Quanlong Huang Gerrit-Reviewer: Zihao Ye Gerrit-Reviewer: ttz <2433038...@qq.com> Gerrit-Comment-Date: Mon, 28 Aug 2023 18:34:47 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-10856: Show client hosts and connections in the web UI
Impala Public Jenkins has submitted this change and it was merged. ( http://gerrit.cloudera.org:8080/20358 ) Change subject: IMPALA-10856: Show client hosts and connections in the web UI .. IMPALA-10856: Show client hosts and connections in the web UI This change adds two tables to display client hosts information and connections information in the sessions Web UI. In the Client Hosts table, include the following information items: -- Hostname, -- Total Connections, -- Total Sessions, -- Total Active Sessions, -- Total Inactive Sessions, -- Open Queries, -- Total Queries In the Connections table, include the following information items: -- Connection ID, -- Connection Type, -- User, -- Delegated User, -- Network Address, -- Session IDs Change-Id: Ie89015b00e1b97a1836eeca205b2c80b32300227 Reviewed-on: http://gerrit.cloudera.org:8080/20358 Reviewed-by: Impala Public Jenkins Tested-by: Impala Public Jenkins --- M be/src/rpc/thrift-server.cc M be/src/rpc/thrift-server.h M be/src/service/impala-http-handler.cc M be/src/service/impala-http-handler.h M be/src/service/impala-server.cc M be/src/service/impala-server.h M be/src/util/debug-util.cc M be/src/util/debug-util.h M www/sessions.tmpl 9 files changed, 312 insertions(+), 10 deletions(-) Approvals: Impala Public Jenkins: Looks good to me, approved; Verified -- To view, visit http://gerrit.cloudera.org:8080/20358 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: merged Gerrit-Change-Id: Ie89015b00e1b97a1836eeca205b2c80b32300227 Gerrit-Change-Number: 20358 Gerrit-PatchSet: 11 Gerrit-Owner: ttz <2433038...@qq.com> Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Quanlong Huang Gerrit-Reviewer: Zihao Ye Gerrit-Reviewer: ttz <2433038...@qq.com>
[Impala-ASF-CR] IMPALA-12409: Don't allow EXTERNAL Iceberg tables to point another Iceberg table in Hive catalog
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/20429 ) Change subject: IMPALA-12409: Don't allow EXTERNAL Iceberg tables to point another Iceberg table in Hive catalog .. Patch Set 1: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/13860/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/20429 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ifb0d7f0e7ec40fba356bd58b43f68d070432de71 Gerrit-Change-Number: 20429 Gerrit-PatchSet: 1 Gerrit-Owner: Zoltan Borok-Nagy Gerrit-Reviewer: Impala Public Jenkins Gerrit-Comment-Date: Mon, 28 Aug 2023 18:06:51 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-12156: Support High Availability for Statestore
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/20372 ) Change subject: IMPALA-12156: Support High Availability for Statestore .. Patch Set 4: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/13859/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/20372 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ibd2c814bbad5c04c1d50c2edaa5b910c82a6fd87 Gerrit-Change-Number: 20372 Gerrit-PatchSet: 4 Gerrit-Owner: Wenzhe Zhou Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Wenzhe Zhou Gerrit-Comment-Date: Mon, 28 Aug 2023 17:50:23 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-12409: Don't allow EXTERNAL Iceberg tables to point another Iceberg table in Hive catalog
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/20429 ) Change subject: IMPALA-12409: Don't allow EXTERNAL Iceberg tables to point another Iceberg table in Hive catalog .. Patch Set 1: Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/9642/ DRY_RUN=true -- To view, visit http://gerrit.cloudera.org:8080/20429 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ifb0d7f0e7ec40fba356bd58b43f68d070432de71 Gerrit-Change-Number: 20429 Gerrit-PatchSet: 1 Gerrit-Owner: Zoltan Borok-Nagy Gerrit-Reviewer: Impala Public Jenkins Gerrit-Comment-Date: Mon, 28 Aug 2023 17:41:18 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-12409: Don't allow EXTERNAL Iceberg tables to point another Iceberg table in Hive catalog
Zoltan Borok-Nagy has uploaded this change for review. (
http://gerrit.cloudera.org:8080/20429
Change subject: IMPALA-12409: Don't allow EXTERNAL Iceberg tables to point
another Iceberg table in Hive catalog
..
IMPALA-12409: Don't allow EXTERNAL Iceberg tables to point another Iceberg
table in Hive catalog
This patch forbids creating an EXTERNAL Iceberg table that points
to another Iceberg catalog. I.e. the following should be forbidden:
CREATE EXTERNAL TABLE ice_ext
STORED BY ICEBERG
TBLPROPERTIES ('iceberg.table_identifier'='db.tbl');
Loading such tables should also raise an error. Users need to query
the original Iceberg tables.
Testing:
* added e2e tests for CREATE EXTERNAL TABLE
* added e2e test about loading such table
Change-Id: Ifb0d7f0e7ec40fba356bd58b43f68d070432de71
---
M fe/src/main/java/org/apache/impala/analysis/CreateTableStmt.java
M fe/src/main/java/org/apache/impala/catalog/IcebergTable.java
M testdata/workloads/functional-query/queries/QueryTest/iceberg-catalogs.test
M testdata/workloads/functional-query/queries/QueryTest/iceberg-insert.test
M testdata/workloads/functional-query/queries/QueryTest/iceberg-negative.test
M tests/query_test/test_iceberg.py
6 files changed, 76 insertions(+), 124 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/29/20429/1
--
To view, visit http://gerrit.cloudera.org:8080/20429
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newchange
Gerrit-Change-Id: Ifb0d7f0e7ec40fba356bd58b43f68d070432de71
Gerrit-Change-Number: 20429
Gerrit-PatchSet: 1
Gerrit-Owner: Zoltan Borok-Nagy
[Impala-ASF-CR] IMPALA-12156: Support High Availability for Statestore
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/20372 ) Change subject: IMPALA-12156: Support High Availability for Statestore .. Patch Set 4: (1 comment) http://gerrit.cloudera.org:8080/#/c/20372/4/bin/start-impala-cluster.py File bin/start-impala-cluster.py: http://gerrit.cloudera.org:8080/#/c/20372/4/bin/start-impala-cluster.py@794 PS4, Line 794: c flake8: E131 continuation line unaligned for hanging indent -- To view, visit http://gerrit.cloudera.org:8080/20372 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ibd2c814bbad5c04c1d50c2edaa5b910c82a6fd87 Gerrit-Change-Number: 20372 Gerrit-PatchSet: 4 Gerrit-Owner: Wenzhe Zhou Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Wenzhe Zhou Gerrit-Comment-Date: Mon, 28 Aug 2023 17:25:53 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-12156: Support High Availability for Statestore
Wenzhe Zhou has posted comments on this change. ( http://gerrit.cloudera.org:8080/20372 ) Change subject: IMPALA-12156: Support High Availability for Statestore .. Patch Set 4: (5 comments) http://gerrit.cloudera.org:8080/#/c/20372/3/be/src/statestore/statestore-subscriber.cc File be/src/statestore/statestore-subscriber.cc: http://gerrit.cloudera.org:8080/#/c/20372/3/be/src/statestore/statestore-subscriber.cc@865 PS3, Line 865: // Try to acquire lock to avoid race with updating catalogd from registration > line too long (91 > 90) Done http://gerrit.cloudera.org:8080/#/c/20372/3/bin/start-impala-cluster.py File bin/start-impala-cluster.py: http://gerrit.cloudera.org:8080/#/c/20372/3/bin/start-impala-cluster.py@793 PS3, Line 793: > flake8: E501 line too long (94 > 90 characters) Done http://gerrit.cloudera.org:8080/#/c/20372/3/tests/custom_cluster/test_statestored_ha.py File tests/custom_cluster/test_statestored_ha.py: http://gerrit.cloudera.org:8080/#/c/20372/3/tests/custom_cluster/test_statestored_ha.py@24 PS3, Line 24: from tests.common.impala_cluster import DEFAULT_CATALOG_SERVICE_PORT > flake8: F401 'tests.common.skip.SkipIf' imported but unused Done http://gerrit.cloudera.org:8080/#/c/20372/3/tests/custom_cluster/test_statestored_ha.py@143 PS3, Line 143: p > flake8: F821 undefined name 'pytest' Done http://gerrit.cloudera.org:8080/#/c/20372/3/tests/custom_cluster/test_statestored_ha.py@320 PS3, Line 320: > flake8: E131 continuation line unaligned for hanging indent Done -- To view, visit http://gerrit.cloudera.org:8080/20372 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ibd2c814bbad5c04c1d50c2edaa5b910c82a6fd87 Gerrit-Change-Number: 20372 Gerrit-PatchSet: 4 Gerrit-Owner: Wenzhe Zhou Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Wenzhe Zhou Gerrit-Comment-Date: Mon, 28 Aug 2023 17:24:55 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-12156: Support High Availability for Statestore
Wenzhe Zhou has uploaded a new patch set (#4). ( http://gerrit.cloudera.org:8080/20372 ) Change subject: IMPALA-12156: Support High Availability for Statestore .. IMPALA-12156: Support High Availability for Statestore To support statestore HA, we allow two statestored instances in an Active-Passive HA pair to be added to an Impala cluster. We add the preemptive behavior for statestored. When HA is enabled, the preemptive behavior allows the statestored with the higher priority to become active and the paired statestored becomes standby. The active statestored acts as the owner of Impala cluster and provides statestore service for the cluster members. To enable catalog HA for a cluster, two statestoreds in the HA pair and all subscribers must be started with starting flag "enable_statestored_ha". - Define new service for Statestore HA. - Negotiate role for HA with peer statestore instance on startup. - Create HA monitor thread: active statestored send heartbeat to standby statestored. standby statestored monitor peer's connection states - Standby statestored send heartbeat to subscribers with request for connection state between active statestore and subscribers. Standby statestored save the connection state as failure detecer. - When standby statestored loss connection with active statestore, check the connection states for active statestore, and take over active role if majority of subscribers lose connections with active statestore. - New active statestored send RPC notification to all subscribers for new active statestored and active catalogd elected by new active staetstored. - New active statestored start to send heartbeat to its peer when it receive handshake from its peer. - Subscriber register to two statestoreds. - Subscriber report connection state for in-active statestore. - Subscriber switch to new active statstore, refuse topic update from standby statestored. Testings: - Added end-to-end unit tests. - TODO: pass core tests Change-Id: Ibd2c814bbad5c04c1d50c2edaa5b910c82a6fd87 --- M be/generated-sources/gen-cpp/CMakeLists.txt M be/src/catalog/catalog-server.cc M be/src/common/global-flags.cc M be/src/runtime/exec-env.cc M be/src/runtime/exec-env.h M be/src/scheduling/admissiond-env.cc M be/src/statestore/statestore-service-client-wrapper.h M be/src/statestore/statestore-subscriber-catalog.cc M be/src/statestore/statestore-subscriber-catalog.h M be/src/statestore/statestore-subscriber-client-wrapper.h M be/src/statestore/statestore-subscriber.cc M be/src/statestore/statestore-subscriber.h M be/src/statestore/statestore-test.cc M be/src/statestore/statestore.cc M be/src/statestore/statestore.h M be/src/statestore/statestored-main.cc M bin/start-impala-cluster.py M common/thrift/StatestoreService.thrift M common/thrift/metrics.json M tests/common/impala_cluster.py M tests/common/impala_service.py A tests/custom_cluster/test_statestored_ha.py 22 files changed, 2,088 insertions(+), 106 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/72/20372/4 -- To view, visit http://gerrit.cloudera.org:8080/20372 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: Ibd2c814bbad5c04c1d50c2edaa5b910c82a6fd87 Gerrit-Change-Number: 20372 Gerrit-PatchSet: 4 Gerrit-Owner: Wenzhe Zhou Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Wenzhe Zhou
[Impala-ASF-CR] IMPALA-12156: Support High Availability for Statestore
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/20372 ) Change subject: IMPALA-12156: Support High Availability for Statestore .. Patch Set 3: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/13858/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/20372 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ibd2c814bbad5c04c1d50c2edaa5b910c82a6fd87 Gerrit-Change-Number: 20372 Gerrit-PatchSet: 3 Gerrit-Owner: Wenzhe Zhou Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Wenzhe Zhou Gerrit-Comment-Date: Mon, 28 Aug 2023 17:12:03 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-12406: OPTIMIZE statement as an alias for INSERT OVERWRITE
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/20405 ) Change subject: IMPALA-12406: OPTIMIZE statement as an alias for INSERT OVERWRITE .. Patch Set 2: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/13857/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/20405 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ief42537499ffe64fafdefe25c8d175539234c4e7 Gerrit-Change-Number: 20405 Gerrit-PatchSet: 2 Gerrit-Owner: Noemi Pap-Takacs Gerrit-Reviewer: Gabor Kaszab Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Noemi Pap-Takacs Gerrit-Reviewer: Tamas Mate Gerrit-Reviewer: Zoltan Borok-Nagy Gerrit-Comment-Date: Mon, 28 Aug 2023 17:10:01 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-12156: Support High Availability for Statestore
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/20372 ) Change subject: IMPALA-12156: Support High Availability for Statestore .. Patch Set 3: (5 comments) http://gerrit.cloudera.org:8080/#/c/20372/3/be/src/statestore/statestore-subscriber.cc File be/src/statestore/statestore-subscriber.cc: http://gerrit.cloudera.org:8080/#/c/20372/3/be/src/statestore/statestore-subscriber.cc@865 PS3, Line 865: // Try to acquire lock to avoid race with updating catalogd from registration thread. line too long (91 > 90) http://gerrit.cloudera.org:8080/#/c/20372/3/bin/start-impala-cluster.py File bin/start-impala-cluster.py: http://gerrit.cloudera.org:8080/#/c/20372/3/bin/start-impala-cluster.py@793 PS3, Line 793: t flake8: E501 line too long (94 > 90 characters) http://gerrit.cloudera.org:8080/#/c/20372/3/tests/custom_cluster/test_statestored_ha.py File tests/custom_cluster/test_statestored_ha.py: http://gerrit.cloudera.org:8080/#/c/20372/3/tests/custom_cluster/test_statestored_ha.py@24 PS3, Line 24: from tests.common.skip import SkipIf flake8: F401 'tests.common.skip.SkipIf' imported but unused http://gerrit.cloudera.org:8080/#/c/20372/3/tests/custom_cluster/test_statestored_ha.py@143 PS3, Line 143: p flake8: F821 undefined name 'pytest' http://gerrit.cloudera.org:8080/#/c/20372/3/tests/custom_cluster/test_statestored_ha.py@320 PS3, Line 320: " flake8: E131 continuation line unaligned for hanging indent -- To view, visit http://gerrit.cloudera.org:8080/20372 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ibd2c814bbad5c04c1d50c2edaa5b910c82a6fd87 Gerrit-Change-Number: 20372 Gerrit-PatchSet: 3 Gerrit-Owner: Wenzhe Zhou Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Wenzhe Zhou Gerrit-Comment-Date: Mon, 28 Aug 2023 16:47:30 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-12156: Support High Availability for Statestore
Hello Impala Public Jenkins, I'd like you to reexamine a change. Please visit http://gerrit.cloudera.org:8080/20372 to look at the new patch set (#3). Change subject: IMPALA-12156: Support High Availability for Statestore .. IMPALA-12156: Support High Availability for Statestore To support statestore HA, we allow two statestored instances in an Active-Passive HA pair to be added to an Impala cluster. We add the preemptive behavior for statestored. When HA is enabled, the preemptive behavior allows the statestored with the higher priority to become active and the paired statestored becomes standby. The active statestored acts as the owner of Impala cluster and provides statestore service for the cluster members. To enable catalog HA for a cluster, two statestoreds in the HA pair and all subscribers must be started with starting flag "enable_statestored_ha". - Define new service for Statestore HA. - Negotiate role for HA with peer statestore instance on startup. - Create HA monitor thread: active statestored send heartbeat to standby statestored. standby statestored monitor peer's connection states - Standby statestored send heartbeat to subscribers with request for connection state between active statestore and subscribers. Standby statestored save the connection state as failure detecer. - When standby statestored loss connection with active statestore, check the connection states for active statestore, and take over active role if majority of subscribers lose connections with active statestore. - New active statestored send RPC notification to all subscribers for new active statestored and active catalogd elected by new active staetstored. - New active statestored start to send heartbeat to its peer when it receive handshake from its peer. - Subscriber register to two statestoreds. - Subscriber report connection state for in-active statestore. - Subscriber switch to new active statstore, refuse topic update from standby statestored. Testings: - Added end-to-end unit tests. - TODO: pass core tests Change-Id: Ibd2c814bbad5c04c1d50c2edaa5b910c82a6fd87 --- M be/generated-sources/gen-cpp/CMakeLists.txt M be/src/catalog/catalog-server.cc M be/src/common/global-flags.cc M be/src/runtime/exec-env.cc M be/src/runtime/exec-env.h M be/src/scheduling/admissiond-env.cc M be/src/statestore/statestore-service-client-wrapper.h M be/src/statestore/statestore-subscriber-catalog.cc M be/src/statestore/statestore-subscriber-catalog.h M be/src/statestore/statestore-subscriber-client-wrapper.h M be/src/statestore/statestore-subscriber.cc M be/src/statestore/statestore-subscriber.h M be/src/statestore/statestore-test.cc M be/src/statestore/statestore.cc M be/src/statestore/statestore.h M be/src/statestore/statestored-main.cc M bin/start-impala-cluster.py M common/thrift/StatestoreService.thrift M common/thrift/metrics.json M tests/common/impala_cluster.py M tests/common/impala_service.py A tests/custom_cluster/test_statestored_ha.py 22 files changed, 2,086 insertions(+), 106 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/72/20372/3 -- To view, visit http://gerrit.cloudera.org:8080/20372 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: Ibd2c814bbad5c04c1d50c2edaa5b910c82a6fd87 Gerrit-Change-Number: 20372 Gerrit-PatchSet: 3 Gerrit-Owner: Wenzhe Zhou Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Wenzhe Zhou
[Impala-ASF-CR] IMPALA-12406: OPTIMIZE statement as an alias for INSERT OVERWRITE
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/20405 ) Change subject: IMPALA-12406: OPTIMIZE statement as an alias for INSERT OVERWRITE .. Patch Set 2: (1 comment) http://gerrit.cloudera.org:8080/#/c/20405/2/tests/query_test/test_iceberg.py File tests/query_test/test_iceberg.py: http://gerrit.cloudera.org:8080/#/c/20405/2/tests/query_test/test_iceberg.py@1279 PS2, Line 1279: flake8: W292 no newline at end of file -- To view, visit http://gerrit.cloudera.org:8080/20405 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ief42537499ffe64fafdefe25c8d175539234c4e7 Gerrit-Change-Number: 20405 Gerrit-PatchSet: 2 Gerrit-Owner: Noemi Pap-Takacs Gerrit-Reviewer: Gabor Kaszab Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Noemi Pap-Takacs Gerrit-Reviewer: Tamas Mate Gerrit-Reviewer: Zoltan Borok-Nagy Gerrit-Comment-Date: Mon, 28 Aug 2023 16:45:28 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-12406: OPTIMIZE statement as an alias for INSERT OVERWRITE
Noemi Pap-Takacs has uploaded this change for review. ( http://gerrit.cloudera.org:8080/20405 Change subject: IMPALA-12406: OPTIMIZE statement as an alias for INSERT OVERWRITE .. IMPALA-12406: OPTIMIZE statement as an alias for INSERT OVERWRITE If an Iceberg table is frequently updated/written to in small batches, a lot of small files are created. This decreases read performance. Similarly, frequent row-level deletes contribute to this problem by creating delete files which have to be merged on read. Currently INSERT OVERWRITE is used as a workaround to rewrite and compact Iceberg tables. OPTIMIZE statement offers a new syntax and an Iceberg specific solution to this problem. This patch introduces the new syntax as an alias for INSERT OVERWRITE. Testing: - FE negative test for non-Iceberg tables and tables with parttition evolution - E2E: normal table, table with delete files, schema evolution Change-Id: Ief42537499ffe64fafdefe25c8d175539234c4e7 --- M fe/src/main/cup/sql-parser.cup M fe/src/main/java/org/apache/impala/analysis/AnalysisContext.java A fe/src/main/java/org/apache/impala/analysis/OptimizeStmt.java M fe/src/main/java/org/apache/impala/planner/PlannerContext.java M fe/src/main/java/org/apache/impala/service/Frontend.java M fe/src/main/jflex/sql-scanner.flex M testdata/workloads/functional-query/queries/QueryTest/iceberg-negative.test A testdata/workloads/functional-query/queries/QueryTest/iceberg-optimize.test M tests/query_test/test_iceberg.py 9 files changed, 240 insertions(+), 8 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/05/20405/2 -- To view, visit http://gerrit.cloudera.org:8080/20405 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newchange Gerrit-Change-Id: Ief42537499ffe64fafdefe25c8d175539234c4e7 Gerrit-Change-Number: 20405 Gerrit-PatchSet: 2 Gerrit-Owner: Noemi Pap-Takacs Gerrit-Reviewer: Gabor Kaszab Gerrit-Reviewer: Noemi Pap-Takacs Gerrit-Reviewer: Tamas Mate Gerrit-Reviewer: Zoltan Borok-Nagy
[Impala-ASF-CR] IMPALA-12395: Override scan cardinality for optimized count star
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/20406 ) Change subject: IMPALA-12395: Override scan cardinality for optimized count star .. Patch Set 2: Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/9641/ DRY_RUN=false -- To view, visit http://gerrit.cloudera.org:8080/20406 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Id5ce967657208057d50bd80adadac29ebb51cbc5 Gerrit-Change-Number: 20406 Gerrit-PatchSet: 2 Gerrit-Owner: Riza Suminto Gerrit-Reviewer: Abhishek Rawat Gerrit-Reviewer: David Rorke Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Riza Suminto Gerrit-Reviewer: Wenzhe Zhou Gerrit-Reviewer: Zoltan Borok-Nagy Gerrit-Comment-Date: Mon, 28 Aug 2023 16:15:12 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-12395: Override scan cardinality for optimized count star
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/20406 ) Change subject: IMPALA-12395: Override scan cardinality for optimized count star .. Patch Set 2: Code-Review+2 -- To view, visit http://gerrit.cloudera.org:8080/20406 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Id5ce967657208057d50bd80adadac29ebb51cbc5 Gerrit-Change-Number: 20406 Gerrit-PatchSet: 2 Gerrit-Owner: Riza Suminto Gerrit-Reviewer: Abhishek Rawat Gerrit-Reviewer: David Rorke Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Riza Suminto Gerrit-Reviewer: Wenzhe Zhou Gerrit-Reviewer: Zoltan Borok-Nagy Gerrit-Comment-Date: Mon, 28 Aug 2023 16:15:12 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-12395: Override scan cardinality for optimized count star
Wenzhe Zhou has posted comments on this change. ( http://gerrit.cloudera.org:8080/20406 ) Change subject: IMPALA-12395: Override scan cardinality for optimized count star .. Patch Set 1: Code-Review+2 Carry +1 from Abhishek -- To view, visit http://gerrit.cloudera.org:8080/20406 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Id5ce967657208057d50bd80adadac29ebb51cbc5 Gerrit-Change-Number: 20406 Gerrit-PatchSet: 1 Gerrit-Owner: Riza Suminto Gerrit-Reviewer: Abhishek Rawat Gerrit-Reviewer: David Rorke Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Riza Suminto Gerrit-Reviewer: Wenzhe Zhou Gerrit-Reviewer: Zoltan Borok-Nagy Gerrit-Comment-Date: Mon, 28 Aug 2023 16:12:59 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-12364: Display memory, disk and network metrics in webUI's query timeline
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/20355 ) Change subject: IMPALA-12364: Display memory, disk and network metrics in webUI's query timeline .. Patch Set 5: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/13856/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/20355 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ifd25e6f0bc9fbd664ec98936daff3f27182dfc7f Gerrit-Change-Number: 20355 Gerrit-PatchSet: 5 Gerrit-Owner: Surya Hebbar Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Kurt Deschler Gerrit-Reviewer: Riza Suminto Gerrit-Reviewer: Surya Hebbar Gerrit-Reviewer: Wenzhe Zhou Gerrit-Comment-Date: Mon, 28 Aug 2023 15:08:26 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-12364: Display memory, disk and network metrics in webUI's query timeline
Surya Hebbar has uploaded a new patch set (#5). ( http://gerrit.cloudera.org:8080/20355 ) Change subject: IMPALA-12364: Display memory, disk and network metrics in webUI's query timeline .. IMPALA-12364: Display memory, disk and network metrics in webUI's query timeline The fragment's plan nodes are enlarged with an animated transition on hovering over the query timeline's fragment diagram. On clicking the plan nodes, total thread and memory usage of the parent fragment are displayed, after accumulating memory and thread usage of all child nodes. In this way, memory and thread usage of multiple fragments can be compared alongside. A fragment's usage can be hidden by clicking on any of the child plan nodes again. Once a fragment's metrics are displayed, they are updated as they are collected from the profile. A grid-line is displayed along with a tooltip on hovering over the fragment diagram, containing the instantaneous time at that position. This grid-line also triggers tooltips and gridlines in other charts. The thread usage is being shown on the additional Y-axis. A warning is displayed on clicking a fragment with less number of samples available. RESOURCE_TRACE_RATIO query option provides the utilization values to be traced within the RuntimeProfile. It contains samples of disk and network usage on each host. These time series counters are available within the profile having the following names. - HostDiskWriteThroughput - HostDiskReadThroughput - HostNetworkRx - HostNetworkTx The additional Y-axis within the utilization chart is used to represent the average of these metrics. The memory units in tooltips and ticks on co-ordinate axes are being displayed in human readable form such as KB, MB, GB and PB for convenience. Both of the charts contain controls to close the chart. Timeticks are being autoscaled during fragment diagram's horizontal zoom. In addition to the scrollbar, hovering on edges of the window allows horizontal scrolling. Change-Id: Ifd25e6f0bc9fbd664ec98936daff3f27182dfc7f --- M www/query_timeline.tmpl M www/scripts/util.js 2 files changed, 590 insertions(+), 157 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/55/20355/5 -- To view, visit http://gerrit.cloudera.org:8080/20355 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: Ifd25e6f0bc9fbd664ec98936daff3f27182dfc7f Gerrit-Change-Number: 20355 Gerrit-PatchSet: 5 Gerrit-Owner: Surya Hebbar Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Kurt Deschler Gerrit-Reviewer: Riza Suminto Gerrit-Reviewer: Surya Hebbar Gerrit-Reviewer: Wenzhe Zhou
[Impala-ASF-CR] IMPALA-12395: Override scan cardinality for optimized count star
Abhishek Rawat has posted comments on this change. ( http://gerrit.cloudera.org:8080/20406 ) Change subject: IMPALA-12395: Override scan cardinality for optimized count star .. Patch Set 1: Code-Review+1 -- To view, visit http://gerrit.cloudera.org:8080/20406 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Id5ce967657208057d50bd80adadac29ebb51cbc5 Gerrit-Change-Number: 20406 Gerrit-PatchSet: 1 Gerrit-Owner: Riza Suminto Gerrit-Reviewer: Abhishek Rawat Gerrit-Reviewer: David Rorke Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Riza Suminto Gerrit-Reviewer: Wenzhe Zhou Gerrit-Reviewer: Zoltan Borok-Nagy Gerrit-Comment-Date: Mon, 28 Aug 2023 14:35:49 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-10856: Show client hosts and connections in the web UI
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/20358 ) Change subject: IMPALA-10856: Show client hosts and connections in the web UI .. Patch Set 10: Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/9640/ DRY_RUN=false -- To view, visit http://gerrit.cloudera.org:8080/20358 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ie89015b00e1b97a1836eeca205b2c80b32300227 Gerrit-Change-Number: 20358 Gerrit-PatchSet: 10 Gerrit-Owner: ttz <2433038...@qq.com> Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Quanlong Huang Gerrit-Reviewer: Zihao Ye Gerrit-Reviewer: ttz <2433038...@qq.com> Gerrit-Comment-Date: Mon, 28 Aug 2023 14:21:18 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-10856: Show client hosts and connections in the web UI
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/20358 ) Change subject: IMPALA-10856: Show client hosts and connections in the web UI .. Patch Set 10: Verified-1 Build failed: https://jenkins.impala.io/job/gerrit-verify-dryrun/9639/ -- To view, visit http://gerrit.cloudera.org:8080/20358 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ie89015b00e1b97a1836eeca205b2c80b32300227 Gerrit-Change-Number: 20358 Gerrit-PatchSet: 10 Gerrit-Owner: ttz <2433038...@qq.com> Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Quanlong Huang Gerrit-Reviewer: Zihao Ye Gerrit-Reviewer: ttz <2433038...@qq.com> Gerrit-Comment-Date: Mon, 28 Aug 2023 13:47:08 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-12383: Fix SingleNodePlanner aggregation limits
Qifan Chen has posted comments on this change. ( http://gerrit.cloudera.org:8080/20379 ) Change subject: IMPALA-12383: Fix SingleNodePlanner aggregation limits .. Patch Set 8: (1 comment) http://gerrit.cloudera.org:8080/#/c/20379/4/fe/src/main/java/org/apache/impala/planner/AggregationNode.java File fe/src/main/java/org/apache/impala/planner/AggregationNode.java: http://gerrit.cloudera.org:8080/#/c/20379/4/fe/src/main/java/org/apache/impala/planner/AggregationNode.java@786 PS4, Line 786: > What's is_serial_plan? I don't understand what case that's supposed to cove The original version of this method deals with only logical properties. The modified version adds physical properties, in particular the inclusion of the only distributed/parallel version of the aggregate operator. As a result, the the serial plan version is excluded. If we can test the serial plan version here, then we will not exclude a piece of functionality existed before. -- To view, visit http://gerrit.cloudera.org:8080/20379 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ic5eec1190e8e182152aa954897b79cc3f219c816 Gerrit-Change-Number: 20379 Gerrit-PatchSet: 8 Gerrit-Owner: Michael Smith Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Joe McDonnell Gerrit-Reviewer: Michael Smith Gerrit-Reviewer: Qifan Chen Gerrit-Reviewer: Quanlong Huang Gerrit-Comment-Date: Mon, 28 Aug 2023 13:06:22 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-5081: Add codegen opt level query option
Daniel Becker has posted comments on this change. ( http://gerrit.cloudera.org:8080/20399 ) Change subject: IMPALA-5081: Add codegen_opt_level query option .. Patch Set 6: (1 comment) Thanks. http://gerrit.cloudera.org:8080/#/c/20399/6/be/src/codegen/llvm-codegen-test.cc File be/src/codegen/llvm-codegen-test.cc: http://gerrit.cloudera.org:8080/#/c/20399/6/be/src/codegen/llvm-codegen-test.cc@673 PS6, Line 673: std::arra Could be constexpr. -- To view, visit http://gerrit.cloudera.org:8080/20399 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I371f8758b6552263e91a1fbfd9a6e1c28e1fa2bd Gerrit-Change-Number: 20399 Gerrit-PatchSet: 6 Gerrit-Owner: Michael Smith Gerrit-Reviewer: Daniel Becker Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Michael Smith Gerrit-Reviewer: Noemi Pap-Takacs Gerrit-Reviewer: Yida Wu Gerrit-Comment-Date: Mon, 28 Aug 2023 11:30:58 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-12399: Pass eventTypeSkipList with OPEN TXN in NotificationEventRequest to avoid receiving OPEN TXN events from HMS
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/20427 ) Change subject: IMPALA-12399: Pass eventTypeSkipList with OPEN_TXN in NotificationEventRequest to avoid receiving OPEN_TXN events from HMS .. Patch Set 2: Verified-1 Build failed: https://jenkins.impala.io/job/gerrit-verify-dryrun/9638/ -- To view, visit http://gerrit.cloudera.org:8080/20427 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Id2f2e1040abce0b00a8a07bfe8b46afcd98290bf Gerrit-Change-Number: 20427 Gerrit-PatchSet: 2 Gerrit-Owner: Anonymous Coward Gerrit-Reviewer: Impala Public Jenkins Gerrit-Comment-Date: Mon, 28 Aug 2023 10:03:19 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-10798: Initial support for reading JSON files
Quanlong Huang has posted comments on this change. ( http://gerrit.cloudera.org:8080/19699 ) Change subject: IMPALA-10798: Initial support for reading JSON files .. Patch Set 30: Code-Review+1 -- To view, visit http://gerrit.cloudera.org:8080/19699 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I31309cb8f2d04722a0508b3f9b8f1532ad49a569 Gerrit-Change-Number: 19699 Gerrit-PatchSet: 30 Gerrit-Owner: Zihao Ye Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Quanlong Huang Gerrit-Reviewer: Zihao Ye Gerrit-Comment-Date: Mon, 28 Aug 2023 09:30:27 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-10856: Show client hosts and connections in the web UI
Quanlong Huang has posted comments on this change. ( http://gerrit.cloudera.org:8080/20358 ) Change subject: IMPALA-10856: Show client hosts and connections in the web UI .. Patch Set 9: Code-Review+2 LGTM. Thanks for improving the WebUI! -- To view, visit http://gerrit.cloudera.org:8080/20358 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ie89015b00e1b97a1836eeca205b2c80b32300227 Gerrit-Change-Number: 20358 Gerrit-PatchSet: 9 Gerrit-Owner: ttz <2433038...@qq.com> Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Quanlong Huang Gerrit-Reviewer: Zihao Ye Gerrit-Reviewer: ttz <2433038...@qq.com> Gerrit-Comment-Date: Mon, 28 Aug 2023 09:29:45 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-10856: Show client hosts and connections in the web UI
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/20358 ) Change subject: IMPALA-10856: Show client hosts and connections in the web UI .. Patch Set 10: Code-Review+2 -- To view, visit http://gerrit.cloudera.org:8080/20358 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ie89015b00e1b97a1836eeca205b2c80b32300227 Gerrit-Change-Number: 20358 Gerrit-PatchSet: 10 Gerrit-Owner: ttz <2433038...@qq.com> Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Quanlong Huang Gerrit-Reviewer: Zihao Ye Gerrit-Reviewer: ttz <2433038...@qq.com> Gerrit-Comment-Date: Mon, 28 Aug 2023 09:30:21 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-10856: Show client hosts and connections in the web UI
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/20358 ) Change subject: IMPALA-10856: Show client hosts and connections in the web UI .. Patch Set 10: Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/9639/ DRY_RUN=false -- To view, visit http://gerrit.cloudera.org:8080/20358 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ie89015b00e1b97a1836eeca205b2c80b32300227 Gerrit-Change-Number: 20358 Gerrit-PatchSet: 10 Gerrit-Owner: ttz <2433038...@qq.com> Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Quanlong Huang Gerrit-Reviewer: Zihao Ye Gerrit-Reviewer: ttz <2433038...@qq.com> Gerrit-Comment-Date: Mon, 28 Aug 2023 09:30:21 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-10856: Show client hosts and connections in the web UI
ttz has posted comments on this change. ( http://gerrit.cloudera.org:8080/20358 ) Change subject: IMPALA-10856: Show client hosts and connections in the web UI .. Patch Set 9: (1 comment) Attentive and detail-oriented! http://gerrit.cloudera.org:8080/#/c/20358/8//COMMIT_MSG Commit Message: http://gerrit.cloudera.org:8080/#/c/20358/8//COMMIT_MSG@21 PS8, Line 21: Connection > Ack Done -- To view, visit http://gerrit.cloudera.org:8080/20358 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ie89015b00e1b97a1836eeca205b2c80b32300227 Gerrit-Change-Number: 20358 Gerrit-PatchSet: 9 Gerrit-Owner: ttz <2433038...@qq.com> Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Quanlong Huang Gerrit-Reviewer: Zihao Ye Gerrit-Reviewer: ttz <2433038...@qq.com> Gerrit-Comment-Date: Mon, 28 Aug 2023 09:12:05 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-10856: Show client hosts and connections in the web UI
ttz has posted comments on this change. ( http://gerrit.cloudera.org:8080/20358 ) Change subject: IMPALA-10856: Show client hosts and connections in the web UI .. Patch Set 9: (1 comment) http://gerrit.cloudera.org:8080/#/c/20358/8//COMMIT_MSG Commit Message: http://gerrit.cloudera.org:8080/#/c/20358/8//COMMIT_MSG@21 PS8, Line 21: Connection > nit: "Connection Type" Ack -- To view, visit http://gerrit.cloudera.org:8080/20358 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ie89015b00e1b97a1836eeca205b2c80b32300227 Gerrit-Change-Number: 20358 Gerrit-PatchSet: 9 Gerrit-Owner: ttz <2433038...@qq.com> Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Quanlong Huang Gerrit-Reviewer: Zihao Ye Gerrit-Reviewer: ttz <2433038...@qq.com> Gerrit-Comment-Date: Mon, 28 Aug 2023 09:01:45 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-10856: Show client hosts and connections in the web UI
Hello Quanlong Huang, Zihao Ye, Impala Public Jenkins, I'd like you to reexamine a change. Please visit http://gerrit.cloudera.org:8080/20358 to look at the new patch set (#9). Change subject: IMPALA-10856: Show client hosts and connections in the web UI .. IMPALA-10856: Show client hosts and connections in the web UI This change adds two tables to display client hosts information and connections information in the sessions Web UI. In the Client Hosts table, include the following information items: -- Hostname, -- Total Connections, -- Total Sessions, -- Total Active Sessions, -- Total Inactive Sessions, -- Open Queries, -- Total Queries In the Connections table, include the following information items: -- Connection ID, -- Connection Type, -- User, -- Delegated User, -- Network Address, -- Session IDs Change-Id: Ie89015b00e1b97a1836eeca205b2c80b32300227 --- M be/src/rpc/thrift-server.cc M be/src/rpc/thrift-server.h M be/src/service/impala-http-handler.cc M be/src/service/impala-http-handler.h M be/src/service/impala-server.cc M be/src/service/impala-server.h M be/src/util/debug-util.cc M be/src/util/debug-util.h M www/sessions.tmpl 9 files changed, 312 insertions(+), 10 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/58/20358/9 -- To view, visit http://gerrit.cloudera.org:8080/20358 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: Ie89015b00e1b97a1836eeca205b2c80b32300227 Gerrit-Change-Number: 20358 Gerrit-PatchSet: 9 Gerrit-Owner: ttz <2433038...@qq.com> Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Quanlong Huang Gerrit-Reviewer: Zihao Ye Gerrit-Reviewer: ttz <2433038...@qq.com>
[Impala-ASF-CR] IMPALA-10856: Show client hosts and connections in the web UI
Quanlong Huang has posted comments on this change. ( http://gerrit.cloudera.org:8080/20358 ) Change subject: IMPALA-10856: Show client hosts and connections in the web UI .. Patch Set 8: Code-Review+1 (1 comment) http://gerrit.cloudera.org:8080/#/c/20358/8//COMMIT_MSG Commit Message: http://gerrit.cloudera.org:8080/#/c/20358/8//COMMIT_MSG@21 PS8, Line 21: Server Name nit: "Connection Type" -- To view, visit http://gerrit.cloudera.org:8080/20358 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ie89015b00e1b97a1836eeca205b2c80b32300227 Gerrit-Change-Number: 20358 Gerrit-PatchSet: 8 Gerrit-Owner: ttz <2433038...@qq.com> Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Quanlong Huang Gerrit-Reviewer: Zihao Ye Gerrit-Reviewer: ttz <2433038...@qq.com> Gerrit-Comment-Date: Mon, 28 Aug 2023 07:57:24 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-10856: Show client hosts and connections in the web UI
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/20358 ) Change subject: IMPALA-10856: Show client hosts and connections in the web UI .. Patch Set 8: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/13855/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/20358 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ie89015b00e1b97a1836eeca205b2c80b32300227 Gerrit-Change-Number: 20358 Gerrit-PatchSet: 8 Gerrit-Owner: ttz <2433038...@qq.com> Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Quanlong Huang Gerrit-Reviewer: Zihao Ye Gerrit-Reviewer: ttz <2433038...@qq.com> Gerrit-Comment-Date: Mon, 28 Aug 2023 07:57:45 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-10856: Show client hosts and connections in the web UI
ttz has posted comments on this change. ( http://gerrit.cloudera.org:8080/20358 ) Change subject: IMPALA-10856: Show client hosts and connections in the web UI .. Patch Set 8: (1 comment) http://gerrit.cloudera.org:8080/#/c/20358/7/www/sessions.tmpl File www/sessions.tmpl: http://gerrit.cloudera.org:8080/#/c/20358/7/www/sessions.tmpl@62 PS7, Line 62: Connection > This might be confused. What about "Connection Type" or "Type" ? Done Naming it "Connection Type" looks good! Similar to "Session Type" in the session table. -- To view, visit http://gerrit.cloudera.org:8080/20358 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ie89015b00e1b97a1836eeca205b2c80b32300227 Gerrit-Change-Number: 20358 Gerrit-PatchSet: 8 Gerrit-Owner: ttz <2433038...@qq.com> Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Quanlong Huang Gerrit-Reviewer: Zihao Ye Gerrit-Reviewer: ttz <2433038...@qq.com> Gerrit-Comment-Date: Mon, 28 Aug 2023 07:36:47 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-10856: Show client hosts and connections in the web UI
Hello Quanlong Huang, Zihao Ye, Impala Public Jenkins, I'd like you to reexamine a change. Please visit http://gerrit.cloudera.org:8080/20358 to look at the new patch set (#8). Change subject: IMPALA-10856: Show client hosts and connections in the web UI .. IMPALA-10856: Show client hosts and connections in the web UI This change adds two tables to display client hosts information and connections information in the sessions Web UI. In the Client Hosts table, include the following information items: -- Hostname, -- Total Connections, -- Total Sessions, -- Total Active Sessions, -- Total Inactive Sessions, -- Open Queries, -- Total Queries In the Connections table, include the following information items: -- Connection ID, -- Server Name, -- User, -- Delegated User, -- Network Address, -- Session IDs Change-Id: Ie89015b00e1b97a1836eeca205b2c80b32300227 --- M be/src/rpc/thrift-server.cc M be/src/rpc/thrift-server.h M be/src/service/impala-http-handler.cc M be/src/service/impala-http-handler.h M be/src/service/impala-server.cc M be/src/service/impala-server.h M be/src/util/debug-util.cc M be/src/util/debug-util.h M www/sessions.tmpl 9 files changed, 312 insertions(+), 10 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/58/20358/8 -- To view, visit http://gerrit.cloudera.org:8080/20358 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: Ie89015b00e1b97a1836eeca205b2c80b32300227 Gerrit-Change-Number: 20358 Gerrit-PatchSet: 8 Gerrit-Owner: ttz <2433038...@qq.com> Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Quanlong Huang Gerrit-Reviewer: Zihao Ye Gerrit-Reviewer: ttz <2433038...@qq.com>
