[Impala-ASF-CR] IMPALA-12023: Skip resource checking on last executor group set
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/19649 ) Change subject: IMPALA-12023: Skip resource checking on last executor group set .. Patch Set 4: Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/9177/ DRY_RUN=false -- To view, visit http://gerrit.cloudera.org:8080/19649 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I5848e4f67939d3dd2fb105c1ae4ca8e15f2e556f Gerrit-Change-Number: 19649 Gerrit-PatchSet: 4 Gerrit-Owner: Riza Suminto Gerrit-Reviewer: Abhishek Rawat Gerrit-Reviewer: Andrew Sherman Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Riza Suminto Gerrit-Reviewer: Wenzhe Zhou Gerrit-Comment-Date: Fri, 24 Mar 2023 03:19:05 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-11966: Enable cache ozone file handles by default
Quanlong Huang has posted comments on this change. ( http://gerrit.cloudera.org:8080/19573 ) Change subject: IMPALA-11966: Enable cache_ozone_file_handles by default .. Patch Set 4: Code-Review+2 -- To view, visit http://gerrit.cloudera.org:8080/19573 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Icc66551f9b87af785a1c30b516ac39f4640638fe Gerrit-Change-Number: 19573 Gerrit-PatchSet: 4 Gerrit-Owner: Michael Smith Gerrit-Reviewer: David Rorke Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Joe McDonnell Gerrit-Reviewer: Michael Smith Gerrit-Reviewer: Quanlong Huang Gerrit-Comment-Date: Fri, 24 Mar 2023 02:47:19 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-11966: Enable cache ozone file handles by default
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/19573 ) Change subject: IMPALA-11966: Enable cache_ozone_file_handles by default .. Patch Set 5: Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/9176/ DRY_RUN=false -- To view, visit http://gerrit.cloudera.org:8080/19573 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Icc66551f9b87af785a1c30b516ac39f4640638fe Gerrit-Change-Number: 19573 Gerrit-PatchSet: 5 Gerrit-Owner: Michael Smith Gerrit-Reviewer: David Rorke Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Joe McDonnell Gerrit-Reviewer: Michael Smith Gerrit-Reviewer: Quanlong Huang Gerrit-Comment-Date: Fri, 24 Mar 2023 02:49:01 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-11966: Enable cache ozone file handles by default
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/19573 ) Change subject: IMPALA-11966: Enable cache_ozone_file_handles by default .. Patch Set 5: Code-Review+2 -- To view, visit http://gerrit.cloudera.org:8080/19573 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Icc66551f9b87af785a1c30b516ac39f4640638fe Gerrit-Change-Number: 19573 Gerrit-PatchSet: 5 Gerrit-Owner: Michael Smith Gerrit-Reviewer: David Rorke Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Joe McDonnell Gerrit-Reviewer: Michael Smith Gerrit-Reviewer: Quanlong Huang Gerrit-Comment-Date: Fri, 24 Mar 2023 02:49:00 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-12023: Skip resource checking on last executor group set
Riza Suminto has posted comments on this change. ( http://gerrit.cloudera.org:8080/19649 ) Change subject: IMPALA-12023: Skip resource checking on last executor group set .. Patch Set 4: query_test.test_tpcds_queries.TestTpcdsQueryWithProcessingCost.test_tpcds_q67a filed with the following error message: E Error occurred on backend 192.168.124.4:27000 E Memory left in process limit: 2.35 GB E Memory left in query limit: 4.00 GB E Query(ad42a9b75ce9737e:bb3c81ac): Limit=4.00 GB Reservation=0 ReservationLimit=3.20 GB OtherMemory=0 Total=0 Peak=0 E Memory is likely oversubscribed. Reducing query concurrency or configuring admission control may help avoid this error. The whole TestTpcdsQueryWithProcessingCost runs with MT_DOP=4. I wonder if that is too tight in dockerised setup. -- To view, visit http://gerrit.cloudera.org:8080/19649 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I5848e4f67939d3dd2fb105c1ae4ca8e15f2e556f Gerrit-Change-Number: 19649 Gerrit-PatchSet: 4 Gerrit-Owner: Riza Suminto Gerrit-Reviewer: Abhishek Rawat Gerrit-Reviewer: Andrew Sherman Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Riza Suminto Gerrit-Reviewer: Wenzhe Zhou Gerrit-Comment-Date: Fri, 24 Mar 2023 02:26:25 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-12015: enable healthz endpoint in admissiond webui
Wenzhe Zhou has posted comments on this change. ( http://gerrit.cloudera.org:8080/19647 ) Change subject: IMPALA-12015: enable healthz endpoint in admissiond webui .. Patch Set 3: Code-Review+1 -- To view, visit http://gerrit.cloudera.org:8080/19647 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Iffb5ee7a00992d1bdba1deddf181ae38aa206140 Gerrit-Change-Number: 19647 Gerrit-PatchSet: 3 Gerrit-Owner: Abhishek Rawat Gerrit-Reviewer: Abhishek Rawat Gerrit-Reviewer: Andrew Sherman Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Wenzhe Zhou Gerrit-Comment-Date: Fri, 24 Mar 2023 02:09:17 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-12015: enable healthz endpoint in admissiond webui
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/19647 ) Change subject: IMPALA-12015: enable healthz endpoint in admissiond webui .. Patch Set 3: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/12687/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/19647 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Iffb5ee7a00992d1bdba1deddf181ae38aa206140 Gerrit-Change-Number: 19647 Gerrit-PatchSet: 3 Gerrit-Owner: Abhishek Rawat Gerrit-Reviewer: Abhishek Rawat Gerrit-Reviewer: Andrew Sherman Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Wenzhe Zhou Gerrit-Comment-Date: Fri, 24 Mar 2023 02:07:00 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-12015: enable healthz endpoint in admissiond webui
Abhishek Rawat has posted comments on this change. ( http://gerrit.cloudera.org:8080/19647 ) Change subject: IMPALA-12015: enable healthz endpoint in admissiond webui .. Patch Set 2: (2 comments) http://gerrit.cloudera.org:8080/#/c/19647/2/be/src/scheduling/admissiond-env.h File be/src/scheduling/admissiond-env.h: http://gerrit.cloudera.org:8080/#/c/19647/2/be/src/scheduling/admissiond-env.h@83 PS2, Line 83: > nit: extra line Done http://gerrit.cloudera.org:8080/#/c/19647/2/be/src/service/impala-http-handler.cc File be/src/service/impala-http-handler.cc: http://gerrit.cloudera.org:8080/#/c/19647/2/be/src/service/impala-http-handler.cc@213 PS2, Line 213: NULL > nit: nullptr Done -- To view, visit http://gerrit.cloudera.org:8080/19647 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Iffb5ee7a00992d1bdba1deddf181ae38aa206140 Gerrit-Change-Number: 19647 Gerrit-PatchSet: 2 Gerrit-Owner: Abhishek Rawat Gerrit-Reviewer: Abhishek Rawat Gerrit-Reviewer: Andrew Sherman Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Wenzhe Zhou Gerrit-Comment-Date: Fri, 24 Mar 2023 01:46:50 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-12015: enable healthz endpoint in admissiond webui
Abhishek Rawat has uploaded a new patch set (#3). ( http://gerrit.cloudera.org:8080/19647 ) Change subject: IMPALA-12015: enable healthz endpoint in admissiond webui .. IMPALA-12015: enable healthz endpoint in admissiond webui Exposed '/healthz' endpoint in admissiond's webserver and metrics webserver. The handler for '/healthz' returns 'OK' if the statestore subscriber service and KRPC service have started on admissiond. Testing: - Manually tested '/healthz' on both admissiond webserver and admissiond metrics webserver - Updated existing 'LdapWebserverTest' to also include test for admissiond's metrics webserver Change-Id: Iffb5ee7a00992d1bdba1deddf181ae38aa206140 --- M be/src/scheduling/admission-control-service.h M be/src/scheduling/admissiond-env.cc M be/src/service/impala-http-handler.cc M fe/src/test/java/org/apache/impala/customcluster/CustomClusterRunner.java M fe/src/test/java/org/apache/impala/customcluster/LdapWebserverTest.java 5 files changed, 64 insertions(+), 28 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/47/19647/3 -- To view, visit http://gerrit.cloudera.org:8080/19647 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: Iffb5ee7a00992d1bdba1deddf181ae38aa206140 Gerrit-Change-Number: 19647 Gerrit-PatchSet: 3 Gerrit-Owner: Abhishek Rawat Gerrit-Reviewer: Abhishek Rawat Gerrit-Reviewer: Andrew Sherman Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Wenzhe Zhou
[Impala-ASF-CR] IMPALA-12023: Skip resource checking on last executor group set
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/19649 ) Change subject: IMPALA-12023: Skip resource checking on last executor group set .. Patch Set 4: Verified-1 Build failed: https://jenkins.impala.io/job/gerrit-verify-dryrun/9175/ -- To view, visit http://gerrit.cloudera.org:8080/19649 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I5848e4f67939d3dd2fb105c1ae4ca8e15f2e556f Gerrit-Change-Number: 19649 Gerrit-PatchSet: 4 Gerrit-Owner: Riza Suminto Gerrit-Reviewer: Abhishek Rawat Gerrit-Reviewer: Andrew Sherman Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Riza Suminto Gerrit-Reviewer: Wenzhe Zhou Gerrit-Comment-Date: Fri, 24 Mar 2023 01:47:13 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-12015: enable healthz endpoint in admissiond webui
Wenzhe Zhou has posted comments on this change. ( http://gerrit.cloudera.org:8080/19647 ) Change subject: IMPALA-12015: enable healthz endpoint in admissiond webui .. Patch Set 2: Code-Review+1 (2 comments) looks good to me. Just two nits; http://gerrit.cloudera.org:8080/#/c/19647/2/be/src/scheduling/admissiond-env.h File be/src/scheduling/admissiond-env.h: http://gerrit.cloudera.org:8080/#/c/19647/2/be/src/scheduling/admissiond-env.h@83 PS2, Line 83: nit: extra line http://gerrit.cloudera.org:8080/#/c/19647/2/be/src/service/impala-http-handler.cc File be/src/service/impala-http-handler.cc: http://gerrit.cloudera.org:8080/#/c/19647/2/be/src/service/impala-http-handler.cc@213 PS2, Line 213: NULL nit: nullptr -- To view, visit http://gerrit.cloudera.org:8080/19647 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Iffb5ee7a00992d1bdba1deddf181ae38aa206140 Gerrit-Change-Number: 19647 Gerrit-PatchSet: 2 Gerrit-Owner: Abhishek Rawat Gerrit-Reviewer: Abhishek Rawat Gerrit-Reviewer: Andrew Sherman Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Wenzhe Zhou Gerrit-Comment-Date: Fri, 24 Mar 2023 01:31:14 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-12015: enable healthz endpoint in admissiond webui
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/19647 ) Change subject: IMPALA-12015: enable healthz endpoint in admissiond webui .. Patch Set 2: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/12686/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/19647 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Iffb5ee7a00992d1bdba1deddf181ae38aa206140 Gerrit-Change-Number: 19647 Gerrit-PatchSet: 2 Gerrit-Owner: Abhishek Rawat Gerrit-Reviewer: Abhishek Rawat Gerrit-Reviewer: Andrew Sherman Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Wenzhe Zhou Gerrit-Comment-Date: Fri, 24 Mar 2023 01:32:58 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-8731: Balance queries across multiple executor groups
Wenzhe Zhou has posted comments on this change. ( http://gerrit.cloudera.org:8080/19630 ) Change subject: IMPALA-8731: Balance queries across multiple executor groups .. Patch Set 5: (6 comments) http://gerrit.cloudera.org:8080/#/c/19630/5//COMMIT_MSG Commit Message: http://gerrit.cloudera.org:8080/#/c/19630/5//COMMIT_MSG@12 PS5, Line 12: '-balance_queries_across_executor_groups=true'. Please mention it's off by default http://gerrit.cloudera.org:8080/#/c/19630/5/be/src/scheduling/admission-controller.cc File be/src/scheduling/admission-controller.cc: http://gerrit.cloudera.org:8080/#/c/19630/5/be/src/scheduling/admission-controller.cc@2569 PS5, Line 2569: } add DCHECK with total_mem_limit - agg_effective_mem_reserved >= 0 and total_slots - agg_slots_in_use >= 0 http://gerrit.cloudera.org:8080/#/c/19630/5/be/src/scheduling/admission-controller.cc@2603 PS5, Line 2603: (mem_reserved, slots_in_use) available memory and slots in descending order http://gerrit.cloudera.org:8080/#/c/19630/5/be/src/scheduling/admission-controller.cc@2606 PS5, Line 2606: GetAvailableMemAndSlots(*a) > GetAvailableMemAndSlots(*b) For available memory and slots, which has high priority? Should we compare them separately? http://gerrit.cloudera.org:8080/#/c/19630/5/tests/custom_cluster/test_executor_groups.py File tests/custom_cluster/test_executor_groups.py: http://gerrit.cloudera.org:8080/#/c/19630/5/tests/custom_cluster/test_executor_groups.py@903 PS5, Line 903: coordinators and multiple executor groups mapped to the same resource pool.""" Could you add more detail for this test? http://gerrit.cloudera.org:8080/#/c/19630/5/tests/custom_cluster/test_executor_groups.py@906 PS5, Line 906: nit: indentation -- To view, visit http://gerrit.cloudera.org:8080/19630 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I25e851fb57c1d820c25cef5316f4ed800e4c6ac5 Gerrit-Change-Number: 19630 Gerrit-PatchSet: 5 Gerrit-Owner: Yifan Zhang Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Riza Suminto Gerrit-Reviewer: Wenzhe Zhou Gerrit-Comment-Date: Fri, 24 Mar 2023 01:22:40 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-12015: enable healthz endpoint in admissiond webui
Abhishek Rawat has uploaded a new patch set (#2). ( http://gerrit.cloudera.org:8080/19647 ) Change subject: IMPALA-12015: enable healthz endpoint in admissiond webui .. IMPALA-12015: enable healthz endpoint in admissiond webui Exposed '/healthz' endpoint in admissiond's webserver and metrics webserver. The handler for '/healthz' returns 'OK' if the statestore subscriber service and KRPC service have started on admissiond. Testing: - Manually tested '/healthz' on both admissiond webserver and admissiond metrics webserver - Updated existing 'LdapWebserverTest' to also include test for admissiond's metrics webserver Change-Id: Iffb5ee7a00992d1bdba1deddf181ae38aa206140 --- M be/src/scheduling/admission-control-service.h M be/src/scheduling/admissiond-env.cc M be/src/scheduling/admissiond-env.h M be/src/service/impala-http-handler.cc M fe/src/test/java/org/apache/impala/customcluster/CustomClusterRunner.java M fe/src/test/java/org/apache/impala/customcluster/LdapWebserverTest.java 6 files changed, 65 insertions(+), 28 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/47/19647/2 -- To view, visit http://gerrit.cloudera.org:8080/19647 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: Iffb5ee7a00992d1bdba1deddf181ae38aa206140 Gerrit-Change-Number: 19647 Gerrit-PatchSet: 2 Gerrit-Owner: Abhishek Rawat Gerrit-Reviewer: Abhishek Rawat Gerrit-Reviewer: Andrew Sherman Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Wenzhe Zhou
[Impala-ASF-CR] IMPALA-12015: enable healthz endpoint in admissiond webui
Abhishek Rawat has posted comments on this change. ( http://gerrit.cloudera.org:8080/19647 ) Change subject: IMPALA-12015: enable healthz endpoint in admissiond webui .. Patch Set 2: (4 comments) http://gerrit.cloudera.org:8080/#/c/19647/1/be/src/scheduling/admissiond-env.h File be/src/scheduling/admissiond-env.h: http://gerrit.cloudera.org:8080/#/c/19647/1/be/src/scheduling/admissiond-env.h@87 PS1, Line 87: > AdmissiondEnv analogous to ExecEnv for impalads. But service_started_ is no Will do. http://gerrit.cloudera.org:8080/#/c/19647/1/be/src/service/impala-http-handler.cc File be/src/service/impala-http-handler.cc: http://gerrit.cloudera.org:8080/#/c/19647/1/be/src/service/impala-http-handler.cc@213 PS1, Line 213: if ((server_ != NULL && server_->IsHealthy()) || > That's fine. Thanks for the input. I've made the above changes. http://gerrit.cloudera.org:8080/#/c/19647/1/fe/src/test/java/org/apache/impala/customcluster/CustomClusterRunner.java File fe/src/test/java/org/apache/impala/customcluster/CustomClusterRunner.java: http://gerrit.cloudera.org:8080/#/c/19647/1/fe/src/test/java/org/apache/impala/customcluster/CustomClusterRunner.java@63 PS1, Line 63: impaladArgs, catalogdArgs, statestoredArgs, "", new HashMap(), > line too long (91 > 90) Done http://gerrit.cloudera.org:8080/#/c/19647/1/fe/src/test/java/org/apache/impala/customcluster/CustomClusterRunner.java@85 PS1, Line 85: public static int StartImpalaCluster(String impaladArgs, String catalogdArgs, > line too long (95 > 90) Done -- To view, visit http://gerrit.cloudera.org:8080/19647 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Iffb5ee7a00992d1bdba1deddf181ae38aa206140 Gerrit-Change-Number: 19647 Gerrit-PatchSet: 2 Gerrit-Owner: Abhishek Rawat Gerrit-Reviewer: Abhishek Rawat Gerrit-Reviewer: Andrew Sherman Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Wenzhe Zhou Gerrit-Comment-Date: Fri, 24 Mar 2023 01:11:55 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-11966: Enable cache ozone file handles by default
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/19573 ) Change subject: IMPALA-11966: Enable cache_ozone_file_handles by default .. Patch Set 4: Verified+1 -- To view, visit http://gerrit.cloudera.org:8080/19573 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Icc66551f9b87af785a1c30b516ac39f4640638fe Gerrit-Change-Number: 19573 Gerrit-PatchSet: 4 Gerrit-Owner: Michael Smith Gerrit-Reviewer: David Rorke Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Joe McDonnell Gerrit-Reviewer: Michael Smith Gerrit-Comment-Date: Thu, 23 Mar 2023 23:25:21 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-7942 (part 2): Add query hints for predicate selectivities
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/18023 ) Change subject: IMPALA-7942 (part 2): Add query hints for predicate selectivities .. Patch Set 23: Verified+1 -- To view, visit http://gerrit.cloudera.org:8080/18023 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I2776b9bbd878b8a21d9c866b400140a454f59e1b Gerrit-Change-Number: 18023 Gerrit-PatchSet: 23 Gerrit-Owner: wangsheng Gerrit-Reviewer: Amogh Margoor Gerrit-Reviewer: Fucun Chu Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Qifan Chen Gerrit-Reviewer: Quanlong Huang Gerrit-Reviewer: Yifan Zhang Gerrit-Reviewer: Zoltan Borok-Nagy Gerrit-Reviewer: wangsheng Gerrit-Comment-Date: Thu, 23 Mar 2023 21:00:18 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-12023: Skip resource checking on last executor group set
Wenzhe Zhou has removed a vote on this change. Change subject: IMPALA-12023: Skip resource checking on last executor group set .. Removed Code-Review-2 by Riza Suminto -- To view, visit http://gerrit.cloudera.org:8080/19649 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: deleteVote Gerrit-Change-Id: I5848e4f67939d3dd2fb105c1ae4ca8e15f2e556f Gerrit-Change-Number: 19649 Gerrit-PatchSet: 4 Gerrit-Owner: Riza Suminto Gerrit-Reviewer: Abhishek Rawat Gerrit-Reviewer: Andrew Sherman Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Riza Suminto Gerrit-Reviewer: Wenzhe Zhou
[Impala-ASF-CR] IMPALA-12023: Skip resource checking on last executor group set
Wenzhe Zhou has posted comments on this change. ( http://gerrit.cloudera.org:8080/19649 ) Change subject: IMPALA-12023: Skip resource checking on last executor group set .. Patch Set 4: Code-Review+2 carry +1 from Andrew -- To view, visit http://gerrit.cloudera.org:8080/19649 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I5848e4f67939d3dd2fb105c1ae4ca8e15f2e556f Gerrit-Change-Number: 19649 Gerrit-PatchSet: 4 Gerrit-Owner: Riza Suminto Gerrit-Reviewer: Abhishek Rawat Gerrit-Reviewer: Andrew Sherman Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Riza Suminto Gerrit-Reviewer: Wenzhe Zhou Gerrit-Comment-Date: Thu, 23 Mar 2023 20:53:49 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-12023: Skip resource checking on last executor group set
Riza Suminto has posted comments on this change. ( http://gerrit.cloudera.org:8080/19649 ) Change subject: IMPALA-12023: Skip resource checking on last executor group set .. Patch Set 4: Code-Review-2 Apologies, it looks like I should not +2 myself. Put -2 in the meantime. -- To view, visit http://gerrit.cloudera.org:8080/19649 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I5848e4f67939d3dd2fb105c1ae4ca8e15f2e556f Gerrit-Change-Number: 19649 Gerrit-PatchSet: 4 Gerrit-Owner: Riza Suminto Gerrit-Reviewer: Abhishek Rawat Gerrit-Reviewer: Andrew Sherman Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Riza Suminto Gerrit-Reviewer: Wenzhe Zhou Gerrit-Comment-Date: Thu, 23 Mar 2023 20:45:32 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-12023: Skip resource checking on last executor group set
Riza Suminto has posted comments on this change. ( http://gerrit.cloudera.org:8080/19649 ) Change subject: IMPALA-12023: Skip resource checking on last executor group set .. Patch Set 3: Code-Review+2 Thank you Wenzhe and Andrew! Carry +2. -- To view, visit http://gerrit.cloudera.org:8080/19649 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I5848e4f67939d3dd2fb105c1ae4ca8e15f2e556f Gerrit-Change-Number: 19649 Gerrit-PatchSet: 3 Gerrit-Owner: Riza Suminto Gerrit-Reviewer: Abhishek Rawat Gerrit-Reviewer: Andrew Sherman Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Riza Suminto Gerrit-Reviewer: Wenzhe Zhou Gerrit-Comment-Date: Thu, 23 Mar 2023 20:31:46 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-12023: Skip resource checking on last executor group set
Wenzhe Zhou has posted comments on this change. ( http://gerrit.cloudera.org:8080/19649 ) Change subject: IMPALA-12023: Skip resource checking on last executor group set .. Patch Set 3: Code-Review+1 -- To view, visit http://gerrit.cloudera.org:8080/19649 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I5848e4f67939d3dd2fb105c1ae4ca8e15f2e556f Gerrit-Change-Number: 19649 Gerrit-PatchSet: 3 Gerrit-Owner: Riza Suminto Gerrit-Reviewer: Abhishek Rawat Gerrit-Reviewer: Andrew Sherman Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Riza Suminto Gerrit-Reviewer: Wenzhe Zhou Gerrit-Comment-Date: Thu, 23 Mar 2023 20:29:52 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-12023: Skip resource checking on last executor group set
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/19649 ) Change subject: IMPALA-12023: Skip resource checking on last executor group set .. Patch Set 4: Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/9175/ DRY_RUN=false -- To view, visit http://gerrit.cloudera.org:8080/19649 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I5848e4f67939d3dd2fb105c1ae4ca8e15f2e556f Gerrit-Change-Number: 19649 Gerrit-PatchSet: 4 Gerrit-Owner: Riza Suminto Gerrit-Reviewer: Abhishek Rawat Gerrit-Reviewer: Andrew Sherman Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Riza Suminto Gerrit-Reviewer: Wenzhe Zhou Gerrit-Comment-Date: Thu, 23 Mar 2023 20:32:24 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-12023: Skip resource checking on last executor group set
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/19649 ) Change subject: IMPALA-12023: Skip resource checking on last executor group set .. Patch Set 4: Code-Review+2 -- To view, visit http://gerrit.cloudera.org:8080/19649 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I5848e4f67939d3dd2fb105c1ae4ca8e15f2e556f Gerrit-Change-Number: 19649 Gerrit-PatchSet: 4 Gerrit-Owner: Riza Suminto Gerrit-Reviewer: Abhishek Rawat Gerrit-Reviewer: Andrew Sherman Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Riza Suminto Gerrit-Reviewer: Wenzhe Zhou Gerrit-Comment-Date: Thu, 23 Mar 2023 20:32:23 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-12023: Skip resource checking on last executor group set
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/19649 ) Change subject: IMPALA-12023: Skip resource checking on last executor group set .. Patch Set 2: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/12685/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/19649 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I5848e4f67939d3dd2fb105c1ae4ca8e15f2e556f Gerrit-Change-Number: 19649 Gerrit-PatchSet: 2 Gerrit-Owner: Riza Suminto Gerrit-Reviewer: Abhishek Rawat Gerrit-Reviewer: Andrew Sherman Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Riza Suminto Gerrit-Reviewer: Wenzhe Zhou Gerrit-Comment-Date: Thu, 23 Mar 2023 20:19:33 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-12023: Skip resource checking on last executor group set
Riza Suminto has posted comments on this change. ( http://gerrit.cloudera.org:8080/19649 ) Change subject: IMPALA-12023: Skip resource checking on last executor group set .. Patch Set 3: (5 comments) http://gerrit.cloudera.org:8080/#/c/19649/1//COMMIT_MSG Commit Message: http://gerrit.cloudera.org:8080/#/c/19649/1//COMMIT_MSG@12 PS1, Line 12: executor group > should be "executor group set" here and below Done http://gerrit.cloudera.org:8080/#/c/19649/1/be/src/util/backend-gflag-util.cc File be/src/util/backend-gflag-util.cc: http://gerrit.cloudera.org:8080/#/c/19649/1/be/src/util/backend-gflag-util.cc@225 PS1, Line 225: (skip_resource_checking_on_last_executor_group > define as skip_resource_checking_on_last_executor_group_set? Done http://gerrit.cloudera.org:8080/#/c/19649/1/be/src/util/backend-gflag-util.cc@227 PS1, Line 227: executor group > executor group set? Done http://gerrit.cloudera.org:8080/#/c/19649/1/be/src/util/backend-gflag-util.cc@229 PS1, Line 229: her g > group set Done http://gerrit.cloudera.org:8080/#/c/19649/1/fe/src/main/java/org/apache/impala/service/Frontend.java File fe/src/main/java/org/apache/impala/service/Frontend.java: http://gerrit.cloudera.org:8080/#/c/19649/1/fe/src/main/java/org/apache/impala/service/Frontend.java@2127 PS1, Line 2127: executor group > executor group set Done -- To view, visit http://gerrit.cloudera.org:8080/19649 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I5848e4f67939d3dd2fb105c1ae4ca8e15f2e556f Gerrit-Change-Number: 19649 Gerrit-PatchSet: 3 Gerrit-Owner: Riza Suminto Gerrit-Reviewer: Abhishek Rawat Gerrit-Reviewer: Andrew Sherman Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Riza Suminto Gerrit-Reviewer: Wenzhe Zhou Gerrit-Comment-Date: Thu, 23 Mar 2023 20:00:58 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-12023: Skip resource checking on last executor group set
Hello Andrew Sherman, Abhishek Rawat, Wenzhe Zhou, Impala Public Jenkins, I'd like you to reexamine a change. Please visit http://gerrit.cloudera.org:8080/19649 to look at the new patch set (#3). Change subject: IMPALA-12023: Skip resource checking on last executor group set .. IMPALA-12023: Skip resource checking on last executor group set This patch adds flag skip_resource_checking_on_last_executor_group_set. If this backend flag is set to true, memory and cpu resource checking will be skipped when a query is being planned against the last (largest) executor group set. Setting true will ensure that query will always get admitted into the last executor group set if it does not fit in any other group set. Testing - Tune test_query_cpu_count_divisor_fraction to run two test case: cpu within limit, and cpu outside limit. - Add test_no_skip_resource_checking Change-Id: I5848e4f67939d3dd2fb105c1ae4ca8e15f2e556f --- M be/src/util/backend-gflag-util.cc M common/thrift/BackendGflags.thrift M fe/src/main/java/org/apache/impala/service/BackendConfig.java M fe/src/main/java/org/apache/impala/service/Frontend.java M tests/custom_cluster/test_executor_groups.py 5 files changed, 46 insertions(+), 2 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/49/19649/3 -- To view, visit http://gerrit.cloudera.org:8080/19649 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: I5848e4f67939d3dd2fb105c1ae4ca8e15f2e556f Gerrit-Change-Number: 19649 Gerrit-PatchSet: 3 Gerrit-Owner: Riza Suminto Gerrit-Reviewer: Abhishek Rawat Gerrit-Reviewer: Andrew Sherman Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Riza Suminto Gerrit-Reviewer: Wenzhe Zhou
[Impala-ASF-CR] IMPALA-12023: Skip resource checking on last executor group set
Hello Andrew Sherman, Abhishek Rawat, Wenzhe Zhou, Impala Public Jenkins, I'd like you to reexamine a change. Please visit http://gerrit.cloudera.org:8080/19649 to look at the new patch set (#2). Change subject: IMPALA-12023: Skip resource checking on last executor group set .. IMPALA-12023: Skip resource checking on last executor group set This patch adds flag skip_resource_checking_on_last_executor_group. If this backend flag is set to true, memory and cpu resource checking will be skipped when a query is being planned against the last (largest) executor group. Setting true will ensure that query will always get admitted into last executor group if it does not fit in any other group. Testing - Tune test_query_cpu_count_divisor_fraction to run two test case: cpu within limit, and cpu outside limit. - Add test_no_skip_resource_checking Change-Id: I5848e4f67939d3dd2fb105c1ae4ca8e15f2e556f --- M be/src/util/backend-gflag-util.cc M common/thrift/BackendGflags.thrift M fe/src/main/java/org/apache/impala/service/BackendConfig.java M fe/src/main/java/org/apache/impala/service/Frontend.java M tests/custom_cluster/test_executor_groups.py 5 files changed, 46 insertions(+), 2 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/49/19649/2 -- To view, visit http://gerrit.cloudera.org:8080/19649 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: I5848e4f67939d3dd2fb105c1ae4ca8e15f2e556f Gerrit-Change-Number: 19649 Gerrit-PatchSet: 2 Gerrit-Owner: Riza Suminto Gerrit-Reviewer: Abhishek Rawat Gerrit-Reviewer: Andrew Sherman Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Riza Suminto Gerrit-Reviewer: Wenzhe Zhou
[Impala-ASF-CR] IMPALA-12015: enable healthz endpoint in admissiond webui
Wenzhe Zhou has posted comments on this change. ( http://gerrit.cloudera.org:8080/19647 ) Change subject: IMPALA-12015: enable healthz endpoint in admissiond webui .. Patch Set 1: (1 comment) http://gerrit.cloudera.org:8080/#/c/19647/1/be/src/service/impala-http-handler.cc File be/src/service/impala-http-handler.cc: http://gerrit.cloudera.org:8080/#/c/19647/1/be/src/service/impala-http-handler.cc@213 PS1, Line 213: if ((server_ != NULL && server_->IsHealthy()) || > I think AdmissionControlService is only applicable for standalone admission That's fine. -- To view, visit http://gerrit.cloudera.org:8080/19647 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Iffb5ee7a00992d1bdba1deddf181ae38aa206140 Gerrit-Change-Number: 19647 Gerrit-PatchSet: 1 Gerrit-Owner: Abhishek Rawat Gerrit-Reviewer: Abhishek Rawat Gerrit-Reviewer: Andrew Sherman Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Wenzhe Zhou Gerrit-Comment-Date: Thu, 23 Mar 2023 19:38:13 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-12023: Skip resource checking on last executor group set
Andrew Sherman has posted comments on this change. ( http://gerrit.cloudera.org:8080/19649 ) Change subject: IMPALA-12023: Skip resource checking on last executor group set .. Patch Set 1: Code-Review+1 -- To view, visit http://gerrit.cloudera.org:8080/19649 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I5848e4f67939d3dd2fb105c1ae4ca8e15f2e556f Gerrit-Change-Number: 19649 Gerrit-PatchSet: 1 Gerrit-Owner: Riza Suminto Gerrit-Reviewer: Abhishek Rawat Gerrit-Reviewer: Andrew Sherman Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Riza Suminto Gerrit-Reviewer: Wenzhe Zhou Gerrit-Comment-Date: Thu, 23 Mar 2023 19:38:14 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-12023: Skip resource checking on last executor group set
Wenzhe Zhou has posted comments on this change. ( http://gerrit.cloudera.org:8080/19649 ) Change subject: IMPALA-12023: Skip resource checking on last executor group set .. Patch Set 1: (5 comments) http://gerrit.cloudera.org:8080/#/c/19649/1//COMMIT_MSG Commit Message: http://gerrit.cloudera.org:8080/#/c/19649/1//COMMIT_MSG@12 PS1, Line 12: executor group should be "executor group set" here and below http://gerrit.cloudera.org:8080/#/c/19649/1/be/src/util/backend-gflag-util.cc File be/src/util/backend-gflag-util.cc: http://gerrit.cloudera.org:8080/#/c/19649/1/be/src/util/backend-gflag-util.cc@225 PS1, Line 225: (skip_resource_checking_on_last_executor_group define as skip_resource_checking_on_last_executor_group_set? http://gerrit.cloudera.org:8080/#/c/19649/1/be/src/util/backend-gflag-util.cc@227 PS1, Line 227: executor group executor group set? http://gerrit.cloudera.org:8080/#/c/19649/1/be/src/util/backend-gflag-util.cc@229 PS1, Line 229: group group set http://gerrit.cloudera.org:8080/#/c/19649/1/fe/src/main/java/org/apache/impala/service/Frontend.java File fe/src/main/java/org/apache/impala/service/Frontend.java: http://gerrit.cloudera.org:8080/#/c/19649/1/fe/src/main/java/org/apache/impala/service/Frontend.java@2127 PS1, Line 2127: executor group executor group set -- To view, visit http://gerrit.cloudera.org:8080/19649 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I5848e4f67939d3dd2fb105c1ae4ca8e15f2e556f Gerrit-Change-Number: 19649 Gerrit-PatchSet: 1 Gerrit-Owner: Riza Suminto Gerrit-Reviewer: Abhishek Rawat Gerrit-Reviewer: Andrew Sherman Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Riza Suminto Gerrit-Reviewer: Wenzhe Zhou Gerrit-Comment-Date: Thu, 23 Mar 2023 19:25:07 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-12015: enable healthz endpoint in admissiond webui
Abhishek Rawat has posted comments on this change. ( http://gerrit.cloudera.org:8080/19647 ) Change subject: IMPALA-12015: enable healthz endpoint in admissiond webui .. Patch Set 1: (1 comment) http://gerrit.cloudera.org:8080/#/c/19647/1/be/src/service/impala-http-handler.cc File be/src/service/impala-http-handler.cc: http://gerrit.cloudera.org:8080/#/c/19647/1/be/src/service/impala-http-handler.cc@213 PS1, Line 213: if ((server_ != NULL && server_->IsHealthy()) || > Both AdmissiondEnv and ExecEnv have AdmissionController member variable. Bu I think AdmissionControlService is only applicable for standalone admission controller. I could move the 'service_started' atomic to AdmissionControlService, but there is no way to get the AdmissionControlService from AdmissionController today. And I don't want to add unnecessary dependency. So we would still basically call AdmissiondEnv::GetInstance()->admission_control_service()->IsHealthy() -- To view, visit http://gerrit.cloudera.org:8080/19647 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Iffb5ee7a00992d1bdba1deddf181ae38aa206140 Gerrit-Change-Number: 19647 Gerrit-PatchSet: 1 Gerrit-Owner: Abhishek Rawat Gerrit-Reviewer: Abhishek Rawat Gerrit-Reviewer: Andrew Sherman Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Wenzhe Zhou Gerrit-Comment-Date: Thu, 23 Mar 2023 19:16:39 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-12023: Skip resource checking on last executor group set
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/19649 ) Change subject: IMPALA-12023: Skip resource checking on last executor group set .. Patch Set 1: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/12684/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/19649 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I5848e4f67939d3dd2fb105c1ae4ca8e15f2e556f Gerrit-Change-Number: 19649 Gerrit-PatchSet: 1 Gerrit-Owner: Riza Suminto Gerrit-Reviewer: Abhishek Rawat Gerrit-Reviewer: Andrew Sherman Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Riza Suminto Gerrit-Reviewer: Wenzhe Zhou Gerrit-Comment-Date: Thu, 23 Mar 2023 19:13:51 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-12023: Skip resource checking on last executor group set
Riza Suminto has uploaded this change for review. ( http://gerrit.cloudera.org:8080/19649 Change subject: IMPALA-12023: Skip resource checking on last executor group set .. IMPALA-12023: Skip resource checking on last executor group set This patch adds flag skip_resource_checking_on_last_executor_group. If this backend flag is set to true, memory and cpu resource checking will be skipped when a query is being planned against the last (largest) executor group. Setting true will ensure that query will always get admitted into last executor group if it does not fit in any other group. Testing - Tune test_query_cpu_count_divisor_fraction to run two test case: cpu within limit, and cpu outside limit. - Add test_no_skip_resource_checking Change-Id: I5848e4f67939d3dd2fb105c1ae4ca8e15f2e556f --- M be/src/util/backend-gflag-util.cc M common/thrift/BackendGflags.thrift M fe/src/main/java/org/apache/impala/service/BackendConfig.java M fe/src/main/java/org/apache/impala/service/Frontend.java M tests/custom_cluster/test_executor_groups.py 5 files changed, 46 insertions(+), 2 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/49/19649/1 -- To view, visit http://gerrit.cloudera.org:8080/19649 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newchange Gerrit-Change-Id: I5848e4f67939d3dd2fb105c1ae4ca8e15f2e556f Gerrit-Change-Number: 19649 Gerrit-PatchSet: 1 Gerrit-Owner: Riza Suminto
[Impala-ASF-CR] IMPALA-12015: enable healthz endpoint in admissiond webui
Wenzhe Zhou has posted comments on this change. ( http://gerrit.cloudera.org:8080/19647 ) Change subject: IMPALA-12015: enable healthz endpoint in admissiond webui .. Patch Set 1: (1 comment) http://gerrit.cloudera.org:8080/#/c/19647/1/be/src/service/impala-http-handler.cc File be/src/service/impala-http-handler.cc: http://gerrit.cloudera.org:8080/#/c/19647/1/be/src/service/impala-http-handler.cc@213 PS1, Line 213: if ((server_ != NULL && server_->IsHealthy()) || > server_ is set only for Impalads and is NULL for admissiond. Both AdmissiondEnv and ExecEnv have AdmissionController member variable. But AdmissionControlService is only defined in AdmissiondEnv. service_started_ should be moved to AdmissionControlService -- To view, visit http://gerrit.cloudera.org:8080/19647 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Iffb5ee7a00992d1bdba1deddf181ae38aa206140 Gerrit-Change-Number: 19647 Gerrit-PatchSet: 1 Gerrit-Owner: Abhishek Rawat Gerrit-Reviewer: Abhishek Rawat Gerrit-Reviewer: Andrew Sherman Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Wenzhe Zhou Gerrit-Comment-Date: Thu, 23 Mar 2023 18:44:09 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-12015: enable healthz endpoint in admissiond webui
Abhishek Rawat has posted comments on this change. ( http://gerrit.cloudera.org:8080/19647 ) Change subject: IMPALA-12015: enable healthz endpoint in admissiond webui .. Patch Set 1: (1 comment) http://gerrit.cloudera.org:8080/#/c/19647/1/be/src/service/impala-http-handler.cc File be/src/service/impala-http-handler.cc: http://gerrit.cloudera.org:8080/#/c/19647/1/be/src/service/impala-http-handler.cc@213 PS1, Line 213: if ((server_ != NULL && server_->IsHealthy()) || > I assume this means that server_ is null in AdmissionD? server_ is set only for Impalads and is NULL for admissiond. The AdmissiondEnv instance is the one which inits AdmissionController and starts all the necessary services: https://github.com/apache/impala/blob/master/be/src/scheduling/admissiond-env.cc#L81 https://github.com/apache/impala/blob/master/be/src/scheduling/admissiond-env.cc#L115 It's possible to move the 'service_ready' atomic to AdmissionController and AdmissiondEnv could set it. -- To view, visit http://gerrit.cloudera.org:8080/19647 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Iffb5ee7a00992d1bdba1deddf181ae38aa206140 Gerrit-Change-Number: 19647 Gerrit-PatchSet: 1 Gerrit-Owner: Abhishek Rawat Gerrit-Reviewer: Abhishek Rawat Gerrit-Reviewer: Andrew Sherman Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Wenzhe Zhou Gerrit-Comment-Date: Thu, 23 Mar 2023 18:35:09 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-12015: enable healthz endpoint in admissiond webui
Wenzhe Zhou has posted comments on this change. (
http://gerrit.cloudera.org:8080/19647 )
Change subject: IMPALA-12015: enable healthz endpoint in admissiond webui
..
Patch Set 1:
(2 comments)
http://gerrit.cloudera.org:8080/#/c/19647/1/be/src/scheduling/admissiond-env.h
File be/src/scheduling/admissiond-env.h:
http://gerrit.cloudera.org:8080/#/c/19647/1/be/src/scheduling/admissiond-env.h@87
PS1, Line 87: std::atomic_bool service_started_{false};
AdmissiondEnv analogous to ExecEnv for impalads. But service_started_ is not
defined in ExecEnv for impalads. Should we move it to AdmissionControlService ?
http://gerrit.cloudera.org:8080/#/c/19647/1/be/src/service/impala-http-handler.cc
File be/src/service/impala-http-handler.cc:
http://gerrit.cloudera.org:8080/#/c/19647/1/be/src/service/impala-http-handler.cc@213
PS1, Line 213: NULL
nit: nullptr
--
To view, visit http://gerrit.cloudera.org:8080/19647
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Iffb5ee7a00992d1bdba1deddf181ae38aa206140
Gerrit-Change-Number: 19647
Gerrit-PatchSet: 1
Gerrit-Owner: Abhishek Rawat
Gerrit-Reviewer: Andrew Sherman
Gerrit-Reviewer: Impala Public Jenkins
Gerrit-Reviewer: Wenzhe Zhou
Gerrit-Comment-Date: Thu, 23 Mar 2023 18:34:22 +
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-11966: Enable cache ozone file handles by default
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/19573 ) Change subject: IMPALA-11966: Enable cache_ozone_file_handles by default .. Patch Set 4: Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/9174/ DRY_RUN=true -- To view, visit http://gerrit.cloudera.org:8080/19573 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Icc66551f9b87af785a1c30b516ac39f4640638fe Gerrit-Change-Number: 19573 Gerrit-PatchSet: 4 Gerrit-Owner: Michael Smith Gerrit-Reviewer: David Rorke Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Joe McDonnell Gerrit-Reviewer: Michael Smith Gerrit-Comment-Date: Thu, 23 Mar 2023 18:15:43 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-12015: enable healthz endpoint in admissiond webui
Andrew Sherman has posted comments on this change. ( http://gerrit.cloudera.org:8080/19647 ) Change subject: IMPALA-12015: enable healthz endpoint in admissiond webui .. Patch Set 1: (1 comment) quick question http://gerrit.cloudera.org:8080/#/c/19647/1/be/src/service/impala-http-handler.cc File be/src/service/impala-http-handler.cc: http://gerrit.cloudera.org:8080/#/c/19647/1/be/src/service/impala-http-handler.cc@213 PS1, Line 213: if ((server_ != NULL && server_->IsHealthy()) || I assume this means that server_ is null in AdmissionD? Can we use admission_controller_ instead of AdmissiondEnv::GetInstance() ? -- To view, visit http://gerrit.cloudera.org:8080/19647 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Iffb5ee7a00992d1bdba1deddf181ae38aa206140 Gerrit-Change-Number: 19647 Gerrit-PatchSet: 1 Gerrit-Owner: Abhishek Rawat Gerrit-Reviewer: Andrew Sherman Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Wenzhe Zhou Gerrit-Comment-Date: Thu, 23 Mar 2023 18:03:07 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-12015: enable healthz endpoint in admissiond webui
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/19647 ) Change subject: IMPALA-12015: enable healthz endpoint in admissiond webui .. Patch Set 1: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/12683/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/19647 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Iffb5ee7a00992d1bdba1deddf181ae38aa206140 Gerrit-Change-Number: 19647 Gerrit-PatchSet: 1 Gerrit-Owner: Abhishek Rawat Gerrit-Reviewer: Andrew Sherman Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Wenzhe Zhou Gerrit-Comment-Date: Thu, 23 Mar 2023 16:54:59 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-12015: enable healthz endpoint in admissiond webui
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/19647 ) Change subject: IMPALA-12015: enable healthz endpoint in admissiond webui .. Patch Set 1: (2 comments) http://gerrit.cloudera.org:8080/#/c/19647/1/fe/src/test/java/org/apache/impala/customcluster/CustomClusterRunner.java File fe/src/test/java/org/apache/impala/customcluster/CustomClusterRunner.java: http://gerrit.cloudera.org:8080/#/c/19647/1/fe/src/test/java/org/apache/impala/customcluster/CustomClusterRunner.java@63 PS1, Line 63: impaladArgs, catalogdArgs, statestoredArgs, "", new HashMap(), ""); line too long (91 > 90) http://gerrit.cloudera.org:8080/#/c/19647/1/fe/src/test/java/org/apache/impala/customcluster/CustomClusterRunner.java@85 PS1, Line 85: String statestoredArgs, String admissiondArgs, Map env, String startArgs) line too long (95 > 90) -- To view, visit http://gerrit.cloudera.org:8080/19647 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Iffb5ee7a00992d1bdba1deddf181ae38aa206140 Gerrit-Change-Number: 19647 Gerrit-PatchSet: 1 Gerrit-Owner: Abhishek Rawat Gerrit-Reviewer: Andrew Sherman Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Wenzhe Zhou Gerrit-Comment-Date: Thu, 23 Mar 2023 16:35:09 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-12015: enable healthz endpoint in admissiond webui
Abhishek Rawat has uploaded this change for review. ( http://gerrit.cloudera.org:8080/19647 Change subject: IMPALA-12015: enable healthz endpoint in admissiond webui .. IMPALA-12015: enable healthz endpoint in admissiond webui Exposed '/healthz' endpoint in admissiond's webserver and metrics webserver. The handler for '/healthz' returns 'OK' if the statestore subscriber service and KRPC service have started on admissiond. Testing: - Manually tested '/healthz' on both admissiond webserver and admissiond metrics webserver - Updated existing 'LdapWebserverTest' to also include test for admissiond's metrics webserver Change-Id: Iffb5ee7a00992d1bdba1deddf181ae38aa206140 --- M be/src/scheduling/admissiond-env.cc M be/src/scheduling/admissiond-env.h M be/src/service/impala-http-handler.cc M fe/src/test/java/org/apache/impala/customcluster/CustomClusterRunner.java M fe/src/test/java/org/apache/impala/customcluster/LdapWebserverTest.java 5 files changed, 58 insertions(+), 28 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/47/19647/1 -- To view, visit http://gerrit.cloudera.org:8080/19647 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newchange Gerrit-Change-Id: Iffb5ee7a00992d1bdba1deddf181ae38aa206140 Gerrit-Change-Number: 19647 Gerrit-PatchSet: 1 Gerrit-Owner: Abhishek Rawat Gerrit-Reviewer: Andrew Sherman Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Wenzhe Zhou
[Impala-ASF-CR] IMPALA-11886: Data cache should support asynchronous writes
Joe McDonnell has submitted this change and it was merged. ( http://gerrit.cloudera.org:8080/19475 ) Change subject: IMPALA-11886: Data cache should support asynchronous writes .. IMPALA-11886: Data cache should support asynchronous writes This patch implements asynchronous writes to the data cache to improve scan performance when a cache miss happens. Previously, writes to the data cache are synchronous with hdfs file reads, and both are handled by remote hdfs IO threads. In other words, if a cache miss occurs, the IO thread needs to take additional responsibility for cache writes, which will lead to scan performance deterioration. This patch uses a thread pool for asynchronous writes, and the number of threads in the pool is determined by the new configuration 'data_cache_num_write_threads'. In asynchronous write mode, the IO thread only needs to copy data to the temporary buffer when storing data into the data cache. The additional memory consumption caused by temporary buffers can be limited, depending on the new configuration 'data_cache_write_buffer_limit'. Testing: - Add test cases for asynchronous data writing to the original DataCacheTest using different number of threads. - Add DataCacheTest,#OutOfWriteBufferLimit Used to test the limit of memory consumed by temporary buffers in the case of asynchronous writes - Add a timer to the MultiThreadedReadWrite function to get the average time of multithreaded writes. Here are some test cases and their time that differ significantly between synchronous and asynchronous: Test case| Policy | Sync/Async | write time in ms MultiThreadedNoMisses| LRU| Sync | 12.20 MultiThreadedNoMisses| LRU| Async | 20.74 MultiThreadedNoMisses| LIRS | Sync |9.42 MultiThreadedNoMisses| LIRS | Async | 16.75 MultiThreadedWithMisses | LRU| Sync | 510.87 MultiThreadedWithMisses | LRU| Async | 10.06 MultiThreadedWithMisses | LIRS | Sync | 1872.11 MultiThreadedWithMisses | LIRS | Async | 11.02 MultiPartitions | LRU| Sync |1.20 MultiPartitions | LRU| Async |5.23 MultiPartitions | LIRS | Sync |1.26 MultiPartitions | LIRS | Async |7.91 AccessTraceAnonymization | LRU| Sync | 1963.89 AccessTraceAnonymization | LRU| Sync | 2073.62 AccessTraceAnonymization | LRU| Async |9.43 AccessTraceAnonymization | LRU| Async | 13.13 AccessTraceAnonymization | LIRS | Sync | 1663.93 AccessTraceAnonymization | LIRS | Sync | 1501.86 AccessTraceAnonymization | LIRS | Async | 12.83 AccessTraceAnonymization | LIRS | Async | 12.74 Change-Id: I878f7486d485b6288de1a9145f49576b7155d312 Reviewed-on: http://gerrit.cloudera.org:8080/19475 Reviewed-by: Joe McDonnell Tested-by: Impala Public Jenkins --- M be/src/runtime/io/data-cache-test.cc M be/src/runtime/io/data-cache-trace.cc M be/src/runtime/io/data-cache.cc M be/src/runtime/io/data-cache.h M be/src/runtime/io/disk-io-mgr.cc M be/src/util/impalad-metrics.cc M be/src/util/impalad-metrics.h M bin/run-all-tests.sh M bin/start-impala-cluster.py M common/thrift/metrics.json 10 files changed, 427 insertions(+), 52 deletions(-) Approvals: Joe McDonnell: Looks good to me, approved Impala Public Jenkins: Verified -- To view, visit http://gerrit.cloudera.org:8080/19475 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: merged Gerrit-Change-Id: I878f7486d485b6288de1a9145f49576b7155d312 Gerrit-Change-Number: 19475 Gerrit-PatchSet: 13 Gerrit-Owner: Anonymous Coward <18770832...@163.com> Gerrit-Reviewer: Anonymous Coward <18770832...@163.com> Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Joe McDonnell
[Impala-ASF-CR] IMPALA-7942 (part 2): Add query hints for predicate selectivities
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/18023 ) Change subject: IMPALA-7942 (part 2): Add query hints for predicate selectivities .. Patch Set 22: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/12682/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/18023 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I2776b9bbd878b8a21d9c866b400140a454f59e1b Gerrit-Change-Number: 18023 Gerrit-PatchSet: 22 Gerrit-Owner: wangsheng Gerrit-Reviewer: Amogh Margoor Gerrit-Reviewer: Fucun Chu Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Qifan Chen Gerrit-Reviewer: Quanlong Huang Gerrit-Reviewer: Yifan Zhang Gerrit-Reviewer: Zoltan Borok-Nagy Gerrit-Reviewer: wangsheng Gerrit-Comment-Date: Thu, 23 Mar 2023 16:07:59 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-7942 (part 2): Add query hints for predicate selectivities
wangsheng has posted comments on this change. (
http://gerrit.cloudera.org:8080/18023 )
Change subject: IMPALA-7942 (part 2): Add query hints for predicate
selectivities
..
Patch Set 22:
(3 comments)
http://gerrit.cloudera.org:8080/#/c/18023/21/fe/src/main/java/org/apache/impala/analysis/Predicate.java
File fe/src/main/java/org/apache/impala/analysis/Predicate.java:
http://gerrit.cloudera.org:8080/#/c/18023/21/fe/src/main/java/org/apache/impala/analysis/Predicate.java@171
PS21, Line 171: public void setSelectivityHint(double selectivityHint) {
> Should we update hasValidSelectivityHint_ as well? Or is it enough to only
Done
Yes, we need to update 'hasValidSelectivityHint_' and 'selectivityHint_' at
same time. So I update this boolean flag
according to input selectivityHint.
http://gerrit.cloudera.org:8080/#/c/18023/21/fe/src/main/java/org/apache/impala/rewrite/BetweenToCompoundRule.java
File fe/src/main/java/org/apache/impala/rewrite/BetweenToCompoundRule.java:
http://gerrit.cloudera.org:8080/#/c/18023/21/fe/src/main/java/org/apache/impala/rewrite/BetweenToCompoundRule.java@68
PS21, Line 68: // Selectivity hint value of this new CompoundPredicate not
been set, so inherited
> Could you add a comment for why we always set the selectivity hint regradle
Done
A new created Predicate's selectivity hint is always false, unless we use hint
in sql. So whether 'BetweenPredicate' been set selectivity hint in sql or not,
we can use this to replace the new created.
http://gerrit.cloudera.org:8080/#/c/18023/21/testdata/workloads/functional-planner/queries/PlannerTest/predicate-selectivity-hint.test
File
testdata/workloads/functional-planner/queries/PlannerTest/predicate-selectivity-hint.test:
http://gerrit.cloudera.org:8080/#/c/18023/21/testdata/workloads/functional-planner/queries/PlannerTest/predicate-selectivity-hint.test@130
PS21, Line 130: select * from tpch.lineitem
> missing the SELECT part here, which causes the test failure
Done
--
To view, visit http://gerrit.cloudera.org:8080/18023
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I2776b9bbd878b8a21d9c866b400140a454f59e1b
Gerrit-Change-Number: 18023
Gerrit-PatchSet: 22
Gerrit-Owner: wangsheng
Gerrit-Reviewer: Amogh Margoor
Gerrit-Reviewer: Fucun Chu
Gerrit-Reviewer: Impala Public Jenkins
Gerrit-Reviewer: Qifan Chen
Gerrit-Reviewer: Quanlong Huang
Gerrit-Reviewer: Yifan Zhang
Gerrit-Reviewer: Zoltan Borok-Nagy
Gerrit-Reviewer: wangsheng
Gerrit-Comment-Date: Thu, 23 Mar 2023 15:47:35 +
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-7942 (part 2): Add query hints for predicate selectivities
wangsheng has uploaded a new patch set (#22). ( http://gerrit.cloudera.org:8080/18023 ) Change subject: IMPALA-7942 (part 2): Add query hints for predicate selectivities .. IMPALA-7942 (part 2): Add query hints for predicate selectivities Currently, Impala only uses simple estimation to compute selectivity for some predicates, and this may lead to worse query plan due to CBO. Hence, we add new hints to reduce such errors. Maybe in the future, we can use histograms to get more precise query plan. This patch adds another query hints: 'SELECTIVITY', we can use this hint to original selectivity computing. Format like this: select col from t where (a=1) /* +SELECTIVITY(0.5) */; Besides, this hint is also valid for compound predicate like this: select col from t where (a=1 and b=2) /* +SELECTIVITY(0.5) */; But pay attention, if we want to use 'SELECTIVITY' hint for predicate, we need to wrap the predicate by braket, even for single binary predicate. Testing: - Added new fe tests in 'PlannerTest' - Added new fe tests in 'AnalyzeStmtsTest' for negative cases Change-Id: I2776b9bbd878b8a21d9c866b400140a454f59e1b --- M fe/src/main/cup/sql-parser.cup M fe/src/main/java/org/apache/impala/analysis/BinaryPredicate.java M fe/src/main/java/org/apache/impala/analysis/CompoundPredicate.java M fe/src/main/java/org/apache/impala/analysis/Expr.java M fe/src/main/java/org/apache/impala/analysis/InPredicate.java M fe/src/main/java/org/apache/impala/analysis/IsNullPredicate.java M fe/src/main/java/org/apache/impala/analysis/Predicate.java M fe/src/main/java/org/apache/impala/rewrite/BetweenToCompoundRule.java M fe/src/main/jflex/sql-scanner.flex M fe/src/test/java/org/apache/impala/analysis/AnalyzeStmtsTest.java M fe/src/test/java/org/apache/impala/planner/PlannerTest.java A testdata/workloads/functional-planner/queries/PlannerTest/predicate-selectivity-hint.test 12 files changed, 427 insertions(+), 5 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/23/18023/22 -- To view, visit http://gerrit.cloudera.org:8080/18023 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: I2776b9bbd878b8a21d9c866b400140a454f59e1b Gerrit-Change-Number: 18023 Gerrit-PatchSet: 22 Gerrit-Owner: wangsheng Gerrit-Reviewer: Amogh Margoor Gerrit-Reviewer: Fucun Chu Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Qifan Chen Gerrit-Reviewer: Quanlong Huang Gerrit-Reviewer: Yifan Zhang Gerrit-Reviewer: Zoltan Borok-Nagy Gerrit-Reviewer: wangsheng
[Impala-ASF-CR] IMPALA-7942 (part 2): Add query hints for predicate selectivities
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/18023 ) Change subject: IMPALA-7942 (part 2): Add query hints for predicate selectivities .. Patch Set 23: Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/9173/ DRY_RUN=false -- To view, visit http://gerrit.cloudera.org:8080/18023 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I2776b9bbd878b8a21d9c866b400140a454f59e1b Gerrit-Change-Number: 18023 Gerrit-PatchSet: 23 Gerrit-Owner: wangsheng Gerrit-Reviewer: Amogh Margoor Gerrit-Reviewer: Fucun Chu Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Qifan Chen Gerrit-Reviewer: Quanlong Huang Gerrit-Reviewer: Yifan Zhang Gerrit-Reviewer: Zoltan Borok-Nagy Gerrit-Reviewer: wangsheng Gerrit-Comment-Date: Thu, 23 Mar 2023 15:48:25 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-11925: Added a new api add write notification log in batch in the MetastoreServiceHandler class
Sai Hemanth Gantasala has posted comments on this change. ( http://gerrit.cloudera.org:8080/19508 ) Change subject: IMPALA-11925: Added a new api add_write_notification_log_in_batch in the MetastoreServiceHandler class .. Patch Set 5: (3 comments) http://gerrit.cloudera.org:8080/#/c/19508/2//COMMIT_MSG Commit Message: http://gerrit.cloudera.org:8080/#/c/19508/2//COMMIT_MSG@9 PS2, Line 9: add_write_notification_log_ > add_write_notification_log_in_batch seems to be the batched version of add_ That's a good suggestion. I don't we would use this API at all in Impala. Since MetastoreServiceHandler class extends AbstractThriftHiveMetastore class and every time a new API is introduced in AbstractThriftHiveMetastore class we'll have to define it in MetastoreServiceHandler class. http://gerrit.cloudera.org:8080/#/c/19508/4//COMMIT_MSG Commit Message: http://gerrit.cloudera.org:8080/#/c/19508/4//COMMIT_MSG@9 PS4, Line 9: In the MetastoreServiceHandler class, the 'add_write_notification_log_ > This is now 42, the limit is 72. Ack http://gerrit.cloudera.org:8080/#/c/19508/4//COMMIT_MSG@19 PS4, Line 19: Change-Id: Ia9a35557c2ed79ed0276c4a418b5292fc6dd6194 > Line too long (74 vs 72). Ack -- To view, visit http://gerrit.cloudera.org:8080/19508 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ia9a35557c2ed79ed0276c4a418b5292fc6dd6194 Gerrit-Change-Number: 19508 Gerrit-PatchSet: 5 Gerrit-Owner: Sai Hemanth Gantasala Gerrit-Reviewer: Aman Sinha Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Daniel Becker Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: John Sherman Gerrit-Reviewer: Quanlong Huang Gerrit-Reviewer: Sai Hemanth Gantasala Gerrit-Comment-Date: Thu, 23 Mar 2023 14:48:01 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-11925: Added a new api add write notification log in batch in the MetastoreServiceHandler class
Hello Quanlong Huang, Aman Sinha, Daniel Becker, Csaba Ringhofer, Impala Public Jenkins, John Sherman, I'd like you to reexamine a change. Please visit http://gerrit.cloudera.org:8080/19508 to look at the new patch set (#5). Change subject: IMPALA-11925: Added a new api add_write_notification_log_in_batch in the MetastoreServiceHandler class .. IMPALA-11925: Added a new api add_write_notification_log_in_batch in the MetastoreServiceHandler class In the MetastoreServiceHandler class, the 'add_write_notification_log_ in_batch' API implementation is missing which is causing test failures. This needs to be fixed by adding implementation of the 'add_write_ notification_log_in_batch' api in the MetastoreServiceHandler class Testing: Added a test in the 'test_hms_apis()' to verify that the 'add_write_ notification_log_in_batch' api in HMS is reachable from impala. The same test can be used in the future to test newly added apis in HMS. Change-Id: Ia9a35557c2ed79ed0276c4a418b5292fc6dd6194 --- M fe/src/main/java/org/apache/impala/catalog/metastore/MetastoreServiceHandler.java M tests/custom_cluster/test_metastore_service.py 2 files changed, 24 insertions(+), 4 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/08/19508/5 -- To view, visit http://gerrit.cloudera.org:8080/19508 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: Ia9a35557c2ed79ed0276c4a418b5292fc6dd6194 Gerrit-Change-Number: 19508 Gerrit-PatchSet: 5 Gerrit-Owner: Sai Hemanth Gantasala Gerrit-Reviewer: Aman Sinha Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Daniel Becker Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: John Sherman Gerrit-Reviewer: Quanlong Huang Gerrit-Reviewer: Sai Hemanth Gantasala
[Impala-ASF-CR] IMPALA-11970: Add query timeline display to Impala WebUI
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/19583 ) Change subject: IMPALA-11970: Add query timeline display to Impala WebUI .. Patch Set 5: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/12681/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/19583 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I8b5826107af0f5a7fe306cb986a875cff261d9db Gerrit-Change-Number: 19583 Gerrit-PatchSet: 5 Gerrit-Owner: Kurt Deschler Gerrit-Reviewer: David Rorke Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Kurt Deschler Gerrit-Reviewer: Quanlong Huang Gerrit-Reviewer: Riza Suminto Gerrit-Reviewer: Wenzhe Zhou Gerrit-Comment-Date: Thu, 23 Mar 2023 14:44:53 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-11970: Add query timeline display to Impala WebUI
Hello Quanlong Huang, Riza Suminto, David Rorke, Wenzhe Zhou, Impala Public Jenkins, I'd like you to reexamine a change. Please visit http://gerrit.cloudera.org:8080/19583 to look at the new patch set (#5). Change subject: IMPALA-11970: Add query timeline display to Impala WebUI .. IMPALA-11970: Add query timeline display to Impala WebUI This change adds a chart to the Impala WebUI for displaying query fragments and plan node timings. The chart list execution timings in a compact form with the plan/fragment tree shown on the left side and a Gantt chart with bars for each execution phase on the right side. Plan node labels are color-coded by fragment. The node list will scroll for large plans so that the scale and legend always stay visible. Exchange nodes have dotted boxes that show when rows are sent from the corresponding senders below. Additionally, 'X' and 'O' symbols are used to node where join builds arrive. By default, fragments are printed with nodes grouped within each fragment for easier timing analysis. A checkbox is provided that changes the ordering so that nodes are printed in plan order similar to the Summary tab. The length of each colored phase interval depicts the longest instance time while instances that complete earlier are shown as grey lines within the phase interval. On the server side, the query profile handler has been modified to return the profile JSON as a document element instead of document content as it is for JSON profile download. This allows the profile to be accessed in full using scripts in templates. Dense profiles will not render correctly as they currently do not contain the necessary timestamps. This limitation will be addressed at a later time with corresponding changes to the dense profile. Example output is attached to the IMPALA-11970 ticket Testing: Manual validation of rendering using TPC-H and TPC-DS queries Will look into adding tests to test_web_pages.py and test_observability.py in a subsequent patch Change-Id: I8b5826107af0f5a7fe306cb986a875cff261d9db --- M be/src/service/impala-http-handler.cc M be/src/service/impala-http-handler.h M be/src/service/impala-server.cc M be/src/util/runtime-profile.cc M www/query_detail_tabs.tmpl A www/query_timeline.tmpl 6 files changed, 524 insertions(+), 8 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/83/19583/5 -- To view, visit http://gerrit.cloudera.org:8080/19583 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: I8b5826107af0f5a7fe306cb986a875cff261d9db Gerrit-Change-Number: 19583 Gerrit-PatchSet: 5 Gerrit-Owner: Kurt Deschler Gerrit-Reviewer: David Rorke Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Kurt Deschler Gerrit-Reviewer: Quanlong Huang Gerrit-Reviewer: Riza Suminto Gerrit-Reviewer: Wenzhe Zhou
[Impala-ASF-CR] IMPALA-8731: Balance queries across multiple executor groups
Riza Suminto has posted comments on this change. ( http://gerrit.cloudera.org:8080/19630 ) Change subject: IMPALA-8731: Balance queries across multiple executor groups .. Patch Set 5: (1 comment) This looks OK to me. I just have 1 ask. http://gerrit.cloudera.org:8080/#/c/19630/5/tests/custom_cluster/test_executor_groups.py File tests/custom_cluster/test_executor_groups.py: http://gerrit.cloudera.org:8080/#/c/19630/5/tests/custom_cluster/test_executor_groups.py@914 PS5, Line 914: balance_queries_across_executor_groups=true Can you add another test where if balance_queries_across_executor_groups=false, both query will run in group1? -- To view, visit http://gerrit.cloudera.org:8080/19630 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I25e851fb57c1d820c25cef5316f4ed800e4c6ac5 Gerrit-Change-Number: 19630 Gerrit-PatchSet: 5 Gerrit-Owner: Yifan Zhang Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Riza Suminto Gerrit-Comment-Date: Thu, 23 Mar 2023 13:20:42 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-12017: Skip memory and cpu limit check if REQUEST POOL is set
Riza Suminto has submitted this change and it was merged. ( http://gerrit.cloudera.org:8080/19645 ) Change subject: IMPALA-12017: Skip memory and cpu limit check if REQUEST_POOL is set .. IMPALA-12017: Skip memory and cpu limit check if REQUEST_POOL is set Memory and cpu limit checking in executor group selection (Frontend.java) should be skipped if REQUEST_POOL query option is set. Setting REQUEST_POOL means user is specifying pool to run the query regardless of memory and cpu limit. Testing: - Add test cases in test_query_cpu_count_divisor_default Change-Id: I14bf7fe71e2dda1099651b3edf62480e1fdbf845 Reviewed-on: http://gerrit.cloudera.org:8080/19645 Reviewed-by: Wenzhe Zhou Tested-by: Impala Public Jenkins Reviewed-by: Riza Suminto --- M fe/src/main/java/org/apache/impala/service/Frontend.java M tests/custom_cluster/test_executor_groups.py 2 files changed, 75 insertions(+), 20 deletions(-) Approvals: Wenzhe Zhou: Looks good to me, but someone else must approve Impala Public Jenkins: Verified Riza Suminto: Looks good to me, approved -- To view, visit http://gerrit.cloudera.org:8080/19645 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: merged Gerrit-Change-Id: I14bf7fe71e2dda1099651b3edf62480e1fdbf845 Gerrit-Change-Number: 19645 Gerrit-PatchSet: 7 Gerrit-Owner: Riza Suminto Gerrit-Reviewer: Abhishek Rawat Gerrit-Reviewer: Andrew Sherman Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Riza Suminto Gerrit-Reviewer: Wenzhe Zhou
[Impala-ASF-CR] IMPALA-12005: Describe executor group set selection in query profile
Riza Suminto has submitted this change and it was merged. ( http://gerrit.cloudera.org:8080/19628 ) Change subject: IMPALA-12005: Describe executor group set selection in query profile .. IMPALA-12005: Describe executor group set selection in query profile This patch adds new profile counters under the Frontend profile node to describe executor group set selection during query planning. It modifies FrontendProfile.java to allow one level of TRuntimeProfileNode nesting under the Frontend profile node. This makes it possible to group profile counters specific to each executor group set in consideration. "fragment-costs" hint is renamed to "segment-costs". A new "cpu-comparison-result" hint is added after "segment-costs" to help navigate how cpu sizing decision is made. This patch also adds some function overloading in runtime-profile.cc to hide TotalTime and InactiveTotalTime that is meaningless for anything under the Frontend profile node. Additional context also added into AnalysisException threw when none of the executor group sets fits the query requirement. This is how the Frontend profile node looks like after running TestExecutorGroups::test_query_cpu_count_divisor_fraction Frontend: Referenced Tables: tpcds_parquet.store_sales - CpuCountDivisor: 0.20 - ExecutorGroupsConsidered: 3 (3) Executor group 1 (root.tiny): Verdict: not enough cpu cores - CpuAsk: 15 (15) - CpuMax: 2 (2) - EffectiveParallelism: 3 (3) - MemoryAsk: 36.83 MB (38617088) - MemoryMax: 64.00 MB (67108864) Executor group 2 (root.small): Verdict: not enough cpu cores - CpuAsk: 25 (25) - CpuMax: 16 (16) - EffectiveParallelism: 5 (5) - MemoryAsk: 36.83 MB (38624004) - MemoryMax: 64.00 MB (67108864) Executor group 3 (root.large): Verdict: Match - CpuAsk: 35 (35) - CpuMax: 192 (192) - EffectiveParallelism: 7 (7) - MemoryAsk: 36.84 MB (38633570) - MemoryMax: 8388608.00 GB (9007199254740992) Testing: - Pass core tests Change-Id: I6c0ac7f5216d631e4439fe97702e21e06d2eda8a Reviewed-on: http://gerrit.cloudera.org:8080/19628 Reviewed-by: Riza Suminto Tested-by: Riza Suminto --- M be/src/service/client-request-state.cc M be/src/service/client-request-state.h M be/src/service/impala-server.cc M be/src/util/runtime-profile.cc M be/src/util/runtime-profile.h M common/thrift/Frontend.thrift M fe/src/main/java/org/apache/impala/planner/PlanFragment.java M fe/src/main/java/org/apache/impala/planner/Planner.java M fe/src/main/java/org/apache/impala/service/Frontend.java M fe/src/main/java/org/apache/impala/service/FrontendProfile.java M testdata/workloads/functional-planner/queries/PlannerTest/tpcds-processing-cost.test M tests/custom_cluster/test_executor_groups.py 12 files changed, 668 insertions(+), 474 deletions(-) Approvals: Riza Suminto: Looks good to me, approved; Verified -- To view, visit http://gerrit.cloudera.org:8080/19628 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: merged Gerrit-Change-Id: I6c0ac7f5216d631e4439fe97702e21e06d2eda8a Gerrit-Change-Number: 19628 Gerrit-PatchSet: 7 Gerrit-Owner: Riza Suminto Gerrit-Reviewer: Abhishek Rawat Gerrit-Reviewer: David Rorke Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Quanlong Huang Gerrit-Reviewer: Riza Suminto Gerrit-Reviewer: Wenzhe Zhou
[Impala-ASF-CR] IMPALA-12005: Describe executor group set selection in query profile
Riza Suminto has posted comments on this change. ( http://gerrit.cloudera.org:8080/19628 ) Change subject: IMPALA-12005: Describe executor group set selection in query profile .. Patch Set 6: Verified+1 Code-Review+2 Carry votes from tip of changes. -- To view, visit http://gerrit.cloudera.org:8080/19628 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I6c0ac7f5216d631e4439fe97702e21e06d2eda8a Gerrit-Change-Number: 19628 Gerrit-PatchSet: 6 Gerrit-Owner: Riza Suminto Gerrit-Reviewer: Abhishek Rawat Gerrit-Reviewer: David Rorke Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Quanlong Huang Gerrit-Reviewer: Riza Suminto Gerrit-Reviewer: Wenzhe Zhou Gerrit-Comment-Date: Thu, 23 Mar 2023 12:47:58 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-12017: Skip memory and cpu limit check if REQUEST POOL is set
Riza Suminto has posted comments on this change. ( http://gerrit.cloudera.org:8080/19645 ) Change subject: IMPALA-12017: Skip memory and cpu limit check if REQUEST_POOL is set .. Patch Set 6: Code-Review+2 Carry +2 -- To view, visit http://gerrit.cloudera.org:8080/19645 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I14bf7fe71e2dda1099651b3edf62480e1fdbf845 Gerrit-Change-Number: 19645 Gerrit-PatchSet: 6 Gerrit-Owner: Riza Suminto Gerrit-Reviewer: Abhishek Rawat Gerrit-Reviewer: Andrew Sherman Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Riza Suminto Gerrit-Reviewer: Wenzhe Zhou Gerrit-Comment-Date: Thu, 23 Mar 2023 12:47:28 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-11908: Parser change for Iceberg metadata querying
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/19483 ) Change subject: IMPALA-11908: Parser change for Iceberg metadata querying .. Patch Set 9: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/12680/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/19483 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I0b5db884b5f3fecbd132fcb2c2cbd6c622ff965b Gerrit-Change-Number: 19483 Gerrit-PatchSet: 9 Gerrit-Owner: Tamas Mate Gerrit-Reviewer: Anonymous Coward Gerrit-Reviewer: Daniel Becker Gerrit-Reviewer: Gabor Kaszab Gerrit-Reviewer: Gergely Fürnstáhl Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Noemi Pap-Takacs Gerrit-Reviewer: Tamas Mate Gerrit-Reviewer: Zoltan Borok-Nagy Gerrit-Comment-Date: Thu, 23 Mar 2023 10:10:22 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-11908: Parser change for Iceberg metadata querying
Daniel Becker has posted comments on this change. ( http://gerrit.cloudera.org:8080/19483 ) Change subject: IMPALA-11908: Parser change for Iceberg metadata querying .. Patch Set 9: (1 comment) http://gerrit.cloudera.org:8080/#/c/19483/6/fe/src/main/java/org/apache/impala/analysis/Analyzer.java File fe/src/main/java/org/apache/impala/analysis/Analyzer.java: http://gerrit.cloudera.org:8080/#/c/19483/6/fe/src/main/java/org/apache/impala/analysis/Analyzer.java@3341 PS6, Line 3341: addMetadataVirtualTable > Thanks for catching this. I did not change it, I added a new condition to t You could mention in the doc comment that tblRefPath is expected to be an Iceberg metadata table. -- To view, visit http://gerrit.cloudera.org:8080/19483 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I0b5db884b5f3fecbd132fcb2c2cbd6c622ff965b Gerrit-Change-Number: 19483 Gerrit-PatchSet: 9 Gerrit-Owner: Tamas Mate Gerrit-Reviewer: Anonymous Coward Gerrit-Reviewer: Daniel Becker Gerrit-Reviewer: Gabor Kaszab Gerrit-Reviewer: Gergely Fürnstáhl Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Noemi Pap-Takacs Gerrit-Reviewer: Tamas Mate Gerrit-Reviewer: Zoltan Borok-Nagy Gerrit-Comment-Date: Thu, 23 Mar 2023 09:55:31 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-11908: Parser change for Iceberg metadata querying
Tamas Mate has posted comments on this change. ( http://gerrit.cloudera.org:8080/19483 ) Change subject: IMPALA-11908: Parser change for Iceberg metadata querying .. Patch Set 9: Hi everyone, thank you for the reviews so far. This patch got two +1s so far, it is a reasonably big change, so I would like to wait for a +2. Anyone would like to and has time to go through it again? -- To view, visit http://gerrit.cloudera.org:8080/19483 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I0b5db884b5f3fecbd132fcb2c2cbd6c622ff965b Gerrit-Change-Number: 19483 Gerrit-PatchSet: 9 Gerrit-Owner: Tamas Mate Gerrit-Reviewer: Anonymous Coward Gerrit-Reviewer: Daniel Becker Gerrit-Reviewer: Gabor Kaszab Gerrit-Reviewer: Gergely Fürnstáhl Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Noemi Pap-Takacs Gerrit-Reviewer: Tamas Mate Gerrit-Reviewer: Zoltan Borok-Nagy Gerrit-Comment-Date: Thu, 23 Mar 2023 09:52:54 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-11908: Parser change for Iceberg metadata querying
Tamas Mate has uploaded a new patch set (#9). ( http://gerrit.cloudera.org:8080/19483 ) Change subject: IMPALA-11908: Parser change for Iceberg metadata querying .. IMPALA-11908: Parser change for Iceberg metadata querying This change extends parsing table references with Iceberg metadata tables. The TableName class has been extended with an extra vTbl field which is filled when a virtual table reference is suspected. This additional field helps to keep the real table in the statement table cache next to the virtual table, which should be loaded so Iceberg metadata tables can be created. Iceberg provides a rich API to query metadata, these Iceberg API tables are accessible through the MetadataTableUtils class. Using these table schemas it is possible to create an Impala table that can be queried later on. Querying a metadata table at this point is expected to throw a NotImplementedException. Testing: - Added E2E test to test it for some tables. Change-Id: I0b5db884b5f3fecbd132fcb2c2cbd6c622ff965b --- M fe/src/main/java/org/apache/impala/analysis/Analyzer.java M fe/src/main/java/org/apache/impala/analysis/FromClause.java A fe/src/main/java/org/apache/impala/analysis/IcebergMetadataTableRef.java M fe/src/main/java/org/apache/impala/analysis/Path.java M fe/src/main/java/org/apache/impala/analysis/TableName.java M fe/src/main/java/org/apache/impala/catalog/IcebergTable.java M fe/src/main/java/org/apache/impala/catalog/IcebergTimeTravelTable.java M fe/src/main/java/org/apache/impala/catalog/iceberg/IcebergCtasTarget.java A fe/src/main/java/org/apache/impala/catalog/iceberg/IcebergMetadataTable.java M fe/src/main/java/org/apache/impala/planner/SingleNodePlanner.java M fe/src/main/java/org/apache/impala/service/IcebergCatalogOpExecutor.java M fe/src/main/java/org/apache/impala/util/IcebergSchemaConverter.java A testdata/workloads/functional-query/queries/QueryTest/iceberg-metadata-tables.test M tests/query_test/test_iceberg.py 14 files changed, 414 insertions(+), 37 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/83/19483/9 -- To view, visit http://gerrit.cloudera.org:8080/19483 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: I0b5db884b5f3fecbd132fcb2c2cbd6c622ff965b Gerrit-Change-Number: 19483 Gerrit-PatchSet: 9 Gerrit-Owner: Tamas Mate Gerrit-Reviewer: Anonymous Coward Gerrit-Reviewer: Daniel Becker Gerrit-Reviewer: Gabor Kaszab Gerrit-Reviewer: Gergely Fürnstáhl Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Noemi Pap-Takacs Gerrit-Reviewer: Tamas Mate Gerrit-Reviewer: Zoltan Borok-Nagy
[Impala-ASF-CR] IMPALA-12017: Skip memory and cpu limit check if REQUEST POOL is set
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/19645 ) Change subject: IMPALA-12017: Skip memory and cpu limit check if REQUEST_POOL is set .. Patch Set 6: Verified+1 -- To view, visit http://gerrit.cloudera.org:8080/19645 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I14bf7fe71e2dda1099651b3edf62480e1fdbf845 Gerrit-Change-Number: 19645 Gerrit-PatchSet: 6 Gerrit-Owner: Riza Suminto Gerrit-Reviewer: Abhishek Rawat Gerrit-Reviewer: Andrew Sherman Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Riza Suminto Gerrit-Reviewer: Wenzhe Zhou Gerrit-Comment-Date: Thu, 23 Mar 2023 09:22:28 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-8731: Balance queries across multiple executor groups
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/19630 ) Change subject: IMPALA-8731: Balance queries across multiple executor groups .. Patch Set 5: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/12679/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/19630 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I25e851fb57c1d820c25cef5316f4ed800e4c6ac5 Gerrit-Change-Number: 19630 Gerrit-PatchSet: 5 Gerrit-Owner: Yifan Zhang Gerrit-Reviewer: Impala Public Jenkins Gerrit-Comment-Date: Thu, 23 Mar 2023 08:40:48 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-11886: Data cache should support asynchronous writes
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/19475 ) Change subject: IMPALA-11886: Data cache should support asynchronous writes .. Patch Set 12: Verified+1 -- To view, visit http://gerrit.cloudera.org:8080/19475 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I878f7486d485b6288de1a9145f49576b7155d312 Gerrit-Change-Number: 19475 Gerrit-PatchSet: 12 Gerrit-Owner: Anonymous Coward <18770832...@163.com> Gerrit-Reviewer: Anonymous Coward <18770832...@163.com> Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Joe McDonnell Gerrit-Comment-Date: Thu, 23 Mar 2023 08:32:11 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-12008: Fix incorrect end time in DML profiles
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/19644 ) Change subject: IMPALA-12008: Fix incorrect end time in DML profiles .. Patch Set 3: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/12678/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/19644 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I9c5dc92c2f8576ceed374d447c0ac05022a2dee6 Gerrit-Change-Number: 19644 Gerrit-PatchSet: 3 Gerrit-Owner: Quanlong Huang Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Quanlong Huang Gerrit-Comment-Date: Thu, 23 Mar 2023 08:21:21 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-12008: Fix incorrect end time in DML profiles
Quanlong Huang has posted comments on this change. ( http://gerrit.cloudera.org:8080/19644 ) Change subject: IMPALA-12008: Fix incorrect end time in DML profiles .. Patch Set 3: (1 comment) http://gerrit.cloudera.org:8080/#/c/19644/2/tests/util/parse_util.py File tests/util/parse_util.py: http://gerrit.cloudera.org:8080/#/c/19644/2/tests/util/parse_util.py@104 PS2, Line 104: > flake8: E302 expected 2 blank lines, found 1 Done -- To view, visit http://gerrit.cloudera.org:8080/19644 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I9c5dc92c2f8576ceed374d447c0ac05022a2dee6 Gerrit-Change-Number: 19644 Gerrit-PatchSet: 3 Gerrit-Owner: Quanlong Huang Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Quanlong Huang Gerrit-Comment-Date: Thu, 23 Mar 2023 07:59:20 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-12008: Fix incorrect end time in DML profiles
Hello Impala Public Jenkins, I'd like you to reexamine a change. Please visit http://gerrit.cloudera.org:8080/19644 to look at the new patch set (#3). Change subject: IMPALA-12008: Fix incorrect end time in DML profiles .. IMPALA-12008: Fix incorrect end time in DML profiles The end time in DML profiles is incorrect that it's actually the time when admission control resources are released. This is correct for normal queries. But for DMLs, coordinator still needs to invoke the updateCatalog RPC of catalogd to finalize the HMS update. The end time should be set after the request finished. This patch fixes the DML end time by not setting it after the admission control resources are released. Instead, it's set after ClientRequestState::WaitInternal() finishes, which makes sure the updateCatalog RPC has finished. For testing, this patch also adds a new debug action in catalogd (catalogd_insert_finish_delay) to inject delays in updateCatalog. Tests - Added e2e test to verify the end time of a DML profile Change-Id: I9c5dc92c2f8576ceed374d447c0ac05022a2dee6 --- M be/src/runtime/coordinator.cc M be/src/service/client-request-state.cc M common/thrift/CatalogService.thrift M fe/src/main/java/org/apache/impala/service/CatalogOpExecutor.java M fe/src/main/java/org/apache/impala/util/DebugUtils.java M tests/query_test/test_observability.py M tests/util/parse_util.py 7 files changed, 63 insertions(+), 4 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/44/19644/3 -- To view, visit http://gerrit.cloudera.org:8080/19644 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: I9c5dc92c2f8576ceed374d447c0ac05022a2dee6 Gerrit-Change-Number: 19644 Gerrit-PatchSet: 3 Gerrit-Owner: Quanlong Huang Gerrit-Reviewer: Impala Public Jenkins
