[Impala-ASF-CR] IMPALA-10036: schedule unpartitioned fragments on random executor
Hello Bikramjeet Vig, I'd like you to reexamine a change. Please visit http://gerrit.cloudera.org:8080/16272 to look at the new patch set (#6). Change subject: IMPALA-10036: schedule unpartitioned fragments on random executor .. IMPALA-10036: schedule unpartitioned fragments on random executor Some plans have unpartitioned fragments that are not the coordinator fragment. It would be best to avoid scheduling these on a coordinator, both to reduce coordinator load and to allow using the coordinator mem estimates for these queries. Testing: Extended the dedicated coordinator admission control test with a query with an unpartitioned fragment. The test failed on master with dedicated coordinator estimates. Manually ran the repro with TPC-DS Q23, which is a more complex query with unpartitioned fragments, and confirmed that could run with low coordinator mem limits. Ran exhaustive tests. Change-Id: I1eee78f24b048968f2fdf91f0da95b78552f8d33 --- M be/src/scheduling/schedule-state.cc M be/src/scheduling/schedule-state.h M be/src/scheduling/scheduler-test-util.cc M be/src/scheduling/scheduler.cc M be/src/scheduling/scheduler.h M tests/custom_cluster/test_admission_controller.py 6 files changed, 104 insertions(+), 45 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/72/16272/6 -- To view, visit http://gerrit.cloudera.org:8080/16272 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: I1eee78f24b048968f2fdf91f0da95b78552f8d33 Gerrit-Change-Number: 16272 Gerrit-PatchSet: 6 Gerrit-Owner: Tim Armstrong Gerrit-Reviewer: Bikramjeet Vig
[Impala-ASF-CR] IMPALA-10036: schedule unpartitioned fragments on random executor
Tim Armstrong has uploaded a new patch set (#4). ( http://gerrit.cloudera.org:8080/16272 ) Change subject: IMPALA-10036: schedule unpartitioned fragments on random executor .. IMPALA-10036: schedule unpartitioned fragments on random executor Some plans have unpartitioned fragments that are not the coordinator fragment. It would be best to avoid scheduling these on a coordinator, both to reduce coordinator load and to allow using the coordinator mem estimates for these queries. Testing: Extended the dedicated coordinator admission control test with a query with an unpartitioned fragment. The test failed on master with dedicated coordinator estimates. Manually ran the repro with TPC-DS Q23, which is a more complex query with unpartitioned fragments, and confirmed that could run with low coordinator mem limits. Change-Id: I1eee78f24b048968f2fdf91f0da95b78552f8d33 --- M be/src/scheduling/schedule-state.cc M be/src/scheduling/schedule-state.h M be/src/scheduling/scheduler-test-util.cc M be/src/scheduling/scheduler.cc M be/src/scheduling/scheduler.h M tests/custom_cluster/test_admission_controller.py 6 files changed, 104 insertions(+), 45 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/72/16272/4 -- To view, visit http://gerrit.cloudera.org:8080/16272 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: I1eee78f24b048968f2fdf91f0da95b78552f8d33 Gerrit-Change-Number: 16272 Gerrit-PatchSet: 4 Gerrit-Owner: Tim Armstrong Gerrit-Reviewer: Bikramjeet Vig
[Impala-ASF-CR] IMPALA-10036: schedule unpartitioned fragments on random executor
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/16272 ) Change subject: IMPALA-10036: schedule unpartitioned fragments on random executor .. Patch Set 6: Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/6211/ DRY_RUN=false -- To view, visit http://gerrit.cloudera.org:8080/16272 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I1eee78f24b048968f2fdf91f0da95b78552f8d33 Gerrit-Change-Number: 16272 Gerrit-PatchSet: 6 Gerrit-Owner: Tim Armstrong Gerrit-Reviewer: Bikramjeet Vig Gerrit-Reviewer: Impala Public Jenkins Gerrit-Comment-Date: Sun, 02 Aug 2020 18:14:12 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-10036: schedule unpartitioned fragments on random executor
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/16272 ) Change subject: IMPALA-10036: schedule unpartitioned fragments on random executor .. Patch Set 4: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/6764/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/16272 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I1eee78f24b048968f2fdf91f0da95b78552f8d33 Gerrit-Change-Number: 16272 Gerrit-PatchSet: 4 Gerrit-Owner: Tim Armstrong Gerrit-Reviewer: Bikramjeet Vig Gerrit-Reviewer: Impala Public Jenkins Gerrit-Comment-Date: Sun, 02 Aug 2020 18:38:22 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-10036: schedule unpartitioned fragments on random executor
Tim Armstrong has posted comments on this change. ( http://gerrit.cloudera.org:8080/16272 ) Change subject: IMPALA-10036: schedule unpartitioned fragments on random executor .. Patch Set 6: Code-Review+2 Carry +2 from Bikram -- To view, visit http://gerrit.cloudera.org:8080/16272 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I1eee78f24b048968f2fdf91f0da95b78552f8d33 Gerrit-Change-Number: 16272 Gerrit-PatchSet: 6 Gerrit-Owner: Tim Armstrong Gerrit-Reviewer: Bikramjeet Vig Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Sun, 02 Aug 2020 19:26:37 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-10036: schedule unpartitioned fragments on random executor
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/16272 ) Change subject: IMPALA-10036: schedule unpartitioned fragments on random executor .. Patch Set 6: Verified-1 Build failed: https://jenkins.impala.io/job/gerrit-verify-dryrun/6211/ -- To view, visit http://gerrit.cloudera.org:8080/16272 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I1eee78f24b048968f2fdf91f0da95b78552f8d33 Gerrit-Change-Number: 16272 Gerrit-PatchSet: 6 Gerrit-Owner: Tim Armstrong Gerrit-Reviewer: Bikramjeet Vig Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Sun, 02 Aug 2020 22:13:13 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-10036: schedule unpartitioned fragments on random executor
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/16272 ) Change subject: IMPALA-10036: schedule unpartitioned fragments on random executor .. Patch Set 6: Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/6212/ DRY_RUN=false -- To view, visit http://gerrit.cloudera.org:8080/16272 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I1eee78f24b048968f2fdf91f0da95b78552f8d33 Gerrit-Change-Number: 16272 Gerrit-PatchSet: 6 Gerrit-Owner: Tim Armstrong Gerrit-Reviewer: Bikramjeet Vig Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Sun, 02 Aug 2020 22:55:56 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-9990: Support SET OWNER for Kudu tables
Fang-Yu Rao has uploaded this change for review. ( http://gerrit.cloudera.org:8080/16273 Change subject: IMPALA-9990: Support SET OWNER for Kudu tables .. IMPALA-9990: Support SET OWNER for Kudu tables KUDU-3090 adds the support for table ownership and exposes the API's of setting owner on creating and altering tables, which allows Impala to also pass to Kudu the new owner of the Kudu table for the ALTER TABLE SET OWNER statement. Specifically, based on the API of AlterTableOptions#setOwner(), this patch stores the ownership information of the Kudu table in the corresponding instance of AlterTableOptions, which will be passed to Kudu via a KuduClient. Testing: - Added an FE test in AnalyzeKuduDDLTest.java to verify the statement could be correctly analyzed. - Added an E2E test in kudu_alter.test to verify the statement could be correctly executed. - Verified that the patch could pass the exhaustive tests in the DEBUG mode. Change-Id: I29d641efc8db314964bc5ee9828a86d4a44ae95c --- M fe/src/main/java/org/apache/impala/service/CatalogOpExecutor.java M fe/src/main/java/org/apache/impala/service/KuduCatalogOpExecutor.java M fe/src/test/java/org/apache/impala/analysis/AnalyzeKuduDDLTest.java M testdata/workloads/functional-query/queries/QueryTest/kudu_alter.test M testdata/workloads/functional-query/queries/QueryTest/kudu_hms_alter.test 5 files changed, 34 insertions(+), 1 deletion(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/73/16273/1 -- To view, visit http://gerrit.cloudera.org:8080/16273 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newchange Gerrit-Change-Id: I29d641efc8db314964bc5ee9828a86d4a44ae95c Gerrit-Change-Number: 16273 Gerrit-PatchSet: 1 Gerrit-Owner: Fang-Yu Rao Gerrit-Reviewer: Andrew Wong Gerrit-Reviewer: Attila Bukor Gerrit-Reviewer: Fang-Yu Rao Gerrit-Reviewer: Tim Armstrong
[Impala-ASF-CR] IMPALA-9990: Support SET OWNER for Kudu tables
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/16273 ) Change subject: IMPALA-9990: Support SET OWNER for Kudu tables .. Patch Set 1: Build Failed https://jenkins.impala.io/job/gerrit-code-review-checks/6765/ : Initial code review checks failed. See linked job for details on the failure. -- To view, visit http://gerrit.cloudera.org:8080/16273 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I29d641efc8db314964bc5ee9828a86d4a44ae95c Gerrit-Change-Number: 16273 Gerrit-PatchSet: 1 Gerrit-Owner: Fang-Yu Rao Gerrit-Reviewer: Andrew Wong Gerrit-Reviewer: Attila Bukor Gerrit-Reviewer: Fang-Yu Rao Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Sun, 02 Aug 2020 23:26:54 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-9990: Support SET OWNER for Kudu tables
Fang-Yu Rao has posted comments on this change. ( http://gerrit.cloudera.org:8080/16273 ) Change subject: IMPALA-9990: Support SET OWNER for Kudu tables .. Patch Set 1: > Patch Set 1: > > Build Failed > > https://jenkins.impala.io/job/gerrit-code-review-checks/6765/ : Initial code > review checks failed. See linked job for details on the failure. The build job failed because AlterTableOptions#setOwner() cannot be resolved (https://jenkins.impala.io/job/gerrit-code-review-checks/6765/artifact/https%3A%5E%5Ejenkins.impala.io%5Ejob%5Eubuntu-16.04-build-only%5E11967%5E.log). It is a bit strange since I thought after the patch for IMPALA-9903 (https://github.com/apache/impala/commit/da2999afd9ddc45d35141649d17db507e03ee9bf) has been merged, we will have a newer version of Kudu that contains the API of AlterTableOptions#setOwner(). In what follows I also provide the related error message. 23:26:41 [ERROR] Failed to execute goal org.apache.maven.plugins:maven-compiler-plugin:3.3:compile (default-compile) on project impala-frontend: Compilation failure 23:26:41 [ERROR] /home/ubuntu/tmp.7Xt03iIWrh/fe/src/main/java/org/apache/impala/service/KuduCatalogOpExecutor.java:[548,22] cannot find symbol 23:26:41 [ERROR] symbol: method setOwner(java.lang.String) 23:26:41 [ERROR] location: variable alterTableOptions of type org.apache.kudu.client.AlterTableOptions -- To view, visit http://gerrit.cloudera.org:8080/16273 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I29d641efc8db314964bc5ee9828a86d4a44ae95c Gerrit-Change-Number: 16273 Gerrit-PatchSet: 1 Gerrit-Owner: Fang-Yu Rao Gerrit-Reviewer: Andrew Wong Gerrit-Reviewer: Attila Bukor Gerrit-Reviewer: Fang-Yu Rao Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Sun, 02 Aug 2020 23:35:03 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-9990: Support SET OWNER for Kudu tables
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/16273 ) Change subject: IMPALA-9990: Support SET OWNER for Kudu tables .. Patch Set 1: Build Failed https://jenkins.impala.io/job/gerrit-code-review-checks/6766/ : Initial code review checks failed. See linked job for details on the failure. -- To view, visit http://gerrit.cloudera.org:8080/16273 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I29d641efc8db314964bc5ee9828a86d4a44ae95c Gerrit-Change-Number: 16273 Gerrit-PatchSet: 1 Gerrit-Owner: Fang-Yu Rao Gerrit-Reviewer: Andrew Wong Gerrit-Reviewer: Attila Bukor Gerrit-Reviewer: Fang-Yu Rao Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Sun, 02 Aug 2020 23:58:54 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-10036: schedule unpartitioned fragments on random executor
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/16272 ) Change subject: IMPALA-10036: schedule unpartitioned fragments on random executor .. Patch Set 6: Build failed: https://jenkins.impala.io/job/gerrit-verify-dryrun/6212/ -- To view, visit http://gerrit.cloudera.org:8080/16272 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I1eee78f24b048968f2fdf91f0da95b78552f8d33 Gerrit-Change-Number: 16272 Gerrit-PatchSet: 6 Gerrit-Owner: Tim Armstrong Gerrit-Reviewer: Bikramjeet Vig Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Mon, 03 Aug 2020 04:06:25 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-10036: schedule unpartitioned fragments on random executor
Tim Armstrong has removed a vote on this change. Change subject: IMPALA-10036: schedule unpartitioned fragments on random executor .. Removed Verified-1 by Impala Public Jenkins -- To view, visit http://gerrit.cloudera.org:8080/16272 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: deleteVote Gerrit-Change-Id: I1eee78f24b048968f2fdf91f0da95b78552f8d33 Gerrit-Change-Number: 16272 Gerrit-PatchSet: 6 Gerrit-Owner: Tim Armstrong Gerrit-Reviewer: Bikramjeet Vig Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tim Armstrong
[Impala-ASF-CR] IMPALA-10036: schedule unpartitioned fragments on random executor
Tim Armstrong has posted comments on this change. ( http://gerrit.cloudera.org:8080/16272 ) Change subject: IMPALA-10036: schedule unpartitioned fragments on random executor .. Patch Set 6: Verified+1 I hit https://issues.apache.org/jira/browse/IMPALA-9923 on a different job on each of the two precommit runs, so I am going to manually verify (i.e. all the tests passed at least once between the two runs, and the only flakiness was that issue). -- To view, visit http://gerrit.cloudera.org:8080/16272 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I1eee78f24b048968f2fdf91f0da95b78552f8d33 Gerrit-Change-Number: 16272 Gerrit-PatchSet: 6 Gerrit-Owner: Tim Armstrong Gerrit-Reviewer: Bikramjeet Vig Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Mon, 03 Aug 2020 04:30:08 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-10036: schedule unpartitioned fragments on random executor
Tim Armstrong has submitted this change and it was merged. ( http://gerrit.cloudera.org:8080/16272 ) Change subject: IMPALA-10036: schedule unpartitioned fragments on random executor .. IMPALA-10036: schedule unpartitioned fragments on random executor Some plans have unpartitioned fragments that are not the coordinator fragment. It would be best to avoid scheduling these on a coordinator, both to reduce coordinator load and to allow using the coordinator mem estimates for these queries. Testing: Extended the dedicated coordinator admission control test with a query with an unpartitioned fragment. The test failed on master with dedicated coordinator estimates. Manually ran the repro with TPC-DS Q23, which is a more complex query with unpartitioned fragments, and confirmed that could run with low coordinator mem limits. Ran exhaustive tests. Change-Id: I1eee78f24b048968f2fdf91f0da95b78552f8d33 Reviewed-on: http://gerrit.cloudera.org:8080/16272 Reviewed-by: Tim Armstrong Tested-by: Tim Armstrong --- M be/src/scheduling/schedule-state.cc M be/src/scheduling/schedule-state.h M be/src/scheduling/scheduler-test-util.cc M be/src/scheduling/scheduler.cc M be/src/scheduling/scheduler.h M tests/custom_cluster/test_admission_controller.py 6 files changed, 104 insertions(+), 45 deletions(-) Approvals: Tim Armstrong: Looks good to me, approved; Verified -- To view, visit http://gerrit.cloudera.org:8080/16272 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: merged Gerrit-Change-Id: I1eee78f24b048968f2fdf91f0da95b78552f8d33 Gerrit-Change-Number: 16272 Gerrit-PatchSet: 7 Gerrit-Owner: Tim Armstrong Gerrit-Reviewer: Bikramjeet Vig Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tim Armstrong
[Impala-ASF-CR] IMPALA-9990: Support SET OWNER for Kudu tables
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/16273 ) Change subject: IMPALA-9990: Support SET OWNER for Kudu tables .. Patch Set 1: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/6767/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/16273 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I29d641efc8db314964bc5ee9828a86d4a44ae95c Gerrit-Change-Number: 16273 Gerrit-PatchSet: 1 Gerrit-Owner: Fang-Yu Rao Gerrit-Reviewer: Andrew Wong Gerrit-Reviewer: Attila Bukor Gerrit-Reviewer: Fang-Yu Rao Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Mon, 03 Aug 2020 05:58:36 + Gerrit-HasComments: No
