[Impala-ASF-CR] IMPALA-8999: make union scheduling work with mt dop
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/14384 ) Change subject: IMPALA-8999: make union scheduling work with mt_dop .. Patch Set 14: Verified+1 -- To view, visit http://gerrit.cloudera.org:8080/14384 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I0d2e9c86b530da3053e49d42b837dca0b1348ff2 Gerrit-Change-Number: 14384 Gerrit-PatchSet: 14 Gerrit-Owner: Tim Armstrong Gerrit-Reviewer: Andrew Sherman Gerrit-Reviewer: Bikramjeet Vig Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Thu, 24 Oct 2019 04:41:57 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-8999: make union scheduling work with mt dop
Impala Public Jenkins has submitted this change and it was merged. ( http://gerrit.cloudera.org:8080/14384 ) Change subject: IMPALA-8999: make union scheduling work with mt_dop .. IMPALA-8999: make union scheduling work with mt_dop This change unifies mt_dop scheduling between the union and scan cases. Testing: Manually checked that fragments with unions get parallelised to the correct degree, both as a result of scans within the fragment and input fragments. Extend TestMtDopAdmissionSlots (renamed to TestMtDopScheduling) to confirm that queries that were not parallelised before are now parallelised. These tests verify the number of instances of each operator using the ExecSummary embedded in the profile. Change-Id: I0d2e9c86b530da3053e49d42b837dca0b1348ff2 Reviewed-on: http://gerrit.cloudera.org:8080/14384 Reviewed-by: Impala Public Jenkins Tested-by: Impala Public Jenkins --- M be/src/scheduling/query-schedule.cc M be/src/scheduling/scheduler-test.cc M be/src/scheduling/scheduler.cc M be/src/scheduling/scheduler.h M fe/src/main/java/org/apache/impala/planner/UnionNode.java D testdata/workloads/functional-query/queries/QueryTest/mt-dop-parquet-admission-slots.test A testdata/workloads/functional-query/queries/QueryTest/mt-dop-parquet-scheduling.test M tests/query_test/test_mt_dop.py 8 files changed, 382 insertions(+), 219 deletions(-) Approvals: Impala Public Jenkins: Looks good to me, approved; Verified -- To view, visit http://gerrit.cloudera.org:8080/14384 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: merged Gerrit-Change-Id: I0d2e9c86b530da3053e49d42b837dca0b1348ff2 Gerrit-Change-Number: 14384 Gerrit-PatchSet: 15 Gerrit-Owner: Tim Armstrong Gerrit-Reviewer: Andrew Sherman Gerrit-Reviewer: Bikramjeet Vig Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tim Armstrong
[Impala-ASF-CR] IMPALA-8999: make union scheduling work with mt dop
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/14384 ) Change subject: IMPALA-8999: make union scheduling work with mt_dop .. Patch Set 14: Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/5129/ DRY_RUN=false -- To view, visit http://gerrit.cloudera.org:8080/14384 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I0d2e9c86b530da3053e49d42b837dca0b1348ff2 Gerrit-Change-Number: 14384 Gerrit-PatchSet: 14 Gerrit-Owner: Tim Armstrong Gerrit-Reviewer: Andrew Sherman Gerrit-Reviewer: Bikramjeet Vig Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Thu, 24 Oct 2019 00:23:23 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-8999: make union scheduling work with mt dop
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/14384 ) Change subject: IMPALA-8999: make union scheduling work with mt_dop .. Patch Set 14: Code-Review+2 -- To view, visit http://gerrit.cloudera.org:8080/14384 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I0d2e9c86b530da3053e49d42b837dca0b1348ff2 Gerrit-Change-Number: 14384 Gerrit-PatchSet: 14 Gerrit-Owner: Tim Armstrong Gerrit-Reviewer: Andrew Sherman Gerrit-Reviewer: Bikramjeet Vig Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Thu, 24 Oct 2019 00:23:22 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-8999: make union scheduling work with mt dop
Bikramjeet Vig has posted comments on this change. ( http://gerrit.cloudera.org:8080/14384 ) Change subject: IMPALA-8999: make union scheduling work with mt_dop .. Patch Set 13: Code-Review+2 -- To view, visit http://gerrit.cloudera.org:8080/14384 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I0d2e9c86b530da3053e49d42b837dca0b1348ff2 Gerrit-Change-Number: 14384 Gerrit-PatchSet: 13 Gerrit-Owner: Tim Armstrong Gerrit-Reviewer: Andrew Sherman Gerrit-Reviewer: Bikramjeet Vig Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Wed, 23 Oct 2019 23:37:14 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-8999: make union scheduling work with mt dop
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/14384 ) Change subject: IMPALA-8999: make union scheduling work with mt_dop .. Patch Set 13: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/4863/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/14384 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I0d2e9c86b530da3053e49d42b837dca0b1348ff2 Gerrit-Change-Number: 14384 Gerrit-PatchSet: 13 Gerrit-Owner: Tim Armstrong Gerrit-Reviewer: Andrew Sherman Gerrit-Reviewer: Bikramjeet Vig Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Wed, 23 Oct 2019 22:46:16 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-8999: make union scheduling work with mt dop
Tim Armstrong has posted comments on this change. ( http://gerrit.cloudera.org:8080/14384 ) Change subject: IMPALA-8999: make union scheduling work with mt_dop .. Patch Set 12: (3 comments) http://gerrit.cloudera.org:8080/#/c/14384/12/be/src/scheduling/scheduler.cc File be/src/scheduling/scheduler.cc: http://gerrit.cloudera.org:8080/#/c/14384/12/be/src/scheduling/scheduler.cc@338 PS12, Line 338: for the scan range > nit: remove? Done http://gerrit.cloudera.org:8080/#/c/14384/12/be/src/scheduling/scheduler.cc@344 PS12, Line 344: Note that this takes into account all of the input fragments, : // not just the leftmost because we expect unions to be symmetrical for purposes of : // planning, unlike joins. > nit: how about: "Note that step 1 is modified to run on fragments with unio Done http://gerrit.cloudera.org:8080/#/c/14384/12/be/src/scheduling/scheduler.cc@392 PS12, Line 392: vector scan_hosts; > nit: maybe add a Dcheck(scan_node_ids.size() == 1 || has_union) so that we Done -- To view, visit http://gerrit.cloudera.org:8080/14384 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I0d2e9c86b530da3053e49d42b837dca0b1348ff2 Gerrit-Change-Number: 14384 Gerrit-PatchSet: 12 Gerrit-Owner: Tim Armstrong Gerrit-Reviewer: Andrew Sherman Gerrit-Reviewer: Bikramjeet Vig Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Wed, 23 Oct 2019 22:01:44 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-8999: make union scheduling work with mt dop
Hello Andrew Sherman, Bikramjeet Vig, Impala Public Jenkins, I'd like you to reexamine a change. Please visit http://gerrit.cloudera.org:8080/14384 to look at the new patch set (#13). Change subject: IMPALA-8999: make union scheduling work with mt_dop .. IMPALA-8999: make union scheduling work with mt_dop This change unifies mt_dop scheduling between the union and scan cases. Testing: Manually checked that fragments with unions get parallelised to the correct degree, both as a result of scans within the fragment and input fragments. Extend TestMtDopAdmissionSlots (renamed to TestMtDopScheduling) to confirm that queries that were not parallelised before are now parallelised. These tests verify the number of instances of each operator using the ExecSummary embedded in the profile. Change-Id: I0d2e9c86b530da3053e49d42b837dca0b1348ff2 --- M be/src/scheduling/query-schedule.cc M be/src/scheduling/scheduler-test.cc M be/src/scheduling/scheduler.cc M be/src/scheduling/scheduler.h M fe/src/main/java/org/apache/impala/planner/UnionNode.java D testdata/workloads/functional-query/queries/QueryTest/mt-dop-parquet-admission-slots.test A testdata/workloads/functional-query/queries/QueryTest/mt-dop-parquet-scheduling.test M tests/query_test/test_mt_dop.py 8 files changed, 382 insertions(+), 219 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/84/14384/13 -- To view, visit http://gerrit.cloudera.org:8080/14384 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: I0d2e9c86b530da3053e49d42b837dca0b1348ff2 Gerrit-Change-Number: 14384 Gerrit-PatchSet: 13 Gerrit-Owner: Tim Armstrong Gerrit-Reviewer: Andrew Sherman Gerrit-Reviewer: Bikramjeet Vig Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tim Armstrong
[Impala-ASF-CR] IMPALA-8999: make union scheduling work with mt dop
Bikramjeet Vig has posted comments on this change. ( http://gerrit.cloudera.org:8080/14384 ) Change subject: IMPALA-8999: make union scheduling work with mt_dop .. Patch Set 12: (4 comments) http://gerrit.cloudera.org:8080/#/c/14384/12/be/src/scheduling/scheduler.cc File be/src/scheduling/scheduler.cc: http://gerrit.cloudera.org:8080/#/c/14384/12/be/src/scheduling/scheduler.cc@338 PS12, Line 338: for the scan range nit: remove? http://gerrit.cloudera.org:8080/#/c/14384/12/be/src/scheduling/scheduler.cc@344 PS12, Line 344: Note that this takes into account all of the input fragments, : // not just the leftmost because we expect unions to be symmetrical for purposes of : // planning, unlike joins. nit: how about: "Note that step 1 is modified to run on fragments with union nodes, by considering all input fragments and not just the leftmost because we expect unions to be symmetrical for purposes of planning, unlike joins." http://gerrit.cloudera.org:8080/#/c/14384/12/be/src/scheduling/scheduler.cc@392 PS12, Line 392: vector scan_hosts; nit: maybe add a Dcheck(scan_node_ids.size() == 1 || has_union) so that we make sure this methods gets another look whenever this changes. http://gerrit.cloudera.org:8080/#/c/14384/11/testdata/workloads/functional-query/queries/QueryTest/mt-dop-parquet-scheduling.test File testdata/workloads/functional-query/queries/QueryTest/mt-dop-parquet-scheduling.test: http://gerrit.cloudera.org:8080/#/c/14384/11/testdata/workloads/functional-query/queries/QueryTest/mt-dop-parquet-scheduling.test@133 PS11, Line 133: > I think this is covered by the tests on l32 and l52 - alltypes has 24 files Got it, thanks for the explanation. -- To view, visit http://gerrit.cloudera.org:8080/14384 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I0d2e9c86b530da3053e49d42b837dca0b1348ff2 Gerrit-Change-Number: 14384 Gerrit-PatchSet: 12 Gerrit-Owner: Tim Armstrong Gerrit-Reviewer: Andrew Sherman Gerrit-Reviewer: Bikramjeet Vig Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Mon, 21 Oct 2019 20:42:40 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-8999: make union scheduling work with mt dop
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/14384 ) Change subject: IMPALA-8999: make union scheduling work with mt_dop .. Patch Set 12: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/4836/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/14384 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I0d2e9c86b530da3053e49d42b837dca0b1348ff2 Gerrit-Change-Number: 14384 Gerrit-PatchSet: 12 Gerrit-Owner: Tim Armstrong Gerrit-Reviewer: Andrew Sherman Gerrit-Reviewer: Bikramjeet Vig Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Sat, 19 Oct 2019 20:55:55 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-8999: make union scheduling work with mt dop
Tim Armstrong has posted comments on this change. ( http://gerrit.cloudera.org:8080/14384 ) Change subject: IMPALA-8999: make union scheduling work with mt_dop .. Patch Set 11: (9 comments) Thanks for the feedback, this should make it a bit less cryptic. http://gerrit.cloudera.org:8080/#/c/14384/11/be/src/scheduling/scheduler.h File be/src/scheduling/scheduler.h: http://gerrit.cloudera.org:8080/#/c/14384/11/be/src/scheduling/scheduler.h@381 PS11, Line 381: For HDFS, this attempts to load balance among instances by computing the average : /// number of bytes per instances and then in a single pass assigning scan ranges to : /// each instance to roughly meet that average. > nit: looks like we forgot to update this in the commit for IMPALA-9015 nice catch http://gerrit.cloudera.org:8080/#/c/14384/11/be/src/scheduling/scheduler.cc File be/src/scheduling/scheduler.cc: http://gerrit.cloudera.org:8080/#/c/14384/11/be/src/scheduling/scheduler.cc@291 PS11, Line 291: one or more scan nodes > for the cases with non-union fragments, we are now considering the parallel We don't actually have plans like this at the moment. The operators that have multiple inputs are joins, which always have an exchange on the RHS, unions, and subplans, which never have a scan on the RHS. That could potentially change, but that's the current state of things. This makes me realise that case 2 is incorrectly described, there's no "or more". http://gerrit.cloudera.org:8080/#/c/14384/11/be/src/scheduling/scheduler.cc@332 PS11, Line 332: Instance selection for an interior fragment > what does instance selection and interior fragment mean here? elaborated a bit http://gerrit.cloudera.org:8080/#/c/14384/11/be/src/scheduling/scheduler.cc@360 PS11, Line 360: hosts > nit: instances_per_host Done http://gerrit.cloudera.org:8080/#/c/14384/11/be/src/scheduling/scheduler.cc@363 PS11, Line 363: only factor in > nit: got confused with this, read it as "only factor... in" vs the intended Done http://gerrit.cloudera.org:8080/#/c/14384/11/be/src/scheduling/scheduler.cc@368 PS11, Line 368: input_fragment_hosts > nit: input_fragment_instances_per_host Done http://gerrit.cloudera.org:8080/#/c/14384/11/fe/src/main/java/org/apache/impala/planner/UnionNode.java File fe/src/main/java/org/apache/impala/planner/UnionNode.java: http://gerrit.cloudera.org:8080/#/c/14384/11/fe/src/main/java/org/apache/impala/planner/UnionNode.java@122 PS11, Line 122: Union fragments are scheduled on the union of hosts that the child fragments run : // on. > nit: Union fragments are scheduled on the union of hosts of all scans in th Done http://gerrit.cloudera.org:8080/#/c/14384/11/testdata/workloads/functional-query/queries/QueryTest/mt-dop-parquet-scheduling.test File testdata/workloads/functional-query/queries/QueryTest/mt-dop-parquet-scheduling.test: http://gerrit.cloudera.org:8080/#/c/14384/11/testdata/workloads/functional-query/queries/QueryTest/mt-dop-parquet-scheduling.test@52 PS11, Line 52: it > nit: if Done http://gerrit.cloudera.org:8080/#/c/14384/11/testdata/workloads/functional-query/queries/QueryTest/mt-dop-parquet-scheduling.test@133 PS11, Line 133: > add a test for a union fragment where the scan ranges for scan node are mor I think this is covered by the tests on l32 and l52 - alltypes has 24 files, so at least 8 ranges per backend. I updated the comments to be specific about that. -- To view, visit http://gerrit.cloudera.org:8080/14384 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I0d2e9c86b530da3053e49d42b837dca0b1348ff2 Gerrit-Change-Number: 14384 Gerrit-PatchSet: 11 Gerrit-Owner: Tim Armstrong Gerrit-Reviewer: Andrew Sherman Gerrit-Reviewer: Bikramjeet Vig Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Sat, 19 Oct 2019 20:11:40 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-8999: make union scheduling work with mt dop
Hello Andrew Sherman, Bikramjeet Vig, Impala Public Jenkins, I'd like you to reexamine a change. Please visit http://gerrit.cloudera.org:8080/14384 to look at the new patch set (#12). Change subject: IMPALA-8999: make union scheduling work with mt_dop .. IMPALA-8999: make union scheduling work with mt_dop This change unifies mt_dop scheduling between the union and scan cases. Testing: Manually checked that fragments with unions get parallelised to the correct degree, both as a result of scans within the fragment and input fragments. Extend TestMtDopAdmissionSlots (renamed to TestMtDopScheduling) to confirm that queries that were not parallelised before are now parallelised. These tests verify the number of instances of each operator using the ExecSummary embedded in the profile. Change-Id: I0d2e9c86b530da3053e49d42b837dca0b1348ff2 --- M be/src/scheduling/query-schedule.cc M be/src/scheduling/scheduler-test.cc M be/src/scheduling/scheduler.cc M be/src/scheduling/scheduler.h M fe/src/main/java/org/apache/impala/planner/UnionNode.java D testdata/workloads/functional-query/queries/QueryTest/mt-dop-parquet-admission-slots.test A testdata/workloads/functional-query/queries/QueryTest/mt-dop-parquet-scheduling.test M tests/query_test/test_mt_dop.py 8 files changed, 380 insertions(+), 219 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/84/14384/12 -- To view, visit http://gerrit.cloudera.org:8080/14384 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: I0d2e9c86b530da3053e49d42b837dca0b1348ff2 Gerrit-Change-Number: 14384 Gerrit-PatchSet: 12 Gerrit-Owner: Tim Armstrong Gerrit-Reviewer: Andrew Sherman Gerrit-Reviewer: Bikramjeet Vig Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tim Armstrong
[Impala-ASF-CR] IMPALA-8999: make union scheduling work with mt dop
Bikramjeet Vig has posted comments on this change. ( http://gerrit.cloudera.org:8080/14384 ) Change subject: IMPALA-8999: make union scheduling work with mt_dop .. Patch Set 11: (9 comments) http://gerrit.cloudera.org:8080/#/c/14384/11/be/src/scheduling/scheduler.h File be/src/scheduling/scheduler.h: http://gerrit.cloudera.org:8080/#/c/14384/11/be/src/scheduling/scheduler.h@381 PS11, Line 381: For HDFS, this attempts to load balance among instances by computing the average : /// number of bytes per instances and then in a single pass assigning scan ranges to : /// each instance to roughly meet that average. nit: looks like we forgot to update this in the commit for IMPALA-9015 http://gerrit.cloudera.org:8080/#/c/14384/11/be/src/scheduling/scheduler.cc File be/src/scheduling/scheduler.cc: http://gerrit.cloudera.org:8080/#/c/14384/11/be/src/scheduling/scheduler.cc@291 PS11, Line 291: one or more scan nodes for the cases with non-union fragments, we are now considering the parallelism of all scan nodes in the fragment as opposed to the left most scan node, what are the cases that this can happen in a plan? Maybe add that as a test case too http://gerrit.cloudera.org:8080/#/c/14384/11/be/src/scheduling/scheduler.cc@332 PS11, Line 332: Instance selection for an interior fragment what does instance selection and interior fragment mean here? http://gerrit.cloudera.org:8080/#/c/14384/11/be/src/scheduling/scheduler.cc@360 PS11, Line 360: hosts nit: instances_per_host http://gerrit.cloudera.org:8080/#/c/14384/11/be/src/scheduling/scheduler.cc@363 PS11, Line 363: only factor in nit: got confused with this, read it as "only factor... in" vs the intended meaning of "only... factor in". maybe say "only consider parallelism" http://gerrit.cloudera.org:8080/#/c/14384/11/be/src/scheduling/scheduler.cc@368 PS11, Line 368: input_fragment_hosts nit: input_fragment_instances_per_host http://gerrit.cloudera.org:8080/#/c/14384/11/fe/src/main/java/org/apache/impala/planner/UnionNode.java File fe/src/main/java/org/apache/impala/planner/UnionNode.java: http://gerrit.cloudera.org:8080/#/c/14384/11/fe/src/main/java/org/apache/impala/planner/UnionNode.java@122 PS11, Line 122: Union fragments are scheduled on the union of hosts that the child fragments run : // on. nit: Union fragments are scheduled on the union of hosts of all scans in the fragment as well as the hosts of all its input fragments http://gerrit.cloudera.org:8080/#/c/14384/11/testdata/workloads/functional-query/queries/QueryTest/mt-dop-parquet-scheduling.test File testdata/workloads/functional-query/queries/QueryTest/mt-dop-parquet-scheduling.test: http://gerrit.cloudera.org:8080/#/c/14384/11/testdata/workloads/functional-query/queries/QueryTest/mt-dop-parquet-scheduling.test@52 PS11, Line 52: it nit: if http://gerrit.cloudera.org:8080/#/c/14384/11/testdata/workloads/functional-query/queries/QueryTest/mt-dop-parquet-scheduling.test@133 PS11, Line 133: add a test for a union fragment where the scan ranges for scan node are more than 4 but the max instances gets bounded by mt_dop -- To view, visit http://gerrit.cloudera.org:8080/14384 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I0d2e9c86b530da3053e49d42b837dca0b1348ff2 Gerrit-Change-Number: 14384 Gerrit-PatchSet: 11 Gerrit-Owner: Tim Armstrong Gerrit-Reviewer: Andrew Sherman Gerrit-Reviewer: Bikramjeet Vig Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Sat, 19 Oct 2019 00:54:34 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-8999: make union scheduling work with mt dop
Tim Armstrong has posted comments on this change. ( http://gerrit.cloudera.org:8080/14384 ) Change subject: IMPALA-8999: make union scheduling work with mt_dop .. Patch Set 11: Exhaustive tests passed -- To view, visit http://gerrit.cloudera.org:8080/14384 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I0d2e9c86b530da3053e49d42b837dca0b1348ff2 Gerrit-Change-Number: 14384 Gerrit-PatchSet: 11 Gerrit-Owner: Tim Armstrong Gerrit-Reviewer: Andrew Sherman Gerrit-Reviewer: Bikramjeet Vig Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Thu, 17 Oct 2019 15:45:58 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-8999: make union scheduling work with mt dop
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/14384 ) Change subject: IMPALA-8999: make union scheduling work with mt_dop .. Patch Set 11: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/4810/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/14384 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I0d2e9c86b530da3053e49d42b837dca0b1348ff2 Gerrit-Change-Number: 14384 Gerrit-PatchSet: 11 Gerrit-Owner: Tim Armstrong Gerrit-Reviewer: Andrew Sherman Gerrit-Reviewer: Bikramjeet Vig Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Thu, 17 Oct 2019 00:23:20 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-8999: make union scheduling work with mt dop
Tim Armstrong has posted comments on this change. ( http://gerrit.cloudera.org:8080/14384 ) Change subject: IMPALA-8999: make union scheduling work with mt_dop .. Patch Set 11: I'm running exhaustive tests still, but this should be ready for review. -- To view, visit http://gerrit.cloudera.org:8080/14384 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I0d2e9c86b530da3053e49d42b837dca0b1348ff2 Gerrit-Change-Number: 14384 Gerrit-PatchSet: 11 Gerrit-Owner: Tim Armstrong Gerrit-Reviewer: Bikramjeet Vig Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Wed, 16 Oct 2019 23:40:47 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-8999: make union scheduling work with mt dop
Hello Bikramjeet Vig, Impala Public Jenkins, I'd like you to reexamine a change. Please visit http://gerrit.cloudera.org:8080/14384 to look at the new patch set (#11). Change subject: IMPALA-8999: make union scheduling work with mt_dop .. IMPALA-8999: make union scheduling work with mt_dop This change unifies mt_dop scheduling between the union and scan cases. Testing: Manually checked that fragments with unions get parallelised to the correct degree, both as a result of scans within the fragment and input fragments. Extend TestMtDopAdmissionSlots (renamed to TestMtDopScheduling) to confirm that queries that were not parallelised before are now parallelised. These tests verify the number of instances of each operator using the ExecSummary embedded in the profile. Change-Id: I0d2e9c86b530da3053e49d42b837dca0b1348ff2 --- M be/src/scheduling/query-schedule.cc M be/src/scheduling/scheduler-test.cc M be/src/scheduling/scheduler.cc M be/src/scheduling/scheduler.h M fe/src/main/java/org/apache/impala/planner/UnionNode.java D testdata/workloads/functional-query/queries/QueryTest/mt-dop-parquet-admission-slots.test A testdata/workloads/functional-query/queries/QueryTest/mt-dop-parquet-scheduling.test M tests/query_test/test_mt_dop.py 8 files changed, 372 insertions(+), 213 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/84/14384/11 -- To view, visit http://gerrit.cloudera.org:8080/14384 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: I0d2e9c86b530da3053e49d42b837dca0b1348ff2 Gerrit-Change-Number: 14384 Gerrit-PatchSet: 11 Gerrit-Owner: Tim Armstrong Gerrit-Reviewer: Bikramjeet Vig Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tim Armstrong
