[Impala-ASF-CR] IMPALA-9073: fix test executor concurrency flakiness
Impala Public Jenkins has submitted this change and it was merged. ( http://gerrit.cloudera.org:8080/14606 ) Change subject: IMPALA-9073: fix test_executor_concurrency flakiness .. IMPALA-9073: fix test_executor_concurrency flakiness The test was checking the incorrect invariant - the slot mechanism only prevents more than than number of queries running on a backend. More queries can run on a cluster since the query's backends are freed up before the query itself finishes. It was a little tricky picking an appropriate metric since there is no strong consistency between the metrics, e.g. decrementing a metric after a backend finishes may race with admitting the next query. So I simply used the same metric used by the admission controller in making decisions, which should be strongly consistent w.r.t. admission control decissions. Also remove the concurrency limit on the coordinator, which seemed inconsistent with the purpose of the test, because we only want concurrency to be limited by the executors. Testing: Looped the test for a bit. Change-Id: I910028919f248a3bf5de345e9eade9dbc4353ebd Reviewed-on: http://gerrit.cloudera.org:8080/14606 Reviewed-by: Impala Public Jenkins Tested-by: Impala Public Jenkins --- M tests/custom_cluster/test_executor_groups.py 1 file changed, 15 insertions(+), 8 deletions(-) Approvals: Impala Public Jenkins: Looks good to me, approved; Verified -- To view, visit http://gerrit.cloudera.org:8080/14606 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: merged Gerrit-Change-Id: I910028919f248a3bf5de345e9eade9dbc4353ebd Gerrit-Change-Number: 14606 Gerrit-PatchSet: 4 Gerrit-Owner: Tim Armstrong Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Sahil Takiar
[Impala-ASF-CR] IMPALA-9073: fix test executor concurrency flakiness
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/14606 ) Change subject: IMPALA-9073: fix test_executor_concurrency flakiness .. Patch Set 3: Verified+1 -- To view, visit http://gerrit.cloudera.org:8080/14606 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I910028919f248a3bf5de345e9eade9dbc4353ebd Gerrit-Change-Number: 14606 Gerrit-PatchSet: 3 Gerrit-Owner: Tim Armstrong Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Sahil Takiar Gerrit-Comment-Date: Fri, 01 Nov 2019 21:12:08 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-9073: fix test executor concurrency flakiness
Sahil Takiar has posted comments on this change. ( http://gerrit.cloudera.org:8080/14606 ) Change subject: IMPALA-9073: fix test_executor_concurrency flakiness .. Patch Set 2: Code-Review+2 -- To view, visit http://gerrit.cloudera.org:8080/14606 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I910028919f248a3bf5de345e9eade9dbc4353ebd Gerrit-Change-Number: 14606 Gerrit-PatchSet: 2 Gerrit-Owner: Tim Armstrong Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Sahil Takiar Gerrit-Comment-Date: Fri, 01 Nov 2019 16:32:26 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-9073: fix test executor concurrency flakiness
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/14606 ) Change subject: IMPALA-9073: fix test_executor_concurrency flakiness .. Patch Set 3: Code-Review+2 -- To view, visit http://gerrit.cloudera.org:8080/14606 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I910028919f248a3bf5de345e9eade9dbc4353ebd Gerrit-Change-Number: 14606 Gerrit-PatchSet: 3 Gerrit-Owner: Tim Armstrong Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Sahil Takiar Gerrit-Comment-Date: Fri, 01 Nov 2019 16:51:07 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-9073: fix test executor concurrency flakiness
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/14606 ) Change subject: IMPALA-9073: fix test_executor_concurrency flakiness .. Patch Set 3: Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/5162/ DRY_RUN=false -- To view, visit http://gerrit.cloudera.org:8080/14606 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I910028919f248a3bf5de345e9eade9dbc4353ebd Gerrit-Change-Number: 14606 Gerrit-PatchSet: 3 Gerrit-Owner: Tim Armstrong Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Sahil Takiar Gerrit-Comment-Date: Fri, 01 Nov 2019 16:51:08 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-9073: fix test executor concurrency flakiness
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/14606 ) Change subject: IMPALA-9073: fix test_executor_concurrency flakiness .. Patch Set 1: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/4923/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/14606 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I910028919f248a3bf5de345e9eade9dbc4353ebd Gerrit-Change-Number: 14606 Gerrit-PatchSet: 1 Gerrit-Owner: Tim Armstrong Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Sahil Takiar Gerrit-Comment-Date: Fri, 01 Nov 2019 00:26:19 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-9073: fix test executor concurrency flakiness
Tim Armstrong has uploaded this change for review. ( http://gerrit.cloudera.org:8080/14606 Change subject: IMPALA-9073: fix test_executor_concurrency flakiness .. IMPALA-9073: fix test_executor_concurrency flakiness The test was checking the incorrect invariant - the slot mechanism only prevents more than than number of queries running on a backend. More queries can run on a cluster since the query's backends are freed up before the query itself finishes. It was a little tricky picking an appropriate metric since there is no strong consistency between the metrics, e.g. decrementing a metric after a backend finishes may race with admitting the next query. So I simply used the same metric used by the admission controller in making decisions, which should be strongly consistent w.r.t. admission control decissions. Also remove the concurrency limit on the coordinator, which seemed inconsistent with the purpose of the test, because we only want concurrency to be limited by the executors. Testing: Looped the test for a bit. Change-Id: I910028919f248a3bf5de345e9eade9dbc4353ebd --- M tests/custom_cluster/test_executor_groups.py 1 file changed, 15 insertions(+), 8 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/06/14606/1 -- To view, visit http://gerrit.cloudera.org:8080/14606 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newchange Gerrit-Change-Id: I910028919f248a3bf5de345e9eade9dbc4353ebd Gerrit-Change-Number: 14606 Gerrit-PatchSet: 1 Gerrit-Owner: Tim Armstrong
