[Impala-ASF-CR] IMPALA-9834: De-flake TestQueryRetries on EC builds
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/16149 ) Change subject: IMPALA-9834: De-flake TestQueryRetries on EC builds .. Patch Set 3: Verified+1 -- To view, visit http://gerrit.cloudera.org:8080/16149 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Id5c73c2cbd0ef369175856c41f36d4b0de4b8d71 Gerrit-Change-Number: 16149 Gerrit-PatchSet: 3 Gerrit-Owner: Sahil Takiar Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Sahil Takiar Gerrit-Reviewer: Thomas Tauber-Marshall Gerrit-Comment-Date: Thu, 09 Jul 2020 21:10:32 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-9834: De-flake TestQueryRetries on EC builds
Impala Public Jenkins has submitted this change and it was merged. ( http://gerrit.cloudera.org:8080/16149 ) Change subject: IMPALA-9834: De-flake TestQueryRetries on EC builds .. IMPALA-9834: De-flake TestQueryRetries on EC builds This patch skips all tests in TestQueryRetries on EC builds. The tests in TestQueryRetries runs queries that run on three instances during regular builds (HDFS, S3, etc.), but only two instances on EC builds. This causes some non-deterministism during the test because killing an impalad in the mini-cluster won't necessarily cause a retry to be triggered. It bumps up the timeout used when waiting for a query to be retried. It improves the assertion in __get_query_id_from_profile so that it dumps the full profile when the assertion fails. This should help debuggability of any test failures that fail in this assertion. Testing: * Ran TestQueryRetries locally Change-Id: Id5c73c2cbd0ef369175856c41f36d4b0de4b8d71 Reviewed-on: http://gerrit.cloudera.org:8080/16149 Reviewed-by: Impala Public Jenkins Tested-by: Impala Public Jenkins --- M tests/custom_cluster/test_query_retries.py 1 file changed, 16 insertions(+), 2 deletions(-) Approvals: Impala Public Jenkins: Looks good to me, approved; Verified -- To view, visit http://gerrit.cloudera.org:8080/16149 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: merged Gerrit-Change-Id: Id5c73c2cbd0ef369175856c41f36d4b0de4b8d71 Gerrit-Change-Number: 16149 Gerrit-PatchSet: 4 Gerrit-Owner: Sahil Takiar Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Sahil Takiar Gerrit-Reviewer: Thomas Tauber-Marshall
[Impala-ASF-CR] IMPALA-9834: De-flake TestQueryRetries on EC builds
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/16149 ) Change subject: IMPALA-9834: De-flake TestQueryRetries on EC builds .. Patch Set 3: Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/6115/ DRY_RUN=false -- To view, visit http://gerrit.cloudera.org:8080/16149 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Id5c73c2cbd0ef369175856c41f36d4b0de4b8d71 Gerrit-Change-Number: 16149 Gerrit-PatchSet: 3 Gerrit-Owner: Sahil Takiar Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Sahil Takiar Gerrit-Reviewer: Thomas Tauber-Marshall Gerrit-Comment-Date: Thu, 09 Jul 2020 16:08:48 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-9834: De-flake TestQueryRetries on EC builds
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/16149 ) Change subject: IMPALA-9834: De-flake TestQueryRetries on EC builds .. Patch Set 3: Code-Review+2 -- To view, visit http://gerrit.cloudera.org:8080/16149 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Id5c73c2cbd0ef369175856c41f36d4b0de4b8d71 Gerrit-Change-Number: 16149 Gerrit-PatchSet: 3 Gerrit-Owner: Sahil Takiar Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Sahil Takiar Gerrit-Reviewer: Thomas Tauber-Marshall Gerrit-Comment-Date: Thu, 09 Jul 2020 16:08:47 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-9834: De-flake TestQueryRetries on EC builds
Thomas Tauber-Marshall has posted comments on this change. ( http://gerrit.cloudera.org:8080/16149 ) Change subject: IMPALA-9834: De-flake TestQueryRetries on EC builds .. Patch Set 2: Code-Review+2 -- To view, visit http://gerrit.cloudera.org:8080/16149 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Id5c73c2cbd0ef369175856c41f36d4b0de4b8d71 Gerrit-Change-Number: 16149 Gerrit-PatchSet: 2 Gerrit-Owner: Sahil Takiar Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Sahil Takiar Gerrit-Reviewer: Thomas Tauber-Marshall Gerrit-Comment-Date: Thu, 09 Jul 2020 16:06:43 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-9834: De-flake TestQueryRetries on EC builds
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/16149 ) Change subject: IMPALA-9834: De-flake TestQueryRetries on EC builds .. Patch Set 2: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/6538/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/16149 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Id5c73c2cbd0ef369175856c41f36d4b0de4b8d71 Gerrit-Change-Number: 16149 Gerrit-PatchSet: 2 Gerrit-Owner: Sahil Takiar Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Sahil Takiar Gerrit-Reviewer: Thomas Tauber-Marshall Gerrit-Comment-Date: Thu, 09 Jul 2020 03:30:15 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-9834: De-flake TestQueryRetries on EC builds
Sahil Takiar has posted comments on this change. ( http://gerrit.cloudera.org:8080/16149 ) Change subject: IMPALA-9834: De-flake TestQueryRetries on EC builds .. Patch Set 1: (4 comments) http://gerrit.cloudera.org:8080/#/c/16149/1//COMMIT_MSG Commit Message: http://gerrit.cloudera.org:8080/#/c/16149/1//COMMIT_MSG@10 PS1, Line 10: EC builds > Are there any other build types where this might be a concern like S3, eg. I haven't seen any flakiness on S3 builds, just EC builds. I think the difference is that the tables are small when EC is enabled because all the data is all compressed using erasure coding. http://gerrit.cloudera.org:8080/#/c/16149/1/tests/custom_cluster/test_query_retries.py File tests/custom_cluster/test_query_retries.py: http://gerrit.cloudera.org:8080/#/c/16149/1/tests/custom_cluster/test_query_retries.py@77 PS1, Line 77: self.__kill_random_impalad() > Do we not have the same problem here of the query possibly running on only Yeah, thats a good point. I just add the skip EC flag to the entire file. I don't think we get much additional coverage running on EC anyway. http://gerrit.cloudera.org:8080/#/c/16149/1/tests/custom_cluster/test_query_retries.py@226 PS1, Line 226: killed_impalad = self.__kill_random_impalad() > Same here Done http://gerrit.cloudera.org:8080/#/c/16149/1/tests/custom_cluster/test_query_retries.py@561 PS1, Line 561: def __validate_runtime_profiles(self, retried_runtime_profile, original_query_id, > Might be nice to add some asserts that the profiles show the query getting Done -- To view, visit http://gerrit.cloudera.org:8080/16149 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Id5c73c2cbd0ef369175856c41f36d4b0de4b8d71 Gerrit-Change-Number: 16149 Gerrit-PatchSet: 1 Gerrit-Owner: Sahil Takiar Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Sahil Takiar Gerrit-Reviewer: Thomas Tauber-Marshall Gerrit-Comment-Date: Thu, 09 Jul 2020 03:14:31 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-9834: De-flake TestQueryRetries on EC builds
Hello Thomas Tauber-Marshall, Impala Public Jenkins, I'd like you to reexamine a change. Please visit http://gerrit.cloudera.org:8080/16149 to look at the new patch set (#2). Change subject: IMPALA-9834: De-flake TestQueryRetries on EC builds .. IMPALA-9834: De-flake TestQueryRetries on EC builds This patch skips all tests in TestQueryRetries on EC builds. The tests in TestQueryRetries runs queries that run on three instances during regular builds (HDFS, S3, etc.), but only two instances on EC builds. This causes some non-deterministism during the test because killing an impalad in the mini-cluster won't necessarily cause a retry to be triggered. It bumps up the timeout used when waiting for a query to be retried. It improves the assertion in __get_query_id_from_profile so that it dumps the full profile when the assertion fails. This should help debuggability of any test failures that fail in this assertion. Testing: * Ran TestQueryRetries locally Change-Id: Id5c73c2cbd0ef369175856c41f36d4b0de4b8d71 --- M tests/custom_cluster/test_query_retries.py 1 file changed, 16 insertions(+), 2 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/49/16149/2 -- To view, visit http://gerrit.cloudera.org:8080/16149 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: Id5c73c2cbd0ef369175856c41f36d4b0de4b8d71 Gerrit-Change-Number: 16149 Gerrit-PatchSet: 2 Gerrit-Owner: Sahil Takiar Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Thomas Tauber-Marshall
[Impala-ASF-CR] IMPALA-9834: De-flake TestQueryRetries on EC builds
Thomas Tauber-Marshall has posted comments on this change. ( http://gerrit.cloudera.org:8080/16149 ) Change subject: IMPALA-9834: De-flake TestQueryRetries on EC builds .. Patch Set 1: (4 comments) http://gerrit.cloudera.org:8080/#/c/16149/1//COMMIT_MSG Commit Message: http://gerrit.cloudera.org:8080/#/c/16149/1//COMMIT_MSG@10 PS1, Line 10: EC builds Are there any other build types where this might be a concern like S3, eg. should we maybe use SkipIfNotHdfsMinicluster.tuned_for_minicluster, or is it okay because its a custom_cluster test so it sort of has to always be run against the minicluster? http://gerrit.cloudera.org:8080/#/c/16149/1/tests/custom_cluster/test_query_retries.py File tests/custom_cluster/test_query_retries.py: http://gerrit.cloudera.org:8080/#/c/16149/1/tests/custom_cluster/test_query_retries.py@77 PS1, Line 77: self.__kill_random_impalad() Do we not have the same problem here of the query possibly running on only two nodes in EC? http://gerrit.cloudera.org:8080/#/c/16149/1/tests/custom_cluster/test_query_retries.py@226 PS1, Line 226: killed_impalad = self.__kill_random_impalad() Same here http://gerrit.cloudera.org:8080/#/c/16149/1/tests/custom_cluster/test_query_retries.py@561 PS1, Line 561: def __validate_runtime_profiles(self, retried_runtime_profile, original_query_id, Might be nice to add some asserts that the profiles show the query getting run on the expected number of backends. -- To view, visit http://gerrit.cloudera.org:8080/16149 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Id5c73c2cbd0ef369175856c41f36d4b0de4b8d71 Gerrit-Change-Number: 16149 Gerrit-PatchSet: 1 Gerrit-Owner: Sahil Takiar Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Thomas Tauber-Marshall Gerrit-Comment-Date: Wed, 08 Jul 2020 17:27:16 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-9834: De-flake TestQueryRetries on EC builds
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/16149 ) Change subject: IMPALA-9834: De-flake TestQueryRetries on EC builds .. Patch Set 1: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/6518/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/16149 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Id5c73c2cbd0ef369175856c41f36d4b0de4b8d71 Gerrit-Change-Number: 16149 Gerrit-PatchSet: 1 Gerrit-Owner: Sahil Takiar Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Thomas Tauber-Marshall Gerrit-Comment-Date: Tue, 07 Jul 2020 21:14:12 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-9834: De-flake TestQueryRetries on EC builds
Sahil Takiar has uploaded this change for review. ( http://gerrit.cloudera.org:8080/16149 Change subject: IMPALA-9834: De-flake TestQueryRetries on EC builds .. IMPALA-9834: De-flake TestQueryRetries on EC builds This patch makes the following changes to de-flake TestQueryRetries on EC builds. It skips any test that scans tpch.lineitem (referred to as a the _shuffle_heavy_query in TestQueryRetries). This query runs on three instances during regular builds (HDFS, S3, etc.), but only two instances on EC builds. This causes some non-deterministism during the test becase killing an impalad in the mini-cluster won't necessarily cause a retry to be triggered. It bumps up the timeout used when waiting for a query to be retried. It improves the assertion in __get_query_id_from_profile so that it dumps the full profile when the assertion fails. This should help debuggability of any test failures that fail in this assertion. Testing: * Ran TestQueryRetries locally Change-Id: Id5c73c2cbd0ef369175856c41f36d4b0de4b8d71 --- M tests/custom_cluster/test_query_retries.py 1 file changed, 11 insertions(+), 2 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/49/16149/1 -- To view, visit http://gerrit.cloudera.org:8080/16149 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newchange Gerrit-Change-Id: Id5c73c2cbd0ef369175856c41f36d4b0de4b8d71 Gerrit-Change-Number: 16149 Gerrit-PatchSet: 1 Gerrit-Owner: Sahil Takiar
