[kudu-CR] rpc-test: fix flakiness in tests that expect timeouts
Adar Dembo has submitted this change and it was merged. Change subject: rpc-test: fix flakiness in tests that expect timeouts .. rpc-test: fix flakiness in tests that expect timeouts The DoTestExpectTimeout() utility function would set a timeout to some number of milliseconds N, and then ask the server to sleep for (N + 50) milliseconds, expecting a timeout. It would then assert that the timeout was returned after some amount of time between N and N+50, but no more than N+50ms. This would be flaky under concurrent load (eg stress threads) because the sleep(50ms) might sometimes actually sleep for an extra 50-100ms. This just changes the test to ask the server to sleep for n+500ms, giving it a lot more budget for sloppiness. I looped TestCallTimeout/0 with 4 stress threads 1000 times in TSAN. Before[1] it failed 4/1000. After[2] it didn't fail. [1] http://dist-test.cloudera.org//job?job_id=todd.1480912345.2054 [2] http://dist-test.cloudera.org//job?job_id=todd.1480912461.12007 Change-Id: Ifff555634968bc92f453b25af4d5c15da21edf7c Reviewed-on: http://gerrit.cloudera.org:8080/5356 Tested-by: Kudu Jenkins Reviewed-by: Adar Dembo --- M src/kudu/rpc/rpc-test-base.h 1 file changed, 2 insertions(+), 2 deletions(-) Approvals: Adar Dembo: Looks good to me, approved Kudu Jenkins: Verified -- To view, visit http://gerrit.cloudera.org:8080/5356 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-MessageType: merged Gerrit-Change-Id: Ifff555634968bc92f453b25af4d5c15da21edf7c Gerrit-PatchSet: 3 Gerrit-Project: kudu Gerrit-Branch: master Gerrit-Owner: Todd Lipcon Gerrit-Reviewer: Adar Dembo Gerrit-Reviewer: Kudu Jenkins Gerrit-Reviewer: Mike Percy
[kudu-CR] rpc-test: fix flakiness in tests that expect timeouts
Adar Dembo has posted comments on this change. Change subject: rpc-test: fix flakiness in tests that expect timeouts .. Patch Set 2: Code-Review+2 -- To view, visit http://gerrit.cloudera.org:8080/5356 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-MessageType: comment Gerrit-Change-Id: Ifff555634968bc92f453b25af4d5c15da21edf7c Gerrit-PatchSet: 2 Gerrit-Project: kudu Gerrit-Branch: master Gerrit-Owner: Todd Lipcon Gerrit-Reviewer: Adar Dembo Gerrit-Reviewer: Kudu Jenkins Gerrit-Reviewer: Mike Percy Gerrit-HasComments: No
[kudu-CR] rpc-test: fix flakiness in tests that expect timeouts
Hello Kudu Jenkins, I'd like you to reexamine a change. Please visit http://gerrit.cloudera.org:8080/5356 to look at the new patch set (#2). Change subject: rpc-test: fix flakiness in tests that expect timeouts .. rpc-test: fix flakiness in tests that expect timeouts The DoTestExpectTimeout() utility function would set a timeout to some number of milliseconds N, and then ask the server to sleep for (N + 50) milliseconds, expecting a timeout. It would then assert that the timeout was returned after some amount of time between N and N+50, but no more than N+50ms. This would be flaky under concurrent load (eg stress threads) because the sleep(50ms) might sometimes actually sleep for an extra 50-100ms. This just changes the test to ask the server to sleep for n+500ms, giving it a lot more budget for sloppiness. I looped TestCallTimeout/0 with 4 stress threads 1000 times in TSAN. Before[1] it failed 4/1000. After[2] it didn't fail. [1] http://dist-test.cloudera.org//job?job_id=todd.1480912345.2054 [2] http://dist-test.cloudera.org//job?job_id=todd.1480912461.12007 Change-Id: Ifff555634968bc92f453b25af4d5c15da21edf7c --- M src/kudu/rpc/rpc-test-base.h 1 file changed, 2 insertions(+), 2 deletions(-) git pull ssh://gerrit.cloudera.org:29418/kudu refs/changes/56/5356/2 -- To view, visit http://gerrit.cloudera.org:8080/5356 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-MessageType: newpatchset Gerrit-Change-Id: Ifff555634968bc92f453b25af4d5c15da21edf7c Gerrit-PatchSet: 2 Gerrit-Project: kudu Gerrit-Branch: master Gerrit-Owner: Todd Lipcon Gerrit-Reviewer: Adar Dembo Gerrit-Reviewer: Kudu Jenkins Gerrit-Reviewer: Mike Percy
[kudu-CR] rpc-test: fix flakiness in tests that expect timeouts
Adar Dembo has posted comments on this change. Change subject: rpc-test: fix flakiness in tests that expect timeouts .. Patch Set 1: (1 comment) http://gerrit.cloudera.org:8080/#/c/5356/1//COMMIT_MSG Commit Message: PS1, Line 12: some Extra 'some' here? -- To view, visit http://gerrit.cloudera.org:8080/5356 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-MessageType: comment Gerrit-Change-Id: Ifff555634968bc92f453b25af4d5c15da21edf7c Gerrit-PatchSet: 1 Gerrit-Project: kudu Gerrit-Branch: master Gerrit-Owner: Todd Lipcon Gerrit-Reviewer: Adar Dembo Gerrit-Reviewer: Kudu Jenkins Gerrit-Reviewer: Mike Percy Gerrit-HasComments: Yes
[kudu-CR] rpc-test: fix flakiness in tests that expect timeouts
Hello Mike Percy, Adar Dembo, I'd like you to do a code review. Please visit http://gerrit.cloudera.org:8080/5356 to review the following change. Change subject: rpc-test: fix flakiness in tests that expect timeouts .. rpc-test: fix flakiness in tests that expect timeouts The DoTestExpectTimeout() utility function would set a timeout to some number of milliseconds N, and then ask the server to sleep for (N + 50) milliseconds, expecting a timeout. It would then assert that the timeout was returned some after some amount of time between N and N+50, but no more than N+50ms. This would be flaky under concurrent load (eg stress threads) because the sleep(50ms) might sometimes actually sleep for an extra 50-100ms. This just changes the test to ask the server to sleep for n+500ms, giving it a lot more budget for sloppiness. I looped TestCallTimeout/0 with 4 stress threads 1000 times in TSAN. Before[1] it failed 4/1000. After[2] it didn't fail. [1] http://dist-test.cloudera.org//job?job_id=todd.1480912345.2054 [2] http://dist-test.cloudera.org//job?job_id=todd.1480912461.12007 Change-Id: Ifff555634968bc92f453b25af4d5c15da21edf7c --- M src/kudu/rpc/rpc-test-base.h 1 file changed, 2 insertions(+), 2 deletions(-) git pull ssh://gerrit.cloudera.org:29418/kudu refs/changes/56/5356/1 -- To view, visit http://gerrit.cloudera.org:8080/5356 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-MessageType: newchange Gerrit-Change-Id: Ifff555634968bc92f453b25af4d5c15da21edf7c Gerrit-PatchSet: 1 Gerrit-Project: kudu Gerrit-Branch: master Gerrit-Owner: Todd Lipcon Gerrit-Reviewer: Adar Dembo Gerrit-Reviewer: Mike Percy