Todd Lipcon has submitted this change and it was merged. Change subject: KUDU-1622. result_tracker: respond to RPCs outside of the lock ......................................................................
KUDU-1622. result_tracker: respond to RPCs outside of the lock This patch started with the goal of addressing the flakiness of ExactlyOnceRpcTest.TestExactlyOnceSemanticsAfterRpcCompleted. The issue was that the MemTracker updates resulting from an RPC response were occurring asynchronously after the RPC was responded, and in two separate increments. Thus, in this test, there was one scenario in which the test would record a partial update and then in a later assertion not match the earlier recorded value. In order to fix this, I ended up at a solution which both fixed the flakiness and reduced lock contention in ResultTracker. We now respond to RPCs after we've done the required MemTracker updates, which also means that we respond to RPCs after dropping the ResultTracker locks. This has been a high contention point in stress tests, so this is a nice benefit. Fixing the flakiness also meant that other 'AssertEventually' workarounds from this test case could now be removed. I looped ExactlyOnceRpcTest.TestExactlyOnceSemanticsAfterRpcCompleted 500 times with 4 stress threads and it passed 100% of the time[1]. Prior to this patch, it would reliably fail at least a couple runs out of 500[2]. I also looped exactly_once_writes-itest 500 times with this patch with 100% success[3]. [1] http://dist-test.cloudera.org//job?job_id=todd.1480928777.29435 [2] http://dist-test.cloudera.org//job?job_id=todd.1480929041.29742 [3] http://dist-test.cloudera.org//job?job_id=todd.1480929367.31010 Change-Id: I065b9e40bc1af81e0871220a0a01461ea35143b5 Reviewed-on: http://gerrit.cloudera.org:8080/5359 Tested-by: Kudu Jenkins Reviewed-by: David Ribeiro Alves <dral...@apache.org> --- M src/kudu/rpc/exactly_once_rpc-test.cc M src/kudu/rpc/result_tracker.cc 2 files changed, 93 insertions(+), 84 deletions(-) Approvals: David Ribeiro Alves: Looks good to me, approved Kudu Jenkins: Verified -- To view, visit http://gerrit.cloudera.org:8080/5359 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-MessageType: merged Gerrit-Change-Id: I065b9e40bc1af81e0871220a0a01461ea35143b5 Gerrit-PatchSet: 4 Gerrit-Project: kudu Gerrit-Branch: master Gerrit-Owner: Todd Lipcon <t...@apache.org> Gerrit-Reviewer: Adar Dembo <a...@cloudera.com> Gerrit-Reviewer: David Ribeiro Alves <dral...@apache.org> Gerrit-Reviewer: Kudu Jenkins Gerrit-Reviewer: Tidy Bot Gerrit-Reviewer: Todd Lipcon <t...@apache.org>
