[ https://issues.apache.org/jira/browse/ZOOKEEPER-1505?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13526854#comment-13526854 ]
Hadoop QA commented on ZOOKEEPER-1505: -------------------------------------- -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12548952/ZOOKEEPER-1505.patch against trunk revision 1415847. +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 3 new or modified tests. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. -1 findbugs. The patch appears to introduce 1 new Findbugs (version 1.3.9) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. +1 core tests. The patch passed core unit tests. +1 contrib tests. The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-ZOOKEEPER-Build/1288//testReport/ Findbugs warnings: https://builds.apache.org/job/PreCommit-ZOOKEEPER-Build/1288//artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html Console output: https://builds.apache.org/job/PreCommit-ZOOKEEPER-Build/1288//console This message is automatically generated. > Multi-thread CommitProcessor > ---------------------------- > > Key: ZOOKEEPER-1505 > URL: https://issues.apache.org/jira/browse/ZOOKEEPER-1505 > Project: ZooKeeper > Issue Type: Improvement > Components: server > Affects Versions: 3.4.3, 3.4.4, 3.5.0 > Reporter: Jay Shrauner > Assignee: Jay Shrauner > Labels: performance, scaling > Fix For: 3.5.0 > > Attachments: ZOOKEEPER-1505.patch, ZOOKEEPER-1505.patch, > ZOOKEEPER-1505.patch > > > CommitProcessor has a single thread that both pulls requests off its queues > and runs all downstream processors. This is noticeably inefficient for > read-intensive workloads, which could be run concurrently. The trick is > handling write transactions. I propose multi-threading this code according to > the following two constraints > - each session must see its requests responded to in order > - all committed transactions must be handled in zxid order, across all > sessions > I believe these cover the only constraints we need to honor. In particular, I > believe we can relax the following: > - it does not matter if the read request in one session happens before or > after the write request in another session > With these constraints, I propose the following threads > - 1 primary queue servicing/work dispatching thread > - 0-N assignable worker threads, where a given session is always assigned > to the same worker thread > By assigning sessions always to the same worker thread (using a simple > sessionId mod number of worker threads), we guarantee the first constraint-- > requests we push onto the thread queue are processed in order. The way we > guarantee the second constraint is we only allow a single commit transaction > to be in flight at a time--the queue servicing thread blocks while a commit > transaction is in flight, and when the transaction completes it clears the > flag. > On a 32 core machine running Linux 2.6.38, achieved best performance with 32 > worker threads for a 56% +/- 5% improvement in throughput (this improvement > was measured on top of that for ZOOKEEPER-1504, not in isolation). > New classes introduced in this patch are: > WorkerService (also in ZOOKEEPER-1504): ExecutorService wrapper that > makes worker threads daemon threads and names then in an easily debuggable > manner. Supports assignable threads (as used here) and non-assignable threads > (as used by NIOServerCnxnFactory). -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira