[ https://issues.apache.org/jira/browse/HBASE-15716?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15316153#comment-15316153 ]
stack commented on HBASE-15716: ------------------------------- Interesting observation was hacking out this lock, I ran then into my being blocked responding... {code} "RpcServer.reader=1,bindAddress=ve0528.halxg.cloudera.com,port=16020" #34 daemon prio=5 os_prio=0 tid=0x00007fa76d886800 nid=0x59f0 runnable [0x00007f9f515e9000] java.lang.Thread.State: RUNNABLE at sun.nio.ch.NativeThread.current(Native Method) at sun.nio.ch.SocketChannelImpl.write(SocketChannelImpl.java:501) - locked <0x00007fa41f096f40> (a java.lang.Object) - locked <0x00007fa41f096f28> (a java.lang.Object) at org.apache.hadoop.hbase.ipc.BufferChain.write(BufferChain.java:105) at org.apache.hadoop.hbase.ipc.RpcServer.channelWrite(RpcServer.java:2401) at org.apache.hadoop.hbase.ipc.RpcServer$Responder.processResponse(RpcServer.java:1072) at org.apache.hadoop.hbase.ipc.RpcServer$Responder.doRespond(RpcServer.java:1136) at org.apache.hadoop.hbase.ipc.RpcServer$Call.sendResponseIfReady(RpcServer.java:570) - locked <0x00007f9fbf7652d0> (a org.apache.hadoop.hbase.ipc.RpcServer$Call) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:139) at org.apache.hadoop.hbase.ipc.SimpleRpcScheduler.dispatch(SimpleRpcScheduler.java:274) at org.apache.hadoop.hbase.ipc.RpcServer$Connection.processRequest(RpcServer.java:1871) at org.apache.hadoop.hbase.ipc.RpcServer$Connection.processOneRpc(RpcServer.java:1762) at org.apache.hadoop.hbase.ipc.RpcServer$Connection.process(RpcServer.java:1608) at org.apache.hadoop.hbase.ipc.RpcServer$Connection.readAndProcess(RpcServer.java:1588) at org.apache.hadoop.hbase.ipc.RpcServer$Listener.doRead(RpcServer.java:838) at org.apache.hadoop.hbase.ipc.RpcServer$Listener$Reader.doRunLoop(RpcServer.java:696) - locked <0x00007fa06a26acc0> (a org.apache.hadoop.hbase.ipc.RpcServer$Listener$Reader) at org.apache.hadoop.hbase.ipc.RpcServer$Listener$Reader.run(RpcServer.java:667) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) {code} Other notes on this synchronization are that as the throughput goes up, this synchronization becomes more of an obstacle. At rates of hundreds of ops a second, the churn in the CSLM shows... I should be able to do an array of volatiles or something sized by handlers/readers? I should also be able to do something with the fact that readpt is always incrementing... will be back. > HRegion#RegionScannerImpl scannerReadPoints synchronization constrains random > read > ---------------------------------------------------------------------------------- > > Key: HBASE-15716 > URL: https://issues.apache.org/jira/browse/HBASE-15716 > Project: HBase > Issue Type: Bug > Components: Performance > Reporter: stack > Assignee: stack > Attachments: 15716.prune.synchronizations.patch, > 15716.prune.synchronizations.v3.patch, 15716.prune.synchronizations.v4.patch, > 15716.prune.synchronizations.v4.patch, 15716.wip.more_to_be_done.patch, > Screen Shot 2016-04-26 at 2.05.45 PM.png, Screen Shot 2016-04-26 at 2.06.14 > PM.png, Screen Shot 2016-04-26 at 2.07.06 PM.png, Screen Shot 2016-04-26 at > 2.25.26 PM.png, Screen Shot 2016-04-26 at 6.02.29 PM.png, Screen Shot > 2016-04-27 at 9.49.35 AM.png, > current-branch-1.vs.NoSynchronization.vs.Patch.png, hits.png, > remove_cslm.patch > > > Here is a [~lhofhansl] special. > When we construct the region scanner, we get our read point and then store it > with the scanner instance in a Region scoped CSLM. This is done under a > synchronize on the CSLM. > This synchronize on a region-scoped Map creating region scanners is the > outstanding point of lock contention according to flight recorder (My work > load is workload c, random reads). -- This message was sent by Atlassian JIRA (v6.3.4#6332)