[
https://issues.apache.org/jira/browse/HADOOP-7105?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13474994#comment-13474994
]
jinglong.liujl commented on HADOOP-7105:
----------------------------------------
There're duplicate synchronized locks in registerChannel(), one is
registerChannel which's in Reader , the other is in socketchannel.register
which's in jdk. It's safty to remove one of it.
As Todd's suggestion, we've add a non-blocking queue in rpc to make accept not
block by reader. And this patch will be released in another issue.
> [IPC] Improvement of lock mechanism in Listener and Reader thread
> -----------------------------------------------------------------
>
> Key: HADOOP-7105
> URL: https://issues.apache.org/jira/browse/HADOOP-7105
> Project: Hadoop Common
> Issue Type: Improvement
> Components: ipc
> Affects Versions: 0.21.0
> Reporter: jinglong.liujl
> Attachments: improveListenerLock2.patch, improveListenerLock.patch
>
>
> In many client cocurrent access, single thread Listener will become
> bottleneck. Many client can't be served, and get connection time out.
> To improve Listener capacity, we make 2 modification.
> 1. Tuning ipc.server.listen.queue.size to a larger value to avoid client
> retry.
> 2. In currently implement, Listener will call registerChannel(), and
> finishAdd() in Reader, which will request Reader synchronized lock. Listener
> will cost too many time in waiting for this lock.
> We have made test,
> ./bin/hadoop org.apache.hadoop.hdfs.NNThroughputBenchmark -op create
> -threads 10000 -files 10000
> case 1 : Currently
> can not pass. and report
> hadoop-rd101.jx.baidu.com/10.65.25.166:59310. Already tried 0 time(s).
> case 2 : tuning back log to 10240
> average cost : 1285.72 ms
> case 3 : tuning back log to 10240 , and improve lock mechanism in patch
> average cost : 941.32 ms
> performance in average cost will improve 26%
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
