[ https://issues.apache.org/jira/browse/HADOOP-7105?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12992507#comment-12992507 ]
jinglong.liujl commented on HADOOP-7105: ---------------------------------------- 1. Firstly, My test is take on baidu's branch and made comperation between with and without this patch. We use NNThroughputBenchmark, but it has a little different with community trunk. it add nn.throughput.bench.rpcmode in NNThroughputBenchmark to make rpc test. I'll create an extra issue for it. 2. Our dfs.name.dir is single SATA diskļ¼ 7 reader threads, backlog is 10240. ./bin/hadoop org.apache.hadoop.hdfs.NNThroughputBenchmark -op create -threads 10000 -files 10000 with nn.throughput.bench.rpcmode is true (means use rpc to call namenode function) > [IPC] Improvement of lock mechanism in Listener and Reader thread > ----------------------------------------------------------------- > > Key: HADOOP-7105 > URL: https://issues.apache.org/jira/browse/HADOOP-7105 > Project: Hadoop Common > Issue Type: Improvement > Components: ipc > Affects Versions: 0.21.0 > Reporter: jinglong.liujl > Attachments: improveListenerLock.patch, improveListenerLock2.patch > > > In many client cocurrent access, single thread Listener will become > bottleneck. Many client can't be served, and get connection time out. > To improve Listener capacity, we make 2 modification. > 1. Tuning ipc.server.listen.queue.size to a larger value to avoid client > retry. > 2. In currently implement, Listener will call registerChannel(), and > finishAdd() in Reader, which will request Reader synchronized lock. Listener > will cost too many time in waiting for this lock. > We have made test, > ./bin/hadoop org.apache.hadoop.hdfs.NNThroughputBenchmark -op create > -threads 10000 -files 10000 > case 1 : Currently > can not pass. and report > hadoop-rd101.jx.baidu.com/10.65.25.166:59310. Already tried 0 time(s). > case 2 : tuning back log to 10240 > average cost : 1285.72 ms > case 3 : tuning back log to 10240 , and improve lock mechanism in patch > average cost : 941.32 ms > performance in average cost will improve 26% -- This message is automatically generated by JIRA. - For more information on JIRA, see: http://www.atlassian.com/software/jira