[IPC] Improvement of lock mechanism in Listener and Reader thread
-----------------------------------------------------------------
Key: HADOOP-7105
URL: https://issues.apache.org/jira/browse/HADOOP-7105
Project: Hadoop Common
Issue Type: Improvement
Components: ipc
Affects Versions: 0.21.0
Reporter: jinglong.liujl
Attachments: improveListenerLock.patch
In many client cocurrent access, single thread Listener will become bottleneck.
Many client can't be served, and get connection time out.
To improve Listener capacity, we make 2 modification.
1. Tuning ipc.server.listen.queue.size to a larger value to avoid client retry.
2. In currently implement, Listener will call registerChannel(), and
finishAdd() in Reader, which will request Reader synchronized lock. Listener
will cost too many time in waiting for this lock.
We have made test,
./bin/hadoop org.apache.hadoop.hdfs.NNThroughputBenchmark -op create -threads
10000 -files 10000
case 1 : Currently
can not pass. and report
hadoop-rd101.jx.baidu.com/10.65.25.166:59310. Already tried 0 time(s).
case 2 : tuning back log to 10240
average cost : 1285.72 ms
case 3 : tuning back log to 10240 , and improve lock mechanism in patch
average cost : 941.32 ms
performance in average cost will improve 26%
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
