[ https://issues.apache.org/jira/browse/HDFS-2243?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13082654#comment-13082654 ]
Eric Caspole commented on HDFS-2243: ------------------------------------ Last week I tried the patch called "hdfs-918-branch20-append.patch" but it didn't seem to help this problem, I am using what seems to be the recommended hdfs for hbase 0.90.3. I can upgrade to hdfs 0.22, it should work with hbase 0.90.3, right? I will upload my screen shot from perf, it was doing 1000tps at that point. I have not seen epoll taking time, at least on that testbed. > DataXceiver per accept seems to be a bottleneck in HBase/YCSB test > ------------------------------------------------------------------ > > Key: HDFS-2243 > URL: https://issues.apache.org/jira/browse/HDFS-2243 > Project: Hadoop HDFS > Issue Type: Bug > Components: data-node > Affects Versions: 0.23.0 > Environment: Using Fedora 14 on a quad core phenom system > Reporter: Eric Caspole > Priority: Minor > > I am running the YCSB benchmark against HBase, sometimes against a single > node, sometimes against a cluster of 6 systems. As the load increases into > thousands of TPS, especially on the single node, I can see that the datanode > runs very high system time and seems to be bottlenecked by how fast it can > create the threads to handle the new connections in DataXceiverServer.run. By > "perf top" I can see the process spends about 12% of all its time in > pthread_create, and in hprof profiles I can see there are tens of thousands > of threads created in just a few minutes of test execution. > Does anyone else observe this bottleneck? Is there a major challenge to using > a thread pool of DataXceivers in this situation? -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira