[
https://issues.apache.org/jira/browse/HDFS-2243?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13082641#comment-13082641
]
Todd Lipcon commented on HDFS-2243:
-----------------------------------
Hi Eric. HDFS-918 addresses part of this problem. HDFS-941 also addresses it
substantially for the HBase workload.
> DataXceiver per accept seems to be a bottleneck in HBase/YCSB test
> ------------------------------------------------------------------
>
> Key: HDFS-2243
> URL: https://issues.apache.org/jira/browse/HDFS-2243
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: data-node
> Affects Versions: 0.23.0
> Environment: Using Fedora 14 on a quad core phenom system
> Reporter: Eric Caspole
> Priority: Minor
>
> I am running the YCSB benchmark against HBase, sometimes against a single
> node, sometimes against a cluster of 6 systems. As the load increases into
> thousands of TPS, especially on the single node, I can see that the datanode
> runs very high system time and seems to be bottlenecked by how fast it can
> create the threads to handle the new connections in DataXceiverServer.run. By
> "perf top" I can see the process spends about 12% of all its time in
> pthread_create, and in hprof profiles I can see there are tens of thousands
> of threads created in just a few minutes of test execution.
> Does anyone else observe this bottleneck? Is there a major challenge to using
> a thread pool of DataXceivers in this situation?
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
