[
https://issues.apache.org/jira/browse/HDFS-5276?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13782347#comment-13782347
]
Suresh Srinivas commented on HDFS-5276:
---------------------------------------
bq. At that point, we remove any thread-locals which belong to threads which no
longer exist.
The counts from the threads, even though they are not running any more, should
be included in stats count. Currently statistics object is passed from the
client to the file system. This implementation may need incompatible changes.
> FileSystem.Statistics got performance issue on multi-thread read/write.
> -----------------------------------------------------------------------
>
> Key: HDFS-5276
> URL: https://issues.apache.org/jira/browse/HDFS-5276
> Project: Hadoop HDFS
> Issue Type: Bug
> Affects Versions: 2.0.4-alpha
> Reporter: Chengxiang Li
> Attachments: DisableFSReadWriteBytesStat.patch,
> HDFSStatisticTest.java, hdfs-test.PNG, jstack-trace.PNG
>
>
> FileSystem.Statistics is a singleton variable for each FS scheme, each
> read/write on HDFS would lead to a AutomicLong.getAndAdd(). AutomicLong does
> not perform well in multi-threads(let's say more than 30 threads). so it may
> cause serious performance issue. during our spark test profile, 32 threads
> read data from HDFS, about 70% cpu time is spent on
> FileSystem.Statistics.incrementBytesRead().
--
This message was sent by Atlassian JIRA
(v6.1#6144)
