[jira] [Commented] (HDFS-5276) FileSystem.Statistics got performance issue on multi-thread read/write.

[ https://issues.apache.org/jira/browse/HDFS-5276?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13799898#comment-13799898 ] Hudson commented on HDFS-5276: -- FAILURE: Integrated in Hadoop-Mapreduce-trunk #1583 (See [htt

[jira] [Commented] (HDFS-5276) FileSystem.Statistics got performance issue on multi-thread read/write.

[ https://issues.apache.org/jira/browse/HDFS-5276?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13799868#comment-13799868 ] Hudson commented on HDFS-5276: -- FAILURE: Integrated in Hadoop-Hdfs-trunk #1557 (See [https://

[jira] [Commented] (HDFS-5276) FileSystem.Statistics got performance issue on multi-thread read/write.

[ https://issues.apache.org/jira/browse/HDFS-5276?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13799858#comment-13799858 ] Hudson commented on HDFS-5276: -- SUCCESS: Integrated in Hadoop-Yarn-trunk #367 (See [https://b

[jira] [Commented] (HDFS-5276) FileSystem.Statistics got performance issue on multi-thread read/write.

[ https://issues.apache.org/jira/browse/HDFS-5276?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13799689#comment-13799689 ] Hudson commented on HDFS-5276: -- SUCCESS: Integrated in Hadoop-trunk-Commit #4632 (See [https:

[jira] [Commented] (HDFS-5276) FileSystem.Statistics got performance issue on multi-thread read/write.

[ https://issues.apache.org/jira/browse/HDFS-5276?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13799625#comment-13799625 ] Colin Patrick McCabe commented on HDFS-5276: I ran mvn eclipse:eclipse locally

[jira] [Commented] (HDFS-5276) FileSystem.Statistics got performance issue on multi-thread read/write.

[ https://issues.apache.org/jira/browse/HDFS-5276?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13794772#comment-13794772 ] Hadoop QA commented on HDFS-5276: - {color:red}-1 overall{color}. Here are the results of t

[jira] [Commented] (HDFS-5276) FileSystem.Statistics got performance issue on multi-thread read/write.

[ https://issues.apache.org/jira/browse/HDFS-5276?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13794744#comment-13794744 ] Andrew Wang commented on HDFS-5276: --- Cool, that all works for me. +1 on the new patch; ma

[jira] [Commented] (HDFS-5276) FileSystem.Statistics got performance issue on multi-thread read/write.

[ https://issues.apache.org/jira/browse/HDFS-5276?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13794740#comment-13794740 ] Colin Patrick McCabe commented on HDFS-5276: thanks for the review, Andrew. bq

[jira] [Commented] (HDFS-5276) FileSystem.Statistics got performance issue on multi-thread read/write.

[ https://issues.apache.org/jira/browse/HDFS-5276?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13794380#comment-13794380 ] Andrew Wang commented on HDFS-5276: --- I really like how this has shaped up. The microbench

[jira] [Commented] (HDFS-5276) FileSystem.Statistics got performance issue on multi-thread read/write.

[ https://issues.apache.org/jira/browse/HDFS-5276?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13794299#comment-13794299 ] Colin Patrick McCabe commented on HDFS-5276: As Binglin pointed out, you cannot

[jira] [Commented] (HDFS-5276) FileSystem.Statistics got performance issue on multi-thread read/write.

[ https://issues.apache.org/jira/browse/HDFS-5276?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13793920#comment-13793920 ] Binglin Chang commented on HDFS-5276: - We can see in no contention case(1 thread): 100M

[jira] [Commented] (HDFS-5276) FileSystem.Statistics got performance issue on multi-thread read/write.

[ https://issues.apache.org/jira/browse/HDFS-5276?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13793917#comment-13793917 ] Binglin Chang commented on HDFS-5276: - My test env (Macbook Pro i7 4core 8 thread hyper

[jira] [Commented] (HDFS-5276) FileSystem.Statistics got performance issue on multi-thread read/write.

[ https://issues.apache.org/jira/browse/HDFS-5276?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13793537#comment-13793537 ] Binglin Chang commented on HDFS-5276: - Hi [~wheat9], volatile cannot be used as atomic

[jira] [Commented] (HDFS-5276) FileSystem.Statistics got performance issue on multi-thread read/write.

[ https://issues.apache.org/jira/browse/HDFS-5276?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13793430#comment-13793430 ] Haohui Mai commented on HDFS-5276: -- [~decster], that's quite impressive. It might be worth

[jira] [Commented] (HDFS-5276) FileSystem.Statistics got performance issue on multi-thread read/write.

[ https://issues.apache.org/jira/browse/HDFS-5276?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13793357#comment-13793357 ] Hudson commented on HDFS-5276: -- FAILURE: Integrated in Hadoop-Mapreduce-trunk #1576 (See [htt

[jira] [Commented] (HDFS-5276) FileSystem.Statistics got performance issue on multi-thread read/write.

[ https://issues.apache.org/jira/browse/HDFS-5276?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13793348#comment-13793348 ] Hudson commented on HDFS-5276: -- FAILURE: Integrated in Hadoop-Hdfs-trunk #1550 (See [https://

[jira] [Commented] (HDFS-5276) FileSystem.Statistics got performance issue on multi-thread read/write.

[ https://issues.apache.org/jira/browse/HDFS-5276?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13793332#comment-13793332 ] Hudson commented on HDFS-5276: -- SUCCESS: Integrated in Hadoop-Yarn-trunk #360 (See [https://b

[jira] [Commented] (HDFS-5276) FileSystem.Statistics got performance issue on multi-thread read/write.

[ https://issues.apache.org/jira/browse/HDFS-5276?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13793300#comment-13793300 ] Hadoop QA commented on HDFS-5276: - {color:red}-1 overall{color}. Here are the results of t

[jira] [Commented] (HDFS-5276) FileSystem.Statistics got performance issue on multi-thread read/write.

[ https://issues.apache.org/jira/browse/HDFS-5276?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13793297#comment-13793297 ] Binglin Chang commented on HDFS-5276: - Did a micro-benchmark only on FileSystem.Statist

[jira] [Commented] (HDFS-5276) FileSystem.Statistics got performance issue on multi-thread read/write.

[ https://issues.apache.org/jira/browse/HDFS-5276?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13793267#comment-13793267 ] Haohui Mai commented on HDFS-5276: -- [~chengxiang li], is it possible to rerun the test wit

[jira] [Commented] (HDFS-5276) FileSystem.Statistics got performance issue on multi-thread read/write.

[ https://issues.apache.org/jira/browse/HDFS-5276?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13793264#comment-13793264 ] Hudson commented on HDFS-5276: -- SUCCESS: Integrated in Hadoop-trunk-Commit #4591 (See [https:

[jira] [Commented] (HDFS-5276) FileSystem.Statistics got performance issue on multi-thread read/write.

[ https://issues.apache.org/jira/browse/HDFS-5276?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13792839#comment-13792839 ] Colin Patrick McCabe commented on HDFS-5276: It looks like someone confused con

[jira] [Commented] (HDFS-5276) FileSystem.Statistics got performance issue on multi-thread read/write.

[ https://issues.apache.org/jira/browse/HDFS-5276?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13792641#comment-13792641 ] Hudson commented on HDFS-5276: -- FAILURE: Integrated in Hadoop-Mapreduce-trunk #1575 (See [htt

[jira] [Commented] (HDFS-5276) FileSystem.Statistics got performance issue on multi-thread read/write.

[ https://issues.apache.org/jira/browse/HDFS-5276?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13792605#comment-13792605 ] Hudson commented on HDFS-5276: -- FAILURE: Integrated in Hadoop-Hdfs-trunk #1549 (See [https://

[jira] [Commented] (HDFS-5276) FileSystem.Statistics got performance issue on multi-thread read/write.

[ https://issues.apache.org/jira/browse/HDFS-5276?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13792523#comment-13792523 ] Hudson commented on HDFS-5276: -- SUCCESS: Integrated in Hadoop-Yarn-trunk #359 (See [https://b

[jira] [Commented] (HDFS-5276) FileSystem.Statistics got performance issue on multi-thread read/write.

[ https://issues.apache.org/jira/browse/HDFS-5276?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13792467#comment-13792467 ] Hudson commented on HDFS-5276: -- SUCCESS: Integrated in Hadoop-trunk-Commit #4585 (See [https:

[jira] [Commented] (HDFS-5276) FileSystem.Statistics got performance issue on multi-thread read/write.

[ https://issues.apache.org/jira/browse/HDFS-5276?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13792027#comment-13792027 ] Hadoop QA commented on HDFS-5276: - {color:green}+1 overall{color}. Here are the results of

[jira] [Commented] (HDFS-5276) FileSystem.Statistics got performance issue on multi-thread read/write.

[ https://issues.apache.org/jira/browse/HDFS-5276?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13791917#comment-13791917 ] Hadoop QA commented on HDFS-5276: - {color:red}-1 overall{color}. Here are the results of t

[jira] [Commented] (HDFS-5276) FileSystem.Statistics got performance issue on multi-thread read/write.

[ https://issues.apache.org/jira/browse/HDFS-5276?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13791120#comment-13791120 ] Binglin Chang commented on HDFS-5276: - The article I refered before [http://java.dzone

[jira] [Commented] (HDFS-5276) FileSystem.Statistics got performance issue on multi-thread read/write.

[ https://issues.apache.org/jira/browse/HDFS-5276?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13788821#comment-13788821 ] Chengxiang Li commented on HDFS-5276: - here is my test environment: [root@node13-1 ~]#

[jira] [Commented] (HDFS-5276) FileSystem.Statistics got performance issue on multi-thread read/write.

[ https://issues.apache.org/jira/browse/HDFS-5276?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13783052#comment-13783052 ] Binglin Chang commented on HDFS-5276: - [~chengxiang li], could you provide your test en

[jira] [Commented] (HDFS-5276) FileSystem.Statistics got performance issue on multi-thread read/write.

[ https://issues.apache.org/jira/browse/HDFS-5276?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13783009#comment-13783009 ] Binglin Chang commented on HDFS-5276: - bq. Why not keep thread-local read statistics an

[jira] [Commented] (HDFS-5276) FileSystem.Statistics got performance issue on multi-thread read/write.

[ https://issues.apache.org/jira/browse/HDFS-5276?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13782500#comment-13782500 ] Colin Patrick McCabe commented on HDFS-5276: bq. The counts from the threads, e

[jira] [Commented] (HDFS-5276) FileSystem.Statistics got performance issue on multi-thread read/write.

[ https://issues.apache.org/jira/browse/HDFS-5276?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13782443#comment-13782443 ] Haohui Mai commented on HDFS-5276: -- And I'm still trying to understand the rational of add

[jira] [Commented] (HDFS-5276) FileSystem.Statistics got performance issue on multi-thread read/write.

[ https://issues.apache.org/jira/browse/HDFS-5276?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13782394#comment-13782394 ] Haohui Mai commented on HDFS-5276: -- [~chengxiang li], can you post the detailed configurat

[jira] [Commented] (HDFS-5276) FileSystem.Statistics got performance issue on multi-thread read/write.

[ https://issues.apache.org/jira/browse/HDFS-5276?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13782347#comment-13782347 ] Suresh Srinivas commented on HDFS-5276: --- bq. At that point, we remove any thread-loca

[jira] [Commented] (HDFS-5276) FileSystem.Statistics got performance issue on multi-thread read/write.

[ https://issues.apache.org/jira/browse/HDFS-5276?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13782307#comment-13782307 ] Colin Patrick McCabe commented on HDFS-5276: bq. How do you know all the thread

[jira] [Commented] (HDFS-5276) FileSystem.Statistics got performance issue on multi-thread read/write.

[ https://issues.apache.org/jira/browse/HDFS-5276?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13782127#comment-13782127 ] Suresh Srinivas commented on HDFS-5276: --- bq. about 70% cpu time is spent on FileSyste

[jira] [Commented] (HDFS-5276) FileSystem.Statistics got performance issue on multi-thread read/write.

[ https://issues.apache.org/jira/browse/HDFS-5276?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13782126#comment-13782126 ] Suresh Srinivas commented on HDFS-5276: --- bq. Increment thread-locals, sum them up on

[jira] [Commented] (HDFS-5276) FileSystem.Statistics got performance issue on multi-thread read/write.

[ https://issues.apache.org/jira/browse/HDFS-5276?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13782107#comment-13782107 ] Andrew Wang commented on HDFS-5276: --- +1 for Colin's suggestion, that was my first thought

[jira] [Commented] (HDFS-5276) FileSystem.Statistics got performance issue on multi-thread read/write.

[ https://issues.apache.org/jira/browse/HDFS-5276?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13782058#comment-13782058 ] Colin Patrick McCabe commented on HDFS-5276: Why not keep thread-local read sta

[jira] [Commented] (HDFS-5276) FileSystem.Statistics got performance issue on multi-thread read/write.

[ https://issues.apache.org/jira/browse/HDFS-5276?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13781656#comment-13781656 ] Binglin Chang commented on HDFS-5276: - Reading so many HDFS files in parallel in one pr

[jira] [Commented] (HDFS-5276) FileSystem.Statistics got performance issue on multi-thread read/write.

[ https://issues.apache.org/jira/browse/HDFS-5276?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13781317#comment-13781317 ] Binglin Chang commented on HDFS-5276: - Hi Chengxiang, could you please provide your tes