[jira] Commented: (HDFS-1110) Namenode heap optimization - reuse objects for commonly used file names

[ https://issues.apache.org/jira/browse/HDFS-1110?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12874117#action_12874117 ] Suresh Srinivas commented on HDFS-1110: --- Dhruba or Jakob, can you please comment... >

[jira] Commented: (HDFS-1110) Namenode heap optimization - reuse objects for commonly used file names

[ https://issues.apache.org/jira/browse/HDFS-1110?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12875240#action_12875240 ] Jakob Homan commented on HDFS-1110: --- Looking good. Review: * If you keep {{NamespaceDedup

[jira] Commented: (HDFS-1110) Namenode heap optimization - reuse objects for commonly used file names

[ https://issues.apache.org/jira/browse/HDFS-1110?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12875247#action_12875247 ] dhruba borthakur commented on HDFS-1110: > The 12 second startup cost compared to th

[jira] Commented: (HDFS-1110) Namenode heap optimization - reuse objects for commonly used file names

[ https://issues.apache.org/jira/browse/HDFS-1110?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12875282#action_12875282 ] Suresh Srinivas commented on HDFS-1110: --- Thanks Jakob. bq. The NameDictionary.lookup(

[jira] Commented: (HDFS-1110) Namenode heap optimization - reuse objects for commonly used file names

[ https://issues.apache.org/jira/browse/HDFS-1110?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12875285#action_12875285 ] Suresh Srinivas commented on HDFS-1110: --- Alternatively we can follow the convention in

[jira] Commented: (HDFS-1110) Namenode heap optimization - reuse objects for commonly used file names

[ https://issues.apache.org/jira/browse/HDFS-1110?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12875356#action_12875356 ] Jakob Homan commented on HDFS-1110: --- bq. Change the method name from lookup to put That so

[jira] Commented: (HDFS-1110) Namenode heap optimization - reuse objects for commonly used file names

[ https://issues.apache.org/jira/browse/HDFS-1110?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12875805#action_12875805 ] Konstantin Shvachko commented on HDFS-1110: --- # variable {{cache}} should read {{na

[jira] Commented: (HDFS-1110) Namenode heap optimization - reuse objects for commonly used file names

[ https://issues.apache.org/jira/browse/HDFS-1110?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12875813#action_12875813 ] Hadoop QA commented on HDFS-1110: - -1 overall. Here are the results of testing the latest a

[jira] Commented: (HDFS-1110) Namenode heap optimization - reuse objects for commonly used file names

[ https://issues.apache.org/jira/browse/HDFS-1110?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12876949#action_12876949 ] Hadoop QA commented on HDFS-1110: - -1 overall. Here are the results of testing the latest a

[jira] Commented: (HDFS-1110) Namenode heap optimization - reuse objects for commonly used file names

[ https://issues.apache.org/jira/browse/HDFS-1110?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12877287#action_12877287 ] Suresh Srinivas commented on HDFS-1110: --- Failed tests TestBalancer and TestLargeDirect

[jira] Commented: (HDFS-1110) Namenode heap optimization - reuse objects for commonly used file names

[ https://issues.apache.org/jira/browse/HDFS-1110?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12877348#action_12877348 ] Konstantin Shvachko commented on HDFS-1110: --- +1 on the patch. We also talked abou

[jira] Commented: (HDFS-1110) Namenode heap optimization - reuse objects for commonly used file names

[ https://issues.apache.org/jira/browse/HDFS-1110?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12877639#action_12877639 ] Konstantin Shvachko commented on HDFS-1110: --- +1 for the patch for 0.20 > Namenode

[jira] Commented: (HDFS-1110) Namenode heap optimization - reuse objects for commonly used file names

[ https://issues.apache.org/jira/browse/HDFS-1110?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12879101#action_12879101 ] Hudson commented on HDFS-1110: -- Integrated in Hadoop-Hdfs-trunk-Commit #310 (See [http://hudso

[jira] Commented: (HDFS-1110) Namenode heap optimization - reuse objects for commonly used file names

[ https://issues.apache.org/jira/browse/HDFS-1110?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12880355#action_12880355 ] Tsz Wo (Nicholas), SZE commented on HDFS-1110: -- Hi Suresh, new commit should be

[jira] Commented: (HDFS-1110) Namenode heap optimization - reuse objects for commonly used file names

[ https://issues.apache.org/jira/browse/HDFS-1110?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12880358#action_12880358 ] Suresh Srinivas commented on HDFS-1110: --- Sure. Next time I commit some thing I will fi

[jira] Commented: (HDFS-1110) Namenode heap optimization - reuse objects for commonly used file names

[ https://issues.apache.org/jira/browse/HDFS-1110?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12861042#action_12861042 ] Suresh Srinivas commented on HDFS-1110: --- Approach: - Commonly used file names are defi

[jira] Commented: (HDFS-1110) Namenode heap optimization - reuse objects for commonly used file names

[ https://issues.apache.org/jira/browse/HDFS-1110?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12861533#action_12861533 ] dhruba borthakur commented on HDFS-1110: Hi suresh, awesome idea. Can we somehow mak

[jira] Commented: (HDFS-1110) Namenode heap optimization - reuse objects for commonly used file names

[ https://issues.apache.org/jira/browse/HDFS-1110?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12861548#action_12861548 ] Suresh Srinivas commented on HDFS-1110: --- Here is an analysis of an fsimage on one of Y

[jira] Commented: (HDFS-1110) Namenode heap optimization - reuse objects for commonly used file names

[ https://issues.apache.org/jira/browse/HDFS-1110?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12861549#action_12861549 ] Suresh Srinivas commented on HDFS-1110: --- Dhruba, see my comments on what can be done i

[jira] Commented: (HDFS-1110) Namenode heap optimization - reuse objects for commonly used file names

[ https://issues.apache.org/jira/browse/HDFS-1110?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12861553#action_12861553 ] Suresh Srinivas commented on HDFS-1110: --- For the above table, 448832 file names are us

[jira] Commented: (HDFS-1110) Namenode heap optimization - reuse objects for commonly used file names

[ https://issues.apache.org/jira/browse/HDFS-1110?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12861567#action_12861567 ] Jakob Homan commented on HDFS-1110: --- bq. Build a tool to generate list names that is used

[jira] Commented: (HDFS-1110) Namenode heap optimization - reuse objects for commonly used file names

[ https://issues.apache.org/jira/browse/HDFS-1110?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12862431#action_12862431 ] Konstantin Shvachko commented on HDFS-1110: --- bq. File names used > 10 times

[jira] Commented: (HDFS-1110) Namenode heap optimization - reuse objects for commonly used file names

[ https://issues.apache.org/jira/browse/HDFS-1110?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12862783#action_12862783 ] Eli Collins commented on HDFS-1110: --- Won't this require a lot of synchronization that prev

[jira] Commented: (HDFS-1110) Namenode heap optimization - reuse objects for commonly used file names

[ https://issues.apache.org/jira/browse/HDFS-1110?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12862842#action_12862842 ] Suresh Srinivas commented on HDFS-1110: --- Access to the map is only during write operat

[jira] Commented: (HDFS-1110) Namenode heap optimization - reuse objects for commonly used file names

[ https://issues.apache.org/jira/browse/HDFS-1110?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12863103#action_12863103 ] dhruba borthakur commented on HDFS-1110: hi suresh, are you saying that we save 470M

[jira] Commented: (HDFS-1110) Namenode heap optimization - reuse objects for commonly used file names

[ https://issues.apache.org/jira/browse/HDFS-1110?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12865661#action_12865661 ] Suresh Srinivas commented on HDFS-1110: --- 470MB was based on my rough calculation. Look

[jira] Commented: (HDFS-1110) Namenode heap optimization - reuse objects for commonly used file names

[ https://issues.apache.org/jira/browse/HDFS-1110?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12865889#action_12865889 ] dhruba borthakur commented on HDFS-1110: Thanks Suresh for the numbers. > agree wit

[jira] Commented: (HDFS-1110) Namenode heap optimization - reuse objects for commonly used file names

[ https://issues.apache.org/jira/browse/HDFS-1110?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12866052#action_12866052 ] Suresh Srinivas commented on HDFS-1110: --- bq. agree with Dhruba that we need to optimiz

[jira] Commented: (HDFS-1110) Namenode heap optimization - reuse objects for commonly used file names

[ https://issues.apache.org/jira/browse/HDFS-1110?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12866060#action_12866060 ] dhruba borthakur commented on HDFS-1110: Awesome. I like this idea because it has no

[jira] Commented: (HDFS-1110) Namenode heap optimization - reuse objects for commonly used file names

[ https://issues.apache.org/jira/browse/HDFS-1110?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12867064#action_12867064 ] Suresh Srinivas commented on HDFS-1110: --- bq. Maybe you can have only one dictionary wh