[
https://issues.apache.org/jira/browse/HADOOP-803?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Raghu Angadi updated HADOOP-803:
--------------------------------
Attachment: NameNodeMemoryHogs.txt
see attached NameNodeMemoryHogs.txt for better formated list. Following is
cut-n-paste of the attachment.
Main FS wide datastructures :
-----------------------------
blockMap : block --> containing nodes : HashMap
activeBLocks : block --> INode : HashMap
INode.children : FileName --> INode : TreeMap (in dir INode).
DN.blocks : block --> self : TreeMap ( in each DataNodeDescriptor )
datanodeMap : String --> DatanodeDescriptor ( linear cost with
number of dataNodes )
Memory cost per Block :
----------------------
Before this patch :
Block Objects : 3 ( one per replica in DN.blocks)
1 ( in blockMap )
1 ( in INode );
TreeMap : 1 ( containingNode TreeSet )
TreeMap$Entry : 3 ( conatiningNodes TreeSet in blockMap )
3 ( in DN.blocks in each of the nodes )
HashMap$Entry : 1 ( blockMap )
1 ( activeBlocks )
After this patch :
Block Objects : 1 ( in blockMap )
1 ( in INode that existed before NameNode restarted )
ArrayList : 1 ( containingNodes Arraylist )
TreeMap$Entry : 3 ( in DN.blocks in each of the nodes )
HashMap$Entry : 1 ( blockMap )
1 ( activeBlocks )
Memory cost per INode :
----------------------
TreeMap : 1 ( children. removed in this issue for regular files )
TreeMap$Entry : 1 ( in parent.children )
INode object : 1
char[128] : 1 (char[] used in String)
String : 1 ( name )
> Reducing memory consumption on Namenode : Part 1
> ------------------------------------------------
>
> Key: HADOOP-803
> URL: https://issues.apache.org/jira/browse/HADOOP-803
> Project: Hadoop
> Issue Type: Bug
> Components: dfs
> Reporter: Raghu Angadi
> Assigned To: Raghu Angadi
> Fix For: 0.11.0
>
> Attachments: block-refs-2.patch, block-refs-3.patch,
> block-refs-5.patch, HADOOP-803-2.patch, HADOOP-803.patch, HADOOP-803_3.patch,
> NameNodeMemoryHogs.txt
>
>
> There appears to be some places in Namenode that allow reducing memory
> consumption without intrusive code or feature changes. This bug is an initial
> attempt making those changes. Please include your thoughts as well.
> One change I am planning to make :
> Currently one copy of each block exists for each of the replicas and one copy
> for blockMap. I think they are all supposed to be same.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
