[ https://issues.apache.org/jira/browse/HDFS-7244?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Amir Langer updated HDFS-7244: ------------------------------ Attachment: SampleCode.tar.gz HDFS-7244.docx attached doc that describes an outline of the idea + Some sample code of how can we use the slab to store BlockInfo objects. > Reduce Namenode memory using Flyweight pattern > ---------------------------------------------- > > Key: HDFS-7244 > URL: https://issues.apache.org/jira/browse/HDFS-7244 > Project: Hadoop HDFS > Issue Type: Improvement > Components: namenode > Reporter: Amir Langer > Attachments: HDFS-7244.docx, SampleCode.tar.gz > > > Using the flyweight pattern can dramatically reduce memory usage in the > Namenode. The pattern also abstracts the actual storage type and allows the > decision of whether it is off-heap or not and what is the serialisation > mechanism to be configured per deployment. > The idea is to move all BlockInfo data (as a first step) to this storage > using the Flyweight pattern. The cost to doing it will be in higher latency > when accessing/modifying a block. The idea is that this will be offset with a > reduction in memory and in the case of off-heap, a dramatic reduction in > memory (effectively, memory used for BlockInfo would reduce to a very small > constant value). > This reduction will also have an huge impact on the latency as GC pauses will > be reduced considerably and may even end up with better latency results than > the original code. > I wrote a stand-alone project as a proof of concept, to show the pattern, the > data structure we can use and what will be the performance costs of this > approach. > see [Slab|https://github.com/langera/slab] > and [Slab performance > results|https://github.com/langera/slab/wiki/Performance-Results]. > Slab abstracts the storage, gives several storage implementations and > implements the flyweight pattern for the application (Namenode in our case). > The stages to incorporate Slab into the Namenode is outlined in the sub-tasks > JIRAs. -- This message was sent by Atlassian JIRA (v6.3.4#6332)