[
https://issues.apache.org/jira/browse/HDFS-7244?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14182061#comment-14182061
]
Edward Bortnikov commented on HDFS-7244:
----------------------------------------
Colin,
As a matter of fact, Daryn Sharp is working on committing HDFS-6658 and the
subtasks to the trunk. This is a fairly self-contained and stable code, of
independent value. I'd suggest to check it in independently. Daryn - please
chime in, you are the committer.
> Reduce Namenode memory using Flyweight pattern
> ----------------------------------------------
>
> Key: HDFS-7244
> URL: https://issues.apache.org/jira/browse/HDFS-7244
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: namenode
> Reporter: Amir Langer
>
> Using the flyweight pattern can dramatically reduce memory usage in the
> Namenode. The pattern also abstracts the actual storage type and allows the
> decision of whether it is off-heap or not and what is the serialisation
> mechanism to be configured per deployment.
> The idea is to move all BlockInfo data (as a first step) to this storage
> using the Flyweight pattern. The cost to doing it will be in higher latency
> when accessing/modifying a block. The idea is that this will be offset with a
> reduction in memory and in the case of off-heap, a dramatic reduction in
> memory (effectively, memory used for BlockInfo would reduce to a very small
> constant value).
> This reduction will also have an huge impact on the latency as GC pauses will
> be reduced considerably and may even end up with better latency results than
> the original code.
> I wrote a stand-alone project as a proof of concept, to show the pattern, the
> data structure we can use and what will be the performance costs of this
> approach.
> see [Slab|https://github.com/langera/slab]
> and [Slab performance
> results|https://github.com/langera/slab/wiki/Performance-Results].
> Slab abstracts the storage, gives several storage implementations and
> implements the flyweight pattern for the application (Namenode in our case).
> The stages to incorporate Slab into the Namenode is outlined in the sub-tasks
> JIRAs.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
