[
https://issues.apache.org/jira/browse/HDFS-6658?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14376532#comment-14376532
]
Charles Lamb commented on HDFS-6658:
------------------------------------
[~daryn],
I spent a couple of hours making a first pass through the patch.
* The BlockReplicaId encodings seem sufficiently large for the foreseeable
future.
* As you point out in the .jpg of your whiteboard, +1 on getting rid of the
triplets.
* You solve the issue of sparse block ids by converting them to scalars and
maintaining the skipBitSet.
Why did you roll your own bitset instead of using the Java bitset?
I'd like to hear more about concurrency in the overall data structure since
that's a problem that [~cmccabe] and I are trying to tackle. Would you be able
to have a phone conversation on Thursday or Friday this week to discuss it?
> Namenode memory optimization - Block replicas list
> ---------------------------------------------------
>
> Key: HDFS-6658
> URL: https://issues.apache.org/jira/browse/HDFS-6658
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: namenode
> Affects Versions: 2.4.1
> Reporter: Amir Langer
> Assignee: Daryn Sharp
> Attachments: BlockListOptimizationComparison.xlsx, BlocksMap
> redesign.pdf, HDFS-6658.patch, HDFS-6658.patch, HDFS-6658.patch, Namenode
> Memory Optimizations - Block replicas list.docx, New primative indexes.jpg,
> Old triplets.jpg
>
>
> Part of the memory consumed by every BlockInfo object in the Namenode is a
> linked list of block references for every DatanodeStorageInfo (called
> "triplets").
> We propose to change the way we store the list in memory.
> Using primitive integer indexes instead of object references will reduce the
> memory needed for every block replica (when compressed oops is disabled) and
> in our new design the list overhead will be per DatanodeStorageInfo and not
> per block replica.
> see attached design doc. for details and evaluation results.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
