[
https://issues.apache.org/jira/browse/HDFS-7844?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14347903#comment-14347903
]
Colin Patrick McCabe commented on HDFS-7844:
--------------------------------------------
{{CommonConfigurationKeys.java}}: add {{hadoop.memory.manager}} configuration
key, which controls whether to use on-heap or off-heap (basically)
{{MemoryManager.java}}: inferface which all memory managers implement. Factory
methods for creating one from a Hadoop {{Configuration}}.
{{ByteArrayMemoryManager.java}}: a "safe" on-heap memory manager. Does
extensive verification of memory accesses to ensure that they're valid.
{{NativeMemoryManager.java}}: off-heap memory manager which delegates to
{{Unsafe}}, which basically calls {{malloc}}.
{{ProbingHashSet.java}}: an off-heap hash table implementation. It uses
probing rather than separate chaining (as suggested by the name), and doubles
in size when it becomes more than half full, to maintain O(1) access.
{{ProbingHashSet#Adaptor}}: a class which deals with storing and loading
entries from the hash table. In the case of the block map, this just means
putting a single 8-byte long (the address of the off-heap BlocksInfo data) into
the correct place in the hash table.
{{ProbingHashSet#Key}}: these are keys that can be compared and so forth. Used
to search for an element, and when we have an element, these are used to
determine what it's hash code is and what else it is identical to.
There is also an iterator provided which can iterate over the whole hash
table... very similar to HashTable#Iterator. One twist is that it can still be
used if the table is modified after the iterator is created. It will return
"reduced consistency" results in that case, but still be useful for many cases.
> Create an off-heap hash table implementation
> --------------------------------------------
>
> Key: HDFS-7844
> URL: https://issues.apache.org/jira/browse/HDFS-7844
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Affects Versions: HDFS-7836
> Reporter: Colin Patrick McCabe
> Assignee: Colin Patrick McCabe
> Attachments: HDFS-7844-scl.001.patch
>
>
> Create an off-heap hash table implementation.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
