[
https://issues.apache.org/jira/browse/HBASE-3857?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13031489#comment-13031489
]
Liyin Tang commented on HBASE-3857:
-----------------------------------
Let me try to answer this:)
1) When writing the HFile, the writer will hold the current inline block index
into memory. If this inline block index is larger than a threshold (default
128KB), it will flush the index into disk. The benefit here is to avoid hold
all the block index during the write operation.
2) But after writing all the data block and the block index is still smaller a
threshold number, there will be only one level of block index, which is the
root level and it is the same as before. So multi-level block index is totally
optional based on block index size.
3) After flushing the each of inline block index onto disk, the Writer will
generate its parent level of block index on the fly, which will point to the
offset of these inline block index. It will be called intermediate level. So
only the leaf level of block index will point to the data block, other level
block index will point to its children level block index. Also for each level
of block index, it will be partitioned into blocks as well. Each index block is
a chunk of block index, which will have the same format of data block.
4) If the intermediate level of block is still larger than the threshold (128
KB), it will generate its parent level. Keep doing this until the current level
of block index is smaller than the threshold. The last one is called root level
block index, which is smaller than the threshold. It works like generating a
balanced tree from bottom to up until the root level is smaller than the
threshold.
5) So when reading the file, it will load back the root level of block index
and hold it into memory. When Reader calls seekTo a key, it will lookup the
tree from root to leaf to find out the data block contains this key. All the
index block will be cached during the searching.
Please let me know there is any concerns about the block index:)
Thanks a lot
> Change the HFile Format
> -----------------------
>
> Key: HBASE-3857
> URL: https://issues.apache.org/jira/browse/HBASE-3857
> Project: HBase
> Issue Type: New Feature
> Reporter: Liyin Tang
> Assignee: Mikhail Bautin
> Attachments: hfile_format_v2_design_draft_0.1.pdf
>
>
> In order to support HBASE-3763 and HBASE-3856, we need to change the format
> of the HFile. The new format proposal is attached here. Thanks for Mikhail
> Bautin for the documentation.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
