[
https://issues.apache.org/jira/browse/HBASE-15477?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15200240#comment-15200240
]
stack commented on HBASE-15477:
-------------------------------
Commit message. Will post to rb when I do up v2 of this patch.
{code}
When we read from HDFS, we overread to pick up the next blocks header.
Doing this saves a seek as we move through the hfile; we save having to
do an explicit seek just to read the block header every time we need to
read the body. We used to read in the next header as part of the
current blocks buffer. This buffer was then what got persisted to
blockcache; so we were over-persisting: our block plus the next blocks'
header (33 bytes).
This patch undoes this over-persisting.
Removes support for version 1 blocks (0.2 was added in hbase-0.92.0).
Not needed any more.
There is an open question on whether checksums should be persisted
when caching. The code seems to say no but if cache is SSD backed or
backed by anything that does not do error correction, we'll want
checksums.
Adds loads of documentation.
M hbase-common/src/main/java/org/apache/hadoop/hbase/io/hfile/BlockType.java
(write) Add writing from a ByteBuff.
M hbase-common/src/main/java/org/apache/hadoop/hbase/nio/ByteBuff.java
(toString) Add one so ByteBuff looks like ByteBuffer when you click on
it in IDE
M hbase-server/src/main/java/org/apache/hadoop/hbase/io/hfile/HFileBlock.java
Remove support for version 1 blocks.
Cleaned up handling of metadata added when we serialize a block to
caches. Metadata is smaller now.
When we serialize (used when caching), do not persist the next blocks
header if present.
Removed a bunch of methods, a few of which had overlapping
functionality and others that exposed too much of our internals.
Also removed a bunch of constructors and unified the constructors we
had left over making them share a common init method.
Shutdown access to defines that should only be used internally here.
Renamed all to do w/ 'EXTRA' and 'extraSerialization' to instead talk
about metadata saved to caches; was unclear previously what EXTRA was
about.
Renamed static final declarations as all uppercase.
(readBlockDataInternal): Redid. Couldn't make sense of it previously.
Undid heavy-duty parse of header by constructing HFileBlock. Other
cleanups. Its 1/3rd the length it used to be. More to do in here.
diff --git
a/hbase-server/src/main/java/org/apache/hadoop/hbase/io/hfile/HFileScanner.java
b/hbase-server/src/main/java/org/apache/hadoop/hbase/io/hfile/HFileScanner.java
Add deprecations of silly methods.
M
hbase-server/src/main/java/org/apache/hadoop/hbase/io/hfile/bucket/BucketCache.java
Do allocation-saving serialization in an less-intrusive way.
TODO: Can we pass HFileBlock a Writer Interface so it can do
serialization?
M
hbase-server/src/test/java/org/apache/hadoop/hbase/io/hfile/CacheTestUtils.java
Don't use HFileBlock internals
M hbase-server/src/test/java/org/apache/hadoop/hbase/io/hfile/TestChecksum.java
M
hbase-server/src/test/java/org/apache/hadoop/hbase/io/hfile/TestHFileBlock.java
M
hbase-server/src/test/java/org/apache/hadoop/hbase/io/hfile/TestHFileBlockIndex.java
M
hbase-server/src/test/java/org/apache/hadoop/hbase/io/hfile/TestHFileEncryption.java
M
hbase-server/src/test/java/org/apache/hadoop/hbase/io/hfile/TestHFileWriterV3.java
API reading a block changed. No need to pass in version 1 compressed
size anymore.
D
hbase-server/src/test/java/org/apache/hadoop/hbase/io/hfile/TestHFileBlockCompatibility.java
Removed. No support for version 1 anymore.
{code}
> Do not save 'next block header' when we cache hfileblocks
> ---------------------------------------------------------
>
> Key: HBASE-15477
> URL: https://issues.apache.org/jira/browse/HBASE-15477
> Project: HBase
> Issue Type: Sub-task
> Components: BlockCache, Performance
> Reporter: stack
> Assignee: stack
> Attachments: 15477.patch
>
>
> When we read from HDFS, we overread to pick up the next blocks header.
> Doing this saves a seek as we move through the hfile; we save having to
> do an explicit seek just to read the block header every time we need to
> read the body. We used to read in the next header as part of the
> current blocks buffer. This buffer was then what got persisted to
> blockcache; so we were over-persisting wrtiting out our block plus the
> next blocks' header (overpersisting 33 bytes). Parse of HFileBlock
> complicated by this extra tail. Fix.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
