[
https://issues.apache.org/jira/browse/HBASE-21917?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16771501#comment-16771501
]
Zheng Hu commented on HBASE-21917:
----------------------------------
bq. IIRC, we want native checksum checking to cut in. Is that happening here?
If it is, small buffers means we cross the native divide many times – each of
which costs – rather than a few times?
Thanks [~stack], yeah, you are right. before this patch , the hadoop's
verifyChunkedSums will try to use native checksum checking if NativeCrc32 is
available. but the bad news is: NativeCrc32 is not a public class in
hadoop-common, our HBase has no access. so i can only uniform those logics
into a general checksum method as the patch described. BTW, the
hadoop-common's checksum looks like this:
{code}
public void verifyChunkedSums(ByteBuffer data, ByteBuffer checksums,
String fileName, long basePos)
throws ChecksumException {
if (type.size == 0) return;
if (data.hasArray() && checksums.hasArray()) {
verifyChunkedSums(
data.array(), data.arrayOffset() + data.position(), data.remaining(),
checksums.array(), checksums.arrayOffset() + checksums.position(),
fileName, basePos); // -------> private method, no access
return;
}
if (NativeCrc32.isAvailable()) {
NativeCrc32.verifyChunkedSums(bytesPerChecksum, type.id, checksums, data,
fileName, basePos); -------> NativeCrc32 is not public, no access.
return;
}
// ...general ByteBuffer checksum verify logic...
{code}
> Make the HFileBlock#validateChecksum can accept ByteBuff as an input.
> ---------------------------------------------------------------------
>
> Key: HBASE-21917
> URL: https://issues.apache.org/jira/browse/HBASE-21917
> Project: HBase
> Issue Type: Sub-task
> Reporter: Zheng Hu
> Assignee: Zheng Hu
> Priority: Major
> Attachments: HBASE-21917.v1.patch, HBASE-21917.v2.patch,
> HBASE-21917.v3.patch, HBASE-21917.v4.patch
>
>
> I've tried to make a patch for HBASE-21879, most of work seems to be fine,
> but the trouble is:
> HFileBlock#validateChecksum can only accept ByteBuffer as its input, while
> after the HBASE-21916, we will use an ourself-defined ByteBuff (which can be
> SingleByteBuff or MultiByteBuff).
> Now, need to create our own ByteBuff checksum validation method, should not
> be so hard but an separate issue will be more clearer.
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
