[ https://issues.apache.org/jira/browse/HBASE-8323?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13814737#comment-13814737 ]
Nicolas Liochon commented on HBASE-8323: ---------------------------------------- I saw it while looking at a server. The class NativeCrc32 is not public in hadoop common, (while PureJavaCrc32C is public, in the java & hadoop meanings of the terms). The easiest way for HBase would be to reuse directly NativeCrc32... [~t...@lipcon.org], is there any problem if NativeCrc32 becomes public, or at least LimitedPrivate? > Low hanging checksum improvements > --------------------------------- > > Key: HBASE-8323 > URL: https://issues.apache.org/jira/browse/HBASE-8323 > Project: HBase > Issue Type: Improvement > Components: Performance > Reporter: Enis Soztutar > > Over at Hadoop land, [~tlipcon] had done some improvements for checksums, a > native implementation for CRC32C (HADOOP-7445) and bulk verify of checksums > (HADOOP-7444). > In HBase, we can do > - Also develop a bulk verify API. Regardless of > hbase.hstore.bytes.per.checksum we always want to verify of the whole > checksum for the hfile block. > - Enable NativeCrc32 to be used as a checksum algo. It is not clear how much > gain we can expect over pure java CRC32. > Though, longer term we should focus on convincing hdfs guys for inline > checksums (HDFS-2699) -- This message was sent by Atlassian JIRA (v6.1#6144)