[
https://issues.apache.org/jira/browse/HBASE-26330?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17530297#comment-17530297
]
Andrew Kyle Purtell edited comment on HBASE-26330 at 4/30/22 1:47 AM:
----------------------------------------------------------------------
I performed a definitive microbenchmark for this issue. See
https://github.com/apurtell/jmh-compression-tests .
In
[blockdata.zip|https://github.com/apurtell/jmh-compression-tests/blob/master/src/main/resources/blockdata.zip]
is 256 MB (258,126,022 bytes exactly) of block data extracted from two HFiles
after processing some Common Crawl Data with IntegrationLoadTestCommonCrawl,
2,680 blocks in total. This data was processed by each new codec implementation
as if the block data were being compressed again for write into an HFile, but
without writing any data, comparing only the CPU time and resource demand of
the codec itself. Java 11, Linux aarch64 VM, Apple M1 Max silicon, but relative
differences are what are interesting. Measured time is the average time in
milliseconds required to compress all blocks of the ~256MB file. This is how
long it would take to write an HFile containing these contents, minus the I/O
overhead of block encoding and actual persistence.
These are the results:
||Codec||Level||Time (milliseconds)||Result (bytes)||Improvement||
|AirCompressor LZ4|-|349.989 ± 2.835|76,999,408|70.17%|
|AirCompressor LZO|-|334.554 ± 3.243|79,369,805|69.25%|
|AirCompressor Snappy|-|364.153 ± 19.718|80,201,763|68.93%|
|AirCompressor Zstandard|3 (effective)|1108.267 ± 8.969|55,129,189|78.64%|
|Brotli|1|593.107 ± 2.376|58,672,319|77.27%|
|Brotli|3|1345.195 ± 27.327|53,917,438|79.11%|
|Brotli|6|2812.411 ± 25.372|48,696,441|81.13%|
|Brotli|10|74615.936 ± 224.854|44,970,710|82.58%|
|LZ4|-|303.045 ± 0.783|76,974,364|70.18%|
|LZMA|1|6410.428 ± 115.065|49,948,535|80.65%|
|LZMA|3|8144.620 ± 152.119|49,109,363|80.97%|
|LZMA|6|43802.576 ± 382.025|46,951,810|81.81%|
|LZMA|9|49821.979 ± 580.110|46,951,810|81.81%|
|Xerial Snappy|-|360.225 ± 2.324|80,749,937|68.72%|
|Zstandard|1|654.699 ± 16.839|56,719,994|78.03%|
|Zstandard|3|839.160 ± 24.906|54,573,095|78.86%|
|Zstandard|5|1594.373 ± 22.384|52,025,485|79.84%|
|Zstandard|7|2308.705 ± 24.744|50,651,554|80.38%|
|Zstandard|9|3659.677 ± 58.018|50,208,425|80.55%|
|Zstandard|12|8705.294 ± 58.080|49,841,446|80.69%|
|Zstandard|15|19785.646 ± 278.080|48,499,508|81.21%|
|Zstandard|18|47702.097 ± 442.670|48,319,879|81.28%|
|Zstandard|22|97799.695 ± 1106.571|48,212,220|81.32%|
Schema design and configuration guidelines at my employer are informed by these
results and similar results from earlier measurements:
- Compression enabled by default on all tables.
- LZ4 implementation (org.apache.hadoop.hbase.io.compress.lz4.Lz4Codec) as
default.
- If pure Java compression is required, use
org.apache.hadoop.hbase.io.compress.aircompressor.Lz4Codec instead.
- For use cases where higher compression efficiencies are required, opt for
Zstandard (org.apache.hadoop.hbase.io.compress.zstd.ZstdCodec).
-- Adjust {{ZstdCodec.ZSTD_LEVEL_KEY}} in table or column family schema to fine
tune for your data.
-- Level 1 is equivalent to LZ4, but LZ4 will perform better.
-- Level 3 is a good fast default.
-- Diminishing returns after level 7.
-- Levels 12-22 are not recommended.
- For web data, Brotli (org.apache.hadoop.hbase.io.compress.brotli.BrotliCodec)
is the superior option.
- When utilizing expensive in time compression, consider setting
{{COMPRESSION}} in table and column schema to LZ4 and
{{COMPRESSION_COMPACT_MAJOR}} to the more aggressive and expensive option.
Thus, only major compaction will incur significant overheads. Flushes and short
compactions will remain very fast. Allocate more long compaction threads than
the default.
- Enable WAL value compression (HBASE-25869) by default.
-- Set SNAPPY codec implementation to AirCompressor snappy
(org.apache.hadoop.hbase.io.compress.aircompressor.SnappyCodec) and configure
WAL value compression to use the SNAPPY codec. It has reasonable performance in
both space and time and as a pure Java implementation will be universally
available.
was (Author: apurtell):
I performed a definitive microbenchmark for this issue. See
https://github.com/apurtell/jmh-compression-tests .
In
[blockdata.zip|https://github.com/apurtell/jmh-compression-tests/blob/master/src/main/resources/blockdata.zip]
is 256 MB (258,126,022 bytes exactly) of block data extracted from two HFiles
after processing some Common Crawl Data with IntegrationLoadTestCommonCrawl,
2,680 blocks in total. This data was processed by each new codec implementation
as if the block data were being compressed again for write into an HFile, but
without writing any data, comparing only the CPU time and resource demand of
the codec itself. Java 11, Linux aarch64 VM, Apple M1 Max silicon, but relative
differences are what are interesting. Measured time is the average time in
milliseconds required to compress all blocks of the ~256MB file. This is how
long it would take to write an HFile containing these contents, minus the I/O
overhead of block encoding and actual persistence.
These are the results:
||Codec||Level||Time (milliseconds)||Result (bytes)||Improvement||
|AirCompressor LZ4|-|349.989 ± 2.835|76,999,408|70.17%|
|AirCompressor LZO|-|334.554 ± 3.243|79,369,805|69.25%|
|AirCompressor Snappy|-|364.153 ± 19.718|80,201,763|68.93%|
|AirCompressor Zstandard|3 (effective)|1108.267 ± 8.969|55,129,189|78.64%|
|Brotli|1|593.107 ± 2.376|58,672,319|77.27%|
|Brotli|3|1345.195 ± 27.327|53,917,438|79.11%|
|Brotli|6|2812.411 ± 25.372|48,696,441|81.13%|
|Brotli|10|74615.936 ± 224.854|44,970,710|82.58%|
|LZ4|-|303.045 ± 0.783|76,974,364|70.18%|
|LZMA|1|6410.428 ± 115.065|49,948,535|80.65%|
|LZMA|3|8144.620 ± 152.119|49,109,363|80.97%|
|LZMA|6|43802.576 ± 382.025|46,951,810|81.81%|
|LZMA|9|49821.979 ± 580.110|46,951,810|81.81%|
|Xerial Snappy|-|360.225 ± 2.324|80,749,937|68.72%|
|Zstandard|1|654.699 ± 16.839|56,719,994|78.03%|
|Zstandard|3|839.160 ± 24.906|54,573,095|78.86%|
|Zstandard|5|1594.373 ± 22.384|52,025,485|79.84%|
|Zstandard|7|2308.705 ± 24.744|50,651,554|80.38%|
|Zstandard|9|3659.677 ± 58.018|50,208,425|80.55%|
|Zstandard|12|8705.294 ± 58.080|49,841,446|80.69%|
|Zstandard|15|19785.646 ± 278.080|48,499,508|81.21%|
|Zstandard|18|47702.097 ± 442.670|48,319,879|81.28%|
|Zstandard|22|97799.695 ± 1106.571|48,212,220|81.32%|
Schema design and configuration guidelines at my employer are informed by these
results and similar results from earlier measurements:
- Compression enabled by default on all tables.
- LZ4 implementation (org.apache.hadoop.hbase.io.compress.lz4.Lz4Codec) as
default.
- If pure Java compression is required, use
org.apache.hadoop.hbase.io.compress.aircompressor.Lz4Codec instead.
- For use cases where higher compression efficiencies are required, opt for
Zstandard (org.apache.hadoop.hbase.io.compress.zstd.ZstdCodec).
-- Adjust {{ZstdCodec.ZSTD_LEVEL_KEY}} in table or column family schema to fine
tune for your data.
-- Level 1 is equivalent to LZ4, but LZ4 will perform better.
-- Level 3 is a good fast default.
-- Diminishing returns after level 7.
-- Levels 12-22 are not recommended.
- For web data, Brotli (org.apache.hadoop.hbase.io.compress.brotli.BrotliCodec)
is the superior option.
- When utilizing expensive in time compression, consider setting
{{COMPRESSION}} in table and column schema to LZ4 and
{{COMPRESSION_COMPACT_MAJOR}} to the more aggressive and expensive option.
Thus, only major compaction will incur significant overheads. Flushes and short
compactions will remain very fast. Allocate more long compaction threads than
the default.
- Enable WAL value compression (HBASE-25869) by default. Set SNAPPY codec
implementation to AirCompressor snappy
(org.apache.hadoop.hbase.io.compress.aircompressor.SnappyCodec) and configure
WAL value compression to use the SNAPPY codec. It has reasonable performance in
both space and time and as a pure Java implementation will be universally
available.
> Document new provided compression codecs
> ----------------------------------------
>
> Key: HBASE-26330
> URL: https://issues.apache.org/jira/browse/HBASE-26330
> Project: HBase
> Issue Type: Sub-task
> Reporter: Andrew Kyle Purtell
> Assignee: Andrew Kyle Purtell
> Priority: Blocker
> Fix For: 2.5.0, 3.0.0-alpha-3
>
>
> Document the new compression codecs:
> - The configuration keys used for setting the codec implementations for the
> various algorithms
> - The provided compression codecs
> - Default schema recommendations (LZ4)
> - Default WAL value compression recommendations (Snappy (aircompressor))
--
This message was sent by Atlassian Jira
(v8.20.7#820007)
