[
https://issues.apache.org/jira/browse/HBASE-26353?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17429758#comment-17429758
]
Andrew Kyle Purtell commented on HBASE-26353:
---------------------------------------------
Just to double check, I re-ran the earlier described test, except when
generating the test data it only emitted:
- 10 million rows
- A 64-bit monotonically increasing row key
- Two values, both 32 bit integers, generated using random number generators
obeying a zipfian distribution (using our RandomDistribution.Zipf with a sigma
of 1.2)
When training the dictionary I gave the trainer the parameters k=32 (bit width
to enter into the dictionary) and d=8 (stride for walking over content, in
bits). This is a good approximation of designing these parameters with intent
in a real use case. The result demonstrates significant speedups in compression
as advertised and allows for achieving a better overall compression by enabling
higher compression levels given an equivalent time budget as a no dictionary
case.
*Integers Only, No Dictionary*
||Level||On Disk Size||Compression||Compaction Time (sec)||
|-|825,486,085|-|-|
|1|261,658,729|68.3%|21|
|3|251,343,431|69.6%|22|
|5|251,968,603|69.5%|25|
|6|251,467,677|69.5%|26|
|7|251,509,580|69.5%|27|
|12|235,410,126|71.5%|51|
*Integers Only, With Dictionary (k=32,d=8)*
||Level||On Disk Size||Compression||Compaction Time (sec)||
|1|248,971,553|69.8%|13|
|3|248,528,035|69.9%|14|
|5|245,846,087|70.2%|16|
|6|245,705,224|70.2%|17|
|7|226,998,954|72.5%|25|
|12|226,796,109|72.5%|39|
|15|226,553,944|72.6%|44|
|-18-|-216,373,878-|-73.8%-|-153-|
|-22-|-216,373,736-|-73.8%-|-165-|
> Support loadable dictionaries in hbase-compression-zstd
> -------------------------------------------------------
>
> Key: HBASE-26353
> URL: https://issues.apache.org/jira/browse/HBASE-26353
> Project: HBase
> Issue Type: Sub-task
> Reporter: Andrew Kyle Purtell
> Assignee: Andrew Kyle Purtell
> Priority: Minor
> Fix For: 2.5.0, 3.0.0-alpha-2
>
>
> ZStandard supports initialization of compressors and decompressors with a
> precomputed dictionary, which can dramatically improve and speed up
> compression of tables with small values. For more details, please see [The
> Case For Small Data
> Compression|https://github.com/facebook/zstd#the-case-for-small-data-compression].
>
> If a table is going to have a lot of small values and the user can put
> together a representative set of files that can be used to train a dictionary
> for compressing those values, a dictionary can be trained with the {{zstd}}
> command line utility, available in any zstandard package for your favorite OS:
> Training:
> {noformat}
> $ zstd --maxdict=1126400 --train-fastcover=shrink \
> -o mytable.dict training_files/*
> Trying 82 different sets of parameters
> ...
> k=674
> d=8
> f=20
> steps=40
> split=75
> accel=1
> Save dictionary of size 1126400 into file mytable.dict
> {noformat}
> Deploy the dictionary file to HDFS or S3, etc.
> Create the table:
> {noformat}
> hbase> create "mytable",
> ... ,
> CONFIGURATION => {
> 'hbase.io.compress.zstd.level' => '6',
> 'hbase.io.compress.zstd.dictionary' => true,
> 'hbase.io.compress.zstd.dictonary.file' =>
> 'hdfs://nn/zdicts/mytable.dict'
> }
> {noformat}
> Now start storing data. Compression results even for small values will be
> excellent.
> Note: Beware, if the dictionary is lost, the data will not be decompressable.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
