[
https://issues.apache.org/jira/browse/FLINK-33819?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17798083#comment-17798083
]
Yue Ma commented on FLINK-33819:
--------------------------------
[~masteryhx] Thanks for replying.
{quote}Linked FLINK-20684 as it has been discussed before.
{quote}
Sorry for missing the previous ticket and creating the duplicated one
{quote}Linked FLINK-11313 which talks about the LZ4 Compression which should be
more usable than Snappy.
{quote}
We did not use LZ4 in the production environment, and for small state jobs, we
directly turn off the compression. And for job with large states, we adopted a
compression algorithm based on Snappy optimization.
{quote}Do you have some test results on it ?
{quote}
Yes. we did some benchmark test for the *SnappyCompression* and *NoCompression*
And result show that and Read Performance. After turning off Compression, State
Benchmark read performance can be improved by *80% to 100%*
We also conducted end-to-end online job testing, and after turning off
Compression, {*}the CPU usage of the job decreased by 16%{*}, while the
Checkpoint Total Size increased by *4-5 times.*
It is obvious that closing Compression is not only about benefits, but also
brings some space amplification. What I want to express is that we may need to
provide such a configuration for users to balance how to exchange space for time
{quote}BTW, If we'd like to introduce such a option, it's better to guarantee
the compalibility.
{quote}
Sorry, I didn't understand the compatibility issue here. I understand that it
is compatible here. After switching the Compression Type, the newly generated
file will be compressed using the new Compress Type, and the existing file can
still be read and written with old Compress Type.
> Support setting CompressType in RocksDBStateBackend
> ---------------------------------------------------
>
> Key: FLINK-33819
> URL: https://issues.apache.org/jira/browse/FLINK-33819
> Project: Flink
> Issue Type: Improvement
> Components: Runtime / State Backends
> Affects Versions: 1.18.0
> Reporter: Yue Ma
> Priority: Major
> Fix For: 1.19.0
>
> Attachments: image-2023-12-14-11-32-32-968.png,
> image-2023-12-14-11-35-22-306.png
>
>
> Currently, RocksDBStateBackend does not support setting the compression
> level, and Snappy is used for compression by default. But we have some
> scenarios where compression will use a lot of CPU resources. Turning off
> compression can significantly reduce CPU overhead. So we may need to support
> a parameter for users to set the CompressType of Rocksdb.
> !image-2023-12-14-11-35-22-306.png!
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
