[jira] [Comment Edited] (FLINK-33819) Support setting CompressType in RocksDBStateBackend
[
https://issues.apache.org/jira/browse/FLINK-33819?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17813740#comment-17813740
]
Piotr Nowojski edited comment on FLINK-33819 at 2/2/24 3:52 PM:
Thanks [~mayuehappy].
{quote}
merged 4f7725aa into master which is not blocked by the performance result.
I also think it's worthy discussing and we could find a better default
behavious in the next version.
{quote}
+1
was (Author: pnowojski):
Thanks [~mayuehappy].
{quote}
merged 4f7725aa into master which is not blocked by the performance result.
I also think it's worthy discussing and we could find a better default
behavious in the next version.
{quote}
+1
Yes
> Support setting CompressType in RocksDBStateBackend
> ---
>
> Key: FLINK-33819
> URL: https://issues.apache.org/jira/browse/FLINK-33819
> Project: Flink
> Issue Type: Improvement
> Components: Runtime / State Backends
>Affects Versions: 1.18.0
>Reporter: Yue Ma
>Assignee: Yue Ma
>Priority: Major
> Labels: pull-request-available
> Fix For: 1.19.0
>
> Attachments: image-2023-12-14-11-32-32-968.png,
> image-2023-12-14-11-35-22-306.png
>
>
> Currently, RocksDBStateBackend does not support setting the compression
> level, and Snappy is used for compression by default. But we have some
> scenarios where compression will use a lot of CPU resources. Turning off
> compression can significantly reduce CPU overhead. So we may need to support
> a parameter for users to set the CompressType of Rocksdb.
> !image-2023-12-14-11-35-22-306.png!
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Comment Edited] (FLINK-33819) Support setting CompressType in RocksDBStateBackend
[ https://issues.apache.org/jira/browse/FLINK-33819?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17806804#comment-17806804 ] Piotr Nowojski edited comment on FLINK-33819 at 1/15/24 1:09 PM: - Thanks for drafting the proposal [~mayuehappy] +1 for making this option configurable. It would be great if someone could test out [~masteryhx]'s suggestion. Configuring no compression for L0 and L1 levels, while keeping lower levels compressed: * By how much the checkpoint would grow? * What about performance? It sounds like this actually could be the default behaviour for us? Would be great to test it out, but not a blocker from my side for implementing this ticket. was (Author: pnowojski): Thanks for drafting the proposal [~mayuehappy] +1 for making this option configurable. > Support setting CompressType in RocksDBStateBackend > --- > > Key: FLINK-33819 > URL: https://issues.apache.org/jira/browse/FLINK-33819 > Project: Flink > Issue Type: Improvement > Components: Runtime / State Backends >Affects Versions: 1.18.0 >Reporter: Yue Ma >Assignee: Yue Ma >Priority: Major > Labels: pull-request-available > Fix For: 1.19.0 > > Attachments: image-2023-12-14-11-32-32-968.png, > image-2023-12-14-11-35-22-306.png > > > Currently, RocksDBStateBackend does not support setting the compression > level, and Snappy is used for compression by default. But we have some > scenarios where compression will use a lot of CPU resources. Turning off > compression can significantly reduce CPU overhead. So we may need to support > a parameter for users to set the CompressType of Rocksdb. > !image-2023-12-14-11-35-22-306.png! -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Comment Edited] (FLINK-33819) Support setting CompressType in RocksDBStateBackend
[ https://issues.apache.org/jira/browse/FLINK-33819?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17806804#comment-17806804 ] Piotr Nowojski edited comment on FLINK-33819 at 1/15/24 1:00 PM: - Thanks for drafting the proposal [~mayuehappy] +1 for making this option configurable. was (Author: pnowojski): +1 for making this option configurable. > Support setting CompressType in RocksDBStateBackend > --- > > Key: FLINK-33819 > URL: https://issues.apache.org/jira/browse/FLINK-33819 > Project: Flink > Issue Type: Improvement > Components: Runtime / State Backends >Affects Versions: 1.18.0 >Reporter: Yue Ma >Assignee: Yue Ma >Priority: Major > Labels: pull-request-available > Fix For: 1.19.0 > > Attachments: image-2023-12-14-11-32-32-968.png, > image-2023-12-14-11-35-22-306.png > > > Currently, RocksDBStateBackend does not support setting the compression > level, and Snappy is used for compression by default. But we have some > scenarios where compression will use a lot of CPU resources. Turning off > compression can significantly reduce CPU overhead. So we may need to support > a parameter for users to set the CompressType of Rocksdb. > !image-2023-12-14-11-35-22-306.png! -- This message was sent by Atlassian Jira (v8.20.10#820010)
