[jira] [Commented] (FLINK-14159) flink rocksdb StreamCompressionDecorator not right
[
https://issues.apache.org/jira/browse/FLINK-14159?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17328267#comment-17328267
]
Flink Jira Bot commented on FLINK-14159:
This major issue is unassigned and itself and all of its Sub-Tasks have not
been updated for 30 days. So, it has been labeled "stale-major". If this ticket
is indeed "major", please either assign yourself or give an update. Afterwards,
please remove the label. In 7 days the issue will be deprioritized.
> flink rocksdb StreamCompressionDecorator not right
> --
>
> Key: FLINK-14159
> URL: https://issues.apache.org/jira/browse/FLINK-14159
> Project: Flink
> Issue Type: Bug
> Components: Runtime / State Backends
>Affects Versions: 1.9.0
>Reporter: jackylau
>Priority: Major
> Labels: stale-major
> Fix For: 1.10.4, 1.11.4, 1.13.0
>
>
> I think the current flink rocksdb StreamCompressionDecorator is not right
> when calling method
> getCompressionDecorator(executionConfig) ,which defalut value is false.That
> is to say, current compression is none.But I find rocksdb using
> {{options.compression}} to specify the compression to use. By default it is
> Snappy, which you can see here
> [https://github.com/facebook/rocksdb/wiki/RocksDB-Tuning-Guide]. And I use
> rocksdb tool sstdump to find it is indeed snappy compression.
> So I think it should be return SnappyStreamCompressionDecorator.INSTANCE
> rather than getCompressionDecorator( executionConfig)
> Coud i commit a PR?
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (FLINK-14159) flink rocksdb StreamCompressionDecorator not right
[
https://issues.apache.org/jira/browse/FLINK-14159?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16939204#comment-16939204
]
jackylau commented on FLINK-14159:
--
[~klion26], It will be at cost of cpu when the state is large and It needn't to
compress twice. so i think it may need some conments on the flink compression
code and ref docs when using rocksdb backend or a extra compress flag to choose
which compression you want (flink or rocksdb)
> flink rocksdb StreamCompressionDecorator not right
> --
>
> Key: FLINK-14159
> URL: https://issues.apache.org/jira/browse/FLINK-14159
> Project: Flink
> Issue Type: Bug
> Components: Runtime / State Backends
>Affects Versions: 1.9.0
>Reporter: jackylau
>Priority: Major
> Fix For: 1.10.0
>
>
> I think the current flink rocksdb StreamCompressionDecorator is not right
> when calling method
> getCompressionDecorator(executionConfig) ,which defalut value is false.That
> is to say, current compression is none.But I find rocksdb using
> {{options.compression}} to specify the compression to use. By default it is
> Snappy, which you can see here
> [https://github.com/facebook/rocksdb/wiki/RocksDB-Tuning-Guide]. And I use
> rocksdb tool sstdump to find it is indeed snappy compression.
> So I think it should be return SnappyStreamCompressionDecorator.INSTANCE
> rather than getCompressionDecorator( executionConfig)
> Coud i commit a PR?
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (FLINK-14159) flink rocksdb StreamCompressionDecorator not right
[
https://issues.apache.org/jira/browse/FLINK-14159?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16937332#comment-16937332
]
Congxian Qiu(klion26) commented on FLINK-14159:
---
[~jackylau] Yes, When we use {{SnappyStreamCompressionDecorator}} and in
FullSnapshot, we'll compress twice(Rocksdb's compression and checkpoint
compression), in incremental snapshot we'll just compress once(The rocksdb's
compression)
> flink rocksdb StreamCompressionDecorator not right
> --
>
> Key: FLINK-14159
> URL: https://issues.apache.org/jira/browse/FLINK-14159
> Project: Flink
> Issue Type: Bug
> Components: Runtime / State Backends
>Affects Versions: 1.9.0
>Reporter: jackylau
>Priority: Major
> Fix For: 1.10.0
>
>
> I think the current flink rocksdb StreamCompressionDecorator is not right
> when calling method
> getCompressionDecorator(executionConfig) ,which defalut value is false.That
> is to say, current compression is none.But I find rocksdb using
> {{options.compression}} to specify the compression to use. By default it is
> Snappy, which you can see here
> [https://github.com/facebook/rocksdb/wiki/RocksDB-Tuning-Guide]. And I use
> rocksdb tool sstdump to find it is indeed snappy compression.
> So I think it should be return SnappyStreamCompressionDecorator.INSTANCE
> rather than getCompressionDecorator( executionConfig)
> Coud i commit a PR?
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (FLINK-14159) flink rocksdb StreamCompressionDecorator not right
[
https://issues.apache.org/jira/browse/FLINK-14159?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16936548#comment-16936548
]
jackylau commented on FLINK-14159:
--
[~klion26] Does that mean checkpoint has twice compression when we use
SnappyStreamCompressionDecorator. The first is checkpoint data, the second is
compressed data and rocksdb compress it again.
But is doesn't matter. Do I understand that right?
> flink rocksdb StreamCompressionDecorator not right
> --
>
> Key: FLINK-14159
> URL: https://issues.apache.org/jira/browse/FLINK-14159
> Project: Flink
> Issue Type: Bug
> Components: Runtime / State Backends
>Affects Versions: 1.9.0
>Reporter: jackylau
>Priority: Major
> Fix For: 1.10.0
>
>
> I think the current flink rocksdb StreamCompressionDecorator is not right
> when calling method
> getCompressionDecorator(executionConfig) ,which defalut value is false.That
> is to say, current compression is none.But I find rocksdb using
> {{options.compression}} to specify the compression to use. By default it is
> Snappy, which you can see here
> [https://github.com/facebook/rocksdb/wiki/RocksDB-Tuning-Guide]. And I use
> rocksdb tool sstdump to find it is indeed snappy compression.
> So I think it should be return SnappyStreamCompressionDecorator.INSTANCE
> rather than getCompressionDecorator( executionConfig)
> Coud i commit a PR?
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (FLINK-14159) flink rocksdb StreamCompressionDecorator not right
[
https://issues.apache.org/jira/browse/FLINK-14159?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16936536#comment-16936536
]
Congxian Qiu(klion26) commented on FLINK-14159:
---
[~jackylau] ah, I've misunderstood what you mean, The
{{getCompressionDecorator(executionConfig)}} did not specific the format of sst
files, but the checkpoint, please ref the doc[1] for more information
[1]
[https://ci.apache.org/projects/flink/flink-docs-stable/ops/state/large_state_tuning.html#compression]
> flink rocksdb StreamCompressionDecorator not right
> --
>
> Key: FLINK-14159
> URL: https://issues.apache.org/jira/browse/FLINK-14159
> Project: Flink
> Issue Type: Bug
> Components: Runtime / State Backends
>Affects Versions: 1.9.0
>Reporter: jackylau
>Priority: Major
> Fix For: 1.10.0
>
>
> I think the current flink rocksdb StreamCompressionDecorator is not right
> when calling method
> getCompressionDecorator(executionConfig) ,which defalut value is false.That
> is to say, current compression is none.But I find rocksdb using
> {{options.compression}} to specify the compression to use. By default it is
> Snappy, which you can see here
> [https://github.com/facebook/rocksdb/wiki/RocksDB-Tuning-Guide]. And I use
> rocksdb tool sstdump to find it is indeed snappy compression.
> So I think it should be return SnappyStreamCompressionDecorator.INSTANCE
> rather than getCompressionDecorator( executionConfig)
> Coud i commit a PR?
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (FLINK-14159) flink rocksdb StreamCompressionDecorator not right
[
https://issues.apache.org/jira/browse/FLINK-14159?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16936290#comment-16936290
]
jackylau commented on FLINK-14159:
--
Hi [~klion26], "If no executionconfig configured, the
{{UncompressedStreamCompressionDecorator}} will be used", but the rocksdb
default compression is snappy. which the symantic is conflict.
> flink rocksdb StreamCompressionDecorator not right
> --
>
> Key: FLINK-14159
> URL: https://issues.apache.org/jira/browse/FLINK-14159
> Project: Flink
> Issue Type: Bug
> Components: Runtime / State Backends
>Affects Versions: 1.9.0
>Reporter: jackylau
>Priority: Major
> Fix For: 1.10.0
>
>
> I think the current flink rocksdb StreamCompressionDecorator is not right
> when calling method
> getCompressionDecorator(executionConfig) ,which defalut value is false.That
> is to say, current compression is none.But I find rocksdb using
> {{options.compression}} to specify the compression to use. By default it is
> Snappy, which you can see here
> [https://github.com/facebook/rocksdb/wiki/RocksDB-Tuning-Guide]. And I use
> rocksdb tool sstdump to find it is indeed snappy compression.
> So I think it should be return SnappyStreamCompressionDecorator.INSTANCE
> rather than getCompressionDecorator( executionConfig)
> Coud i commit a PR?
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (FLINK-14159) flink rocksdb StreamCompressionDecorator not right
[
https://issues.apache.org/jira/browse/FLINK-14159?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16935785#comment-16935785
]
Congxian Qiu(klion26) commented on FLINK-14159:
---
hi, [~jackylau] Do you want to use {{SnappyStreamCompressionDecorator}} in
Flink, currently, we'll return {{SnappyStreamCompressionDecorator.INSTANCE}} if
executionConfig configured, please ref to
{{AbstractStateBsackend}}#getCompressionDecorator, If no executionconfig
configured, the {{UncompressedStreamCompressionDecorator}} will be used. thanks
> flink rocksdb StreamCompressionDecorator not right
> --
>
> Key: FLINK-14159
> URL: https://issues.apache.org/jira/browse/FLINK-14159
> Project: Flink
> Issue Type: Bug
> Components: Runtime / State Backends
>Affects Versions: 1.9.0
>Reporter: jackylau
>Priority: Major
> Fix For: 1.10.0
>
>
> I think the current flink rocksdb StreamCompressionDecorator is not right
> when calling method
> getCompressionDecorator(executionConfig) ,which defalut value is false.That
> is to say, current compression is none.But I find rocksdb using
> {{options.compression}} to specify the compression to use. By default it is
> Snappy, which you can see here
> [https://github.com/facebook/rocksdb/wiki/RocksDB-Tuning-Guide]. And I use
> rocksdb tool sstdump to find it is indeed snappy compression.
> So I think it should be return SnappyStreamCompressionDecorator.INSTANCE
> rather than getCompressionDecorator( executionConfig)
> Coud i commit a PR?
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (FLINK-14159) flink rocksdb StreamCompressionDecorator not right
[
https://issues.apache.org/jira/browse/FLINK-14159?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16934884#comment-16934884
]
jackylau commented on FLINK-14159:
--
who is responsible for , and who can see this issue [~lzljs3620320]
> flink rocksdb StreamCompressionDecorator not right
> --
>
> Key: FLINK-14159
> URL: https://issues.apache.org/jira/browse/FLINK-14159
> Project: Flink
> Issue Type: Bug
> Components: Runtime / State Backends
>Affects Versions: 1.9.0
>Reporter: jackylau
>Priority: Major
> Fix For: 1.10.0
>
>
> I think the current flink rocksdb StreamCompressionDecorator is not right
> calling method
> getCompressionDecorator(executionConfig) which defalut value is false.That is
> to say, current compression is none.But I find rocksdb using
> {{options.compression}} to specify the compression to use. By default it is
> Snappy, which you can see here
> [https://github.com/facebook/rocksdb/wiki/RocksDB-Tuning-Guide]. And I use
> rocksdb tool sstdump to find it is indeed snappy compression.
> So I think it should be return SnappyStreamCompressionDecorator.INSTANCE
> rather than getCompressionDecorator( executionConfig)
> Coud i commit a PR?
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
