[jira] [Commented] (FLINK-15311) Lz4BlockCompressionFactory should use native compressor instead of java unsafe
[ https://issues.apache.org/jira/browse/FLINK-15311?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17000971#comment-17000971 ] Yingjie Cao commented on FLINK-15311: - Fix via 5a30c79b0db7c99784fbb40346b7682d0ac64740 on release-1.10. Fix via 444a54437070870c0217d73656566274414e72fa on master. > Lz4BlockCompressionFactory should use native compressor instead of java unsafe > -- > > Key: FLINK-15311 > URL: https://issues.apache.org/jira/browse/FLINK-15311 > Project: Flink > Issue Type: Improvement > Components: Runtime / Network >Reporter: Jingsong Lee >Assignee: Yingjie Cao >Priority: Blocker > Labels: pull-request-available > Fix For: 1.10.0 > > Time Spent: 40m > Remaining Estimate: 0h > > According to: > [https://lz4.github.io/lz4-java/1.7.0/lz4-compression-benchmark/] > Java java unsafe compressor has lower performance than native lz4 compressor. > After FLINK-14845 , we use lz4 compression for shuffler. > In testing, I found shuffle using java unsafe compressor. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (FLINK-15311) Lz4BlockCompressionFactory should use native compressor instead of java unsafe
[ https://issues.apache.org/jira/browse/FLINK-15311?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16999889#comment-16999889 ] zhijiang commented on FLINK-15311: -- After discussed offline, this improvement is very critical for our motivation to bring compression at the beginning. Then still make it as a blocker for release-1.10, but change the type to improvement instead. > Lz4BlockCompressionFactory should use native compressor instead of java unsafe > -- > > Key: FLINK-15311 > URL: https://issues.apache.org/jira/browse/FLINK-15311 > Project: Flink > Issue Type: Bug > Components: Runtime / Network >Reporter: Jingsong Lee >Assignee: Yingjie Cao >Priority: Blocker > Fix For: 1.10.0 > > > According to: > [https://lz4.github.io/lz4-java/1.7.0/lz4-compression-benchmark/] > Java java unsafe compressor has lower performance than native lz4 compressor. > After FLINK-14845 , we use lz4 compression for shuffler. > In testing, I found shuffle using java unsafe compressor. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (FLINK-15311) Lz4BlockCompressionFactory should use native compressor instead of java unsafe
[ https://issues.apache.org/jira/browse/FLINK-15311?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16999797#comment-16999797 ] zhijiang commented on FLINK-15311: -- I guess it belongs to performance improvement, not a bug, because it does not affect the compression function and stable issue. Should it be a blocker for the release? > Lz4BlockCompressionFactory should use native compressor instead of java unsafe > -- > > Key: FLINK-15311 > URL: https://issues.apache.org/jira/browse/FLINK-15311 > Project: Flink > Issue Type: Bug > Components: Runtime / Network >Reporter: Jingsong Lee >Priority: Critical > Fix For: 1.10.0 > > > According to: > [https://lz4.github.io/lz4-java/1.7.0/lz4-compression-benchmark/] > Java java unsafe compressor has lower performance than native lz4 compressor. > After FLINK-14845 , we use lz4 compression for shuffler. > In testing, I found shuffle using java unsafe compressor. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (FLINK-15311) Lz4BlockCompressionFactory should use native compressor instead of java unsafe
[ https://issues.apache.org/jira/browse/FLINK-15311?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16998946#comment-16998946 ] Yingjie Cao commented on FLINK-15311: - [~lzljs3620320] Thanks for reporting this issue. I will check why native implementation is not used. > Lz4BlockCompressionFactory should use native compressor instead of java unsafe > -- > > Key: FLINK-15311 > URL: https://issues.apache.org/jira/browse/FLINK-15311 > Project: Flink > Issue Type: Bug > Components: Runtime / Network >Reporter: Jingsong Lee >Priority: Critical > Fix For: 1.10.0 > > > According to: > [https://lz4.github.io/lz4-java/1.7.0/lz4-compression-benchmark/] > Java java unsafe compressor has lower performance than native lz4 compressor. > After FLINK-14845 , we use lz4 compression for shuffler. > In testing, I found shuffle using java unsafe compressor. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (FLINK-15311) Lz4BlockCompressionFactory should use native compressor instead of java unsafe
[ https://issues.apache.org/jira/browse/FLINK-15311?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16998943#comment-16998943 ] Jingsong Lee commented on FLINK-15311: -- CC: [~kevin.cyj] > Lz4BlockCompressionFactory should use native compressor instead of java unsafe > -- > > Key: FLINK-15311 > URL: https://issues.apache.org/jira/browse/FLINK-15311 > Project: Flink > Issue Type: Bug > Components: Runtime / Network >Reporter: Jingsong Lee >Priority: Critical > Fix For: 1.10.0 > > > According to: > [https://lz4.github.io/lz4-java/1.7.0/lz4-compression-benchmark/] > Java java unsafe compressor has lower performance than native lz4 compressor. > After FLINK-14845 , we use lz4 compression for shuffler. > In testing, I found shuffle using java unsafe compressor. -- This message was sent by Atlassian Jira (v8.3.4#803005)
