[
https://issues.apache.org/jira/browse/FLINK-14845?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16991107#comment-16991107
]
zhijiang commented on FLINK-14845:
--
[~pnowojski] Yes, exactly we would verify the effects via benchmark
[
https://issues.apache.org/jira/browse/FLINK-14845?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16990786#comment-16990786
]
Piotr Nowojski commented on FLINK-14845:
Thanks [~kevin.cyj] and [~zjwang] for your efforts!
[
https://issues.apache.org/jira/browse/FLINK-14845?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16990697#comment-16990697
]
Yingjie Cao commented on FLINK-14845:
-
Fix via 66d4d7da2d8b717f420509d9785fad0880562f10 on master.
[
https://issues.apache.org/jira/browse/FLINK-14845?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16985873#comment-16985873
]
Yingjie Cao commented on FLINK-14845:
-
Based on the above discussions, I have given an
[
https://issues.apache.org/jira/browse/FLINK-14845?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16984097#comment-16984097
]
Yingjie Cao commented on FLINK-14845:
-
> So I think it's fine to not make it pluggable in the first
[
https://issues.apache.org/jira/browse/FLINK-14845?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16983552#comment-16983552
]
Piotr Nowojski commented on FLINK-14845:
Regarding the org.lz4:lz4-java dependency. This one is
[
https://issues.apache.org/jira/browse/FLINK-14845?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16983338#comment-16983338
]
Yingjie Cao commented on FLINK-14845:
-
> Would it be more complicate to request a buffer from the
[
https://issues.apache.org/jira/browse/FLINK-14845?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16982446#comment-16982446
]
Piotr Nowojski commented on FLINK-14845:
For the output, writing back the compressed bytes in
[
https://issues.apache.org/jira/browse/FLINK-14845?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16982271#comment-16982271
]
Yingjie Cao commented on FLINK-14845:
-
> Where are you going to compress data into?
Each compressor
[
https://issues.apache.org/jira/browse/FLINK-14845?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16980729#comment-16980729
]
Piotr Nowojski commented on FLINK-14845:
Maybe before going further I have a couple of
[
https://issues.apache.org/jira/browse/FLINK-14845?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16979871#comment-16979871
]
Yingjie Cao commented on FLINK-14845:
-
[~pnowojski] I totally agree with you. Then I will work on
[
https://issues.apache.org/jira/browse/FLINK-14845?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16979249#comment-16979249
]
Piotr Nowojski commented on FLINK-14845:
[~kevin.cyj], I think I would prefer the 3rd option,
[
https://issues.apache.org/jira/browse/FLINK-14845?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16979002#comment-16979002
]
Yingjie Cao commented on FLINK-14845:
-
[~pnowojski] , thanks for proposing these options.
#
[
https://issues.apache.org/jira/browse/FLINK-14845?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16978343#comment-16978343
]
Jingsong Lee commented on FLINK-14845:
--
> doesn't Blink work on batches of records? Couldn't the
[
https://issues.apache.org/jira/browse/FLINK-14845?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16978325#comment-16978325
]
Piotr Nowojski commented on FLINK-14845:
Re, [~lzljs3620320]: doesn't Blink work on batches of
[
https://issues.apache.org/jira/browse/FLINK-14845?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16978297#comment-16978297
]
Chesnay Schepler commented on FLINK-14845:
--
[~kevin.cyj] Cluster partitions may also be read by
[
https://issues.apache.org/jira/browse/FLINK-14845?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16978288#comment-16978288
]
Yingjie Cao commented on FLINK-14845:
-
[~chesnay] A choice maybe make the compression codec cluster
[
https://issues.apache.org/jira/browse/FLINK-14845?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16978284#comment-16978284
]
Yingjie Cao commented on FLINK-14845:
-
[~pnowojski] You are right, taking both Blocking and
[
https://issues.apache.org/jira/browse/FLINK-14845?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16978268#comment-16978268
]
Chesnay Schepler commented on FLINK-14845:
--
How would this work with cluster partitions?
[
https://issues.apache.org/jira/browse/FLINK-14845?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16978257#comment-16978257
]
Jingsong Lee commented on FLINK-14845:
--
[~pnowojski] put forward a good thought for the future. I
[
https://issues.apache.org/jira/browse/FLINK-14845?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16978243#comment-16978243
]
Piotr Nowojski commented on FLINK-14845:
+1 for the future. Could be useful also for
[
https://issues.apache.org/jira/browse/FLINK-14845?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16978241#comment-16978241
]
Yingjie Cao commented on FLINK-14845:
-
Doing compression and decompression with task threads may be
[
https://issues.apache.org/jira/browse/FLINK-14845?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16978203#comment-16978203
]
Stephan Ewen commented on FLINK-14845:
--
For combining compression and low latency, we need to see
[
https://issues.apache.org/jira/browse/FLINK-14845?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16978184#comment-16978184
]
Yingjie Cao commented on FLINK-14845:
-
[~lzljs3620320] For small data, for example, dozens of byte,
[
https://issues.apache.org/jira/browse/FLINK-14845?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16978175#comment-16978175
]
Jingsong Lee commented on FLINK-14845:
--
Thanks [~kevin.cyj]
A little confuse about point 8, Are
[
https://issues.apache.org/jira/browse/FLINK-14845?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16978153#comment-16978153
]
Yingjie Cao commented on FLINK-14845:
-
To verify how many improvements we could gain from
[
https://issues.apache.org/jira/browse/FLINK-14845?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16978019#comment-16978019
]
Jingsong Lee commented on FLINK-14845:
--
[~sewen] Sorry for misunderstanding. What I want to say is
[
https://issues.apache.org/jira/browse/FLINK-14845?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16977982#comment-16977982
]
Kurt Young commented on FLINK-14845:
[~sewen] Currently we set all shuffle type to `BLOCKING` in
[
https://issues.apache.org/jira/browse/FLINK-14845?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16977693#comment-16977693
]
Stephan Ewen commented on FLINK-14845:
--
[~lzljs3620320] Quick question for clarification: In the
[
https://issues.apache.org/jira/browse/FLINK-14845?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16977441#comment-16977441
]
Stephan Ewen commented on FLINK-14845:
--
Kurt and me had a quick chat about this.
On the receiver
[
https://issues.apache.org/jira/browse/FLINK-14845?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16977096#comment-16977096
]
Jingsong Lee commented on FLINK-14845:
--
Big +1 for this feature.
There will be a lot of shuffles
31 matches
Mail list logo
