[jira] [Commented] (SPARK-30602) SPIP: Support push-based shuffle to improve shuffle efficiency

[ https://issues.apache.org/jira/browse/SPARK-30602?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17786077#comment-17786077 ] Mridul Muralidharan commented on SPARK-30602: - @MasterDDT, currently there is no active

[jira] [Commented] (SPARK-30602) SPIP: Support push-based shuffle to improve shuffle efficiency

[ https://issues.apache.org/jira/browse/SPARK-30602?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17785989#comment-17785989 ] Mitesh commented on SPARK-30602: Is there a plan to support push-based shuffle on Spark standalone

[jira] [Commented] (SPARK-30602) SPIP: Support push-based shuffle to improve shuffle efficiency

[ https://issues.apache.org/jira/browse/SPARK-30602?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17516298#comment-17516298 ] Min Shen commented on SPARK-30602: -- [~pan3793] , I guess you were referring to the earlier screenshot

[jira] [Commented] (SPARK-30602) SPIP: Support push-based shuffle to improve shuffle efficiency

[ https://issues.apache.org/jira/browse/SPARK-30602?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17516280#comment-17516280 ] Cheng Pan commented on SPARK-30602: --- Hi [~mshen] [~mridulm80] Why your reporter shows that Push-based

[jira] [Commented] (SPARK-30602) SPIP: Support push-based shuffle to improve shuffle efficiency

[ https://issues.apache.org/jira/browse/SPARK-30602?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17440194#comment-17440194 ] Jin Xing commented on SPARK-30602: -- Hi [~mshen]  Thanks a lot for the feature of Remote Shuffle

[jira] [Commented] (SPARK-30602) SPIP: Support push-based shuffle to improve shuffle efficiency

[ https://issues.apache.org/jira/browse/SPARK-30602?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17391745#comment-17391745 ] Min Shen commented on SPARK-30602: -- [~mridulm80], thanks for shepherding this work and your reviews on

[jira] [Commented] (SPARK-30602) SPIP: Support push-based shuffle to improve shuffle efficiency

[ https://issues.apache.org/jira/browse/SPARK-30602?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17391324#comment-17391324 ] Mridul Muralidharan commented on SPARK-30602: - Thanks for all the work in getting this

[jira] [Commented] (SPARK-30602) SPIP: Support push-based shuffle to improve shuffle efficiency

[ https://issues.apache.org/jira/browse/SPARK-30602?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17391276#comment-17391276 ] Gengliang Wang commented on SPARK-30602: [~zhouyejoe] Great, thanks! > SPIP: Support push-based

[jira] [Commented] (SPARK-30602) SPIP: Support push-based shuffle to improve shuffle efficiency

[ https://issues.apache.org/jira/browse/SPARK-30602?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17390184#comment-17390184 ] Ye Zhou commented on SPARK-30602: - [~Gengliang.Wang] This is the last PR(Subtask 9) to be merged 

[jira] [Commented] (SPARK-30602) SPIP: Support push-based shuffle to improve shuffle efficiency

[ https://issues.apache.org/jira/browse/SPARK-30602?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17389984#comment-17389984 ] Gengliang Wang commented on SPARK-30602: [~mshen] The expected 3.2 RC date is in August. What is

[jira] [Commented] (SPARK-30602) SPIP: Support push-based shuffle to improve shuffle efficiency

[ https://issues.apache.org/jira/browse/SPARK-30602?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17322476#comment-17322476 ] Min Shen commented on SPARK-30602: -- We have published the production results of push-based shuffle on

[jira] [Commented] (SPARK-30602) SPIP: Support push-based shuffle to improve shuffle efficiency

[ https://issues.apache.org/jira/browse/SPARK-30602?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17305738#comment-17305738 ] Min Shen commented on SPARK-30602: -- Just an update for where we are: The team at LinkedIn has been

[jira] [Commented] (SPARK-30602) SPIP: Support push-based shuffle to improve shuffle efficiency

[ https://issues.apache.org/jira/browse/SPARK-30602?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17198680#comment-17198680 ] Apache Spark commented on SPARK-30602: -- User 'otterc' has created a pull request for this issue:

[jira] [Commented] (SPARK-30602) SPIP: Support push-based shuffle to improve shuffle efficiency

[ https://issues.apache.org/jira/browse/SPARK-30602?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17198678#comment-17198678 ] Apache Spark commented on SPARK-30602: -- User 'otterc' has created a pull request for this issue:

[jira] [Commented] (SPARK-30602) SPIP: Support push-based shuffle to improve shuffle efficiency

[ https://issues.apache.org/jira/browse/SPARK-30602?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17198679#comment-17198679 ] Chandni Singh commented on SPARK-30602: --- The reference PR for the consolidated change:

[jira] [Commented] (SPARK-30602) SPIP: Support push-based shuffle to improve shuffle efficiency

[ https://issues.apache.org/jira/browse/SPARK-30602?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17183622#comment-17183622 ] Mridul Muralidharan commented on SPARK-30602: - SPIP proposal document:

[jira] [Commented] (SPARK-30602) SPIP: Support push-based shuffle to improve shuffle efficiency

[ https://issues.apache.org/jira/browse/SPARK-30602?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17151133#comment-17151133 ] qingwu.fu commented on SPARK-30602: --- hi [~mshen], could I have the access auth to see the source code?

[jira] [Commented] (SPARK-30602) SPIP: Support push-based shuffle to improve shuffle efficiency

[ https://issues.apache.org/jira/browse/SPARK-30602?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17144060#comment-17144060 ] Min Shen commented on SPARK-30602: -- Also want to share the production results we have so far. We have

[jira] [Commented] (SPARK-30602) SPIP: Support push-based shuffle to improve shuffle efficiency

[ https://issues.apache.org/jira/browse/SPARK-30602?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17144010#comment-17144010 ] Min Shen commented on SPARK-30602: -- Our paper summarizing the work on this new push-based shuffle was

[jira] [Commented] (SPARK-30602) SPIP: Support push-based shuffle to improve shuffle efficiency

[ https://issues.apache.org/jira/browse/SPARK-30602?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17043829#comment-17043829 ] Min Shen commented on SPARK-30602: -- [~shanyu], I have listed a few key differences between Riffle and

[jira] [Commented] (SPARK-30602) SPIP: Support push-based shuffle to improve shuffle efficiency

[ https://issues.apache.org/jira/browse/SPARK-30602?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17030114#comment-17030114 ] shanyu zhao commented on SPARK-30602: - Thanks for the effort Min! Riffle seems to only do map side

[jira] [Commented] (SPARK-30602) SPIP: Support push-based shuffle to improve shuffle efficiency

[ https://issues.apache.org/jira/browse/SPARK-30602?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17026134#comment-17026134 ] Tyson Condie commented on SPARK-30602: --  The design looks to bring in some good optimizations from

[jira] [Commented] (SPARK-30602) SPIP: Support push-based shuffle to improve shuffle efficiency

[ https://issues.apache.org/jira/browse/SPARK-30602?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17021672#comment-17021672 ] Hyukjin Kwon commented on SPARK-30602: -- Can you send the email to the dev list to discuss? If you