[jira] [Commented] (SPARK-32922) Add support for ShuffleBlockFetcherIterator to read from merged shuffle partitions and to fallback to original shuffle blocks if encountering failures
[ https://issues.apache.org/jira/browse/SPARK-32922?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17381620#comment-17381620 ] Apache Spark commented on SPARK-32922: -- User 'otterc' has created a pull request for this issue: https://github.com/apache/spark/pull/33378 > Add support for ShuffleBlockFetcherIterator to read from merged shuffle > partitions and to fallback to original shuffle blocks if encountering failures > -- > > Key: SPARK-32922 > URL: https://issues.apache.org/jira/browse/SPARK-32922 > Project: Spark > Issue Type: Sub-task > Components: Shuffle, Spark Core >Affects Versions: 3.1.0 >Reporter: Min Shen >Assignee: Chandni Singh >Priority: Major > Fix For: 3.2.0 > > > With the extended MapOutputTracker, the reducers can now get the task input > data from the merged shuffle partitions for more efficient shuffle data > fetch. The reducers should also be able to fallback to fetching the original > unmarked blocks if it encounters failures when fetching the merged shuffle > partitions. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-32922) Add support for ShuffleBlockFetcherIterator to read from merged shuffle partitions and to fallback to original shuffle blocks if encountering failures
[ https://issues.apache.org/jira/browse/SPARK-32922?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17381619#comment-17381619 ] Apache Spark commented on SPARK-32922: -- User 'otterc' has created a pull request for this issue: https://github.com/apache/spark/pull/33378 > Add support for ShuffleBlockFetcherIterator to read from merged shuffle > partitions and to fallback to original shuffle blocks if encountering failures > -- > > Key: SPARK-32922 > URL: https://issues.apache.org/jira/browse/SPARK-32922 > Project: Spark > Issue Type: Sub-task > Components: Shuffle, Spark Core >Affects Versions: 3.1.0 >Reporter: Min Shen >Assignee: Chandni Singh >Priority: Major > Fix For: 3.2.0 > > > With the extended MapOutputTracker, the reducers can now get the task input > data from the merged shuffle partitions for more efficient shuffle data > fetch. The reducers should also be able to fallback to fetching the original > unmarked blocks if it encounters failures when fetching the merged shuffle > partitions. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-32922) Add support for ShuffleBlockFetcherIterator to read from merged shuffle partitions and to fallback to original shuffle blocks if encountering failures
[ https://issues.apache.org/jira/browse/SPARK-32922?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17358768#comment-17358768 ] Chandni Singh commented on SPARK-32922: --- Splitting the changes into ESS server side/client side changes as per the comment here: https://github.com/apache/spark/pull/32140#issuecomment-856099709 > Add support for ShuffleBlockFetcherIterator to read from merged shuffle > partitions and to fallback to original shuffle blocks if encountering failures > -- > > Key: SPARK-32922 > URL: https://issues.apache.org/jira/browse/SPARK-32922 > Project: Spark > Issue Type: Sub-task > Components: Shuffle, Spark Core >Affects Versions: 3.1.0 >Reporter: Min Shen >Priority: Major > > With the extended MapOutputTracker, the reducers can now get the task input > data from the merged shuffle partitions for more efficient shuffle data > fetch. The reducers should also be able to fallback to fetching the original > unmarked blocks if it encounters failures when fetching the merged shuffle > partitions. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-32922) Add support for ShuffleBlockFetcherIterator to read from merged shuffle partitions and to fallback to original shuffle blocks if encountering failures
[ https://issues.apache.org/jira/browse/SPARK-32922?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17319866#comment-17319866 ] Apache Spark commented on SPARK-32922: -- User 'otterc' has created a pull request for this issue: https://github.com/apache/spark/pull/32140 > Add support for ShuffleBlockFetcherIterator to read from merged shuffle > partitions and to fallback to original shuffle blocks if encountering failures > -- > > Key: SPARK-32922 > URL: https://issues.apache.org/jira/browse/SPARK-32922 > Project: Spark > Issue Type: Sub-task > Components: Shuffle, Spark Core >Affects Versions: 3.1.0 >Reporter: Min Shen >Priority: Major > > With the extended MapOutputTracker, the reducers can now get the task input > data from the merged shuffle partitions for more efficient shuffle data > fetch. The reducers should also be able to fallback to fetching the original > unmarked blocks if it encounters failures when fetching the merged shuffle > partitions. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-32922) Add support for ShuffleBlockFetcherIterator to read from merged shuffle partitions and to fallback to original shuffle blocks if encountering failures
[ https://issues.apache.org/jira/browse/SPARK-32922?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17319865#comment-17319865 ] Apache Spark commented on SPARK-32922: -- User 'otterc' has created a pull request for this issue: https://github.com/apache/spark/pull/32140 > Add support for ShuffleBlockFetcherIterator to read from merged shuffle > partitions and to fallback to original shuffle blocks if encountering failures > -- > > Key: SPARK-32922 > URL: https://issues.apache.org/jira/browse/SPARK-32922 > Project: Spark > Issue Type: Sub-task > Components: Shuffle, Spark Core >Affects Versions: 3.1.0 >Reporter: Min Shen >Priority: Major > > With the extended MapOutputTracker, the reducers can now get the task input > data from the merged shuffle partitions for more efficient shuffle data > fetch. The reducers should also be able to fallback to fetching the original > unmarked blocks if it encounters failures when fetching the merged shuffle > partitions. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-32922) Add support for ShuffleBlockFetcherIterator to read from merged shuffle partitions and to fallback to original shuffle blocks if encountering failures
[ https://issues.apache.org/jira/browse/SPARK-32922?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17249274#comment-17249274 ] Apache Spark commented on SPARK-32922: -- User 'otterc' has created a pull request for this issue: https://github.com/apache/spark/pull/30768 > Add support for ShuffleBlockFetcherIterator to read from merged shuffle > partitions and to fallback to original shuffle blocks if encountering failures > -- > > Key: SPARK-32922 > URL: https://issues.apache.org/jira/browse/SPARK-32922 > Project: Spark > Issue Type: Sub-task > Components: Shuffle, Spark Core >Affects Versions: 3.1.0 >Reporter: Min Shen >Priority: Major > > With the extended MapOutputTracker, the reducers can now get the task input > data from the merged shuffle partitions for more efficient shuffle data > fetch. The reducers should also be able to fallback to fetching the original > unmarked blocks if it encounters failures when fetching the merged shuffle > partitions. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-32922) Add support for ShuffleBlockFetcherIterator to read from merged shuffle partitions and to fallback to original shuffle blocks if encountering failures
[ https://issues.apache.org/jira/browse/SPARK-32922?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17249272#comment-17249272 ] Apache Spark commented on SPARK-32922: -- User 'otterc' has created a pull request for this issue: https://github.com/apache/spark/pull/30768 > Add support for ShuffleBlockFetcherIterator to read from merged shuffle > partitions and to fallback to original shuffle blocks if encountering failures > -- > > Key: SPARK-32922 > URL: https://issues.apache.org/jira/browse/SPARK-32922 > Project: Spark > Issue Type: Sub-task > Components: Shuffle, Spark Core >Affects Versions: 3.1.0 >Reporter: Min Shen >Priority: Major > > With the extended MapOutputTracker, the reducers can now get the task input > data from the merged shuffle partitions for more efficient shuffle data > fetch. The reducers should also be able to fallback to fetching the original > unmarked blocks if it encounters failures when fetching the merged shuffle > partitions. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
