[jira] [Commented] (SPARK-32920) Add support in Spark driver to coordinate the finalization of the push/merge phase in push-based shuffle for a given shuffle and the initiation of the reduce stage
[ https://issues.apache.org/jira/browse/SPARK-32920?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17398617#comment-17398617 ] Apache Spark commented on SPARK-32920: -- User 'Ngone51' has created a pull request for this issue: https://github.com/apache/spark/pull/33738 > Add support in Spark driver to coordinate the finalization of the push/merge > phase in push-based shuffle for a given shuffle and the initiation of the > reduce stage > --- > > Key: SPARK-32920 > URL: https://issues.apache.org/jira/browse/SPARK-32920 > Project: Spark > Issue Type: Sub-task > Components: Shuffle, Spark Core >Affects Versions: 3.1.0 >Reporter: Min Shen >Assignee: Venkata krishnan Sowrirajan >Priority: Major > Fix For: 3.2.0 > > > With push-based shuffle, we are currently decoupling map task executions from > the shuffle block push process. Thus, when all map tasks finish, we might > want to wait for some small extra time to allow more shuffle blocks to get > pushed and merged. This requires some extra coordination in the Spark driver > when it transitions from a shuffle map stage to the corresponding reduce > stage. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-32920) Add support in Spark driver to coordinate the finalization of the push/merge phase in push-based shuffle for a given shuffle and the initiation of the reduce stage
[ https://issues.apache.org/jira/browse/SPARK-32920?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17398616#comment-17398616 ] Apache Spark commented on SPARK-32920: -- User 'Ngone51' has created a pull request for this issue: https://github.com/apache/spark/pull/33738 > Add support in Spark driver to coordinate the finalization of the push/merge > phase in push-based shuffle for a given shuffle and the initiation of the > reduce stage > --- > > Key: SPARK-32920 > URL: https://issues.apache.org/jira/browse/SPARK-32920 > Project: Spark > Issue Type: Sub-task > Components: Shuffle, Spark Core >Affects Versions: 3.1.0 >Reporter: Min Shen >Assignee: Venkata krishnan Sowrirajan >Priority: Major > Fix For: 3.2.0 > > > With push-based shuffle, we are currently decoupling map task executions from > the shuffle block push process. Thus, when all map tasks finish, we might > want to wait for some small extra time to allow more shuffle blocks to get > pushed and merged. This requires some extra coordination in the Spark driver > when it transitions from a shuffle map stage to the corresponding reduce > stage. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-32920) Add support in Spark driver to coordinate the finalization of the push/merge phase in push-based shuffle for a given shuffle and the initiation of the reduce stage
[ https://issues.apache.org/jira/browse/SPARK-32920?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17386346#comment-17386346 ] Apache Spark commented on SPARK-32920: -- User 'Ngone51' has created a pull request for this issue: https://github.com/apache/spark/pull/33495 > Add support in Spark driver to coordinate the finalization of the push/merge > phase in push-based shuffle for a given shuffle and the initiation of the > reduce stage > --- > > Key: SPARK-32920 > URL: https://issues.apache.org/jira/browse/SPARK-32920 > Project: Spark > Issue Type: Sub-task > Components: Shuffle, Spark Core >Affects Versions: 3.1.0 >Reporter: Min Shen >Assignee: Venkata krishnan Sowrirajan >Priority: Major > Fix For: 3.2.0 > > > With push-based shuffle, we are currently decoupling map task executions from > the shuffle block push process. Thus, when all map tasks finish, we might > want to wait for some small extra time to allow more shuffle blocks to get > pushed and merged. This requires some extra coordination in the Spark driver > when it transitions from a shuffle map stage to the corresponding reduce > stage. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-32920) Add support in Spark driver to coordinate the finalization of the push/merge phase in push-based shuffle for a given shuffle and the initiation of the reduce stage
[ https://issues.apache.org/jira/browse/SPARK-32920?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17383669#comment-17383669 ] Apache Spark commented on SPARK-32920: -- User 'venkata91' has created a pull request for this issue: https://github.com/apache/spark/pull/33426 > Add support in Spark driver to coordinate the finalization of the push/merge > phase in push-based shuffle for a given shuffle and the initiation of the > reduce stage > --- > > Key: SPARK-32920 > URL: https://issues.apache.org/jira/browse/SPARK-32920 > Project: Spark > Issue Type: Sub-task > Components: Shuffle, Spark Core >Affects Versions: 3.1.0 >Reporter: Min Shen >Assignee: Venkata krishnan Sowrirajan >Priority: Major > Fix For: 3.2.0 > > > With push-based shuffle, we are currently decoupling map task executions from > the shuffle block push process. Thus, when all map tasks finish, we might > want to wait for some small extra time to allow more shuffle blocks to get > pushed and merged. This requires some extra coordination in the Spark driver > when it transitions from a shuffle map stage to the corresponding reduce > stage. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-32920) Add support in Spark driver to coordinate the finalization of the push/merge phase in push-based shuffle for a given shuffle and the initiation of the reduce stage
[ https://issues.apache.org/jira/browse/SPARK-32920?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17379305#comment-17379305 ] Apache Spark commented on SPARK-32920: -- User 'venkata91' has created a pull request for this issue: https://github.com/apache/spark/pull/33303 > Add support in Spark driver to coordinate the finalization of the push/merge > phase in push-based shuffle for a given shuffle and the initiation of the > reduce stage > --- > > Key: SPARK-32920 > URL: https://issues.apache.org/jira/browse/SPARK-32920 > Project: Spark > Issue Type: Sub-task > Components: Shuffle, Spark Core >Affects Versions: 3.1.0 >Reporter: Min Shen >Assignee: Venkata krishnan Sowrirajan >Priority: Major > Fix For: 3.2.0 > > > With push-based shuffle, we are currently decoupling map task executions from > the shuffle block push process. Thus, when all map tasks finish, we might > want to wait for some small extra time to allow more shuffle blocks to get > pushed and merged. This requires some extra coordination in the Spark driver > when it transitions from a shuffle map stage to the corresponding reduce > stage. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-32920) Add support in Spark driver to coordinate the finalization of the push/merge phase in push-based shuffle for a given shuffle and the initiation of the reduce stage
[ https://issues.apache.org/jira/browse/SPARK-32920?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17379303#comment-17379303 ] Apache Spark commented on SPARK-32920: -- User 'venkata91' has created a pull request for this issue: https://github.com/apache/spark/pull/33303 > Add support in Spark driver to coordinate the finalization of the push/merge > phase in push-based shuffle for a given shuffle and the initiation of the > reduce stage > --- > > Key: SPARK-32920 > URL: https://issues.apache.org/jira/browse/SPARK-32920 > Project: Spark > Issue Type: Sub-task > Components: Shuffle, Spark Core >Affects Versions: 3.1.0 >Reporter: Min Shen >Assignee: Venkata krishnan Sowrirajan >Priority: Major > Fix For: 3.2.0 > > > With push-based shuffle, we are currently decoupling map task executions from > the shuffle block push process. Thus, when all map tasks finish, we might > want to wait for some small extra time to allow more shuffle blocks to get > pushed and merged. This requires some extra coordination in the Spark driver > when it transitions from a shuffle map stage to the corresponding reduce > stage. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-32920) Add support in Spark driver to coordinate the finalization of the push/merge phase in push-based shuffle for a given shuffle and the initiation of the reduce stage
[ https://issues.apache.org/jira/browse/SPARK-32920?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17246804#comment-17246804 ] Apache Spark commented on SPARK-32920: -- User 'venkata91' has created a pull request for this issue: https://github.com/apache/spark/pull/30691 > Add support in Spark driver to coordinate the finalization of the push/merge > phase in push-based shuffle for a given shuffle and the initiation of the > reduce stage > --- > > Key: SPARK-32920 > URL: https://issues.apache.org/jira/browse/SPARK-32920 > Project: Spark > Issue Type: Sub-task > Components: Shuffle, Spark Core >Affects Versions: 3.1.0 >Reporter: Min Shen >Assignee: Apache Spark >Priority: Major > > With push-based shuffle, we are currently decoupling map task executions from > the shuffle block push process. Thus, when all map tasks finish, we might > want to wait for some small extra time to allow more shuffle blocks to get > pushed and merged. This requires some extra coordination in the Spark driver > when it transitions from a shuffle map stage to the corresponding reduce > stage. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-32920) Add support in Spark driver to coordinate the finalization of the push/merge phase in push-based shuffle for a given shuffle and the initiation of the reduce stage
[ https://issues.apache.org/jira/browse/SPARK-32920?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17246803#comment-17246803 ] Apache Spark commented on SPARK-32920: -- User 'venkata91' has created a pull request for this issue: https://github.com/apache/spark/pull/30691 > Add support in Spark driver to coordinate the finalization of the push/merge > phase in push-based shuffle for a given shuffle and the initiation of the > reduce stage > --- > > Key: SPARK-32920 > URL: https://issues.apache.org/jira/browse/SPARK-32920 > Project: Spark > Issue Type: Sub-task > Components: Shuffle, Spark Core >Affects Versions: 3.1.0 >Reporter: Min Shen >Priority: Major > > With push-based shuffle, we are currently decoupling map task executions from > the shuffle block push process. Thus, when all map tasks finish, we might > want to wait for some small extra time to allow more shuffle blocks to get > pushed and merged. This requires some extra coordination in the Spark driver > when it transitions from a shuffle map stage to the corresponding reduce > stage. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
