[
https://issues.apache.org/jira/browse/SPARK-24817?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16566159#comment-16566159
]
Erik Erlandson commented on SPARK-24817:
----------------------------------------
I'm curious about what the {{barrier}} invocations inside {{mapPartitions}}
closures imply about communications between executors, for example executors
running on pods in a kube cluster. It is possible that whatever is allowing
shuffle data to transfer between executors will also allow these {{barrier}}
coordinations to work, but we had to create a headless service for executors to
register properly with the driver pod, and if every executor pod needs
something like that for barrier to work, it will be an impact for kube backend
support.
> Implement BarrierTaskContext.barrier()
> --------------------------------------
>
> Key: SPARK-24817
> URL: https://issues.apache.org/jira/browse/SPARK-24817
> Project: Spark
> Issue Type: New Feature
> Components: Spark Core
> Affects Versions: 2.4.0
> Reporter: Jiang Xingbo
> Priority: Major
>
> Implement BarrierTaskContext.barrier(), to support global sync between all
> the tasks in a barrier stage. The global sync shall finish immediately once
> all tasks in the same barrier stage reaches the same barrier.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
