Xiangrui Meng created SPARK-30667: ------------------------------------- Summary: Support simple all gather in barrier task context Key: SPARK-30667 URL: https://issues.apache.org/jira/browse/SPARK-30667 Project: Spark Issue Type: New Feature Components: PySpark, Spark Core Affects Versions: 3.0.0 Reporter: Xiangrui Meng
Currently we offer task.barrier() to coordinate tasks in barrier mode. Tasks can see all IP addresses from BarrierTaskContext. It would be simpler to integrate with distributed frameworks like TensorFlow DistributionStrategy if we provide all gather that can let tasks share additional information with others, e.g., an available port. Note that with all gather, tasks are share their IP addresses as well. -- This message was sent by Atlassian Jira (v8.3.4#803005) --------------------------------------------------------------------- To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org