Davies Liu created SPARK-3030:
---------------------------------
Summary: reuse python worker
Key: SPARK-3030
URL: https://issues.apache.org/jira/browse/SPARK-3030
Project: Spark
Issue Type: Improvement
Components: PySpark
Reporter: Davies Liu
Currently, it will fork an Python worker for each task, it will better if we
can reuse the worker for later tasks.
This will be very useful for large dataset with big broadcast, so it does not
need to sending broadcast to worker again and again. Also it can reduce the
overhead of launch a task.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
