Wenchen Fan created SPARK-27666:
-----------------------------------
Summary: Stop python runner threads when task finishes
Key: SPARK-27666
URL: https://issues.apache.org/jira/browse/SPARK-27666
Project: Spark
Issue Type: Improvement
Components: SQL
Affects Versions: 3.0.0
Reporter: Wenchen Fan
When a task finishes, we should stop the threads that are spawned by the Python
runner immediately.
https://github.com/apache/spark/pull/24542 fixed the problem that, if the
Python runner thread releases the block lock at the end, and the block lock is
already released when task finishes, we will hit assertion error.
If we can stop the python runner thread when task finishes, we don't need
https://github.com/apache/spark/pull/24542 any more.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
