[
https://issues.apache.org/jira/browse/SPARK-27666?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
wuyi updated SPARK-27666:
-------------------------
Summary: Do not release lock while TaskContext already completed (was:
Stop python runner threads when task finishes)
> Do not release lock while TaskContext already completed
> -------------------------------------------------------
>
> Key: SPARK-27666
> URL: https://issues.apache.org/jira/browse/SPARK-27666
> Project: Spark
> Issue Type: Improvement
> Components: SQL
> Affects Versions: 3.0.0
> Reporter: Wenchen Fan
> Priority: Major
>
> When a task finishes, we should stop the threads that are spawned by the
> Python runner immediately.
> https://github.com/apache/spark/pull/24542 fixed the problem that, if the
> Python runner thread releases the block lock at the end, and the block lock
> is already released when task finishes, we will hit assertion error.
> If we can stop the python runner thread when task finishes, we don't need
> https://github.com/apache/spark/pull/24542 any more.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
