[ https://issues.apache.org/jira/browse/SPARK-35498?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17350273#comment-17350273 ]
Apache Spark commented on SPARK-35498: -------------------------------------- User 'WeichenXu123' has created a pull request for this issue: https://github.com/apache/spark/pull/32644 > Add an API "inheritable_thread_target" which return a wrapped thread target > for pyspark pin thread mode > ------------------------------------------------------------------------------------------------------- > > Key: SPARK-35498 > URL: https://issues.apache.org/jira/browse/SPARK-35498 > Project: Spark > Issue Type: Improvement > Components: PySpark > Affects Versions: 3.2.0 > Reporter: Weichen Xu > Priority: Major > > In pyspark, user may create some threads, not via `Thread` object, but via > some parallel helper function such as: > `thread_pool.imap_unordered` > In this case, we need create a function wrapper, used in pin thread mode, The > wrapper function, before calling original thread target, it inherits the > inheritable properties specific to JVM thread such as > ``InheritableThreadLocal``, and after original thread target return, > garbage-collects the Python thread instance and also closes the connection > which finishes JVM thread correctly. -- This message was sent by Atlassian Jira (v8.3.4#803005) --------------------------------------------------------------------- To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org