squito commented on issue #24898: [SPARK-22340][PYTHON] Add a mode to pin Python thread into JVM's URL: https://github.com/apache/spark/pull/24898#issuecomment-529485760 I think the biggest problem with the pinned mode is that the lifecycle of the python & jvm threads aren't tied together at all. If you make many short-lived threads in python, I don't py4j will automatically clean up the corresponding java thread when the python thread dies. that seems like a big problem. I suppose there is also the reverse problem -- the python threads don't die if the jvm thread dies -- but I think for pyspark that isn't a problem, since spark itself manages those threads and there are never many of them. It seems OK to me that the JVM will have a few more active threads, as long as they correspond to threads in active use in python, and that set doesn't just keep growing forever.
---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services --------------------------------------------------------------------- To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
