[
https://issues.apache.org/jira/browse/SPARK-52076?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Hyukjin Kwon updated SPARK-52076:
---------------------------------
Issue Type: Bug (was: Improvement)
> Explicitly closes ExecutePlanResponseReattachableIterator after usage
> ---------------------------------------------------------------------
>
> Key: SPARK-52076
> URL: https://issues.apache.org/jira/browse/SPARK-52076
> Project: Spark
> Issue Type: Bug
> Components: Connect
> Affects Versions: 4.0.0
> Reporter: Hyukjin Kwon
> Priority: Major
>
> There could be a deadlock:
> ```
> Dumping Threads....
> File "/.../versions/3.9.21/lib/python3.9/threading.py", line 937, in
> _bootstrap
> self._bootstrap_inner()
> File "/.../versions/3.9.21/lib/python3.9/threading.py", line 980, in
> _bootstrap_inner
> self.run()
> File "/.../versions/3.9.21/lib/python3.9/threading.py", line 917, in run
> self._target(*self._args, **self._kwargs)
> File "/.../versions/3.9.21/lib/python3.9/concurrent/futures/thread.py",
> line 85, in _worker
> del work_item
> File "/.../python/pyspark/sql/connect/client/reattach.py", line 347, in
> __del__
> return self.close()
> File "/.../python/pyspark/sql/connect/client/reattach.py", line 343, in
> close
> self._release_all()
> File "/.../python/pyspark/sql/connect/client/reattach.py", line 241, in
> _release_all
> with self._lock:
> ---------------
> File "/.../versions/3.9.21/lib/python3.9/threading.py", line 937, in
> _bootstrap
> self._bootstrap_inner()
> File "/.../versions/3.9.21/lib/python3.9/threading.py", line 980, in
> _bootstrap_inner
> self.run()
> File "/.../versions/3.9.21/lib/python3.9/threading.py", line 917, in run
> self._target(*self._args, **self._kwargs)
> File
> "/.../versions/pyspark-dev-3.9/lib/python3.9/site-packages/grpc/_channel.py",
> line 1751, in channel_spin
> event = state.channel.next_call_event()
> ---------------
> File "<string>", line 44, in <module>
> File "/.../python/pyspark/sql/connect/session.py", line 890, in stop
> self.client.close()
> File "/.../python/pyspark/sql/connect/client/core.py", line 1234, in close
> ExecutePlanResponseReattachableIterator.shutdown()
> File "/.../python/pyspark/sql/connect/client/reattach.py", line 82, in
> shutdown
> cls._get_or_create_release_thread_pool().shutdown()
> File "/.../versions/3.9.21/lib/python3.9/concurrent/futures/thread.py",
> line 235, in shutdown
> t.join()
> File "/.../versions/3.9.21/lib/python3.9/threading.py", line 1060, in join
> self._wait_for_tstate_lock()
> File "/.../versions/3.9.21/lib/python3.9/threading.py", line 1080, in
> _wait_for_tstate_lock
> if lock.acquire(block, timeout):
> File "<string>", line 1, in <module>
> File "<string>", line 1, in <module>
> ```
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
