Hyukjin Kwon created SPARK-52076:
------------------------------------
Summary: Explicitly closes ExecutePlanResponseReattachableIterator
after usage
Key: SPARK-52076
URL: https://issues.apache.org/jira/browse/SPARK-52076
Project: Spark
Issue Type: Improvement
Components: Connect
Affects Versions: 4.0.0
Reporter: Hyukjin Kwon
There could be a deadlock:
```
Dumping Threads....
File "/.../versions/3.9.21/lib/python3.9/threading.py", line 937, in
_bootstrap
self._bootstrap_inner()
File "/.../versions/3.9.21/lib/python3.9/threading.py", line 980, in
_bootstrap_inner
self.run()
File "/.../versions/3.9.21/lib/python3.9/threading.py", line 917, in run
self._target(*self._args, **self._kwargs)
File "/.../versions/3.9.21/lib/python3.9/concurrent/futures/thread.py", line
85, in _worker
del work_item
File "/.../python/pyspark/sql/connect/client/reattach.py", line 347, in
__del__
return self.close()
File "/.../python/pyspark/sql/connect/client/reattach.py", line 343, in close
self._release_all()
File "/.../python/pyspark/sql/connect/client/reattach.py", line 241, in
_release_all
with self._lock:
---------------
File "/.../versions/3.9.21/lib/python3.9/threading.py", line 937, in
_bootstrap
self._bootstrap_inner()
File "/.../versions/3.9.21/lib/python3.9/threading.py", line 980, in
_bootstrap_inner
self.run()
File "/.../versions/3.9.21/lib/python3.9/threading.py", line 917, in run
self._target(*self._args, **self._kwargs)
File
"/.../versions/pyspark-dev-3.9/lib/python3.9/site-packages/grpc/_channel.py",
line 1751, in channel_spin
event = state.channel.next_call_event()
---------------
File "<string>", line 44, in <module>
File "/.../python/pyspark/sql/connect/session.py", line 890, in stop
self.client.close()
File "/.../python/pyspark/sql/connect/client/core.py", line 1234, in close
ExecutePlanResponseReattachableIterator.shutdown()
File "/.../python/pyspark/sql/connect/client/reattach.py", line 82, in
shutdown
cls._get_or_create_release_thread_pool().shutdown()
File "/.../versions/3.9.21/lib/python3.9/concurrent/futures/thread.py", line
235, in shutdown
t.join()
File "/.../versions/3.9.21/lib/python3.9/threading.py", line 1060, in join
self._wait_for_tstate_lock()
File "/.../versions/3.9.21/lib/python3.9/threading.py", line 1080, in
_wait_for_tstate_lock
if lock.acquire(block, timeout):
File "<string>", line 1, in <module>
File "<string>", line 1, in <module>
```
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
