Julien Danjou <jul...@danjou.info> added the comment:
> Is there a real-world situation where it's specifically necessary or even > beneficial to utilize ThreadPoolExecutor at this point after thread > finalization rather than earlier in the program? Not that it doesn't exist, > but to me it intuitively seems very odd to utilize an executor within an > atexit hook, which are intended to just be resource finalization/cleanup > functions called at interpreter shutdown. Assuming there is a genuine use > case I'm not seeing, it may be worth weighing against the decision to convert > the executors to not use daemon threads, as I presently don't think there's a > way to (safely) allow that behavior without reverting back to using daemon > threads. To put that in perspective, here is the original issue that trigged this bug for me: Traceback (most recent call last): File "/root/project/ddtrace/profiling/scheduler.py", line 50, in flush exp.export(events, start, self._last_export) File "/root/project/ddtrace/profiling/exporter/http.py", line 186, in export self._upload(client, self.endpoint_path, body, headers) File "/root/project/ddtrace/profiling/exporter/http.py", line 189, in _upload self._retry_upload(self._upload_once, client, path, body, headers) File "/root/project/.tox/py39-profile-gevent/lib/python3.9/site-packages/tenacity/__init__.py", line 423, in __call__ do = self.iter(retry_state=retry_state) File "/root/project/.tox/py39-profile-gevent/lib/python3.9/site-packages/tenacity/__init__.py", line 360, in iter return fut.result() File "/root/.pyenv/versions/3.9.0/lib/python3.9/concurrent/futures/_base.py", line 433, in result return self.__get_result() File "/root/.pyenv/versions/3.9.0/lib/python3.9/concurrent/futures/_base.py", line 389, in __get_result raise self._exception File "/root/project/.tox/py39-profile-gevent/lib/python3.9/site-packages/tenacity/__init__.py", line 426, in __call__ result = fn(*args, **kwargs) File "/root/project/ddtrace/profiling/exporter/http.py", line 193, in _upload_once client.request("POST", path, body=body, headers=headers) File "/root/.pyenv/versions/3.9.0/lib/python3.9/http/client.py", line 1255, in request self._send_request(method, url, body, headers, encode_chunked) File "/root/.pyenv/versions/3.9.0/lib/python3.9/http/client.py", line 1301, in _send_request self.endheaders(body, encode_chunked=encode_chunked) File "/root/.pyenv/versions/3.9.0/lib/python3.9/http/client.py", line 1250, in endheaders self._send_output(message_body, encode_chunked=encode_chunked) File "/root/.pyenv/versions/3.9.0/lib/python3.9/http/client.py", line 1010, in _send_output self.send(msg) File "/root/.pyenv/versions/3.9.0/lib/python3.9/http/client.py", line 950, in send self.connect() File "/root/.pyenv/versions/3.9.0/lib/python3.9/http/client.py", line 921, in connect self.sock = self._create_connection( File "/root/project/.tox/py39-profile-gevent/lib/python3.9/site-packages/gevent/socket.py", line 88, in create_connection addrs = list(getaddrinfo(host, port, 0, SOCK_STREAM)) File "/root/project/.tox/py39-profile-gevent/lib/python3.9/site-packages/gevent/_socketcommon.py", line 247, in getaddrinfo addrlist = get_hub().resolver.getaddrinfo(host, port, family, type, proto, flags) File "/root/project/.tox/py39-profile-gevent/lib/python3.9/site-packages/gevent/hub.py", line 841, in _get_resolver self._resolver = self.resolver_class(hub=self) # pylint:disable=not-callable File "/root/project/.tox/py39-profile-gevent/lib/python3.9/site-packages/gevent/resolver/thread.py", line 39, in __init__ self.pool = hub.threadpool File "/root/project/.tox/py39-profile-gevent/lib/python3.9/site-packages/gevent/hub.py", line 865, in _get_threadpool self._threadpool = self.threadpool_class(self.threadpool_size, hub=self) File "/root/project/.tox/py39-profile-gevent/lib/python3.9/site-packages/gevent/hub.py", line 860, in threadpool_class return GEVENT_CONFIG.threadpool File "/root/project/.tox/py39-profile-gevent/lib/python3.9/site-packages/gevent/_config.py", line 50, in getter return self.settings[setting_name].get() File "/root/project/.tox/py39-profile-gevent/lib/python3.9/site-packages/gevent/_config.py", line 146, in get self.value = self.validate(self._default()) File "/root/project/.tox/py39-profile-gevent/lib/python3.9/site-packages/gevent/_config.py", line 248, in validate return self._import_one_of([self.shortname_map.get(x, x) for x in value]) File "/root/project/.tox/py39-profile-gevent/lib/python3.9/site-packages/gevent/_config.py", line 223, in _import_one_of return self._import_one(candidates[-1]) File "/root/project/.tox/py39-profile-gevent/lib/python3.9/site-packages/gevent/_config.py", line 237, in _import_one module = importlib.import_module(module) File "/root/.pyenv/versions/3.9.0/lib/python3.9/importlib/__init__.py", line 127, in import_module return _bootstrap._gcd_import(name[level:], package, level) File "<frozen importlib._bootstrap>", line 1030, in _gcd_import File "<frozen importlib._bootstrap>", line 1007, in _find_and_load File "<frozen importlib._bootstrap>", line 986, in _find_and_load_unlocked File "<frozen importlib._bootstrap>", line 680, in _load_unlocked File "<frozen importlib._bootstrap_external>", line 790, in exec_module File "<frozen importlib._bootstrap>", line 228, in _call_with_frames_removed File "/root/project/.tox/py39-profile-gevent/lib/python3.9/site-packages/gevent/threadpool.py", line 748, in <module> class ThreadPoolExecutor(concurrent.futures.ThreadPoolExecutor): File "/root/.pyenv/versions/3.9.0/lib/python3.9/concurrent/futures/__init__.py", line 49, in __getattr__ from .thread import ThreadPoolExecutor as te File "/root/.pyenv/versions/3.9.0/lib/python3.9/concurrent/futures/thread.py", line 37, in <module> threading._register_atexit(_python_exit) File "/root/.pyenv/versions/3.9.0/lib/python3.9/threading.py", line 1370, in _register_atexit raise RuntimeError("can't register atexit after shutdown") RuntimeError: can't register atexit after shutdown What's happening is that the ddtrace library registers an atexit hook that does an HTTP call. As the application runs using gevent, some gevent modules that were not loaded are loaded during the atexit() hook and the loading of `concurrent.futures.thread` is done very late, at the point where the interpreter is shutting down. I'm totally fine blaming gevent here if you prefer. The problem is that there's nothing preventing any library call to be made in an atexit() hook, and any library could decide to use `concurent.futures` without the library user being able to do anything about it, except maybe, making sure `concurent.futures` is loaded very early in the program. However, having to load this library even if you don't use it to be sure it does not break would break the separation of principles. At at least, at this stage, it might be the responsibility of Python to make sure all `threading._register_at_exit` calls are done whatever if the library is used or not (i.e. that'd mean loading `concurent.futures` with `threading` unconditionally). ---------- _______________________________________ Python tracker <rep...@bugs.python.org> <https://bugs.python.org/issue42647> _______________________________________ _______________________________________________ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com