[issue36476] Runtime finalization assumes all other threads have exited.

[STINNER Victor]

Change by STINNER Victor :


--
components: +Subinterpreters

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue36476] Runtime finalization assumes all other threads have exited.

[Nick Coghlan]

Nick Coghlan  added the comment:

Thinking about that idea further, I don't think that change would help much, 
since the relevant operations should already be checking for thread termination 
when they attempt to reacquire the GIL.

That means what we're missing is:

1. When daemon threads still exist after the non-daemon threads terminate, 
deliberately giving them additional time to run (and hence terminate)
2. Explicitly attempting to kick daemon threads out of blocking system calls by 
sending them signals to provoke EINTR (I have no idea if there's a windows 
equivalent for this, but we should be able to use pthread_kill on POSIX 
systems. However, choosing *which* wakeup signal to send could be fraught with 
compatibility problems)

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue36476] Runtime finalization assumes all other threads have exited.

[Nick Coghlan]

Nick Coghlan  added the comment:

Perhaps we need a threading.throw() API, similar to the one we have for 
generators and coroutines?

If we had that, then Py_FinalizeEx() could gain a few new features:

* throw SystemExit into all daemon threads and then give them a chance to 
terminate before calling any atexit handlers (printing a warning if some of the 
threads don't exit)
* throw SystemExit into all daemon and non-daemon threads after running atexit 
handlers (printing a warning if any such threads exist at all, along with 
another warning if some of the threads don't exit)

Adding that would require an enhancement to the PendingCall machinery, though, 
since most pending calls are only processed in the main thread (there's no way 
to route them to specific child threads).

A simpler alternative would be to have an atomic "terminate_threads" counter in 
the ceval runtime state that was incremented to 1 to request that SystemExit be 
raised in daemon threads, and then to 2 to request that SystemExit be raised in 
all still running threads. When a thread received that request to exit, it 
would set a new flag in the thread state to indicate it was terminating, and 
then raise SystemExit. (The thread running Py_FinalizeEx would set that flag in 
advance so it wasn't affected, and other threads would use it to ensure they 
only raised SystemExit once). The runtime cost of this would just be another 
_Py_atomic_load_relaxed call in the eval_breaker branch. (Probably inside 
`make_pending_calls`, so it gets triggered both by the eval_breaker logic, and 
by explicit calls to `Py_MakePendingCalls`).

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue36476] Runtime finalization assumes all other threads have exited.

[Antoine Pitrou]

Antoine Pitrou  added the comment:

> 1. Python daemon threads

I think the answer is to document a bit more clearly that they can pose all 
kinds of problems.  Perhaps we could even display a visible warning when people 
create daemon threads.

> 2. Python threads created in atexit handlers

We could run the "join non-daemon threads" routine a *second time* after atexit 
handlers have been called.  It probably can't hurt (unless people do silly 
things?).

> 3. non-Python threads accessing the C-API

This one I don't know how to handle. By construction, a non-Python thread can 
do anything it wants, and we cannot add guards against this at the beginning of 
each C API function. I think that when someone calls the C API, we're clearly 
in the realm of "consenting adults".

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue36476] Runtime finalization assumes all other threads have exited.

[Eric Snow]

Eric Snow  added the comment:

So I see 3 things to address here:

1. Python daemon threads
2. Python threads created in atexit handlers
3. non-Python threads accessing the C-API

Possible solutions (starting point for discussion):

1. stop them at the point we stop waiting for non-daemon threads (at the 
beginning of finalization)
2. disallow them?  do one more pass of wait-for-threads?
3. cause all (external) attempts to access the C-API to fail once finalization 
begins

Regarding daemon threads, the docs already say "Daemon threads are abruptly 
stopped at shutdown." [1]  So let's force them to stop.  Can we do that?  If we 
*can* simply kill the threads, can we do so without leaking resources?  
Regardless, the mechanism we currently use (check for finalizing each(?) time 
through the eval loop) mostly works fine.  The problem is when C code called 
from Python in a daemon thread blocks long enough that it makes C-API calls (or 
even the eval loop) *after* we've started cleaning up the runtime state.  So if 
there was a way to interrupt that blocking code, that would probably be good 
enough.

The other two possible solutions are, I suppose, a bit more drastic.  What are 
the alternatives?


[1] https://docs.python.org/3/library/threading.html#thread-objects

--
nosy: +nanjekyejoannah, ncoghlan, pablogsal, pitrou, tim.peters, vstinner

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue36476] Runtime finalization assumes all other threads have exited.

[Eric Snow]

Eric Snow  added the comment:

To put it another way:

(from issue33608#msg358748)

> The docs [1] aren't super clear about it, but there are some fundamental
> assumptions we make about runtime finalization:
>
> * no use of the C-API while Py_FinalizeEx() is executing (except for a
> few helpers like Py_Initialized)
> * only a small portion of the C-API is available afterward (at least
> until Py_Initialize() is run)
>
> I guess the real question is what to do about this?
> 
>[1] https://docs.python.org/3/c-api/init.html#c.Py_FinalizeEx

Adding to that list:

* no other Python threads are running once we start finalizing the runtime (not 
far into Py_FinalizeEx())

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue36476] Runtime finalization assumes all other threads have exited.

[Eric Snow]

Eric Snow  added the comment:

Analysis by @pconnell:

* https://bugs.python.org/issue33608#msg357169
* https://bugs.python.org/issue33608#msg357170
* https://bugs.python.org/issue33608#msg357179

tl;dr daemon threads and external C-API access during/after runtime 
finalization are causing crashes.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue36476] Runtime finalization assumes all other threads have exited.

[Eric Snow]

Change by Eric Snow :


--
nosy: +pconnell

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue36476] Runtime finalization assumes all other threads have exited.

[Eric Snow]

Eric Snow  added the comment:

Problems with lingering threads during/after runtime finalization continue to 
be a problem.  I'm going to use this issue as the focal point for efforts to 
resolve this.


Related issues:
* #36479 "Exit threads when interpreter is finalizing rather than runtime."
* #24770 "Py_Finalize() doesn't stop daemon threads"
* #23592 "SIGSEGV on interpreter shutdown, with daemon threads running wild"
* #37127 "Handling pending calls during runtime finalization may cause 
problems."
* #33608 "Add a cross-interpreter-safe mechanism to indicate that an object may 
be destroyed."
* #36818 "Add PyInterpreterState.runtime."
* #36724 "Clear _PyRuntime at exit"
* #14073 "allow per-thread atexit()"
* #1596321 "KeyError at exit after 'import threading' in other thread"
* #37266 "Daemon threads must be forbidden in subinterpreters"
* #31517 "MainThread association logic is fragile"

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue36476] Runtime finalization assumes all other threads have exited.

[Eric Snow]

Change by Eric Snow :


--
stage:  -> needs patch
versions: +Python 3.9 -Python 3.7

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue36476] Runtime finalization assumes all other threads have exited.

[Eric Snow]

Eric Snow  added the comment:

Adding to the list:

* any OS threads created by an extension module or embedding application

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue36476] Runtime finalization assumes all other threads have exited.

[Eric Snow]

Eric Snow  added the comment:

FYI, I've opened issue36477 to deal with the subinterpreters case.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue36476] Runtime finalization assumes all other threads have exited.

[Eric Snow]

New submission from Eric Snow :

Among the first 3 things that happen in Py_FinalizeEx() are, in order:

1. wait for all non-daemon threads (of the main interpreter) to finish
2. call registered atexit funcs
3. mark the runtime as finalizing

At that point the only remaining Python threads are:

* the main thread (where finalization is happening)
* daemon threads
* non-daemon threads created in atexit functions
* any threads belonging to subinterpreters

The next time any of those threads (aside from main) acquire the GIL, we expect 
that they will exit via a call to PyThread_exit_thread() (caveat: issue 
#36475).  However, we have no guarantee on when that will happen, if ever.  
Such lingering threads can cause problems, including crashes and deadlock (see 
issue #36469).

I don't know what else we can do, beyond what we're already doing.  Any ideas?

--
components: Interpreter Core
messages: 339143
nosy: eric.snow
priority: normal
severity: normal
status: open
title: Runtime finalization assumes all other threads have exited.
type: behavior
versions: Python 3.7, Python 3.8

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com