[issue46070] broken subinterpreters

[STINNER Victor]

STINNER Victor  added the comment:

Using attached bug.py, it's possible to trigger the crash on the main branch. I 
modified the reproducer to use the "_asyncio" extension which still uses the 
old API PyModule_Create() with PyModuleDef.m_size = -1.

--
Added file: https://bugs.python.org/file50497/bug.py

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue46070] broken subinterpreters

[STINNER Victor]

STINNER Victor  added the comment:

I can sometimes reproduce the crash on Windows with Python 3.9. Call stack 
(most recent to oldest frames):

* PyObject_GC_UnTrack() - crash on _PyGCHead_SET_NEXT(prev, next) because prev 
is dangling pointer (0x1fe64dd5250), Visual Studio is unable to read memory
* meth_dealloc() -- delete _sre_compile() method object
* (...)
* PyDict_SetItem() -- set "compile" to None
* _PyModule_ClearDict() -- clear the "_sre" module dict
* _PyModule_Clear()
* _PyImport_Clenaup()
* Py_EndInterpreter()
* (...)
* run_in_subinterp()
* (...)
* t_bootstrap()

The crash occurs in meth_dealloc(), when deallocating the _sre_compile() method 
object stored in _sre module dictionary as the attribute "compile".

The PyGC_Head.prev pointer is a dangling pointer.

On Python 3.9, the "re" module is not imported at startup, but it's imported 
indirectly by "import importlib.util" via "import typing". On Python 3.10, the 
re module is no longer imported by "import importlib.util".

The crash is random. Sometimes, I need 3 or 4 tries. Sometimes, it crash using 
-X dev. Sometimes, it crash immediately. When debugging in Visual Stuido, the 
crash seems easier to reproduce.

On Python 3.9, the _sre exetnsion uses the old API: PyModule_Create() with 
PyModuleDef.m_size = -1.

On Python 3.10, the _sre extension has been converted to multiphase init API: 
PyModuleDef_Init() with PyModuleDef.m_size = sizeof(_sremodulestate). Moreover, 
"import importlib.util" no longer imports indirectly the "re" module.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue46070] broken subinterpreters

[Ben Steffensmeier]

Ben Steffensmeier  added the comment:

We have been seeing intermittent crashes on jep that we tracked down to the 
same change (d0d29655ff).

I have created a sample program using _testcapi that crashes about 50% of the 
time when run on Windows with Python 3.9.9. We have not been able to reproduce 
problems on any other OS.

See also: https://github.com/ninia/jep/issues/366

--
nosy: +bsteffensmeier
Added file: https://bugs.python.org/file50494/win_py399_crash_reproducer.py

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue46070] broken subinterpreters

[Nathan Jensen]

Change by Nathan Jensen :


--
nosy: +ndjensen

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue46070] broken subinterpreters

[Petr Viktorin]

Petr Viktorin  added the comment:

That was a fix for GH-17350, which might need to be reverted as well.

Victor, could you take another look at GH-17350? I must admit I (still) don't 
understand this change; what would break if it was reverted (along with the 
fixup from bpo-44050)?

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue46070] broken subinterpreters

[graysky]

graysky  added the comment:

While this is being evaluated, can someone give an opinion about the sanity of 
simply reverting https://hg.python.org/lookup/d0d29655ff for now in order to 
use 3.10.1?  Thanks.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue46070] broken subinterpreters

[graysky]

graysky  added the comment:

@Eric - I have not seen this on 3.8 or 3.9.  No data before 3.8.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue46070] broken subinterpreters

[Petr Viktorin]

Petr Viktorin  added the comment:

Interned strings were broken in GH-20058, see bpo-46006. Maybe that's also the 
issue here?

--
nosy: +vstinner

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue46070] broken subinterpreters

[Eric Snow]

Eric Snow  added the comment:

(related: bpo-44059)

Presumably the problem relates to global state used in different interpreters 
leading to an inconsistent state in the crashing extension (or its 
dependencies).

@graysky, do you know if this was a problem before Python 3.8?

--
components: +Subinterpreters
nosy: +eric.snow, petr.viktorin, shihai1991
stage:  -> test needed
type:  -> behavior
versions: +Python 3.11, Python 3.9

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue46070] broken subinterpreters

[graysky]

New submission from graysky :

Seems as though cpython is broken when working with subinterpreters. The 
problematic change could be (d0d29655ff) affecting import.c.[1] Reverting this 
commit and rebuilding python fixes the issues on my system with some scripts 
that import sqlite, for example, the Kodi plugin YouTube[2] and the IMDB 
Trailers plugin. Others have reported similar breakage with other python 
code[3].

Example output when the bug manifests:

ERROR : Traceback (most recent call last):
ERROR :   File "/usr/lib/python3.10/sqlite3/dbapi2.py", line 83, in 
register_adapters_and_converters
ERROR : 
ERROR : register_converter("timestamp", convert_timestamp)
ERROR : 
ERROR : KeyError
ERROR : : 
ERROR : 'timepart_full'
ERROR :  
ERROR : Exception ignored deletion of interned string failed
ERROR : :


References:
1. 
https://github.com/python/cpython/commit/d0d29655ffc43d426ad68542d8de8304f7f1346a
2. https://github.com/anxdpanic/plugin.video.youtube/issues/255
3. https://bbs.archlinux.org/viewtopic.php?id=272121

--
components: Interpreter Core
messages: 408520
nosy: graysky
priority: normal
severity: normal
status: open
title: broken subinterpreters
versions: Python 3.10

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com