[issue43895] Unnecessary Cache of Shared Object Handles
Change by Ian H : -- keywords: +patch pull_requests: +24282 stage: -> patch review pull_request: https://github.com/python/cpython/pull/25487 ___ Python tracker <https://bugs.python.org/issue43895> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue43895] Unnecessary Cache of Shared Object Handles
Ian H added the comment: Proposed patch is in https://github.com/python/cpython/pull/25487. -- ___ Python tracker <https://bugs.python.org/issue43895> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue43895] Unnecessary Cache of Shared Object Handles
New submission from Ian H : While working on another project I noticed that there's a cache of shared object handles kept inside _PyImport_FindSharedFuncptr. See https://github.com/python/cpython/blob/b2b6cd00c6329426fc3b34700f2e22155b44168c/Python/dynload_shlib.c#L51-L55. It appears to be an optimization to work around poor caching of shared object handles in old libc implementations. After some testing, I have been unable to find any meaningful performance difference from this cache, so I propose we remove it to save the space. My initial tests were on Linux (Ubuntu 18.04). I saw no discernible difference in the time for running the Python test suite with a single thread. Running the test suite using a single thread shows a lot of variance, but after running with and without the cache 40 times the mean times with/without the cache was nearly the same. Interpreter startup time also appears to be unaffected. This was all with a debug build, so I'm in the process of collecting data with a release build to see if that changes anything. -- messages: 391453 nosy: Ian.H priority: normal severity: normal status: open title: Unnecessary Cache of Shared Object Handles versions: Python 3.10 ___ Python tracker <https://bugs.python.org/issue43895> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue43870] C API Functions Bypass __import__ Override
New submission from Ian H : Some of the import-related C API functions are documented as bypassing an override to builtins.__import__. This appears to be the case, but the documentation is incomplete in this regard. For example, PyImport_ImportModule is implemented by calling PyImport_Import which does respect an override to builtins.__import__, but PyImport_ImportModule doesn't mention respecting an override. On the other hand some routines (like PyImport_ImportModuleLevelObject) do not respect an override to the builtin import. Is this something that people are open to having fixed? I've been working on an academic project downstream that involved some overrides to the __import__ machinery (I haven't figured out a way to do this with just import hooks) and having some modules skip going through our override threw us for a bad debugging loop. The easiest long-term fix from our perspective is to patch the various PyImport routines to always respect an __import__ override. This technically is a backwards compatibility break, but I'm unsure if anyone is actually relying on the fact that specific C API functions bypass builtins.__import__ entirely. It seems more likely that the current behavior will cause bugs downstream like it did for us. -- messages: 391220 nosy: Ian.H priority: normal severity: normal status: open title: C API Functions Bypass __import__ Override type: behavior versions: Python 3.9 ___ Python tracker <https://bugs.python.org/issue43870> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue43819] ExtensionFileLoader Does Not Implement invalidate_caches
New submission from Ian H : Currently there's no easy way to get at the internal cache of module spec objects for compiled extension modules. See https://github.com/python/cpython/blob/20ac34772aa9805ccbf082e700f2b033291ff5d2/Python/import.c#L401-L415. For example, these module spec objects continue to be cached even if we call importlib.invalidate_caches. ExtensionFileLoader doesn't implement the corresponding method for this. The comment in the C file referenced above implies this is done this way to avoid re-initializing extension modules. I'm not sure if this can be fixed, but I figured I'd ask for input. Our use-case is an academic project where we've been experimenting with building an interface for linker namespaces into Python to allow for (among other things) loading multiple copies of any module without explicit support from that module. We've been able to do this without having custom builds of Python. We've instead gone the route of overriding some of the import machinery at runtime. To make this work we need a way to prevent caching of previous import-related information about a specific extension module. We currently have to rely on an unfortunate hack to get access to the internal cache of module spec objects for extension modules and modify that dictionary manually. What we have works, but any sort of alternative would be welcome. -- messages: 390905 nosy: Ian.H priority: normal severity: normal status: open title: ExtensionFileLoader Does Not Implement invalidate_caches type: behavior versions: Python 3.9 ___ Python tracker <https://bugs.python.org/issue43819> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
