[issue46965] Enable informing callee it's awaited via vector call flag
Dino Viehland added the comment: Doh, sorry about that link, this one goes to a specific commit: https://github.com/facebookincubator/cinder/blob/6863212ada4b569c15cd95c4e7a838f254c8ccfb/Python/ceval.c#L6642 I do think a new opcode is a good way to go, and that could just be emitted by the compiler when it recognizes the pattern. I think we mainly avoided that because we had some issues around performance testing when we updated the byte code version and the peek was negligible, but with improved call performance in 3.11 that may not be the case anymore. It's probably possible to keep most of gather in Python if necessary, there'd still need to be a C wrapper which could flow the wrapper in and the wait handle creation would need to be possible from Python (which slightly scares me). There's probably also a perf win from the C implementation - I'll see if @v2m has any data on that. -- ___ Python tracker <https://bugs.python.org/issue46965> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue46965] Enable informing callee it's awaited via vector call flag
New submission from Dino Viehland : The idea here is to add a new flag to the vectorcall nargs that indicates the call is being awaited: _Py_AWAITED_CALL_MARKER. This flag will allow the callee to know that it's being eagerly evaluated. When the call is eagerly evaluated the callee can potentially avoid various amounts of overhead. For a coroutine the function can avoid creating the coroutine object and instead returns a singleton instance of a wait handle indicating eager execution has occurred: https://github.com/facebookincubator/cinder/blob/cinder/3.8/Python/ceval.c#L6617 This gives a small win by reducing the overhead of allocating the co-routine object. For something like gather much more significant wins can be achieved. If all of the inputs have already been computed the creation of tasks and scheduling of them to the event loop can be elided. An example implementation of this is available in Cinder: https://github.com/facebookincubator/cinder/blob/cinder/3.8/Modules/_asynciomodule.c#L7103 Again the gather implementation uses the singleton wait handle object to return the value indicating the computation completed synchronously. We've used this elsewhere in Cinder as well - for example if we have an "AsyncLazyValue" which lazily performs a one-time computation of a value and caches it. Therefore the common case becomes that the value is already available, and the await can be performed without allocating any intermediate values. -- assignee: dino.viehland messages: 414782 nosy: Mark.Shannon, carljm, dino.viehland, gvanrossum, itamaro priority: normal severity: normal stage: needs patch status: open title: Enable informing callee it's awaited via vector call flag type: performance versions: Python 3.11 ___ Python tracker <https://bugs.python.org/issue46965> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue30533] missing feature in inspect module: getmembers_static
Change by Dino Viehland : -- stage: patch review -> resolved status: open -> closed ___ Python tracker <https://bugs.python.org/issue30533> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue30533] missing feature in inspect module: getmembers_static
Dino Viehland added the comment: New changeset c2bb29ce9ae4adb6a8123285ad3585907cd4cc73 by Weipeng Hong in branch 'main': bpo-30533: Add docs for `inspect.getmembers_static` (#29874) https://github.com/python/cpython/commit/c2bb29ce9ae4adb6a8123285ad3585907cd4cc73 -- ___ Python tracker <https://bugs.python.org/issue30533> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue30533] missing feature in inspect module: getmembers_static
Change by Dino Viehland : -- stage: patch review -> resolved status: open -> closed versions: +Python 3.11 -Python 3.9 ___ Python tracker <https://bugs.python.org/issue30533> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue30533] missing feature in inspect module: getmembers_static
Dino Viehland added the comment: New changeset af8c8caaf5e07c02202d736a31f6a2f7e27819b8 by Weipeng Hong in branch 'main': bpo-30533:Add function inspect.getmembers_static that does not call properties or dynamic properties. (#20911) https://github.com/python/cpython/commit/af8c8caaf5e07c02202d736a31f6a2f7e27819b8 -- nosy: +dino.viehland ___ Python tracker <https://bugs.python.org/issue30533> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue43636] test_descr fails randomly when executed with -R :
Dino Viehland added the comment: @vstinner - The fix doesn't change the behavior of _PyType_Lookup and instead just fixes a previous unrelated bug. The condition will typically ever be hit once (at startup) as it's very unlikely that versions will wrap, so there should be no performance difference after the fix. -- resolution: fixed -> stage: resolved -> status: closed -> open ___ Python tracker <https://bugs.python.org/issue43636> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue43636] test_descr fails randomly when executed with -R :
Dino Viehland added the comment:
I think the issue here is that in assign_version_tag there's this code:
if (type->tp_version_tag == 0) {
// Wrap-around or just starting Python - clear the whole cache
type_cache_clear(cache, 1);
return 1;
}
the return 1 is incorrect, it should be return 0 as a valid version tag hasn't
been assigned.
--
___
Python tracker
<https://bugs.python.org/issue43636>
___
___
Python-bugs-list mailing list
Unsubscribe:
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue43636] test_descr fails randomly when executed with -R :
Dino Viehland added the comment:
And it looks like we have an entry with a 0 version, but with a valid name:
(type_cache_entry) $3 = {
version = 0
name = 0x000100ec44f0
value = NULL
}
--
___
Python tracker
<https://bugs.python.org/issue43636>
___
___
Python-bugs-list mailing list
Unsubscribe:
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue43636] test_descr fails randomly when executed with -R :
Dino Viehland added the comment: It's probably worth having an assert that the version tag is valid, that'll make it easier to see what's going wrong in the cache hit case. We should have the version tag being 0 now when it's not valid. I may not be able to debug it anymore tonight, but if no will look tomorrow. -- ___ Python tracker <https://bugs.python.org/issue43636> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue43636] test_descr fails randomly when executed with -R :
Dino Viehland added the comment: Looking! -- ___ Python tracker <https://bugs.python.org/issue43636> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue43452] Microoptimize PyType_Lookup for cache hits
Dino Viehland added the comment:
Setup a micro-benchmark, foo.c:
#define PY_SSIZE_T_CLEAN
#include
#include
#include
int
main(int argc, char *argv[])
{
wchar_t *program = Py_DecodeLocale(argv[0], NULL);
if (program == NULL) {
fprintf(stderr, "Fatal error: cannot decode argv[0]\n");
exit(1);
}
Py_SetProgramName(program); /* optional but recommended */
Py_Initialize();
PyObject *pName = PyUnicode_DecodeFSDefault("foo");
if (pName == NULL) { printf("no foo\n"); PyErr_Print(); }
PyObject *pModule = PyImport_Import(pName);
if (pModule == NULL) { printf("no mod\n"); PyErr_Print(); return 0; }
PyObject *cls = PyObject_GetAttrString(pModule, "C");
if (cls == NULL) { printf("no cls\n"); }
PyObject *fs[20];
for(int i = 0; i<20; i++) {
char buf[4];
sprintf(buf, "f%d", i);
fs[i] = PyUnicode_DecodeFSDefault(buf);
}
for(int i = 0; i<1; i++) {
for(int j = 0; j<20; j++) {
if(_PyType_Lookup(cls, fs[j])==NULL) {
printf("Uh oh\n");
}
}
}
if (Py_FinalizeEx() < 0) {
exit(120);
}
PyMem_RawFree(program);
return 0;
}
Lib/foo.py:
import time
class C:
pass
for i in range(20):
setattr(C, f"f{i}", lambda self: None)
obj hash: 0m6.222s
str hash: 0m6.327s
baseline: 0m6.784s
--
___
Python tracker
<https://bugs.python.org/issue43452>
___
___
Python-bugs-list mailing list
Unsubscribe:
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue43452] Microoptimize PyType_Lookup for cache hits
Change by Dino Viehland : -- keywords: +patch pull_requests: +23572 stage: -> patch review pull_request: https://github.com/python/cpython/pull/24804 ___ Python tracker <https://bugs.python.org/issue43452> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue43452] Microoptimize PyType_Lookup for cache hits
New submission from Dino Viehland : The common case going through _PyType_Lookup is to have a cache hit. There's some small tweaks which can make this a little cheaper: 1) the name field identity is used for a cache hit, and is kept alive by the cache. So there's no need to read the hash code of the name - instead the address can be used as the hash. 2) There's no need to check if the name is cachable on the lookup either, it probably is, and if it is, it'll be in the cache. 3) If we clear the version tag when invalidating a type then we don't actually need to check for a valid version tag bit. -- components: Interpreter Core messages: 388377 nosy: dino.viehland priority: normal severity: normal status: open title: Microoptimize PyType_Lookup for cache hits versions: Python 3.10 ___ Python tracker <https://bugs.python.org/issue43452> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue42199] bytecode_helper assertNotInBytecode fails too eagerly
Dino Viehland added the comment: New changeset 6e799be0a18d0bb5bbbdc77cd3c30a229d31dfb4 by Max Bernstein in branch 'master': bpo-42199: Fix bytecode_helper assertNotInBytecode (#23031) https://github.com/python/cpython/commit/6e799be0a18d0bb5bbbdc77cd3c30a229d31dfb4 -- nosy: +dino.viehland ___ Python tracker <https://bugs.python.org/issue42199> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue40255] Fixing Copy on Writes from reference counting
Dino Viehland added the comment: I think there's other cases of performance related features being hidden under an ifdef. Computed gotos show up that way, although probably more because it's a compiler extension that's not supported everywhere. Pymalloc is also very similar in that it implies an ABI change as well. I wonder if it would be worth it to introduce an ABI flag for this as well? On the one hand is it a slightly different contract, on the other hand using extensions that don't support the immortalization actually work just fine. -- nosy: +dino.viehland ___ Python tracker <https://bugs.python.org/issue40255> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue26067] test_shutil fails when gid name is missing
Dino Viehland added the comment: New changeset 52268941f37e3e27bd01792b081877ec3bc9ce12 by Matthias Braun in branch 'master': bpo-26067: Do not fail test_shutil / chown when gid/uid cannot be resolved (#19032) https://github.com/python/cpython/commit/52268941f37e3e27bd01792b081877ec3bc9ce12 -- nosy: +dino.viehland ___ Python tracker <https://bugs.python.org/issue26067> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue39551] mock patch should match behavior of import from when module isn't present in sys.modules
Dino Viehland added the comment: I like that idea, let me see if I can find a way to do that. This is a little bit different in that it's implicitly trying to find a module, and supports dotting through non-packages as well, but maybe there's a way to leverage importlib and still keep the existing behavior. -- ___ Python tracker <https://bugs.python.org/issue39551> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue39551] mock patch should match behavior of import from when module isn't present in sys.modules
Dino Viehland added the comment: Sorry, publish may not necessarily be the best term. When you do "from foo import bar" and foo is a package Python will set the bar module onto the foo module. That leads to the common mistake of doing "import foo" and then "foo.bar.baz" and later removing the thing that do "from foo import bar" and having things blow up later. Without additional support there's no way to patch the immutable module. We can provide a mode where we enable the patching for testing and disable it in a production environment though. That basically just involves passing a proxy object down to Mock. And monkey patching the mutable module is perfectly fine. The thing that's doing the ignoring of the assignment is the import system. So it's now okay if the package raises an AttributeError. There's not really a great way to work around this other than just bypassing mock's resolution of the object here - i.e. replacing mock.patch along with _get_target, _importer, and _dot_lookup and calling mock._patch directly, which isn't very nice (but is do-able). And while this is a strange way to arrive at a module existing in sys.modules but not being on the package it is something that can happen in the normal course of imports, hence the reason why the import system handles this by checking sys.modules today. It's also just turning what is currently an error while mocking into a success case with a simple few line change. -- stage: needs patch -> patch review ___ Python tracker <https://bugs.python.org/issue39551> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue39551] mock patch should match behavior of import from when module isn't present in sys.modules
Dino Viehland added the comment:
My actual scenario involves a custom module loader where the modules are
published are completely immutable (it ends up publishing an object which isn't
a subtype of module). It can still have normal Python modules as a child which
aren't immutable, so they could still be patched by Mock (or it could have
immutable sub-packages which Mock wouldn't be able to patch).
So imagine something like this:
immutable_package\__init__.py
__immutable__ = True
x = 2
immutable_package\x.py
y = 2
Doing a "from immutable_package import x" would normally publish "x" as a child
onto the package. But because the package is immutable, this is impossible,
and the assignment is ignored with a warning.
When Mock gets a call to patch on something like "immutable_package.x.y", it's
not going to find x, even though if I were to write "from immutable_package.x
import y" or "from immutable_package import x" it would succeed.
Cases can be contrived without all of this though where the child isn't
published on it's parent, but it requires
x/__init__.py
from x.pkg import child
x/pkg/__init__.py:
x = 1
x/pkg/child.py:
from unittest.mock import patch
y = 42
@patch('x.pkg.child.y', 100)
def f():
print(y)
f()
"python -m x" will fail without the patch but succeed with it.
--
___
Python tracker
<https://bugs.python.org/issue39551>
___
___
Python-bugs-list mailing list
Unsubscribe:
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue39551] mock patch should match behavior of import from when module isn't present in sys.modules
Dino Viehland added the comment: It's related to bpo-39336. If you have an immutable package which doesn't allow it's children to be published on it when following the chain of packages it ends up not arriving at the final module. But you can also hit it if you end up doing the patch during a relative import, although that seems much less likely. But it generally seems matching the behavior of imports would be good to me. -- stage: patch review -> needs patch ___ Python tracker <https://bugs.python.org/issue39551> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue17636] Modify IMPORT_FROM to fallback on sys.modules
Change by Dino Viehland : -- pull_requests: +17722 pull_request: https://github.com/python/cpython/pull/18347 ___ Python tracker <https://bugs.python.org/issue17636> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue39551] mock patch should match behavior of import from when module isn't present in sys.modules
Change by Dino Viehland : -- keywords: +patch pull_requests: +17721 stage: needs patch -> patch review pull_request: https://github.com/python/cpython/pull/18347 ___ Python tracker <https://bugs.python.org/issue39551> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue39551] mock patch should match behavior of import from when module isn't present in sys.modules
New submission from Dino Viehland : The fix for bpo-17636 added support for falling back to sys.modules when a module isn't directly present on the module. But mock doesn't have the same behavior - it'll try the import, and then try to get the value off the object. If it's not there it just errors out. Instead it should also consult sys.modules to be consistent with import semantics. -- assignee: dino.viehland components: Tests messages: 361366 nosy: dino.viehland priority: normal severity: normal stage: needs patch status: open title: mock patch should match behavior of import from when module isn't present in sys.modules type: behavior versions: Python 3.9 ___ Python tracker <https://bugs.python.org/issue39551> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue39459] test_import: test_unwritable_module() fails on AMD64 Fedora Stable Clang Installed 3.x
Dino Viehland added the comment: I guess the update to lib.pyproj probably just makes the files show up when opening the solution in Visual Studio then. -- ___ Python tracker <https://bugs.python.org/issue39459> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue39459] test_import: test_unwritable_module() fails on AMD64 Fedora Stable Clang Installed 3.x
Dino Viehland added the comment: New changeset 0cd5bff6b7da3118d0c5a88fc2b80f80eb7c3059 by Dino Viehland in branch 'master': bpo-39459: include missing test files in windows installer https://github.com/python/cpython/commit/0cd5bff6b7da3118d0c5a88fc2b80f80eb7c3059 -- ___ Python tracker <https://bugs.python.org/issue39459> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue39459] test_import: test_unwritable_module() fails on AMD64 Fedora Stable Clang Installed 3.x
Dino Viehland added the comment: Nope, thank you for pointing that out. I've updated them now with PR 18241 -- ___ Python tracker <https://bugs.python.org/issue39459> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue39459] test_import: test_unwritable_module() fails on AMD64 Fedora Stable Clang Installed 3.x
Change by Dino Viehland : -- pull_requests: +17621 pull_request: https://github.com/python/cpython/pull/18241 ___ Python tracker <https://bugs.python.org/issue39459> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue39459] test_import: test_unwritable_module() fails on AMD64 Fedora Stable Clang Installed 3.x
Dino Viehland added the comment: I've added the files to the makefile and AMD64 Fedora Stable Clang Installed 3.x was passing. -- resolution: -> fixed stage: patch review -> resolved status: open -> closed ___ Python tracker <https://bugs.python.org/issue39459> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue39459] test_import: test_unwritable_module() fails on AMD64 Fedora Stable Clang Installed 3.x
Change by Dino Viehland : -- pull_requests: +17590 pull_request: https://github.com/python/cpython/pull/18211 ___ Python tracker <https://bugs.python.org/issue39459> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue39459] test_import: test_unwritable_module() fails on AMD64 Fedora Stable Clang Installed 3.x
Dino Viehland added the comment: Ahh, that's probably it Brett, I didn't know that was there, thanks! -- ___ Python tracker <https://bugs.python.org/issue39459> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue39459] test_import: test_unwritable_module() fails on AMD64 Fedora Stable Clang Installed 3.x
Dino Viehland added the comment: The curious thing about this is other tests in CircularImportTests are importing packages from test.test_import.data in the exact same way. -- ___ Python tracker <https://bugs.python.org/issue39459> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue38076] Make struct module PEP-384 compatible
Dino Viehland added the comment: One more data point: Backporting this change to Python 3.6 (I just happened to have it applied there already, so I haven't tried it on 3.7 or 3.8) has no crash and no hangs in multiprocessing on Linux. So something definitely changed in multiproessing which is causing the hang on shutdown, and forces us into this code path where we crash as well. -- ___ Python tracker <https://bugs.python.org/issue38076> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue39336] Immutable module type can't be used as package in custom loader
Change by Dino Viehland : -- resolution: -> fixed stage: patch review -> resolved status: open -> closed ___ Python tracker <https://bugs.python.org/issue39336> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue39336] Immutable module type can't be used as package in custom loader
Dino Viehland added the comment: New changeset 9b6fec46513006d7b06fcb645cca6e4f5bf7c7b8 by Dino Viehland in branch 'master': bpo-39336: Allow packages to not let their child modules be set on them (#18006) https://github.com/python/cpython/commit/9b6fec46513006d7b06fcb645cca6e4f5bf7c7b8 -- ___ Python tracker <https://bugs.python.org/issue39336> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue38076] Make struct module PEP-384 compatible
Dino Viehland added the comment: With either fix, or with both, on Linux I still see this hang at shutdown. Victor mentioned the fact that he had to hit Ctrl-C on Linux to see this, and I have to do the same thing. Then with the fixes in place the original test case still hangs on shutdown. On Python 3.7 (I don't readily have 3.8 available) at least this just runs and completes with no ctrl-C and no crashes. So while either of the fixes may be good to prevent the crashes, there's still probably some underlying issue in multiprocessing. I haven't tested on Mac OS/X. It looks like the clearing was originally introduced here: https://bugs.python.org/issue10241 Interestingly there was a similar issue w/ _tkinter, which also used PyType_FromSpec (although it sounds like it was just a ref count issue on the type). Unfortunately there's no associated test cases added to verify the behavior. Antoine and Neil are both now on the PR which removes the collection behavior so hopefully they can chime in on the safety of that fix. -- ___ Python tracker <https://bugs.python.org/issue38076> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue38076] Make struct module PEP-384 compatible
Dino Viehland added the comment: https://github.com/python/cpython/pull/18038 is a partial fix for this. I think it fixes the crash at shutdown, although I'm still seeing a hang on master on Linux which is different then earlier versions of Python. I seem to have a really bogus stack trace when I attach to it so I'm not quite certain what's going on there. -- ___ Python tracker <https://bugs.python.org/issue38076> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue38076] Make struct module PEP-384 compatible
Change by Dino Viehland : -- pull_requests: +17437 stage: resolved -> patch review pull_request: https://github.com/python/cpython/pull/18038 ___ Python tracker <https://bugs.python.org/issue38076> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue38076] Make struct module PEP-384 compatible
Dino Viehland added the comment:
And here's a variation which doesn't involve any instances from the module:
import _struct
class C:
def __init__(self):
self.pack = _struct.pack
def __del__(self):
self.pack('I', -42)
_struct.x = C()
--
___
Python tracker
<https://bugs.python.org/issue38076>
___
___
Python-bugs-list mailing list
Unsubscribe:
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue38076] Make struct module PEP-384 compatible
Dino Viehland added the comment:
This is a relatively simple repro of the underlying problem:
import _struct
s = _struct.Struct('i')
class C:
def __del__(self):
s.pack(42, 100)
_struct.x = C()
It's a little bit different in that it is actually causing the module to
attempt to throw an exception instead of do a type check.
--
___
Python tracker
<https://bugs.python.org/issue38076>
___
___
Python-bugs-list mailing list
Unsubscribe:
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue38076] Make struct module PEP-384 compatible
Dino Viehland added the comment: It seems problematic that_PyInterpreterState_ClearModules runs before all instances from a module have been cleared. If PyState_FindModule is no longer able to return module state then there's no way for a module to reliably work at shutdown other than having all instances hold onto the module (or module state) directly from all of their insatances. Certainly that would mimic more closely what happens w/ pure Python instances and modules - the type will hold onto the functions which will hold onto the module global state. -- nosy: +dino.viehland ___ Python tracker <https://bugs.python.org/issue38076> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue38076] Make struct module PEP-384 compatible
Change by Dino Viehland : -- nosy: +eelizondo -dino.viehland ___ Python tracker <https://bugs.python.org/issue38076> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue39336] Immutable module type can't be used as package in custom loader
Dino Viehland added the comment: I think the warning shouldn't be too bad. It looks like ImportWarnings are filtered by default already, and the extra overhead of raising a warning in this case probably is nothing compared to the actual work in loading the module. -- ___ Python tracker <https://bugs.python.org/issue39336> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue39336] Immutable module type can't be used as package in custom loader
Change by Dino Viehland : -- keywords: +patch pull_requests: +17406 stage: needs patch -> patch review pull_request: https://github.com/python/cpython/pull/18006 ___ Python tracker <https://bugs.python.org/issue39336> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue39336] Immutable module type can't be used as package in custom loader
Change by Dino Viehland : -- nosy: +brett.cannon ___ Python tracker <https://bugs.python.org/issue39336> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue39336] Immutable module type can't be used as package in custom loader
New submission from Dino Viehland :
I'm trying to create a custom module type for a custom loader where the
returned modules are immutable. But I'm running into an issue where the
immutable module type can't be used as a module for a package. That's because
the import machinery calls setattr to set the module as an attribute on it's
parent in _boostrap.py
# Set the module as an attribute on its parent.
parent_module = sys.modules[parent]
setattr(parent_module, name.rpartition('.')[2], module)
I'd be okay if for these immutable module types they simply didn't have their
children packages published on them.
A simple simulation of this is a package which replaces its self with an object
which doesn't support adding arbitrary attributes:
x/__init__.py:
import sys
class MyMod(object):
__slots__ = ['__builtins__', '__cached__', '__doc__', '__file__',
'__loader__', '__name__', '__package__', '__path__', '__spec__']
def __init__(self):
for attr in self.__slots__:
setattr(self, attr, globals()[attr])
sys.modules['x'] = MyMod()
x/y.py:
# Empty file
>>> from x import y
Traceback (most recent call last):
File "", line 1, in
File "", line 983, in _find_and_load
File "", line 971, in _find_and_load_unlocked
AttributeError: 'MyMod' object has no attribute 'y'
There's a few different options I could see on how this could be supported:
1) Simply handle the attribute error and allow things to continue
2) Add ability for the modules loader to perform the set, and fallback to
setattr if one isn't available. Such as:
getattr(parent_module, 'add_child_module', setattr)(parent_module,
name.rpartition('.')[2], module)
3) Add the ability for the module type to handle the setattr:
getattr(type(parent_module), 'add_child_module',
fallback)(parent_module,
, name.rpartition('.')[2], module)
--
assignee: dino.viehland
components: Interpreter Core
messages: 36
nosy: dino.viehland
priority: normal
severity: normal
stage: needs patch
status: open
title: Immutable module type can't be used as package in custom loader
type: behavior
___
Python tracker
<https://bugs.python.org/issue39336>
___
___
Python-bugs-list mailing list
Unsubscribe:
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue38500] Provide a way to get/set PyInterpreterState.frame_eval without needing to access interpreter internals
Dino Viehland added the comment: Adding the getter/setters seems perfectly reasonable to me, and I agree they should be underscore prefixed as well. -- ___ Python tracker <https://bugs.python.org/issue38500> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue36971] Add subsections in C API "Common Object Structures" page
Dino Viehland added the comment: @BTaskaya Seems done, I'll go ahead and close it -- stage: patch review -> resolved status: open -> closed ___ Python tracker <https://bugs.python.org/issue36971> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue38140] Py_tp_dictoffset / Py_tp_finalize are unsettable in stable API
Change by Dino Viehland : -- resolution: -> fixed stage: patch review -> resolved status: open -> closed ___ Python tracker <https://bugs.python.org/issue38140> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue38140] Py_tp_dictoffset / Py_tp_finalize are unsettable in stable API
Dino Viehland added the comment: New changeset 3368f3c6ae4140a0883e19350e672fd09c9db616 by Dino Viehland (Eddie Elizondo) in branch 'master': bpo-38140: Make dict and weakref offsets opaque for C heap types (#16076) https://github.com/python/cpython/commit/3368f3c6ae4140a0883e19350e672fd09c9db616 -- ___ Python tracker <https://bugs.python.org/issue38140> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue38176] test_threading leaked [1, 1, 1] references: test_threads_join
Change by Dino Viehland : -- stage: patch review -> resolved ___ Python tracker <https://bugs.python.org/issue38176> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue38176] test_threading leaked [1, 1, 1] references: test_threads_join
Change by Dino Viehland : -- resolution: -> fixed status: open -> closed ___ Python tracker <https://bugs.python.org/issue38176> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue38176] test_threading leaked [1, 1, 1] references: test_threads_join
Change by Dino Viehland : -- keywords: +patch pull_requests: +15768 stage: -> patch review pull_request: https://github.com/python/cpython/pull/16158 ___ Python tracker <https://bugs.python.org/issue38176> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue38176] test_threading leaked [1, 1, 1] references: test_threads_join
Change by Dino Viehland : -- assignee: -> dino.viehland nosy: +dino.viehland ___ Python tracker <https://bugs.python.org/issue38176> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue38116] Make select module PEP-384 compatible
Dino Viehland added the comment: New changeset f919054e539a5c1afde1b31c9fd7a8f5b2313311 by Dino Viehland in branch 'master': bpo-38116: Convert select module to PEP-384 (#15971) https://github.com/python/cpython/commit/f919054e539a5c1afde1b31c9fd7a8f5b2313311 -- ___ Python tracker <https://bugs.python.org/issue38116> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue38140] Py_tp_dictoffset / Py_tp_finalize are unsettable in stable API
Change by Dino Viehland : -- nosy: +petr.viktorin ___ Python tracker <https://bugs.python.org/issue38140> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue38152] AST change introduced tons of reference leaks
Change by Dino Viehland : -- assignee: -> dino.viehland nosy: +dino.viehland ___ Python tracker <https://bugs.python.org/issue38152> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue38138] test_importleak is leaking references
Change by Dino Viehland : -- keywords: +patch pull_requests: +15676 stage: -> patch review pull_request: https://github.com/python/cpython/pull/16053 ___ Python tracker <https://bugs.python.org/issue38138> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue38138] test_importleak is leaking references
Change by Dino Viehland : -- assignee: -> dino.viehland ___ Python tracker <https://bugs.python.org/issue38138> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue38140] Py_tp_dictoffset / Py_tp_finalize are unsettable in stable API
New submission from Dino Viehland : This makes it impossible to port certain types to the stable ABI and remove statics from the interpreter. -- assignee: dino.viehland components: Interpreter Core messages: 352167 nosy: dino.viehland, eric.snow priority: normal severity: normal status: open title: Py_tp_dictoffset / Py_tp_finalize are unsettable in stable API versions: Python 3.9 ___ Python tracker <https://bugs.python.org/issue38140> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue38113] Remove statics from ast.c
Change by Dino Viehland : -- pull_requests: +15608 pull_request: https://github.com/python/cpython/pull/15975 ___ Python tracker <https://bugs.python.org/issue38113> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue38113] Remove statics from ast.c
Dino Viehland added the comment: Remove statics to make more compatible with subinterpreters. -- components: +Interpreter Core -Extension Modules title: Make ast module PEP-384 compatible -> Remove statics from ast.c ___ Python tracker <https://bugs.python.org/issue38113> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue38116] Make select module PEP-384 compatible
Change by Dino Viehland : -- keywords: +patch pull_requests: +15604 stage: -> patch review pull_request: https://github.com/python/cpython/pull/15971 ___ Python tracker <https://bugs.python.org/issue38116> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue38116] Make select module PEP-384 compatible
New submission from Dino Viehland : Remove static PyTypeObject* definitions and move static data to _selectstate to help improve compatibility w/ subinterpreters and re-use by alternate implementations. -- assignee: dino.viehland components: Extension Modules messages: 351913 nosy: dino.viehland, eric.snow priority: normal severity: normal status: open title: Make select module PEP-384 compatible versions: Python 3.9 ___ Python tracker <https://bugs.python.org/issue38116> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue38113] Make ast module PEP-384 compatible
Change by Dino Viehland : -- nosy: +eric.snow ___ Python tracker <https://bugs.python.org/issue38113> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue38113] Make ast module PEP-384 compatible
Change by Dino Viehland : -- keywords: +patch pull_requests: +15592 stage: -> patch review pull_request: https://github.com/python/cpython/pull/15957 ___ Python tracker <https://bugs.python.org/issue38113> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue38113] Make ast module PEP-384 compatible
New submission from Dino Viehland : Remove various static state from the ast module and make it PEP-384 compatible. This will help make it be more compatible w/ subinterpreters and make it re-usable by 3rd party implementations. -- assignee: dino.viehland components: Extension Modules messages: 351888 nosy: dino.viehland priority: normal severity: normal status: open title: Make ast module PEP-384 compatible versions: Python 3.9 ___ Python tracker <https://bugs.python.org/issue38113> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue36781] Optimize sum() for bools
Dino Viehland added the comment: Sorry, it seemed like a perfectly reasonable change when I was looking at the PR! -- ___ Python tracker <https://bugs.python.org/issue36781> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue36781] Optimize sum() for bools
Dino Viehland added the comment: New changeset 88bdb9280b251d753f1b1c8d9183de0fff003622 by Dino Viehland (Serhiy Storchaka) in branch 'master': bpo-36781: Optimize sum() for bools. (#13074) https://github.com/python/cpython/commit/88bdb9280b251d753f1b1c8d9183de0fff003622 -- nosy: +dino.viehland ___ Python tracker <https://bugs.python.org/issue36781> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue38075] Make random module PEP-384 compatible
Dino Viehland added the comment: There's several different reasons why these changes are beneficial. It'll help make the modules more compatible with PEP 554 in the future, enable more code re-use across Python VMs, reduce code maintenance going forward, and may also help with performance oriented changes in the future. On the PEP 554 front it is necessary to remove the static state that is used in the modules as we can't have sharing across the interpreters. The modules that are loaded into the different interpreter will need to not share any state on so all of their state needs to be per-instance state (there's also just an argument that static global state is just kind of ugly). On the re-use across VMs front it'll mean that alternative VMs will only need to stick to the stable API and well be able to use these modules as-is without having to re-implement them. It may not even be strictly necessary that these modules stick to 100% of the stable API if the places where they go outside dramatically help the implementation (for example PR 15805 uses _PyBytesWriter still). Sticking to the stable API will also naturally make it somewhat easier to make changes that impact the non-stable API. By it's very nature the stable API is going to change less (theoretically never) and therefore these modules will be less impacted by updates which are attempting to improve performance on the core. One example of that is the recent vector call support where many of the types needed to be updated (although trivially). So the final, and probably most speculative area, is the possibility of unlocking performance in the core runtime per Victor's stable API work: https://pythoncapi.readthedocs.io/. By having less dependencies on the core implementation details it should be easier to evolve those going forward an making it easier to unlock future performance gains in the core. -- ___ Python tracker <https://bugs.python.org/issue38075> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue36971] Add subsections in C API "Common Object Structures" page
Dino Viehland added the comment: New changeset 9669931e5e76cf4b6ae6d3d66e699b5fd6ffe931 by Dino Viehland (Jeroen Demeyer) in branch 'master': bpo-36971: add subsections in C API "Common Object Structures" page (#13446) https://github.com/python/cpython/commit/9669931e5e76cf4b6ae6d3d66e699b5fd6ffe931 -- nosy: +dino.viehland ___ Python tracker <https://bugs.python.org/issue36971> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue37725] "make clean" should remove PGO task data
Dino Viehland added the comment: New changeset c6bbcd258302b4b9b3d4f3c39bb5f7ff0120ec67 by Dino Viehland (Neil Schemenauer) in branch 'master': bpo-37725: have "make clean" remove PGO task data (#15033) https://github.com/python/cpython/commit/c6bbcd258302b4b9b3d4f3c39bb5f7ff0120ec67 -- nosy: +dino.viehland ___ Python tracker <https://bugs.python.org/issue37725> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue38076] Make struct module PEP-384 compatible
Change by Dino Viehland : -- keywords: +patch pull_requests: +15456 stage: -> patch review pull_request: https://github.com/python/cpython/pull/15805 ___ Python tracker <https://bugs.python.org/issue38076> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue38076] Make struct module PEP-384 compatible
New submission from Dino Viehland : Make struct module PEP-384 compatible -- assignee: dino.viehland components: Extension Modules messages: 351524 nosy: dino.viehland, eric.snow priority: normal severity: normal status: open title: Make struct module PEP-384 compatible versions: Python 3.9 ___ Python tracker <https://bugs.python.org/issue38076> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue38075] Make random module PEP-384 compatible
Change by Dino Viehland : -- keywords: +patch pull_requests: +15450 stage: -> patch review pull_request: https://github.com/python/cpython/pull/15798 ___ Python tracker <https://bugs.python.org/issue38075> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue38075] Make random module PEP-384 compatible
New submission from Dino Viehland : Make random module PEP-384 compatible -- assignee: dino.viehland components: Extension Modules messages: 351509 nosy: dino.viehland, eric.snow priority: normal severity: normal status: open title: Make random module PEP-384 compatible versions: Python 3.9 ___ Python tracker <https://bugs.python.org/issue38075> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue38074] Make zlib PEP-384 compatible
Change by Dino Viehland : -- keywords: +patch pull_requests: +15445 stage: -> patch review pull_request: https://github.com/python/cpython/pull/15792 ___ Python tracker <https://bugs.python.org/issue38074> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue38074] Make zlib PEP-384 compatible
New submission from Dino Viehland : Make zlib PEP-384 compatible -- components: Extension Modules messages: 351501 nosy: dino.viehland, eric.snow priority: normal severity: normal status: open title: Make zlib PEP-384 compatible versions: Python 3.9 ___ Python tracker <https://bugs.python.org/issue38074> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue38073] Make pwd module PEP-384 compatible
New submission from Dino Viehland : Make the pwd module PEP-384 compatible -- assignee: dino.viehland components: Extension Modules messages: 351495 nosy: dino.viehland, eric.snow priority: normal severity: normal status: open title: Make pwd module PEP-384 compatible versions: Python 3.9 ___ Python tracker <https://bugs.python.org/issue38073> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue38073] Make pwd module PEP-384 compatible
Change by Dino Viehland : -- keywords: +patch pull_requests: +15443 stage: -> patch review pull_request: https://github.com/python/cpython/pull/15790 ___ Python tracker <https://bugs.python.org/issue38073> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue38072] Make grp module PEP-384 compatible
Change by Dino Viehland : -- keywords: +patch pull_requests: +15441 stage: -> patch review pull_request: https://github.com/python/cpython/pull/15788 ___ Python tracker <https://bugs.python.org/issue38072> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue38072] Make grp module PEP-384 compatible
Change by Dino Viehland : -- assignee: dino.viehland components: Extension Modules nosy: dino.viehland, eric.snow priority: normal severity: normal status: open title: Make grp module PEP-384 compatible versions: Python 3.9 ___ Python tracker <https://bugs.python.org/issue38072> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue38071] Make termios PEP-384 compatible
Change by Dino Viehland : -- keywords: +patch pull_requests: +15438 stage: -> patch review pull_request: https://github.com/python/cpython/pull/15785 ___ Python tracker <https://bugs.python.org/issue38071> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue38071] Make termios PEP-384 compatible
Change by Dino Viehland : -- components: Extension Modules nosy: dino.viehland, eric.snow priority: normal severity: normal status: open title: Make termios PEP-384 compatible type: enhancement versions: Python 3.9 ___ Python tracker <https://bugs.python.org/issue38071> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue38069] Make _posixsubprocess PEP-384 compatible
Change by Dino Viehland : -- keywords: +patch pull_requests: +15433 stage: -> patch review pull_request: https://github.com/python/cpython/pull/15780 ___ Python tracker <https://bugs.python.org/issue38069> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue38069] Make _posixsubprocess PEP-384 compatible
Change by Dino Viehland : -- assignee: dino.viehland components: Extension Modules nosy: dino.viehland, eric.snow priority: normal severity: normal status: open title: Make _posixsubprocess PEP-384 compatible versions: Python 3.9 ___ Python tracker <https://bugs.python.org/issue38069> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue26185] zipfile.ZipInfo slots can raise unexpected AttributeError
Dino Viehland added the comment: New changeset 992347d7376765fe3f4fc958fb1be193ba21f6c3 by Dino Viehland (Mickaël Schoentgen) in branch 'master': bpo-26185: Fix repr() on empty ZipInfo object (#13441) https://github.com/python/cpython/commit/992347d7376765fe3f4fc958fb1be193ba21f6c3 -- nosy: +dino.viehland ___ Python tracker <https://bugs.python.org/issue26185> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue36839] Support the buffer protocol in code objects
Dino Viehland added the comment:
The 20MB of savings is actually the amount of byte code that exists in the IG
code base. I was just measuring the web site code, and not the other various
Python code in the process (e.g. no std lib code, no 3rd party libraries,
etc...). The IG code base is pretty monolithic and starting up the site
requires about half of the code to get imported. So I think the 20MB per
process is a pretty realistic number.
I've also created a C extension and the object implementing the buffer protocol
looks like:
typedef struct {
PyObject_HEAD
const char* data;
size_t size;
Py_ssize_t hash;
CIceBreaker *breaker;
size_t exports;
PyObject* code_obj; /* borrowed reference, the code object keeps us alive */
} CIceBreakerCode;
All of the modules are currently getting compiled into a single memory mapped
file and then these objects get created which implement the buffer protocol for
each function. So the overhead it just takes a byte code w/ 16 opcodes before
it breaks even, so it is significantly lighter weight than using a memoryview
object.
It's certainly true that the byte code isn't the #1 source of memory here (the
code objects themselves are pretty big), but in the serialized state it ends up
representing 25% of the serialized data. I would expect when you add in ref
counts and typing information it's not quite as good, but reducing the overhead
of code by 20% is still a pretty nice win.
I can't make any promises about open sourcing the import system, but I can
certainly look into that as well.
--
___
Python tracker
<https://bugs.python.org/issue36839>
___
___
Python-bugs-list mailing list
Unsubscribe:
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue36839] Support the buffer protocol in code objects
Dino Viehland added the comment: Well given that we're one day away from 3.8 Beta 1 I'm not going to rush this in :) In particular the discussion has made me wonder if I can also do more with sharing strings using the legacy unicode strings (which I don't believe will require any runtime changes). So I'll keep pushing on this and hopefully be able to demonstrate an even large win from the concept. -- ___ Python tracker <https://bugs.python.org/issue36839> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue36839] Support the buffer protocol in code objects
Dino Viehland added the comment: In the Instagram case there's about 20mb of byte code total and there are 3-4 dozen worker processes running per server. The byte code also represents the second largest section of memory as far as serialized code objects are concerned, the only larger one is strings (which are about 25mb). Anyway, that leads to around 1gb of memory savings per server, which is the reason why this was an interesting optimization to pursue when it's applied to many servers. -- ___ Python tracker <https://bugs.python.org/issue36839> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue36839] Support the buffer protocol in code objects
Dino Viehland added the comment: Sure, but immutable/const is almost always a language level guarantee. The only case where that's not true is when you have OS/hardware level memory protection and that doesn't apply to any of Python's existing byte codes. So from a Python perspective, code objects are remaining immutable - they can only be created by objects which expose the read-only buffer protocol. So for example passing in a memoryview(b'abc') will work here while a memoryview(bytearray(b'abc')) will fail. And because when asking for a non read-write view the buffer implementer needs to be consistent for all callers (https://docs.python.org/3/c-api/buffer.html#c.PyBUF_WRITABLE) it seems that this invariant should hold for all objects being passed in. Could someone create a buffer object which still allows the underlying memory to be written? Sure. But I can use ctypes to modify byte code today as well with something like "ctypes.cast(id(f.__code__.co_code) + 32, ctypes.POINTER(ctypes.c_char)) [0] = 101" So people will still be able to do nasty things, but there are certainly guards in place to strongly discourage them from doing so. -- ___ Python tracker <https://bugs.python.org/issue36839> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue37001] symtable.symtable doesn't accept bytes which leads to a mismatch from compile()
Change by Dino Viehland : -- stage: patch review -> resolved status: open -> closed ___ Python tracker <https://bugs.python.org/issue37001> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue37001] symtable.symtable doesn't accept bytes which leads to a mismatch from compile()
Dino Viehland added the comment: New changeset 415406999d7c09af9f3dcacfb4578b9e97b2ce77 by Dino Viehland in branch 'master': bpo-37001: Makes symtable.symtable have parity with compile for input (#13483) https://github.com/python/cpython/commit/415406999d7c09af9f3dcacfb4578b9e97b2ce77 -- ___ Python tracker <https://bugs.python.org/issue37001> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue36839] Support the buffer protocol in code objects
Dino Viehland added the comment: The PR actually checks that the buffer is read-only (this was also a concern that Mark Shannon had). And the Python buffer protocol says that you need to consistently hand out read-only buffers. So while someone could create a buffer and mutate it outside of the buffer protocol it should be really read-only. As far as the ref counts, it's not just the ref counts for the code byte strings that are potentially problematic. But it's the ref counts on all of the random other objects which the code objects are on the same page as, as well as other random read-write data that could be on those pages. There's also an additional benefit in that code objects not loaded before forking can continue to share their memory as well. -- ___ Python tracker <https://bugs.python.org/issue36839> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue37001] symtable.symtable doesn't accept bytes which leads to a mismatch from compile()
Change by Dino Viehland : -- keywords: +patch pull_requests: +13395 stage: -> patch review ___ Python tracker <https://bugs.python.org/issue37001> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue37002] PyType_FromSpec can't create immutable types
New submission from Dino Viehland : This is another catch of using PyType_FromSpec (similar to https://bugs.python.org/issue26979 but also completely different). Because PyType_FromSpec produces a heap type it gets the Py_TPFLAGS_HEAPTYPE flag set on the newly produced type. To enforce the immutability of built-in types type_setattro checks this flag. Therefore these types end up being mutable, e.g: import _ssl _ssl.SSLError.x = 42 In addition to not providing parity for replacing PyType_Ready in the stable API, it also means that this could presumably bleed across sub-interpreters or just provide surprising and bad results. -- components: Interpreter Core messages: 343100 nosy: dino.viehland, eric.snow, vstinner priority: normal severity: normal status: open title: PyType_FromSpec can't create immutable types versions: Python 3.8, Python 3.9 ___ Python tracker <https://bugs.python.org/issue37002> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue37001] symtable.symtable doesn't accept bytes which leads to a mismatch from compile()
New submission from Dino Viehland : symtable is useful when combined with compile() to AST to understand what the names bind to. But symtable.symtable() doesn't accept a bytes object, while compile does. Ultimately these feed down to the same API, and could easily lead to subtle mismatches due to encodings. The workaround seems to be to use the tokenize.detect_encoding to discover the encoding and then do the encoding from Python, but this seems wasteful. -- assignee: dino.viehland components: Library (Lib) messages: 343096 nosy: dino.viehland priority: normal severity: normal status: open title: symtable.symtable doesn't accept bytes which leads to a mismatch from compile() versions: Python 3.8 ___ Python tracker <https://bugs.python.org/issue37001> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue36929] Other Python _io implementations may not expose _io in their type names
Dino Viehland added the comment: New changeset ccb7ca728e09b307f9e9fd36ec40353137e68a3b by Dino Viehland (Max Bernstein) in branch 'master': bpo-36929: Modify io/re tests to allow for missing mod name (#13392) https://github.com/python/cpython/commit/ccb7ca728e09b307f9e9fd36ec40353137e68a3b -- nosy: +dino.viehland ___ Python tracker <https://bugs.python.org/issue36929> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue36839] Support the buffer protocol in code objects
Change by Dino Viehland : -- nosy: +eric.snow ___ Python tracker <https://bugs.python.org/issue36839> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue36839] Support the buffer protocol in code objects
Change by Dino Viehland : -- keywords: +patch pull_requests: +13093 stage: needs patch -> patch review ___ Python tracker <https://bugs.python.org/issue36839> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
