[issue47144] Allow setting __classcell__
Douglas Raillard added the comment: EDIT: empty_list() is a class method of List, not ContainerBase -- ___ Python tracker <https://bugs.python.org/issue47144> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue47144] Allow setting __classcell__
Douglas Raillard added the comment: > Would bpo-47143 "Add functools.copy_class() which updates closures" solve > your use case? This looks like a similar issue indeed. If I'm able to copy a class "cleanly" (as defined in this other thread), that may solve the problem (hard to tell precisely until trying it though, the devil is in the details). > Would you elaborate your use case? As you probably expect, it's a relatively tricky metaprogramming bit of code. It's implementing something akin to higher kinded types, similar in spirit to the following example where you can have e.g. containers that "know" the item type they contain: class ContainerBase: ITEM = None @classmethod def apply(cls, typ): ... @classmethod def empty_list(cls): ... class List(ContainerBase): pass class ListOfInt(List.apply(int)): pass The goal is to allow "ListOfInt" to define custom methods if we want in its body. "ListOfInt" cannot return instances of itself [1] so I'm trying to "cherry pick" its method and graft them onto the type that is actually instantiated. [1] We e.g. want to create an empty list of the right type. "List" already provides the method for such factory, but it will not return an actual instance of ListOfInt, since it does not even know this class exists. What it will use is an internal type that it created using the "ITEM" class attribute in e.g. List.__init_subclass__, and that is what will get instantiated. This "internal class" could still be customized by ListOfInt to have some custom methods though, which is what I'm trying to achieve. -- ___ Python tracker <https://bugs.python.org/issue47144> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue47144] Allow setting __classcell__
New submission from Douglas Raillard :
The cell object __classcell__ currently cannot be set by code invoked by
metaclass.__new__. Attempts to do so will be caught by builtin___build_class__
in bltimodule.c:
} else {
const char *msg =
"__class__ set to %.200R defining %.200R as %.200R";
PyErr_Format(PyExc_TypeError, msg, cell_cls, name, cls);
}
This is unfortunate as there is a use-case for such trickery: if the method of
a class A are only used to be grafted onto another class B (monkey patching),
A.__classcell__ should be set to B so that super() works as expected.
This can be useful in contexts where e.g. A.__new__ returns instances of B. B
is not necessarily something under direct user control (it can be dynamically
created by inheriting from a "template"), so A would be a better place for
users to define custom method.
--
components: Interpreter Core
messages: 416172
nosy: douglas-raillard-arm
priority: normal
severity: normal
status: open
title: Allow setting __classcell__
versions: Python 3.11
___
Python tracker
<https://bugs.python.org/issue47144>
___
___
Python-bugs-list mailing list
Unsubscribe:
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue45830] Custom pickler memory leak
New submission from Douglas Raillard :
The following script exhibits the memory leak. It only happens if
"dispatch_table" is set _before_ calling super().__init__, which is pretty
unexpected.
import pickle
import io
import gc
import tracemalloc
tracemalloc.start(10)
snap = tracemalloc.take_snapshot()
class MyPickler(pickle.Pickler):
def __init__(self, *args, **kwargs):
# Swapping the next 2 lines "solves" the memory leak for some reason
self.dispatch_table = dict()
super().__init__(*args, **kwargs)
l=[]
for i in range(1):
if i % 1000 == 0:
print('='*80)
snap2 = tracemalloc.take_snapshot()
stats=snap2.compare_to(snap, 'lineno')
for s in stats[:10]:
print(s)
snap = snap2
f = io.BytesIO()
MyPickler(f)
gc.collect()
The output of the last iteration is as follow. The leak of 562 kiB is apparent:
testmem.py:12: size=562 KiB (+62.5 KiB), count=9000 (+1000), average=64
B
/usr/lib/python3.10/tracemalloc.py:125: size=2376 B (-72 B), count=33
(-1), average=72 B
/usr/lib/python3.10/tracemalloc.py:129: size=72 B (+72 B), count=1
(+1), average=72 B
/usr/lib/python3.10/tracemalloc.py:502: size=252 B (+28 B), count=9
(+1), average=28 B
/usr/lib/python3.10/tracemalloc.py:498: size=2104 B (+0 B), count=36
(+0), average=58 B
/home/dourai01/Work/lisa/lisa/testmem.py:10: size=1844 B (+0 B),
count=9 (+0), average=205 B
/usr/lib/python3.10/tracemalloc.py:193: size=1680 B (+0 B), count=35
(+0), average=48 B
/usr/lib/python3.10/tracemalloc.py:547: size=1256 B (+0 B), count=3
(+0), average=419 B
/usr/lib/python3.10/tracemalloc.py:226: size=832 B (+0 B), count=2
(+0), average=416 B
/usr/lib/python3.10/tracemalloc.py:173: size=800 B (+0 B), count=2
(+0), average=400 B
If "dispatch_table" is set after calling super().__init__, there is no leak
anymore:
/usr/lib/python3.10/tracemalloc.py:135: size=740 B (+740 B), count=7
(+7), average=106 B
/usr/lib/python3.10/tracemalloc.py:125: size=2088 B (-656 B), count=29
(-4), average=72 B
/usr/lib/python3.10/tracemalloc.py:136: size=320 B (+320 B), count=1
(+1), average=320 B
/usr/lib/python3.10/tracemalloc.py:132: size=0 B (-256 B), count=0 (-1)
/usr/lib/python3.10/tracemalloc.py:129: size=72 B (+72 B), count=1
(+1), average=72 B
/usr/lib/python3.10/tracemalloc.py:498: size=2008 B (+48 B), count=34
(+1), average=59 B
/usr/lib/python3.10/tracemalloc.py:193: size=1584 B (+48 B), count=33
(+1), average=48 B
/usr/lib/python3.10/tracemalloc.py:502: size=196 B (-28 B), count=7
(-1), average=28 B
/usr/lib/python3.10/tracemalloc.py:126: size=84 B (+28 B), count=3
(+1), average=28 B
--
components: Library (Lib)
messages: 406478
nosy: douglas-raillard-arm
priority: normal
severity: normal
status: open
title: Custom pickler memory leak
type: resource usage
versions: Python 3.10
___
Python tracker
<https://bugs.python.org/issue45830>
___
___
Python-bugs-list mailing list
Unsubscribe:
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue22239] asyncio: nested event loop
Douglas Raillard added the comment: Drive by comment: I landed on this thread for the exact same reason: > This situation is very frequent when e.g. a library is designed to be > async-first, and also provides a blocking API which just wraps the async code > by running it until complete. The library in question is "devlib", which abstracts over SSH/adb/local shell. We cannot make a "full" switch to async as it would be a big breaking change. To workaround that, I came up with a decorator that wraps a corountine, and "replaces" it such that: @asyncf async def f(...): ... # Blocking call under its "normal" name, for backward compat f() # Used in an async environment await f.asyn() This allows converting bit by bit the whole library, with full backward compatibility for both users and internal calls. On top of that, that library is heavily used in jupyter notebooks, so all in all, nest-asyncio is impossible to avoid. -- nosy: +douglas-raillard-arm ___ Python tracker <https://bugs.python.org/issue22239> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue44749] LOAD_NAME not using PyObject_GetItem when globals() is a dict subclass
Douglas Raillard added the comment: I ended up with a slightly different API that allows me to preemptively set some global names before the code runs in the module, so my use case sort of vanished. I don't think there is a real need of beating the dead horse here. My only suggestion on the topic is that since the feature is not documented and is not really usable anyway, we might as well remove the slow path in LOAD_GLOBAL (unless there is a use case for having LOAD_GLOBAL and not LOAD_NAME I'm missing) -- ___ Python tracker <https://bugs.python.org/issue44749> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue44749] LOAD_NAME not using PyObject_GetItem when globals() is a dict subclass
Douglas Raillard added the comment: Looks like it is, for some reason I did not find these previous issues when looking for existing ones. -- ___ Python tracker <https://bugs.python.org/issue44749> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue44749] LOAD_NAME not using PyObject_GetItem when globals() is a dict subclass
New submission from Douglas Raillard :
Re-raising the bug reported by Kevin Shweh:
thread: https://bugs.python.org/issue14385
message: https://bugs.python.org/msg337245
Here is a copy for easier reference:
The patch for this issue changed LOAD_GLOBAL to use PyObject_GetItem when
globals() is a dict subclass, but LOAD_NAME, STORE_GLOBAL, and DELETE_GLOBAL
weren't changed. (LOAD_NAME uses PyObject_GetItem for builtins now, but not for
globals.)
This means that global lookup doesn't respect overridden __getitem__ inside a
class statement (unless you explicitly declare the name global with a global
statement, in which case LOAD_GLOBAL gets used instead of LOAD_NAME).
I don't have a strong opinion on whether STORE_GLOBAL or DELETE_GLOBAL should
respect overridden __setitem__ or __delitem__, but the inconsistency between
LOAD_GLOBAL and LOAD_NAME seems like a bug that should be fixed.
For reference, in the following code, the first 3 exec calls successfully print
5, and the last exec call fails, due to the LOAD_GLOBAL/LOAD_NAME inconsistency:
class Foo(dict):
def __getitem__(self, index):
return 5 if index == 'y' else super().__getitem__(index)
exec('print(y)', Foo())
exec('global y; print(y)', Foo())
exec('''
class UsesLOAD_NAME:
global y
print(y)''', Foo())
exec('''
class UsesLOAD_NAME:
print(y)''', Foo())
I encountered the same issue when trying to create a way to "instantiate"
modules with some globals replaced by user-defined values to make a
dependency-injection system. I therefore want to lookup some names in a
separate dict rather than getting the value normally bound in that module
(typically by an import statement).
--
components: Interpreter Core
messages: 398299
nosy: douglas-raillard-arm
priority: normal
severity: normal
status: open
title: LOAD_NAME not using PyObject_GetItem when globals() is a dict subclass
versions: Python 3.10
___
Python tracker
<https://bugs.python.org/issue44749>
___
___
Python-bugs-list mailing list
Unsubscribe:
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue43460] Exception copy error
Douglas Raillard added the comment: The solution based on the signature is something along those lines: class E(BaseException): def __new__(cls, *args, **kwargs): """ Fix exception copying. Turn all the keyword arguments into positional arguments, so that the :exc:`BaseException` machinery has all the parameters for a valid call to ``__new__``, instead of missing all the keyword arguments. """ sig = inspect.signature(cls.__init__) bound_args = sig.bind_partial(*args, **kwargs) bound_args.apply_defaults() args = tuple(bound_args.arguments.values()) return super().__new__(cls, *args) def __init__(self, x): self.x=x But there are a many shortcomings to that approach: * What if super().__new__() consumes arguments before passing the rest to __init__() ? This fix is blind to that since it only cares about __init__ signature * What if inspect.signature() does not provide a signature (extension modules) ? * Defaults are "hardcoded" in the args, so the object will always be restored with the defaults of the time it was created. This is a breaking change, as currently the defaults used when restoring the instance are the current ones. * Also uses more memory for args (and for pickle files), since it contains all the defaults -- ___ Python tracker <https://bugs.python.org/issue43460> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue43460] Exception copy error
New submission from Douglas Raillard : Instances of subclasses of BaseException created with keyword argument fail to copy properly as demonstrated by: import copy class E(BaseException): def __init__(self, x): self.x=x # works fine e = E(None) copy.copy(e) # raises e = E(x=None) copy.copy(e) This seems to affect all Python versions I've tested (3.6 <= Python <= 3.9). I've currently partially worked around the issue with a custom pickler that just restores __dict__, but: * "args" is not part of __dict__, and setting "args" key in __dict__ does not create a "working object" (i.e. the key is set, but is ignored for all intents and purposes except direct lookup in __dict__) * pickle is friendly: you can provide a custom pickler that chooses the reduce function for each single class. copy module is much less friendly: copyreg.pickle() only allow registering custom functions for specific classes. That means there is no way (that I know) to make copy.copy() select a custom reduce for a whole subclass tree. One the root of the issue: * exception from the standard library prevent keyword arguments (maybe because of that issue ?), but there is no such restriction on user-defined classes. * the culprit is BaseException_reduce() (in Objects/exceptions.c) [1] It seems that the current behavior is a consequence of the __dict__ being created lazily, I assume for speed and memory efficiency There seems to be a few approaches that would solve the issue: * keyword arguments passed to the constructor could be fused with the positional arguments in BaseException_new (using the signature, but signature might be not be available for extension types I suppose) * keyword arguments could just be stored like "args" in a "kwargs" attribute in PyException_HEAD, so they are preserved and passed again to __new__ when the instance is restored upon copying/pickling. * the fact that keyword arguments were used could be saved as a bool in PyException_HEAD. When set, this flag would make BaseException_reduce() only use __dict__ and not "args". This would technically probably be a breaking change, but the only cases I can think of where this would be observable are a bit far fetched (if __new__ or __init__ have side effects beyond storing attributes in __dict__). [1] https://github.com/python/cpython/blob/master/Objects/exceptions.c#L134 -- messages: 388427 nosy: douglas-raillard-arm priority: normal severity: normal status: open title: Exception copy error type: behavior ___ Python tracker <https://bugs.python.org/issue43460> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue43102] namedtuple's __new__.__globals__['__builtins__'] is None
Douglas Raillard added the comment: Thanks for looking into this issue -- ___ Python tracker <https://bugs.python.org/issue43102> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue43102] namedtuple's __new__.__globals__['__builtins__'] is None
Douglas Raillard added the comment: I did hit the issue while porting a tool to Python 3.9: https://github.com/ARM-software/lisa/pull/1585/commits/a4cd3aa1ad339ebfe59cc9e2ae290bb3788c900d It basically infers valid Python expressions from type annotations (some sort of inverse type checker) and run them while serializing all intermediate subexpressions for debugging. This allows a whole test suite to be plain classes and functions (usable out of context) to be assembled into "pipelines" without extra user work. The reason I ran into that problem are: 1. I'm scanning whole modules for annotations, so it is exposed to lots of things 2. In order to be able to infer expressions, it has to augment existing annotations when it is unambiguous. In our case, since __new__ is more or less a classmethod (i.e. it takes the class as first argument even if strictly speaking it's a staticmethod), it added an annotation with the class name (extracted from __qualname__). It implements PEP563 to evaluate the annotions, which describes how to get the globals for a function and class and that they can be fed to eval(). Using typing.get_type_hints() is not really possible since annotations need to be augmented, and fancy type hints like Optional are unusable for my purpose since there typing.get_args() was only added to Python 3.8 (I need to support >= 3.6). Maybe an alternative would be to use a module-level types.MappingProxyType instance ? This way there is no extra object created for each namedtuple and since it's read-only it should be as safe as None. -- versions: +Python 3.9 -Python 3.10 ___ Python tracker <https://bugs.python.org/issue43102> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue43102] namedtuple's __new__.__globals__['__builtins__'] is None
New submission from Douglas Raillard :
When creating a namedtuple such as this one:
from collections import namedtuple
class C(namedtuple('C', ('hello', 'world'))):
pass
print(C.__new__.__globals__)
The globals' dict of __new__ contains a "__builtins__" key which is set to None
in collections/__init__.py:
namespace = {
'_tuple_new': tuple_new,
'__builtins__': None,
'__name__': f'namedtuple_{typename}',
}
When such globals are used with eval(), it will raise a TypeError such as:
>>> eval('X', {'__builtins__': None})
Traceback (most recent call last):
File "", line 1, in
File "", line 1, in
TypeError: 'NoneType' object is not subscriptable
If an empty dict was used instead, we get the expected exception:
>>> eval('X', {'__builtins__': {}})
Traceback (most recent call last):
File "", line 1, in
File "", line 1, in
NameError: name 'X' is not defined
Given that both ways allow preventing references to any builtin, please
consider switching to an empty dict. Also, even though this is documented as
implementation detail, this would agree more with the current documentation
stating:
The value of __builtins__ is normally either this module or the value of
this module’s __dict__ attribute
https://docs.python.org/3/library/builtins.html
--
components: Library (Lib)
messages: 386145
nosy: douglas-raillard-arm
priority: normal
severity: normal
status: open
title: namedtuple's __new__.__globals__['__builtins__'] is None
type: behavior
versions: Python 3.9
___
Python tracker
<https://bugs.python.org/issue43102>
___
___
Python-bugs-list mailing list
Unsubscribe:
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
