[issue45184] Add `pop` function to remove context manager from (Async)ExitStack
Andreas H. added the comment: Inside the discussion an ExitPool class is sketched (https://mail.python.org/archives/list/python-id...@python.org/message/66W55FRCYMYF73TVMDMWDLVIZK4ZDHPD/), which provides this removal of context managers. What I learned is that this would have different cleanup mode (atexit style), as compared to present ExitStack cleanup (nested style). So contrary to what I was originally thinking ExitPool functionality would be close to, but not a strict superset of ExitStack functionality. Still such an ExitPool functionality would be extremely useful. -- ___ Python tracker <https://bugs.python.org/issue45184> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue46333] ForwardRef.__eq__ does not respect module parameter
Change by Andreas H. : -- pull_requests: +29443 pull_request: https://github.com/python/cpython/pull/31283 ___ Python tracker <https://bugs.python.org/issue46333> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue46369] get_type_hints does not evaluate ForwardRefs inside NewType
Andreas H. added the comment: Allright. B) sounds good to me. I dont think I have time today, so please feel to tackle the issue. If not I can look at it the next week. -- ___ Python tracker <https://bugs.python.org/issue46369> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue46373] TypedDict and NamedTuple do not evaluate cross-module ForwardRef in all cases
New submission from Andreas H. :
TypedDict does not resolve cross-module ForwardRefs when the ForwardRef is not
a direct one.
In other words the fix GH-27017 (issue 41249) for TypedDict seems incomplete.
The same issue seem to exist for NamedTuple.
Example:
#module.py
TD = typing.TypedDict("TD", {'test': typing.List[typing.Optional['Y']]})
class Y:
pass
# other module
class TDSub(module.TD):
a: int
get_type_hints(TDSub)
# -> Exception NameError: Y not found
On the other hand, with direct ForwardRef, as e.g. in
TD = typing.TypedDict("TD", {'test': 'Y' } )
it works (that was indeed fixed by GH-27017)
Same issue exists for NamedTuple. There neither of the above works, i.e.
cross-module ForwardRefs are never resolve (but they could - als NamedTuple has
the __module__ member set with to calling module). I am not sure if inheritance
for NamedTuple is supported so I do not know if it is really a bug.
The problem in the code is that in TypedDict the `module` parameter is passed
only onto the immediate ForwardRef. One option could be to recursively walk the
type and search for unpatched ForwardRefs and set the module parameter.
On the other hand, the retroactive patching of forward refs is problematic as
it may mess with the caching mechanism im typing.py. There may be a type with
ForwardRef already in cache (and used by more than one user), but only for one
user the `module` is to be updated. So probably a correct implementation is
tricky, or some other way has to be found to update the `module` of ForwardRefs
(e.g. by copying the type tree).
For NamedTuple the whole mechanism of passing the `module` parameter to the
ForwardRefs is not done (not even for direct ForwardRef ones).
Not sure how important this (likely not very) is as I do not use TypedDict and
NamedTuple. This is just to report it.
--
components: Library (Lib)
messages: 410547
nosy: AlexWaygood, Jelle Zijlstra, andreash, gvanrossum, kj, kumaraditya303,
sobolevn
priority: normal
severity: normal
status: open
title: TypedDict and NamedTuple do not evaluate cross-module ForwardRef in all
cases
type: behavior
versions: Python 3.10, Python 3.11
___
Python tracker
<https://bugs.python.org/issue46373>
___
___
Python-bugs-list mailing list
Unsubscribe:
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue46371] A better way to resolve ForwardRefs in type aliases across modules
New submission from Andreas H. :
(De)Serialization of in-memory data structures is an important application.
However there is a rather unpleasant issue with ForwardRefs.
One cannot export type aliases when they contain ForwardRefs (and expect
things to work).
Consider the example:
Json = Union[ List['Json'], Dict[str, 'Json'], int, float, bool, None ]
When used in another module the containing ForwardRefs cannot be resolved, or
in the worst case, resolve to
wrong types if in the caller namespace contains a symbol with the same name as
ForwardRef refers to.
This of course only applies to inspection-based tools which utilize the
run-time information, not the static type checkers.
Type aliases sometimes have to be used sometimes - they cannot be avoided in
all cases,
especially with recursive types as in the example above.
There are several options to improve the situation. These are all that came to
my mind and I want to expose them to discussion.
1. Guard with NewType
Json = NewType('Json', Union[ List['Json'], Dict[str, 'Json'], int, float,
bool, None ] )
Advantage: This could allow automatic cross-module ForwardRef resolution
provided issue 46369 [1] is acknowledged as a bug and fixed.
Disadvantage: Does not create a true type alias, but a sub-type. Type casts
have to be used all the time, e.g. `data = Json(bool)`.
So can only applied to a subset of use-cases (but is IMO a clean solution
when it fits).
2. Use the `module` parameter of ForwardRef
Json = Union[ List[ForwardRef('Json', module=__name__)], Dict[str,
ForwardRef('Json', module=__name__)], int, float, bool, None ] )
Advantage: Works in 3.10.
Disadvantage: Would require issue 46333 [2] to be fixed. ForwardRef is not
meant to be instatiated by the user,
also `module` parameter is currently completely internal.
3. Modify ForwardRef so that it accepts a fully qualified name
Json = Union[ List[__name__+'.Json'], Dict[str, __name__+'.Json'], int,
float, bool, None ] )
Advantage: This is only a tiny change (because ForwardRef has the `module`
parameter). ForwardRef would stay internal. Less ugly than 2.
Disadvantage: Still a bit ugly. Would also require issue 46333 [2] to be
fixed. Relative module specification (as in relative imports)
would not work.
4. Manual evaluation
Json = Union[ List['Json'], Dict[str, 'Json'], int, float, bool, None ]
resolve_type_alias(Json, globals(), locals() )
def resolve_type_alias(type, globalns, localns):
class dummy:
test: type
typing.get_type_hints(dummy, globalns, localns) # Note: this modifies
ForwardRefs in-place
Advantage: Works in many versions.
Disadvantage: Requires user to explicily call function after the last
referenced type is defined
(this may be physically separated from the type alias definition, which
does not feel like a good solution especially since
this ForwardRef export problem is only technical, and not even close to
beeing obvious to most people)
5. Make `get_type_hints()` to work with type-aliases (essentially integrate
above `resolve_type_alias`). The return value
of get_type_hints() would be the same as argument, just with ForwardRefs
in-place resolved.
Json = Union[ List['Json'], Dict[str, 'Json'], int, float, bool, None ]
get_type_hints(Json, globals(), locals())
Advantage: same as 4) but hides useful (but ugly) code
Disadvantage: same as 4)
6. Make all types in typing (such as List, Dict, Union, etc...) to capture
their calling module and pass this information to ForwardRef, when
one is to be created. Then already during construction of ForwardRef the
`module` will be correctly set.
Json = Union[ List['Json'], Dict[str, 'Json'], int, float, bool, None ]
Advantage: This requires no user intervention. Things will "just work"
Disadvantage: This is rather big change. It is incompatible with the caching
used inside typing.py (the new __module__ parameter would
need to be taken into account in __hash__ and/or __eq__). And maybe has
other issues I do not see at the moment.
7. Extend `TypeAlias` hint so it can be used in bracket way similar to e.g.
`Annotated`
Json = TypeAlias[ Union[ List['Json'], Dict[str, 'Json'], int, float,
bool, None ] ]
I know, presently it is supposed to be used as `Json: TypeAlias = Union[
]`. But that is of no help at run-time, because
the variable Json contains no run-time information. So even if TypeAlias
would capture the calling module,
this information is not passed on to the variable `Json`. This is different
for the bracket notation TypeAlias[ .. ].
Advantage: Similar usage to Annotated. Would not require a big change such
as in 4).
Disadvantage: TypeAlias is not supposed to be used that way.
8. Introduce function to define a type alias, which sets
[issue46369] get_type_hints does not evaluate ForwardRefs inside NewType
New submission from Andreas H. :
Consider the following:
NewT = typing.NewType("NewT", typing.List[typing.Optional['Z']] )
class Z:
pass
Now get_type_hints() does not resolve the ForwardRef within NewType (but it
does so for TypedDict, dataclasses, NamedTuple).
Neither of the following works.
1)
class dummy:
test: NewT
get_type_hints(test,None,None)
print( NewT.__supertype__.__args__[0].__args__[0]__.__forward_evaluated__ )
# --> False
Note: investigating the return value of get_type_hints does not change the
outcome.
get_type_hints() patches ForwardRefs in-place.
2)
get_type_hints(NewT,None,None)
# --> TypeError is not a module, class, method, or function
For Python 3.10+ a workaround exists, but requires access to implementation
details of NewType:
class dummy:
test: NewT.__supertype__
get_type_hints( dummy, globalns=sys.modules[NewT.__module__].__dict__,
localns=None )
Possible solution could be
A) to extent `get_type_hints` to explicitly handle NewType (basically call
_eval_type for the __supertype__ member).
That makes approach 2) work (but not 1)
or B) to extend _eval_type() to handle process NewType as well. This would
make 1) work (but not 2).
I guess, since NewType is supposed to be semantically a subclass of the
referred type, 2) is probably the preferred approach, which would suggest A).
Strictly speaking this issue exits in all Python versions that have NewType,
but it is easier to fix in 3.10 because there NewType has the __module__ member.
--
components: Library (Lib)
messages: 410528
nosy: andreash, gvanrossum, kj
priority: normal
severity: normal
status: open
title: get_type_hints does not evaluate ForwardRefs inside NewType
type: behavior
versions: Python 3.10, Python 3.11
___
Python tracker
<https://bugs.python.org/issue46369>
___
___
Python-bugs-list mailing list
Unsubscribe:
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue46333] ForwardRef.__eq__ does not respect module parameter
Andreas H. added the comment: Ah, let me add one point: PEP563 (-> `from __future__ import annotations`) is also not helping. Even with PEP563 enabled, the JSON example Json = Union[ List['Json'], Dict[str, 'Json'], int, float, bool, None ] needs to be written in exact the same way as without PEP563. In other words there are cases where `ForwardRef` cannot be avoided. And unforntunately these are the cases where we have the ForwardRef missing context issue. -- ___ Python tracker <https://bugs.python.org/issue46333> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue46333] ForwardRef.__eq__ does not respect module parameter
Andreas H. added the comment:
Yeah, sure. The use-case is (de)serialization. Right now I use the library
cattr, but there are many others.
If you are interested there is related discussion in the cattr board [1].
The original problem is how to define the types for serialization.
1. If everything is a class, things work well, but not if type aliases are
used:
2. Type aliases sometimes have to be used - they cannot be avoided in all
cases,
especially with recursive types. The famous example is
Json = Union[ List['Json'], Dict[str, 'Json'], int, float, bool, None ]
(Note: even though mypy does not support this construct, pylance meanwhile
does [2])
3. `typing.Annotated` seems to be made for specifying additional information
such as value ranges, right to be used
in (de)serialization + validation contexts. Often these will just be type
aliases (not used as class members).
Combination is possible with typing.NewType.
The problem is that the implicit `ForwardRef('Json')` cannot be automatically
resolved (as it only a name with no context).
There is really no way this could be handle inside a library such as cattr.
When one wants to separate interface from implementation this issue is even
more complicated. The module where the serialization function is called is
typically different from the module with the type definition (This is probably
more the norm than the exception)
The option I expored is to explicitly create the ForwardRef and specify the
module parameter (even though I have to admit that I also did read that the
ForwardRef is only for internal use).
Json = Union[ List[ForwardRef(Json',module=__name__)], Dict[str,
ForwardRef(Json',module=__name__)], int, float, bool, None ]
Ugly, but this is better than nothing.
A (worse) alternative is to do
Json = Union[ List['Json'], Dict[str, 'Json'], int, float, bool, None ]
typing._eval_type(Json, globals(), locals())
That works since it puts the ForwardRefs into "evaluated" state
(self.__forward_value__ is then set)
A third, but future, alternative could be to automatically decompose the
argument of ForwardRef into module and type. Then one could write
Json = Union[ List[__name__+'.Json'], Dict[str, __name__+'.Json'], int,
float, bool, None ]
and all above problems would be solved.
Then ForwardRef could stay internal and this is more readable. Even though, the
static type checkers would need to understand `__name__+'TYPE'` constructs to
be useful.
Anyhow, it would be nice to have a solution here.
[1]: https://github.com/python-attrs/cattrs/issues/201
[2]:
https://devblogs.microsoft.com/python/pylance-introduces-five-new-features-that-enable-type-magic-for-python-developers/
--
___
Python tracker
<https://bugs.python.org/issue46333>
___
___
Python-bugs-list mailing list
Unsubscribe:
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue46333] ForwardRef.__eq__ does not respect module parameter
Andreas H. added the comment: I will give it a try. -- ___ Python tracker <https://bugs.python.org/issue46333> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue46333] ForwardRef.__eq__ does not respect module parameter
New submission from Andreas H. :
The __eq__ method of ForwardRef does not take into account the module
parameter.
However, ForwardRefs with dissimilar module parameters are referring to
different types even if they have different name. Thus also the ForwardRef's
with same name but different module, should not be considered equal.
Consider the following code
from typing import *
ZZ = Optional['YY']
YY = int
YY = Tuple[Optional[ForwardRef("YY", module=__name__)], int]
print( YY.__args__[0].__args__[0].__forward_module__ )
# this prints None, but should print __main__ (or whatever __name__ contains)
When the first ForwardRef is not created, the program behaves correctly
#ZZ = Optional['YY']
YY = int
YY = Tuple[Optional[ForwardRef("YY", module=__name__)], int]
print( YY.__args__[0].__args__[0].__forward_module__ )
# this prints __main__ (or whatever __name__ contains)
The issue is that the line `ZZ = Optional['YY']` creates a cache entry, which
is re-used instead of the second definition `Optional[ForwardRef("YY",
module=__name__)]` and thus shadows the different argument of ForwardRef.
This problem could be fixed if the __eq__ method of FowardRef also checks for
module equality.
i.e. in ForwardRef.__eq__ in typing.py replace
return self.__forward_arg__ == other.__forward_arg__
with
return self.__forward_arg__ == other.__forward_arg__ and
self.__forward__module__ == other.__forward__module__
Ideally, (and for consistency reasons) the `__repr__` method of `ForwardRef`
would also include the module arg if it present:
Change:
def __repr__(self):
return f'ForwardRef({self.__forward_arg__!r})'
to
def __repr__(self):
if self.__forward_module__ is None:
return f'ForwardRef({self.__forward_arg__!r})'
else:
return f'ForwardRef({self.__forward_arg__!r},
module={self.__forward__module!r})'
--
components: Library (Lib)
messages: 410221
nosy: andreash, gvanrossum, kj
priority: normal
severity: normal
status: open
title: ForwardRef.__eq__ does not respect module parameter
type: behavior
versions: Python 3.10, Python 3.9
___
Python tracker
<https://bugs.python.org/issue46333>
___
___
Python-bugs-list mailing list
Unsubscribe:
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue45184] Add `pop` function to remove context manager from (Async)ExitStack
Andreas H. added the comment: I see your point. But even with `pop` or `remove` it is still a stack or stack-like. In the normal case the context managers are still released in reverse order as they were added. Order cannot be changed arbitrarily. There is just the additional function of removing a single context manager prematurely(e.g. for graceful error recovery and such). I would perhaps say that a stack is the "wrong" solution to the problem of "programmatically combining context managers" [this is from the official documentaion] in the first place. I write wrong in quotes because it is of course not really wrong, as one wants the reverse exit order. But to adequately address the dynamic case one needs in my opinion the ability to prematurely remove context managers. Otherwise the use is limited. Reimplemeting the desired functionality with dicts or lists does not seem appealing to me as the code will be 90% the same to ExitStack. It will then also make ExitStack obsolete. So why not integrate it there? The unsymmetry of being able to add context managers but not being able to remove them also seems odd to me. -- ___ Python tracker <https://bugs.python.org/issue45184> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue45184] Add `pop` function to remove context manager from (Async)ExitStack
New submission from Andreas H. :
Currently it is not possible to remove context managers from an ExitStack (or
AsyncExitStack).
Workarounds are difficult and generally do accesses implementation details of
(Async)ExitStack.
See e.g. https://stackoverflow.com/a/37607405. It could be done as follows:
class AsyncExitStackWithPop(contextlib.AsyncExitStack):
"""Same as AsyncExitStack but with pop, i.e. removal functionality"""
async def pop(self, cm):
callbacks = self._exit_callbacks
self._exit_callbacks = collections.deque()
found = None
while callbacks:
cb = callbacks.popleft()
if cb[1].__self__ == cm:
found = cb
else:
self._exit_callbacks.append(cb)
if not found:
raise KeyError("context manager not found")
if found[0]:
return found[1](None,None,None)
else:
return await found[1](None, None, None)
The alternative is re-implementation of ExitStack with pop functionality, but
that is also very difficult to get right (especially with exceptions). Which is
probably the reason why there is ExitStack in the library at all.
So I propose to augment (Async)ExitStack with a `pop` method like above or
similar to the above.
Use-Cases:
An example is a component that manages several connections to network services.
During run-time the network services might need to change (i.e. some be
disconnected and some be connected according to business logic), or handle
re-connection events (ie. graceful response to network errors).
It is not too hard to imagine more use cases.
Essentially every case where dynamic resource management is needed and where
single resources are managable with python context managers.
--
components: Library (Lib)
messages: 401703
nosy: andreash, ncoghlan, yselivanov
priority: normal
severity: normal
status: open
title: Add `pop` function to remove context manager from (Async)ExitStack
type: enhancement
versions: Python 3.10, Python 3.11, Python 3.6, Python 3.7, Python 3.8, Python
3.9
___
Python tracker
<https://bugs.python.org/issue45184>
___
___
Python-bugs-list mailing list
Unsubscribe:
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue44795] asyncio.run does not allow for graceful shutdown of main task
New submission from Andreas H. :
The issue is that the main task (which was supplied to asyncio.run) has no
chance to clean up its "own" sub-tasks and handle
possible exceptions that occur during the sub-task clean up. It prevents a
graceful shutdown.
There is no way to prevent the current printing of the "unhandled" exeption,
even though the sub-task exception was catched by the main task. (See example
below)
-- Current behavior --
When asyncio.run() receives an (unhanded) exception, all tasks are cancelled
simultaneously.
If any task generates an exception during its clean-up phase this is printed to
the log, even though this exception is handled by the main task.
-- Expected behavior --
asyncio.run() should first cancel the main task, wait for it to complete its
shutdown (and possible cancel its own sub-tasks, with exception catching), and
*afterwards* cancel the remaining tasks.
-- Example Code --
For instance realize a graceful shutdown of a webserver when SIGTERM signal
handler raises a SystemExit exception.
import os
import asyncio
import logging
async def main():
logging.basicConfig(level=logging.INFO)
async def sub_task():
logging.info('sub_task: enter')
try:
while True:
await asyncio.sleep(1)
logging.info('some_task: action')
finally:
logging.info('sub_task: cleanup')
await asyncio.sleep(3)
logging.info('sub_task: cleanup generates exception')
raise ValueError()
logging.info('sub_task: cleanup end')
task = asyncio.create_task(sub_task())
try:
while True:
await asyncio.sleep(1)
except Exception as e:
logging.info(f"Main: exception {repr(e)} received: something went
wrong: cancelling sub-task")
task.cancel()
finally:
logging.info("Main: cleanup")
try:
await task
except Exception as e:
logging.info(f"Main: catched exception {repr(e)} from await
sub_task")
try:
asyncio.run( main() )
except KeyboardInterrupt:
pass
-- Script Output with Ctrl+C manually generating an KeyboardInterrupt exception
--
INFO:root:sub_task: enter
INFO:root:some_task: action
<--- CtrlC pressed here
INFO:root:Main: exception CancelledError() received: something went wrong:
cancelling sub-task
INFO:root:Main: cleanup
INFO:root:sub_task: cleanup
INFO:root:sub_task: cleanup generates exception
INFO:root:Main: catched exception ValueError() from await sub_task
ERROR:asyncio:unhandled exception during asyncio.run() shutdown
task: .sub_task() done, defined at
D:\Benutzer\projekte\iep\apps\data_player\_signals_test\test.py:10>
exception=ValueError()>
Traceback (most recent call last):
File
"C:\Users\z0013xar\AppData\Local\Continuum\anaconda3\lib\asyncio\runners.py",
line 43, in run
return loop.run_until_complete(main)
File
"C:\Users\z0013xar\AppData\Local\Continuum\anaconda3\lib\asyncio\base_events.py",
line 574, in run_until_complete
self.run_forever()
File
"C:\Users\z0013xar\AppData\Local\Continuum\anaconda3\lib\asyncio\base_events.py",
line 541, in run_forever
self._run_once()
File
"C:\Users\z0013xar\AppData\Local\Continuum\anaconda3\lib\asyncio\base_events.py",
line 1750, in _run_once
event_list = self._selector.select(timeout)
File "C:\Users\z0013xar\AppData\Local\Continuum\anaconda3\lib\selectors.py",
line 323, in select
r, w, _ = self._select(self._readers, self._writers, [], timeout)
File "C:\Users\z0013xar\AppData\Local\Continuum\anaconda3\lib\selectors.py",
line 314, in _select
r, w, x = select.select(r, w, w, timeout)
KeyboardInterrupt
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "D:\Benutzer\projekte\iep\apps\data_player\_signals_test\test.py", line
14, in sub_task
await asyncio.sleep(1)
File
"C:\Users\z0013xar\AppData\Local\Continuum\anaconda3\lib\asyncio\tasks.py",
line 595, in sleep
return await future
concurrent.futures._base.CancelledError
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "D:\Benutzer\projekte\iep\apps\data_player\_signals_test\test.py", line
34, in main
await task
File "D:\Benutzer\projekte\iep\apps\data_player\_signals_test\test.py", line
20, in sub_task
raise ValueError()
ValueError
-- Expected Output --
Same as above but without
"ERROR:asyncio:unhandled exception during asyncio.run() shutdown"
and following traceback
--
components: asyncio
messages: 398638
nosy: andreash, asvetlov, yselivanov
priority: normal
severity: normal
status: o
