[issue45184] Add `pop` function to remove context manager from (Async)ExitStack
Andreas H. added the comment: Inside the discussion an ExitPool class is sketched (https://mail.python.org/archives/list/python-id...@python.org/message/66W55FRCYMYF73TVMDMWDLVIZK4ZDHPD/), which provides this removal of context managers. What I learned is that this would have different cleanup mode (atexit style), as compared to present ExitStack cleanup (nested style). So contrary to what I was originally thinking ExitPool functionality would be close to, but not a strict superset of ExitStack functionality. Still such an ExitPool functionality would be extremely useful. -- ___ Python tracker <https://bugs.python.org/issue45184> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue46333] ForwardRef.__eq__ does not respect module parameter
Change by Andreas H. : -- pull_requests: +29443 pull_request: https://github.com/python/cpython/pull/31283 ___ Python tracker <https://bugs.python.org/issue46333> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue46369] get_type_hints does not evaluate ForwardRefs inside NewType
Andreas H. added the comment: Allright. B) sounds good to me. I dont think I have time today, so please feel to tackle the issue. If not I can look at it the next week. -- ___ Python tracker <https://bugs.python.org/issue46369> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue46373] TypedDict and NamedTuple do not evaluate cross-module ForwardRef in all cases
New submission from Andreas H. : TypedDict does not resolve cross-module ForwardRefs when the ForwardRef is not a direct one. In other words the fix GH-27017 (issue 41249) for TypedDict seems incomplete. The same issue seem to exist for NamedTuple. Example: #module.py TD = typing.TypedDict("TD", {'test': typing.List[typing.Optional['Y']]}) class Y: pass # other module class TDSub(module.TD): a: int get_type_hints(TDSub) # -> Exception NameError: Y not found On the other hand, with direct ForwardRef, as e.g. in TD = typing.TypedDict("TD", {'test': 'Y' } ) it works (that was indeed fixed by GH-27017) Same issue exists for NamedTuple. There neither of the above works, i.e. cross-module ForwardRefs are never resolve (but they could - als NamedTuple has the __module__ member set with to calling module). I am not sure if inheritance for NamedTuple is supported so I do not know if it is really a bug. The problem in the code is that in TypedDict the `module` parameter is passed only onto the immediate ForwardRef. One option could be to recursively walk the type and search for unpatched ForwardRefs and set the module parameter. On the other hand, the retroactive patching of forward refs is problematic as it may mess with the caching mechanism im typing.py. There may be a type with ForwardRef already in cache (and used by more than one user), but only for one user the `module` is to be updated. So probably a correct implementation is tricky, or some other way has to be found to update the `module` of ForwardRefs (e.g. by copying the type tree). For NamedTuple the whole mechanism of passing the `module` parameter to the ForwardRefs is not done (not even for direct ForwardRef ones). Not sure how important this (likely not very) is as I do not use TypedDict and NamedTuple. This is just to report it. -- components: Library (Lib) messages: 410547 nosy: AlexWaygood, Jelle Zijlstra, andreash, gvanrossum, kj, kumaraditya303, sobolevn priority: normal severity: normal status: open title: TypedDict and NamedTuple do not evaluate cross-module ForwardRef in all cases type: behavior versions: Python 3.10, Python 3.11 ___ Python tracker <https://bugs.python.org/issue46373> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue46371] A better way to resolve ForwardRefs in type aliases across modules
New submission from Andreas H. : (De)Serialization of in-memory data structures is an important application. However there is a rather unpleasant issue with ForwardRefs. One cannot export type aliases when they contain ForwardRefs (and expect things to work). Consider the example: Json = Union[ List['Json'], Dict[str, 'Json'], int, float, bool, None ] When used in another module the containing ForwardRefs cannot be resolved, or in the worst case, resolve to wrong types if in the caller namespace contains a symbol with the same name as ForwardRef refers to. This of course only applies to inspection-based tools which utilize the run-time information, not the static type checkers. Type aliases sometimes have to be used sometimes - they cannot be avoided in all cases, especially with recursive types as in the example above. There are several options to improve the situation. These are all that came to my mind and I want to expose them to discussion. 1. Guard with NewType Json = NewType('Json', Union[ List['Json'], Dict[str, 'Json'], int, float, bool, None ] ) Advantage: This could allow automatic cross-module ForwardRef resolution provided issue 46369 [1] is acknowledged as a bug and fixed. Disadvantage: Does not create a true type alias, but a sub-type. Type casts have to be used all the time, e.g. `data = Json(bool)`. So can only applied to a subset of use-cases (but is IMO a clean solution when it fits). 2. Use the `module` parameter of ForwardRef Json = Union[ List[ForwardRef('Json', module=__name__)], Dict[str, ForwardRef('Json', module=__name__)], int, float, bool, None ] ) Advantage: Works in 3.10. Disadvantage: Would require issue 46333 [2] to be fixed. ForwardRef is not meant to be instatiated by the user, also `module` parameter is currently completely internal. 3. Modify ForwardRef so that it accepts a fully qualified name Json = Union[ List[__name__+'.Json'], Dict[str, __name__+'.Json'], int, float, bool, None ] ) Advantage: This is only a tiny change (because ForwardRef has the `module` parameter). ForwardRef would stay internal. Less ugly than 2. Disadvantage: Still a bit ugly. Would also require issue 46333 [2] to be fixed. Relative module specification (as in relative imports) would not work. 4. Manual evaluation Json = Union[ List['Json'], Dict[str, 'Json'], int, float, bool, None ] resolve_type_alias(Json, globals(), locals() ) def resolve_type_alias(type, globalns, localns): class dummy: test: type typing.get_type_hints(dummy, globalns, localns) # Note: this modifies ForwardRefs in-place Advantage: Works in many versions. Disadvantage: Requires user to explicily call function after the last referenced type is defined (this may be physically separated from the type alias definition, which does not feel like a good solution especially since this ForwardRef export problem is only technical, and not even close to beeing obvious to most people) 5. Make `get_type_hints()` to work with type-aliases (essentially integrate above `resolve_type_alias`). The return value of get_type_hints() would be the same as argument, just with ForwardRefs in-place resolved. Json = Union[ List['Json'], Dict[str, 'Json'], int, float, bool, None ] get_type_hints(Json, globals(), locals()) Advantage: same as 4) but hides useful (but ugly) code Disadvantage: same as 4) 6. Make all types in typing (such as List, Dict, Union, etc...) to capture their calling module and pass this information to ForwardRef, when one is to be created. Then already during construction of ForwardRef the `module` will be correctly set. Json = Union[ List['Json'], Dict[str, 'Json'], int, float, bool, None ] Advantage: This requires no user intervention. Things will "just work" Disadvantage: This is rather big change. It is incompatible with the caching used inside typing.py (the new __module__ parameter would need to be taken into account in __hash__ and/or __eq__). And maybe has other issues I do not see at the moment. 7. Extend `TypeAlias` hint so it can be used in bracket way similar to e.g. `Annotated` Json = TypeAlias[ Union[ List['Json'], Dict[str, 'Json'], int, float, bool, None ] ] I know, presently it is supposed to be used as `Json: TypeAlias = Union[ ]`. But that is of no help at run-time, because the variable Json contains no run-time information. So even if TypeAlias would capture the calling module, this information is not passed on to the variable `Json`. This is different for the bracket notation TypeAlias[ .. ]. Advantage: Similar usage to Annotated. Would not require a big change such as in 4). Disadvantage: TypeAlias is not supposed to be used that way. 8. Introduce function to define a type alias, which sets
[issue46369] get_type_hints does not evaluate ForwardRefs inside NewType
New submission from Andreas H. : Consider the following: NewT = typing.NewType("NewT", typing.List[typing.Optional['Z']] ) class Z: pass Now get_type_hints() does not resolve the ForwardRef within NewType (but it does so for TypedDict, dataclasses, NamedTuple). Neither of the following works. 1) class dummy: test: NewT get_type_hints(test,None,None) print( NewT.__supertype__.__args__[0].__args__[0]__.__forward_evaluated__ ) # --> False Note: investigating the return value of get_type_hints does not change the outcome. get_type_hints() patches ForwardRefs in-place. 2) get_type_hints(NewT,None,None) # --> TypeError is not a module, class, method, or function For Python 3.10+ a workaround exists, but requires access to implementation details of NewType: class dummy: test: NewT.__supertype__ get_type_hints( dummy, globalns=sys.modules[NewT.__module__].__dict__, localns=None ) Possible solution could be A) to extent `get_type_hints` to explicitly handle NewType (basically call _eval_type for the __supertype__ member). That makes approach 2) work (but not 1) or B) to extend _eval_type() to handle process NewType as well. This would make 1) work (but not 2). I guess, since NewType is supposed to be semantically a subclass of the referred type, 2) is probably the preferred approach, which would suggest A). Strictly speaking this issue exits in all Python versions that have NewType, but it is easier to fix in 3.10 because there NewType has the __module__ member. -- components: Library (Lib) messages: 410528 nosy: andreash, gvanrossum, kj priority: normal severity: normal status: open title: get_type_hints does not evaluate ForwardRefs inside NewType type: behavior versions: Python 3.10, Python 3.11 ___ Python tracker <https://bugs.python.org/issue46369> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue46333] ForwardRef.__eq__ does not respect module parameter
Andreas H. added the comment: Ah, let me add one point: PEP563 (-> `from __future__ import annotations`) is also not helping. Even with PEP563 enabled, the JSON example Json = Union[ List['Json'], Dict[str, 'Json'], int, float, bool, None ] needs to be written in exact the same way as without PEP563. In other words there are cases where `ForwardRef` cannot be avoided. And unforntunately these are the cases where we have the ForwardRef missing context issue. -- ___ Python tracker <https://bugs.python.org/issue46333> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue46333] ForwardRef.__eq__ does not respect module parameter
Andreas H. added the comment: Yeah, sure. The use-case is (de)serialization. Right now I use the library cattr, but there are many others. If you are interested there is related discussion in the cattr board [1]. The original problem is how to define the types for serialization. 1. If everything is a class, things work well, but not if type aliases are used: 2. Type aliases sometimes have to be used - they cannot be avoided in all cases, especially with recursive types. The famous example is Json = Union[ List['Json'], Dict[str, 'Json'], int, float, bool, None ] (Note: even though mypy does not support this construct, pylance meanwhile does [2]) 3. `typing.Annotated` seems to be made for specifying additional information such as value ranges, right to be used in (de)serialization + validation contexts. Often these will just be type aliases (not used as class members). Combination is possible with typing.NewType. The problem is that the implicit `ForwardRef('Json')` cannot be automatically resolved (as it only a name with no context). There is really no way this could be handle inside a library such as cattr. When one wants to separate interface from implementation this issue is even more complicated. The module where the serialization function is called is typically different from the module with the type definition (This is probably more the norm than the exception) The option I expored is to explicitly create the ForwardRef and specify the module parameter (even though I have to admit that I also did read that the ForwardRef is only for internal use). Json = Union[ List[ForwardRef(Json',module=__name__)], Dict[str, ForwardRef(Json',module=__name__)], int, float, bool, None ] Ugly, but this is better than nothing. A (worse) alternative is to do Json = Union[ List['Json'], Dict[str, 'Json'], int, float, bool, None ] typing._eval_type(Json, globals(), locals()) That works since it puts the ForwardRefs into "evaluated" state (self.__forward_value__ is then set) A third, but future, alternative could be to automatically decompose the argument of ForwardRef into module and type. Then one could write Json = Union[ List[__name__+'.Json'], Dict[str, __name__+'.Json'], int, float, bool, None ] and all above problems would be solved. Then ForwardRef could stay internal and this is more readable. Even though, the static type checkers would need to understand `__name__+'TYPE'` constructs to be useful. Anyhow, it would be nice to have a solution here. [1]: https://github.com/python-attrs/cattrs/issues/201 [2]: https://devblogs.microsoft.com/python/pylance-introduces-five-new-features-that-enable-type-magic-for-python-developers/ -- ___ Python tracker <https://bugs.python.org/issue46333> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue46333] ForwardRef.__eq__ does not respect module parameter
Andreas H. added the comment: I will give it a try. -- ___ Python tracker <https://bugs.python.org/issue46333> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue46333] ForwardRef.__eq__ does not respect module parameter
New submission from Andreas H. : The __eq__ method of ForwardRef does not take into account the module parameter. However, ForwardRefs with dissimilar module parameters are referring to different types even if they have different name. Thus also the ForwardRef's with same name but different module, should not be considered equal. Consider the following code from typing import * ZZ = Optional['YY'] YY = int YY = Tuple[Optional[ForwardRef("YY", module=__name__)], int] print( YY.__args__[0].__args__[0].__forward_module__ ) # this prints None, but should print __main__ (or whatever __name__ contains) When the first ForwardRef is not created, the program behaves correctly #ZZ = Optional['YY'] YY = int YY = Tuple[Optional[ForwardRef("YY", module=__name__)], int] print( YY.__args__[0].__args__[0].__forward_module__ ) # this prints __main__ (or whatever __name__ contains) The issue is that the line `ZZ = Optional['YY']` creates a cache entry, which is re-used instead of the second definition `Optional[ForwardRef("YY", module=__name__)]` and thus shadows the different argument of ForwardRef. This problem could be fixed if the __eq__ method of FowardRef also checks for module equality. i.e. in ForwardRef.__eq__ in typing.py replace return self.__forward_arg__ == other.__forward_arg__ with return self.__forward_arg__ == other.__forward_arg__ and self.__forward__module__ == other.__forward__module__ Ideally, (and for consistency reasons) the `__repr__` method of `ForwardRef` would also include the module arg if it present: Change: def __repr__(self): return f'ForwardRef({self.__forward_arg__!r})' to def __repr__(self): if self.__forward_module__ is None: return f'ForwardRef({self.__forward_arg__!r})' else: return f'ForwardRef({self.__forward_arg__!r}, module={self.__forward__module!r})' -- components: Library (Lib) messages: 410221 nosy: andreash, gvanrossum, kj priority: normal severity: normal status: open title: ForwardRef.__eq__ does not respect module parameter type: behavior versions: Python 3.10, Python 3.9 ___ Python tracker <https://bugs.python.org/issue46333> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue45184] Add `pop` function to remove context manager from (Async)ExitStack
Andreas H. added the comment: I see your point. But even with `pop` or `remove` it is still a stack or stack-like. In the normal case the context managers are still released in reverse order as they were added. Order cannot be changed arbitrarily. There is just the additional function of removing a single context manager prematurely(e.g. for graceful error recovery and such). I would perhaps say that a stack is the "wrong" solution to the problem of "programmatically combining context managers" [this is from the official documentaion] in the first place. I write wrong in quotes because it is of course not really wrong, as one wants the reverse exit order. But to adequately address the dynamic case one needs in my opinion the ability to prematurely remove context managers. Otherwise the use is limited. Reimplemeting the desired functionality with dicts or lists does not seem appealing to me as the code will be 90% the same to ExitStack. It will then also make ExitStack obsolete. So why not integrate it there? The unsymmetry of being able to add context managers but not being able to remove them also seems odd to me. -- ___ Python tracker <https://bugs.python.org/issue45184> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue45184] Add `pop` function to remove context manager from (Async)ExitStack
New submission from Andreas H. : Currently it is not possible to remove context managers from an ExitStack (or AsyncExitStack). Workarounds are difficult and generally do accesses implementation details of (Async)ExitStack. See e.g. https://stackoverflow.com/a/37607405. It could be done as follows: class AsyncExitStackWithPop(contextlib.AsyncExitStack): """Same as AsyncExitStack but with pop, i.e. removal functionality""" async def pop(self, cm): callbacks = self._exit_callbacks self._exit_callbacks = collections.deque() found = None while callbacks: cb = callbacks.popleft() if cb[1].__self__ == cm: found = cb else: self._exit_callbacks.append(cb) if not found: raise KeyError("context manager not found") if found[0]: return found[1](None,None,None) else: return await found[1](None, None, None) The alternative is re-implementation of ExitStack with pop functionality, but that is also very difficult to get right (especially with exceptions). Which is probably the reason why there is ExitStack in the library at all. So I propose to augment (Async)ExitStack with a `pop` method like above or similar to the above. Use-Cases: An example is a component that manages several connections to network services. During run-time the network services might need to change (i.e. some be disconnected and some be connected according to business logic), or handle re-connection events (ie. graceful response to network errors). It is not too hard to imagine more use cases. Essentially every case where dynamic resource management is needed and where single resources are managable with python context managers. -- components: Library (Lib) messages: 401703 nosy: andreash, ncoghlan, yselivanov priority: normal severity: normal status: open title: Add `pop` function to remove context manager from (Async)ExitStack type: enhancement versions: Python 3.10, Python 3.11, Python 3.6, Python 3.7, Python 3.8, Python 3.9 ___ Python tracker <https://bugs.python.org/issue45184> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue44795] asyncio.run does not allow for graceful shutdown of main task
New submission from Andreas H. : The issue is that the main task (which was supplied to asyncio.run) has no chance to clean up its "own" sub-tasks and handle possible exceptions that occur during the sub-task clean up. It prevents a graceful shutdown. There is no way to prevent the current printing of the "unhandled" exeption, even though the sub-task exception was catched by the main task. (See example below) -- Current behavior -- When asyncio.run() receives an (unhanded) exception, all tasks are cancelled simultaneously. If any task generates an exception during its clean-up phase this is printed to the log, even though this exception is handled by the main task. -- Expected behavior -- asyncio.run() should first cancel the main task, wait for it to complete its shutdown (and possible cancel its own sub-tasks, with exception catching), and *afterwards* cancel the remaining tasks. -- Example Code -- For instance realize a graceful shutdown of a webserver when SIGTERM signal handler raises a SystemExit exception. import os import asyncio import logging async def main(): logging.basicConfig(level=logging.INFO) async def sub_task(): logging.info('sub_task: enter') try: while True: await asyncio.sleep(1) logging.info('some_task: action') finally: logging.info('sub_task: cleanup') await asyncio.sleep(3) logging.info('sub_task: cleanup generates exception') raise ValueError() logging.info('sub_task: cleanup end') task = asyncio.create_task(sub_task()) try: while True: await asyncio.sleep(1) except Exception as e: logging.info(f"Main: exception {repr(e)} received: something went wrong: cancelling sub-task") task.cancel() finally: logging.info("Main: cleanup") try: await task except Exception as e: logging.info(f"Main: catched exception {repr(e)} from await sub_task") try: asyncio.run( main() ) except KeyboardInterrupt: pass -- Script Output with Ctrl+C manually generating an KeyboardInterrupt exception -- INFO:root:sub_task: enter INFO:root:some_task: action <--- CtrlC pressed here INFO:root:Main: exception CancelledError() received: something went wrong: cancelling sub-task INFO:root:Main: cleanup INFO:root:sub_task: cleanup INFO:root:sub_task: cleanup generates exception INFO:root:Main: catched exception ValueError() from await sub_task ERROR:asyncio:unhandled exception during asyncio.run() shutdown task: .sub_task() done, defined at D:\Benutzer\projekte\iep\apps\data_player\_signals_test\test.py:10> exception=ValueError()> Traceback (most recent call last): File "C:\Users\z0013xar\AppData\Local\Continuum\anaconda3\lib\asyncio\runners.py", line 43, in run return loop.run_until_complete(main) File "C:\Users\z0013xar\AppData\Local\Continuum\anaconda3\lib\asyncio\base_events.py", line 574, in run_until_complete self.run_forever() File "C:\Users\z0013xar\AppData\Local\Continuum\anaconda3\lib\asyncio\base_events.py", line 541, in run_forever self._run_once() File "C:\Users\z0013xar\AppData\Local\Continuum\anaconda3\lib\asyncio\base_events.py", line 1750, in _run_once event_list = self._selector.select(timeout) File "C:\Users\z0013xar\AppData\Local\Continuum\anaconda3\lib\selectors.py", line 323, in select r, w, _ = self._select(self._readers, self._writers, [], timeout) File "C:\Users\z0013xar\AppData\Local\Continuum\anaconda3\lib\selectors.py", line 314, in _select r, w, x = select.select(r, w, w, timeout) KeyboardInterrupt During handling of the above exception, another exception occurred: Traceback (most recent call last): File "D:\Benutzer\projekte\iep\apps\data_player\_signals_test\test.py", line 14, in sub_task await asyncio.sleep(1) File "C:\Users\z0013xar\AppData\Local\Continuum\anaconda3\lib\asyncio\tasks.py", line 595, in sleep return await future concurrent.futures._base.CancelledError During handling of the above exception, another exception occurred: Traceback (most recent call last): File "D:\Benutzer\projekte\iep\apps\data_player\_signals_test\test.py", line 34, in main await task File "D:\Benutzer\projekte\iep\apps\data_player\_signals_test\test.py", line 20, in sub_task raise ValueError() ValueError -- Expected Output -- Same as above but without "ERROR:asyncio:unhandled exception during asyncio.run() shutdown" and following traceback -- components: asyncio messages: 398638 nosy: andreash, asvetlov, yselivanov priority: normal severity: normal status: o