[issue46843] PersistentTaskGroup API
Joongi Kim added the comment: I have released the new version of aiotools with rewritten TaskGroup and PersistentTaskGroup. https://aiotools.readthedocs.io/en/latest/aiotools.taskgroup.html aiotools.TaskGroup has small additions to asyncio.TaskGroup: a naming API and `current_taskgroup` context variable. aiotools.PersistentTaskGroup is what I've described here, highlighting both async-with usage and long-lived object usage and `all_ptaskgroups()` classmethod for the monitoring purpose except the two-phase graceful shutdown (future TODO). -- ___ Python tracker <https://bugs.python.org/issue46843> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue46875] Missing name in TaskGroup.__repr__()
Joongi Kim added the comment: Ah, I'm confused with aiotools.TaskGroup (originated from EdgeDB's TaskGroup) code while browsing both aiotools and stdlib asyncio.TaskGroup source codes. The naming facility seems to be intentionally removed when ported to the stdlib. So I am closing this and sorry fo the noise. Though, is there any particular reason to remove it? My guess is that you think that TaskGroup is more like a control-flow structure which does not need to be named, just like we don't name "for loop" for instance. -- stage: -> resolved status: open -> closed ___ Python tracker <https://bugs.python.org/issue46875> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue46875] Missing name in TaskGroup.__repr__()
New submission from Joongi Kim : The __repr__() method in asyncio.TaskGroup does not include self._name. I think this is a simple overlook, because asyncio.Task includes the task name in __repr__(). :wink: https://github.com/python/cpython/blob/345572a1a02/Lib/asyncio/taskgroups.py#L28-L42 I'll make a simple PR to fix it. -- components: asyncio messages: 414162 nosy: achimnol, asvetlov, gvanrossum, yselivanov priority: normal severity: normal status: open title: Missing name in TaskGroup.__repr__() versions: Python 3.11 ___ Python tracker <https://bugs.python.org/issue46875> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue46843] PersistentTaskGroup API
Joongi Kim added the comment: I have updated the PersistentTaskGroup implementation referring asyncio.TaskGroup and added more detailed test cases, which works with the latest Python 3.11 GitHub checkout. https://github.com/achimnol/aiotools/pull/36/files Please have a look at the class docstring. There are two different usage: async context manager vs. attributes of long-lived objects. One of the point is to "revive" asyncio.gather() with return_exceptions=True but let it handle/report exceptions immediately with customizable exception handler. Currently two-phase shutdown is not implemented yet as I'm still thinking about how to adapt the current implementation. -- ___ Python tracker <https://bugs.python.org/issue46843> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue46843] PersistentTaskGroup API
Joongi Kim added the comment: Short summary: PersistentTaskGroup shares the followings from TaskGroup: - It uses WeakSet to keep track of child tasks. - After exiting the async context manager scope (or the shutdown procedure), it ensures that all tasks are complete or cancelled. PersistentTaskGroup differs in that: - It keeps running after all tasks successfully finish unless it is explicitly shutdown or the parent task is cancelled. - It is one of the main use cases that shutdown() method is called separately. The shutdown procedure may be triggered from different task contexts. - It provides two-phase cancellation with a configurable grace period. - It does not propagate unhandled exceptions and cancellations from child tasks to the outside of the task group and sibling tasks but calls a customizable fallback exception handler. -> This could be done without modifying TaskGroup. The API looks similar to TaskGroup with minor modification. The semantics of a PersistentTaskGroup more resembles a nested event loop, in that it has its own set of tasks, it keeps running until closed, and it has its own fallback exception handler. Note that current aiotools implementation lacks many details, such as two-phase cancellation. I'm going to implement more soon. -- ___ Python tracker <https://bugs.python.org/issue46843> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue46843] PersistentTaskGroup API
Joongi Kim added the comment: > And just a question: I'm just curious about what happens if belonging tasks > see the cancellation raised from their inner tasks. Sibling tasks should not > be cancelled, and the outer task group should not be cancelled, unless the > task group itself has requested cancellation. Could the new cancellation > counter help this? To achieve this by distinguishing cancellation from inner/outer tasks, TaskGroup._on_task_done() should be modified to skip setting _on_completed_fut because it should keep running. Swallowing exceptions in child tasks can be done without modifying TaskGroup, but this part requires changes of TaskGroup. Another difference is the usage. Instead of relying on the async context manager interface, we would call "TaskGroup.shutdown()" separately from either directly in signal handlers or from cleanup methods of long-lived objects that have task groups as attributes. And I also want to perform two-phase cancellation: instead of cancelling all tasks immediately as in current _abort(), have a configurable grace period until they have chances to complete and then cancel with additional timeout on cancellation itself to prevent hangs. -- ___ Python tracker <https://bugs.python.org/issue46843> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue46843] PersistentTaskGroup API
Joongi Kim added the comment: > As for errors in siblings aborting the TaskGroup, could you apply a wrapper > to the scheduled coroutines to swallow and log any errors yourself? Yes, this could be a simplest way to implement PersistentTaskGroup if TaskGroup supports "persistent" option to keep it running. And just a question: I'm just curious about what happens if belonging tasks see the cancellation raised from their inner tasks. Sibling tasks should not be cancelled, and the outer task group should not be cancelled, unless the task group itself has requested cancellation. Could the new cancellation counter help this? -- ___ Python tracker <https://bugs.python.org/issue46843> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue46843] PersistentTaskGroup API
Joongi Kim added the comment: Good to hear that TaskGroup already uses WeakSet. When all tasks finish, PersistentTaskGroup should not finish and wait for future tasks, unless explicitly cancelled or shutdown. Could this be also configured with asyncio.TaskGroup? I'm also ok with adding a simple option for such behavior to asyncio.TaskGroup instead of adding a whole new API/class. -- ___ Python tracker <https://bugs.python.org/issue46843> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue46843] PersistentTaskGroup API
Joongi Kim added the comment: Anoter case: https://github.com/lablup/backend.ai-manager/pull/533 https://github.com/lablup/backend.ai-agent/pull/341 When shutting down the application, I'd like to explicitly cancel the shielded tasks, while keep them shielded before shutdown. So I inserted `ptaskgroup.create_task()` inside `asyncio.shield()`, so that the tasks are not cancelled upon the cancellation of their callers but they get cancelled when the server shuts down. This pattern is conveniently implemented with PersistentTaskGroup. -- ___ Python tracker <https://bugs.python.org/issue46843> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue46844] Implicit binding of PersistentTaskGroup (or virtual event loops)
Joongi Kim added the comment: Updated the title to reduce confusion. -- title: Context-based TaskGroup for legacy libraries -> Implicit binding of PersistentTaskGroup (or virtual event loops) ___ Python tracker <https://bugs.python.org/issue46844> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue46844] Context-based TaskGroup for legacy libraries
Joongi Kim added the comment: I have added more about my stories in bpo-46843. I think the suggestion of implicit taskgroup binding with the current asyncio.TaskGroup has no point but it would have more meaning with PersistentTaskGroup. So, if we treat PersistentTaskGroup as a "nested, hierarchical virtual event loop" to repeat and group shutdown procedures for different task sets separately, the point may look a little bit clearer. It is more like assigning a virtual event loop to different modules and libraries, while keeping the behavior of asyncio.create_task() same. The difference is that the caller controls when these virtual loops are terminated and in what order. Does this make sense better? -- ___ Python tracker <https://bugs.python.org/issue46844> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue46843] PersistentTaskGroup API
Joongi Kim added the comment: Here is one another story. When handling message queues in distributed applications, I use the following pattern frequently for graceful shutdown: * Use a sentinel object to signal the end of queue. * Enqueue the sentinel object when: - The server is shutting down. (i.e., cancelled explicitly) - The connection peer has sent an explicit termination message. (e.g., EOF) * Wait until all enqueued messages before the sentinal object to be processed. - I'd like to impose a shutdown timeout on here using a persistent task group, by spawning all handler tasks of this queue into it. -- ___ Python tracker <https://bugs.python.org/issue46843> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue46843] PersistentTaskGroup API
Joongi Kim added the comment: I ended up with the following conclusion: - The new abstraction should not cancel sibling tasks and itself upon unhandled execption but loudly report such errors (and the fallback error handler should be customizable). - Nesting task groups will give additional benefits such as orderly shutdown of different task groups. Empty up message queues before shutting down netweork connections, etc. You may take my suggestion as "let's have a hierarchical nested virtual event loops to group tasks". PersistentTaskGroup actually shares many characteristics with the event loop while itself is not an event loop. So I came up with WeakSet with task decorators to handle exceptions by my own, and this is the current rudimentary implementation of PersistentTaskGroup in aiotools. And I discovered from the additional search results that the same pattern ---managing sporadic tasks using WeakSet and writing a proper cancellation loop of them---appear quite commonly in many different asyncio applications and libraries. So that's why I think this should be an intrinsic/essential abstraction. -- ___ Python tracker <https://bugs.python.org/issue46843> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue46843] PersistentTaskGroup API
Joongi Kim added the comment: This particular experience, https://github.com/lablup/backend.ai-agent/pull/331, has actually motivated me to suggest PersistentTaskGroup. The program subscribes the event stream of Docker daemon using aiohttp as an asyncio task, and this should be kept running throughout the whole application lifetime. I first applied aiotools.TaskGroup to ensure shutdown of spawned event handler tasks, but I missed that it cancels all sibling tasks if one of the spawned tasks bubbles up an unhandled exception. This has caused silent termination of the subscriber task and led to a bug. We could debug this issue by inspecting aiomonitor and checking the existence of this task. After this issue, I began to think we need a proper abstraction of a long-running task group (NOTE: the task group is long-running. The lifetime of internal tasks does not matter). Another case is that https://github.com/lablup/backend.ai/issues/330. One of our customer site has suffered from excessive CPU usage by our program. We could identify the issue by aiomonitor, and the root cause was the indefinite accumulation of peridoically created asyncio tasks to measure the disk usage of user directories, when there are too many files in them. Since the number of tasks have exceeded 10K, it was very difficult to group and distinguish individual asyncio tasks in aiomonitor. I thought that it would be nice if we could group such tasks into long-running groups and view task statistics separately. -- ___ Python tracker <https://bugs.python.org/issue46843> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue46843] PersistentTaskGroup API
Joongi Kim added the comment: @gvanrossum As you mentioned, the event loop currently plays the role of the top-level task group already, even without introducing yet another top-level task. For instance, asyncio.run() includes necessary shutdown procedures to cancel all belonging unfinished tasks and async generators. However, I think we should provide an abstraction to organize the shutdown procedures in a *hierarchical* manner. For example, we could cancel all event handler tasks before cancelling all HTTP handler tasks upon a web server shutdown. This prevents any potential races between theses two different task sets. I think you could agree with the necessity of orderly release of underlying resources during shutdown in general. Currently asyncio.Task.all_tasks() is just a list created from WeakSet and we cannot guarantee which tasks will be cancelled first. Yes, this can be done by manually writing codes to declare multiple WeakSets and a for-loop to cancel the contained tasks by enumerating over them, just like asyncio.run() does. With the new addition of TaskGroup and ExceptionGroup, this code does not require core changes of Python. But I believe that this hierarchical persistent task group abstraction should be an essential part of the API and asyncio tutorials when writing server applications. asyncio.run() could be written by users, but I think the core devs have agreed with that it is an essential abstraction to be included in the stdlib. I'd like argue that hierarchical persistent task groups is the same case. Though I named it "PersistentTaskGroup" because it looks similar to TaskGroup, but this name may be misleading. In PersistentTaskGroup, even when all tasks finish successfully, it does NOT terminate but keeps waiting for new tasks to be spawned. It terminates only when the outer task is cancelled or its shutdown() method is called. Note that belonging tasks may be either short-running or long-running, and this does not matter. The point is to shutdown any remaining tasks in an orderly manner. If you don't like the naming, please suggest alternatives. -- ___ Python tracker <https://bugs.python.org/issue46843> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue46622] Add an async variant of lru_cache for coroutines.
Change by Joongi Kim : -- nosy: +achimnol ___ Python tracker <https://bugs.python.org/issue46622> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue46843] PersistentTaskGroup API
Joongi Kim added the comment: @yselivanov @asvetlov I think this API suggestion would require more refining and discussion in depths, and probably it may be better to undergo the PEP writing and review process. Or I might need to have a separate discussion thread somewhere else (maybe discuss.python.org?). Since I'm just a newbie in terms of Python core/stdlib development, could one of you guide me with what you think as the right way? -- ___ Python tracker <https://bugs.python.org/issue46843> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue46843] PersistentTaskGroup API
Joongi Kim added the comment: Some search results from cs.github.com with the input "asyncio task weakset", which may be replaced/simplified with PersistentTaskGroup: - https://github.com/Textualize/textual/blob/38efc821737e3158a8c4c7ef8ecfa953dc7c0ba8/src/textual/message_pump.py#L43 - https://github.com/aiokitchen/aiomisc/blob/59abd4434e6d134537490db699f89a51df1e6bbc/aiomisc/entrypoint.py#L132 - https://github.com/anki/cozmo-python-sdk/blob/dd29edef18748fcd816550469195323842a7872e/src/cozmo/event.py#L102 - https://github.com/aio-libs/aiohttp-sse/blob/db7d49bfc8a4907d9a8e7696a85b9772e1c550eb/examples/graceful_shutdown.py#L50 - https://github.com/mosquito/aiormq/blob/9c6c0dfc771ea8f6e79b7532177640c2692c640f/aiormq/base.py#L18 https://github.com/mars-project/mars/blob/d1a14cc4a1cb96e40e1d81eef38113b0c9221a84/mars/lib/aio/_runners.py#L57 -- ___ Python tracker <https://bugs.python.org/issue46843> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue46843] PersistentTaskGroup API
Joongi Kim added the comment: Example use cases: * Implement an event iteration loop to fetch events and dispatch the handlers depending on the event type (e.g., WebSocket connections, message queues, etc.) - https://github.com/aio-libs/aiohttp/pull/2885 - https://github.com/lablup/backend.ai-manager/pull/533 - https://github.com/lablup/backend.ai-agent/pull/341 - https://github.com/lablup/backend.ai-agent/pull/331 * Separate monitoring of event handler tasks by the event sources. - aiomonitor extension to count currently ongoing tasks and extract the most frequent task stack frames * Separate the fallback exception handlers by each persistent task group, instead of using the single "global" event loop exception handler. -- ___ Python tracker <https://bugs.python.org/issue46843> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue46843] PersistentTaskGroup API
Joongi Kim added the comment: I think people may ask "why in stdlib?". My reasons are: - We are adding new asyncio APIs in 3.11 such as TaskGroup, so I think it is a good time to add another one, as long as it does not break existing stuffs. - I believe that long-running task sets are equally representative use-case for real-world asyncio applications, particularly for servers. Why not to have intrinsic support for them? - PersistentTaskGroup is going to be universally adopted throughout my 70+K LoC asyncio codebase, for instance, in every aiohttp.Application context, plugin contexts and modules, etc. Of course, the name "PersistentTaskGroup" may look quite long, and I'm completely open with alternative suggestions. I also welcome suggestions on changes to its functional semantics based on your experience and knowledge. -- ___ Python tracker <https://bugs.python.org/issue46843> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue46843] PersistentTaskGroup API
Joongi Kim added the comment: So I have more things in mind. Basically PersistentTaskGroup resemble TaskGroup in that: - It has the same "create_task()" method. - It has an explicit "cancel()" or "shutdown()" method. - Exiting of the context manager means that all tasks of it have either completed or cancelled. TaskGroup is intended to be used for a short-lived set of tasks, while PersistentTaskGroup is intended for a long-running set of tasks though individual tasks may be short-lived. Thus, adding globally accessible monitoring facility for plain TaskGroup would not be that useful. In contrast, it is super-useful to have a monitoring feature in PersistentTaskGroup! In aiomonitor, we can enumerate the currently running asyncio tasks by reading asyncio.Task.all_tasks(). This has saved my life several times when debugging real-world server applications. I think we can go further by having asyncio.PersistentTaskGroup.all_task_groups() which works in the same way. If we make different modules and libraries to use different persistent task groups, then we could keep track of their task statistics separately. -- ___ Python tracker <https://bugs.python.org/issue46843> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue46844] Context-based TaskGroup for legacy libraries
Joongi Kim added the comment: Ok, let me be clear: Patching asyncio.create_task() to support this opt-in contextual task group binding is not an ultimate goal of this issue. If it becomes possible to override/extend the task factory at runtime with any event loop implementation, then it's ok to implement this feature request as a 3rd-party library. I also don't want to bloat the stdlib with version-specific branches, if there are alternative ways to achieve the same goal. I just wanted to check out your opinons and potential alternative approaches to implement it. -- ___ Python tracker <https://bugs.python.org/issue46844> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue46844] Context-based TaskGroup for legacy libraries
Joongi Kim added the comment: Ah, and this use case also requires that TaskGroup should have an option like `return_exceptions=True` which makes it not to cancel sibling tasks upon unhandled exceptions, as I suggested in PersistentTaskGroup (bpo-46843). -- ___ Python tracker <https://bugs.python.org/issue46844> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue46844] Context-based TaskGroup for legacy libraries
Joongi Kim added the comment: An example would be like: tg = asyncio.TaskGroup() ... async with tg: with asyncio.TaskGroupBinder(tg): # just a hypothetical API asyncio.create_task(...) # equivalent to tg.create_task(...) await some_library.some_work() # all tasks are bound to tg asyncio.create_task(...) # fire-and-forget (not bound to tg) If TaskGroup supports enumeration/counting of its own tasks and asyncio allows enumeration of TaskGroups just like asyncio.Task.all_tasks(), we could extend aiomonitor to provide per-taskgroup statistics. In my projects, we have multiple cases to find and fix bugs in customer sites using aiomonitor and I'm willing to improve aiomonitor to support task groups as well. -- ___ Python tracker <https://bugs.python.org/issue46844> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue46844] Context-based TaskGroup for legacy libraries
Joongi Kim added the comment: My propsal is to opt-in the taskgroup binding for asyncio.create_task() under a specific context, not changing the defautl behavior. -- ___ Python tracker <https://bugs.python.org/issue46844> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue46844] Context-based TaskGroup for legacy libraries
Joongi Kim added the comment: It is also useful to write debugging/monitoring codes for asyncio applications. For instance, we could "group" tasks from different libraries and count them. -- ___ Python tracker <https://bugs.python.org/issue46844> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue46844] Context-based TaskGroup for legacy libraries
Joongi Kim added the comment: Conceptually it is similar to replace malloc using LD_PRELOAD or LD_LIBRARY_PATH manipulation. When I cannot modify the executable/library binaries, this allows replacing the functionality of specific functions. If we could assign a specific (persistent) task group to all asyncio tasks spawned by a black-box code (when the black-box itself does not use task groups), we could achieve the full application-level transparency on the timing of task cancellation. -- ___ Python tracker <https://bugs.python.org/issue46844> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue46844] Context-based TaskGroup for legacy libraries
Joongi Kim added the comment: The main benefit is that any legacy code that I cannot modify can be upgraded to TaskGroup-based codes, which offers a better machinary for exception handling and propagation. There may be different ways to visit this issue: allow replacing the task factory in asyncio at runtime. Then I could just implement my own snippet to transfer the "ownership" of the task to a specific task group. -- ___ Python tracker <https://bugs.python.org/issue46844> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue46844] Context-based TaskGroup for legacy libraries
New submission from Joongi Kim : Along with bpo-46843 and the new asyncio.TaskGroup API, I would like to suggest addition of context-based TaskGroup feature. Currently asyncio.create_task() just creates a new task directly attached to the event loop, while asyncio.TaskGroup.create_task() creates a new task managed by the TaskGroup instance. It would be ideal to all existing asyncio codes to migrate to use TaskGroup, but this is impractical. An alternative approach is to implicitly bind asyncio.create_task() under a specific context to a specific task group, probably using contextvars. I believe that this approach would allow more control over tasks implicitly spawned by 3rd-party libraries that cannot control. How about your thoughts? -- components: asyncio messages: 413881 nosy: achimnol, asvetlov, gvanrossum, yselivanov priority: normal severity: normal status: open title: Context-based TaskGroup for legacy libraries type: enhancement versions: Python 3.11 ___ Python tracker <https://bugs.python.org/issue46844> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue46843] PersistentTaskGroup API
New submission from Joongi Kim : I'm now tracking the recent addition and discussion of TaskGroup and cancellation scopes. It's interesting! :) I would like to suggest to have a different mode of operation in asyncio.TaskGroup, which I named "PersistentTaskGroup". AFAIK, TaskGroup targets to replace asyncio.gather, ensuring completion or cancellation of all tasks within the context manager scope. I believe that a "safe" asyncio application should consist of a nested tree of task groups, which allow us to explicitly state when tasks of different purposes and contexts terminate. For example, a task group for database transactions should be shutdown before a task group for HTTP handlers is shutdown. To this end, in server applications with many sporadically spawned tasks throughout the whole process lifetime, there are different requirements for a task group that manages such task sets. The tasks should *not* be cancelled upon the unhandled exceptions of sibling tasks in the task group, while we need an explicit "fallback" exception handler for those (just like "return_exceptions=True" in asyncio.gather). The tasks belong to the task group but their references should not be kept forever to prevent memory leak (I'd suggest using weakref.WeakSet). When terminating the task group itself, the ongoing tasks should be cancelled. The cancellation process upon termination may happend in two phases: cancel request with initial timeout + additional limited waiting of cancellations. (This is what Guido has mentioned in the discussion in bpo-46771.) An initial sketch of PersistentTaskGroup is on aiotools: https://github.com/achimnol/aiotools/blob/main/src/aiotools/ptaskgroup.py Currently has no two-phase cancellation because it would require Python 3.11 with asyncio.Task.uncancel(). As Andrew has left a comment (https://github.com/achimnol/aiotools/issues/29#issuecomment-997437030), I think it is the time to revisit the concrete API design and whether to include PersistentTaskGroup in the stdlib or not. -- components: asyncio messages: 413880 nosy: achimnol, asvetlov, gvanrossum, yselivanov priority: normal severity: normal status: open title: PersistentTaskGroup API type: enhancement versions: Python 3.11 ___ Python tracker <https://bugs.python.org/issue46843> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue45416] "loop argument must agree with lock" instantiating asyncio.Condition
Change by Joongi Kim : -- keywords: +patch nosy: +Joongi Kim nosy_count: 6.0 -> 7.0 pull_requests: +27160 stage: -> patch review pull_request: https://github.com/python/cpython/pull/28850 ___ Python tracker <https://bugs.python.org/issue45416> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue44738] io_uring as a new backend to selectors and asyncio
Joongi Kim added the comment: As in the previous discussion, instead of tackling stdlib right away, it would be nice to evaluate the approach using 3rd-party libs, such as trio and/or async-tokio, or maybe a new library. I have a strong feeling that we need to improve the async file I/O. AFAIK, aiofiles is the only choice we have and it uses a thread pool, which involves many more context switches than required. -- ___ Python tracker <https://bugs.python.org/issue44738> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue44738] io_uring as a new backend to selectors and asyncio
Joongi Kim added the comment: Ah, yes, but one year has passed so it may be another chance to discuss its adoption, as new advances like tokio_uring became available. -- ___ Python tracker <https://bugs.python.org/issue44738> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue44738] io_uring as a new backend to selectors and asyncio
New submission from Joongi Kim : This is a rough early idea suggestion on adding io_uring as an alternative I/O multiplexing mechanism in Python (maybe selectors and asyncio). io_uring is a relatively new I/O mechanism introduced in Linux kernel 5.1 or later. https://lwn.net/Articles/776703/ https://lwn.net/Articles/810414/ https://blogs.oracle.com/linux/post/an-introduction-to-the-io_uring-asynchronous-io-framework The advantages of io_uring over epoll: - completion-based - less number of syscalls - higher performance (https://twitter.com/hielkedv/status/1218891982636027905) - file I/O support including read/write/stat/open/close I'm not sure that io_uring would bring actual performance improvements to Python (and asyncio) or not yet. We need some exploration and prototyping, but still technically it would be a nice-to-have feature, considering Python's recent speed-up optimizations. Also io_uring is also intended to support high-speed storage devices such as NVMe, and may be a good addition to asyncio in terms of improved async file I/O support. Here are existing attempts to incorporate uring in other languages: - liburing (C, https://github.com/axboe/liburing) - iou, tokio-uring (Rust, https://tokio.rs/blog/2021-07-tokio-uring) I don't have any estimation on the efforts and time required to do the work, but just want to spark the discussion. :) -- components: asyncio messages: 398215 nosy: achimnol, asvetlov, corona10, njs, yselivanov priority: normal severity: normal status: open title: io_uring as a new backend to selectors and asyncio type: enhancement versions: Python 3.11 ___ Python tracker <https://bugs.python.org/issue44738> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue44343] Adding the "with" statement support to ContextVar
Joongi Kim added the comment: After checking out PEP-567 (https://www.python.org/dev/peps/pep-0567/), I'm adding njs to the nosy list. -- nosy: +njs ___ Python tracker <https://bugs.python.org/issue44343> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue44343] Adding the "with" statement support to ContextVar
New submission from Joongi Kim :
This is just an idea: ContextVar.set() and ContextVar.reset() looks naturally
mappable with the "with" statement.
For example:
a = ContextVar('a')
token = a.set(1234)
...
a.reset(token)
could be naturally rewritten as:
a = ContextVar('a')
with a.set(1234):
...
Is there any particular reason *not* to do this?
If not, I'd like make a PR to add this API.
Naming suggestions of this API are welcome, but it also seems possible to keep
it "set()" if we retain the reference to the ContextVar instance in the Token
instance.
--
components: Library (Lib)
messages: 395302
nosy: achimnol
priority: normal
severity: normal
status: open
title: Adding the "with" statement support to ContextVar
type: enhancement
versions: Python 3.11
___
Python tracker
<https://bugs.python.org/issue44343>
___
___
Python-bugs-list mailing list
Unsubscribe:
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue41229] Asynchronous generator memory leak
Change by Joongi Kim : -- pull_requests: +22115 pull_request: https://github.com/python/cpython/pull/23217 ___ Python tracker <https://bugs.python.org/issue41229> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue41229] Asynchronous generator memory leak
Change by Joongi Kim : -- nosy: +Joongi Kim nosy_count: 6.0 -> 7.0 pull_requests: +20687 pull_request: https://github.com/python/cpython/pull/21545 ___ Python tracker <https://bugs.python.org/issue41229> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue41229] Asynchronous generator memory leak
Change by Joongi Kim : -- nosy: +njs ___ Python tracker <https://bugs.python.org/issue41229> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue41229] Asynchronous generator memory leak
Joongi Kim added the comment: I've searched the Python documentation and the docs must be updated to explicitly state the necessity of aclose(). refs) https://docs.python.org/3/reference/expressions.html#asynchronous-generator-functions https://www.python.org/dev/peps/pep-0525/ I'm not sure that what the original authors' intention is, but for me, it looks like that calling aclose() is an optional thing and the responsibility to call aclose() on async generators is left to the asyncgen-shutdown handler of the event loop. The example in this issue show that we need to aclose asyncgens whenever we are done with it, even far before shutting down the event loop. -- ___ Python tracker <https://bugs.python.org/issue41229> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue41229] Asynchronous generator memory leak
Joongi Kim added the comment: >From the given example, if I add "await q.aclose()" after "await >q.asend(123456)" it does not leak the memory. This is a good example showing that we should always wrap async generators with explicit "aclosing" context manager (which does not exist yet in the stdlib). I'm already doing so by writing a custom library: https://github.com/achimnol/aiotools/blob/ef7bf0ce/src/aiotools/context.py#L152 We may need to update the documentation to recommend explicit aclosing of async generators. -- nosy: +achimnol ___ Python tracker <https://bugs.python.org/issue41229> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue41320] async process closing after event loop closed
Change by Joongi Kim : -- nosy: +achimnol ___ Python tracker <https://bugs.python.org/issue41320> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue30064] BaseSelectorEventLoop.sock_{recv, sendall}() don't remove their callbacks when canceled
Joongi Kim added the comment: And I suspect that this issue is something simliar to what I did in a recent janus PR: https://github.com/aio-libs/janus/blob/ec8592b91254971473b508313fb91b01623f13d7/janus/__init__.py#L84 to give a chance for specific callbacks to execute via an extra context switch. -- ___ Python tracker <https://bugs.python.org/issue30064> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue30064] BaseSelectorEventLoop.sock_{recv, sendall}() don't remove their callbacks when canceled
Joongi Kim added the comment: I just encountered this issue when doing "sys.exit(1)" on a Click-based CLI program that internally uses asyncio event loop via wrapped via a context manager, on Python 3.8.2. Using uvloop or adding "time.sleep(0.1)" before "sys.exit(1)" removes the error. -- nosy: +achimnol ___ Python tracker <https://bugs.python.org/issue30064> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue38599] Deprecate creation of asyncio object when the loop is not running
Joongi Kim added the comment: It is also generating deprecation warning: > /opt/python/3.8.0/lib/python3.8/asyncio/queues.py:48: DeprecationWarning: The > loop argument is deprecated since Python 3.8, and scheduled for removal in > Python 3.10. > self._finished = locks.Event(loop=loop) -- nosy: +achimnol ___ Python tracker <https://bugs.python.org/issue38599> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue33221] Add stats for asyncio task usage.
Joongi Kim <m...@daybreaker.info> added the comment: I like trio-style instrumentation API because it could be used for more generic purposes, not only for statistics. This stats or instrumentation API will greatly help me to utilize external monitoring services such as Datadog in my production deployments. -- ___ Python tracker <rep...@bugs.python.org> <https://bugs.python.org/issue33221> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue33221] Add stats for asyncio task usage.
Change by Joongi Kim <m...@daybreaker.info>: -- nosy: +achimnol ___ Python tracker <rep...@bugs.python.org> <https://bugs.python.org/issue33221> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue33177] make install hangs on macOS when there is an existing Python app
Joongi Kim <m...@daybreaker.info> added the comment: I found that the reason was my Python 3.6.4 installed via the official-installer has the permission of "root:wheel" and pyenv is running in my plain user privilege. Using chown command to change the permissions to "joongi:admin" and retrying worked. What I'm curious now is: why didn't "rm -r /Application/Python\ 3.6" command explicitly display the permission error and just the whole installation process hanged up? -- ___ Python tracker <rep...@bugs.python.org> <https://bugs.python.org/issue33177> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue33177] make install hangs on macOS when there is an existing Python app
New submission from Joongi Kim <m...@daybreaker.info>: I have installed Python 3.6.4 for macOS by downloading from the official site (www.python.org) and then tried installing 3.6.5 using pyenv. Then the installation process hangs here: https://user-images.githubusercontent.com/555156/38078784-57e44462-3378-11e8-8011-9579afc3c811.png There is a 2-years old issue in pyenv (https://github.com/pyenv/pyenv/issues/512) but this may have to be fixed from here. -- components: Installation, macOS messages: 314639 nosy: achimnol, ned.deily, ronaldoussoren priority: normal severity: normal status: open title: make install hangs on macOS when there is an existing Python app type: behavior versions: Python 3.6 ___ Python tracker <rep...@bugs.python.org> <https://bugs.python.org/issue33177> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue32526] Closing async generator while it is running does not raise an exception
Change by Joongi Kim <daybreake...@gmail.com>: -- keywords: +patch pull_requests: +5035 stage: -> patch review ___ Python tracker <rep...@bugs.python.org> <https://bugs.python.org/issue32526> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue32528] Change base class for futures.CancelledError
Joongi Kim <m...@daybreaker.info> added the comment: I strongly agree to have discretion of CancelledError and other general exceptions, though I don't have concrete ideas on good unobtrusive ways to achieve this. If I write my codes carefully I could control most of cancellation explicitly, but it is still hard to control it in 3rd-party libraries that I depend on. Often they just raise random errors, or CancelledError is swallowed. Also it would be nice to have some documentation and examples on how to write "cancellation-friendly" coroutine codes. -- ___ Python tracker <rep...@bugs.python.org> <https://bugs.python.org/issue32528> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue32528] Change base class for futures.CancelledError
Change by Joongi Kim <m...@daybreaker.info>: -- nosy: +achimnol ___ Python tracker <rep...@bugs.python.org> <https://bugs.python.org/issue32528> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue32526] Closing async generator while it is running does not raise an exception
New submission from Joongi Kim <m...@daybreaker.info>: Here is the minimal example code: https://gist.github.com/achimnol/965a6aecf7b1f96207abf11469b68965 Just run this code using "python -m pytest -s test.py" to see what happens. (My setup uses Python 3.6.4 and pytest 3.3.2 on macOS High Sierra 10.13.2) I tried the same logic using synchornous APIs such as threading.Thread / time.sleep instead of loop.create_task / asyncio.sleep, it raised "ValueError: generator already executing" when tried to close the generator. -- ___ Python tracker <rep...@bugs.python.org> <https://bugs.python.org/issue32526> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue32526] Closing async generator while it is running does not raise an exception
Change by Joongi Kim <m...@daybreaker.info>: -- components: asyncio nosy: achimnol, asvetlov, njs, yselivanov priority: normal severity: normal status: open title: Closing async generator while it is running does not raise an exception type: behavior versions: Python 3.6 ___ Python tracker <rep...@bugs.python.org> <https://bugs.python.org/issue32526> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
