[issue26081] Implement asyncio Future in C to improve performance

Changes by Donald Stufft : -- pull_requests: +881 ___ Python tracker ___ ___

[issue26081] Implement asyncio Future in C to improve performance

Changes by Yury Selivanov : -- priority: release blocker -> normal resolution: -> fixed status: open -> closed ___ Python tracker

[issue26081] Implement asyncio Future in C to improve performance

Roundup Robot added the comment: New changeset 345904bd0456 by Yury Selivanov in branch '3.6': Issue #26081: Fix refleak in _asyncio.Future.__iter__().throw. https://hg.python.org/cpython/rev/345904bd0456 New changeset b95aa07d by Yury Selivanov in branch 'default': Merge 3.6 (issue #26081)

[issue26081] Implement asyncio Future in C to improve performance

Yury Selivanov added the comment: This patch introduced multiple refleaks in test_asyncgen. -- priority: normal -> release blocker resolution: fixed -> status: closed -> open ___ Python tracker

[issue26081] Implement asyncio Future in C to improve performance

Yury Selivanov added the comment: Yes, I think it's a good idea. -- ___ Python tracker ___ ___

[issue26081] Implement asyncio Future in C to improve performance

INADA Naoki added the comment: How about changing module name? _asyncio_speedup for example. -- ___ Python tracker ___

[issue26081] Implement asyncio Future in C to improve performance

Yury Selivanov added the comment: I mean another optimization possibility. -- ___ Python tracker ___ ___

[issue26081] Implement asyncio Future in C to improve performance

Yury Selivanov added the comment: Thank you, INADA! Next task -- optimize asyncio.Task in C in 3.7. Another 10-15% performance improvement. -- ___ Python tracker

[issue26081] Implement asyncio Future in C to improve performance

Changes by Berker Peksag : -- stage: patch review -> resolved versions: +Python 3.7 ___ Python tracker ___

[issue26081] Implement asyncio Future in C to improve performance

INADA Naoki added the comment: I close this issue for now. Further improvements can be new issue. -- resolution: -> fixed status: open -> closed ___ Python tracker

[issue26081] Implement asyncio Future in C to improve performance

INADA Naoki added the comment: I've committed the patch with trivial fixes (adding curly braces to if statements). And I'm sorry, I committed with wrong issue number. https://hg.python.org/cpython/rev/678424183b38 (3.6) https://hg.python.org/cpython/rev/f8815001a390 (default) I fixed NEWS

[issue26081] Implement asyncio Future in C to improve performance

Yury Selivanov added the comment: I quickly looked over the patch and I think it's good. If anything we still have time to hunt down any bugs or even revert this before 3.6 final. INADA, feel free to commit it before Monday to 3.6 and default branches. --

[issue26081] Implement asyncio Future in C to improve performance

INADA Naoki added the comment: Now I understand tp_dealloc, tp_finalize and subtype_dealloc. Attached patch passes tests. -- Added file: http://bugs.python.org/file45023/fastfuture4.patch ___ Python tracker

[issue26081] Implement asyncio Future in C to improve performance

INADA Naoki added the comment: Fixed overriding Future._repr_info(). But I failed to implement overridable Future.__del__ in C yet. (FYI, fastfuture2.patch passes tests by mix-in __del__ and __repr__) -- Added file: http://bugs.python.org/file45012/fastfuture3.patch

[issue26081] Implement asyncio Future in C to improve performance

INADA Naoki added the comment: fastfuture3-wip.patch is work in progress implementation of implementing __repr__ and __del__ in C. I post it to avoid duplicated works. Known TODOs: * Support overriding Future._repr_info() * Fix __del__ is not called (Research how tp_del, tp_finalize, and

[issue26081] Implement asyncio Future in C to improve performance

INADA Naoki added the comment: fixed -- Added file: http://bugs.python.org/file44993/fastfuture2.patch ___ Python tracker ___

[issue26081] Implement asyncio Future in C to improve performance

INADA Naoki added the comment: FutureIter_throw is wrong, maybe. Removing FutureIter_send and FutureIter_throw from FutureIter_methods solves the segv and test passed. -- ___ Python tracker

[issue26081] Implement asyncio Future in C to improve performance

Yury Selivanov added the comment: INADA, would you be able to take a look? -- ___ Python tracker ___ ___

[issue26081] Implement asyncio Future in C to improve performance

Yury Selivanov added the comment: The most recent patch segfaults... Will try to debug. -- ___ Python tracker ___

[issue26081] Implement asyncio Future in C to improve performance

Changes by Rémi Cardona : -- nosy: +RemiCardona ___ Python tracker ___ ___ Python-bugs-list

[issue26081] Implement asyncio Future in C to improve performance

INADA Naoki added the comment: This is the patch. And git branch is here https://github.com/methane/cpython/pull/5 -- Added file: http://bugs.python.org/file44670/fastfuture.patch ___ Python tracker

[issue26081] Implement asyncio Future in C to improve performance

Yury Selivanov added the comment: > Implementing __del__ and __repr__ in C is bit hard task to me. > I can't do it in this week. (maybe I can't do it in this month too.) NP. I'll take a look myself after you upload the next iteration of the patch... --

[issue26081] Implement asyncio Future in C to improve performance

INADA Naoki added the comment: I'm working on fixing points you commented. Wait a minute. Implementing __del__ and __repr__ in C is bit hard task to me. I can't do it in this week. (maybe I can't do it in this month too.) On Thu, Sep 15, 2016 at 7:37 AM, Yury Selivanov

[issue26081] Implement asyncio Future in C to improve performance

Yury Selivanov added the comment: INADA, would you be able to address my last review comments? Also, I'm wondering what if we could implement __del__ and __repr__ in C too, so that we could drop BaseFuture class? -- ___ Python tracker

[issue26081] Implement asyncio Future in C to improve performance

Ned Deily added the comment: This change touches a lot of files and affect both the unix* and Windows build processes so, yeah, I think it's too risky to go in to b1. Let's get it in as soon as possible after b1. -- ___ Python tracker

[issue26081] Implement asyncio Future in C to improve performance

Guido van Rossum added the comment: Yeah, let's do this in 3.6b2. -- ___ Python tracker ___ ___

[issue26081] Implement asyncio Future in C to improve performance

Yury Selivanov added the comment: > Yury: What do you think of the code? How solid is it? (The issue I found was > due to my own very recent changes to _blocking.) The code looks fine, I can fix the remaining nits myself. I've left a couple of comments in review. > Ned: Is it better to do

[issue26081] Implement asyncio Future in C to improve performance

Guido van Rossum added the comment: Yury: What do you think of the code? How solid is it? (The issue I found was due to my own very recent changes to _blocking.) Ned: Is it better to do this in 3.6b1 or to wait for 3.6b2? -- nosy: +ned.deily ___

[issue26081] Implement asyncio Future in C to improve performance

INADA Naoki added the comment: Sorry, again. fixed. Now this passes `./python -m test.test_asyncio` -- Added file: http://bugs.python.org/file44547/fastfuture.patch ___ Python tracker

[issue26081] Implement asyncio Future in C to improve performance

Guido van Rossum added the comment: Thanks! I can't review the whole thing, but I patched it in and tried running the asyncio/examples/crawl.py example, like so: $ ~/src/cpython36/python.exe examples/crawl.py xkcd.com -q Exception RuntimeError('yield was used instead of yield from in task >

[issue26081] Implement asyncio Future in C to improve performance

INADA Naoki added the comment: Oh, I'm sorry. I usually working on git, and convert git diff to hg diff when posting patch. I've used `patch -p1` instead of `hg import --no-edit` to apply git patch into hg workdir. I wonder if Rietveld accepts git diff format... -- Added file:

[issue26081] Implement asyncio Future in C to improve performance

Guido van Rossum added the comment: The actual _futures module appears missing from your latest patch -- what's up with that? -- ___ Python tracker ___

[issue26081] Implement asyncio Future in C to improve performance

Changes by INADA Naoki : Added file: http://bugs.python.org/file44531/fastfuture.patch ___ Python tracker ___

[issue26081] Implement asyncio Future in C to improve performance

INADA Naoki added the comment: There are only two weeks until 3.6 beta. Yury, could you review this again? Or should I implement freelist before review? Implementing freelist may be easy, but measuring the effect of freelist from realistic application is not easy. --

[issue26081] Implement asyncio Future in C to improve performance

INADA Naoki added the comment: Implemented FutureIter -- Added file: http://bugs.python.org/file44081/futures.patch ___ Python tracker ___

[issue26081] Implement asyncio Future in C to improve performance

Yury Selivanov added the comment: > Implementing full behavior of generator seems difficult to me. I'll implement minimum implementation in next patch. Sure, but you have to implement send() and throw(). -- ___ Python tracker

[issue26081] Implement asyncio Future in C to improve performance

INADA Naoki added the comment: > Left a couple of comments; the important one -- Future.__await__ (and > Future.__iter__) should always return a *new* instance of a generator-like > object (tied to the Future object). Implementing full behavior of generator seems difficult to me. I'll

[issue26081] Implement asyncio Future in C to improve performance

Yury Selivanov added the comment: I'd also think about implementing asyncio.Handle in C (with a freelist). -- ___ Python tracker ___

[issue26081] Implement asyncio Future in C to improve performance

Yury Selivanov added the comment: > What parts of Future are performance critical? Maybe it is worth to implement > in C only the most critical code. Basically everything. Contrary to @contextmanager, Futures are the building blocks of asyncio, so instantiation + awaiting on them + setting

[issue26081] Implement asyncio Future in C to improve performance

Serhiy Storchaka added the comment: See also a discussion on Python-Dev about rewriting contextlib.contextmanager in C: https://mail.python.org/pipermail/python-dev/2016-August/thread.html#145786 . What parts of Future are performance critical? Maybe it is worth to implement in C only the

[issue26081] Implement asyncio Future in C to improve performance

Yury Selivanov added the comment: > Yury, could you review this before 3.6a4? Left a couple of comments; the important one -- Future.__await__ (and Future.__iter__) should always return a *new* instance of a generator-like object (tied to the Future object). --

[issue26081] Implement asyncio Future in C to improve performance

Changes by Berker Peksag : -- stage: needs patch -> patch review versions: -Python 3.5 ___ Python tracker ___

[issue26081] Implement asyncio Future in C to improve performance

INADA Naoki added the comment: Yury, could you review this before 3.6a4? -- ___ Python tracker ___ ___

[issue26081] Implement asyncio Future in C to improve performance

Yury Selivanov added the comment: Yes. Most people will use vanilla asyncio anyways. -- ___ Python tracker ___

[issue26081] Implement asyncio Future in C to improve performance

Marco Paolini added the comment: THe guys developing uvloop say their implementation is already quite fast [1]. Is it worth integrating it? [1] https://github.com/MagicStack/uvloop -- nosy: +mpaolini ___ Python tracker

[issue26081] Implement asyncio Future in C to improve performance

INADA Naoki added the comment: > asyncio uses loop.create_future() to create sockets. I'd suggest you to > create two base test classes: one that monkeypatches loop.create_future to > return pure python Future in its setUp method; an another, that makes > create_future to return a C version

[issue26081] Implement asyncio Future in C to improve performance

INADA Naoki added the comment: Thanks. I'll working on test_asyncio in next few days. -- ___ Python tracker ___

[issue26081] Implement asyncio Future in C to improve performance

Yury Selivanov added the comment: > Before working on it, could someone give me idea to run whole test_asyncio with and without C version Future easily? asyncio uses loop.create_future() to create sockets. I'd suggest you to create two base test classes: one that monkeypatches

[issue26081] Implement asyncio Future in C to improve performance

INADA Naoki added the comment: In my patch, test_asyncio runs against C version Future. I saw how test_json tests against C version and pure Python version. But test_asyncio is more larger than test_json. Before working on it, could someone give me idea to run whole test_asyncio with and

[issue26081] Implement asyncio Future in C to improve performance

INADA Naoki added the comment: OK. Here is current version. -- Added file: http://bugs.python.org/file43670/futures.patch ___ Python tracker ___

[issue26081] Implement asyncio Future in C to improve performance

Yury Selivanov added the comment: > I'm working on this. Some bugs are fixed, but doesn't pass tests for now. Thanks a lot! I couldn't find time to finish this myself. I can definitely help you and review the patch once it's ready. > Yury, could you explain what the comment "This isn't a

[issue26081] Implement asyncio Future in C to improve performance

INADA Naoki added the comment: Should I send pull request to github.com/python/asyncio? Or should I post patch here? -- ___ Python tracker ___

[issue26081] Implement asyncio Future in C to improve performance

INADA Naoki added the comment: Passing all tests now. Yury, could you explain what the comment "This isn't a Future class; it's a BaseFuture" means? Should it be "_futures.Future" or "_futures.BaseFuture"? -- ___ Python tracker

[issue26081] Implement asyncio Future in C to improve performance

INADA Naoki added the comment: I'm working on this. Some bugs are fixed, but doesn't pass tests for now. https://github.com/methane/cpython/pull/5 -- nosy: +methane ___ Python tracker

[issue26081] Implement asyncio Future in C to improve performance

Guido van Rossum added the comment: We should really try to get this into 3.6. --Guido (mobile) -- ___ Python tracker ___

[issue26081] Implement asyncio Future in C to improve performance

Changes by Giampaolo Rodola' : -- nosy: +giampaolo.rodola ___ Python tracker ___ ___

[issue26081] Implement asyncio Future in C to improve performance

New submission from Yury Selivanov: Some info on this: https://github.com/python/asyncio/issues/282#issuecomment-155957235 Long story short, Future implemented in C can speedup some asyncio code up to 25%. I'm attaching a patch with a WIP implementation. There are some failing assertions