[issue22448] call_at/call_later with Timer cancellation can result in (practically) unbounded memory usage.

STINNER Victor added the comment: > Please commit your change to the tulip repo too. Oh sorry. In fact, I made the commit but I forgot to push my change :-p -- ___ Python tracker __

[issue22448] call_at/call_later with Timer cancellation can result in (practically) unbounded memory usage.

Changes by Yury Selivanov : -- resolution: -> fixed status: open -> closed ___ Python tracker ___ ___ Python-bugs-list mailing list U

[issue22448] call_at/call_later with Timer cancellation can result in (practically) unbounded memory usage.

Yury Selivanov added the comment: Eh, I knew something was wrong. Thanks. NUMBER_OF_TASKS 10 ITERATIONS -> 2000 out of 2000 2 loops: 0.045037232999675325 1 loop: 0.045182990999819594 TOTAL_BENCH_TIME 91.36706805229187 Please commit your change to the tulip repo too. --

[issue22448] call_at/call_later with Timer cancellation can result in (practically) unbounded memory usage.

STINNER Victor added the comment: Hum you don't reset start between the two benchmarks. -- ___ Python tracker ___ ___ Python-bugs-list

[issue22448] call_at/call_later with Timer cancellation can result in (practically) unbounded memory usage.

Yury Selivanov added the comment: typo: > 2 loops is always about 30-40% slower. 2 loops is always about 30-40% faster. -- ___ Python tracker ___ __

[issue22448] call_at/call_later with Timer cancellation can result in (practically) unbounded memory usage.

Yury Selivanov added the comment: Victor, Here's an updated benchmark results: NUMBER_OF_TASKS 1 ITERATIONS -> 2000 out of 2000 2 loops: 0.004267875499863294 1 loop: 0.007916624497738667 TOTAL_BENCH_TIME 15.975227117538452 NUMBER_OF_TASKS 10 ITERATIONS ->

[issue22448] call_at/call_later with Timer cancellation can result in (practically) unbounded memory usage.

Roundup Robot added the comment: New changeset b85ed8bb7523 by Victor Stinner in branch '3.4': Issue #22448: asyncio, cleanup _run_once(), only iterate once to remove delayed https://hg.python.org/cpython/rev/b85ed8bb7523 New changeset 8e9df3414185 by Victor Stinner in branch 'default': (Merge 3

[issue22448] call_at/call_later with Timer cancellation can result in (practically) unbounded memory usage.

STINNER Victor added the comment: "... on average faster 2-3% ... slower for 2-3%" 3% is meaningless. On a microbenchmark, you can say that it's faster or slower if the difference is at least 10%. -- ___ Python tracker

[issue22448] call_at/call_later with Timer cancellation can result in (practically) unbounded memory usage.

Yury Selivanov added the comment: Victor, I've done some additional testing. Here's a test that Joshua wrote for the code review: https://gist.github.com/1st1/b38ac6785cb01a679722 It appears that single loop approach works a bit faster for smaller collections of tasks. On a list of 1 task

[issue22448] call_at/call_later with Timer cancellation can result in (practically) unbounded memory usage.

Yury Selivanov added the comment: > IMO it makes the code simpler and easier to understand. But it's a tad slower, like 2-3% ;) You can test it yourself, we only tested it on huge tasks list of 1M items. FWIW, I'm not opposed to your patch. -- ___

[issue22448] call_at/call_later with Timer cancellation can result in (practically) unbounded memory usage.

Yury Selivanov added the comment: Victor, During the code review we tried the single loop approach. At the end Joshua wrote a small benchmark to test if it's really faster to do it in one loop or not. Turned out that single loop approach is not faster than loop+comprehension (but it's not muc

[issue22448] call_at/call_later with Timer cancellation can result in (practically) unbounded memory usage.

STINNER Victor added the comment: > Turned out that single loop approach is not faster than loop+comprehension IMO it makes the code simpler and easier to understand. -- ___ Python tracker

[issue22448] call_at/call_later with Timer cancellation can result in (practically) unbounded memory usage.

STINNER Victor added the comment: Sorry, I'm coming later after the review and the commit, but I worry about performances of _run_once() since it's the core of asyncio. Yury proposed to only iterate once on self._scheduled when removing delayed calls, and I have the same concern. Here is a pa

[issue22448] call_at/call_later with Timer cancellation can result in (practically) unbounded memory usage.

Yury Selivanov added the comment: Hm, strange, usually roundup robot closes issues. Anyways, closed now. Thanks again, Joshua. -- resolution: -> fixed status: open -> closed ___ Python tracker ___

[issue22448] call_at/call_later with Timer cancellation can result in (practically) unbounded memory usage.

Joshua Moore-Oliva added the comment: Also - should I close this issue now that a patch has been committed? -- ___ Python tracker ___

[issue22448] call_at/call_later with Timer cancellation can result in (practically) unbounded memory usage.

Joshua Moore-Oliva added the comment: > Hm. That sounds like you won't actually be interoperable with other > asyncio-using code. asyncio code can be interoperated with by spinning off an asyncio coroutine that on completion calls a callback that reschedules a non-asyncio coroutine. I assume

[issue22448] call_at/call_later with Timer cancellation can result in (practically) unbounded memory usage.

Roundup Robot added the comment: New changeset 2a868c9f8f15 by Yury Selivanov in branch '3.4': asyncio: Improve canceled timer handles cleanup. Closes issue #22448. https://hg.python.org/cpython/rev/2a868c9f8f15 New changeset a6aaacb2b807 by Yury Selivanov in branch 'default': asyncio: Improve c

[issue22448] call_at/call_later with Timer cancellation can result in (practically) unbounded memory usage.

Guido van Rossum added the comment: On Sat, Sep 20, 2014 at 3:38 PM, Joshua Moore-Oliva wrote: > > Joshua Moore-Oliva added the comment: > > > I will try to review later tonight. > > Thanks! > > > That makes sense when using gevent, but not when using asyncio or > Trollius. Nothing will make ev

[issue22448] call_at/call_later with Timer cancellation can result in (practically) unbounded memory usage.

Joshua Moore-Oliva added the comment: > I will try to review later tonight. Thanks! > That makes sense when using gevent, but not when using asyncio or Trollius. > Nothing will make events run if you don't use yield [from]. Yes, I am aware of that. I have written a small custom library using

[issue22448] call_at/call_later with Timer cancellation can result in (practically) unbounded memory usage.

Guido van Rossum added the comment: I will try to review later tonight. One thing though: > I tend to write my code with the thought that any non standard library function can yield That makes sense when using geven, but not when using asyncio or Trollius. Nothing will make events run if you d

[issue22448] call_at/call_later with Timer cancellation can result in (practically) unbounded memory usage.

Joshua Moore-Oliva added the comment: My patch is ready for review, if I followed the process correctly I think you should have received an email https://codereview.appspot.com/145220043 > By the way I just looked at wait_for.py; it has a bug where do_work() isn't > using yield-from with the

[issue22448] call_at/call_later with Timer cancellation can result in (practically) unbounded memory usage.

Guido van Rossum added the comment: By the way I just looked at wait_for.py; it has a bug where do_work() isn't using yield-from with the sleep() call. But that may well be the issue you were trying to debug, and this does not change my opinion about the issue -- I am still looking forward to

[issue22448] call_at/call_later with Timer cancellation can result in (practically) unbounded memory usage.

Guido van Rossum added the comment: We can merge the changes into 3.4 and 3.5 for you, it's just a simple copy (the codebases are identical). However, the 3.4.2 release candidate is apparently in 2 days, so I think you've missed that train already. On Sat, Sep 20, 2014 at 9:02 AM, Joshua Moore-O

[issue22448] call_at/call_later with Timer cancellation can result in (practically) unbounded memory usage.

Joshua Moore-Oliva added the comment: > You can contribute upstream to the Tulip project first. Will I be writing a patch and tests for tulip, and then separate a patch and tests for python 3.4? Or will I submit to tulip, and then the changes will get merged from tulip into python by some oth

[issue22448] call_at/call_later with Timer cancellation can result in (practically) unbounded memory usage.

Guido van Rossum added the comment: Hi Joshua, This is indeed a problem -- I just never expected that you would be having that many events and and canceling the majority. I am sorry you had to debug this. :-( This was anticipated by the author of pyftpdlib (Giampaolo Roloda'), who proposed

[issue22448] call_at/call_later with Timer cancellation can result in (practically) unbounded memory usage.

New submission from Joshua Moore-Oliva: The core issue stems from the implementation of Timer cancellation. (which features like asyncio.wait_for build upon). BaseEventLoop stores scheduled events in an array backed heapq named _scheduled. Once an event has been scheduled with call_at, cancel