[issue31620] asyncio.Queue leaks memory if the queue is empty and consumers poll it frequently

[Andrew Svetlov]

Andrew Svetlov  added the comment:


New changeset c62f0cb3b1f6f9ca4ce463b1c99b0543bdfa38d6 by Andrew Svetlov (Suren 
Nihalani) in branch 'master':
bpo-31620: have asyncio/queues not leak memory when you've exceptions during 
waiting (#3813)
https://github.com/python/cpython/commit/c62f0cb3b1f6f9ca4ce463b1c99b0543bdfa38d6


--
nosy: +asvetlov

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue31620] asyncio.Queue leaks memory if the queue is empty and consumers poll it frequently

[Roundup Robot]

Change by Roundup Robot :


--
pull_requests: +4283

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue31620] asyncio.Queue leaks memory if the queue is empty and consumers poll it frequently

[Andrew Svetlov]

Andrew Svetlov  added the comment:


New changeset ac4f6d4448fb6f9affb817bafb8357450fe43349 by Andrew Svetlov (Miss 
Islington (bot)) in branch '3.6':
bpo-31620: have asyncio/queues not leak memory when you've exceptions during 
waiting (GH-3813) (#4326)
https://github.com/python/cpython/commit/ac4f6d4448fb6f9affb817bafb8357450fe43349


--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue31620] asyncio.Queue leaks memory if the queue is empty and consumers poll it frequently

[Andrew Svetlov]

Andrew Svetlov  added the comment:

Sorry for late closing the issue, PR was merged a month ago.

--
resolution:  -> fixed
stage: patch review -> resolved
status: open -> closed
versions: +Python 3.7

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue31620] asyncio.Queue leaks memory if the queue is empty and consumers poll it frequently

[Zack Elan]

New submission from Zack Elan :

Repro: Call asyncio.wait_for(some_queue.get(), some_timeout) repeatedly, with 
no items in the queue, so that the call times out each time.

Expected: No increase in memory while polling an empty queue

Actual: The queue holds on to pending "getter" futures until a item passes 
through the queue, which clears the pending tasks out.

Use case:

I have producer and consumer asyncio.Tasks, linked by an asyncio.Queue. The 
producer is idle most of the time and pushes messages very infrequently. The 
consumer task polls the queue with wait_for and a timeout so that it's able to 
record "yep, I'm still idle" when the wait_for times out.

Attached script has a minimal repro. The producer emits an item every 60 
seconds and the consumer polls every second. At the 59th second the _getters 
member of the queue is holding on to 59 pending futures. By varying the 
intervals this can leak an arbitrary amount of memory, for example producer 
emitting an item once a day vs. consumer polling once a second will leak 86,400 
futures.

>From the attached script:

2017-09-28 10:09:12,699 Queue  is idle
...
2017-09-28 10:10:09,784 Queue  is idle
2017-09-28 10:10:10,785 Queue  is idle
2017-09-28 10:10:11,699 Received 0 from 
2017-09-28 10:10:12,700 Queue  is idle
2017-09-28 10:10:13,702 Queue  is idle

Noticed this in 3.6.1, though based on no changes to 
https://github.com/python/cpython/blob/master/Lib/asyncio/queues.py between the 
3.6 branch and master I suspect it also affects 3.6.x and 3.7.

--
components: asyncio
files: test.py
messages: 303264
nosy: yselivanov, zackelan
priority: normal
severity: normal
status: open
title: asyncio.Queue leaks memory if the queue is empty and consumers poll it 
frequently
type: resource usage
versions: Python 3.6
Added file: https://bugs.python.org/file47174/test.py

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue31620] asyncio.Queue leaks memory if the queue is empty and consumers poll it frequently

[Surenkumar Nihalani]

Surenkumar Nihalani  added the comment:

I am working on fixing this.

--
nosy: +snihalani

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue31620] asyncio.Queue leaks memory if the queue is empty and consumers poll it frequently

[Surenkumar Nihalani]

Change by Surenkumar Nihalani :


--
keywords: +patch
pull_requests: +3797
stage:  -> patch review

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue31620] asyncio.Queue leaks memory if the queue is empty and consumers poll it frequently

[Caleb Hattingh]

Caleb Hattingh  added the comment:

This looks like a dupe, or at least quite closely related to 
https://bugs.python.org/issue26259. If the PR resolves both issues that one 
should be closed too.

--
nosy: +cjrh

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com