[issue29861] multiprocessing Pool keeps objects (tasks, args, results) alive too long

2017-05-05 Thread STINNER Victor 

STINNER Victor added the comment:


New changeset fd6094cdebb5736745d164e0207de2d4cb0b50dc by Victor Stinner in 
branch '2.7':
Relax test timing (bpo-29861) to avoid sporadic failures (#1120) (#1472)
https://github.com/python/cpython/commit/fd6094cdebb5736745d164e0207de2d4cb0b50dc


--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue29861] multiprocessing Pool keeps objects (tasks, args, results) alive too long

2017-05-05 Thread STINNER Victor 

STINNER Victor added the comment:

> New changeset 685cdb9acc3fca04a9897d88b89771ddfd50e772 by Antoine Pitrou in 
> branch 'master':
> Relax test timing (bpo-29861) to avoid sporadic failures (#1120)

Oh, this change wasn't backported to 2.7 and caused the bpo-30269. I proposed a 
backport: https://github.com/python/cpython/pull/1472 I will merge it once 
tests pass ;-)

--
nosy: +haypo

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue29861] multiprocessing Pool keeps objects (tasks, args, results) alive too long

2017-05-05 Thread STINNER Victor 

Changes by STINNER Victor :


--
pull_requests: +1571

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue29861] multiprocessing Pool keeps objects (tasks, args, results) alive too long

2017-04-14 Thread Antoine Pitrou 

Antoine Pitrou added the comment:


New changeset 47f24a018f6a9c3ed996d2b83a7ebd0d1f905827 by Antoine Pitrou in 
branch '3.5':
Relax test timing (bpo-29861) to avoid sporadic failures (#1120) (#1133)
https://github.com/python/cpython/commit/47f24a018f6a9c3ed996d2b83a7ebd0d1f905827


--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue29861] multiprocessing Pool keeps objects (tasks, args, results) alive too long

2017-04-14 Thread Antoine Pitrou 

Antoine Pitrou added the comment:


New changeset 413a8913aee255d0635e4ddbb9c343d9e5e76fea by Antoine Pitrou in 
branch '3.6':
Relax test timing (bpo-29861) to avoid sporadic failures (#1120) (#1132)
https://github.com/python/cpython/commit/413a8913aee255d0635e4ddbb9c343d9e5e76fea


--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue29861] multiprocessing Pool keeps objects (tasks, args, results) alive too long

2017-04-14 Thread Antoine Pitrou 

Changes by Antoine Pitrou :


--
pull_requests: +1267

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue29861] multiprocessing Pool keeps objects (tasks, args, results) alive too long

2017-04-14 Thread Antoine Pitrou 

Changes by Antoine Pitrou :


--
pull_requests: +1266

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue29861] multiprocessing Pool keeps objects (tasks, args, results) alive too long

2017-04-14 Thread Antoine Pitrou 

Antoine Pitrou added the comment:


New changeset 685cdb9acc3fca04a9897d88b89771ddfd50e772 by Antoine Pitrou in 
branch 'master':
Relax test timing (bpo-29861) to avoid sporadic failures (#1120)
https://github.com/python/cpython/commit/685cdb9acc3fca04a9897d88b89771ddfd50e772


--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue29861] multiprocessing Pool keeps objects (tasks, args, results) alive too long

2017-04-13 Thread Antoine Pitrou 

Changes by Antoine Pitrou :


--
pull_requests: +1258

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue29861] multiprocessing Pool keeps objects (tasks, args, results) alive too long

2017-04-13 Thread Antoine Pitrou 

Antoine Pitrou added the comment:

Ok, I can reproduce now.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue29861] multiprocessing Pool keeps objects (tasks, args, results) alive too long

2017-04-13 Thread Antoine Pitrou 

Antoine Pitrou added the comment:

I can't reproduce here, on Ubuntu 16.04, after running the test 500 times.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue29861] multiprocessing Pool keeps objects (tasks, args, results) alive too long

2017-04-12 Thread Xiang Zhang 

Xiang Zhang added the comment:

Hi, Antoine, after this change, I sometimes see tests fail for 3.5 branch, for 
example 
http://buildbot.python.org/all/builders/x86%20Ubuntu%20Shared%203.5/builds/194/steps/test/logs/stdio.

--
nosy: +xiang.zhang

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue29861] multiprocessing Pool keeps objects (tasks, args, results) alive too long

2017-03-24 Thread Antoine Pitrou 

Changes by Antoine Pitrou :


--
resolution:  -> fixed
stage: needs patch -> resolved
status: open -> closed

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue29861] multiprocessing Pool keeps objects (tasks, args, results) alive too long

2017-03-24 Thread Antoine Pitrou 

Antoine Pitrou added the comment:


New changeset 5084ff7ddfe68969d95af12075016f309718aea8 by Antoine Pitrou in 
branch '2.7':
bpo-29861: release references to multiprocessing Pool tasks (#743) (#803)
https://github.com/python/cpython/commit/5084ff7ddfe68969d95af12075016f309718aea8


--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue29861] multiprocessing Pool keeps objects (tasks, args, results) alive too long

2017-03-24 Thread Antoine Pitrou 

Changes by Antoine Pitrou :


--
pull_requests: +708

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue29861] multiprocessing Pool keeps objects (tasks, args, results) alive too long

2017-03-24 Thread Antoine Pitrou 

Antoine Pitrou added the comment:


New changeset 80cb6ed4db9bae09de1e9ad6001d11cb45a4c1cc by Antoine Pitrou in 
branch '3.5':
bpo-29861: release references to multiprocessing Pool tasks (#743) (#801)
https://github.com/python/cpython/commit/80cb6ed4db9bae09de1e9ad6001d11cb45a4c1cc


--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue29861] multiprocessing Pool keeps objects (tasks, args, results) alive too long

2017-03-24 Thread Antoine Pitrou 

Antoine Pitrou added the comment:


New changeset cc3331fec8b7a61c3f06c097eac85bfa38490758 by Antoine Pitrou in 
branch '3.6':
bpo-29861: release references to multiprocessing Pool tasks (#743) (#800)
https://github.com/python/cpython/commit/cc3331fec8b7a61c3f06c097eac85bfa38490758


--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue29861] multiprocessing Pool keeps objects (tasks, args, results) alive too long

2017-03-24 Thread Antoine Pitrou 

Changes by Antoine Pitrou :


--
pull_requests: +706

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue29861] multiprocessing Pool keeps objects (tasks, args, results) alive too long

2017-03-24 Thread Antoine Pitrou 

Changes by Antoine Pitrou :


--
pull_requests: +705

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue29861] multiprocessing Pool keeps objects (tasks, args, results) alive too long

2017-03-24 Thread Antoine Pitrou 

Antoine Pitrou added the comment:


New changeset 8988945cdc27ffa86ba8c624e095b51c459f5154 by Antoine Pitrou in 
branch 'master':
bpo-29861: release references to multiprocessing Pool tasks (#743)
https://github.com/python/cpython/commit/8988945cdc27ffa86ba8c624e095b51c459f5154


--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue29861] multiprocessing Pool keeps objects (tasks, args, results) alive too long

2017-03-20 Thread Antoine Pitrou 

Changes by Antoine Pitrou :


--
pull_requests: +656

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue29861] multiprocessing Pool keeps objects (tasks, args, results) alive too long

2017-03-20 Thread Antoine Pitrou 

Antoine Pitrou added the comment:

Quick patch below.  I'll make a PR once I have time to :-)

diff --git a/Lib/multiprocessing/pool.py b/Lib/multiprocessing/pool.py
index ffdf426..945afa2 100644
--- a/Lib/multiprocessing/pool.py
+++ b/Lib/multiprocessing/pool.py
@@ -128,6 +128,8 @@ def worker(inqueue, outqueue, initializer=None, 
initargs=(), maxtasks=None,
 util.debug("Possible encoding error while sending result: %s" % (
 wrapped))
 put((job, i, (False, wrapped)))
+
+task = job = result = func = args = kwds = None
 completed += 1
 util.debug('worker exiting after %d tasks' % completed)
 
@@ -402,6 +404,8 @@ class Pool(object):
 if set_length:
 util.debug('doing set_length()')
 set_length(i+1)
+finally:
+task = taskseq = job = None
 else:
 util.debug('task handler got sentinel')
 
@@ -445,6 +449,7 @@ class Pool(object):
 cache[job]._set(i, obj)
 except KeyError:
 pass
+task = job = obj = None
 
 while cache and thread._state != TERMINATE:
 try:
@@ -461,6 +466,7 @@ class Pool(object):
 cache[job]._set(i, obj)
 except KeyError:
 pass
+task = job = obj = None
 
 if hasattr(outqueue, '_reader'):
 util.debug('ensuring that outqueue is not full')

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue29861] multiprocessing Pool keeps objects (tasks, args, results) alive too long

2017-03-20 Thread Antoine Pitrou 

New submission from Antoine Pitrou:

The various workers in multiprocessing.Pool keep a reference to the last 
encountered task or task result.  This means some data may be kept alive even 
after the caller is done with them, as long as some other task doesn't clobber 
the relevant variables.

Specifically, Pool._handle_tasks(), Pool._handle_results() and the toplevel 
worker() function fail to clear references at the end of each loop.

Originally reported at https://github.com/dask/distributed/issues/956

--
components: Library (Lib)
messages: 289894
nosy: davin, pitrou, sbt
priority: normal
severity: normal
stage: needs patch
status: open
title: multiprocessing Pool keeps objects (tasks, args, results) alive too long
type: resource usage
versions: Python 2.7, Python 3.5, Python 3.6, Python 3.7

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com