[issue995907] memory leak with threads and enhancement of the timer class
Charles-François Natali added the comment: OK, I just created #17956 for ScheduledExecutor, closing this one. -- resolution: - duplicate stage: test needed - committed/rejected status: open - closed superseder: - add ScheduledExecutor ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue995907 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue995907] memory leak with threads and enhancement of the timer class
Charles-François Natali added the comment:
I'm attaching a proof of concept code for a ScheduledExecutor
interface, and a ScheduledThreadPoolExecutor implementation
(unfortunately I can't upload it as a mercurial diff for now).
Here's what the API looks like:
from concurrent.futures import ScheduledThreadPoolExecutor
import time
def say(text):
print({}: {}.format(time.ctime(), text))
with ScheduledThreadPoolExecutor(5) as p:
p.schedule(1, say, 'hello 1')
f = p.schedule_fixed_rate(0, 2, say, 'hello 2')
p.schedule_fixed_delay(0, 3, say, 'hello 3')
time.sleep(6)
say(cancelling: %s % f)
f.cancel()
time.sleep(10)
say(shutting down)
schedule() is for one-shot, schedule_fixed_rate() for fixed rate
scheduling (i.e. there will be no drift due to the task execution
time), and schedule_fixed_delay() is for fixed delay (i.e. there will
always be a fixed amount of time between two invokations).
Random notes:
- the scheduling is handled by a new SchedQueue in the queue module:
sched would have been useful, but actually it can't be used here: it
stops as soon as the queue is empty, when it calls the wait function
it won't wake up if a new task is enqueued, etc. Also, I guess such a
queue could be useful in general.
- I had to create a DelayedFuture subclass, which is returned by
schedule_XXX methods. The main differences with raw Future are that it
has a scheduled time and period attributes, and supports
reinitialization (a future can only be run once). It can be cancelled,
and also supports result/exception retrieval.
- I don't know if a process-based counterpart
(ScheduledProcessPoolExecutor) is really useful. I didn't look at it
for now.
--
Added file: http://bugs.python.org/file30199/scheduled.diff
Added file: http://bugs.python.org/file30200/test.py
___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue995907
___diff -ur cpython.orig/Lib/concurrent/futures/_base.py
cpython-b3e1be1493a5/Lib/concurrent/futures/_base.py
--- cpython.orig/Lib/concurrent/futures/_base.py2013-05-07
08:21:21.0 +
+++ cpython-b3e1be1493a5/Lib/concurrent/futures/_base.py2013-05-10
16:35:16.0 +
@@ -6,7 +6,10 @@
import collections
import logging
import threading
-import time
+try:
+from time import monotonic as time
+except ImportError:
+from time import time
FIRST_COMPLETED = 'FIRST_COMPLETED'
FIRST_EXCEPTION = 'FIRST_EXCEPTION'
@@ -188,7 +191,7 @@
before the given timeout.
if timeout is not None:
-end_time = timeout + time.time()
+end_time = timeout + time()
with _AcquireFutures(fs):
finished = set(
@@ -204,7 +207,7 @@
if timeout is None:
wait_timeout = None
else:
-wait_timeout = end_time - time.time()
+wait_timeout = end_time - time()
if wait_timeout 0:
raise TimeoutError(
'%d (of %d) futures unfinished' % (
@@ -390,11 +393,11 @@
elif self._state == FINISHED:
return self.__get_result()
-self._condition.wait(timeout)
+gotit = self._condition.wait(timeout)
if self._state in [CANCELLED, CANCELLED_AND_NOTIFIED]:
raise CancelledError()
-elif self._state == FINISHED:
+elif gotit:
return self.__get_result()
else:
raise TimeoutError()
@@ -423,11 +426,11 @@
elif self._state == FINISHED:
return self._exception
-self._condition.wait(timeout)
+gotit = self._condition.wait(timeout)
if self._state in [CANCELLED, CANCELLED_AND_NOTIFIED]:
raise CancelledError()
-elif self._state == FINISHED:
+elif gotit:
return self._exception
else:
raise TimeoutError()
@@ -499,6 +502,39 @@
self._condition.notify_all()
self._invoke_callbacks()
+
+class DelayedFuture(Future):
+A future whose execution can be delayed, and periodic.
+
+def __init__(self, sched_time, period=0, delay=0):
+super().__init__()
+self._sched_time = sched_time
+if period 0:
+# step 0 = fixed rate
+self._step = period
+elif delay 0:
+# step 0 = fixed delay
+self._step = -delay
+else:
+# step == 0 = one-shot
+self._step = 0
+
+def is_periodic(self):
+return self._step != 0
+
+def get_sched_time(self):
+return self._sched_time
+
+def rearm(self):
+Re-arm the future, and update its scheduled execution time.
+with self._condition:
+if self._step 0:
+self._sched_time += self._step
+
[issue995907] memory leak with threads and enhancement of the timer class
Charles-François Natali added the comment: IMO, this shouldn't be implemented atop thread, but ought to be a regular thread pool: this way, you won't get behind if some task takes too long to execute, the thread pool can start new threads as needed, and we get the general work submit/cancel (through future) for free. Also, it would probably deserve a new interface in concurrent.futures, as ScheduledExecutor, with new schedule(delay, fn, *args, **kwargs) and schedule_periodic(delay, fn, *args, **kwargs) for one-shot and periodic calls. It would be much more consistant than an ad-hoc implementation in the threading module. -- nosy: +neologix ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue995907 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue995907] memory leak with threads and enhancement of the timer class
R. David Murray added the comment: I take your point; I knew there was something bothering me about how the tasks were handled but I didn't consciously see the bug. I like the idea of a ScheduledExecutor. Yael, thanks a lot for working through this, but I think we should probably close this issue and open a new one for adding a ScheduledExecutor to concurrent.futures, and make see-also link from the Timer class to it. Would you be interested in working on it? -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue995907 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue995907] memory leak with threads and enhancement of the timer class
R. David Murray added the comment: Review comments added. -- nosy: +r.david.murray ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue995907 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue995907] memory leak with threads and enhancement of the timer class
Yael added the comment: Can you please review the patch? thanks! -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue995907 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue995907] memory leak with threads and enhancement of the timer class
Yael added the comment: Added a class Threading.TimerPool. This new class spawns one thread, and that thread is running as long as there are active timers. -- keywords: +patch Added file: http://bugs.python.org/file29853/mywork.patch ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue995907 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue995907] memory leak with threads and enhancement of the timer class
Yael added the comment: I am working on a patch for a new class that uses a single background thread, it should be ready soon. One unintended consequence of this change is that with one thread, multiple timers that have the same timeout will no longer run in parallel, but one after the other. -- nosy: +yael ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue995907 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue995907] memory leak with threads and enhancement of the timer class
Changes by Mark Lawrence breamore...@yahoo.co.uk: -- versions: +Python 3.2 -Python 2.7 ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue995907 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue995907] memory leak with threads and enhancement of the timer class
Daniel Diniz aja...@gmail.com added the comment: Can't quite understand what the problem is supposed to be, comments? -- nosy: +ajaksu2 stage: - test needed type: - feature request versions: +Python 2.7 -Python 2.3 ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue995907 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue995907] memory leak with threads and enhancement of the timer class
Antoine Pitrou pit...@free.fr added the comment: I can't understand problem number 1 either. I suspect it is a misunderstanding by the poster, since he is talking about a destructor of the c++ based library and there's no c++ code in the interpreter. As for number 2, it is a legitimate feature request. However, since the current Timer class is very inefficient (one separate thread for each timer), it would make sense to rephrase it as a more general suggestion to re-implement the Timer class using a single background thread and a heapq-based priority queue. It would be a good project for someone wanting to start contributing to the interpreter. -- nosy: +pitrou ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue995907 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
