Thank you for all for all the answers so far, particularly to Ilya and
Jean-Paul who provided some very helpful code samples.
It's interesting to realise that, by avoiding locking, we can end up
with a much more efficient implementation. I'll have to figure out how
widely we can apply this technique - and how often it's going to be
worth rewriting things to allow that. Thanks for some useful pointers!
Richard
On 12/03/18 20:00, Jean-Paul Calderone wrote:
On Mon, Mar 12, 2018 at 3:52 PM, Ilya Skriblovsky
<ilyaskriblov...@gmail.com <mailto:ilyaskriblov...@gmail.com>> wrote:
Hi, Richard,
I've used class like this to cache the result of Expensive
Calculation:
class DeferredCache:
pending = None
result = None
failure = None
def __init__(self, expensive_func):
self.expensive_func = expensive_func
def __call__(self):
if self.pending is None:
def on_ready(result):
self.result = result
def on_fail(failure):
self.failure = failure
self.pending =
defer.maybeDeferred(self.expensive_func).addCallbacks(on_ready,
on_fail)
return self.pending.addCallback(self._return_result)
This seems like basically a correct answer to me. However, I suggest
one small change.
You probably want to create and return a new Deferred for each
result. If you don't, then your internal `pending` Deferred is now
reachable by application code.
As written, an application might (very, very reasonably):
d = getResource()
d.addCallback(long_async_operation)
Now `pending` has `long_async_operation` as a callback on its chain.
This will prevent anyone else from getting a result until
`long_async_operation` is done.
You can fix this by:
result = Deferred()
self.pending.addCallback(self._return_result).chainDeferred(result)
return result
Now the application can only reach `result`. Nothing they do to
`result` will make much difference to `pending` because
`chainDeferred` only puts `callback` (and `errback`) onto `pending`'s
callback chain. `callback` and `errback` don't wait on anything.
You have to be a little careful with `chainDeferred` because it
doesn't have the recursion-avoidance logic that implicit chaining
has. However, that doesn't matter in this particular case because the
chain depth is fixed at two (`pending` and `result`). The problems
only arise if you extend the chain out in this direction without bound.
Jean-Paul
def _return_result(self, _):
return self.failure or self.result
Using it you can get rid of DeferredLocks:
deferred_cache = DeferredCache(do_expensive_calculation)
def getResource():
return deferred_cache()
It will start `expensive_func` on the first call. The second and
consequtive calls will return deferreds that resolves with the
result when expensive_func is done. If you call it when result is
already here, it will return alread-fired deferred.
Of course, it will require some more work if you need to pass
arguments to `expensive_func` and memoize results per arguments
values.
-- ilya
_______________________________________________
Twisted-Python mailing list
Twisted-Python@twistedmatrix.com
https://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python
