@Guido: I agree, that’s a much cleaner solution to pass the executor. However, I think the last line should be future.add_done_callback(callback) return newf
not executor.submit. I’ll rewrite it like this and resubmit tonight for discussion. Sent from my iPhone > On Jan 26, 2018, at 11:59 AM, Guido van Rossum <gu...@python.org> wrote: > > @Bar: I don't know about exposing _chain_future(). Certainly it's overkill > for what the OP wants -- their PR only cares about chaining > concurrent.future.Future. > > @Daniel: I present the following simpler solution -- it requires you to > explicitly pass the executor, but since 'fn' is being submitted to an > executor, EIBTI. > > def then(executor, future, fn): > newf = concurrent.futures.Future() > def callback(fut): > f = executor.submit(fn, fut) > try: > newf.set_result(f.result()) > except CancelledError: > newf.cancel() > except Exception as err: > newf.set_exception(err) > return executor.submit(callback) > > I can't quite follow your reasoning about worker threads (and did you realize > that because of the GIL, Python doesn't actually use multiple cores?). But I > suppose it doesn't matter whether I understand that -- your point is that you > want the 'fn' function submitted to the executor, not run as a "done > callback". And that's reasonable. But modifying so much code just so the > Future can know which to executor it belongs so you can make then() a method > seems overkill. > >> On Fri, Jan 26, 2018 at 8:54 AM, Daniel Collins <dancollin...@gmail.com> >> wrote: >> So, just going point by point: >> >> Yes, absolutely put this off for 3.8. I didn’t know the freeze was so close >> or I would have put the 3.8 tag on originally. >> >> Yes, absolutely it is only meant for concurrent.futures futures, it only >> changes async where async uses concurrent.futures futures. >> >> Here’s a more fleshed out description of the use case: >> >> Assume you have two functions. Function a(x: str)->AResult fetches an >> AResult object from a web resource, function b(y: AResult) performs some >> computationally heavy work on AResult. >> >> Assume you’re calling a 10 times with a threadpoolexecutor with 2 worker >> theads. If you were to schedule a as future using submit, and b as a >> callback, the executions would look like this: >> >> ExecutorThread: b*10 >> Worker1: a*5 >> Worker2: a*5 >> >> This only gets worse as more work (b) is scheduled as a callback for the >> result from a. >> >> Now you could resolve this by, instead of submitting b as a callback, >> submitting the following lambda: >> >> lambda x: executor.submit(b, x) >> >> But then you wouldn’t have easy access to this new future. You would have to >> build a lot of boilerplate code to collect that future into some external >> collection, and this would only get worse the deeper the nesting goes. >> >> With this syntax on the other hand, if you run a 10 times using submit, but >> then run a_fut.then(b) for each future, execution instead looks like this: >> >> ExecutorThread: >> Worker1: a*5 b*5 >> Worker2: a*5 b*5 >> >> You can also do additional depth easily. Suppose you want to run 3 c >> operations (processes the output of b) for each b operation. Then you could >> call this like >> >> b_fut = a_fut.then(b) >> >> for i in range(3): >> b_fut.then(c) >> >> And the execution would look like this: >> >> ExecutorThread: >> Worker1: a*5 b*5 c*15 >> Worker2: a*5 b*5 c*15 >> >> Which would be very difficult to do otherwise, and distributes the load >> across the workers, while having direct access to the outputs of the calls >> to c. >> >> -dancollins34 >> >> Sent from my iPhone >> >>> On Jan 26, 2018, at 1:07 AM, Guido van Rossum <gu...@python.org> wrote: >>> >>> I really don't want to distract Yury with this. Let's consider this (or >>> something that addresses the same need) for 3.8. >>> >>> To be clear this is meant as a feature for concurrent.futures.Future, not >>> for asyncio.Future. (It's a bit confusing since you also change asyncio.) >>> >>> Also to be honest I don't understand the use case *or* the semantics very >>> well. You have some explaining to do... >>> >>> (Also, full links: https://bugs.python.org/issue32672; >>> https://github.com/python/cpython/pull/5335) >>> >>>> On Thu, Jan 25, 2018 at 8:38 PM, Daniel Collins <dancollin...@gmail.com> >>>> wrote: >>>> Hello all, >>>> >>>> So, first time posting here. I’ve been bothered for a while about the lack >>>> of the ability to chain futures in python, such that the next future will >>>> execute upon the first’s completion. So I submitted a pr to do this. >>>> This would add the .then(self, fn) method to concurrent.futures.Future. >>>> Thoughts? >>>> >>>> -dancollins34 >>>> >>>> Github PR #5335 >>>> bugs.python.org issue #32672 >>>> >>>> _______________________________________________ >>>> Python-ideas mailing list >>>> Python-ideas@python.org >>>> https://mail.python.org/mailman/listinfo/python-ideas >>>> Code of Conduct: http://python.org/psf/codeofconduct/ >>>> >>> >>> >>> >>> -- >>> --Guido van Rossum (python.org/~guido) > > > > -- > --Guido van Rossum (python.org/~guido)
_______________________________________________ Python-ideas mailing list Python-ideas@python.org https://mail.python.org/mailman/listinfo/python-ideas Code of Conduct: http://python.org/psf/codeofconduct/
