> FWIW, I agree with Andrew here. Being able to swap a > ThreadPoolExecutor or ProcessPoolExecutor with a serial version using > the same API can have benefits in various situations. One is > easier debugging (in case the problem you have to debug isn't a race > condition, of course :-)). Another is writing a library a command-line > tool or library where the final decision of whether to parallelize > execution (e.g. through a command-line option for a CLI tool) is up > to the user, not the library developer.
After Andrew explained his own use case for it with isolating bugs to ensure that the issue wasn't occurring as a result of parallelism, threads, processes, etc; I certainly can see how it would be useful. I could also see a use case in a CLI tool for a conveniently similar parallel and non-parallel version, although I'd likely prefer just having an entirely separate implementation. Particularly if the parallel version includes diving a large, computationally intensive task into many sub-tasks (more common for PPE), that seems like it could result in significant additional unneeded overhead for the non-parallel version. I think at this point, it's potential usefulness is clear though. But, IMO, the main question is now the following: would it be better *initially* placed in the standard library or on PyPI (which could eventually transition into stdlib if it sees widespread usage)? > It seems there are two possible design decisions for a serial executor: > - one is to execute the task immediately on `submit()` > - another is to execute the task lazily on `result()` To me, it seems like the latter would be more useful for debugging purposes, since that would be more similar to how the submitted task/function would actually be executed. ``submit()`` could potentially "fake" the process of scheduling the execution of the function, but without directly executing it; perhaps with something like this: ``executor.submit()`` => create a pending item => add pending item to dict => add callable to call queue => fut.result() => check if in pending items => get from top of call queue => run work item => pop from pending items => set result/exception => return result (skip last three if fut is not in/associated with a pending item). IMO, that would be similar enough to the general workflow followed in the executors without any of the parallelization. On Sun, Feb 16, 2020 at 6:29 AM Antoine Pitrou <solip...@pitrou.net> wrote: > On Sat, 15 Feb 2020 14:16:39 -0800 > Andrew Barnert via Python-ideas > <python-ideas@python.org> wrote: > > > On Feb 15, 2020, at 13:36, Jonathan Crall <erote...@gmail.com> wrote: > > > > > > Also, there is no duck-typed class that behaves like an executor, but > does its processing in serial. Often times a develop will want to run a > task in parallel, but depending on the environment they may want to disable > threading or process execution. To address this I use a utility called a > `SerialExecutor` which shares an API with > ThreadPoolExecutor/ProcessPoolExecutor but executes processes sequentially > in the same python thread: > > > > This makes sense. I think most futures-and-executors frameworks in other > languages have a serial/synchronous/immediate/blocking executor just like > this. (And the ones that don’t, it’s usually because they have a different > way to specify the same functionality—e.g., in C++, you only use executors > via the std::async function, and you can just pass a launch option instead > of an executor to run synchronously.) > > FWIW, I agree with Andrew here. Being able to swap a > ThreadPoolExecutor or ProcessPoolExecutor with a serial version using > the same API can have benefits in various situations. One is > easier debugging (in case the problem you have to debug isn't a race > condition, of course :-)). Another is writing a library a command-line > tool or library where the final decision of whether to parallelize > execution (e.g. through a command-line option for a CLI tool) is up > to the user, not the library developer. > > It seems there are two possible design decisions for a serial executor: > - one is to execute the task immediately on `submit()` > - another is to execute the task lazily on `result()` > > This could for example be controlled by a constructor argument to > SerialExecutor. > > Regards > > Antoine. > > _______________________________________________ > Python-ideas mailing list -- python-ideas@python.org > To unsubscribe send an email to python-ideas-le...@python.org > https://mail.python.org/mailman3/lists/python-ideas.python.org/ > Message archived at > https://mail.python.org/archives/list/python-ideas@python.org/message/PCDN4JMKR7VCWXTEZSMWWIY55NTT3JOM/ > Code of Conduct: http://python.org/psf/codeofconduct/ >
_______________________________________________ Python-ideas mailing list -- python-ideas@python.org To unsubscribe send an email to python-ideas-le...@python.org https://mail.python.org/mailman3/lists/python-ideas.python.org/ Message archived at https://mail.python.org/archives/list/python-ideas@python.org/message/BOCI4YLSJU5KLLFQSGVEPLHQKJIRW4XE/ Code of Conduct: http://python.org/psf/codeofconduct/
