I'm not sure I get the benefits of this. You can write the same thing even more
simply by directly using `ThreadPoolExecutor`. You don't even need
`as_completed` as in your `concurrent_future_example`, because you don't want
any of the values as they're completed, you only want them after they're all
available. In fact, you don't even need to deal with futures. (You don't even
need an executor; you could just use a `multiprocessing.[Thread]Pool`, but
let's stick with the executor here.)
def main():
with concurrent.futures.ThreadPoolExecutor(max_workers=5) as executor:
succeeded = executor.map(extract_and_load, URLs)
print(f"Successfully completed {sum(1 for result in succeeded if
result)}")
I think this meets your own design considerations better than your version.
It's even less invasive—not even a decorator, just use the existing function
as-is. Since there is no decorator, it doesn't get in the way of serial code.
And so on.
If you need to have environment variables that control the parallelism, you can
wrap `ThreadPoolExecutor` trivially. Just write an `__init__` that looks up the
environment variables before calling `super()`; you don't need to build a whole
different abstraction on top of it.
Of course it's also more flexible—if you do need more complicated concurrency
later (e.g., if processing results takes long enough that it's worth handling
them as they come in instead of waiting for them all), you have the futures,
which can be composed in various ways—but it doesn't in any way force you to
use that flexibility if you don't need it.
If you find this useful anyway, because you have a team that doesn't want to
learn even the basic use of futures and executors, of course that's fine. And
if you put it on PyPI, maybe others will find it useful as well. But I don't
think there's any need for it in the stdlib or anything.
_______________________________________________
Python-ideas mailing list -- [email protected]
To unsubscribe send an email to [email protected]
https://mail.python.org/mailman3/lists/python-ideas.python.org/
Message archived at
https://mail.python.org/archives/list/[email protected]/message/P4SXRCWX464ROR35GHNF33DCUEILBFUA/
Code of Conduct: http://python.org/psf/codeofconduct/
