On Sep 4, 2019, at 19:52, Bar Harel <[email protected]> wrote:
>
> I'm sorry but I truly fail to see the complication:
>
> sem = Semaphore(10) # line num 1 somewhere near executor creation
> sem.acquire() # line number 2, right before submit
> future = executor.sumbit(...)
> future.add_done_callback(lambda x: sem.release()) # line number 3, right
> after submit.
>
> It's only 3 lines of code, barely noticeable,
Can you really barely notice the difference between this:
future = x.submit(func, arg)
… and this:
sem.acquire()
future = x.submit(func, arg)
future.add_done_callback(lambda x: sem.release())
You’ve turned 1 line of code into 3, and the number of characters or tokens or
concepts to think about have all increased even more; the actual function is
now buried among the boilerplate. I can’t see how you can call that “barely
noticeable”.
There’s also the fact that you have to learn about semaphores to do it this
way. To you or me that may seem trivial, but one of the appeals of
concurrent.futures is that you don’t have to think about synchronization,
because it’s all tied up in the queue, which means it can even be used by rank
novices who don’t even know the difference between a condition variable and a
barrier. (I’m not saying people shouldn’t learn how to use semaphores. They
should also learn how to build their own thread pools, and understand how
futures work under the covers, and so on, but the fact that they can make their
code concurrent, and do it correctly, even before they’ve learned all that is
pretty nice, and I think this proposal is a simple extension that extends that
niceness.)
I have taught novices how to use executors in a few minutes. Often they just
need to look at the parallel-downloader example and they get it. And I’m pretty
sure I could also explain to them how and why to limit concurrency by passing a
max_queue_len parameter in a few minutes. In fact, I’ve seen a novice find the
equivalent feature in Ruby in a couple minutes of web searching and add it to
his program, only stumbling over trying to figure out the deal with all the
different fail policies (IIRC, Ruby has all the options from Java and
more—block, raise, discard and return an empty future, run synchronously in
your thread, pass the method to an arbitrary fail function… I think either of
the first two would satisfy 90% of the uses, so that complexity isn’t needed.
At least if nobody’s asked for it.)
> quite clear, with minimal overhead.
But the max_queue_len is even clearer, and has even less overhead (on top of a
whole lot less boilerplate).
> Just think the general idea might be a premature optimisation.
It’s not about optimization—as I already said in my previous email (the one you
seem to be replying to, although it’s not the one you quoted); the performance
difference is unlikely to matter in most code.
It’s about readability, boilerplate, novice-friendliness, etc. These are much
bigger wins than saving a microsecond in a process that’s slow enough to
execute on another thread.
All that being said, despite it making no difference in the vast majority of
real-world uses, it still seems a little perverse to insist on using the slower
code here. I will gladly write slower code where it doesn’t matter if it makes
it easier for more people to understand my code, but I rarely write slower code
because of some matter of abstract principle, and especially not if it makes it
harder for more people to understand my code.
> If already, I would have either changed the signature of the executor
> creation to have a new 'max_queue' keyword-only argument, or allow you to
> enter the queue as an input,
If you would have added it in the original version, how is adding it now any
different? I assume the only reason you would have added it is that it’s a
nicer API. So what counters that in the opposite direction? It’s not like it’s
going to cause any backward compatibility issues or anything, is it?
> but I still believe that any attempt to create such an interface will just
> cause more complications than the simple 3 line solution.
Why would it cause complications? It’s dead simple to design and implement,
dead simple to understand, and relies on well-understood and well-rested
behavior that’s been part of the queue and mp modules since long before
concurrent even existed.
_______________________________________________
Python-ideas mailing list -- [email protected]
To unsubscribe send an email to [email protected]
https://mail.python.org/mailman3/lists/python-ideas.python.org/
Message archived at
https://mail.python.org/archives/list/[email protected]/message/BBSEGWPNCHL3GRL5VRRQSEQM3VJLPGE5/
Code of Conduct: http://python.org/psf/codeofconduct/
