This is a bad idea in the generator itself, as commented earlier by others here.
>From a cross implementation perspective, in Jython, different threads can call next on a non running generator, *so long as they coordinate with each other external to any use of this generator*, and this works fine. But any reliance on gi_running, as seen here, can only be considered to be possible help in detecting such races; it would not even come close to preventing a race: https://github.com/jythontools/jython/blob/master/src/org/python/core/PyGenerator.java#L146 (We don't even bother making gi_running a volatile to get actual test-and-set style semantics, because really it makes no sense to pretend otherwise; and why pay the performance penalty?) The idea of putting generators behind a queue sounds reasonably workable - the semantics then are the right ones, although implementing this efficiently is the trick here. On Sun, Apr 16, 2017 at 11:08 PM, Nick Coghlan <ncogh...@gmail.com> wrote: > On 17 April 2017 at 08:00, Paul Moore <p.f.mo...@gmail.com> wrote: > > On 15 April 2017 at 10:45, Nick Coghlan <ncogh...@gmail.com> wrote: > >> So I'd be opposed to trying to make generator objects natively thread > >> aware - as Stephen notes, the GIL is an implementation detail of > >> CPython, so it isn't OK to rely on it when defining changes to > >> language level semantics (in this case, whether or not it's OK to have > >> multiple threads all calling the same generator without some form of > >> external locking). > >> > >> However, it may make sense to explore possible options for offering a > >> queue.AutoQueue type, where the queue always has a defined maximum > >> size (defaulting to 1), disallows explicit calls to put(), and > >> automatically populates itself based on an iterator supplied to the > >> constructors. Once the input iterator raises StopIteration, then the > >> queue will start reporting itself as being empty. > > > > +1 A generator that can have values pulled from it on different > > threads sounds like a queue to me, so the AutoQueue class that wraps a > > generator seems like a natural abstraction to work with. It also means > > that the cost for thread safety is only paid by those applications > > that need it. > > If someone did build something like this, it would be interesting to > benchmark it against a more traditional producer thread model, where > one thread is responsible for adding work items to the queue, while > others are responsible for draining them. > > The trick is that an auto-queue would borrow execution time from the > consumer threads when new values are needed, so you'd theoretically > get fewer context switches between threads, but at the cost of > changing the nature of the workload in a given thread, and hence > messing with the working set of objects it has active. > > It may also pair well with the concurrent.futures.Executor model, > which is already good for "go handle this predefined list of tasks", > but currently less useful as a replacement for a message queue with a > pool of workers. > > Setting the latter up yourself is currently still a bit tedious, since: > > 1. we don't have a standard threading Pool abstraction in the standard > library, just the one tucked away as part of multiprocessing > 2. while queue.Queue has native support for worker pools, we don't > provide a pre-assembled version that makes it easy to say "here is the > producer, here are the consumers, wire them together for me" > > There are good reasons for that (mainly that it's hard to come up with > an abstraction that's useful in its own right without becoming so > complex that you're on the verge of reinventing a task manager like > celery or a distributed computation manager like dask), but at the > same time, the notion of "input queue, worker pool, output queue" is > one that comes up a *lot* across different concurrency models, so > there's potential value in providing a low-barrier-to-entry > introduction to that idiom as part of the standard library. > > Cheers, > Nick. > > -- > Nick Coghlan | ncogh...@gmail.com | Brisbane, Australia > _______________________________________________ > Python-ideas mailing list > Python-ideas@python.org > https://mail.python.org/mailman/listinfo/python-ideas > Code of Conduct: http://python.org/psf/codeofconduct/ >
_______________________________________________ Python-ideas mailing list Python-ideas@python.org https://mail.python.org/mailman/listinfo/python-ideas Code of Conduct: http://python.org/psf/codeofconduct/
