Re: [Python-ideas] Including the unparse module in the standard library
On Mon, Jul 16, 2018 at 12:24 PM, Brett Cannon wrote: > > Since it isn't necessary for Python to function, I would say we probably > don''t want to pull it up. Then the maintenance burden grows much more. > might make sense to put it on pypi though, if someone want to take responsibility for it. -CHB -- Christopher Barker, Ph.D. Oceanographer Emergency Response Division NOAA/NOS/OR(206) 526-6959 voice 7600 Sand Point Way NE (206) 526-6329 fax Seattle, WA 98115 (206) 526-6317 main reception chris.bar...@noaa.gov ___ Python-ideas mailing list Python-ideas@python.org https://mail.python.org/mailman/listinfo/python-ideas Code of Conduct: http://python.org/psf/codeofconduct/
Re: [Python-ideas] Including the unparse module in the standard library
On Thu, 12 Jul 2018 at 11:21 Andre Roberge wrote: > In the cPython repository, there is an unparse module in the Tools section. > https://github.com/python/cpython/blob/master/Tools/parser/unparse.py > > However, as it is not part of the standard library, it cannot be easily > used; to do so, one needs to make a local copy in a place from where it can > be imported. > > This module can be useful for people using the ast module to create and > parse trees, modify them ... and who want to convert the result back into > source code. Since it is obviously maintained to be compatible with the > current Python version, > Maintained doesn't mean it's check on very often, nor designed well or updated. ;) > would it be possible to include the unparse module in the standard library? > Since it isn't necessary for Python to function, I would say we probably don''t want to pull it up. Then the maintenance burden grows much more. ___ Python-ideas mailing list Python-ideas@python.org https://mail.python.org/mailman/listinfo/python-ideas Code of Conduct: http://python.org/psf/codeofconduct/
Re: [Python-ideas] Do not block threads when pickle/unpickle
If you've for example an application with a GUI. Every time bigger objects are pickled/unpickled the complete GUI blocks. Or if you've a server application with multiple network connections and the application should have a guaranteed response time, then it is impossible if one single client could lead the server to block for a longer time if it sends a big object which takes long to unpickle. Just 2 simple examples. Am 2018-07-16 um 20:05 schrieb Joseph Jevnik: The GIL must be held to allocate memory for Python objects and to invoke the Python code to deserialize user defined picklable objects. I don't think there is a long span of time where the code could leave the GIL released. The Python implementation is just pausing to let other Python threads run, but it is not actually able to parallelize. The same would be true of the C implementation, is there a reason to want to pause the thread to let another thread run? On Mon, Jul 16, 2018 at 1:56 PM, Martin Bammer wrote: Hi, the old and slow python implementation of pickle didn't block background thread. But the newer C-implementation blocks other threads while dump/load is running. Wouldn't it be possible to allow other threads during this time? Especially could load/loads release the GIL, because Python objects are not available to the Python code until these functions have finished? Regards, Martin ___ Python-ideas mailing list Python-ideas@python.org https://mail.python.org/mailman/listinfo/python-ideas Code of Conduct: http://python.org/psf/codeofconduct/ ___ Python-ideas mailing list Python-ideas@python.org https://mail.python.org/mailman/listinfo/python-ideas Code of Conduct: http://python.org/psf/codeofconduct/
Re: [Python-ideas] Do not block threads when pickle/unpickle
Hi, On Mon, 16 Jul 2018 19:56:34 +0200 Martin Bammer wrote: > Hi, > > the old and slow python implementation of pickle didn't block background > thread. > > But the newer C-implementation blocks other threads while dump/load is > running. This is a fair comment. Please open an issue on the bug tracker at https://bugs.python.org/ Regards Antoine. ___ Python-ideas mailing list Python-ideas@python.org https://mail.python.org/mailman/listinfo/python-ideas Code of Conduct: http://python.org/psf/codeofconduct/
Re: [Python-ideas] Do not block threads when pickle/unpickle
The GIL must be held to allocate memory for Python objects and to invoke the Python code to deserialize user defined picklable objects. I don't think there is a long span of time where the code could leave the GIL released. The Python implementation is just pausing to let other Python threads run, but it is not actually able to parallelize. The same would be true of the C implementation, is there a reason to want to pause the thread to let another thread run? On Mon, Jul 16, 2018 at 1:56 PM, Martin Bammer wrote: > Hi, > > the old and slow python implementation of pickle didn't block background > thread. > > But the newer C-implementation blocks other threads while dump/load is > running. > > Wouldn't it be possible to allow other threads during this time? > > Especially could load/loads release the GIL, because Python objects are not > available to the Python code until these functions have finished? > > Regards, > > Martin > > > ___ > Python-ideas mailing list > Python-ideas@python.org > https://mail.python.org/mailman/listinfo/python-ideas > Code of Conduct: http://python.org/psf/codeofconduct/ ___ Python-ideas mailing list Python-ideas@python.org https://mail.python.org/mailman/listinfo/python-ideas Code of Conduct: http://python.org/psf/codeofconduct/
[Python-ideas] Do not block threads when pickle/unpickle
Hi, the old and slow python implementation of pickle didn't block background thread. But the newer C-implementation blocks other threads while dump/load is running. Wouldn't it be possible to allow other threads during this time? Especially could load/loads release the GIL, because Python objects are not available to the Python code until these functions have finished? Regards, Martin ___ Python-ideas mailing list Python-ideas@python.org https://mail.python.org/mailman/listinfo/python-ideas Code of Conduct: http://python.org/psf/codeofconduct/
Re: [Python-ideas] The future of Python parallelism. The GIL. Subinterpreters. Actors.
On Mon, 16 Jul 2018 18:00:37 +0100 MRAB wrote: > Could you explicitly share an object in a similar way to how you > explicitly open a file? > > The shared object's refcount would be incremented and the sharing > function would return a proxy to the shared object. > > Refcounting in the thread/process would be done on the proxy. > > When the proxy is closed or garbage-collected, the shared object's > refcount would be decremented. > > The shared object could be garbage-collected when its refcount drops to > zero. Yes, I'm assuming that would be how shareable buffers could be implemented: a per-interpreter proxy (with a regular Python refcount) mediating access to a shared object (which could have an atomic / thread-safe refcount). As for how shareable buffers could be useful, see my work on PEP 574: https://www.python.org/dev/peps/pep-0574/ Regards Antoine. ___ Python-ideas mailing list Python-ideas@python.org https://mail.python.org/mailman/listinfo/python-ideas Code of Conduct: http://python.org/psf/codeofconduct/
Re: [Python-ideas] The future of Python parallelism. The GIL. Subinterpreters. Actors.
On 2018-07-16 05:24, Chris Angelico wrote: On Mon, Jul 16, 2018 at 1:21 PM, Nathaniel Smith wrote: On Sun, Jul 15, 2018 at 6:00 PM, Chris Angelico wrote: On Mon, Jul 16, 2018 at 10:31 AM, Nathaniel Smith wrote: On Sun, Jul 8, 2018 at 11:27 AM, David Foster wrote: * The Actor model can be used with some effort via the “multiprocessing” module, but it doesn’t seem that streamlined and forces there to be a separate OS process per line of execution, which is relatively expensive. What do you mean by "the Actor model"? Just shared-nothing concurrency? (My understanding is that in academia it means shared-nothing + every thread/process/whatever gets an associated queue + queues are globally addressable + queues have unbounded buffering + every thread/process/whatever is implemented as a loop that reads messages from its queue and responds to them, with no internal concurrency. I don't know why this particular bundle of features is considered special. Lots of people seem to use it in looser sense though.) Shared-nothing concurrency is, of course, the very easiest way to parallelize. But let's suppose you're trying to create an online multiplayer game. Since it's a popular genre at the moment, I'll go for a battle royale game (think PUBG, H1Z1, Fortnite, etc). A hundred people enter; one leaves. The game has to let those hundred people interact, which means that all hundred people have to be connected to the same server. And you have to process everyone's movements, gunshots, projectiles, etc, etc, etc, fast enough to be able to run a server "tick" enough times per second - I would say 32 ticks per second is an absolute minimum, 64 is definitely better. So what happens when the processing required takes more than one CPU core for 1/32 seconds? A shared-nothing model is either fundamentally impossible, or a meaningless abstraction (if you interpret it to mean "explicit queues/pipes for everything"). What would the "Actor" model do here? "Shared-nothing" is a bit of jargon that means there's no *implicit* sharing; your threads can still communicate, the communication just has to be explicit. I don't know exactly what algorithms your hypothetical game needs, but they might be totally fine in a shared-nothing approach. It's not just for embarrassingly parallel problems. Right, so basically it's the exact model that Python *already* has for multiprocessing - once you go to separate processes, nothing is implicitly shared, and everything has to be done with queues. Ideally, I would like to be able to write my code as a set of functions, then easily spin them off as separate threads, and have them able to magically run across separate CPUs. Unicorns not being a thing, I'm okay with warping my code a bit around the need for parallelism, but I'm not sure how best to do that. Assume here that we can't cheat by getting most of the processing work done with the GIL released (eg in Numpy), and it actually does require Python-level parallelism of CPU-heavy work. If you need shared-memory threads, on multiple cores, for CPU-bound logic, where the logic is implemented in Python, then yeah, you basically need a free-threaded implementation of Python. Jython is such an implementation. PyPy could be if anyone were interested in funding it [1], but apparently no-one is. Probably removing the GIL from CPython is impossible. (I'd be happy to be proven wrong.) Sorry I don't have anything better to report. (This was a purely hypothetical example.) There could be some interesting results from using the GIL only for truly global objects, and then having other objects guarded by arena locks. The trouble is that, in CPython, as soon as you reference any read-only object from the globals, you need to raise its refcount. ISTR someone mentioned something along the lines of sys.eternalize(obj) to flag something as "never GC this thing, it no longer has a refcount", which would then allow global objects to be referenced in a truly read-only way (eg to call a function). Sadly, I'm not expert enough to actually look into implementing it, but it does seem like a very cool concept. It also fits into the "warping my code a bit" category (eg eternalizing a small handful of key objects, and paying the price of "well, now they can never be garbage collected"), with the potential to then parallelize more easily. Could you explicitly share an object in a similar way to how you explicitly open a file? The shared object's refcount would be incremented and the sharing function would return a proxy to the shared object. Refcounting in the thread/process would be done on the proxy. When the proxy is closed or garbage-collected, the shared object's refcount would be decremented. The shared object could be garbage-collected when its refcount drops to zero. The good news is that there are many, many situations where you don't actually need "shared-memory threads, on multiple cores, for CPU-bound logic, where the logic is
Re: [Python-ideas] The future of Python parallelism. The GIL. Subinterpreters. Actors.
On Mon, 16 Jul 2018 07:00:34 +0200 Stephan Houben wrote: > What about the following model: you have N Python interpreters, each with > their own GIL. Each *Python* object belongs to precisely one interpreter. This is roughly what Eric's subinterpreters approach tries to do. Regards Antoine. ___ Python-ideas mailing list Python-ideas@python.org https://mail.python.org/mailman/listinfo/python-ideas Code of Conduct: http://python.org/psf/codeofconduct/
Re: [Python-ideas] The future of Python parallelism. The GIL. Subinterpreters. Actors.
On Sun, 15 Jul 2018 20:21:56 -0700 Nathaniel Smith wrote: > > If you need shared-memory threads, on multiple cores, for CPU-bound > logic, where the logic is implemented in Python, then yeah, you > basically need a free-threaded implementation of Python. Jython is > such an implementation. PyPy could be if anyone were interested in > funding it [1], but apparently no-one is. Probably removing the GIL > from CPython is impossible. (I'd be happy to be proven wrong.) It's not that it's impossible, it's that everyone trying to remove it ended up with a 30-40% slowdown in a single-threaded mode (*). Perhaps Larry manages to do better, though ;-) (*) a figure which I assume is highly workload-dependent Regards Antoine. ___ Python-ideas mailing list Python-ideas@python.org https://mail.python.org/mailman/listinfo/python-ideas Code of Conduct: http://python.org/psf/codeofconduct/
Re: [Python-ideas] The future of Python parallelism. The GIL. Subinterpreters. Actors.
Nick Coghlan wrote: > It was never extended beyond Windows, and a Windows-only solution > doesn't meet the needs of a lot of folks interested in more efficient > exploitation of multiple local CPU cores. On the other hand Windows has a higher need for a better multi-core story. A reasonable Unix-only solution already exists with fork() if you don't need a lot of shared memory, multi-processing on Windows is just not on the same level. ___ Python-ideas mailing list Python-ideas@python.org https://mail.python.org/mailman/listinfo/python-ideas Code of Conduct: http://python.org/psf/codeofconduct/
