On Fri, Oct 24, 2008 at 10:40 AM, Andy O'Meara <[EMAIL PROTECTED]> wrote: >> > 2) Barriers to "free threading". As Jesse describes, this is simply >> > just the GIL being in place, but of course it's there for a reason. >> > It's there because (1) doesn't hold and there was never any specs/ >> > guidance put forward about what should and shouldn't be done in multi- >> > threaded apps >> >> No, it's there because it's necessary for acceptable performance >> when multiple threads are running in one interpreter. Independent >> interpreters wouldn't mean the absence of a GIL; it would only >> mean each interpreter having its own GIL. >> > > I see what you're saying, but let's note that what you're talking > about at this point is an interpreter containing protection from the > client level violating (supposed) direction put forth in python > multithreaded guidelines. Glenn Linderman's post really gets at > what's at hand here. It's really important to consider that it's not > a given that python (or any framework) has to be designed against > hazardous use. Again, I refer you to the diagrams and guidelines in > the QuickTime API: > > http://developer.apple.com/technotes/tn/tn2125.html > > They tell you point-blank what you can and can't do, and it's that's > simple. Their engineers can then simply create the implementation > around those specs and not weigh any of the implementation down with > sync mechanisms. I'm in the camp that simplicity and convention wins > the day when it comes to an API. It's safe to say that software > engineers expect and assume that a thread that doesn't have contact > with other threads (except for explicit, controlled message/object > passing) will run unhindered and safely, so I raise an eyebrow at the > GIL (or any internal "helper" sync stuff) holding up an thread's > performance when the app is designed to not need lower-level global > locks. > > Anyway, let's talk about solutions. My company looking to support > python dev community endeavor that allows the following: > > - an app makes N worker threads (using the OS) > > - each worker thread makes its own interpreter, pops scripts off a > work queue, and manages exporting (and then importing) result data to > other parts of the app. Generally, we're talking about CPU-bound work > here. > > - each interpreter has the essentials (e.g. math support, string > support, re support, and so on -- I realize this is open-ended, but > work with me here). > > Let's guesstimate about what kind of work we're talking about here and > if this is even in the realm of possibility. If we find that it *is* > possible, let's figure out what level of work we're talking about. > >From there, I can get serious about writing up a PEP/spec, paid > support, and so on.
Point of order! Just for my own sanity if anything :) I think some minor clarifications are in order. What are "threads" within Python: Python has built in support for POSIX light weight threads. This is what most people are talking about when they see, hear and say "threads" - they mean Posix Pthreads (http://en.wikipedia.org/wiki/POSIX_Threads) this is not what you (Adam) seem to be asking for. PThreads are attractive due to the fact they exist within a single interpreter, can share memory all "willy nilly", etc. Python does in fact, use OS-Level pthreads when you request multiple threads. The Global Interpreter Lock is fundamentally designed to make the interpreter easier to maintain and safer: Developers do not need to worry about other code stepping on their namespace. This makes things thread-safe, inasmuch as having multiple PThreads within the same interpreter space modifying global state and variable at once is, well, bad. A c-level module, on the other hand, can sidestep/release the GIL at will, and go on it's merry way and process away. POSIX Threads/pthreads/threads as we get from Java, allow unsafe programming styles. These programming styles are of the "shared everything deadlock lol" kind. The GIL *partially* protects against some of the pitfalls. You do not seem to be asking for pthreads :) http://www.python.org/doc/faq/library/#can-t-we-get-rid-of-the-global-interpreter-lock http://en.wikipedia.org/wiki/Multi-threading However, then there are processes. The difference between threads and processes is that they do *not share memory* but they can share state via shared queues/pipes/message passing - what you seem to be asking for - is the ability to completely fork independent Python interpreters, with their own namespace and coordinate work via a shared queue accessed with pipes or some other communications mechanism. Correct? Multiprocessing, as it exists within python 2.6 today actually forks (see trunk/Lib/multiprocessing/forking.py) a completely independent interpreter per process created and then construct pipes to inter-communicate, and queue to do work coordination. I am not suggesting this is good for you - I'm trying to get to exactly what you're asking for. Fundamentally, allowing total free-threading with Posix threads, using the same Java-Model for control is a recipe for pain - we're just repeating mistakes instead of solving a problem, ergo - Adam Olsen's work. Monitors, Actors, etc have all been discussed, proposed and are being worked on. So, just to clarify - Andy, do you want one interpreter, $N threads (e.g. PThreads) or the ability to fork multiple "heavyweight" processes? Other bits for reading: http://www.boddie.org.uk/python/pprocess.html (as an alternative the multiprocessing) http://smparkes.net/tag/dramatis/ http://osl.cs.uiuc.edu/parley/ http://candygram.sourceforge.net/ -- http://mail.python.org/mailman/listinfo/python-list