On Apr 17, 9:19 am, sturlamolden <[EMAIL PROTECTED]> wrote: > On 17 Apr, 10:25, "Martin v. Löwis" <[EMAIL PROTECTED]> wrote: > > > help progress at all. I think neither was the case in this thread - > > the guy claimed that he actually did something about the GIL, and > > now we are all waiting for him to also tell us what it is that he > > did. > > Ok, I did not remove the GIL, but I found a way to remove its > notorious side effect on SMPs. So I am going to reveal it now. Forgive > my strange sence of humor at 3 o'clock in the morning. > > First, think about this: > > (1) What is the GIL? > (2) Where does it live? > (3) How can it be manually released? > > The answer to this is: > > (1) A critical section (a lock/mutex object) > (2) As a global object in Python25.dll on my computer > (3) Using Python's C API or calling methods in a ctypes.CDLL object > > The Python C API has the ability to embed Python interpreters. You do > this by importing Python25.dll into the process. ctypes has the > ability to call functions in a DLL. So is it possible to embed Python > in Python? And what would be consequence be? > > First, if I try to load a DLL more than once, Windows will detect this > and just give me a handle to the currently imported library. So by > importing Python25.dll with ctypes, I can just make my Python > interpreter talk to itself through its own CAPI. Yes I can create sub > interpreters using PyNew_Interpreter, but they all share the same GIL, > so it's not useful here. > > So here is what I suddendly realized, and please don't laugh, its so > simple its almost silly: > > I make some copies of Python25.dll, and call them Python25-1.dll, > Python25-2.dll, Python25-3.dll, Python25-4.dll, etc. Then I load them > all into my current Python process as ctypes.CDLL objects. Tada! I now > have a pool of independent Python interpreters, not sharing the GIL, > but still living in the same process. > > I can make the different interpreters talk to each other, including > manipulating each other's GIL, using the using ctypes and Python's C > API for embedding. > > So why does this circumvent the GIL? Because ctypes releases it before > calling functions form a CDLL object. > > If I use my main interpreter to delegate a task to one of its embedded > 'children', its GIL will be released while it is waiting for the > answer. Associating each embedded interpreter with a threading.Thread > is all that remains. The GIL is released while the thread operating > the child interpreter is blocked. > > An there you have the answer. It's really very simple :-)
Interesting. Windows specific, but there's other ways to do the same thing more portably. The bigger issue is that you can't share any objects. This effectively gives you a multiprocess model - a bit cheaper than that, but not enough to really supply GIL-free threading. -- http://mail.python.org/mailman/listinfo/python-list