Re: [Python-Dev] GIL removal question

2011-08-17 Thread Sturla Molden
Den 10.08.2011 13:43, skrev Guido van Rossum: They have a specific plan, based on Software Transactional Memory: http://morepypy.blogspot.com/2011/06/global-interpreter-lock-or-how-to-kill.html Microsoft's experiment to use STM in .NET failed though. And Linux got rid of the BKL without STM.

Re: [Python-Dev] GIL removal question

2011-08-13 Thread Stefan Behnel
Guido van Rossum, 12.08.2011 23:38: On Fri, Aug 12, 2011 at 12:57 PM, Rene Nejsum wrote: I think I understand the background and need for GIL. Without it Python programs would have been cluttered with lock/synchronized statements and C-extensions would be harder to write. No, sorry, the first

Re: [Python-Dev] GIL removal question

2011-08-13 Thread Guido van Rossum
On Sat, Aug 13, 2011 at 2:12 AM, Stefan Behnel stefan...@behnel.de wrote: Guido van Rossum, 12.08.2011 23:38: On Fri, Aug 12, 2011 at 12:57 PM, Rene Nejsum wrote: I think I understand the background and need for GIL. Without it Python programs would have been cluttered with lock/synchronized

Re: [Python-Dev] GIL removal question

2011-08-13 Thread Antoine Pitrou
On Sat, 13 Aug 2011 09:08:16 -0400 Guido van Rossum gu...@python.org wrote: And, though mostly off-topic, the worst problem with C code, calling back into Python, and the GIL that I have seen (several times): Suppose you are calling some complex C library that creates threads itself, where

Re: [Python-Dev] GIL removal question

2011-08-13 Thread Guido van Rossum
On Sat, Aug 13, 2011 at 8:43 AM, Antoine Pitrou solip...@pitrou.net wrote: On Sat, 13 Aug 2011 09:08:16 -0400 Guido van Rossum gu...@python.org wrote: And, though mostly off-topic, the worst problem with C code, calling back into Python, and the GIL that I have seen (several times): Suppose

Re: [Python-Dev] GIL removal question

2011-08-13 Thread Nick Coghlan
On Sun, Aug 14, 2011 at 9:26 AM, Guido van Rossum gu...@python.org wrote: These days we have PyGILState_Ensure(): http://docs.python.org/dev/c-api/init.html#PyGILState_Ensure and even dedicated documentation: http://docs.python.org/dev/c-api/init.html#non-python-created-threads ;) That is

Re: [Python-Dev] GIL removal question

2011-08-13 Thread Sturla Molden
Den 13.08.2011 17:43, skrev Antoine Pitrou: These days we have PyGILState_Ensure(): http://docs.python.org/dev/c-api/init.html#PyGILState_Ensure With the most recent Cython (0.15) we can just do: with gil: suite to ensure holding the GIL. And similarly from a thread holding the GIL

Re: [Python-Dev] GIL removal question

2011-08-12 Thread Antoine Pitrou
On Fri, 12 Aug 2011 09:32:23 -0500 VanL van.lindb...@gmail.com wrote: On 8/11/2011 2:11 PM, Sturla Molden wrote: (b) another threading model (e.g. one interpreter per thread, as in Tcl, Erlang, or .NET app domains). We are close to this, in that we already have baked-in support for

Re: [Python-Dev] GIL removal question

2011-08-12 Thread Xavier Morel
On 2011-08-11, at 21:11 , Sturla Molden wrote: (b) another threading model (e.g. one interpreter per thread, as in Tcl, Erlang, or .NET app domains). Nitpick: this is not correct re. erlang. While it is correct that it uses another threading model (one could even say no threading model),

Re: [Python-Dev] GIL removal question

2011-08-12 Thread Rene Nejsum
My two danish kroner on GIL issues…. I think I understand the background and need for GIL. Without it Python programs would have been cluttered with lock/synchronized statements and C-extensions would be harder to write. Thanks to Sturla Molden for he's explanation earlier in this thread.

Re: [Python-Dev] GIL removal question

2011-08-12 Thread Sturla Molden
Den 12.08.2011 18:51, skrev Xavier Morel: * Erlang uses erlang processes, which are very cheap preempted *processes* (no shared memory). There have always been tens to thousands to millions of erlang processes per interpreter source contention within the interpreter going back to pre-SMP by

Re: [Python-Dev] GIL removal question

2011-08-12 Thread Sturla Molden
Den 12.08.2011 18:57, skrev Rene Nejsum: My two danish kroner on GIL issues…. I think I understand the background and need for GIL. Without it Python programs would have been cluttered with lock/synchronized statements and C-extensions would be harder to write. Thanks to Sturla Molden for

Re: [Python-Dev] GIL removal question

2011-08-12 Thread Aaron Westendorf
Even in the Erlang model, the afore-mentioned issues of bus contention put a cap on the number of threads you can run in any given application assuming there's any amount of cross-thread synchronization. I wrote a blog post on this subject with respect to my experience in tuning RabbitMQ on NUMA

Re: [Python-Dev] GIL removal question

2011-08-12 Thread Xavier Morel
On 2011-08-12, at 20:59 , Sturla Molden wrote: Den 12.08.2011 18:51, skrev Xavier Morel: * Erlang uses erlang processes, which are very cheap preempted *processes* (no shared memory). There have always been tens to thousands to millions of erlang processes per interpreter source contention

Re: [Python-Dev] GIL removal question

2011-08-12 Thread Guido van Rossum
On Fri, Aug 12, 2011 at 12:57 PM, Rene Nejsum r...@stranden.com wrote: I think I understand the background and need for GIL. Without it Python programs would have been cluttered with lock/synchronized statements and C-extensions would be harder to write. No, sorry, the first half of this is

Re: [Python-Dev] GIL removal question

2011-08-12 Thread Rene Nejsum
Thank you for the clarification, I should have been more precise... On 12/08/2011, at 23.38, Guido van Rossum wrote: On Fri, Aug 12, 2011 at 12:57 PM, Rene Nejsum r...@stranden.com wrote: I think I understand the background and need for GIL. Without it Python programs would have been

Re: [Python-Dev] GIL removal question

2011-08-12 Thread Greg Ewing
Sturla Molden wrote: With one interpreter per thread, and a malloc that does not let threads share memory pages (one heap per thread), Python could do the same. Wouldn't that be more or less equivalent to running each thread in a separate process? -- Greg

Re: [Python-Dev] GIL removal question

2011-08-11 Thread Sturla Molden
Den 09.08.2011 11:33, skrev Марк Коренберг: Probably I want to re-invent a bicycle. I want developers to say me why we can not remove GIL in that way: 1. Remove GIL completely with all current logick. 2. Add it's own RW-locking to all mutable objects (like list or dict) 3. Add RW-locks to every

Re: [Python-Dev] GIL removal question

2011-08-10 Thread David Beazley
Message: 1 Date: Tue, 9 Aug 2011 15:31:47 +0600 From: ? socketp...@gmail.com To: python-dev@python.org Subject: [Python-Dev] GIL removal question Message-ID: CAEmTpZGe2J6poDUW3sihHS3LHDdQ3cq5gWqfty_=z5w8r0r...@mail.gmail.com Content-Type: text/plain; charset=UTF-8

Re: [Python-Dev] GIL removal question

2011-08-10 Thread Nick Coghlan
On Wed, Aug 10, 2011 at 9:09 PM, David Beazley d...@dabeaz.com wrote: You're forgetting step 5. 5. Put fine-grain locks around all reference counting operations (or rewrite all of Python's memory management and garbage collection from scratch). ... After implementing the aforementioned step

Re: [Python-Dev] GIL removal question

2011-08-10 Thread David Beazley
On Aug 10, 2011, at 6:15 AM, Nick Coghlan wrote: On Wed, Aug 10, 2011 at 9:09 PM, David Beazley d...@dabeaz.com wrote: You're forgetting step 5. 5. Put fine-grain locks around all reference counting operations (or rewrite all of Python's memory management and garbage collection from

Re: [Python-Dev] GIL removal question

2011-08-10 Thread Guido van Rossum
On Wed, Aug 10, 2011 at 7:32 AM, David Beazley d...@dabeaz.com wrote: On Aug 10, 2011, at 6:15 AM, Nick Coghlan wrote: On Wed, Aug 10, 2011 at 9:09 PM, David Beazley d...@dabeaz.com wrote: You're forgetting step 5. 5. Put fine-grain locks around all reference counting operations (or

Re: [Python-Dev] GIL removal question

2011-08-10 Thread Maciej Fijalkowski
On Wed, Aug 10, 2011 at 1:43 PM, Guido van Rossum gu...@python.org wrote: On Wed, Aug 10, 2011 at 7:32 AM, David Beazley d...@dabeaz.com wrote: On Aug 10, 2011, at 6:15 AM, Nick Coghlan wrote: On Wed, Aug 10, 2011 at 9:09 PM, David Beazley d...@dabeaz.com wrote: You're forgetting step 5.

Re: [Python-Dev] GIL removal question

2011-08-10 Thread Vlad Riscutia
Removing GIL is interesting work and probably multiple people are willing to contribute. Threading and synchronization is a deep topic and it might be that if just one person toys around with removing GIL he might not see performance improvement (not meaning to offend anyone who tried this,

Re: [Python-Dev] GIL removal question

2011-08-10 Thread Brian Curtin
On Wed, Aug 10, 2011 at 11:14, Vlad Riscutia riscutiav...@gmail.com wrote: Removing GIL is interesting work and probably multiple people are willing to contribute. Threading and synchronization is a deep topic and it might be that if just one person toys around with removing GIL he might not

Re: [Python-Dev] GIL removal question

2011-08-10 Thread Eric Snow
On Wed, Aug 10, 2011 at 10:19 AM, Brian Curtin brian.cur...@gmail.com wrote: On Wed, Aug 10, 2011 at 11:14, Vlad Riscutia riscutiav...@gmail.com wrote: Removing GIL is interesting work and probably multiple people are willing to contribute. Threading and synchronization is a deep topic and it

Re: [Python-Dev] GIL removal question

2011-08-10 Thread Raymond Hettinger
On Aug 10, 2011, at 4:15 AM, Nick Coghlan wrote: After implementing the aforementioned step 5, you will find that the performance of everything, including the threaded code, will be quite a bit worse. Frankly, this is probably the most significant obstacle to have any kind of GIL-less

Re: [Python-Dev] GIL removal question

2011-08-10 Thread Maciej Fijalkowski
On Wed, Aug 10, 2011 at 7:19 PM, Raymond Hettinger raymond.hettin...@gmail.com wrote: On Aug 10, 2011, at 4:15 AM, Nick Coghlan wrote: After implementing the aforementioned step 5, you will find that the performance of everything, including the threaded code, will be quite a bit worse.  

[Python-Dev] GIL removal question

2011-08-09 Thread Марк Коренберг
Probably I want to re-invent a bicycle. I want developers to say me why we can not remove GIL in that way: 1. Remove GIL completely with all current logick. 2. Add it's own RW-locking to all mutable objects (like list or dict) 3. Add RW-locks to every context instance 4. use RW-locks when

[Python-Dev] GIL removal question

2011-08-09 Thread Марк Коренберг
Probably I want to re-invent a bicycle. I want developers to say me why we can not remove GIL in that way: 1. Remove GIL completely with all current logick. 2. Add it's own RW-locking to all mutable objects (like list or dict) 3. Add RW-locks to every context instance 4. use RW-locks when

Re: [Python-Dev] GIL removal question

2011-08-09 Thread Stefan Behnel
Марк Коренберг, 09.08.2011 11:31: In a summary: Please say clearly why, actually, my variant is not still implemented. This question comes up on the different Python lists every once in a while. In general, if you want something to be implemented in a specific way, feel free to provide the