Re: Status of Python threading support (GIL removal)?

"Paul Rubin" wrote: > "Hendrik van Rooyen" writes: > > I think that this is because (like your link has shown) the problem > > is really not trivial, and also because the model that can bring > > sanity to the party (independent threads/processes that communicate >

Re: Status of Python threading support (GIL removal)?

"Hendrik van Rooyen" writes: > I think that this is because (like your link has shown) the problem > is really not trivial, and also because the model that can bring > sanity to the party (independent threads/processes that communicate > with queued messages) is seen as inefficient at small scale.

Re: Status of Python threading support (GIL removal)?

In article , Jeremy Sanders wrote: >Jesse Noller wrote: >> >> Sorry, you're incorrect. I/O Bound threads do in fact, take advantage >> of multiple cores. > >I don't know whether anyone else brought this up, but it looks >like Python has problems with even this form of threading > >http://www.dab

Re: Status of Python threading support (GIL removal)?

In article <90303b55-8686-4d56-b89c-01e31d0a6...@l8g2000vbp.googlegroups.com>, =?windows-1252?Q?Jure_Erzno=9Enik?= wrote: > >So, recently I started writing a part of this new system in Python. A >report generator to be exact. Let's not go into existing offerings, >they are insufficient for our ne

Re: Status of Python threading support (GIL removal)?

Jesse Noller wrote: > Sorry, you're incorrect. I/O Bound threads do in fact, take advantage > of multiple cores. I don't know whether anyone else brought this up, but it looks like Python has problems with even this form of threading http://www.dabeaz.com/python/GIL.pdf It's certainly a very

Re: Status of Python threading support (GIL removal)?

Jure Erznožnik wrote: > On Jun 20, 1:36 am, a...@pythoncraft.com (Aahz) wrote: >> You should put up or shut up -- I've certainly seen multi-core speedup >> with threaded software, so show us your benchmarks! >> -- > > Sorry, no intent to offend anyone here. Flame wars are not my thing. > > I have

Re: Status of Python threading support (GIL removal)?

Christian Heimes wrote: > Hard computations gain more speed from carefully crafted C or Fortran > code that utilizes features like the L1 and L2 CPU cache, SIMD etc. or > parallelized algorithms. If you start sharing values between multiple > cores you have a serious problem. > > Oh, and use NumPy

Re: Status of Python threading support (GIL removal)?

"Kay Schluehr" wrote: > This implies that people stay defensive concerning concurrency ( like > me right now ) and do not embrace it like e.g. Erlang does. Sometimes > there is a radical change in the way we design applications and a > language is the appropriate medium to express it succinctly.

Re: Status of Python threading support (GIL removal)?

On Jun 21, 9:32 am, OdarR wrote: > > Do you think multiprocessing can help you seriously ? > Can you benefit from multiple cpu ? > > did you try to enhance your code with numpy ? > > Olivier > (installed a backported multiprocessing on his 2.5.1 Python, but need > installation of Xcode first) Mul

Re: Status of Python threading support (GIL removal)?

On 21 juin, 03:27, Jure Erznožnik wrote: > Add: > Carl, Olivier & co. - You guys know exactly what I wanted. > Others: Going back to C++ isn't what I had in mind when I started > initial testing for my project. Do you think multiprocessing can help you seriously ? Can you benefit from multiple cp

Re: Status of Python threading support (GIL removal)?

Look, guys, here's the thing: In the company I work at we decided to rewrite our MRP system in Python. I was one of the main proponents of it since it's nicely cross platform and allows for quite rapid application development. The language and it's built in functions are simply great. The oppositio

Re: Status of Python threading support (GIL removal)?

On Jun 20, 8:18 pm, s...@pobox.com wrote: >     Carl> Maybe you don't intend to sound like you're saying "shut up and >     Carl> use C", but to me, that's how you come off.  If you're going to >     Carl> advise someone to use C, at least try to show some understanding >     Carl> for their concer

Re: Status of Python threading support (GIL removal)?

On 20 Jun., 17:28, Stefan Behnel wrote: > Kay Schluehr wrote: > >> You might want to read about "The Problem with Threads": > > >>http://www.eecs.berkeley.edu/Pubs/TechRpts/2006/EECS-2006-1.pdf > > >> and then decide to switch to an appropriate concurrency model for your use > >> case. > > > and t

Re: Status of Python threading support (GIL removal)?

Carl> I'm sure you think you're trying to be helpful, but you're coming Carl> off as really presumptuous with this casual dismissal of their Carl> concerns. My apologies, but in most cases there is more than one way to skin a cat. Trust me, if removing the global interpreter lock was

Re: Status of Python threading support (GIL removal)?

Add: Carl, Olivier & co. - You guys know exactly what I wanted. Others: Going back to C++ isn't what I had in mind when I started initial testing for my project. -- http://mail.python.org/mailman/listinfo/python-list

Re: Status of Python threading support (GIL removal)?

On Jun 20, 6:36 am, s...@pobox.com wrote: >     Carl> Here's the thing: not everyone complaining about the GIL is trying >     Carl> to get the "raw power of their machines."  They just want to take >     Carl> advantage of multiple cores so that their Python program runs >     Carl> faster. > > If

Re: Status of Python threading support (GIL removal)?

-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 Jean-Paul Calderone wrote: > On Sat, 20 Jun 2009 00:07:27 GMT, Lie Ryan wrote: >> [snip] >> >> Perhaps we should have more built-in/stdlib operations that can release >> GIL safely to release GIL by default? And perhaps some builtin/stdlib >> should r

Re: Status of Python threading support (GIL removal)?

Kay Schluehr wrote: >> You might want to read about "The Problem with Threads": >> >> http://www.eecs.berkeley.edu/Pubs/TechRpts/2006/EECS-2006-1.pdf >> >> and then decide to switch to an appropriate concurrency model for your use >> case. > > and to a programming language that supports it. Maybe

Re: Status of Python threading support (GIL removal)?

Carl> Here's the thing: not everyone complaining about the GIL is trying Carl> to get the "raw power of their machines." They just want to take Carl> advantage of multiple cores so that their Python program runs Carl> faster. If their code is CPU-bound it's likely that rewriting

Re: Status of Python threading support (GIL removal)?

On 20 juin, 11:02, Carl Banks wrote: > Here's the thing: not everyone complaining about the GIL is trying to > get the "raw power of their machines."  They just want to take > advantage of multiple cores so that their Python program runs > faster. > > It would be rude and presumptuous to tell such

Re: Status of Python threading support (GIL removal)?

> You might want to read about "The Problem with Threads": > > http://www.eecs.berkeley.edu/Pubs/TechRpts/2006/EECS-2006-1.pdf > > and then decide to switch to an appropriate concurrency model for your use > case. and to a programming language that supports it. -- http://mail.python.org/mailman/

Re: Status of Python threading support (GIL removal)?

On Jun 19, 4:35 pm, a...@pythoncraft.com (Aahz) wrote: > In article <157e0345-74e0-4144-a2e6-2b4cc854c...@z7g2000vbh.googlegroups.com>, > Carl Banks   wrote: > >I wish Pythonistas would be more willing to acknowledge the (few) > >drawbacks of the language (or implementation, in this case) instead o

Re: Status of Python threading support (GIL removal)?

On Jun 19, 4:42 pm, Christian Heimes wrote: > OdarR schrieb: > > > On 19 juin, 21:41, Carl Banks wrote: > >> He's saying that if your code involves extensions written in C that > >> release the GIL, the C thread can run on a different core than the > >> Python-thread at the same time.  The GIL is

Re: Status of Python threading support (GIL removal)?

> Ross Ridge (RR) wrote: >RR> By definition an I/O bound thread isn't CPU bound so won't benefit from >RR> improved CPU resources. But doing I/O is not the same as being I/O bound. And Python allows multithreading when a thread does I/O even if that thread is not I/O bound but CPU bound. See

Re: Status of Python threading support (GIL removal)?

> Jure Erznožnik (JE) wrote: >JE> I have shown my benchmarks. See first post and click on the link. >JE> That's the reason I started this discussion. >JE> All I'm saying is that you can get threading benefit, but only if the >JE> threading in question is implemented in C plugin. >JE> I have

Re: Status of Python threading support (GIL removal)?

> Jure Erznožnik (JE) wrote: >JE> Sorry, just a few more thoughts: >JE> Does anybody know why GIL can't be made more atomic? I mean, use >JE> different locks for different parts of code? >JE> This way there would be way less blocking and the plugin interface >JE> could remain the same (the in

Re: Status of Python threading support (GIL removal)?

On Sat, 20 Jun 2009 00:07:27 GMT, Lie Ryan wrote: [snip] Perhaps we should have more built-in/stdlib operations that can release GIL safely to release GIL by default? And perhaps some builtin/stdlib should receive an optional argument that instruct them to release GIL and by passing this argume

Re: Status of Python threading support (GIL removal)?

Jure Erznožnik wrote: > On Jun 20, 1:36 am, a...@pythoncraft.com (Aahz) wrote: >> You should put up or shut up -- I've certainly seen multi-core speedup >> with threaded software, so show us your benchmarks! >> -- > > Sorry, no intent to offend anyone here. Flame wars are not my thing. > > I have

Re: Status of Python threading support (GIL removal)?

Jesse Noller wrote: > Sorry, you're incorrect. I/O Bound threads do in fact, take advantage > of multiple cores. wrote: >Incorrect. They take advantage of OS threading support where another >thread can run while one is blocked for I/O. That is not equal to >running on multiple cores (though it

Re: Status of Python threading support (GIL removal)?

Aahz wrote: > In article <157e0345-74e0-4144-a2e6-2b4cc854c...@z7g2000vbh.googlegroups.com>, > Carl Banks wrote: >> I wish Pythonistas would be more willing to acknowledge the (few) >> drawbacks of the language (or implementation, in this case) instead of >> all this rationalization. > > Pleas

Re: Status of Python threading support (GIL removal)?

On Jun 20, 1:36 am, a...@pythoncraft.com (Aahz) wrote: > > You should put up or shut up -- I've certainly seen multi-core speedup > with threaded software, so show us your benchmarks! > -- Sorry, no intent to offend anyone here. Flame wars are not my thing. I have shown my benchmarks. See first p

Re: Status of Python threading support (GIL removal)?

OdarR schrieb: > On 19 juin, 21:41, Carl Banks wrote: >> He's saying that if your code involves extensions written in C that >> release the GIL, the C thread can run on a different core than the >> Python-thread at the same time. The GIL is only required for Python >> code, and C code that uses t

Re: Status of Python threading support (GIL removal)?

In article , =?windows-1252?Q?Jure_Erzno=9Enik?= wrote: >On Jun 19, 11:59=A0pm, Jesse Noller wrote: >> >> Sorry, you're incorrect. I/O Bound threads do in fact, take advantage >> of multiple cores. > >Incorrect. They take advantage of OS threading support where another >thread can run while one

Re: Status of Python threading support (GIL removal)?

In article <157e0345-74e0-4144-a2e6-2b4cc854c...@z7g2000vbh.googlegroups.com>, Carl Banks wrote: > >I wish Pythonistas would be more willing to acknowledge the (few) >drawbacks of the language (or implementation, in this case) instead of >all this rationalization. Please provide more evidence

Re: Status of Python threading support (GIL removal)?

In article , =?windows-1252?Q?Jure_Erzno=9Enik?= wrote: > >I do aggree though that threading is important. Regardless of any >studies showing that threads suck, they are here and they offer >relatively simple concurrency. IMHO they should never have been >crippled like this. Even though GIL solve

Re: Status of Python threading support (GIL removal)?

On Fri, Jun 19, 2009 at 6:10 PM, Jure Erznožnik wrote: > On Jun 19, 11:59 pm, Jesse Noller wrote: >> On Fri, Jun 19, 2009 at 12:50 PM, OdarR wrote: >> > On 19 juin, 16:16, Martin von Loewis > >> If you know that your (C) code is thread safe on its own, you can >> >> release the GIL around long-run

Re: Status of Python threading support (GIL removal)?

On 19 Jun, 21:41, Carl Banks wrote: > > (Note: I'm not talking about releasing the GIL for I/O operations, > it's not the same thing.  I'm talking about the ability to run > computations on multiple cores at the same time, not to block in 50 > threads at the same time.  Multiple cores aren't going

Re: Status of Python threading support (GIL removal)?

On Jun 19, 11:59 pm, Jesse Noller wrote: > On Fri, Jun 19, 2009 at 12:50 PM, OdarR wrote: > > On 19 juin, 16:16, Martin von Loewis >> If you know that your (C) code is thread safe on its own, you can > >> release the GIL around long-running algorithms, thus using as many > >> CPUs as you have ava

Re: Status of Python threading support (GIL removal)?

On Jun 19, 11:45 pm, OdarR wrote: > On 19 juin, 21:05, Christian Heimes wrote: > > > I've seen a single Python process using the full capacity of up to 8 > > CPUs. The application is making heavy use of lxml for large XSL > > transformations, a database adapter and my own image processing library

Re: Status of Python threading support (GIL removal)?

On Fri, Jun 19, 2009 at 12:50 PM, OdarR wrote: > On 19 juin, 16:16, Martin von Loewis > If you know that your (C) code is thread safe on its own, you can >> release the GIL around long-running algorithms, thus using as many >> CPUs as you have available, in a single process. > > what do you mean ?

Re: Status of Python threading support (GIL removal)?

On 19 juin, 21:41, Carl Banks wrote: > He's saying that if your code involves extensions written in C that > release the GIL, the C thread can run on a different core than the > Python-thread at the same time.  The GIL is only required for Python > code, and C code that uses the Python API.  C cod

Re: Status of Python threading support (GIL removal)?

Sorry, just a few more thoughts: Does anybody know why GIL can't be made more atomic? I mean, use different locks for different parts of code? This way there would be way less blocking and the plugin interface could remain the same (the interpreter would know what lock it used for the plugin, so t

Re: Status of Python threading support (GIL removal)?

On 19 juin, 21:05, Christian Heimes wrote: > I've seen a single Python process using the full capacity of up to 8 > CPUs. The application is making heavy use of lxml for large XSL > transformations, a database adapter and my own image processing library > based upon FreeImage. interesting... > O

Re: Status of Python threading support (GIL removal)?

Thanks guys, for all the replies. They were some very interesting reading / watching. Seems to me, the Unladen-Swallow might in time produce code which will have this problem lessened a bit. Their roadmap suggests at least modifying the GIL principles if not fully removing it. On top of this, they

Re: Status of Python threading support (GIL removal)?

On Jun 19, 11:08 am, OdarR wrote: > On 19 juin, 19:13, s...@pobox.com wrote: > > >     Olivier> what do you mean ? > > >     Olivier> Cpython can't benefit from multi-core without multiple > >     Olivier> processes. > > > It can, precisely as Martin indicated.  Only one thread at a time can hold

Re: Status of Python threading support (GIL removal)?

On Jun 19, 6:53 am, Ben Charrow wrote: > Jure Erznožnik wrote: > > See here for introduction: > >http://groups.google.si/group/comp.lang.python/browse_thread/thread/3... > > > Digging through my problem, I discovered Python isn't exactly thread > > safe and to solve the issue, there's this Global

Re: Status of Python threading support (GIL removal)?

OdarR wrote: > I don't see such improvement in the Python library, or maybe you can > indicate us some meaningfull example...? > > I currently only use CPython, with PIL, Reportlab...etc. > I don't see improvement on a Core2duo CPU and Python. How to proceed > (following what you wrote) ? I've se

Re: Status of Python threading support (GIL removal)?

On 19 juin, 19:13, s...@pobox.com wrote: >     Olivier> what do you mean ? > >     Olivier> Cpython can't benefit from multi-core without multiple >     Olivier> processes. > > It can, precisely as Martin indicated.  Only one thread at a time can hold > the GIL.  That doesn't mean that multiple thr

Re: Status of Python threading support (GIL removal)?

>> If you know that your (C) code is thread safe on its own, you can >> release the GIL around long-running algorithms, thus using as many >> CPUs as you have available, in a single process. Olivier> what do you mean ? Olivier> Cpython can't benefit from multi-core without mu

Re: Status of Python threading support (GIL removal)?

CPython itself can't... but the c extension can. Mine did. On Fri, Jun 19, 2009 at 9:50 AM, OdarR wrote: > On 19 juin, 16:16, Martin von Loewis > If you know that your (C) code is thread safe on its own, you can > > release the GIL around long-running algorithms, thus using as many > > CPUs as

Re: Status of Python threading support (GIL removal)?

On 19 juin, 16:16, Martin von Loewis If you know that your (C) code is thread safe on its own, you can > release the GIL around long-running algorithms, thus using as many > CPUs as you have available, in a single process. what do you mean ? Cpython can't benefit from multi-core without multiple

Re: Status of Python threading support (GIL removal)?

Jure Erznožnik wrote: See here for introduction: http://groups.google.si/group/comp.lang.python/browse_thread/thread/370f8a1747f0fb91 Digging through my problem, I discovered Python isn't exactly thread safe and to solve the issue, there's this Global Interpreter Lock (GIL) in place. Effectively

Re: Status of Python threading support (GIL removal)?

Jure Erznožnik wrote: > See here for introduction: > http://groups.google.si/group/comp.lang.python/browse_thread/thread/370f8a1747f0fb91 > > Digging through my problem, I discovered Python isn't exactly thread > safe and to solve the issue, there's this Global Interpreter Lock > (GIL) in place. >

Re: Status of Python threading support (GIL removal)?

On 19 juin, 11:52, Jure Erznožnik wrote: > See here for > introduction:http://groups.google.si/group/comp.lang.python/browse_thread/thread/3... > > Digging through my problem, I discovered Python isn't exactly thread > safe and to solve the issue, there's this Global Interpreter Lock > (GIL) in p

Re: Status of Python threading support (GIL removal)?

Digging through my problem, I discovered Python isn't exactly thread safe and to solve the issue, there's this Global Interpreter Lock (GIL) in place. It's the opposite: Python is exactly thread safe precisely because it has the GIL in place. Is there any other way to work around the issue asi

Re: Status of Python threading support (GIL removal)?

Digging through my problem, I discovered Python isn't exactly thread safe and to solve the issue, there's this Global Interpreter Lock (GIL) in place. It's the opposite: Python is exactly thread safe precisely because it has the GIL in place. Is there any other way to work around the issue asi

Re: Status of Python threading support (GIL removal)?

Jure Erznožnik wrote: > See here for introduction: > http://groups.google.si/group/comp.lang.python/browse_thread/thread/370f8a1747f0fb91 > > Digging through my problem, I discovered Python isn't exactly thread > safe and to solve the issue, there's this Global Interpreter Lock > (GIL) in place. >

Re: Status of Python threading support (GIL removal)?

This is a very long-running issue, that has been discussed many times. Here are the two sides to keeping the gil or removing it: Remove the GIL: - True multi-threaded programming - Scalable performance across a multi-core machine - Unfortunately, this causes a slow-down in single core/th

Status of Python threading support (GIL removal)?

See here for introduction: http://groups.google.si/group/comp.lang.python/browse_thread/thread/370f8a1747f0fb91 Digging through my problem, I discovered Python isn't exactly thread safe and to solve the issue, there's this Global Interpreter Lock (GIL) in place. Effectively, this causes the interp