Re: multi threading in multi processor (computer)

[Martin Christensen]

-BEGIN PGP SIGNED MESSAGE-
Hash: SHA1

> "Irmen" == Irmen de Jong <[EMAIL PROTECTED]> writes:
Irmen> Naah.  What about: http://www.razorvine.net/img/killGIL.jpg

Some people have too much spare time and too weird senses of
humour...

Fortunately for the rest of us. :-) This one actually made me laugh
out loud.

Martin

- -- 
Homepage:   http://www.cs.auc.dk/~factotum/
GPG public key: http://www.cs.auc.dk/~factotum/gpgkey.txt
-BEGIN PGP SIGNATURE-
Version: GnuPG v1.2.5 (GNU/Linux)
Comment: Using Mailcrypt+GnuPG 

iEYEARECAAYFAkISBcwACgkQYu1fMmOQldUg2QCgq1ATLCJWqAS7SBsHpcXTduma
xjMAoII+AzDwkp2F2NZvw4PUrBUx+GDh
=Yqjf
-END PGP SIGNATURE-
-- 
http://mail.python.org/mailman/listinfo/python-list

Re: multi threading in multi processor (computer)

[Irmen de Jong]

Leif K-Brooks wrote:
Irmen de Jong wrote:
the GIL must die.
I couldn't resist:
http://www.razorvine.net/img/GIL.jpg

Neither could I:
http://ecritters.biz/diegil.png
(In case it's not entirely obvious, the stick figure just slices the GIL 
into two pieces with his sword, causing its blood to splatter on the wall.)
Naah.
What about:
http://www.razorvine.net/img/killGIL.jpg
--Irmen
--
http://mail.python.org/mailman/listinfo/python-list

Re: Kill GIL (was Re: multi threading in multi processor (computer))

[Paul Rubin]

[EMAIL PROTECTED] (Aahz) writes:
> >[phr] The day is coming when even cheap computers have multiple cpu's.
> >See hyperthreading and the coming multi-core P4's, and the finally
> >announced Cell processor.
> >
> >Conclusion: the GIL must die.
> 
> It's not clear to what extent these processors will perform well with
> shared memory space.  One of the things I remember most about Bruce
> Eckel's discussions of Java and threading is just how broken Java's
> threading model is in certain respects when it comes to CPU caches
> failing to maintain cache coherency.

Um???  I'm not experienced with multiprocessors but I thought that
maintaining cache coherency was a requirement.  What's the deal?  If
coherency isn't maintained, is it really multiprocessing?

> It's always going to be true that getting fully scaled performance
> will require more CPUs with non-shared memory -- that's going to
> mean IPC with multiple processes instead of threads.

But unless you use shared memory, the context switch overhead from
IPC becomes a bad bottleneck.

See http://poshmodule.sourceforge.net/posh/html/node1.html
for an interesting scheme of working around the GIL by spreading
naturally multi-threaded applications into multiple processes
(using shared memory).  It would simplify things a lot if you could
just use threads.
-- 
http://mail.python.org/mailman/listinfo/python-list

Re: multi threading in multi processor (computer)

[Leif K-Brooks]

Irmen de Jong wrote:
the GIL must die.
I couldn't resist:
http://www.razorvine.net/img/GIL.jpg
Neither could I:
http://ecritters.biz/diegil.png
(In case it's not entirely obvious, the stick figure just slices the GIL 
into two pieces with his sword, causing its blood to splatter on the wall.)
--
http://mail.python.org/mailman/listinfo/python-list

Re: Kill GIL (was Re: multi threading in multi processor (computer))

[Courageous]

>Killing the GIL is proposing a silver bullet where there is no werewolf-ly,

About the only reason for killing the GIL is /us/. We, purists,
pythonistas, language nuts, or what not, who for some reason or
other simply hate the idea of the GIL. I'd view it as an artistic
desire, unurgent, something to plan for the future canvas upon
which our painting is painted...

C//

-- 
http://mail.python.org/mailman/listinfo/python-list

Re: Kill GIL (was Re: multi threading in multi processor (computer))

[Jack Diederich]

On Sat, Feb 12, 2005 at 07:13:17PM -0500, Aahz wrote:
> In article <[EMAIL PROTECTED]>,
> Paul Rubin   wrote:
> >
> >The day is coming when even cheap computers have multiple cpu's.
> >See hyperthreading and the coming multi-core P4's, and the finally
> >announced Cell processor.

I'm looking forward to Multi-core P4s (and Opterons).  The Cell is a
non-starter for general purpose computing.  Arstechnica has a couple
good pieces on it, the upshot is that it is one normal processor
with eight strange floating point co-processors hanging off it.

> >Conclusion: the GIL must die.
> 
> It's not clear to what extent these processors will perform well with
> shared memory space.  One of the things I remember most about Bruce
> Eckel's discussions of Java and threading is just how broken Java's
> threading model is in certain respects when it comes to CPU caches
> failing to maintain cache coherency.  It's always going to be true that
> getting fully scaled performance will require more CPUs with non-shared
> memory -- that's going to mean IPC with multiple processes instead of
> threads.
> 
> Don't get me wrong -- I'm probably one of the bigger boosters of threads.
> But it bugs me when people think that getting rid of the GIL will be the
> Holy Grail of Python performance.  No way.  No how.  No time.

"Me Too!"  for a small number of processors (four) it is easier (and
usually sufficient) to pipeline functional parts into different
processes than it is to thread the whole monkey.  As a bonus this usually
gives you scaling across machines (and not just CPUs) for cheap.  I'm 
aware there are some problems where this isn't true.  From reading this
thread every couple months on c.l.py for the last few years it is my 
opinion that the number of people who think threading is the only solution
to their problem greatly outnumber the number of people who actually have 
such a problem (like, nearly all of them).

Killing the GIL is proposing a silver bullet where there is no werewolf-ly,

-Jack

-- 
http://mail.python.org/mailman/listinfo/python-list

Kill GIL (was Re: multi threading in multi processor (computer))

[Aahz]

In article <[EMAIL PROTECTED]>,
Paul Rubin   wrote:
>
>The day is coming when even cheap computers have multiple cpu's.
>See hyperthreading and the coming multi-core P4's, and the finally
>announced Cell processor.
>
>Conclusion: the GIL must die.

It's not clear to what extent these processors will perform well with
shared memory space.  One of the things I remember most about Bruce
Eckel's discussions of Java and threading is just how broken Java's
threading model is in certain respects when it comes to CPU caches
failing to maintain cache coherency.  It's always going to be true that
getting fully scaled performance will require more CPUs with non-shared
memory -- that's going to mean IPC with multiple processes instead of
threads.

Don't get me wrong -- I'm probably one of the bigger boosters of threads.
But it bugs me when people think that getting rid of the GIL will be the
Holy Grail of Python performance.  No way.  No how.  No time.
-- 
Aahz ([EMAIL PROTECTED])   <*> http://www.pythoncraft.com/

"The joy of coding Python should be in seeing short, concise, readable
classes that express a lot of action in a small amount of clear code -- 
not in reams of trivial code that bores the reader to death."  --GvR
-- 
http://mail.python.org/mailman/listinfo/python-list

Re: multi threading in multi processor (computer)

[Irmen de Jong]

Paul Rubin wrote:
John Lenton <[EMAIL PROTECTED]> writes:
and buying more, cheap computers gives you more processing power than
buying less, multi-processor computers. 

The day is coming when even cheap computers have multiple cpu's.
See hyperthreading and the coming multi-core P4's, and the finally
announced Cell processor.
Conclusion: the GIL must die.
I think I agree with this.
And I couldn't resist:
http://www.razorvine.net/img/GIL.jpg
;-)
--Irmen
--
http://mail.python.org/mailman/listinfo/python-list

Re: multi threading in multi processor (computer)

[Paul Rubin]

John Lenton <[EMAIL PROTECTED]> writes:
> and buying more, cheap computers gives you more processing power than
> buying less, multi-processor computers. 

The day is coming when even cheap computers have multiple cpu's.
See hyperthreading and the coming multi-core P4's, and the finally
announced Cell processor.

Conclusion: the GIL must die.
-- 
http://mail.python.org/mailman/listinfo/python-list

Re: multi threading in multi processor (computer)

[Aahz]

In article <[EMAIL PROTECTED]>,
[EMAIL PROTECTED] <[EMAIL PROTECTED]> wrote:
>
>Is anyone has experiance in running python code to run multi thread
>parallel in multi processor. Is it possible ?

Yes.  The GIL prevents multiple Python threads from running
simultaneously, but C extensions can release the GIL; all I/O functions
in the standard library do, so threading Python makes sense for e.g. web
spiders.  See the slides on my website for more info.

>Can python manage which cpu shoud do every thread?

Nope.
-- 
Aahz ([EMAIL PROTECTED])   <*> http://www.pythoncraft.com/

"The joy of coding Python should be in seeing short, concise, readable
classes that express a lot of action in a small amount of clear code -- 
not in reams of trivial code that bores the reader to death."  --GvR
-- 
http://mail.python.org/mailman/listinfo/python-list

Re: multi threading in multi processor (computer)

[John Lenton]

On Wed, Feb 09, 2005 at 07:56:27AM -0800, [EMAIL PROTECTED] wrote:
> Hello Pierre,
> 
> That's a pity, since when we have to run parallel, with single
> processor is really  not efficient. To use more computers I think is
> cheaper than to buy super computer in developt country.

and buying more, cheap computers gives you more processing power than
buying less, multi-processor computers. So the best thing you can do
is learn to leverage some distributed computing scheme. Take a look at
Pyro, and its Event server.

-- 
John Lenton ([EMAIL PROTECTED]) -- Random fortune:
When the cup is full, carry it level.


signature.asc
Description: Digital signature
-- 
http://mail.python.org/mailman/listinfo/python-list

Re: multi threading in multi processor (computer)

[Alan Kennedy]

[EMAIL PROTECTED]
That's a pity, since when we have to run parallel, with single
processor is really  not efficient. To use more computers I think is
cheaper than to buy super computer in developt country.
Although cpython has a GIL that prevents multiple python threads *in the 
same python process* from running *inside the python interpreter* at the 
same time (I/O is not affected, for example), this can be gotten around 
by using multiple processes, each bound to a different CPU, and using 
some form of IPC (pyro, CORBA, bespoke, etc) to communicate between 
those processes.

This solution is not ideal, because it will probably involve 
restructuring your app. Also, all of the de/serialization involved in 
the IPC will slow things down, unless you're using POSH, a shared memory 
based system that requires System V IPC.

http://poshmodule.sf.net
Alternatively, you could simply use either jython or ironpython, both of 
which have no central interpreter lock (because they rely on JVM/CLR 
garbage collection), and thus will support transparent migration of 
threads to multiple processors in a multi-cpu system, if the underlying 
VM supports that.

http://www.jython.org
http://www.ironpython.com
And you shouldn't have to restructure your code, assuming that it is 
already thread-safe?

For interest, I thought I'd mention PyLinda, a distributed object system 
that takes a completely different, higher level, approach to object 
distribution: it creates "tuple space", where objects live. The objects 
can be located and sent messages. But (Py)Linda hides most of gory 
details of how objects actually get distributed, and the mechanics of 
actually connecting with those remote objects.

http://www-users.cs.york.ac.uk/~aw/pylinda/
HTH,
--
alan kennedy
--
email alan:  http://xhaus.com/contact/alan
--
http://mail.python.org/mailman/listinfo/python-list

Re: multi threading in multi processor (computer)

[[EMAIL PROTECTED]]

Hello Pierre,

That's a pity, since when we have to run parallel, with single
processor is really  not efficient. To use more computers I think is
cheaper than to buy super computer in developt country.

Sincerely Yours,
pujo aji

-- 
http://mail.python.org/mailman/listinfo/python-list

Re: multi threading in multi processor (computer)

[Jeremy Jones]

Pierre Barbier de Reuille wrote:
[EMAIL PROTECTED] a écrit :
Hello,
Is anyone has experiance in running python code to run multi thread
parallel in multi processor. Is it possible ?
Can python manage which cpu shoud do every thread?
Sincerely Yours,
Pujo
There's just no way you can use Python in a multi-processor environment,
This isn't exactly correct.  There is no way with plain, out of the box 
Python (and writing plain Python code) to take full advantage of 
multiple processors using a single process.  A single plain vanilla 
Python process will saturate at most one CPU.  I think that's what you 
were trying to express, but I thought it would be best to clarify.  The 
machine I'm running on right now is a dual-CPU machine.  I can 
*definitely* run Python on it.  I haven't tried threading just yet since 
it's a new-to-me machine.  But if I were to watch gkrellm, I would 
expect to see a CPU intensive (multithreaded) process flip back and 
forth between the two CPUs, taking its turn totally saturating both of 
them, one at a time.


because the GIL (Global Interpreter Lock) will prevent two threads 
from running concurrently. When I saw this discussed, the Python 
developper were more into multi-process systems when it comes to 
multi-processors.
I think I even heard some discussion about efficient inter-process 
messaging system, but I can't remember where :o)

Hope it'll help.
Pierre
Jeremy Jones
-- 
http://mail.python.org/mailman/listinfo/python-list

Re: multi threading in multi processor (computer)

[Pierre Barbier de Reuille]

[EMAIL PROTECTED] a écrit :
Hello,
Is anyone has experiance in running python code to run multi thread
parallel in multi processor. Is it possible ?
Can python manage which cpu shoud do every thread?
Sincerely Yours,
Pujo
There's just no way you can use Python in a multi-processor environment, 
because the GIL (Global Interpreter Lock) will prevent two threads from 
running concurrently. When I saw this discussed, the Python developper 
were more into multi-process systems when it comes to multi-processors.
I think I even heard some discussion about efficient inter-process 
messaging system, but I can't remember where :o)

Hope it'll help.
Pierre
--
http://mail.python.org/mailman/listinfo/python-list