[issue5840] "Thread State and the Global Interpreter Lock" section of the docs doesn't cover TLS APIs

[Irit Katriel]

Irit Katriel  added the comment:

Indeed, this API was deprecated in 3.7:

https://docs.python.org/3/c-api/init.html#thread-local-storage-tls-api

--
nosy: +iritkatriel
resolution:  -> out of date
stage: needs patch -> resolved
status: open -> closed

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue5840] "Thread State and the Global Interpreter Lock" section of the docs doesn't cover TLS APIs

[Cheryl Sabella]

Cheryl Sabella  added the comment:

I believe this issue is superseded by PEP539 and issue25658?

--
nosy: +csabella

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue5840] Thread State and the Global Interpreter Lock section of the docs doesn't cover TLS APIs

[Antoine Pitrou]

Antoine Pitrou pit...@free.fr added the comment:

Actually, I don't think the TLS APIs should be documented. They are quite 
internal, and used only for the PyGILState APIs.

--
nosy: +pitrou

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue5840
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue5840] Thread State and the Global Interpreter Lock section of the docs doesn't cover TLS APIs

[Antoine Pitrou]

Antoine Pitrou pit...@free.fr added the comment:

They are used by pyOpenSSL and pygobject/pyglib, though :/

--

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue5840
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue5840] Thread State and the Global Interpreter Lock section of the docs doesn't cover TLS APIs

[Georg Brandl]

Georg Brandl ge...@python.org added the comment:

Committed init.rst.diff in r72037.  Next step: document TLS APIs.

--

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue5840
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue5840] Thread State and the Global Interpreter Lock section of the docs doesn't cover TLS APIs

[Jean-Paul Calderone]

New submission from Jean-Paul Calderone exar...@divmod.com:

Python includes several APIs for manipulating TLS:

  PyAPI_FUNC(int) PyThread_create_key(void);
  PyAPI_FUNC(void) PyThread_delete_key(int);
  PyAPI_FUNC(int) PyThread_set_key_value(int, void *);
  PyAPI_FUNC(void *) PyThread_get_key_value(int);
  PyAPI_FUNC(void) PyThread_delete_key_value(int key);

These are not covered in the threading documentation, though.  In fact,
the threading documentation goes so far as to claim that they /don't/ exist:

While most thread packages have a way to store “per-thread global
data,” Python’s internal platform independent thread abstraction doesn’t
support this yet.

It would be great to have these APIs covered in the documentation.  One
subtlety in particular tripped me up and took a long time to track down
by reading various parts of the CPython source - when a thread exits,
its TLS values are not destroyed or reclaimed in any way.  On top of
this, the keys used by the TLS APIs are thread IDs, and thread IDs can
be re-used.  This means that a newer thread can see values from an older
thread in TLS, which was extremely surprising to me, led to a very
obscure threading bug.

--
assignee: georg.brandl
components: Documentation
messages: 86501
nosy: exarkun, georg.brandl
severity: normal
status: open
title: Thread State and the Global Interpreter Lock section of the docs 
doesn't cover TLS APIs
type: feature request

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue5840
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue5840] Thread State and the Global Interpreter Lock section of the docs doesn't cover TLS APIs

[Jeroen Ruigrok van der Werven]

Jeroen Ruigrok van der Werven asmo...@in-nomine.org added the comment:

This is at least relevant to 2.7, reflect this in the issue.

Patch to remove the notion we don't support TLS yet.

We will need to see where to add the current functions in the overall
scheme of threading documentation.

--
keywords: +patch
nosy: +asmodai
stage:  - needs patch
versions: +Python 2.7
Added file: http://bugs.python.org/file13767/init.rst.diff

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue5840
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue5840] Thread State and the Global Interpreter Lock section of the docs doesn't cover TLS APIs

[Georg Brandl]

Georg Brandl ge...@python.org added the comment:

Looks good.

--

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue5840
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

Re: Will Python 3.0 remove the global interpreter lock (GIL)

[TheFlyingDutchman]

2

On Sep 19, 5:08 pm, Terry Reedy [EMAIL PROTECTED] wrote:
 Terry Reedy [EMAIL PROTECTED] wrote in message



 This assumes that comparing versions of 1.5 is still relevant.  As far as I
 know, his patch has not been maintained to apply against current Python.
 This tells me that no one to date really wants to dump the GIL at the cost
 of half Python's speed.  Of course not.  The point of dumping the GIL is to
 use multiprocessors to get more speed!  So with two cores and extra
 overhead, Stein-patched 1.5 would not even break even.

Is the only point in getting rid of the GIL to allow multi-threaded
applications?

Can't multiple threads also provide a performance boost versus
multiple processes on a single-core machine?


 So now this question for you:  CPython 2.5 runs too slow in 2007: true or
 false?

Ugh, I guess I have to agree with Steven D'Aprano - it depends.


 If you answer false, then there is no need for GIL removal.

OK, I can see that.

 If you answer true, then cutting its speed for 90+% of people is bad.

OK, have to agree. Sounds like it could be a good candidate for a
fork. One question - is it a computer science maxim that an
interpreter that implements multi-threading will always be slower when
running single threaded apps?


 And another question: why should such people spend time they do not have to
 make Python worse for themselves?

I can't make an argument for someone doing something for free that
they don't have the time for. Ditto for doing something for free that
they don't want to do. But it does seem that if they give a reason for
why it's the wrong thing to do, it's fair to make a counter-argument.
Although I agree with Steven D'Aprano's theme in that it should be a
cordial rebuttal and not a demand.


-- 
http://mail.python.org/mailman/listinfo/python-list

Re: Will Python 3.0 remove the global interpreter lock (GIL)

[Hendrik van Rooyen]

 Steven D'Aprano [EMAIL PROTECTED] wrote:

 
 I think a better question is, how much faster/slower would Stein's code 
 be on today's processors, versus CPython being hand-simulated in a giant 
 virtual machine made of clockwork?

This obviously depends on whether or not the clockwork is orange

- Hendrik

-- 
http://mail.python.org/mailman/listinfo/python-list

Re: Will Python 3.0 remove the global interpreter lock (GIL)

[Paul Boddie]

On 20 Sep, 00:59, TheFlyingDutchman [EMAIL PROTECTED] wrote:

 Paul it's a pleasure to see that you are not entirely against
 complaints.

Well, it seems to me that I'm usually the one making them. ;-)

 The very fastest Intel processor of the last 1990's that I found came
 out in October 1999 and had a speed around 783Mhz. Current fastest
 processors are something like 3.74 Ghz, with larger caches.

True, although you're paying silly money for a 3.8 GHz CPU with a
reasonable cache. However, as always, you can get something not too
far off for a reasonable sum. When I bought my CPU two or so years
ago, there was a substantial premium for as little as 200 MHz over the
3.0 GHz CPU I went for, and likewise a 3.4 GHz CPU seems to be had for
a reasonable price these days in comparison to the unit with an extra
400 MHz.

Returning to the subject under discussion, though, one big difference
between then and now is the availability of dual core CPUs, and these
seem to be fairly competitive on price with single cores, although the
frequencies of each core are lower and you have to decide whether you
believe the AMD marketing numbers: is a dual 2.2 GHz core CPU 4200+
or not, for example? One can argue whether it's better to have two
cores, especially for certain kinds of applications (and CPython,
naturally), but if I were compiling lots of stuff, the ability to do a
make -j2 and have a decent speed-up would almost certainly push me
in the direction of multicore units, especially if the CPU consumed
less power. And if anyone thinks all this parallelism is just
hypothetical, they should take a look at distcc to see a fairly clear
roadmap for certain kinds of workloads.

 Memory is also faster and larger. It appears that someone running a non-GIL
 implementation of CPython today would have significantly faster
 performance than a GIL CPython implementation of the late 1990's.
 Correct me if I am wrong, but it seems that saying non-GIL CPython is
 too slow, while once valid, has become invalid due to the increase in
 computing power that has taken place.

Although others have picked over these arguments, I can see what
you're getting at: even if we take a fair proportion of the increase
in computing power since the late 1990s, rather than 100% of it,
CPython without the GIL would still faster and have more potential for
further speed increases in more parallel architectures, rather than
running as fast as possible on a sequential architecture where not
even obscene amounts of money will buy you significantly better
performance. But I don't think it's so interesting to consider this
situation as merely a case of removing the GIL and using lots of
threads.

Let us return to the case I referenced above: even across networks,
where the communications cost is significantly higher than that of
physical memory, distributed compilation can provide a good
performance curve. Now I'm not arguing that every computational task
can be distributed in such a way, but we can see that some
applications of parallelisation are mature, even mainstream. There are
also less extreme cases: various network services can be scaled up
relatively effectively by employing multiple processes, as is the UNIX
way; some kinds of computation can be done in separate processes and
the results collected later on - we do this with relational databases
all the time. So, we already know that monolithic multithreaded
processes are not the only answer. (Java put an emphasis on extensive
multithreading and sandboxing because of the requirements of running
different people's code side-by-side on embedded platforms with
relatively few operating system conveniences, as well as on Microsoft
Windows, of course.)

If the programmer cost in removing the GIL and maintaining a GIL-free
CPython ecosystem is too great, then perhaps it is less expensive to
employ other, already well-understood mechanisms instead. Of course,
there's no global programmer lock, so everyone interested in doing
something about removing the GIL, writing their own Python
implementation, or whatever they see to be the solution can freely do
so without waiting for someone else to get round to it. Like those
more readily parallelisable applications mentioned above, more stuff
can get done provided that everyone doesn't decide to join the same
project. A lesson from the real world, indeed.

Paul

-- 
http://mail.python.org/mailman/listinfo/python-list

Re: Will Python 3.0 remove the global interpreter lock (GIL)

[Bruno Desthuilliers]

TheFlyingDutchman a écrit :
(snip)

 I am confused about the benefits/disadvantages of the GIL removal.
 Is it correct that the GIL is preventing CPython from having threads?
 
 Is it correct that the only issue with the GIL is the prevention of
 being able to do multi-threading?

http://docs.python.org/lib/module-thread.html
http://docs.python.org/lib/module-threading.html

-- 
http://mail.python.org/mailman/listinfo/python-list

Re: Will Python 3.0 remove the global interpreter lock (GIL)

[Chris Mellon]

On 9/19/07, TheFlyingDutchman [EMAIL PROTECTED] wrote:
 On Sep 19, 5:08 pm, Terry Reedy [EMAIL PROTECTED] wrote:
  Terry Reedy [EMAIL PROTECTED] wrote in message

 This is a little confusing because google groups does not show your
 original post (not uncommon for them to lose a post in a thread - but
 somehow still reflect the fact that it exists in the total-posts
 number that they display) that you are replying to.


 
  This assumes that comparing versions of 1.5 is still relevant.  As far as I
  know, his patch has not been maintained to apply against current Python.
  This tells me that no one to date really wants to dump the GIL at the cost
  of half Python's speed.  Of course not.  The point of dumping the GIL is to
  use multiprocessors to get more speed!  So with two cores and extra
  overhead, Stein-patched 1.5 would not even break even.
 
  Quad (and more) cores are a different matter.  Hence, I think, the
  resurgence of interest.

 I am confused about the benefits/disadvantages of the GIL removal.
 Is it correct that the GIL is preventing CPython from having threads?


No. Python has threads, and they're wrappers around true OS level
system threads. What the GIL does is prevent *Python* code in those
threads from running concurrently.

 Is it correct that the only issue with the GIL is the prevention of
 being able to do multi-threading?


This sentence doesn't parse in a way that makes sense.

 If you only planned on writing single-threaded applications would GIL-
 removal have no benefit?


Yes.

 Can threading have a performance benefit on a single-core machine
 versus  running multiple processes?


A simple question with a complicated answer. With the qualifier can,
I have to say yes to be honest although you will only see absolute
performance increases on a single core from special purposed APIs that
call into C code anyway - and the GIL doesn't effect those so GIL
removal won't have an effect on the scalability of those operations.

Pure CPU bound threads (all pure Python code) will not increase
performance on a single core (there's CPU level concurrency that can,
but not OS level threads). You can improve *perceived* performance
this way (latency at the expense of throughput), but not raw
performance.

Very, very few operations are CPU bound these days, and even fewer of
the ones were Python is involved. The largest benefits to the desktop
user of multiple cores are the increase in cross-process performance
(multitasking), not single applications.

Servers vary more widely. However, in general, there's not a huge
benefit to faster threading when you can use multiple processes
instead. Python is not especially fast in terms of pure CPU time, so
if you're CPU bound anyway moving your CPU bound code into C (or
something else) is likely to reap far more benefits - and sidestepping
the GIL in the process.

In short, I think any problem that would be directly addressed by
removing the GIL is better addressed by other solutions.

  So now this question for you:  CPython 2.5 runs too slow in 2007: true or
  false?

 I guess I gotta go with Steven D'Aprano - both true and false
 depending on your situation.

  If you answer false, then there is no need for GIL removal.

 OK, I see that.

  If you answer true, then cutting its speed for 90+% of people is bad.

 OK, seems reasonable, assuming that multi-threading cannot be
 implemented without a performance hit on single-threaded applications.
 Is that a computer science maxim - giving an interpreted language
 multi-threading will always negatively impact the performance of
 single-threaded applications?


It's not a maxim, per se - it's possible to have lockless concurrency,
although when you do this it's more like the shared nothing process
approach - but in general, yes. The cost of threading is the cost of
the locking needed to ensure safety, and the amount of locking is
proportional to the amount of shared state. Most of the common uses of
threading in the real world do not improve absolute performance and
won't no matter how many cores you use.

 
  | Most people are not currently bothered by the GIL and would not want its
  | speed halved.
 
  And another question: why should such people spend time they do not have to
  make Python worse for themselves?
 
 Saying they don't have time to make a change, any change, is always
 valid in my book. I cannot argue against that. Ditto for them saying
 they don't want to make a change with no explanation. But it seems if
 they make statements about why a change is not good, then it is fair
 to make a counter-argument. I do agree with the theme of Steven
 D'Aprano's comments in that it should be a cordial counter-argument
 and not a demand.


 --
 http://mail.python.org/mailman/listinfo/python-list

-- 
http://mail.python.org/mailman/listinfo/python-list

Re: Will Python 3.0 remove the global interpreter lock (GIL)

[Grant Edwards]

On 2007-09-20, TheFlyingDutchman [EMAIL PROTECTED] wrote:

 Is the only point in getting rid of the GIL to allow multi-threaded
 applications?

That's the main point.

 Can't multiple threads also provide a performance boost versus
 multiple processes on a single-core machine?

That depends on the algorithm, the code, and the
synchronization requirements.

 OK, have to agree. Sounds like it could be a good candidate
 for a fork. One question - is it a computer science maxim that
 an interpreter that implements multi-threading will always be
 slower when running single threaded apps?

I presume you're referring to Amdahl's law.

  http://en.wikipedia.org/wiki/Amdahl's_law

Remember there are reasons other than speed on a
multi-processor platorm for wanting to do multi-threading.
Sometimes it just maps onto the application better than
a single-threaded solution.

-- 
Grant Edwards   grante Yow! I want you to MEMORIZE
  at   the collected poems of
   visi.comEDNA ST VINCENT MILLAY
   ... BACKWARDS!!
-- 
http://mail.python.org/mailman/listinfo/python-list

Re: Will Python 3.0 remove the global interpreter lock (GIL)

[Paul Rubin]

Steven D'Aprano [EMAIL PROTECTED] writes:
 That's why your comparatively wimpy site preferred to throw extra web 
 servers at the job of serving webpages rather than investing in smarter, 
 harder-working programmers to pull the last skerricks of performance out 
 of the hardware you already had.

The compute intensive stuff (image rendering and crunching) has
already had most of those skerricks pulled out.  It is written in C
and assembler (not by us).  Only a small part of our stuff is written
in Python: it just happens to be the part I'm involved with.

 But Python speed ups don't come for free. For instance, I'd *really* 
 object if Python ran twice as fast for users with a quad-core CPU, but 
 twice as slow for users like me with only a dual-core CPU.

Hmm.  Well if the tradeoff were selectable at python configuration
time, then this option would certainly be worth doing.  You might not
have a 4-core cpu today but you WILL have one soon.

 What on earth makes you think that would be anything more than a 
 temporary, VERY temporary, shutdown? My prediction is that the last of 
 the machines wouldn't have even been unplugged

Of course that example was a reductio ad absurdum.  In reality they'd
use the speedup to compute 2x as much stuff, rather than ever powering
any servers down.  Getting the extra computation is more valuable than
saving the electricity.  It's just easier to put a dollar value on
electricity than on computation in an example like this.  It's also
the case for our specfiic site that our server cluster is in large
part a disk farm and not just a compute farm, so even if we sped up
the software infinitely we'd still need a lot of boxes to bolt the
disks into and keep them spinning.

 Now there's a thought... given that Google:
 
 (1) has lots of money;
 (2) uses Python a lot;
 (3) already employs both Guido and (I think...) Alex Martelli and 
 possibly other Python gurus;
 (4) is not shy in investing in Open Source projects;
 (5) and most importantly uses technologies that need to be used across 
 multiple processors and multiple machines
 
 one wonders if Google's opinion of where core Python development needs to 
 go is the same as your opinion?

I think Google's approach has been to do cpu-intensive tasks in other
languages, primarily C++.  It would still be great if they put some
funding into PyPy development, since I think I saw something about the
EU funding being interrupted.
-- 
http://mail.python.org/mailman/listinfo/python-list

Re: Will Python 3.0 remove the global interpreter lock (GIL)

[Chris Mellon]

On 20 Sep 2007 07:43:18 -0700, Paul Rubin
http://phr.cx@nospam.invalid wrote:
 Steven D'Aprano [EMAIL PROTECTED] writes:
  That's why your comparatively wimpy site preferred to throw extra web
  servers at the job of serving webpages rather than investing in smarter,
  harder-working programmers to pull the last skerricks of performance out
  of the hardware you already had.

 The compute intensive stuff (image rendering and crunching) has
 already had most of those skerricks pulled out.  It is written in C
 and assembler (not by us).  Only a small part of our stuff is written
 in Python: it just happens to be the part I'm involved with.


That means that this part is also unaffected by the GIL.

  But Python speed ups don't come for free. For instance, I'd *really*
  object if Python ran twice as fast for users with a quad-core CPU, but
  twice as slow for users like me with only a dual-core CPU.

 Hmm.  Well if the tradeoff were selectable at python configuration
 time, then this option would certainly be worth doing.  You might not
 have a 4-core cpu today but you WILL have one soon.

  What on earth makes you think that would be anything more than a
  temporary, VERY temporary, shutdown? My prediction is that the last of
  the machines wouldn't have even been unplugged

 Of course that example was a reductio ad absurdum.  In reality they'd
 use the speedup to compute 2x as much stuff, rather than ever powering
 any servers down.  Getting the extra computation is more valuable than
 saving the electricity.  It's just easier to put a dollar value on
 electricity than on computation in an example like this.  It's also
 the case for our specfiic site that our server cluster is in large
 part a disk farm and not just a compute farm, so even if we sped up
 the software infinitely we'd still need a lot of boxes to bolt the
 disks into and keep them spinning.


I think this is instructive, because it's pretty typical of GIL
complaints. Someone gives an example where the GIL is limited, but
upon inspection it turns out that the actual bottleneck is elsewhere,
that the GIL is being sidestepped anyway, and that the supposed
benefits of removing the GIL wouldn't materialize because the problem
space isn't really as described.

  Now there's a thought... given that Google:
 
  (1) has lots of money;
  (2) uses Python a lot;
  (3) already employs both Guido and (I think...) Alex Martelli and
  possibly other Python gurus;
  (4) is not shy in investing in Open Source projects;
  (5) and most importantly uses technologies that need to be used across
  multiple processors and multiple machines
 
  one wonders if Google's opinion of where core Python development needs to
  go is the same as your opinion?

 I think Google's approach has been to do cpu-intensive tasks in other
 languages, primarily C++.  It would still be great if they put some
 funding into PyPy development, since I think I saw something about the
 EU funding being interrupted.
 --

At the really high levels of scalability, such as across a server
farm, threading is useless. The entire point of threads, rather than
processes, is that you've got shared, mutable state. A shared nothing
process (or Actor, if you will) model is the only one that makes sense
if you really want to scale because it's the only one that allows you
to distribute over machines. The fact that it also scales very well
over multiple cores (better than threads, in many cases) is just
gravy.

The only hard example I've seen given of the GIL actually limiting
scalability is on single server, high volume Django sites, and I don't
think that the architecture of those sites is very scalable anyway.
-- 
http://mail.python.org/mailman/listinfo/python-list

Re: Will Python 3.0 remove the global interpreter lock (GIL)

[Paul Rubin]

Chris Mellon [EMAIL PROTECTED] writes:
 No. Python has threads, and they're wrappers around true OS level
 system threads. What the GIL does is prevent *Python* code in those
 threads from running concurrently.

Well, C libraries can release the GIL if they are written for thread
safety, but as far as I know, most don't release it.  For example I
don't think cElementTree releases the GIL, and it's a huge CPU
consumer in some of the code I run, despite being written in C pretty
carefully.  Also, many of the most basic builtin types (such as dicts)
are implemented in C and don't release the GIL.

 Very, very few operations are CPU bound these days, and even fewer of
 the ones were Python is involved. The largest benefits to the desktop
 user of multiple cores are the increase in cross-process performance
 (multitasking), not single applications.

If you add up all the CPU cycles being used by Python everywhere,
I wonder how many of them are on desktops and how many are on servers.

 Python is not especially fast in terms of pure CPU time, so
 if you're CPU bound anyway moving your CPU bound code into C (or
 something else) is likely to reap far more benefits - and sidestepping
 the GIL in the process.

If moving code into C is so easy, why not move all the code there
instead of just the CPU-bound code?  Really, coding in C adds a huge
cost in complexity and unreliability.  Python makes life a lot better
for developers, and so reimplementing Python code in C should be seen
as a difficult desperation measure rather than an easy way to get
speedups.  Therefore, Python's slowness is a serious weakness and not
just a wart with an easy workaround.

 In short, I think any problem that would be directly addressed by
 removing the GIL is better addressed by other solutions.

It does sound like removing the GIL from CPython would have very high
costs in more than one area.  Is my hope that Python will transition
from CPython to PyPy overoptimistic?
-- 
http://mail.python.org/mailman/listinfo/python-list

Re: Will Python 3.0 remove the global interpreter lock (GIL)

[Paul Rubin]

Chris Mellon [EMAIL PROTECTED] writes:
  The compute intensive stuff (image rendering and crunching) has
  already had most of those skerricks pulled out.  It is written in C
  and assembler
 That means that this part is also unaffected by the GIL.

Right, it was a counterexample against the speed doesn't matter
meme, not specifically against the GIL.  And that code is fast because
someone undertook comparatively enormous effort to code it in messy,
unsafe languages instead of Python, because Python is so slow.

 At the really high levels of scalability, such as across a server
 farm, threading is useless. The entire point of threads, rather than
 processes, is that you've got shared, mutable state. A shared nothing
 process (or Actor, if you will) model is the only one that makes sense
 if you really want to scale because it's the only one that allows you
 to distribute over machines. The fact that it also scales very well
 over multiple cores (better than threads, in many cases) is just
 gravy.

In reality you want to organize the problem so that memory intensive
stuff is kept local, and that's where you want threads, to avoid the
communications costs of serializing stuff between processes, either
between boxes or between cores.  If communications costs could be
ignored there would be no need for gigabytes of ram in computers.
We'd just use disks for everything.  As it is, we use tons of ram,
most of which is usually twiddling its thumbs doing nothing (as DJ
Bernstein put it) because the cpu isn't addressing it at that instant.
The memory just sits there waiting for the cpu to access it.  We
actually can get better-than-linear speedups by designing the hardware
to avoid this.  See:
  http://cr.yp.to/snuffle/bruteforce-20050425.pdf
for an example.

 The only hard example I've seen given of the GIL actually limiting
 scalability is on single server, high volume Django sites, and I don't
 think that the architecture of those sites is very scalable anyway.

The stuff I'm doing now happens to work ok with multiple processes but
would have been easier to write with threads.
-- 
http://mail.python.org/mailman/listinfo/python-list

Re: Will Python 3.0 remove the global interpreter lock (GIL)

[Terry Reedy]

Paul Rubin http://phr.cx@NOSPAM.invalid wrote in message 
news:[EMAIL PROTECTED]
| funding into PyPy development, since I think I saw something about the
| EU funding being interrupted.

As far as I know, the project was completed and promised funds paid.  But I 
don't know of any major follow-on funding, which I am sure they could use. 



-- 
http://mail.python.org/mailman/listinfo/python-list

Re: Will Python 3.0 remove the global interpreter lock (GIL)

[Terry Reedy]

Paul Rubin http://phr.cx@NOSPAM.invalid wrote in message 
news:[EMAIL PROTECTED]
| It does sound like removing the GIL from CPython would have very high
| costs in more than one area.  Is my hope that Python will transition
| from CPython to PyPy overoptimistic?

I presume you mean 'will the leading edge reference version transition...
Or more plainly, will Guido switch to PyPy for further development of 
Python?  I once thought so, but 1) Google sped the arrival of Py3.0 by 
hiring Guido with a major chunk of time devoted to Python development, so 
he started before PyPy was even remotely ready (and it still is not); and 
2) PyPy did not focus only or specifically on being a CPython replacement 
but became an umbrella for a variety of experiment (including, for 
instance, a Scheme frontend). 



-- 
http://mail.python.org/mailman/listinfo/python-list

Re: Will Python 3.0 remove the global interpreter lock (GIL)

[Bruno Desthuilliers]

Ben Finney a écrit :
(snip)
 One common response to that is Processes are expensive on Win32. My
 response to that is that if you're programming on Win32 and expecting
 the application to scale well, you already have problems that must
 first be addressed that are far more fundamental than the GIL.

Lol ! +1 QOTW !

-- 
http://mail.python.org/mailman/listinfo/python-list

Re: Will Python 3.0 remove the global interpreter lock (GIL)

[Grant Edwards]

On 2007-09-19, Steven D'Aprano [EMAIL PROTECTED] wrote:
 On Tue, 18 Sep 2007 18:09:26 -0700, TheFlyingDutchman wrote:

 How much faster/slower would Greg Stein's code be on today's
 processors versus CPython running on the processors of the
 late 1990's? 

 I think a better question is, how much faster/slower would
 Stein's code be on today's processors, versus CPython being
 hand-simulated in a giant virtual machine made of clockwork?

That depends on whether you have the steam-driven model or the
water-wheel driven model.  The steam-drive one is quite a bit
faster once you get it going, but the water-wheel model has a
much shorter start-up time (though it is more weather
dependent).

-- 
Grant Edwards   grante Yow! Hey, waiter!  I want
  at   a NEW SHIRT and a PONY TAIL
   visi.comwith lemon sauce!
-- 
http://mail.python.org/mailman/listinfo/python-list

Re: Will Python 3.0 remove the global interpreter lock (GIL)

[Steven D'Aprano]

On Tue, 18 Sep 2007 18:09:26 -0700, TheFlyingDutchman wrote:

 How much faster/slower would Greg Stein's code be on today's processors
 versus CPython running on the processors of the late 1990's? 

I think a better question is, how much faster/slower would Stein's code 
be on today's processors, versus CPython being hand-simulated in a giant 
virtual machine made of clockwork?



-- 
Steven.
-- 
http://mail.python.org/mailman/listinfo/python-list

Re: Will Python 3.0 remove the global interpreter lock (GIL)

[TheFlyingDutchman]

On Sep 19, 8:51 am, Steven D'Aprano [EMAIL PROTECTED]
cybersource.com.au wrote:
 On Tue, 18 Sep 2007 18:09:26 -0700, TheFlyingDutchman wrote:
  How much faster/slower would Greg Stein's code be on today's processors
  versus CPython running on the processors of the late 1990's?

 I think a better question is, how much faster/slower would Stein's code
 be on today's processors, versus CPython being hand-simulated in a giant
 virtual machine made of clockwork?

 --
 Steven.

Steven, You forgot this part:

And if you decide to answer, please add a true/false response
to this statement - CPython in the late 1990's ran too slow'.


-- 
http://mail.python.org/mailman/listinfo/python-list

Re: Will Python 3.0 remove the global interpreter lock (GIL)

[Paul Boddie]

On 19 Sep, 03:09, TheFlyingDutchman [EMAIL PROTECTED] wrote:

 How much faster/slower would Greg Stein's code be on today's
 processors versus CPython running on the processors of the late
 1990's? And if you decide to answer, please add a true/false response
 to this statement - CPython in the late 1990's ran too slow.

Too slow for what? And what's the fixation with CPython, anyway? Other
implementations of Python 2.x don't have the GIL. Contrary to popular
folklore, Jython has been quite a reasonable implementation of Python
for about half as long as CPython has been around, if you don't mind
the JVM. I'm sure people have lots of complaints about Jython like
they do about CPython and the GIL, thinking that complaining about it
is going to make the situation better, or that they're imparting some
kind of wisdom to which the people who actually wrote the code must
be oblivious, but nobody is withholding the code from anyone who wants
to actually improve it.

And there we learn something: that plenty of people are willing to
prod others into providing them with something that will make their
lives better, their jobs easier, and their profits greater, but not so
many are interested in contributing back to the cause and taking on
very much of the work themselves. Anyway, the response to your
statement is false. Now you'll have to provide us with the insight
we're all missing. Don't disappoint!

Paul

-- 
http://mail.python.org/mailman/listinfo/python-list

Re: Will Python 3.0 remove the global interpreter lock (GIL)

[TheFlyingDutchman]

On Sep 19, 3:41 pm, Paul Boddie [EMAIL PROTECTED] wrote:
 On 19 Sep, 03:09, TheFlyingDutchman [EMAIL PROTECTED] wrote:



  How much faster/slower would Greg Stein's code be on today's
  processors versus CPython running on the processors of the late
  1990's? And if you decide to answer, please add a true/false response
  to this statement - CPython in the late 1990's ran too slow.

 Too slow for what? And what's the fixation with CPython, anyway? Other
 implementations of Python 2.x don't have the GIL. Contrary to popular
 folklore, Jython has been quite a reasonable implementation of Python
 for about half as long as CPython has been around, if you don't mind
 the JVM. I'm sure people have lots of complaints about Jython like
 they do about CPython and the GIL, thinking that complaining about it
 is going to make the situation better, or that they're imparting some
 kind of wisdom to which the people who actually wrote the code must
 be oblivious, but nobody is withholding the code from anyone who wants
 to actually improve it.


 And there we learn something: that plenty of people are willing to
 prod others into providing them with something that will make their
 lives better, their jobs easier, and their profits greater, but not so
 many are interested in contributing back to the cause and taking on
 very much of the work themselves. Anyway, the response to your
 statement is false. Now you'll have to provide us with the insight
 we're all missing. Don't disappoint!

 Paul

Paul it's a pleasure to see that you are not entirely against
complaints.

The very fastest Intel processor of the last 1990's that I found came
out in October 1999 and had a speed around 783Mhz. Current fastest
processors are something like 3.74 Ghz, with larger caches. Memory is
also faster and larger. It appears that someone running a non-GIL
implementation of CPython today would have significantly faster
performance than a GIL CPython implementation of the late 1990's.
Correct me if I am wrong, but it seems that saying non-GIL CPython is
too slow, while once valid, has become invalid due to the increase in
computing power that has taken place.

-- 
http://mail.python.org/mailman/listinfo/python-list

Re: Will Python 3.0 remove the global interpreter lock (GIL)

[Terry Reedy]

Terry Reedy [EMAIL PROTECTED] wrote in message 
news:[EMAIL PROTECTED]
|
| TheFlyingDutchman [EMAIL PROTECTED] wrote in message
| news:[EMAIL PROTECTED]

| Since Guido wrote that, there have been put forth more ideas and interest
| and promises of efforts to remove or revise the GIL or do other things to
| make using multiple cores easier.  (The later being the point of the
| concern over GIL.)

A few days ago, an undergraduate posted on the dev list that he just 
started an independent study project on removing the GIL.  Maybe we'll get 
a report early next year.

Guido also said that he is willing to make changes to the CPython internals 
to aid multiproccessor usage [as long, presumably, as it does not cut speed 
in half].

|| How much faster/slower would Greg Stein's code be on today's
|| processors versus CPython running on the processors of the late
|| 1990's?
|
| Perhaps a bit faster, though processor speeds have not increased so much
| the last couple of years.

This assumes that comparing versions of 1.5 is still relevant.  As far as I 
know, his patch has not been maintained to apply against current Python. 
This tells me that no one to date really wants to dump the GIL at the cost 
of half Python's speed.  Of course not.  The point of dumping the GIL is to 
use multiprocessors to get more speed!  So with two cores and extra 
overhead, Stein-patched 1.5 would not even break even.

Quad (and more) cores are a different matter.  Hence, I think, the 
resurgence of interest.

||And if you decide to answer, please add a true/false response
|| to this statement - CPython in the late 1990's ran too slow.
|
| False by late 1990's standards, True by today's standards ;-).

So now this question for you:  CPython 2.5 runs too slow in 2007: true or 
false?

If you answer false, then there is no need for GIL removal.
If you answer true, then cutting its speed for 90+% of people is bad.

| Most people are not currently bothered by the GIL and would not want its
| speed halved.

And another question: why should such people spend time they do not have to 
make Python worse for themselves?

Terry Jan Reedy




-- 
http://mail.python.org/mailman/listinfo/python-list

Re: Will Python 3.0 remove the global interpreter lock (GIL)

[Steven D'Aprano]

On Wed, 19 Sep 2007 11:07:48 -0700, TheFlyingDutchman wrote:

 On Sep 19, 8:51 am, Steven D'Aprano [EMAIL PROTECTED]
 cybersource.com.au wrote:
 On Tue, 18 Sep 2007 18:09:26 -0700, TheFlyingDutchman wrote:
  How much faster/slower would Greg Stein's code be on today's
  processors versus CPython running on the processors of the late
  1990's?

 I think a better question is, how much faster/slower would Stein's code
 be on today's processors, versus CPython being hand-simulated in a
 giant virtual machine made of clockwork?

 --
 Steven.
 
 Steven, You forgot this part:
 
 And if you decide to answer, please add a true/false response to this
 statement - CPython in the late 1990's ran too slow'.


No, I ignored it, because it doesn't have a true/false response. It's a 
malformed request. Too slow for what task? Compared to what 
alternative? Fast and slow are not absolute terms, they are relative. A 
sloth is fast compared to continental drift, but slow compared to the 
space shuttle.

BUT even if we all agreed that CPython was (or wasn't) too slow in the 
late 1990s, why on earth do you imagine that is important? It is no 
longer the late 1990s, it is now 2007, and we are not using Python 1.4 
any more.



-- 
Steven.
-- 
http://mail.python.org/mailman/listinfo/python-list

Re: Will Python 3.0 remove the global interpreter lock (GIL)

[Steven D'Aprano]

On Wed, 19 Sep 2007 15:59:59 -0700, TheFlyingDutchman wrote:

 Paul it's a pleasure to see that you are not entirely against
 complaints.

I'm not against complaints either, so long as they are well-thought out. 
I've made a few of my own over the years, some of which may have been 
less well-thought out than others.


 The very fastest Intel processor of the last 1990's that I found came
 out in October 1999 and had a speed around 783Mhz. Current fastest
 processors are something like 3.74 Ghz, with larger caches. Memory is
 also faster and larger. It appears that someone running a non-GIL
 implementation of CPython today would have significantly faster
 performance than a GIL CPython implementation of the late 1990's.

That's an irrelevant comparison. It's a STUPID comparison. The two 
alternatives aren't non-GIL CPython on 2007 hardware versus GIL 
CPython on 1999 hardware because we aren't using GIL CPython on 1999 
hardware, we're using it on 2007 hardware. *That's* the alternative to 
the non-GIL CPython that you need to compare against.

Why turn your back on eight years of faster hardware? What's the point of 
getting rid of the GIL unless it leads to faster code? Get the speed and 
performance of 1999 today! doesn't seem much of a selling point in 2007.


 Correct me if I am wrong, but it seems that saying non-GIL CPython is
 too slow, while once valid, has become invalid due to the increase in
 computing power that has taken place.

You're wrong, because the finishing line has shifted -- performance we 
were satisfied with in 1998 would be considered unbearable to work with 
in 2007.

I remember in 1996 (give or take a year) being pleased that my new 
computer allowed my Pascal compiler to compile a basic, bare-bones GUI 
text editor in a mere two or four hours, because it used to take up to 
half a day on my older computer. Now, I expect to compile a basic text 
editor in minutes, not hours.

According to http://linuxreviews.org/gentoo/compiletimes/

the whole of Openoffice-ximian takes around six hours to compile. Given 
the speed of my 1996 computer, it would probably take six YEARS to 
compile something of Openoffice's complexity.


As a purely academic exercise, we might concede that the non-GIL version 
of CPython 1.5 running on a modern, dual-core CPU with lots of RAM will 
be faster than CPython 2.5 running on an eight-year old CPU with minimal 
RAM. But so what? That's of zero practical interest for anyone running 
CPython 2.5 on a modern PC.

If you are running a 1999 PC, your best bet is to stick with the standard 
CPython 1.5 including the GIL, because it is faster than the non-GIL 
version.

If you are running a 2007 PC, your best bet is *still* to stick with the 
standard CPython (version 2.5 now, not 1.5), because it will still be 
faster than the non-GIL version (unless you have four or more processors, 
and maybe not even then).

Otherwise, there's always Jython or IronPython.



-- 
Steven.
-- 
http://mail.python.org/mailman/listinfo/python-list

Re: Will Python 3.0 remove the global interpreter lock (GIL)

[Paul Rubin]

TheFlyingDutchman [EMAIL PROTECTED] writes:
 The very fastest Intel processor of the last 1990's that I found came
 out in October 1999 and had a speed around 783Mhz. Current fastest
 processors are something like 3.74 Ghz, with larger caches. Memory is
 also faster and larger. It appears that someone running a non-GIL
 implementation of CPython today would have significantly faster
 performance than a GIL CPython implementation of the late 1990's.
 Correct me if I am wrong, but it seems that saying non-GIL CPython is
 too slow, while once valid, has become invalid due to the increase in
 computing power that has taken place.

This reasoning is invalid.  For one thing, disk and memory sizes and
network bandwith have increased by a much larger factor than CPU speed
since the late 1990's.  A big disk drive in 1999 was maybe 20gb; today
it's 750gb, almost 40x larger, way outstripping the 5x cpu mhz
increase.  A fast business network connection was a 1.4 mbit/sec T-1
line, today it's often 100 mbit or more, again far oustripping CPU
mhz.  If Python was just fast enough to firewall your T1 net
connection or index your 20gb hard drive in 1999, it's way too slow to
do the same with today's net connections and hard drives, just because
of that change in the hardware landscape.  We have just about stopped
seeing increases in cpu mhz: that 3.74ghz speed was probably reached a
couple years ago.  We get cpu speed increases now through parallelism,
not mhz.  Intel and AMD both have 4-core cpu's now and Intel has a
16-core chip coming.  Python is at a serious disadvantage compared
with other languages if the other languages keep up with developments
and Python does not.

Also, Python in the late 90's was pitched as a scripting language,
intended for small throwaway tasks, while today it's used for complex
applications, and the language has evolved accordingly.  CPython is
way behind the times, not only from the GIL, but because its slow
bytecode interpreter, its non-compacting GC, etc.  The platitude that
performance doesn't matter, that programmer time is more valuable than
machine time, etc. is at best an excuse for laziness.  And more and
more often, in the application areas where Python is deployed, it's
just plain wrong.  Take web servers: a big site like Google has
something like a half million of them.  Even the comparatively wimpy
site where I work has a couple thousand.  If each server uses 150
watts of power (plus air conditioning), then if making the software 2x
faster lets us shut down 1000 of them, the savings in electricity
bills alone is larger than my salary.  Of course that doesn't include
environmental benefits, hardware and hosting costs, the costs and
headaches of administering that many boxes, etc.  For a lot of Python
users, significant speedups are a huge win.

However, I don't think fixing CPython (through GIL removal or anything
else) is the answer, and Jython isn't the answer either.  Python's
future is in PyPy, or should be.  Why would a self-respecting Python
implementation be written in (yikes) C or (yucch) Java, if it can be
written in Python?  So I hope that PyPy's future directions include
true parallelism.
-- 
http://mail.python.org/mailman/listinfo/python-list

Re: Will Python 3.0 remove the global interpreter lock (GIL)

[Steven D'Aprano]

On Wed, 19 Sep 2007 19:14:39 -0700, Paul Rubin wrote:

 We get cpu speed increases now through parallelism, not mhz.  Intel and
 AMD both have 4-core cpu's now and Intel has a 16-core chip coming. 
 Python is at a serious disadvantage compared with other languages if the
 other languages keep up with developments and Python does not.

I think what you mean to say is that Python _will be_ at a serious 
disadvantage if other languages keep up and Python doesn't. Python can't 
be at a disadvantage _now_ because of what happens in the future.

Although, with the rapid take-up of multi-core CPUs, the future is 
*really close*, so I welcome the earlier comment from Terry Reedy that 
Guido has said he is willing to make changes to the CPython internals to 
support multiprocessors, and that people have begun to investigate 
practical methods of removing the GIL (as opposed to just bitching about 
it for the sake of bitching).


 The platitude that performance doesn't matter

Who on earth says that? I've never heard anyone say that.

What I've heard people say is that _machine_ performance isn't the only 
thing that needs to be maximized, or even the most important thing. 
Otherwise we'd all be writing hand-optimized assembly language, and there 
would be a waiting line of about five years to get access to the few 
programmers capable of writing that hand-optimized assembly language.


 that programmer time is more valuable than machine time

Programmer time is more valuable than machine time in many cases, 
especially when tasks are easily parallisable across many machines. 
That's why your comparatively wimpy site preferred to throw extra web 
servers at the job of serving webpages rather than investing in smarter, 
harder-working programmers to pull the last skerricks of performance out 
of the hardware you already had.


 etc. is at best an excuse for laziness.

What are you doing about solving the problem? Apart from standing on the 
side-lines calling out Get yer lazy behinds movin', yer lazy bums!!! at 
the people who aren't even convinced there is a problem that needs 
solving?


 And more and more often, in the
 application areas where Python is deployed, it's just plain wrong.  Take
 web servers: a big site like Google has something like a half million of
 them.  Even the comparatively wimpy site where I work has a couple
 thousand.  If each server uses 150 watts of power (plus air
 conditioning), then if making the software 2x faster lets us shut down
 1000 of them, 

What on earth makes you think that would be anything more than a 
temporary, VERY temporary, shutdown? My prediction is that the last of 
the machines wouldn't have even been unplugged before management decided 
that running twice as fast, or servicing twice as many people at the same 
speed, is more important than saving on the electricity bill, and they'd 
be plugged back in.


 the savings in electricity bills alone is larger than my
 salary.  Of course that doesn't include environmental benefits, hardware
 and hosting costs, the costs and headaches of administering that many
 boxes, etc.  For a lot of Python users, significant speedups are a huge
 win.

Oh, I wouldn't say No thanks! to a Python speed up. My newest PC has a 
dual-core CPU (no cutting edge for me...) and while Python is faster on 
it than it was on my old PC, it isn't twice as fast.

But Python speed ups don't come for free. For instance, I'd *really* 
object if Python ran twice as fast for users with a quad-core CPU, but 
twice as slow for users like me with only a dual-core CPU.

I'd also object if the cost of Python running twice as fast was for the 
startup time to quadruple, because I already run a lot of small scripts 
where the time to launch the interpreter is a significant fraction of the 
total run time. If I wanted something like Java, that runs fast once it 
is started but takes a LONG time to actually start, I know where to find 
it.

I'd also object if the cost of Python running twice as fast was for Guido 
and the rest of the Python-dev team to present me with their wages bill 
for six months of development. I'm grateful that somebody is paying their 
wages, but if I had to pay for it myself it wouldn't be done. It simply 
isn't that important to me (and even if it was, I couldn't afford it).

Now there's a thought... given that Google:

(1) has lots of money;
(2) uses Python a lot;
(3) already employs both Guido and (I think...) Alex Martelli and 
possibly other Python gurus;
(4) is not shy in investing in Open Source projects;
(5) and most importantly uses technologies that need to be used across 
multiple processors and multiple machines

one wonders if Google's opinion of where core Python development needs to 
go is the same as your opinion?



-- 
Steven.
-- 
http://mail.python.org/mailman/listinfo/python-list

Re: Will Python 3.0 remove the global interpreter lock (GIL)

[TheFlyingDutchman]

On Sep 19, 8:54 pm, Steven D'Aprano [EMAIL PROTECTED]
cybersource.com.au wrote:
 On Wed, 19 Sep 2007 19:14:39 -0700, Paul Rubin wrote:



  etc. is at best an excuse for laziness.

 What are you doing about solving the problem? Apart from standing on the
 side-lines calling out Get yer lazy behinds movin', yer lazy bums!!! at
 the people who aren't even convinced there is a problem that needs
 solving?

He's trying to convince the developers that there is a problem. That
is not the same as your strawman argument.


  And more and more often, in the
  application areas where Python is deployed, it's just plain wrong.  Take
  web servers: a big site like Google has something like a half million of
  them.  Even the comparatively wimpy site where I work has a couple
  thousand.  If each server uses 150 watts of power (plus air
  conditioning), then if making the software 2x faster lets us shut down
  1000 of them,

 What on earth makes you think that would be anything more than a
 temporary, VERY temporary, shutdown? My prediction is that the last of
 the machines wouldn't have even been unplugged before management decided
 that running twice as fast, or servicing twice as many people at the same
 speed, is more important than saving on the electricity bill, and they'd
 be plugged back in.

Plugging back in 1000 servers would be preferable to buying and
plugging in 2000 new servers which is what would occur if the software
in this example had not been sped up 2x and management had still
desired a 2x speed up in system performance as you suggest.

-- 
http://mail.python.org/mailman/listinfo/python-list

Re: Will Python 3.0 remove the global interpreter lock (GIL)

[TheFlyingDutchman]

On Sep 19, 5:08 pm, Terry Reedy [EMAIL PROTECTED] wrote:
 Terry Reedy [EMAIL PROTECTED] wrote in message

This is a little confusing because google groups does not show your
original post (not uncommon for them to lose a post in a thread - but
somehow still reflect the fact that it exists in the total-posts
number that they display) that you are replying to.



 This assumes that comparing versions of 1.5 is still relevant.  As far as I
 know, his patch has not been maintained to apply against current Python.
 This tells me that no one to date really wants to dump the GIL at the cost
 of half Python's speed.  Of course not.  The point of dumping the GIL is to
 use multiprocessors to get more speed!  So with two cores and extra
 overhead, Stein-patched 1.5 would not even break even.

 Quad (and more) cores are a different matter.  Hence, I think, the
 resurgence of interest.

I am confused about the benefits/disadvantages of the GIL removal.
Is it correct that the GIL is preventing CPython from having threads?

Is it correct that the only issue with the GIL is the prevention of
being able to do multi-threading?

If you only planned on writing single-threaded applications would GIL-
removal have no benefit?

Can threading have a performance benefit on a single-core machine
versus  running multiple processes?

 So now this question for you:  CPython 2.5 runs too slow in 2007: true or
 false?

I guess I gotta go with Steven D'Aprano - both true and false
depending on your situation.

 If you answer false, then there is no need for GIL removal.

OK, I see that.

 If you answer true, then cutting its speed for 90+% of people is bad.

OK, seems reasonable, assuming that multi-threading cannot be
implemented without a performance hit on single-threaded applications.
Is that a computer science maxim - giving an interpreted language
multi-threading will always negatively impact the performance of
single-threaded applications?


 | Most people are not currently bothered by the GIL and would not want its
 | speed halved.

 And another question: why should such people spend time they do not have to
 make Python worse for themselves?

Saying they don't have time to make a change, any change, is always
valid in my book. I cannot argue against that. Ditto for them saying
they don't want to make a change with no explanation. But it seems if
they make statements about why a change is not good, then it is fair
to make a counter-argument. I do agree with the theme of Steven
D'Aprano's comments in that it should be a cordial counter-argument
and not a demand.


-- 
http://mail.python.org/mailman/listinfo/python-list

Re: Will Python 3.0 remove the global interpreter lock (GIL)

[TheFlyingDutchman]

On Sep 2, 5:38 pm, Eduardo O. Padoan [EMAIL PROTECTED]
wrote:
  No.http://www.artima.com/weblogs/viewpost.jsp?thread=211430

 Ops, I meant:http://www.artima.com/forums/threaded.jsp?forum=106thread=211200

 --http://www.advogato.org/person/eopadoan/
 Bookmarks:http://del.icio.us/edcrypt

No. We're not changing the CPython implementation much. Getting rid
of the GIL would be a massive rewrite of the interpreter because all
the internal data structures (and the reference counting operations)
would have to be made thread-safe. This was tried once before (in the
late '90s by Greg Stein) and the resulting interpreter ran twice as
slow.

How much faster/slower would Greg Stein's code be on today's
processors versus CPython running on the processors of the late
1990's? And if you decide to answer, please add a true/false response
to this statement - CPython in the late 1990's ran too slow.

-- 
http://mail.python.org/mailman/listinfo/python-list

Re: Will Python 3.0 remove the global interpreter lock (GIL)

[Terry Reedy]

TheFlyingDutchman [EMAIL PROTECTED] wrote in message 
news:[EMAIL PROTECTED]
| On Sep 2, 5:38 pm, Eduardo O. Padoan [EMAIL PROTECTED]
| wrote:
|   No.http://www.artima.com/weblogs/viewpost.jsp?thread=211430
| 
|  Ops, I 
meant:http://www.artima.com/forums/threaded.jsp?forum=106thread=211200
| 
|  --http://www.advogato.org/person/eopadoan/
|  Bookmarks:http://del.icio.us/edcrypt
|
| No. We're not changing the CPython implementation much. Getting rid
| of the GIL would be a massive rewrite of the interpreter because all
| the internal data structures (and the reference counting operations)
| would have to be made thread-safe. This was tried once before (in the
| late '90s by Greg Stein) and the resulting interpreter ran twice as
| slow.

Since Guido wrote that, there have been put forth more ideas and interest 
and promises of efforts to remove or revise the GIL or do other things to 
make using multiple cores easier.  (The later being the point of the 
concern over GIL.)

| How much faster/slower would Greg Stein's code be on today's
| processors versus CPython running on the processors of the late
| 1990's?

Perhaps a bit faster, though processor speeds have not increased so much 
the last couple of years.

|And if you decide to answer, please add a true/false response
| to this statement - CPython in the late 1990's ran too slow.

False by late 1990's standards, True by today's standards ;-).

Most people are not currently bothered by the GIL and would not want its 
speed halved.

In any case, any of the anti-GIL people are free to update Stein's code and 
distribute a GIl-less version of CPython.  (Or to use Jython or 
IronPython.)

tjr





-- 
http://mail.python.org/mailman/listinfo/python-list

Re: Will Python 3.0 remove the global interpreter lock (GIL)

[Evan Klitzke]

On Sun, 2007-09-02 at 17:21 -0700, llothar wrote:
 I'm afraid that the GIL is killing the usefullness of python for some
 types of applications now where 4,8 oder 64 threads on a chip are here
 or comming soon.
 
 What is the status about that for the future of python?

The GIL is an implementation specific issue with CPython. It will not be
removed in CPython for the forseeable future, but you can already get a
GIL-free interpreter with Jython and IronPython. AFAIK there are plans
to remove the GIL in PyPy.

According to the last PyPy release announcement, they're running at
about half the speed of CPython, and have a preliminary JIT that can
translate certain integer operations into assembly, and will be expanded
upon in future releases. If you're looking for a progressive alternative
to CPython, I'd keep an eye on that project ;-)

-- 
Evan Klitzke [EMAIL PROTECTED]

-- 
http://mail.python.org/mailman/listinfo/python-list

Re: Will Python 3.0 remove the global interpreter lock (GIL)

[Michele Simionato]

On Sep 3, 2:21 am, llothar [EMAIL PROTECTED] wrote:
 I'm afraid that the GIL is killing the usefullness of python for some
 types of applications now where 4,8 oder 64 threads on a chip are here
 or comming soon.

 What is the status about that for the future of python?

This is FAQ. You will find thousands of discussion on the net about
that.
My personal opinion (and I am not the only one in the Python
community) is that
if you want to scale the way to go is to use processes, not threads,
so removing the GIL would be a waste of effort anyway.
Look at the 'processing' module in PyPI.

 Michele Simionato

-- 
http://mail.python.org/mailman/listinfo/python-list

Re: Will Python 3.0 remove the global interpreter lock (GIL)

[king kikapu]

On Sep 3, 9:15 am, Michele Simionato [EMAIL PROTECTED]
wrote:
 On Sep 3, 2:21 am, llothar [EMAIL PROTECTED] wrote:


 My personal opinion (and I am not the only one in the Python
 community) is that
 if you want to scale the way to go is to use processes, not threads,
 so removing the GIL would be a waste of effort anyway.
 Look at the 'processing' module in PyPI.

  Michele Simionato


I second that. You may also look here,
http://www.parallelpython.com/

I tested it and work as expected. You can see all your processing-
cores work nicely and balanced.

-- 
http://mail.python.org/mailman/listinfo/python-list

Re: Will Python 3.0 remove the global interpreter lock (GIL)

[Ben Finney]

Michele Simionato [EMAIL PROTECTED] writes:

 On Sep 3, 2:21 am, llothar [EMAIL PROTECTED] wrote:
  I'm afraid that the GIL is killing the usefullness of python for
  some types of applications now where 4,8 oder 64 threads on a chip
  are here or comming soon.
 
 This is FAQ. You will find thousands of discussion on the net about
 that.

 My personal opinion (and I am not the only one in the Python
 community) is that if you want to scale the way to go is to use
 processes, not threads, so removing the GIL would be a waste of
 effort anyway.

Yes. Processes are cheap on well-designed operating systems, and using
processes to subdivide your processor usage encourages simple, modular
interfaces between parts of a program. Threads, while also cheap, are
much more difficult and fiddly to program correctly, hence are rarely
an overall win.

One common response to that is Processes are expensive on Win32. My
response to that is that if you're programming on Win32 and expecting
the application to scale well, you already have problems that must
first be addressed that are far more fundamental than the GIL.

-- 
 \Pinky, are you pondering what I'm pondering? Umm, I think |
  `\   so, Brain, but three men in a tub? Ooh, that's unsanitary!  -- |
_o__)_Pinky and The Brain_ |
Ben Finney
-- 
http://mail.python.org/mailman/listinfo/python-list

Re: Will Python 3.0 remove the global interpreter lock (GIL)

[king kikapu]

I was wondering (and maybe i still do) about this GIL problem. I am
relatively new to Python (less than a year) and when i started to
think about it i said: Oh, this IS a problem. But when i dig a
little more, i found that Ah, maybe it isn't.
I strongly believe that the best usage of multiple cores processor
will be achieved if programming languages are modified to support this
on their hearts. Code blocks that would be identified by the
compiler and run in parallel and such things. Laboratories are working
on these stuff but i do not expect something in the very-near future.

So, as i mentioned above, there are solutions for that right now
(parallel python and others) that enabled us with little effort to
spawn a new python interpreter, thus allowing the OS to schedule it on
a different core and do the job this way relatively cheap.
I wouldn't recommend going to IronPython despite the fact that the CLR
better utilize MP. The reason for this is that i would NEVER give up
the freedom that CPython gives me by exchange better usage of the MP
and platform lock-in.

-- 
http://mail.python.org/mailman/listinfo/python-list

Re: Will Python 3.0 remove the global interpreter lock (GIL)

[king kikapu]

I was wondering (and maybe i still do) about this GIL problem. I am
relatively new to Python (less than a year) and when i started to
think about it i said: Oh, this IS a problem. But when i dig a
little more, i found that Ah, maybe it isn't.
I strongly believe that the best usage of multiple cores processor
will be achieved if programming languages are modified to support this
on their hearts. Code blocks that would be identified by the
compiler and run in parallel and such things. Laboratories are working
on these stuff but i do not expect something in the very-near future.

So, as i mentioned above, there are solutions for that right now
(parallel python and others) that enabled us with little effort to
spawn a new python interpreter, thus allowing the OS to schedule it on
a different core and do the job this way relatively cheap.
I wouldn't recommend going to IronPython despite the fact that the CLR
better utilize MP. The reason for this is that i would NEVER give up
the freedom that CPython gives me by exchange better usage of the MP
and platform lock-in.

-- 
http://mail.python.org/mailman/listinfo/python-list

Re: Will Python 3.0 remove the global interpreter lock (GIL)

[king kikapu]

I was wondering (and maybe i still do) about this GIL problem. I am
relatively new to Python (less than a year) and when i started to
think about it i said: Oh, this IS a problem. But when i dig a
little more, i found that Ah, maybe it isn't.
I strongly believe that the best usage of multiple cores processor
will be achieved if programming languages are modified to support this
on their hearts. Code blocks that would be identified by the
compiler and run in parallel and such things. Laboratories are working
on these stuff but i do not expect something in the very-near future.

So, as i mentioned above, there are solutions for that right now
(parallel python and others) that enabled us with little effort to
spawn a new python interpreter, thus allowing the OS to schedule it on
a different core and do the job this way relatively cheap.
I wouldn't recommend going to IronPython despite the fact that the CLR
better utilize MP. The reason for this is that i would NEVER give up
the freedom that CPython gives me by exchange better usage of the MP
and platform lock-in.

-- 
http://mail.python.org/mailman/listinfo/python-list

Will Python 3.0 remove the global interpreter lock (GIL)

[llothar]

I'm afraid that the GIL is killing the usefullness of python for some
types of applications now where 4,8 oder 64 threads on a chip are here
or comming soon.

What is the status about that for the future of python?

I know that at the moment allmost nobody in the scripting world has
solved this problem, but it bites and it bites hard. Only groovy as a
Java Plugin has support but i never tried it. Writing an interpreter
that does MT this seems to be extremely difficult to do it right, with
lots of advanced stuff like CAS and lock free programming.

Even Smalltalk and Common Lisp didn't get it until know (with the
exception of certain experiments).

-- 
http://mail.python.org/mailman/listinfo/python-list

Re: Will Python 3.0 remove the global interpreter lock (GIL)

[Eduardo O. Padoan]

On 9/2/07, llothar [EMAIL PROTECTED] wrote:
 I'm afraid that the GIL is killing the usefullness of python for some
 types of applications now where 4,8 oder 64 threads on a chip are here
 or comming soon.

 What is the status about that for the future of python?

 I know that at the moment allmost nobody in the scripting world has
 solved this problem, but it bites and it bites hard. Only groovy as a
 Java Plugin has support but i never tried it. Writing an interpreter
 that does MT this seems to be extremely difficult to do it right, with
 lots of advanced stuff like CAS and lock free programming.

 Even Smalltalk and Common Lisp didn't get it until know (with the
 exception of certain experiments).


No. http://www.artima.com/weblogs/viewpost.jsp?thread=211430

-- 
http://www.advogato.org/person/eopadoan/
Bookmarks: http://del.icio.us/edcrypt
-- 
http://mail.python.org/mailman/listinfo/python-list

Re: Will Python 3.0 remove the global interpreter lock (GIL)

[Eduardo O. Padoan]

 No. http://www.artima.com/weblogs/viewpost.jsp?thread=211430

Ops, I meant:
http://www.artima.com/forums/threaded.jsp?forum=106thread=211200


-- 
http://www.advogato.org/person/eopadoan/
Bookmarks: http://del.icio.us/edcrypt
-- 
http://mail.python.org/mailman/listinfo/python-list

Re: Will Python 3.0 remove the global interpreter lock (GIL)

[llothar]

On 3 Sep., 07:38, Eduardo O. Padoan [EMAIL PROTECTED]
wrote:
  No.http://www.artima.com/weblogs/viewpost.jsp?thread=211430

 Ops, I meant:http://www.artima.com/forums/threaded.jsp?forum=106thread=211200


Thanks. I whish there would be a project for rewritting the C
interpreter
to make it better and more useable for threading use.

But the CPU infrastructure is also not perfect enough so maybe it's
good to
wait with this a few more years until Intel and AMD know what they are
doing.


-- 
http://mail.python.org/mailman/listinfo/python-list

Re: Will Python 3.0 remove the global interpreter lock (GIL)

[Luis M . González]

On Sep 2, 11:16 pm, llothar [EMAIL PROTECTED] wrote:
 On 3 Sep., 07:38, Eduardo O. Padoan [EMAIL PROTECTED]
 wrote:

   No.http://www.artima.com/weblogs/viewpost.jsp?thread=211430

  Ops, I 
  meant:http://www.artima.com/forums/threaded.jsp?forum=106thread=211200

 Thanks. I whish there would be a project for rewritting the C
 interpreter
 to make it better and more useable for threading use.

 But the CPU infrastructure is also not perfect enough so maybe it's
 good to
 wait with this a few more years until Intel and AMD know what they are
 doing.


I read somewhere that PYPY won't have the interpreter lock (I may be
wrong though).
Check it out: http://codespeak.net/pypy


-- 
http://mail.python.org/mailman/listinfo/python-list

Re: global interpreter lock

[Tommy . Ryding]

Thanks,
PyThread_release_lock() is implemented in all thread_xxx.h files.

So I thought right but specified my question like a stupid ;)

//T

-- 
http://mail.python.org/mailman/listinfo/python-list

global interpreter lock

[Tommy . Ryding]

I just need confirmation that I think right.

Is the files thread_xxx.h (xxx = nt, os2 or whatever) responsible for
the
global interpreter lock in a multithreaded environment?

I'm currently writing my own thread_VW for VxWorks, thats why I'm
asking. 

//Tommy

-- 
http://mail.python.org/mailman/listinfo/python-list

Re: global interpreter lock

[Jeremy Jones]

[EMAIL PROTECTED] wrote:

I just need confirmation that I think right.

Is the files thread_xxx.h (xxx = nt, os2 or whatever) responsible for
the
global interpreter lock in a multithreaded environment?

I'm currently writing my own thread_VW for VxWorks, thats why I'm
asking. 

//Tommy

  

Someone can correct me if I'm wrong, but the lock actually lives in 
ceval.c, around here:



802 PyThread_release_lock(interpreter_lock);
803
804 /* Other threads may run now */
805
806 PyThread_acquire_lock(interpreter_lock, 1);


This was taken from what appears to be a 2.4.1 release rather than a CVS 
checkout.  It looks like the PyThread_type_lock is defined in the 
thread_xxx.h files, though.

HTH,

- jmj
-- 
http://mail.python.org/mailman/listinfo/python-list

Re: global interpreter lock

[Michael Sparks]

Stephen Thorne wrote:

 On 15/09/05, Michael Sparks [EMAIL PROTECTED] wrote:
 At the moment, one option that springs to mind is this:
 yield WaitDataAvailable(inbox)
 
 Twisted supports this.
 
 help(twisted.internet.defer.waitForDeferred)

Thanks for this. I'll take a look and either we'll use that or we'll use
something that maps cleanly. (Reason for pause is because running on
mobiles is important to us.) Thanks for the example too :)

Best Regards,


Michael.
-- 
[EMAIL PROTECTED], http://kamaelia.sourceforge.net/ 
British Broadcasting Corporation, Research and Development
Kingswood Warren, Surrey KT20 6NP

This message (and any attachments) may contain personal views
which are not the views of the BBC unless specifically stated.

-- 
http://mail.python.org/mailman/listinfo/python-list

Re: global interpreter lock

[Michele Simionato]

It looks like I am reinventing Twisted and/or Kamelia.
This is code I wrote just today to simulate Python 2.5
generator in current Python:

import Queue

class coroutine(object):
def __init__(self, *args, **kw):
self.queue = Queue.Queue()
self.it = self.__cor__(*args, **kw)
def start(self):
return self.it.next()
def next(self):
return self.send(None)
def __iter__(self):
return self
def send(self, *args):
self.queue.put(args)
return self.it.next()
def recv(self):
return self.queue.get()
@classmethod
def generator(cls, gen):
return type(gen.__name__, (cls,), dict(__cor__=gen))

@coroutine.generator
def consumer(self, N):
for i in xrange(N):
yield i
cmd = self.recv()
if cmd == exit: break

c = consumer(100)
print c.start()
for cmd in [, , , , exit]:
print c.send(cmd)

 
Michele Simionato

-- 
http://mail.python.org/mailman/listinfo/python-list

Re: global interpreter lock

[Michael Sparks]

Michele Simionato wrote:

 It looks like I am reinventing Twisted and/or Kamaelia.

If it's /fun/ , is that a problem ? ;) (Interesting implementation BTW :)

FWIW, I've about a year ago it wasn't clear if we would be able to release
our stuff, so as part of a presentation I included a minimalistic decorator
based version of our system that has some similarities to yours. (The idea
was then at least the ideas had been shared, if not the main code - which
had approval)

Posted below in case it's of interest:

import copy
def wrapgenerator(bases=object, **attrs):
   def decorate(func):
   class statefulgenerator(bases):
  __doc__ = func.__doc__
  def __init__(self,*args):
   super(statefulgenerator, self) __init__(*args)
   self.func=func(self,*args) 
   for k in attrs.keys():
 self.__dict__[k] = copy.deepcopy(attrs[k])
   self.next=self.__iter__().next
  def __iter__(self): return iter(self.func)
   return statefulgenerator
   return decorate

class com(object):
  def __init__(_, *args):
 # Default queues
 _.queues = {inbox:[],control:[],
 outbox:[], signal:[]}
  def send(_,box,obj): _.queues[box].append(obj)
  def dataReady(_,box): return len(_.queues[box])0
  def recv(_, box): # NB. Exceptions aren't caught
X=_.queues[box][0]
del _.queues[box][0]
return X


A sample component written using this approach then looks like this:

@wrapgenerator(com)
def forwarder(self):
   Simple data forwarding generator
   while 1:
  if self.dataReady(inbox):
 self.send(outbox,self.recv(inbox))
  elif self.dataReady(control):
 if self.recv(control) == shutdown:
break
  yield 1
   self.send(signal,shutdown)
   yield 0

Since we're not actualy using this approach, there's likely to be border
issues here. I'm not actually sure I like this particular approach, but it
was an interesting experiment.

Best Regards,


Michael.

-- 
http://mail.python.org/mailman/listinfo/python-list

Re: global interpreter lock

[Michael Sparks]

Paul Rubin wrote:
...
 I don't see how generators substitute for microthreads.  In your example
 from another post:

I've done some digging and found what you mean by microthreads -
specifically I suspect you're referring to the microthreads package for
stackless? (I tend to view an activated generator as having a thread of
control, and since it's not a true thread, but is similar, I tend to view
that as a microthread. However your term and mine don't co-incide, and it
appears to cause confusion, so I'll switch my definition to match yours,
given the microthreads package, etc)

You're right, generators aren't a substitue for microthreads. However I do
see them as being a useful alternative to microthreads. Indeed the fact
that you're limited to a single stack frame I think has actually helped our
architecture.

The reason I say this is because it naturally encourages small components
which are highly focussed in what they do. For example, when I was
originally looking at how to wrap network handling up, it was logical to
want to do this: 

[ writing something probably implementable using greenlets, but definitely
  pseudocode ]

@Nestedgenerator
def runProtocol(...)
   while:
   data = get_data_from_connection( ... )

# Assume non-blocking socket
def get_data_from_connection(...)
try:
data = sock.recv()
return data
except ... :
Yield(WaitSocketDataReady(sock))
except ... :
return failure

Of something - you get the idea (the above code is naff, but that's because
it's late here) - the operation that would block normally you yield inside
until given a message.

The thing about this is that we wouldn't have resulted in the structure we
do have - which is to have components for dealing with connected sockets,
listening sockets and so on. We've been able to reuse the connected socket
code between systems much more cleanly that we would have done (I
suspect) than if we'd been able to nest yields (as I once asked about here)
or have true co-routines.

At some point it would be interesing to rewrite our entire system based on
greenlets and see if that works out with more or less reuse. (And more or
less ability to make code more parallel or not)


[re-arranging order slightly of comments ]
class encoder(component):
   def __init__(self, **args):
   self.encoder = unbreakable_encryption.encoder(**args)
   def main(self):
  while 1:
  if self.dataReady(inbox):
 data = self.recv(inbox)
 encoded = self.encoder.encode(data)
 self.send(encoded, outbox)
  yield 1
 
...
 In that particular example, the yield is only at the end, so the
 generator isn't doing anything that an ordinary function closure
 couldn't:
 
def main(self):
def run_event():
if self.dataReady(inbox):
   data = self.recv(inbox)
   encoded = self.encoder.encode(data)
   self.send(encoded, outbox)
return run_event

Indeed, in particular we can currently rewrite that particular example as:

class encoder(component):
   def __init__(self, **args):
   self.encoder = unbreakable_encryption.encoder(**args)
   def mainLoop(self):
  if self.dataReady(inbox):
 data = self.recv(inbox)
 encoded = self.encoder.encode(data)
 self.send(encoded, outbox)
  return 1

That's a bad example though. A more useful example is probably something
more like this:

class Multicast_sender(Axon.Component.component):
   def __init__(self, local_addr, local_port, remote_addr, remote_port):
   super(Multicast_sender, self).__init__()
   self.local_addr = local_addr
   self.local_port = local_port
   self.remote_addr = remote_addr
   self.remote_port = remote_port

   def main(self):
   sock = socket.socket(socket.AF_INET, socket.SOCK_DGRAM,
socket.IPPROTO_UDP)
   sock.bind((self.local_addr,self.local_port))
   sock.setsockopt(socket.IPPROTO_IP, socket.IP_MULTICAST_TTL, 10)
   while 1:
  if self.dataReady(inbox):
 data = self.recv()
 l = sock.sendto(data, (self.remote_addr,self.remote_port) );
  yield 1

With a bit of fun with decorators, that can actually be collapsed into
something more like:

@component
def Multicast_sender(self, local_addr, local_port, remote_addr,
remote_port):
   sock = socket.socket(socket.AF_INET, socket.SOCK_DGRAM,
socket.IPPROTO_UDP)
   sock.bind((self.local_addr,self.local_port))
   sock.setsockopt(socket.IPPROTO_IP, socket.IP_MULTICAST_TTL, 10)
   while 1:
  if self.dataReady(inbox):
 data = self.recv()
 l = sock.sendto(data, (self.remote_addr,self.remote_port) );
  yield 1





 You've got the main method creating a generator that has its own
 event loop that yields after each event it processes. 

Re: global interpreter lock

[Michael Sparks]

[ Second time lucky... ]
Paul Rubin wrote:
...
 I don't see how generators substitute for microthreads.  In your example
 from another post:

I've done some digging and found what you mean by microthreads -
specifically I suspect you're referring to the microthreads package for
stackless? (I tend to view an activated generator as having a thread of
control, and since it's not a true thread, but is similar, I tend to view
that as a microthread. However your term and mine don't co-incide, and it
appears to cause confusion, so I'll switch my definition to match yours,
given the microthreads package, etc)

The reason I say this is because it naturally encourages small components
which are highly focussed in what they do. For example, when I was
originally looking at how to wrap network handling up, it was logical to
want to do this: 

[ writing something probably implementable using greenlets, but definitely
  pseudocode ]

@Nestedgenerator
def runProtocol(...)
   while:
   data = get_data_from_connection( ... )

# Assume non-blocking socket
def get_data_from_connection(...)
try:
data = sock.recv()
return data
except ... :
Yield(WaitSocketDataReady(sock))
except ... :
return failure

Of something - you get the idea (the above code is naff, but that's because
it's late here) - the operation that would block normally you yield inside
until given a message.

The thing about this is that we wouldn't have resulted in the structure we
do have - which is to have components for dealing with connected sockets,
listening sockets and so on. We've been able to reuse the connected socket
code between systems much more cleanly that we would have done (I
suspect) than if we'd been able to nest yields (as I once asked about here)
or have true co-routines.

At some point it would be interesing to rewrite our entire system based on
greenlets and see if that works out with more or less reuse. (And more or
less ability to make code more parallel or not)

[re-arranging order slightly of comments ]
class encoder(component):
   def __init__(self, **args):
   self.encoder = unbreakable_encryption.encoder(**args)
   def main(self):
  while 1:
  if self.dataReady(inbox):
 data = self.recv(inbox)
 encoded = self.encoder.encode(data)
 self.send(encoded, outbox)
  yield 1
 
...
 In that particular example, the yield is only at the end, so the
 generator isn't doing anything that an ordinary function closure
 couldn't:
 
def main(self):
def run_event():
if self.dataReady(inbox):
   data = self.recv(inbox)
   encoded = self.encoder.encode(data)
   self.send(encoded, outbox)
return run_event


Indeed, in particular we can currently rewrite that particular example as:

class encoder(component):
   def __init__(self, **args):
   self.encoder = unbreakable_encryption.encoder(**args)
   def mainLoop(self):
  if self.dataReady(inbox):
 data = self.recv(inbox)
 encoded = self.encoder.encode(data)
 self.send(encoded, outbox)
  return 1

And that will work today. (We have a 3 callback form available for people
who aren't very au fait with generators, or are just more comfortable with
callbacks)


That's a bad example though. A more useful example is probably something
more like this: (changed example from accidental early post)
...
  center = list(self.rect.center)
  self.image = self.original
  current = self.image
  scale = 1.0
  angle = 1
  pos = center
  while 1:
self.image = current
if self.dataReady(imaging):
   self.image = self.recv(imaging)
   current = self.image
if self.dataReady(scaler):
   # Scaling
   scale = self.recv(scaler)
w,h = self.image.get_size()
self.image = pygame.transform.scale(self.image, (w*scale,
h*scale))
if self.dataReady(rotator):
   angle = self.recv(rotator)
   # Rotation
self.image = pygame.transform.rotate(self.image, angle)
if self.dataReady(translation):
   # Translation
   pos = self.recv(translation)
self.rect = self.image.get_rect()
self.rect.center = pos
yield 1


(this code is from Kamaelia.UI.Pygame.BasicSprite)

Can it be transformed to something event based? Yes of course. Is it clear
what's happening though? I would say yes. Currently we encourage the user
to look to see if data is ready before taking it, simply because it's the
simplest interface that we can guarantee consistency with.

For example, currently the exception based equivalent would be:
try:
pos = self.recv(translation)
except IndexError:
pass

Which 

Re: global interpreter lock

[Stephen Thorne]

On 15/09/05, Michael Sparks [EMAIL PROTECTED] wrote:
 At the moment, one option that springs to mind is this:
 yield WaitDataAvailable(inbox)

Twisted supports this.

help(twisted.internet.defer.waitForDeferred)

example usage is:

@deferredGenerator
def thingummy():
  thing = waitForDeferred(makeSomeRequestResultingInDeferred())
  yield thing
  thing = thing.getResult()
  print thing #the result! hoorj!

With the new generator syntax, it becomes somewhat less clunky,
allowing for the syntax:

@defgen
def foo():
   somereturnvalue = yield SomeLongRunningOperation()
   print somereturnvalue

http://svn.twistedmatrix.com/cvs/sandbox/radix/newdefgen.py?view=markuprev=14348

-- 
Stephen Thorne
Development Engineer
-- 
http://mail.python.org/mailman/listinfo/python-list

Re: global interpreter lock

[Paul Rubin]

Michael Sparks [EMAIL PROTECTED] writes:
  But I think to do it on Erlang's scale, Python needs user-level
  microthreads and not just OS threads.  
 
 You've just described Kamaelia* BTW, except substitute micro-thread
 with generator :-) (Also we call the queues outboxes and inboxes, and 
 the combination of a generator in a class with inboxes and outboxes
 components)
* http://kamaelia.sf.net/

I don't see how generators substitute for microthreads.  In your example
from another post:

   class encoder(component):
  def __init__(self, **args):
  self.encoder = unbreakable_encryption.encoder(**args)
  def main(self):
 while 1:
 if self.dataReady(inbox):
data = self.recv(inbox)
encoded = self.encoder.encode(data)
self.send(encoded, outbox)
 yield 1

You've got the main method creating a generator that has its own
event loop that yields after each event it processes.  Notice the kludge

 if self.dataReady(inbox):
data = self.recv(inbox)

instead of just saying something like:

data = self.get_an_event(inbox)

where .get_an_event blocks (i.e. yields) if no event is pending.
The reason for that is that Python generators aren't really coroutines
and you can't yield except from the top level function in the generator.

In that particular example, the yield is only at the end, so the
generator isn't doing anything that an ordinary function closure
couldn't:

   def main(self):
   def run_event():
   if self.dataReady(inbox):
  data = self.recv(inbox)
  encoded = self.encoder.encode(data)
  self.send(encoded, outbox)
   return run_event

Now instead of calling .next on a generator every time you want to let
your microthread run, just call the run_event function that main has
returned.  However, I suppose there's times when you'd want to read an
event, do something with it, yield, read another event, and do
something different with it, before looping.  In that case you can use
yields in different parts of that state machine.  But it's not that
big a deal; you could just use multiple functions otherwise.

All in all, maybe I'm missing something but I don't see generators as
being that much help here.  With first-class continuations like
Stackless used to have, the story would be different, of course.

-- 
http://mail.python.org/mailman/listinfo/python-list

Re: global interpreter lock

[Mike Meyer]

Dennis Lee Bieber [EMAIL PROTECTED] writes:
 On Wed, 31 Aug 2005 22:44:06 -0400, Mike Meyer [EMAIL PROTECTED] declaimed
 the following in comp.lang.python:
 I don't know what Ada offers. Java gives you pseudo-monitors. I'm

   From the days of mil-std 1815, Ada has supported tasks which
 communicate via rendezvous... The receiving task waits on an accept
 statement (simplified -- there is a means to wait on multiple different
 accepts, and/or time-out). The sending task calls the entry (looks
 like a regular procedure call with in and/or out parameters -- matches
 the signature of the waiting accept). As with accept, there are
 selective entry calls, wherein which ever task is waiting on the
 matching accept will be invoked. During the rendezvous, the sending
 task blocks until the receiving task exits the accept block -- at
 which point both tasks may proceed concurrently.

Thank you for providing the description. That was sufficient context
that Google found the GNAT documentation, which was very detailed.

Based on that, it seems that entry/accept are just a synchronization
construct - with some RPC semantics thrown in.

   As you might notice -- data can go both ways: in at the top of the
 rendezvous, and out at the end.

   Tasks are created by declaring them (there are also task types, so
 one can easily create a slew of identical tasks). 

 procedure xyz is

   a : task;   -- not real Ada, again, simplified
   b : task;

 begin -- the tasks begin execution here
   -- do stuff in the procedure itself, maybe call task entries
 end;

The problem is that this doesn't really provide any extra protection
for the programmer. You get language facilities that will provide the
protection, but the programmer has to remember to use them in every
case. If you forget to declare a method as protected, then nothing
stops two tasks from entering it and screwing up the objects data with
unsynchronized access. This should be compared to SCOOP, where trying
to do something like that is impossible.

   Thanks again,
   mike

-- 
Mike Meyer [EMAIL PROTECTED]  http://www.mired.org/home/mwm/
Independent WWW/Perforce/FreeBSD/Unix consultant, email for more information.
-- 
http://mail.python.org/mailman/listinfo/python-list

Re: global interpreter lock

[Bryan Olson]

Mike Meyer wrote:
  Bryan Olson writes:
 System support for threads has advanced far beyond what Mr. Meyer
 dealt with in programming the Amiga.
 
  I don't think it has - but see below.
 
 In industry, the two major camps are Posix threads, and Microsoft's
 Win32 threads (on NT or better).  Some commercial Unix vendors have
 mature support for Posix threads; on Linux, the NPTL is young but
 clearly the way to move forward.
 
  I haven't looked at Win32 threading. Maybe it's better than Posix
  threads. Sure, Posix threads is better than what I dealt with 10 years
  ago, but there's no way I'd call it advanced beyond that model. They
  aren't even as good as the Python Threading/Queue model.

Ever looked under the hood to see what happens when you wait
with a timeout on a Python queue/semaphore?

With Python threads/queues how do I wait for two queues (or
locks or semaphores) at one call? (I know some methods to
accomplish the same effect, but they suck.)

 Java and Ada will wrap the native thread package, which
 C(++) offers it directly.
 
  Obviously, any good solution will wrap the native threads [...]

I recommend looking at how software that implements
sophisticated services actually words. Many things one
might think to be obvious turn out not to be true.


-- 
--Bryan
-- 
http://mail.python.org/mailman/listinfo/python-list

Re: global interpreter lock

[Paul Rubin]

Mike Meyer [EMAIL PROTECTED] writes:
  Sure. I tried to be helpful there, but maybe I need to be more
  specific. The ref from my previous post, Google-able as The
  C10K problem is good but now a little dated.
 
 That appears to be a discussion on squeezing the most out of a network
 server, touching on threading models only so far as to mention what's
 available for popular languages on some popular server OS's.

The C10K paper is really interesting as are several of the papers that
it links to.  I didn't realize that opening a disk file couldn't be
done really asynchronously, though I'd consider that an OS bug.
Harder to deal with: if you mmap some data and touch an address that's
not in ram, it's inherently impossible (in the normal notion of user
processes) for the executing thread to continue doing any processing
while waiting for the page fault to be serviced.  

So if you're doing any significant amount of paging (such as using an
mmapped database with some infrequently accessed bits), you really
can't use your whole computer (all the cpu cycles) without multiple
threads or processes, even with a single processor.
-- 
http://mail.python.org/mailman/listinfo/python-list

Re: global interpreter lock

[Peter Hansen]

Dennis Lee Bieber wrote:
 On Thu, 01 Sep 2005 06:15:38 GMT, Bryan Olson [EMAIL PROTECTED]
 declaimed the following in comp.lang.python:
With Python threads/queues how do I wait for two queues (or
 
   Why have two queues? Use one queue and tag the items with the
 sender's id (or return queue).

I've faced the same issue, and it stems from having existing classes 
which do not already support this sort of mechanism.  Sure, you can 
sometimes design/redesign stuff to do exactly that, but much of the time 
you don't want to go rewriting some existing, stable code that waits on 
one queue just because you now need to throw another into the mix.

-Peter
-- 
http://mail.python.org/mailman/listinfo/python-list

Re: global interpreter lock

[Bryan Olson]

Dennis Lee Bieber wrote:
   Well, at that point, you could substitute waiting on a queue with
  waiting on a socket and still have the same problem -- regardless of
  the nature of the language/libraries for threading; it's a problem with
  the design of the classes as applied to a threaded environment.

It's a problem for all levels; pretty much any of them can botch
it. Not only do we want to be able to wait on two queues or two
sockets, we might have good cause to wait on two queues *and*
two sockets. Win32 provides WaitMultipleObjects which lets the
programmer efficiently wait on many objects of various types at
one call; the Linux kernel Gurus are looking at supporting a
similar feature.

Library designers can botch the functionality by wrapping the
waitable object in their own classes with their own special wait
operation, and hiding the OS object.


-- 
--Bryan
-- 
http://mail.python.org/mailman/listinfo/python-list

Re: global interpreter lock

[Mike Meyer]

Bryan Olson [EMAIL PROTECTED] writes:
 Mike Meyer wrote:
   Bryan Olson writes:
  System support for threads has advanced far beyond what Mr. Meyer
  dealt with in programming the Amiga.
  
   I don't think it has - but see below.
  
  In industry, the two major camps are Posix threads, and Microsoft's
  Win32 threads (on NT or better).  Some commercial Unix vendors have
  mature support for Posix threads; on Linux, the NPTL is young but
  clearly the way to move forward.
  
   I haven't looked at Win32 threading. Maybe it's better than Posix
   threads. Sure, Posix threads is better than what I dealt with 10 years
   ago, but there's no way I'd call it advanced beyond that model. They
   aren't even as good as the Python Threading/Queue model.
 With Python threads/queues how do I wait for two queues (or
 locks or semaphores) at one call? (I know some methods to
 accomplish the same effect, but they suck.)

By not as good as, I meant the model they provide isn't as managable
as the one provided by Queue/Threading. Like async I/O,
Queue/Threading provides a better model at the cost of
generality. There are things it doesn't do well, and others it doesn't
do at all. If it didn't have those problems, I wouldn't be looking for
alternatives.

  Java and Ada will wrap the native thread package, which
  C(++) offers it directly.
   Obviously, any good solution will wrap the native threads [...]
 I recommend looking at how software that implements
 sophisticated services actually words. Many things one
 might think to be obvious turn out not to be true.

Instead of making vague assertions, why don't you provide us with
facts? I.e. - what are the things you think are obvious that turned
out not to be true? Name some software that implements sophisticated
services that we can go look out. And so on...

 mike
-- 
Mike Meyer [EMAIL PROTECTED]  http://www.mired.org/home/mwm/
Independent WWW/Perforce/FreeBSD/Unix consultant, email for more information.
-- 
http://mail.python.org/mailman/listinfo/python-list

Re: global interpreter lock

[Bryan Olson]

Mike Meyer wrote:
  Bryan Olson writes:
 With Python threads/queues how do I wait for two queues (or
 locks or semaphores) at one call? (I know some methods to
 accomplish the same effect, but they suck.)
 
  By not as good as, I meant the model they provide isn't as managable
  as the one provided by Queue/Threading. Like async I/O,
  Queue/Threading provides a better model at the cost of
  generality.

I can't tell why you think that.

  Instead of making vague assertions, why don't you provide us
  with facts?

Yeah, I'll keep doing that. You're the one proclaiming a 'model'
to be more manageable with no evidence.

  I.e. - what are the things you think are obvious that turned
  out not to be true? Name some software that implements sophisticated
  services that we can go look out. And so on...

Thought we went over that. Look at the popular relational-
database engines. Implementing such a service with one line of
execution and async I/O is theoretically possible, but I've not
heard of anyone who has managed to do it. MySQL, PostgreSQL,
IBPhoenix, MaxDB, all have multiple simultaneous threads and/or
processes (as do the competitive commercial database engines,
though you can't look under the hood so easily).


-- 
--Bryan
-- 
http://mail.python.org/mailman/listinfo/python-list

Re: global interpreter lock

[[EMAIL PROTECTED]]

phil hunt wrote:
 Some times concurrency is the best (or only) way to do a job. Other
 times, it's more trouble than its worth. A good programmer will know
 which is which, and will not use an overly complex solution for the
 project he is writing.

Also, a good programmer won't conflate concurrency with threads.

-- 
http://mail.python.org/mailman/listinfo/python-list

Re: global interpreter lock

[Bryan Olson]

Bengt Richter wrote:
  Bryan Olson  wrote:
 For a long time,
 the most sophisticated software services generally have used
 multiple lines of execution, and now that's mostly in the form
 of threads. No one actually disagrees, but they go right on
 knocking the modern methods.
 
  I think Mike is asking for references/citations/links to the
  concurrency systems and modern methods you are talking about ;-)
  (I'd be interested too ;-)

Sure. I tried to be helpful there, but maybe I need to be more
specific. The ref from my previous post, Google-able as The
C10K problem is good but now a little dated. System support for
threads has advanced far beyond what Mr. Meyer dealt with in
programming the Amiga. In industry, the two major camps are
Posix threads, and Microsoft's Win32 threads (on NT or better).
Some commercial Unix vendors have mature support for Posix
threads; on Linux, the NPTL is young but clearly the way to move
forward. Java and Ada will wrap the native thread package, which
C(++) offers it directly. Microsoft's threading now works really
well. The WaitMultipleObjects idea is a huge winner.


-- 
--Bryan
-- 
http://mail.python.org/mailman/listinfo/python-list

Re: global interpreter lock

[Mike Meyer]

Bryan Olson [EMAIL PROTECTED] writes:
 Bengt Richter wrote:
   Bryan Olson  wrote:
  For a long time,
  the most sophisticated software services generally have used
  multiple lines of execution, and now that's mostly in the form
  of threads. No one actually disagrees, but they go right on
  knocking the modern methods.
   I think Mike is asking for references/citations/links to the
   concurrency systems and modern methods you are talking about ;-)
   (I'd be interested too ;-)
 Sure. I tried to be helpful there, but maybe I need to be more
 specific. The ref from my previous post, Google-able as The
 C10K problem is good but now a little dated.

That appears to be a discussion on squeezing the most out of a network
server, touching on threading models only so far as to mention what's
available for popular languages on some popular server OS's.

 System support for threads has advanced far beyond what Mr. Meyer
 dealt with in programming the Amiga.

I don't think it has - but see below.

 In industry, the two major camps are Posix threads, and Microsoft's
 Win32 threads (on NT or better).  Some commercial Unix vendors have
 mature support for Posix threads; on Linux, the NPTL is young but
 clearly the way to move forward.

I haven't looked at Win32 threading. Maybe it's better than Posix
threads. Sure, Posix threads is better than what I dealt with 10 years
ago, but there's no way I'd call it advanced beyond that model. They
aren't even as good as the Python Threading/Queue model.

 Java and Ada will wrap the native thread package, which
 C(++) offers it directly.

Obviously, any good solution will wrap the native threads
package. Just like it wraps the I/O package. That doesn't make the
native threads package good. You also have to deal with the cost the
compiler pays when you implement threading primitives as library
calls. I'm surprised the paper didn't mention that problem, as it
effects performance so directly.

I don't know what Ada offers. Java gives you pseudo-monitors. I'm
almost willing to call them advanced, but they are still don't
really help much. There are some good threading models
available. Posix threads isn't one of them, nor is Java's
synchronized.

mike
-- 
Mike Meyer [EMAIL PROTECTED]  http://www.mired.org/home/mwm/
Independent WWW/Perforce/FreeBSD/Unix consultant, email for more information.
-- 
http://mail.python.org/mailman/listinfo/python-list

Re: global interpreter lock

[phil hunt]

On Sun, 28 Aug 2005 20:34:07 GMT, Bryan Olson [EMAIL PROTECTED] wrote:
phil hunt wrote:
  Yes, find solutions. Don't find dangerous dead-ends that look like
  solutions but which will give you lots of trouble.

If concurrency is a dead end, why do the programs that provide
the most sophisticated services of any in the world rely on it
so heavily?

Some times concurrency is the best (or only) way to do a job. Other 
times, it's more trouble than its worth. A good programmer will know 
which is which, and will not use an overly complex solution for the 
project he is writing.

-- 
Email: zen19725 at zen dot co dot uk


-- 
http://mail.python.org/mailman/listinfo/python-list

Re: global interpreter lock

[phil hunt]

On Sun, 28 Aug 2005 19:25:55 -0400, Mike Meyer [EMAIL PROTECTED] wrote:
Bryan Olson [EMAIL PROTECTED] writes:
 phil hunt wrote:
   Yes, find solutions. Don't find dangerous dead-ends that look like
   solutions but which will give you lots of trouble.
 If concurrency is a dead end, why do the programs that provide
 the most sophisticated services of any in the world rely on it
 so heavily?

I don't know what Phil is saying, but I'm not calling concurrency a
dead end. 

In general it isn't. However, in many programs, it might be, in that 
by using it you might end up with a very complex program that fails
unpredictably and if hard to debug: if you get in that situation, 
you may have to start again, in which case your previous work will 
have been a dead end.

(Actually I would suggest that knowing when to throw something away 
and start again is something that differentiates between good and 
bad programmers).

I'm calling the tools available in most programming
languages for dealing with it primitive.

We need better tools.

I agree.

-- 
Email: zen19725 at zen dot co dot uk


-- 
http://mail.python.org/mailman/listinfo/python-list

Re: global interpreter lock

[Bengt Richter]

On Tue, 30 Aug 2005 05:15:34 GMT, Bryan Olson [EMAIL PROTECTED] wrote:

Mike Meyer wrote:
  Bryan Olson [EMAIL PROTECTED] writes:
   Bryan Olson writes:
   Trivially, an 'if' statement that depends upon input
  data is statically predictable. Use of async I/O means makes the
  programs execution dependent upon external timing.
 Mike Meyer wrote:
[...]

   [...] I'm calling the tools available in most programming
   languages for dealing with it primitive.
   We need better tools.
 Agreed, but if 'select' is someone's idea of the state of the
 art, they have little clue as to the tools already available.
 
  Well, share!

Uh, where have you been? I keep explaining that concurrency
systems have improved vastly in recent years. For a long time,
the most sophisticated software services generally have used
multiple lines of execution, and now that's mostly in the form
of threads. No one actually disagrees, but they go right on
knocking the modern methods.

I think Mike is asking for references/citations/links to the
concurrency systems and modern methods you are talking about ;-)
(I'd be interested too ;-)

Regards,
Bengt Richter
-- 
http://mail.python.org/mailman/listinfo/python-list

Re: global interpreter lock

[Mike Meyer]

[EMAIL PROTECTED] (Bengt Richter) writes:
 On Tue, 30 Aug 2005 05:15:34 GMT, Bryan Olson [EMAIL PROTECTED] wrote:
Mike Meyer wrote:
  Bryan Olson [EMAIL PROTECTED] writes:
   Bryan Olson writes:
   Trivially, an 'if' statement that depends upon input
  data is statically predictable. Use of async I/O means makes the
  programs execution dependent upon external timing.
 Mike Meyer wrote:
 [...]

   [...] I'm calling the tools available in most programming
   languages for dealing with it primitive.
   We need better tools.
 Agreed, but if 'select' is someone's idea of the state of the
 art, they have little clue as to the tools already available.
 
  Well, share!

Uh, where have you been? I keep explaining that concurrency
systems have improved vastly in recent years. For a long time,
the most sophisticated software services generally have used
multiple lines of execution, and now that's mostly in the form
of threads. No one actually disagrees, but they go right on
knocking the modern methods.

 I think Mike is asking for references/citations/links to the
 concurrency systems and modern methods you are talking about ;-)
 (I'd be interested too ;-)

Yup. I know systems are getting more concurrent. I also find that the
tools in popular languages for dealing with concurrency suck. I know
of some of these tools myself, but they either have restrictions on
the problems they can solve (like async I/O) or don't integrate well
with Python (like SCOOP).

So I'm definitely interested in learning about other alternatives!

  mike
-- 
Mike Meyer [EMAIL PROTECTED]  http://www.mired.org/home/mwm/
Independent WWW/Perforce/FreeBSD/Unix consultant, email for more information.
-- 
http://mail.python.org/mailman/listinfo/python-list

Re: global interpreter lock

[Bryan Olson]

Mike Meyer wrote:
  Bryan Olson writes:
  phil hunt wrote:
 
   What's important is *predictability*, e.g. which instruction will
   the computer execute next?
  
   If you only have one thread, you can tell by looking at the code
   what gets executed next. It's very simple.
 Not really. Trivially, an 'if' statement that depends upon input
 data is statically predictable. Use of async I/O means makes the
 programs execution dependent upon external timing.
 
 
  Yes, but that depenency is tied to a single point - the select
  call. The paths after that are statically predictable. This makes the
  code very managable.

Wow -- I could not disagree more. Returning back to some single
point for every possibly-blocking operation is painful to manage
even for simple GUIs, and humanly intractable for sophisticated
services.

Select is certainly useful, but it scales badly and isn't as
general as better tools.

  [...] I'm calling the tools available in most programming
  languages for dealing with it primitive.
 
  We need better tools.

Agreed, but if 'select' is someone's idea of the state of the
art, they have little clue as to the tools already available.
Bringing the tools to Python remains a bit of challenge, largely
because so many Pythoners are unaware.


-- 
--Bryan
-- 
http://mail.python.org/mailman/listinfo/python-list

Re: global interpreter lock

[Mike Meyer]

Bryan Olson [EMAIL PROTECTED] writes:
 Mike Meyer wrote:
   Bryan Olson writes:
   phil hunt wrote:
What's important is *predictability*, e.g. which instruction will
the computer execute next?
If you only have one thread, you can tell by looking at the code
what gets executed next. It's very simple.
  Not really. Trivially, an 'if' statement that depends upon input
  data is statically predictable. Use of async I/O means makes the
  programs execution dependent upon external timing.
   Yes, but that depenency is tied to a single point - the select
   call. The paths after that are statically predictable. This makes the
   code very managable.
 Wow -- I could not disagree more. Returning back to some single
 point for every possibly-blocking operation is painful to manage
 even for simple GUIs, and humanly intractable for sophisticated
 services.

I'd be interested in what you're trying to do that winds up as
unmanagable. There are clearly things select+async IO is unsuitable
for. You may be running into problems because you're trying to use it
in such an environment. For instance, it's not clear to me that it
will work well for any kind of GUI programming, though I've had good
look with it for command line interfaces.

 Select is certainly useful, but it scales badly and isn't as
 general as better tools.

It can't take advantage of multiple CPUs. I've not run into scaling
problems on single-CPU systems.

   [...] I'm calling the tools available in most programming
   languages for dealing with it primitive.
   We need better tools.
 Agreed, but if 'select' is someone's idea of the state of the
 art, they have little clue as to the tools already available.

Well, share! If you know of tools that make dealing with concurrent
code more manageable than an async I/O loop and have fewer
restrictions, I'm certainly interested in hearing about them.

 Bringing the tools to Python remains a bit of challenge, largely
 because so many Pythoners are unaware.

Well, the way to fix that is to talk about them!

  mike
-- 
Mike Meyer [EMAIL PROTECTED]  http://www.mired.org/home/mwm/
Independent WWW/Perforce/FreeBSD/Unix consultant, email for more information.
-- 
http://mail.python.org/mailman/listinfo/python-list

Re: global interpreter lock

[Bryan Olson]

Mike Meyer wrote:
  Bryan Olson [EMAIL PROTECTED] writes:
   Bryan Olson writes:
   Trivially, an 'if' statement that depends upon input
  data is statically predictable. Use of async I/O means makes the
  programs execution dependent upon external timing.
 Mike Meyer wrote:
   Yes, but that depenency is tied to a single point - the select
   call. The paths after that are statically predictable. This makes the
   code very managable.
 Wow -- I could not disagree more. Returning back to some single
 point for every possibly-blocking operation is painful to manage
 even for simple GUIs, and humanly intractable for sophisticated
 services.
 
  I'd be interested in what you're trying to do that winds up as
  unmanagable.

I'd like to call a utility in someone else's code, but it might
do something that could block. I find re-writing everyone's code
into a state-machine that can trap back to the central I/O loop
and later resume where it left off, to be hard to manage. When
I'm writing my own base classes, I find it hard to support the
back-to-the-I/O-loop-and-resume thing so that method over-rides
can call blocking operations when they need to.


  There are clearly things select+async IO is unsuitable
  for. You may be running into problems because you're trying to use it
  in such an environment. For instance, it's not clear to me that it
  will work well for any kind of GUI programming, though I've had good
  look with it for command line interfaces.

Uh, not sure where you're coming from there. Are you unaware of
the 'event loop' in the GUI's, or unaware that it's async I/O.

 Select is certainly useful, but it scales badly and isn't as
 general as better tools.
 
  It can't take advantage of multiple CPUs. I've not run into scaling
  problems on single-CPU systems.

Select() is linear-time in the number of sockets to be checked
(not just the number found to be ready). There's a good write-up
of the problem and solutions Google-able as The C10K problem.


   [...] I'm calling the tools available in most programming
   languages for dealing with it primitive.
   We need better tools.
 Agreed, but if 'select' is someone's idea of the state of the
 art, they have little clue as to the tools already available.
 
  Well, share!

Uh, where have you been? I keep explaining that concurrency
systems have improved vastly in recent years. For a long time,
the most sophisticated software services generally have used
multiple lines of execution, and now that's mostly in the form
of threads. No one actually disagrees, but they go right on
knocking the modern methods.


-- 
--Bryan
-- 
http://mail.python.org/mailman/listinfo/python-list

Re: global interpreter lock

[Bryan Olson]

phil hunt wrote:
  It's not the number of paths that's important.

Absolutely right. Non-trivial software always has too many paths
to consider them individually, so we have to reason generally.

  What's important is *predictability*, e.g. which instruction will
  the computer execute next?
 
  If you only have one thread, you can tell by looking at the code
  what gets executed next. It's very simple.

Not really. Trivially, an 'if' statement that depends upon input
data is statically predictable. Use of async I/O means makes the
programs execution dependent upon external timing.


  If you have 2 threads you can easily have a timing-based situation
  that occurs rarely but which causes your program to behave wierdly.
  This sort of bug is very hard to reproduce and therefore to fix.

So we need to learn to avoid it.


[...]
  Yes, find solutions. Don't find dangerous dead-ends that look like
  solutions but which will give you lots of trouble.

If concurrency is a dead end, why do the programs that provide
the most sophisticated services of any in the world rely on it
so heavily?


-- 
--Bryan
-- 
http://mail.python.org/mailman/listinfo/python-list

Re: global interpreter lock

[Bryan Olson]

Piet van Oostrum wrote:
 Paul Rubin http://[EMAIL PROTECTED] (PR) wrote:
 PR Really, the essence of programming is to find ways of organizing the
 PR program to stay reliable and maintainable in the face of that
 PR combinatorial explosion.  That means facing the problem and finding
 PR solutions, not running away.  The principle is no different for
 PR threads than it is for if statements.
 
  The principle is (more or less) similar, but for parallel programs it 
is an
  order of magnitude more complicated. Compare the correctness proofs of
  parallel programs with those of sequential programs.

That's an artifact of what the research community is trying to
accomplish with the proof. Proving non-trivial programs correct
is currently beyond the state of the art.


-- 
--Bryan
-- 
http://mail.python.org/mailman/listinfo/python-list

Re: global interpreter lock

[Mike Meyer]

Bryan Olson [EMAIL PROTECTED] writes:
phil hunt wrote:
   What's important is *predictability*, e.g. which instruction will
   the computer execute next?
  
   If you only have one thread, you can tell by looking at the code
   what gets executed next. It's very simple.
 Not really. Trivially, an 'if' statement that depends upon input
 data is statically predictable. Use of async I/O means makes the
 programs execution dependent upon external timing.

Yes, but that depenency is tied to a single point - the select
call. The paths after that are statically predictable. This makes the
code very managable.

   If you have 2 threads you can easily have a timing-based situation
   that occurs rarely but which causes your program to behave wierdly.
   This sort of bug is very hard to reproduce and therefore to fix.
 So we need to learn to avoid it.

No, we need tools that make it impossible to write codde that triggers
it. Async I/O is one such tool, but it has lots of other limitations
that make it unsuitable for many applications.

 [...]
   Yes, find solutions. Don't find dangerous dead-ends that look like
   solutions but which will give you lots of trouble.
 If concurrency is a dead end, why do the programs that provide
 the most sophisticated services of any in the world rely on it
 so heavily?

I don't know what Phil is saying, but I'm not calling concurrency a
dead end. I'm calling the tools available in most programming
languages for dealing with it primitive.

We need better tools.

   mike
-- 
Mike Meyer [EMAIL PROTECTED]  http://www.mired.org/home/mwm/
Independent WWW/Perforce/FreeBSD/Unix consultant, email for more information.
-- 
http://mail.python.org/mailman/listinfo/python-list

Re: global interpreter lock

[Piet van Oostrum]

 Paul Rubin http://[EMAIL PROTECTED] (PR) wrote:

PR [EMAIL PROTECTED] (phil hunt) writes:
 Let's see. Reality is that writing correct programs is hard. Writing
 correct programs that use concurrency is even harder, because of the
 exponential explosion of the order that operations can happen
 in. Personally, I'm willing to use anything I can find that makes
 those tasks easier.
 
 Indeed so. Use threading (or whatever) when one has to, use an 
 asynchronous single-threaded process whenever you can.

PR This is silly.  You could say the exact same thing about if
PR statements.  The number of paths through the program is exponential in
PR the number of if statements executed.  So we better get rid of if
PR statements.

PR Really, the essence of programming is to find ways of organizing the
PR program to stay reliable and maintainable in the face of that
PR combinatorial explosion.  That means facing the problem and finding
PR solutions, not running away.  The principle is no different for
PR threads than it is for if statements.

The principle is (more or less) similar, but for parallel programs it is an
order of magnitude more complicated. Compare the correctness proofs of
parallel programs with those of sequential programs.
-- 
Piet van Oostrum [EMAIL PROTECTED]
URL: http://www.cs.uu.nl/~piet [PGP 8DAE142BE17999C4]
Private email: [EMAIL PROTECTED]
-- 
http://mail.python.org/mailman/listinfo/python-list

Re: global interpreter lock

[[EMAIL PROTECTED]]

Mike Meyer wrote:
 Bryan Olson [EMAIL PROTECTED] writes:
  The issue here is whether to confuse reality with what one might
  wish reality to be.

 Let's see. Reality is that writing correct programs is hard. Writing
 correct programs that use concurrency is even harder, because of the
 exponential explosion of the order that operations can happen
 in.

And dont forget:
Writing concurrent programs without protected memory between execution
contexts is even harder than with it.

-- 
http://mail.python.org/mailman/listinfo/python-list

Re: global interpreter lock

[phil hunt]

On Thu, 25 Aug 2005 00:56:10 -0400, Mike Meyer [EMAIL PROTECTED] wrote:
 The issue here is whether to confuse reality with what one might
 wish reality to be.

Let's see. Reality is that writing correct programs is hard. Writing
correct programs that use concurrency is even harder, because of the
exponential explosion of the order that operations can happen
in. Personally, I'm willing to use anything I can find that makes
those tasks easier.

Indeed so. Use threading (or whatever) when one has to, use an 
asynchronous single-threaded process whenever you can.


-- 
Email: zen19725 at zen dot co dot uk


-- 
http://mail.python.org/mailman/listinfo/python-list

Re: global interpreter lock

[Paul Rubin]

[EMAIL PROTECTED] (phil hunt) writes:
 Let's see. Reality is that writing correct programs is hard. Writing
 correct programs that use concurrency is even harder, because of the
 exponential explosion of the order that operations can happen
 in. Personally, I'm willing to use anything I can find that makes
 those tasks easier.
 
 Indeed so. Use threading (or whatever) when one has to, use an 
 asynchronous single-threaded process whenever you can.

This is silly.  You could say the exact same thing about if
statements.  The number of paths through the program is exponential in
the number of if statements executed.  So we better get rid of if
statements.

Really, the essence of programming is to find ways of organizing the
program to stay reliable and maintainable in the face of that
combinatorial explosion.  That means facing the problem and finding
solutions, not running away.  The principle is no different for
threads than it is for if statements.
-- 
http://mail.python.org/mailman/listinfo/python-list

Re: global interpreter lock

[phil hunt]

On 26 Aug 2005 14:35:03 -0700, Paul Rubin http://[EMAIL PROTECTED] wrote:
[EMAIL PROTECTED] (phil hunt) writes:
 Let's see. Reality is that writing correct programs is hard. Writing
 correct programs that use concurrency is even harder, because of the
 exponential explosion of the order that operations can happen
 in. Personally, I'm willing to use anything I can find that makes
 those tasks easier.
 
 Indeed so. Use threading (or whatever) when one has to, use an 
 asynchronous single-threaded process whenever you can.

This is silly.  You could say the exact same thing about if
statements.  The number of paths through the program is exponential in
the number of if statements executed.  So we better get rid of if
statements.

It's not the number of paths that's important.

What's important is *predictability*, e.g. which instruction will
the computer execute next?

If you only have one thread, you can tell by looking at the code 
what gets executed next. It's very simple.

If you have 2 threads you can easily have a timing-based situation
that occurs rarely but which causes your program to behave wierdly.
This sort of bug is very hard to reproduce and therefore to fix.

Really, the essence of programming is to find ways of organizing the
program to stay reliable and maintainable in the face of that
combinatorial explosion. 

Yes, and introducing code that makes randomly-occurring bugs more
likely makes debugging inherently harder.

 That means facing the problem and finding
solutions, not running away.

Yes, find solutions. Don't find dangerous dead-ends that look like 
solutions but which will give you lots of trouble.

-- 
Email: zen19725 at zen dot co dot uk


-- 
http://mail.python.org/mailman/listinfo/python-list

Re: global interpreter lock

[Mike Meyer]

Paul Rubin http://[EMAIL PROTECTED] writes:

 [EMAIL PROTECTED] (phil hunt) writes:
 Let's see. Reality is that writing correct programs is hard. Writing
 correct programs that use concurrency is even harder, because of the
 exponential explosion of the order that operations can happen
 in. Personally, I'm willing to use anything I can find that makes
 those tasks easier.
 Indeed so. Use threading (or whatever) when one has to, use an 
 asynchronous single-threaded process whenever you can.
 This is silly.  You could say the exact same thing about if
 statements.  The number of paths through the program is exponential in
 the number of if statements executed.  So we better get rid of if
 statements.

The number of paths through a program isn't exponential in the number
of if statements, it's multiplicative. Each if statement multiplies
the number of paths through the program by 2, no matter how many other
statements you have.

On the other hand, with threads, the number of possible execution
orders is the number of threads raised to the power of the number of
instructions (assuming that instructions are atomic, which is probably
false) in the shared code segment. It's a *much* nastier problem.

 Really, the essence of programming is to find ways of organizing the
 program to stay reliable and maintainable in the face of that
 combinatorial explosion.  That means facing the problem and finding
 solutions, not running away.  The principle is no different for
 threads than it is for if statements.

Correct. But choosing to use a tool that has a less complex model but
solves the problem is *not* running away. If it were, you'd have to
call using if statements rather than a goto running away, because
that's that's exactly what you're doing.

I do agree that we should face the problems and look for solutions,
because some problems can't be solved with async I/O. That's why I
posted the article titled Static vs. dynamic checking for support of
concurrent programming - I'm trying to find out if one potential
solution could be adapted for Python.

   mike
-- 
Mike Meyer [EMAIL PROTECTED]  http://www.mired.org/home/mwm/
Independent WWW/Perforce/FreeBSD/Unix consultant, email for more information.
-- 
http://mail.python.org/mailman/listinfo/python-list

Re: global interpreter lock

[Bryan Olson]

Mike Meyer wrote:
  Bryan Olson writes:
 
 Mike Meyer wrote:
   The rule I follow in choosing my tools is Use the least complex tool
   that will get the job done.
 
 Even if a more complex tool could do the job better?
 
  In that case, the simpler model isn't necessarily getting the job
  done. I purposely didn't refine the word job just so this would be
  the case.

I didn't ask about any particular case. You stated a general
rule you follow, and I think that rule is nuts.


 Now I've gotten off-topic. Threads are winning, and the industry
 is going to multiple processors even for PC-class machines.
 Might as well learn to use that power.
 
  I own too many orphans to ever confuse popularity with technical
  superiority.

The issue here is whether to confuse reality with what one might
wish reality to be.

  I've learned how to use threads, and done some
  non-trivial thread proramming, and hope to never repeat that
  experience. It was the second most difficult programming task I've
  ever attempted(*).

Great -- lets see it!  Can you point out what parts were so
hard? How would you have solved the same problems without
threads?


  As I said above, the real problem isn't threads per
  se, it's that the model for programming them in popular languages is
  still primitive. So far, to achieve the non-repitition goal, I've used
  async I/O, restricted my use of real threads in popular languages to
  trivial cases, and started using servers so someone else gets tod eal
  with these issues.

Then maybe we should listen to those other people. Is there a
successful single-line-of-execution async-I/O based server that
provides a service as sophisticated as the modern relational-
database engines? Why do you think that is?


-- 
--Bryan
-- 
http://mail.python.org/mailman/listinfo/python-list

Re: global interpreter lock

[Mike Meyer]

Bryan Olson [EMAIL PROTECTED] writes:
 Mike Meyer wrote:
   Bryan Olson writes:
  Mike Meyer wrote:
The rule I follow in choosing my tools is Use the least complex tool
that will get the job done.
  Even if a more complex tool could do the job better?
   In that case, the simpler model isn't necessarily getting the job
   done. I purposely didn't refine the word job just so this would be
   the case.
 I didn't ask about any particular case. You stated a general
 rule you follow, and I think that rule is nuts.

You're entitled to write code as complex and unmanagable as you
wish. Me, I'll stick with the simplest thing that solve the problem.

  Now I've gotten off-topic. Threads are winning, and the industry
  is going to multiple processors even for PC-class machines.
  Might as well learn to use that power.
   I own too many orphans to ever confuse popularity with technical
   superiority.
 The issue here is whether to confuse reality with what one might
 wish reality to be.

Let's see. Reality is that writing correct programs is hard. Writing
correct programs that use concurrency is even harder, because of the
exponential explosion of the order that operations can happen
in. Personally, I'm willing to use anything I can find that makes
those tasks easier.

   I've learned how to use threads, and done some
   non-trivial thread proramming, and hope to never repeat that
   experience. It was the second most difficult programming task I've
   ever attempted(*).
 Great -- lets see it!  Can you point out what parts were so
 hard? How would you have solved the same problems without
 threads?

Google for aws amiga web server. Somebody is liable to still have
the source around. The hard part was dealing with the making sure that
every sequence of operations that actually happened was correct, of
course. The web server I wrote after that used async i/o, thus
avoiding the problem completely.

   As I said above, the real problem isn't threads per
   se, it's that the model for programming them in popular languages is
   still primitive. So far, to achieve the non-repitition goal, I've used
   async I/O, restricted my use of real threads in popular languages to
   trivial cases, and started using servers so someone else gets tod eal
   with these issues.
 Then maybe we should listen to those other people.

Yes, we probably should. I do. The problem is, the designers of
popular languages apparently don't, so I'm stuck with lousy tools like
thread libraries (meaning you get no compile-time help in avoiding the
problems that plague concurrent programs) and Java's pseudo-monitors.

 Is there a successful single-line-of-execution async-I/O based
 server that provides a service as sophisticated as the modern
 relational- database engines? Why do you think that is?

I don't know - is there? There have certainly been some sophisticated
network servers using async I/O and a single thread of execution. Are
they as sophisticated as a modern relational database? I dunno. Then
again, I already know that async i/o with a single thread of execution
isn't as powerful as threads, so there are almost certainly problem
areas where it isn't suitable and threads are. So what? That doesn't
make the async I/O model any less useful. Of course, if you're only
able to learn one tool, you should probably learn the most powerful
one you can. But just because you only know how to use a hammer
doesn't automatically make everything you encounter a nail.

mike
-- 
Mike Meyer [EMAIL PROTECTED]  http://www.mired.org/home/mwm/
Independent WWW/Perforce/FreeBSD/Unix consultant, email for more information.
-- 
http://mail.python.org/mailman/listinfo/python-list

Re: global interpreter lock

[Mike Meyer]

Paul Rubin http://[EMAIL PROTECTED] writes:
 Mike Meyer [EMAIL PROTECTED] writes:
 Even simpler to program in is the model used by Erlang. It's more CSP
 than threading, though, as it doesn't have shared memory as part of
 the model. But if you can use the simpler model to solve your problem
 - you probably should.
 Well, ok, the Python equivalent would be wrapping every shareable
 object in its own thread, that communicates with other threads through
 Queues.  This is how some Pythonistas suggest writing practically all
 multi-threaded Python code.  It does a reasonable job of avoiding
 synchronization headaches and it's not that hard to code that way.

This sort of feels like writing your while loops/etc. with if and
goto. Sure, they really are that at the hardware level, but you'd like
the constructs you work with to be at a higher level. It's not really
that bad, because Queue is a higher level construct, but it's still
not quite not as good as it could be.

 But I think to do it on Erlang's scale, Python needs user-level
 microthreads and not just OS threads.  Maybe Python 3000 can add some
 language support, though an opportunity was missed when Python's
 generator syntax got defined the way it did.

I'm not sure we need to go as far as Erlang does. On the other hand,
I'm also not sure we can get a much better threading model without
language support of some kind. Threading and Queues are all well and
good, but they still leave the programmer handling primitive threading
objects.

mike
-- 
Mike Meyer [EMAIL PROTECTED]  http://www.mired.org/home/mwm/
Independent WWW/Perforce/FreeBSD/Unix consultant, email for more information.
-- 
http://mail.python.org/mailman/listinfo/python-list

Re: global interpreter lock

[Mike Meyer]

Dennis Lee Bieber [EMAIL PROTECTED] writes:
 On Sat, 20 Aug 2005 22:30:43 -0400, Mike Meyer [EMAIL PROTECTED] declaimed
 the following in comp.lang.python:
 with these issues. If I ever find myself having to have non-trivial
 threads again, I'll check the state of the threading models in other
 languages, and make a serious push for implementing parts of the
 program in a less popular language with a less primitive threading
 model.
   The June edition of SIGPLAN Notices (the PLDI'05 proceeding issue)
 has a paper titled Threads Cannot Be Implemented As a Library -- which
 is primarily concerned with the problems of threading being done, well,
 via an add-on library (as opposed to a native part of the language
 specification: C#, Ada, Java).

Thanks for the reference.  A litte googling turns up a copy published
via HP at URL:
http://www.hpl.hp.com/techreports/2004/HPL-2004-209.html .

   I suspect Python falls into the library category.

Well, that's what it's got now, so that seem likely.

  mike
-- 
Mike Meyer [EMAIL PROTECTED]  http://www.mired.org/home/mwm/
Independent WWW/Perforce/FreeBSD/Unix consultant, email for more information.
-- 
http://mail.python.org/mailman/listinfo/python-list

Re: global interpreter lock

[Michael Sparks]

Paul Rubin wrote:
 Mike Meyer [EMAIL PROTECTED] writes:
 Even simpler to program in is the model used by Erlang. It's more CSP
 than threading, though, as it doesn't have shared memory as part of
 the model. But if you can use the simpler model to solve your problem
 - you probably should.
 
 Well, ok, the Python equivalent would be wrapping every shareable
 object in its own thread, that communicates with other threads through
 Queues.  This is how some Pythonistas suggest writing practically all
 multi-threaded Python code.  It does a reasonable job of avoiding
 synchronization headaches and it's not that hard to code that way.
 
 But I think to do it on Erlang's scale, Python needs user-level
 microthreads and not just OS threads.  

You've just described Kamaelia* BTW, except substitute micro-thread
with generator :-) (Also we call the queues outboxes and inboxes, and 
the combination of a generator in a class with inboxes and outboxes
components)
   * http://kamaelia.sf.net/

For those who really want threads as well, theres a threaded component based
class that uses Queues instead :)

Best Regards,


Michael.

-- 
http://mail.python.org/mailman/listinfo/python-list

Re: global interpreter lock

[[EMAIL PROTECTED]]

km wrote:

 is true parallelism possible in python ? or atleast in the coming versions ?
 is global interpreter lock a bane in this context ?

I've had absolutely zero problems implementing truly parallel programs
in python.  All of my parallel programs have been multiprocess
architectures, though--the GIL doesn't affect multiprocess
architectures.

Good support for multiple process architectures was one of the things
that initially lead me to pick Python over Java in the first place
(Java was woefully lacking in support facilities for this kind of
architecture at that time; it's improved somewhat since then but still
requires some custom C coding).  I don't have much desire to throw out
decades of work by OS implementors on protected memory without a pretty
darn good reason.

-- 
http://mail.python.org/mailman/listinfo/python-list

Re: global interpreter lock

[Alan Kennedy]

[km]
 is true parallelism possible in python ? 

cpython:no.
jython: yes.
ironpython: yes.

  or atleast in the coming versions ?

cpython:unknown.
pypy:   don't have time to research. Anyone know?

 is global interpreter lock a bane in this context ? 

beauty/bane-is-in-the-eye-of-the-beholder-ly y'rs

-- 
alan kennedy
--
email alan:  http://xhaus.com/contact/alan
-- 
http://mail.python.org/mailman/listinfo/python-list

Re: global interpreter lock

[Donn Cave]

Quoth Paul Rubin http://[EMAIL PROTECTED]:
| Mike Meyer [EMAIL PROTECTED] writes:

| The real problem is that the concurrency models available in currently
| popular languages are still at the goto stage of language
| development.  Better models exist, have existed for decades, and are
| available in a variety of languages.
|
| But Python's threading system is designed to be like Java's, and
| actual Java implementations seem to support concurrent threads just fine.

I don't see a contradiction here.  goto is just fine, too --
you can write excellent programs with goto.  20 years of one very
successful software engineering crusade against this feature have
made it a household word for brokenness, but most current programming
languages have more problems in that vein that pass without question.
If you want to see progress, it's important to remember that goto
was a workable, useful, powerful construct that worked fine in the
right hands - and that wasn't enough.

Anyway, to return to the subject, I believe if you follow this
subthread back you will see that it has diverged a little from
simply whether or how Python could support SMP.

Mike, care to mention an example or two of the better models you
had in mind there?

Donn Cave, [EMAIL PROTECTED]
-- 
http://mail.python.org/mailman/listinfo/python-list

Re: global interpreter lock

[Bryan Olson]

Mike Meyer wrote:
  The real problem is that the concurrency models available in currently
  popular languages are still at the goto stage of language
  development.  Better models exist, have existed for decades, and are
  available in a variety of languages.

That's not the real problem; it's a different and arguable
problem. The GIL isn't even part of Python's threading model;
it's part of the implementation.

  It's not that these languages are for thread-phobes, either. They
  don't lose power any more than Python looses power by not having a
  goto. They languages haven't taken off for reasons unrelated to the
  threading model(*).
 
  The rule I follow in choosing my tools is Use the least complex tool
  that will get the job done.

Even if a more complex tool could do the job better?

  Given that the threading models in
  popular languages are complex and hard to work with, I look elsewhere
  for solutions. I've had good luck using async I/O in lieue of
  theards. It's won't solve every problem, but where it does, it's much
  simpler to work with.

I've found single-line-of-execution async I/O to be worse than
threads. I guess that puts me in the Tannenbaum camp and not the
Ousterhout camp. Guido and Tannenbaum worked together on Amoeba
(and other stuff), which featured threads with semaphores and
seemed to work well.

Now I've gotten off-topic. Threads are winning, and the industry
is going to multiple processors even for PC-class machines.
Might as well learn to use that power.


-- 
--Bryan
-- 
http://mail.python.org/mailman/listinfo/python-list

Re: global interpreter lock

[Alan Kennedy]

[Bryan Olson]
I don't see much point in trying to convince programmers that
they don't really want concurrent threads. They really do. Some
don't know how to use them, but that's largely because they
haven't had them. I doubt a language for thread-phobes has much
of a future.

[Mike Meyer]
 The real problem is that the concurrency models available in currently
 popular languages are still at the goto stage of language
 development.  Better models exist, have existed for decades, and are
 available in a variety of languages.

I think that having a concurrency mechanism that doesn't use goto will 
require a fundamental redesign of the underlying execution hardware, 
i.e. the CPU.

All modern CPUs allow flow control through the use of 
machine-code/assembly instructions which branch, either conditionally or 
unconditionally, to either a relative or absolute memory address, i.e. a 
GOTO.

Modern languages wrap this goto nicely using constructs such as 
generators, coroutines or continuations, which allow preservation and 
restoration of the execution context, e.g. through closures, evaluation 
stacks, etc. But underneath the hood, they're just gotos. And I have no 
problem with that.

To really have parallel execution with clean modularity requires a 
hardware redesign at the CPU level, where code units, executing in 
parallel, are fed a series of data/work-units. When they finish 
processing an individual unit, it gets passed (physically, at a hardware 
level) to another code unit, executing in parallel on another execution 
unit/CPU. To achieve multi-stage processing of data would require 
breaking up the processing into a pipeline of modular operations, which 
communicate through dedicated hardware channels.

I don't think I've described it very clearly above, but you can read a 
good high-level overview of a likely model from the 1980's, the 
Transputer, here

http://en.wikipedia.org/wiki/Transputer

Transputers never took off, for a variety of technical and commercial 
reasons, even though there was full high-level programming language 
support in the form of Occam: I think it was just too brain-bending for 
most programmers at the time. (I personally *almost* took on the task of 
developing a debugger for transputer arrays for my undergrad thesis in 
1988, but when I realised the complexity of the problem, I picked a 
hypertext project instead ;-)

http://en.wikipedia.org/wiki/Occam_programming_language

IMHO, python generators (which BTW are implemented with a JVM goto 
instruction in jython 2.2) are a nice programming model that fits neatly 
with this hardware model. Although not today.

-- 
alan kennedy
--
email alan:  http://xhaus.com/contact/alan
-- 
http://mail.python.org/mailman/listinfo/python-list

Re: global interpreter lock

[Mike Meyer]

Donn Cave [EMAIL PROTECTED] writes:

 Quoth Paul Rubin http://[EMAIL PROTECTED]:
 | Mike Meyer [EMAIL PROTECTED] writes:
 | The real problem is that the concurrency models available in currently
 | popular languages are still at the goto stage of language
 | development.  Better models exist, have existed for decades, and are
 | available in a variety of languages.
 | But Python's threading system is designed to be like Java's, and
 | actual Java implementations seem to support concurrent threads just fine.
 I don't see a contradiction here.  goto is just fine, too --
 you can write excellent programs with goto.

Right. The only thing wrong with goto is that we've since found
better ways to describe program flow. These ways are less complex,
hence easier to use and understand.

 Mike, care to mention an example or two of the better models you
 had in mind there?

I've seen a couple of such, but have never been able to find the one I
really liked in Google again :-(. That leaves Eiffel's SCOOP (aka
Concurrent Eiffel).  You can find a short intro at URL:
http://archive.eiffel.com/doc/manuals/technology/concurrency/short/page.html
.

Even simpler to program in is the model used by Erlang. It's more CSP
than threading, though, as it doesn't have shared memory as part of
the model. But if you can use the simpler model to solve your problem
- you probably should.

   mike
-- 
Mike Meyer [EMAIL PROTECTED]  http://www.mired.org/home/mwm/
Independent WWW/Perforce/FreeBSD/Unix consultant, email for more information.
-- 
http://mail.python.org/mailman/listinfo/python-list

Re: global interpreter lock

[Mike Meyer]

Bryan Olson [EMAIL PROTECTED] writes:
 Mike Meyer wrote:
   The real problem is that the concurrency models available in currently
   popular languages are still at the goto stage of language
   development.  Better models exist, have existed for decades, and are
   available in a variety of languages.
 That's not the real problem; it's a different and arguable
 problem. The GIL isn't even part of Python's threading model;
 it's part of the implementation.

Depends on what point you consider the problem.

   It's not that these languages are for thread-phobes, either. They
   don't lose power any more than Python looses power by not having a
   goto. They languages haven't taken off for reasons unrelated to the
   threading model(*).
   The rule I follow in choosing my tools is Use the least complex tool
   that will get the job done.
 Even if a more complex tool could do the job better?

In that case, the simpler model isn't necessarily getting the job
done. I purposely didn't refine the word job just so this would be
the case.

 Now I've gotten off-topic. Threads are winning, and the industry
 is going to multiple processors even for PC-class machines.
 Might as well learn to use that power.

I own too many orphans to ever confuse popularity with technical
superiority. I've learned how to use threads, and done some
non-trivial thread proramming, and hope to never repeat that
experience. It was the second most difficult programming task I've
ever attempted(*). As I said above, the real problem isn't threads per
se, it's that the model for programming them in popular languages is
still primitive. So far, to achieve the non-repitition goal, I've used
async I/O, restricted my use of real threads in popular languages to
trivial cases, and started using servers so someone else gets tod eal
with these issues. If I ever find myself having to have non-trivial
threads again, I'll check the state of the threading models in other
languages, and make a serious push for implementing parts of the
program in a less popular language with a less primitive threading
model.

mike

*) The most difficult task was writing horizontal microcode, which
also had serious concurrency issues in the form of device settling
times. I dealt with that by inventing a programming model that hid
most of the timing details from the programmer. It occasionally lost a
cycle, but the people who used it after me were *very* happy with it
compared to the previous model.

-- 
Mike Meyer [EMAIL PROTECTED]  http://www.mired.org/home/mwm/
Independent WWW/Perforce/FreeBSD/Unix consultant, email for more information.
-- 
http://mail.python.org/mailman/listinfo/python-list

Re: global interpreter lock

[Paul Rubin]

Mike Meyer [EMAIL PROTECTED] writes:
 Even simpler to program in is the model used by Erlang. It's more CSP
 than threading, though, as it doesn't have shared memory as part of
 the model. But if you can use the simpler model to solve your problem
 - you probably should.

Well, ok, the Python equivalent would be wrapping every shareable
object in its own thread, that communicates with other threads through
Queues.  This is how some Pythonistas suggest writing practically all
multi-threaded Python code.  It does a reasonable job of avoiding
synchronization headaches and it's not that hard to code that way.

But I think to do it on Erlang's scale, Python needs user-level
microthreads and not just OS threads.  Maybe Python 3000 can add some
language support, though an opportunity was missed when Python's
generator syntax got defined the way it did.

I've been reading a bit about Erlang and am impressed with it.  Here
is a good thesis about parallelizing Erlang, link courtesy of Ulf
Wiger on comp.lang.functional:

  http://www.erlang.se/publications/xjobb/0089-hedqvist.pdf

The thesis also gives a good general description of how Erlang works.
-- 
http://mail.python.org/mailman/listinfo/python-list

Re: global interpreter lock

[Donn Cave]

Quoth Mike Meyer [EMAIL PROTECTED]:
[... wandering from the nominal topic ...]

| *) The most difficult task was writing horizontal microcode, which
| also had serious concurrency issues in the form of device settling
| times. I dealt with that by inventing a programming model that hid
| most of the timing details from the programmer. It occasionally lost a
| cycle, but the people who used it after me were *very* happy with it
| compared to the previous model.

My favorite concurrency model comes with a Haskell variant called
O'Haskell, and it was last seen calling itself Timber with some
added support for time as an event source.  The most on topic thing
about it -- its author implemented a robot controller in Timber, and
the robot is a little 4-wheeler called ... Timbot.

Donn Cave, [EMAIL PROTECTED]
-- 
http://mail.python.org/mailman/listinfo/python-list

global interpreter lock

[km]

Hi all,

is true parallelism possible in python ? or atleast in the coming versions ?
is global interpreter lock a bane in this context ? 

regards,
KM
-- 
http://mail.python.org/mailman/listinfo/python-list

Re: global interpreter lock

[Paul Rubin]

km [EMAIL PROTECTED] writes:
 is true parallelism possible in python ? or atleast in the coming versions ?
 is global interpreter lock a bane in this context ? 

http://poshmodule.sf.net
-- 
http://mail.python.org/mailman/listinfo/python-list

Re: global interpreter lock

[Robin Becker]

Paul Rubin wrote:
 km [EMAIL PROTECTED] writes:
 
is true parallelism possible in python ? or atleast in the coming versions ?
is global interpreter lock a bane in this context ? 
 
 
 http://poshmodule.sf.net

Is posh maintained? The page mentions 2003 as the last date.
-- 
Robin Becker

-- 
http://mail.python.org/mailman/listinfo/python-list