On 9/17/05, "Martin v. Löwis" <[EMAIL PROTECTED]> wrote: > > > Instead, the issue mainly died because nobody provided > working code (along with a strategy on what to do with > the existing extension modules). >
When I first started writing python code I had just come out of about 6 years of kernel development (1994-2000). These 6 years span the times when we were making changes to UP kernels to work correctly and efficiently on SMP machines, implementing kernel threads, and thread-safing libc and other changes needed to move from the now defunct user space DCE-threads to posix threads including kernel threads and even MxN support. So, I was very familiar with thread programming. I architected, designed, and developed a bunch of distributed servers and services using python and Pyro and it all was done using nice and cool thread programming practices and techniques. It was nice to see Python supported threads. Close to a year into this when we had a lot of functionality implemented I started running scalability tests and I was shocked and appalled to find out that our architecture did not scale well on SMP machines because we depended on threads. And that's when I discovered the GIL and twisted and other process based python techniques used to get around the GIL. It was too late for us to redo things and we've been in a holding pattern since then waiting to see what will hit us first, the need to scale a lot, which means we'd have to drop some time to implement some changes to support process based scalability, or no GIL. I have my money on process based scalability. :( I'm sure that has happened to a lot of people because nobody finds about the GIL in the beginning of their python development experience. If I started writing a new complex app from scratch, now I know exactly what to do. One big problem is that a lot of people defend the GIL with the premise that the GIL is OK because we release the GIL in C extensions and during IO. That is true, the GIL is not as bad as it would've been if the CPython interpreter extensions didn't do that but it's also true that it hurts performance on SMP machines in more than one way. When I found out about this it dawn on me that the performance could've been worse although it was still way below what we expected. I don't remember exactly how but what I remember is that when I presented the problem to Guido he just told me to put up or shut up. At first I was insulted but then I realized that it was a fair response given the complexity of the problem, previous history I didn't know about, the fairly good compromise of releasing the GIL in extensions, and the lack of a large percentage of python developers asking for better performance on SMP machines. I couldn't put up so ... I think I've gone beyond my quota :-) PS The GIL is probably my only peeve about the CPython interpreter. So don't get me wrong, I love the language and I'm always grateful for all the hard work you guys put in developing such great language and implementation. -- Luis P Caamano Atlanta, GA USA _______________________________________________ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
