Re: [Python-Dev] Reworking the GIL
Hello, I built something very similar for my company last year, and it’s been running flawlessly in production at a few customer sites since, with avg. CPU usage ~50% around the clock. I even posted about it on the Python mailing list [1] where there was almost no resonance at that time. I never posted code, though -- nobody seemed to be too interested. I am well aware that your current work is a lot more far-reaching than what I’ve done, which is basically just a FIFO scheduler. I even added scheduling priorities later which don’t work too great because the amount of time used for a tick can vary by several orders of magnitude, as you know. Thought you might be interested. Regards Stefan [1] http://mail.python.org/pipermail/python-dev/2008-March/077814.html [2] http://www.bestinclass.dk/index.php/2009/10/python-vs-clojure-evolving/ [3] www.dabeaz.com/python/GIL.pdf PS On a slightly different note, I came across some Python bashing [2] yesterday and somehow from there to David Beazley’s presentation about the GIL [3]. While I don’t mind the bashing, the observations about the GIL seem quite unfair to me because David’s measurements have been made on Mac OS X with its horribly slow pthreads functions. I was not able to measure any slowdown on Linux. ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Reworking the GIL
Stefan Ring wrote: [2] http://www.bestinclass.dk/index.php/2009/10/python-vs-clojure-evolving/ [3] www.dabeaz.com/python/GIL.pdf PS On a slightly different note, I came across some Python bashing [2] yesterday and somehow from there to David Beazley’s presentation about the GIL [3]. While I don’t mind the bashing, the observations about the GIL seem quite unfair to me because David’s measurements have been made on Mac OS X with its horribly slow pthreads functions. I was not able to measure any slowdown on Linux. We care about Mac OS X though, so even if the contention wasn't as bad on a different OS, the Mac downsides matter. With the GIL updates in place, it would be interesting to see that analysis redone for 2.7/3.2 though. Regards, Nick. P.S. As far as interest in the idea goes, the GIL is one of those areas where it takes a fairly rare combination of interest, expertise and established credibility to propose a change and get assent to it. You'll notice that even Antoine had to resort to the if nobody objects soon, I'm checking this in tactic to garner any responses. It's an area where even those with relevant expertise still have to put aside a fair chunk of time in order to properly review a proposed change :) -- Nick Coghlan | ncogh...@gmail.com | Brisbane, Australia --- ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Removal of intobject.h in 3.1
Eric Smith wrote: M.-A. Lemburg wrote: Since package developers are just starting to port things to 3.x and many appear to be considering supporting both 2.7 and 3.1 (including myself), I find it a bit strange that such an import aliasing header was removed in 3.1. There's some discussion of this at http://bugs.python.org/issue7353 You might want to comment there. Thanks for pointing me to the that ticket. Looks like Guido already commented on this, so intobject.h could be revived in some form. Since this file would only be used by extension modules and possibly also include other helper macros, perhaps it'd be better to to rename the file to py2compat.h or something along those lines ?! We could then also have a py2compat.c to hold corresponding C code, e.g. to provide compatibility wrappers of new APIs that implement different semantics in 3.x. -- Marc-Andre Lemburg eGenix.com Professional Python Services directly from the Source (#1, Nov 23 2009) Python/Zope Consulting and Support ...http://www.egenix.com/ mxODBC.Zope.Database.Adapter ... http://zope.egenix.com/ mxODBC, mxDateTime, mxTextTools ...http://python.egenix.com/ ::: Try our new mxODBC.Connect Python Database Interface for free ! eGenix.com Software, Skills and Services GmbH Pastor-Loeh-Str.48 D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg Registered at Amtsgericht Duesseldorf: HRB 46611 http://www.egenix.com/company/contact/ ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Removal of intobject.h in 3.1
Martin v. Löwis wrote:
IMHO, that's not really a good way to encourage people to try to provide
a smooth upgrade to the 3.x branch. Much to the contrary. 3.x should make
it easier for developers by providing more standard helpers like
the removed intobject.h header file.
I think it's better than it sounds. The macros (unfortunately) allowed
to make non-obvious mistakes. Now that they are gone, people need to
really think of what precisely they want to do.
For example, consider
if (PyInt_Check(o)){
long val = PyInt_AsLong(o);
// process
} else if (PyLong_Check(o)) {
long long val = PyLong_AsLongLong(o);
// check for overflow
// process
}
With intobject.h, this code would continue to compile, but work
incorrectly, as the second case will never be executed. It would
be better to port this as
#if Py2.x
if (PyInt_Check(o)){
long val = PyInt_AsLong(o);
// process
} else
#endif
if (PyLong_Check(o)) {
i.e. eliminating the int case altogether.
Sure, but that assumes that the original code already had support
for Python longs, which a lot of code doesn't.
In an ideal world, developers would add that code to their
extensions right away. In the real world, where developers only
have limited resources available, you'll get more 3.x ports
by making such ports as painless as possible while at the
same time not forcing them to alienate their 2.x user base.
The long support could then be added in later releases
of the extensions, giving the developers more time adapt.
For another example,
long foo = PyInt_AsLong(Foo);
has a hidden error in 3.x, with intobject: PyLong_AsLong might
overflow, which the 2.x case doesn't.
That's not quite true: PyInt_AsLong(obj) will try the
nb_int slot on non-integer objects which can return errors
(it returns -1 and sets the error message).
So eliminating intobject.h likely helps avoiding subtle errors.
In the long run, yes. In the short run, other criteria are
more important, IMHO.
--
Marc-Andre Lemburg
eGenix.com
Professional Python Services directly from the Source (#1, Nov 23 2009)
Python/Zope Consulting and Support ...http://www.egenix.com/
mxODBC.Zope.Database.Adapter ... http://zope.egenix.com/
mxODBC, mxDateTime, mxTextTools ...http://python.egenix.com/
::: Try our new mxODBC.Connect Python Database Interface for free !
eGenix.com Software, Skills and Services GmbH Pastor-Loeh-Str.48
D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg
Registered at Amtsgericht Duesseldorf: HRB 46611
http://www.egenix.com/company/contact/
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe:
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Reworking the GIL
I built something very similar for my company last year, and it’s been running
flawlessly in production at a few customer sites since, with avg. CPU usage
~50%
around the clock. I even posted about it on the Python mailing list [1] where
there was almost no resonance at that time. I never posted code, though --
nobody seemed to be too interested.
I've never bothered to make this tidy and nice, especially the
function naming (PySpecial_*) leaves some things to be desired. It's
not too bad, though; it just doesn't have commit-ready quality. I
don't worry about this anymore, so I just post what I have. Maybe
someone can make use of it.
--- Python-2.5.2/Include/pythread.h.scheduling 2006-06-13 17:04:24.0 +0200
+++ Python-2.5.2/Include/pythread.h 2008-10-16 14:46:07.0 +0200
@@ -40,6 +40,20 @@
PyAPI_FUNC(void *) PyThread_get_key_value(int);
PyAPI_FUNC(void) PyThread_delete_key_value(int key);
+#ifndef _POSIX_THREADS
+#error Requires POSIX threads
+#endif
+
+PyAPI_FUNC(void *) PyThread_mutex_alloc(void);
+PyAPI_FUNC(void) PyThread_mutex_free(void *);
+PyAPI_FUNC(void) PyThread_mutex_lock(void *);
+PyAPI_FUNC(void) PyThread_mutex_unlock(void *);
+
+PyAPI_FUNC(void *) PyThread_cond_alloc(void);
+PyAPI_FUNC(void) PyThread_cond_free(void *);
+PyAPI_FUNC(void) PyThread_cond_wait(void *, void *);
+PyAPI_FUNC(void) PyThread_cond_signal(void *);
+
#ifdef __cplusplus
}
#endif
--- Python-2.5.2/Python/thread.c.scheduling 2006-07-21 09:59:47.0 +0200
+++ Python-2.5.2/Python/thread.c 2008-10-16 14:46:07.0 +0200
@@ -155,6 +155,56 @@
#endif
*/
+void *PyThread_mutex_alloc(void)
+{
+ pthread_mutex_t *m = malloc(sizeof(pthread_mutex_t));
+ if (pthread_mutex_init(m, NULL))
+ Py_FatalError(PyThread_mutex_alloc: pthread_mutex_init failed);
+ return m;
+}
+
+void PyThread_mutex_free(void *m)
+{
+ if (pthread_mutex_destroy(m))
+ Py_FatalError(PyThread_mutex_free: pthread_mutex_destroy failed);
+ free(m);
+}
+
+void PyThread_mutex_lock(void *m)
+{
+ pthread_mutex_lock(m);
+}
+
+void PyThread_mutex_unlock(void *m)
+{
+ pthread_mutex_unlock(m);
+}
+
+void *PyThread_cond_alloc(void)
+{
+ pthread_cond_t *c = malloc(sizeof(pthread_cond_t));
+ if (pthread_cond_init(c, NULL))
+ Py_FatalError(PyThread_cond_alloc: pthread_cond_init failed);
+ return c;
+}
+
+void PyThread_cond_free(void *c)
+{
+ if (pthread_cond_destroy(c))
+ Py_FatalError(PyThread_cond_free: pthread_cond_destroy failed);
+ free(c);
+}
+
+void PyThread_cond_wait(void *c, void *m)
+{
+ pthread_cond_wait(c, m);
+}
+
+void PyThread_cond_signal(void *c)
+{
+ pthread_cond_signal(c);
+}
+
/* return the current thread stack size */
size_t
PyThread_get_stacksize(void)
--- Python-2.5.2/Python/ceval.c.scheduling 2008-01-23 21:09:39.0 +0100
+++ Python-2.5.2/Python/ceval.c 2008-10-16 14:47:07.0 +0200
@@ -210,7 +210,31 @@
#endif
#include pythread.h
-static PyThread_type_lock interpreter_lock = 0; /* This is the GIL */
+typedef void *PySpecial_cond_type;
+
+struct special_linkstruct {
+ PySpecial_cond_type wait;
+ struct special_linkstruct *queue_next, *free_next;
+ int in_use;
+};
+
+typedef void *PySpecial_lock_type;
+
+typedef struct {
+ PySpecial_lock_type the_lock;
+ struct special_linkstruct *wait_queue, *wait_last, *free_queue;
+} PySpecialSemaphore;
+
+void
+PySpecial_init(PySpecialSemaphore *s)
+{
+ s-the_lock = PyThread_mutex_alloc();
+ s-wait_queue = NULL;
+ s-wait_last = NULL;
+ s-free_queue = NULL;
+}
+
+static PySpecialSemaphore *interpreter_lock = NULL; /* This is the GIL */
static long main_thread = 0;
int
@@ -219,26 +243,100 @@
return interpreter_lock != 0;
}
+static PySpecialSemaphore *allocate_special(void)
+{
+ PySpecialSemaphore *s = malloc(sizeof(PySpecialSemaphore));
+ PySpecial_init(s);
+ return s;
+}
+
+static struct special_linkstruct *allocate_special_linkstruct(void)
+{
+ struct special_linkstruct *ls = malloc(sizeof(struct special_linkstruct));
+ ls-wait = PyThread_cond_alloc();
+ ls-queue_next = NULL;
+ ls-free_next = NULL;
+ ls-in_use = 0;
+ return ls;
+}
+
+static void PySpecial_Lock(PySpecialSemaphore *s)
+{
+ struct special_linkstruct *ls;
+
+ PyThread_mutex_lock(s-the_lock);
+
+ if (!s-free_queue)
+ s-free_queue = allocate_special_linkstruct();
+
+ ls = s-free_queue;
+ s-free_queue = ls-free_next;
+
+ if (!s-wait_queue)
+ {
+ ls-in_use = 1;
+ s-wait_queue = ls;
+ s-wait_last = ls;
+ PyThread_mutex_unlock(s-the_lock);
+ return;
+ }
+
+ assert(s-wait_queue != ls);
+ assert(s-wait_last != ls);
+ assert(s-wait_last-queue_next == NULL);
+ assert(!ls-in_use);
+ s-wait_last-queue_next = ls;
+ s-wait_last = ls;
+ ls-in_use = 1;
+
+ while (s-wait_queue != ls)
+ PyThread_cond_wait(ls-wait, s-the_lock);
+
+ PyThread_mutex_unlock(s-the_lock);
+}
+
+static void PySpecial_Unlock(PySpecialSemaphore *s)
+{
+ struct special_linkstruct *ls;
+
+ PyThread_mutex_lock(s-the_lock);
+ ls = s-wait_queue;
+ assert(ls-in_use);
+
+ s-wait_queue = ls-queue_next;
+ if (s-wait_queue)
+ {
+
Re: [Python-Dev] Removal of intobject.h in 3.1
In an ideal world, developers would add that code to their extensions right away. In the real world, where developers only have limited resources available, you'll get more 3.x ports by making such ports as painless as possible while at the same time not forcing them to alienate their 2.x user base. Unfortunately, such 3.x code would be full of bugs. For another example, long foo = PyInt_AsLong(Foo); has a hidden error in 3.x, with intobject: PyLong_AsLong might overflow, which the 2.x case doesn't. That's not quite true: PyInt_AsLong(obj) will try the nb_int slot on non-integer objects which can return errors (it returns -1 and sets the error message). However, the 2.x code will typically assume that the object is an int object, in which case PyInt_AsLong can never fail. So with intobject.h, the code will happily compile, but then fail to detect an exception at run-time - causing other difficult-to-find bugs. Regards, Martin ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Removal of intobject.h in 3.1
On Mon, Nov 23, 2009 at 10:44 AM, M.-A. Lemburg m...@egenix.com wrote: Thanks for pointing me to the that ticket. Looks like Guido already commented on this, so intobject.h could be revived in some form. I'm wondering how a resurrected intobject.h should be used: would Linux distributors (for example) package up intobject.h with the rest of the python-devel RPM/package/whatever so that Python extension modules could just include it directly, or would it be better to encourage the extension writers to make a copy of intobject.h to add to the source for their extension? In the first case, intobject.h would become a requirement for those extensions, so we'd presumably end up committed to either maintaining intobject.h for the lifetime of Python 3.x, or causing some pain when it does eventually get deleted. I quite like Benjamin's suggestion (in the issue tracker) of putting it in Doc/includes. Since this file would only be used by extension modules and possibly also include other helper macros, perhaps it'd be better to to rename the file to py2compat.h or something along those lines ?! We could then also have a py2compat.c to hold corresponding C code, e.g. to provide compatibility wrappers of new APIs that implement different semantics in 3.x. This is also an interesting idea. Mark ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Removal of intobject.h in 3.1
Mark Dickinson wrote: On Mon, Nov 23, 2009 at 10:44 AM, M.-A. Lemburg m...@egenix.com wrote: Thanks for pointing me to the that ticket. Looks like Guido already commented on this, so intobject.h could be revived in some form. I'm wondering how a resurrected intobject.h should be used: would Linux distributors (for example) package up intobject.h with the rest of the python-devel RPM/package/whatever so that Python extension modules could just include it directly, or would it be better to encourage the extension writers to make a copy of intobject.h to add to the source for their extension? In the first case, intobject.h would become a requirement for those extensions, so we'd presumably end up committed to either maintaining intobject.h for the lifetime of Python 3.x, or causing some pain when it does eventually get deleted. I quite like Benjamin's suggestion (in the issue tracker) of putting it in Doc/includes. I'm not sure whether Doc/ is a good location for such things. In the past, we've put such files in Misc/, e.g. the old Makefile.pre.in based extension mechanism was shipped that way: Extension writers would simply copy the file to their extension directory, create a Setup.in file and then have the users run make -f Makefile.pre.in boot to have a Makefile created which then built the extension. However, I'm not sure whether using the copy mechanism really works out that well: it's easier to just have the file(s) in the standard Python include file locations and then have extension writers use #include py2compat.h where necessary. Extension writer can then still use a private copy if needed, but most of the time they are probably better served by using the Python provided standard one. Since this file would only be used by extension modules and possibly also include other helper macros, perhaps it'd be better to to rename the file to py2compat.h or something along those lines ?! We could then also have a py2compat.c to hold corresponding C code, e.g. to provide compatibility wrappers of new APIs that implement different semantics in 3.x. This is also an interesting idea. -- Marc-Andre Lemburg eGenix.com Professional Python Services directly from the Source (#1, Nov 23 2009) Python/Zope Consulting and Support ...http://www.egenix.com/ mxODBC.Zope.Database.Adapter ... http://zope.egenix.com/ mxODBC, mxDateTime, mxTextTools ...http://python.egenix.com/ ::: Try our new mxODBC.Connect Python Database Interface for free ! eGenix.com Software, Skills and Services GmbH Pastor-Loeh-Str.48 D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg Registered at Amtsgericht Duesseldorf: HRB 46611 http://www.egenix.com/company/contact/ ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Removal of intobject.h in 3.1
M.-A. Lemburg mal at egenix.com writes: We could then also have a py2compat.c to hold corresponding C code, e.g. to provide compatibility wrappers of new APIs that implement different semantics in 3.x. If the semantic differences are embodied in the builtin object types I wonder how you can make a compatibility wrapper (short of reimplementing the 2.x types). An int is not a long, a bytes is not a str is not a unicode. I'm not sure concealing the differences between 2.x and 3.x behind such a wrapper is a good idea. It would be better if people became aware of / learnt about the new semantics. Regards Antoine. ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Removal of intobject.h in 3.1
I'm not sure concealing the differences between 2.x and 3.x behind such a wrapper is a good idea. It would be better if people became aware of / learnt about the new semantics. +1. When porting a larger code base, I always came up with a custom set of macros, specific to the way the modules were written in the package I ported. In particular for PyString_, some of the code meant to deal with bytes, other code with text. There was no way to tell, short of reading and understanding the code. As you'll have to review the complete code, anyway, the savings you gain by using macros somebody else wrote are really minor. Regards, Martin ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
