Re: python 3 problem: how to convert an extension method into a class Method
On 26/02/2013 18:38, Peter Otten wrote: Robin Becker wrote: ...3: $ python -m timeit -s 'from new import instancemethod from math import sqrt class A(int): pass A.m = instancemethod(sqrt, None, A) a = A(42) ' 'a.m()' 100 loops, best of 3: 0.5 usec per loop $ python -m timeit -s 'from math import sqrt class A(int): def m(self): return sqrt(self) a = A(42) ' 'a.m()' 100 loops, best of 3: 0.473 usec per loop this analysis might be relevant if I wanted to use sqrt. However, in my case the method takes py C utf8 bytes50 20 usec unicode 39 15 here py refers to a native python method and C to the extension method after adding to the class. Both are called via an instance of the class. -- Robin Becker -- http://mail.python.org/mailman/listinfo/python-list
Re: python 3 problem: how to convert an extension method into a class Method
On Wed, Feb 27, 2013 at 9:36 PM, Robin Becker ro...@reportlab.com wrote: However, in my case the method takes py C utf8 bytes50 20 usec unicode 39 15 here py refers to a native python method and C to the extension method after adding to the class. Both are called via an instance of the class. Which raises the obvious question: Does it even matter? Will the saving of a few microseconds really make a difference? Python's best feature is its clarity of code, not its blazing performance; its performance goal is fast enough, and for many MANY purposes, you won't be able to tell the difference between that and awesome. Don't sacrifice your code's clarity to the little tin god of efficiency until you're sure you actually get something back. ChrisA -- http://mail.python.org/mailman/listinfo/python-list
Re: python 3 problem: how to convert an extension method into a class Method
On 27/02/2013 10:49, Chris Angelico wrote: On Wed, Feb 27, 2013 at 9:36 PM, Robin Becker ro...@reportlab.com wrote: However, in my case the method takes py C utf8 bytes50 20 usec unicode 39 15 here py refers to a native python method and C to the extension method after adding to the class. Both are called via an instance of the class. Which raises the obvious question: Does it even matter? Will the saving of a few microseconds really make a difference? Python's best feature is its clarity of code, not its blazing performance; its performance goal is fast enough, and for many MANY purposes, you won't be able to tell the difference between that and awesome. Don't sacrifice your code's clarity to the little tin god of efficiency until you're sure you actually get something back. ChrisA in fact this is the stringWidth function and it's used thousands of times. I think when we did benchmark tests it came out as 1 or 2 as a cpu hog. Since it's comparatively easy to code it's an obvious choice to move to C. -- Robin Becker -- http://mail.python.org/mailman/listinfo/python-list
Re: python 3 problem: how to convert an extension method into a class Method
Robin Becker wrote: On 26/02/2013 18:38, Peter Otten wrote: Robin Becker wrote: ...3: $ python -m timeit -s 'from new import instancemethod from math import sqrt class A(int): pass A.m = instancemethod(sqrt, None, A) a = A(42) ' 'a.m()' 100 loops, best of 3: 0.5 usec per loop $ python -m timeit -s 'from math import sqrt class A(int): def m(self): return sqrt(self) a = A(42) ' 'a.m()' 100 loops, best of 3: 0.473 usec per loop this analysis might be relevant if I wanted to use sqrt. However, in my case the method takes py C utf8 bytes50 20 usec unicode 39 15 here py refers to a native python method and C to the extension method after adding to the class. Both are called via an instance of the class. I think you misunderstood. You compare the time it takes to run the function coded in C and its Python equivalent -- that difference is indeed significant. But what I was trying to measure was the difference between two ways to wrap the C function: Given a function cfunc implemented in C and the two ways of turning it into a method (1) class A(object): pass def method(self, ...): return cfunc(self, ...) A.method = method (2) class A(object): pass A.method = new.instancemethod(cfunc, None, A) I interpreted my timeit results as an indication that both ways have roughly the same overhead (method (1) being 0.027 usec faster). I don't have your code available, that's why I picked math.sqrt as an example for cfunc. I expect that you will get a similar result with your actual cfunc and therefore can (and should IMO) use method (1) in both Python 2 and 3 -- but of course not being able to measure it myself it may turn out I'm wrong. -- http://mail.python.org/mailman/listinfo/python-list
Re: python 3 problem: how to convert an extension method into a class Method
On 27/02/2013 11:14, Peter Otten wrote: I think you misunderstood. You compare the time it takes to run the function coded in C and its Python equivalent -- that difference is indeed significant. indeed. The function call overhead there looks pretty small so perhaps that's the way forward. But what I was trying to measure was the difference between two ways to wrap the C function: I expect that you will get a similar result with your actual cfunc and therefore can (and should IMO) use method (1) in both Python 2 and 3 -- but of course not being able to measure it myself it may turn out I'm wrong. Thanks, I suspect function calls have got faster in later pythons so my fear of the wrapper is outdated. As others have pointed out I should probably be creating a method in C then I should be able to do something like class A: def meth(self): . from extension import c_dummy_class A.meth = c_dummy_class.c_meth -- Robin Becker -- http://mail.python.org/mailman/listinfo/python-list
Re: python 3 problem: how to convert an extension method into a class Method
On Wed, 27 Feb 2013 21:49:04 +1100, Chris Angelico wrote: On Wed, Feb 27, 2013 at 9:36 PM, Robin Becker ro...@reportlab.com wrote: However, in my case the method takes py C utf8 bytes50 20 usec unicode 39 15 here py refers to a native python method and C to the extension method after adding to the class. Both are called via an instance of the class. Which raises the obvious question: Does it even matter? Will the saving of a few microseconds really make a difference? Python's best feature is its clarity of code, not its blazing performance; its performance goal is fast enough, and for many MANY purposes, you won't be able to tell the difference between that and awesome. Don't sacrifice your code's clarity to the little tin god of efficiency until you're sure you actually get something back. While this is true, I point out that we're not really talking about a few microseconds, but over 15µs per call, more than doubling the speed of the function call. That's not insignificant, and calls to that function could well be a bottleneck determining the application's total processing time. Robin's very first sentence in this thread states: In python 2 I was able to improve speed of reportlab using a C extension to optimize some heavily used methods. which suggests rather strongly that, yes, it does matter. Re-writing critical sections of code in C is a well-known, recommended approach to speeding up Python. You will see it in the Python library, where there are accelerated C versions of modules such as decimal, pickle and bisect. There's a standard library module in CPython for calling C code directly from your Python code (ctypes), and at least two others (cffi from PyPy, and weave), at least two projects for interfacing Python with C or C++ (boost, swig), and three projects for writing C code using Python-like syntax (pyrex, cython and numba). -- Steven -- http://mail.python.org/mailman/listinfo/python-list
python 3 problem: how to convert an extension method into a class Method
In python 2 I was able to improve speed of reportlab using a C extension to optimize some heavily used methods. so I was able to do this class A: . def method(self,...): try: from extension import c_method import new A.method = new.instancemethod(c_method,None,A) except: pass and if the try succeeds our method is bound as a class method ie is unbound and works fine when I call it. In python 3 this doesn't seem to work at all. In fact the new module is gone. The types.MethodType stuff doesn't seem to work. Is there a way in Python 3.3 to make this happen? This particular method is short, but is called many times so adding python wrapping layers is not a good way forward. If the above cannot be made to work (another great victory for Python 3) then is there a way to bind an external method to the instance without incurring too much overhead. Alternatively could it make sense to implement an accelerated basetype that just contains the accelerated methods of class A. I could then imagine doing something like try: from extension import class c_baseA as baseA except: class baseA: def method() class A(baseA): . presumably I then get some kind of penalty for the base class lookup, but how bad is that? -- Robin Becker -- http://mail.python.org/mailman/listinfo/python-list
Re: python 3 problem: how to convert an extension method into a class Method
On 02/26/2013 12:21 PM, Robin Becker wrote: In python 2 I was able to improve speed of reportlab using a C extension to optimize some heavily used methods. so I was able to do this class A: That creates an old-style class in Python 2.x. They've been obsolete for many years. You want to derive from object: class A(object): Once you do, Python 2 and Python 3 should work the same. I can't help with the rest, unfortunately. . def method(self,...): try: from extension import c_method import new A.method = new.instancemethod(c_method,None,A) except: pass and if the try succeeds our method is bound as a class method ie is unbound and works fine when I call it. In python 3 this doesn't seem to work at all. In fact the new module is gone. The types.MethodType stuff doesn't seem to work. Is there a way in Python 3.3 to make this happen? This particular method is short, but is called many times so adding python wrapping layers is not a good way forward. If the above cannot be made to work (another great victory for Python 3) then is there a way to bind an external method to the instance without incurring too much overhead. Alternatively could it make sense to implement an accelerated basetype that just contains the accelerated methods of class A. I could then imagine doing something like try: from extension import class c_baseA as baseA except: class baseA: def method() class A(baseA): . presumably I then get some kind of penalty for the base class lookup, but how bad is that? -- DaveA -- http://mail.python.org/mailman/listinfo/python-list
Re: python 3 problem: how to convert an extension method into a class Method
Robin Becker wrote: In python 2 I was able to improve speed of reportlab using a C extension to optimize some heavily used methods. so I was able to do this class A: . def method(self,...): try: from extension import c_method import new A.method = new.instancemethod(c_method,None,A) except: pass and if the try succeeds our method is bound as a class method ie is unbound and works fine when I call it. In python 3 this doesn't seem to work at all. In fact the new module is gone. The types.MethodType stuff doesn't seem to work. Is there a way in Python 3.3 to make this happen? This particular method is short, but is called many times so adding python wrapping layers is not a good way forward. If the above cannot be made to work (another great victory for Python 3) then is there a way to bind an external method to the instance without incurring too much overhead. Hm, according to my random measurement your clever approach incurs more overhead than the straight-forward way that continues to work in Python 3: $ python -m timeit -s 'from new import instancemethod from math import sqrt class A(int): pass A.m = instancemethod(sqrt, None, A) a = A(42) ' 'a.m()' 100 loops, best of 3: 0.5 usec per loop $ python -m timeit -s 'from math import sqrt class A(int): def m(self): return sqrt(self) a = A(42) ' 'a.m()' 100 loops, best of 3: 0.473 usec per loop -- http://mail.python.org/mailman/listinfo/python-list
Re: python 3 problem: how to convert an extension method into a class Method
On 26/02/2013 18:38, Peter Otten wrote: Robin Becker wrote: In python 2 I was able to improve speed of reportlab using a C extension to optimize some heavily used methods. so I was able to do this class A: . def method(self,...): try: from extension import c_method import new A.method = new.instancemethod(c_method,None,A) except: pass and if the try succeeds our method is bound as a class method ie is unbound and works fine when I call it. In python 3 this doesn't seem to work at all. In fact the new module is gone. The types.MethodType stuff doesn't seem to work. Is there a way in Python 3.3 to make this happen? This particular method is short, but is called many times so adding python wrapping layers is not a good way forward. If the above cannot be made to work (another great victory for Python 3) then is there a way to bind an external method to the instance without incurring too much overhead. Hm, according to my random measurement your clever approach incurs more overhead than the straight-forward way that continues to work in Python 3: $ python -m timeit -s 'from new import instancemethod from math import sqrt class A(int): pass A.m = instancemethod(sqrt, None, A) a = A(42) ' 'a.m()' 100 loops, best of 3: 0.5 usec per loop $ python -m timeit -s 'from math import sqrt class A(int): def m(self): return sqrt(self) a = A(42) ' 'a.m()' 100 loops, best of 3: 0.473 usec per loop c:\Users\Mark\MyPythonpython Python 3.3.0 (v3.3.0:bd8afb90ebf2, Sep 29 2012, 10:55:48) [MSC v.1600 32 bit (Intel)] on win32 Type help, copyright, credits or license for more information. import new Traceback (most recent call last): File stdin, line 1, in module ImportError: No module named 'new' -- Cheers. Mark Lawrence -- http://mail.python.org/mailman/listinfo/python-list
Re: python 3 problem: how to convert an extension method into a class Method
Mark Lawrence wrote: On 26/02/2013 18:38, Peter Otten wrote: Robin Becker wrote: In python 2 I was able to improve speed of reportlab using a C extension to optimize some heavily used methods. so I was able to do this class A: . def method(self,...): try: from extension import c_method import new A.method = new.instancemethod(c_method,None,A) except: pass and if the try succeeds our method is bound as a class method ie is unbound and works fine when I call it. In python 3 this doesn't seem to work at all. In fact the new module is gone. The types.MethodType stuff doesn't seem to work. Is there a way in Python 3.3 to make this happen? This particular method is short, but is called many times so adding python wrapping layers is not a good way forward. If the above cannot be made to work (another great victory for Python 3) then is there a way to bind an external method to the instance without incurring too much overhead. Hm, according to my random measurement your clever approach incurs more overhead than the straight-forward way that continues to work in Python 3: $ python -m timeit -s 'from new import instancemethod from math import sqrt class A(int): pass A.m = instancemethod(sqrt, None, A) a = A(42) ' 'a.m()' 100 loops, best of 3: 0.5 usec per loop $ python -m timeit -s 'from math import sqrt class A(int): def m(self): return sqrt(self) a = A(42) ' 'a.m()' 100 loops, best of 3: 0.473 usec per loop c:\Users\Mark\MyPythonpython Python 3.3.0 (v3.3.0:bd8afb90ebf2, Sep 29 2012, 10:55:48) [MSC v.1600 32 bit (Intel)] on win32 Type help, copyright, credits or license for more information. import new Traceback (most recent call last): File stdin, line 1, in module ImportError: No module named 'new' I did the timing in Python 2 of course, to demonstrate that the feature the OP is missing in Python 3 offers no advantage in the Python version where it /is/ available. Does my previous post make sense now? -- http://mail.python.org/mailman/listinfo/python-list
Re: python 3 problem: how to convert an extension method into a class Method
On 02/26/2013 09:21 AM, Robin Becker wrote: In python 2 I was able to improve speed of reportlab using a C extension to optimize some heavily used methods. so I was able to do this class A: . def method(self,...): try: from extension import c_method import new A.method = new.instancemethod(c_method,None,A) except: pass and if the try succeeds our method is bound as a class method ie is unbound and works fine when I call it. In python 3 this doesn't seem to work at all. In fact the new module is gone. The types.MethodType stuff doesn't seem to work. Is there a way in Python 3.3 to make this happen? This particular method is short, but is called many times so adding python wrapping layers is not a good way forward. Dumb question, but have you tried just assigning it? In Py3 methods are just normal functions... 8-- class A(): pass A.method = c_method 8-- -- ~Ethan~ -- http://mail.python.org/mailman/listinfo/python-list
Re: python 3 problem: how to convert an extension method into a class Method
Ethan Furman wrote: On 02/26/2013 09:21 AM, Robin Becker wrote: In python 2 I was able to improve speed of reportlab using a C extension to optimize some heavily used methods. so I was able to do this class A: . def method(self,...): try: from extension import c_method import new A.method = new.instancemethod(c_method,None,A) except: pass and if the try succeeds our method is bound as a class method ie is unbound and works fine when I call it. In python 3 this doesn't seem to work at all. In fact the new module is gone. The types.MethodType stuff doesn't seem to work. Is there a way in Python 3.3 to make this happen? This particular method is short, but is called many times so adding python wrapping layers is not a good way forward. Dumb question, but have you tried just assigning it? In Py3 methods are just normal functions... 8-- class A(): pass A.method = c_method 8-- The problem is that functions implemented in C don't support the descriptor protocol (they don't have a __get__() method). So from math import sqrt class A(int): ... pass ... A.c = sqrt A.py = lambda self: sqrt(self) a = A(42) a.py() 6.48074069840786 a.c() Traceback (most recent call last): File stdin, line 1, in module TypeError: sqrt() takes exactly one argument (0 given) -- http://mail.python.org/mailman/listinfo/python-list
Re: python 3 problem: how to convert an extension method into a class Method
Am 26.02.2013 21:19, schrieb Ethan Furman: Dumb question, but have you tried just assigning it? In Py3 methods are just normal functions... 8-- class A(): pass A.method = c_method 8-- That doesn't work with builtin functions because they don't implement the descriptor protocol: Python 3.3.0 (v3.3.0:bd8afb90ebf2, Feb 8 2013, 00:38:29) [GCC 4.7.2] on linux Type help, copyright, credits or license for more information. class Example: ... id = id ... Example().id() Traceback (most recent call last): File stdin, line 1, in module TypeError: id() takes exactly one argument (0 given) You can use PyInstanceMethod_Type to wrap a builtin method in something that acts like a normal method. I implemented the type for Python 3000 when I removed the unbound method object. It's not available to pure Python code except for testing: import _testcapi class Example: ... id = _testcapi.instancemethod(id) ... Example().id() 140525206026320 The C code is rather simple and small. You can easily re-implement in Python, too. -- http://mail.python.org/mailman/listinfo/python-list
Re: python 3 problem: how to convert an extension method into a class Method
On Tue, 26 Feb 2013 17:21:16 +, Robin Becker wrote: In python 2 I was able to improve speed of reportlab using a C extension to optimize some heavily used methods. so I was able to do this class A: . def method(self,...): try: from extension import c_method import new A.method = new.instancemethod(c_method,None,A) except: pass Why are you suppressing and ignoring arbitrary errors here? That doesn't sound good. Surely a better way would be: import new try: from extension import c_method except ImportError: pass else: A.method = new.instancemethod(c_method, None, A) and if the try succeeds our method is bound as a class method ie is unbound and works fine when I call it. In python 3 this doesn't seem to work at all. In fact the new module is gone. The types.MethodType stuff doesn't seem to work. I've never tried this with a function written in C, but for one written in Python all you need is this: A.method = c_method Try that and see if it works with your function written in C. I expect that it will, provided that the function is written as a method descriptor. I don't know enough about C extensions to tell you how to do that, sorry. -- Steven -- http://mail.python.org/mailman/listinfo/python-list
