Maciej Fijalkowski, 13.12.2012 19:23: > On Thu, Dec 13, 2012 at 7:21 PM, Stefan Behnel wrote: >> Maciej Fijalkowski, 13.12.2012 09:13: >>> On Thu, Dec 13, 2012 at 9:35 AM, Stefan Behnel wrote: >>>> Maciej Fijalkowski, 12.12.2012 20:10: >>>>> On Wed, Dec 12, 2012 at 7:06 PM, Joe Hillenbrand wrote: >>>>>> I was able to fix the issue with scrapy. >>>>>> >>>>>> https://github.com/joehillen/scrapy/commit/8778af5c5be50a5d746751352f8d710d1f24681c >>>>>> >>>>>> Unfortunately, scrapy takes twice as long in PyPy than in CPython. I >>>>>> suspect >>>>>> this is because lxml is twice as slow in PyPy vs CPython, which I found >>>>>> in >>>>>> lxml's benchmarks. >>>>>> >>>>>> Should lxml be added to the set of speed tests? >>>>> >>>>> no. lxml uses cpyext (CPython extension compatibility) that is and >>>>> will forever be slow. >>>> >>>> Well, I don't think it would be hard for any PyPy core developer to make it >>>> twice as fast. Shouldn't be more than a day's work. >>> >>> I'm not so sure, we wouldn't know until someone tries it. What >>> optimizations did you have in mind? >> >> Anything that creates a proper fast-path in the ref-counting functions and >> that generally takes pressure off them, e.g. by keeping PyObjects alive in >> a weakref dict as long as the corresponding PyPy object lives, so that >> useless re-allocation cycles are avoided. I'm sure that really simple >> changes can bring a substantial improvement here. > > short term allocations are usually very cheap.
In the profile I posted the last time we discussed this, I think it was pretty clear that most of the time is not currently being spent in the allocation but in the deallocation. More than 50% of the overall runtime in this case: http://cython.org/callgrind-pypy-nbody.png Plus, connecting the lifetime of PyObjects to that of PyPy objects would fix the problem that PyObjects can die prematurely and take C state with them: http://docs.cython.org/src/userguide/pypy.html My intuition was to add a fastpath to Py_DECREF() that would do (close to) nothing if the PyPy object is still alive. Either that, or move this whole decision into C by somehow increasing the C level refcount during the lifetime of the PyPy object and decreasing it when the PyPy object dies. The latter approach (if doable) is obviously preferable from a C point of view because it would improve the hit-count of the "common case" tests in the INCREF/DECREF C macros, thus avoiding unnecessary calls into PyPy all together by using inlined code. That would give it about the same speed as in CPython for objects that are being reused in C code more than once for which a PyPy object reference exists (certainly not an unusual case). Stefan _______________________________________________ pypy-dev mailing list [email protected] http://mail.python.org/mailman/listinfo/pypy-dev
