Anyone tracking the homepage of speed.pypy.org will notice a drop in the pypy-vs-cpython ratio from ~4.1x to ~3.5x. TL;DR: I am redoing the cpython runs and the results changed. They will change a bit more going forward.
Part of the motivation to move to benchmarker2 was to udate all the things and get gcc14 into the builds. This worked and the recent releases of pypy8 on linux use gcc14. But... I wrote some scripting to run the benchmarker2 buildbot since I want it to run inside a vm, pinning to cpus dedicated only to benchmarking. This is the https://github.com/pypy/buildbot/blob/main/start-benchmarker2.sh script. It had a typo: the PATH addition was not properly escaped. All the benchmarks run since benchmarker2 started have used the system gcc 8.5 rather than the /opt/rh/gcc-toolset-14 gcc14. I noticed this when trying to properly run cpython 3.11 and 3.12 with both our legacy benchmarks and the pyperformance ones. Rather than the manual benchmark runs I have been doing in the past, I added a buildbot runner for CPython. This exposed a further typo in the RPATH escaping I was using, so the enable-shared builds of CPython may not have been using the libpython I wanted. Where we stand now? I reran the 3.11 and 3.12 CPython benchmarks with the RPATH fix and replaced the old results. This moved the needle on the pypy vs cpython comparisons and dropped PyPy from 4.1x to 3.5x CPython, where it was on the older benchmarker machine. But that was all with gcc8.5. So now there is a new environment 'benchmarker2-gcc14' (hidden under the "historical" accordion on the comparison, timeline, changes pages). This uses gcc14. Eventually, when we have enough results to get some kind of a timeline, I will flip the names, benchmarker2 will become benchmarker2-gcc8.5 and benchmarker2-gcc14 will become benchmarker2. I think the bottom line will stay somewhere around 3.5x for legacy benchmarks. The whole point of this exercise is to slowly move to pyperformance benchmark comparisons. Sorry for the mess. Matti _______________________________________________ pypy-dev mailing list -- [email protected] To unsubscribe send an email to [email protected] https://mail.python.org/mailman3//lists/pypy-dev.python.org Member address: [email protected]
