Thank you Matti for your answer. ----- Mail original ----- > De: "Matti Picus" <[email protected]> > À: "PIERRE AUGIER" <[email protected]> > Cc: "hpy-dev" <[email protected]> > Envoyé: Vendredi 15 Novembre 2024 16:59:44 > Objet: Re: [hpy-dev] Future of HPy? Faster CPython, and comparison between > different Python interpreters
>> Is it (PyPy) also measured with https://pyperformance.readthedocs.io/ ? > > We run a different version of those benchmarks at speed.pypy.org, > where you can also see a comparison to CPython 3.11 and CPython 3.12. > Our benchmarks still run on Python2.7 (well, PyPy 2.7) since PyPy is > written in RPython which is a python2.7 variant. IMHO, it's a bit unfortunate that this implementation detail has such consequence. It would be better if the different Python implementations can work together on the same benchmark tool (I guess https://pyperformance.readthedocs.io/, maybe adapted to be more friendly with alternative implementations). >> Do you think it would be interesting and feasible to have a website showing >> fair >> comparisons between different Python interpreters ? > > Much ink has been spilled over what is "fair". The different > interpreters have different warmup characteristics. Speed is not > always the only factor: some people care about memory usage or other > parameters. And even getting consistent benchmarks over time is > difficult. If the goal is to track performance over time in a > sandboxed consistent environment, I think there is value in the > proposal. But a one-off "hey X is N.NN faster than Y" is going to be > problematic. Yes, one number is clearly insuffisant to summarize the performance of an implementation. I still think that we have an issue about the communication on the performance comparison between different Python implementations, which is bad for the global understanding of the community that users would benefit to have an ecosystem more open to alternative implementations. First, there are different websites for the benchmarks of the different implementations with (slightly) different benchmarks, methodologies and bases for comparison: - https://pypy.org/ and https://speed.pypy.org/ for PyPy (based on PyPy benchmarks, compatible with Python 2.7) - https://github.com/faster-cpython/benchmarking-public for CPython (based on https://github.com/python/pyperformance) - A figure on https://github.com/oracle/graalpython for GraalPy (based on https://github.com/python/pyperformance) - https://speed.python.org/ (not up-to-date, complicated and not very readable figures) Second, it is very difficult with what we have to go beyond comparisons with 1 number "X is N.NN faster than Y", especially for people that do not know the details of pyperformance benchmarks. It would be much better to be able to present something much more quantitative with few numbers associated with groups of benchmarks for different types of applications (for example, Pure Python, numerics with C extensions, startup, web servers, ...). For some cases, it would also be useful to have simple comparisons with other technologies (based on Python or on another language). Of course, I realize that I just point out a difficulty and do not bring any solution, but it might be a first step to improve the situation. > I will let others answer as for HPy. > Matti > > > On Fri, Nov 15, 2024 at 5:32 PM PIERRE AUGIER > <[email protected]> wrote: >> >> Hi HPy developers, >> >> I send this email here since it can reach people interested in HPy, and >> PyPy/GraalPy developers. >> >> CPython 3.13 has an experimental JIT. The results in term of global >> performance >> can be followed here https://github.com/faster-cpython/benchmarking-public. >> >> It is still very modest (best results are something like 1.4 time faster than >> CPython 3.10). However, there are other plans to improve CPython new JIT >> (https://github.com/faster-cpython/ideas/blob/main/3.14/README.md, see also >> https://github.com/faster-cpython/ideas/issues/701#issuecomment-2405979384) >> so >> we can guess that CPython 3.14 and 3.15 will be (a bit?) faster. >> >> In https://github.com/faster-cpython/ideas/blob/main/3.14/README.md, I read >> with >> interest that HPy is mentioned! >> >> #### HPy-like C API >> >> Tagged integers will require new internal APIs to pass opaque references, >> so we >> might as well add a new, consistent API modeled on >> [HPy](https://hpyproject.org/). >> We will use this internally to start with, but we expect tools like >> [Cython](https://cython.org/) and >> [Pybind11](https://github.com/pybind/pybind11) will want to use it to >> avoid the >> overhead of going through the `PyObject *` based API. >> >> >> So I was wondering about the state of HPy. For example, is there a plan to >> port >> Numpy 2 in HPy? What about HPy support in Cython? Do you still think HPy >> could >> become more popular and used? >> >> >> Another related question is about the performance comparison between >> different >> interpreters. It seems to me that the difference of performance between >> CPython >> and other alternative interpreters is a strong argument for HPy. >> >> Seeing the performance comparisons with different versions and commits of >> CPython, I was wondering about similar comparisons with other interpreters >> (in >> practice PyPy and GraalPy). It would be very interesting to know from fair >> benchmarks about the performance of different interpreters. >> >> In https://github.com/oracle/graalpython, it's written "GraalPy is ~4x faster >> than CPython on the [official Python Performance Benchmark >> Suite](https://pyperformance.readthedocs.io/)". >> >> In https://pypy.org/, there is written "PyPy is 4.4 times faster than CPython >> 3.7", but comparing to CPython 3.7 does not make sense nowadays. Is it also >> measured with https://pyperformance.readthedocs.io/ ? >> >> Do you think it would be interesting and feasible to have a website showing >> fair >> comparisons between different Python interpreters ? >> >> I guess a simple solution would be to use faster-cpython workflows by asking >> them to also consider stable releases of PyPy and GraalPy. I'm going to ask >> in >> https://github.com/faster-cpython/ideas. >> >> Would there be a good alternative solution to get such data? >> >> Pierre Augier >> >> _______________________________________________ >> hpy-dev mailing list -- [email protected] >> To unsubscribe send an email to [email protected] >> https://mail.python.org/mailman3/lists/hpy-dev.python.org/ > > Member address: [email protected] _______________________________________________ hpy-dev mailing list -- [email protected] To unsubscribe send an email to [email protected] https://mail.python.org/mailman3/lists/hpy-dev.python.org/ Member address: [email protected]
