> Would it be possible rerun the tests with the current setup for say the last 1000 revisions or perhaps a subset of these (e.g. every 10th revision) to try to binary search for the revision which introduced the change ?
Every run takes 1-2 h so doing 1000 would be certainly time-consuming :) That's why from now on I am trying to invest in daily builds for master, so we can answer that exact question if we detect regressions in the future. On Wed, 14 Oct 2020 at 15:04, M.-A. Lemburg <m...@egenix.com> wrote: > On 14.10.2020 16:00, Pablo Galindo Salgado wrote: > >> Would it be possible to get the data for older runs back, so that > > it's easier to find the changes which caused the slowdown ? > > > > Unfortunately no. The reasons are that that data was misleading because > > different points were computed with a different version of pyperformance > and > > therefore with different packages (and therefore different code). So the > points > > could not be compared among themselves. > > > > Also, past data didn't include 3.9 commits because the data gathering > was not > > automated and it didn't run in a long time :( > > Make sense. > > Would it be possible rerun the tests with the current > setup for say the last 1000 revisions or perhaps a subset of these > (e.g. every 10th revision) to try to binary search for the revision which > introduced the change ? > > > On Wed, 14 Oct 2020 at 14:57, M.-A. Lemburg <m...@egenix.com > > <mailto:m...@egenix.com>> wrote: > > > > Hi Pablo, > > > > thanks for pointing this out. > > > > Would it be possible to get the data for older runs back, so that > > it's easier to find the changes which caused the slowdown ? > > > > Going to the timeline, it seems that the system only has data > > for Oct 14 (today): > > > > > https://speed.python.org/timeline/#/?exe=12&ben=regex_dna&env=1&revs=1000&equid=off&quarts=on&extr=on&base=none > > > > In addition to unpack_sequence, the regex_dna test has slowed > > down a lot compared to Py3.8. > > > > > https://github.com/python/pyperformance/blob/master/pyperformance/benchmarks/bm_unpack_sequence.py > > > https://github.com/python/pyperformance/blob/master/pyperformance/benchmarks/bm_regex_dna.py > > > > Thanks. > > > > On 14.10.2020 15:16, Pablo Galindo Salgado wrote: > > > Hi! > > > > > > I have updated the branch benchmarks in the pyperformance server > and now they > > > include 3.9. There are > > > some benchmarks that are faster but on the other hand some > benchmarks are > > > substantially slower, pointing > > > at a possible performance regression in 3.9 in some aspects. In > particular > > some > > > tests like "unpack sequence" are > > > almost 20% slower. As there are some other tests were 3.9 is > faster, is > > not fair > > > to conclude that 3.9 is slower, but > > > this is something we should look into in my opinion. > > > > > > You can check these benchmarks I am talking about by: > > > > > > * Go here: https://speed.python.org/comparison/ > > > * In the left bar, select "lto-pgo latest in branch '3.9'" and > "lto-pgo latest > > > in branch '3.8'" > > > * To better read the plot, I would recommend to select a > "Normalization" > > to the > > > 3.8 branch (this is in the top part of the page) > > > and to check the "horizontal" checkbox. > > > > > > These benchmarks are very stable: I have executed them several > times over the > > > weekend yielding the same results and, > > > more importantly, they are being executed on a server specially > prepared to > > > running reproducible benchmarks: CPU affinity, > > > CPU isolation, CPU pinning for NUMA nodes, CPU frequency is fixed, > CPU > > governor > > > set to performance mode, IRQ affinity is > > > disabled for the benchmarking CPU nodes...etc so you can trust > these numbers. > > > > > > I kindly suggest for everyone interested in trying to improve the > 3.9 (and > > > master) performance, to review these benchmarks > > > and try to identify the problems and fix them or to find what > changes > > introduced > > > the regressions in the first place. All benchmarks > > > are the ones being executed by the pyperformance suite > > > (https://github.com/python/pyperformance) so you can execute them > > > locally if you need to. > > > > > > --- > > > > > > On a related note, I am also working on the speed.python.org > > <http://speed.python.org> > > > <http://speed.python.org> server to provide more automation and > > > ideally some integrations with GitHub to detect performance > regressions. For > > > now, I have done the following: > > > > > > * Recompute benchmarks for all branches using the same version of > > > pyperformance (except master) so they can > > > be compared with each other. This can only be seen in the > "Comparison" > > > tab: https://speed.python.org/comparison/ > > > * I am setting daily builds of the master branch so we can detect > performance > > > regressions with daily granularity. These > > > daily builds will be located in the "Changes" and "Timeline" > tabs > > > (https://speed.python.org/timeline/). > > > * Once the daily builds are working as expected, I plan to work on > trying to > > > automatically comment or PRs or on bpo if > > > we detect that a commit has introduced some notable performance > regression. > > > > > > Regards from sunny London, > > > Pablo Galindo Salgado. > > > > > > _______________________________________________ > > > python-committers mailing list -- python-committ...@python.org > > <mailto:python-committ...@python.org> > > > To unsubscribe send an email to python-committers-le...@python.org > > <mailto:python-committers-le...@python.org> > > > > https://mail.python.org/mailman3/lists/python-committers.python.org/ > > > Message archived at > > > https://mail.python.org/archives/list/python-committ...@python.org/message/G3LB4BCAY7T7WG22YQJNQ64XA4BXBCT4/ > > > Code of Conduct: https://www.python.org/psf/codeofconduct/ > > > > > > > -- > > Marc-Andre Lemburg > > eGenix.com > > > > Professional Python Services directly from the Experts (#1, Oct 14 > 2020) > > >>> Python Projects, Coaching and Support ... > https://www.egenix.com/ > > >>> Python Product Development ... > https://consulting.egenix.com/ > > > ________________________________________________________________________ > > > > ::: We implement business ideas - efficiently in both time and costs > ::: > > > > eGenix.com Software, Skills and Services GmbH Pastor-Loeh-Str.48 > > D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg > > Registered at Amtsgericht Duesseldorf: HRB 46611 > > https://www.egenix.com/company/contact/ > > https://www.malemburg.com/ > > > > -- > Marc-Andre Lemburg > eGenix.com > > Professional Python Services directly from the Experts (#1, Oct 14 2020) > >>> Python Projects, Coaching and Support ... https://www.egenix.com/ > >>> Python Product Development ... https://consulting.egenix.com/ > ________________________________________________________________________ > > ::: We implement business ideas - efficiently in both time and costs ::: > > eGenix.com Software, Skills and Services GmbH Pastor-Loeh-Str.48 > D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg > Registered at Amtsgericht Duesseldorf: HRB 46611 > https://www.egenix.com/company/contact/ > https://www.malemburg.com/ > >
_______________________________________________ Python-Dev mailing list -- python-dev@python.org To unsubscribe send an email to python-dev-le...@python.org https://mail.python.org/mailman3/lists/python-dev.python.org/ Message archived at https://mail.python.org/archives/list/python-dev@python.org/message/LBEAVPI5WT6ZV5RKCKHW3EWXLDY534IQ/ Code of Conduct: http://python.org/psf/codeofconduct/
