It looks like they aren't run on the run that runs on every pull request https://github.com/sympy/sympy/blob/958cc95aebc35ced70d870586e68d0468684c1a0/asv.conf.actions.json#L67
There are some separate asv.conf.slow.json configurations in the benchmarks repo for people who want to run the slow benchmarks. The benchmarks are still run in the normal asv way of running along a history of commits from time to time, but this is only done manually when someone is interested in doing it. A lot of these benchmarks are only there historically. Many of the SymPy benchmarks predate asv and were moved from the SymPy repo to the benchmarks repo. They need to be cleaned up so that they aren't as slow. In general, I wouldn't necessarily take the SymPy benchmarks to be the gold standard in benchmark suites. There is quite a lot of work that needs to be done there to clean things up and write more/better benmarks, but no one has really had time to do it except on a case by case basis when a benchmark is needed for something or other. It's hard because writing good benchmarks is hard and also because there are literally tens of thousands of functions and behaviors in SymPy that could be benchmarked, so choosing the right things to benchmark is important. Aaron Meurer On Thu, Jul 25, 2024 at 11:40 AM Asish Kumar <officialasishku...@gmail.com> wrote: > > I was recently checking https://github.com/sympy/sympy_benchmarks and had > some questions: > > 1. Are slow_benchmarks are also run by asv? > 2. If yes, why it's in a seperate folder? > 3. If No, why it's there? > > On Tuesday, July 16, 2024 at 12:00:44 AM UTC+5:30 asme...@gmail.com wrote: >> >> Looking at your linked GitHub action, you run the benchmark in the >> last 5 commits. SymPy's CI for PRs only runs the benchmarks for the PR >> and master. >> >> Aside from this, it's dependent on how long your benchmarks take to >> run. Someone should probably write a guide on how to make a good >> benchmark. Ideally a benchmark shouldn't take more than a second to >> run, and that's at the very high end (10 or 100 ms is better), because >> it needs to be run multiple times to get an average runtime. >> Benchmarks that run too quickly are also bad because you are basically >> benchmarking noise. >> >> Also, if TARDIS has to be recompiled every time in order to install it >> into the virtual environment, that could be a factor (SymPy is pure >> Python, so installation time is not an issue). If this is an issue you >> might consider caching nightly wheels for recent commits somewhere. >> >> Aaron Meurer >> >> On Mon, Jul 15, 2024 at 1:14 AM Asish Kumar >> <officiala...@gmail.com> wrote: >> > >> > Hi there, >> > >> > I'm a GSoC student with TARDIS-SN this summer and I had a issue with the >> > long time it takes to run benchmarks. Currently my benchmarks are getting >> > run for 1.5 hrs (for only a single commit). I was wondering how sympy does >> > it? >> > >> > I checked the benchmark actions of sympy and was pretty surprised to see >> > that it only takes 20 mins to run the whole benchmark. >> > >> > Thank you >> > Asish Kumar >> > >> > -- >> > You received this message because you are subscribed to the Google Groups >> > "sympy" group. >> > To unsubscribe from this group and stop receiving emails from it, send an >> > email to sympy+un...@googlegroups.com. >> > To view this discussion on the web visit >> > https://groups.google.com/d/msgid/sympy/1ff21b7f-382e-4d73-90fd-85316b9ca5bdn%40googlegroups.com. > > -- > You received this message because you are subscribed to the Google Groups > "sympy" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to sympy+unsubscr...@googlegroups.com. > To view this discussion on the web visit > https://groups.google.com/d/msgid/sympy/34cb7637-4acd-4ade-8b14-b684edaf31d2n%40googlegroups.com. -- You received this message because you are subscribed to the Google Groups "sympy" group. To unsubscribe from this group and stop receiving emails from it, send an email to sympy+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/sympy/CAKgW%3D6Lt%3DEmbk20xhY5q5YFMCyAOtRU2QspKj5Hy9WL71e%3DjSw%40mail.gmail.com.
