On Fri, Apr 15, 2022 at 6:54 AM Oscar Benjamin <oscar.j.benja...@gmail.com> wrote: > > On Fri, 11 Mar 2022 at 14:53, praneeth ratna <praneeth.ra...@gmail.com> wrote: > > > > Hi all, > > > > As mentioned in the project description here , > > https://github.com/sympy/sympy/wiki/GSoC-Ideas#benchmarks-and-performance , > > the sympy_benchmarks misses some benchmarks which are needed to be shifted > > fro the sympy main repo. Also the sympy_benchmarks misses benchmarks for > > combinatorics, series , stats and many other modules which can be > > added.(Please correct me here) > > Also I did not have a clear understanding of what the new implementation of > > benchmarks do and how it should be done. > > I think that with some of the project ideas the intention is not > necessarily that anyone has a clear idea of exactly what should be > done but that someone wanting to do a project would do some research > themselves to come up with an idea. > > In relation to benchmarks the immediate issues I see are: > > 1. Some old benchmarking code is in the main repo and should be moved > over to the benchmarks repo. > 2. The benchmarks repo is currently broken because some test fails so > no PR can be merged there until that is fixed. > 3. The benchmarks are affected by caching which means that many of the > timings reported are completely unrepresentative. > > Bigger issues:
For me the biggest issue with the benchmarks right now is that they aren't very effective towards actually making SymPy faster, and preventing it from getting slower (which are related but distinct things). So any ideas suggested for a benchmarking project should, in my opinion, be somehow linked to this end goal. This includes things like - Making the benchmarks easier to run (including running them automatically) - Making it easier to add benchmarks - Making it easier to interpret the results of benchmarks - Making the benchmarks suite itself more likely to catch performance regressions > > 1. The benchmarks are based on ASV which is awkward to use. It's > unnecessarily difficult even just to run a single benchmark. A better > framework should be found. Just a small note on this point. I've done a small bit of research on this myself and have come up with nothing. It's possible something better does exist out there, but it's also likely that it doesn't and if we want something better we will have to write it ourselves. > 2. The benchmarks mostly only measure things that are already quite > fast because otherwise they would take too long to run. To guide > future improvements we really need benchmarks for things that are > currently slow. > 3. The benchmarks are too tied up with sympy itself. I would rather > have an independent benchmark suite that collects good examples and > can be used by other projects as well. For this you need benchmarks > that are language agnostic and not simply written in sympy code. (I > also think it would be better if a lot of the test suite was like this > as well.) > 4. The things that are tested in the benchmarks are a bit random. It > would be better to focus on core features like differentiation, > integration, linear algebra, polynomials, solvers, numerical > evaluation. > > A big part of the difficulty in making good benchmarks is that it > really requires having some understanding of what are the core > operations that should or could be made faster. This is true, but there's also enough work to be done on the benchmarking tooling itself that most of the work could focus on that, rather than necessarily adding too many actual benchmarks, which could be done later. Aaron Meurer > > -- > Oscar > > -- > You received this message because you are subscribed to the Google Groups > "sympy" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to sympy+unsubscr...@googlegroups.com. > To view this discussion on the web visit > https://groups.google.com/d/msgid/sympy/CAHVvXxRPFjmZBrkJpusB-WER3V%3DMd%2B%2BP_X8HKAuHHJr3PPXe7Q%40mail.gmail.com. -- You received this message because you are subscribed to the Google Groups "sympy" group. To unsubscribe from this group and stop receiving emails from it, send an email to sympy+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/sympy/CAKgW%3D6%2BseAxDJQpMOXiubj7c_2sDJFRJp2D_8LVVKE6WOLpOoQ%40mail.gmail.com.
