On 21 October 2010 01:33, David Roe <r...@math.harvard.edu> wrote:
> There are a number of tickets in trac about performance regressions in
> Sage. I'm sure there are far more performance regressions which we don't
> know about because nobody noticed.
I agree, and I've seen some comments from William that writing code
one way or another can change things by a factor of 100.
> As someone writing library code, it's generally not obvious that one is
> about to introduce a performance regression (otherwise you'd probably not do
> it).
Agreed.
> Consequently, I've been thinking recently about how to detect performance
> regressions automatically. There are really two parts to the problem:
> gathering timing data on the Sage library, and analyzing that data to
> determine if regression have occurred (and how serious they are).
>
>
> Data gathering:
>
> One could modify local/bin/sage-doctest to allow the option of changing each
> doctest by wrapping it in a "timeit()" call. This would then generate a
> timing datum for each doctest line.
> * these timings vary from run to run (presumably due to differing levels of
> load on the machine). I don't know how to account for this, but usually
> it's a fairly small effect (on the order of 10% error).
They would differ by a lot more than 10%. One of my machines is a Sage
buildbot client. If that is building Sage, and I'm not, Sage will take
about and hour to build and test. If I'm building Sage at the same or
similar time, or will increase that by a factor of at least two.
What is needed is to measure CPU time used. That should be relatively
stable and not depend too much on system load, though even there I
would not be surprised by changes of +/- 10%.
> * if you're testing against a previous version of sage, the doctest
> structure will have changed because people wrote more doctests. And doctest
> lines depend on each other: you define variables that are used in later
> lines. So inserting a line could make timings of later lines incomparable
> to the exact same line without the inserted line. We might be able to parse
> the lines and check that various objects are actually the same (across
> different versions of sage, so this would require either a version-invariant
> hash or saving in one version, loading in the other and comparing. And you
> would have to do that for each variable that occurs in the line), but that
> seems to be getting too complicated...
Getting a checksum of each doctest would be easy. I suggest we use:
$ cksum sometest.py | awk '{print $1}'
because that will be totally portable across all platforms. 'cksum' is
32-bit checksum that's part of the POSIX standard and the algorithm is
defined. So there's no worry about whether one has an md5 program, and
if so what its called.
> One way of handling these problems is to create a relational database to put
> timings in. This could also be interesting for benchmarketing purposes: we
> could have timings on various machines, and we highlight performance
> improvements, in addition to watching for performance regressions.
Sounds good. But it could be a lot of work to implement.
> So, here's a first draft for a database schema to put timing data into.
> I've included a description of each table, along with a description of
> columns I thought were non-obvious. I'm definitely interested in
> suggestsion for improving this schema.
>
> Table: Hosts
> # computer information; including identifying data to determine when running
> on same host
> col: id
> col: identifying_data # some way to uniquely identify the computer on which
> a test is being run. Presumably the output of some unix function, but I
> don't know what.
One needs to be a bit careful here, as any of these changes could
cause a change in system performance
1) Upgrade of CPU speed.
2) Upgrade of operating system
Mathworks list on their site methods they use for licensing purposes
http://www.mathworks.com/support/solutions/en/data/1-171PI/
we could use some of them as a starting point.
> Table: Sage_version
> # a table giving each existing version of Sage an id
> There are also questions of how this would be distributed. Presumably the
> data part wouldn't come standard with sage. Maybe an optional spkg? Of
> course, you're mainly interested in comparing DIFFERENT versions of sage on
> the SAME computer, which doesn't really fit with how we normally distribute
> code and data.
>
>
> Analysis:
> Once we have the database set up and have gathered some data, you can do all
> kinds of things with it. I'm most interested in how to find speed
> regressions, and I can certainly imagine writing code to do so. You have
> data from a previous version of sage (or your own, before you applied some
> patch(es)) and you run a regression test with "sage -rt" or something; this
> generates the same kind of timing data and then you look for slowdowns,
> either absolute or relative to the average ratio of your current run to the
> previous run for a given line of code. It can then print out a sorted list
> of lines with slowdowns, or ones above a certain threshold, or...
>
> I'd like to hear more ideas on how to improve all of this.
> David
That sounds good, but a lot of work. Perhaps more than necessary and
perhaps more than one person will have time to do.
Here's something a bit simpler
1) Change 'make ptestlong' so it computes a checksum of each doctest,
performs the tests, writes the CPU time to a data file. This means
there's no need to write any more tests.
2) Add a new target to the makefile "make speedtest". That would
compute the checksum of all tests, and compare the checksums to those
in the datafile created in (1) above.
3) "make speedtest" would then execute any tests which have the same
checksum as stored in a reference file. Any test that takes
significantly longer or shorter, is then flagged. Those could then be
reported to sage-devel.
This would avoid the need for a relational database, and be quite easy
to implement. Any regressions should be seen by several people.
Dave
--
To post to this group, send an email to sage-devel@googlegroups.com
To unsubscribe from this group, send an email to
sage-devel+unsubscr...@googlegroups.com
For more options, visit this group at http://groups.google.com/group/sage-devel
URL: http://www.sagemath.org
