On Fri, 2007-08-10 at 12:52 -0700, William Stein wrote:
> Here's a crazy idea. We decide on an extension to the doctest syntax
> that states that a doctest fails if it takes over a certain amount of time
> to run on a machine with at least a 2Ghz processor. For example,
>
> sage: 2 + 2 # takes at most 5s
>
> In doctesting, if the comment includes the phrase "takes at most [xxx]s",
> then local/bin/sage-doctest will run the doctest but also time it, and if it
> takes more than [xxx] seconds, it flags it as an error.
>
> To implement the above, we could replace the above line by the
> following Python equivalent lines (which gets run by Python's doctest):
>
> >>> t = walltime()
> >>> 2 + 2
> >>> if FASTCPU and walltime(t) > 5: assert False, "test xxx takes
> too long to run"
>
> What do you think? This has the feel of a good but easy to use idea
> to me, which other
> people will poke full of holes, but with sufficient discussion might
> lead to something
> actually useful.
>
> William
This doesn't sound like a good idea to me, if for no other reason than
the fact that either FASTCPU will need to change over time, which will
require updating the time it is supposed to take for all the tests to
run, or the tests will have to be run on a specific machine to really
have any meaning.
Here is a high level description of another possible idea, ignoring most
implementation details for a moment. To test the speed of my SAGE
installation, I simply run a function benchmark(). This runs lots of
test code, and probably takes at least an hour to run, and returns a
Benchmark object, say, which contains lots of information about how long
various tests took to run.
Now if I want to compare the speed to a different SAGE installation, I
can load a Benchmark instance from disk, as in:
sage: b1 = benchmark()
sage: b2 = load('sage2.6.4-benchmark')
sage: b1
(Benchmark instance created with SAGE version:2.7.3; branch: sage-main;
processor: something-or-other)
sage: b2
(Benchmark instance created with SAGE version:2.6.4; branch: sage-main;
processor: something-or-other)
sage: b1.compare(b2)
--The following tests ran faster under version 2.7.3 ...
[some information about tests and timings.]
[...]
--The following tests ran faster under version 2.6.4 ...
[more info]
or maybe instead
sage: b1.compare(b2)
testname time1 time2 difference
... ... ... ...
... ... ... ...
(etc)
An automated test could then be written to pick up on things that
significantly slow down between releases. For example, maybe when sage
-test is run, it can be supplied with timing data from a previous run,
and produce warnings if anything is slower that it used to be.
--~--~---------~--~----~------------~-------~--~----~
To post to this group, send email to sage-devel@googlegroups.com
To unsubscribe from this group, send email to [EMAIL PROTECTED]
For more options, visit this group at http://groups.google.com/group/sage-devel
URLs: http://sage.scipy.org/sage/ and http://modular.math.washington.edu/sage/
-~----------~----~----~----~------~----~------~--~---
