> Also, I was talking to Craig Citro about this and he had the
> interesting idea of creating some kind of a "test object" which would
> be saved and then could be run into future versions of Sage and re-run
> in. The idea of saving the tests that are run, and then running the
> exact same tests (rather than worrying about correlation of files and
> tests) will make catching regressions much easier.
Hi,
Wow, that's an *extremely* good idea! Nice work, Craig.
Basically, we could have one object that has:
(a) list of tests that got run.
(b) for each of several machines and sage versions:
- how long each test took
Regarding (a), this gets extracted from the doctests somehow for
starters, though could have some other tests thrown if we want.
I could easily imagine storing the above as a single entry in a
MongoDB collection (say):
{'tests':[ordered list of input blocks of code that could be
extracted from doctests],
'timings':[{'machine':'sage.math.washington.edu',
'version':'sage-4.6.alpha3', 'timings':[a list of floats]},
{'machine':'bsd.math.washington.edu',
'version':'sage-4.5.3', 'timings':[a list of floats]}]
Note that the ordered list of input blocks could stored using GridFS,
since it's bigger than 4MB:
wst...@sage:~/build/sage-4.6.alpha3/devel/sage$ sage -grep "sage:" > a
wst...@sage:~/build/sage-4.6.alpha3/devel/sage$ ls -lh a
-rw-r--r-- 1 wstein wstein 9.7M 2010-10-25 11:41 a
wst...@sage:~/build/sage-4.6.alpha3/devel/sage$ wc -l a
133579 a
Alternatively, the list of input blocks could be stored in its own
collection, which would just get named by the tests field:
{'tests':'williams_test_suite_2010-10-25'}
The latter is nice, since it would make it much easier to make a web
app that allows for browsing through
the timing results, e.g,. sort them by longest to slowest, and easily
click to get the input that took a long time.
Another option: have exactly one collection for each test suite, and
have all other data be in that collection:
Collection name: "williams_test_suite-2010-10-25"
Documents:
* A document with a unique id, starting at 0, for each actual test
{'id':0, 'code':'factor(2^127+1)'}
* A document for each result of running the tests on an actual platform:
{'machine':'bsd.math.washington.edu', 'version':'sage-4.5.3',
'timings':{0:1.3, 1:0.5,...} }
Here, the timings are stored as a mapping from id's to floats.
I think timing should all be done using the "timeit" modulo, since
that is the Python standard module designed for exactly the purpose of
benchmarking. That ends the discussion about CPU time versus walltime,
etc.
The architecture of separating out the tests from the
timing/recording/web_view framework is very nice, because it makes it
easy to add in additional test suites. E.g., I've had companies ask
me: "Could paid support include the statement: 'in all future releases
of Sage, the following commands will run in at most the following
times on the following hardware'?" And they have specific commands
they care about.
-- William
--
To post to this group, send an email to sage-devel@googlegroups.com
To unsubscribe from this group, send an email to
sage-devel+unsubscr...@googlegroups.com
For more options, visit this group at http://groups.google.com/group/sage-devel
URL: http://www.sagemath.org
