On Mon, Oct 25, 2010 at 11:54 AM, William Stein <wst...@gmail.com> wrote:
>> Also, I was talking to Craig Citro about this and he had the
>> interesting idea of creating some kind of a "test object" which would
>> be saved and then could be run into future versions of Sage and re-run
>> in. The idea of saving the tests that are run, and then running the
>> exact same tests (rather than worrying about correlation of files and
>> tests) will make catching regressions much easier.
>
> Hi,
>
> Wow, that's an *extremely* good idea! Nice work, Craig.
> Basically, we could have one object that has:
>
> (a) list of tests that got run.
> (b) for each of several machines and sage versions:
> - how long each test took
>
> Regarding (a), this gets extracted from the doctests somehow for
> starters, though could have some other tests thrown if we want.
>
> I could easily imagine storing the above as a single entry in a
> MongoDB collection (say):
>
> {'tests':[ordered list of input blocks of code that could be
> extracted from doctests],
> 'timings':[{'machine':'sage.math.washington.edu',
> 'version':'sage-4.6.alpha3', 'timings':[a list of floats]},
> {'machine':'bsd.math.washington.edu',
> 'version':'sage-4.5.3', 'timings':[a list of floats]}]
>
> Note that the ordered list of input blocks could stored using GridFS,
> since it's bigger than 4MB:
>
> wst...@sage:~/build/sage-4.6.alpha3/devel/sage$ sage -grep "sage:" > a
> wst...@sage:~/build/sage-4.6.alpha3/devel/sage$ ls -lh a
> -rw-r--r-- 1 wstein wstein 9.7M 2010-10-25 11:41 a
> wst...@sage:~/build/sage-4.6.alpha3/devel/sage$ wc -l a
> 133579 a
>
> Alternatively, the list of input blocks could be stored in its own
> collection, which would just get named by the tests field:
>
> {'tests':'williams_test_suite_2010-10-25'}
>
> The latter is nice, since it would make it much easier to make a web
> app that allows for browsing through
> the timing results, e.g,. sort them by longest to slowest, and easily
> click to get the input that took a long time.
>
> Another option: have exactly one collection for each test suite, and
> have all other data be in that collection:
>
> Collection name: "williams_test_suite-2010-10-25"
>
> Documents:
>
> * A document with a unique id, starting at 0, for each actual test
> {'id':0, 'code':'factor(2^127+1)'}
>
> * A document for each result of running the tests on an actual platform:
> {'machine':'bsd.math.washington.edu', 'version':'sage-4.5.3',
> 'timings':{0:1.3, 1:0.5,...} }
> Here, the timings are stored as a mapping from id's to floats.
+1. My only hesitance with this is that it requires either an internet
connection or mongodb to participate, both of which are "optional"
features of Sage. Of course, people could store their timings locally
as lists of dicts, and then optionally upload to mongodb (requiring
only pymongo, a much ligher dependency, or even via a json front end).
> I think timing should all be done using the "timeit" modulo, since
> that is the Python standard module designed for exactly the purpose of
> benchmarking. That ends the discussion about CPU time versus walltime,
> etc.
>
> The architecture of separating out the tests from the
> timing/recording/web_view framework is very nice, because it makes it
> easy to add in additional test suites. E.g., I've had companies ask
> me: "Could paid support include the statement: 'in all future releases
> of Sage, the following commands will run in at most the following
> times on the following hardware'?" And they have specific commands
> they care about.
OTOH, I think that generating timing data as part of a standard test
run could be very valuable. This precludes the use of timeit in
general for long-ish running tests. Not that the two ideas are
incompatible.
- Robert
--
To post to this group, send an email to sage-devel@googlegroups.com
To unsubscribe from this group, send an email to
sage-devel+unsubscr...@googlegroups.com
For more options, visit this group at http://groups.google.com/group/sage-devel
URL: http://www.sagemath.org
