On 10/25/2010 01:54 PM, William Stein wrote:
>> Also, I was talking to Craig Citro about this and he had the
>> interesting idea of creating some kind of a "test object" which would
>> be saved and then could be run into future versions of Sage and re-run
>> in. The idea of saving the tests that are run, and then running the
>> exact same tests (rather than worrying about correlation of files and
>> tests) will make catching regressions much easier.
>
> Wow, that's an *extremely* good idea! Nice work, Craig.
> Basically, we could have one object that has:
>
> (a) list of tests that got run.
> (b) for each of several machines and sage versions:
> - how long each test took
>
> Regarding (a), this gets extracted from the doctests somehow for
> starters, though could have some other tests thrown if we want.
>
> I could easily imagine storing the above as a single entry in a
> MongoDB collection (say):
>
> {'tests':[ordered list of input blocks of code that could be
> extracted from doctests],
> 'timings':[{'machine':'sage.math.washington.edu',
> 'version':'sage-4.6.alpha3', 'timings':[a list of floats]},
> {'machine':'bsd.math.washington.edu',
> 'version':'sage-4.5.3', 'timings':[a list of floats]}]
>
> Note that the ordered list of input blocks could stored using GridFS,
> since it's bigger than 4MB:
>
> wst...@sage:~/build/sage-4.6.alpha3/devel/sage$ sage -grep "sage:" > a
> wst...@sage:~/build/sage-4.6.alpha3/devel/sage$ ls -lh a
> -rw-r--r-- 1 wstein wstein 9.7M 2010-10-25 11:41 a
> wst...@sage:~/build/sage-4.6.alpha3/devel/sage$ wc -l a
> 133579 a
>
> Alternatively, the list of input blocks could be stored in its own
> collection, which would just get named by the tests field:
>
> {'tests':'williams_test_suite_2010-10-25'}
>
> The latter is nice, since it would make it much easier to make a web
> app that allows for browsing through
> the timing results, e.g,. sort them by longest to slowest, and easily
> click to get the input that took a long time.
>
> Another option: have exactly one collection for each test suite, and
> have all other data be in that collection:
>
> Collection name: "williams_test_suite-2010-10-25"
>
> Documents:
>
> * A document with a unique id, starting at 0, for each actual test
> {'id':0, 'code':'factor(2^127+1)'}
>
> * A document for each result of running the tests on an actual platform:
> {'machine':'bsd.math.washington.edu', 'version':'sage-4.5.3',
> 'timings':{0:1.3, 1:0.5,...} }
> Here, the timings are stored as a mapping from id's to floats.
This last option seems most "natural" to me, though identical inputs
that appear in multiple suites would generally(?) get different ids in
the collections. Would it be better to use a hash of the 'code' for the
'id', or can the database automatically ensure that different ids imply
different inputs?
Disclaimer: I'm not familiar with MongoDB. Here's a brief introduction:
http://www.mongodb.org/display/DOCS/Introduction
In your experience, are queries fast? For example, if we wanted to see
how timings vary across Sage versions and machines for a specific input?
> I think timing should all be done using the "timeit" modulo, since
> that is the Python standard module designed for exactly the purpose of
> benchmarking. That ends the discussion about CPU time versus walltime,
> etc.
>
> The architecture of separating out the tests from the
> timing/recording/web_view framework is very nice, because it makes it
> easy to add in additional test suites. E.g., I've had companies ask
> me: "Could paid support include the statement: 'in all future releases
> of Sage, the following commands will run in at most the following
> times on the following hardware'?" And they have specific commands
> they care about.
--
To post to this group, send an email to sage-devel@googlegroups.com
To unsubscribe from this group, send an email to
sage-devel+unsubscr...@googlegroups.com
For more options, visit this group at http://groups.google.com/group/sage-devel
URL: http://www.sagemath.org
