I think if you set both number and repeat to 1 in sage.misc.sage_timeit, it
will only run once (though I could be wrong).
We should think about a way to automate uploading of timing data if someone
doesn't have MongoDB installed. For example, we could have the test script
which ran doctests have the option of sending an e-mail somewhere. Or make
pymongo standard in Sage.
David
On Mon, Oct 25, 2010 at 23:26, Robert Bradshaw <rober...@math.washington.edu
> wrote:
> On Mon, Oct 25, 2010 at 11:54 AM, William Stein <wst...@gmail.com> wrote:
> >> Also, I was talking to Craig Citro about this and he had the
> >> interesting idea of creating some kind of a "test object" which would
> >> be saved and then could be run into future versions of Sage and re-run
> >> in. The idea of saving the tests that are run, and then running the
> >> exact same tests (rather than worrying about correlation of files and
> >> tests) will make catching regressions much easier.
> >
> > Hi,
> >
> > Wow, that's an *extremely* good idea! Nice work, Craig.
> > Basically, we could have one object that has:
> >
> > (a) list of tests that got run.
> > (b) for each of several machines and sage versions:
> > - how long each test took
> >
> > Regarding (a), this gets extracted from the doctests somehow for
> > starters, though could have some other tests thrown if we want.
> >
> > I could easily imagine storing the above as a single entry in a
> > MongoDB collection (say):
> >
> > {'tests':[ordered list of input blocks of code that could be
> > extracted from doctests],
> > 'timings':[{'machine':'sage.math.washington.edu',
> > 'version':'sage-4.6.alpha3', 'timings':[a list of floats]},
> > {'machine':'bsd.math.washington.edu',
> > 'version':'sage-4.5.3', 'timings':[a list of floats]}]
> >
> > Note that the ordered list of input blocks could stored using GridFS,
> > since it's bigger than 4MB:
> >
> > wst...@sage:~/build/sage-4.6.alpha3/devel/sage$ sage -grep "sage:" > a
> > wst...@sage:~/build/sage-4.6.alpha3/devel/sage$ ls -lh a
> > -rw-r--r-- 1 wstein wstein 9.7M 2010-10-25 11:41 a
> > wst...@sage:~/build/sage-4.6.alpha3/devel/sage$ wc -l a
> > 133579 a
> >
> > Alternatively, the list of input blocks could be stored in its own
> > collection, which would just get named by the tests field:
> >
> > {'tests':'williams_test_suite_2010-10-25'}
> >
> > The latter is nice, since it would make it much easier to make a web
> > app that allows for browsing through
> > the timing results, e.g,. sort them by longest to slowest, and easily
> > click to get the input that took a long time.
> >
> > Another option: have exactly one collection for each test suite, and
> > have all other data be in that collection:
> >
> > Collection name: "williams_test_suite-2010-10-25"
> >
> > Documents:
> >
> > * A document with a unique id, starting at 0, for each actual test
> > {'id':0, 'code':'factor(2^127+1)'}
> >
> > * A document for each result of running the tests on an actual platform:
> > {'machine':'bsd.math.washington.edu', 'version':'sage-4.5.3',
> > 'timings':{0:1.3, 1:0.5,...} }
> > Here, the timings are stored as a mapping from id's to floats.
>
> +1. My only hesitance with this is that it requires either an internet
> connection or mongodb to participate, both of which are "optional"
> features of Sage. Of course, people could store their timings locally
> as lists of dicts, and then optionally upload to mongodb (requiring
> only pymongo, a much ligher dependency, or even via a json front end).
>
> > I think timing should all be done using the "timeit" modulo, since
> > that is the Python standard module designed for exactly the purpose of
> > benchmarking. That ends the discussion about CPU time versus walltime,
> > etc.
> >
> > The architecture of separating out the tests from the
> > timing/recording/web_view framework is very nice, because it makes it
> > easy to add in additional test suites. E.g., I've had companies ask
> > me: "Could paid support include the statement: 'in all future releases
> > of Sage, the following commands will run in at most the following
> > times on the following hardware'?" And they have specific commands
> > they care about.
>
> OTOH, I think that generating timing data as part of a standard test
> run could be very valuable. This precludes the use of timeit in
> general for long-ish running tests. Not that the two ideas are
> incompatible.
>
> - Robert
>
> --
> To post to this group, send an email to sage-devel@googlegroups.com
> To unsubscribe from this group, send an email to
> sage-devel+unsubscr...@googlegroups.com<sage-devel%2bunsubscr...@googlegroups.com>
> For more options, visit this group at
> http://groups.google.com/group/sage-devel
> URL: http://www.sagemath.org
>
--
To post to this group, send an email to sage-devel@googlegroups.com
To unsubscribe from this group, send an email to
sage-devel+unsubscr...@googlegroups.com
For more options, visit this group at http://groups.google.com/group/sage-devel
URL: http://www.sagemath.org
