On Wed, Dec 1, 2010 at 11:32 AM, David Kirkby <david.kir...@onetel.net> wrote: > On 1 December 2010 18:18, David Roe <r...@math.harvard.edu> wrote: >> I disagree that doctests should need to be independently verified. > > I think we will have to agree to differ then.
I agree with David Roe. I also would like to encourage David Kirkby (or anybody else) to independently test as many examples as they can, and if they uncover any issues, open a ticket and post a patch. Also, if they are refereeing new patches, do some testing of your own. I always do! If anything, this independent checking should be the referee's job -- even if the author claimed to check things independently, the referee would do well to double check some tests. So David K., I hope you'll continue to "put your money where you mouth" is and referee a lot of patches. You've done a massive amount already. Keep up the good work. But let's not make Sage too much more bureaucratic. If anything, it's already too bureaucratic. I personally can hardly stand to submit anything to Sage anymore because of this. I do think it would be good to start using nosetest (http://somethingaboutorange.com/mrl/projects/nose/0.11.2/) to automatically run all functions that start with "test_" in all files, in addition to doctests. This is how I've been testing the purple-sage library (http://code.google.com/p/purplesage/), and for many cases it does result in me writing much more comprehensive test suites. Notetest is also very nice because it can run all the tests in a given file in parallel. Also, when a test in a file fails, it can drop you into a debugging shell right there with the failed test. This is all something that we should start doing in addition to aiming for 100% doctest coverage for the sage library... >> Of course, if we had an arbitrarily large amount of time to write doctests, >> then it would be a laudible goal. Even now, I think there are situations >> where it would be reasonable to ask this of the author of a patch: if there >> was some indication of inconsistency for example. And if someone wants to >> go through the Sage library adding such consistency checks, I think that's a >> great way to improve Sage. > > So you admit it would improve sage to check the tests. It's hard to deny. >> But it's already difficult enough to get code >> refereed without adding a requirement that code have such consistency >> checks. > > It would probably be a bit easier to convince reviewers if your > doctests can be verified. When people review, they should try to verify tests however they want. >> The doctests that you object to fill two important roles: >> 1) they provide an example to a reader of the documentation how to use the >> function. > > Yes, perhaps a confusing one if the answer is wrong. An embarrassing > one if the examples are wrong. > >> 2) they provide a check so that if some change to the Sage library breaks >> something, we find out when testing. > >> Until we have 100% doctest coverage, I think that's plenty. > > 100% covered of unverified tests is not worth a lot to me. What do you > propose we do when we get 100% coverage - go back and check if the > rests are valid or not? What a waste of time that would be. Verifying correctness of tests is not a waste of time. > It would be less overall effort to do the tests correctly the first time. People presumably *think* they are doing tests correctly. The point is that you're wanting authors to submit "proofs" that they did independent verification of results, and I think that is too much bureaucracy. But asking referees to check claimed examples -- that makes sense! In particular, if I referee some code, and it turns out somebody finds that the examples were just wrong, then I as the referee will be pretty embarrassed. > If you are going to give an example, how much longer does it take to > check if they are consistent with Mathematica or similar software? Or > chose an integral from a book? That does raise an issue: one problems is that most of Sage isn't calculus. Most code I write these days isn't available in any other software... A lot of what Sage does is available only in Magma say, which many people don't even have access to. > -- > To post to this group, send an email to sage-devel@googlegroups.com > To unsubscribe from this group, send an email to > sage-devel+unsubscr...@googlegroups.com > For more options, visit this group at > http://groups.google.com/group/sage-devel > URL: http://www.sagemath.org > -- William Stein Professor of Mathematics University of Washington http://wstein.org -- To post to this group, send an email to sage-devel@googlegroups.com To unsubscribe from this group, send an email to sage-devel+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/sage-devel URL: http://www.sagemath.org
