On 1 December 2010 20:01, William Stein <wst...@gmail.com> wrote: > On Wed, Dec 1, 2010 at 11:32 AM, David Kirkby <david.kir...@onetel.net> wrote: >> On 1 December 2010 18:18, David Roe <r...@math.harvard.edu> wrote: >>> I disagree that doctests should need to be independently verified. >> >> I think we will have to agree to differ then. > > I agree with David Roe.
I thought you would. > I also would like to encourage David Kirkby (or anybody else) to > independently test as many examples as they can, and if they uncover > any issues, open a ticket and post a patch. For me personally, as a non-mathematician I'd have a problem with just accepting a doctest, which I probably can't verify myself. In some cases I can using Mathematica, and have done on some occasions. But in the case of http://trac.sagemath.org/sage_trac/ticket/10187 I could not. But in this case I'll trust the author when he says he has verified this with Maple. I think Sage is better for that change. > Also, if they are > refereeing new patches, do some testing of your own. I always do! > If anything, this independent checking should be the referee's job -- > even if the author claimed to check things independently, the referee > would do well to double check some tests. You have an advantage over me. Of course, I could decline to give a positive review until a mathematician has said the patch is OK. That would delay the ticket more of course. > So David K., I hope you'll continue to "put your money where you > mouth" is and referee a lot of patches. You've done a massive amount > already. Keep up the good work. But as I say, I'm restricted somewhat when people add tests I'm not convinced of. > But let's not make Sage too much more bureaucratic. If anything, it's > already too bureaucratic. I personally can hardly stand to submit > anything to Sage anymore because of this. I realise you now have PSage. I feel its a shame you did not complete the Cygwin port first, but that's your choice. I can understand your reasons. > I do think it would be good to start using nosetest > (http://somethingaboutorange.com/mrl/projects/nose/0.11.2/) to > automatically run all functions that start with "test_" in all files, I suggested 'nose' was added a long time ago http://groups.google.com/group/sage-devel/browse_thread/thread/928632557a8a041c/f8bc25a249ea4483?hl=en&lnk=gst&q=nose#f8bc25a249ea4483 the only person to reply (Robert Bradshaw) disagreed. >> It would probably be a bit easier to convince reviewers if your >> doctests can be verified. > > When people review, they should try to verify tests however they want. But one could make life a lot easier for a reviewer by picking something when the results can be verified easily. If one writes a test to show how to use function X, then the input probably does not matter too much. So chose an input where the output can be verified, rather than some input where it can't be. >>> Until we have 100% doctest coverage, I think that's plenty. >> >> 100% covered of unverified tests is not worth a lot to me. What do you >> propose we do when we get 100% coverage - go back and check if the >> rests are valid or not? What a waste of time that would be. > > Verifying correctness of tests is not a waste of time. I don't know what the current coverage is, but lets say for argument it needs another 1000 tests to get 100% coverage. It's better to verify those 1000 tests now, rather than wait to we get 100% coverage, then go back and verify them. >> It would be less overall effort to do the tests correctly the first time. > > People presumably *think* they are doing tests correctly. The point > is that you're wanting authors to submit "proofs" that they did > independent verification of results, and I think that is too much > bureaucracy. No, I'm not suggesting a formal proof. In the case of the patch here http://trac.sagemath.org/sage_trac/attachment/ticket/10187/trac_10187_fix_easy_doctests.patch lines 345 & 346 was added, as a test, with nothing to say why. The author has now said Maple 12 gives the same answer - I believe him in this case. I rather suspect the input, which shows how to use the taylor function, could be any of numerous inputs. The one chosen sage: taylor(gamma(1/3+x),x,0,3) gives a huge output which is going to be next to impossible to verify analytically. I rather suspect using a different series, where the output was well known, would have been more logical. > But asking referees to check claimed examples -- > that makes sense! In particular, if I referee some code, and it > turns out somebody finds that the examples were just wrong, then I as > the referee will be pretty embarrassed. Yes, but using examples like sage: taylor(gamma(1/3+x),x,0,3) makes it almost impossible for a referee to check it, as the output is huge. > >> If you are going to give an example, how much longer does it take to >> check if they are consistent with Mathematica or similar software? Or >> chose an integral from a book? > > That does raise an issue: one problems is that most of Sage isn't > calculus. Most code I write these days isn't available in any other > software... > A lot of what Sage does is available only in Magma say, which many > people don't even have access to. Fair enough. But you can at least state the Sage output is consistent with that from Magma. In any case, you stated only a week or so ago that Magma 2.13 is now installed on sage.math http://groups.google.com/group/sage-devel/msg/8e473e24b0e48772?hl=en It's a shame the license of Wolfram Alpha does not allow for testing software like Sage. (This was debated some time ago on sage-devel). Otherwise that would give a nice easy way to verify *some* results. "is 100001 prime" http://www.wolframalpha.com/input/?i=is+100001+prime > William Stein > Professor of Mathematics > University of Washington > http://wstein.org I appreciate in many cases it's not going to be possible to verify by other means. One has to be extra careful about the code then. Dave -- To post to this group, send an email to sage-devel@googlegroups.com To unsubscribe from this group, send an email to sage-devel+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/sage-devel URL: http://www.sagemath.org
