On Sun, Apr 29, 2012 at 7:35 AM, David Kirkby <david.kir...@onetel.net> wrote: > The fact there are functions in Sage untested is worrying, and I think > its fair to say most people would like to see 100% doctest coverage, > but IMHO, we need a plan for this to happen, and the plan inforced, > otherwise this will not happen soon.
To take things down a completely different path, I might argue that part of the problem is that there is diminishing return to doctests as we approach the 100% mark. In particular, I think it would be worth starting to look at other forms of coverage, e.g. line (or even branch) coverage--I feel much more confident in a method that has high branch coverage and no doctest than the other way around (and a lot of the painful-to-doctest methods are the internal, underscore methods that are tested indirectly elsewhere). I would also like to see more random fuzz testing (like you've done to see if you can crash things with random input) and correctness testing (e.g. do the same computation (or verify deep theorems and conjectures) with random inputs using different algorithms (this could perhaps even be semi-automated for any method that has an algorithm parameter). Thoughts? > So I'm suggesting we collect ideas for a plan. > > Two things that do NOT appear to work, at least in isolation are: > > A) Saying coverage should be x % in release Y, then hoping it is. > B) Saying coverage should be x % by the date DD/MM/YY, then hoping it is. > > Both of these have been tried, and niether are working. The graph here > > http://thales.math.uqam.ca/~labbes/blogue/2011/03/evolution-of-the-overall-doctest-coverage-of-sage/ > > seems to indicate that progress on this has slowed over the last year > or so, where progress was about 4.7% per year. (I'm not actually sure > I agree with his line of best fit, which would suggest to me the > situation has slowed even more.) > > > A few ideas that might work, are below. I would add, I'm not > suggesting they are all very good (in particular 1 and 4 are a bit > excessive), but I mention them anyway. > > 1) Having a release, where the only aim is to increase doctest > coverage. No patches are accepted, unless they increase coverage. > 2) Finding sections of code which are not tested, and depreciating > them, unless someone writes the doctests, in which case the > depreciation can be removed. > 3) Having a release, where anyone can submit a patch which gets > reviewed, but the release manager does not merge it unless the author > of the patch can provide another patch which increase doctest > coverage. > 4) Finding individuals that have written code that is not tested, and > not merging any more patches from them unless they first add tests to > the ALL code they have already written. > 5) Finding individuals that have written code that is not tested, and > not merging any more patches from them unless they first add tests to > a number (say 5) tests for all the code they have written which is > untested. > 6) Not accepting any patches, unless a equal number of lines of code > are submitted which add doctests. > 7) Paying individuals to just write tests. > > I believe if we want to achieve 100% coverage, we need to do more than > A and B above, as they are proved not to work. > > Can anyone suggest anything else, or have any comments. 7') Some kind of gamification. This could be as simple as a monthly leader-board for whoever contributed the most doctests, the patchbot giving preference to patches increasing coverage, or whatever. Carrots are more enjoyable than sticks. 7'') Personal bounties of some kind. E.g. I will referee any ticket of your choosing if you contribute 100 new, unrelated doctests. 8) Make it easier (and unobtrusive) to contribute doctests. - Robert -- To post to this group, send an email to sage-devel@googlegroups.com To unsubscribe from this group, send an email to sage-devel+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/sage-devel URL: http://www.sagemath.org
