David W. Van Couvering wrote: > I like the idea of having it as a release barrier, and I also like the > idea of getting an email saying "Code Coverage Regression" and printing > out the package(s) that have regressed below a low-water mark.
I'm not sure a release barrier will work. If the coverage is low in a certain area and no-one has the itch to work on it then is there no release? Think about how the coverage gets low in the first place. Someone contributes an improvement with some amount of testing. I think it's reasonable to reject such a contribution if there are no-tests. Without tests there is no easy way for a committer to determine if the feature even works. Now if tests are contributed that shows the feature basically works, but have low code coverage, is there really a justification to reject the contribution? The feature basically works according to the original contributor's itch, it's someone else's itch that more tests exist. One can always request more tests from the contributor, but I'm not sure you can force it. > What I am at a loss for is what the low-water mark should be. I think > whatever we choose, we are going to have some immediate regressions. > Then the question becomes, how much work are we willing to put into this > to get it fixed. > > One approach that comes to mind is to set a reachable goal for each > release as a step along the way to our ultimate goal. For right now, a > regression could be if any package goes 10% below what our current > baseline is. Then we try to raise the water each release and re-set our > baseline. Not sure how we can get people to scratch the code coverage itch. It seems we can't get a lot of folks interested in fixing the 150+ bugs out there, never mind writing more tests that might uncover more bugs. I would love it if we could find a way. Bottom line is that if people don't care about code coverage they are not going to work on improving it. Dan.
