Re: Monitoring/improving code coverage (was Re: code coverage results for trunk - svn revision no 390306)
Daniel John Debrunner wrote: >Not sure how we can get people to scratch the code coverage itch. It >seems we can't get a lot of folks interested in fixing the 150+ bugs out >there, never mind writing more tests that might uncover more bugs. I >would love it if we could find a way. > >Bottom line is that if people don't care about code coverage they are >not going to work on improving it. > > Well, there is scratch your own itch and then there is the "Apache Way". >From http://www.apache.org/foundation/how-it-works.html *Philosophy* While there is not an official list, these six principles have been cited as the core beliefs of philosophy behind the foundation, which is normally referred to as "The Apache Way": - collaborative software development - commercial-friendly standard license - consistently high quality software - respectful, honest, technical-based interaction - faithful implementation of standards - security as a mandatory feature All of the ASF projects share these principles. Code coverage, reducing bug backlog and addressing functional testing and security issues are part of our responsibility as an ASF project.Where do we rate for each of the six principles on a scale of 1-10 for embedded and client? Where should we be? How do we get there? Kathey
Re: Monitoring/improving code coverage (was Re: code coverage results for trunk - svn revision no 390306)
I think it would be great to have more awareness about code coverage, esepcially if it gets worse. Getting people to work on it is a different question, and I agree it may be difficult. I know I would be alarmed if someone checks in a complicated feature and the code coverage for those packages is say 10%, and at a minimum a discussion should ensue. But as it stands we don't even know when that happens. I liked having a goal for each release of improving the code coverage by some modest amount -- incremental improvement. Another area where I think there would be value in increasing awareness is if we have a complexity analysis tool and some package jumps in complexity by 50% after a checkin... But that's another itch for another email thread. David Daniel John Debrunner wrote: David W. Van Couvering wrote: I like the idea of having it as a release barrier, and I also like the idea of getting an email saying "Code Coverage Regression" and printing out the package(s) that have regressed below a low-water mark. I'm not sure a release barrier will work. If the coverage is low in a certain area and no-one has the itch to work on it then is there no release? Think about how the coverage gets low in the first place. Someone contributes an improvement with some amount of testing. I think it's reasonable to reject such a contribution if there are no-tests. Without tests there is no easy way for a committer to determine if the feature even works. Now if tests are contributed that shows the feature basically works, but have low code coverage, is there really a justification to reject the contribution? The feature basically works according to the original contributor's itch, it's someone else's itch that more tests exist. One can always request more tests from the contributor, but I'm not sure you can force it. What I am at a loss for is what the low-water mark should be. I think whatever we choose, we are going to have some immediate regressions. Then the question becomes, how much work are we willing to put into this to get it fixed. One approach that comes to mind is to set a reachable goal for each release as a step along the way to our ultimate goal. For right now, a regression could be if any package goes 10% below what our current baseline is. Then we try to raise the water each release and re-set our baseline. Not sure how we can get people to scratch the code coverage itch. It seems we can't get a lot of folks interested in fixing the 150+ bugs out there, never mind writing more tests that might uncover more bugs. I would love it if we could find a way. Bottom line is that if people don't care about code coverage they are not going to work on improving it. Dan.
Re: Monitoring/improving code coverage (was Re: code coverage results for trunk - svn revision no 390306)
Backing out the code was a suggestion, not a rule. What is appropriate I think depends upon the situation. My main point is it would be good to have the information made available to us in a timely and "in-your-face" way -- we don't have to go searching for it, and we find out right away -- so we can discuss it as a community and try to make the right decision. David Daniel John Debrunner wrote: David W. Van Couvering wrote: I think the same could be done for code coverage regressions. If a new chunk of code is checked in and the numbers drop way way down for a given module, I think that is cause for concern and a committer could reasonably insist that the feature is not sufficiently tested and back out the change, or at least block any future checkins in that area until enough tests are provided. So we could have applied this rule to JDBC 4.0 checkins. JDBC 4.0 code was checked in, the code coverage numbers decreased and could not increase until the tests could run under JDBC 4.0. In this case requiring the JDBC 4.0 changes be backed out until all the tests ran under JBDC 4.0 seems too drastic. Goes against the model of incremental development. Dan.
Re: Monitoring/improving code coverage (was Re: code coverage results for trunk - svn revision no 390306)
David W. Van Couvering wrote: > I like the idea of having it as a release barrier, and I also like the > idea of getting an email saying "Code Coverage Regression" and printing > out the package(s) that have regressed below a low-water mark. I'm not sure a release barrier will work. If the coverage is low in a certain area and no-one has the itch to work on it then is there no release? Think about how the coverage gets low in the first place. Someone contributes an improvement with some amount of testing. I think it's reasonable to reject such a contribution if there are no-tests. Without tests there is no easy way for a committer to determine if the feature even works. Now if tests are contributed that shows the feature basically works, but have low code coverage, is there really a justification to reject the contribution? The feature basically works according to the original contributor's itch, it's someone else's itch that more tests exist. One can always request more tests from the contributor, but I'm not sure you can force it. > What I am at a loss for is what the low-water mark should be. I think > whatever we choose, we are going to have some immediate regressions. > Then the question becomes, how much work are we willing to put into this > to get it fixed. > > One approach that comes to mind is to set a reachable goal for each > release as a step along the way to our ultimate goal. For right now, a > regression could be if any package goes 10% below what our current > baseline is. Then we try to raise the water each release and re-set our > baseline. Not sure how we can get people to scratch the code coverage itch. It seems we can't get a lot of folks interested in fixing the 150+ bugs out there, never mind writing more tests that might uncover more bugs. I would love it if we could find a way. Bottom line is that if people don't care about code coverage they are not going to work on improving it. Dan.
Re: Monitoring/improving code coverage (was Re: code coverage results for trunk - svn revision no 390306)
David W. Van Couvering wrote: > I think the same could be done for code coverage regressions. If a new > chunk of code is checked in and the numbers drop way way down for a > given module, I think that is cause for concern and a committer could > reasonably insist that the feature is not sufficiently tested and back > out the change, or at least block any future checkins in that area until > enough tests are provided. So we could have applied this rule to JDBC 4.0 checkins. JDBC 4.0 code was checked in, the code coverage numbers decreased and could not increase until the tests could run under JDBC 4.0. In this case requiring the JDBC 4.0 changes be backed out until all the tests ran under JBDC 4.0 seems too drastic. Goes against the model of incremental development. Dan.
Re: Monitoring/improving code coverage (was Re: code coverage results for trunk - svn revision no 390306)
I think the model we are following with functionality regressions works well. We are not necessarily preventing future checkins unless things are perceived to be Out of Hand by one or more committers. I think the same could be done for code coverage regressions. If a new chunk of code is checked in and the numbers drop way way down for a given module, I think that is cause for concern and a committer could reasonably insist that the feature is not sufficiently tested and back out the change, or at least block any future checkins in that area until enough tests are provided. So I propose having a flag raised when numbers go say 20% below current baseline for a given package, and then it is up to the committers to decide what action is necessary. David Kathey Marsden wrote: Rick Hillegas wrote: In previous lives, I've seen code-coverage metrics generated on, say, a monthly basis and used as a release barrier. I do not think they are appropriate as a barrier to checkin. I think since we seem to be going for very long release cycles instead of the release early, release often model, release barriers are not very effective for maintaining quality on the trunk. Also in open source, developers come and go so the wait until release model doesn't tend to work that well in that regard either. If Linux could release a new kernel twice a day, I tend to think that the trunk should be "ready for release" at any time for completed functionality and even incremental functionality could be covered. Kathey
Re: Monitoring/improving code coverage (was Re: code coverage results for trunk - svn revision no 390306)
Rick Hillegas wrote: > In previous lives, I've seen code-coverage metrics generated on, say, > a monthly basis and used as a release barrier. I do not think they are > appropriate as a barrier to checkin. > I think since we seem to be going for very long release cycles instead of the release early, release often model, release barriers are not very effective for maintaining quality on the trunk. Also in open source, developers come and go so the wait until release model doesn't tend to work that well in that regard either. If Linux could release a new kernel twice a day, I tend to think that the trunk should be "ready for release" at any time for completed functionality and even incremental functionality could be covered. Kathey
Re: Monitoring/improving code coverage (was Re: code coverage results for trunk - svn revision no 390306)
I like the idea of having it as a release barrier, and I also like the idea of getting an email saying "Code Coverage Regression" and printing out the package(s) that have regressed below a low-water mark. What I am at a loss for is what the low-water mark should be. I think whatever we choose, we are going to have some immediate regressions. Then the question becomes, how much work are we willing to put into this to get it fixed. One approach that comes to mind is to set a reachable goal for each release as a step along the way to our ultimate goal. For right now, a regression could be if any package goes 10% below what our current baseline is. Then we try to raise the water each release and re-set our baseline. David Rick Hillegas wrote: In previous lives, I've seen code-coverage metrics generated on, say, a monthly basis and used as a release barrier. I do not think they are appropriate as a barrier to checkin. Regards, -Rick Kathey Marsden wrote: David W. Van Couvering wrote: Did I ask this before? Do we want to agree upon a "low water mark" for code coverage and send out a "Quality Regression" email if our testing coverage falls below that mark? I think this would have a lot of value. This sounds like an interesting idea. Code coverage is an important quality data point. What kind of granularity would it have? Would it be just the overall number or would individual packages or files be flagged? Also for areas that have poor coverage, how could we encourage numbers to be brought up before or during enhancements? Kathey
Re: Monitoring/improving code coverage (was Re: code coverage results for trunk - svn revision no 390306)
In previous lives, I've seen code-coverage metrics generated on, say, a monthly basis and used as a release barrier. I do not think they are appropriate as a barrier to checkin. Regards, -Rick Kathey Marsden wrote: David W. Van Couvering wrote: Did I ask this before? Do we want to agree upon a "low water mark" for code coverage and send out a "Quality Regression" email if our testing coverage falls below that mark? I think this would have a lot of value. This sounds like an interesting idea. Code coverage is an important quality data point. What kind of granularity would it have? Would it be just the overall number or would individual packages or files be flagged? Also for areas that have poor coverage, how could we encourage numbers to be brought up before or during enhancements? Kathey
Monitoring/improving code coverage (was Re: code coverage results for trunk - svn revision no 390306)
David W. Van Couvering wrote: > > Did I ask this before? Do we want to agree upon a "low water mark" > for code coverage and send out a "Quality Regression" email if our > testing coverage falls below that mark? I think this would have a lot > of value. This sounds like an interesting idea. Code coverage is an important quality data point. What kind of granularity would it have? Would it be just the overall number or would individual packages or files be flagged? Also for areas that have poor coverage, how could we encourage numbers to be brought up before or during enhancements? Kathey
