On 9/13/06, Kathey Marsden <[EMAIL PROTECTED]> wrote:
The reason I am so regression averse is that there is a tremendous impact and magnification of regressions downstream when Derby is used as a component of a larger piece of software.
I agree, and I would be surprised if there was anyone who disagreed. It is in the interest of anyone who wants to use of Derby to be able to rely on it as a part of their software architecture. Therefore, it is in our best interest to keep regressions to a minimum so that developers feel they can rely on Derby. So, the question is: what can we, the Derby community, do about it when faced with regressions like the ones we've seen with the recent optimizer fixes? We must be fair and not fault of the developer who made the original changes. Additional tests which exercised the changes were added but they were not enough to uncover all the problems. So our tests did not catch the problem, and it appears that a downstream consumer's tests did not either. So it wasn't until it got into a third party's hands, hands which did something we did not - or could not - anticipate, that a problem was uncovered. Is there anything that could have prevented the regression? Even if the changes were flagged as potentially regression-causing, what can we do? The developer making the change already made a best effort at testing the change. Many eyes looked at the code and deemed it sound. And so, that code was checked in. But yet there was a regression. Could we have prevented it? The only way that we might have prevented it that has occurred to me is that if changes are being made to areas that are considered 'sensitive,' that we monitor code coverage around the changes: have all the new paths through the new or changed code been tested and verified to work properly? Maybe other people have better suggestions, but putting the burden of refining tests to conform to code coverage results seems an undue burden on developers when functional tests that they have crafted to match the change work and pass, and no one else is able to find issues. Doing so would raise the bar considerably for working in an area flagged as 'sensitive', which means you are less likely to get any contributors in that area. I agree that it is not a good thing that we introduced a regression into 10.1, but I think we can set a great example by fixing the regressions quickly and making a new release where the regressions are fixed. If we can somehow make the time between when the regression is identified, fixed, and then available in a released version approach zero, then we as a community should be able to sleep at night without having to worry about whether or not a recent change might have caused a regression. :-) I would love to hear other people's opinions on the issue, especially with regards to improvements to how we can test and verify changes with a high regression potential. cheers, andrew p.s. A debate for elsewhere would be to discuss how SQL's declarative style impedes functional testing of implementations written in imperative languages.
