On Thu, Mar 13, 2014 at 12:42:26PM -0400, David Nalley wrote: > Radical proposition: > > Because we have two problems, of different nature, we are in a > difficult situation. This is a possible solution, and I'd appreciate > you reading and considering it. Feedback is welcome. I propose that > after we enter the RC stage that we not entertain any bugs as blockers > that don't have automated test cases associated with them. This means > that you are still welcome to do manual testing of your pet feature > and the things that are important to you; during the testing window > (or anytime really). However, if the automation suite isn't also > failing then we consider the release as high enough quality to ship. > This isn't something we can codify, but the PMC can certainly adopt > this attitude as a group when voting. Which also means that we can > deviate from it. If you brought up a blocker for release - we should > be immediately looking at how we can write a test for that behavior. > This should also mean several other behaviors need to become a valid > part of our process. We need to ensure that things are well tested > before allowing a merge. This means we need a known state of master, > and we need to perform testing that allows us to confirm that a patch > does no harm. We also need to insist on implementation of > comprehensive tests for every inbound feature. >
Sounds like a good mechanism to improve tests - without automating bugs I think we've done a grave disservice to our testing efforts. When I was active, I marked up quite a few bugs with the label - integration-test - to be covered under automated tests. Perhaps a starting point for picking up those efforts. I'd also avoid tagging every feature as 'Automation' as that beats the purpose. I see several blanket bugs (which are tasks really) marked as automation. We need to avoid creating these for the sake of creating them. More pointed, specific bugs to be caught, analysed whether they are automatable need to be triaged into these buckets. -- Prasanna., ------------------------ Powered by BigRock.com
