On 6 March 2013 17:13, Scott Kitterman <ubu...@kitterman.com> wrote: > On Tuesday, March 05, 2013 08:07:36 PM Jono Bacon wrote: >> On Tue, Mar 5, 2013 at 7:43 PM, Scott Kitterman <ubu...@kitterman.com> > wrote: >> > What percentage of code in the default install is covered by automated >> > tests? >> I am not sure, maybe the QA team can weigh in on this. >> >> > For a relatively small project, such approaches are conceivable. For >> > something the size of an installed Ubuntu (pick your favorite flavor) >> > system, I think we're a long ways away from being sufficiently >> > instrumented with tests. >> I agree: I think automated testing is going to be essential here to >> *assure* quality across the system. I think we would need ensure that >> we have good test coverage across the core components. > > I agree. My main point is that such things are pre-requisites to a new > release model. We don't have them, so we should get them before we change to > something we're not ready to support.
I don't see them as strict pre-requisites. Its like this: if you have 500% coverage you can still ship broken code. (100% coverage is not complete coverage, because all (open source) coverage tools todate can at most report branch coverage, not domain and range coverage (though a http://en.wikipedia.org/wiki/Symbolic_execution based coverage analysis would be pretty close). Once you have complete coverage all you know is that you have claimed that your code should do what it does, *not* that that is correct: you can still violate the expectations of third party libraries and that leads to bugs. What not enough tests *actually* means is that individual developers making changes need to think harder. While there is a safety barrier of 'noone really uses this' there is no driver for contributors to pay attention to detail : they can fix any issue by another upload. Exactly what conclusion to draw from that I'm not sure; will leave it as an exercise for the reader. -Rob -- Robert Collins <rbtcoll...@hp.com> Distinguished Technologist HP Cloud Services -- ubuntu-devel mailing list ubuntu-devel@lists.ubuntu.com Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-devel
