Generally +1 on this, but I'm also intrigued by Martin's statistic... do we currently weight test failures by how likely they are to fail (i.e. how likely they are flaky)? That seems like it would be a great metric to use to decide which to fix first.
On 03/28/2016 01:29 AM, David Cheney wrote: > I know if we didn't retry constantly, the Juju tests'd never pass. But > by retrying, there is no impetus to fix them. > > How about we stop retrying flaky tests? The blocked build get's the grease. > > On Mon, Mar 28, 2016 at 5:23 PM, Martin Packman > <martin.pack...@canonical.com> wrote: >> On 27/03/2016, David Cheney <david.che...@canonical.com> wrote: >>> Hi Martin, >>> >>> I was told that the Go 1.6 tests were voting, so these bugs should be >>> blocking bugs. Is this not the case ? >> The tests are voting, and giving blesses, so no blocking bugs, but a >> lot of the remaining issues are low-occurrence failures. Basically the >> unit tests pass generally given the three attempts, but overall fail a >> lot from a number of issues that all happen only occasionally. >> >> For instance, bug 1553292: >> >> <http://reports.vapour.ws/releases/issue/56c36c5b749a567aa9496178> >> >> This is maybe ~5% chance of failing, but given the number of jobs now >> using go 1.5+ that's still six failures in the last week. >> >> We have enough issues like this that CI spends a lot more time >> retesting on go 1.6 than we do on go 1.2 with the same unit tests. >> >> Martin -- - Katherine -- Juju-dev mailing list Juju-dev@lists.ubuntu.com Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/juju-dev
