My point is not to advocate for a specific solution but rather to suggest that *any* sensible incremental approach will produce real results.
--Mark Ramm On Mon, Mar 28, 2016 at 7:51 PM, David Cheney <david.che...@canonical.com> wrote: > On Tue, Mar 29, 2016 at 10:42 AM, Mark Ramm-Christensen > (Canonical.com) <mark.ramm-christen...@canonical.com> wrote: > > Never a good time to stop feature work entirely and fix what amounts to a > > race prone set of tests. > > > > > > But I would advocate building in some practices to improve the situation > > incrementally: > > > > fixing one major issue per team per week > > SGTM. How do we know which of the millions of private lists of bugs > are the critical ones? Which of the hundred "critical", "papercut", > "urgent" LP tags are the critical ones? > > > promoting all issues which fail CI more than x times per week to Critical > > blocking all branches on fridays except for fixes for bugs on the top > issues > > list > > Which timezone is friday ? > > > or some other similar policy > > > > Over time over time any of the above policies will bring the total > number of > > test failures down significantly, and would still allow progress on > feature > > work. > > > > On Mon, Mar 28, 2016 at 1:05 PM, Nate Finch <nate.fi...@canonical.com> > > wrote: > >> > >> I'll just note that we've had flaky tests for as long as I've been > working > >> on Juju, and there's never a "good" time to fix them. :) > >> > >> On Mon, Mar 28, 2016 at 11:48 AM Aaron Bentley > >> <aaron.bent...@canonical.com> wrote: > >>> > >>> -----BEGIN PGP SIGNED MESSAGE----- > >>> Hash: SHA256 > >>> > >>> On 2016-03-28 09:03 AM, Katherine Cox-Buday wrote: > >>> > Generally +1 on this, but I'm also intrigued by Martin's > >>> > statistic... do we currently weight test failures by how likely > >>> > they are to fail (i.e. how likely they are flaky)? That seems like > >>> > it would be a great metric to use to decide which to fix first. > >>> > >>> We don't do it on the likelihood of failure, but we do it on the > >>> frequency of failure. > >>> > >>> http://reports.vapour.ws/releases/top-issues > >>> > >>> I report on these on the cross-team call, and once the 2.0 settles > >>> down, I'll be reporting them on the release call again. > >>> > >>> Aaron > >>> -----BEGIN PGP SIGNATURE----- > >>> Version: GnuPG v2 > >>> > >>> iQEcBAEBCAAGBQJW+VJcAAoJEK84cMOcf+9hWrwH/0JradfscIE0wnt+yCW9nNCR > >>> 9hTHI2U19v1VuP6pWI4UiC7srfojPI8EXXEXrrAhF9rT8tpVK4EcJRJK9RvWvvz5 > >>> BEquHMS0+eROFOqDJFavEB8hU7BKHErzkSwSG8uKq7JuwHs9gNtQO9z9fIhVKjnr > >>> aP4z2IliCqbYfXbupfSTD8TmqhI0AipQymTg3QB4C3sJdXzc5GjzIIckUo/X7aJj > >>> zH1tEtlwOdP0c9F+8ZVs1j6AAkb+uDGc/1Qr4MT1kInqGkli2UNF4TOX/AihNPyH > >>> iwYgq6O7uOkijFTrL9obRfbXxIFw1WCc9cYzxbRYnGfQff47Dyj7/BUStPPH0i0= > >>> =8FQ6 > >>> -----END PGP SIGNATURE----- > >>> > >>> -- > >>> Juju-dev mailing list > >>> Juju-dev@lists.ubuntu.com > >>> Modify settings or unsubscribe at: > >>> https://lists.ubuntu.com/mailman/listinfo/juju-dev > >> > >> > >> -- > >> Juju-dev mailing list > >> Juju-dev@lists.ubuntu.com > >> Modify settings or unsubscribe at: > >> https://lists.ubuntu.com/mailman/listinfo/juju-dev > >> > > > > > > -- > > Juju-dev mailing list > > Juju-dev@lists.ubuntu.com > > Modify settings or unsubscribe at: > > https://lists.ubuntu.com/mailman/listinfo/juju-dev > > >
-- Juju-dev mailing list Juju-dev@lists.ubuntu.com Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/juju-dev
