+1, the advantages are appealing. Though I am afraid that this will probably reduce the incentive to fix flaky tests.
-Meng On Thu, Mar 29, 2018 at 9:45 AM, Benno Evers <bev...@mesosphere.com> wrote: > Hi all, > > if you're regularly running Mesos unit tests, e.g. because you've set up a > CI system, you probably noticed that there is a lot of noise in the results > due to flaky tests. > > As a measure to ease the pain, what do you think about adding a `FLAKY` > label to known flaky unit tests, similar to how we have `ROOT`, `INTERNET`, > `DISABLED`, etc. right now? > > The advantages, in my opinion, would be: > - Looking at test results, it would be immediately visible whether a test > failure was known flaky or not without going to JIRA > - People who want to reduce noise can disable all known flaky tests by a > simple gtest filter > - People who want to can still run the flaky tests easier than if they get > disabled outright > - With a little bit of scripting, it would be possible to add logic like > "for flaky tests, run them 10 times and only report a failure if more than > x% of the runs fail." > > What do you think? > > Best regards, > -- > Benno Evers > Software Engineer, Mesosphere >
