Hi all, if you're regularly running Mesos unit tests, e.g. because you've set up a CI system, you probably noticed that there is a lot of noise in the results due to flaky tests.
As a measure to ease the pain, what do you think about adding a `FLAKY` label to known flaky unit tests, similar to how we have `ROOT`, `INTERNET`, `DISABLED`, etc. right now? The advantages, in my opinion, would be: - Looking at test results, it would be immediately visible whether a test failure was known flaky or not without going to JIRA - People who want to reduce noise can disable all known flaky tests by a simple gtest filter - People who want to can still run the flaky tests easier than if they get disabled outright - With a little bit of scripting, it would be possible to add logic like "for flaky tests, run them 10 times and only report a failure if more than x% of the runs fail." What do you think? Best regards, -- Benno Evers Software Engineer, Mesosphere
