Re: Adding a `FLAKY` label to flaky unit tests

[Meng Zhu]

+1, the advantages are appealing.

Though I am afraid that this will probably reduce the incentive to fix
flaky tests.

-Meng

On Thu, Mar 29, 2018 at 9:45 AM, Benno Evers  wrote:

> Hi all,
>
> if you're regularly running Mesos unit tests, e.g. because you've set up a
> CI system, you probably noticed that there is a lot of noise in the results
> due to flaky tests.
>
> As a measure to ease the pain, what do you think about adding a `FLAKY`
> label to known flaky unit tests, similar to how we have `ROOT`, `INTERNET`,
> `DISABLED`, etc. right now?
>
> The advantages, in my opinion, would be:
>  - Looking at test results, it would be immediately visible whether a test
> failure was known flaky or not without going to JIRA
>  - People who want to reduce noise can disable all known flaky tests by a
> simple gtest filter
>  - People who want to can still run the flaky tests easier than if they get
> disabled outright
>  - With a little bit of scripting, it would be possible to add logic like
> "for flaky tests, run them 10 times and only report a failure if more than
> x% of the runs fail."
>
> What do you think?
>
> Best regards,
> --
> Benno Evers
> Software Engineer, Mesosphere
>

Adding a `FLAKY` label to flaky unit tests

[Benno Evers]

Hi all,

if you're regularly running Mesos unit tests, e.g. because you've set up a
CI system, you probably noticed that there is a lot of noise in the results
due to flaky tests.

As a measure to ease the pain, what do you think about adding a `FLAKY`
label to known flaky unit tests, similar to how we have `ROOT`, `INTERNET`,
`DISABLED`, etc. right now?

The advantages, in my opinion, would be:
 - Looking at test results, it would be immediately visible whether a test
failure was known flaky or not without going to JIRA
 - People who want to reduce noise can disable all known flaky tests by a
simple gtest filter
 - People who want to can still run the flaky tests easier than if they get
disabled outright
 - With a little bit of scripting, it would be possible to add logic like
"for flaky tests, run them 10 times and only report a failure if more than
x% of the runs fail."

What do you think?

Best regards,
-- 
Benno Evers
Software Engineer, Mesosphere

Re: Adding a `FLAKY` label to flaky unit tests

[Vinod Kone]

Would the CI run FLAKY tests or will it filter it out? I'm assuming it
still does based on your observation above.

What are the other reasons tests are DISABLED today?

On Thu, Mar 29, 2018 at 10:35 AM, Meng Zhu  wrote:

> +1, the advantages are appealing.
>
> Though I am afraid that this will probably reduce the incentive to fix
> flaky tests.
>
> -Meng
>
> On Thu, Mar 29, 2018 at 9:45 AM, Benno Evers 
> wrote:
>
> > Hi all,
> >
> > if you're regularly running Mesos unit tests, e.g. because you've set up
> a
> > CI system, you probably noticed that there is a lot of noise in the
> results
> > due to flaky tests.
> >
> > As a measure to ease the pain, what do you think about adding a `FLAKY`
> > label to known flaky unit tests, similar to how we have `ROOT`,
> `INTERNET`,
> > `DISABLED`, etc. right now?
> >
> > The advantages, in my opinion, would be:
> >  - Looking at test results, it would be immediately visible whether a
> test
> > failure was known flaky or not without going to JIRA
> >  - People who want to reduce noise can disable all known flaky tests by a
> > simple gtest filter
> >  - People who want to can still run the flaky tests easier than if they
> get
> > disabled outright
> >  - With a little bit of scripting, it would be possible to add logic like
> > "for flaky tests, run them 10 times and only report a failure if more
> than
> > x% of the runs fail."
> >
> > What do you think?
> >
> > Best regards,
> > --
> > Benno Evers
> > Software Engineer, Mesosphere
> >
>

Re: Adding a `FLAKY` label to flaky unit tests

[Andrei Budnik]

I have a couple of questions:
1) What would be the criteria for removing `FLAKY` label from a test? Who
will take care of removing this label?
2) Do we expect that most of our tests will eventually get `FLAKY` label?

On Thu, Mar 29, 2018 at 7:35 PM, Meng Zhu  wrote:

> +1, the advantages are appealing.
>
> Though I am afraid that this will probably reduce the incentive to fix
> flaky tests.
>
> -Meng
>
> On Thu, Mar 29, 2018 at 9:45 AM, Benno Evers 
> wrote:
>
> > Hi all,
> >
> > if you're regularly running Mesos unit tests, e.g. because you've set up
> a
> > CI system, you probably noticed that there is a lot of noise in the
> results
> > due to flaky tests.
> >
> > As a measure to ease the pain, what do you think about adding a `FLAKY`
> > label to known flaky unit tests, similar to how we have `ROOT`,
> `INTERNET`,
> > `DISABLED`, etc. right now?
> >
> > The advantages, in my opinion, would be:
> >  - Looking at test results, it would be immediately visible whether a
> test
> > failure was known flaky or not without going to JIRA
> >  - People who want to reduce noise can disable all known flaky tests by a
> > simple gtest filter
> >  - People who want to can still run the flaky tests easier than if they
> get
> > disabled outright
> >  - With a little bit of scripting, it would be possible to add logic like
> > "for flaky tests, run them 10 times and only report a failure if more
> than
> > x% of the runs fail."
> >
> > What do you think?
> >
> > Best regards,
> > --
> > Benno Evers
> > Software Engineer, Mesosphere
> >
>