On Tue, Oct 13, 2009 at 1:00 PM, Julie Parent <jpar...@chromium.org> wrote:
> We did this on my last project to deal with flaky test infrastructure. It > worked well in that test failures were pretty much guaranteed to be real (we > ran tests 3 times and only reported failure if a test failed all 3 times), > but it did definitely make us stop caring about flaky tests. > Idealizing this some more, what if it could auto-add flaky tests to > test_expectations, or at least generate a daily report that someone could > then manually add? This would make us keep track of the flakiness and would > only have the slowdown in test run times until a flaky test is added to > test_expectations. > I agree that this would address the slowdown issue. It's more work though. :) Tony also pointed out that rerunning tests would not allow you to distinguish tests that consistently pass in isolation, but fail when run in the whole suite. I think as long as we don't automatically add to test_expectations though, this should be fine. Whoever goes to fix the test can check if it consistently fails/passes only in isolation. Another possibility would be to rerun unexpected failures, but continue to turn the tree red. This will provide more immediate feedback to people trying to green to tree as to whether tests are flaky or regressions. It would help with sheriffing/gardening without the other downsides. Ojan > On Tue, Oct 13, 2009 at 12:02 PM, Ojan Vafai <o...@chromium.org> wrote: > >> We could rerun any unexpected fail/crash/timeout tests. If they pass the >> second time around, then the tree turns orange instead of red. This has come >> up many times, but we've never agreed on whether it's a good idea. >> Pros: >> -Easier to distinguish between flakiness and new failures >> -Tree will be much greener >> -Try bots will be much more reliable >> -Increase in overall team sanity >> >> Cons: >> -Almost guaranteed to sweep some new flakiness under the rug >> -If enough flakiness accumulates it will affect cycle time (each >> timeout/crash test we rerun takes seconds) >> >> Shouldn't be too hard to implement. >> >> Ojan >> >> On Tue, Oct 13, 2009 at 11:32 AM, Peter Kasting <pkast...@google.com>wrote: >> >>> On Tue, Oct 13, 2009 at 11:17 AM, Nicolas Sylvain <nsylv...@chromium.org >>> > wrote: >>> >>>> It's because the sheriff don't notice the new failing tests, because >>>> most of the time the gardener does a good job of updating the list at the >>>> same time as the merge, so the tree stays mostly green. >>>> See http://src.chromium.org/viewvc/chrome?view=rev&revision=28727 >>>> >>> >>> In my case I was watching jparent's commits, so I stand by my assertion. >>> >>> PK >>> >>> >>> >> >> >> >> > --~--~---------~--~----~------------~-------~--~----~ Chromium Developers mailing list: chromium-dev@googlegroups.com View archives, change email options, or unsubscribe: http://groups.google.com/group/chromium-dev -~----------~----~----~----~------~----~------~--~---
