Given the feedback I've seen here, how about we do the following: Specifically mark timing sensitive tests. If a marked tests fails, re-run it. If it passes the second time, consider it a normal pass and keep bots green. If it fails the second time, turn bots red. The easiest way to mark the tests seems to be to move them into a specific directory.
Any opposition or better ideas? Thanks, Julie On Thu, Dec 3, 2009 at 6:49 AM, Gustavo Noronha Silva <g...@gnome.org> wrote: > On Wed, 2009-12-02 at 14:51 -0800, Julie Parent wrote: > > As Eric just said to me in person, another option is to just re-run > > *any* failing test twice, and only turn tree red if it fails twice. > > (Chromium just recently started doing this, and it has greatly > > improved our tree greenness). This obviously doesn't explicitly > > identify timing dependent tests, but it solves the bigger issues that > > flaky tests cause. > > But that would turn moot the point of the suggestion, I think. Having > only tests that are expected to fail under special conditions be tested > twice makes sense, but if a test that isn't expected to fail under > special conditions fails, we should see that as a failure. > > To give you a bit of insight into this, in GTK+ we used to have tests > that only failed when they were preceded by a specific test. This was > very important information that would be lost if it was run after > itself, and thus passed. The problems that caused this were missing > support in DRT, and a couple of times real bugs that caused crashes. > > This is to say I think we would be better served by only running > known-time-dependent tests twice. > > Thanks, > > -- > Gustavo Noronha Silva <g...@gnome.org> > GNOME Project > > _______________________________________________ > webkit-dev mailing list > webkit-dev@lists.webkit.org > http://lists.webkit.org/mailman/listinfo.cgi/webkit-dev >
_______________________________________________ webkit-dev mailing list webkit-dev@lists.webkit.org http://lists.webkit.org/mailman/listinfo.cgi/webkit-dev
