Makes sense to me ...
On Thu, Oct 4, 2012 at 2:46 PM, Ojan Vafai <o...@chromium.org> wrote: > TL;DR: We should add a NeedsRebaseline keyword to TestExpectations and add > garden-o-matic tooling for it for the cases where someone commits a > change/test that they know will need new results for different ports (e.g. > any patch that changes the rendering of pixel tests). > > A common pattern that I see across ports is that someone will add something > like the following in a patch that changes the results of a pixel test: > // Needs rebaseline after r23456 > webkit.org/b/12345 path/to/test.html [ Failure ] > > This has a couple problems: > -Often the correct expectation is something like [ Missing Failure > ImageOnlyFailure ]. So, even though the test is listed, the bot turns red. > -The tooling can't give you a list of all the tests that are expected to > only need a rebaseline. > -Related to the above, people often forget about these lines and don't do > the rebaseline. > > We should add [ NeedsRebasline ], which is equivalent to [ Missing Failure > ImageOnlyFailure ]. I'm thinking it should not include Timeout/Crash since > those would need a solution other than a rebaseline (e.g. something is wrong > with the test or patch). > > In garden-o-matic, we can make a tab specifically for tests that need > rebaseline and give some indication whether the original patch that line was > added in has run on all the relevant bots. This way the people keeping the > tree green can also make sure that NeedsRebaseline lines don't get > forgotten. > > If it continues to be a problem we could even setup and automated nag bot to > email people who leave in NeedsRebaseline lines for more than a week. > > In the long-run, we should make it so you can grab the new results of the > EWS bots and don't need to add lines to TestExpectations at all. In the > short-term though, this is a way we can handle pixel tests without making > the tree red all the time. > > As a side note, we should also get rid of Missing as a valid expectation. A > test should either be NeedsRebaseline or have an expected result. > > Ojan > > _______________________________________________ > webkit-dev mailing list > webkit-dev@lists.webkit.org > http://lists.webkit.org/mailman/listinfo/webkit-dev > _______________________________________________ webkit-dev mailing list webkit-dev@lists.webkit.org http://lists.webkit.org/mailman/listinfo/webkit-dev