On Jun 6, 2012 10:36 PM, "Peter Kasting" <pkast...@chromium.org> wrote: > > On Wed, Jun 6, 2012 at 10:22 PM, Ryosuke Niwa <rn...@webkit.org> wrote: >> >> Now that everyone knows the problem, I propose to rename FAIL to DIFF. >> >> FAIL should mean that the test fails, not that it fails with image, text, or image and text failures. >> >> DIFF, on the other hand, has no ambiguity. It can't be interpreted as timeout, crash, or pass but can easily be associated with image and text differences. > > > I don't think DIFF is any better. It sounds like it means the output is "different than what we wanted", thus it effectively means "didn't pass", and one would expect it to match MISSING/CRASH/TIMEOUT as much as one would expect FAIL to.
The output being different implies that we have an output, which isn't true when DRT/WTR crashes or times out. > Personally I'd prefer to resolve this -- if we need to -- by removing FAIL entirely. Being explicit about your expectations isn't a bad thing. Plus, the number of cases that are truly TEXT IMAGE IMAGE+TEXT seems likely to be small. People use FAIL when they don't know what to expect; e.g. adding or rebaselining tests. zit's utterky unreasonable to expect patch authors to add TEXT IMAGE TEXT+IMAGE to every test they're expecting to rebaseline. I also think it's a bad practice to add test expections just to keep bots green. It's much better if the tests started to fail on the waterfall so that people who pay attention to bots can rebaseline them since most people forget about rebaselining tests once their patches are landed. - Ryosuke
_______________________________________________ webkit-dev mailing list webkit-dev@lists.webkit.org http://lists.webkit.org/mailman/listinfo.cgi/webkit-dev
