On Tue, Mar 9, 2010 at 9:19 AM, Robert O'Callahan <rob...@ocallahan.org>wrote:
> > Mozilla has been using this technique for years. Perhaps we can pick their >> brains for some good tricks. Or, dare I say it, share some tests. >> >> >> > Hi, hope I'm not crashing the party, and sorry I'm late :-). Let me just > say a few things about reftests... > > Maciej mentioned that a reftest can assert that <b> and > style="font-weight:bold" are equivalent, but would not catch a bug that > completely disabled font-weight. That is true. However, in our reftest > framework you can also assert that two pages render differently, so you can > test that font-weight:normal renders differently from font-weight:bold, > which would catch a bug like that. If you look in the Mozilla reftests, > there are many tests with 'sanity' in the name ensuring that pages that > should render differently, do. > > You could still miss a weird bug like font-weight:bold making the text > italic instead. But in practice, we hardly ever encounter bugs that cause > some reftests to render incorrectly and still all tests pass. > > Over time we have learned a number of tricks for writing reftests. Here are > a few: > -- In general we require tests to match pixel-perfectly. It's always > tempting to allow a fudge factor, but tiny differences can indicate > significant bugs. > -- We have nifty SVG filter tricks to help find those tiny differences, > e.g. > http://mxr.mozilla.org/mozilla-central/source/layout/tools/reftest/reftest-analyzer.xhtml > -- If some pixels of a test are unpredictable or platform-dependent, but > not relevant to the test, you can censor them out by placing an opaque > element over them in the test and reference. > This and several of the other tips in here seem like they can apply nicely to our existing layout tests as well. > -- In other tests with unpredictable pixel values, e.g. video, we use SVG > filters to threshold pixel channel values. > -- The Web usually has More Than One Way To Do It. E.g. for CSS gradients, > a lot of our reference pages use canvas to render a reference gradient. > -- Many Web features have particular cases that are easy to reduce to > reftests even when general cases aren't. For example, box-shadow and > text-shadow with no blur are trivial to test with reftests. You can create > gradients using repeated stops to achieve abrupt transitions and test them > against colored boxes. You could use a similar trick to test patterns in SVG > text. > -- Aggressive subpixel antialiasing is a real pain. For example, wrapping > text in an overflow:hidden container isn't always a no-op even if the > container is sized to its contents. Using sans-serif fonts helps, using CSS > padding helps more. > -- For speed, conciseness and overall expediency, you can often pack many > variations of a test into a single reftest page, and in practice there's no > significant downside to that. > > One might argue that although reftests are a good way to test what they > test, they don't provide enough broad functional testing, e.g. to make sure > that blurs "look right". I don't have good data to refute or confirm that. > However, if we want to, we can get an image test just by making the > reference page a PNG ... but I can only see two cases out of over 4000 where > we felt we needed to do that. > > Overall, I'm extremely enthusiastic about reftests! As others mentioned, > we'd like to get official CSS and SVG test suites using reftests; they're > the best approach I know of for writing robust automated layout and > rendering tests across platforms and browsers. > > Rob > -- > "He was pierced for our transgressions, he was crushed for our iniquities; > the punishment that brought us peace was upon him, and by his wounds we are > healed. We all, like sheep, have gone astray, each of us has turned to his > own way; and the LORD has laid on him the iniquity of us all." [Isaiah > 53:5-6] > > _______________________________________________ > webkit-dev mailing list > webkit-dev@lists.webkit.org > http://lists.webkit.org/mailman/listinfo.cgi/webkit-dev > >
_______________________________________________ webkit-dev mailing list webkit-dev@lists.webkit.org http://lists.webkit.org/mailman/listinfo.cgi/webkit-dev