On May 17, 2010, at 3:44 PM, James Robinson wrote: > Leopard and Snow Leopard have subtle differences in the way they render > antialiased text. This does not affect the text metrics but does cause > slight pixel differences. The majority of our existing pixel test baselines > appear to have been generated on Leopard, but there is a growing minority of > tests that have been generated on Snow Leopard as well as a small number of > baselines that are out of date or just plain incorrect. This means that when > run with --tolerance=0, there are a significant number of failures on Snow > Leopard that do not have anything to do with the behavior they are intended > to test. Here are some concrete numbers (taken around r59000, the exact > results vary slightly from revision to revision but not hugely). > > On a Leopard machine, running the tests with -p --tolerance=0 yields 120 > failures. With the default tolerance (0.1%) there are 68 failures. > > On Snow Leopard, with -p --tolerance=0 there are 3934 failures. With the > default tolerance (0.1%) there are 144 failures. > > Based on this, it seems reasonable to assume that there are ~3814 pixel test > results in platform/mac that are Leopard specific. There is also a smaller > number of results generated from Snow Leopard machines that fail on Leopard. > I think this is an unhealthy state for the project as it prevents people > developing on a Mac on Snow Leopard from running the pixel tests without > getting a huge number of spurious failures. Currently pretty much all Mac > developers are likely to be on Snow Leopard, except for Google employees who > will be moving to Snow Leopard in the near future. The pixel tests are our > only coverage for lots of the repaint logic currently. > > I think fundamentally there is a short term and a longer term problem here. > The short term problem is that since the Mac pixel results are in such a bad > state for people on Snow Leopard, people making changes to the code are not > likely to update the pixel results at all which leads to an even worse mess > in platform/mac. The longer term problem is that since the pixel tests are > only run on build bots by one port (Chromium) and not run on build.webkit.org > by anyone, the pixel tests are likely to languish.
Thanks for doing this, James. At one point we had pixel bots at Apple, but they fell by the wayside. > To resolve the immediate problem of Leopard-specific results in platform/mac, > I'd like to move all of the Leopard-specific results currently in > platform/mac to platform/mac-leopard and to generate new Snow Leopard > specific results for those tests in platform/mac. Specifically my plan is to > do this: > > For each test T with pixel results in platform/mac that fails on Leopard or > Snow Leopard with -p --tolerance=0: > - if T passes pixel tests exactly on Leopard and fails with a <0.5% pixel > difference on Snow Leopard, assume the only pixel difference is due to text > antialiasing and do the following: > - svn mv the current platform/mac pixel results to platform/mac-leopard > - add new pixel results generated on Snow Leopard into platform/mac > - there should be around 3800 tests in this category > - if T passes pixel tests exactly on Snow Leopard and fails with a <0.5% > pixel difference on Leopard: > - generate new pixel baselines for Leopard and add to platform/mac-leopard > - there should be around 50 tests in this category > - if T fails on both Leopard and Snow Leopard, remove the current > expectations and file a bug > - there should be around 70 tests in this category > > After this is done it should be possible to run the full layout test suite > with --tolerance=0 on Snow Leopard and have no failures. Assuming this > happens, I would like to switch the default value of --tolerance from 0.1 to > 0.0 to try to keep regressions from creeping back in. You may find that some of the text antialiasing differences are hardware-dependent, so I'm not sure if a tolerance of 0 would be possible. > Does this sound like a good plan? The biggest impact on the project will be > a number of large commits in LayoutTests/platform/ to do the svn move and svn > add operations to move results from platform/mac -> platform/mac-leopard and > to add new Snow Leopard results to platform/mac. This can be done over a > weekend and mostly by a script to try to minimize impact. This sounds like a great short-term plan. I think a longer-term plan would be to move as many tests to ref-tests as possible, which eliminates any text-aliasing issues. Simon
_______________________________________________ webkit-dev mailing list webkit-dev@lists.webkit.org http://lists.webkit.org/mailman/listinfo.cgi/webkit-dev
