We finally have correct baseline images checked in for Chromium-Skia on SnowLeopard (hooray). Now we face the daunting task of rebaselining for Chromium-Skia on Leopard.
The problem is that chromium-mac-leopard has ~4500 image diffs compared to chromium-mac. The vast majority of these diffs are due to very minor differences in text rendering between Leopard and SnowLeopard. (Details are in http://code.google.com/p/chromium/issues/detail?id=100904 ) I poked around in LayoutTests/platform (details are at the bottom of this email) and found that indeed there are 4000-5000 baseline images that distinguish between Leopard and SnowLeopard. This is presumably why layout tests generally pass on both platforms. Before Cary and I spend 2-3 days committing all these new baseline images so that we can re-enable test_expectations for SnowLeopard... is this the right approach? It seems pretty inefficient to clog the WebKit repository with all these baseline images with minor pixel value differences. Here are the various approaches I can think of... what's the Hive-Mind-Approved approach? - Commit 4500 new baseline images for SnowLeopard - pro: known to work, will catch any regressions that come later - con: takes a long time to commit, chews up disk space and bandwidth for all developers, future minor changes may require yet another set of new baselines - Leave all SnowLeopard tests marked as "PASS FAIL" (or maybe mark them "SKIP") in test_expectations - pro: known to work, quick and easy, doesn't clog repo space and developer update bandwidth, future minor changes won't break any bots - con: will not catch any regressions that come later on SnowLeopard - Remove descriptive text from all these tests, so that text rendering is only evaluated in tests specifically for that purpose - pro: prevents this problem for future OS versions, should allow for lots more baseline images to be shared across platforms - con: a lot of work to replace all existing baseline images, must coordinate across community of Chromium/WebKit developers, tests will be more difficult to interpret without text - Figure out how our test pages can be rendered with a completely cross-platform pixel-equivalent font - pro: similar to above but tests keep their descriptive text - con: similar to above but more technically challenging - Augment our pixel-diff tools to allow for comparison masks (only pay attention to pixel diffs within this rectangle) - pro: existing baseline images can stay in place, and perhaps be shared with new OS versions and platforms - con: requires modification of pixel-diff tools, need to add comparison mask to each test definition Details on how I determined that there are already 4000-5000 baseline images that distinguish between Leopard and SnowLeopard for existing platforms (Safari and chromium-cg): ~/src/webkit/rebaseline-greenify-bots/WebKit/LayoutTests$ mkdir tests ~/src/webkit/rebaseline-greenify-bots/WebKit/LayoutTests$ find platform -name *.png >tests/all-pngs ~/src/webkit/rebaseline-greenify-bots/WebKit/LayoutTests$ wc -l tests/all-pngs 45529 tests/all-pngs ~/src/webkit/rebaseline-greenify-bots/WebKit/LayoutTests$ cat tests/all-pngs | awk -F '/' '{print $2}' | sort | uniq > tests/list-all-platforms ~/src/webkit/rebaseline-greenify-bots/WebKit/LayoutTests$ grep mac tests/list-all-platforms >tests/list-mac-platforms ~/src/webkit/rebaseline-greenify-bots/WebKit/LayoutTests$ mkdir tests/by-platform ~/src/webkit/rebaseline-greenify-bots/WebKit/LayoutTests$ for PLATFORM in $(cat tests/list-mac-platforms); do grep ^platform/$PLATFORM/ tests/all-pngs | sed s/platform\\/$PLATFORM\\/// >tests/by-platform/$PLATFORM ; done ~/src/webkit/rebaseline-greenify-bots/WebKit/LayoutTests$ wc -l tests/by-platform/* 1050 tests/by-platform/chromium-cg-mac 1575 tests/by-platform/chromium-cg-mac-leopard 114 tests/by-platform/chromium-cg-mac-snowleopard 114 tests/by-platform/chromium-gpu-cg-mac 135 tests/by-platform/chromium-gpu-mac 4909 tests/by-platform/chromium-mac 79 tests/by-platform/chromium-mac-leopard 131 tests/by-platform/chromium-mac-snowleopard 7631 tests/by-platform/mac 4435 tests/by-platform/mac-leopard 247 tests/by-platform/mac-snowleopard 7 tests/by-platform/mac-wk2 20427 total ~/src/webkit/rebaseline-greenify-bots/WebKit/LayoutTests$ cat tests/by-platform/chromium-cg-mac tests/by-platform/chromium-cg-mac-snowleopard tests/by-platform/mac tests/by-platform/mac-snowleopard | sort | uniq >tests/list-snowleopard-cg-images ~/src/webkit/rebaseline-greenify-bots/WebKit/LayoutTests$ cat tests/by-platform/chromium-cg-mac-leopard tests/by-platform/mac-leopard | sort | uniq >tests/list-leopard-cg-images ~/src/webkit/rebaseline-greenify-bots/WebKit/LayoutTests$ cat tests/list-leopard-cg-images tests/list-snowleopard-cg-images | sort | uniq >tests/list-unique-cg-images ~/src/webkit/rebaseline-greenify-bots/WebKit/LayoutTests$ wc -l tests/*-cg-images 5120 tests/list-leopard-cg-images 7735 tests/list-snowleopard-cg-images 7744 tests/list-unique-cg-images 20599 total
_______________________________________________ webkit-dev mailing list webkit-dev@lists.webkit.org http://lists.webkit.org/mailman/listinfo.cgi/webkit-dev
