On 5/23/12 2:30 PM, "Maciej Stachowiak" <m...@apple.com> wrote:
> >On May 23, 2012, at 2:16 PM, Dirk Pranke <dpra...@chromium.org> wrote: > >> On Wed, May 23, 2012 at 1:41 PM, Ryosuke Niwa <rn...@webkit.org> wrote: >>>>> The only sane argument I've heard so far to gate pixel tests is that >>>>>the >>>>> correctness of such tests need to be manually inspected, which >>>>>requires >>>>> a >>>>> lot of manual labor and is very error prone. >>>> >>>> I'm assuming the above includes the ongoing maintenance cost of >>>> keeping pixel tests up to date, as well as the cost at the initial >>>> checkin. >>> >>> I'm not concerned of those. Once the correct expected result is >>>checked in, >>> it's pretty easy to rebaseline tests per rendering engine changes >>>assuming >>> people who are rebaselining tests know what they're doing. >> >> You should be concerned; keeping pixel tests up-to-date is clearly a >> non-zero cost that only the chromium port thus far has been willing to >> bear, and I suspect that the cost of updating baselines is >> substantially higher than the cost of the initial review over time >> (since it's a recurring cost). > >Are you concerned just about the actual pixel results or also about >keeping render tree dumps up to date? We can address the pixel result >issue by introducing a new test that dumps its render tree but does not >do pixel testing. > >I think there is a high value to importing standards test suites >wholesale, even if they overlap with our existing coverage. Picking and >choosing subsets makes things more complicated. If there are significant >externalities to adding particular kinds of tests, I would prefer we >mitigate those externalities rather than run fewer tests. As a side note to this discussion, there is talk in the W3C community regarding their test approval process. At the recent working group meetings in Germany the idea was floated to simply approve all tests that are currently waiting for review (and doing this going forward, e.g. no longer requiring approval upon submission). Apparently, not enough people are reviewing tests, and as a result, tests can linger for months (or longer) before ever being looked at. Once browser vendors start implementing features, associated tests will be revisited for that area. This has by no means been decided, but something we should consider if it ultimately does come to fruition. If W3C tests are no longer reviewed, this would mean importing tests without any knowledge of their accuracy - though that will still allow us to catch regressions. Dirk, I've updated the process on the Wiki page with the feedback you provided. I hope I captured it all - I included everything that appeared to have agreement between you and Ryosuke. Feel free to modify it directly if I missed anything, or let me know and I can refine it further. Jacob > >Regards, >Maciej > >_______________________________________________ >webkit-dev mailing list >webkit-dev@lists.webkit.org >http://lists.webkit.org/mailman/listinfo.cgi/webkit-dev _______________________________________________ webkit-dev mailing list webkit-dev@lists.webkit.org http://lists.webkit.org/mailman/listinfo.cgi/webkit-dev