On Wed, Sep 19, 2012 at 2:00 PM, Dirk Pranke <[email protected]> wrote: > On Wed, Sep 19, 2012 at 1:54 PM, Allan Sandfeld Jensen <[email protected]> > wrote: >> On Wednesday 19 September 2012, Dirk Pranke wrote: >>> After some limited amount of thought, I'm inclined to agree with >>> Ryosuke and Ossy here. If we have tests that don't depend on checking >>> the metrics, can they just be dumpAsText tests or reftests instead? >>> >> They could be. The tests I had in mind and made a conversion for are however >> examples of tests from external test suites. We have many tests of this >> nature, and converting them all the a webkit specific framework would be a >> lot >> of work, no one would actually do, and would mean we could not share and >> update the tests as easily with other members of W3C. >> >> This is exactly why I asked for what kind of guidelines we should make for >> when we should use baselines without metrics. For instance we could decide >> new >> tests cases made for WebKit bugs should alway use dumpAsText, and restrict >> this feature to only test cases from an external source, or only older tests. >> > > I see. My understanding is that the W3C is supposed to be moving in > the direction of more reftests everywhere, so hopefully this becomes > less of an issue. What to do with the legacy tests that may not get > updated is a different issue, of course. > > My personal hope / theory / claim is that we should make the diff > smarter about using the information we have to detect real issues, > rather than throwing away information and then potentially missing > issues.
Sorry, I realize this answer isn't as helpful as it could be. Personally, I'm not a fan of adding new (external) test suites that aren't text-only tests or reftests, and so I'm a bit negative here. However, I think the value depends on the test suite in question and what it's testing, so it's hard for me to give a blanket statement. If you can point to specific tests you'd like to try this for I can comment in more detail, but I'm not really okay with a blanket approval for using --ignore-metrics whenever we feel like it yet. -- Dirk _______________________________________________ webkit-dev mailing list [email protected] http://lists.webkit.org/mailman/listinfo/webkit-dev
