[webkit-dev] Baselines ignoring metrics
Hello WebKit, Some time ago I wrote a patch that would make the test runners detect baseline that had been generated with the --ignore-metrics flag, see https://bugs.webkit.org/show_bug.cgi?id=94815 . This makes it possible to generate baselines that would always be testing ignoring metrics, and which would help consolidate more expected results between platforms - see the second patch in the bug for an example of that. The cases I expect this to make sense in, are tests for css selectors, parsing and other places the final layout is not part of what is being tested. Ideally such new tests should be written using dumpText, but we have many old test- cases that has not been written that way, and many more that are imported from from external test suites which never will be. Should we apply the patch and start using ignore-metric baselines, and should we make some guidelines for what kind of tests this should be allowed on? Best regards `Allan Sandfeld ___ webkit-dev mailing list webkit-dev@lists.webkit.org http://lists.webkit.org/mailman/listinfo/webkit-dev
Re: [webkit-dev] Baselines ignoring metrics
As ossy has suggested in the bug, we should just convert those tests that don't require metrics equivalency to dumpAsText tests. On Sep 19, 2012 2:53 AM, Allan Sandfeld Jensen k...@carewolf.com wrote: Hello WebKit, Some time ago I wrote a patch that would make the test runners detect baseline that had been generated with the --ignore-metrics flag, see https://bugs.webkit.org/show_bug.cgi?id=94815 . This makes it possible to generate baselines that would always be testing ignoring metrics, and which would help consolidate more expected results between platforms - see the second patch in the bug for an example of that. The cases I expect this to make sense in, are tests for css selectors, parsing and other places the final layout is not part of what is being tested. Ideally such new tests should be written using dumpText, but we have many old test- cases that has not been written that way, and many more that are imported from from external test suites which never will be. Should we apply the patch and start using ignore-metric baselines, and should we make some guidelines for what kind of tests this should be allowed on? Best regards `Allan Sandfeld ___ webkit-dev mailing list webkit-dev@lists.webkit.org http://lists.webkit.org/mailman/listinfo/webkit-dev ___ webkit-dev mailing list webkit-dev@lists.webkit.org http://lists.webkit.org/mailman/listinfo/webkit-dev
Re: [webkit-dev] Baselines ignoring metrics
After some limited amount of thought, I'm inclined to agree with Ryosuke and Ossy here. If we have tests that don't depend on checking the metrics, can they just be dumpAsText tests or reftests instead? I thought the initial motivation for --ignore-metrics was for new ports to at least confirm that the render trees had the right shape even if they didn't have the right metrics. It's unclear to me if this has ever been useful for people, though. On a related note (which I thought I mentioned in some ignore-metrics-related bug, but apparently at least not this one), I've been working on various render-tree-parsing-related features, and I've been thinking that it might be nice to have a smarter diff that could distinguish between fields that actually rendered differently due to different intrinsic metrics vs. fields that rendered differently because of different derived metrics (e.g., two text fields are different sizes because the text renders differently vs. two containing blocks that are also different because they contain the different text fields; if both container and containing block are off by one pixel, on the the contained block's diff is really interesting). But, I don't think (but am not yet 100% sure) that the existing rendertree format contains the information needed to do this accurately. -- Dirk On Wed, Sep 19, 2012 at 1:08 PM, Ryosuke Niwa rn...@webkit.org wrote: As ossy has suggested in the bug, we should just convert those tests that don't require metrics equivalency to dumpAsText tests. On Sep 19, 2012 2:53 AM, Allan Sandfeld Jensen k...@carewolf.com wrote: Hello WebKit, Some time ago I wrote a patch that would make the test runners detect baseline that had been generated with the --ignore-metrics flag, see https://bugs.webkit.org/show_bug.cgi?id=94815 . This makes it possible to generate baselines that would always be testing ignoring metrics, and which would help consolidate more expected results between platforms - see the second patch in the bug for an example of that. The cases I expect this to make sense in, are tests for css selectors, parsing and other places the final layout is not part of what is being tested. Ideally such new tests should be written using dumpText, but we have many old test- cases that has not been written that way, and many more that are imported from from external test suites which never will be. Should we apply the patch and start using ignore-metric baselines, and should we make some guidelines for what kind of tests this should be allowed on? Best regards `Allan Sandfeld ___ webkit-dev mailing list webkit-dev@lists.webkit.org http://lists.webkit.org/mailman/listinfo/webkit-dev ___ webkit-dev mailing list webkit-dev@lists.webkit.org http://lists.webkit.org/mailman/listinfo/webkit-dev ___ webkit-dev mailing list webkit-dev@lists.webkit.org http://lists.webkit.org/mailman/listinfo/webkit-dev
Re: [webkit-dev] Baselines ignoring metrics
On Wednesday 19 September 2012, Dirk Pranke wrote: After some limited amount of thought, I'm inclined to agree with Ryosuke and Ossy here. If we have tests that don't depend on checking the metrics, can they just be dumpAsText tests or reftests instead? They could be. The tests I had in mind and made a conversion for are however examples of tests from external test suites. We have many tests of this nature, and converting them all the a webkit specific framework would be a lot of work, no one would actually do, and would mean we could not share and update the tests as easily with other members of W3C. This is exactly why I asked for what kind of guidelines we should make for when we should use baselines without metrics. For instance we could decide new tests cases made for WebKit bugs should alway use dumpAsText, and restrict this feature to only test cases from an external source, or only older tests. Best regards `Allan Sandfeld ___ webkit-dev mailing list webkit-dev@lists.webkit.org http://lists.webkit.org/mailman/listinfo/webkit-dev
Re: [webkit-dev] Baselines ignoring metrics
On Wed, Sep 19, 2012 at 1:54 PM, Allan Sandfeld Jensen k...@carewolf.com wrote: On Wednesday 19 September 2012, Dirk Pranke wrote: After some limited amount of thought, I'm inclined to agree with Ryosuke and Ossy here. If we have tests that don't depend on checking the metrics, can they just be dumpAsText tests or reftests instead? They could be. The tests I had in mind and made a conversion for are however examples of tests from external test suites. We have many tests of this nature, and converting them all the a webkit specific framework would be a lot of work, no one would actually do, and would mean we could not share and update the tests as easily with other members of W3C. This is exactly why I asked for what kind of guidelines we should make for when we should use baselines without metrics. For instance we could decide new tests cases made for WebKit bugs should alway use dumpAsText, and restrict this feature to only test cases from an external source, or only older tests. I see. My understanding is that the W3C is supposed to be moving in the direction of more reftests everywhere, so hopefully this becomes less of an issue. What to do with the legacy tests that may not get updated is a different issue, of course. My personal hope / theory / claim is that we should make the diff smarter about using the information we have to detect real issues, rather than throwing away information and then potentially missing issues. -- Dirk ___ webkit-dev mailing list webkit-dev@lists.webkit.org http://lists.webkit.org/mailman/listinfo/webkit-dev
Re: [webkit-dev] Baselines ignoring metrics
On Wed, Sep 19, 2012 at 2:00 PM, Dirk Pranke dpra...@chromium.org wrote: On Wed, Sep 19, 2012 at 1:54 PM, Allan Sandfeld Jensen k...@carewolf.com wrote: On Wednesday 19 September 2012, Dirk Pranke wrote: After some limited amount of thought, I'm inclined to agree with Ryosuke and Ossy here. If we have tests that don't depend on checking the metrics, can they just be dumpAsText tests or reftests instead? They could be. The tests I had in mind and made a conversion for are however examples of tests from external test suites. We have many tests of this nature, and converting them all the a webkit specific framework would be a lot of work, no one would actually do, and would mean we could not share and update the tests as easily with other members of W3C. This is exactly why I asked for what kind of guidelines we should make for when we should use baselines without metrics. For instance we could decide new tests cases made for WebKit bugs should alway use dumpAsText, and restrict this feature to only test cases from an external source, or only older tests. I see. My understanding is that the W3C is supposed to be moving in the direction of more reftests everywhere, so hopefully this becomes less of an issue. What to do with the legacy tests that may not get updated is a different issue, of course. My personal hope / theory / claim is that we should make the diff smarter about using the information we have to detect real issues, rather than throwing away information and then potentially missing issues. Sorry, I realize this answer isn't as helpful as it could be. Personally, I'm not a fan of adding new (external) test suites that aren't text-only tests or reftests, and so I'm a bit negative here. However, I think the value depends on the test suite in question and what it's testing, so it's hard for me to give a blanket statement. If you can point to specific tests you'd like to try this for I can comment in more detail, but I'm not really okay with a blanket approval for using --ignore-metrics whenever we feel like it yet. -- Dirk ___ webkit-dev mailing list webkit-dev@lists.webkit.org http://lists.webkit.org/mailman/listinfo/webkit-dev
Re: [webkit-dev] Baselines ignoring metrics
On Wednesday 19 September 2012, Dirk Pranke wrote: If you can point to specific tests you'd like to try this for I can comment in more detail, but I'm not really okay with a blanket approval for using --ignore-metrics whenever we feel like it yet. In the referenced bug I added an example patch converting all the tests under fast/selectors. They are mostly from the css selectors suite plus a few extra. Since they all use text, almost none of the results are shared between platforms, but they could all be with this patch. `Allan ___ webkit-dev mailing list webkit-dev@lists.webkit.org http://lists.webkit.org/mailman/listinfo/webkit-dev
