Re: [webkit-dev] CSS 2.1 Test Suite
The best way to judge whether a reference result is correct is to submit the result to the W3C CSS 2.1 test suite and have it reviewed. The only way this test suite will get more reference results is if people like us volunteer to submit references. If it's useful to us to have these 'homebrew reference results' then it will be useful to everyone else who uses the suite. The previous thread mentioned checking Mozilla to see if their test suite had references for particular tests. If that's the case, then we should either encourage them to submit the references to the W3C, or just do that ourselves on their behalf. Alan From: Ryosuke Niwa rn...@webkit.org As we have previous discussed following https://lists.webkit.org/pipermail/webkit-dev/2012-March/019782.html, it's hard to judge whether a given reference result is enough to cover everything the test intends to test. e.g. you can have a bug such that both the test and the reference file ends up having the same rendering result. - Ryosuke On Tue, Apr 10, 2012 at 2:26 PM, Robert Hogan li...@roberthogan.net wrote: Hi there, We currently add tests from the CSS 2.1 suite as we fix them. They get added to the css2.1/20110323 folder in LayoutTests. Most of them don't have native reference tests yet in the suite so we (mostly I) have been adding homebrew reference results to the folder to avoid generating pixel results on all platforms. (see http://webkitmemes.tumblr.com/post/20788159625 !) These reference-results are easily removed once superseded but it might be cleaner just to move them, and the associated css tests, to a folder of their own in fast/css. That will allow css2.1/20110323 to be a clean import that the 8500 or so passing tests can be added to in 20 or 30 batches.[1] It will also make NRWT's reftests harness work with the suite. Does anyone object to that approach? The only thing going against it seems to be the principle that imported tests should be stored separately and together but this is more a case of using them to fix bugs and prevent future regressions while allowing a clean import of the CSS 2.1 test suite to take place in parallel. The problem this does not solve is how we avoid creating pixel results for tests that already pass but which do not have reftests of their own. Again I would be in favour of putting these in fast/css and keeping them there until reftests are added to the suite. This would allow us to prevent them regressing and come up with a reftest for them that can be submitted to the CSS test suite guys. The end result would be that we only directly import to the css2.1 folder those tests in the CSS test suite that have ref tests native to the suite. Thanks, Robert [1] I keep a local and relatively up to date copy of the passing and failing tests in separate folders in my checkout. Yes I know I should create bugs for them and get started on landing the passes. ___ webkit-dev mailing list webkit-dev@lists.webkit.org http://lists.webkit.org/mailman/listinfo.cgi/webkit-dev ___ webkit-dev mailing list webkit-dev@lists.webkit.org http://lists.webkit.org/mailman/listinfo.cgi/webkit-dev
Re: [webkit-dev] CSS 2.1 Test Suite
On Wed, Apr 11, 2012 at 12:11 PM, Alan Stearns stea...@adobe.com wrote: The best way to judge whether a reference result is correct is to submit the result to the W3C CSS 2.1 test suite and have it reviewed. The only way this test suite will get more reference results is if people like us volunteer to submit references. Right. We might want to add some wiki page on how to do this (step by step instruction) since not everyone contributing to the WebKit is familiar with their procedure. The previous thread mentioned checking Mozilla to see if their test suite had references for particular tests. If that's the case, then we should either encourage them to submit the references to the W3C, or just do that ourselves on their behalf. That'll be nice. - Ryosuke ___ webkit-dev mailing list webkit-dev@lists.webkit.org http://lists.webkit.org/mailman/listinfo.cgi/webkit-dev
Re: [webkit-dev] CSS 2.1 Test Suite
On Wednesday 11 April 2012 20:11:23 Alan Stearns wrote: The best way to judge whether a reference result is correct is to submit the result to the W3C CSS 2.1 test suite and have it reviewed. The only way this test suite will get more reference results is if people like us volunteer to submit references. If it's useful to us to have these 'homebrew reference results' then it will be useful to everyone else who uses the suite. Agreed. This will help us land the tests that already pass and won't slow down the effort to fix the css tests that we don't. We can agree to only import css tests with reftests and get NRWT working on them. I hope to do this in: https://bugs.webkit.org/show_bug.cgi?id=83048 However I do think we need a decision on how we: 1. Land fixes for currently broken tests that don't have a reftest. 2. Clean up the existing css2.1/20110323 folder. If it's just a case of living with pixel results for now I'm happy with that. But I think allowing them to live outside css2.1/20110323 would encourage me and others to write reftests while we're fixing the tests' results on WebKit. From listening to Maciej, Ojan and Ryosuke it is not an option to keep homebrew reftests in css2.1/20110323 for two good reasons: it breaks important assumptions in the way the reftest harness works, and it is better to keep imported test suites clean and unmodified. So it's either 1 or 2 above I think. I would prefer 2 as it won't bloat git checkouts so much and will make fixes easier to land. ___ webkit-dev mailing list webkit-dev@lists.webkit.org http://lists.webkit.org/mailman/listinfo.cgi/webkit-dev
Re: [webkit-dev] CSS 2.1 Test Suite
On Wed, Apr 11, 2012 at 12:25 PM, Robert Hogan li...@roberthogan.net wrote: On Wednesday 11 April 2012 20:11:23 Alan Stearns wrote: The best way to judge whether a reference result is correct is to submit the result to the W3C CSS 2.1 test suite and have it reviewed. The only way this test suite will get more reference results is if people like us volunteer to submit references. If it's useful to us to have these 'homebrew reference results' then it will be useful to everyone else who uses the suite. Agreed. This will help us land the tests that already pass and won't slow down the effort to fix the css tests that we don't. We can agree to only import css tests with reftests and get NRWT working on them. I hope to do this in: https://bugs.webkit.org/show_bug.cgi?id=83048 However I do think we need a decision on how we: 1. Land fixes for currently broken tests that don't have a reftest. 2. Clean up the existing css2.1/20110323 folder. If it's just a case of living with pixel results for now I'm happy with that. But I think allowing them to live outside css2.1/20110323 would encourage me and others to write reftests while we're fixing the tests' results on WebKit. From listening to Maciej, Ojan and Ryosuke it is not an option to keep homebrew reftests in css2.1/20110323 for two good reasons: it breaks important assumptions in the way the reftest harness works, and it is better to keep imported test suites clean and unmodified. So it's either 1 or 2 above I think. I would prefer 2 as it won't bloat git checkouts so much and will make fixes easier to land. What does clean up the existing folder entail? -- Dirk ___ webkit-dev mailing list webkit-dev@lists.webkit.org http://lists.webkit.org/mailman/listinfo.cgi/webkit-dev
Re: [webkit-dev] CSS 2.1 Test Suite
On Wed, Apr 11, 2012 at 12:25 PM, Robert Hogan li...@roberthogan.netwrote: On Wednesday 11 April 2012 20:11:23 Alan Stearns wrote: The best way to judge whether a reference result is correct is to submit the result to the W3C CSS 2.1 test suite and have it reviewed. The only way this test suite will get more reference results is if people like us volunteer to submit references. If it's useful to us to have these 'homebrew reference results' then it will be useful to everyone else who uses the suite. Agreed. This will help us land the tests that already pass and won't slow down the effort to fix the css tests that we don't. We can agree to only import css tests with reftests and get NRWT working on them. I hope to do this in: https://bugs.webkit.org/show_bug.cgi?id=83048 However I do think we need a decision on how we: 1. Land fixes for currently broken tests that don't have a reftest. 2. Clean up the existing css2.1/20110323 folder. If it's just a case of living with pixel results for now I'm happy with that. But I think allowing them to live outside css2.1/20110323 would encourage me and others to write reftests while we're fixing the tests' results on WebKit. I'd argue for just using pixel results for now, and submit patches (to W3C) to convert them to reftests. - Ryosuke ___ webkit-dev mailing list webkit-dev@lists.webkit.org http://lists.webkit.org/mailman/listinfo.cgi/webkit-dev
Re: [webkit-dev] CSS 2.1 Test Suite
On Tuesday 10 April 2012 22:35:13 Ryosuke Niwa wrote: As we have previous discussed following https://lists.webkit.org/pipermail/webkit-dev/2012-March/019782.html, it's hard to judge whether a given reference result is enough to cover everything the test intends to test. e.g. you can have a bug such that both the test and the reference file ends up having the same rendering result. Definitely - and this will certainly be the case with some of the css tests, but a minority. A *lot* of them are along the lines of 'this text/box is green'. For the tests we write to fix failing CSS tests this will definitely be something to watch out for in review. ___ webkit-dev mailing list webkit-dev@lists.webkit.org http://lists.webkit.org/mailman/listinfo.cgi/webkit-dev
Re: [webkit-dev] CSS 2.1 Test Suite
On Wednesday 11 April 2012 20:27:23 Dirk Pranke wrote: What does clean up the existing folder entail? Just move any -expected.html file out of there and generate pixel results. Or move the test and its -expected.html into a folder in fast/css. ___ webkit-dev mailing list webkit-dev@lists.webkit.org http://lists.webkit.org/mailman/listinfo.cgi/webkit-dev
[webkit-dev] CSS 2.1 Test Suite
Hi there, We currently add tests from the CSS 2.1 suite as we fix them. They get added to the css2.1/20110323 folder in LayoutTests. Most of them don't have native reference tests yet in the suite so we (mostly I) have been adding homebrew reference results to the folder to avoid generating pixel results on all platforms. (see http://webkitmemes.tumblr.com/post/20788159625 !) These reference-results are easily removed once superseded but it might be cleaner just to move them, and the associated css tests, to a folder of their own in fast/css. That will allow css2.1/20110323 to be a clean import that the 8500 or so passing tests can be added to in 20 or 30 batches.[1] It will also make NRWT's reftests harness work with the suite. Does anyone object to that approach? The only thing going against it seems to be the principle that imported tests should be stored separately and together but this is more a case of using them to fix bugs and prevent future regressions while allowing a clean import of the CSS 2.1 test suite to take place in parallel. The problem this does not solve is how we avoid creating pixel results for tests that already pass but which do not have reftests of their own. Again I would be in favour of putting these in fast/css and keeping them there until reftests are added to the suite. This would allow us to prevent them regressing and come up with a reftest for them that can be submitted to the CSS test suite guys. The end result would be that we only directly import to the css2.1 folder those tests in the CSS test suite that have ref tests native to the suite. Thanks, Robert [1] I keep a local and relatively up to date copy of the passing and failing tests in separate folders in my checkout. Yes I know I should create bugs for them and get started on landing the passes. ___ webkit-dev mailing list webkit-dev@lists.webkit.org http://lists.webkit.org/mailman/listinfo.cgi/webkit-dev
Re: [webkit-dev] CSS 2.1 Test Suite
As we have previous discussed following https://lists.webkit.org/pipermail/webkit-dev/2012-March/019782.html, it's hard to judge whether a given reference result is enough to cover everything the test intends to test. e.g. you can have a bug such that both the test and the reference file ends up having the same rendering result. - Ryosuke On Tue, Apr 10, 2012 at 2:26 PM, Robert Hogan li...@roberthogan.net wrote: Hi there, We currently add tests from the CSS 2.1 suite as we fix them. They get added to the css2.1/20110323 folder in LayoutTests. Most of them don't have native reference tests yet in the suite so we (mostly I) have been adding homebrew reference results to the folder to avoid generating pixel results on all platforms. (see http://webkitmemes.tumblr.com/post/20788159625 !) These reference-results are easily removed once superseded but it might be cleaner just to move them, and the associated css tests, to a folder of their own in fast/css. That will allow css2.1/20110323 to be a clean import that the 8500 or so passing tests can be added to in 20 or 30 batches.[1] It will also make NRWT's reftests harness work with the suite. Does anyone object to that approach? The only thing going against it seems to be the principle that imported tests should be stored separately and together but this is more a case of using them to fix bugs and prevent future regressions while allowing a clean import of the CSS 2.1 test suite to take place in parallel. The problem this does not solve is how we avoid creating pixel results for tests that already pass but which do not have reftests of their own. Again I would be in favour of putting these in fast/css and keeping them there until reftests are added to the suite. This would allow us to prevent them regressing and come up with a reftest for them that can be submitted to the CSS test suite guys. The end result would be that we only directly import to the css2.1 folder those tests in the CSS test suite that have ref tests native to the suite. Thanks, Robert [1] I keep a local and relatively up to date copy of the passing and failing tests in separate folders in my checkout. Yes I know I should create bugs for them and get started on landing the passes. ___ webkit-dev mailing list webkit-dev@lists.webkit.org http://lists.webkit.org/mailman/listinfo.cgi/webkit-dev ___ webkit-dev mailing list webkit-dev@lists.webkit.org http://lists.webkit.org/mailman/listinfo.cgi/webkit-dev
