[webkit-dev] trac.webkit.org seems down?
http://trac.webkit.org/ is failing to load for me. ___ webkit-dev mailing list webkit-dev@lists.webkit.org http://lists.webkit.org/mailman/listinfo.cgi/webkit-dev
Re: [webkit-dev] webkitPostMessage
[+dslomov]
I found a thread on public-webapps that seems related to this
question. Dmitry, do you know what current status is here? I'd like
to make sure we're on a path towards interoperability with other
browsers.
Thanks,
Adam
On Tue, Apr 10, 2012 at 9:08 PM, Adam Barth aba...@webkit.org wrote:
I'm trying to understand why we have both DOMWindow.webkitPostMessage
and DOMWindow.postMessage. I'm also trying to understand the
following comment in {JS,V8}DOMWindowCustom.cpp:
// This function has variable arguments and can be:
// Per current spec:
// postMessage(message, targetOrigin)
// postMessage(message, targetOrigin, {sequence of transferrables})
// Legacy non-standard implementations in webkit allowed:
// postMessage(message, {sequence of transferrables}, targetOrigin);
Specifically:
1) Can we remove webkitPostMessage? If we can't remove it now, is
there a time in the future at which we can remove it?
2) Can we adopt the behavior in the specification (and drop the
non-standard behavior)? If not, should we change the specification to
match our behavior?
Many thanks,
Adam
___
webkit-dev mailing list
webkit-dev@lists.webkit.org
http://lists.webkit.org/mailman/listinfo.cgi/webkit-dev
Re: [webkit-dev] trac.webkit.org seems down?
Take it easy, Eric! No trac, no SVN server, no commit, no regression, no problem. ;-) Eric Seidel írta: http://trac.webkit.org/ is failing to load for me. ___ webkit-dev mailing list webkit-dev@lists.webkit.org http://lists.webkit.org/mailman/listinfo.cgi/webkit-dev
Re: [webkit-dev] trac.webkit.org seems down?
Resent from the right address. On Wed, Apr 11, 2012 at 12:47 AM, Ryosuke Niwa * wrote: Observing the same. Seems like it's time to go sleep... On Wed, Apr 11, 2012 at 12:35 AM, Eric Seidel e...@webkit.org wrote: http://trac.webkit.org/ is failing to load for me. ___ webkit-dev mailing list webkit-dev@lists.webkit.org http://lists.webkit.org/mailman/listinfo.cgi/webkit-dev ___ webkit-dev mailing list webkit-dev@lists.webkit.org http://lists.webkit.org/mailman/listinfo.cgi/webkit-dev
Re: [webkit-dev] trac.webkit.org seems down?
The trac/svn server was overloaded for about an hour and recovered on its own around 1am PDT. -Bill On Apr 11, 2012, at 12:50 AM, Ryosuke Niwa wrote: Resent from the right address. On Wed, Apr 11, 2012 at 12:47 AM, Ryosuke Niwa * wrote: Observing the same. Seems like it's time to go sleep... On Wed, Apr 11, 2012 at 12:35 AM, Eric Seidel e...@webkit.org wrote: http://trac.webkit.org/ is failing to load for me. ___ webkit-dev mailing list webkit-dev@lists.webkit.org http://lists.webkit.org/mailman/listinfo.cgi/webkit-dev ___ webkit-dev mailing list webkit-dev@lists.webkit.org http://lists.webkit.org/mailman/listinfo.cgi/webkit-dev ___ webkit-dev mailing list webkit-dev@lists.webkit.org http://lists.webkit.org/mailman/listinfo.cgi/webkit-dev
Re: [webkit-dev] CSS 2.1 Test Suite
The best way to judge whether a reference result is correct is to submit the result to the W3C CSS 2.1 test suite and have it reviewed. The only way this test suite will get more reference results is if people like us volunteer to submit references. If it's useful to us to have these 'homebrew reference results' then it will be useful to everyone else who uses the suite. The previous thread mentioned checking Mozilla to see if their test suite had references for particular tests. If that's the case, then we should either encourage them to submit the references to the W3C, or just do that ourselves on their behalf. Alan From: Ryosuke Niwa rn...@webkit.org As we have previous discussed following https://lists.webkit.org/pipermail/webkit-dev/2012-March/019782.html, it's hard to judge whether a given reference result is enough to cover everything the test intends to test. e.g. you can have a bug such that both the test and the reference file ends up having the same rendering result. - Ryosuke On Tue, Apr 10, 2012 at 2:26 PM, Robert Hogan li...@roberthogan.net wrote: Hi there, We currently add tests from the CSS 2.1 suite as we fix them. They get added to the css2.1/20110323 folder in LayoutTests. Most of them don't have native reference tests yet in the suite so we (mostly I) have been adding homebrew reference results to the folder to avoid generating pixel results on all platforms. (see http://webkitmemes.tumblr.com/post/20788159625 !) These reference-results are easily removed once superseded but it might be cleaner just to move them, and the associated css tests, to a folder of their own in fast/css. That will allow css2.1/20110323 to be a clean import that the 8500 or so passing tests can be added to in 20 or 30 batches.[1] It will also make NRWT's reftests harness work with the suite. Does anyone object to that approach? The only thing going against it seems to be the principle that imported tests should be stored separately and together but this is more a case of using them to fix bugs and prevent future regressions while allowing a clean import of the CSS 2.1 test suite to take place in parallel. The problem this does not solve is how we avoid creating pixel results for tests that already pass but which do not have reftests of their own. Again I would be in favour of putting these in fast/css and keeping them there until reftests are added to the suite. This would allow us to prevent them regressing and come up with a reftest for them that can be submitted to the CSS test suite guys. The end result would be that we only directly import to the css2.1 folder those tests in the CSS test suite that have ref tests native to the suite. Thanks, Robert [1] I keep a local and relatively up to date copy of the passing and failing tests in separate folders in my checkout. Yes I know I should create bugs for them and get started on landing the passes. ___ webkit-dev mailing list webkit-dev@lists.webkit.org http://lists.webkit.org/mailman/listinfo.cgi/webkit-dev ___ webkit-dev mailing list webkit-dev@lists.webkit.org http://lists.webkit.org/mailman/listinfo.cgi/webkit-dev
Re: [webkit-dev] CSS 2.1 Test Suite
On Wed, Apr 11, 2012 at 12:11 PM, Alan Stearns stea...@adobe.com wrote: The best way to judge whether a reference result is correct is to submit the result to the W3C CSS 2.1 test suite and have it reviewed. The only way this test suite will get more reference results is if people like us volunteer to submit references. Right. We might want to add some wiki page on how to do this (step by step instruction) since not everyone contributing to the WebKit is familiar with their procedure. The previous thread mentioned checking Mozilla to see if their test suite had references for particular tests. If that's the case, then we should either encourage them to submit the references to the W3C, or just do that ourselves on their behalf. That'll be nice. - Ryosuke ___ webkit-dev mailing list webkit-dev@lists.webkit.org http://lists.webkit.org/mailman/listinfo.cgi/webkit-dev
Re: [webkit-dev] CSS 2.1 Test Suite
On Wednesday 11 April 2012 20:11:23 Alan Stearns wrote: The best way to judge whether a reference result is correct is to submit the result to the W3C CSS 2.1 test suite and have it reviewed. The only way this test suite will get more reference results is if people like us volunteer to submit references. If it's useful to us to have these 'homebrew reference results' then it will be useful to everyone else who uses the suite. Agreed. This will help us land the tests that already pass and won't slow down the effort to fix the css tests that we don't. We can agree to only import css tests with reftests and get NRWT working on them. I hope to do this in: https://bugs.webkit.org/show_bug.cgi?id=83048 However I do think we need a decision on how we: 1. Land fixes for currently broken tests that don't have a reftest. 2. Clean up the existing css2.1/20110323 folder. If it's just a case of living with pixel results for now I'm happy with that. But I think allowing them to live outside css2.1/20110323 would encourage me and others to write reftests while we're fixing the tests' results on WebKit. From listening to Maciej, Ojan and Ryosuke it is not an option to keep homebrew reftests in css2.1/20110323 for two good reasons: it breaks important assumptions in the way the reftest harness works, and it is better to keep imported test suites clean and unmodified. So it's either 1 or 2 above I think. I would prefer 2 as it won't bloat git checkouts so much and will make fixes easier to land. ___ webkit-dev mailing list webkit-dev@lists.webkit.org http://lists.webkit.org/mailman/listinfo.cgi/webkit-dev
Re: [webkit-dev] CSS 2.1 Test Suite
On Wed, Apr 11, 2012 at 12:25 PM, Robert Hogan li...@roberthogan.net wrote: On Wednesday 11 April 2012 20:11:23 Alan Stearns wrote: The best way to judge whether a reference result is correct is to submit the result to the W3C CSS 2.1 test suite and have it reviewed. The only way this test suite will get more reference results is if people like us volunteer to submit references. If it's useful to us to have these 'homebrew reference results' then it will be useful to everyone else who uses the suite. Agreed. This will help us land the tests that already pass and won't slow down the effort to fix the css tests that we don't. We can agree to only import css tests with reftests and get NRWT working on them. I hope to do this in: https://bugs.webkit.org/show_bug.cgi?id=83048 However I do think we need a decision on how we: 1. Land fixes for currently broken tests that don't have a reftest. 2. Clean up the existing css2.1/20110323 folder. If it's just a case of living with pixel results for now I'm happy with that. But I think allowing them to live outside css2.1/20110323 would encourage me and others to write reftests while we're fixing the tests' results on WebKit. From listening to Maciej, Ojan and Ryosuke it is not an option to keep homebrew reftests in css2.1/20110323 for two good reasons: it breaks important assumptions in the way the reftest harness works, and it is better to keep imported test suites clean and unmodified. So it's either 1 or 2 above I think. I would prefer 2 as it won't bloat git checkouts so much and will make fixes easier to land. What does clean up the existing folder entail? -- Dirk ___ webkit-dev mailing list webkit-dev@lists.webkit.org http://lists.webkit.org/mailman/listinfo.cgi/webkit-dev
Re: [webkit-dev] CSS 2.1 Test Suite
On Wed, Apr 11, 2012 at 12:25 PM, Robert Hogan li...@roberthogan.netwrote: On Wednesday 11 April 2012 20:11:23 Alan Stearns wrote: The best way to judge whether a reference result is correct is to submit the result to the W3C CSS 2.1 test suite and have it reviewed. The only way this test suite will get more reference results is if people like us volunteer to submit references. If it's useful to us to have these 'homebrew reference results' then it will be useful to everyone else who uses the suite. Agreed. This will help us land the tests that already pass and won't slow down the effort to fix the css tests that we don't. We can agree to only import css tests with reftests and get NRWT working on them. I hope to do this in: https://bugs.webkit.org/show_bug.cgi?id=83048 However I do think we need a decision on how we: 1. Land fixes for currently broken tests that don't have a reftest. 2. Clean up the existing css2.1/20110323 folder. If it's just a case of living with pixel results for now I'm happy with that. But I think allowing them to live outside css2.1/20110323 would encourage me and others to write reftests while we're fixing the tests' results on WebKit. I'd argue for just using pixel results for now, and submit patches (to W3C) to convert them to reftests. - Ryosuke ___ webkit-dev mailing list webkit-dev@lists.webkit.org http://lists.webkit.org/mailman/listinfo.cgi/webkit-dev
Re: [webkit-dev] CSS 2.1 Test Suite
On Tuesday 10 April 2012 22:35:13 Ryosuke Niwa wrote: As we have previous discussed following https://lists.webkit.org/pipermail/webkit-dev/2012-March/019782.html, it's hard to judge whether a given reference result is enough to cover everything the test intends to test. e.g. you can have a bug such that both the test and the reference file ends up having the same rendering result. Definitely - and this will certainly be the case with some of the css tests, but a minority. A *lot* of them are along the lines of 'this text/box is green'. For the tests we write to fix failing CSS tests this will definitely be something to watch out for in review. ___ webkit-dev mailing list webkit-dev@lists.webkit.org http://lists.webkit.org/mailman/listinfo.cgi/webkit-dev
Re: [webkit-dev] CSS 2.1 Test Suite
On Wednesday 11 April 2012 20:27:23 Dirk Pranke wrote: What does clean up the existing folder entail? Just move any -expected.html file out of there and generate pixel results. Or move the test and its -expected.html into a folder in fast/css. ___ webkit-dev mailing list webkit-dev@lists.webkit.org http://lists.webkit.org/mailman/listinfo.cgi/webkit-dev
[webkit-dev] Process for making changes that affect layout test results
Given the recent discussion on test_expectations.txt, perhaps the answer to my question is still up in the air. I'm working on a change that I expect to require changing the expectations for about 75 tests on chromium win and linux. https://trac.webkit.org/wiki/Rebaseline seems to only cover the gardening work to rebaseline after the commit. I cannot find any wiki pages that describe what the original author is expected to do when making visual changes. Should I attempt to rebaseline manually? Should I mark the tests as failing? Should I just check in and let the bots go red? Thanks, Tony ___ webkit-dev mailing list webkit-dev@lists.webkit.org http://lists.webkit.org/mailman/listinfo.cgi/webkit-dev
Re: [webkit-dev] Process for making changes that affect layout test results
On Wed, Apr 11, 2012 at 1:57 PM, Tony Payne tpa...@chromium.org wrote: Given the recent discussion on test_expectations.txt, perhaps the answer to my question is still up in the air. I'm working on a change that I expect to require changing the expectations for about 75 tests on chromium win and linux. https://trac.webkit.org/wiki/Rebaseline seems to only cover the gardening work to rebaseline after the commit. I cannot find any wiki pages that describe what the original author is expected to do when making visual changes. Should I attempt to rebaseline manually? Should I mark the tests as failing? Should I just check in and let the bots go red? Just land the patch and rebaseline the tests. Please also coordinate with Chromium port's WebKit gardener when landing this patch. Also, does this patch only affect Chromium Windows and Linux, and not GTK, Qt, Windows, etc...? If the answer is no, and will affect other non-Chromium ports, then you're also responsible for rebaselining or coordinating with other ports to make sure you don't break tests on their ports as well. - Ryosuke ___ webkit-dev mailing list webkit-dev@lists.webkit.org http://lists.webkit.org/mailman/listinfo.cgi/webkit-dev
Re: [webkit-dev] Process for making changes that affect layout test results
All code I'm changing is inside of #if PLATFORM(CHROMIUM) blocks. Thanks for the quick answer. Tony On Wed, Apr 11, 2012 at 2:03 PM, Ryosuke Niwa rn...@webkit.org wrote: On Wed, Apr 11, 2012 at 1:57 PM, Tony Payne tpa...@chromium.org wrote: Given the recent discussion on test_expectations.txt, perhaps the answer to my question is still up in the air. I'm working on a change that I expect to require changing the expectations for about 75 tests on chromium win and linux. https://trac.webkit.org/wiki/Rebaseline seems to only cover the gardening work to rebaseline after the commit. I cannot find any wiki pages that describe what the original author is expected to do when making visual changes. Should I attempt to rebaseline manually? Should I mark the tests as failing? Should I just check in and let the bots go red? Just land the patch and rebaseline the tests. Please also coordinate with Chromium port's WebKit gardener when landing this patch. Also, does this patch only affect Chromium Windows and Linux, and not GTK, Qt, Windows, etc...? If the answer is no, and will affect other non-Chromium ports, then you're also responsible for rebaselining or coordinating with other ports to make sure you don't break tests on their ports as well. - Ryosuke ___ webkit-dev mailing list webkit-dev@lists.webkit.org http://lists.webkit.org/mailman/listinfo.cgi/webkit-dev
Re: [webkit-dev] Process for making changes that affect layout test results
Typically, if you're working on Chromium Linux or Win, you'd include the new expected results for that platform in your initial commit/code-review as well. On Wed, Apr 11, 2012 at 2:09 PM, Tony Payne tpa...@chromium.org wrote: All code I'm changing is inside of #if PLATFORM(CHROMIUM) blocks. Thanks for the quick answer. Tony On Wed, Apr 11, 2012 at 2:03 PM, Ryosuke Niwa rn...@webkit.org wrote: On Wed, Apr 11, 2012 at 1:57 PM, Tony Payne tpa...@chromium.org wrote: Given the recent discussion on test_expectations.txt, perhaps the answer to my question is still up in the air. I'm working on a change that I expect to require changing the expectations for about 75 tests on chromium win and linux. https://trac.webkit.org/wiki/Rebaseline seems to only cover the gardening work to rebaseline after the commit. I cannot find any wiki pages that describe what the original author is expected to do when making visual changes. Should I attempt to rebaseline manually? Should I mark the tests as failing? Should I just check in and let the bots go red? Just land the patch and rebaseline the tests. Please also coordinate with Chromium port's WebKit gardener when landing this patch. Also, does this patch only affect Chromium Windows and Linux, and not GTK, Qt, Windows, etc...? If the answer is no, and will affect other non-Chromium ports, then you're also responsible for rebaselining or coordinating with other ports to make sure you don't break tests on their ports as well. - Ryosuke ___ webkit-dev mailing list webkit-dev@lists.webkit.org http://lists.webkit.org/mailman/listinfo.cgi/webkit-dev ___ webkit-dev mailing list webkit-dev@lists.webkit.org http://lists.webkit.org/mailman/listinfo.cgi/webkit-dev
[webkit-dev] NPAPI plugin may be broken in mac latest sources?
Yesterday I updated to the latest webkit sources for the mac (`Tools\Scripts update-webkit`) and rebuilt. Now, when I run Safari using them (`Tools\Scripts\run-safari --debug`) and open a file for my plug-in (which works fine), then go Back/Forward QUICKLY, NPP_NewStream() is never called for Forward. If I go Back/Forward SLOWLY, then it gets called for Forward. In between Back and Forward, the NP_Shutdown() .. NP_Initialize, NPP_New is called in both cases, and I've verified they're all returning no error (or valid values). The release of Safari I have (5.1.5) does not exhibit this behavior (I'm on Lion). I don't see anything in the bug database about this; before I enter a bug, is this known? Thanks, Rudi ___ webkit-dev mailing list webkit-dev@lists.webkit.org http://lists.webkit.org/mailman/listinfo.cgi/webkit-dev
Re: [webkit-dev] NPAPI plugin may be broken in mac latest sources?
The first step is to file a bug. Then we can CC the right people and triage your troubles. http://www.webkit.org/quality/reporting.html On Wed, Apr 11, 2012 at 3:02 PM, Rudi Sherry rshe...@adobe.com wrote: Yesterday I updated to the latest webkit sources for the mac (`Tools\Scripts update-webkit`) and rebuilt. Now, when I run Safari using them (`Tools\Scripts\run-safari --debug`) and open a file for my plug-in (which works fine), then go Back/Forward QUICKLY, NPP_NewStream() is never called for Forward. If I go Back/Forward SLOWLY, then it gets called for Forward. In between Back and Forward, the NP_Shutdown() .. NP_Initialize, NPP_New is called in both cases, and I've verified they're all returning no error (or valid values). The release of Safari I have (5.1.5) does not exhibit this behavior (I'm on Lion). I don't see anything in the bug database about this; before I enter a bug, is this known? Thanks, Rudi ___ webkit-dev mailing list webkit-dev@lists.webkit.org http://lists.webkit.org/mailman/listinfo.cgi/webkit-dev ___ webkit-dev mailing list webkit-dev@lists.webkit.org http://lists.webkit.org/mailman/listinfo.cgi/webkit-dev
Re: [webkit-dev] Process for making changes that affect layout test results
Is the best way to do that to use run_webkit_tests to generate a new baseline or does Tools/Scripts/webkitpy/tool/commands/rebaseline.py support pulling new baselines from the trybots? Thanks, Tony On Wed, Apr 11, 2012 at 2:13 PM, Ojan Vafai o...@chromium.org wrote: Typically, if you're working on Chromium Linux or Win, you'd include the new expected results for that platform in your initial commit/code-review as well. On Wed, Apr 11, 2012 at 2:09 PM, Tony Payne tpa...@chromium.org wrote: All code I'm changing is inside of #if PLATFORM(CHROMIUM) blocks. Thanks for the quick answer. Tony On Wed, Apr 11, 2012 at 2:03 PM, Ryosuke Niwa rn...@webkit.org wrote: On Wed, Apr 11, 2012 at 1:57 PM, Tony Payne tpa...@chromium.org wrote: Given the recent discussion on test_expectations.txt, perhaps the answer to my question is still up in the air. I'm working on a change that I expect to require changing the expectations for about 75 tests on chromium win and linux. https://trac.webkit.org/wiki/Rebaseline seems to only cover the gardening work to rebaseline after the commit. I cannot find any wiki pages that describe what the original author is expected to do when making visual changes. Should I attempt to rebaseline manually? Should I mark the tests as failing? Should I just check in and let the bots go red? Just land the patch and rebaseline the tests. Please also coordinate with Chromium port's WebKit gardener when landing this patch. Also, does this patch only affect Chromium Windows and Linux, and not GTK, Qt, Windows, etc...? If the answer is no, and will affect other non-Chromium ports, then you're also responsible for rebaselining or coordinating with other ports to make sure you don't break tests on their ports as well. - Ryosuke ___ webkit-dev mailing list webkit-dev@lists.webkit.org http://lists.webkit.org/mailman/listinfo.cgi/webkit-dev ___ webkit-dev mailing list webkit-dev@lists.webkit.org http://lists.webkit.org/mailman/listinfo.cgi/webkit-dev
[webkit-dev] importing test suites (was: CSS 2.1 Test Suite)
On Wed, Apr 11, 2012 at 12:31 PM, Robert Hogan li...@roberthogan.net wrote: On Wednesday 11 April 2012 20:27:23 Dirk Pranke wrote: What does clean up the existing folder entail? Just move any -expected.html file out of there and generate pixel results. Or move the test and its -expected.html into a folder in fast/css. Okay, following a conversation on #irc, I think I understand this better. There do appear to be differences of opinion over what to do here, so I will try to describe this from ground zero. The CSS test suite has ~9000 tests. the correctness of which is established manually (and visually). We would like to import these tests and run them automatically. Some small number of tests (a few hundred?) do have reference html, but most don't. Since most tests do not come with reference html or expected PNGs, we have to add one or the other. Currently we have added ~400 or so to the tree, and when we add them we have been writing -expected.html reference results to avoid generating pixel results where possible. This raises at least two questions, which have been confusing the discussion until now: 1) What is the best way to add these tests to the tree? Should we add them one-at-a-time with -expected files that we have created? Should we add them all at once and SKIP the tests until we have generated -expected results for each test? Or should we only import subsets that have official -expected files? 2) Is it acceptable for someone other than the test author to write -expected html references? What bar do we need to have to ensure that the reference file will catch the bug the initial test is intended to catch? Do we (or should we) maintain a similar bar for pixel tests today? Personally, I don't care that much about (1), as long as we can track the provenance of the tests (where they came from) ... I would be happy with us importing tests one at a time and adding our own results, and hopefully feeding them back up stream to the W3C. I care much more about (2): I believe we should not be introducing new tests that only have pixel tests for results unless absolutely necessary - the cost of maintaining pixel test results vastly outweighs the potential benefit we get by having a reference file that won't break the same way the test breaks. I believe it is reasonably for someone other than the test author to write a reference html for a test and get it to be good enough that it will provide more value than a pixel test result, and we should be encouraging that. I believe we're hoping to discuss this more broadly at the contributors meeting, but it might be good to discuss here first. Thoughts? -- Dirk ___ webkit-dev mailing list webkit-dev@lists.webkit.org http://lists.webkit.org/mailman/listinfo.cgi/webkit-dev
Re: [webkit-dev] Process for making changes that affect layout test results
use run-webkit-tests (or attempt to pull the results from the trybots by hand). You cannot pull new baselines from the try bots using rebaseline.py. -- Dirk On Wed, Apr 11, 2012 at 3:26 PM, Tony Payne tpa...@chromium.org wrote: Is the best way to do that to use run_webkit_tests to generate a new baseline or does Tools/Scripts/webkitpy/tool/commands/rebaseline.py support pulling new baselines from the trybots? Thanks, Tony On Wed, Apr 11, 2012 at 2:13 PM, Ojan Vafai o...@chromium.org wrote: Typically, if you're working on Chromium Linux or Win, you'd include the new expected results for that platform in your initial commit/code-review as well. On Wed, Apr 11, 2012 at 2:09 PM, Tony Payne tpa...@chromium.org wrote: All code I'm changing is inside of #if PLATFORM(CHROMIUM) blocks. Thanks for the quick answer. Tony On Wed, Apr 11, 2012 at 2:03 PM, Ryosuke Niwa rn...@webkit.org wrote: On Wed, Apr 11, 2012 at 1:57 PM, Tony Payne tpa...@chromium.org wrote: Given the recent discussion on test_expectations.txt, perhaps the answer to my question is still up in the air. I'm working on a change that I expect to require changing the expectations for about 75 tests on chromium win and linux. https://trac.webkit.org/wiki/Rebaseline seems to only cover the gardening work to rebaseline after the commit. I cannot find any wiki pages that describe what the original author is expected to do when making visual changes. Should I attempt to rebaseline manually? Should I mark the tests as failing? Should I just check in and let the bots go red? Just land the patch and rebaseline the tests. Please also coordinate with Chromium port's WebKit gardener when landing this patch. Also, does this patch only affect Chromium Windows and Linux, and not GTK, Qt, Windows, etc...? If the answer is no, and will affect other non-Chromium ports, then you're also responsible for rebaselining or coordinating with other ports to make sure you don't break tests on their ports as well. - Ryosuke ___ webkit-dev mailing list webkit-dev@lists.webkit.org http://lists.webkit.org/mailman/listinfo.cgi/webkit-dev ___ webkit-dev mailing list webkit-dev@lists.webkit.org http://lists.webkit.org/mailman/listinfo.cgi/webkit-dev ___ webkit-dev mailing list webkit-dev@lists.webkit.org http://lists.webkit.org/mailman/listinfo.cgi/webkit-dev
Re: [webkit-dev] Is converting pixel tests to reftests kosher for imported libraries?
On Fri, Mar 9, 2012 at 2:49 PM, Sam Weinig wei...@apple.com wrote: On Mar 7, 2012, at 4:41 PM, Ojan Vafai wrote: I just did a first pass a greening the Chromium Lion bot: http://trac.webkit.org/changeset/110096. Of these hundreds of tests, ~99% of them are perfect candidates for being reftests (e.g. they contain one line of text and a solid box or two under the text), but most of them are in the CSS imported test suites. Is it kosher to convert them to reftests or should we leave pixel tests from imported test suites alone? If we want to make these ref tests, it probably makes more sense to do that work with the CSS WG, so that they can be part of the standard test suite. Until then, I think we should keep them regular pixel tests. Note that this thread (to resurrect it just-after-easter because it's timely) is directly relevant to the note I just sent out about importing test suites. I, at least, would like some clarification ... if we are importing tests that have no accompanying expected result (and are expected to be inspected manually for correctness), is it acceptable to write reference html for the tests, or do they have to be imported as pixel tests? I personally think it's acceptable, but I understand that there might be a difference of opinion. https://bugs.webkit.org/show_bug.cgi?id=72987 is an example of this. -- Dirk ___ webkit-dev mailing list webkit-dev@lists.webkit.org http://lists.webkit.org/mailman/listinfo.cgi/webkit-dev
Re: [webkit-dev] Is converting pixel tests to reftests kosher for imported libraries?
On Wed, Apr 11, 2012 at 3:46 PM, Dirk Pranke dpra...@chromium.org wrote: On Fri, Mar 9, 2012 at 2:49 PM, Sam Weinig wei...@apple.com wrote: On Mar 7, 2012, at 4:41 PM, Ojan Vafai wrote: I just did a first pass a greening the Chromium Lion bot: http://trac.webkit.org/changeset/110096. Of these hundreds of tests, ~99% of them are perfect candidates for being reftests (e.g. they contain one line of text and a solid box or two under the text), but most of them are in the CSS imported test suites. Is it kosher to convert them to reftests or should we leave pixel tests from imported test suites alone? If we want to make these ref tests, it probably makes more sense to do that work with the CSS WG, so that they can be part of the standard test suite. Until then, I think we should keep them regular pixel tests. Note that this thread (to resurrect it just-after-easter because it's timely) is directly relevant to the note I just sent out about importing test suites. I, at least, would like some clarification ... if we are importing tests that have no accompanying expected result (and are expected to be inspected manually for correctness), is it acceptable to write reference html for the tests, or do they have to be imported as pixel tests? Put differently, we either need to add an -expected html or an -expected png. Does it have to be the latter? I personally think it's acceptable, but I understand that there might be a difference of opinion. https://bugs.webkit.org/show_bug.cgi?id=72987 is an example of this. -- Dirk ___ webkit-dev mailing list webkit-dev@lists.webkit.org http://lists.webkit.org/mailman/listinfo.cgi/webkit-dev
Re: [webkit-dev] importing test suites (was: CSS 2.1 Test Suite)
On Wednesday 11 April 2012 23:29:56 Dirk Pranke wrote: 1) What is the best way to add these tests to the tree? Should we add them one-at-a-time with -expected files that we have created? Should we add them all at once and SKIP the tests until we have generated -expected results for each test? Or should we only import subsets that have official -expected files? Here is a good example of this in action: http://trac.webkit.org/changeset/113076 The tests all come unmodified from the CSS test suite, the -expected.html files were created by me. The CSS test suite does not used the -expected.html semantic. ___ webkit-dev mailing list webkit-dev@lists.webkit.org http://lists.webkit.org/mailman/listinfo.cgi/webkit-dev
Re: [webkit-dev] importing test suites (was: CSS 2.1 Test Suite)
+1 on not introducing new pixel tests and allowing someone other than the test author to create the -expected file. We may also be able to streamline some of this process by implementing some helper scripts. Ultimately, someone will still have to review new files manually, but scripts should be able to speed up the process. Alan Stearns and I plan to present some ideas around W3C and WebKit test integration at the contributors meeting to elicit more feedback and recruit people to support the effort. On 4/11/12 3:29 PM, Dirk Pranke dpra...@chromium.org wrote: On Wed, Apr 11, 2012 at 12:31 PM, Robert Hogan li...@roberthogan.net wrote: On Wednesday 11 April 2012 20:27:23 Dirk Pranke wrote: What does clean up the existing folder entail? Just move any -expected.html file out of there and generate pixel results. Or move the test and its -expected.html into a folder in fast/css. Okay, following a conversation on #irc, I think I understand this better. There do appear to be differences of opinion over what to do here, so I will try to describe this from ground zero. The CSS test suite has ~9000 tests. the correctness of which is established manually (and visually). We would like to import these tests and run them automatically. Some small number of tests (a few hundred?) do have reference html, but most don't. Since most tests do not come with reference html or expected PNGs, we have to add one or the other. Currently we have added ~400 or so to the tree, and when we add them we have been writing -expected.html reference results to avoid generating pixel results where possible. This raises at least two questions, which have been confusing the discussion until now: 1) What is the best way to add these tests to the tree? Should we add them one-at-a-time with -expected files that we have created? Should we add them all at once and SKIP the tests until we have generated -expected results for each test? Or should we only import subsets that have official -expected files? 2) Is it acceptable for someone other than the test author to write -expected html references? What bar do we need to have to ensure that the reference file will catch the bug the initial test is intended to catch? Do we (or should we) maintain a similar bar for pixel tests today? Personally, I don't care that much about (1), as long as we can track the provenance of the tests (where they came from) ... I would be happy with us importing tests one at a time and adding our own results, and hopefully feeding them back up stream to the W3C. I care much more about (2): I believe we should not be introducing new tests that only have pixel tests for results unless absolutely necessary - the cost of maintaining pixel test results vastly outweighs the potential benefit we get by having a reference file that won't break the same way the test breaks. I believe it is reasonably for someone other than the test author to write a reference html for a test and get it to be good enough that it will provide more value than a pixel test result, and we should be encouraging that. I believe we're hoping to discuss this more broadly at the contributors meeting, but it might be good to discuss here first. Thoughts? -- Dirk ___ webkit-dev mailing list webkit-dev@lists.webkit.org http://lists.webkit.org/mailman/listinfo.cgi/webkit-dev ___ webkit-dev mailing list webkit-dev@lists.webkit.org http://lists.webkit.org/mailman/listinfo.cgi/webkit-dev
Re: [webkit-dev] importing test suites (was: CSS 2.1 Test Suite)
On Wed, Apr 11, 2012 at 3:52 PM, Jacob Goldstein jac...@adobe.com wrote: +1 on not introducing new pixel tests and allowing someone other than the test author to create the -expected file. We may also be able to streamline some of this process by implementing some helper scripts. Ultimately, someone will still have to review new files manually, but scripts should be able to speed up the process. -1 on that. As I said on other threads about this topic, determining whether a reference file adequately detect all bugs a test is intended to test is hard, and losing the test coverage at the cost of lowering maintenance cost is not necessary a good thing. Also, adding a reference file would mean that either we're adding -ref.html / -noref.html files or modifying reftest.list. If doing the former, then we can't use this approach in any directory where we use reftest.list at the moment because we explicitly prohibit mixing naming convention and reftest.list. Modifying reftest.list is essentially modifying the test suite, and it seems like there is a consensus that we don't want to do it. - Ryosuke ___ webkit-dev mailing list webkit-dev@lists.webkit.org http://lists.webkit.org/mailman/listinfo.cgi/webkit-dev
Re: [webkit-dev] importing test suites (was: CSS 2.1 Test Suite)
I understand, all valid points. Don't people currently fix tests that are failing and that they didn't author? If so, isn't that similar as generating a new ref file in that you first have to understand what is being tested? There would be the old pixel file to go from, so in many cases this may not be too hard. On the other hand, having the test author create the ref file is definitely preferable. What would we do in the case where a test author is no longer available? From: Ryosuke Niwa rn...@webkit.orgmailto:rn...@webkit.org Date: Wed, 11 Apr 2012 16:00:44 -0700 To: Jacob Goldstein jac...@adobe.commailto:jac...@adobe.com Cc: Dirk Pranke dpra...@chromium.orgmailto:dpra...@chromium.org, Robert Hogan li...@roberthogan.netmailto:li...@roberthogan.net, webkit-dev@lists.webkit.orgmailto:webkit-dev@lists.webkit.org webkit-dev@lists.webkit.orgmailto:webkit-dev@lists.webkit.org Subject: Re: [webkit-dev] importing test suites (was: CSS 2.1 Test Suite) On Wed, Apr 11, 2012 at 3:52 PM, Jacob Goldstein jac...@adobe.commailto:jac...@adobe.com wrote: +1 on not introducing new pixel tests and allowing someone other than the test author to create the -expected file. We may also be able to streamline some of this process by implementing some helper scripts. Ultimately, someone will still have to review new files manually, but scripts should be able to speed up the process. -1 on that. As I said on other threads about this topic, determining whether a reference file adequately detect all bugs a test is intended to test is hard, and losing the test coverage at the cost of lowering maintenance cost is not necessary a good thing. Also, adding a reference file would mean that either we're adding -ref.html / -noref.html files or modifying reftest.list. If doing the former, then we can't use this approach in any directory where we use reftest.list at the moment because we explicitly prohibit mixing naming convention and reftest.list. Modifying reftest.list is essentially modifying the test suite, and it seems like there is a consensus that we don't want to do it. - Ryosuke ___ webkit-dev mailing list webkit-dev@lists.webkit.org http://lists.webkit.org/mailman/listinfo.cgi/webkit-dev
Re: [webkit-dev] importing test suites (was: CSS 2.1 Test Suite)
On Wed, Apr 11, 2012 at 4:00 PM, Ryosuke Niwa rn...@webkit.org wrote: On Wed, Apr 11, 2012 at 3:52 PM, Jacob Goldstein jac...@adobe.com wrote: +1 on not introducing new pixel tests and allowing someone other than the test author to create the -expected file. We may also be able to streamline some of this process by implementing some helper scripts. Ultimately, someone will still have to review new files manually, but scripts should be able to speed up the process. -1 on that. As I said on other threads about this topic, determining whether a reference file adequately detect all bugs a test is intended to test is hard, and losing the test coverage at the cost of lowering maintenance cost is not necessary a good thing. Also, adding a reference file would mean that either we're adding -ref.html / -noref.html files or modifying reftest.list. If doing the former, then we can't use this approach in any directory where we use reftest.list at the moment because we explicitly prohibit mixing naming convention and reftest.list. Modifying reftest.list is essentially modifying the test suite, and it seems like there is a consensus that we don't want to do it. This is a quibble compared to the first paragraph, but the last two paragraphs are merely implementation details (especially if we think we're generating reftest.list from link tags embedded in the tests). I think if we agree that it's okay to add -ref.html / -noref.html files for tests we can revisit what the best way to manage such a process is. I think the initial guideline was established when we thought that an imported test suite would come with all of the needed reference files, and in such a case I agree we should leave it as stock as possible. -- Dirk ___ webkit-dev mailing list webkit-dev@lists.webkit.org http://lists.webkit.org/mailman/listinfo.cgi/webkit-dev
Re: [webkit-dev] importing test suites (was: CSS 2.1 Test Suite)
On Wed, Apr 11, 2012 at 4:05 PM, Jacob Goldstein jac...@adobe.com wrote: Don't people currently fix tests that are failing and that they didn't author? We don't fix imported tests if you meant modifying html files, etc... to fix a broken test file. If so, isn't that similar as generating a new ref file in that you first have to understand what is being tested? There would be the old pixel file to go from, so in many cases this may not be too hard. Yes, but determining whether a pixel result is correct or not is easier than determining whether a reference file is correct AND adequately addresses all intended test purposes. On the other hand, having the test author create the ref file is definitely preferable. What would we do in the case where a test author is no longer available? Someone needs to figure out the intended purpose of the test and make a call. I've done quite few of those for editing tests we have (not imported tests); I've converted them to dumpAsText and dump-as-markup tests. - Ryosuke ___ webkit-dev mailing list webkit-dev@lists.webkit.org http://lists.webkit.org/mailman/listinfo.cgi/webkit-dev
Re: [webkit-dev] importing test suites (was: CSS 2.1 Test Suite)
On Wed, Apr 11, 2012 at 4:15 PM, Ryosuke Niwa rn...@webkit.org wrote: On Wed, Apr 11, 2012 at 4:05 PM, Jacob Goldstein jac...@adobe.com wrote: Don't people currently fix tests that are failing and that they didn't author? We don't fix imported tests if you meant modifying html files, etc... to fix a broken test file. If so, isn't that similar as generating a new ref file in that you first have to understand what is being tested? There would be the old pixel file to go from, so in many cases this may not be too hard. Yes, but determining whether a pixel result is correct or not is easier than determining whether a reference file is correct AND adequately addresses all intended test purposes. It may sometimes be easier to tell if a pixel result is correct or not, but not always: I've stared at plenty of failing pixel tests where there were no visible changes at all. Moreover, I think that requiring a reference file to address all intended test purposes is too high of a bar to allow it to replace a PNG; I'm not even sure if it's a practical goal at all, especially given the limited amount of experience we have with reference tests. I think there is some lower bar of goodness that we can reach in practice through experience. -- Dirk ___ webkit-dev mailing list webkit-dev@lists.webkit.org http://lists.webkit.org/mailman/listinfo.cgi/webkit-dev
Re: [webkit-dev] Is converting pixel tests to reftests kosher for imported libraries?
On Wed, Apr 11, 2012 at 3:48 PM, Dirk Pranke dpra...@chromium.org wrote: On Wed, Apr 11, 2012 at 3:46 PM, Dirk Pranke dpra...@chromium.org wrote: On Fri, Mar 9, 2012 at 2:49 PM, Sam Weinig wei...@apple.com wrote: On Mar 7, 2012, at 4:41 PM, Ojan Vafai wrote: I just did a first pass a greening the Chromium Lion bot: http://trac.webkit.org/changeset/110096. Of these hundreds of tests, ~99% of them are perfect candidates for being reftests (e.g. they contain one line of text and a solid box or two under the text), but most of them are in the CSS imported test suites. Is it kosher to convert them to reftests or should we leave pixel tests from imported test suites alone? If we want to make these ref tests, it probably makes more sense to do that work with the CSS WG, so that they can be part of the standard test suite. Until then, I think we should keep them regular pixel tests. Note that this thread (to resurrect it just-after-easter because it's timely) is directly relevant to the note I just sent out about importing test suites. I, at least, would like some clarification ... if we are importing tests that have no accompanying expected result (and are expected to be inspected manually for correctness), is it acceptable to write reference html for the tests, or do they have to be imported as pixel tests? Put differently, we either need to add an -expected html or an -expected png. Does it have to be the latter? I strongly prefer adding -expected.html files. While there is a chance the reference may not cover all the code paths the original test intended, I believe our overall test coverage will be better with more reftests and fewer pixel tests, not to mention our project-wide sanity from spending less time doing expectations file management. Pixel tests accidentally let bugs through all the time because not all ports run them and because it's often hard to tell if a new result is correct. I agree with the sentiment that we should be upstreaming these to the W3C, but I don't see why we would require upstreaming them first instead of committing them locally and then upstreaming them. If there are people willing to do the work, lets allow them do it either way. I personally think it's acceptable, but I understand that there might be a difference of opinion. https://bugs.webkit.org/show_bug.cgi?id=72987 is an example of this. -- Dirk ___ webkit-dev mailing list webkit-dev@lists.webkit.org http://lists.webkit.org/mailman/listinfo.cgi/webkit-dev
Re: [webkit-dev] PSA: Chromium layout test fallbacks AKA down with graphs, long live trees
This change has been made. I will note, however, that the fallback order for all WebKit ports is still not a tree :( The webkit2 ports will use results in platform/wk2 in addition to their non-wk2 fallback paths :( -- Dirk On Mon, Mar 26, 2012 at 12:00 PM, Ojan Vafai o...@chromium.org wrote: Sometime this week or next, we plan to change the Chromium layout test fallback order. We'll be changing it so all chromium ports fallback to platform/chromium, then to platform/mac. They will never fallback to platform/win or platform/mac-* as they do today. The benefits of this change are that it makes the fallback order much easier to reason about for both human beings and code. I believe it makes the fallback order for all WebKit ports a tree instead of a complex graph. This will have minimal impact on the number of expected result files in the tree since only a few hundred chromium tests currently fallback to platform/win and/or platform/mac-*. Ojan ___ webkit-dev mailing list webkit-dev@lists.webkit.org http://lists.webkit.org/mailman/listinfo.cgi/webkit-dev ___ webkit-dev mailing list webkit-dev@lists.webkit.org http://lists.webkit.org/mailman/listinfo.cgi/webkit-dev
Re: [webkit-dev] Process for making changes that affect layout test results
On Wed, Apr 11, 2012 at 4:56 PM, Tony Payne tpa...@chromium.org wrote: Some of these test result changes turn out to be the effect of incorrect test cases. For example, compositing/color-matching/image-color-matching.html compares an image without a color profile with an image tagged with what it calls a default RGB profile. However, this default RGB profile is not sRGB, so the test is wrongly[1] assuming these images should be displayed the same. I'd like to replace all of the resources containing random RGB profiles with sRGB profiles instead. 1) Is this a good idea? Is there a reason why these image resources have non-standard profiles in them? I don't know the answer to this, but hopefully someone else will. 2) How do I coordinate this, as changing the image resources seems like it will affect all the ports? To the extent you can notify other port maintainers, I'm sure they would appreciate it. Otherwise, treat it the same way you would treat any other change ... It will just be more painful :). -- Dirk Tony [1] http://www.w3.org/Graphics/Color/sRGB.html On Wed, Apr 11, 2012 at 3:36 PM, Dirk Pranke dpra...@chromium.org wrote: use run-webkit-tests (or attempt to pull the results from the trybots by hand). You cannot pull new baselines from the try bots using rebaseline.py. -- Dirk On Wed, Apr 11, 2012 at 3:26 PM, Tony Payne tpa...@chromium.org wrote: Is the best way to do that to use run_webkit_tests to generate a new baseline or does Tools/Scripts/webkitpy/tool/commands/rebaseline.py support pulling new baselines from the trybots? Thanks, Tony On Wed, Apr 11, 2012 at 2:13 PM, Ojan Vafai o...@chromium.org wrote: Typically, if you're working on Chromium Linux or Win, you'd include the new expected results for that platform in your initial commit/code-review as well. On Wed, Apr 11, 2012 at 2:09 PM, Tony Payne tpa...@chromium.org wrote: All code I'm changing is inside of #if PLATFORM(CHROMIUM) blocks. Thanks for the quick answer. Tony On Wed, Apr 11, 2012 at 2:03 PM, Ryosuke Niwa rn...@webkit.org wrote: On Wed, Apr 11, 2012 at 1:57 PM, Tony Payne tpa...@chromium.org wrote: Given the recent discussion on test_expectations.txt, perhaps the answer to my question is still up in the air. I'm working on a change that I expect to require changing the expectations for about 75 tests on chromium win and linux. https://trac.webkit.org/wiki/Rebaseline seems to only cover the gardening work to rebaseline after the commit. I cannot find any wiki pages that describe what the original author is expected to do when making visual changes. Should I attempt to rebaseline manually? Should I mark the tests as failing? Should I just check in and let the bots go red? Just land the patch and rebaseline the tests. Please also coordinate with Chromium port's WebKit gardener when landing this patch. Also, does this patch only affect Chromium Windows and Linux, and not GTK, Qt, Windows, etc...? If the answer is no, and will affect other non-Chromium ports, then you're also responsible for rebaselining or coordinating with other ports to make sure you don't break tests on their ports as well. - Ryosuke ___ webkit-dev mailing list webkit-dev@lists.webkit.org http://lists.webkit.org/mailman/listinfo.cgi/webkit-dev ___ webkit-dev mailing list webkit-dev@lists.webkit.org http://lists.webkit.org/mailman/listinfo.cgi/webkit-dev ___ webkit-dev mailing list webkit-dev@lists.webkit.org http://lists.webkit.org/mailman/listinfo.cgi/webkit-dev
Re: [webkit-dev] Is converting pixel tests to reftests kosher for imported libraries?
On Wed, Apr 11, 2012 at 4:45 PM, Ojan Vafai o...@chromium.org wrote: I agree with the sentiment that we should be upstreaming these to the W3C, but I don't see why we would require upstreaming them first instead of committing them locally and then upstreaming them. How do we know whether a reference file came from W3C repository or not. (Maybe by the fact it's named *-expected.html?) Also, there are directories with reftest.list but without reference files for some tests. The last time I checked, you were opposed to having bot reftest.list and *-expected.html / *-expected-mismatch.html files. Have you changed your opinion on this? - Ryosuke ___ webkit-dev mailing list webkit-dev@lists.webkit.org http://lists.webkit.org/mailman/listinfo.cgi/webkit-dev
