Re: [webkit-dev] run-webkit-tests question; hashes when comparing ref test output
On Thu, Jan 22, 2015 at 1:56 PM, Darin Adler da...@apple.com wrote: On Jan 22, 2015, at 12:58 PM, Dirk Pranke dpra...@chromium.org wrote: Of course, Chromium was more willing to incur the cost of keeping all of the pixel tests up to date when trivial things changed the output This comment, and much of the discussion, seems to be about pixel tests. But what does this have to do with reference tests? The message has nothing to do with whether the test is a pixel test or a ref test. My reply was intended to explain why we had the hashes didn't match but diff passed message, and the discrepancy around fuzzy matching; on the Chromium ports, if run-webkit-tests had printed this message, it would've served as a warning to the user/developer that there was something weird going on. If you all decide you wanted to keep fuzzy matching enabled (and hence that message would happen regularly and could be safely ignored), I would either delete the message outright or at least make it be a debug message. You could also decide you wanted to do fuzzy matching only on ref tests, or only on pixel tests, and change the logic accordingly, of course. -- Dirk ___ webkit-dev mailing list webkit-dev@lists.webkit.org https://lists.webkit.org/mailman/listinfo/webkit-dev
Re: [webkit-dev] Any old-run-webkit-tests clients left?
iOS was the only one I was aware of that was left. -- Dirk On Mon, Jul 28, 2014 at 12:25 PM, David Farler dfar...@apple.com wrote: Hi, Are there any ports or users of old-run-webkit-tests left besides the iOS Simulator? I'm on the verge of completing iOS Simulator support for webkitpy and run-webkit-tests so, once the new port is stable, I'd like to consider removing this code. David ___ webkit-dev mailing list webkit-dev@lists.webkit.org https://lists.webkit.org/mailman/listinfo/webkit-dev ___ webkit-dev mailing list webkit-dev@lists.webkit.org https://lists.webkit.org/mailman/listinfo/webkit-dev
Re: [webkit-dev] Anyone else having these problems with run-webkit-tests?
On Sun, Mar 23, 2014 at 12:52 PM, Benjamin Poulain benja...@webkit.orgwrote: Same here :( On 3/23/14, 12:15 PM, Darin Adler wrote: When I use run-webkit-tests to run the entire test suite, on a debug build of TOT WebKit, on Mavericks, I’m having the following problems: - Towards the end of the run, the tests run slowly, over a second per test on my 3.5GHz i7 iMac with tons of memory, which is about 10X too slow I think. The sample shows the vast majority of the time is spent in JavaScript garbage collection. This is running mostly the svg/custom tests. Yep. The time from 31k to 33k is a long as from 0 to 31k :( - Towards the end of the run, instead of the 8 parallel copies of DumpRenderTree, only 1 copy of DumpRenderTree seems to run. This is running mostly the svg/custom tests. I think the problem is we can only run DumpRenderTree in parallel on different folders. Toward the end, everything left is one or two slow folders. The problem is partially that the sharding is folder-at-a-time, partially that the svg folder is big and slow, and partially that svg comes near the end of the alphabet (i.e., we don't start the big slow directory until close to the end of run, so there ends up being one big long pole). At some point we added code to run-webkit-tests to have a list of slow directories that got started towards the beginning. I don't remember if we did that before or after the Blink fork, but it would be easy to port it over from Blink if it was after. There is also the --experimental-fully-parallel flag that you can use, which shards a test at a time and get you the full parallelization until the last 8 tests. The downside to this is that you'll probably have a lot more flakiness. Hope this is useful, -- Dirk ___ webkit-dev mailing list webkit-dev@lists.webkit.org https://lists.webkit.org/mailman/listinfo/webkit-dev
Re: [webkit-dev] Anyone else having these problems with run-webkit-tests?
On Mon, Mar 24, 2014 at 3:32 PM, Bem Jones-Bey bjone...@adobe.com wrote: On Mar 24, 2014, at 14:54 , Dirk Pranke dpra...@chromium.org wrote: On Sun, Mar 23, 2014 at 12:52 PM, Benjamin Poulain benja...@webkit.orgwrote: Same here :( On 3/23/14, 12:15 PM, Darin Adler wrote: When I use run-webkit-tests to run the entire test suite, on a debug build of TOT WebKit, on Mavericks, I’m having the following problems: - Towards the end of the run, the tests run slowly, over a second per test on my 3.5GHz i7 iMac with tons of memory, which is about 10X too slow I think. The sample shows the vast majority of the time is spent in JavaScript garbage collection. This is running mostly the svg/custom tests. Yep. The time from 31k to 33k is a long as from 0 to 31k :( - Towards the end of the run, instead of the 8 parallel copies of DumpRenderTree, only 1 copy of DumpRenderTree seems to run. This is running mostly the svg/custom tests. I think the problem is we can only run DumpRenderTree in parallel on different folders. Toward the end, everything left is one or two slow folders. The problem is partially that the sharding is folder-at-a-time, partially that the svg folder is big and slow, and partially that svg comes near the end of the alphabet (i.e., we don't start the big slow directory until close to the end of run, so there ends up being one big long pole). At some point we added code to run-webkit-tests to have a list of slow directories that got started towards the beginning. I don't remember if we did that before or after the Blink fork, but it would be easy to port it over from Blink if it was after. I'm pretty sure this was after the Blink fork. I did a quick look through the history, and I found this: http://src.chromium.org/viewvc/blink?revision=152697view=revision Is that the correct change? If it will make tests run faster in WebKit land as well, I'm more than happy to port it myself. :-) Yup, that's the one. Obviously the whole virtual test suite thing is less relevant, but you can use it for any directory you want. -- Dirk ___ webkit-dev mailing list webkit-dev@lists.webkit.org https://lists.webkit.org/mailman/listinfo/webkit-dev
Re: [webkit-dev] Importing W3C tests for HTML template elements
Ryosuke and I discussed this a bit over IRC. Ryosuke's main concern was that supporting multiple document roots adds a fair amount of complexity to NRWT. Conceptually, it's probably easier to add support to run the W3C's new server (known as wptserve) and then maybe use it for *all* imported tests from the W3C. If it turns out that wptserve is too slow, and you would prefer to run only some of the directories over http (i.e., the 10k+ CSS tests don't need http), you'll probably need to modify how wptserve is run (and NRWT) as well, anyway, so the patch would be different. Ryosuke, please let me know if I've misstated your thinking at all. -- Dirk On Mon, Jan 6, 2014 at 11:23 AM, Ryosuke Niwa rn...@webkit.org wrote: I don't think we should do this given that the python server has been added to W3C testing harness, and they're gonna convert all existing tests to use that instead: http://lists.w3.org/Archives/Public/public-test-infra/2014JanMar/.html We should simply wait for that effort to take place and add a support for the python server instead. - R. Niwa On Fri, Dec 6, 2013 at 12:25 AM, youenn fablet youe...@gmail.com wrote: As long as the newly imported tests use relative URLs, alias may be used as a workaround. I will give it a try. Bug entry is at https://bugs.webkit.org/show_bug.cgi?id=125339 Any further help appreciated, Youenn 2013/12/6 Darin Adler da...@apple.com If that's really ends up being super hard we can always put yet another third-party or imported directory inside the http directory as previously suggested. it's annoying to have three different places for imported tests and code, but not something I want to hold us up for a long time. -- Darin Sent from my iPhone On Dec 5, 2013, at 5:51 PM, Ryosuke Niwa rn...@webkit.org wrote: On Wed, Dec 4, 2013 at 11:19 AM, Darin Adler da...@apple.com wrote: On Dec 4, 2013, at 6:48 AM, youenn fablet youe...@gmail.com wrote: I am planning to add some XHR tests from https://github.com/w3c/web-platform-tests. My initial plan was to add them in a subdirectory of LayoutTests/http/tests/w3c. If adding them into LayoutTests/imported/w3c, that would probably require updating the test scripts to start/stop the HTTP test server for that particular sub-folder. Any preference? I’d prefer LayoutTests/imported/w3c. Although I’m not so happy about the different terminology we are using for “imported” vs. “ThirdParty”, which seems like the same concept at the top level of the directory structure. One trickiness to it is that we don't currently run any HTTP test in parallel and the document root of the HTTP server is set to LayoutTests/http/tests so we might need to modify that or restart the HTTP server whenever we're running HTTP tests outside of LayoutTests/http. - R. Niwa ___ webkit-dev mailing list webkit-dev@lists.webkit.org https://lists.webkit.org/mailman/listinfo/webkit-dev ___ webkit-dev mailing list webkit-dev@lists.webkit.org https://lists.webkit.org/mailman/listinfo/webkit-dev
Re: [webkit-dev] Importing W3C tests for HTML template elements
The way I got around this when I was first working on it was to simply map imported/w3c onto a subdirectory of the document root in apache; it's a two line change. For some time I've toyed with the idea of changing the DocumentRoot to just be LayoutTests/, so that any test could be run over http directly. A lot of tests and test results would need to be updated for this, but I think it could simplify the test organization a fair amount and open up some interesting possibilities. -- Dirk On Thu, Dec 5, 2013 at 6:25 PM, Darin Adler da...@apple.com wrote: If that's really ends up being super hard we can always put yet another third-party or imported directory inside the http directory as previously suggested. it's annoying to have three different places for imported tests and code, but not something I want to hold us up for a long time. -- Darin Sent from my iPhone On Dec 5, 2013, at 5:51 PM, Ryosuke Niwa rn...@webkit.org wrote: On Wed, Dec 4, 2013 at 11:19 AM, Darin Adler da...@apple.com wrote: On Dec 4, 2013, at 6:48 AM, youenn fablet youe...@gmail.com wrote: I am planning to add some XHR tests from https://github.com/w3c/web-platform-tests. My initial plan was to add them in a subdirectory of LayoutTests/http/tests/w3c. If adding them into LayoutTests/imported/w3c, that would probably require updating the test scripts to start/stop the HTTP test server for that particular sub-folder. Any preference? I’d prefer LayoutTests/imported/w3c. Although I’m not so happy about the different terminology we are using for “imported” vs. “ThirdParty”, which seems like the same concept at the top level of the directory structure. One trickiness to it is that we don't currently run any HTTP test in parallel and the document root of the HTTP server is set to LayoutTests/http/tests so we might need to modify that or restart the HTTP server whenever we're running HTTP tests outside of LayoutTests/http. - R. Niwa ___ webkit-dev mailing list webkit-dev@lists.webkit.org https://lists.webkit.org/mailman/listinfo/webkit-dev ___ webkit-dev mailing list webkit-dev@lists.webkit.org https://lists.webkit.org/mailman/listinfo/webkit-dev
Re: [webkit-dev] Importing W3C tests for HTML template elements
On Wed, Dec 4, 2013 at 11:19 AM, Darin Adler da...@apple.com wrote: On Dec 4, 2013, at 6:48 AM, youenn fablet youe...@gmail.com wrote: I am planning to add some XHR tests from https://github.com/w3c/web-platform-tests. My initial plan was to add them in a subdirectory of LayoutTests/http/tests/w3c. If adding them into LayoutTests/imported/w3c, that would probably require updating the test scripts to start/stop the HTTP test server for that particular sub-folder. Any preference? I’d prefer LayoutTests/imported/w3c. Although I’m not so happy about the different terminology we are using for “imported” vs. “ThirdParty”, which seems like the same concept at the top level of the directory structure. Might be nice to think about a more elegant way to trigger the HTTP server when needed. Even requiring some kind of trigger file in each directory would be better than the hard-coded path rule we use now. I'd prefer/recommend LayoutTests/imported/w3c as well (and just modify the test scripts to deal w/ the additional http servers). As you start to pull in more of the w3c's tests, there will probably need to be some overall cleanup and refactoring of how stuff like this works and is shared (e.g., duplication of files under resources/ , paths mapped in the http server, etc.). Blink has the same issues. I'd be happy to help out with this where possible to try and keep things relatively consistent. -- Dirk ___ webkit-dev mailing list webkit-dev@lists.webkit.org https://lists.webkit.org/mailman/listinfo/webkit-dev
Re: [webkit-dev] Proposed feature: Network Service Discovery
On Fri, Aug 30, 2013 at 10:06 AM, Oliver Hunt oli...@apple.com wrote: On Aug 30, 2013, at 9:15 AM, Brendan Long s...@brendanlong.com wrote: On 08/29/2013 05:45 PM, Benjamin Poulain wrote: Can you explain a bit what it is for? What are the common use cases? This would be useful for certain kinds of web apps. For example,a music website like Pandora or Spotify could allow users to include music on their local network. Or a service like Netflix could include local network movies (on networked hard drives, or DVR's) in their search results, and play them from the same interface. Here's my concern - if you say a service like x might want to search for something, that is better described as a random website. That may be something the user wants, alternatively it could be something evil. It could also be something evil embedded in an ad on the site a user trusts. My concern here is that as a web spec this essentially acts as a way for arbitrary web content from any source to perform a network scan of your local machine and get data about your internal network topology and services from inside your firewall. That's a really scary concept to me. While there are certainly security concerns that need to be clearly thought through and addressed, the spec isn't as broad as you make it sound. It picks up services that are advertising themselves, after all; you can't probe. (Unless you've noticed something in the spec I haven't; I've scanned the spec, but not read it super-carefully). Another use case for this is for devices like AppleTVs and ChromeCast ... receivers advertise themselves on the local network, and the browser (and browser-based apps) can identify available receivers that you can send media to. The draft does contain the sentence Web pages should not be able to communicate with Local-networked Services that have not been authorized by the user thereby maintaining the user's privacy in the use cases section; this should definite be emphasized and fleshed out, in a security section. -- Dirk ___ webkit-dev mailing list webkit-dev@lists.webkit.org https://lists.webkit.org/mailman/listinfo/webkit-dev
Re: [webkit-dev] Proposed feature: Network Service Discovery
On Fri, Aug 30, 2013 at 3:48 PM, Oliver Hunt oli...@apple.com wrote: On Aug 30, 2013, at 12:44 PM, Dirk Pranke dpra...@chromium.org wrote: On Fri, Aug 30, 2013 at 10:06 AM, Oliver Hunt oli...@apple.com wrote: On Aug 30, 2013, at 9:15 AM, Brendan Long s...@brendanlong.com wrote: On 08/29/2013 05:45 PM, Benjamin Poulain wrote: Can you explain a bit what it is for? What are the common use cases? This would be useful for certain kinds of web apps. For example,a music website like Pandora or Spotify could allow users to include music on their local network. Or a service like Netflix could include local network movies (on networked hard drives, or DVR's) in their search results, and play them from the same interface. Here's my concern - if you say a service like x might want to search for something, that is better described as a random website. That may be something the user wants, alternatively it could be something evil. It could also be something evil embedded in an ad on the site a user trusts. My concern here is that as a web spec this essentially acts as a way for arbitrary web content from any source to perform a network scan of your local machine and get data about your internal network topology and services from inside your firewall. That's a really scary concept to me. While there are certainly security concerns that need to be clearly thought through and addressed, the spec isn't as broad as you make it sound. It picks up services that are advertising themselves, after all; you can't probe. (Unless you've noticed something in the spec I haven't; I've scanned the spec, but not read it super-carefully). Define advertise? Bonjour like? UPnP? Yes (the spec explicitly lists zeroconf, upnp, and dial). The draft does contain the sentence Web pages should not be able to communicate with Local-networked Services that have not been authorized by the user thereby maintaining the user's privacy in the use cases section; this should definite be emphasized and fleshed out, in a security section. How does the user know what they're doing? If there's an ad/unescaped comment containing something malicious should a remote site be able to know what services you have in your internal network? I'm not sure I understand your question, but I'm talking about the user having to opt-in to disclosing services, similar to the opt-ins we do for geolocation, media capture, local files, etc., e.g., Spotify would like to know if you have any local media receivers, etc. ... -- Dirk ___ webkit-dev mailing list webkit-dev@lists.webkit.org https://lists.webkit.org/mailman/listinfo/webkit-dev
Re: [webkit-dev] Adding new entries to TestExpectations can be harmful
I may be missing context, or just slow, but how does that link illustrate things being harmful ? It looks like you're mostly adding the bug numbers inline and/or taking skipped tests and making them flaky ? -- Dirk On Wed, May 22, 2013 at 6:11 PM, Ryosuke Niwa rn...@webkit.org wrote: And there is an overwhelming evidence: http://trac.webkit.org/changeset?reponame=new=150559%40trunk%2FLayoutTests%2Fplatform%2Fmac%2FTestExpectationsold=150476%40trunk%2FLayoutTests%2Fplatform%2Fmac%2FTestExpectations - R. Niwa ___ webkit-dev mailing list webkit-dev@lists.webkit.org https://lists.webkit.org/mailman/listinfo/webkit-dev ___ webkit-dev mailing list webkit-dev@lists.webkit.org https://lists.webkit.org/mailman/listinfo/webkit-dev
Re: [webkit-dev] obtaining webkit.org address?
It seems like you could get nearly both things if you just had people list both their @webkit.org and @company.com addresses (or at least a company affiliation, if not an actual address) in committers.py, right? -- Dirk On Mon, Apr 29, 2013 at 10:26 AM, Brian Weinstein bweinst...@apple.comwrote: What Simon said is correct. We would prefer that contributors use their company affiliated email for committing. However, if there is a need for a webkit.org email (some people don't have access to their work email when not in the office, for example), please email me with your request for an @webkit.org email with the username you would like and where you would like it to be forwarded. A further note is, if you are given a @webkit.org email and have a company affiliation, the company affiliated email should be used for committing, the @webkit.org email is meant for mailing lists. Thanks! Brian On Apr 29, 2013, at 10:17 AM, Simon Fraser simon.fra...@apple.com wrote: I don't think WebKit has a strict policy on this. I would actually prefer that we phase out webkit.org email addresses. I like to be able to determine what someone's affiliation is. Simon On Apr 28, 2013, at 9:10 PM, Glenn Adams gl...@skynav.com wrote: On Sun, Apr 28, 2013 at 8:46 PM, Glenn Adams gl...@skynav.com wrote: And if one prefers to use a webkit.org address, like you are doing? A little follow-up: when I got my SVN account and credentials earlier this year as a committer, I expected to be given or at least asked if I wanted a webkit.org address, to which I would have said yes. However, I wasn't asked and the credentials went through with my company address. As my company is basically just me, I would prefer to use a webkit.org address in order to identify better with the community as such. So I'm just now following up on this inquiry about how to get a community address. On Sun, Apr 28, 2013 at 11:55 AM, Antonio Gomes toniki...@webkit.org wrote: Previously people used to get SVN accounts associated to a @webkit.org alias. Today it seems like it is preferable to get a company email as alias, and credential to write-access SVN. On Sunday, April 28, 2013, Glenn Adams wrote: How does a committer/reviewer obtain a webkit.org address? I notice that the majority of existing committers and reviewers have either a webkit.org or a chromium.org address listed in contributors.json. I think it an important part of being part of the WK community to be able to identify oneself as being in that community, and having a usable webkit.org address seems a good way to effect that. G. ___ webkit-dev mailing list webkit-dev@lists.webkit.org https://lists.webkit.org/mailman/listinfo/webkit-dev ___ webkit-dev mailing list webkit-dev@lists.webkit.org https://lists.webkit.org/mailman/listinfo/webkit-dev ___ webkit-dev mailing list webkit-dev@lists.webkit.org https://lists.webkit.org/mailman/listinfo/webkit-dev
Re: [webkit-dev] Test frame size for reftests
On Wed, Apr 24, 2013 at 10:30 AM, L. David Baron dba...@dbaron.org wrote: On Wednesday 2013-04-24 09:55 -0700, Darin Adler wrote: On Apr 23, 2013, at 11:36 AM, Benjamin Poulain benja...@webkit.org wrote: The current mode works fine for some ref-tests but is really annoying for others. I would love to have a way to compare the entire page instead of the viewport. Maybe it could just be an API on testrunner? Like: testRunner.dumpEntirePagePixels()? What’s the downside of just doing this for all reftests? Would it make them slower? Would it cause failures? I’d love it if we could just do it without it having to be opt-in. Tests that rely on this wouldn't be sharable with other browsers or usable in W3C test suites, at least based on current practice. (I think we had some sort of agreement within some set of people involved with W3C reftests to use 800x600. Mozilla is currently using 800x1000, though I'd like to switch to the smaller size.) I think this raises two related questions, both of which would be helped by concrete examples: 1) What sort of test are you writing where 800x600 isn't big enough to test what you need to test? 2) What sort of test are you writing where there's needs to be something on the page not included in the test *and* will render differently in different browsers? I can posit the existence of both sorts of things, but I'm not sure when they arise in practice? -- Dirk ___ webkit-dev mailing list webkit-dev@lists.webkit.org https://lists.webkit.org/mailman/listinfo/webkit-dev
Re: [webkit-dev] Test frame size for reftests
On Wed, Apr 24, 2013 at 3:29 PM, Elliott Sprehn espr...@chromium.orgwrote: On Wed, Apr 24, 2013 at 11:11 AM, Dirk Pranke dpra...@chromium.orgwrote: ... I think this raises two related questions, both of which would be helped by concrete examples: 1) What sort of test are you writing where 800x600 isn't big enough to test what you need to test? 2) What sort of test are you writing where there's needs to be something on the page not included in the test *and* will render differently in different browsers? I can posit the existence of both sorts of things, but I'm not sure when they arise in practice? I've seen tests before that have lots of form controls where a bunch were outside the viewport. Sure, you *can* write such tests. Did you *have to*? -- Dirk ___ webkit-dev mailing list webkit-dev@lists.webkit.org https://lists.webkit.org/mailman/listinfo/webkit-dev
Re: [webkit-dev] Test frame size for reftests
On Wed, Apr 24, 2013 at 3:35 PM, Elliott Sprehn espr...@chromium.orgwrote: On Wed, Apr 24, 2013 at 3:31 PM, Dirk Pranke dpra...@chromium.org wrote: On Wed, Apr 24, 2013 at 3:29 PM, Elliott Sprehn espr...@chromium.orgwrote: On Wed, Apr 24, 2013 at 11:11 AM, Dirk Pranke dpra...@chromium.orgwrote: ... I think this raises two related questions, both of which would be helped by concrete examples: 1) What sort of test are you writing where 800x600 isn't big enough to test what you need to test? 2) What sort of test are you writing where there's needs to be something on the page not included in the test *and* will render differently in different browsers? I can posit the existence of both sorts of things, but I'm not sure when they arise in practice? I've seen tests before that have lots of form controls where a bunch were outside the viewport. Sure, you *can* write such tests. Did you *have to*? Certainly not, I just know such tests already exist. Right, so in the absence of tests that *need* bigger viewports, I'd side w/ David and say that we should fix tests to fit within 800x600 so that they are portable to other browsers and meet the W3C's criteria. Looking at the entire page would be a bad idea, as it would let us tolerate non-conformant tests. -- Dirk ___ webkit-dev mailing list webkit-dev@lists.webkit.org https://lists.webkit.org/mailman/listinfo/webkit-dev
Re: [webkit-dev] Test frame size for reftests
On Wed, Apr 24, 2013 at 3:49 PM, Darin Adler da...@apple.com wrote: On Apr 24, 2013, at 3:45 PM, Dirk Pranke dpra...@chromium.org wrote: in the absence of tests that *need* bigger viewports, I'd side w/ David and say that we should fix tests to fit within 800x600 so that they are portable to other browsers and meet the W3C's criteria. Makes sense. I believe the 800x600 viewport makes it easy to accidentally write incorrect ref tests that look good when you read the HTML in the test file and the expected file, but are no good. It would be nice to find a simple way to spot flawed ref tests easily so even an inexperienced beginner will notice such mistakes. I'm not sure I'm understanding you here. You mean it's easy to write a test that is bigger than 800x600, and perhaps we should generate some sort of warning for this? -- Dirk ___ webkit-dev mailing list webkit-dev@lists.webkit.org https://lists.webkit.org/mailman/listinfo/webkit-dev
Re: [webkit-dev] Test frame size for reftests
On Wed, Apr 24, 2013 at 4:00 PM, Darin Adler da...@apple.com wrote: On Apr 24, 2013, at 3:52 PM, Dirk Pranke dpra...@chromium.org wrote: You mean it's easy to write a test that is bigger than 800x600, and perhaps we should generate some sort of warning for this? It’s easy to write a test where part of what we’re trying to test is outside 800x600, right. And it would be good if this mistake was easy to notice. Yeah, seems like we could do either or both of modify DRT/WTR to draw a big border at 800x600 and log errors to stdout/stderr if the page wanted to be overflow that. -- Dirk ___ webkit-dev mailing list webkit-dev@lists.webkit.org https://lists.webkit.org/mailman/listinfo/webkit-dev
Re: [webkit-dev] Compiling WebKit up to 25% faster in Windows?
On Tue, Apr 16, 2013 at 3:58 PM, Brent Fulgham bfulg...@gmail.com wrote: Hi Daniel, I'm afraid I don't quite understand the nature of the change you are proposing: 1. Is it sufficient to supply the full path to the include files (e.g., change Foo.h to WebCore/html/Foo.h) to achieve these gains? 2. ... or ... is it sufficient to copy all header files to a common include target path (e.g., $(BuildDir)/include) to achieve these gains? I understand that in either case, we would want to remove the various compiler include path directives so that there are fewer places for it to search. I believe he's proposing the former -- (1) -- along with the necessary changes to all the code generation scripts to do the right thing. -- Dirk ___ webkit-dev mailing list webkit-dev@lists.webkit.org https://lists.webkit.org/mailman/listinfo/webkit-dev
[webkit-dev] Layout Tests and the W3C
Hi all, Those of you who are subscribed to blink-dev@ will see that I just sent out a note entitled Blink, Testing and the W3C, describing stuff I'm working on to get the W3C tests running regularly as part of the layout tests. The W3C has gotten to the point where they have many thousands of ref tests, a lot of which are newer versions of old test suites that we imported once upon a time (meaning that we can replace a lot of pixel tests with ref tests). That work leverages a bunch of work Rebecca Hauck has been working on that actually started pre-Blink-fork, and in fact currently only exists as a patch posted to bugs.webkit.org ( https://bugs.webkit.org/show_bug.cgi?id=111513 ). I would like to get something resembling the same process working for both WebKit and Blink, since we use basically the same tools. There will probably be some differences, of course, e.g., in how we mirror repos and track issues, but I would hope we'd all agree the more tests we can share (both between WebKit and Blink and with other vendors like FF and IE) the happier everyone will be, right? We haven't really worked out the processes for how this'll work yet; Rebecca and I were starting to work on it when the fork happened. Assuming there is still interest, I'll start sketching out how things might work for WebKit and report back. Also, if we actually start to make progress on this, I'd be happy to talk about it at the WebKit committer's meeting if there's interest there as well. Thoughts? -- Dirk ___ webkit-dev mailing list webkit-dev@lists.webkit.org https://lists.webkit.org/mailman/listinfo/webkit-dev
Re: [webkit-dev] Parallel JavaScript: Why a separate ParallelArray types
On Fri, Apr 12, 2013 at 10:44 PM, Filip Pizlo fpi...@apple.com wrote: On Apr 12, 2013, at 10:43 PM, Ryosuke Niwa rn...@webkit.org wrote: Perhaps we can come up with some JS API for shared memory lock and propose it in TC39 or WebApps WG? I think that would be wonderful. Anyone else interested? Feel free to write something up, but as I said before, I suspect any such proposal is DOA in either group. If you wanted to get something adopted here, I think you would have to build something, ship it, and then have it become so successful that the standards groups would be forced to reconsider their positions. -- Dirk ___ webkit-dev mailing list webkit-dev@lists.webkit.org https://lists.webkit.org/mailman/listinfo/webkit-dev
Re: [webkit-dev] Parallel JavaScript: Why a separate ParallelArray types
On Fri, Apr 12, 2013 at 1:50 PM, Filip Pizlo fpi...@apple.com wrote: I'm curious: would you want to use ParallelArray, if you had the flexibility of building a different abstraction? I find ParallelArray to be an awkward abstraction to begin with. I like work queues and such. The whole idea that the only parallelism you get is from arrays feels narrow-minded. I think the needs of the HPC / GPU-accelerating crowd are quite different from the people looking for general concurrency solutions to the multi-core problem. For what it's worth, NRWT is (IMO) an interesting example of a highly-parallel program. In a sense, the problem is perfectly suited to map/reduce, and the performance of NRWT scales linearly well up to (at least) 48-core machines; the necessarily-serial part of the program is quite small. However, in order to turn it into a usable program (where you can actually tell what's going on and manage it well), you do need to actually monitor tasks and queues and provide problem-specific feedback to the user. I'm not sure what the message of that is, exactly ... -- Dirk ___ webkit-dev mailing list webkit-dev@lists.webkit.org https://lists.webkit.org/mailman/listinfo/webkit-dev
Re: [webkit-dev] Rewriting binding code generator, maybe?
On Thu, Apr 11, 2013 at 9:51 AM, Darin Adler da...@apple.com wrote: On Apr 11, 2013, at 5:02 AM, Thiago Marcos P. Santos tmpsan...@gmail.com wrote: Would be great if both projects could share the same generators. The part that would be OK to share would be the parsing. The actual code generation needs to change all the time as the internal architecture of WebKit and of the JavaScript engine and other things we’re binding change, so we should not try to change that. I'm not sure how useful a suggestion this is, but ANTLR has a pretty strong framework for separating parsing from code generation and seems like it would be an ideal fit for this, except that the tool itself is written in Java. Perhaps that is a show-stopper? I'm not actually aware of any similar parser-generator tools that have similar architectures but are in more WebKit/Blink-friendly languages, but maybe they exist? It certainly would be nice not to have to roll your own framework here. -- Dirk ___ webkit-dev mailing list webkit-dev@lists.webkit.org https://lists.webkit.org/mailman/listinfo/webkit-dev
Re: [webkit-dev] Rewriting binding code generator, maybe?
On Thu, Apr 11, 2013 at 10:29 AM, Leandro Pereira lean...@profusion.mobiwrote: dpranke, On 04/11/2013 02:12 PM, Dirk Pranke wrote: I'm not sure how useful a suggestion this is, but ANTLR has a pretty strong framework for separating parsing from code generation and seems like it would be an ideal fit for this, except that the tool itself is written in Java. Perhaps that is a show-stopper? I'm not actually aware of any similar parser-generator tools that have similar architectures but are in more WebKit/Blink-friendly languages, but maybe they exist? Pyparsing is a nice little library that requires no code generation (grammar is written in pure Python) and is very simple to use. Their website lives at http://pyparsing.wikispaces.**com/http://pyparsing.wikispaces.com/ . That's pretty much the opposite of what I was getting at (I was talking about ANTLR's use of StringTemplate to separate code generation from the building of the AST). -- Dirk ___ webkit-dev mailing list webkit-dev@lists.webkit.org https://lists.webkit.org/mailman/listinfo/webkit-dev
Re: [webkit-dev] Is there a plan for supporting multi-process and WebCL in webkit
On Wed, Apr 10, 2013 at 1:17 PM, Filip Pizlo fpi...@apple.com wrote: I.e. if you believe that (A) is not a solvable problem, then this constitutes an argument against ParallelArrays, and not against inferring that a normal array should behave like a ParallelArray. I believe that there are a class of solvable problems for soundness, but it's hard if not impossible to do it generally; that was my main concern. There are certainly languages that make this problem easier, for example (Haskell, J). I agree that the JIT+runtime bailout strategy you discuss gives you more options here. - Adding hints to the language results in programming models that nobody uses. Have you ever used OpenMP? I'm familiar enough with it. I was referring to things more like restrict, const, and pure which give you (or the compiler) more guarantees about what it is safe to infer about the parameters and functions. I'm not sure these are the same sorts of things you don't like. Hope my answers help! :-) Yes, thank you for expounding. -- Dirk ___ webkit-dev mailing list webkit-dev@lists.webkit.org https://lists.webkit.org/mailman/listinfo/webkit-dev
Re: [webkit-dev] Is there a plan for supporting multi-process and WebCL in webkit
On Wed, Apr 10, 2013 at 7:37 AM, Filip Pizlo fpi...@apple.com wrote: On Apr 10, 2013, at 2:29 AM, Pozdnyakov, Mikhail mikhail.pozdnya...@intel.com wrote: Why not infer ParallelArrays automatically? Sorry, did not get it. Could you please elaborate? Normal JS arrays. You use them with pure computation. Profiler notices this. And then the same things that you have in the ParallelArrays proposal now work except you don't need a new type. Compiler writers have been chasing this problem for 50 years, with (AFAIK) very limited success for languages that don't give you compile-time annotations to help you along. I'm not aware of any work on JIT compilers doing this. Why do you think we could do better? (Honest question, I'm not trying to be snarky, nor am I an expert). ___ webkit-dev mailing list webkit-dev@lists.webkit.org https://lists.webkit.org/mailman/listinfo/webkit-dev
Re: [webkit-dev] Is there a plan for supporting multi-process and WebCL in webkit
Right, I get how ParallelArrays would work. I was asking Filip how you
might infer that an array could be a ParallelArray.
-- Dirk
On Wed, Apr 10, 2013 at 12:45 PM, Oliver Hunt oli...@apple.com wrote:
The parallel arrays apis aren't a magic make my code parallel wand.
They don't make general code parallel, they simply provide a bunch of
functions like map, etc where if you restrict your list of language
features to a specific subset, they'll vectorise it.
For instance
ParallelArray([1,2,3]).map(function(a,v) { return v * 2; })
would be vectorised
ParallelArray([1,2,3]).map(function(a,v) { return v.x * 2; })
wouldn't be.
This isn't solve autovectorisation, this is we've been given a specific
operation to perform on each element of an array, parallelise it if
possible, and what's possible is arbitrarily limited, and require a
distinct type that isn't an array.
--Oliver
On Apr 10, 2013, at 12:37 PM, Dirk Pranke dpra...@chromium.org wrote:
On Wed, Apr 10, 2013 at 7:37 AM, Filip Pizlo fpi...@apple.com wrote:
On Apr 10, 2013, at 2:29 AM, Pozdnyakov, Mikhail
mikhail.pozdnya...@intel.com wrote:
Why not infer ParallelArrays automatically?
Sorry, did not get it. Could you please elaborate?
Normal JS arrays. You use them with pure computation. Profiler notices
this. And then the same things that you have in the ParallelArrays proposal
now work except you don't need a new type.
Compiler writers have been chasing this problem for 50 years, with (AFAIK)
very limited success for languages that don't give you compile-time
annotations to help you along. I'm not aware of any work on JIT compilers
doing this. Why do you think we could do better? (Honest question, I'm not
trying to be snarky, nor am I an expert).
___
webkit-dev mailing list
webkit-dev@lists.webkit.org
https://lists.webkit.org/mailman/listinfo/webkit-dev
___
webkit-dev mailing list
webkit-dev@lists.webkit.org
https://lists.webkit.org/mailman/listinfo/webkit-dev
Re: [webkit-dev] Is there a plan for supporting multi-process and WebCL in webkit
Right, you need some way to ensure that the functions applied to the array
are pure, the array doesn't mutate, etc. I thought perhaps your
ParallelArray() type would be ensuring this (e.g., by enforcing type
checking and freezing the array), but maybe I misunderstood you? Even then,
I'm not sure how easy ensuring the functions were pure would be.
Plus there's the problem of figuring how best to parallelize the
operations, which is also often non-trivial.
-- Dirk
On Wed, Apr 10, 2013 at 12:58 PM, Oliver Hunt oli...@apple.com wrote:
I'm not sure what you mean here: a parallel array is simply an array of
numbers; there is nothing magic about it.
If we _really_ wanted to be blunt about it, we could simply say (in
Array.map)
if (length = giant function.isEasyPeasy() everything in the array
is a number)
return doHardcoreMap();
// carry on doing regular map
...
Understand that the parallel APIs have to do all of this as well, the only
thing they lose is the is everything a number check
--Oliver
On Apr 10, 2013, at 12:50 PM, Dirk Pranke dpra...@chromium.org wrote:
Right, I get how ParallelArrays would work. I was asking Filip how you
might infer that an array could be a ParallelArray.
-- Dirk
On Wed, Apr 10, 2013 at 12:45 PM, Oliver Hunt oli...@apple.com wrote:
The parallel arrays apis aren't a magic make my code parallel wand.
They don't make general code parallel, they simply provide a bunch of
functions like map, etc where if you restrict your list of language
features to a specific subset, they'll vectorise it.
For instance
ParallelArray([1,2,3]).map(function(a,v) { return v * 2; })
would be vectorised
ParallelArray([1,2,3]).map(function(a,v) { return v.x * 2; })
wouldn't be.
This isn't solve autovectorisation, this is we've been given a
specific operation to perform on each element of an array, parallelise it
if possible, and what's possible is arbitrarily limited, and require a
distinct type that isn't an array.
--Oliver
___
webkit-dev mailing list
webkit-dev@lists.webkit.org
https://lists.webkit.org/mailman/listinfo/webkit-dev
Re: [webkit-dev] Rewriting binding code generator, maybe?
On Wed, Apr 10, 2013 at 4:59 PM, Timothy Hatcher timo...@apple.com wrote: On Apr 10, 2013, at 6:32 PM, Ryosuke Niwa rn...@webkit.org wrote: Can we rewrite CodeGenerator*.pm in Ruby or Python? I feel that the current code is very hard to understand and hack on. In particular we have CodeGenerator.pm and CodeGeneratorJS.pm (CodeGeneratorV8.pm has been removed), and we need to merge them anyway. They can't be merged. We also have CodeGeneratorObjC.pm. (And internally we have CodeGeneratorSafari.pm.) I've seen many people expressing their inability to improve the binding code because of its being written in Perl. I've also seen people express their frustration for our tools currently written in Ruby and Python. I personally find Perl fine for this task and don't see a need to rewrite things just because. I'm sure Perl is a fine tool for the job, if you happen to know Perl, but Perl code is generally much less approachable than Ruby, and Ruby less so than Python. Of course many if not most of the Python zealots have gone elsewhere :). I believe in Blink-land there may be a reasonably concerted effort to move the stuff Chromium still uses to Python where possible, and obviously this is one such case (as Eric alludes to earlier), so it does seem like it would be nice to be able share the code and make the rewrite generic if it wasn't too much additional work, there was interest in doing so, and the rewrite was gonna happen anyway. Otherwise I'm quite sympathetic to the ain't broke argument here :) -- Dirk ___ webkit-dev mailing list webkit-dev@lists.webkit.org https://lists.webkit.org/mailman/listinfo/webkit-dev
Re: [webkit-dev] Tools/BuildSlaveSupport/.../TestFailures/run-unittests.html
TestFailures, as I understand it, provides a summarize view of current tree failures clustered by the changes that might have caused the failures, kind of like garden-o-matic's initial page. I believe I broke it at one point accidentally, asked if anyone used it, and Ossy said that he did (there might've been others). To my shame, I never got around to fixing my breakage, and it looks kinda still broken: http://build.webkit.org/TestFailures/ Seems like it would be good to fix it :). I think aroben built it and it has been neglected since he stopped working on WebKit full time. -- Dirk On Tue, Apr 9, 2013 at 10:08 PM, Glenn Adams gl...@skynav.com wrote: Does any bot (or person) run this? Regularly? I just noticed today it reports a number of failures (using r148074). Mozilla/5.0 (Macintosh; Intel Mac OS X 10_8_3) AppleWebKit/537.37+ (KHTML, like Gecko) Version/6.0.3 Safari/536.28.10 Tests completed in 705 milliseconds. 513 tests of 545 passed, 32 failed. May I assume there would be no objection if I attempt to fix those failures? ___ webkit-dev mailing list webkit-dev@lists.webkit.org https://lists.webkit.org/mailman/listinfo/webkit-dev ___ webkit-dev mailing list webkit-dev@lists.webkit.org https://lists.webkit.org/mailman/listinfo/webkit-dev
Re: [webkit-dev] Cleaning House
On Thu, Apr 4, 2013 at 12:30 AM, Geoffrey Garen gga...@apple.com wrote: Hi folks. Since we no longer need to support the Chromium port, let's take the opportunity to streamline. Hopefully, this will make development easier and more coherent for everyone. Adam and Eric offered to do some of this cleanup, but I think it's healthier for people who will continue to be a part of WebKit to decide what gets cleaned up, and execute on the plan. Below is a high-level view of some improvements we're planning over the coming weeks. Concepts we plan to remove: Layering violations in WebCore/platform, where a Page* or Frame* is passed to a function Supplementable and Supplement #if USE(GOOGLEURL) #if USE(V8) #if !USE(JSC) #if PLATFORM(CHROMIUM) Skia DOMFileSystem WebLayer and its scrolling implementation Features #defines that haven't gained traction Specific files we plan to remove: .gyp build files WebCore/bindings/v8 WebCore/bindings/scripts/*v8* LayoutTests/platform/chromium* WebKit/chromium WTF/wtf/chromium WebCore/platform/chromium WebCore/*Chromium* Source/Platform/chromium ManualTests/chromium/ Tools/BuildSlaveSupport/chromium/ Tools/DumpRenderTree/chromium/ Also: Adopt libc++ Please let me know if you have other suggestions for improvements, or if you see something in the list that shouldn't be there. FWIW, mrobinson has been working on a GYP build for the GTK port, so I wouldn't delete all of the .gyp files (at least not w/o them weighing in on it). I thought there was some interest at Apple in also using GYP, but perhaps things have (or will) change given that interop w/ Chromium is no longer the big plus it would've been. -- Dirk ___ webkit-dev mailing list webkit-dev@lists.webkit.org https://lists.webkit.org/mailman/listinfo/webkit-dev
Re: [webkit-dev] Cleaning House
On Thu, Apr 4, 2013 at 12:50 PM, Geoffrey Garen gga...@apple.com wrote: To clarify: (1) The EWS bots are still running. (2) The mac and mac-wk2 EWS bots are running tests, and passing. (3) The cr-linux bots are running tests, and failing. If we're OK with item (3), we can go ahead with cleaning house, and break the cr-* EWS bots entirely, while we work on making the mac and mac-wk2 EWS bots faster. I've posted a patch to turn off the tests on the cr-* EWS bots (though we'll still try to compile). https://bugs.webkit.org/show_bug.cgi?id=113959. Obviously, we can simply turn the bots off whenever they're no longer useful as well. -- Dirk ___ webkit-dev mailing list webkit-dev@lists.webkit.org https://lists.webkit.org/mailman/listinfo/webkit-dev
Re: [webkit-dev] Compiling WebKit up to 25% faster in Windows?
On Tue, Mar 26, 2013 at 11:47 AM, Ryosuke Niwa rn...@webkit.org wrote: On Tue, Mar 26, 2013 at 11:42 AM, Jarred Nicholls jar...@webkit.orgwrote: On Tue, Mar 26, 2013 at 2:37 PM, Ryosuke Niwa rn...@webkit.org wrote: On Tue, Mar 26, 2013 at 11:21 AM, Daniel Bratell brat...@opera.comwrote: Is this something that has been talked about in the past, and would you be interested in replacing the long list of directories to search for every include with paths (relative some good base) directly in the include directives? Using explicit paths to include files has been talked about in the past; e.g. https://lists.webkit.org/pipermail/webkit-dev/2011-November/018632.html The most convincing counter argument I can remember is that it'll make refactoring harder because you'll have to update all #include's when you move headers. A few tailored scripts make that an easier job. We're all engineers after all. We have been talking about this for years, and we even have bugs: https://bugs.webkit.org/show_bug.cgi?id=61772 Given that, I'm somewhat skeptical if we ever get around to it in foreseeable future. Scripts that update include paths from Source/WebCore/foo.h to Source/Platform/foo.h are substantially easier than something to modify XCode projects robustly ;) If we have consensus that we should just switch to paths relative to Source (or maybe a couple different options), that would be (IMO) a big win. It sounds like Daniel co. have already done the big bang conversion. On Tue, Mar 26, 2013 at 11:58 AM, Geoffrey Garen gga...@apple.com wrote: In WebKit include directives are without path, and instead the compiler is given a very long list of directories to search through. That process takes a lot of time in Windows. It must take some time in OSX and in Linux too but probably less. Can we make a first-order improvement just by making sure that paths are searched in order of likelihood? That actually sounds harder and more fragile to me than switch to full paths across the board. -- Dirk ___ webkit-dev mailing list webkit-dev@lists.webkit.org https://lists.webkit.org/mailman/listinfo/webkit-dev
Re: [webkit-dev] Compiling WebKit up to 25% faster in Windows?
On Tue, Mar 26, 2013 at 1:29 PM, Benjamin Poulain benja...@webkit.orgwrote: On Tue, Mar 26, 2013 at 1:20 PM, Dirk Pranke dpra...@chromium.org wrote If we have consensus that we should just switch to paths relative to Source (or maybe a couple different options), that would be (IMO) a big win. It sounds like Daniel co. have already done the big bang conversion. I think using full path would be a serious hit regarding hackability. I would rather spend some time tweaking my compiler to cache each directory content than waste time finding where is every single header I need to include. Interesting. I have the exact opposite experience, having to paw around to figure out where Font.h actually lives rather than just seeing WebCore/platform/graphics/Font.h. At any rate, to be clear, I would be in favor of that change but I'm not expecting it to happen :). -- Dirk ___ webkit-dev mailing list webkit-dev@lists.webkit.org https://lists.webkit.org/mailman/listinfo/webkit-dev
Re: [webkit-dev] Compiling WebKit up to 25% faster in Windows?
On Tue, Mar 26, 2013 at 1:42 PM, Alexis Menard ale...@webkit.org wrote: On Tue, Mar 26, 2013 at 5:40 PM, Dirk Pranke dpra...@chromium.org wrote: On Tue, Mar 26, 2013 at 1:29 PM, Benjamin Poulain benja...@webkit.org wrote: On Tue, Mar 26, 2013 at 1:20 PM, Dirk Pranke dpra...@chromium.org wrote If we have consensus that we should just switch to paths relative to Source (or maybe a couple different options), that would be (IMO) a big win. It sounds like Daniel co. have already done the big bang conversion. I think using full path would be a serious hit regarding hackability. I would rather spend some time tweaking my compiler to cache each directory content than waste time finding where is every single header I need to include. Interesting. I have the exact opposite experience, having to paw around to figure out where Font.h actually lives rather than just seeing WebCore/platform/graphics/Font.h. But a modern IDE can easily solve that for you. You just click on the header and it opens it for you. Xcode, QtCreator handles it very well. That said if you don't use an IDE then I understand your point :). I think to me the fact that the paths are spelled out is both one less action I have to do to get my question answered (I don't have to click), and a helpful reminder of the general layout of the code (and which directories I might need to care about in relation to a particular file). -- Dirk ___ webkit-dev mailing list webkit-dev@lists.webkit.org https://lists.webkit.org/mailman/listinfo/webkit-dev
Re: [webkit-dev] New script to import W3C CSS tests
On Thu, Mar 7, 2013 at 8:46 AM, Rebecca Hauck rha...@adobe.com wrote: Hi Dirk Everyone, Dirk, your question about this in IRC the other day was timely indeed. I've recently written a batch of tests for CSS Regions that I intend to submit to both the W3C and Webkit (and have planned more to come). Rather than push two slightly different versions of the same tests to each repo, and to address how these tests are to be maintained over time, I took a stab automating the import and just submitted a patch [1]. I know this conversation was started a while ago as I attended the WK Contributor's meeting last spring where my colleagues Jacob Goldstein and Alan Stearns led the discussion. I was sort of new onto the scene then and was pretty much a lurker in the meeting, so you probably don't remember me. I recall that much of the debate was about handling failing tests, approved vs. submitted, duplicates, etc. Since all I've done is automate the mechanics of the import itself, I have not solved a lot of those issues, but the conversion of tests certainly had its own set of things to consider. I figured I would put this one piece in place, get feedback on it, and hopefully resurrect the discussion on fitting this into the bigger picture. A great goal would be to have this script run on an automated schedule and to extend it to deal with test results, expected failures, etc. All of the dirty details are in the ChangeLog of the patch and in the script headers, but I basically tried to cover #8 from last spring's meeting notes [2]. I think the script covers all of those bullets, with that last bullet as sort of. I also got a little inspiration from the import-w3c-performance-wg-tests that already exists. I followed a few of their steps, but had to add a few layers to handle the added complexity of the CSS test suites. Dirk, when you have a chance, take a look. Review it with gusto as you say. And anyone else who's interested, your feedback is welcome. I fairly certain this will go a few rounds before it's settled. :) Thanks, Rebecca! It will be great if we can start regularly importing and running the W3C's tests. (I'll review the patch and post any other comments on the bug). -- Dirk ___ webkit-dev mailing list webkit-dev@lists.webkit.org https://lists.webkit.org/mailman/listinfo/webkit-dev
Re: [webkit-dev] New script to import W3C CSS tests
On Thu, Mar 7, 2013 at 3:04 PM, Tony Gentilcore to...@chromium.org wrote: I also got a little inspiration from the import-w3c-performance-wg-tests that already exists. I followed a few of their steps, but had to add a few layers to handle the added complexity of the CSS test suites. Is there any way we can merge the two scripts so there is only one import/exporter than handles both performance and CSS test suites? That is a good question, and something I'll be looking at as part of this :). More generally, I'm trying to look into importing and running tests from the w3c across the board as one of my next projects. Rebecca beat me to it! -- Dirk -Tony ___ webkit-dev mailing list webkit-dev@lists.webkit.org https://lists.webkit.org/mailman/listinfo/webkit-dev ___ webkit-dev mailing list webkit-dev@lists.webkit.org https://lists.webkit.org/mailman/listinfo/webkit-dev
Re: [webkit-dev] Best practices for landing new/changed layout test expectations?
On Tue, Feb 26, 2013 at 1:03 PM, Ryosuke Niwa rn...@webkit.org wrote: On Tue, Feb 26, 2013 at 12:47 PM, Dirk Pranke dpra...@chromium.org wrote: On Tue, Feb 26, 2013 at 2:11 AM, Ryosuke Niwa rn...@webkit.org wrote: On Tue, Feb 26, 2013 at 1:55 AM, Tom Hudson tomhud...@google.com wrote: On Mon, Feb 25, 2013 at 10:34 PM, Ryosuke Niwa rn...@webkit.org wrote: It should be fairly straight forward to create a tool that analyzes files changed in each commit and deduce which tests' expected results have been changed. The tool can then fetch results from each port' bot for those tests and automatically land them. It can then comment on the bug automatically about these rebaseline commits. There is no need to add remove entries from TestExpectation files. Wait, what? For some reason neither I nor the mailing list archives got your initial message, nor Silvia or Tom's responses, nor your responses (at least as of the time of me writing this), so I feel like I've missed a radical shift in this thread, and maybe I missed some of the context. https://lists.webkit.org/pipermail/webkit-dev/2013-February/023967.html This link doesn't point to any of those messages, but perhaps it's not that important. You're proposing that we automatically land updated baselines without review and then somehow update bugs, have people go back and look at the updated bugs to see if the baseline changes represent actual regressions or just expected changes? Right. Given that the commit already contains information as to which tests have been rebaselined, a script should be able to fetch new baselines for those affected tests on each platform and land them or upload as patches as needed. It's possible that we could fetch and cluster new baselines based on what changed in the initial commit. I would be concerned that there could be a fair amount of noise in either direction (tests that changed on the initial platform didn't on others, and others did), and you'd also have to figure out how to cluster changes since most builds on the bots contain multiple changes. But, you could probably use some of garden-o-matic's results to help here. That said, I'm not sure this workflow would actually improve things much over garden-o-matic. I am quite a bit more reluctant to automatically land any such changes; it seems like it would be hard if not impossible to tell (programmatically) whether a baseline changed as expected or if it represented a regression. If we were to work on new tooling, I would be much more in favor of pushing this up to an EWS-time step like Ossy suggests. -- Dirk ___ webkit-dev mailing list webkit-dev@lists.webkit.org https://lists.webkit.org/mailman/listinfo/webkit-dev
Re: [webkit-dev] Best practices for landing new/changed layout test expectations?
On Mon, Feb 25, 2013 at 12:38 PM, Eric Seidel esei...@google.com wrote: I've noticed as of late several different approaches being used when adding/changing LayoutTests which need rebaselining on other platforms. Obviously we cannot expect developers to test/rebaseline on all platforms before landing given our current infrastructure. But what should we expect them to do? Currently some folks add failing expectations to other ports TestExpectations. Some add [skip]. Some even add them to the (new) global TestExpectations file. What's the proper course of action? I checked: http://trac.webkit.org/wiki/Keeping%20the%20Tree%20Green http://trac.webkit.org/wiki/Creating%20and%20Submitting%20Layout%20Tests%20and%20Patches http://trac.webkit.org/wiki/TestExpectations and didn't see the I expect this to fail/need rebaseline on other ports case discussed. Memory says that the last time this was raised [1], the consensus was to land the change, turn the tree red, notify as many gardeners / port maintainers as possible, and rebaseline as quickly as possible. I.e., don't add entries to TestExpectations. Of course, such a policy wouldn't play nicely w/ the EWS bots, and I think this discussion predated everyone switching over to TestExpectations (and certainly predated the generic expectations file). I don't find the above entirely satisfactory, but I also don't have any great alternatives to endorse given the existing tooling. Also, if the test fails generically (all ports), it probably shouldn't be landed. I'm not sure why you couldn't at least update the port you're developing on; if you don't, how do you know your fix is working at all? (Of course, there could be a case where a high-priority fix might break some low-priority tests). I remember some discussion of a [rebaseline] keyword in TestExpectations, but I'm not sure that ever made it in? The [ NeedsRebaseline ] enhancement is, as of yet, unimplemented and unclaimed [2]. It shouldn't be too hard for someone to try it if they're looking for a reason to explore the NRWT code :) -- Dirk [1] https://lists.webkit.org/pipermail/webkit-dev/2012-April/020250.html [2] https://bugs.webkit.org/show_bug.cgi?id=100415 ___ webkit-dev mailing list webkit-dev@lists.webkit.org https://lists.webkit.org/mailman/listinfo/webkit-dev
Re: [webkit-dev] Best practices for landing new/changed layout test expectations?
On Mon, Feb 25, 2013 at 1:22 PM, Ryosuke Niwa rn...@webkit.org wrote: On Mon, Feb 25, 2013 at 12:52 PM, Dirk Pranke dpra...@google.com wrote: On Mon, Feb 25, 2013 at 12:38 PM, Eric Seidel esei...@google.com wrote: I've noticed as of late several different approaches being used when adding/changing LayoutTests which need rebaselining on other platforms. Obviously we cannot expect developers to test/rebaseline on all platforms before landing given our current infrastructure. But what should we expect them to do? Currently some folks add failing expectations to other ports TestExpectations. Some add [skip]. Some even add them to the (new) global TestExpectations file. What's the proper course of action? I checked: http://trac.webkit.org/wiki/Keeping%20the%20Tree%20Green http://trac.webkit.org/wiki/Creating%20and%20Submitting%20Layout%20Tests%20and%20Patches http://trac.webkit.org/wiki/TestExpectations and didn't see the I expect this to fail/need rebaseline on other ports case discussed. Memory says that the last time this was raised [1], the consensus was to land the change, turn the tree red, notify as many gardeners / port maintainers as possible, and rebaseline as quickly as possible. I.e., don't add entries to TestExpectations. Of course, such a policy wouldn't play nicely w/ the EWS bots, and I think this discussion predated everyone switching over to TestExpectations (and certainly predated the generic expectations file). I don't find the above entirely satisfactory, but I also don't have any great alternatives to endorse given the existing tooling. Also, if the test fails generically (all ports), it probably shouldn't be landed. I'm not sure why you couldn't at least update the port you're developing on; if you don't, how do you know your fix is working at all? (Of course, there could be a case where a high-priority fix might break some low-priority tests). Adding a new feature behind a build flag disabled everywhere by default. I remember some discussion of a [rebaseline] keyword in TestExpectations, but I'm not sure that ever made it in? The [ NeedsRebaseline ] enhancement is, as of yet, unimplemented and unclaimed [2]. It shouldn't be too hard for someone to try it if they're looking for a reason to explore the NRWT code :) I object to adding such a thing. People add and forget about these entries way too often: The rationale for adding the keyword was precisely this ... it would make it easier to programmatically identify tests needing updated baselines and alert people :). Do you have an alternative you'd suggest for keeping people from forgetting (particularly in the case where we have an expected failure waiting on a bug fix)? -- Dirk ___ webkit-dev mailing list webkit-dev@lists.webkit.org https://lists.webkit.org/mailman/listinfo/webkit-dev
Re: [webkit-dev] Best practices for landing new/changed layout test expectations?
On Mon, Feb 25, 2013 at 1:33 PM, Ryosuke Niwa rn...@webkit.org wrote: On Mon, Feb 25, 2013 at 1:29 PM, Dirk Pranke dpra...@google.com wrote: On Mon, Feb 25, 2013 at 1:22 PM, Ryosuke Niwa rn...@webkit.org wrote: On Mon, Feb 25, 2013 at 12:52 PM, Dirk Pranke dpra...@google.com wrote: I remember some discussion of a [rebaseline] keyword in TestExpectations, but I'm not sure that ever made it in? The [ NeedsRebaseline ] enhancement is, as of yet, unimplemented and unclaimed [2]. It shouldn't be too hard for someone to try it if they're looking for a reason to explore the NRWT code :) I object to adding such a thing. People add and forget about these entries way too often: The rationale for adding the keyword was precisely this ... it would make it easier to programmatically identify tests needing updated baselines and alert people :). Do you have an alternative you'd suggest for keeping people from forgetting (particularly in the case where we have an expected failure waiting on a bug fix)? Just turn bots red. People maintaining bots will notice and rebaseline them as needed. If there's a significant time lapse between failing test and bug fix, it seems like a bad idea to leave the tree red. Are you suggesting we should land a failling baseline in the meantime? -- Dirk ___ webkit-dev mailing list webkit-dev@lists.webkit.org https://lists.webkit.org/mailman/listinfo/webkit-dev
Re: [webkit-dev] Getting involved
Hi Jason, On http://trac.webkit.org/wiki there are a bunch of pages under Layout Tests and Performance Tests that describe much of the infrastructure. These days most of it is written in Python. I don't know what projects Benjamin has in mind so I don't know if that will match what he was referring to, but hopefully this will be helpful. -- Dirk On Mon, Feb 18, 2013 at 3:44 PM, Jason Anderssen janders...@exactal.com wrote: Hello Benjamin, Thank you for a starting point and would like to explore further with this. Please point me in the right direction to how I can help in the cleaning of the WebKit testing infrastructure please. This will also allow me to understand how all you guys work together and understand the procedures etc.. Cheers Jason From: Benjamin Poulain benja...@webkit.org Date: Tue, 19 Feb 2013 08:52:17 +1000 To: Jason Anderssen janders...@exactal.com Cc: webkit-dev@lists.webkit.org webkit-dev@lists.webkit.org Subject: Re: [webkit-dev] Getting involved Hi Jason, On Mon, Feb 18, 2013 at 2:45 PM, Jason Anderssen janders...@exactal.com wrote: With a background in development (both windows and MAC) I was wanting to get involved in an open source project that I believe is going to make the world a greater place. So I have decided to get involved with the WebKit project. I would like to start of by performing the boring work of reformatting code to suite the guidelines as stipulated on the webkit.org website (and double check that they are still up to date), and would like recommendations to which parts of the project could do with the re-formatting first up. ( I feel this will get me to know the framework, and structure of the code base well, before I can start inputting my coding and help. ) Any feedback to where to start would be very much appreciated. Also I have the code all built, however if I ./run-safari I get Can't find built framework at /Users/jason/research/webkit/WebKitBuild/Release/JavaScriptCore.framework/Versions/A/JavaScriptCore. I will work out the problem, but in case someone knows exactly what I have done wrong, it may save me some time. First welcome to WebKit :) I am sure there is a better use of your time than fixing the coding style. A side project of mine is improving and cleaning some of WebKit testing infrastructure. There are some fairly easy tasks related to that if you are interested. I am sure others will have ideas of easy things to do to get started. For your build/run issues, it will be easier to ask on IRC #webkit than on the mailing list. Benjamin ___ webkit-dev mailing list webkit-dev@lists.webkit.org https://lists.webkit.org/mailman/listinfo/webkit-dev ___ webkit-dev mailing list webkit-dev@lists.webkit.org https://lists.webkit.org/mailman/listinfo/webkit-dev
Re: [webkit-dev] Testing feature suggestion: animation/interaction pixel-results on the fly
On Wed, Feb 13, 2013 at 11:33 PM, Benjamin Poulain benja...@webkit.org wrote:
On Wed, Feb 13, 2013 at 11:16 PM, Dirk Pranke dpra...@chromium.org wrote:
Those changes are not harmless. There are people monitoring tests
results
full time in order to keep WebKit in good shape. No other part of WebKit
require continuous attention.
I'm sorry, but either I don't understand Dongsung's suggestion, or I
don't understand your criticism. What does this sentence paragraph
mean? Are you suggesting someone needs to look at this type of test
full time in a way that we don't have to look at the other tests?
My lack of sleep is likely to blame here :)
All I am saying is tests have a hight cost when they start failing, we
should be careful how we design them so we can easily triage bad tests,
flaky tests, and real errors.
No argument here :).
Case 1: CSS Filters Shaders
I wanted this test functionality when I commented
http://webkit.org/b/97859#c19
If I want to make gaussian blur test, I prefer using 'getPixel' test as
follows,
Why wasn't a ref-test a better solution in this particular case?
I'm curious, what would you imagine the ref test contains?
If I am not mistaken, the composition operations are parallel with the ones
of SVG and Canvas (aren't they?).
I would have attempted comparing the 3 implementations as it seems to me the
pixels values should be the same.
That was a genuine question about that case :)
Case 2: Fixed Position Element
[...]
function repeatedlyCalledDuringScrolling() {
ASSERT(getPixel(15, 9) == white);
ASSERT(getPixel(15, 10) == green);
ASSERT(getPixel(9, 15) == white);
ASSERT(getPixel(10, 15) == green);
}
I think this shows what I said about correctness and readability:
-Asserting the correctness of the test and the result becomes close to
impossible for the reader. One has to review the full code to have a
chance
of understanding an error.
-You cannot cover non trivial cases (images, text, form elements, etc).
-And it is inefficient. You have to render each frame on the UIProcess,
move
it to the WebProcess, and box it for JavaScript to process (with pixel
format conversions depending on your graphics system)
Of the ideas raised, I think this is one of my least favorite for
testing
fixed positioning.
Isn't his suggestion the equivalent of what we do today in text-only
tests? i.e., printing pass or fail and making you have to look at
the test itself to see what's being tested?
If the correctness of the rendering depends on those 4 specific pixels
having those four specific values, how exactly are you going to verify
that by looking at it?
Again, I think I'm just not understanding you here?
When looking at a test test, you follow the flow to know what it is supposed
to do and where it breaks.
How are you supposed to know, _by reading the code_, that the color at
position 15, 9 should be white?
is the code the test code, or the source code we're testing? Of
course, without any larger context, those assertions are impossible to
interpret. I figured he was just indicating what he meant by a
programmatic check, and a real test case would be more informative.
And of course it's easy to write bad tests that are neither readable
nor maintainable. But we also have lots of pixel tests that are
difficult for people not intimately familiar with the code being
tested to evaluate for correctness, too.
Lastly, I don't think he was suggesting that this would replace all
pixel tests; as you say, it is more well suited for some times of
tests than others.
As far as efficiency goes, even moving the picture back and forth
between two processes is still going to be far faster than running a
second reftest and comparing the output of the two pages.
-- Dirk
___
webkit-dev mailing list
webkit-dev@lists.webkit.org
https://lists.webkit.org/mailman/listinfo/webkit-dev
Re: [webkit-dev] Is the wxWidgets port maintained?
On Mon, Feb 11, 2013 at 7:44 PM, Benjamin Poulain benja...@webkit.org wrote: I am sorry, I should have given more context. There is visibly a growing discontent in the community about the cost imposed from small ports. Just two weeks ago, there were 2 threads discussing the cost of peripheral ports. I am convinced a part of this is technical. The project has not changed its policies while the number of ports was growing. While duplicated code and interfaces was okay when there were only 3 ports, it has become a pain when we have 7+ ports to updates ... In his email WebKit Wishes, Eric said It can’t be the job of the core maintainers to care about all the peripheral ports which contribute very little core code. I think it was an entirely reasonable question to ask if the wx port was being maintained, but I'm surprised by how this thread has evolved. There is a lot of discussion going on about the cost of so many ports, but not much about the benefits. Speaking personally, even before I joined Google, I was drawn to WebKit partially because it was used on such a wide variety of projects and in so many different ways. I was fortunate to be able to get a job that allows me to contribute to it, and I have found the work that I've done to help maintain the peripheral ports, while not pain free, quite rewarding (although I would be quite happy if it was less costly, of course). My point is that I think that lots of ports are part of what makes WebKit the goodness it is. Maybe I'm alone here, or at best part of a minority, but I wanted us to not lose sight of this idea. -- Dirk ___ webkit-dev mailing list webkit-dev@lists.webkit.org https://lists.webkit.org/mailman/listinfo/webkit-dev
Re: [webkit-dev] Is the wxWidgets port maintained?
On Mon, Feb 11, 2013 at 1:41 PM, Kevin Ollivier kev...@theolliviers.com wrote: Hi Martin, On Feb 11, 2013, at 12:47 PM, Martin Robinson wrote: On Mon, Feb 11, 2013 at 11:27 AM, Kevin Ollivier kev...@theolliviers.com wrote: The project is not dead per-se, but it's true that it's not very active these days. Recently, I haven't had a lot of time to devote to the port. The other big factor has been that I've been the only active contributor to the port pretty much since its inception, which has made patch submission pretty difficult due to the challenge of finding a reviewer with knowledge of the wx code. As a result, the wx activity on trunk will look pretty sparse since I'm making most of my changes on the wx port's git repo on gitorious.org rather than submitting patches for trunk. When it comes to trunk, I basically just periodically submit build fixes for any build breakages in order to merge and update my git mirror. Regardless of whether the Wx port stays in the tree, I think you should consider changing your build system. I believe the only port that uses waf is Wx, so using cmake or gyp will make your work a lot easier. Actually, it will, if anything, increase the workload. Because I use waf, I am able to use Python to auto-generate the list of sources to build. In other words, I tell it to build all sources in a defined list of base source directories, along with all sources in baseDirectory/portName subdirs (where in this case I set portName to wx) and FileNameportName.cpp (e.g. FileNameWX.cpp if the port is wx), along with some additional similar rules for USE defines, like CF. Then if there are exceptions to these rules, I just filter them out of the results or explicitly add files when I need to, say if I need to compile a single Mac port source file. Since the WebKit tree is well-structured, this approach works quite well, with me having to define exceptions fairly rarely. The main issue I run into seems to be derived sources files that are no longer built / used but are still being generated. The performance hit of this is about 1 added second to my build, though on slower machine s it might be a couple seconds. For me, it's negligible given the benefit I get from it. As an example, I'm still at the linking phase right now on Mac (dealing with the removal of the libWebKitLibrarySnowLeopard.a and a couple other things), but my patch so far to catch up with 6 months of changes in trunk is right now at 19KB, and about 30 lines or so of that is actually the build system changes. The number of exceptions I needed to add to the build over that time is 4, and I added about as many build directories to the list as well. I suspect if we were to accumulate the changes to cmake and gyp files over those 6 months, it will probably be well above 30-40 lines. Sure, I wouldn't have written all those lines myself, but sharing work with other developers is still more work for any individual developer than having the computer do the work for you. :) Everyone tells me how great cmake and gyp is, but I'm not sure any of them have taken any time out to actually investigate whether or not waf is as good as or better than their preferred tool. (It's not as good as auto-generating project files, but it's also in Python so integrating it with gyp would probably be a simple task.) I've often wanted to take some time out to get other ports building with it, as it probably would take a day or two at most, but I lack both the time and expertise in other ports to spend much time on that. Much of the code in there already is port-agnostic, with most wx-specific bits in a port_name == wx check, but of course there's the issue of the various port-specific build configurations and such for each port. I have to admit, I thought waf was more like scons and less like cmake and gyp, i.e., I thought it was a portable build system rather than a meta-build system (and had never looked at it in detail accordingly). I am taking another look at it now :) -- Dirk ___ webkit-dev mailing list webkit-dev@lists.webkit.org https://lists.webkit.org/mailman/listinfo/webkit-dev
Re: [webkit-dev] Is the wxWidgets port maintained?
On Mon, Feb 11, 2013 at 3:26 PM, Kevin Ollivier kev...@theolliviers.com wrote: On Feb 11, 2013, at 2:01 PM, Martin Robinson wrote: On Mon, Feb 11, 2013 at 1:41 PM, Kevin Ollivier kev...@theolliviers.com wrote: Actually, it will, if anything, increase the workload. Because I use waf, I am able to use Python to auto-generate the list of sources to build. In other words, I tell it to build all sources in a defined list of base source directories, along with all sources in baseDirectory/portName subdirs (where in this case I set portName to wx) and FileNameportName.cpp (e.g. FileNameWX.cpp if the port is wx), along with some additional similar rules for USE defines, like CF. Then if there are exceptions to these rules, I just filter them out of the results or explicitly add files when I need to, say if I need to compile a single Mac port source file. Since the WebKit tree is well-structured, this approach works quite well, with me having to define exceptions fairly rarely. The main issue I run into seems to be derived sources files that are no longer built / used but are still being generated. The performance hit of this is about 1 added second to my build, though on slower machi n es it might be a couple seconds. For me, it's negligible given the benefit I get from it. If I understand correctly, gyp is also capable of this kind of wildcard inclusion and exclusion. This will be the tool that allows the gyp build to be shared among many ports with the same source lists. Yes, but that is not what I see when I check, say, Source/WebCore/WebCore.gypi. Which, BTW, has had around 500 revisions in it over the past 6 months in comparison to the ~10 lines of changes to source files in my patch-in-progress. So while gyp itself might have that feature in it, for whatever reason, the WebKit projects do not utilize that feature right now, so in practice, a switch to gyp means a switch away from rule-based compilation. There's two different things going on here: ease of maintenance, and reproducibility. Using wildcards in the (meta-)build file itself can make things easier but make it harder to get reproducible builds since you might accidentally include files in the tree you didn't want. When you're on a minority port, having wildcards certainly makes your life easier because you don't have to track every change we make explicitly, and it's probably easier for you to ensure you're doing things correctly in your own checkouts, so you don't have to worry about reproducibility as much. On the other hand, if you were to convert to using the same gypi files we were using, then you wouldn't have to track every change we make, either; you'd get them for free. -- Dirk ___ webkit-dev mailing list webkit-dev@lists.webkit.org https://lists.webkit.org/mailman/listinfo/webkit-dev
Re: [webkit-dev] Is the wxWidgets port maintained?
On Mon, Feb 11, 2013 at 4:30 PM, Kevin Ollivier kev...@theolliviers.com wrote: On Feb 11, 2013, at 3:33 PM, Dirk Pranke wrote: On Mon, Feb 11, 2013 at 3:26 PM, Kevin Ollivier kev...@theolliviers.com wrote: On Feb 11, 2013, at 2:01 PM, Martin Robinson wrote: On Mon, Feb 11, 2013 at 1:41 PM, Kevin Ollivier kev...@theolliviers.com wrote: Actually, it will, if anything, increase the workload. Because I use waf, I am able to use Python to auto-generate the list of sources to build. In other words, I tell it to build all sources in a defined list of base source directories, along with all sources in baseDirectory/portName subdirs (where in this case I set portName to wx) and FileNameportName.cpp (e.g. FileNameWX.cpp if the port is wx), along with some additional similar rules for USE defines, like CF. Then if there are exceptions to these rules, I just filter them out of the results or explicitly add files when I need to, say if I need to compile a single Mac port source file. Since the WebKit tree is well-structured, this approach works quite well, with me having to define exceptions fairly rarely. The main issue I run into seems to be derived sources files that are no longer built / used but are still being generated. The performance hit of this is about 1 added second to my build, though on slower mac hin es it might be a couple seconds. For me, it's negligible given the benefit I get from it. If I understand correctly, gyp is also capable of this kind of wildcard inclusion and exclusion. This will be the tool that allows the gyp build to be shared among many ports with the same source lists. Yes, but that is not what I see when I check, say, Source/WebCore/WebCore.gypi. Which, BTW, has had around 500 revisions in it over the past 6 months in comparison to the ~10 lines of changes to source files in my patch-in-progress. So while gyp itself might have that feature in it, for whatever reason, the WebKit projects do not utilize that feature right now, so in practice, a switch to gyp means a switch away from rule-based compilation. There's two different things going on here: ease of maintenance, and reproducibility. Using wildcards in the (meta-)build file itself can make things easier but make it harder to get reproducible builds since you might accidentally include files in the tree you didn't want. When you're on a minority port, having wildcards certainly makes your life easier because you don't have to track every change we make explicitly, and it's probably easier for you to ensure you're doing things correctly in your own checkouts, so you don't have to worry about reproducibility as much. If the filesystem always follows clearly defined rules, then I'd argue that reproducibility would not be an issue, though. WebKit currently doesn't always, but the vast majority of times it does. The cases where it doesn't are few and far between, and I don't think fixes would be all that difficult. You can always create specific locations, like perhaps a Shared directory, for any special cases that should explicitly be added by more than one, but a subset of, ports, and can't be properly encapsulated into a feature or use define. (For example, the WhateverNone.cpp are one of the special exceptions I have to account for, and they could go in there.) I think the concern is really that this approach seems 'implicit' rather than 'explicit'. With clear filesystem rules, though, it's not actually implicit. A file going into one of the directories listed as a primary build dir means it is to be built by all ports. So long as the project defines said rules, the act of putting it there is explicit, and expresses intent. This also, BTW, addresses the problem of old stuff being left in the tree. To remove it from the build, you have to remove it, so you can just leave it sitting in the tree for someone to stumble across later. (I'm pretty sure a number of the exceptions I define in waf are actually examples of this.) I'm not sure we're talking about the same sort of concerns. The case I'm talking about when you have a makefile that says something like SRCS=*.cpp. Then, I have to remember to save a local copy of a file as foo.cpp.orig rather than foo_orig.cpp, and if I want to rename a file I have to actually rename it rather than copy it (or at least remember to delete it). I think you're talking more about conventions over which directories contain files for each port? On the other hand, if you were to convert to using the same gypi files we were using, then you wouldn't have to track every change we make, either; you'd get them for free. Yes, but this mechanism relies on WebKit's policies about fixing others' build systems to be effective. waf, on the other hand, removes that task almost entirely. Plus, see my other comments about all the other benefits of using a meta-build system. (And BTW
Re: [webkit-dev] Testing feature suggestion: animation/interaction pixel-results on the fly
On Sat, Feb 9, 2013 at 1:57 AM, Benjamin Poulain benja...@webkit.org wrote: On Sat, Feb 9, 2013 at 1:24 AM, noam.rosent...@nokia.com wrote: I did not say they cannot be tested with the two methods suggested :) It's more about a preference to move some of those decisions from the test infrastructure to the test itself, but potentially those bugs could be tested in either way. Examples for bugs I've encountered/reviewed and can use better in-motion testing (note that those are specific to Qt/EFL, but I'm sure there are tons of bugs like this that come up for Apple/Google as well) http://trac.webkit.org/changeset/140825 http://trac.webkit.org/changeset/142112 http://trac.webkit.org/changeset/134953 https://bugs.webkit.org/show_bug.cgi?id=109179 Controlling the clock and programmatically sampling the end result would definitely make those more testable, but of course any progress in this area would be beneficial and my preference to a canvas-based API is more of an opinion. To explain my concerns: Sometime, I look at a failing test, and think: what the f**k is this supposed to test?. Then I have to dig a ton of logs, and finally read the change to understand a the single JS statement in the whole script that make the test useful. This is the situation I am afraid with a solution where pixels would be evaluated from JavaScript. You can test, but you lack visibility why something is correct or incorrect. Text tests, ref-tests and pixel tests have a great readability, you can quickly evaluate their correctness. This is important in my opinion, I don't think we want more opaque output like the RenderTree dump. I am not against your plan if the readability of the tests can be good. I'd be happy to review patches toward testing dynamic changes in webpages. A counter-argument, of course, that there are a lot of pixel tests that would be a lot better as text-only tests that were verifying certain aspects of how the page rendered programmatically. Perhaps -- and this is a tangent to this thread -- we could also investigate ways of writing tests that did render pngs but told NRWT to ignore them (e.g., dumpAsTextAndIgnoredImage() or something), or conveyed which parts of the image were important ... -- Dirk ___ webkit-dev mailing list webkit-dev@lists.webkit.org https://lists.webkit.org/mailman/listinfo/webkit-dev
Re: [webkit-dev] Testing feature suggestion: animation/interaction pixel-results on the fly
On Fri, Feb 8, 2013 at 11:12 AM, Ryosuke Niwa rn...@webkit.org wrote: On Fri, Feb 8, 2013 at 5:35 AM, noam.rosent...@nokia.com wrote: we don't currently have a solution in webkit's test infrastructure for testing animations on the fly, or in general for testing multiple image results in the same test. (If this was discussed before and I'm unaware of that discussion, please stop me here...) This is because ImageDiff works on the single-test level, and you can't explicitly make image comparisons in the same test at different points in time. This has before caused us several regressions that were not caught by layout tests. I'd like to propose a solution, and would welcome some feedback on whether it's a good one... The idea is that you would be able to programatically retrieve the current snapshot into a canvas ImageData, and then compare the pixel results with JavaScript in the LayoutTest. Something like: I had similar thoughts but my counter proposal is to let DRT/WTR generate multiple actual results either in the form of multiple layers in PNG or multiple PNG images. The advantage of this latter approach is that we can have multiple reference files as well. e.g. if a green box is to be moved from (0,0) to (50,0) and to (100, 0), we could create three reference files that correspond to each state. But perhaps there is a good reason you didn't choose this approach. Could you elaborate on the reason you picked this particular API? I also appreciate the problem -- and would be interesting in solving it -- but I'm a bit concerned that either proposal is at least somewhat nondeterministic (or flaky), and I'm not sure how you'd be able to control which frames of the animation you were getting well enough. But I'm not much of expert here so I'm not sure how often it matters. -- Dirk ___ webkit-dev mailing list webkit-dev@lists.webkit.org https://lists.webkit.org/mailman/listinfo/webkit-dev
Re: [webkit-dev] Gated trunk, experiences from OpenStack
On Tue, Feb 5, 2013 at 9:46 AM, Martin Robinson mrobin...@webkit.org wrote: On Tue, Feb 5, 2013 at 9:28 AM, Adam Barth aba...@webkit.org wrote: Do you know how they got rid of flakiness in their tests? We've spent a bunch of effort fixing flaky tests (and in marking the remaining flaky tests as flaky), but there's still a long tail of flakiness. I wonder if that sort of thing might be different for OpenStack if they have a different approach to testing than we do. Another useful thing is to know the number of tests in OpenStack. WebKit has more tests than any other project I've worked on. There are two other related aspects that make our tests flaky: 1) They're very high level integration tests (mostly), which, as they cover large swaths of code in each test, are much more susceptible to flakiness than method-level unit tests. 2) They weren't generally written to be run in parallel, and thus we often have to be concerned with system-level resource contention. -- Dirk ___ webkit-dev mailing list webkit-dev@lists.webkit.org https://lists.webkit.org/mailman/listinfo/webkit-dev
Re: [webkit-dev] Common build system (was Re: WebKit Wishes)
I have looked at YAML off and on over the years, and I'm not sure that it would be much of an improvement in this case. I do believe that dropping the strict python syntax could make some things easier to read. I don't have a fully-baked proposal in mind, and I don't know what the perf hit would be. I will also note that -- like in many languages -- it's easy to write hard-to-read files. GYP has never had much of a style guide, and it probably could greatly benefit from one. It is important to make a fair apples-to-apples comparison for whatever we end up looking at. I suspect that given any tool or language, you can find some things that are easy to express and some that are harder. Even the existing chromium gyp files are not necessarily a model of clean code or good style. Also, one thing that's important to realize about GYP's syntax is that it is very set-based; the order things appear in the files is much less important than in many other languages (especially in how they handle conditionals, i.e., if/then/else). This has many advantages, but does also have some disadvantages. -- Dirk On Tue, Feb 5, 2013 at 12:16 PM, Eric Seidel e...@webkit.org wrote: I'm curious if YAML was ever considered? I have very limited experience with YAML, except for Google App Engine config files. It's very python parse-able? :) On Tue, Feb 5, 2013 at 11:55 AM, Mark Mentovai m...@chromium.org wrote: You’re not supposed to use arbitrary Python, it’s highly discouraged. We have a linter that keeps you from doing things you’re not supposed to do (like this), but it slows things down, so it’s not part of the “standard” GYP run that developers normally use. It can run as a pre-commit script or test on the bot or something else. Used as we’re using it, GYP basically is simple JSON, except the rules about commas are looser and # introduces comments. On Tue, Feb 5, 2013 at 2:47 PM, Ryosuke Niwa rn...@webkit.org wrote: On Tue, Feb 5, 2013 at 6:09 AM, Mark Mentovai m...@chromium.org wrote: The parser (and the grammar) works the way it does because it’s just Python This works great for people who like Python syntax but not for someone like myself who dislikes Python syntax. I also find it particularly annoying that people can use whatever Python constructs they want to use in GYP. It dramatically reduces language portability because you need to support quite a few Python constructs and quirks in order to correctly parse GYP. I personally would have much preferred for it be a simple JSON file. - R. Niwa ___ webkit-dev mailing list webkit-dev@lists.webkit.org https://lists.webkit.org/mailman/listinfo/webkit-dev ___ webkit-dev mailing list webkit-dev@lists.webkit.org https://lists.webkit.org/mailman/listinfo/webkit-dev ___ webkit-dev mailing list webkit-dev@lists.webkit.org https://lists.webkit.org/mailman/listinfo/webkit-dev
Re: [webkit-dev] Gated trunk, experiences from OpenStack
On Tue, Feb 5, 2013 at 3:34 PM, Tim Ansell mit...@mithis.com wrote: On 6 February 2013 07:17, Dirk Pranke dpra...@chromium.org wrote: On Tue, Feb 5, 2013 at 9:46 AM, Martin Robinson mrobin...@webkit.org wrote: On Tue, Feb 5, 2013 at 9:28 AM, Adam Barth aba...@webkit.org wrote: Do you know how they got rid of flakiness in their tests? We've spent a bunch of effort fixing flaky tests (and in marking the remaining flaky tests as flaky), but there's still a long tail of flakiness. I wonder if that sort of thing might be different for OpenStack if they have a different approach to testing than we do. From what I can see they have a pretty similar goal to us. I personally don't know where our test flakyness comes from, so can't really comment on how we could fix it. Another useful thing is to know the number of tests in OpenStack. WebKit has more tests than any other project I've worked on. There are two other related aspects that make our tests flaky: 1) They're very high level integration tests (mostly), which, as they cover large swaths of code in each test, are much more susceptible to flakiness than method-level unit tests. While OpenStack doesn't have anywhere near the number of integration tests WebKit does, it does have large integration tests. Infact, one of their tests brings up a whole cloud stack and checks that you can operate the cluster. 2) They weren't generally written to be run in parallel, and thus we often have to be concerned with system-level resource contention. Neither where OpenStack's originally. They made heavy use of tool called testr ( http://pypi.python.org/pypi/testrepository ) which has a mode to automatically find when two tests are interfering with each other. testr also has a bunch of other useful features, like only re-running tests which are currently failing and keeping a database of test runs and allowing stat collection. Ah, the testr isolation bisection does look interesting. I have done a little work along those lines but haven't gotten very far. -- Dirk ___ webkit-dev mailing list webkit-dev@lists.webkit.org https://lists.webkit.org/mailman/listinfo/webkit-dev
Re: [webkit-dev] Common build system (was Re: WebKit Wishes)
On Sun, Feb 3, 2013 at 9:20 PM, Maciej Stachowiak m...@apple.com wrote: On Feb 3, 2013, at 8:18 PM, Mike Lawther mikelawt...@chromium.org wrote: Hi Maciej! On 31 January 2013 11:15, Maciej Stachowiak m...@apple.com wrote: It's far simplest for us if: (a) There is an Xcode project (or a Makefile) that builds the Mac port checked in to source control. (b) The generated project invokes only tools that are part of the default Mac OS X install. For b), does 'default' include an Xcode install? From my memory of setting up a Mac dev box Xcode was needed to compile. The limitation in (a) is all about how the build fleet is managed, and yes, the machines in question have the Mac OS X user install as well as developer tools installed, and it is ok to rely on that. To clarify: the things that are not kosher are submitting a binary tool as part of the module, downloading things as part of the build, or relying on something to be installed out of band before building. It is ok for the build to rely on scripts, so long as the relevant interpreter is part of the default install. Any such script would have to be in each directory under Source/ since these build separately without access to the source of the others. I should mention that there's a lot of interest right now at Apple in the possibility of switching to Gyp. We think it's likely closer to what we need than CMake, and we suspect there is probably somewhat better probability of modifying it if necessary. If we find someone at Apple who is able to work on such a project in the short term (not completely impossible), would any Google folks be interested in collaborating? Ideally we could use assistance from someone who (a) understands how the Chromium WebKit Gyp-based build works; and (b) is willing and able to change Gyp if necessary. Mark is the gyp-guru, but I would be happy to help out as well. -- Dirk ___ webkit-dev mailing list webkit-dev@lists.webkit.org https://lists.webkit.org/mailman/listinfo/webkit-dev
Re: [webkit-dev] Using ninja (was Re:Common build system (was Re: WebKit Wishes))
The short answer is, you can't. The fastest path is probably to get a working set of gyp files for the apple mac wk2. I'm going to start working on this just to see how far of we are (Adam's work from a year or two ago had JSC and WebCore building, but wasn't too functional beyond that; even so, it's probably a good starting point). -- Dirk On Sun, Feb 3, 2013 at 1:22 PM, Dean Jackson d...@apple.com wrote: OK, this sounds fantastic. And I've noticed how much faster Chromium incrementally builds using ninja when I've done that. So, ignoring the discussion of a single build system for a moment, how can I, as a developer using the OS X + WK2 port, living mostly in Xcode for editing and debugging, use ninja? I need the idiot's guide :) (Note: I am an idiot, but not so much an idiot to realise that the answer involves lots of work and probably updating some old GYP files that you and Adam were testing with, etc etc. I'm just selfishly thinking that cutting even 30s off each incremental rebuild would make me so much happier that I'd be willing to put up with other inconveniences.) Dean On 03/02/2013, at 4:54 PM, Eric Seidel e...@webkit.org wrote: +1 Ninja is beyond-words amazing. http://martine.github.com/ninja/ For better or worse, it is not designed to use human-editable build files, but rather to be used by a meta build system, like GYP or CMake. So using ninja is really an orthogonal discussion to the single build system discussion for WebKit. :) Were the WebKit project to convert to using a single meta-build system, ninja would become an option many users might choose. I'm told most Chromium hackers have GYP set to output ninja files these days, with the exception of some folks who still want the MSVC build environment. For WebKit ports already using CMake, they should definitely try ninja today! Anyway, my wish was not about arguing for a specific build solution. I'm instead noting that for the project to continue to move quickly, we need to stop needing to edit 8 build systems for every file move/addition. Whether that's GYP or CMake or something else, I don't really care. Adam and I tried GYP-for-WebKIt a while back. But any of these solutions will require buy-in from Apple, as they will have to do the largest amount of work converting to use something other than XCode. On Sat, Feb 2, 2013 at 8:20 PM, Nico Weber tha...@chromium.org wrote: On Sat, Feb 2, 2013 at 4:58 PM, Adam Barth aba...@webkit.org wrote: Ninja has extremely fast incremental builds and can be generated by GYP. Here are some stats from a year ago: https://plus.google.com/101038813433650812235/posts/irc26fhRtPC Ninja has gotten even faster since then. If you're interested in trying it out, you can play around with incremental builds of the Chromium port on Mac or Linux. You can also look at the build output from the chromium bots. Empty build in 1s: http://build.webkit.org/builders/Chromium%20Linux%20Release/builds/66807/steps/compile-webkit/logs/stdio Build with a few files changed in 15s: http://build.webkit.org/builders/Chromium%20Linux%20Release/builds/66800/steps/compile-webkit/logs/stdio …and this is on fairly slow bots. On my SSD-equipped laptop, I can do incremental rebuilds of all of chrome after touching one (cpp or mm) file in 2-6s. ___ webkit-dev mailing list webkit-dev@lists.webkit.org https://lists.webkit.org/mailman/listinfo/webkit-dev ___ webkit-dev mailing list webkit-dev@lists.webkit.org https://lists.webkit.org/mailman/listinfo/webkit-dev
Re: [webkit-dev] Common build system (was Re: WebKit Wishes)
On Thu, Jan 31, 2013 at 11:48 AM, Hugo Parente Lima hugo.l...@openbossa.org wrote: On Wednesday, January 30, 2013 04:15:48 PM Maciej Stachowiak wrote: On Jan 30, 2013, at 3:24 PM, Dirk Pranke dpra...@chromium.org wrote: On Wed, Jan 30, 2013 at 1:50 PM, Filip Pizlo fpi...@apple.com wrote: Thanks for sharing this. On Jan 30, 2013, at 1:28 PM, Eric Seidel e...@webkit.org wrote: I wish we only had one build system (it were easy to add/remove/move files). I believe changes like http://trac.webkit.org/changeset/74849 are an unhealthy sign for the project. Adam is not the only person who has chosen to empty files instead of removing them. The pain of updating 8 build system is so great, we jump through hoops to avoid it. Which means it took us months to move JavaScriptCore/wtf to WTF, and will take us years to kill WebCore/platform. +1 This is a hard problem. It is a problem worth solving. Do you have more thoughts on this, particularly since you know quite well how both Xcode and gyp work? I suspect this is one of those things that it would be hard to achieve consensus on since there are so many stakeholders. But it may be fruitful to have a what if discussion about what this might look like. I think we can solve this problem if we agree that we want to. I think we haven't in the past mostly because we couldn't reach a consensus that it was worth solving enough to really try. I would love to see this fixed and would be glad to work on it. I think we should at least pursue this far enough to fully understand what our options are and what the costs and tradeoffs might be; does anyone disagree, and is anyone else willing to help pitch in? I think there are several possible ways we could solve this. One would be to switch to a common meta-build system. My understanding is that Apple's internal production build processes impose certain constraints here that I don't fully understand, but I know we've discussed the possibility of checking in generated project files as a workaround. Maybe there are other options as well to those constraints? I would love to discuss this further w/ someone from Apple ... It's far simplest for us if: (a) There is an Xcode project (or a Makefile) that builds the Mac port checked in to source control. (b) The generated project invokes only tools that are part of the default Mac OS X install. It may not be completely impossible to violate these requirements but it will require a lot of bureaucracy. (Also, just to get this out of the way, I don't think gyp needs to be the solution). Another alternative would be to write a script that did support at least the common use cases (add/move/delete files). There have been attempts in the past, but they have foundered on at least some perceived skepticism over how well this would work w/ XCode. That said, I don't think we've really pushed this to see. At some point this script might turn into a meta-meta-build system, which might be silly but also be the shortest path to the finish line. I suggest if there is interest in this we can start a new thread to discuss further ... My preference would be to use a common meta-build system with a comfortably human-readable and human-editable syntax, and checked in generated project files for those ports that need them. I think a key to making this work is to get Chromium and the Apple Mac port onto a common build system, which will probably require both Google and Apple ponying up at least one person to work on this project for a reasonable period of time. I think the plausible meta-build-systems to use would be CMake and Gyp, but in both cases it may be necessary to modify them to work well for everyone. I'd also like to add that I think a key related issue to common build system is common feature configuration. The many different ways ports control their feature flags is super confusing. I've long wanted to implement common configuration management, but have not had time. I have a patch trying to solve this issue for CMake based ports[1], the patch still on going, but even a change affecting just 2-3 ports using the same build system is a bit hard to get a consensus, so you can imagine how hard will be to get a consensus over all WebKit ports. [1] https://bugs.webkit.org/show_bug.cgi?id=103162 This is slightly off-topic, but I had thought that no one was actually using CMake; maybe I was mistaken and just none of the ports that build on webkit.org are? It looks like Blackberry and maybe a WinCE port uses CMake? Anyone else? -- Dirk ___ webkit-dev mailing list webkit-dev@lists.webkit.org https://lists.webkit.org/mailman/listinfo/webkit-dev
Re: [webkit-dev] Common build system (was Re: WebKit Wishes)
On Thu, Jan 31, 2013 at 12:10 PM, Patrick Gansterer par...@paroga.com wrote: Am 31.01.2013 um 21:07 schrieb Dirk Pranke: On Thu, Jan 31, 2013 at 11:48 AM, Hugo Parente Lima hugo.l...@openbossa.org wrote: On Wednesday, January 30, 2013 04:15:48 PM Maciej Stachowiak wrote: On Jan 30, 2013, at 3:24 PM, Dirk Pranke dpra...@chromium.org wrote: On Wed, Jan 30, 2013 at 1:50 PM, Filip Pizlo fpi...@apple.com wrote: Thanks for sharing this. On Jan 30, 2013, at 1:28 PM, Eric Seidel e...@webkit.org wrote: I wish we only had one build system (it were easy to add/remove/move files). I believe changes like http://trac.webkit.org/changeset/74849 are an unhealthy sign for the project. Adam is not the only person who has chosen to empty files instead of removing them. The pain of updating 8 build system is so great, we jump through hoops to avoid it. Which means it took us months to move JavaScriptCore/wtf to WTF, and will take us years to kill WebCore/platform. +1 This is a hard problem. It is a problem worth solving. Do you have more thoughts on this, particularly since you know quite well how both Xcode and gyp work? I suspect this is one of those things that it would be hard to achieve consensus on since there are so many stakeholders. But it may be fruitful to have a what if discussion about what this might look like. I think we can solve this problem if we agree that we want to. I think we haven't in the past mostly because we couldn't reach a consensus that it was worth solving enough to really try. I would love to see this fixed and would be glad to work on it. I think we should at least pursue this far enough to fully understand what our options are and what the costs and tradeoffs might be; does anyone disagree, and is anyone else willing to help pitch in? I think there are several possible ways we could solve this. One would be to switch to a common meta-build system. My understanding is that Apple's internal production build processes impose certain constraints here that I don't fully understand, but I know we've discussed the possibility of checking in generated project files as a workaround. Maybe there are other options as well to those constraints? I would love to discuss this further w/ someone from Apple ... It's far simplest for us if: (a) There is an Xcode project (or a Makefile) that builds the Mac port checked in to source control. (b) The generated project invokes only tools that are part of the default Mac OS X install. It may not be completely impossible to violate these requirements but it will require a lot of bureaucracy. (Also, just to get this out of the way, I don't think gyp needs to be the solution). Another alternative would be to write a script that did support at least the common use cases (add/move/delete files). There have been attempts in the past, but they have foundered on at least some perceived skepticism over how well this would work w/ XCode. That said, I don't think we've really pushed this to see. At some point this script might turn into a meta-meta-build system, which might be silly but also be the shortest path to the finish line. I suggest if there is interest in this we can start a new thread to discuss further ... My preference would be to use a common meta-build system with a comfortably human-readable and human-editable syntax, and checked in generated project files for those ports that need them. I think a key to making this work is to get Chromium and the Apple Mac port onto a common build system, which will probably require both Google and Apple ponying up at least one person to work on this project for a reasonable period of time. I think the plausible meta-build-systems to use would be CMake and Gyp, but in both cases it may be necessary to modify them to work well for everyone. I'd also like to add that I think a key related issue to common build system is common feature configuration. The many different ways ports control their feature flags is super confusing. I've long wanted to implement common configuration management, but have not had time. I have a patch trying to solve this issue for CMake based ports[1], the patch still on going, but even a change affecting just 2-3 ports using the same build system is a bit hard to get a consensus, so you can imagine how hard will be to get a consensus over all WebKit ports. [1] https://bugs.webkit.org/show_bug.cgi?id=103162 This is slightly off-topic, but I had thought that no one was actually using CMake; maybe I was mistaken and just none of the ports that build on webkit.org are? It looks like Blackberry and maybe a WinCE port uses CMake? Anyone else? EFL uses CMake too. 4 EFL bots @ http://build.webkit.org 1 WinCE bot @ http://build.webkit.org 1 EFL bot as EWS Ah, I thought EFL was using Autotools. Thanks for the correction. -- Dirk ___ webkit-dev
Re: [webkit-dev] Common build system (was Re: WebKit Wishes)
On Thu, Jan 31, 2013 at 9:14 AM, Patrick Gansterer par...@paroga.com wrote: Am 31.01.2013 um 17:17 schrieb Maciej Stachowiak: On Jan 31, 2013, at 1:56 AM, Patrick Gansterer par...@paroga.com wrote: Am 31.01.2013 um 10:37 schrieb Ryosuke Niwa: On Thu, Jan 31, 2013 at 1:17 AM, Jochen Eisinger joc...@chromium.org wrote: Another option is to add a webkit-patch command for modifying the build files. That way, the syntax doesn't need to be overly human friendly. There was also some attempt to write a tool to add files automatically: https://bugs.webkit.org/show_bug.cgi?id=61772 I would expect that such a tool becomes easier if it would only modify one source of truth and generates all other artifacts such as Xcode projects from it. I don't want build file's syntax to be so human unfriendly that I need a tool for it. Often times, these syntax problems can be improved dramatically by simple changes. e.g. we had a similar discussion about TestExpectation syntax, and I'm much happier with the new syntax even though the new syntax is functionally equivalent to the old one, and two syntaxes are very similar. On Thu, Jan 31, 2013 at 1:17 AM, Mark Rowe mr...@apple.com wrote: I’ve experimented with this in the past and you’re right that it shouldn’t be particularly difficult to do. However, I suspect that the task would be similar in scope to defining an improved syntax for gyp. And if the syntax is the primary sticking point with gyp then it’d seem preferable to tackle initially. Yeah. In fact, we can just come up with whatever syntax we like and convert it to the existing gyp format if the syntax was the biggest issue. Do we want to define the whole build system (including information how to invoke the generators) with the new system, or is a simple list for the input files sufficient? IMHO adding a new generator build step happens very rarely. So maybe we can spit the input file list (mainly *.cpp and *.idl) into new files. Then GYP and CMake can read them and generate the build system out of them directly (like they to already today) instead of listing the files in the *.gpyi and *.cmake. This might work for other systems like qmake too. For XCode we can maybe have a template XCode project and generate the work XCode project with a script. This script then only need to fill in the files from the input file list into the template XCode project. Defining the feature flags can be done like Maciej suggested with Port.h files. I think it would be better to adapt an existing meta build system to our needs than to make one from scratch, unless we find that completely impractical. In particular, gyp and cmake both know how to handle generated sources, and while it may not be super common to make a new type of generated source, it's bad for hackability of the project of doing so is super hard. We get a lot of hackability benefits from using various kinds of generated sources, many first introduced in the days when we had a lot fewer build systems. So in my mind, they are already ahead of a hypothetical simple system. Do you want to kick the requirements of the smaller ports from trunk or do you think that e.g. a qmake generate for GYP makes sense? AFAIK e.g. QtWebKit is shipped with Qt, which uses qmake as build system, where CMake/GYP is not an option. It could certainly make sense for us to add Autotools and Qmake generators to GYP; I'm less certain if a CMake generator makes much sense, but I haven't thought about it as much. I'm not super familiar with any of these three tools, so I could be dead wrong. I completely agree that creating a new meta meta build system isn't a good idea, but sharing the common parts (which reduce the daily productivity) might be a step in the right direction. Using simple text files which contain the list of files (like the gpyi files already do today) isn't a new build system. It only offers the existing meta build systems (CMake, GYP, autotools, qmake) to use a common base. The remaining build systems can be ported to one of these systems or be adopted to use this file lists too. -- Patrick I suspect that we would quickly find that we would want some sort of support for conditionals and/or file inclusion in our simple text files, at which point you basically get a meta-meta-build system :). I don't actually think such a thing is that bad of an idea, but it's all in the details. I would like to find a solution where all of the ports were able to retain integration with their native build environments one way or another. -- Dirk ___ webkit-dev mailing list webkit-dev@lists.webkit.org https://lists.webkit.org/mailman/listinfo/webkit-dev
Re: [webkit-dev] Exporting symbols (was Re: Common build system (was Re: WebKit Wishes))
On Thu, Jan 31, 2013 at 9:48 AM, Ryosuke Niwa rn...@webkit.org wrote: On Thu, Jan 31, 2013 at 9:43 AM, Alexey Proskuryakov a...@apple.com wrote: 31.01.2013, в 0:17, Ryosuke Niwa rn...@webkit.org написал(а): I did it for WTF and JSC with kevino@. https://bugs.webkit.org/show_bug.cgi?id=72855 I hope I had had time to extend the work to WebCore, which was my original goal :-/ I did it through an automation but the tool was so slow and hacky that it was not capable for large WebCore codebase. Seeing that the number of exported symbol is about 2700, manual work with regular expression will also work if it is OK to do the work incrementally... and if you are patient enough :-) 2700 is a lot of symbols to export! Perhaps we can reduce the number of exported symbols? For example, we can move more code from WebKit into WebCore to share more code between ports so that the API surface WebCore has to expose will be smaller and more well defined. One thing to keep an eye on with WEBCORE_EXPORT is that it can increase the number of exports for each port, because it would export symbols that are only needed for other ports. I'm not sure I understand the concern here. Is the idea that some ports might build WebCore as a DLL (and need exports) and others wouldn't? Or the idea that the API that is needed to be exported might vary from port to port in such a way that too many symbols for WebCore might be exported on some ports? Can we make the macro starter and specify ports under which a given symbol needs to be exported? e.g. WEBCORE_EXPORT(MAC, QT) I'm not sure how this helps, but I think that's because of my previous comment. I could see something like this maybe being needed in my later scenario (where MAC might need a symbol exported from WebCore that Qt needed to keep from being exported)? Is that a hard requirement, or an optimization? -- Dirk ___ webkit-dev mailing list webkit-dev@lists.webkit.org https://lists.webkit.org/mailman/listinfo/webkit-dev
Re: [webkit-dev] Exporting symbols (was Re: Common build system (was Re: WebKit Wishes))
On Thu, Jan 31, 2013 at 2:52 PM, Alexey Proskuryakov a...@apple.com wrote: 31.01.2013, в 14:45, Dirk Pranke dpra...@chromium.org написал(а): One thing to keep an eye on with WEBCORE_EXPORT is that it can increase the number of exports for each port, because it would export symbols that are only needed for other ports. I'm not sure I understand the concern here. Is the idea that some ports might build WebCore as a DLL (and need exports) and others wouldn't? Or the idea that the API that is needed to be exported might vary from port to port in such a way that too many symbols for WebCore might be exported on some ports? The latter. I suspect that it's somewhat less of an issue for JavaScriptCore, because its exports are mostly used from cross-platform code. I see. Some concrete examples would be helpful here (to me, at least). What components and ports are we talking about, and what sorts of symbols? -- Dirk ___ webkit-dev mailing list webkit-dev@lists.webkit.org https://lists.webkit.org/mailman/listinfo/webkit-dev
Re: [webkit-dev] Exporting symbols (was Re: Common build system (was Re: WebKit Wishes))
On Thu, Jan 31, 2013 at 3:12 PM, Alexey Proskuryakov a...@webkit.org wrote: 31.01.2013, в 14:57, Dirk Pranke dpra...@chromium.org написал(а): One thing to keep an eye on with WEBCORE_EXPORT is that it can increase the number of exports for each port, because it would export symbols that are only needed for other ports. I'm not sure I understand the concern here. Is the idea that some ports might build WebCore as a DLL (and need exports) and others wouldn't? Or the idea that the API that is needed to be exported might vary from port to port in such a way that too many symbols for WebCore might be exported on some ports? The latter. I suspect that it's somewhat less of an issue for JavaScriptCore, because its exports are mostly used from cross-platform code. I see. Some concrete examples would be helpful here (to me, at least). What components and ports are we talking about, and what sorts of symbols? I don't have concrete examples, and I don't know how big an effect on performance increasing the number of exports would have. It's something to keep an eye on, not a reason to avoid trying. I'm not parsing your reply properly -- avoid trying what? A common EXPORT macro that is set as appropriate by each port? -- Dirk ___ webkit-dev mailing list webkit-dev@lists.webkit.org https://lists.webkit.org/mailman/listinfo/webkit-dev
Re: [webkit-dev] WebKit Wishes
On Wed, Jan 30, 2013 at 1:50 PM, Filip Pizlo fpi...@apple.com wrote: Thanks for sharing this. On Jan 30, 2013, at 1:28 PM, Eric Seidel e...@webkit.org wrote: I wish we only had one build system (it were easy to add/remove/move files). I believe changes like http://trac.webkit.org/changeset/74849 are an unhealthy sign for the project. Adam is not the only person who has chosen to empty files instead of removing them. The pain of updating 8 build system is so great, we jump through hoops to avoid it. Which means it took us months to move JavaScriptCore/wtf to WTF, and will take us years to kill WebCore/platform. +1 This is a hard problem. It is a problem worth solving. Do you have more thoughts on this, particularly since you know quite well how both Xcode and gyp work? I suspect this is one of those things that it would be hard to achieve consensus on since there are so many stakeholders. But it may be fruitful to have a what if discussion about what this might look like. I think we can solve this problem if we agree that we want to. I think we haven't in the past mostly because we couldn't reach a consensus that it was worth solving enough to really try. I would love to see this fixed and would be glad to work on it. I think we should at least pursue this far enough to fully understand what our options are and what the costs and tradeoffs might be; does anyone disagree, and is anyone else willing to help pitch in? I think there are several possible ways we could solve this. One would be to switch to a common meta-build system. My understanding is that Apple's internal production build processes impose certain constraints here that I don't fully understand, but I know we've discussed the possibility of checking in generated project files as a workaround. Maybe there are other options as well to those constraints? I would love to discuss this further w/ someone from Apple ... (Also, just to get this out of the way, I don't think gyp needs to be the solution). Another alternative would be to write a script that did support at least the common use cases (add/move/delete files). There have been attempts in the past, but they have foundered on at least some perceived skepticism over how well this would work w/ XCode. That said, I don't think we've really pushed this to see. At some point this script might turn into a meta-meta-build system, which might be silly but also be the shortest path to the finish line. I suggest if there is interest in this we can start a new thread to discuss further ... -- Dirk ___ webkit-dev mailing list webkit-dev@lists.webkit.org https://lists.webkit.org/mailman/listinfo/webkit-dev
Re: [webkit-dev] build.webkit.org down
This is the output of the test bots, right? I'm guessing this is due to (a) us running pixel tests and (b) not checking in the expected failing baselines. If I ever get back to working on supporting -failing.png, this could go down *a lot*. Alternatively, we could just decide to check in the existing failures as -expected and call it good enough ... -- Dirk On Fri, Jan 25, 2013 at 9:35 AM, Eric Seidel e...@webkit.org wrote: I'm surprised that chromium mac is 3x the size of Apple Mac... debug even! Chromium build output should be much smaller... at least as of late. On Fri, Jan 25, 2013 at 9:30 AM, William Siegrist wsiegr...@apple.com wrote: Here are the largest results sets on the master in GB. We currently store 6.8TB of total results, going back 14 months, and 1.1TB of archives going back 1 week. 1500Apple MountainLion (Leaks) 1018Chromium Mac Release (Tests) 857 Chromium Linux Release (Tests) 532 Chromium Win Release (Tests) 371 Apple MountainLion Debug WK2 (Tests) 324 Apple Lion Debug WK2 (Tests) 299 Chromium Linux Release (Grid Layout) 173 GTK Linux 64-bit Release 171 Chromium Android Release (Tests) 160 EFL Linux 64-bit Debug WK2 158 Apple Lion (Leaks) 145 EFL Linux 64-bit Release 113 EFL Linux 64-bit Debug 105 GTK Linux 32-bit Release 94 GTK Linux 64-bit Debug 94 GTK Linux 64-bit Release WK2 (Tests) 85 EFL Linux 64-bit Release WK2 80 Apple Lion Release WK1 (Tests) 77 Apple Lion Release WK2 (Tests) 60 Apple Lion Debug WK1 (Tests) 60 Apple MountainLion Debug WK1 (Tests) 59 Apple MountainLion Release WK1 (Tests) 53 Qt Linux Release Archives are already pruning after 1 week. Next week, I'm going to start pruning results older than 14 months since we don't have much more space. If anyone thinks you need more than 14 months of results, let me know soon and we can see what our options are. -Bill ___ webkit-dev mailing list webkit-dev@lists.webkit.org http://lists.webkit.org/mailman/listinfo/webkit-dev ___ webkit-dev mailing list webkit-dev@lists.webkit.org http://lists.webkit.org/mailman/listinfo/webkit-dev ___ webkit-dev mailing list webkit-dev@lists.webkit.org http://lists.webkit.org/mailman/listinfo/webkit-dev
Re: [webkit-dev] Please avoid rolling out patches speculatively and reland them ASAP if you had to
The build cop / gardener / sheriff / whatever may not have local or easy access to a bot that reproduces the problem ... rolling it out might be the only feasible way to test in that case. On Tue, Dec 11, 2012 at 2:20 PM, Oliver Hunt oli...@apple.com wrote: On Dec 11, 2012, at 2:17 PM, Peter Kasting pkast...@chromium.org wrote: On Tue, Dec 11, 2012 at 2:14 PM, Oliver Hunt oli...@apple.com wrote: I don't understand why anyone is _speculatively_ rolling out patches. You should only be rolling it out if you _know_ the patch is bad. Sometimes something bad happens to the tree, the sheriff doesn't know which patch is responsible, and the change authors are not present to ask for help. In a case like this the sheriff has to either do speculative rollouts or leave the tree broken. Ideally, of course, change authors are around when something like this happens. But maybe the bustage doesn't happen until much later, due to some subtle/latent issue, or maybe the change author is in fact irresponsible. Or the sheriff could actually see if rolling out a patch locally fixes the problem. I'm not sure why they're considering not testing to be a valid behaviour for someone who is ostensibly meant to be keeping things going in the face of people who aren't testing. PK ___ webkit-dev mailing list webkit-dev@lists.webkit.org http://lists.webkit.org/mailman/listinfo/webkit-dev ___ webkit-dev mailing list webkit-dev@lists.webkit.org http://lists.webkit.org/mailman/listinfo/webkit-dev
Re: [webkit-dev] Proposal: Make Ninja the default build-system for build-webkit --chromium
On Mon, Dec 10, 2012 at 1:30 AM, Jochen Eisinger joc...@chromium.org wrote: Will the buildbots use ninja or the native build tools? My only concern is that we're catching problems with e.g. MSVS only after we roll the WebKit deps in chromium and one of the MSVS bots starts failing. Eric is only suggesting changing update-webkit and build-webkit, which means that only the webkit.org bots would be affected. As long as the chromium.org canaries are still using chromium checkouts (and the native build systems), we'll still have coverage. Of course, your point is still valid for other scenarios where we don't have coverage of what we use on the official builds, but as Nico pointed out in a separate thread, so far this hasn't been a frequent problem. -- Dirk ___ webkit-dev mailing list webkit-dev@lists.webkit.org http://lists.webkit.org/mailman/listinfo/webkit-dev
Re: [webkit-dev] r- your own patches [was: Re: RenderArena: Teaching an old dog new tricks]
On Fri, Nov 16, 2012 at 7:42 AM, Julien Chaffraix julien.chaffr...@gmail.com wrote: Seconded. I also think only the one who submitted the patch can clear the r? flag. Others should NOT do that, please, even you are a reviewer. You can r- the patch if you believe it is bad. I disagree with that. You seem to think that patches falls into either good or bad. However the reality is more complex and there are levels of goodness and badness. I use r- for patches that I really think are not in the right direction or shouldn't be landed: it is a statement in this direction. Clearing the flag is for patches that are close enough but still not up to our standards and that I want to kick off the review queue. This is my reasoning on that and other people likely have different views. However I don't think it's unreasonable to clear the flag instead of r-'ing in some contexts. I would be very confused if anyone but me cleared the r? on one of my bugs, unless it was obvious that the patch was old and buried and I had just forgotten about it. If I think a patch has a few flaws but is on the right track (and I just have questions), I will often leave the r? as-is and expect the contributor to figure it out (meaning that someone else might then be okay to r+ it. I will usually only r- a patch if I think the patch should not be landed as-is and I don't want someone else to r+ it. -- Dirk ___ webkit-dev mailing list webkit-dev@lists.webkit.org http://lists.webkit.org/mailman/listinfo/webkit-dev
Re: [webkit-dev] commit-queue bot missing a dependency? (ImportError: No module named logilab.astng.utils)
Argh, yeah. Unfortunately, I'm not sure what triggers this problem. It's easy enough to fix as soon as I can find someone w/ access to the CQ. -- Dirk On Tue, Nov 13, 2012 at 7:50 AM, Adam Barth aba...@webkit.org wrote: Sounds related to a recent changed made by dpranke. Adam On Tue, Nov 13, 2012 at 7:33 AM, Jussi Kukkonen jussi.kukko...@intel.com wrote: commit-queue bot seems to be missing a pylint dependency? This is what I got for my rebaseline in https://bugs.webkit.org/show_bug.cgi?id=102059: --- Failed to run ['/mnt/git/webkit-commit-queue/Tools/Scripts/webkit-patch', '--status-host=queues.webkit.org', '-... exit_code: 2 Last 500 characters of output: yle/checkers/python.py, line 31, in module from webkitpy.thirdparty.autoinstalled.pylint import lint File /mnt/git/webkit-commit-queue/Tools/Scripts/webkitpy/thirdparty/autoinstalled/pylint/lint.py, line 31, in module from pylint.checkers import utils File /mnt/git/webkit-commit-queue/Tools/Scripts/webkitpy/thirdparty/autoinstalled/pylint/checkers/__init__.py, line 44, in module from logilab.astng.utils import ASTWalker ImportError: No module named logilab.astng.utils Full output: http://queues.webkit.org/results/14823414 --- - jussi ___ webkit-dev mailing list webkit-dev@lists.webkit.org http://lists.webkit.org/mailman/listinfo/webkit-dev ___ webkit-dev mailing list webkit-dev@lists.webkit.org http://lists.webkit.org/mailman/listinfo/webkit-dev
Re: [webkit-dev] generic test expectations?
We don't currently support port-specific reftests (or at least, not very well), and we certainly should be trying to minimize where they occur. Perhaps we should discuss why you need them (in a separate thread with a separate subject line)? It sounds like this largely has to do with what features are enabled and/or supported? Perhaps said tests should just be skipped in the meantime? At any rate, we encourage people these days to check in expected failures rather than suppressing them using the TestExpectations files. I am hoping this week to finally get back to working on the -failing feature so you can at least distinguish expectations that are known to be failing/incorrect more easily. -- Dirk On Tue, Nov 13, 2012 at 11:02 AM, Glenn Adams gl...@skynav.com wrote: That would seem to work only if the test(s) fail the same way on all ports. In the case that I'm working from, I'm using reftests, where I know the correct expectations, but the actual behavior will (does) differ on different ports (when the corresponding feature is committed). I would like to be able to (independently) commit new reftests *and* their known good expectation counterparts (that should apply on all ports), and then subsequently commit an independent patch that implements the expected behavior (on some but not all ports), and the comment further follow-on patches that fix the failing ports (possibly by writing new expectations for those specific ports). On Tue, Nov 13, 2012 at 11:52 AM, Ryosuke Niwa rn...@webkit.org wrote: It is customary to add a failing test expectation (i.e. *-expected.txt file that contains the said failure) in such cases. On Tue, Nov 13, 2012 at 10:49 AM, Glenn Adams gl...@skynav.com wrote: Just checking, but I don't see a way to add test expectations that apply generically (to all ports). It would be nice to have something like LayoutTests/platform/generic/TestExpectations to which one could add new tests that are known to fail everywhere (e.g., because the code that implements a feature that is tested by those tests is not yet committed), but which will (at some point in the near future) not fail (when the code that is to be tested is committed). At present, it seems that if one wishes to do this, then it is necessary to add entries to the each base port expectations (i.e., chromium, mac, win, etc), which is rather annoying. If there is no objection to adding such a generic platform expectations file, then I will undertake to do so. ___ webkit-dev mailing list webkit-dev@lists.webkit.org http://lists.webkit.org/mailman/listinfo/webkit-dev ___ webkit-dev mailing list webkit-dev@lists.webkit.org http://lists.webkit.org/mailman/listinfo/webkit-dev
Re: [webkit-dev] generic test expectations?
On Tue, Nov 13, 2012 at 11:29 AM, Glenn Adams gl...@skynav.com wrote: On Tue, Nov 13, 2012 at 12:09 PM, Dirk Pranke dpra...@chromium.org wrote: We don't currently support port-specific reftests (or at least, not very well), and we certainly should be trying to minimize where they occur. Hmm, I actually used port specific reftest expectation files in a recent patch [1] (since rolled out), and it appeared to work (as a way to effectively rebaseline those expectations). So something seems to be working. [1] http://trac.webkit.org/changeset/133529 I expect it'll sort of work, but it won't be robust; you may hit weird behavior and/or bugs. We really haven't beaten on this aspect of things, and I don't know yet how much we want to. At any rate, we encourage people these days to check in expected failures rather than suppressing them using the TestExpectations files. The problem is essentially a chicken and egg problem. I don't know what the per-port failures will be ahead of time, but I do know the set of correct expectations. Since I am (independently) unable to build/test all ports run by build bots, I would like to commit the set of tests plus known good expectations as a preliminary step (with a generic skip all tests for all ports), and then subsequently commit the feature itself, and then subsequently override the generic skip on a port specific basis, effectively re-enabling the tests on a port by port basis as I refine the feature patch (as needed) to handle port specific behavioral differences. I think this is a reasonable approach. I would be interested to hear if others had alternatives they preferred. -- Dirk ___ webkit-dev mailing list webkit-dev@lists.webkit.org http://lists.webkit.org/mailman/listinfo/webkit-dev
[webkit-dev] do you want WK1 and WK2 keywords in the TestExpectations files?
Hi all, I occasionally get asked if the TestExpectations syntax supports a way to distinguish different results for WebKit1 and WebKit2 via keywords. We currently don't do this, and different ports have worked around this in slightly different ways by using dedicated wk2-specific TestExpectations files and sometimes wk1-specific TestExpectations files. However, this is a little awkward and gets worse if you also need to support different expectations for multiple different configs (e.g., mac-lion vs mac-snowleopard vs mac-mountainlion). So, it seems like WK1 and WK2 keywords might be useful. However, I don't really want to add more divergence between ports, so it would be nice to have everyone agree to use it if we were to add it. What do you all think? Would you like this feature, and would you all use it ? (Since I don't regularly switch between WK1 and WK2 I don't have a strong feeling here beyond what I've written above). -- Dirk ___ webkit-dev mailing list webkit-dev@lists.webkit.org http://lists.webkit.org/mailman/listinfo/webkit-dev
Re: [webkit-dev] generic test expectations?
On Tue, Nov 13, 2012 at 12:02 PM, Tony Chang t...@chromium.org wrote: On Tue, Nov 13, 2012 at 11:35 AM, Dirk Pranke dpra...@chromium.org wrote: On Tue, Nov 13, 2012 at 11:29 AM, Glenn Adams gl...@skynav.com wrote: On Tue, Nov 13, 2012 at 12:09 PM, Dirk Pranke dpra...@chromium.org wrote: We don't currently support port-specific reftests (or at least, not very well), and we certainly should be trying to minimize where they occur. Hmm, I actually used port specific reftest expectation files in a recent patch [1] (since rolled out), and it appeared to work (as a way to effectively rebaseline those expectations). So something seems to be working. [1] http://trac.webkit.org/changeset/133529 I expect it'll sort of work, but it won't be robust; you may hit weird behavior and/or bugs. We really haven't beaten on this aspect of things, and I don't know yet how much we want to. I don't think we should support port specific ref test results. That kind of misses the point of using a ref test in the first place. I mean, you may as well check in port specific pixel results which are easier to review for correctness. It may be the case that a ref test is not appropriate for what you're trying to test. I think that there are probably cases where we will have differences in results because of (legal and entirely correct or permissible) differences in the implementations and in this case a reftest can still be an improvement over maintaining N platform-specific pixel versions. The obvious example is when we haven't implemented features yet (or have bugs). The W3C's spec for handling reftests also gives you a way to say a test passes if any of these N references may match, which is fairly consistent with the idea that platform specific references are okay in some cases. As to whether pixel-tests are easier to review for correctness than reftests or not, I think it depends on the test. -- Dirk ___ webkit-dev mailing list webkit-dev@lists.webkit.org http://lists.webkit.org/mailman/listinfo/webkit-dev
Re: [webkit-dev] do you want WK1 and WK2 keywords in the TestExpectations files?
On Tue, Nov 13, 2012 at 12:29 PM, Dirk Pranke dpra...@chromium.org wrote: On Tue, Nov 13, 2012 at 12:13 PM, Darin Adler da...@apple.com wrote: I’d prefer an directory-based overlay/inheritance approach to sharing WebKit1- vs. WebKit2-specific expectations over in-file keywords. I’d like to share more than just a central expectations file, so we could share expected results or even WebKit1 or WebKit2-specific tests. We pretty much have this today (with platform/wk2 and platform/mac-wk2). You're saying you'd prefer to add platform/wk1, platform/mac-wk1, platform/mac-lion-wk1, and platform/mac-lion-wk2 if/where necessary (and no keywords), right? One specific example to motivate this ... imagine a test that we want to skip everywhere except current (and future) mac wk2. This would require a Skip in platform/mac/TestExpectations, a Pass in platform/mac-wk2/TestExpectations, and then a Skip in (a newly created, since we don't currently have this) platform/mac-lion-wk2/TestExpectations. -- Dirk ___ webkit-dev mailing list webkit-dev@lists.webkit.org http://lists.webkit.org/mailman/listinfo/webkit-dev
[webkit-dev] sharing more test references
Hi all, Currently, we have ~8000 pixel tests in the tree, and ~800 reference tests. I would like to make that first number a lot smaller. It turns out that we have a reasonably large number of tests that produce the exact same pixel results. On chromium-mac on 10.8, for example, there are 2048 tests that share a result with some other test. 50 of them, for example, draw a green square in the upper left corner of the page (e.g., for svg/custom/root-element.html). The way ref tests are currently implemented, there is no direct way to say use test X as a reference for test Y (although this is in the W3C specs for ref tests, which suggests using link tags in the test html). Previous conversations on this topic have concluded with us thinking that we don't want to give up the convention of having an -expected file next to each test, and I still think this is a good idea. However, we can probably have our cake and eat it too in many cases by simply doing something like creating an -expected.html file that contains solely iframe src=path-to-X. So, I would like to : 1) document that as an accepted pattern somewhere 2) start building up a directory of shared references, e.g., LayoutTests/references/green-square.html 3) start converting existing pixel tests to use these. Anyone think this won't, work, otherwise object, or have better ideas? -- Dirk ___ webkit-dev mailing list webkit-dev@lists.webkit.org http://lists.webkit.org/mailman/listinfo/webkit-dev
Re: [webkit-dev] sharing more test references
On Tue, Nov 13, 2012 at 4:33 PM, Darin Adler da...@apple.com wrote: On Nov 13, 2012, at 4:01 PM, Dirk Pranke dpra...@chromium.org wrote: It turns out that we have a reasonably large number of tests that produce the exact same pixel results. On chromium-mac on 10.8, for example, there are 2048 tests that share a result with some other test. 50 of them, for example, draw a green square in the upper left corner of the page (e.g., for svg/custom/root-element.html). It seems to me that when we find a pattern like this, we should create a hand-coded reference test result. The fact that so many tests produce the same pixels means that each reference can be used to move a lot of tests from the pixel test to the reference test category. I don’t think we need to do that iframe thing you said, as long as there are large sets of tests with the same result, since if the payoff for each reference is big enough, it should be affordable to hand-write the reference file. I don't think I'm understanding you. Wouldn't the fact that there are a large set of tests with the same result be an argument *for* doing the iframe thing? What is the advantage to having 50 copies of a hand-coded green square in upper left corner reference test? -- Dirk ___ webkit-dev mailing list webkit-dev@lists.webkit.org http://lists.webkit.org/mailman/listinfo/webkit-dev
Re: [webkit-dev] sharing more test references
On Tue, Nov 13, 2012 at 4:59 PM, Darin Adler da...@apple.com wrote: On Nov 13, 2012, at 4:56 PM, Dirk Pranke dpra...@chromium.org wrote: Wouldn't the fact that there are a large set of tests with the same result be an argument *for* doing the iframe thing? The simple hand-coded green square in upper left corner should be simple, perhaps even simpler than the iframe thing. What is the advantage to having 50 copies of a hand-coded green square in upper left corner reference test? Tests standing alone and being independent, easy to move around, revise, and understand individually rather than as part of a suite. Got it. It seems like referencing a well-known result makes things easier to understand, not harder, once you see it at least once, but I imagine it certainly depends on the complexity of the result. E.g., PASSED is better than iframe src='path-to-test-containing-the-word-passed'. Past about four lines it seems like the iframe would win. I don’t have a strong objection to your iframe technique, but I’d start simpler and do it only if it’s really needed. I will also note that there are a large number of tests where we seem to have duplicate results, e.g., dom/html/level1 and dom/xhtml/level1 where the results are basically the same between the two suites, and having the xhtml results just be iframe'd versions of the html one seems like it would make sense. ___ webkit-dev mailing list webkit-dev@lists.webkit.org http://lists.webkit.org/mailman/listinfo/webkit-dev
Re: [webkit-dev] sharing more test references
On Tue, Nov 13, 2012 at 5:06 PM, Eric Seidel e...@webkit.org wrote: !DOCTYPE html body style=margin: 0px div style=height: 100px; width: 100px; background-color: green Does seem pretty simple. !DOCTYPE html body style=margin: 0px svgrect width=100px height=100px fill=greensvg is even shorter. :) I support getting rid of pixel tests. I suspect that some very dumb scripts could turn large chunks of these existing pixel-tests into ref-tests. I doubt that those would be the interesting ones though (where platforms have divergent results). I've been spending a fair amount of time working on this, actually. I think it's harder than you might think. I'm happy to talk further about it. From what I can tell, we get most of divergence between platforms from the fact that we render text differently everywhere and we tend to render controls differently everywhere. Most of the time the differences are unrelated to what's actually being tested, unfortunately :(. -- Dirk ___ webkit-dev mailing list webkit-dev@lists.webkit.org http://lists.webkit.org/mailman/listinfo/webkit-dev
[webkit-dev] Does anyone still use the TestFailures app?
http://build.webkit.org/TestFailures/ I think Adam Roben was working on this a year or so ago. It appears to be broken at the moment (it's likely that I broke it, in fact), but before I spend much time fixing it I thought I'd check. I've never actually used it myself, so I'm not sure what all it was supposed to do; it looks like it overlaps in functionality some with the flakiness dashboard, but was probably written before the flakiness dashboard worked with the build.webkit.org bots and everyone was converted to using NRWT. If anyone is still using it (or would if it was actually working) in preference to the flakiness dashboard, can you let me know why? Ideally I'd like to get rid of it and roll any good features it had into the flakiness dashboard, but I'm happy to fix it and/or keep it around if it does other things I'm not aware of or if people are still using it. Cheers, -- Dirk ___ webkit-dev mailing list webkit-dev@lists.webkit.org http://lists.webkit.org/mailman/listinfo/webkit-dev
Re: [webkit-dev] Does anyone still use the TestFailures app?
On Mon, Nov 5, 2012 at 12:47 PM, Osztrogonac Csaba o...@inf.u-szeged.hu wrote: Hi, We use it and http://build.webkit.sed.hu/TestFailures/ for gardening, it is the first step we usually do if determining who broke what about the waterfall isn't trivial. On http://build.webkit.sed.hu/TestFailures/ we use a very old copy of test failures and it still works fine. Dirk Pranke írta: http://build.webkit.org/TestFailures/ I think Adam Roben was working on this a year or so ago. It appears to be broken at the moment (it's likely that I broke it, in fact), but before I spend much time fixing it I thought I'd check. It works for me more or less, but I got strange link names: http/tests/security/cross-origin-plugin-private-browsing-toggled.html: [object DocumentFragment] Maybe one of the garden-o-matic patches broke it somehow. I've never actually used it myself, so I'm not sure what all it was supposed to do; it looks like it overlaps in functionality some with the flakiness dashboard, but was probably written before the flakiness dashboard worked with the build.webkit.org bots and everyone was converted to using NRWT. If anyone is still using it (or would if it was actually working) in preference to the flakiness dashboard, can you let me know why? We still use it, because it is very simple, it works almost always, it isn't hakced day by day and its output is very very simple. We get the result - which revision broke a given test, which are the related bug reports - with _one_ click on the name of the slave. It is more complicated to do same thing on flakiness dashboard: - select webkit.org from group - select a given slave - select tests with wrong expectations - (unselect flaky) - find manually the last good revision for test by test but it is _impossible_ if the breakage is too old Ideally I'd like to get rid of it and roll any good features it had into the flakiness dashboard, but I'm happy to fix it and/or keep it around if it does other things I'm not aware of or if people are still using it. Please don't remove this good and simple tool, we use it day by day. Thanks, Ossy! I guess I'll figure out how to fix it and go from there :). -- Dirk ___ webkit-dev mailing list webkit-dev@lists.webkit.org http://lists.webkit.org/mailman/listinfo/webkit-dev
[webkit-dev] does anyone still use 'webkit-patch rebaseline-server'?
It is/was intended to be useful for quickly reviewing and rebaselining a bunch of failures in a local (on-disk) checkout. It's been largely unmaintained and ignored for quite some time in favor of garden-o-matic. I have recently started to land some patches that will make garden-o-matic work locally as well as with the bots, and it will ultimately replace rebaseline-server; I'm attempting to consolidate all of our different change-reviewing UIs so that they are more consistent and share more code. The local garden-o-matic should roughly work now, but the UI isn't well-tuned for this use case and there's some things left do to. If you have any features you would like to add to the garden-o-matic UI (or want to make sure are transferred over from rebaseline-server), now would be a good time to mention them. And, if no one speaks up to say that rebaseline-server is still being used, I will assume I can delete it when I'm ready to do so. Thanks! -- Dirk ___ webkit-dev mailing list webkit-dev@lists.webkit.org http://lists.webkit.org/mailman/listinfo/webkit-dev
Re: [webkit-dev] DRT/WTR should clear the cache at the beginning of each test?
On Mon, Oct 29, 2012 at 5:48 AM, Maciej Stachowiak m...@apple.com wrote: On Oct 28, 2012, at 10:09 PM, Dirk Pranke dpra...@chromium.org wrote: On Sun, Oct 28, 2012 at 6:32 AM, Maciej Stachowiak m...@apple.com wrote: I think the nature of loader and cache code is that it's very hard to make tests which always fail deterministically when regressions are introduced, as opposed to randomly. The reason for this is that bugs in these areas are often timing-dependent. I think it's likely this tendency to fail randomly will be the case whether or not the tests are trying to explicitly test the cache or are just incidentally doing so in the course of other things. I am not familiar with the loader and caching code in webkit, but I know enough about similar problem spaces to be puzzled by why it's impossible to write tests that can adequately test the code. Has anyone claimed that? I think impossible to write tests that can adequately test the code is not a position that anyone in this thread has taken, certainly not me above. My claim is only that many classes of loader and cache bugs, when first introduced, are likely to cause nondeterministic test failures. And further, this is likely to be the case even if tests are written to target that subsystem. That's not the same as saying adequate tests are impossible. I'm sorry, I didn't mean impossible literally. Please strike that, as it sounds like it has just made a confusing situation worse. But, you did claim that it would be very hard to make tests that always fail deterministically, and I don't see why that's true? Testing things that are timing-dependent only require that you be able to control or simulate time. It may be that this is hard to do with layout tests, but it's pretty straightforward with unit tests that allow you to control the layers above and below the cache. It just means to have good testing of some areas of the code, we need a good way of dealing with nondeterministic failures. This is backwards. If you *don't* have good testing, more of your failures are likely to show up sporadically, which leads you to want to build tools for them. Randomized testing is a helpful tool to use *alongside* focused testing to ensure coverage, but should not be used as a replacement. What I personally would most wish for is good tools to catch when a test starts failing nondeterministically, and to identify the revision where the failures began. The reason we hate random failures is that they are hard to track down and diagnose. But some types of bugs are unlikely to manifest in a purely deterministic way. It would be good if we had a reliable and useful way to catch those types of bugs. This is a fine idea -- and I'm always happy to talk about ways we can improve our test tooling, please feel free to start a separate thread on these issues -- but I don't want to lose sight of the main issue here. I think the problem I identified -- that it's overly hard to track down and diagnose regressions that cause tests to fail only part of the time -- is more important and more fundamental than any of the three problems that you cite below. Our test infrastructure ultimately exists to help us notice and promptly fix regressions, and for some types of regressions, namely those that do not manifest 100% of the time, it is not working so well. The problems you mention are all secondary consequences of that fundamental problem, in my opinion. First of all, this isn't an either/or situation. We should be capable of addressing all of these issues in parallel. Second, I don't see how the existence of bugs in the code, the lack of test isolation, or the lack of good test coverage for certain layers of the code follow from not having good tools to triage intermittent failures? That seems like putting the cart before the horse. Third, are you familiar with the flakiness dashboard? http://test-results.appspot.com/dashboards/flakiness_dashboard.html#group=%40ToT%20-%20webkit.orgbuilder=Apple%20Lion%20Debug%20WK1%20(Tests) Does it not do exactly what you're describing? Are there things that you would like added? If it would be helpful for us to have a meeting or something to help explain how this works, I'm sure we could set one up. - Maciej It sounds like we've identified three existing problems - please correct me if I'm misstating them: 1. There appears to be a bug in the caching code that is causing tests for other parts of the system to fail randomly. 2. DRT and WTR on some ports are implemented in a way that is causing the system to be more fragile than some of us would like it to be, and there doesn't seem to be an a priori need for this to be the case; indeed some ports already don't do this. 3. We don't apparently have dedicated test coverage for caching and the loader that people think is good enough, and getting such tests might be hard. P.S. I do think your problem
Re: [webkit-dev] DRT/WTR should clear the cache at the beginning of each test?
On Oct 26, 2012, at 11:11 PM, Ryosuke Niwa rn...@webkit.org wrote: I’m sure Antti, Alexey, and others who have worked on the loader and other parts of WebKit are happy to write those tests or list the kind of things they want to test. Heck, I don’t mind writing those tests if someone could make a list. I totally sympathize with the sentiment to reduce the test flakiness but loader and cache code have historically been under-tested, and we’ve had a number of bugs detected only by running non-loader tests consecutively. On the contrary, we’ve had this DRT behavior for ages. Is there any reason we can’t wait for another couple of weeks or months until we add more loader cache tests before making the behavior change? Please correct me if I'm misinformed, but it's been three months since this issue was first raised, and it doesn't sound like they've been writing those tests or are happy to do so, and despite people asking on this thread, they haven't been listing the kinds of tests they think they need. Have we actually made any progress here, or was the issue dropped until Ami raised it again? It seems like the latter to me ... again, please correct me if this is being actively worked on, because that would change the whole tenor of this debate. On Sun, Oct 28, 2012 at 6:32 AM, Maciej Stachowiak m...@apple.com wrote: I think the nature of loader and cache code is that it's very hard to make tests which always fail deterministically when regressions are introduced, as opposed to randomly. The reason for this is that bugs in these areas are often timing-dependent. I think it's likely this tendency to fail randomly will be the case whether or not the tests are trying to explicitly test the cache or are just incidentally doing so in the course of other things. I am not familiar with the loader and caching code in webkit, but I know enough about similar problem spaces to be puzzled by why it's impossible to write tests that can adequately test the code. Is the caching disk-based, and maybe running tests in parallel screwing with things? If so, then maybe the fact that we now run tests in parallel is why this is a problem now and hasn't been before? Or maybe the fact that a given process doesn't always see the same tests in the same order is the problem? Unfortunately, it's very tempting when a test is failing randomly to blame the test rather than to investigate whether there is an actual regression affecting it. And sometimes it really is the test's fault. But sometimes it is a genuine bug in the code. On the other hand, nondetermisitic test failures make it harder to use test infrastructure in general. These are difficult things to reconcile. The original philosophy of WebKit tests is to test end-to-end under relatively realistic conditions, but at the same time unpredictability makes it hard to stay at zero regressions. Exactly. Personally, the cost of unpredictability in the test infrastructure is so much higher than the value we're getting (implicitly) that this is a no-brainer to me. There are some tradeoffs (like running tests in parallel) that are worth it, but this isn't one of them. I am happy to explain further my thinking and standards if there's interest. Hopefully that partially answers Alexey's questions about where we should draw the line in trying to make our tests deterministic and hermetic: do everything you reasonably can. We're not picking on caching here. I think making different ports do testing under different conditions makes it more likely that some contributors will introduce regressions without noticing, leaving it for others to clean up. So it's regrettable if we go that way because we are unable to reach consensus. I agree that it is bad to have different ports behaving differently, and I would like to avoid that as well. I don't want any port suffering from flaky tests, but I also don't think it's reasonable to have one group force that on everyone else indefinitely, either. I am also fine with having some way to test systems more non-deterministically in a way to expose more bugs, but that needs to be clearly separated from the other testing we do; it is an unfair cost to impose on the rest of the system otherwise and should be tolerated only if we have no other choice. We have other choices. Creating some special opt-in --antti mode would be even worse, as it's almost certain that failures would creep into a mode that nobody runs. This comment (and Antti's suggestion, below) makes me think that you didn't understand my virtual test suite suggestion; that's not surprising, since Apple doesn't actual use this feature of NRWT yet. A virtual test suite is a way of saying (re-)run the tests under directory X with command-line flags Y and Z, and put the results in a new directory. For example, Chromium runs all of the tests in fast/canvas twice, once normally using the regular software code path, and once with a
Re: [webkit-dev] DRT/WTR should clear the cache at the beginning of each test?
On Sun, Oct 28, 2012 at 2:47 PM, Ryosuke Niwa rn...@webkit.org wrote: On Sun, Oct 28, 2012 at 2:09 PM, Dirk Pranke dpra...@chromium.org wrote: On Oct 26, 2012, at 11:11 PM, Ryosuke Niwa rn...@webkit.org wrote: I’m sure Antti, Alexey, and others who have worked on the loader and other parts of WebKit are happy to write those tests or list the kind of things they want to test. Heck, I don’t mind writing those tests if someone could make a list. I totally sympathize with the sentiment to reduce the test flakiness but loader and cache code have historically been under-tested, and we’ve had a number of bugs detected only by running non-loader tests consecutively. On the contrary, we’ve had this DRT behavior for ages. Is there any reason we can’t wait for another couple of weeks or months until we add more loader cache tests before making the behavior change? Please correct me if I'm misinformed, but it's been three months since this issue was first raised, and it doesn't sound like they've been writing those tests or are happy to do so, and despite people asking on this thread, they haven't been listing the kinds of tests they think they need. I don't think anyone else had suggested adding tests as an option or set a deadline until I suggested yesterday (or when I did in my original reply to the thread). In fact, since Ami posted his reply on October 26th 1:20AM (PST), many contributors from non-PST timezones haven't even had a chance to read his post during normal business hours. Given that I'd think it's totally unreasonable to land the patch as is without giving people reasonable amount of time (~one week) to respond to this thread. Both you and Eric U suggesting adding new tests for this in the original thread on 8/9; in fact, this whole issue got a fair amount of discussion then, and this round hasn't really added anything new. I'm happy to wait a little longer if others want to come up with some other suggestions; I apologize if my previous response sounded like I was throwing down a gauntlet or otherwise not open to ideas; that was definitely not my intent. Rather, I was attempting to say that unless someone else has other ideas, the right path forward seems fairly clear to me and that I intended to proceed down it. Does that seem more reasonable to you? -- Dirk ___ webkit-dev mailing list webkit-dev@lists.webkit.org http://lists.webkit.org/mailman/listinfo/webkit-dev
Re: [webkit-dev] DRT/WTR should clear the cache at the beginning of each test?
I don't know that there was consensus that every port had to be updated at the same time; in fact Balazs said Qt and EFL already clear the cache. I think you should just land the change for Chromium and let others update their ports as needed. The value in reduced flakiness and more predictability outweighs anything else in my book. Test coverage that you can't explain or rely on doesn't count for much to me. -- Dirk On Fri, Oct 26, 2012 at 1:20 AM, Ami Fischman fisch...@chromium.org wrote: This thread stalled out because although there seemed to be majority agreement that hermetic/repeatable tests are a good thing, there was a requirement that all ports be updated to the new behavior at the same time, and I'm only competent to do the chromium DRT (see https://bugs.webkit.org/show_bug.cgi?id=93195 for details). Is anyone interested in stepping up and doing the equivalent (clear caches between tests) for the mac and/or gtk ports' DRTs? On Wed, Aug 8, 2012 at 2:35 PM, Dirk Pranke dpra...@chromium.org wrote: On Wed, Aug 8, 2012 at 10:47 AM, Ojan Vafai o...@chromium.org wrote: See https://bugs.webkit.org/show_bug.cgi?id=93195. media/W3C/video/networkState/networkState_during_progress.html and media/video-poster-blocked-by-willsendrequest.html are flaky on all platforms because they behave differently if the loaded resource is cached. Every time I've taken a stab at reducing test flakiness, I've come across at least a few tests that pass when run as part of the test suite, but fail when run by themselves (or in parallel) because they accidentally expect an image or something to be in the cache. I think it would make the tests more maintainable if we cleared the cache before each test run. This is *not* before each page load though. So tests that do multiple page loads will still test cross-navigation caching behavior. While it's true that we could one-off fix each of these tests, it's usually very time consuming to figure out that caching is the problem, that's assuming anyone takes the time to look into why the test is flaky in the first place. Any objections? Given that the way we run tests in parallel in NRWT means that different processes get different lists of tests each time, it sounds like we may be getting a fair amount of nondeterminism from the cache not being cleared between tests. That seems bad, so I'm in favor of clearing the cache :) -- Dirk ___ webkit-dev mailing list webkit-dev@lists.webkit.org http://lists.webkit.org/mailman/listinfo/webkit-dev
Re: [webkit-dev] DRT/WTR should clear the cache at the beginning of each test?
On Fri, Oct 26, 2012 at 11:38 AM, Ryosuke Niwa rn...@webkit.org wrote: On Fri, Oct 26, 2012 at 11:33 AM, Elliott Sprehn espr...@chromium.org wrote: On Fri, Oct 26, 2012 at 11:17 AM, Ryosuke Niwa rn...@webkit.org wrote: ... I agree this is a good change but it appears that we should add more cache/loader tests before changing DRT's behavior given that there are active contributors who rely on the current DRT behaviors to detect regressions. Can we add a flag to control this behavior? Then Antti could run the tests without cache clearing when modifying things possibly related to the cache code. We could even run a separate cr-linux bot like we do for debug builds. I think having a set of tests that tests loaders/caches explicitly is more useful. I think having a set of tests for loaders and caches would be more useful as well, but I don't think it's fair to make that a requirement to changing the default behavior here, especially since it's not clear who all would be best suited to writing those tests or what the extent of that work is. I think Eliot's suggestion is a good one. I think the overall cost to the project by having flakiness in the tests probably outweighs the value we get in mysterious additional coverage, and it seems like having a flag would be a good compromise. -- Dirk - Ryosuke ___ webkit-dev mailing list webkit-dev@lists.webkit.org http://lists.webkit.org/mailman/listinfo/webkit-dev ___ webkit-dev mailing list webkit-dev@lists.webkit.org http://lists.webkit.org/mailman/listinfo/webkit-dev
Re: [webkit-dev] A new test in a patch passes locally fails on ews
I think there is some general interest in a feature like this. I need to sync up w/ Adam and Eric and figure out what all might be required here, so we might do this but I can't say for sure yet ... -- Dirk On Wed, Oct 24, 2012 at 8:40 AM, Adam Barth aba...@webkit.org wrote: I don't have any current plans to implement that service. If someone else would like to administer the machines and add these features, I'm happy to review patches. Adam On Wed, Oct 24, 2012 at 6:38 AM, Pravin D pravind@gmail.com wrote: Wanted to know if the we are going to go with the plan to attach a mailing service to the bot for the failing test cases anything soon ? As only Mac and cr-linux EWS bots run test cases its a becoming little difficult to figure out whats the failure and whether its platform specific or not (i'm using Qt-linux and Windows). On Fri, Oct 5, 2012 at 11:20 PM, Eric Seidel e...@webkit.org wrote: I think that's an interesting idea. The bots don't have a mail server. :) But we could presumably wire up some sort of service for them. On Thu, Oct 4, 2012 at 6:41 PM, Emil A Eklund e...@chromium.org wrote: What if we mail the zip files to the person that uploaded the patch? That way the responsibility of managing the storage is shifted to the author and the author still benefits from the results from all platforms. ___ webkit-dev mailing list webkit-dev@lists.webkit.org http://lists.webkit.org/mailman/listinfo/webkit-dev ___ webkit-dev mailing list webkit-dev@lists.webkit.org http://lists.webkit.org/mailman/listinfo/webkit-dev ___ webkit-dev mailing list webkit-dev@lists.webkit.org http://lists.webkit.org/mailman/listinfo/webkit-dev ___ webkit-dev mailing list webkit-dev@lists.webkit.org http://lists.webkit.org/mailman/listinfo/webkit-dev ___ webkit-dev mailing list webkit-dev@lists.webkit.org http://lists.webkit.org/mailman/listinfo/webkit-dev
Re: [webkit-dev] A new test in a patch passes locally fails on ews
I think I did a bunch of pruning at some point and the zip files on the EWS bots are way smaller than they used to be; there's probably even more we could do, but certainly for some class of patches there will be too many failures and we'd be looking at many megabytes of zip files that we don't necessarily want to stick bugzilla with. -- Dirk On Wed, Oct 24, 2012 at 12:51 PM, Eric Seidel e...@webkit.org wrote: Now with feeling. On Wed, Oct 24, 2012 at 12:51 PM, Eric Seidel esei...@google.com wrote: We used to upload zips to bugs. But there is a built-in cutoff on that feature. I suspect that the zip size from the bots has just ballooned above that cut-off and fixing LayoutTestResults to only include the relevant files would probably make the EWSes start uploading results.zip files again. :) On Wed, Oct 24, 2012 at 12:33 PM, Dirk Pranke dpra...@chromium.org wrote: I think there is some general interest in a feature like this. I need to sync up w/ Adam and Eric and figure out what all might be required here, so we might do this but I can't say for sure yet ... -- Dirk On Wed, Oct 24, 2012 at 8:40 AM, Adam Barth aba...@webkit.org wrote: I don't have any current plans to implement that service. If someone else would like to administer the machines and add these features, I'm happy to review patches. Adam On Wed, Oct 24, 2012 at 6:38 AM, Pravin D pravind@gmail.com wrote: Wanted to know if the we are going to go with the plan to attach a mailing service to the bot for the failing test cases anything soon ? As only Mac and cr-linux EWS bots run test cases its a becoming little difficult to figure out whats the failure and whether its platform specific or not (i'm using Qt-linux and Windows). On Fri, Oct 5, 2012 at 11:20 PM, Eric Seidel e...@webkit.org wrote: I think that's an interesting idea. The bots don't have a mail server. :) But we could presumably wire up some sort of service for them. On Thu, Oct 4, 2012 at 6:41 PM, Emil A Eklund e...@chromium.org wrote: What if we mail the zip files to the person that uploaded the patch? That way the responsibility of managing the storage is shifted to the author and the author still benefits from the results from all platforms. ___ webkit-dev mailing list webkit-dev@lists.webkit.org http://lists.webkit.org/mailman/listinfo/webkit-dev ___ webkit-dev mailing list webkit-dev@lists.webkit.org http://lists.webkit.org/mailman/listinfo/webkit-dev ___ webkit-dev mailing list webkit-dev@lists.webkit.org http://lists.webkit.org/mailman/listinfo/webkit-dev ___ webkit-dev mailing list webkit-dev@lists.webkit.org http://lists.webkit.org/mailman/listinfo/webkit-dev ___ webkit-dev mailing list webkit-dev@lists.webkit.org http://lists.webkit.org/mailman/listinfo/webkit-dev ___ webkit-dev mailing list webkit-dev@lists.webkit.org http://lists.webkit.org/mailman/listinfo/webkit-dev
Re: [webkit-dev] build-webkit stopped working for me
On Sun, Oct 21, 2012 at 2:17 PM, Maciej Stachowiak m...@apple.com wrote: Apple did not ship the last release of Safari to SnowLeopard and we have no plans to maintain SnowLeopard support on trunk. We haven't actively ripped out SL-specific ifdefs because we were under the impression that the Chromium port still targets SL and sometimes uses Mac code paths. This is correct; Chromium still supports SL. -- Dirk ___ webkit-dev mailing list webkit-dev@lists.webkit.org http://lists.webkit.org/mailman/listinfo/webkit-dev
Re: [webkit-dev] [chromium] as of r131699, the chromium port no longer uses any baselines from LayoutTests/platform/mac
It's not a tree, it's still a DAG, because of the wk2 baselines as Adam says. Note that my change didn't really change the shape of the tree (previously all the chromium ports fell back to platform/chromium then platform/mac, then the generic results. I just removed a node). -- Dirk On Wed, Oct 17, 2012 at 10:11 PM, Adam Barth aba...@webkit.org wrote: We should update the graph. It might be a tree now. The only complication might be the wk2 baselines. Adam On Wed, Oct 17, 2012 at 9:53 PM, Eric Seidel e...@webkit.org wrote: Does that mean our fallback graph is now finally a tree?? :) https://docs.google.com/drawings/d/1z65SKkWrD4Slm6jobIphHwwRADyUtjOAxwGBVKBY8Kc/edit On Wed, Oct 17, 2012 at 8:15 PM, Dirk Pranke dpra...@chromium.org wrote: All of the Chromium ports will use baselines in their port-specific directories, then fall back through various paths to platform/chromium and then to next to the test. Which means you Apple folks can feel free to break things in platform/mac to your hearts' content :). Let me know if you see any weirdness or problems. Thanks! -- Dirk ___ webkit-dev mailing list webkit-dev@lists.webkit.org http://lists.webkit.org/mailman/listinfo/webkit-dev ___ webkit-dev mailing list webkit-dev@lists.webkit.org http://lists.webkit.org/mailman/listinfo/webkit-dev ___ webkit-dev mailing list webkit-dev@lists.webkit.org http://lists.webkit.org/mailman/listinfo/webkit-dev
[webkit-dev] [chromium] as of r131699, the chromium port no longer uses any baselines from LayoutTests/platform/mac
All of the Chromium ports will use baselines in their port-specific directories, then fall back through various paths to platform/chromium and then to next to the test. Which means you Apple folks can feel free to break things in platform/mac to your hearts' content :). Let me know if you see any weirdness or problems. Thanks! -- Dirk ___ webkit-dev mailing list webkit-dev@lists.webkit.org http://lists.webkit.org/mailman/listinfo/webkit-dev
[webkit-dev] Chromium Mountain Lion (Mac 10.8) baselines ...
Hi all, You can largely ignore this if you never modify Chromium baselines or TestExpectations. But if you do modify them ... I'm working on getting our new 10.8/MountainLion bot green. It looks like Apple changed their text rendering a bit in the latest release and so we need lots more baselines. In addition, the Chromium project will currently fall back on the Apple baselines in platform/mac, and we've decided that we probably shouldn't (so that Apple folks don't have to worry about accidentally breaking the Chromium port when they change their baselines in platform/mac), so I'll be landing copies of those baselines into platform/chromium-mac as well. So, you'll be seeing a lot of churn under LayoutTests :( Sorry ! Please *do not* modify LayoutTests/platform/chromium-mac-mountainlion/TestExpectations without coordinating with me. Largely, you should pretend it (and the 10.8 bot) don't exist until I say otherwise. -- Dirk PS. As an aside, this once again raises the question of whether maintaining pixel tests (generally) is worth the cost to the project. I would like to decrease our dependence on them, and am working on a number of ideas that might help here, but don't have anything ready to solve the problem right now. More on these efforts soon ... ___ webkit-dev mailing list webkit-dev@lists.webkit.org http://lists.webkit.org/mailman/listinfo/webkit-dev
Re: [webkit-dev] add NeedsRebaseline keyword to TestExpectations as a way to hande updating pixel tests?
Makes sense to me ... On Thu, Oct 4, 2012 at 2:46 PM, Ojan Vafai o...@chromium.org wrote: TL;DR: We should add a NeedsRebaseline keyword to TestExpectations and add garden-o-matic tooling for it for the cases where someone commits a change/test that they know will need new results for different ports (e.g. any patch that changes the rendering of pixel tests). A common pattern that I see across ports is that someone will add something like the following in a patch that changes the results of a pixel test: // Needs rebaseline after r23456 webkit.org/b/12345 path/to/test.html [ Failure ] This has a couple problems: -Often the correct expectation is something like [ Missing Failure ImageOnlyFailure ]. So, even though the test is listed, the bot turns red. -The tooling can't give you a list of all the tests that are expected to only need a rebaseline. -Related to the above, people often forget about these lines and don't do the rebaseline. We should add [ NeedsRebasline ], which is equivalent to [ Missing Failure ImageOnlyFailure ]. I'm thinking it should not include Timeout/Crash since those would need a solution other than a rebaseline (e.g. something is wrong with the test or patch). In garden-o-matic, we can make a tab specifically for tests that need rebaseline and give some indication whether the original patch that line was added in has run on all the relevant bots. This way the people keeping the tree green can also make sure that NeedsRebaseline lines don't get forgotten. If it continues to be a problem we could even setup and automated nag bot to email people who leave in NeedsRebaseline lines for more than a week. In the long-run, we should make it so you can grab the new results of the EWS bots and don't need to add lines to TestExpectations at all. In the short-term though, this is a way we can handle pixel tests without making the tree red all the time. As a side note, we should also get rid of Missing as a valid expectation. A test should either be NeedsRebaseline or have an expected result. Ojan ___ webkit-dev mailing list webkit-dev@lists.webkit.org http://lists.webkit.org/mailman/listinfo/webkit-dev ___ webkit-dev mailing list webkit-dev@lists.webkit.org http://lists.webkit.org/mailman/listinfo/webkit-dev
Re: [webkit-dev] Tips on why run-webkit-tests is not working for me?
Think Eric and Alexey identified the problem here ... -- Dirk On Wed, Oct 3, 2012 at 9:44 AM, Alexey Proskuryakov a...@webkit.org wrote: 03.10.2012, в 9:19, Eric Seidel e...@webkit.org написал(а): Perhaps make is missing? or the java directory its trying to build is missing and it's just printing the wrong path? Isn't make part of Xcode command line utilities? If that's the issue indeed, the fix is to change the script to use xcrun make here, and installing command line utilities is a workaround. - WBR, Alexey Proskuryakov ___ webkit-dev mailing list webkit-dev@lists.webkit.org http://lists.webkit.org/mailman/listinfo/webkit-dev ___ webkit-dev mailing list webkit-dev@lists.webkit.org http://lists.webkit.org/mailman/listinfo/webkit-dev
Re: [webkit-dev] Tips on why run-webkit-tests is not working for me?
On Wed, Oct 3, 2012 at 3:39 PM, Darin Adler da...@apple.com wrote: On Oct 3, 2012, at 9:10 AM, Ryosuke Niwa rn...@webkit.org wrote: Could you tell us what your directory structure look like and where you're executing that command? Looks like a path confusion. My WebKit source tree is in ~/Safari/OpenSource and my build products are in ~/Builds. I am running the script inside the ~/Safari/OpenSource directory. On Oct 3, 2012, at 9:11 AM, Ojan Vafai o...@chromium.org wrote: I'm guessing there's something about your checkout that is confusing the code that finds the path to run-webkit-tests. Does /Users/darin/Safari/Internal/Tools/Scripts/../../../OpenSource/Tools/Scripts/run-webkit-tests actually exist or is that path incorrect? That path is correct, although I don’t know where it’s coming from. Maybe related to how I originally checked out my tree from Subversion. On Oct 3, 2012, at 9:19 AM, Eric Seidel e...@webkit.org wrote: It looks like it's in Mac._build_java_test_support: I don’t think I have Java installed on this computer. Maybe that’s somehow related to the problem. I would suggest running test-webkitpy (which will go through and delete any orphaned .pyc files which could be confusing things). OK, I’ll do that when I get a chance. Perhaps make is missing? I often use make from the command line to build, so it’s in my path. I wish the error message made it clearer what file couldn’t be found. The thing that changed most recently on my machine is how Xcode is installed, so I’m guessing the problem is somehow related to that, since I’ve run the tests many times in the past. I can look into making the error clearer, but it's trying to run /usr/bin/make, -C /Users/darin/Safari/Internal/Tools/Scripts/../../../OpenSource/LayoutTests/java] (approximately). It's probably bad that it's hardcoded to look in /usr/bin; is that your problem? Try changing the lines of code Eric linked to ... -- Dirk ___ webkit-dev mailing list webkit-dev@lists.webkit.org http://lists.webkit.org/mailman/listinfo/webkit-dev
Re: [webkit-dev] Tips on why run-webkit-tests is not working for me?
On Wed, Oct 3, 2012 at 4:18 PM, Dirk Pranke dpra...@chromium.org wrote: On Wed, Oct 3, 2012 at 3:39 PM, Darin Adler da...@apple.com wrote: On Oct 3, 2012, at 9:10 AM, Ryosuke Niwa rn...@webkit.org wrote: Could you tell us what your directory structure look like and where you're executing that command? Looks like a path confusion. My WebKit source tree is in ~/Safari/OpenSource and my build products are in ~/Builds. I am running the script inside the ~/Safari/OpenSource directory. On Oct 3, 2012, at 9:11 AM, Ojan Vafai o...@chromium.org wrote: I'm guessing there's something about your checkout that is confusing the code that finds the path to run-webkit-tests. Does /Users/darin/Safari/Internal/Tools/Scripts/../../../OpenSource/Tools/Scripts/run-webkit-tests actually exist or is that path incorrect? That path is correct, although I don’t know where it’s coming from. Maybe related to how I originally checked out my tree from Subversion. On Oct 3, 2012, at 9:19 AM, Eric Seidel e...@webkit.org wrote: It looks like it's in Mac._build_java_test_support: I don’t think I have Java installed on this computer. Maybe that’s somehow related to the problem. I would suggest running test-webkitpy (which will go through and delete any orphaned .pyc files which could be confusing things). OK, I’ll do that when I get a chance. Perhaps make is missing? I often use make from the command line to build, so it’s in my path. I wish the error message made it clearer what file couldn’t be found. The thing that changed most recently on my machine is how Xcode is installed, so I’m guessing the problem is somehow related to that, since I’ve run the tests many times in the past. I can look into making the error clearer, but it's trying to run /usr/bin/make, -C /Users/darin/Safari/Internal/Tools/Scripts/../../../OpenSource/LayoutTests/java] (approximately). It's probably bad that it's hardcoded to look in /usr/bin; is that your problem? Try changing the lines of code Eric linked to ... And if you're still having problems feel free to contact me off-list (on #irc or whatever). -- Dirk ___ webkit-dev mailing list webkit-dev@lists.webkit.org http://lists.webkit.org/mailman/listinfo/webkit-dev
Re: [webkit-dev] Skipped files are going away
Last call on this ... I will be removing the files later today unless something goes awry :). -- Dirk On Fri, Sep 21, 2012 at 3:41 PM, Dirk Pranke dpra...@chromium.org wrote: Hi all, Now that we're on the new TestExpectations syntax, I'm planning to remove the Skipped files (and support for them from NRWT). I have a change posted that will add minimal support for the syntax to old-run-webkit-tests in https://bugs.webkit.org/show_bug.cgi?id=97276 ; basically any file or directory listed in a TestExpectations file will be skipped *regardless of the expectation listed*. I can make things smarter if need be, but it's not clear to me how many people if any are still using ORWT so the need doesn't seem to be there. Please comment on the bug ASAP if you disagree and would like ORWT to be smarter in this area :). Once that change lands, my plan is to copy all of the entries from the existing Skipped files over into their corresponding TestExpectations files, and then delete the Skipped files, and then delete the code that supports the Skipped files in NRWT and ORWT. I hope to do this early next week. If you have any objections, now would be a good time to speak up! Note that all of this is being done in the oft-requested name of simplifying the number of different ways we have to handle failing tests. Also, I will remind people that the new TestExpectations syntax is a superset of the old Skipped file syntax, so there shouldn't be any good reason to keep the old files around. However, if you know something I don't, please mention it :). Thanks, -- Dirk ___ webkit-dev mailing list webkit-dev@lists.webkit.org http://lists.webkit.org/mailman/listinfo/webkit-dev
Re: [webkit-dev] Skipped files are going away
Now landed in http://trac.webkit.org/changeset/129788 . Let me know if there are any problems. -- Dirk On Thu, Sep 27, 2012 at 10:15 AM, Dirk Pranke dpra...@chromium.org wrote: Last call on this ... I will be removing the files later today unless something goes awry :). -- Dirk On Fri, Sep 21, 2012 at 3:41 PM, Dirk Pranke dpra...@chromium.org wrote: Hi all, Now that we're on the new TestExpectations syntax, I'm planning to remove the Skipped files (and support for them from NRWT). I have a change posted that will add minimal support for the syntax to old-run-webkit-tests in https://bugs.webkit.org/show_bug.cgi?id=97276 ; basically any file or directory listed in a TestExpectations file will be skipped *regardless of the expectation listed*. I can make things smarter if need be, but it's not clear to me how many people if any are still using ORWT so the need doesn't seem to be there. Please comment on the bug ASAP if you disagree and would like ORWT to be smarter in this area :). Once that change lands, my plan is to copy all of the entries from the existing Skipped files over into their corresponding TestExpectations files, and then delete the Skipped files, and then delete the code that supports the Skipped files in NRWT and ORWT. I hope to do this early next week. If you have any objections, now would be a good time to speak up! Note that all of this is being done in the oft-requested name of simplifying the number of different ways we have to handle failing tests. Also, I will remind people that the new TestExpectations syntax is a superset of the old Skipped file syntax, so there shouldn't be any good reason to keep the old files around. However, if you know something I don't, please mention it :). Thanks, -- Dirk ___ webkit-dev mailing list webkit-dev@lists.webkit.org http://lists.webkit.org/mailman/listinfo/webkit-dev
Re: [webkit-dev] Skipped files are going away
Sorry for the long stream of messages on this ... in case it's not clear, all I did with this change was cut and paste the entries from one file into the other (since the TestExpectations file syntax is a superset of the Skipped syntax) and do some minimal de-duping so that run-webkit-tests --lint-test-files would pass. Different ports use different organization methods for these files, and I didn't attempt to follow each port, nor reorganize entries that were being skipped but really should've been failures or flakes or something else. I leave each port to do their own cleanup, but of course I'll be happy to troubleshoot/answer questions, etc :). -- Dirk On Thu, Sep 27, 2012 at 12:24 PM, Dirk Pranke dpra...@chromium.org wrote: Now landed in http://trac.webkit.org/changeset/129788 . Let me know if there are any problems. -- Dirk On Thu, Sep 27, 2012 at 10:15 AM, Dirk Pranke dpra...@chromium.org wrote: Last call on this ... I will be removing the files later today unless something goes awry :). -- Dirk On Fri, Sep 21, 2012 at 3:41 PM, Dirk Pranke dpra...@chromium.org wrote: Hi all, Now that we're on the new TestExpectations syntax, I'm planning to remove the Skipped files (and support for them from NRWT). I have a change posted that will add minimal support for the syntax to old-run-webkit-tests in https://bugs.webkit.org/show_bug.cgi?id=97276 ; basically any file or directory listed in a TestExpectations file will be skipped *regardless of the expectation listed*. I can make things smarter if need be, but it's not clear to me how many people if any are still using ORWT so the need doesn't seem to be there. Please comment on the bug ASAP if you disagree and would like ORWT to be smarter in this area :). Once that change lands, my plan is to copy all of the entries from the existing Skipped files over into their corresponding TestExpectations files, and then delete the Skipped files, and then delete the code that supports the Skipped files in NRWT and ORWT. I hope to do this early next week. If you have any objections, now would be a good time to speak up! Note that all of this is being done in the oft-requested name of simplifying the number of different ways we have to handle failing tests. Also, I will remind people that the new TestExpectations syntax is a superset of the old Skipped file syntax, so there shouldn't be any good reason to keep the old files around. However, if you know something I don't, please mention it :). Thanks, -- Dirk ___ webkit-dev mailing list webkit-dev@lists.webkit.org http://lists.webkit.org/mailman/listinfo/webkit-dev
Re: [webkit-dev] Settings and Preferences in layout tests
On a related note, there is a gradual movement to pass more command line flags along with each test to DRT during run-webkit-tests (to toggle between pixel tests and non, change timeout values, etc.), and being able to ensure we reset the state to the default between each test is kinda necessary for this to work. -- Dirk On Wed, Sep 26, 2012 at 2:00 PM, Eric Seidel e...@webkit.org wrote: When overridePreferences was added back in the day: http://trac.webkit.org/changeset/39212 The contract was that DRT would call resetDefaults between tests. I don't know what the current status of resetToDefaults implementations in other ports or in non-WK1 architectures. On Wed, Sep 26, 2012 at 3:53 PM, Brady Eidson beid...@apple.com wrote: On Sep 26, 2012, at 1:48 PM, Ryosuke Niwa rn...@webkit.org wrote: On Wed, Sep 26, 2012 at 1:44 PM, Simon Fraser simon.fra...@apple.com wrote: First, direct calls on testRunner that just set preferences should be migrated to internals.settings or testRunner.overridePreference calls, I think (I don't know if either is preferred). I support the idea of unifying the approaches and just use internals.settings. However, the last time I checked, Alexey had some concerns about using internals due to settings may not be properly propagated to WebKit2 layer. Has this concern been addressed? In general I prefer the overridePreference() calls whenever they exist. internals.settings are not exposed in any real-world product whereas preferences exist in each platform's WebKit-layer API that they expose to their embedders in some form. ~Brady ___ webkit-dev mailing list webkit-dev@lists.webkit.org http://lists.webkit.org/mailman/listinfo/webkit-dev ___ webkit-dev mailing list webkit-dev@lists.webkit.org http://lists.webkit.org/mailman/listinfo/webkit-dev ___ webkit-dev mailing list webkit-dev@lists.webkit.org http://lists.webkit.org/mailman/listinfo/webkit-dev
[webkit-dev] the switchover to the new TestExpectations syntax is complete; the old syntax is no longer supported
As of http://trac.webkit.org/changeset/129265 . (There are still some things to clean up in the code now that the old syntax is gone, but hardly anyone will care about that). I am not aware of any open bugs or issues with the new syntax. Speak now or I'll assume we're all good :). Cheers, -- Dirk ___ webkit-dev mailing list webkit-dev@lists.webkit.org http://lists.webkit.org/mailman/listinfo/webkit-dev
[webkit-dev] Skipped files are going away
Hi all, Now that we're on the new TestExpectations syntax, I'm planning to remove the Skipped files (and support for them from NRWT). I have a change posted that will add minimal support for the syntax to old-run-webkit-tests in https://bugs.webkit.org/show_bug.cgi?id=97276 ; basically any file or directory listed in a TestExpectations file will be skipped *regardless of the expectation listed*. I can make things smarter if need be, but it's not clear to me how many people if any are still using ORWT so the need doesn't seem to be there. Please comment on the bug ASAP if you disagree and would like ORWT to be smarter in this area :). Once that change lands, my plan is to copy all of the entries from the existing Skipped files over into their corresponding TestExpectations files, and then delete the Skipped files, and then delete the code that supports the Skipped files in NRWT and ORWT. I hope to do this early next week. If you have any objections, now would be a good time to speak up! Note that all of this is being done in the oft-requested name of simplifying the number of different ways we have to handle failing tests. Also, I will remind people that the new TestExpectations syntax is a superset of the old Skipped file syntax, so there shouldn't be any good reason to keep the old files around. However, if you know something I don't, please mention it :). Thanks, -- Dirk ___ webkit-dev mailing list webkit-dev@lists.webkit.org http://lists.webkit.org/mailman/listinfo/webkit-dev
Re: [webkit-dev] Bug numbers in TestExpectations
The URL is already optional. However, I would encourage people to actually create useful bugs rather than not adding comments to the TestExpectations/Skipped files. I'm not a big fan of having to look in two different places to figure out what's broken, and having bugs for things makes it easier to run reports and queries against the bug database that are actually complete. -- Dirk On Thu, Sep 20, 2012 at 10:34 AM, Alexey Proskuryakov a...@webkit.org wrote: I've been repeatedly finding it that bug numbers in TestExpectations don't lead to bugs that are helpful in the context of the specific test. Often the bug doesn't have any information beyond what's already in the expectation (that the test is skipped of failing, without a posted diff or any other useful detail). Other times, the linked bug is one that the person was working on at the time of changing expectations, so the bug link is more like see related bug , and try to figure out how exactly it's related, not this failure is tracked by . Free form comments worked great for both cases in Skipped files. It appears that the rigid format that requires putting a bug URL is causing more harm than good in practice. I suggest making the URL optional. - WBR, Alexey Proskuryakov ___ webkit-dev mailing list webkit-dev@lists.webkit.org http://lists.webkit.org/mailman/listinfo/webkit-dev ___ webkit-dev mailing list webkit-dev@lists.webkit.org http://lists.webkit.org/mailman/listinfo/webkit-dev
Re: [webkit-dev] Should we close the tree? (was: Re: the new TestExpectations syntax is landing soon)
This should be fixed now. I apologize for the problems. As Ojan said, rolling it out would've been tricky because the changes to the expectation syntax depended on this change. In retrospect it might've been better to have more time elapse between the initial change and the changes to the syntax but I didn't want to impose multiple changes to the format on people for an extended period of time. Some might wonder why this got through ... we had unit tests for both the python side and the html side of this, but not an integration test that ensured that the right thing went all the way through (it's not actually clear how to test this without being overly invasive on your checkout, but I'm going to keep thinking about it). Unfortunately, this also wasn't caught in a review and the tree happened to be fairly green yesterday so I missed it in garden-o-matic and on the bots (which don't really use results.html anyway) :(. -- Dirk On Thu, Sep 20, 2012 at 11:14 AM, Ojan Vafai o...@chromium.org wrote: Fixing the problem will likely take less time at this point than rolling the patch out as there are a number of dependent patches. The fix should be in soon. On Thu, Sep 20, 2012 at 11:10 AM, Geoffrey Garen gga...@apple.com wrote: I'd prefer to see the patch rolled out. Geoff On Sep 20, 2012, at 11:07 AM, Alexey Proskuryakov a...@webkit.org wrote: Now tracked as https://bugs.webkit.org/show_bug.cgi?id=97182. I think that we should close the tree if resolving this takes any significant time. Not being able to see how exactly tests are failing on other platforms is unacceptable. - WBR, Alexey Proskuryakov 20.09.2012, в 3:54, Osztrogonac Csaba o...@inf.u-szeged.hu написал(а): Unfortunately r129047 broke the results.html, see https://bugs.webkit.org/show_bug.cgi?id=96845#c9 for details. Dirk Pranke írta: These changes are now starting to land ... as of r129047, TEXT, IMAGE+TEXT, and AUDIO are no longer legal keywords in the TestExpectations syntax ... you should use FAIL instead. I will be landing the support for the new syntax as quickly as I can to minimize the transition period. Apologies for the inconvenience. -- Dirk On Wed, Sep 12, 2012 at 4:29 PM, Dirk Pranke dpra...@chromium.org wrote: Hi all, The new format of the much-debated TestExpectations syntax will be landing soon (hopefully in the next couple days). For those of who have forgotten / repressed the earlier debates, the new syntax looks something like: webkit.org/b/12345 [ Mac Vista] fast/html/keygen.html [ ImageOnlyFailure ] Andis documented in full at https://trac.webkit.org/wiki/TestExpectations#NewSyntaxNotquiteyetlanded . ( The [ and ] characters are delimiters, not EBNF optional markers, although those sections are in fact optional :) ). Note that the new syntax means that Skipped files are a syntactic subset of TestExpectations files, and I plan to convert all of the Skipped files to TestExpectations files via copy and paste shortly after the new syntax is landed, and then drop support for Skipped files (I will update ORWT to use the new files and treat any entry as a Skip). The plan for landing these changes is: 1) Add support for parsing the new lines and converting them back into the old format (internally) so that both syntaxes are supported 2) Convert all the existing files over 3) Make sure things aren't broken :) 4) Drop support for the old syntax I plan for this to all happen quickly, in less than a day. This means that if you have patches posted that modify those files they may become stale and need to be updated. Changes from the old syntax: 1. We use URLs (a specific whitelisted set; let me know if you want to add to it) instead of BUGWK12345 etc. 2. We use bug(dpranke) instead of BUGDPRANKE 3. We use CamelCase instead of SHOUTING 4. We use Failure to represent what used to be TEXT, IMAGE+TEXT, and AUDIO - these failures will be indistinguishable in the new world, meaning that you can't distinguish between text only and both image and text. Since only Chromium runs pixel tests by default, this shouldn't be a big deal. 5. We use ImageOnlyFailure to represent what used to be IMAGE 6. We use [ and ] for delimiters instead of : and = 7. We use # instead of // as a comment 7. WontFix will now imply Skip, i.e., tests marked WontFix will automatically be Skipped 8. WontFix and Skip will not require (or even allow) any other expectations, i.e., you can't say [ WontFix Crash ]. If you want to indicate that the test will crash if you actually run it, use a comment. 9. WontFix, Skip, Slow, and Rebaseline all move from the left hand side to the right. The only keywords on the left restrict which configurations the lines apply to. I will send out follow-up emails as this stuff lands. Please let me know if you have any questions. Thanks! -- Dirk
Re: [webkit-dev] Baselines ignoring metrics
After some limited amount of thought, I'm inclined to agree with Ryosuke and Ossy here. If we have tests that don't depend on checking the metrics, can they just be dumpAsText tests or reftests instead? I thought the initial motivation for --ignore-metrics was for new ports to at least confirm that the render trees had the right shape even if they didn't have the right metrics. It's unclear to me if this has ever been useful for people, though. On a related note (which I thought I mentioned in some ignore-metrics-related bug, but apparently at least not this one), I've been working on various render-tree-parsing-related features, and I've been thinking that it might be nice to have a smarter diff that could distinguish between fields that actually rendered differently due to different intrinsic metrics vs. fields that rendered differently because of different derived metrics (e.g., two text fields are different sizes because the text renders differently vs. two containing blocks that are also different because they contain the different text fields; if both container and containing block are off by one pixel, on the the contained block's diff is really interesting). But, I don't think (but am not yet 100% sure) that the existing rendertree format contains the information needed to do this accurately. -- Dirk On Wed, Sep 19, 2012 at 1:08 PM, Ryosuke Niwa rn...@webkit.org wrote: As ossy has suggested in the bug, we should just convert those tests that don't require metrics equivalency to dumpAsText tests. On Sep 19, 2012 2:53 AM, Allan Sandfeld Jensen k...@carewolf.com wrote: Hello WebKit, Some time ago I wrote a patch that would make the test runners detect baseline that had been generated with the --ignore-metrics flag, see https://bugs.webkit.org/show_bug.cgi?id=94815 . This makes it possible to generate baselines that would always be testing ignoring metrics, and which would help consolidate more expected results between platforms - see the second patch in the bug for an example of that. The cases I expect this to make sense in, are tests for css selectors, parsing and other places the final layout is not part of what is being tested. Ideally such new tests should be written using dumpText, but we have many old test- cases that has not been written that way, and many more that are imported from from external test suites which never will be. Should we apply the patch and start using ignore-metric baselines, and should we make some guidelines for what kind of tests this should be allowed on? Best regards `Allan Sandfeld ___ webkit-dev mailing list webkit-dev@lists.webkit.org http://lists.webkit.org/mailman/listinfo/webkit-dev ___ webkit-dev mailing list webkit-dev@lists.webkit.org http://lists.webkit.org/mailman/listinfo/webkit-dev ___ webkit-dev mailing list webkit-dev@lists.webkit.org http://lists.webkit.org/mailman/listinfo/webkit-dev
