I filed https://github.com/w3c/web-platform-tests/issues/8391 for the WPT server performances issues. As of upstreaming the tests, would the following solve identified issues and keep the upstream process lightweight enough? - Meta tag each upstreamed test to identify this is an upstreamed version of a WebKit test (and maybe which WebKit test). - Keep relative paths for upstreamed tests. y
Le ven. 17 nov. 2017 à 23:02, youenn fablet <youe...@gmail.com> a écrit : > Thanks for taking the time to share this additional information. > I think this is helpful to make progress. > Please see inline for some more comments related to the points you are > bringing to the table. > > Stepping back from the WPT server specific issue, being optimistic and > assuming that we are able to run WPT tests with good enough performances. > Migration potential downsides I heard so far: > - Prioritization of tests to investigate might be harder. > - Sharing tests with other teams might be harder if subresource links are > not relative. > These two seem like solvable issues to me. > > Le ven. 17 nov. 2017 à 10:09, Alexey Proskuryakov <a...@apple.com> a écrit : > >> >> 17 нояб. 2017 г., в 9:18, youenn fablet <youe...@gmail.com> написал(а): >> >> >> Chris recently noticed that some heavily used files (testharness*) were >> cacheable through Apache but not WPT. >> This is now fixed and should improve WPT performances. >> >> >> This is part of <https://bugs.webkit.org/show_bug.cgi?id=178277>, >> another part is that the server is 10x slower than Apache. >> > > Other measurements showed 3x slower, which makes it still worthwhile to > explore. > We need to be cautious here though since optimization is all about chasing > where time is actually spent. > > If we can prove to ourselves that this is an important issue, we should > discuss with the WPT community to see how to fix this issue. > In addition to better caching, other optimization like > https://github.com/w3c/wptserve/pull/86 may bring some additional benefit. > > If we do not find any good solution at WPT server level, we still have the > option to run some tests as file-based. > Ryosuke mentioned the possibility to classify tests at import time by > checking their results when served through WPT server and as file URLs. > > I just tested on my MacBook Pro, and WPT tests took 23% of time while >> being only 9% of the total count. Taking in mind that WebKit own tests have >> higher value due to the way we choose what to test (see below), that's not >> a great story in my opinion. >> > > These numbers are difficult to interpret. > WPT authoring style is multiple tests in a single file, which is bad for > stability but potentially good for performances. > WebKit usually prefers one file/one test. > > If we want to talk to WPT community about this, we need to provide some > more tangible numbers. > We could try to run a large subset of WPT tests that run the same through > Apache and WPT. > I just did that on a small subset of IDB tests. This seems to show > something like a 25% slowdown. > > One other thing that we discussed before was the operational complexity of >> running WPT tests. We frequently need to share tests with people who don't >> work on WebKit directly, but have the need to edit and run our tests. >> Inability to drag and drop a local copy into a Safari window is a deterrent >> to addressing problems caught by the tests. I think that the response we >> got was that tests will continue to require a server to run. >> > > Tests are available at https://w3c-test.org which makes it easy to share > through any tool supporting hyperlinks. > A webarchive can also be made so that it is easy to share and probably > edit such tests. > Tools like jsfiddle are also a great way to create/share/edit tests. > I received several bug reports on bugzilla using it and this proved to be > efficient. > > >> Let me explain why I think that WebKit tests are often more valuable as >> regression tests than WPT tests are. We add tests as we fix bugs, so we >> know that the tests are generally for problems that have a high impact on >> users and developers - that's because someone actually discovered the >> problem, and someone prioritized it highly enough to fix. We also know that >> our tests cover code that is error prone, which is why we had bugs in the >> first place. Of course, anything can be broken, but certain things are less >> likely to. Compliance tests written for specs are also valuable, but at >> some point we need to prioritize which tests to investigate and even to run. >> > > I don't really see why we should prioritize the tests to run when all of > them provide clear value to some WebKit members. > I agree that we need to prioritize tests we investigate. There can be a > solution inside WPT, like adding WebKit specific metadata so that: > - WPT contributors would communicate with WebKit members whenever changing > such tests > - WebKit contributors would prioritize WPT-WebKit-metadata failing tests > > That said, if these tests are so beneficial to WebKit, they are > potentially very useful to other teams as well. > And vice-versa, we might find really good WPT tests that show useful > crashes and failures in WebKit. > I am experiencing that nowadays with WPT service worker tests. >
_______________________________________________ webkit-dev mailing list webkit-dev@lists.webkit.org https://lists.webkit.org/mailman/listinfo/webkit-dev