Hi Dean, On the run of Safari that was used for this report, the infrastructure test for ahem was actually passing: https://wpt.fyi/results/infrastructure/assumptions?sha=67152fdecd&product=chrome[stable]&product=edge[stable]&product=firefox[stable]&product=safari[experimental]
Are you sure that Ahem is the explanation for the failures, do you have a test that you think is actually passing and the wpt.fyi results are wrong? Clearly, having screenshots would make it easier to understand a situation like this, and it's something we've discussed a bit today: https://github.com/web-platform-tests/wpt.fyi/issues/57 On Fri, Oct 12, 2018 at 3:01 AM Dean Jackson <[email protected]> wrote: > > It turns out that many (most?) of the CSS failures are because we no longer > expose user-installed fonts, e.g. Ahem. > > Options: > > - update lots of tests to load Ahem via @font-face (yuck) > - allow Ahem to be used if installed (weird to special case one font, but > probably ok) > > Dean > > > On 12 Oct 2018, at 03:26, Philip Jägenstedt <[email protected]> wrote: > > > > Alright, I've written a one-off script [1] to find the Safari-only > > failures, and here's the output: > > https://gist.github.com/foolip/4d410ce79416bcdce71feb212159a02e > > > > Barring bugs, each of linked tests or one of its subtests should be > > failing in Safari Technology Preview and passing in stable versions of > > Chrome, Edge and Firefox. > > > > Numerically, most of the failures are in css (622), encoding (135) and > > html (60). With css, it's mostly css/CSS2. > > > > I hope looking through this may be of use to you! > > > > [1] https://github.com/foolip/ad-hoc-wpt-results-analysis > > > > On Mon, Oct 8, 2018 at 11:50 PM Philip Jägenstedt <[email protected]> > > wrote: > >> > >> That filtering capability unfortunately does not yet exist on wpt.fyi > >> but it's a high priority and actively being worked on: > >> https://github.com/web-platform-tests/wpt.fyi/issues/201 > >> > >> FWIW, I suspect that these purposes, comparing to the stable versions > >> of all *other* browsers might be the most useful: > >> https://wpt.fyi/results/?product=chrome%5Bstable%5D&product=edge%5Bstable%5D&product=firefox%5Bstable%5D&product=safari%5Bexperimental%5D&aligned > >> > >> Again, no way to filter on wpt.fyi, but I'll see if I can download the > >> full results and write a quick script. > >> > >> On Thu, Oct 4, 2018 at 11:49 PM Ryosuke Niwa <[email protected]> wrote: > >>> > >>> Thanks for the intriguing data, Philip. > >>> > >>> Is there a way to get a list of tests where all other browsers pass but > >>> Safari / WebKit fail? > >>> > >>> That would allow us to quickly identify the set of tests we can fix to > >>> improve the interoperability across browsers right away. > >>> > >>> - R. Niwa > >>> > >>> On Tue, Oct 2, 2018 at 3:45 AM Philip Jägenstedt <[email protected]> > >>> wrote: > >>>> > >>>> Hi WebKittens, > >>>> > >>>> Fresh off the bots, I'm excited to report more robust Safari results, > >>>> and that Safari WPT pass rates are clearly improving! Thanks to the > >>>> hard work of Mike Pennisi [1] we now have the first Safari 12 results: > >>>> https://wpt.fyi/results/?sha=ee2e69bfb1&product=safari-12.0 > >>>> > >>>> This uses the same setup as for Safari Technology Preview, which has > >>>> been running for a while [2] and are the results you see on the > >>>> "experimental" view: > >>>> https://wpt.fyi/results/?label=experimental > >>>> > >>>> This appears much more robust than the Safari 11 data we've collected > >>>> from Sauce Labs, and we can see a massive improvement between Safari > >>>> 11 and 12: > >>>> https://wpt.fyi/results/?sha=ee2e69bfb1&product=safari-11.1&product=safari-12.0&diff > >>>> > >>>> This lumps together infrastructure improvements as well as Safari > >>>> 11->12 improvements, but improvements in service-workers/ [3] stands > >>>> out, as well as in webdriver/, referrer-policy/, css/css-align/, and > >>>> others. (The effect of moving away from Sauce is mainly less > >>>> timeouts.) > >>>> > >>>> Also very interesting is to compare Safari 12 stable to TP: > >>>> https://wpt.fyi/results/?sha=ee2e69bfb1&product=safari-12.0&product=safari-12.1&diff > >>>> > >>>> One can tell that work is going in canvas-related things, > >>>> web-animations/, css/css-logical/ and more! \o/ > >>>> > >>>> I hope you'll all find these results valuable, and please report bugs > >>>> or feature requests here: > >>>> https://github.com/web-platform-tests/wpt.fyi/issues > >>>> > >>>> P.S. We're also trying to use use these diff views to spot > >>>> regressions. It's a bit hard to use, [4] but a fix in in progress [5] > >>>> and I might check back here when that works. I'll append to the end of > >>>> this email a non-exhaustive list of possible regressions already > >>>> possible to spot. > >>>> > >>>> [1] https://github.com/web-platform-tests/results-collection/issues/604 > >>>> [2] https://wpt.fyi/test-runs?labels=safari,experimental > >>>> [3] > >>>> https://wpt.fyi/results/service-workers?sha=ee2e69bfb1&product=safari-11.1&product=safari-12.0&diff=true > >>>> [4] https://github.com/web-platform-tests/wpt.fyi/issues/411 > >>>> [5] https://github.com/web-platform-tests/wpt.fyi/pull/609 > >>>> > >>>> P.P.S. Possible regressions in Safari TP: > >>>> https://wpt.fyi/results/css/vendor-imports/mozilla/mozilla-central-reftests/shapes1?sha=ee2e69bfb1&product=safari-12.0&product=safari-12.1 > >>>> https://wpt.fyi/results/service-workers/service-worker/extendable-event-async-waituntil.https.html?sha=ee2e69bfb1&product=safari-12.0&product=safari-12.1 > >>>> https://wpt.fyi/results/service-workers/service-worker/skip-waiting-installed.https.html?sha=ee2e69bfb1&product=safari-12.0&product=safari-12.1 > >>>> _______________________________________________ > >>>> webkit-dev mailing list > >>>> [email protected] > >>>> https://lists.webkit.org/mailman/listinfo/webkit-dev > > _______________________________________________ > > webkit-dev mailing list > > [email protected] > > https://lists.webkit.org/mailman/listinfo/webkit-dev > _______________________________________________ webkit-dev mailing list [email protected] https://lists.webkit.org/mailman/listinfo/webkit-dev
