On Wed, Oct 17, 2018 at 11:53 PM Boris Zbarsky <bzbar...@mit.edu> wrote: > > On 10/13/18 3:27 AM, Philip Jägenstedt wrote: > > Fiddling with these rules can reveal lots > > more potential issues, and if you like I could provide reports on that too. > > I would be pretty interested in that, yes. In particular, a report > where there is 1 "not PASS and not FAIL" and 3 "PASS" would be pretty > helpful, I suspect.
Rerunning my script it's apparent that unreliable Edge results [1] leads to the same tests being considered lone failures or not for the other browsers. So, I've use the same set of runs for this report of what you suggested: https://gist.github.com/foolip/e6014c9bcc8ca405219bf18542eb5d69 It's not a long list, so I checked them all and they are timeouts. This is sometimes the failure mode for genuine problems, so looking over these might be valuable. > By the way, I recently found some tests that fail when run directly but > pass in the harness. :( For example > http://w3c-test.org/html/infrastructure/common-dom-interfaces/collections/htmlallcollection.html > fails various subtests in all browsers due to the <div id="log"> being > in the DOM when running directly. Not really sure what we can do with that. That's a bit odd, the <div id="log"> is in the markup and would be when running manually or under automation. Are you sure that explains the difference? If it does, then just removing it from the markup and adapting any affected tests would be the way to go. I updated the test pretty recently, if you're confident it's broken can you file a wpt issue and assign me? [1] https://github.com/web-platform-tests/results-collection/issues/563 _______________________________________________ dev-platform mailing list dev-platform@lists.mozilla.org https://lists.mozilla.org/listinfo/dev-platform
