On Mon, Sep 11, 2017 at 10:33 PM Robert O'Callahan <rob...@ocallahan.org> wrote:
> On Tue, Sep 12, 2017 at 11:38 AM, Andrew Halberstadt < > ahalberst...@mozilla.com> wrote: > >> I don't think so, that data already exists and is query-able from >> ActiveData: >> https://activedata.allizom.org/tools/query.html#query_id=8pDOpeni > > > That query tells you about disabled tests, but doesn't know about *why* a > test was disabled. E.g. you can't distinguish tests disabled because > they're not expected to work on some (or all) platforms from tests that > were disabled for intermittent failures that should, in principle, be fixed. > > Rob > True, though I don't know that gps' proposal would solve that either. But this is a good idea, and is easy to solve from a technical standpoint. We'd just need to agree on some standard manifest keys: [test_foo.html] skip-if = <condition> reason = {'intermittent', 'fail', ... } bugs = 1234567, 1234576 When we log the skip, we make sure this metadata makes it into the structured log. Then tools like ActiveData or a manifest parsing mach command can query it. Reftest and wpt complicate matters a bit, but we could figure out something similar for them. -Andrew _______________________________________________ dev-platform mailing list dev-platform@lists.mozilla.org https://lists.mozilla.org/listinfo/dev-platform
