On 9/12/17 7:02 AM, James Graham wrote:
On 12/09/17 14:55, Andrew Halberstadt wrote:
On Mon, Sep 11, 2017 at 10:33 PM Robert O'Callahan <rob...@ocallahan.org>
wrote:

On Tue, Sep 12, 2017 at 11:38 AM, Andrew Halberstadt <
ahalberst...@mozilla.com> wrote:

I don't think so, that data already exists and is query-able from
ActiveData:
https://activedata.allizom.org/tools/query.html#query_id=8pDOpeni


That query tells you about disabled tests, but doesn't know about *why* a
test was disabled. E.g. you can't distinguish tests disabled because
they're not expected to work on some (or all) platforms from tests that
were disabled for intermittent failures that should, in principle, be fixed.

Rob


True, though I don't know that gps' proposal would solve that either.

But this is a good idea, and is easy to solve from a technical standpoint.
We'd just need to agree on some standard manifest keys:

I'm pretty sure that the problem isn't technical, but actually getting people to do that consistently (plus retrofitting the data onto thousands of currently disabled tests). You would at least have to add a lint and a free pass for all the existing tests.

If you don't need to get it perfect, you can always do an hg blame and look at the commit message that disabled a test, then test it for whatever makes sense -- mentioning "intermittent", the author being a sheriff, or something similar. Or at least gathering them all together so you can skim through them and mark which ones sound like they were disabled for being intermittent.

I'm not just handwaving here, or at least not completely -- I have something like this set up for semi-automatically generating Spidermonkey release notes on changed API functions, where it looks at the revision where each function was added or removed and adds the bug number to the appropriate list, with some attempt made to avoid whitespace changes and reordering within the header files. (It doesn't work very well, mind.)

_______________________________________________
dev-platform mailing list
dev-platform@lists.mozilla.org
https://lists.mozilla.org/listinfo/dev-platform

Reply via email to