On 9/12/17 7:02 AM, James Graham wrote:
On 12/09/17 14:55, Andrew Halberstadt wrote:
On Mon, Sep 11, 2017 at 10:33 PM Robert O'Callahan
<rob...@ocallahan.org>
wrote:
On Tue, Sep 12, 2017 at 11:38 AM, Andrew Halberstadt <
ahalberst...@mozilla.com> wrote:
I don't think so, that data already exists and is query-able from
ActiveData:
https://activedata.allizom.org/tools/query.html#query_id=8pDOpeni
That query tells you about disabled tests, but doesn't know about
*why* a
test was disabled. E.g. you can't distinguish tests disabled because
they're not expected to work on some (or all) platforms from tests that
were disabled for intermittent failures that should, in principle,
be fixed.
Rob
True, though I don't know that gps' proposal would solve that either.
But this is a good idea, and is easy to solve from a technical
standpoint.
We'd just need to agree on some standard manifest keys:
I'm pretty sure that the problem isn't technical, but actually getting
people to do that consistently (plus retrofitting the data onto
thousands of currently disabled tests). You would at least have to add
a lint and a free pass for all the existing tests.
If you don't need to get it perfect, you can always do an hg blame and
look at the commit message that disabled a test, then test it for
whatever makes sense -- mentioning "intermittent", the author being a
sheriff, or something similar. Or at least gathering them all together
so you can skim through them and mark which ones sound like they were
disabled for being intermittent.
I'm not just handwaving here, or at least not completely -- I have
something like this set up for semi-automatically generating
Spidermonkey release notes on changed API functions, where it looks at
the revision where each function was added or removed and adds the bug
number to the appropriate list, with some attempt made to avoid
whitespace changes and reordering within the header files. (It doesn't
work very well, mind.)
_______________________________________________
dev-platform mailing list
dev-platform@lists.mozilla.org
https://lists.mozilla.org/listinfo/dev-platform