On 10/15/2014 5:15 PM, Trevor Saunders wrote:
Hi,
This morning tomcat decided to back bug 982842 and a bunch of dependant
bugs out for breaking some of the gaia device tests. As I understand
things, this is not the first time something like that has happened.
However I think that was a mistake, it treated those tests as tier 1
when they pretty clearly do not meet the requirements in
https://wiki.mozilla.org/Sheriffing/Job_Visibility_Policy for tier 1
tests. Those tests aren't even on treeherder / tbpl, much less runnable
from try. Also
https://developer.mozilla.org/en-US/Firefox_OS/Platform/Automated_testing/Gaia_unit_tests
doesn't document how these tests can be run with an emulator or
device. The visibility rules exist in part to make sure that tier 1
tests can be easily reproduced and confirmed to work, however these
tests don't come close to being easy to run. The best explanation I've
heard for this state of afairs is that people want to get these tests to
meet the requirements, but given that we have accepted this excuse for
very few tests in the past I don't think that's good enough here. So,
until someone gets these tests to be visible on treeherder I don't think
we should treat them as a pseudo tier 1 test suite.
Trev
Oftentimes, when on-device tests break, there are also real regressions
on the phones themselves. At which point, how is it any different from
when we backout a patch for nightly bustage that our automation didn't
catch?
Note that I'm not offering any opinion on how sensible it is that we
lack the ability to catch regressions like these in our CI, but I'm not
agreeing that backing out was the wrong decision under the
circumstances. QA has always had the ability to request backouts for
functional regressions regardless of what product they're affecting.
-Ryan
_______________________________________________
dev-platform mailing list
dev-platform@lists.mozilla.org
https://lists.mozilla.org/listinfo/dev-platform
