On Wed, Oct 15, 2014 at 05:36:00PM -0400, Ryan VanderMeulen wrote: > On 10/15/2014 5:15 PM, Trevor Saunders wrote: > >Hi, > > > >This morning tomcat decided to back bug 982842 and a bunch of dependant > >bugs out for breaking some of the gaia device tests. As I understand > >things, this is not the first time something like that has happened. > >However I think that was a mistake, it treated those tests as tier 1 > >when they pretty clearly do not meet the requirements in > >https://wiki.mozilla.org/Sheriffing/Job_Visibility_Policy for tier 1 > >tests. Those tests aren't even on treeherder / tbpl, much less runnable > >from try. Also > >https://developer.mozilla.org/en-US/Firefox_OS/Platform/Automated_testing/Gaia_unit_tests > >doesn't document how these tests can be run with an emulator or > >device. The visibility rules exist in part to make sure that tier 1 > >tests can be easily reproduced and confirmed to work, however these > >tests don't come close to being easy to run. The best explanation I've > >heard for this state of afairs is that people want to get these tests to > >meet the requirements, but given that we have accepted this excuse for > >very few tests in the past I don't think that's good enough here. So, > >until someone gets these tests to be visible on treeherder I don't think > >we should treat them as a pseudo tier 1 test suite. > > > >Trev > > > > Oftentimes, when on-device tests break, there are also real regressions on > the phones themselves. At which point, how is it any different from when we > backout a patch for nightly bustage that our automation didn't catch?
For one thing most things we regress but don't catch with tests we fix in place. IME its only when things are horribly broken that we revert things immediately, and there doesn't seem to be evidence for this being more than a regression of some sort. > Note that I'm not offering any opinion on how sensible it is that we lack > the ability to catch regressions like these in our CI, but I'm not agreeing > that backing out was the wrong decision under the circumstances. QA has > always had the ability to request backouts for functional regressions > regardless of what product they're affecting. I'm not aware of a rule stating that, and I tend to disagree with it assuming it does exist somewhere. As for Ed's point on visibility rules not being primarily about backout that's reasonable, but I'm not aware of a better document, links welcome :) Trev > > -Ryan > _______________________________________________ > dev-platform mailing list > dev-platform@lists.mozilla.org > https://lists.mozilla.org/listinfo/dev-platform
signature.asc
Description: Digital signature
_______________________________________________ dev-platform mailing list dev-platform@lists.mozilla.org https://lists.mozilla.org/listinfo/dev-platform
