On Wed, Oct 15, 2014 at 2:15 PM, Trevor Saunders <tbsaunde+mozi...@tbsaunde.org> wrote: > This morning tomcat decided to back bug 982842 and a bunch of dependant > bugs out for breaking some of the gaia device tests. As I understand > things, this is not the first time something like that has happened. > However I think that was a mistake, it treated those tests as tier 1 > when they pretty clearly do not meet the requirements in > https://wiki.mozilla.org/Sheriffing/Job_Visibility_Policy for tier 1 > tests. Those tests aren't even on treeherder / tbpl, much less runnable > from try. Also > https://developer.mozilla.org/en-US/Firefox_OS/Platform/Automated_testing/Gaia_unit_tests > doesn't document how these tests can be run with an emulator or > device. The visibility rules exist in part to make sure that tier 1 > tests can be easily reproduced and confirmed to work, however these > tests don't come close to being easy to run. The best explanation I've > heard for this state of afairs is that people want to get these tests to > meet the requirements, but given that we have accepted this excuse for > very few tests in the past I don't think that's good enough here. So, > until someone gets these tests to be visible on treeherder I don't think > we should treat them as a pseudo tier 1 test suite.
There's nothing magical about functionality that's tested in our automation. I.e. it's not on average any more or less important that a lot of other functionality that's not tested there. The only thing that's different about things in automation is that it's much easier for us to see when it breaks, and what caused the breakage. But any type of regression is cause for backout. When patches that cause regressions are allowed to sit in the tree that causes lost productivity for a lot of people. It definitely sucks a lot that we don't have automated testing for more stuff in B2G. It's a long slug but it's slowly getting better. This is a huge problem not just for people like you who have a harder time to see if you cause regression. But it's an even bigger problem for B2G developers who are having to work on a code base that is constantly seeing regressions. Regressions that sit in the tree make it dramatically much harder to write and test other patches. It's generally much better to back the offending patch out to allow everyone else to go at full speed. / Jonas _______________________________________________ dev-platform mailing list dev-platform@lists.mozilla.org https://lists.mozilla.org/listinfo/dev-platform
