Re: treating B2G device tests as tier 1

On Thu, Oct 16, 2014 at 10:20 PM, Boris Zbarsky wrote: > I think what Bobby was actually asking for is this: > > If a patch lands and is green in continuous integration but then we have > some other tests somewhere (hidden, run manually, whatever) that start > failing because of this patch, and w

Re: treating B2G device tests as tier 1

On 2014-10-16, 4:20 PM, Boris Zbarsky wrote: On 10/16/14, 4:01 PM, Ehsan Akhgari wrote: Sure. But you're just describing why tests are useful and an absolute necessity. :-) I think what Bobby was asking for is a much stronger ask that is not really attainable. I think what Bobby was actuall

Re: treating B2G device tests as tier 1

On 10/16/14, 4:01 PM, Ehsan Akhgari wrote: Sure. But you're just describing why tests are useful and an absolute necessity. :-) I think what Bobby was asking for is a much stronger ask that is not really attainable. I think what Bobby was actually asking for is this: If a patch lands and is

Re: treating B2G device tests as tier 1

On Thu, Oct 16, 2014 at 1:01 PM, Ehsan Akhgari wrote: > On 2014-10-16, 3:56 PM, Dale Harvey wrote: >> >> >> >> On 16 October 2014 20:55, Ehsan Akhgari > > wrote: >> >> On 2014-10-16, 1:52 PM, Bobby Holley wrote: >> >> On Thu, Oct 16, 2014 at 7:04 PM, Ehs

Re: treating B2G device tests as tier 1

On 2014-10-16, 3:56 PM, Dale Harvey wrote: On 16 October 2014 20:55, Ehsan Akhgari mailto:ehsan.akhg...@gmail.com>> wrote: On 2014-10-16, 1:52 PM, Bobby Holley wrote: On Thu, Oct 16, 2014 at 7:04 PM, Ehsan Akhgari mailto:ehsan.akhg...@gmail.com>

Re: treating B2G device tests as tier 1

On 16 October 2014 20:55, Ehsan Akhgari wrote: > On 2014-10-16, 1:52 PM, Bobby Holley wrote: > >> On Thu, Oct 16, 2014 at 7:04 PM, Ehsan Akhgari > > wrote: >> >> I don't think it's reasonable to assume that the person doing the >> backout has the time or th

Re: treating B2G device tests as tier 1

On 2014-10-16, 1:52 PM, Bobby Holley wrote: On Thu, Oct 16, 2014 at 7:04 PM, Ehsan Akhgari mailto:ehsan.akhg...@gmail.com>> wrote: I don't think it's reasonable to assume that the person doing the backout has the time or the expertise to add a test for the broken functionality. Not

Re: treating B2G device tests as tier 1

On Thu, Oct 16, 2014 at 7:04 PM, Ehsan Akhgari wrote: I don't think it's reasonable to assume that the person doing the backout > has the time or the expertise to add a test for the broken functionality. > Not the sheriff certainly, but I think if the regression is severe enough to warrant this

Re: treating B2G device tests as tier 1

On Thu, Oct 16, 2014 at 10:04 AM, Ehsan Akhgari wrote: > On 2014-10-16, 2:51 AM, Bobby Holley wrote: >> >> This has irked me before too. An obvious compromise would that the >> backout proceeds, but it must include a test that would have failed on >> CI when the patch was landed. This puts the onu

Re: treating B2G device tests as tier 1

On 2014-10-16, 2:51 AM, Bobby Holley wrote: This has irked me before too. An obvious compromise would that the backout proceeds, but it must include a test that would have failed on CI when the patch was landed. This puts the onus on the owners of the broken functionality to make sure that this s

Re: treating B2G device tests as tier 1

This has irked me before too. An obvious compromise would that the backout proceeds, but it must include a test that would have failed on CI when the patch was landed. This puts the onus on the owners of the broken functionality to make sure that this supposedly-critical functionality has basic smo

Re: treating B2G device tests as tier 1

On 2014-10-15, 10:19 PM, Karl Tomlinson wrote: Jonas Sicking writes: But any type of regression is cause for backout. While I agree regressions are bad, this isn't the usual process. If it were, then I wouldn't bother filing bugs, but merely back out the offending change. There is some kind

Re: treating B2G device tests as tier 1

Jonas Sicking writes: > But any type of regression is cause for backout. While I agree regressions are bad, this isn't the usual process. If it were, then I wouldn't bother filing bugs, but merely back out the offending change. There is some kind test for whether the regression costs more than

Re: treating B2G device tests as tier 1

On Wed, Oct 15, 2014 at 2:15 PM, Trevor Saunders wrote: > This morning tomcat decided to back bug 982842 and a bunch of dependant > bugs out for breaking some of the gaia device tests. As I understand > things, this is not the first time something like that has happened. > However I think that wa

Re: treating B2G device tests as tier 1

On Wed, Oct 15, 2014 at 05:36:00PM -0400, Ryan VanderMeulen wrote: > On 10/15/2014 5:15 PM, Trevor Saunders wrote: > >Hi, > > > >This morning tomcat decided to back bug 982842 and a bunch of dependant > >bugs out for breaking some of the gaia device tests. As I understand > >things, this is not th

Re: treating B2G device tests as tier 1

On 15/10/2014 22:15, Trevor Saunders wrote: However I think that was a mistake, it treated those tests as tier 1 when they pretty clearly do not meet the requirements in https://wiki.mozilla.org/Sheriffing/Job_Visibility_Policy ... The visibility rules exist in part to make sure that tier 1 tes

Re: treating B2G device tests as tier 1

On 10/15/2014 5:15 PM, Trevor Saunders wrote: Hi, This morning tomcat decided to back bug 982842 and a bunch of dependant bugs out for breaking some of the gaia device tests. As I understand things, this is not the first time something like that has happened. However I think that was a mistake,

treating B2G device tests as tier 1

Hi, This morning tomcat decided to back bug 982842 and a bunch of dependant bugs out for breaking some of the gaia device tests. As I understand things, this is not the first time something like that has happened. However I think that was a mistake, it treated those tests as tier 1 when they pret