Re: New backout policy for Ts regressions on mozilla-inbound

[Ehsan Akhgari]

I have documented this here: <
https://developer.mozilla.org/en-US/docs/Developer_Guide/Committing_Rules_and_Responsibilities#Dealing_with_performance_regressions
>

--
Ehsan



On Thu, Oct 18, 2012 at 2:05 PM, Ehsan Akhgari wrote:

> Hi everyone,
>
> As part of our efforts to get more value out of the Talos test suite for
> preventing performance regressions, we believe that we are now ready to put
> a first set of measures against startup time regressions.  We will start by
> imposing a new backout policy for mozilla-inbound checkins for regressions
> more than 4% on any given platform.  If your patch falls in a range which
> causes more than 4% Ts regression, it will be backed out by our sheriffs
> together with the rest of the patches in that range, and you can only
> reland after you fix the regression by testing locally or on the try server.
>
> The 4% threshold has been chosen based on anecdotal evidence on the most
> recent Ts regressions that we have seen, and is too generous, but we will
> be working to improve the reporting and regression detection systems
> better, and as those get improved, we would feel more comfortable with
> imposing this policy on other Talos tests with tighter thresholds.
>
> Please let me know if you have any questions.
>
> Cheers,
> --
> Ehsan
> 
>
___
dev-platform mailing list
dev-platform@lists.mozilla.org
https://lists.mozilla.org/listinfo/dev-platform

Re: New backout policy for Ts regressions on mozilla-inbound

[Ehsan Akhgari]

On 2012-10-19 12:13 PM, Dao wrote:

On 18.10.2012 20:05, Ehsan Akhgari wrote:

If your patch falls in a range which
causes more than 4% Ts regression, it will be backed out by our sheriffs
together with the rest of the patches in that range, and you can only
reland after you fix the regression by testing locally or on the try
server.


The last point seems excessive. If multiple patches are in the range and
you think yours is likely innocent, I think you should be allowed to
re-land. Worst case is you'll get backed out a second time, but this
seems like it's going to have less time overhead than manually messing
with talos.


Sure, we still allow good judgement, so if you believe your patch was 
innocent, then obviously there is no regression for you to fix, so you 
can just reland.


And if Talos proves that your patch is indeed at fault, you will get 
backed out again!  ;-)


Ehsan

___
dev-platform mailing list
dev-platform@lists.mozilla.org
https://lists.mozilla.org/listinfo/dev-platform

Re: New backout policy for Ts regressions on mozilla-inbound

[Justin Lebar]

>> If your patch falls in a range which
>> causes more than 4% Ts regression, it will be backed out by our sheriffs
>> together with the rest of the patches in that range, and you can only
>> reland after you fix the regression by testing locally or on the try
>> server.

Our tools for comparing talos results are in a pitiful state.  I know
that improving them is part of the SfN effort, but I want to emphasize
that if we're backing out patches for Ts regressions now, we need
these tools yesterday.

-Justin

On Fri, Oct 19, 2012 at 12:13 PM, Dao  wrote:
> On 18.10.2012 20:05, Ehsan Akhgari wrote:
>>
>> If your patch falls in a range which
>> causes more than 4% Ts regression, it will be backed out by our sheriffs
>> together with the rest of the patches in that range, and you can only
>> reland after you fix the regression by testing locally or on the try
>> server.
>
>
> The last point seems excessive. If multiple patches are in the range and you
> think yours is likely innocent, I think you should be allowed to re-land.
> Worst case is you'll get backed out a second time, but this seems like it's
> going to have less time overhead than manually messing with talos.
>
>
> ___
> dev-platform mailing list
> dev-platform@lists.mozilla.org
> https://lists.mozilla.org/listinfo/dev-platform
___
dev-platform mailing list
dev-platform@lists.mozilla.org
https://lists.mozilla.org/listinfo/dev-platform

Re: New backout policy for Ts regressions on mozilla-inbound

[Dao]

On 18.10.2012 20:05, Ehsan Akhgari wrote:

If your patch falls in a range which
causes more than 4% Ts regression, it will be backed out by our sheriffs
together with the rest of the patches in that range, and you can only
reland after you fix the regression by testing locally or on the try server.


The last point seems excessive. If multiple patches are in the range and 
you think yours is likely innocent, I think you should be allowed to 
re-land. Worst case is you'll get backed out a second time, but this 
seems like it's going to have less time overhead than manually messing 
with talos.


___
dev-platform mailing list
dev-platform@lists.mozilla.org
https://lists.mozilla.org/listinfo/dev-platform

Re: New backout policy for Ts regressions on mozilla-inbound

[Chris AtLee]

On 18/10/12 06:44 PM, Justin Lebar wrote:
>> Do we still have the bug where a test that finishes first, but is from a
>> later cset (say a later cset IMPROVES Ts by 4% or more) would make us
>> think we regressed it on an earlier cset if that earlier talos run
>> finishes later?
>>
>> Such that we set graph points by the time the test finished, not time
>> the push was, etc.
> 
> https://bugzilla.mozilla.org/show_bug.cgi?id=688534

That applies to the rendering of the graphs on graphs.m.o only. The
regression detection uses the push time to order the results.
___
dev-platform mailing list
dev-platform@lists.mozilla.org
https://lists.mozilla.org/listinfo/dev-platform

Re: New backout policy for Ts regressions on mozilla-inbound

[Ehsan Akhgari]

On 2012-10-19 11:39 AM, Armen Zambrano G. wrote:

Is there a place where this will be document?
I would like to keep an eye on what gets added or point people at it.


Not that I know of!  Please feel free to start a wiki page or something. 
 ;-)


Ehsan

___
dev-platform mailing list
dev-platform@lists.mozilla.org
https://lists.mozilla.org/listinfo/dev-platform

Re: New backout policy for Ts regressions on mozilla-inbound

[Armen Zambrano G.]

Is there a place where this will be document?
I would like to keep an eye on what gets added or point people at it.

This is awesome!
Looking forward to see how it goes.

thanks Ehsan

On 2012-10-18 2:05 PM, Ehsan Akhgari wrote:

Hi everyone,

As part of our efforts to get more value out of the Talos test suite for
preventing performance regressions, we believe that we are now ready to put
a first set of measures against startup time regressions.  We will start by
imposing a new backout policy for mozilla-inbound checkins for regressions
more than 4% on any given platform.  If your patch falls in a range which
causes more than 4% Ts regression, it will be backed out by our sheriffs
together with the rest of the patches in that range, and you can only
reland after you fix the regression by testing locally or on the try server.

The 4% threshold has been chosen based on anecdotal evidence on the most
recent Ts regressions that we have seen, and is too generous, but we will
be working to improve the reporting and regression detection systems
better, and as those get improved, we would feel more comfortable with
imposing this policy on other Talos tests with tighter thresholds.

Please let me know if you have any questions.

Cheers,
--
Ehsan




___
dev-platform mailing list
dev-platform@lists.mozilla.org
https://lists.mozilla.org/listinfo/dev-platform

Re: New backout policy for Ts regressions on mozilla-inbound

[Justin Lebar]

> Do we still have the bug where a test that finishes first, but is from a
> later cset (say a later cset IMPROVES Ts by 4% or more) would make us
> think we regressed it on an earlier cset if that earlier talos run
> finishes later?
>
> Such that we set graph points by the time the test finished, not time
> the push was, etc.

https://bugzilla.mozilla.org/show_bug.cgi?id=688534

On Thu, Oct 18, 2012 at 6:26 PM, Justin Wood (Callek)  wrote:
> Ehsan Akhgari wrote:
>> Hi everyone,
>>
>> As part of our efforts to get more value out of the Talos test suite for
>> preventing performance regressions, we believe that we are now ready to put
>> a first set of measures against startup time regressions.  We will start by
>> imposing a new backout policy for mozilla-inbound checkins for regressions
>> more than 4% on any given platform.  If your patch falls in a range which
>> causes more than 4% Ts regression, it will be backed out by our sheriffs
>> together with the rest of the patches in that range, and you can only
>> reland after you fix the regression by testing locally or on the try server.
>>
>> The 4% threshold has been chosen based on anecdotal evidence on the most
>> recent Ts regressions that we have seen, and is too generous, but we will
>> be working to improve the reporting and regression detection systems
>> better, and as those get improved, we would feel more comfortable with
>> imposing this policy on other Talos tests with tighter thresholds.
>>
>> Please let me know if you have any questions.
>>
>
> Do we still have the bug where a test that finishes first, but is from a
> later cset (say a later cset IMPROVES Ts by 4% or more) would make us
> think we regressed it on an earlier cset if that earlier talos run
> finishes later?
>
> Such that we set graph points by the time the test finished, not time
> the push was, etc.
>
> ~Justin Wood (Callek)
>
> ___
> dev-platform mailing list
> dev-platform@lists.mozilla.org
> https://lists.mozilla.org/listinfo/dev-platform
___
dev-platform mailing list
dev-platform@lists.mozilla.org
https://lists.mozilla.org/listinfo/dev-platform

Re: New backout policy for Ts regressions on mozilla-inbound

[Justin Wood (Callek)]

Ehsan Akhgari wrote:
> Hi everyone,
> 
> As part of our efforts to get more value out of the Talos test suite for
> preventing performance regressions, we believe that we are now ready to put
> a first set of measures against startup time regressions.  We will start by
> imposing a new backout policy for mozilla-inbound checkins for regressions
> more than 4% on any given platform.  If your patch falls in a range which
> causes more than 4% Ts regression, it will be backed out by our sheriffs
> together with the rest of the patches in that range, and you can only
> reland after you fix the regression by testing locally or on the try server.
> 
> The 4% threshold has been chosen based on anecdotal evidence on the most
> recent Ts regressions that we have seen, and is too generous, but we will
> be working to improve the reporting and regression detection systems
> better, and as those get improved, we would feel more comfortable with
> imposing this policy on other Talos tests with tighter thresholds.
> 
> Please let me know if you have any questions.
> 

Do we still have the bug where a test that finishes first, but is from a
later cset (say a later cset IMPROVES Ts by 4% or more) would make us
think we regressed it on an earlier cset if that earlier talos run
finishes later?

Such that we set graph points by the time the test finished, not time
the push was, etc.

~Justin Wood (Callek)

___
dev-platform mailing list
dev-platform@lists.mozilla.org
https://lists.mozilla.org/listinfo/dev-platform

New backout policy for Ts regressions on mozilla-inbound

[Ehsan Akhgari]

Hi everyone,

As part of our efforts to get more value out of the Talos test suite for
preventing performance regressions, we believe that we are now ready to put
a first set of measures against startup time regressions.  We will start by
imposing a new backout policy for mozilla-inbound checkins for regressions
more than 4% on any given platform.  If your patch falls in a range which
causes more than 4% Ts regression, it will be backed out by our sheriffs
together with the rest of the patches in that range, and you can only
reland after you fix the regression by testing locally or on the try server.

The 4% threshold has been chosen based on anecdotal evidence on the most
recent Ts regressions that we have seen, and is too generous, but we will
be working to improve the reporting and regression detection systems
better, and as those get improved, we would feel more comfortable with
imposing this policy on other Talos tests with tighter thresholds.

Please let me know if you have any questions.

Cheers,
--
Ehsan

___
dev-platform mailing list
dev-platform@lists.mozilla.org
https://lists.mozilla.org/listinfo/dev-platform