Re: New backout policy for Ts regressions on mozilla-inbound
I have documented this here: https://developer.mozilla.org/en-US/docs/Developer_Guide/Committing_Rules_and_Responsibilities#Dealing_with_performance_regressions -- Ehsan http://ehsanakhgari.org/ On Thu, Oct 18, 2012 at 2:05 PM, Ehsan Akhgari ehsan.akhg...@gmail.comwrote: Hi everyone, As part of our efforts to get more value out of the Talos test suite for preventing performance regressions, we believe that we are now ready to put a first set of measures against startup time regressions. We will start by imposing a new backout policy for mozilla-inbound checkins for regressions more than 4% on any given platform. If your patch falls in a range which causes more than 4% Ts regression, it will be backed out by our sheriffs together with the rest of the patches in that range, and you can only reland after you fix the regression by testing locally or on the try server. The 4% threshold has been chosen based on anecdotal evidence on the most recent Ts regressions that we have seen, and is too generous, but we will be working to improve the reporting and regression detection systems better, and as those get improved, we would feel more comfortable with imposing this policy on other Talos tests with tighter thresholds. Please let me know if you have any questions. Cheers, -- Ehsan http://ehsanakhgari.org/ ___ dev-platform mailing list dev-platform@lists.mozilla.org https://lists.mozilla.org/listinfo/dev-platform
Re: New backout policy for Ts regressions on mozilla-inbound
Is there a place where this will be document? I would like to keep an eye on what gets added or point people at it. This is awesome! Looking forward to see how it goes. thanks Ehsan On 2012-10-18 2:05 PM, Ehsan Akhgari wrote: Hi everyone, As part of our efforts to get more value out of the Talos test suite for preventing performance regressions, we believe that we are now ready to put a first set of measures against startup time regressions. We will start by imposing a new backout policy for mozilla-inbound checkins for regressions more than 4% on any given platform. If your patch falls in a range which causes more than 4% Ts regression, it will be backed out by our sheriffs together with the rest of the patches in that range, and you can only reland after you fix the regression by testing locally or on the try server. The 4% threshold has been chosen based on anecdotal evidence on the most recent Ts regressions that we have seen, and is too generous, but we will be working to improve the reporting and regression detection systems better, and as those get improved, we would feel more comfortable with imposing this policy on other Talos tests with tighter thresholds. Please let me know if you have any questions. Cheers, -- Ehsan http://ehsanakhgari.org/ ___ dev-platform mailing list dev-platform@lists.mozilla.org https://lists.mozilla.org/listinfo/dev-platform
Re: New backout policy for Ts regressions on mozilla-inbound
On 2012-10-19 11:39 AM, Armen Zambrano G. wrote: Is there a place where this will be document? I would like to keep an eye on what gets added or point people at it. Not that I know of! Please feel free to start a wiki page or something. ;-) Ehsan ___ dev-platform mailing list dev-platform@lists.mozilla.org https://lists.mozilla.org/listinfo/dev-platform
Re: New backout policy for Ts regressions on mozilla-inbound
On 18/10/12 06:44 PM, Justin Lebar wrote: Do we still have the bug where a test that finishes first, but is from a later cset (say a later cset IMPROVES Ts by 4% or more) would make us think we regressed it on an earlier cset if that earlier talos run finishes later? Such that we set graph points by the time the test finished, not time the push was, etc. https://bugzilla.mozilla.org/show_bug.cgi?id=688534 That applies to the rendering of the graphs on graphs.m.o only. The regression detection uses the push time to order the results. ___ dev-platform mailing list dev-platform@lists.mozilla.org https://lists.mozilla.org/listinfo/dev-platform
Re: New backout policy for Ts regressions on mozilla-inbound
If your patch falls in a range which causes more than 4% Ts regression, it will be backed out by our sheriffs together with the rest of the patches in that range, and you can only reland after you fix the regression by testing locally or on the try server. Our tools for comparing talos results are in a pitiful state. I know that improving them is part of the SfN effort, but I want to emphasize that if we're backing out patches for Ts regressions now, we need these tools yesterday. -Justin On Fri, Oct 19, 2012 at 12:13 PM, Dao d...@design-noir.de wrote: On 18.10.2012 20:05, Ehsan Akhgari wrote: If your patch falls in a range which causes more than 4% Ts regression, it will be backed out by our sheriffs together with the rest of the patches in that range, and you can only reland after you fix the regression by testing locally or on the try server. The last point seems excessive. If multiple patches are in the range and you think yours is likely innocent, I think you should be allowed to re-land. Worst case is you'll get backed out a second time, but this seems like it's going to have less time overhead than manually messing with talos. ___ dev-platform mailing list dev-platform@lists.mozilla.org https://lists.mozilla.org/listinfo/dev-platform ___ dev-platform mailing list dev-platform@lists.mozilla.org https://lists.mozilla.org/listinfo/dev-platform
Re: New backout policy for Ts regressions on mozilla-inbound
On 2012-10-19 12:13 PM, Dao wrote: On 18.10.2012 20:05, Ehsan Akhgari wrote: If your patch falls in a range which causes more than 4% Ts regression, it will be backed out by our sheriffs together with the rest of the patches in that range, and you can only reland after you fix the regression by testing locally or on the try server. The last point seems excessive. If multiple patches are in the range and you think yours is likely innocent, I think you should be allowed to re-land. Worst case is you'll get backed out a second time, but this seems like it's going to have less time overhead than manually messing with talos. Sure, we still allow good judgement, so if you believe your patch was innocent, then obviously there is no regression for you to fix, so you can just reland. And if Talos proves that your patch is indeed at fault, you will get backed out again! ;-) Ehsan ___ dev-platform mailing list dev-platform@lists.mozilla.org https://lists.mozilla.org/listinfo/dev-platform
New backout policy for Ts regressions on mozilla-inbound
Hi everyone, As part of our efforts to get more value out of the Talos test suite for preventing performance regressions, we believe that we are now ready to put a first set of measures against startup time regressions. We will start by imposing a new backout policy for mozilla-inbound checkins for regressions more than 4% on any given platform. If your patch falls in a range which causes more than 4% Ts regression, it will be backed out by our sheriffs together with the rest of the patches in that range, and you can only reland after you fix the regression by testing locally or on the try server. The 4% threshold has been chosen based on anecdotal evidence on the most recent Ts regressions that we have seen, and is too generous, but we will be working to improve the reporting and regression detection systems better, and as those get improved, we would feel more comfortable with imposing this policy on other Talos tests with tighter thresholds. Please let me know if you have any questions. Cheers, -- Ehsan http://ehsanakhgari.org/ ___ dev-platform mailing list dev-platform@lists.mozilla.org https://lists.mozilla.org/listinfo/dev-platform
Re: New backout policy for Ts regressions on mozilla-inbound
Ehsan Akhgari wrote: Hi everyone, As part of our efforts to get more value out of the Talos test suite for preventing performance regressions, we believe that we are now ready to put a first set of measures against startup time regressions. We will start by imposing a new backout policy for mozilla-inbound checkins for regressions more than 4% on any given platform. If your patch falls in a range which causes more than 4% Ts regression, it will be backed out by our sheriffs together with the rest of the patches in that range, and you can only reland after you fix the regression by testing locally or on the try server. The 4% threshold has been chosen based on anecdotal evidence on the most recent Ts regressions that we have seen, and is too generous, but we will be working to improve the reporting and regression detection systems better, and as those get improved, we would feel more comfortable with imposing this policy on other Talos tests with tighter thresholds. Please let me know if you have any questions. Do we still have the bug where a test that finishes first, but is from a later cset (say a later cset IMPROVES Ts by 4% or more) would make us think we regressed it on an earlier cset if that earlier talos run finishes later? Such that we set graph points by the time the test finished, not time the push was, etc. ~Justin Wood (Callek) ___ dev-platform mailing list dev-platform@lists.mozilla.org https://lists.mozilla.org/listinfo/dev-platform
