Re: Sheriff Highlights and Summary in February 2017

[Andreas Tolfsen]

Also sprach Andrew Halberstadt :

> I don't have any data to back this up, but my suspicion is that a
> large percentage of backouts had try runs, but said try runs
> didn't run the jobs that failed and caused the backout.

I ascribe to this explanation.  This has happened to me a few times
because I have not run all the tests relevant to a change.  It has
either been due to forgetfulness when constructing the try syntax,
or because Marionette has been picked up and used in some new,
inventive way in a component not covered by my default syntax.

> For example, you could either:
> 
> A) Do a full try run then push, almost guaranteeing you won't be
> backed out. But then you run every job twice and take longer to
> complete your bug, a significant cost.
> 
> B) Do a partial try run, running X% of the jobs yielding a Y%
> chance of being backed out.

A third option would be to introduce a new try syntax to have the CI
pick out the right test jobs to run based on the files changed.

Using mach it is possible to extract (known) test dependency
information about source files:

% ./mach file-info dep-tests testing/marionette/driver.js
testing/marionette/driver.js:
Test file patterns:
dom/network/tests/marionette/**
testing/marionette/**
dom/events/test/marionette/**

testing/marionette/harness/marionette_harness/tests/unit/**
dom/system/gonk/tests/marionette/**

If it was possible to determine what try syntax covers those
directories, it might be possible to run an intermediary pre-step on
TaskCluster to generate the appropriate try syntax.

It strikes me as an attractive feature if our CI could figure out
what the appropriate tests to run are based on the changes in the
push.  I don’t know how well this would work or how accurate the
test dependency information from mach is, but it seems worth
experimenting with.

As an aside, I filed a bug to have `./mach test SOURECFILE` pick up
the known test dependencies a few weeks ago:

https://bugzilla.mozilla.org/show_bug.cgi?id=1345942

___
dev-platform mailing list
dev-platform@lists.mozilla.org
https://lists.mozilla.org/listinfo/dev-platform

Re: Sheriff Highlights and Summary in February 2017

[jmaher]

On Friday, March 10, 2017 at 3:46:14 PM UTC-5, Kris Maglione wrote:
> On Fri, Mar 10, 2017 at 01:55:40PM +, David Burns wrote:
> >I went back and did some checks with autoland to servo and the results are
> >negligible. So from 01 February 2017 to 10 March 2017 (as of sending this
> >email). I have removed merge commits from the numbers.
> >
> >Autoland:
> >Total Servo Sync Pushes: 152
> >Total Pushes: 1823
> >Total Backouts: 144
> >Percentage of backouts: 7.8990674712
> >Percentage of backouts without Servo: 8.61759425494
> >
> >Mozilla-Inbound:
> >Total Pushes: 1472
> >Total Backouts: 166
> >Percentage of backouts: 11.277173913
> 
> Is there any way you can get these numbers in terms of patches, rather than 
> pushes? Or, ideally, in terms of bugs landed and backed out? Pushes to 
> inbound 
> still often have patches for more than one bug, so if 4 bugs bets pushed to 
> inbound in one push, and 4 land on autoland as separate pushes, and one gets 
> backed out from each branch, the comparison isn't very useful.

I have been asking the same question and from some initial data, it looks like 
we would have 2075 bugs changed with 166 backouts, or a 8.0% backout rate.  I 
think David is working on validating this data, but to me it shows that code 
quality is the same between branches, we are just landing more bugs on inbound.

There is more guess work for the sheriffs when it comes to backing out things 
with multibug pushes, I would be curious how many times (in the last few 
months) we have had a backout on inbound that didn't fix the problem due to 
multi bug pushes.
___
dev-platform mailing list
dev-platform@lists.mozilla.org
https://lists.mozilla.org/listinfo/dev-platform

Re: Sheriff Highlights and Summary in February 2017

[Kris Maglione]

On Fri, Mar 10, 2017 at 01:55:40PM +, David Burns wrote:

I went back and did some checks with autoland to servo and the results are
negligible. So from 01 February 2017 to 10 March 2017 (as of sending this
email). I have removed merge commits from the numbers.

Autoland:
Total Servo Sync Pushes: 152
Total Pushes: 1823
Total Backouts: 144
Percentage of backouts: 7.8990674712
Percentage of backouts without Servo: 8.61759425494

Mozilla-Inbound:
Total Pushes: 1472
Total Backouts: 166
Percentage of backouts: 11.277173913


Is there any way you can get these numbers in terms of patches, rather than 
pushes? Or, ideally, in terms of bugs landed and backed out? Pushes to inbound 
still often have patches for more than one bug, so if 4 bugs bets pushed to 
inbound in one push, and 4 land on autoland as separate pushes, and one gets 
backed out from each branch, the comparison isn't very useful.

___
dev-platform mailing list
dev-platform@lists.mozilla.org
https://lists.mozilla.org/listinfo/dev-platform

Re: Sheriff Highlights and Summary in February 2017

[Chris Peterson]

I seem to recall gps saying, a few years ago, that the cost per push or 
Try run was about $36. It would be good to know the current cost so 
developers can feel more comfortable spending Mozilla money on 
"unnecessary" Try runs before landing.



On 3/10/2017 6:47 AM, Andrew Halberstadt wrote:

I don't have any data to back this up, but my suspicion is that a large
percentage
of backouts had try runs, but said try runs didn't run the jobs that failed
and caused
the backout. Imo, these kinds of backouts are (more) acceptable because it
means
the developer was trying to avoid doing a full try run, a practice that
should be
cheaper overall in the long run (and if done properly).

For example, you could either:
A) Do a full try run then push, almost guaranteeing you won't be backed
out. But
then you run every job twice and take longer to complete your bug, a
significant
cost.

B) Do a partial try run, running X% of the jobs yielding a Y% chance of
being
backed out.

There's some sweet spot between no try run, and try: -all which is the most
cost
effective (both in terms of AWS bill and developer time).

That being said, I think this is an issue of tools rather than policy.
Things like being
smarter about running jobs based on files changed and improving interfaces
to
selecting jobs on try, should help with backout rates. But the single
biggest thing we
could do is getting rid of integration branches altogether (and instead get
autoland to
merge directly from try). In this world, backouts would hardly even exist
anymore.

I believe we're already headed in this direction, albeit slowly.

-Andrew

On Fri, Mar 10, 2017 at 8:55 AM, David Burns  wrote:


I went back and did some checks with autoland to servo and the results are
negligible. So from 01 February 2017 to 10 March 2017 (as of sending this
email). I have removed merge commits from the numbers.

Autoland:
Total Servo Sync Pushes: 152
Total Pushes: 1823
Total Backouts: 144
Percentage of backouts: 7.8990674712
Percentage of backouts without Servo: 8.61759425494

Mozilla-Inbound:
Total Pushes: 1472
Total Backouts: 166
Percentage of backouts: 11.277173913


I will look into why, with more pushes, is resulting in fewer backouts. The
thing to note is that autoland, by its nature, does not allow us to fail
forward like inbound without having to get a sheriff to land the code.

I think, and this is my next area to investigate, is the 1 bug per push
(the autoland model) could be helping with the percentage of backouts being
lower.

David

On 7 March 2017 at 21:29, Chris Peterson  wrote:


On 3/7/2017 3:38 AM, Joel Maher wrote:


One large difference I see between autoland and mozilla-inbound is that

on

autoland we have many single commits/push whereas mozilla-inbound it is
fewer.  I see the Futurama data showing pushes and the sheriff report
showing total commits.



autoland also includes servo commits imported from GitHub that won't

break

Gecko. (They might break the linux64-stylo builds, but they won't be

backed

out for that.)

___
dev-platform mailing list
dev-platform@lists.mozilla.org
https://lists.mozilla.org/listinfo/dev-platform


___
dev-platform mailing list
dev-platform@lists.mozilla.org
https://lists.mozilla.org/listinfo/dev-platform



___
dev-platform mailing list
dev-platform@lists.mozilla.org
https://lists.mozilla.org/listinfo/dev-platform

Re: Sheriff Highlights and Summary in February 2017

[Andrew Halberstadt]

I don't have any data to back this up, but my suspicion is that a large
percentage
of backouts had try runs, but said try runs didn't run the jobs that failed
and caused
the backout. Imo, these kinds of backouts are (more) acceptable because it
means
the developer was trying to avoid doing a full try run, a practice that
should be
cheaper overall in the long run (and if done properly).

For example, you could either:
A) Do a full try run then push, almost guaranteeing you won't be backed
out. But
then you run every job twice and take longer to complete your bug, a
significant
cost.

B) Do a partial try run, running X% of the jobs yielding a Y% chance of
being
backed out.

There's some sweet spot between no try run, and try: -all which is the most
cost
effective (both in terms of AWS bill and developer time).

That being said, I think this is an issue of tools rather than policy.
Things like being
smarter about running jobs based on files changed and improving interfaces
to
selecting jobs on try, should help with backout rates. But the single
biggest thing we
could do is getting rid of integration branches altogether (and instead get
autoland to
merge directly from try). In this world, backouts would hardly even exist
anymore.

I believe we're already headed in this direction, albeit slowly.

-Andrew

On Fri, Mar 10, 2017 at 8:55 AM, David Burns  wrote:

> I went back and did some checks with autoland to servo and the results are
> negligible. So from 01 February 2017 to 10 March 2017 (as of sending this
> email). I have removed merge commits from the numbers.
>
> Autoland:
> Total Servo Sync Pushes: 152
> Total Pushes: 1823
> Total Backouts: 144
> Percentage of backouts: 7.8990674712
> Percentage of backouts without Servo: 8.61759425494
>
> Mozilla-Inbound:
> Total Pushes: 1472
> Total Backouts: 166
> Percentage of backouts: 11.277173913
>
>
> I will look into why, with more pushes, is resulting in fewer backouts. The
> thing to note is that autoland, by its nature, does not allow us to fail
> forward like inbound without having to get a sheriff to land the code.
>
> I think, and this is my next area to investigate, is the 1 bug per push
> (the autoland model) could be helping with the percentage of backouts being
> lower.
>
> David
>
> On 7 March 2017 at 21:29, Chris Peterson  wrote:
>
> > On 3/7/2017 3:38 AM, Joel Maher wrote:
> >
> >> One large difference I see between autoland and mozilla-inbound is that
> on
> >> autoland we have many single commits/push whereas mozilla-inbound it is
> >> fewer.  I see the Futurama data showing pushes and the sheriff report
> >> showing total commits.
> >>
> >
> > autoland also includes servo commits imported from GitHub that won't
> break
> > Gecko. (They might break the linux64-stylo builds, but they won't be
> backed
> > out for that.)
> >
> > ___
> > dev-platform mailing list
> > dev-platform@lists.mozilla.org
> > https://lists.mozilla.org/listinfo/dev-platform
> >
> ___
> dev-platform mailing list
> dev-platform@lists.mozilla.org
> https://lists.mozilla.org/listinfo/dev-platform
>
___
dev-platform mailing list
dev-platform@lists.mozilla.org
https://lists.mozilla.org/listinfo/dev-platform

Re: Sheriff Highlights and Summary in February 2017

[David Burns]

I went back and did some checks with autoland to servo and the results are
negligible. So from 01 February 2017 to 10 March 2017 (as of sending this
email). I have removed merge commits from the numbers.

Autoland:
Total Servo Sync Pushes: 152
Total Pushes: 1823
Total Backouts: 144
Percentage of backouts: 7.8990674712
Percentage of backouts without Servo: 8.61759425494

Mozilla-Inbound:
Total Pushes: 1472
Total Backouts: 166
Percentage of backouts: 11.277173913


I will look into why, with more pushes, is resulting in fewer backouts. The
thing to note is that autoland, by its nature, does not allow us to fail
forward like inbound without having to get a sheriff to land the code.

I think, and this is my next area to investigate, is the 1 bug per push
(the autoland model) could be helping with the percentage of backouts being
lower.

David

On 7 March 2017 at 21:29, Chris Peterson  wrote:

> On 3/7/2017 3:38 AM, Joel Maher wrote:
>
>> One large difference I see between autoland and mozilla-inbound is that on
>> autoland we have many single commits/push whereas mozilla-inbound it is
>> fewer.  I see the Futurama data showing pushes and the sheriff report
>> showing total commits.
>>
>
> autoland also includes servo commits imported from GitHub that won't break
> Gecko. (They might break the linux64-stylo builds, but they won't be backed
> out for that.)
>
> ___
> dev-platform mailing list
> dev-platform@lists.mozilla.org
> https://lists.mozilla.org/listinfo/dev-platform
>
___
dev-platform mailing list
dev-platform@lists.mozilla.org
https://lists.mozilla.org/listinfo/dev-platform

Re: Sheriff Highlights and Summary in February 2017

[Lawrence Mandel]

I don't know that it's useful to modify your report if you're confident in
your estimates below. Thanks for the additional information.

Lawrence

On Tue, Mar 7, 2017 at 4:03 AM, Carsten Book  wrote:

> Hi Lawrence,
>
> most (i would say 95 %) of the backouts are for Code issues - this include
> bustages and test failures.
>
> From this Code Issues i would guess its about 2/3 for breaking tests and
> 1/3 Build Bustages.
>
> The other backout reasons are merge conflicts / backout requests for
> changes causes new regressions out of our test suites / backout requests
> from developers etc -
>
> In February the backout rate was (excluding the servo merge with 8315
> changesets from the 13242) = 297 in 4927 changesets = ~ 6 %
> In January the backout rate = 302 backouts in 4121 changesets = 7 %
>
> We had a much higher backout rate in the past i think - so it stabilized
> now with this 6-7 % backout rate in the last months,
>
> If you think its useful, i can provide for the next monthly report a more
> detailed analysis like x % =  backouts because builds bustages, y %=
> backouts for test failures etc .
>
> Cheers,
>  -Tomcat
>
>
> On Mon, Mar 6, 2017 at 11:13 PM, Lawrence Mandel 
> wrote:
>
>> Hi Tomcat,
>>
>> Do you have any more details about the reasons why the 297 changesets
>> needed to be backed out?
>>
>> Thanks,
>>
>> Lawrence
>>
>> On Wed, Mar 1, 2017 at 6:05 AM, Carsten Book  wrote:
>>
>>> Hi,
>>>
>>> We will be more active in 2017 and inform more about whats happening in
>>> Sheriffing and since its already March :)
>>>
>>> In February we had about 13242 changesets landed on mozilla-central
>>> monitored by Sheriffs.
>>>
>>> (The high number of changes is because of the merge from servo to
>>> mozilla-central with about 8315 changesets).
>>>
>>> 297 changesets were backed out in February.
>>>
>>> Beside this Sheriffs took park in doing uplifts and checkin-needed bugs.
>>>
>>> The Current Orangefactor is 10.43 (7250 test failures failures in 695
>>> pushes in the last 7 days).
>>> You can find the list of top Intermittent failure bugs here
>>> https://brasstacks.mozilla.com/orangefactor/
>>>
>>> You can find more statistics here: https://futurama.theautomatedt
>>> ester.co.uk/
>>>
>>> A big thanks to the Team especially our Community Sheriffs for handling
>>> the new Tier 2 Stylo Build on integration trees  and mozilla-central  and
>>> the teamwork with the Developers!
>>>
>>> If you want also to be a Community Sheriffs, as part of more blogging
>>> about Sheriffing i published a blog post about it :
>>> https://blog.mozilla.org/tomcat/2017/02/27/community-sheriffs/
>>>
>>> Let us know when you have any Question or Feedback about Sheriffing.
>>>
>>> Cheers,
>>>  -Tomcat
>>>
>>
>>
>
___
dev-platform mailing list
dev-platform@lists.mozilla.org
https://lists.mozilla.org/listinfo/dev-platform

Re: [Sheriffs] Sheriff Highlights and Summary in February 2017

[Wes Kocher]

Wonder if autoland's backout rate will go up when autoland is detached from
mozreview (letting someone autoland directly from bugzilla is in the plans,
I believe).

On Tue, Mar 7, 2017 at 4:47 AM, Carsten Book <cb...@mozilla.com> wrote:

> Hi,
>
> i agree with Sebastian - the integration of submitting to try and getting
> the results in mozreview helps i think a lot.
>
> Also this is all great feedback and will add a more detailed area about
> backouts (where they happen like inbound/autoland, and reason) in the next
> sheriff monthly report.
>
> Cheers,
> - Tomcat
>
> On Tue, Mar 7, 2017 at 1:27 PM, Sebastian Hengst <a...@gmx.de> wrote:
>
>> Mozreview shows the results of Try pushes (if they have been requested)
>> near the top where also the landing can be requested which makes it less
>> likely that issues get missed (pushing to Try and not checking the results
>> is not that rare).
>>
>> Mozreview is also the only repository to which the person who gets backed
>> out the most (my subjection impression) pushes to (doesn't mean the second
>> push will stick).
>>
>> Inbound is also often used for platform-wide changes which might have
>> bitrotted between the Try push and the push to inbound.
>>
>> Sebastian
>>
>>  Original-Nachricht 
>> Betreff: Re: [Sheriffs] Sheriff Highlights and Summary in February 2017
>> Von: David Burns <dbu...@mozilla.com>
>> Datum: 2017-03-07 12:23
>>
>>> One thing that we have also noticed is that the backout rate on autoland
>>> is lower than inbound.
>>>
>>> In the last 7 days backout rate is averaging (merges have been removed):
>>>
>>>   * Autoland 6%.(24 backouts out of 381 pushes)
>>>   * Inbound 12% (30 backouts out of 251 pushes)
>>>
>>> I don't have graphs to show this but when I look at
>>> https://futurama.theautomatedtester.co.uk/ <
>>> https://futurama.theautomatedtester.co.uk/>   each week I have seen
>>> this result consistenly for about a month.
>>>
>>> David
>>>
>>>
>>>
>>> On 7 March 2017 at 09:03, Carsten Book <cb...@mozilla.com >> cb...@mozilla.com>> wrote:
>>>
>>> Hi Lawrence,
>>>
>>> most (i would say 95 %) of the backouts are for Code issues - this
>>> include bustages and test failures.
>>>
>>>  From this Code Issues i would guess its about 2/3 for breaking
>>> tests and 1/3 Build Bustages.
>>>
>>> The other backout reasons are merge conflicts / backout requests for
>>> changes causes new regressions out of our test suites / backout
>>> requests from developers etc -
>>>
>>> In February the backout rate was (excluding the servo merge with
>>> 8315 changesets from the 13242) = 297 in 4927 changesets = ~ 6 %
>>> In January the backout rate = 302 backouts in 4121 changesets = 7 %
>>>
>>> We had a much higher backout rate in the past i think - so it
>>> stabilized now with this 6-7 % backout rate in the last months,
>>>
>>> If you think its useful, i can provide for the next monthly report a
>>> more detailed analysis like x % =  backouts because builds bustages,
>>> y %= backouts for test failures etc .
>>>
>>> Cheers,
>>>   -Tomcat
>>>
>>>
>>> On Mon, Mar 6, 2017 at 11:13 PM, Lawrence Mandel
>>> <lman...@mozilla.com <mailto:lman...@mozilla.com>> wrote:
>>>
>>> Hi Tomcat,
>>>
>>> Do you have any more details about the reasons why the 297
>>> changesets needed to be backed out?
>>>
>>> Thanks,
>>>
>>> Lawrence
>>>
>>> On Wed, Mar 1, 2017 at 6:05 AM, Carsten Book <cb...@mozilla.com
>>> <mailto:cb...@mozilla.com>> wrote:
>>>
>>> Hi,
>>>
>>> We will be more active in 2017 and inform more about whats
>>> happening in Sheriffing and since its already March :)
>>>
>>> In February we had about 13242 changesets landed on
>>> mozilla-central monitored by Sheriffs.
>>>
>>> (The high number of changes is because of the merge from
>>> servo to mozilla-central with about 8315 changesets).
>>>
>>> 297 changesets were backed out in February.
>>>
>>

Re: Sheriff Highlights and Summary in February 2017

[David Burns]

This is just a rough number of how many pushes had a backout and how many
didn't. I don't have any data on whether this is a full or partial backout.

If there are multiple bugs in a push on inbound, a sheriff may revert the
entire push (or might not depending on how obvious the error is and
availability of the person pushing.)

As jmaher pointed out, and so did you, pushes are more succinct on autoland
which could be mean its simpler to know what to revert. Other than doing
some A/B testing to prove the hypothesis of the sheriffs in this thread, a
lot of the data is going to be anecdotal as to why there is a difference.

David

On 7 March 2017 at 12:57, Boris Zbarsky  wrote:

> On 3/7/17 6:23 AM, David Burns wrote:
>
>>- Autoland 6%.(24 backouts out of 381 pushes)
>>- Inbound 12% (30 backouts out of 251 pushes)
>>
>
> Were those full backouts or partial backouts?
>
> That is, how are we counting a multi-bug push to inbound where one of the
> bugs gets backed out?  Note that such a thing probably doesn't happen on
> autoland.
>
> -Boris
> ___
> dev-platform mailing list
> dev-platform@lists.mozilla.org
> https://lists.mozilla.org/listinfo/dev-platform
>
___
dev-platform mailing list
dev-platform@lists.mozilla.org
https://lists.mozilla.org/listinfo/dev-platform

Re: Sheriff Highlights and Summary in February 2017

[Boris Zbarsky]

On 3/7/17 6:23 AM, David Burns wrote:

   - Autoland 6%.(24 backouts out of 381 pushes)
   - Inbound 12% (30 backouts out of 251 pushes)


Were those full backouts or partial backouts?

That is, how are we counting a multi-bug push to inbound where one of 
the bugs gets backed out?  Note that such a thing probably doesn't 
happen on autoland.


-Boris
___
dev-platform mailing list
dev-platform@lists.mozilla.org
https://lists.mozilla.org/listinfo/dev-platform

Re: [Sheriffs] Sheriff Highlights and Summary in February 2017

[Carsten Book]

Hi,

i agree with Sebastian - the integration of submitting to try and getting
the results in mozreview helps i think a lot.

Also this is all great feedback and will add a more detailed area about
backouts (where they happen like inbound/autoland, and reason) in the next
sheriff monthly report.

Cheers,
- Tomcat

On Tue, Mar 7, 2017 at 1:27 PM, Sebastian Hengst <a...@gmx.de> wrote:

> Mozreview shows the results of Try pushes (if they have been requested)
> near the top where also the landing can be requested which makes it less
> likely that issues get missed (pushing to Try and not checking the results
> is not that rare).
>
> Mozreview is also the only repository to which the person who gets backed
> out the most (my subjection impression) pushes to (doesn't mean the second
> push will stick).
>
> Inbound is also often used for platform-wide changes which might have
> bitrotted between the Try push and the push to inbound.
>
> Sebastian
>
>  Original-Nachricht ----
> Betreff: Re: [Sheriffs] Sheriff Highlights and Summary in February 2017
> Von: David Burns <dbu...@mozilla.com>
> Datum: 2017-03-07 12:23
>
>> One thing that we have also noticed is that the backout rate on autoland
>> is lower than inbound.
>>
>> In the last 7 days backout rate is averaging (merges have been removed):
>>
>>   * Autoland 6%.(24 backouts out of 381 pushes)
>>   * Inbound 12% (30 backouts out of 251 pushes)
>>
>> I don't have graphs to show this but when I look at
>> https://futurama.theautomatedtester.co.uk/ <https://futurama.theautomated
>> tester.co.uk/>   each week I have seen this result consistenly for about
>> a month.
>>
>> David
>>
>>
>>
>> On 7 March 2017 at 09:03, Carsten Book <cb...@mozilla.com > cb...@mozilla.com>> wrote:
>>
>> Hi Lawrence,
>>
>> most (i would say 95 %) of the backouts are for Code issues - this
>> include bustages and test failures.
>>
>>  From this Code Issues i would guess its about 2/3 for breaking
>> tests and 1/3 Build Bustages.
>>
>> The other backout reasons are merge conflicts / backout requests for
>> changes causes new regressions out of our test suites / backout
>> requests from developers etc -
>>
>> In February the backout rate was (excluding the servo merge with
>> 8315 changesets from the 13242) = 297 in 4927 changesets = ~ 6 %
>> In January the backout rate = 302 backouts in 4121 changesets = 7 %
>>
>> We had a much higher backout rate in the past i think - so it
>> stabilized now with this 6-7 % backout rate in the last months,
>>
>> If you think its useful, i can provide for the next monthly report a
>> more detailed analysis like x % =  backouts because builds bustages,
>> y %= backouts for test failures etc .
>>
>> Cheers,
>>   -Tomcat
>>
>>
>> On Mon, Mar 6, 2017 at 11:13 PM, Lawrence Mandel
>> <lman...@mozilla.com <mailto:lman...@mozilla.com>> wrote:
>>
>> Hi Tomcat,
>>
>> Do you have any more details about the reasons why the 297
>> changesets needed to be backed out?
>>
>> Thanks,
>>
>> Lawrence
>>
>> On Wed, Mar 1, 2017 at 6:05 AM, Carsten Book <cb...@mozilla.com
>> <mailto:cb...@mozilla.com>> wrote:
>>
>> Hi,
>>
>> We will be more active in 2017 and inform more about whats
>> happening in Sheriffing and since its already March :)
>>
>> In February we had about 13242 changesets landed on
>> mozilla-central monitored by Sheriffs.
>>
>> (The high number of changes is because of the merge from
>> servo to mozilla-central with about 8315 changesets).
>>
>> 297 changesets were backed out in February.
>>
>> Beside this Sheriffs took park in doing uplifts and
>> checkin-needed bugs.
>>
>> The Current Orangefactor is 10.43 (7250 test failures
>> failures in 695 pushes in the last 7 days).
>> You can find the list of top Intermittent failure bugs here
>> https://brasstacks.mozilla.com/orangefactor/
>> <https://brasstacks.mozilla.com/orangefactor/>
>>
>> You can find more statistics here:
>> https://futurama.theautomatedtester.co.uk/
>> <https://futurama.theautomatedtester.co.uk/>

Re: Sheriff Highlights and Summary in February 2017

[David Burns]

One thing that we have also noticed is that the backout rate on autoland is
lower than inbound.

In the last 7 days backout rate is averaging (merges have been removed):

   - Autoland 6%.(24 backouts out of 381 pushes)
   - Inbound 12% (30 backouts out of 251 pushes)

I don't have graphs to show this but when I look at https://futurama.
theautomatedtester.co.uk/   each week I have seen this result consistenly
for about a month.

David


On 7 March 2017 at 09:03, Carsten Book  wrote:

> Hi Lawrence,
>
> most (i would say 95 %) of the backouts are for Code issues - this include
> bustages and test failures.
>
> From this Code Issues i would guess its about 2/3 for breaking tests and
> 1/3 Build Bustages.
>
> The other backout reasons are merge conflicts / backout requests for
> changes causes new regressions out of our test suites / backout requests
> from developers etc -
>
> In February the backout rate was (excluding the servo merge with 8315
> changesets from the 13242) = 297 in 4927 changesets = ~ 6 %
> In January the backout rate = 302 backouts in 4121 changesets = 7 %
>
> We had a much higher backout rate in the past i think - so it stabilized
> now with this 6-7 % backout rate in the last months,
>
> If you think its useful, i can provide for the next monthly report a more
> detailed analysis like x % =  backouts because builds bustages, y %=
> backouts for test failures etc .
>
> Cheers,
>  -Tomcat
>
>
> On Mon, Mar 6, 2017 at 11:13 PM, Lawrence Mandel 
> wrote:
>
>> Hi Tomcat,
>>
>> Do you have any more details about the reasons why the 297 changesets
>> needed to be backed out?
>>
>> Thanks,
>>
>> Lawrence
>>
>> On Wed, Mar 1, 2017 at 6:05 AM, Carsten Book  wrote:
>>
>>> Hi,
>>>
>>> We will be more active in 2017 and inform more about whats happening in
>>> Sheriffing and since its already March :)
>>>
>>> In February we had about 13242 changesets landed on mozilla-central
>>> monitored by Sheriffs.
>>>
>>> (The high number of changes is because of the merge from servo to
>>> mozilla-central with about 8315 changesets).
>>>
>>> 297 changesets were backed out in February.
>>>
>>> Beside this Sheriffs took park in doing uplifts and checkin-needed bugs.
>>>
>>> The Current Orangefactor is 10.43 (7250 test failures failures in 695
>>> pushes in the last 7 days).
>>> You can find the list of top Intermittent failure bugs here
>>> https://brasstacks.mozilla.com/orangefactor/
>>>
>>> You can find more statistics here: https://futurama.theautomatedt
>>> ester.co.uk/
>>>
>>> A big thanks to the Team especially our Community Sheriffs for handling
>>> the new Tier 2 Stylo Build on integration trees  and mozilla-central  and
>>> the teamwork with the Developers!
>>>
>>> If you want also to be a Community Sheriffs, as part of more blogging
>>> about Sheriffing i published a blog post about it :
>>> https://blog.mozilla.org/tomcat/2017/02/27/community-sheriffs/
>>>
>>> Let us know when you have any Question or Feedback about Sheriffing.
>>>
>>> Cheers,
>>>  -Tomcat
>>>
>>
>>
>
___
dev-platform mailing list
dev-platform@lists.mozilla.org
https://lists.mozilla.org/listinfo/dev-platform

Re: Sheriff Highlights and Summary in February 2017

[Carsten Book]

Hi Lawrence,

most (i would say 95 %) of the backouts are for Code issues - this include
bustages and test failures.

>From this Code Issues i would guess its about 2/3 for breaking tests and
1/3 Build Bustages.

The other backout reasons are merge conflicts / backout requests for
changes causes new regressions out of our test suites / backout requests
from developers etc -

In February the backout rate was (excluding the servo merge with 8315
changesets from the 13242) = 297 in 4927 changesets = ~ 6 %
In January the backout rate = 302 backouts in 4121 changesets = 7 %

We had a much higher backout rate in the past i think - so it stabilized
now with this 6-7 % backout rate in the last months,

If you think its useful, i can provide for the next monthly report a more
detailed analysis like x % =  backouts because builds bustages, y %=
backouts for test failures etc .

Cheers,
 -Tomcat


On Mon, Mar 6, 2017 at 11:13 PM, Lawrence Mandel 
wrote:

> Hi Tomcat,
>
> Do you have any more details about the reasons why the 297 changesets
> needed to be backed out?
>
> Thanks,
>
> Lawrence
>
> On Wed, Mar 1, 2017 at 6:05 AM, Carsten Book  wrote:
>
>> Hi,
>>
>> We will be more active in 2017 and inform more about whats happening in
>> Sheriffing and since its already March :)
>>
>> In February we had about 13242 changesets landed on mozilla-central
>> monitored by Sheriffs.
>>
>> (The high number of changes is because of the merge from servo to
>> mozilla-central with about 8315 changesets).
>>
>> 297 changesets were backed out in February.
>>
>> Beside this Sheriffs took park in doing uplifts and checkin-needed bugs.
>>
>> The Current Orangefactor is 10.43 (7250 test failures failures in 695
>> pushes in the last 7 days).
>> You can find the list of top Intermittent failure bugs here
>> https://brasstacks.mozilla.com/orangefactor/
>>
>> You can find more statistics here: https://futurama.theautomatedt
>> ester.co.uk/
>>
>> A big thanks to the Team especially our Community Sheriffs for handling
>> the new Tier 2 Stylo Build on integration trees  and mozilla-central  and
>> the teamwork with the Developers!
>>
>> If you want also to be a Community Sheriffs, as part of more blogging
>> about Sheriffing i published a blog post about it :
>> https://blog.mozilla.org/tomcat/2017/02/27/community-sheriffs/
>>
>> Let us know when you have any Question or Feedback about Sheriffing.
>>
>> Cheers,
>>  -Tomcat
>>
>
>
___
dev-platform mailing list
dev-platform@lists.mozilla.org
https://lists.mozilla.org/listinfo/dev-platform

Re: Sheriff Highlights and Summary in February 2017

[Lawrence Mandel]

Hi Tomcat,

Do you have any more details about the reasons why the 297 changesets
needed to be backed out?

Thanks,

Lawrence

On Wed, Mar 1, 2017 at 6:05 AM, Carsten Book  wrote:

> Hi,
>
> We will be more active in 2017 and inform more about whats happening in
> Sheriffing and since its already March :)
>
> In February we had about 13242 changesets landed on mozilla-central
> monitored by Sheriffs.
>
> (The high number of changes is because of the merge from servo to
> mozilla-central with about 8315 changesets).
>
> 297 changesets were backed out in February.
>
> Beside this Sheriffs took park in doing uplifts and checkin-needed bugs.
>
> The Current Orangefactor is 10.43 (7250 test failures failures in 695
> pushes in the last 7 days).
> You can find the list of top Intermittent failure bugs here
> https://brasstacks.mozilla.com/orangefactor/
>
> You can find more statistics here: https://futurama.
> theautomatedtester.co.uk/
>
> A big thanks to the Team especially our Community Sheriffs for handling
> the new Tier 2 Stylo Build on integration trees  and mozilla-central  and
> the teamwork with the Developers!
>
> If you want also to be a Community Sheriffs, as part of more blogging
> about Sheriffing i published a blog post about it :
> https://blog.mozilla.org/tomcat/2017/02/27/community-sheriffs/
>
> Let us know when you have any Question or Feedback about Sheriffing.
>
> Cheers,
>  -Tomcat
>
___
dev-platform mailing list
dev-platform@lists.mozilla.org
https://lists.mozilla.org/listinfo/dev-platform

Sheriff Highlights and Summary in February 2017

[Carsten Book]

Hi,

We will be more active in 2017 and inform more about whats happening in
Sheriffing and since its already March :)

In February we had about 13242 changesets landed on mozilla-central
monitored by Sheriffs.

(The high number of changes is because of the merge from servo to
mozilla-central with about 8315 changesets).

297 changesets were backed out in February.

Beside this Sheriffs took park in doing uplifts and checkin-needed bugs.

The Current Orangefactor is 10.43 (7250 test failures failures in 695
pushes in the last 7 days).
You can find the list of top Intermittent failure bugs here
https://brasstacks.mozilla.com/orangefactor/

You can find more statistics here:
https://futurama.theautomatedtester.co.uk/

A big thanks to the Team especially our Community Sheriffs for handling the
new Tier 2 Stylo Build on integration trees  and mozilla-central  and the
teamwork with the Developers!

If you want also to be a Community Sheriffs, as part of more blogging about
Sheriffing i published a blog post about it :
https://blog.mozilla.org/tomcat/2017/02/27/community-sheriffs/

Let us know when you have any Question or Feedback about Sheriffing.

Cheers,
 -Tomcat
___
dev-platform mailing list
dev-platform@lists.mozilla.org
https://lists.mozilla.org/listinfo/dev-platform