Re: Making try faster for debugging intermittents

2016-07-13 Thread Armen Zambrano G. 

On 2016-07-01 06:10 AM, Mike Hommey wrote:

On Fri, Jul 01, 2016 at 03:07:43PM +1000, Xidorn Quan wrote:

On Fri, Jul 1, 2016 at 1:02 PM, Karl Tomlinson  wrote:


William Lachance writes:


As part of a larger effort to improve the experience around
debugging intermittents, I've been looking at reducing the time it
takes for common "try" workloads for developers (so that
e.g. retriggering a job to reproduce a failure can happen faster).



Also, accounts of specific try workloads of this type which are
annoying/painful would be helpful. :) I think I have a rough idea
of the particular type of try push I'm looking for (not pushed by
release operations, at least one retrigger) but it would be great
to get firsthand confirmation of that.


One thing that might be helpful is enabling running only tests on
try with a designated build that has already been created.

Often tests are modified to add logging, after which the same
build could be run with the new version of the test, thus saving
waiting for a build.



FWIW, there's a bug about this:
https://bugzilla.mozilla.org/show_bug.cgi?id=1240644


You can already run tests with arbitrary test tarballs (that you could
create locally), but I can't find where it's documented, which may
explain why it's not well known.

(CCing Armen, who would know)

Mike



I totally missed this.

It requires checking out mozci and using the trigger.py script (only 
Buildbot - old CI).

We want to teach |mach try| to help developers with this case scenario.
It would work with both Buildbot and TaskCluster (new CI).

regards,
Armen
--
Zambrano Gasparnian, Armen
Automation & Tools Engineer
http://armenzg.blogspot.ca
___
dev-platform mailing list
dev-platform@lists.mozilla.org
https://lists.mozilla.org/listinfo/dev-platform

Re: Making try faster for debugging intermittents

2016-07-06 Thread William Lachance 

On 2016-07-01 6:10 AM, Mike Hommey wrote:

> > One thing that might be helpful is enabling running only tests on
> > try with a designated build that has already been created.
> >
> > Often tests are modified to add logging, after which the same
> > build could be run with the new version of the test, thus saving
> > waiting for a build.
> >

>
> FWIW, there's a bug about this:
> https://bugzilla.mozilla.org/show_bug.cgi?id=1240644

You can already run tests with arbitrary test tarballs (that you could
create locally), but I can't find where it's documented, which may
explain why it's not well known.

(CCing Armen, who would know)


Another angle to this is the one-click loaner support we're working on, 
which will give you access to a virtual machine running in automation 
with the tests ready to be run. Should be helpful for the use case of 
"let me run my test with a bunch of debugging code enabled in an 
automation-style environment".


This work is being tracked in this bug:

https://bugzilla.mozilla.org/show_bug.cgi?id=1262260

Will
___
dev-platform mailing list
dev-platform@lists.mozilla.org
https://lists.mozilla.org/listinfo/dev-platform

Re: Making try faster for debugging intermittents

2016-07-01 Thread Mike Hommey 
On Fri, Jul 01, 2016 at 03:07:43PM +1000, Xidorn Quan wrote:
> On Fri, Jul 1, 2016 at 1:02 PM, Karl Tomlinson  wrote:
> 
> > William Lachance writes:
> >
> > > As part of a larger effort to improve the experience around
> > > debugging intermittents, I've been looking at reducing the time it
> > > takes for common "try" workloads for developers (so that
> > > e.g. retriggering a job to reproduce a failure can happen faster).
> >
> > > Also, accounts of specific try workloads of this type which are
> > > annoying/painful would be helpful. :) I think I have a rough idea
> > > of the particular type of try push I'm looking for (not pushed by
> > > release operations, at least one retrigger) but it would be great
> > > to get firsthand confirmation of that.
> >
> > One thing that might be helpful is enabling running only tests on
> > try with a designated build that has already been created.
> >
> > Often tests are modified to add logging, after which the same
> > build could be run with the new version of the test, thus saving
> > waiting for a build.
> >
> 
> FWIW, there's a bug about this:
> https://bugzilla.mozilla.org/show_bug.cgi?id=1240644

You can already run tests with arbitrary test tarballs (that you could
create locally), but I can't find where it's documented, which may
explain why it's not well known.

(CCing Armen, who would know)

Mike
___
dev-platform mailing list
dev-platform@lists.mozilla.org
https://lists.mozilla.org/listinfo/dev-platform

Re: Making try faster for debugging intermittents

2016-06-30 Thread Xidorn Quan 
On Fri, Jul 1, 2016 at 1:02 PM, Karl Tomlinson  wrote:

> William Lachance writes:
>
> > As part of a larger effort to improve the experience around
> > debugging intermittents, I've been looking at reducing the time it
> > takes for common "try" workloads for developers (so that
> > e.g. retriggering a job to reproduce a failure can happen faster).
>
> > Also, accounts of specific try workloads of this type which are
> > annoying/painful would be helpful. :) I think I have a rough idea
> > of the particular type of try push I'm looking for (not pushed by
> > release operations, at least one retrigger) but it would be great
> > to get firsthand confirmation of that.
>
> One thing that might be helpful is enabling running only tests on
> try with a designated build that has already been created.
>
> Often tests are modified to add logging, after which the same
> build could be run with the new version of the test, thus saving
> waiting for a build.
>

FWIW, there's a bug about this:
https://bugzilla.mozilla.org/show_bug.cgi?id=1240644

- Xidorn
___
dev-platform mailing list
dev-platform@lists.mozilla.org
https://lists.mozilla.org/listinfo/dev-platform

Re: Making try faster for debugging intermittents

2016-06-30 Thread Karl Tomlinson 
William Lachance writes:

> As part of a larger effort to improve the experience around
> debugging intermittents, I've been looking at reducing the time it
> takes for common "try" workloads for developers (so that
> e.g. retriggering a job to reproduce a failure can happen faster).

> Also, accounts of specific try workloads of this type which are
> annoying/painful would be helpful. :) I think I have a rough idea
> of the particular type of try push I'm looking for (not pushed by
> release operations, at least one retrigger) but it would be great
> to get firsthand confirmation of that.

One thing that might be helpful is enabling running only tests on
try with a designated build that has already been created.

Often tests are modified to add logging, after which the same
build could be run with the new version of the test, thus saving
waiting for a build.
___
dev-platform mailing list
dev-platform@lists.mozilla.org
https://lists.mozilla.org/listinfo/dev-platform

Making try faster for debugging intermittents

2016-06-30 Thread William Lachance 
As part of a larger effort to improve the experience around debugging 
intermittents, I've been looking at reducing the time it takes for 
common "try" workloads for developers (so that e.g. retriggering a job 
to reproduce a failure can happen faster).


Of course, the common advice of "profile before you optimize" applies 
here. That is to say, we want to spend time optimizing what makes this 
particular task painful, rather than optimizing try in general.


To start with, I've been using a manually generated dump of the jobs 
posted to try on treeherder in a 10 day span. You can see some 
preliminary results of me manipulating the data to get some rough 
information (last job for each try push, and total machine time for each 
job) here:


http://nbviewer.jupyter.org/url/people.mozilla.org/%7Ewlachance/try%20analysis.ipynb

Before I go much further, I'd love to know if anyone has done similar 
explorations in the recent past and if there are any aggregation points 
other than Treeherder (taskcluster? buildbot?) which are storing this 
information: my working assumption is that treeherder is the best place 
to analyze this information (since it acts as a central point of 
aggregation), but I'm open to being proven wrong.


Also, accounts of specific try workloads of this type which are 
annoying/painful would be helpful. :) I think I have a rough idea of the 
particular type of try push I'm looking for (not pushed by release 
operations, at least one retrigger) but it would be great to get 
firsthand confirmation of that.


Will
___
dev-platform mailing list
dev-platform@lists.mozilla.org
https://lists.mozilla.org/listinfo/dev-platform