Re: Experiment with running debug tests less often on mozilla-inbound the week of August 25

[Jeff Gilbert]

I was just going to ask about this. I glanced through the mozconfigs in the 
tree for at least Linux debug, but it looks like it only has --enable-debug, 
not even -O1. Maybe it's buried somewhere in there, but I didn't find it with a 
quick look.

I took a look at the build log for WinXP debug, and --enable-opt is only 
present on the configure line for nspr, whereas --enable-debug is in a number 
of other places.

Can we get confirmation for whether debug builds are (partially?) optimized? If 
not, we should do that. (Unless I'm missing a reason not to, especially if we 
only care about pass/fail, and not crash stacks/debugability)

-Jeff

- Original Message -
From: "Kyle Huey" 
To: "Joshua Cranmer 🐧" 
Cc: "dev-platform" 
Sent: Tuesday, August 19, 2014 3:56:27 PM
Subject: Re: Experiment with running debug tests less often on mozilla-inbound  
the week of August 25

I'm pretty sure the debug builds on our CI infrastructure are built
with optimization.

- Kyle

On Tue, Aug 19, 2014 at 3:42 PM, Joshua Cranmer 🐧  wrote:
> On 8/19/2014 5:25 PM, Ehsan Akhgari wrote:
>>
>> Yep, the debug tests indeed take more time, mostly because they run more
>> checks.
>
>
> Actually, the bigger cause in the slowdown is probably that debug tests
> don't have any optimizations, not more checks. An atomic increment on a
> debug build invokes something like a hundred instructions (including several
> call instructions) whereas the equivalent operation on an opt build is just
> one.
>
> --
> Joshua Cranmer
> Thunderbird and DXR developer
> Source code archæologist
>
>
> ___
> dev-platform mailing list
> dev-platform@lists.mozilla.org
> https://lists.mozilla.org/listinfo/dev-platform
___
dev-platform mailing list
dev-platform@lists.mozilla.org
https://lists.mozilla.org/listinfo/dev-platform
___
dev-platform mailing list
dev-platform@lists.mozilla.org
https://lists.mozilla.org/listinfo/dev-platform

DXR Intern Presentation

[Erik Rose]

Marcell has been my intern on the DXR project these past months, and it's been 
a privilege to work with him. From JavaScript analysis to Elasticsearch to 
writing Haskell in Python, he's cut an impressive swath through the codebase, 
and you can expect to see a lot of his handiwork deployed in the next several 
weeks. He'll be giving a bit of a preview in a wide-ranging presentation this 
Thursday. Oh, and there might be dinosaurs. And robots.

8/21 @1:30PM PST in Mountain View (will be streamed on AirMo)

Topics include...
- Type signature search
- JavaScript support almost done
- Moving to Elasticsearch
- DXR fan art!

We hope to see you there! (There will also be an accompanying blog post going 
up after the talk on mvcisback.github.io.)

Cheers,
Erik
DXR Lead
___
dev-platform mailing list
dev-platform@lists.mozilla.org
https://lists.mozilla.org/listinfo/dev-platform

Re: Non-Trivial SpecialPowers Usage Considered Harmful

[Bobby Holley]

On Mon, Aug 18, 2014 at 1:14 PM, Jed Davis  wrote:

> Bobby Holley  writes:
> [...]
> > If you find yourself itching to do something complicated, write a
> > mochitest-chrome test. The default template [2] now generates html files
> > (rather then XUL files), so the ergonomics there should be easier than
> > before.
> >
> > If you don't want to write a mochitest-chrome test for some reason, you
> can
> > also use SpecialPowers.loadChromeScript(uri), which lets mochitest-plain
> > asynchronously load a privileged JS file in its own privileged scope.
>
> On e10s-enabled platforms, does loadChromeScript run the script in the
> parent process?


It operates over the message manager, so I believe that it does:
http://mxr.mozilla.org/mozilla-central/source/testing/specialpowers/content/SpecialPowersObserverAPI.js#346
___
dev-platform mailing list
dev-platform@lists.mozilla.org
https://lists.mozilla.org/listinfo/dev-platform

Re: Running mochitests from a copy of the objdir?

[Neil]

Gregory Szorc wrote:


On 8/19/14 2:11 PM, Neil wrote:


Gregory Szorc wrote:


I think the underlying bug here is "mach mochitest doesn't work for
comm-central." It should. I'm sure there is a bug on file somewhere.


Fair enough, but for when it does work, what's the mach version of 
python runtests.py?


mach mochitest? mach mochitest-plain?

Under the scenes mach is using straight Python to construct the 
mochitest harness/runner and invoke it directly. No extra processes 
necessary.


You can run |mach mach-debug-commands| to locate where each command is 
defined in source. Or, load 
https://hg.mozilla.org/mozilla-central/file/149d3ce6e020/testing/mochitest/mach_commands.py 
for the mochitest commands.


Right, but 
http://hg.mozilla.org/mozilla-central/file/149d3ce6e020/testing/mochitest/mach_commands.py#l290 
suggests that the mach defaults aren't the same as the runtest.py 
defaults, which are the ones that I want.


--
Warning: May contain traces of nuts.
___
dev-platform mailing list
dev-platform@lists.mozilla.org
https://lists.mozilla.org/listinfo/dev-platform

Re: Experiment with running debug tests less often on mozilla-inbound the week of August 25

[Kyle Huey]

I'm pretty sure the debug builds on our CI infrastructure are built
with optimization.

- Kyle

On Tue, Aug 19, 2014 at 3:42 PM, Joshua Cranmer 🐧  wrote:
> On 8/19/2014 5:25 PM, Ehsan Akhgari wrote:
>>
>> Yep, the debug tests indeed take more time, mostly because they run more
>> checks.
>
>
> Actually, the bigger cause in the slowdown is probably that debug tests
> don't have any optimizations, not more checks. An atomic increment on a
> debug build invokes something like a hundred instructions (including several
> call instructions) whereas the equivalent operation on an opt build is just
> one.
>
> --
> Joshua Cranmer
> Thunderbird and DXR developer
> Source code archæologist
>
>
> ___
> dev-platform mailing list
> dev-platform@lists.mozilla.org
> https://lists.mozilla.org/listinfo/dev-platform
___
dev-platform mailing list
dev-platform@lists.mozilla.org
https://lists.mozilla.org/listinfo/dev-platform

Re: Experiment with running debug tests less often on mozilla-inbound the week of August 25

[Joshua Cranmer 🐧]

On 8/19/2014 5:25 PM, Ehsan Akhgari wrote:
Yep, the debug tests indeed take more time, mostly because they run 
more checks.


Actually, the bigger cause in the slowdown is probably that debug tests 
don't have any optimizations, not more checks. An atomic increment on a 
debug build invokes something like a hundred instructions (including 
several call instructions) whereas the equivalent operation on an opt 
build is just one.


--
Joshua Cranmer
Thunderbird and DXR developer
Source code archæologist

___
dev-platform mailing list
dev-platform@lists.mozilla.org
https://lists.mozilla.org/listinfo/dev-platform

Re: Experiment with running debug tests less often on mozilla-inbound the week of August 25

[Jonathan Griffin]

No, fx-team is not affected by this experiment; we intend to target 
mozilla-inbound only for this 1-week trial.  The reason is that the 
number of commits on m-i seems larger than fx-team, and therefore the 
impacts should be more visible.


Jonathan

On 8/19/2014 3:19 PM, Matthew N. wrote:

On 8/19/14 12:22 PM, Jonathan Griffin wrote:

To assess the impact of doing this, we will be performing an experiment
the week of August 25, in which we will run debug tests on
mozilla-inbound on most desktop platforms every other run, instead of
every run as we do now.  Debug tests on linux64 will continue to run
every time.  Non-desktop platforms and trees other than mozilla-inbound
will not be affected.


To clarify, is fx-team affected by this change? I ask because you 
mention "desktop" and that is where the desktop front-end team does 
landings. I suspect fx-team landings are less likely to hit debug-only 
issues than mozilla-inbound as fx-team has much fewer C++ changes and 
anecdotally JS-only changes seem to trigger debug-only failures less 
often.



This approach is based on the premise that the number of debug-only
platform-specific failures on desktop is low enough to be manageable,
and that the extra burden this imposes on the sheriffs will be small
enough compared to the improvement in test slave metrics to justify the
cost.


FWIW, I think fx-team is more desktop-specific (although Android 
front-end stuff also lands there and I'm not familiar with that).


MattN
___
dev-platform mailing list
dev-platform@lists.mozilla.org
https://lists.mozilla.org/listinfo/dev-platform


___
dev-platform mailing list
dev-platform@lists.mozilla.org
https://lists.mozilla.org/listinfo/dev-platform

Re: Experiment with running debug tests less often on mozilla-inbound the week of August 25

[Trevor Saunders]

On Tue, Aug 19, 2014 at 02:49:48PM -0700, Jonathan Griffin wrote:
> On 8/19/2014 2:41 PM, Ehsan Akhgari wrote:
> >On 2014-08-19, 3:57 PM, Jeff Gilbert wrote:
> >>I would actually say that debug tests are more important for continuous
> >>integration than opt tests. At least in code I deal with, we have a ton
> >>of asserts to guarantee behavior, and we really want test coverage with
> >>these via CI. If a test passes on debug, it should almost certainly pass
> >>on opt, just faster. The opposite is not true.
> >>
> >>"They take a long time and then break" is part of what I believe caused
> >>us to not bother with debug testing on much of Android and B2G, which we
> >>still haven't completely fixed. It should be unacceptable to ship
> >>without CI on debug tests, but here we are anyways. (This is finally
> >>nearly fixed, though there is still some work to do)
> >>
> >>I'm not saying running debug tests less often is on the same scale of
> >>bad, but I would like to express my concerns about heading in that
> >>direction.
> >
> >I second this.  I'm curious to know why you picked debug tests for this
> >experiment.  Would it not make more sense to run opt tests on desktop
> >platforms on every other run?
> >
> Just based on the fact that they take longer and thus running them less
> frequently would have a larger impact.  If there's a broad consensus that
> debug runs are more valuable, we could switch to running opt tests less
> frequently instead.

It seems to me our goal here is basically to pick so that the expected
time to detect bustage is minimized without increasing the maximum time
it can take to detect bustage.  That is take p(d) to be the probability
only debug tests will fail, p(o) the probability only opt tests will
fail, and p(b) the probability both will fail.  Then take t(d) and t(o)
the time for a debug and opt test to run respectively.  Now you want to
decide which to run first debug or opt.  you'd expect that if you choose
debug you'd expect to detect bustage in
(p(d) + p(b)) * t(d) + p(o) * (t(o) + t(d))
which simplifies to
t(d) + p(o) * t(o)
On the other hand if you choose to test opt first you get
t(o) + p(d) * t(d)

I suspect we all agree t(d) > t(o) and it seems likely p(d) > p(o), but
it should be clear which is the better choice depends on the exact
values of those numbers (and  this is not a good model of reality in
many ways).

Trev

> 
> Jonathan
> ___
> dev-platform mailing list
> dev-platform@lists.mozilla.org
> https://lists.mozilla.org/listinfo/dev-platform
___
dev-platform mailing list
dev-platform@lists.mozilla.org
https://lists.mozilla.org/listinfo/dev-platform

Re: Experiment with running debug tests less often on mozilla-inbound the week of August 25

[David Burns]

I know this is tangential but the small changes are the least tested 
changes in my experience. The try push requirement for checkin-needed 
has had a wonderful impact on the amount of times the tree is closed[1]. 
The tree is less likely to be closed these days.


David

[1] http://futurama.theautomatedtester.co.uk/

On 19/08/2014 22:04, Ralph Giles wrote:

On 2014-08-19 1:55 PM, Benoit Girard wrote:

Perhaps we should instead promote checkin-needed (or a similar simple)
to coalesce simple changes together.

I would prefer to use 'checkin-needed' for more things, but am blocked
by the try-needed requirement. We need some way to bless small changes
for inbound without a try push. Look up the author's commit access maybe?

  -r
___
dev-platform mailing list
dev-platform@lists.mozilla.org
https://lists.mozilla.org/listinfo/dev-platform


___
dev-platform mailing list
dev-platform@lists.mozilla.org
https://lists.mozilla.org/listinfo/dev-platform

Re: Experiment with running debug tests less often on mozilla-inbound the week of August 25

[Ehsan Akhgari]

On 2014-08-19, 5:49 PM, Jonathan Griffin wrote:

On 8/19/2014 2:41 PM, Ehsan Akhgari wrote:

On 2014-08-19, 3:57 PM, Jeff Gilbert wrote:

I would actually say that debug tests are more important for
continuous integration than opt tests. At least in code I deal with,
we have a ton of asserts to guarantee behavior, and we really want
test coverage with these via CI. If a test passes on debug, it should
almost certainly pass on opt, just faster. The opposite is not true.

"They take a long time and then break" is part of what I believe
caused us to not bother with debug testing on much of Android and
B2G, which we still haven't completely fixed. It should be
unacceptable to ship without CI on debug tests, but here we are
anyways. (This is finally nearly fixed, though there is still some
work to do)

I'm not saying running debug tests less often is on the same scale of
bad, but I would like to express my concerns about heading in that
direction.


I second this.  I'm curious to know why you picked debug tests for
this experiment.  Would it not make more sense to run opt tests on
desktop platforms on every other run?


Just based on the fact that they take longer and thus running them less
frequently would have a larger impact.  If there's a broad consensus
that debug runs are more valuable, we could switch to running opt tests
less frequently instead.


Yep, the debug tests indeed take more time, mostly because they run more 
checks.  :-)  The checks in opt builds are not exactly a subset of the 
ones in debug builds, but they are close.  Based on that, I think 
running opt tests on every other push is a more conservative one, and I 
support it more.  That being said, for this one week limited trial, 
given that the sheriffs will help backfill the skipped tests, I don't 
care very strongly about this, as long as it doesn't set the precedence 
that we can ignore debug tests!


Cheers,
Ehsan

___
dev-platform mailing list
dev-platform@lists.mozilla.org
https://lists.mozilla.org/listinfo/dev-platform

Re: Experiment with running debug tests less often on mozilla-inbound the week of August 25

[Matthew N.]

On 8/19/14 12:22 PM, Jonathan Griffin wrote:

To assess the impact of doing this, we will be performing an experiment
the week of August 25, in which we will run debug tests on
mozilla-inbound on most desktop platforms every other run, instead of
every run as we do now.  Debug tests on linux64 will continue to run
every time.  Non-desktop platforms and trees other than mozilla-inbound
will not be affected.


To clarify, is fx-team affected by this change? I ask because you 
mention "desktop" and that is where the desktop front-end team does 
landings. I suspect fx-team landings are less likely to hit debug-only 
issues than mozilla-inbound as fx-team has much fewer C++ changes and 
anecdotally JS-only changes seem to trigger debug-only failures less often.



This approach is based on the premise that the number of debug-only
platform-specific failures on desktop is low enough to be manageable,
and that the extra burden this imposes on the sheriffs will be small
enough compared to the improvement in test slave metrics to justify the
cost.


FWIW, I think fx-team is more desktop-specific (although Android 
front-end stuff also lands there and I'm not familiar with that).


MattN
___
dev-platform mailing list
dev-platform@lists.mozilla.org
https://lists.mozilla.org/listinfo/dev-platform

RE: dev-platform Digest, Vol 103, Issue 47

[Look, Yuriy]

Thank you, Boris, this explains my problem.


On Tue, 19 Aug 2014 15:10:38 -0400  Boris Zbarsky  wrote:
>On 8/19/14, 2:52 PM, Look, Yuriy wrote:
>> I am building my code against Gecko 31 for Windows and a public C++ method 
>> nsIDocument::GetDocShell defined in include\nsIDocument.h on line 1188 does 
>> not get resolved by Visual Studio 2010 linker.

>This is a non-virtual non-inline method, with default (hidden) visibility.  So 
>it can only be used from inside libxul, no?

>> This is the only one that does not get resolved for me, even though my code 
>> uses quite a few, including several more methods of nsIDocument interface.

>Are those also non-virtual and non-inline?

>-Boris


The contents of this e-mail are intended for the named addressee only. It 
contains information that may be confidential. Unless you are the named 
addressee or an authorized designee, you may not copy or use it, or disclose it 
to anyone else. If you received it in error please notify us immediately and 
then destroy it
___
dev-platform mailing list
dev-platform@lists.mozilla.org
https://lists.mozilla.org/listinfo/dev-platform

Re: Experiment with running debug tests less often on mozilla-inbound the week of August 25

[Jonathan Griffin]

On 8/19/2014 2:41 PM, Ehsan Akhgari wrote:

On 2014-08-19, 3:57 PM, Jeff Gilbert wrote:
I would actually say that debug tests are more important for 
continuous integration than opt tests. At least in code I deal with, 
we have a ton of asserts to guarantee behavior, and we really want 
test coverage with these via CI. If a test passes on debug, it should 
almost certainly pass on opt, just faster. The opposite is not true.


"They take a long time and then break" is part of what I believe 
caused us to not bother with debug testing on much of Android and 
B2G, which we still haven't completely fixed. It should be 
unacceptable to ship without CI on debug tests, but here we are 
anyways. (This is finally nearly fixed, though there is still some 
work to do)


I'm not saying running debug tests less often is on the same scale of 
bad, but I would like to express my concerns about heading in that 
direction.


I second this.  I'm curious to know why you picked debug tests for 
this experiment.  Would it not make more sense to run opt tests on 
desktop platforms on every other run?


Just based on the fact that they take longer and thus running them less 
frequently would have a larger impact.  If there's a broad consensus 
that debug runs are more valuable, we could switch to running opt tests 
less frequently instead.


Jonathan
___
dev-platform mailing list
dev-platform@lists.mozilla.org
https://lists.mozilla.org/listinfo/dev-platform

Re: Running mochitests from a copy of the objdir?

[Gregory Szorc]

On 8/19/14 2:11 PM, Neil wrote:

Gregory Szorc wrote:


I think the underlying bug here is "mach mochitest doesn't work for
comm-central." It should. I'm sure there is a bug on file somewhere.


Fair enough, but for when it does work, what's the mach version of
python runtests.py?


mach mochitest? mach mochitest-plain?

Under the scenes mach is using straight Python to construct the 
mochitest harness/runner and invoke it directly. No extra processes 
necessary.


You can run |mach mach-debug-commands| to locate where each command is 
defined in source. Or, load 
https://hg.mozilla.org/mozilla-central/file/149d3ce6e020/testing/mochitest/mach_commands.py 
for the mochitest commands.

___
dev-platform mailing list
dev-platform@lists.mozilla.org
https://lists.mozilla.org/listinfo/dev-platform

Re: Experiment with running debug tests less often on mozilla-inbound the week of August 25

[Jonathan Griffin]

I also agree about coalescing better.  We are looking at ways to do that 
in conjunction with 
https://wiki.mozilla.org/Auto-tools/Projects/Autoland, which we'll have 
a prototype of by the end of the quarter.  In this model, commits that 
are going through autoland could be coalesced when landing on inbound, 
which would reduce slave load on all platforms.


Until that's deployed and in widespread use, we have other options to 
decrease slave load, and this experiment is the simplest.  It won't 
result in reduced test coverage, since sheriffs will backfill in the 
case of a regression.  Essentially, we're not running tests that would 
have passed anyway.


Depending on feedback we receive after this experiment, we may opt to 
change our approach in the future:  i.e., run tests every Nth opt build 
instead of debug build, or try to identify sets of "never failing" tests 
and just run those less frequently, or always include at least one 
flavor of Windows, OSX and Linux on every commit, etc.


Regards,

Jonathan


On 8/19/2014 1:55 PM, Benoit Girard wrote:

I completely agree with Jeff Gilbert on this one.

I think we should try to coalesce -better-. I just checked the current
state of mozilla-inbound and it doesn't feel any of the current patch
really need their own set of tests because they're are not time
sensitive or sufficiently complex. Right now developers are asked to
create bugs for their own change with their own patch. This leads to a
lot of little patches being landed by individual developers which
seems to reflect the current state of mozilla-inbound.

Perhaps we should instead promote checkin-needed (or a similar simple)
to coalesce simple changes together. Opting into this means that your
patch may take significantly longer to get merged if it's landed with
another bad patch and should only be used when that's acceptable.
Right now developers with commit access are not encouraged to make use
of checkin-needed AFAIK. If we started recommending against individual
landings for simple changes, and improved the process, we could
probably significantly cut the number of tests jobs by cutting the
number of pushes.

On Tue, Aug 19, 2014 at 3:57 PM, Jeff Gilbert  wrote:

I would actually say that debug tests are more important for continuous 
integration than opt tests. At least in code I deal with, we have a ton of 
asserts to guarantee behavior, and we really want test coverage with these via 
CI. If a test passes on debug, it should almost certainly pass on opt, just 
faster. The opposite is not true.

"They take a long time and then break" is part of what I believe caused us to 
not bother with debug testing on much of Android and B2G, which we still haven't 
completely fixed. It should be unacceptable to ship without CI on debug tests, but here 
we are anyways. (This is finally nearly fixed, though there is still some work to do)

I'm not saying running debug tests less often is on the same scale of bad, but 
I would like to express my concerns about heading in that direction.

-Jeff

- Original Message -
From: "Jonathan Griffin" 
To: dev-platform@lists.mozilla.org
Sent: Tuesday, August 19, 2014 12:22:21 PM
Subject: Experiment with running debug tests less often on mozilla-inbound  
the week of August 25

Our pools of test slaves are often at or over capacity, and this has the
effect of increasing job coalescing and test wait times.  This, in turn,
can lead to longer tree closures caused by test bustage, and can cause
try runs to be very slow to complete.

One of the easiest ways to mitigate this is to run tests less often.

To assess the impact of doing this, we will be performing an experiment
the week of August 25, in which we will run debug tests on
mozilla-inbound on most desktop platforms every other run, instead of
every run as we do now.  Debug tests on linux64 will continue to run
every time.  Non-desktop platforms and trees other than mozilla-inbound
will not be affected.

This approach is based on the premise that the number of debug-only
platform-specific failures on desktop is low enough to be manageable,
and that the extra burden this imposes on the sheriffs will be small
enough compared to the improvement in test slave metrics to justify the
cost.

While this experiment is in progress, we will be monitoring job
coalescing and test wait times, as well as impacts on sheriffs and
developers.  If the experiment causes sheriffs to be unable to perform
their job effectively, it can be terminated prematurely.

We intend to use the data we collect during the experiment to inform
decisions about additional tooling we need to make this or a similar
plan permanent at some point in the future, as well as validating the
premise on which this experiment is based.

After the conclusion of this experiment, a follow-up post will be made
which will discuss our findings.  If you have any concerns, feel free to
reach out to me.

Jonathan

___
de

Re: Experiment with running debug tests less often on mozilla-inbound the week of August 25

[Ehsan Akhgari]

On 2014-08-19, 3:57 PM, Jeff Gilbert wrote:

I would actually say that debug tests are more important for continuous 
integration than opt tests. At least in code I deal with, we have a ton of 
asserts to guarantee behavior, and we really want test coverage with these via 
CI. If a test passes on debug, it should almost certainly pass on opt, just 
faster. The opposite is not true.

"They take a long time and then break" is part of what I believe caused us to 
not bother with debug testing on much of Android and B2G, which we still haven't 
completely fixed. It should be unacceptable to ship without CI on debug tests, but here 
we are anyways. (This is finally nearly fixed, though there is still some work to do)

I'm not saying running debug tests less often is on the same scale of bad, but 
I would like to express my concerns about heading in that direction.


I second this.  I'm curious to know why you picked debug tests for this 
experiment.  Would it not make more sense to run opt tests on desktop 
platforms on every other run?


Cheers,
Ehsan

___
dev-platform mailing list
dev-platform@lists.mozilla.org
https://lists.mozilla.org/listinfo/dev-platform

Re: Running mochitests from a copy of the objdir?

[Neil]

Gregory Szorc wrote:

I think the underlying bug here is "mach mochitest doesn't work for 
comm-central." It should. I'm sure there is a bug on file somewhere.


Fair enough, but for when it does work, what's the mach version of 
python runtests.py?


--
Warning: May contain traces of nuts.
___
dev-platform mailing list
dev-platform@lists.mozilla.org
https://lists.mozilla.org/listinfo/dev-platform

Re: Running mochitests from a copy of the objdir?

[Neil]

Jonathan Griffin wrote:


Can you show us the command-line you're using?


../path/to/python runtests.py

--
Warning: May contain traces of nuts.
___
dev-platform mailing list
dev-platform@lists.mozilla.org
https://lists.mozilla.org/listinfo/dev-platform

Re: Experiment with running debug tests less often on mozilla-inbound the week of August 25

[Ralph Giles]

On 2014-08-19 1:55 PM, Benoit Girard wrote:
> Perhaps we should instead promote checkin-needed (or a similar simple)
> to coalesce simple changes together.

I would prefer to use 'checkin-needed' for more things, but am blocked
by the try-needed requirement. We need some way to bless small changes
for inbound without a try push. Look up the author's commit access maybe?

 -r
___
dev-platform mailing list
dev-platform@lists.mozilla.org
https://lists.mozilla.org/listinfo/dev-platform

Re: Experiment with running debug tests less often on mozilla-inbound the week of August 25

[Benoit Girard]

I completely agree with Jeff Gilbert on this one.

I think we should try to coalesce -better-. I just checked the current
state of mozilla-inbound and it doesn't feel any of the current patch
really need their own set of tests because they're are not time
sensitive or sufficiently complex. Right now developers are asked to
create bugs for their own change with their own patch. This leads to a
lot of little patches being landed by individual developers which
seems to reflect the current state of mozilla-inbound.

Perhaps we should instead promote checkin-needed (or a similar simple)
to coalesce simple changes together. Opting into this means that your
patch may take significantly longer to get merged if it's landed with
another bad patch and should only be used when that's acceptable.
Right now developers with commit access are not encouraged to make use
of checkin-needed AFAIK. If we started recommending against individual
landings for simple changes, and improved the process, we could
probably significantly cut the number of tests jobs by cutting the
number of pushes.

On Tue, Aug 19, 2014 at 3:57 PM, Jeff Gilbert  wrote:
> I would actually say that debug tests are more important for continuous 
> integration than opt tests. At least in code I deal with, we have a ton of 
> asserts to guarantee behavior, and we really want test coverage with these 
> via CI. If a test passes on debug, it should almost certainly pass on opt, 
> just faster. The opposite is not true.
>
> "They take a long time and then break" is part of what I believe caused us to 
> not bother with debug testing on much of Android and B2G, which we still 
> haven't completely fixed. It should be unacceptable to ship without CI on 
> debug tests, but here we are anyways. (This is finally nearly fixed, though 
> there is still some work to do)
>
> I'm not saying running debug tests less often is on the same scale of bad, 
> but I would like to express my concerns about heading in that direction.
>
> -Jeff
>
> - Original Message -
> From: "Jonathan Griffin" 
> To: dev-platform@lists.mozilla.org
> Sent: Tuesday, August 19, 2014 12:22:21 PM
> Subject: Experiment with running debug tests less often on mozilla-inbound
>   the week of August 25
>
> Our pools of test slaves are often at or over capacity, and this has the
> effect of increasing job coalescing and test wait times.  This, in turn,
> can lead to longer tree closures caused by test bustage, and can cause
> try runs to be very slow to complete.
>
> One of the easiest ways to mitigate this is to run tests less often.
>
> To assess the impact of doing this, we will be performing an experiment
> the week of August 25, in which we will run debug tests on
> mozilla-inbound on most desktop platforms every other run, instead of
> every run as we do now.  Debug tests on linux64 will continue to run
> every time.  Non-desktop platforms and trees other than mozilla-inbound
> will not be affected.
>
> This approach is based on the premise that the number of debug-only
> platform-specific failures on desktop is low enough to be manageable,
> and that the extra burden this imposes on the sheriffs will be small
> enough compared to the improvement in test slave metrics to justify the
> cost.
>
> While this experiment is in progress, we will be monitoring job
> coalescing and test wait times, as well as impacts on sheriffs and
> developers.  If the experiment causes sheriffs to be unable to perform
> their job effectively, it can be terminated prematurely.
>
> We intend to use the data we collect during the experiment to inform
> decisions about additional tooling we need to make this or a similar
> plan permanent at some point in the future, as well as validating the
> premise on which this experiment is based.
>
> After the conclusion of this experiment, a follow-up post will be made
> which will discuss our findings.  If you have any concerns, feel free to
> reach out to me.
>
> Jonathan
>
> ___
> dev-platform mailing list
> dev-platform@lists.mozilla.org
> https://lists.mozilla.org/listinfo/dev-platform
> ___
> dev-platform mailing list
> dev-platform@lists.mozilla.org
> https://lists.mozilla.org/listinfo/dev-platform
___
dev-platform mailing list
dev-platform@lists.mozilla.org
https://lists.mozilla.org/listinfo/dev-platform

Re: New QE-Verify Flag in Bugzilla

[Clint Talbert]

Hello all,

I know that there is a bunch of confusion in the wake of the recent 
reorganization of b2g, so let me set things straight. There is no new QE 
department. The QA team is still the same folks, although part of it is in the 
B2G org, part of it is in the Services org, and the core QA team are still in 
the Platform org.

This little "QE" moniker that was used here is indicating the aspirational 
shift that we'd like to make to move Quality Analysis at Mozilla toward a more 
"Quality Engineering" orientation. This means we are trying to shift toward 
deep diving into the technology behind the features we're testing, making 
better use of automation, being more proactive, being more data & metrics 
driven, being more experimental in ways that we do the work we do (with 
community, trying new approaches), etc. 

This "engineering" orientation toward Quality will be a hallmark of the unified 
voice for Quality at Mozilla that we are striving to create. I know that many 
developers across the org are also interested in this, and we invite you to be 
a part of that too.

There will be much more to come on all of this, and you should start to see 
some small things changing in the Quality teams as we move toward that 
"Engineering" direction. We aren't going to change the names of all the "QA" 
things right now, we are going to focus on earning the "engineering" moniker 
first before we change anything else.

Hope that clears up the confusion.

Clint

- Original Message -
From: "Jared Wein" 
To: "Marc Schifer" 
Cc: firefox...@mozilla.org, "dev-platform" , 
"dev-quality" , firefox-...@mozilla.org
Sent: Tuesday, August 19, 2014 11:03:58 AM
Subject: Re: New QE-Verify Flag in Bugzilla

Hi Marc,

Can you shed some background on the change in name from QA to QE? Does QE stand 
for Quality Engineering, whereas QA is Quality Assurance? What are the 
differences between the two? Do we now have two different teams/departments?

Thanks,
Jared

- Original Message -
> From: "Marc Schifer" 
> To: "dev-quality" 
> Cc: firefox...@mozilla.org, "dev-platform" , 
> firefox-...@mozilla.org
> Sent: Thursday, August 14, 2014 5:45:52 PM
> Subject: New QE-Verify Flag in Bugzilla
> 
> A short while ago we had a discussion about how to stream line some QA
> processes (Desktop QA - Bugzilla keywords: Proposal on dev-quality) and
> reduce the bug spam generated by setting keywords and whiteboard tags
> (Bugspam generated by the new process on firefox-dev). The final resolution
> of this discussion was to have a new Bugzilla flag created, qe-verify, that
> would replace the current use of QA+/- whiteboard tags and the verifyme
> keyword. This will give you the ability to easily filter out any activity on
> the flag instead of needing to have multiple filters for keywords and
> whiteboard activity as well simplify our queries for tracking work to be
> done in QE. The use of other keywords (qawanted, qaurgent, steps-wanted,
> regression-window-wanted, testcase-wanted) will remain the same.
> 
> Currently this is only implemented for Firefox QE, other teams are more then
> welcome to adopt if so desired.
> 
> Details:
>  New Flag: qe-verify[?|+|-]
>   qe-verify[?] = triage request
>   qe-verify[+] = bug to be verified
>   qe-verify[-] = bug will not/can not be verified
> 
>  Deprecate use of:
>   Whiteboard tag: QA[?|+|-]
>   Keyword   : verifyme
> 
>  The component list the flag can be set for is:
> 
>   Core   -- Any --
>   Firefox-- Any --
>   Firefox for Android-- Any --
>   Firefox for Metro  -- Any --
>   Firefox Health Report  -- Any --
>   Firefox OS -- Any --
>   Loop   Client
>   Loop   General
>   Mozilla Localizations  -- Any --
>   Mozilla QA Mozmill Tests
>   Mozilla Services   -- Any --
>   NSPR   -- Any --
>   NSS-- Any --
>   Plugins-- Any --
>   Snippets   -- Any --
>   Toolkit-- Any --
> 
> If there is a component that I missed please let me or the bugzilla team know
> so we can fix it.
> 
> This message was posted to dev-quality and has been crossposted to on
> firefox-dev and dev-platform. Please follow up on dev-quality.
> 
> Thank You.
>  Marc S.
> Firefox Q.E. Manager
> 
> ___
> firefox-dev mailing list
> firefox-...@mozilla.org
> https://mail.mozilla.org/listinfo/firefox-dev
> 
___
dev-platform mailing list
dev-platform@lists.mozilla.org
https://lists.mozilla.org/listinfo/dev-platform
___
dev-platform mailing list
dev-platform@lists.mozilla.org
https://lists.mozilla.org/listinfo/dev-platform

Re: Experiment with running debug tests less often on mozilla-inbound the week of August 25

[Jeff Gilbert]

I would actually say that debug tests are more important for continuous 
integration than opt tests. At least in code I deal with, we have a ton of 
asserts to guarantee behavior, and we really want test coverage with these via 
CI. If a test passes on debug, it should almost certainly pass on opt, just 
faster. The opposite is not true.

"They take a long time and then break" is part of what I believe caused us to 
not bother with debug testing on much of Android and B2G, which we still 
haven't completely fixed. It should be unacceptable to ship without CI on debug 
tests, but here we are anyways. (This is finally nearly fixed, though there is 
still some work to do)

I'm not saying running debug tests less often is on the same scale of bad, but 
I would like to express my concerns about heading in that direction.

-Jeff

- Original Message -
From: "Jonathan Griffin" 
To: dev-platform@lists.mozilla.org
Sent: Tuesday, August 19, 2014 12:22:21 PM
Subject: Experiment with running debug tests less often on mozilla-inbound  
the week of August 25

Our pools of test slaves are often at or over capacity, and this has the 
effect of increasing job coalescing and test wait times.  This, in turn, 
can lead to longer tree closures caused by test bustage, and can cause 
try runs to be very slow to complete.

One of the easiest ways to mitigate this is to run tests less often.

To assess the impact of doing this, we will be performing an experiment 
the week of August 25, in which we will run debug tests on 
mozilla-inbound on most desktop platforms every other run, instead of 
every run as we do now.  Debug tests on linux64 will continue to run 
every time.  Non-desktop platforms and trees other than mozilla-inbound 
will not be affected.

This approach is based on the premise that the number of debug-only 
platform-specific failures on desktop is low enough to be manageable, 
and that the extra burden this imposes on the sheriffs will be small 
enough compared to the improvement in test slave metrics to justify the 
cost.

While this experiment is in progress, we will be monitoring job 
coalescing and test wait times, as well as impacts on sheriffs and 
developers.  If the experiment causes sheriffs to be unable to perform 
their job effectively, it can be terminated prematurely.

We intend to use the data we collect during the experiment to inform 
decisions about additional tooling we need to make this or a similar 
plan permanent at some point in the future, as well as validating the 
premise on which this experiment is based.

After the conclusion of this experiment, a follow-up post will be made 
which will discuss our findings.  If you have any concerns, feel free to 
reach out to me.

Jonathan

___
dev-platform mailing list
dev-platform@lists.mozilla.org
https://lists.mozilla.org/listinfo/dev-platform
___
dev-platform mailing list
dev-platform@lists.mozilla.org
https://lists.mozilla.org/listinfo/dev-platform

Experiment with running debug tests less often on mozilla-inbound the week of August 25

[Jonathan Griffin]

Our pools of test slaves are often at or over capacity, and this has the 
effect of increasing job coalescing and test wait times.  This, in turn, 
can lead to longer tree closures caused by test bustage, and can cause 
try runs to be very slow to complete.


One of the easiest ways to mitigate this is to run tests less often.

To assess the impact of doing this, we will be performing an experiment 
the week of August 25, in which we will run debug tests on 
mozilla-inbound on most desktop platforms every other run, instead of 
every run as we do now.  Debug tests on linux64 will continue to run 
every time.  Non-desktop platforms and trees other than mozilla-inbound 
will not be affected.


This approach is based on the premise that the number of debug-only 
platform-specific failures on desktop is low enough to be manageable, 
and that the extra burden this imposes on the sheriffs will be small 
enough compared to the improvement in test slave metrics to justify the 
cost.


While this experiment is in progress, we will be monitoring job 
coalescing and test wait times, as well as impacts on sheriffs and 
developers.  If the experiment causes sheriffs to be unable to perform 
their job effectively, it can be terminated prematurely.


We intend to use the data we collect during the experiment to inform 
decisions about additional tooling we need to make this or a similar 
plan permanent at some point in the future, as well as validating the 
premise on which this experiment is based.


After the conclusion of this experiment, a follow-up post will be made 
which will discuss our findings.  If you have any concerns, feel free to 
reach out to me.


Jonathan

___
dev-platform mailing list
dev-platform@lists.mozilla.org
https://lists.mozilla.org/listinfo/dev-platform

Re: nsIDocument::GetDocShell is not present in any Gecko 31/Windows .lib files

[Boris Zbarsky]

On 8/19/14, 2:52 PM, Look, Yuriy wrote:

I am building my code against Gecko 31 for Windows and a public C++ method 
nsIDocument::GetDocShell defined in include\nsIDocument.h on line 1188 does not 
get resolved by Visual Studio 2010 linker.


This is a non-virtual non-inline method, with default (hidden) 
visibility.  So it can only be used from inside libxul, no?



This is the only one that does not get resolved for me, even though my code 
uses quite a few, including several more methods of nsIDocument interface.


Are those also non-virtual and non-inline?

-Boris
___
dev-platform mailing list
dev-platform@lists.mozilla.org
https://lists.mozilla.org/listinfo/dev-platform

Re: Disabling a few mochitests this week, and upcoming changes next week to browser-chrome

[Gavin Sharp]

On Tue, Aug 19, 2014 at 9:06 AM, Ed Morley  wrote:
> Agreed that this case is possibly slightly different - however this change
> is merely exposing already flaky tests - and this change is very much
> overdue. If the tests mentioned are really that important, then could we
> please get people allocated to fixing them?

We have allocated people to fixing them. These bugs were explicitly
assigned in the 34.2 iteration:
Bug 947574
Bug 1002439
Bug 1041537

and we've made some significant progress in tracking them down. But as
you know, fixing these issues can be relatively difficult.

> Ultimately no one individual test should have the right to destroy the SnR
> for a suite - and if we don't have this policy as a rapid (and enforceable
> in practice) means to control rogue tests, then the only other hammer the
> sheriffs have is to hide entire suites.

If we're reduced to interacting with each other with hammers, then
we've failed. Firefox peers will not ignore needinfo or review
requests. If you feel that they are, then you can escalate to me as
module owner. I do not ignore requests, and am quite responsive to
email and IRC pings.

Gavin
___
dev-platform mailing list
dev-platform@lists.mozilla.org
https://lists.mozilla.org/listinfo/dev-platform

nsIDocument::GetDocShell is not present in any Gecko 31/Windows .lib files

[Look, Yuriy]

Hi,

I am building my code against Gecko 31 for Windows and a public C++ method 
nsIDocument::GetDocShell defined in include\nsIDocument.h on line 1188 does not 
get resolved by Visual Studio 2010 linker.  This is the only one that does not 
get resolved for me, even though my code uses quite a few, including several 
more methods of nsIDocument interface.  Your help is very much appreciated.

Thank you,

Yuriy Look | Software Developer | Compuware APM
yuriy.l...@compuware.com   313  227 3107
...
Simply Smarter APM | 
Compuware.com/APM
 | APM 
blog | 
dynaTrace Free 
Trial

The contents of this e-mail are intended for the named addressee only. It 
contains information that may be confidential. Unless you are the named 
addressee or an authorized designee, you may not copy or use it, or disclose it 
to anyone else. If you received it in error please notify us immediately and 
then destroy it
___
dev-platform mailing list
dev-platform@lists.mozilla.org
https://lists.mozilla.org/listinfo/dev-platform

Re: Disabling a few mochitests this week, and upcoming changes next week to browser-chrome

[Gavin Sharp]

(I'm trying to move this thread to firefox-dev, please send followups there.)

> Bug 1041569 - has been needinfo'd for 4 weeks, it is only disabling this on 
> win debug, I don't know how to reduce that anymore, especially if nobody 
> wants to look at it.

Not what I see - a needinfo request was responded to on July 25th, and
then there was no further activity (and no pending requests). Similar
stories for the other bugs.

Driving these bugs to completion is a shared responsibility - I am
happy to help, but if this is important to you then you have some
responsibility for communicating that importance and following up
directly when you're not getting the results you need. That means you
sometimes need to pester people, and need to escalate to a module
owner (per the policy you linked to). There was no followup or
escalation in those bugs.

Gavin


On Tue, Aug 19, 2014 at 9:00 AM, jmaher  wrote:
> On Tuesday, August 19, 2014 11:46:08 AM UTC-4, Gavin Sharp wrote:
>> > Going with our test disabling policy 
>> > (https://wiki.mozilla.org/Sheriffing/Test_Disabling_Policy), I am going to 
>> > get ready and disable tests on Friday.
>>
>>
>>
>> A few issues here:
>>
>> - This particular case (we're making broad changes to how the tests
>>
>> run that causes many new failures) was not what that policy was meant
>>
>> to cover. We need some leeway to handle this situation differently.
>>
>> - This policy should probably clarify that any test disabling patches
>>
>> still require module owner/peer review.
>>
>>
>>
>> > You can see all the bugs that are blocking bug 992911 
>> > (https://bugzilla.mozilla.org/showdependencytree.cgi?id=992911&hide_resolved=1)
>>
>>
>>
>> We cannot disable all of the tests in that list wholesale. At least
>>
>> bug 1041569, bug 1041583, bug 1017187, bug 1001820, and bug 963075
>
> Bug 1041569 - has been needinfo'd for 4 weeks, it is only disabling this on 
> win debug, I don't know how to reduce that anymore, especially if nobody 
> wants to look at it.
>
> bug 1041583 - no activity in 4+ week as well.  We are only disabling it on 
> windows debug
>
> bug 1017187 - no reply in 4+ weeks, this is disabling the test on all debug 
> branches.
>
> bug 1001820 - idle for 3 weeks until last week and it was recommended to 
> disable the test (in fact it appears to be disabled)
>
> bug 963075 - no activity for 4 weeks, we need to disable 2 pdf tests on opt 
> builds only.
>
> All in all we are changing 12 tests in the manifests.
>
> It is obvious based on the lack of response in the bugs that fixing these 
> bugs are not a priority, changing the way we run these tests will only give 
> us more reliable results and less churn when we need to adjust chunking or 
> jobs.
> ___
> dev-platform mailing list
> dev-platform@lists.mozilla.org
> https://lists.mozilla.org/listinfo/dev-platform
___
dev-platform mailing list
dev-platform@lists.mozilla.org
https://lists.mozilla.org/listinfo/dev-platform

Re: New QE-Verify Flag in Bugzilla

[Jared Wein]

Hi Marc,

Can you shed some background on the change in name from QA to QE? Does QE stand 
for Quality Engineering, whereas QA is Quality Assurance? What are the 
differences between the two? Do we now have two different teams/departments?

Thanks,
Jared

- Original Message -
> From: "Marc Schifer" 
> To: "dev-quality" 
> Cc: firefox...@mozilla.org, "dev-platform" , 
> firefox-...@mozilla.org
> Sent: Thursday, August 14, 2014 5:45:52 PM
> Subject: New QE-Verify Flag in Bugzilla
> 
> A short while ago we had a discussion about how to stream line some QA
> processes (Desktop QA - Bugzilla keywords: Proposal on dev-quality) and
> reduce the bug spam generated by setting keywords and whiteboard tags
> (Bugspam generated by the new process on firefox-dev). The final resolution
> of this discussion was to have a new Bugzilla flag created, qe-verify, that
> would replace the current use of QA+/- whiteboard tags and the verifyme
> keyword. This will give you the ability to easily filter out any activity on
> the flag instead of needing to have multiple filters for keywords and
> whiteboard activity as well simplify our queries for tracking work to be
> done in QE. The use of other keywords (qawanted, qaurgent, steps-wanted,
> regression-window-wanted, testcase-wanted) will remain the same.
> 
> Currently this is only implemented for Firefox QE, other teams are more then
> welcome to adopt if so desired.
> 
> Details:
>  New Flag: qe-verify[?|+|-]
>   qe-verify[?] = triage request
>   qe-verify[+] = bug to be verified
>   qe-verify[-] = bug will not/can not be verified
> 
>  Deprecate use of:
>   Whiteboard tag: QA[?|+|-]
>   Keyword   : verifyme
> 
>  The component list the flag can be set for is:
> 
>   Core   -- Any --
>   Firefox-- Any --
>   Firefox for Android-- Any --
>   Firefox for Metro  -- Any --
>   Firefox Health Report  -- Any --
>   Firefox OS -- Any --
>   Loop   Client
>   Loop   General
>   Mozilla Localizations  -- Any --
>   Mozilla QA Mozmill Tests
>   Mozilla Services   -- Any --
>   NSPR   -- Any --
>   NSS-- Any --
>   Plugins-- Any --
>   Snippets   -- Any --
>   Toolkit-- Any --
> 
> If there is a component that I missed please let me or the bugzilla team know
> so we can fix it.
> 
> This message was posted to dev-quality and has been crossposted to on
> firefox-dev and dev-platform. Please follow up on dev-quality.
> 
> Thank You.
>  Marc S.
> Firefox Q.E. Manager
> 
> ___
> firefox-dev mailing list
> firefox-...@mozilla.org
> https://mail.mozilla.org/listinfo/firefox-dev
> 
___
dev-platform mailing list
dev-platform@lists.mozilla.org
https://lists.mozilla.org/listinfo/dev-platform

Re: Running mochitests from a copy of the objdir?

[Jonathan Griffin]

Hi Neil,

Can you show us the command-line you're using?

Jonathan

On 8/19/2014 1:53 AM, Neil wrote:

Gregory Szorc wrote:


On 8/18/2014 4:45 PM, Neil wrote:



Time was that you could just python runtests.py to run mochitests.

Then we needed modules that you don't get in the default python, so 
you had to invoke python from the virtualenv instead.


Now that doesn't work either, because it's trying to run .mozconfig, 
so my questions are a) why and b) how do I stop it? (I found 
--binary but that didn't seem to be enough on its own.)



Can you please describe your workflow

It's quite simple: python runtests.py had all the right defaults for 
me, while the Makefile targets pass a bunch of parameters that I don't 
want, and mach doesn't work in comm-central yet, so I don't know how 
badly that would work out.


(as opposed to giving you hints on how to subtly hack around the 
existing implementation)



I'd prefer hints on how to fix the build system, but I'll take hacks too.



___
dev-platform mailing list
dev-platform@lists.mozilla.org
https://lists.mozilla.org/listinfo/dev-platform

Re: Running mochitests from a copy of the objdir?

[Gregory Szorc]

On 8/19/14 1:53 AM, Neil wrote:

Gregory Szorc wrote:


On 8/18/2014 4:45 PM, Neil wrote:



Time was that you could just python runtests.py to run mochitests.

Then we needed modules that you don't get in the default python, so
you had to invoke python from the virtualenv instead.

Now that doesn't work either, because it's trying to run .mozconfig,
so my questions are a) why and b) how do I stop it? (I found --binary
but that didn't seem to be enough on its own.)


Can you please describe your workflow


It's quite simple: python runtests.py had all the right defaults for me,
while the Makefile targets pass a bunch of parameters that I don't want,
and mach doesn't work in comm-central yet, so I don't know how badly
that would work out.


(as opposed to giving you hints on how to subtly hack around the
existing implementation)


I'd prefer hints on how to fix the build system, but I'll take hacks too.


Well, mach seems to be working for people doing m-c development. I think 
the underlying bug here is "mach mochitest doesn't work for 
comm-central." It should. I'm sure there is a bug on file somewhere.


___
dev-platform mailing list
dev-platform@lists.mozilla.org
https://lists.mozilla.org/listinfo/dev-platform

Re: Disabling a few mochitests this week, and upcoming changes next week to browser-chrome

[Ed Morley]

On 19/08/2014 16:46, Gavin Sharp wrote:

A few issues here:
- This particular case (we're making broad changes to how the tests
run that causes many new failures) was not what that policy was meant
to cover. We need some leeway to handle this situation differently.


Agreed that this case is possibly slightly different - however this 
change is merely exposing already flaky tests - and this change is very 
much overdue. If the tests mentioned are really that important, then 
could we please get people allocated to fixing them? You can't exactly 
say that Joel hasn't given sufficient notification to the lists & yet 
we've had virtually no interested in resolving the issues.



- This policy should probably clarify that any test disabling patches
still require module owner/peer review.


The policy does state:
"In the rare case we are disabling the majority of the tests (either at 
once or slowly over time) for a given feature, we need to get the module 
owner to sign off on the current state of the tests."


However it intentionally doesn't state it for smaller cases (I'm not 
including the above in this). Ultimately disabling a test is a last 
resort, and if done according to the policy, will have been performed after:
"In the case we go another 2 days with no response from a module owner, 
we will disable the test."


...in which case either the module owner will ignore the review request 
(which makes the policy pointless, since the whole idea was to set 
expectations and consequences that would actually be enforced), give it 
r- (which isn't acceptable, given they've not acted on the bug in a 
reasonable timeframe), or give it r+ (in which case, great, but this is 
the state we should be in anyway).


Ultimately no one individual test should have the right to destroy the 
SnR for a suite - and if we don't have this policy as a rapid (and 
enforceable in practice) means to control rogue tests, then the only 
other hammer the sheriffs have is to hide entire suites.


Kind regards,

Ed
___
dev-platform mailing list
dev-platform@lists.mozilla.org
https://lists.mozilla.org/listinfo/dev-platform

Re: Disabling a few mochitests this week, and upcoming changes next week to browser-chrome

[jmaher]

On Tuesday, August 19, 2014 11:46:08 AM UTC-4, Gavin Sharp wrote:
> > Going with our test disabling policy 
> > (https://wiki.mozilla.org/Sheriffing/Test_Disabling_Policy), I am going to 
> > get ready and disable tests on Friday.
> 
> 
> 
> A few issues here:
> 
> - This particular case (we're making broad changes to how the tests
> 
> run that causes many new failures) was not what that policy was meant
> 
> to cover. We need some leeway to handle this situation differently.
> 
> - This policy should probably clarify that any test disabling patches
> 
> still require module owner/peer review.
> 
> 
> 
> > You can see all the bugs that are blocking bug 992911 
> > (https://bugzilla.mozilla.org/showdependencytree.cgi?id=992911&hide_resolved=1)
> 
> 
> 
> We cannot disable all of the tests in that list wholesale. At least
> 
> bug 1041569, bug 1041583, bug 1017187, bug 1001820, and bug 963075

Bug 1041569 - has been needinfo'd for 4 weeks, it is only disabling this on win 
debug, I don't know how to reduce that anymore, especially if nobody wants to 
look at it.

bug 1041583 - no activity in 4+ week as well.  We are only disabling it on 
windows debug

bug 1017187 - no reply in 4+ weeks, this is disabling the test on all debug 
branches.

bug 1001820 - idle for 3 weeks until last week and it was recommended to 
disable the test (in fact it appears to be disabled)

bug 963075 - no activity for 4 weeks, we need to disable 2 pdf tests on opt 
builds only.

All in all we are changing 12 tests in the manifests.

It is obvious based on the lack of response in the bugs that fixing these bugs 
are not a priority, changing the way we run these tests will only give us more 
reliable results and less churn when we need to adjust chunking or jobs.  
___
dev-platform mailing list
dev-platform@lists.mozilla.org
https://lists.mozilla.org/listinfo/dev-platform

Re: Disabling a few mochitests this week, and upcoming changes next week to browser-chrome

[Gavin Sharp]

> Going with our test disabling policy 
> (https://wiki.mozilla.org/Sheriffing/Test_Disabling_Policy), I am going to 
> get ready and disable tests on Friday.

A few issues here:
- This particular case (we're making broad changes to how the tests
run that causes many new failures) was not what that policy was meant
to cover. We need some leeway to handle this situation differently.
- This policy should probably clarify that any test disabling patches
still require module owner/peer review.

> You can see all the bugs that are blocking bug 992911 
> (https://bugzilla.mozilla.org/showdependencytree.cgi?id=992911&hide_resolved=1)

We cannot disable all of the tests in that list wholesale. At least
bug 1041569, bug 1041583, bug 1017187, bug 1001820, and bug 963075
need alternate solutions (i.e. more targeted disabling or fixes).

Gavin

On Tue, Aug 19, 2014 at 7:07 AM, jmaher  wrote:
> It has been 3+ weeks since Vaibhav and I found the remaining issues with 
> --run-by-dir (https://bugzilla.mozilla.org/show_bug.cgi?id=992911) for 
> browser-chrome.  Since then a few issues have been fixed, and many have been 
> ignored.
>
> Going with our test disabling policy 
> (https://wiki.mozilla.org/Sheriffing/Test_Disabling_Policy), I am going to 
> get ready and disable tests on Friday.  This weekend I can sort out any 
> remaining issues (there are always new issues that show up as the product 
> changes) and turn on --run-by-dir next week.
>
> You can see all the bugs that are blocking bug 992911 
> (https://bugzilla.mozilla.org/showdependencytree.cgi?id=992911&hide_resolved=1)
>
> Happy hacking,
> -jmaher
> ___
> dev-platform mailing list
> dev-platform@lists.mozilla.org
> https://lists.mozilla.org/listinfo/dev-platform
___
dev-platform mailing list
dev-platform@lists.mozilla.org
https://lists.mozilla.org/listinfo/dev-platform

Re: Disabling a few mochitests this week, and upcoming changes next week to browser-chrome

[Kyle Huey]

Since these are front-end tests you'd be better off sending this to firefox-dev.

- Kyle

On Tue, Aug 19, 2014 at 7:07 AM, jmaher  wrote:
> It has been 3+ weeks since Vaibhav and I found the remaining issues with 
> --run-by-dir (https://bugzilla.mozilla.org/show_bug.cgi?id=992911) for 
> browser-chrome.  Since then a few issues have been fixed, and many have been 
> ignored.
>
> Going with our test disabling policy 
> (https://wiki.mozilla.org/Sheriffing/Test_Disabling_Policy), I am going to 
> get ready and disable tests on Friday.  This weekend I can sort out any 
> remaining issues (there are always new issues that show up as the product 
> changes) and turn on --run-by-dir next week.
>
> You can see all the bugs that are blocking bug 992911 
> (https://bugzilla.mozilla.org/showdependencytree.cgi?id=992911&hide_resolved=1)
>
> Happy hacking,
> -jmaher
> ___
> dev-platform mailing list
> dev-platform@lists.mozilla.org
> https://lists.mozilla.org/listinfo/dev-platform
___
dev-platform mailing list
dev-platform@lists.mozilla.org
https://lists.mozilla.org/listinfo/dev-platform

Disabling a few mochitests this week, and upcoming changes next week to browser-chrome

[jmaher]

It has been 3+ weeks since Vaibhav and I found the remaining issues with 
--run-by-dir (https://bugzilla.mozilla.org/show_bug.cgi?id=992911) for 
browser-chrome.  Since then a few issues have been fixed, and many have been 
ignored.

Going with our test disabling policy 
(https://wiki.mozilla.org/Sheriffing/Test_Disabling_Policy), I am going to get 
ready and disable tests on Friday.  This weekend I can sort out any remaining 
issues (there are always new issues that show up as the product changes) and 
turn on --run-by-dir next week.

You can see all the bugs that are blocking bug 992911 
(https://bugzilla.mozilla.org/showdependencytree.cgi?id=992911&hide_resolved=1)

Happy hacking,
-jmaher
___
dev-platform mailing list
dev-platform@lists.mozilla.org
https://lists.mozilla.org/listinfo/dev-platform

Re: Running mochitests from a copy of the objdir?

[Neil]

Gregory Szorc wrote:


On 8/18/2014 4:45 PM, Neil wrote:
 


Time was that you could just python runtests.py to run mochitests.

Then we needed modules that you don't get in the default python, so you had to 
invoke python from the virtualenv instead.

Now that doesn't work either, because it's trying to run .mozconfig, so my 
questions are a) why and b) how do I stop it? (I found --binary but that didn't 
seem to be enough on its own.)
   


Can you please describe your workflow

It's quite simple: python runtests.py had all the right defaults for me, 
while the Makefile targets pass a bunch of parameters that I don't want, 
and mach doesn't work in comm-central yet, so I don't know how badly 
that would work out.



(as opposed to giving you hints on how to subtly hack around the existing 
implementation)


I'd prefer hints on how to fix the build system, but I'll take hacks too.

--
Warning: May contain traces of nuts.
___
dev-platform mailing list
dev-platform@lists.mozilla.org
https://lists.mozilla.org/listinfo/dev-platform