On 06/16/2017 10:31 AM, Dan Mosedale wrote:
2017-06-16 6:22 GMT-07:00 Ehsan Akhgari <ehsan.akhg...@gmail.com
<mailto:ehsan.akhg...@gmail.com>>:
On 06/15/2017 04:37 PM, Nathan Froyd wrote:
Would it be feasible to use headless mode for mochitests (or
reftests,
etc. etc.)?
Running the tests that we rely on for correctness on an
environment that looks this different than the environment that
our users run sounds like a bad thing to do. What would be the
advantage of doing so?
Two advantages would be:
* it would likely be much faster, allowing folks to get a
substantially tighter feedback loop of developing and running a set of
tests. My expectation, based on previous experience in the
web-development world, is that people would run tests more often
during the development cycle, which would catch other errors more
quickly during development, making development more efficient.
The speed argument is a good one, but I haven't seen any numbers, so I
don't know what difference we are talking about. :-) It could be that
the results may actually be very impressive, and it differ per OS and
per test suite, etc. But I think we should first gather some evidence
as to how much faster these builds are. Hopefully they shouldn't be
_that_ much faster, because otherwise we'd have huge overheads in our
widget layer which we should probably do something about! (But of
course I'm cautious to make too many assumptions here as we are dealing
with OS code.)
* it would allow you to run (some part of a) test suite without having
to use a VM, a separate machine, or some other hack to avoid messing
up all the tests that are focus-dependent. This would also likely
lead to more frequent local test runs.
Oh certainly, I think for Firefox developers using Mac and Windows, this
is absolutely fantastic news! (Linux developers have indeed been spoiled
in this department!) My question was more about running tests in the
infrastructure, I should have been more clear.
The concerns about differing environment do sound reasonable.
One conceivable way forward might be to start by adding add headless
testing infrastructure to the tree so that developers could use them
locally as "smoke tests", and then run the real tests locally or by
try before landing. At the same time, they could added as Tier > 1
tests so that we could collect data to see what the level of failure
differences between those tests and normal mochitests are to figure
out next steps.
I suggest before spending a lot of time on that, let's first gather some
evidence that speed-wise that would be a worth-while endeavor (and if we
do have that evidence already then great!) For the purpose of local
testing, I suppose people can just do it when the builds start to work
on the respective platform, and just like running tests under xvfb (or
on your local machine anyway for that matter) there's no guarantee that
the baseline configuration will pass out of the box for all tests across
the tree, and if you come across tests that don't you file bugs etc.
_______________________________________________
dev-platform mailing list
dev-platform@lists.mozilla.org
https://lists.mozilla.org/listinfo/dev-platform
