2017-06-16 6:22 GMT-07:00 Ehsan Akhgari <ehsan.akhg...@gmail.com>: > On 06/15/2017 04:37 PM, Nathan Froyd wrote: > >> Would it be feasible to use headless mode for mochitests (or reftests, >> etc. etc.)? >> > Running the tests that we rely on for correctness on an environment that > looks this different than the environment that our users run sounds like a > bad thing to do. What would be the advantage of doing so?
Two advantages would be: * it would likely be much faster, allowing folks to get a substantially tighter feedback loop of developing and running a set of tests. My expectation, based on previous experience in the web-development world, is that people would run tests more often during the development cycle, which would catch other errors more quickly during development, making development more efficient. * it would allow you to run (some part of a) test suite without having to use a VM, a separate machine, or some other hack to avoid messing up all the tests that are focus-dependent. This would also likely lead to more frequent local test runs. The concerns about differing environment do sound reasonable. One conceivable way forward might be to start by adding add headless testing infrastructure to the tree so that developers could use them locally as "smoke tests", and then run the real tests locally or by try before landing. At the same time, they could added as Tier > 1 tests so that we could collect data to see what the level of failure differences between those tests and normal mochitests are to figure out next steps. Dan _______________________________________________ dev-platform mailing list dev-platform@lists.mozilla.org https://lists.mozilla.org/listinfo/dev-platform
