Another pain point: running all relevant tests. Many features have relevant tests across many test suites, sometimes spread across different directories. When people are hacking on a feature, they should have a way to run all relevant tests for that feature. All too often I've submitted something to Try (or worse landed without Try) only to find out I wasn't locally running all tests relevant for the thing that changed.
I did some work in bug 987360 around test "tagging." e.g. "mach test fhr" would run all tests tagged as relevant to Firefox Health Report. Of course, this subject overlaps with "intelligent scheduling of automation jobs based on what changed." A corollary is making tests easier to run. The N variants of mochitest-* mach commands IMO should be excised and consolidated into `mach mochitest`. This is better, but itself less ideal than `mach test` (which exists today and automatically runs the test runner relevant to the requested tests). `mach test` is less ideal than `mach <filename>`, which realizes the argument is a test file and invokes the test harness appropriate for that test. FWIW, I don't think the documentation around the mach commands for test selection is that great. e.g. I'm not sure how many people realize that they can run `mach xpcshell-test test_foo.js` from the topsrcdir and all `test_foo.js` files under source control are executed. Perhaps we could add some docs to `mach help` or drop some inline hints that people don't have to type so much. On Thu, Mar 12, 2015 at 3:51 PM, Jonathan Griffin <jgrif...@mozilla.com> wrote: > The A-Team is embarking on a project to improve the developer experience > when running unittests locally. This project will address the following > frequently-heard complaints: > > * Locally developers often use mach to run tests, but tests in CI use > mozharness, which can result in different behaviors. > * It's hard to reproduce a try job because it's hard to set up the test > environment and difficult to figure out which command-line arguments to > use. > * It's difficult to run tests from a tests.zip package if you don't have a > build on that machine and thus can't use mach. > * It's difficult to run tests through a debugger using a downloaded build. > > The quintessential use case here is making it easy to reproduce a try run > locally, without a local build, using a syntax something like: > > * runtests --try 2844bc3a9227 > > Ideally, this would download the appropriate build and tests.zip package, > bootstrap the test environment, and run the tests using the exact same > arguments as are used on try, optionally running it through an appropriate > debugger. You would be able to substitute a local build and/or local > tests.zip package if desired. You would be able to override command-line > arguments used in CI if you wanted to, otherwise the tests would be run > using the same args as in CI. > > What other use cases would you like us to address, which aren't derivatives > of the above issues? > > Thanks for your input, > > Jonathan > _______________________________________________ > dev-platform mailing list > dev-platform@lists.mozilla.org > https://lists.mozilla.org/listinfo/dev-platform > _______________________________________________ dev-platform mailing list dev-platform@lists.mozilla.org https://lists.mozilla.org/listinfo/dev-platform
