> The thing about intermittent test failures is that they seem to show up on treeherder but not when you run the tests locally. So the process of fixing them can require lots of pushes to github to trigger new treeherder test runs. And since test runs take a long time, it can be hard to iterate quickly on these bugs.
Just in case someone finds it useful: Some time ago the main reason why we couldn't reproduce intermittent Gij failures in Messages app locally was that local environment was too fast comparing to Treeherder. So we started running/debugging such tests inside _slow_ vagrant box - almost all intermittent Gif tests became perma-red there. At that time I used gaia-vagrant [1] with tweaked RAM and number of CPU cores [2].+ :gasolin's foxbox [3] can be tweaked as well. [1] https://github.com/azasypkin/gaia-vagrant [2] https://github.com/azasypkin/gaia-vagrant/blob/master/Vagrantfile#L58-L60 [3] https://github.com/gasolin/foxbox On Wed, Nov 11, 2015 at 8:34 PM, David Flanagan <[email protected]> wrote: > Michael and others are making good progress fixing intermittent failures > (see his "Orange is the new bad" thread.) He's doing this with a patch in > bug 1222215 that turns off automatic retries for failing tests. Hopefully > he will be able to actually land that soon. > > But until that is landed, if you are working on a test, it is worth trying > out your test with that patch applied so you can see if it is failing > intermittently. You can probably just cherry-pick this commit: > https://github.com/nullaus/gaia/commit/735dd67cdb565f152e86f43506836a5f6e134b71 > > The thing about intermittent test failures is that they seem to show up on > treeherder but not when you run the tests locally. So the process of fixing > them can require lots of pushes to github to trigger new treeherder test > runs. And since test runs take a long time, it can be hard to iterate > quickly on these bugs. I worked yesterday on one of these bugs and realized > that I could make the tests runs go faster by turning off the tests I > didn't care about. > > If you edit tests/taskcluster/tasks.yml you can comment out (with #) > almost all of the lines in that file, leaving only > > marionette_js_tests: > chunks: 2 > > Note that I changed from 40 chunks to 2 chunks. This is because the next > useful change is to tests/taskcluster/tasks/marionette_js_tests.yml > > where you can add: > > APP: gallery # or whatever app you're working on tests for > > at line 23, as part of the `env:` section. > > When you push those changes to your PR, the test run that is triggered > will run the usual "Gaia decision opt" and "Bootstrap" tests, then it will > only the Gij integration tests, in two chunks, for your one app. It still > isn't fast, but it should be a lot faster than running a full set of tests. > > David > > _______________________________________________ > dev-fxos mailing list > [email protected] > https://lists.mozilla.org/listinfo/dev-fxos > >
_______________________________________________ dev-fxos mailing list [email protected] https://lists.mozilla.org/listinfo/dev-fxos
