We have landed bug 1222215 and the Gaia Marionette JS retry logic is finally gone \o/.To make this happen we: - increased the default script/seach timeout for the marionette js client [1] - made the marionette app.launch API more robust [2] - made client.helper.waitForElement and waitforElementToDisappear configurable [3][4] - disabled about 40 tests [5]
Special thanks to Aus and Gareth (among others) for helping out with this effort. With the additional 40 tests disabled, we now have ~90 out of ~380 tests disabled, which is 25%. And since we are currently in between releases (so to speak) now would be a great time to fix this up. I highly encourage you to go into our disabled list [5], and start to pick off the tests that are relevant to your module. A great way to get quick results on treeherder for an intermittent test is to use this commit [6] (replacing the TEST_FILES var with your test path) on top of your test patch. Also note that because we disabled the retry logic, Gij will fail more often. Please be a good Gaia citizen and keep an eye on the blue runs on gaia [7] and gaia-master [8]. If we catch intermittents there, we can prevent headaches for other teams once they reach b2g-inbound and m-c. I'll put some tips and tricks in the P.S. section of this email, but feel free to reach out to Aus, Gareth or myself anytime for help with these pesky intermittents. Long story short, if you are using assert you should probably use waitFor instead. But reality can be a bit more complicated, so we are at your disposal. Thanks! Michael 1.) https://bugzilla.mozilla.org/show_bug.cgi?id=1233892 2.) https://bugzilla.mozilla.org/show_bug.cgi?id=1233823 3.) https://bugzilla.mozilla.org/show_bug.cgi?id=1233880 4.) https://bugzilla.mozilla.org/show_bug.cgi?id=1235531 5.) https://github.com/mozilla-b2g/gaia/blob/master/shared/test/integration/tbpl-manifest.json 6.) https://github.com/mozilla-b2g/gaia/commit/5ca9082a33c9 7.) https://treeherder.mozilla.org/#/jobs?repo=gaia 8.) https://treeherder.mozilla.org/#/jobs?repo=gaia-master P.S. In case anyone is interested, here are a couple of common problems we saw while going through these tests. First, writing to settings or contacts DB is erratically slow. Haven't had time to investigate why yet, so make sure to set large timeouts when waiting for these operations to complete. Similarly, opening a new window (dialog, popup, activity) is always slower than device, so make sure to wait for windows to show up before switching to them. Also, all tests can fail with the following messages "RangeError: port should be >= 0 and < 65536: 65537, Error: Not connected. To write data you must call connect first.". This is the dreaded harness issue that has been around for a long time, but thankfully it only happens 2-3 out of a hundred times now. And if you see "Crash detected but error running stackwalk," it means it's time to submit a PR with some console.logs in them. Hope that helps :) On Sun, Dec 20, 2015 at 6:45 PM, Andreas Tolfsen <[email protected]> wrote: > On 18 December 2015 at 17:01, Michael Henretty <[email protected]> > wrote: > > > > On Fri, Dec 18, 2015 at 10:56 AM, Armen Zambrano Gasparnian > > <[email protected]> wrote: > >> > >> Would separating intermittent test jobs into their own separate job help > >> here? > >> That way the retry logic can be removed for the more stable tests. > > > > This is not a bad idea. However, I think the worst part about the > "internal" > > retry logic is that it hides how bad an intermittent test really is. So > > perhaps we could move all the disabled/intermittent tests [1] to a job > that > > gets hidden on b2g-inbound. > > Yes, it is possible to make that specific job a tier-2 (or 3), i.e. > hidden, job on Treeherder. > > However I cannot tell if this approach is better than just disabling > these specific tests. It’s my impression that they have been > intermittent for such a long time, that they clearly aren’t considered > valuable by developers. >
_______________________________________________ dev-fxos mailing list [email protected] https://lists.mozilla.org/listinfo/dev-fxos
