Some more details on how we're approaching this problem from the
infrastructure side:
Releng recently gave us the ability to run select jobs on faster VM's
than the default, see
https://bugzilla.mozilla.org/show_bug.cgi?id=1031083. We have B2G
emulator media mochitests scheduled on cedar using these faster VM's.
After fixing a minor problem with these, we'll be able to see if these
faster VM's solve the problem. Local experiments suggest they do, but
it will take a number of runs in buildbot to be sure.
If that doesn't fix the problem, we have the option of trying still
faster VM's (at greater cost), or trying to run the tests on real
hardware. The disadvantage of running the tests on real hardware is
that such hardware doesn't scale very readily and is already stretched
pretty thin, and the emulator doesn't currently run on our linux
hardware slaves, and will require some amount of work to fix.
This work is being tracked in
https://bugzilla.mozilla.org/show_bug.cgi?id=994920.
Jonathan
On 8/28/2014 3:06 PM, Randell Jesup wrote:
I wrote in April:
The B2G emulator design is causing all sorts of problems. We just fixed
the #2 orange which was caused by the Audio channel StartPlaying()
taking up to 20 seconds to run (and we "fixed" it by effectively
removing some timeouts). However, we just wasted half a week trying to
land AEC & MediaStreamGraph improvements. We still haven't landed due
to yet another B2G emulator orange, but the solution we used for the M10
problem doesn't fix the fundamental problems with B2G emulator.
You can read the earlier thread (starting 7-apr) about this issue. We
wallpapered over the issues (including turning down 'fake' audio
generation to 1/10th realtime and letting it underflow).
The problems with the b2g emulator have just gotten worse as we add more
tests and make changes to improve the system that give the emulators
fits.
Right now, we're looking at being blocked from landing important
improvements (that make things *not* fail due to perf timeouts in
real-user-scenarios) because b2g-emulator chokes on anything even
smelling of realtime data. It can stall for 10's of seconds (see
above), or even minutes. Even running a single test can cause other,
unrelated tests to perma-orange.
The stuff we've had to do (like turning down audio generation) to block
oranges in the current setup makes the tests very non-real-world, and so
greatly diminishes their utility anyways.
There was work being done to move media and other semi-realtime tests to
faster hardware; that is happening but it's not ready yet. (For reference,
in April tests showed that a b2g emulator mochitest that took <10
seconds on my Xeon took 350-450 seconds on tbpl.)
The fundamental problem is that b2g-emulator can't deal safely with any
sort of realtime or semi-realtime data unless run on a fast machine.
The architecture for the emulator setup means the effective CPU power is
dependent on the machine running the test, and that varies a lot (and
tbpl machines are WAY slower than my 2.5 year old desktop). Combine
that with Debug being much slower, and it's recipe for disaster for any
sort of time-dependent tests.
...
So, what do we do? Because if we do nothing, it will only get worse.
So we've done nothing (that's landed at least), and it has gotten worse,
and we're at the breaking point where b2g emulator (especially debug)
for media tests (especially webrtc) is providing negative value, and
blocking critically important improvements.
We've just landed bug 1059867 to disable most webrtc tests on the emulator
until we can get them running on hardware that has the power to run them
(or other fixes make them viable again (bug 1059878)). We may need to
consider similar measures for other media tests (webaudio, etc). In the
meantime, we're going to try to run local emulator pull/build/mochitest
cronjobs on faster desktop machines (perhaps mine) on a daily or perhaps
continuous basis. (Poor man's tbpl - maybe I'll un-mothball tinderbox
for some nostalgic flames...)
Also note that webrtc tests do run on the b2g desktop tbpl runs, so we
have some coverage.
I hope we can find a better solution than "run it on my dev machine"
sometime soon (very soon!), but right now that's better than playing
whack-a-random-timeout or just increasing run times to infinity.
P.S. there are some interesting threads of stuff that could help a lot,
like the comment Jay Wang made in April about SpecialPowers.exactGC
taking 3-10s per instance on b2g debug, and tons of them being run (one
test took 102s to finish, and had 90 gc's which mostly took ~10s each).
Bug 1012516
_______________________________________________
dev-platform mailing list
dev-platform@lists.mozilla.org
https://lists.mozilla.org/listinfo/dev-platform
