Re: B2G emulator issues

2014-08-28 Thread Randell Jesup
I wrote in April: The B2G emulator design is causing all sorts of problems. We just fixed the #2 orange which was caused by the Audio channel StartPlaying() taking up to 20 seconds to run (and we fixed it by effectively removing some timeouts). However, we just wasted half a week trying to land

Re: B2G emulator issues

2014-08-28 Thread Jonathan Griffin
Some more details on how we're approaching this problem from the infrastructure side: Releng recently gave us the ability to run select jobs on faster VM's than the default, see https://bugzilla.mozilla.org/show_bug.cgi?id=1031083. We have B2G emulator media mochitests scheduled on cedar

Re: B2G emulator issues

2014-08-28 Thread Nils Ohlmeier
Thanks Jonathan for the update. I would like to point out that at least for the WebRTC tests which do test the connection between two WebRTC clients we theoretically also have the option to split the tests into two half's (which we do for steeplechase tests already anyhow), and then start two

Re: B2G emulator issues

2014-05-23 Thread JW Wang
On Tuesday, April 8, 2014 11:45:15 PM UTC+8, Mike Habicher wrote: In my experience running tests locally, a single mochitest run on the ARM emulator (hardware: Thinkpad X220, 16GB RAM, SSD) where everything was built with 'B2G_DEBUG=0 B2G_NOOPT=0' will run in 2 to 3 minutes. The same test,

Re: B2G emulator issues

2014-04-15 Thread Vicamo Yang
I ran crashtest/reftest/marionette/xpcshell/mochitest on emulator-x86-kk, have filed related bugs and make them block bug 753928. Basically: 1) need to carry --emulator x86 automatically (bug 996443) 2) to add x86 emulator for xpcshell tests (bug 996473) 3) PROCESS-CRASH at the end of

Re: B2G emulator issues

2014-04-15 Thread Randell Jesup
I ran crashtest/reftest/marionette/xpcshell/mochitest on emulator-x86-kk, have filed related bugs and make them block bug 753928. Basically: 1) need to carry --emulator x86 automatically (bug 996443) 2) to add x86 emulator for xpcshell tests (bug 996473) 3) PROCESS-CRASH at the end of

Re: B2G emulator issues

2014-04-15 Thread Vicamo Yang
於 4/15/14, 9:42 PM, Randell Jesup 提到: I ran crashtest/reftest/marionette/xpcshell/mochitest on emulator-x86-kk, have filed related bugs and make them block bug 753928. Basically: 1) need to carry --emulator x86 automatically (bug 996443) 2) to add x86 emulator for xpcshell tests (bug 996473)

Re: B2G emulator issues

2014-04-09 Thread Thomas Zimmermann
Hi That is what the emulator is already doing. If we start emulating HW down to individual CPU cycles, it'll only get slower. :( I think this is wrong in some way. Otherwise I wouldn't see this: 1) running on TBPL (AWS) the internal timings reported show the specific test going from 30

Re: B2G emulator issues

2014-04-09 Thread Randell Jesup
That is what the emulator is already doing. If we start emulating HW down to individual CPU cycles, it'll only get slower. :( I think this is wrong in some way. Otherwise I wouldn't see this: 1) running on TBPL (AWS) the internal timings reported show the specific test going from 30

Re: B2G emulator issues

2014-04-08 Thread Thomas Zimmermann
Hi, Thanks for bringing up this issue. One option (very, very painful, and even slower) would be a proper device simulator which simulates both the CPU and the system hardware (of *some* B2G phone). This would produce the most realistic result with an emulator. That is what the emulator

Re: B2G emulator issues

2014-04-08 Thread Mike Habicher
On 14-04-07 08:49 PM, Ehsan Akhgari wrote: On 2014-04-07, 8:03 PM, Robert O'Callahan wrote: When you say debug, you mean the emulator is running a FirefoxOS debug build, not that the emulator itself is built debug --- right? Given that, is it a correct summary to say that the problem is that

Re: B2G emulator issues

2014-04-08 Thread Randell Jesup
Hi, Thanks for bringing up this issue. One option (very, very painful, and even slower) would be a proper device simulator which simulates both the CPU and the system hardware (of *some* B2G phone). This would produce the most realistic result with an emulator. That is what the emulator

Re: B2G emulator issues

2014-04-08 Thread Jonathan Griffin
On 4/8/2014 1:05 AM, Thomas Zimmermann wrote: There are tests that instruct the emulator to trigger certain HW events. We can't run them on actual phones. To me, the idea of switching to a x86-based emulator seems to be the most promising solution. What would be necessary? Best regards Thomas

Re: B2G emulator issues

2014-04-08 Thread Karl Tomlinson
Randell Jesup writes: 1) running on TBPL (AWS) the internal timings reported show the specific test going from 30 seconds to 450 seconds with the patch. 2) on my local system, the test self-reports ~10 seconds, with or without the patch. Note: the timer in question is

B2G emulator issues

2014-04-07 Thread Randell Jesup
The B2G emulator design is causing all sorts of problems. We just fixed the #2 orange which was caused by the Audio channel StartPlaying() taking up to 20 seconds to run (and we fixed it by effectively removing some timeouts). However, we just wasted half a week trying to land AEC

Re: B2G emulator issues

2014-04-07 Thread Jonathan Griffin
How easy is it to identify CPU-sensitive tests? I think the most practical solution (at least in the near term) is to find that set of tests, and run only that set on a faster VM, or on real hardware (like our ix slaves). Jonathan On 4/7/2014 3:16 PM, Randell Jesup wrote: The B2G emulator

Re: B2G emulator issues

2014-04-07 Thread Geoffrey Brown
On 4/7/2014 3:16 PM, Randell Jesup wrote: The B2G emulator design is causing all sorts of problems. We just fixed That sounds very similar to some of the failures seen on the Android 2.3 emulator. Many media-related mochitests intermittently time out on the Android 2.3 emulator when run on

Re: B2G emulator issues

2014-04-07 Thread Randell Jesup
How easy is it to identify CPU-sensitive tests? Easy for some (most but not all media tests). Almost all getUserMedia/PeerConnection tests. ICE/STUN/TURN tests. Not that easy for some. And some may be only indirectly sensitive - timeouts in delay-the-rendering code, TCP/DNS/SPDY timers, etc,

Re: B2G emulator issues

2014-04-07 Thread Robert O'Callahan
When you say debug, you mean the emulator is running a FirefoxOS debug build, not that the emulator itself is built debug --- right? Given that, is it a correct summary to say that the problem is that the emulator is just too slow? Applying time dilation might make tests green but we'd be left

Re: B2G emulator issues

2014-04-07 Thread Ehsan Akhgari
On 2014-04-07, 8:03 PM, Robert O'Callahan wrote: When you say debug, you mean the emulator is running a FirefoxOS debug build, not that the emulator itself is built debug --- right? Given that, is it a correct summary to say that the problem is that the emulator is just too slow? Applying time

Re: B2G emulator issues

2014-04-07 Thread Shih-Chiang Chien
Why don’t we just switch to x86 emulator? x86 emulator runs way faster than the ARM emulator. Best Regards, Shih-Chiang Chien Mozilla Taiwan On Apr 8, 2014, at 8:49 AM, Ehsan Akhgari ehsan.akhg...@gmail.com wrote: On 2014-04-07, 8:03 PM, Robert O'Callahan wrote: When you say debug, you mean