Re: [b2g] Eideticker results for app startup on 2.0 (aurora) and b2g-inbound now coming in
On 14-07-17 04:22 PM, William Lachance wrote: This brings up the point that we should probably come up with some way of making sure that we're always using the same workload between our three perf testing frameworks (b2gperf, make test-perf, and eideticker), or at least being able to audit that's the case. Anyway, at the least these results should be able to track relative improvements with some reliability. :) Update: tracking this in bug 1040705: https://bugzilla.mozilla.org/show_bug.cgi?id=1040705 Will ___ dev-b2g mailing list dev-b2g@lists.mozilla.org https://lists.mozilla.org/listinfo/dev-b2g
[b2g] Eideticker results for app startup on 2.0 (aurora) and b2g-inbound now coming in
Just wanted to announce that we now have Eideticker startup time results coming in for these two branches for the Flame (512M configuration), with videos: 2.0: http://eideticker.mozilla.org/b2g/#/flame-512/2.0/b2g-camera-startup/timetostableframe b2g-inbound: http://eideticker.mozilla.org/b2g/#/flame-512/b2g-inbound/b2g-camera-startup/timetostableframe These tests use a visual comparison of live video to determine when things have loaded, which is similar to the methodology used by some of our partners. Will ___ dev-b2g mailing list dev-b2g@lists.mozilla.org https://lists.mozilla.org/listinfo/dev-b2g
Re: [b2g] Eideticker results for app startup on 2.0 (aurora) and b2g-inbound now coming in
We're currently using settings that should be about equivalent to the light reference workload: https://github.com/mozilla-b2g/gaia/blob/master/test_media/reference-workload/makeReferenceWorkload.sh#L31 Although this is mostly defined adhoc inside the test harness: https://github.com/mozilla/eideticker/blob/master/src/eideticker/eideticker/b2gtestmixins.py This brings up the point that we should probably come up with some way of making sure that we're always using the same workload between our three perf testing frameworks (b2gperf, make test-perf, and eideticker), or at least being able to audit that's the case. Anyway, at the least these results should be able to track relative improvements with some reliability. :) Will On 14-07-17 01:51 PM, Milan Sreckovic wrote: This is great. What’s the “size” of the user data on the phone? One of the reference-workload-medium or one of the other ones? -- - Milan On Jul 17, 2014, at 12:07 , William Lachance wlacha...@mozilla.com wrote: Just wanted to announce that we now have Eideticker startup time results coming in for these two branches for the Flame (512M configuration), with videos: 2.0: http://eideticker.mozilla.org/b2g/#/flame-512/2.0/b2g-camera-startup/timetostableframe b2g-inbound: http://eideticker.mozilla.org/b2g/#/flame-512/b2g-inbound/b2g-camera-startup/timetostableframe These tests use a visual comparison of live video to determine when things have loaded, which is similar to the methodology used by some of our partners. Will ___ dev-b2g mailing list dev-b2g@lists.mozilla.org https://lists.mozilla.org/listinfo/dev-b2g ___ dev-b2g mailing list dev-b2g@lists.mozilla.org https://lists.mozilla.org/listinfo/dev-b2g
[b2g] Eideticker results for the Flame now available
Just wanted to give a heads up that we're now generating startup test results for loading various Gaia apps on the Flame using Eideticker: http://eideticker.mozilla.org/b2g/#/flame/b2g-contacts-startup/timetostableframe For those not familiar with it already, Eideticker uses a high speed camera to measure application performance. But not only does Eideticker generate numbers, you can also dig into what it captured to see whether or not a regression is user perceivable. You can click on the datapoints in the chart above to get a video and even see a frame-by-frame analysis of what eideticker is measuring. Take, for example, this test run where we're measuring startup time of the contacts application: http://eideticker.mozilla.org/b2g/detail.html?id=2b007f8cfd8b11e3923c10ddb19eacac#/framecannyentropies I wrote up a blog post talking more about this (as well as some short-term plans) here: http://wrla.ch/blog/2014/06/end-of-q2-eideticker-update-flame-tests-future-plans/ Will ___ dev-b2g mailing list dev-b2g@lists.mozilla.org https://lists.mozilla.org/listinfo/dev-b2g
Re: [b2g] a proposal on one tool to rule them all
[ Bcc'ing auto-tools, in case a-team peoples missed this ] Hi Dave, This sounds really cool. I suspect that posting this on dev.platform might yield more comments... personally I only have some minor comments which are hopefully still useful: On 04/07/2014 01:34 PM, Dave Huseby wrote: Here is what I'm thinking in rough priority order: 1. Unify event signaling for all sources of interesting events. Right now we have a bunch of interesting perf-related events (e.g. checkerboarding, memory pressures, zRAM caching, power consumption spikes, slow event handling, etc) that are reported in different ways. For instance, Ben Kelly showed us how to spot memory pressure events in the log output. We learned that the graphics pipeline knows when checkerboarding occurs and could report that. We saw how the powertool can see when power consumption spikes and could report it. zRAM caching events reported from B2G. RIL, WiFi, GPS, etc all reported from B2G in the logs. So I know there was an effort to add performance counters to marionette last year, which sounds similar to what you are talking about. I think we eventually abandoned that effort, for what reason I don't know. CC'ing mdas, who IIRC was working on it -- hopefully she can clarify. The first step I think we should take is figuring out a way to report interesting events via a unified mechanism. For events generated on the device, I think the Profiler::addMarker is the good way to go. Not only would it be a single, central reporting mechanism available from both C++ and JS, it would also get those events into the profiling capture buffer. The profiling capture mechanism has a small circular buffer for storing stack traces and markers. There's no reason we couldn't have a thread transmitting event markers back to the host over a socket in realtime. For events generated off of the device (e.g. powertool events, eideticker events, orangutan input driver events), they could also be reported over a socket to the event consumer. So for orangutan, the thing that counts the most is when we dispatch the event to the kernel subsystem to the device, not when we trigger the program off the device. Would it be possible for external programs like this to report something to gecko, then gecko would report them over the socket? I don't want to bikeshed the implementation too much, just curious about your thoughts here... 2. Build a single consumer of the event stream. Once we get event reporting over a socket, the next step is to build a centralized consumer of the events. NodeJS is probably a good choice for this tool. It should probably function as a daemon on the host machine and be capable of detecting devices and executing adb commands to set up the port forwarding automagically. The event consumer will listen on a single port--like a web server--and accept multiple incoming connections from event sources. So the way android port-forwarding works, it is currently possible for the host to connect to a tcp socket on the device easily over usb (this is how we do marionette), but not the reverse. So I suspect it would be easier for gecko to emit events over a socket, then have some process on the host hook into that. 3. Build a configurable dispatch mechanism for the event consumer. The last piece involves building a configurable dispatch mechanism for hooking up actions to certain events. The first--and maybe the only--action we will want is to gather the profiling data for the last N seconds whenever we see an interesting event. For instance, when I'm optimizing idle power draw, a power spike event from the powertool should trigger dumping the profiling data for the last N seconds. I'll also want the profiling data when I see a transient memory spikes and checkerboarding events, and... Now that I think about it, grabbing the profiler data in response to a marker added from JS would be tremendously helpful for debugging JS event ordering and dispatching issues (i.e. the gaia window manager). Proposal: To keep this simple enough that it could be implemented quickly, I propose a very simple event transmission mechanism. The data on the socket will flow in only one direction--from the event source to the consumer. Events will contain JSON encoded data: a timestamp with sub-ms resolution, a subject string, and a body string. The subject is what triggers an action, the body is used for any optional/extended data. We'll use TCP. So the ateam has been working on adding something called structured logging to all our frameworks, which uses a json protocol much like what you're describing. It would be great if you could use something similar here, so all the python tools we're creating for reading/parsing structured logs could be reused here. I believe we also have some existing javascript helper methods in the tree for creating these types of json messages. Here's a description of the format we're using:
Re: [b2g] Is it Possible to use Orangfuzz on the Simulator or B2G Desktop?
On 13-07-31 01:38 PM, Gary Kwong wrote: Is it Possible to use Orangfuzz on the Simulator or B2G Desktop? Since orangfuzz is based on the Orangutan framework [1], I've only tried it on a Unagi test device, and it should work on a Geeksphone Keon as well. I don't think it works as-is on the Simulator / B2G desktop, but wlach (cc'ed) might know more. Apparently it needs at least the /dev/input interface which real hardware provides. Yeah, orangutan won't work on anything but a real device (or emulator). It may be possible to rework orangfuzz to use marionette action chain events (https://developer.mozilla.org/en-US/docs/Marionette/Marionette#Action_Chain_Methods), but I'm not sure how difficult that would be. Will ___ dev-b2g mailing list dev-b2g@lists.mozilla.org https://lists.mozilla.org/listinfo/dev-b2g
Re: [b2g] Fwd: Mobile Automation Deep Dive Meeting tomorrow Wed Jan 22, 1PM PST
Unfortunately as I posted elsewhere, this meeting had to be canceled because Joel's internet connection wasn't working. We'll carry over this agenda to the next meeting (Wednesday February 6). I'll post another announcement to dev.planning in just under two weeks! Will On 01/22/2013 08:34 PM, Dietrich Ayala wrote: Might interest Gaia B2G developers. -- Forwarded message -- From: William Lachance wlacha...@mozilla.com Date: Tue, Jan 22, 2013 at 5:24 PM Subject: Mobile Automation Deep Dive Meeting tomorrow Wed Jan 22, 1PM PST To: dev-plann...@lists.mozilla.org Tomorrow is our second mobile automation deep dive meetings, where we discuss in-depth an aspect of mobile automated testing for FirefoxOS and Firefox for Android. The idea is to both expand the common base of knowledge on the state of the art in this area (as we continue to add new mobile test harnesses, frameworks, and platforms to the mix) as well as come up with plans/ideas which could help us in the future. The focus tomorrow will be on how we can reduce the amount of time it takes to get a new mobile platform up and running. Our recent experiences with getting Android tests running on the Pandaboard suggest to show that there's plenty of room for improvement. Joel Maher (main developer from the A-team working on that project) will do a presentation on how things went down, as well as his ideas for making things better in the future. After that, we'll have a hopefully productive group discussion and produce a long list of bugs and action items. :) For more information, see our agenda here: https://etherpad.mozilla.org/**mobile-deepdivehttps://etherpad.mozilla.org/mobile-deepdive Anyone with interest in this topic is more than welcome to join us. Vidyo: tor-spadina Phone: 650-903-0800 or 650-215-1282 x92 Conf# 95589 (US/INTL) 1-800-707-2533 (pin 369) Conf# 95589 (US) IRC:irc://irc.mozilla.org:6697/#**ateamhttp://irc.mozilla.org:6697/#ateam Will __**_ dev-planning mailing list dev-plann...@lists.mozilla.org https://lists.mozilla.org/**listinfo/dev-planninghttps://lists.mozilla.org/listinfo/dev-planning ___ dev-b2g mailing list dev-b2g@lists.mozilla.org https://lists.mozilla.org/listinfo/dev-b2g
