As mentioned in a previous thread [1], we still use the 319MB Flame configuration as it's the only comparable baseline we have between v2.2 and v2.5. We can't drop using the 319MB Flame for v2.5 unless we build a new baseline with a different device configuration against v2.2.
[1] https://groups.google.com/d/msg/mozilla.dev.fxos/uDrAUHGTq18/79YaFJHQCQAJ Thanks, Eli Perelman On Fri, Oct 2, 2015 at 12:58 PM, Justin D'Arcangelo <[email protected] > wrote: > Correction, my bad. Eli just informed me that his numbers earlier were > from the 319mb Flame configuration. I was under the impression that we were > no longer supporting that config. In addition, I was looking at > “fullyLoaded” and not “visuallyLoaded”. > > We have a patch on the way that should be optimizing the way we fetch > album art and should drastically cut down on memory usage. Once that has > landed, I will re-test under 319mb and see where we stand. > > -Justin > > > On Oct 2, 2015, at 1:51 PM, Justin D'Arcangelo <[email protected]> > wrote: > > Here’s the latest Raptor numbers, run locally, on Flame: > > OGA: > > jdarcangelo-20869:gaia Justin$ raptor test coldlaunch --app music-oga > --runs 5 > [Cold Launch: music-oga.gaiamobile.org] Preparing to start testing... > [Cold Launch: music-oga.gaiamobile.org] Priming application > [Cold Launch: music-oga.gaiamobile.org] Starting run 1 > [Cold Launch: music-oga.gaiamobile.org] Run 1 complete > [Cold Launch: music-oga.gaiamobile.org] Starting run 2 > [Cold Launch: music-oga.gaiamobile.org] Run 2 complete > [Cold Launch: music-oga.gaiamobile.org] Starting run 3 > [Cold Launch: music-oga.gaiamobile.org] Run 3 complete > [Cold Launch: music-oga.gaiamobile.org] Starting run 4 > [Cold Launch: music-oga.gaiamobile.org] Run 4 complete > [Cold Launch: music-oga.gaiamobile.org] Starting run 5 > [Cold Launch: music-oga.gaiamobile.org] Run 5 complete > [Cold Launch: music-oga.gaiamobile.org] Results from > music-oga.gaiamobile.org > > | Metric | Mean | Median | Min | Max | StdDev | > p95 | > | --------------------- | -------- | ------ | ------ | ------ | ------ | > -------- | > | navigationLoaded | 737 | 739 | 715 | 752 | 14.071 | > 737 | > | navigationInteractive | 900.600 | 893 | 880 | 924 | 16.305 | > 900.600 | > | visuallyLoaded | 1123 | 1121 | 1076 | 1170 | 32.323 | > 1123 | > | contentInteractive | 1123.400 | 1121 | 1077 | 1171 | 32.327 | > 1123.400 | > | fullyLoaded | 2478.400 | 2474 | 2450 | 2517 | 23.466 | > 2478.400 | > | uss | 14.542 | 14.645 | 14.301 | 14.738 | 0.182 | > 14.542 | > | rss | 34.377 | 34.480 | 34.133 | 34.574 | 0.185 | > 34.377 | > | pss | 18.775 | 18.877 | 18.534 | 18.978 | 0.186 | > 18.775 | > > [Cold Launch: music-oga.gaiamobile.org] Testing complete > jdarcangelo-20869:gaia Justin$ > > > NGA: > > jdarcangelo-20869:gaia Justin$ raptor test coldlaunch --app music --runs 5 > [Cold Launch: music.gaiamobile.org] Preparing to start testing... > [Cold Launch: music.gaiamobile.org] Priming application > [Cold Launch: music.gaiamobile.org] Starting run 1 > [Cold Launch: music.gaiamobile.org] Run 1 complete > [Cold Launch: music.gaiamobile.org] Starting run 2 > [Cold Launch: music.gaiamobile.org] Run 2 complete > [Cold Launch: music.gaiamobile.org] Starting run 3 > [Cold Launch: music.gaiamobile.org] Run 3 complete > [Cold Launch: music.gaiamobile.org] Starting run 4 > [Cold Launch: music.gaiamobile.org] Run 4 complete > [Cold Launch: music.gaiamobile.org] Starting run 5 > [Cold Launch: music.gaiamobile.org] Run 5 complete > [Cold Launch: music.gaiamobile.org] Results from music.gaiamobile.org > > | Metric | Mean | Median | Min | Max | StdDev | > p95 | > | --------------------- | -------- | ------ | ------ | ------ | ------ | > -------- | > | navigationLoaded | 804.200 | 808 | 742 | 858 | 37.151 | > 804.200 | > | navigationInteractive | 826.400 | 827 | 773 | 879 | 33.643 | > 826.400 | > | visuallyLoaded | 1655.600 | 1625 | 1597 | 1788 | 68.002 | > 1655.600 | > | contentInteractive | 1656 | 1625 | 1598 | 1789 | 68.220 | > 1656 | > | fullyLoaded | 1656.600 | 1625 | 1598 | 1790 | 68.482 | > 1656.600 | > | uss | 20.751 | 20.855 | 19.855 | 21.309 | 0.491 | > 20.751 | > | pss | 24.955 | 25.103 | 24.015 | 25.555 | 0.518 | > 24.955 | > | rss | 40.479 | 40.594 | 39.578 | 41.035 | 0.493 | > 40.479 | > > [Cold Launch: music.gaiamobile.org] Testing complete > jdarcangelo-20869:gaia Justin$ > > > I’m actually seeing ~800ms *improvement* with the NGA app in > “fullyLoaded”. Maybe Eli can answer this, but is “fullyLoaded” the number I > should be looking at? I’m also only seeing a ~6MB regression in USS. > All-in-all, this doesn’t seem “that bad” for what is essentially an > unoptimized app. Again, I’m confident that we can improve upon these > numbers significantly in the next 2-3 weeks. > > It also looks like the Gecko-wide perf regression is now gone (I hope). > So, maybe we can start trusting these numbers again. > > -Justin > > > On Oct 2, 2015, at 1:16 PM, Fabrice Desré <[email protected]> wrote: > > On 10/02/2015 09:49 AM, Justin D'Arcangelo wrote: > > I would also like to add that this policy of immediately pouncing on devs > who attempt to try something new that may cause the perf numbers to > momentarily dip is part of why we seem to have a culture problem in FxOS > dev where everyone is afraid to take any kind of risks. If we are not > allowed to have a 2-3 week window to optimize after a huge landing such as > this, then how are we supposed to experiment or take risks? > > > You have all the time you want if you don't put dogfooders at risk. No > one is saying that you should not take the risk to try something new > (side note, you spent enough time on spark & flyweb to know that). But > when it comes to shipping there is a minimum bar to meet, and with > basically a x2 memory usage we are not meeting it in this app yet, > sorry. Feel free to ship a new app alongside the existing one instead > and ask people to try it, since we can't do A/B testing. > > The problem we have is that most people don't care enough about having a > stable nightly, which is why we haven't updated dogfooders for more than > a month now. > > Fabrice > -- > Fabrice Desré > b2g team > Mozilla Corporation > > > > > _______________________________________________ > dev-fxos mailing list > [email protected] > https://lists.mozilla.org/listinfo/dev-fxos > >
_______________________________________________ dev-fxos mailing list [email protected] https://lists.mozilla.org/listinfo/dev-fxos
