Correction, my bad. Eli just informed me that his numbers earlier were from the 319mb Flame configuration. I was under the impression that we were no longer supporting that config. In addition, I was looking at “fullyLoaded” and not “visuallyLoaded”.
We have a patch on the way that should be optimizing the way we fetch album art and should drastically cut down on memory usage. Once that has landed, I will re-test under 319mb and see where we stand. -Justin > On Oct 2, 2015, at 1:51 PM, Justin D'Arcangelo <[email protected]> > wrote: > > Here’s the latest Raptor numbers, run locally, on Flame: > > OGA: > > jdarcangelo-20869:gaia Justin$ raptor test coldlaunch --app music-oga --runs 5 > [Cold Launch: music-oga.gaiamobile.org <http://music-oga.gaiamobile.org/>] > Preparing to start testing... > [Cold Launch: music-oga.gaiamobile.org <http://music-oga.gaiamobile.org/>] > Priming application > [Cold Launch: music-oga.gaiamobile.org <http://music-oga.gaiamobile.org/>] > Starting run 1 > [Cold Launch: music-oga.gaiamobile.org <http://music-oga.gaiamobile.org/>] > Run 1 complete > [Cold Launch: music-oga.gaiamobile.org <http://music-oga.gaiamobile.org/>] > Starting run 2 > [Cold Launch: music-oga.gaiamobile.org <http://music-oga.gaiamobile.org/>] > Run 2 complete > [Cold Launch: music-oga.gaiamobile.org <http://music-oga.gaiamobile.org/>] > Starting run 3 > [Cold Launch: music-oga.gaiamobile.org <http://music-oga.gaiamobile.org/>] > Run 3 complete > [Cold Launch: music-oga.gaiamobile.org <http://music-oga.gaiamobile.org/>] > Starting run 4 > [Cold Launch: music-oga.gaiamobile.org <http://music-oga.gaiamobile.org/>] > Run 4 complete > [Cold Launch: music-oga.gaiamobile.org <http://music-oga.gaiamobile.org/>] > Starting run 5 > [Cold Launch: music-oga.gaiamobile.org <http://music-oga.gaiamobile.org/>] > Run 5 complete > [Cold Launch: music-oga.gaiamobile.org <http://music-oga.gaiamobile.org/>] > Results from music-oga.gaiamobile.org <http://music-oga.gaiamobile.org/> > > | Metric | Mean | Median | Min | Max | StdDev | p95 > | > | --------------------- | -------- | ------ | ------ | ------ | ------ | > -------- | > | navigationLoaded | 737 | 739 | 715 | 752 | 14.071 | 737 > | > | navigationInteractive | 900.600 | 893 | 880 | 924 | 16.305 | > 900.600 | > | visuallyLoaded | 1123 | 1121 | 1076 | 1170 | 32.323 | 1123 > | > | contentInteractive | 1123.400 | 1121 | 1077 | 1171 | 32.327 | > 1123.400 | > | fullyLoaded | 2478.400 | 2474 | 2450 | 2517 | 23.466 | > 2478.400 | > | uss | 14.542 | 14.645 | 14.301 | 14.738 | 0.182 | > 14.542 | > | rss | 34.377 | 34.480 | 34.133 | 34.574 | 0.185 | > 34.377 | > | pss | 18.775 | 18.877 | 18.534 | 18.978 | 0.186 | > 18.775 | > > [Cold Launch: music-oga.gaiamobile.org <http://music-oga.gaiamobile.org/>] > Testing complete > jdarcangelo-20869:gaia Justin$ > > > NGA: > > jdarcangelo-20869:gaia Justin$ raptor test coldlaunch --app music --runs 5 > [Cold Launch: music.gaiamobile.org <http://music.gaiamobile.org/>] Preparing > to start testing... > [Cold Launch: music.gaiamobile.org <http://music.gaiamobile.org/>] Priming > application > [Cold Launch: music.gaiamobile.org <http://music.gaiamobile.org/>] Starting > run 1 > [Cold Launch: music.gaiamobile.org <http://music.gaiamobile.org/>] Run 1 > complete > [Cold Launch: music.gaiamobile.org <http://music.gaiamobile.org/>] Starting > run 2 > [Cold Launch: music.gaiamobile.org <http://music.gaiamobile.org/>] Run 2 > complete > [Cold Launch: music.gaiamobile.org <http://music.gaiamobile.org/>] Starting > run 3 > [Cold Launch: music.gaiamobile.org <http://music.gaiamobile.org/>] Run 3 > complete > [Cold Launch: music.gaiamobile.org <http://music.gaiamobile.org/>] Starting > run 4 > [Cold Launch: music.gaiamobile.org <http://music.gaiamobile.org/>] Run 4 > complete > [Cold Launch: music.gaiamobile.org <http://music.gaiamobile.org/>] Starting > run 5 > [Cold Launch: music.gaiamobile.org <http://music.gaiamobile.org/>] Run 5 > complete > [Cold Launch: music.gaiamobile.org <http://music.gaiamobile.org/>] Results > from music.gaiamobile.org <http://music.gaiamobile.org/> > > | Metric | Mean | Median | Min | Max | StdDev | p95 > | > | --------------------- | -------- | ------ | ------ | ------ | ------ | > -------- | > | navigationLoaded | 804.200 | 808 | 742 | 858 | 37.151 | > 804.200 | > | navigationInteractive | 826.400 | 827 | 773 | 879 | 33.643 | > 826.400 | > | visuallyLoaded | 1655.600 | 1625 | 1597 | 1788 | 68.002 | > 1655.600 | > | contentInteractive | 1656 | 1625 | 1598 | 1789 | 68.220 | 1656 > | > | fullyLoaded | 1656.600 | 1625 | 1598 | 1790 | 68.482 | > 1656.600 | > | uss | 20.751 | 20.855 | 19.855 | 21.309 | 0.491 | > 20.751 | > | pss | 24.955 | 25.103 | 24.015 | 25.555 | 0.518 | > 24.955 | > | rss | 40.479 | 40.594 | 39.578 | 41.035 | 0.493 | > 40.479 | > > [Cold Launch: music.gaiamobile.org <http://music.gaiamobile.org/>] Testing > complete > jdarcangelo-20869:gaia Justin$ > > > I’m actually seeing ~800ms *improvement* with the NGA app in “fullyLoaded”. > Maybe Eli can answer this, but is “fullyLoaded” the number I should be > looking at? I’m also only seeing a ~6MB regression in USS. All-in-all, this > doesn’t seem “that bad” for what is essentially an unoptimized app. Again, > I’m confident that we can improve upon these numbers significantly in the > next 2-3 weeks. > > It also looks like the Gecko-wide perf regression is now gone (I hope). So, > maybe we can start trusting these numbers again. > > -Justin > > >> On Oct 2, 2015, at 1:16 PM, Fabrice Desré <[email protected] >> <mailto:[email protected]>> wrote: >> >> On 10/02/2015 09:49 AM, Justin D'Arcangelo wrote: >>> I would also like to add that this policy of immediately pouncing on devs >>> who attempt to try something new that may cause the perf numbers to >>> momentarily dip is part of why we seem to have a culture problem in FxOS >>> dev where everyone is afraid to take any kind of risks. If we are not >>> allowed to have a 2-3 week window to optimize after a huge landing such as >>> this, then how are we supposed to experiment or take risks? >> >> You have all the time you want if you don't put dogfooders at risk. No >> one is saying that you should not take the risk to try something new >> (side note, you spent enough time on spark & flyweb to know that). But >> when it comes to shipping there is a minimum bar to meet, and with >> basically a x2 memory usage we are not meeting it in this app yet, >> sorry. Feel free to ship a new app alongside the existing one instead >> and ask people to try it, since we can't do A/B testing. >> >> The problem we have is that most people don't care enough about having a >> stable nightly, which is why we haven't updated dogfooders for more than >> a month now. >> >> Fabrice >> -- >> Fabrice Desré >> b2g team >> Mozilla Corporation >
_______________________________________________ dev-fxos mailing list [email protected] https://lists.mozilla.org/listinfo/dev-fxos
