Here’s the latest Raptor numbers, run locally, on Flame: OGA:
jdarcangelo-20869:gaia Justin$ raptor test coldlaunch --app music-oga --runs 5 [Cold Launch: music-oga.gaiamobile.org] Preparing to start testing... [Cold Launch: music-oga.gaiamobile.org] Priming application [Cold Launch: music-oga.gaiamobile.org] Starting run 1 [Cold Launch: music-oga.gaiamobile.org] Run 1 complete [Cold Launch: music-oga.gaiamobile.org] Starting run 2 [Cold Launch: music-oga.gaiamobile.org] Run 2 complete [Cold Launch: music-oga.gaiamobile.org] Starting run 3 [Cold Launch: music-oga.gaiamobile.org] Run 3 complete [Cold Launch: music-oga.gaiamobile.org] Starting run 4 [Cold Launch: music-oga.gaiamobile.org] Run 4 complete [Cold Launch: music-oga.gaiamobile.org] Starting run 5 [Cold Launch: music-oga.gaiamobile.org] Run 5 complete [Cold Launch: music-oga.gaiamobile.org] Results from music-oga.gaiamobile.org | Metric | Mean | Median | Min | Max | StdDev | p95 | | --------------------- | -------- | ------ | ------ | ------ | ------ | -------- | | navigationLoaded | 737 | 739 | 715 | 752 | 14.071 | 737 | | navigationInteractive | 900.600 | 893 | 880 | 924 | 16.305 | 900.600 | | visuallyLoaded | 1123 | 1121 | 1076 | 1170 | 32.323 | 1123 | | contentInteractive | 1123.400 | 1121 | 1077 | 1171 | 32.327 | 1123.400 | | fullyLoaded | 2478.400 | 2474 | 2450 | 2517 | 23.466 | 2478.400 | | uss | 14.542 | 14.645 | 14.301 | 14.738 | 0.182 | 14.542 | | rss | 34.377 | 34.480 | 34.133 | 34.574 | 0.185 | 34.377 | | pss | 18.775 | 18.877 | 18.534 | 18.978 | 0.186 | 18.775 | [Cold Launch: music-oga.gaiamobile.org] Testing complete jdarcangelo-20869:gaia Justin$ NGA: jdarcangelo-20869:gaia Justin$ raptor test coldlaunch --app music --runs 5 [Cold Launch: music.gaiamobile.org] Preparing to start testing... [Cold Launch: music.gaiamobile.org] Priming application [Cold Launch: music.gaiamobile.org] Starting run 1 [Cold Launch: music.gaiamobile.org] Run 1 complete [Cold Launch: music.gaiamobile.org] Starting run 2 [Cold Launch: music.gaiamobile.org] Run 2 complete [Cold Launch: music.gaiamobile.org] Starting run 3 [Cold Launch: music.gaiamobile.org] Run 3 complete [Cold Launch: music.gaiamobile.org] Starting run 4 [Cold Launch: music.gaiamobile.org] Run 4 complete [Cold Launch: music.gaiamobile.org] Starting run 5 [Cold Launch: music.gaiamobile.org] Run 5 complete [Cold Launch: music.gaiamobile.org] Results from music.gaiamobile.org | Metric | Mean | Median | Min | Max | StdDev | p95 | | --------------------- | -------- | ------ | ------ | ------ | ------ | -------- | | navigationLoaded | 804.200 | 808 | 742 | 858 | 37.151 | 804.200 | | navigationInteractive | 826.400 | 827 | 773 | 879 | 33.643 | 826.400 | | visuallyLoaded | 1655.600 | 1625 | 1597 | 1788 | 68.002 | 1655.600 | | contentInteractive | 1656 | 1625 | 1598 | 1789 | 68.220 | 1656 | | fullyLoaded | 1656.600 | 1625 | 1598 | 1790 | 68.482 | 1656.600 | | uss | 20.751 | 20.855 | 19.855 | 21.309 | 0.491 | 20.751 | | pss | 24.955 | 25.103 | 24.015 | 25.555 | 0.518 | 24.955 | | rss | 40.479 | 40.594 | 39.578 | 41.035 | 0.493 | 40.479 | [Cold Launch: music.gaiamobile.org] Testing complete jdarcangelo-20869:gaia Justin$ I’m actually seeing ~800ms *improvement* with the NGA app in “fullyLoaded”. Maybe Eli can answer this, but is “fullyLoaded” the number I should be looking at? I’m also only seeing a ~6MB regression in USS. All-in-all, this doesn’t seem “that bad” for what is essentially an unoptimized app. Again, I’m confident that we can improve upon these numbers significantly in the next 2-3 weeks. It also looks like the Gecko-wide perf regression is now gone (I hope). So, maybe we can start trusting these numbers again. -Justin > On Oct 2, 2015, at 1:16 PM, Fabrice Desré <[email protected]> wrote: > > On 10/02/2015 09:49 AM, Justin D'Arcangelo wrote: >> I would also like to add that this policy of immediately pouncing on devs >> who attempt to try something new that may cause the perf numbers to >> momentarily dip is part of why we seem to have a culture problem in FxOS dev >> where everyone is afraid to take any kind of risks. If we are not allowed to >> have a 2-3 week window to optimize after a huge landing such as this, then >> how are we supposed to experiment or take risks? > > You have all the time you want if you don't put dogfooders at risk. No > one is saying that you should not take the risk to try something new > (side note, you spent enough time on spark & flyweb to know that). But > when it comes to shipping there is a minimum bar to meet, and with > basically a x2 memory usage we are not meeting it in this app yet, > sorry. Feel free to ship a new app alongside the existing one instead > and ask people to try it, since we can't do A/B testing. > > The problem we have is that most people don't care enough about having a > stable nightly, which is why we haven't updated dogfooders for more than > a month now. > > Fabrice > -- > Fabrice Desré > b2g team > Mozilla Corporation
_______________________________________________ dev-fxos mailing list [email protected] https://lists.mozilla.org/listinfo/dev-fxos
