There definitely could be a regression in Gecko. As Justin pointed out, we had a pretty serious regression for the past few weeks that was only resolved today, but my results were gathered after the regression resolution.
For Flame performance automation, we test every application against each build we get from PVT. Each application is tested 30 times, and the metrics for every run are stored in a database which is where the dashboards get their data. That also means we can re-query to slice the data in different ways. The data I provided today represents the 95th percentile of visuallyLoaded and the mean of USS. We typically don't use mean for launch times as they are not informative of the standard deviation of results like a percentile value can. Regardless, here is the current standard deviation of launch times for said apps: Calendar: 49ms Camera: 157ms Clock: 31ms Contacts: 224ms Dialer: 22ms Email: 152ms FM: 33ms Gallery: 52ms Music: 47ms SMS: 44ms Settings: 44ms Video: 41ms Test Startup Limit: 116ms Now, I'm not a statistician in the least, but standard deviation representing the "normal distribution", most apps exhibit a stable launch time within about 40-50ms. The 95th percentile gives us a metric that users can care about: "95% of the time my app loads faster than X milliseconds". Interestingly enough, the Test Startup Limit app which we use to determine a general launch baseline, exhibits swings of 116ms in its normal distribution which is concerning, even though it hasn't really regressed its performance. Gareth wrote that app and maybe can give us some insight into what it does that other apps don't that may cause instability in launch time but over consistently achieving its goal. I'm happy to help provide whatever insight into this that is needed. :) Thanks, Eli Perelman On Fri, Oct 2, 2015 at 3:46 PM, <[email protected]> wrote: > And another calm app - Video. I landed an L10n/Intl refactor this morning > with good perf results [0], but the tests were likely done before. > > If you look at Video changelog [1] in the recent months, there's really > nothing between November 2014 and my change today. > > So, where does 200ms and 900kb regression come from? (side note - 900kb > should not be acceptable unless significant chunk of new data has been > added to startup path). > > I believe that those three apps - Video, FM and Clock should see > performance/memory between 2.2 and 2.5 improvement. Out of them, Clock is > the hardest to "revert", but FM and Video should be fairly easy to test the > same gaia between two geckos and two gaias against the same gecko. > > And I believe we should do this before we jump to conclusions because the > perf impact on those apps seems to be bigger than for some other apps and > if it's all noise it's a red herring. And if it's Gecko impact, we should > fix it in Gecko. > > zb. > > > [0] https://bugzilla.mozilla.org/show_bug.cgi?id=1197454#c6 > [1] > https://github.com/mozilla-b2g/gaia/commits/master/apps/video/js/video.js > _______________________________________________ > dev-fxos mailing list > [email protected] > https://lists.mozilla.org/listinfo/dev-fxos >
_______________________________________________ dev-fxos mailing list [email protected] https://lists.mozilla.org/listinfo/dev-fxos
