I agree with the sentiment about allowing tradeoffs. Improving performance in startup time can often mean consuming more memory, and it's hard to improve both. But that's why these numbers are here: for app owners and release decision makers to decide what tradeoffs to make for the best experience while still keeping perceived performance the best they can.
Like I said before, it is possible that there are platform changes that have caused performance regressions in apps along the release, and we have tried to file bugs along the way to ensure these were not known at the last minute. Maybe some confusion has arisen about the implied assumption between what numbers are published and their cause: I have provided launch time metrics for each app as they are executed in our automation against regular builds. Just because there is a regression in launch time for an app does not mean the app in question was at fault. Doing due diligence with bisections, etc. will be the determinate of root cause, but these are the numbers nonetheless. :) Eli Perelman On Mon, Oct 5, 2015 at 3:46 AM, Christopher Lord <[email protected]> wrote: > I'll preface what I say with the hopefully obvious statement that we > should always aim for everything to be better. That said, however, I'd take > a 2mb memory regression and a half-second startup time regression if it > meant the app was polished and performed well. > > Obviously I want both, but it'd be unfair to compare an app that's buggy > and unpolished that has marginally better memory use and/or startup time to > a superior app that provides a more polished experience. Especially if > we're talking sub-second startup and minimal memory use as it is. I think > this partially applies to the music app, which now looks/feels more > polished and snappy than before. > > Have you guys used an Android phone recently? Their startup time for apps > is generally atrocious compared to ours (even on high-end devices) - we > shouldn't drop the ball, but it's not where we compare badly. Given we > aren't targeting 256mb devices anymore, I'd gladly have all our apps use > double the memory they did in 2.2 if it meant we had a consistent 60Hz > update, consistent transitions and snappy response. > > I also agree that it's unfair to be jumping on devs over such small > differences when this is far from a fair comparison (platform changes will > have significant effects on these numbers). > > --Chris > > On Sat, Oct 3, 2015 at 12:32 AM, <[email protected]> wrote: > >> I tried to test my hypothesis using Flame with a build from today (from >> taskcluster, flame-kk-eng) testing test-startup-limit app (no changes) and >> FM app (minimal changes): >> >> Here are my results: https://pastebin.mozilla.org/8848224 >> >> The bottom line is that I did not reproduce your results for Flame, 319mb >> for FM. >> >> I used Gecko from today and tested FM and test-startup-limit from 2.2 >> against 2.5. >> >> So, one should read my read my results as "if we exclude Gecko changes, >> that's what should have happened". >> >> My results, if anything can be significant, indicate: >> >> 1) fullyLoaded improvement in FM on both 319/512 between 2.2 and 2.5 >> 2) visuallyLoaded no change to minimal improvement in FM on both 319/512 >> between 2.2 and 2.5 >> 3) No noticable performance change between 2.2 and 2.5 in >> test-startup-limit >> 4) Potentially a minor memory win in test-startup-limit in 319 scenario >> 5) Significant memory win in FM between 2.2 and 2.5 in both 319 and 512 >> scenarios >> >> Based on that, I would say that we might be seeing Gecko regression >> because your results for FM differ significantly from mine and the only >> change is that you used 2.2 gecko while I used 2.5 gecko. >> >> Where full tests show 175ms regression in visuallyLoaded and 140kb >> regression in memory usage, my results show 140ms improvement in >> visuallyLoaded and 850kb improvement in memory usage. >> >> Also, while your results for 2.5 match mine, your results for FM 2.2 show >> significant difference that can (only?) be attributed to Gecko changes: >> >> Your results (Gecko 2.2 and FM 2.2): >> FM v2.2 cold launch: 604ms >> FM v2.2 USS: 10.37MB >> >> My results (Gecko 2.5 an FM 2.2): >> FM v2.2 cold launch: 782ms (visuallyLoaded, median) >> FM v2.2 USS: 11.31MB (USS, median) >> >> The consistency in mine and your results for 2.5 for FM make me feel a >> bit more confident with the data. The discrepancy in 2.2 results make me >> worried about Gecko impact on the results. >> >> If the same app (FM from 2.2) run on two different geckos (2.2 and 2.5) >> give 180ms difference in results and 1mb difference in memory usage, it >> would indicate that quite possibly many of the regressions we observe come >> from Gecko and not Gaia changes. >> >> zb. >> _______________________________________________ >> dev-fxos mailing list >> [email protected] >> https://lists.mozilla.org/listinfo/dev-fxos >> > > > _______________________________________________ > dev-fxos mailing list > [email protected] > https://lists.mozilla.org/listinfo/dev-fxos > >
_______________________________________________ dev-fxos mailing list [email protected] https://lists.mozilla.org/listinfo/dev-fxos
