I would also like to add that this policy of immediately pouncing on devs who attempt to try something new that may cause the perf numbers to momentarily dip is part of why we seem to have a culture problem in FxOS dev where everyone is afraid to take any kind of risks. If we are not allowed to have a 2-3 week window to optimize after a huge landing such as this, then how are we supposed to experiment or take risks?
In a worse-case scenario, we have the old Music app in the dev_apps folder that we can switch back to at a moment’s notice. But, we should be encouraging devs by giving them time to optimize after taking a huge risk. We are shipping to Mozillians (foxfooders) right now, who presumably understand that we are trying to make FxOS better. -Justin > On Oct 2, 2015, at 12:41 PM, Justin D'Arcangelo <[email protected]> > wrote: > > I understand, but the whole point of doing the switch now was to ensure that > there were more people using the app to make sure there were no showstopper > bugs that we weren’t aware of. If we don’t get people using the app for a few > weeks, its possible that some major bugs could slip through. Especially in > the case of an app like Music where everyone possesses a wide variety of > media files. It would be impossible for any of the Music app devs or QA > testers to check every possible media file out there. However, with > dogfooders using the app, its more likely that more types of media files will > get tested. > > -Justin > > >> On Oct 2, 2015, at 12:36 PM, Fabrice Desré <[email protected]> wrote: >> >> Think about people dogfooding. It's barely acceptable to suddenly switch >> to a much worse version of any app, even I your target is some arbitrary >> deadline in the future and you're confident to fix issues. You would not >> do that on a live website right? >> >> On 10/02/2015 09:31 AM, Justin D'Arcangelo wrote: >>> I feel like this is the 3rd or 4th time I’ve had to give this explanation, >>> but at least in the case of Music NGA, we merely landed a completely new, >>> feature-complete app this week. The optimization phase of the code had not >>> yet begun, hence the reason for the increase in the perf numbers. However, >>> prior to landing, I *did* run Raptor every day for the past 2 weeks on >>> Flame. In my Raptor results, Music NGA was coming out ~500ms *faster* than >>> the old app. However, as I noted in the bug, I do not trust the numbers >>> because of the OS-wide perf regression that was causing *both* Music apps >>> to take about 3-4 seconds to launch. >>> >>> This week, the focus has been mainly on identifying and quickly addressing >>> any bugs that came up after the initial testing of the app. I feel that we >>> have things somewhat under control as far as broken functionality goes. >>> Yesterday, we started working on optimizations. There are several areas >>> where we are completely unoptimized at the moment: >>> >>> - album art caching/loading >>> - thumbnail sizes >>> - script loading >>> - view caching >>> >>> All of these items will address the memory usage, startup time or both. So, >>> please do not assume that we spent weeks optimizing the app before landing >>> this week. We merely reached a state of “feature-complete” with a new >>> codebase. We hope to meet or beat the prior app’s numbers before the v2.5 >>> deadline. >>> >>> Thanks! >>> >>> -Justin >>> >>> >>>> On Oct 2, 2015, at 12:01 PM, Fabrice Desré <[email protected]> wrote: >>>> >>>> All the nga apps (music, contacts, sms) show significant regressions. Is >>>> that only a lack of optimizations in these apps, in the bridge they all >>>> use or design flaws in nga itself? >>>> In any case, we have to stop porting new apps to nga until these >>>> questions are answered. >>>> >>>> Fabrice >>>> >>>> On 10/02/2015 07:51 AM, Eli Perelman wrote: >>>>> Hello fxos, >>>>> >>>>> With deadlines for v2.5 approaching, I thought I would take a couple >>>>> minutes and summarize the current state of performance for Gaia. At the >>>>> outset of v2.5 we captured metrics of v2.2 and have used that as the >>>>> baseline to determine whether applications have regressed their >>>>> performance since. Any applications whose performance has significantly >>>>> regressed since v2.2 will need approval to not block as major increases >>>>> will block v2.5. >>>>> >>>>> Enough of the chatter, here's the data: >>>>> >>>>> Calendar v2.2 cold launch: 1454ms >>>>> Calendar current cold launch: 1638ms (~180ms regression) >>>>> Calendar v2.2 USS: 14.01MB >>>>> Calendar current USS: 13.99MB (good) >>>>> >>>>> Camera v2.2 cold launch: 1492ms >>>>> Camera current cold launch: 2090ms (~600ms regression) >>>>> Camera v2.2 USS: 13.83MB >>>>> Camera current USS: 16.05MB (~2.2MB regression) >>>>> >>>>> Clock v2.2 cold launch: 1232ms >>>>> Clock current cold launch: 1260ms (acceptable) >>>>> Clock v2.2 USS: 13.98MB >>>>> Clock current USS: 14.95MB (~1MB regression) >>>>> >>>>> Contacts v2.2 cold launch: 773ms >>>>> Contacts current cold launch: 1246ms (~475ms regression) >>>>> Contacts v2.2 USS: 18.26MB >>>>> Contacts current USS: 20.04MB (~1.75MB regression) >>>>> >>>>> Dialer v2.2 cold launch: 851ms >>>>> Dialer current cold launch: 944ms (~90ms regression, still under 1000ms) >>>>> Dialer v2.2 USS: 17.48MB >>>>> Dialer current USS: 13.04MB (good!) >>>>> >>>>> Email v2.2 cold launch: 2129ms >>>>> Email current cold launch: 606ms (good!) >>>>> Email v2.2 USS: 16.17MB >>>>> Email current USS: 15.78MB (good) >>>>> >>>>> FM v2.2 cold launch: 604ms >>>>> FM current cold launch: 783ms (~175ms regression) >>>>> FM v2.2 USS: 10.37MB >>>>> FM current USS: 10.51MB (acceptable) >>>>> >>>>> Gallery v2.2 cold launch: 1113ms >>>>> Gallery current cold launch: 1207ms (~90ms regression) >>>>> Gallery v2.2 USS: 17.71MB >>>>> Gallery current USS: 18.98MB (~1.25MB regression) >>>>> >>>>> Music v2.2 cold launch: 1066ms >>>>> Music current cold launch: 1717ms (~650ms regression) >>>>> Music v2.2 USS: 13.37MB >>>>> Music current USS: 29.49MB (~16.12MB regression) >>>>> >>>>> SMS v2.2 cold launch: 1340ms >>>>> SMS current cold launch: 1630ms (~290ms regression) >>>>> SMS v2.2 USS: 12.86MB >>>>> SMS current USS: 19.94MB (~7MB regression) >>>>> >>>>> Settings v2.2 cold launch: 2474ms >>>>> Settings current cold launch: 2950ms (~475ms regression) >>>>> Settings v2.2 USS: 17.18MB >>>>> Settings current USS: 17.54MB (acceptable) >>>>> >>>>> Video v2.2 cold launch: 1115ms >>>>> Video current cold launch: 1309ms (~190ms regression) >>>>> Video v2.2 USS: 12.13MB >>>>> Video current USS: 13MB (acceptable) >>>>> >>>>> TLDR; there seem to be quite a few serious regressions across many >>>>> applications, in both cold launch time and USS memory usage. As a >>>>> comparison, the Test Startup Limit app when first captured started off >>>>> in the 880ms range, spent a good chunk of June and July around 620ms and >>>>> is now around 850ms. >>>>> >>>>> If anyone has any questions about the data or needs additional >>>>> information, please let me know. >>>>> >>>>> Also, kudos to the Email team for the massive improvement in both launch >>>>> time and memory. >>>>> >>>>> Thanks, >>>>> >>>>> Eli Perelman >>>>> >>>>> >>>>> _______________________________________________ >>>>> dev-fxos mailing list >>>>> [email protected] >>>>> https://lists.mozilla.org/listinfo/dev-fxos >>>>> >>>> >>>> >>>> -- >>>> Fabrice Desré >>>> b2g team >>>> Mozilla Corporation >>>> _______________________________________________ >>>> dev-fxos mailing list >>>> [email protected] >>>> https://lists.mozilla.org/listinfo/dev-fxos >>> >> >> >> -- >> Fabrice Desré >> b2g team >> Mozilla Corporation > _______________________________________________ dev-fxos mailing list [email protected] https://lists.mozilla.org/listinfo/dev-fxos
