Gecko has a pretty substantial number of metrics used for measuring startup performance.
We have probes in Telemetry [0], StartupTimeline [1], various uses of MozAfterPaint, both in talos tests [2] and in production code [3][4]. We also have first paint in compositor [5], before-first-paint [6], timeToNonBlankPaint [7] which should not be confused with firstNonBlankPaint [8] and probably a number of other probes that register different timestamps and are used all around to mean different things. Some measure layout, others composition. Many of them are misused by capturing the timestamp in the callback to event listener fired asynchronously post-event. We end up seeing huge swings in some "first-paint" [9] that are not reproducible in another "first-paint" [10], and we know that things like WebRender may not affect some of our "first-paint" because it measures only part of paint that WebRender doesn't affect[11]. It doesn't help that some of them are chrome-only while others are available in content. I believe that, while we can recognize the complexity of the systems in play and how this complexity explains why different probes ended up being constituted, this situation is counter productive to our ability to understand the performance implication of our changes in product. I'd like to suggest establishing a single set of timestamps with unique names to represent various stages of the product launch. Those timestamps would be described based on the user-perceived results and as such serve us as best-effort approximations of the impact of any change on the user-perceived experience. In particular, my proposal is based on WICG Paint Timing proposal [12] and establishes the 5 major timestamps to be executed at the latest event that contributes to the user-perceived outcome. For example, when selecting when to mark "xPaint" event, we will use the consumer notion of the term "paint" and mark it after *all* operations required for the paint to happen are done - layout, composition, rendering and paint. My proposal is also based on the work on negotiated performance milestones established for Firefox OS project [13]. The proposed milestones are: 1) firstPaint This milestone happens when the first paint that is influenced by the data for the measured object is completed by the engine (and likely submitted to the graphic driver). In the context of an HTML document, the first paint that is affected by the document's background or title is completed. 2) firstContentfulPaint The first contentful paint of browser.xul happens when the first paint that includes layout of DOM data from browser.xul is completed. 3) visuallyCompletedPaint This milestones is achieved after the first paint with the above-the-fold part of the DOM ready is submitted. This event may require the document to inform the engine that all the items in the visual field are ready, and the next paint captures the timestamp. 4) chromeInteractive This milestone is achieved when the app reports the UI to be ready to be interacted with. The definition of what constitutes of the UI being ready is up to the app, and may just include the URL bar being ready to receive URL input, or may wait for all core parts of the product to have event handlers attached (urlbar, tabbar, main menu etc.) This milestone may be reached before (3), but not after (5). 5) fullyLoaded This milestone also may require data from the document and it should mark the timestamp when all startup operations are completed. This should include delayed startup operations that may have waited for previous stages to be achieved, but should not wait for non-startup delayed operations like periodic updates of data from AMO etc. The last milestone is a good moment to reliably measure memory consumption (possibly after performing GC), take a screenshot for any tests that compare UI between starts and so on. Generally speaking, (5) is when we naively can say "the app is fully launched". ================== This system would require us to provide a way for each document to inform the engine when some of the later stages are reached. Then the engine would take the next full paint and capture the timestamp. Such timestamp list would be available for chrome and behind a flag for content, to be used by tests. The value of such milestone approach lies not only in unification of reading, but also easier hooking of code into the bootstrap process. Having such a consistent multi-stage bootstrap allows developers to decide at which stage their code has to be executed to delay only what has to be affected by it and in result developing a culture of adding code that doesn't affect early stages of the bootstrap unnecessarily. Lastly of course, the value comes in our ability to say that all telemetry probes, talos tests, tp6 tests etc. and automation can now rely on a single set of timestamps which would increase developers ability to understand and verify how their patches affect the user perceived experience. I do not expect us to place all of them at once, but I believe that there's a value in having a conversation and establishing the list of marks we want to unify around and then start moving toward them. In the end, the bootstrap timestamps of browser.xul should be read in a very similar fashion to how we read bootstrap timestamps of amazon.com with the difference being that later stages like "visuallyLoaded" or "interactive" may be triggered by browser.xul's JS code, while for amazon, we'll either not capture them, or use some custom code in testing machinery to decide when the hero elements are interactive. zb. [0] http://searchfox.org/mozilla-central/source/toolkit/components/startup/StartupTimeline.cpp#51 [1] http://searchfox.org/mozilla-central/source/toolkit/components/startup/StartupTimeline.h#15 [2] http://searchfox.org/mozilla-central/source/testing/talos/talos/tests/tpaint/file_tpaint_win.html#4 [3] http://searchfox.org/mozilla-central/source/browser/base/content/browser.js#1388 [4] http://searchfox.org/mozilla-central/source/browser/base/content/tabbrowser.xml#1124 [5] http://searchfox.org/mozilla-central/source/gfx/layers/ipc/UiCompositorControllerMessageTypes.h#24 [6] http://searchfox.org/mozilla-central/source/dom/ipc/TabChild.cpp#2847 [7] http://searchfox.org/mozilla-central/source/dom/performance/PerformanceTiming.h#249 [8] http://searchfox.org/mozilla-central/source/layout/base/nsPresContext.h#1430 [9] https://telemetry.mozilla.org/new-pipeline/evo.html#!aggregates=75th-percentile!95th-percentile&cumulative=0&end_date=null&keys=&max_channel_version=nightly%252F57&measure=SIMPLE_MEASURES_FIRSTPAINT&min_channel_version=nightly%252F54&processType=*&product=Firefox&sanitize=1&sort_keys=submissions&start_date=null&trim=1&use_submission_date=0 [10] https://treeherder.mozilla.org/perf.html#/graphs?series=mozilla-central,1517547,1,1&series=mozilla-central,1535589,1,1&series=mozilla-central,1535590,1,1&series=mozilla-central,1535587,1,1&series=mozilla-central,1535588,1,1 [11] https://bugzilla.mozilla.org/show_bug.cgi?id=1388157#c15 [12] https://w3c.github.io/paint-timing/ [13] https://developer.mozilla.org/en-US/docs/Archive/B2G_OS/Developing_Gaia/Raptor/Responsiveness_guidelines _______________________________________________ dev-platform mailing list dev-platform@lists.mozilla.org https://lists.mozilla.org/listinfo/dev-platform
