TL;DR - Firefox does pretty well when compared to Chrome. The Presto project is a Mozilla platform initiative that intends to look into any performance differences between Firefox and other UserAgents in order to highlight areas that we should look into improving and to clear any prejudice that may be caused by FUD or past performance differences that are no longer true.
We've begun this project by doing a set of performance runs on WebPageTest.com in order to compare Firefox's performance against Chrome on the home pages of the Alexa 100 (the top 100 web sites worldwide). We've made a visualization tool which plots all of the runs on a graph, so we may easily compare them: http://valenting.github.io/presto-plot/plot.html - click on a domain link to see results for each site. - you are able to view the results sorted or unsorted, or filter by the browser version - you can also compare metrics other than load time - such as speed Index, number of DNS queries, etc - you can also compare the browsers on several connectivity profiles (Cable, 3G, Dialup) - You can click on each individual point to go to the WPT run and view the results in greater detail --------------- Initial results: The results consistently show that Firefox is faster than Chrome for both of our major metrics (load time and speedIndex). On a majority of domains Firefox is faster on both first loads and reloads. When we analyze 3G connectivity runs, Firefox's speedup less substantial, and we encounter additional domains where Chrome has an edge. Dial up connectivity is a bit more difficult to analyze. Because of the large size of most websites, browsers are unable to load the website within 5 minutes, or it might time out. Chrome seems to have more successful data points, and faster loads on dial up, but it is difficult to reach a conclusion based on a limited data set. WPT also has Nightly builds - which default to e10s ON. The several data points we have show similar performance to non-e10s builds (which is good, considering Nightly builds have more checks and assertions) ----------------------- Interesting metrics: Load time - is measured as the time from the start of the initial navigation until the beginning of the window load event (onload). Speed Index - is the metric recommended PageSpeedTest.com. Speed index measures how fast a page is displayed on screen. Details: https://goo.gl/7ha6eE Activity time - measured as the time from the start of the initial navigation until there was 2 seconds of no network activity after Document Complete. This will usually include any activity that is triggered by javascript after the main page loads. For most metrics a lower score is better (faster). ------------------ Error sources: Websites may return different content depending on the UA string. While this optimization makes sense for a lot of websites, in this situation it is difficult to determine if the browser's performance or the website's optimizations have more impact on the page load. Since we do not log onto any sites, the data here for sites which rely on logins to display full data (Facebook, etc) are not representative of most users' experience and will generally have a different (much more heavily JS and XHR-based) profile in reality. For several data sets we may observe a certain spike in the data points – runs that take 3x-7x times longer than usual. This may be due to a number of reasons: a higher load on the network in the data center, a higher load on the VMs on which the browsers are running, or the fact that websites serve variable content – different images, different ads. It may be that some of the page loads don't load the website completely, or encounter errors. WPT also saves a screenshot of the page, along with data on all of the resources it loads, so we may look at really fast loads to make sure they are complete. WPT has a maximum load time of 5 minutes. Any page load that takes longer than that will be recorded as a 0 – so while it may seem to load faster, that is not the case. Other errors may also be recorded as a 0. WPT only loads one tab page at a time, whereas the load time on a user's may be affected by his activity in other tabs. e10s certainly helps in this area. We only tested the performance of the landing page. It would be possible to simulate user activity and navigation once the landing page is loaded, but for this we'd need to write separate scripts for each domain we test. Location may be an issue, as the RTT to the server or the CDN may vary. All of the tests we ran were made on the WPT datacenter in Dulles VA. ------------------ The stats are merely informative. All of the data is is publicly available to anyone who has an inclination towards statistics. Setup: webpagetest.meteor.com - an api that returns a list of IDs in JSON form, representing all the WPT tests run for a given domain (source and API: https://github.com/valenting/webpagetest ) plot.html - when you click a domain it gets all of the IDs for tests run against that domains, then parses the results and plots the results. (source: https://github.com/valenting/presto-plot ) _______________________________________________ dev-platform mailing list dev-platform@lists.mozilla.org https://lists.mozilla.org/listinfo/dev-platform