On Friday, 23 June 2017 10:05:42 UTC-4, Ehsan Akhgari wrote: > On Fri, Jun 23, 2017 at 4:19 AM, Chris Peterson <cpeter...@mozilla.com> > wrote: > > > > > > > On 6/23/17 12:17 AM, Ehsan Akhgari wrote: > > > > But to speak of a more direct measurement of performance, let's look at > > our progress on Speedometer V2 > > <https://github.com/WebKit/webkit/tree/master/PerformanceTests/Speedometer>.
Ehsan, were you comparing against http://speedometer2.benj.me? Or this http://browserbench.org/Speedometer/? Or using the AWFY code? (It uses local proxy) > > Today, I measured our progress so far on this benchmark by comparing > > Firefox 53, 54, 55.0b3 (latest beta as of this writing) and the latest > > Nightly, all x64 builds, on the reference hardware > > <https://www.amazon.com/dp/B01K1IO3QW>. This is the result (numbers are > > the reported benchmark score, higher is better): > > > > [image: Speedometer improvements] > > > > > > How do these Speedometer V2 scores map to the results on AWFY? AWFY shows > > many Speedometer sub-tests, but no score in the range of 70.21. AWFY > > machine #36 is the reference hardware. > > > > https://arewefastyet.com/#machine=36&view=breakdown&suite=speedometer-misc > > > > Armen has been investigating the difference. The Speedometer benchmark is > extremely sensitive to anything else that is going on on the machine at the > time you are running the tests, see for example > https://bugzilla.mozilla.org/show_bug.cgi?id=1373396#c3 where he discovered > that turning off the "Shut off display after 15 minutes" setting improves > our benchmark score by about 10 points or so! I think there are other > investigations ongoing to dig into the remaining difference as well. > Currently, machine #36 is testing against non-PGO builds from inbound (to catch regressions). I'm looking into adding mozilla-central PGO builds to the mix. I assume PGO builds can run slightly faster than non-PGO builds. Once we have PGO builds we will be able to see if there are anymore machine configration changes required (I doubt it). Direct link to the speedometer score on machine #36: https://arewefastyet.com/#machine=36&view=single&suite=speedometer-misc&subtest=score > (Also note that there is a speed difference on Speedometer between Nightly > and Beta, where Nightly with the same code will be a bit slower than Beta > since some Nightly specific features do show up in Speedometer profiles > currently, for example things like bug 1375568 are currently Nightly only, > and there are also debugging checks like < > https://searchfox.org/mozilla-central/rev/3291398f10dcbe192fb52e74974b172616c018aa/ipc/chromium/src/base/pickle.h#26> > that show up a bit in profiles as well. I think that explains why 55.0b3 > is scoring so high comparing to 56.0a1 there.) > > Cheers, > -- > Ehsan _______________________________________________ dev-platform mailing list dev-platform@lists.mozilla.org https://lists.mozilla.org/listinfo/dev-platform
