On Monday, February 4, 2013 1:39:46 PM UTC-8, Brian Smith wrote: > Also, I want to echo khuey's comment: It seems like a lot of the argument > against PGO is that, while our benchmarks are faster, users won't actually > notice any difference. If that is true, then I agree with khuey that that is > a massive systemic failure; we shouldn't be guiding development based on > benchmarks that don't correlate positively with user-visible improvement. If > all of our benchmarks showing the benefits of PGO are useless and there > really isn't any difference between PGO and non-PGO builds, then I'm not > going to push for us to continue doing PGO builds any more. But, in that case > I hope we also come up with a plan for making better benchmarks. > > And, also, if PGO doesn't have a significant positive performance difference, > I would be very curious as to why not. Is PGO snake oil in general? Is there > something about our codebase that is counter-productive to PGO? And, if the > latter, then is there anything we can do to undo that counter-productivity?
My measurements convinced me that PGO [1] is not useless, even if it's not also incredibly useful. First, benchmarks do matter, even if we think they are not measuring the right thing. Also, benchmarks usually have problems but aren't complete disasters, and optimizing for imperfect benchmarks has yielded real gains. Second, I did find a page that did dynamic rendering that had a 14% higher frame rate with PGO builds. Interestingly, Chrome, which I'm told does not use PGO for the same reasons we find it difficult to use, matches our PGO frame rate, which suggests there is something we could do in the code to make it faster. (With that code, would PGO make it 14% faster yet, or not?) Admittedly, that page isn't practical, but it is some sort of real page. One thing I did notice today is that on a MacBook Air rendering time is nontrivial, so we may be able to observe a CPU usage difference and/or pageload time difference on similar Windows hardware. For the most part, at some point we just need to make our choice about the inevitable tradeoffs, see what happens, and revise if necessary. But I'm personally very happy that the discussion is going on for so long, because it seems like more and more ideas for both alternative paths and for figuring out which way is better coming up. Correct me if I'm wrong, but as I see it, this discussion has turned into a great example of collaborative analysis and idea generation. Dave [1] as applied in our current setup, to our current code base, yada yada yada... _______________________________________________ dev-platform mailing list dev-platform@lists.mozilla.org https://lists.mozilla.org/listinfo/dev-platform
