Thanks to all those who took part in my Flash speed test a few weeks ago. I'd like to share back the results, but I must caveat that I only got a small amount of data, so these results are not statically significant.
The Purpose ========= I wanted to know how consistent the timing is in Flash across the plethora of web browsers out there in the wild. I had been working on a app that somewhat reliest on consistent timing, hence my curiosity. The Method ========= I wrote a very simple app using the flex framework, that did the following: 1. Set the framerate to x 2. Measure the system time (getTimer()) 3. Add an event listener to Event.ENTER_FRAME 4. on each frame enter, increment a counter, until the counter == x * 10 5. Measure the system time again, and deduct the value from (2) 6. Compute the implied framerate as (x * 10)/ value from (5) 7. Capture OS/Browser setup from UserAgent string, using JS. 8. send data to server (GAE) I did this for x=24 and for x=1000 Known Experimental Deficiencies ========================= a) getTimer may not be accurate b) incrementing a counter once per frame is not representative of the real computational load most apps would place per frame c) individual system performance varies with time, e.g. with the load placed by other apps, with GC, etc d) I don't know enough about the internals on the Flash player, or the browsers, to really know what's going on in there, and what else might make this experiment flawed (unknown unknowns...) e) Some of the data points came from the same PCs (including mine) f) Most tests were done on developers' machines, which are not representative of the average machine g) I didn't test anything between x=24 and x=1000, and there may be speeds between 24 and 1000 that operate very differently. Results ====== Data Set: data points (n) = 134 n By OS - Mac: 28, Win: 100, Linux: 6 n By x - x=1000 FPS: 73, x=24 FPS: 61 Range of results: x=24: min=16.5, max=24.2 x=1000: min=24.3, max=998.9 Results by OS/browser: (the data gets spotty here, so I'll summarize) x=1000 OS = Mac n= 10 results: min = 24.3, max=58.0, mean = 49.6 results: 7 @ 56-58, 2 @ 24-26, 1...@46 x=24 OS = Mac n= 18 results: min = 16.5, max=24.1, mean = 21.5 Safari 4 on Intel: mean=23.9, n=4 Safari 4 on PPC: mean=16.5, n=1 Safari 3 on Intel: mean=19.4, n=3 FF3.0.z on Intel: mean = 20.6, n=5 (other data points are one-offs, so not quoted) x=1000 OS = Win XP / Vista (excluded Win 2000, and 'Windows') n= 56 results: min = 54.6, max=998.9, mean = 586 FF 3.0.8: mean = 930, n=21, min=780, max=998.9 IE 6.z & 7.z & 8.z: mean = 62.6, n=13, min=54.7 max=64.1 Chrome: mean = 187.3, n=5, min=173.5 max = 199.7 x=24 OS = Win XP / Vista (excluded Win 2000, and 'Windows') n= 40 results: min = 20.7, max=24.2, mean =23.1 FF 3.0.8: mean =23.3 , n=17, min=21.6, max=24.2 IE 6.z & 7.z & 8.z: mean =23.1 , n=11, min=20.7 max=24.2 Chrome: mean =24.2 , n=2, min=24.169 max =24.174 x=1000 OS=Linux n=3 mean=80.9 ,min=66.8, max=105 x=24 OS=Linux n=3 mean = 24.0, min=23.8, max=24.1 Discussion ======== None of this data is statistically significant, as there isn't enough of it, but a few patterns emerge nonetheless. To prevent me peppering this section with qualifying words, like 'maybe', 'perhaps', etc I shall present these observations as facts, though they clearly ARE NOT. 1) You can't get higher than 60 FPS on Mac when you request 1000 FPS 2) Requesting 24 FPS on a Mac gives varied results. Safari 4 seems pretty reliable, unless the machine is running PowerPC. FF 3.0.z delivers a mean on 20.6 FPS, which 14% slow of the target 24 FPS 3) Windows is very different to Mac 4) The max FPS on windows is highly dependent on the browser. 5) The actual FPS on windows when FPS 1000 is requested for a specific browser is pretty consistent - FF 3.0.8 is clustered around 900, Chrome is clustered around 180, IE is clustered around 60 6) The actual FPS on Windows when FPS 24 is requested is pretty reliable on all browsers, averaging 23.1 FPS 7) The actual FPS on Windows and Mac when requesting FPS 24 never exceeded 24.2 FPS - i.e. you 'never' get a speed that's too fast. To summarize these point observations: don't rely on actually getting the FPS that you request, and don't rely on what you do get being consistent across browsers and OSs. It appears that FPS 24 is a fairly safe bet (but FPS 40 might be too, for all I know). If you can, write apps that don't require the FPS to be correct for the app to function properly (i.e. dont try to derive time from a count of ENTER_FRAME, either implicitly or explicitly). Test extensively across browsers and OSs, especially cross Windows-Mac. I'd be interested to hear what you all think - what important things have I missed? cheers tom
