On Sat, Aug 1, 2009 at 4:23 PM, Eric Seidel <e...@webkit.org> wrote: > Seems chromium tests are even slower: 10k * mean = 1,634s >
For completeness sake, the Chromium numbers include the pixel tests. > 50% (5000) of the tests are < 0.016s, assuming they all take that long, > that's at most 80seconds. > 40% (4000) of the tests are between 0.016 and 0.077, that's at most 311 > seconds. > > 1,634 (total) - 80 - 311 leaves at least 1243 seconds accounted by the last > 10% of tests! > > Assuming you and I both did our math correctly, the bulk of our delay is > from a few slow tests, specifically less > than 10% of the tests are accounting for 1243s (76%) of the time! > > -eric > > On Sat, Aug 1, 2009 at 12:52 PM, Ojan Vafai <o...@chromium.org> wrote: > >> SUMMARY >> I doubt there's much room for improvement in the test harness itself >> outside of running tests in parallel. However, there's clearly a lot to be >> gained by making individual tests faster. For Chromium runs, the median time >> to run a test is ~16ms. It's not apples to apples, but I would be surprised >> if there were a significant difference between this and WebKit's >> run-webkit-tests. >> >> With 16ms / test and 10,000 tests, it would take ~2.7 minutes to run the >> whole test suite. >> >> DETAILS >> For the Chromium run_webkit_test.py, here's the stats in seconds from a >> TestShell run on Windows Debug. FWIW, we spit these stats out for every run. >> The eventual goal was to track this in a set of graphs, but I never got >> around to implementing that. >> >> Median: 0.0160000324249 >> Mean: 0.163495878646 >> 90th percentile: 0.077999830246 >> 99th percentile: 3.1099998951 >> Standard deviation: 0.519017629449 >> >> Here's the times for the five slowest top-level directories: >> LayoutTests\editing took 43.5 seconds to run 976 tests. >> LayoutTests\svg took 53.7 seconds to run 891 tests. >> LayoutTests\fast took 378.7 seconds to run 3695 tests. >> LayoutTests\http took 396.9 seconds to run 632 tests. >> LayoutTests\media took 596.3 seconds to run 84 tests. >> >> This might be Chromium specific, but I'm seeing many of the the media >> layout tests taking 8-10 seconds. >> >> Ojan >> >> On Sat, Aug 1, 2009 at 12:49 PM, Eric Seidel <e...@webkit.org> wrote: >> >>> Yeah, I stared a slowest run before going to bed, and got the following: >>> The 10 slowest tests: >>> >>> 9.58 secs: editing/selection/move-left-right.html >>> 9.37 secs: fast/js/array-filter.html >>> 7.71 secs: editing/selection/extend-selection.html >>> 6.82 secs: fast/js/sort-randomly.html >>> 6.25 secs: http/tests/cache/subresource-expiration.html >>> 5.77 secs: fast/js/array-enumerators-functions.html >>> 5.41 secs: fast/events/click-count.html >>> 5.39 secs: fast/js/toString-and-valueOf-override.html >>> 5.26 secs: fast/js/try-catch-crash.html >>> 5.05 secs: http/tests/navigation/slowmetaredirect-basic.html >>> >>> that's 10% of our time right there. :) I bet some of those have 5 second >>> timers and should be easy to fix. I'll take a look, anyone else who would >>> like to help would be most welcome. :) >>> >>> I remember Ojan finding that running tests in parallel was the biggest >>> overall win, but there is definitely a lot of low-hanging fruit in the >>> slowest tests too. I'm curious what the fastest tests run in. I would >>> expect the harness cost to be near-zero, but I'm not sure. (i.e. what's the >>> runtime of run-webkit-tests with 10000 empty test. I would expect only a >>> few seconds.) >>> >>> -eric >>> >>> On Fri, Jul 31, 2009 at 11:21 PM, Maciej Stachowiak <m...@apple.com>wrote: >>> >>>> >>>> On Jul 31, 2009, at 10:25 PM, Eric Seidel wrote: >>>> >>>> 681.70s total testing time >>>>> >>>>> That's 11.5 minutes for every patch I want to land. (because I run the >>>>> layout tests before landing anything, as part of bugzilla-tool). >>>>> >>>>> I'm very interested in any suggestions folks have to make that number >>>>> smaller. I know Chromium runs the layout tests in parallel using their >>>>> python run_webkit_tests.py harness. Try-bots would be another solution to >>>>> the "make sure it passes all the tests before landing". >>>>> >>>>> Again, very interested in suggestions as to how to improve this. >>>>> >>>> >>>> We're working on try-bots for the ports of particular interest to Apple. >>>> But independent of that I think speeding up the layout tests is a >>>> worthwhile >>>> goal. >>>> >>>> Besides parallelism for better performance on multi-core systems, here >>>> are some other ideas: >>>> >>>> - Look at the slowest tests and see if they can be made to run faster >>>> (I'm running with --slowest as we speak). >>>> - Look at the subdirectories that take the most time - it's possible >>>> that in some cases we can consolidate multiple tests to make the whole test >>>> suite run faster, if sheer number of tests is the bottleneck. >>>> >>>> I think a good goal to shoot for would be for the full test suite to run >>>> in under 5 minutes on a reasonably modern laptop. That should allow for >>>> reasonable use during development with some room for growth. >>>> >>>> Regards, >>>> Maciej >>>> >>>> >>> >>> _______________________________________________ >>> webkit-dev mailing list >>> webkit-dev@lists.webkit.org >>> http://lists.webkit.org/mailman/listinfo.cgi/webkit-dev >>> >>> >> >
_______________________________________________ webkit-dev mailing list webkit-dev@lists.webkit.org http://lists.webkit.org/mailman/listinfo.cgi/webkit-dev
