Seems chromium tests are even slower: 10k * mean = 1,634s 50% (5000) of the tests are < 0.016s, assuming they all take that long, that's at most 80seconds. 40% (4000) of the tests are between 0.016 and 0.077, that's at most 311 seconds.
1,634 (total) - 80 - 311 leaves at least 1243 seconds accounted by the last 10% of tests! Assuming you and I both did our math correctly, the bulk of our delay is from a few slow tests, specifically less than 10% of the tests are accounting for 1243s (76%) of the time! -eric On Sat, Aug 1, 2009 at 12:52 PM, Ojan Vafai <o...@chromium.org> wrote: > SUMMARY > I doubt there's much room for improvement in the test harness itself > outside of running tests in parallel. However, there's clearly a lot to be > gained by making individual tests faster. For Chromium runs, the median time > to run a test is ~16ms. It's not apples to apples, but I would be surprised > if there were a significant difference between this and WebKit's > run-webkit-tests. > > With 16ms / test and 10,000 tests, it would take ~2.7 minutes to run the > whole test suite. > > DETAILS > For the Chromium run_webkit_test.py, here's the stats in seconds from a > TestShell run on Windows Debug. FWIW, we spit these stats out for every run. > The eventual goal was to track this in a set of graphs, but I never got > around to implementing that. > > Median: 0.0160000324249 > Mean: 0.163495878646 > 90th percentile: 0.077999830246 > 99th percentile: 3.1099998951 > Standard deviation: 0.519017629449 > > Here's the times for the five slowest top-level directories: > LayoutTests\editing took 43.5 seconds to run 976 tests. > LayoutTests\svg took 53.7 seconds to run 891 tests. > LayoutTests\fast took 378.7 seconds to run 3695 tests. > LayoutTests\http took 396.9 seconds to run 632 tests. > LayoutTests\media took 596.3 seconds to run 84 tests. > > This might be Chromium specific, but I'm seeing many of the the media > layout tests taking 8-10 seconds. > > Ojan > > On Sat, Aug 1, 2009 at 12:49 PM, Eric Seidel <e...@webkit.org> wrote: > >> Yeah, I stared a slowest run before going to bed, and got the following: >> The 10 slowest tests: >> >> 9.58 secs: editing/selection/move-left-right.html >> 9.37 secs: fast/js/array-filter.html >> 7.71 secs: editing/selection/extend-selection.html >> 6.82 secs: fast/js/sort-randomly.html >> 6.25 secs: http/tests/cache/subresource-expiration.html >> 5.77 secs: fast/js/array-enumerators-functions.html >> 5.41 secs: fast/events/click-count.html >> 5.39 secs: fast/js/toString-and-valueOf-override.html >> 5.26 secs: fast/js/try-catch-crash.html >> 5.05 secs: http/tests/navigation/slowmetaredirect-basic.html >> >> that's 10% of our time right there. :) I bet some of those have 5 second >> timers and should be easy to fix. I'll take a look, anyone else who would >> like to help would be most welcome. :) >> >> I remember Ojan finding that running tests in parallel was the biggest >> overall win, but there is definitely a lot of low-hanging fruit in the >> slowest tests too. I'm curious what the fastest tests run in. I would >> expect the harness cost to be near-zero, but I'm not sure. (i.e. what's the >> runtime of run-webkit-tests with 10000 empty test. I would expect only a >> few seconds.) >> >> -eric >> >> On Fri, Jul 31, 2009 at 11:21 PM, Maciej Stachowiak <m...@apple.com>wrote: >> >>> >>> On Jul 31, 2009, at 10:25 PM, Eric Seidel wrote: >>> >>> 681.70s total testing time >>>> >>>> That's 11.5 minutes for every patch I want to land. (because I run the >>>> layout tests before landing anything, as part of bugzilla-tool). >>>> >>>> I'm very interested in any suggestions folks have to make that number >>>> smaller. I know Chromium runs the layout tests in parallel using their >>>> python run_webkit_tests.py harness. Try-bots would be another solution to >>>> the "make sure it passes all the tests before landing". >>>> >>>> Again, very interested in suggestions as to how to improve this. >>>> >>> >>> We're working on try-bots for the ports of particular interest to Apple. >>> But independent of that I think speeding up the layout tests is a worthwhile >>> goal. >>> >>> Besides parallelism for better performance on multi-core systems, here >>> are some other ideas: >>> >>> - Look at the slowest tests and see if they can be made to run faster >>> (I'm running with --slowest as we speak). >>> - Look at the subdirectories that take the most time - it's possible that >>> in some cases we can consolidate multiple tests to make the whole test suite >>> run faster, if sheer number of tests is the bottleneck. >>> >>> I think a good goal to shoot for would be for the full test suite to run >>> in under 5 minutes on a reasonably modern laptop. That should allow for >>> reasonable use during development with some room for growth. >>> >>> Regards, >>> Maciej >>> >>> >> >> _______________________________________________ >> webkit-dev mailing list >> webkit-dev@lists.webkit.org >> http://lists.webkit.org/mailman/listinfo.cgi/webkit-dev >> >> >
_______________________________________________ webkit-dev mailing list webkit-dev@lists.webkit.org http://lists.webkit.org/mailman/listinfo.cgi/webkit-dev
