On 6 February 2013 07:17, Dirk Pranke <[email protected]> wrote: > On Tue, Feb 5, 2013 at 9:46 AM, Martin Robinson <[email protected]> > wrote: > > On Tue, Feb 5, 2013 at 9:28 AM, Adam Barth <[email protected]> wrote: > >> Do you know how they got rid of flakiness in their tests? We've spent > >> a bunch of effort fixing flaky tests (and in marking the remaining > >> flaky tests as flaky), but there's still a long tail of flakiness. I > >> wonder if that sort of thing might be different for OpenStack if they > >> have a different approach to testing than we do. >
>From what I can see they have a pretty similar goal to us. I personally don't know where our test flakyness comes from, so can't really comment on how we could fix it. > > > > Another useful thing is to know the number of tests in OpenStack. > > WebKit has more tests than any other project I've worked on. > > > > There are two other related aspects that make our tests flaky: > > 1) They're very high level integration tests (mostly), which, as they > cover large swaths of code in each test, are much more susceptible to > flakiness than method-level unit tests. > While OpenStack doesn't have anywhere near the number of integration tests WebKit does, it does have large integration tests. Infact, one of their tests brings up a whole cloud stack and checks that you can operate the cluster. > 2) They weren't generally written to be run in parallel, and thus we > often have to be concerned with system-level resource contention. > Neither where OpenStack's originally. They made heavy use of tool called * testr* ( http://pypi.python.org/pypi/testrepository ) which has a mode to automatically find when two tests are interfering with each other. testr also has a bunch of other useful features, like only re-running tests which are currently failing and keeping a database of test runs and allowing stat collection. We too could use testr if our tests output the subunit format. The subunit format was originally developed for python and has excellent python support so I think it should be pretty trivial to add. Tim 'mithro' Ansell
_______________________________________________ webkit-dev mailing list [email protected] https://lists.webkit.org/mailman/listinfo/webkit-dev
