On 11/08/2012 02:51 AM, Ojan Vafai wrote:
On Wed, Nov 7, 2012 at 12:41 PM, Dirk Pranke <[email protected]
<mailto:[email protected]>> wrote:
On Tue, Nov 6, 2012 at 11:58 PM, Z(an Dobers(ek
<[email protected] <mailto:[email protected]>> wrote:
> Hi WebKitties,
>
> I'd like to see in what scale the --test-list option in NRWT is
used and
> whether anyone would object a change in how its usage behaves.
>
> Currently the test gathering takes into account any paths that
were passed
> in through the command line and any paths listed in the file to
which a
> possible --test-file option points[1]. These paths are then
expanded if
> necessary (due to platform-specific tests and virtual test
suites) and all
> the tests to which these paths apply are gathered[2]. All the
gathered tests
> are then either sorted into a natural order or randomized (if the
> --randomize-order option is used)[3].
>
> I'd like to propose a change in the behavior when using the
--test-list
> option. When used, the tests would be gathered only from the
paths listed in
> the file to which the option points. Also, the gathered tests
would be
> sorted to follow the order of the paths in the test list file.
For instance,
> consider the following two entries being placed in the test list
file:
>
> c/d/e.html
> a/b/
>
> The current behavior would gather all the tests to which these
two paths
> apply and sort them in this order (on the premise that no other
paths are
> passed through the command line arguments):
>
> a/b/1.html
> a/b/2.html
> c/d/e.html
>
> The new behavior would place the c/d/e.html test at the top of
the list but
> sort the other two tests in order, like this:
>
> c/d/e.html
> a/b/1.html
> a/b/2.html
>
> The idea behind this is to be able to specify the exact order in
which the
> tests should be run. I'm currently playing around with a tool
that randomly
> chooses a certain number of tests, runs them and bisects down
the first
> regression that occurs (if any) to the smallest possible ordered
list of
> tests that reproduces that regression. The proposed change makes
this a
> simple task from the test listing perspective and I'd argue that
exact test
> ordering is the main point of an option such as --test-list.
>
> Currently I'm using an untested workaround of adding a new
option that
> specifies the tests should be reordered back to the original
order that the
> test list file provided, but I'd like to move on with
implementing the
> change in behavior if everyone feels it's OK and won't tamper
with his/her
> workflow.
>
> -Z
>
>
> [1]
>
http://trac.webkit.org/browser/trunk/Tools/Scripts/webkitpy/layout_tests/controllers/layout_test_finder.py#L46
> [2]
>
http://trac.webkit.org/browser/trunk/Tools/Scripts/webkitpy/layout_tests/port/base.py#L576
> [3]
>
http://trac.webkit.org/browser/trunk/Tools/Scripts/webkitpy/layout_tests/controllers/manager.py#L309
>
Hi,
I have used the option from time to time, and I have also wanted from
time to time the ability to run tests in a specific order. This change
would be fine by me. If people want to ensure that tests run in the
old order, they can always sort the test list.
I have used this in the past to identify which test caused test
flakiness (e.g. I have 100 tests and I know the last test depends on
one of the other 99 to run first in order to pass). The problem with
this last sentence is that it's hard to know what order
run-webkit-tests would choose, so it's hard to replicate it. It would
make that use-case a little harder if we made this change. Otherwise,
I don't care either way.
Exactly for the reason Ojan mentioned I think the best would be to add
an option for defining the desired ordering behavior. Paths on command
line and in --test-list should imply to use the ordering as it is passed
but it would be possible to force the current method. I hope it would
not add too much complexity.
-kbalazs
_______________________________________________
webkit-dev mailing list
[email protected]
http://lists.webkit.org/mailman/listinfo/webkit-dev
