Hi Fred,
On 26.05.2011 02:15, Fred Moyer wrote:
> +1
>
> Do you have commit privileges on Apache-Test?
Surprise, yes :)
Committed as r1127812.
Regards,
Rainer
> On Fri, May 20, 2011 at 1:51 PM, Rainer Jung <rainer.j...@kippdata.de> wrote:
>> I noticed, that the SMOKE change in 1.36 also introduced a new sorting
>> behaviour. In the change the test suite was running by default using the
>> repeat order, that means the tests were ordered alpabetically.
>>
>> Now the default order is random.
>>
>> Furthermore the -order=rotate (A B C A B C) is gone, and -order=repeat
>> now behaves like previously -order=rotate (it seems it previously was A
>> A B B C C).
>>
>> I suggest:
>>
>> - keep behaviour and name of repeat as it is now
>> - don't try to reimplement old repeat, because it might be tricky in
>> combination with the new way iterations are done (SMOKE)
>> - switch back to the new *repeat* (=old rotate) as the default for
>> non-SMOKE tests, i.e. run the tests by default in sorted order and only
>> do random if -order=random is explicitely given.
>> - stay with random as default for SMOKE
>>
>> The following one line change should do that:
>>
>> Index: Apache-Test/lib/Apache/TestSort.pm
>> ===================================================================
>> --- Apache-Test/lib/Apache/TestSort.pm (revision 1125303)
>> +++ Apache-Test/lib/Apache/TestSort.pm (working copy)
>> @@ -54,7 +54,7 @@
>> sub run {
>> my($self, $list, $args) = @_;
>>
>> - my $order = $args->{order} || 'random';
>> + my $order = $args->{order} || 'repeat';
>> if ($order =~ /^\d+$/) {
>> #dont want an explicit -seed option but env var can be a pain
>> #so if -order is number assume it is the random seed
>>
>>
>> Furthermore the READMEs and Changelog should get updated:
>>
>> ´Index: README
>> ===================================================================
>> --- README (revision 1125303)
>> +++ README (working copy)
>> @@ -77,18 +77,20 @@
>>
>> It's also possible that a test will pass when it's run after a
>> particular test, but if moved to run after a different state it may
>> -fail. For this reason it's important to ran the randomization smoking
>> -testing with -order=random option.
>> +fail. For this reason by default the tests run in random order.
>>
>> Since it's important to be able to reproduce the problem with the
>> random testing, whenever -order=random is used, the used seed is
>> printed to STDERR. Which can be then fed into the future tests with:
>> via APACHE_TEST_SEED environment variable.
>>
>> +By adding the option -order=repeat, the tests will be run in
>> +alphabetical order.
>> +
>> Combining these two important smoke testing techiques, one can run
>> tests with:
>>
>> - t/SMOKE -times=N -order=(repeat|rotate|random)
>> + t/SMOKE -times=N -order=(repeat|random)
>>
>> For example, to run the mod_rewrite tests 5 times, one would:
>>
>> @@ -98,8 +100,7 @@
>> modes:
>>
>> - randomize all tests
>> -- repeat the same test N times in sequence
>> -- rotate the whole tests suite N times
>> +- repeat the whole tests suite N times
>>
>> For configuration options and default settings run:
>>
>> Index: Apache-Test/README
>> ===================================================================
>> --- Apache-Test/README (revision 1125303)
>> +++ Apache-Test/README (working copy)
>> @@ -203,18 +203,15 @@
>>
>> run all tests 10 times in a random order (the seed is autogenerated
>> and reported)
>> -% t/TEST -times=10 -order=random
>> +% t/SMOKE -times=10 -order=random
>>
>> run all tests 10 times in a random order using the seed obtained from
>> the previous random run (e.g. 2352211):
>> -% t/TEST -times=10 -order=2352211
>> +% t/SMOKE -times=10 -order=2352211
>>
>> -rotate all tests 10 times (a, b, c, a, b, c)
>> -% t/TEST -times=10 -order=rotate
>> +repeat all tests 10 times (a, b, c, a, b, c)
>> +% t/SMOKE -times=10 -order=repeat
>>
>> -repeat all tests 10 times (a, a, b, b, c, c)
>> -% t/TEST -times=10 -order=repeat
>> -
>> When certain tests fail when running with -times option, you want to
>> find out the minimal sequence of tests that lead to the
>> failure. Apache::TestSmoke helps to ease this task, simply run:
>> Index: Apache-Test/Changes
>> ===================================================================
>> --- Apache-Test/Changes (revision 1125303)
>> +++ Apache-Test/Changes (working copy)
>> @@ -8,6 +8,9 @@
>>
>> =item 1.37-dev
>>
>> +Run t/TEST tests by default in alphabetical order and
>> +only t/SMOKE tests by default in random order.
>> +
>> Add t_file_watch_for to Apache::TestUtil [Torsten Foertsch]
>>
>> Add $boolean parameter to Apache::TestHandler::ok and
>> Apache::TestHandler::ok1
>>
>> Regards,
>>
>> Rainer
