Re: [PATCH] recreatable shuffled tests for prove
On 7/25/05, Michael G Schwern [EMAIL PROTECTED] wrote:
prove --shuffle --list=5,4,0,1,2,3 t # the shuffle list is predetermined
I'm not sure I see the utility in that last one that significantly beats
out just reordering the arguments to prove. Do you have a use case? And
what happens when the number of args is the list?
The whole point of this option is to allow the reproduction of a
certain order even in a Perl that was not compiled with the same
configuration (mainly with respect to the random number generator). I
patched the POD also with the section below.
** PART OF THE PATCH
***
=head1 RECREATABLE SHUFFLED TESTS
Shuffling tests is useful for stressing a test suite with respect
to the independency of execution order. But if a subtle bug is
found with shuffling, there must be a way to reproduce the run.
Enter the Cseed and Clist options that go together with
the Ishuffle option.
The shuffling is controled by the Perl random number seed, that
is reported every time a shuffling is done. For example, if
$ prove -b -D --shuffle 0 1 2 3 4
outputs
# shuffle seed: 87829589
2
3
1
4
0
then
$ prove -b -D -s --seed=87829589 0 1 2 3 4
must repeat the same run every time. This is valid for Perl
interpreters compiled with the same configuration. This is not
strictly necessary, but different configurations of
the pseudo-random generation (different libraries or even different
library versions) and different architectures may give different
results.
In such cases, Cprove can report the permutation list computed
by shuffling by using the Idebug option:
$ prove -b -D -d -s seed:79617309 0 1 2 3 4
# $Test::Harness::Switches: -Iblib\arch -Iblib\lib
# shuffle seed: 87829589
# shuffle list: 1,2,0,3,4
1
2
0
3
4
Instead of giving the seed for shuffling, the list can be predetermined
with the Clist argument.
$ prove -b -D -d -s --list=1,2,0,3,4 0 1 2 3 4
will run the same sequence everywhere, without concern for
differences between random number generators.
** END ***
In conclusion, we don't need --list in similar Perl builds, --seed
should be enough. There is also the issue of a very large list of
tests, which was not approached by this patch: it just prints a very
very long line when reporting the shuffle permutation in such cases.
Another issue you mentioned And what happens when the number of args
is the list? points to a verification of the suitability of the
given list as a permutation. To check it out, a sub can be written
which checks that the list is really a permutation of 0..N-1, where N
is the number of tests. Something like this would do
# check($n, @list) returns whether @list is a permutation of 0..$n-1
sub check {
my $n = shift;
my %h;
for (@_) {
return 0 if ($_ 0 || $_ = $n);
return 0 if $h{$_}++;
}
return keys %h == $n;
}
I am not sure whether this verification is desirable or practical. In
order to be correct, yes. But trying to reproduce shuffled tests will
always fall apart if the number of test varies or the original order
of the test scripts. There are many involved factors: I count on
perl-qa to help revealing what is worth checking or not, so the patch
can be tuned.
If I may suggest something. --shuffle should print the seed it used
my $seed = $Seed || int rand(2**$Config{randbits});
print STDERR Using seed: $seed\n;
srand $seed;
that way you can repeat a failed --shuffle test without having to first
remember to set --seed.
It does this indeed. This is essential for reproducing failed tests.
Instead of writing to STDERR, I included it in the test output like
this
print # shuffle seed: $shuffle_seed\n;
We cannot afford to have an optional printing of this information or
to lose that information in STDERR. If the test fails, maybe it would
be too late to report the seed that caused the situation.
(Instead of int rand(2**$Config{randbits}) I used the arbitrary int
rand(1E8). Yours is a more clever and portable expression.)
Best regards,
Adriano.
Re: HTTP::Recorder
Le mardi 12 juillet 2005 à 19:35, Ian Langworth écrivait: I'd like to improve HTTP::Recorder. I've contacted Linda Julien (http://search.cpan.org/~leira/) via her CPAN email address, but I've received no response. The module hasn't been touched in over a year and every RT ticket seems to have gone unanswered (http://rt.cpan.org/NoAuth/Bugs.html?Dist=HTTP-Recorder). Suggestions? As the author of HTTP::Proxy, a tool used by HTTP::Recorder, I'd be very happy to help. :-) A while ago, I discussed with Linda about using filter objects rather than a LWP::UA subclass for HTTP::Recorder, and she told me that the first version of HTTP::Recorder actually used filters. I think that using filters may be more appropriate (though I can't prove it right now :-). For instance, you could make HTTP::Recorder only record some sites, and not everything you visit. I've also fiddled with the idea of having a more general HTTP::Proxy::Filter class, that would contain both a HTTP::Proxy::HeaderFilter and a HTTP::ProxyBodyFilter, for complicated needs that must work both with the headers and the body. Follow-up to the HTTP::Proxy mailing list. PS: There is also a HTTP::Recorder mailing-list, but I do not know if it's still alive. -- Philippe BooK Bruhat Destroy the little and you destroy the large. (Moral from Groo The Wanderer #55 (Epic))
Re: [PATCH] recreatable shuffled tests for prove
On Tue, Jul 26, 2005 at 08:51:01AM -0300, Adriano Ferreira ([EMAIL PROTECTED]) wrote: The whole point of this option is to allow the reproduction of a certain order even in a Perl that was not compiled with the same This option has to be able to handle the case of a set of 1000 tests, all randomized, so we're talking about needing a file that contains the order. -- Andy Lester = [EMAIL PROTECTED] = www.petdance.com = AIM:petdance
Re: [PATCH] recreatable shuffled tests for prove
On Tue, Jul 26, 2005 at 08:51:01AM -0300, Adriano Ferreira wrote: Instead of giving the seed for shuffling, the list can be predetermined with the Clist argument. $ prove -b -D -d -s --list=1,2,0,3,4 0 1 2 3 4 will run the same sequence everywhere, without concern for differences between random number generators. Yeah, that's exactly what I was worried about. Why not just write: prove -b -D -d 1 2 0 3 4 this even avoids having to write special code to handle Andy's worry about large lists of arguments. cat list | xargs prove prove is a command line utility. Use the command line. I am not sure whether this verification is desirable or practical. In order to be correct, yes. But trying to reproduce shuffled tests will always fall apart if the number of test varies or the original order of the test scripts. There are many involved factors: I count on perl-qa to help revealing what is worth checking or not, so the patch can be tuned. Why would the number of test files vary when you trying to reproduce a previous test run? -- Michael G Schwern [EMAIL PROTECTED] http://www.pobox.com/~schwern Don't try the paranormal until you know what's normal. -- Lords and Ladies by Terry Prachett
Re: Test harnesses?
On 25 Jul 2005, at 22:29, Peter Kay wrote: http://qa.perl.org/test-modules.html has a bunch of test modules listed. However, there are no harnesses listed. I know Test::Harness, and I'm going to go read about Test::Builder, but what other meta-testing modules are there? [snip] All depends on your definition of harness I guess :-) Is Apache::Test one? Is Test::Class? Test::Base? Test::LectroTest? Test::Inline? One of the things that makes Perl's standard testing framework interesting is that everything is so decoupled. As long as something talks TAP you can plug it in. As well as Test::Harness, you might want to look at: Test::Harness::Straps Test::TAP::HTMLMatrix Test::TAP::Model Apart from the TAPish stuff the only other Perl testing framework that seemed to get any traction at all was the JUnit based Test::Unit, which has it's own test runners as well as being able to output TAP. Seems dead in the water now though. The only other thing that occurs is FIT frameworks, of which Perl has two: Test::FIT Test::C2FIT Neither seems to have really caught on. People seem to prefer to grow domain-specific languages in Perl based on Test::Builder instead. Cheers, Adrian
Re: [PATCH] recreatable shuffled tests for prove
On 7/26/05, Michael G Schwern [EMAIL PROTECTED] wrote: Yeah, that's exactly what I was worried about. Why not just write: prove -b -D -d 1 2 0 3 4 this even avoids having to write special code to handle Andy's worry about large lists of arguments. I see your point and agree. Probably, --seed is enough together with the remark to not wait for this to work the same everywhere (but only in the same Perl configuration wrt 'rand' and 'srand'). In the same machine, --seed can be used to reproduce the same run or output the same list with the -D switch. This output can be saved and reused just as you said. Before you wrote this message, I was thinking about some nice way to implement retrieving the list from a file and that only makes the problem and implementation worse. For example, it calls for an implementation of 'slurp', the issue of checking mentioned in previous messages, etc. prove is a command line utility. Use the command line. Ok. prove should stay simple. of the test scripts. There are many involved factors: I count on perl-qa to help revealing what is worth checking or not, so the patch can be tuned. Why would the number of test files vary when you trying to reproduce a previous test run? They should not. But in a given setting, some confusion about new or removed files could cause problems because --list would use just indices 0..N-1. I was thinking about problems that were in fact introduced by --list. Without it, they are not concerns.
Re: Test harnesses?
On Tue, Jul 26, 2005 at 04:38:45PM +0100, Adrian Howard wrote: One of the things that makes Perl's standard testing framework interesting is that everything is so decoupled. As long as something talks TAP you can plug it in. As well as Test::Harness, you might want to look at: Test::Harness::Straps Test::TAP::HTMLMatrix Test::TAP::Model The coupling problem comes from the issue that the TAP parser/runner (Test::Harness::Straps) is still tightly coupled to a single formatter (Test::Harness). The original idea of THS was to hand a THS object a Test::Harness::Formatter object (Test::Harness would be reduced to one of these, one that produced HTML or XML is another example) and go but I got bogged down worrying about how to write the callback interface. Now that I've figured that out (don't use callbacks, hand the strap a formatter object and use that) maybe I'll take another whack at finishing it up. -- Michael G Schwern [EMAIL PROTECTED] http://www.pobox.com/~schwern You are wicked and wrong to have broken inside and peeked at the implementation and then relied upon it. -- tchrist in [EMAIL PROTECTED]
Re: [PATCH] recreatable shuffled tests for prove
Here is another patch. No --list anymore. Just --seed. There is also a new test script t/prove-shuffle.t. It touches the MANIFEST and tweaks t/prove-globbing.t which depends on distribution files matching t/prove*.t. Adriano. prove-patch Description: Binary data
