Re: A smoke a day keeps the doctor away (Daily Build and Smoke Test)
On Sun, Feb 18, 2001 at 06:26:00PM -0500, [EMAIL PROTECTED] wrote: > I'd like to set up automated daily build and smoke tests for bleedperl > and the other active trees. How many configuration permutations were you proposing? > It will be fairly simple to write and distribute a cron job which > wraps "make test" and emails if anything fails. For multiple configurations some sort of summary analysis really is necessary (although H Merjin Brand seems to have put a lot of effort into getting this completely under control) Nicholas Clark
Re: Untested libraries.
On Sat, Apr 21, 2001 at 10:32:35PM +0100, Michael G Schwern wrote: > Devel::SelfStubber hmm. yes. I was meaning to get around to this about 3 years ago. Stage 1 (the test for SelfLoader) was added. IIRC there were bugs in SelfStubber and how it coped with __DATA__ and how it coped if you called it with more than one file to stub. > PerlIO if Jarkko-of-Borg assimilates PerlIO::gzip (which I wasn't sure he was thinking about) it's got (currently) 98 tests that have picked up some of the bugs in the core PerlIO system. Are there prizes for every regression test added? :-) Nicholas Clark
Re: Untested libraries.
On Sun, Apr 22, 2001 at 01:54:31AM +0100, Michael G Schwern wrote: > On Sat, Apr 21, 2001 at 07:21:47PM -0500, Jarkko Hietaniemi wrote: > > > Net::Ping > > > > Maybe hard to test anything meaningful portably. > > localhost is always an option, no? If nothing else we can make sure > an object can be set up and torn down. Net::Ping has broken badly in > the past. I believe that the hive mind of p5p found that "localhost" certainly wasn't always an option (in that we couldn't guarantee a hostname lookup to work on that name (Win32?) and I wouldn't be surprised that 127.0.0.1 doesn't work everywhere. *However* that is not an excuse not to make a test that attempts to lookup localhost, and if that fails go "1..0 skipping", else charge ahead and test what it can. [I think. Someone's going to suggest why this is stupid] > > Err, nothing much would work in bleadperl if PerlIO didn't since > > it's the "stdio" layer... But subtleties in the implementation aren't tested. EG Nothing in the core realised that a new bug had been introduced in the past 3 weeks: open FOO, "<:foo", "foo" or die "Failed" should fail if the layer foo fails to push (during the open). Currently the open doesn't return failure if the push failed. Nick I-S is aware of this bug. > Great, shall we remove the t/base tests then? ;) No. BTW the first test that a negative floating point number correctly converts from a PV to an NV is one of storable's, which is near the end of lib/*.t :-( We do now have a test that 1 + 1 == 2. > PerlIO claims it can do things, for example, the :crlf in the > SYNOPSIS. That's a starting point to test against. I agree. I'm not quite sure how to write such a test, else I'd have a go. Nicholas Clark
Re: Untested modules update: The Magic Number is 27
On Thu, Dec 27, 2001 at 09:11:26AM -0800, Benjamin Stuhl wrote:
> --- Michael G Schwern <[EMAIL PROTECTED]> wrote:
> [... and the shears went snip, snip, ker-snip ...]
> > #B::Bytecode[module broken, can't test]
> > #B::C [ditto]
> > #B::CC [ditto ditto]
> > #B::Concise [[EMAIL PROTECTED]]
> > #B::Disassembler[Wolfgang Laun]
> > #B::Lint[[EMAIL PROTECTED]]
> > #B::Stackobj[[EMAIL PROTECTED]]
> > #B::Xref[[EMAIL PROTECTED]]
> > Byteloader
>
> Byteloader is untestable without a working B::Bytecode (we
> have no way of generating input files for it. Anyway,
> Byteloader/B::Bytecode are actually speed _losses_, so IMHO
> they should probably simply be killed - they're just wasted
> space in the distribution. (IIRC, I was the last person to
> seriously hack on them.) And if they get removed from the
> core, they obviously don't need tests ;-).
I think I agree on all of this.
("think" in that I've read it twice and can't see any obvious reason why I'm
going to be able think of any counter arguments between now and perl6)
> > Dynaloader
>
> What sort of test does Dynaloader need? If dynaloading
> doesn't work, no dynamic extensions (Socket, Fcntl, IO,
> POSIX, ...) will work, and the errors they'll spew are
> pretty obviously loading problems.
XSLoader::load 'POSIX', $VERSION;
XSLoader::load 'Fcntl', $VERSION;
XSLoader::load 'Socket', $VERSION;
XSLoader::load 'IO', $VERSION;
although that's a biased picture as it appears that many other core extensions
are using its interface directly, rather than via the featherweight XSLoader.
(including Encode.pm, I18N.pm, threads.pm threads/shared.pm)
>From its pod:
DynaLoader Interface Summary
@dl_library_path
@dl_resolve_using
@dl_require_symbols
$dl_debug
@dl_librefs
@dl_modules
Implemented in:
bootstrap($modulename) Perl
@filepaths = dl_findfile(@names) Perl
$flags = $modulename->dl_load_flags Perl
$symref = dl_find_symbol_anywhere($symbol) Perl
$libref = dl_load_file($filename, $flags) C
$status = dl_unload_file($libref) C
$symref = dl_find_symbol($libref, $symbol) C
@symbols = dl_undef_symbols()C
dl_install_xsub($name, $symref [, $filename])C
$message = dl_error C
so I guess that as much as that as possible, preferable all needs testing
(in an ideal world. As I found with Benchmark.t, trying to test everything
gets quite big. And it had less things than Dynaloader.)
Nicholas Clark
Re: More Test::Builder::Test stuff
On Sun, Feb 17, 2002 at 02:00:06PM +, Mark Fowler wrote: > On Sun, 17 Feb 2002, Mark Fowler wrote: > > > I'd really like to see that it still works on a system that > > doesn't have Term::ANSIColor installed on it (it should turn colouring > > into a no-op and skip tests, but I can't test that.) > > D'Oh! No it shouldn't. As color should work (just without colouring) on > systems that don't have Term::ANSIColor installed then the tests shouldn't > be skipped at all, they should be carried out to explicitly check that > then still work without it. > > Code updated so that it should do just that. I think that if you have enough disk space you should be able to build yourself a second perl install that doesn't have Term::ANSIColor installed, by configuring perl to use a different prefix and ensuring it doesn't have any of the "standard" places to find libraries in INC. Alternatively, if your Term::ANSIColor is installed in 5.6.1's directories, just build yourself a 5.005_03 in the regular place :-) Nicholas Clark -- EMCFT http://www.ccl4.org/~nick/CV.html
Re: [Patch docs] perlsub. Re: [ID 20020227.012], [ID 20020227.018]
On Thu, Mar 07, 2002 at 01:25:55AM +0300, Anton Tagunov wrote:
> Hi! How about this?
>
> --- pod/perlsub.pod.origWed Feb 20 18:02:38 2002
> +++ pod/perlsub.pod Thu Mar 7 00:30:53 2002
> @@ -169,7 +169,7 @@
>
> Like the flattened incoming parameter list, the return list is also
> flattened on return. So all you have managed to do here is stored
> -everything in C<@a> and made C<@b> an empty list. See
> +everything in C<@a> and made C<@b> an empty array. See
Not sure. I'd like to find a way to phrase that without describing @b as
either "list" or array. It's set to an empty list, but it is an array.
And finding a way of saying that without using either word feels best.
> L for alternatives.
>
> A subroutine may be called using an explicit C<&> prefix. The
> @@ -727,8 +727,8 @@
>
> sub ioqueue {
> local (*READER, *WRITER);# not my!
> -pipe(READER, WRITER);or die "pipe: $!";
> -return (*READER, *WRITER);
> +pipe(READER, WRITER) or die "pipe: $!";
> +return (*READER{IO}, *WRITER{IO});
> }
> ($head, $tail) = ioqueue();
>
> --end of patch--
>
>
> Have I been too bold with adding {IO}? To tell the truth I do
> not understand what is happening completely, so it's again a
> blindfolded strike. I guess that with my patch instead of
> typeglobs this sample starts returning references to file handles.
>
> Is that okay? Is that smth to recommend to people?
Not sure. I'm not sure what's going on with the IO, but it's late an I'm
tired. However, I'm replying because I can see a syntax error that the
patch removes, and as nothing has been applied, that syntax error remained.
Was there ever a feasible plan on how to automate syntax checking of all
the code examples in the pod snippets?
[I remember this rant I had about perlipc once. (rant-and-patch, IIRC)
I also remember another thing that fell off my to-do list - try to make
some of the perlipc examples into regression tests. That was related to
the SOCKS5-rant]
Nicholas Clark
--
Even better than the real thing:http://nms-cgi.sourceforge.net/
Re: How to test for the absence of an infinite loop?
On Mon, Mar 18, 2002 at 03:57:55AM -0500, Mark-Jason Dominus wrote:
>
> Michael The Schwern <[EMAIL PROTECTED]> says:
> > Use alarm and skip the test if $Config{d_alarm} is false (see
> > t/op/alarm.t for an example). If you think the infinite loop is due
> > to a programming glitch, as opposed to a cross-platform issue, this
> > will be enough.
>
> Thanks very much!
>
> > The only other thing I can think of is to set up a parent process
> > which forks off a child
>
> ARRRGH. NO FORK.
On Mon, Mar 18, 2002 at 04:33:48PM +, Nick Ing-Simmons wrote:
> FWIW system() does create a new process on Win32, but fork() does not
> (in only creates a new thread). I have no idea if flock-faking stuff
> on Win32 allows recursive locks by different threads of the same process ...
One of my TODOs (preferably sooner rather than later) was to see if this
bit of ext/Socket/socketpair.t could be generalised enough to end up in
test.pl and provide a portable self-destruct timer
(for infinite loops and other infinite hangs):
BEGIN {
chdir 't' if -d 't';
@INC = '../lib';
require Config; import Config;
$can_fork = $Config{'d_fork'}
|| ($^O eq 'MSWin32' && $Config{useithreads}
&& $Config{ccflags} =~ /-DPERL_IMPLICIT_SYS\b/);
if ($^O eq "hpux" or $Config{'extensions'} !~ /\bSocket\b/ &&
!(($^O eq 'VMS') && $Config{d_socket})) {
print "1..0\n";
exit 0;
}
# Too many things in this test will hang forever if something is wrong,
# so we need a self destruct timer. And IO can hang despite an alarm.
# This is convoluted, but we must fork before Test::More, else child's
# Test::More thinks that it ran no tests, and prints a message to that
# effect
if( $can_fork) {
my $parent = $$;
$child = fork;
die "Fork failed" unless defined $child;
if (!$child) {
$SIG{INT} = sub {exit 0}; # You have 60 seconds. Your time starts now.
my $must_finish_by = time + 60;
my $remaining;
while ($remaining = time - $must_finish_by) {
sleep $remaining;
}
warn "Something unexpectedly hung during testing";
kill "INT", $parent or die "Kill failed: $!";
exit 1;
}
}
}
The idea being that a test killing itself after elapsed time (and failing
to return some results from that test) is better than a test hanging forever
and causing a whole automatic test run to fail.
[and in the case of a busy hang, rather than an idle hang, munching shared
CPU]
Also, did the Windows Gurus work out why the above didn't like Win95?
Nicholas Clark
--
Even better than the real thing:http://nms-cgi.sourceforge.net/
Re: Test::Harness skip reason report change?
On Fri, Apr 26, 2002 at 11:39:26AM +0400, Ilya Martynov wrote: > Looks nice. But I'm curious if it will be possible to see skip reasons > with new formatting if one test had several different skip > reasons. Currently it just emits 'skipped: various reasons'. on the "looks nice". I hadn't thought of the better list of reasons, but I agree with what that too. However, if there are 50+ different skip reasons, would it be better to have some sort of cutoff on the amount of extra data per test to show. Something like 10, and then it a ... line and tells you to run the test to see the full output? Nicholas Clark -- Even better than the real thing:http://nms-cgi.sourceforge.net/
Re: [perl #15479] perl 5.8.0 segfault
On Wed, Jul 31, 2002 at 01:20:25PM +0200, Rafael Garcia-Suarez wrote: > Wasn't there a .t file to run separate perl interpreters > and test for core dumps ? I keep forgetting that I need to remember to ask this. Is there a FAQ for regression test writing? Well, an guide to "so I want to write a regression test, explaining how to do it, how perl5's tests are structured to reduce interdependencies, use Test::More; when Test::More is not appropriate.." And where did the p5p FAQ get to? Nicholas Clark
Re: RFC: Test::ManyParams
On Fri, Aug 02, 2002 at 08:16:17AM +0200, Janek Schleicher wrote: > Ilya Martynov wrote at Fri, 02 Aug 2002 07:42:44 +0200: > > >>>>>> On Wed, 31 Jul 2002 21:52:17 +0200, Janek Schleicher <[EMAIL PROTECTED]> >said: > > > > JS> [..snip..] > > > > JS> Thinking in general, > > JS> there could be also some other features included. > > JS> Let's think we'd like to test the creation of big pictures, > > JS> perhaps 5_000 x 5_000. > > JS> It could take a while to make a test for all pixels, > > JS> but we also would like to test some of them (randomly choosen), > > JS> to avoid systematic error. > > > > Test results should be easily reproducible. I don't think having > > randomly choosen tests is good idea. I think having randomly chosen repeatable tests is an excellent idea. Over the course of many people making many test runs explore far more of parameter space than any single systematic test permutation device could hope to achieve. > srand could be our friend. Which is how I'm doing it at work now. I call srand with a random number. (I'm getting mine from /dev/urandom, but I suspect that calling rand() and using that to prime srand will achieve sufficient randomness for these purposes. (ie you get to run one of 65536 sequences, which is better than running 1 of 1 sequence) My tests at work generate a seed this way before starting. They call srand with the seed, and store the seed. If the test fails, they print out what the seed was on STDERR. (as a hack in an END block that checks the exit code after Test::Builder has set it in its END block) If the test is run with no arguments it randomises (as above). Else it treats the argument as a chosen seed, and uses that to call srand() to repeat a previous run. This way I can make test a few times and it runs different random parameters each time. And if a test fails, it prints out the seed, and I can re-run repeatedly by hand until I work out why there is a bug. And fix it. I've found bugs this way that I wouldn't have spotted early by just running a fixed set of parameters in my tests each time. Mainly because I make test many times during the day as I make each incremental change (so each make test has to be fast) which immediately picks up on big bugs, but the incremental effect can find really obscure bugs that only crop up for a small proportion of possible input. > However, it's not important for me that the parameters are really randomly choosen - > > allthough I still would prefer to have some before I release a module - > but I'd like to write tests to avoid systematic mistakes, > while it would need too long to test all scenarios. This is the problem that I have, and I think I've found a solution that works for me. Nicholas Clark
Re: RFC: Test::ManyParams
On Tue, Aug 06, 2002 at 06:42:14AM +0200, Janek Schleicher wrote: > I think the most fair way to handle it, > is to give a warning when Test::ManyParams is used twice with seeding setting > in the same package. Yes. This sounds like what I'm doing at work - if two attempts to call the seeding function are made, then a warning is issued. I can't think of a better way, and it does seem to work. The only problem I found was "unauthorised" use of rand() before the official call to srand() had been made. I'm not sure of the best way to stamp that out - possibly by overloading CORE::GLOBAL::rand at BEGIN time. Nicholas Clark -- Even better than the real thing:http://nms-cgi.sourceforge.net/
Re: Testing POSIX locale support
On Mon, Aug 26, 2002 at 04:05:26PM -0400, John Peacock wrote: > I have a module, Math::Currency, which tries to be very smart with regard > to POSIX locale settings when defining the default currency formatting. > However, since I can only test on _my_ systems, I am limited in my ability > to guess all of the strange locale settings that may be in use. > > Specifically, in order to test that the locale stuff is working, I need to > have two different locales installed to switch between. Currently, I am > using en_US and en_GB, but obviously that ignores most of the planet. This doesn't answer any of your questions (sorry) but IIRC on p5p it was found that fa_IR caused quite a lot of "fun", as it used a character for the decimal separator that was multibyte. (certainly for standard numerics, which is what core perl was concerned with. No idea about monetary values, and as I don't understand any Farsi or have any Farsi fonts installed, it's not going really to help if I have a look.) So you may wish to install fa_IR and test with that. Nicholas Clark -- Even better than the real thing:http://nms-cgi.sourceforge.net/
Re: Fwd: [ken@mathforum.org: Re: [Inline 0.43] insecure dependency when tainting]
On Thu, Aug 29, 2002 at 12:29:58AM -0700, Michael G Schwern wrote:
> Also, Nick's example is a little odd. You usually don't want '.' (ie. t/)
> in your @INC. It's more like this:
>
> BEGIN {
> if($ENV{PERL_CORE}) {
> chdir 't';
> @INC = '../lib';
> }
> }
>
> but in some cases you need to include something more. For example,
> MakeMaker does this:
>
> BEGIN {
> if($ENV{PERL_CORE}) {
> chdir 't';
> @INC = ('../lib', 'lib');
> }
> else {
> unshift @INC, 't/lib';
> }
> }
>
> so it can see specialty helper modules in t/lib/.
> On Thursday, August 29, 2002, at 06:51 AM, Nicholas Clark wrote:
> >You'll often see regression tests in the core start like this:
> >
> >sub BEGIN {
> >if ($ENV{PERL_CORE}){
> > chdir('t') if -d 't';
> > @INC = ('.', '../lib');
> >} else {
> > unshift @INC, 't';
> >}
For information, my example was from the top of t/integer.t in Storable
It looks like the same bit of code was cargo-cult cut & pasted (er, by
me I think) from the other Storable tests (such as freeze.t) where they
have to require 'st-dump.pl'; to get a library of useful test routines.
I'm not sure if Storable was actually the best (cleanest, simplest) thing to
snag the example code from.
Nicholas Clark
Re: Eliminating the core-test BEGIN block (was Re: [ken@mathforum.org: Re: [Inline 0.43] insecure dependency when tainting])
On Mon, Sep 02, 2002 at 03:56:14PM -0700, Michael G Schwern wrote: > While the result is still ugly, it means we can expand and alter the > requirements for running a core test. For example, the PERL_CORE > environment variable should be set (t/TestInit.pm currently doesn't). So > the full command is really something like this: > > cd t; > PERL_CORE=1 ./perl -I../lib path/to/test.t > > Which could be reduced somewhat to: > > ./perl -It -MTestInit t/path/to/test.t > > and perhaps reduced a little further by moving/linking TestInit.pm into the > top level directory. > > ./perl -MTestInit t/path/to/test.t I think that that would be a good idea. > but that will cause trouble when running tests with -T (because . is no > longer in the path). ../perl -MTestInit t/path/to/test.t Too late for "-T" option at t/path/to/test.t line 1. So you already have to change the command line to run a -T test by hand. So I don't see this as a problem: ../perl -TI. -MTestInit t/path/to/test.t 1..1 ok 1 Although writing it -T -I. is probably clearer Nicholas Clark
Re: Why not a smoke db ?
On Sat, Sep 21, 2002 at 05:50:45PM +0200, alian wrote:
> After see a smoke report from Nicolas Clark using ccache, I took a look
> at ccache web site. And find http://build.samba.org.
Oooh, they've got a Cray in theirs.
> So I've begin something like this, thing I would see on http://qa.perl.org:
> http://www.alianwebserver.com/cgi-bin/smoke_db
>
> Comments are welcome.
Part in English, part in French? :-)
I suppose that I should be thankful that it's not part Finnish, part Tamil.
On Mon, Sep 23, 2002 at 04:59:21PM +0200, Abe Timmerman wrote:
> Op een mooie herfstdag (Monday 23 September 2002 16:14), schreef Alain Barbet:
>
> > Jarkko Hietaniemi wrote:
> > http://www.alianwebserver.com/cgi-bin/smoke_db
> >
> > > Looks great! Some suggestions/questions:
> > > - freebsd os version missing?
> >
> > Yes it's missing info from Nicolas Clark reports.
>
> That is due to the combination of 1.13 and changed Config.pm (should be solved
> in 1.15).
And as I seem to be running 1.06 (with tweaks) this would explain why you're
not seeing it. I'll try to send a patch to merge the tweaks back in, plus
then try to get around to making the changes mentioned at YAPC::EU
(splitting the commands run for make regen_headers out into a script, calling
that script instead of make regen_headers in the smoke harness, and the
option of multiple build trees where copying (or hard link forestry) is
faster than make distclean)
But that's at least number 3 on my todo list (after sorting out the
spoofathon, getting the script for my YAPC::EU talk written down and
online)
> > > - at least for the linuxes and bsds
> > > it would be useful to have also
> > > the CPU architecture in the
> > > report (not all boxes are x86)
Nor are all Solaris boxes sparc.
(or x86 for that matter - it was internally ported to alpha to check 64 bit
cleanliness before true 64 bit sparc chips were available)
> > k, but this is a missing info from smoke report.
>
> I can add $Config{archname} to the smokereports if you all want that.
That would be good.
Nicholas Clark
Re: [ Memory ] Re: Thought
On Wed, Oct 02, 2002 at 05:06:53PM +0200, Tels wrote: > On 02-Oct-02 H.Merijn Brand carved into stone: Going to be a bit hard to change, then :-) > > As a start for a `normal' interface to the interperters internals, this > > time no open hack, but a neat interface. > > Very open to additions. > > Useful? > > Is it accurate? > > > Feedback please > > > > > > NAME > > Devel-Internals - Perl extension for internal interpreter > > statistics > > > > SYNOPSIS > > use Devel::Internals; > > > > my $end = sbrk (); > > Can you please use a better name? sbrk() doesn't mean anything to me. Trouble is that it's entirely accurate. If you know what sbrk() is in Unix, then you know that this would return it. The value is not directly memory used or anything like that. It's the current sbrk() :-) Nicholas Clark
Re: [ Memory ] Re: Thought
On Wed, Oct 02, 2002 at 05:07:20PM -0400, Michael G Schwern wrote: > On Wed, Oct 02, 2002 at 10:26:32PM +0200, H.Merijn Brand wrote: > > > I realize that sbrk() is a familiar concept to deep C programmers on > > > BSD-style systems, but in order for this to be useful to Perl programmers, > > > or even users not familiar with BSD, you're going to have to name it > > > something more descriptive than sbrk() and explain better what it does. Is it going to be useful (see below) to people who don't know this sort of thing? > > That's why there /is/ an Internals.pm: to hide the gory details and > > implementation specifics to the user. > > But you're not. You're just exposing sbrk(), which is a gory detail. My > sbrk man page describes sbrk as being used to find "the current location of > the program break" which means nothing to me. Nor does "returns the current > memory top". > > It'll be even worse on an OS which doesn't have sbrk, which means no sbrk(2) > man page to look at. > > Does sbrk() just return the current number of bytes allocated to the > program? If it only returns the value from sbrk(), damn well call it sbrk. That's what it is; don't invent a new name. It will only cause confusion later for anyone who does migrate from perl to dealing with unix guts, or needs to talk to someone who does know what sbrk() is. (Oh. we're going to call this concept pels. You may hear other people call them pixels...) The reason I'm saying it might not be much use to people unfamiliar with the internals of unix is precisely because it does return a precisely defined but not directly useful value - the highest address on the heap. If you read it, call perl code that creates a known perl structure, and read it again, doesn't go up directly and exactly related to the amount of memory that structure needed. Depending on how much other memory there is free, it may not go up at all, or it may go up more if the allocation system grabs a big block of free space. However, the function returns something that may not be sbrk() on a system without sbrk(), don't call it sbrk, because it ain't. (And do document what it does return on each different system) Nicholas Clark
Re: [ Memory ] Re: Thought
On Thu, Oct 03, 2002 at 02:35:14PM -0400, Benjamin Goldberg wrote: > If "what it does" is merely return the value of the C sbrk() function, > then IMHO, sbrk() is a perfectly good name. > > However, as to other possible names -- how about ProcessSize() ? I'm > not sure if this is really a valid description of what sbrk() returns, > though. I'm not certain it's as accurate as we might like to think. I am under the impression (quite possibly wrong, as I don't have the source code handy) that 1: Doug Lea's malloc will do anonymous mmap() to satisfy really big requests 2: glib malloc is a variant derived from Doug Lea's code Hence is it plausible that systems exist where the malloc that perl calls may normally get memory from the system with sbrk(), but occasionally use somewhere other than the system heap. So perl code might cause more memory to be used, but if we report back sbrk() as process size we're actually claiming a bit more for our "fact" than we can actually be sure of. If we call it sbrk() we can divorce ourselves from actually claiming that it is the one true source of memory :-) Not that this is helpful - I guess we want a direct sbrk() interface if sbrk() is on the system, and a process size interface that makes the best estimate it can. But I'm not writing this thing, and so far I guess I've not wanted it *that* much. [mm. maybe I have. number 3 on my hitlist of "what slows perl" is RAM. So seeing if a data structure change reduces RAM is interesting to me, the encode compiler, and everyone building perl from source] Nicholas Clark -- Even better than the real thing:http://nms-cgi.sourceforge.net/
Re: [ Memory ] Re: Thought
On Sat, Oct 05, 2002 at 03:50:34PM +0200, Tels wrote: > Nevertheless, making your data structures smaller will help, even if in > some particulare case it doesn't reduce the heap usage directly. Yes, I found this to be true, but on my hitlist of things that make your perl script slow, it was number 3, with raw CPU usage as number 2, and Ops as number 1. [ie play Ops golf, even if you think you'll be using more CPU or more RAM with your way of doing it in less Ops. And unlike conventional perl golf, that's sum total Ops encountered at runtime, not total Ops used] I did a talk about optimising perl code at YAPC in Munich. I'm going to use this as an opportunity to blatantly plug the fact that I've put it online: http://www.ccl4.org/~nick/P/Fast_Enough/ because it might contain one or two ideas that people haven't thought of. People who saw it at YAPC may not want to read it *yet*, because I've not incorporated more bits based the several other good ideas *I* hadn't thought of that other people told me. Nicholas Clark -- Even better than the real thing:http://nms-cgi.sourceforge.net/
Re: Some upcoming changes to Test::Builder
On Mon, Oct 14, 2002 at 09:00:42PM -0400, Michael G Schwern wrote: > 5.8.0's threads are giving me serious headaches. When 5.8.1 comes out I > might drop support for 5.8.0's threads just so I can remove a large volume > of work-around code. Leaving support for 5.005 threads in? I'm confused. Or do you mean dropping all automatic threading support? In which case, how does one test threaded code? [not that I've written any] Will there be an explicit thread testing module? Nicholas Clark
Re: Test::Class - comments wanted
On Sun, Oct 13, 2002 at 04:01:10PM -0700, David Wheeler wrote: > On Sunday, October 13, 2002, at 10:05 AM, Tony Bowden wrote: > > >> Makes it simpler for people who prefer the 'no_plan' style of > >> testing > > > >Maybe this is what I just don't get. I'm not one of those people, so I > >don't really understand why people might prefer it. Especially here > >where there's such a natural way to specify them, and you're only > >counting them per method, rather than over the entire test. > > I have to agree with Tony. I think it's important to explicitly > indicate the number of tests that a given method runs, and to be > explicit about saying when you're not sure how many tests there will > be. In that regard, I like the current design better, although I would > have no complaint if you decided to change the string "no_plan" to > something else. Me too. In that I'd not like "no plan" to be the default. I'd like it to be necessary to explicitly choose no plan. I'm not sure if it is helpful (or even a good interface) but it's possible to count the number of arguments in @_, and thereby distinguish between no arguments and 1 undef argument. It's just that saying "pass a literal undef for no plan" is about as clear as "no_plan". Empty string is probably clearer, and quite easy to test for (with length, after a defined || croak test) Nicholas Clark -- Even better than the real thing:http://nms-cgi.sourceforge.net/
Re: [ANNOUNCE] tv (Test::Verbose 0.001)
On Thu, Oct 24, 2002 at 09:57:48AM -0400, Barrie Slaymaker wrote: > New! Improved! As Seen On TV!! > > ahem. > > tv is a command bundled with Test::Verbose that puts an easier, smarter > interface around the ExtUtils testing system. But does it come complete with a free Schwern to write your tests for you? That's what I want. :-) [To be honest, I don't care if it's a Schwern (closure) who merely puts it onto his (captured lexical) $TODO, as long as there's also a reference to Chromatic to actually get a round tuit. That still gets my tests written for me with maximal laziness] Nicholas Clark -- INTERCAL better than perl? http://www.perl.org/advocacy/spoofathon/
callbacks at the end of Test::More ?
I'm intending to write a module to work with Test::More to provide repeatable "random" tests (ie take advantage of the pseudo random nature of rand). I discussed it with Schwern at YAPC::EU and he thinks it will work. However, I'd like to be able to cleanly print out my random number seed to STDERR (or whatever Test::Builder's correct name for this is) if it believes that any tests have failed, and I can't see a clean way to do this. When I was experimenting with this idea at work I hacked things by having an END block that checked $?, and if it was non-zero printing out the seed. However, this doesn' seem to be a good idea for production code. Is there a way to achieve such a callback from Test::Builder? It looks like there isn't, as the logical place would be in Test::Builder::_ending after it has worked out the number of passing and failing tests. If so, could I make a feature request for some way to register such a callback. Nicholas Clark -- Brainfuck better than perl? http://www.perl.org/advocacy/spoofathon/
Re: callbacks at the end of Test::More ?
On Wed, Nov 06, 2002 at 07:18:14PM -0500, Michael G Schwern wrote:
> On Sat, Oct 26, 2002 at 04:22:49PM +0100, Nicholas Clark wrote:
> > However, I'd like to be able to cleanly print out my random number seed to
> > STDERR (or whatever Test::Builder's correct name for this is) if it believes
> > that any tests have failed, and I can't see a clean way to do this.
>
> In an END block, grab the T::B object and check the summary().
>
> #!/usr/bin/perl -w
>
> use Test::More tests => 1;
>
> pass('foo');
>
> END {
> my $failed = grep!$_, Test::More->builder->summary;
> diag $failed ? "Failed\n" : "Passed\n";
> }
>
> The only trouble there is if the test as a whole fails during the ending,
> like because the wrong number of tests were run or the exit code was odd,
> the above logic is too simple to know that. You have to go through the
> sort of logic _ending() does.
I think that will do fine for what I need. I'm only reporting extra
diagnostics if all tests failed.
I now have this wierd thought that I may have asked you this question in
Munich and you told me this answer before. If so, oops.
>
> Not sure to add in an ending callback or an is_passing() method. Or both.
>
>
> BTW How does one get the current srand if you use the one set by Perl?
I don't think it's possible. I think I'm going to have to overload rand
and do the srand thing by hand. (and probably overload srand in case anyone
else is attempting to go behind my back)
Nicholas Clark
Re: [RFC] Module::Husbandry
On Fri, Dec 20, 2002 at 08:17:16AM -0500, darren chamberlain wrote:
> This is great. I keep meaning to do something like this myself, get
> about halfway there, and then end up doing something else. Good work.
>
> - Using Template.{t,pm} is a little counter-intuitive -- this looks
> like it's a ragular Perl module, instead of a template (despite the
> name). I'd use something like _skeleton.pm and _skeleton.t for
> these files.
Aargh. A bad pun has landed in my head and demands that I share it:
tePMlate.pm
Hopefully it's clear that something with that name is not intended for
production.
Nicholas Clark
Re: Meta testing
On Mon, Mar 03, 2003 at 12:14:57AM +, Fergal Daly wrote: > Any comments or suggestions? No, sorry. I don't know much of the details of the innards, and it's a complex problem. But I don't think anyone has replied, so I thought that I should avoid you suffering Warnock's Dilemma any longer. Nicholas Clark
Re: Scrutinizing CPAN distributions (was Testing for valid path names...)
On Mon, Aug 18, 2003 at 11:25:14PM +1000, Andrew Savige wrote: > Four completely normal peeps there. ;-) Judging by the popularity of > cricket and golf statistics, I think there are a lot of lurkers too. So in a few years time we'll have people quoting Wisden's CPAN almanac? Yes, bioperl now including the entire human genome as a prereq has added a new sharp upwards spike to CPAN's mean prereq size/module size ratio. We've not seen anything like this since Meta added a dependency on the entire Linux 2.7.15 source tree back on April 2nd 2005 :-) Nicholas Clark
Re: blocks and subplans again
On Wed, Aug 20, 2003 at 03:52:42AM -0700, Michael G Schwern wrote: > On Wed, Aug 20, 2003 at 10:34:08AM +0100, Fergal Daly wrote: > > On Wednesday 20 August 2003 08:23, Michael G Schwern wrote: > > > You don't want subtests to have to know any state, such as how far to > > > indent. Why? Consider: > > > > Something has to keep state > > The state of the overall test? No, that's something which should be > avoided. See below. > > A plan counts the number of tests and sub-blocks to expect. The 1..8 at the > > end would be calculated by summing the plans of all blocks. > > I've yet to see a real use-case for plans of plans. I don't like no-plans of no-plans. I think that there is the possibility of missing tests not getting spotted. If the outer test is a noplan ok 1 ok 2 ok 3 1..3 ok 1 ok 2 ok ok 1..5 then I think that we miss that test 3 of the nested test failed. Or get confused about which test was the missing test Or if I've got that wrong, I think that we also get confused with 2 adjacent sub tests. ok 1 ok 2 ok 3 1..3 ok ok ok ok ok 1..3 Nicholas Clark
Re: blocks and subplans again
On Thu, Aug 21, 2003 at 12:17:39PM -0700, Michael G Schwern wrote: > Not if you introduce an end tag (though I'd rather not). Why not? I'd like the ability to specify a variant of no plan that is roughly "it ain't over until the fat lady sings" and then if I don't print out a singing fat lady as the last bit of output then that's a fail. I've had things exit cleanly midway from a no_plan test script, and the harness has been none the wiser. But I just do stupid things. Nicholas Clark
Re: Phalanx / CPANTS / Kwalitee
On Mon, Oct 13, 2003 at 10:28:29AM +0200, Thomas Klausner wrote: > Obviously, this list is far from complete, which is why I request some > feedback on more hints for Kwalitee from all of you! Something I'd be curious about: Modules with lower case names which aren't pragmas. (although how you determine this is hard) Nicholas Clark
Re: Taint mode testing and project Phalanx
On Mon, Oct 20, 2003 at 10:27:34PM -0700, Michael G Schwern wrote:
> On Tue, Oct 21, 2003 at 12:24:03AM -0500, Dave Rolsky wrote:
> > Not to mention that it's buggy as hell. For example, in various versions
> > of Perl I've used there have been rather serious bugs in the regex engine
> > when taint mode is on, even when dealing with untainted variables!
>
> I've never hit anything like this. Do you have examples?
http://rt.perl.org/rt2/Ticket/Display.html?id=24248
variations on the theme of
#!perl -T
{
local $ENV{PATH} = "/bin";
my $r = "foo";
$ARGV[0] =~ /($r)/;
my $c = "echo $1";
system $c;
}
__END__
http://rt.perl.org/rt2/Ticket/Display.html?id=22270
where I don't agree with any of the explainations (IIRC) and stand by the
bug. (But ran out of time to find a better explaination)
Nicholas Clark
Re: No more code coverage
On Wed, Oct 22, 2003 at 10:01:26AM +0100, Tim Bunce wrote: > I don't have time in the short term to work on the (albeit fairly > trivial) change to Storable. If someone can do that and get a new > release out then I'll look deeper into the performance issues then. In reading this thread over the course of the past few days I've missed what the needed change to Storable would be. Could you remind me what it is? Or better still submit a wishlist bug against the core. (I don't think submitting it on rt.cpan.org will mail anyone apart from Abhijit) No, I don't think that I have time to work on this either right now, but there's always next month. Nicholas Clark
[EMAIL PROTECTED]: Re: XS and strings]
Another thing to scan for Kwalitee? [probably quite hard] Nicholas Clark - Forwarded message from Marcus Holland-Moritz <[EMAIL PROTECTED]> - Mailing-List: contact [EMAIL PROTECTED]; run by ezmlm Precedence: bulk list-help: <mailto:[EMAIL PROTECTED]> list-unsubscribe: <mailto:[EMAIL PROTECTED]> list-post: <mailto:[EMAIL PROTECTED]> Delivered-To: mailing list [EMAIL PROTECTED] Delivered-To: [EMAIL PROTECTED] X-Authenticated: #13357003 Date: Fri, 2 Jan 2004 17:36:27 +0100 From: Marcus Holland-Moritz <[EMAIL PROTECTED]> To: Alberto Manuel Brandão Simões <[EMAIL PROTECTED]> Cc: [EMAIL PROTECTED] Subject: Re: XS and strings In-Reply-To: <[EMAIL PROTECTED]> X-Mailer: Sylpheed version 0.9.7claws (GTK+ 1.2.10; i686-pc-linux-gnu) X-Spam-Check-By: one.develooper.com X-Spam-Status: No, hits=-1.0 required=7.0 tests=CARRIAGE_RETURNS,IN_REP_TO,QUOTED_EMAIL_TEXT,REFERENCES,SPAM_PHRASE_00_01 version=2.44 X-SMTPD: qpsmtpd/0.26, http://develooper.com/code/qpsmtpd/ On 2004-01-02, at 12:51:04 +, Alberto Manuel Brandão Simões wrote: > On Thu, 2004-01-01 at 21:50, Marcus Holland-Moritz wrote: > > First, don't pass NULL as RETVAL. This looks to me like you want to > > return an undefined value. Use XSRETURN_UNDEF instead. > > True. I was browsing the CPAN yesterday. A lot of people seem to assume that it's valid to return NULL as an AV*. It isn't. It even causes a segmentation violation. -- Power corrupts. And atomic power corrupts atomically. - End forwarded message -
Re: testers.cpan.org ideas
On Tue, Mar 09, 2004 at 01:14:10PM +, Leon Brocard wrote: > Does anyone have any features they'd like to see on the website? I'm > looking at extracting more information (Perl version, platform) and > having pages (and thus RSS) per author. Version would certainly be a good thing to add. Most of the other significant compile time options are summariesed by the architecture string. Nicholas Clark
Re: Devel::Cover - require 5.8?
On Sat, Apr 03, 2004 at 03:04:49PM -0800, Ovid wrote: > 2. Hash ordering is now guaranteed to be random (unless you specify different > compile time > options to Perl, something I chose not to do). Many of our tests assume that hash > order is > deterministic -- which it appeared to be prior to 58 (remember, "not guaranteed to > return in a > particular order" is not the same as "non-deterministic"). I had to rewrite much of > our test > suite (which takes around two hours to run!) to find and fix all of these. 5.8.2 and later have hashes that are deterministic until the data is pathological. And the pathological checks kick in very late. Only 5.8.1 has true random hashes (not 5.8.0) and even without compile time changes (IIRC) you can always force deterministic hashes with environment variables. > 5. Localizing tied arrays and hashes is broken in 5.8. We tie and localize %ENV. I wasn't aware of this. Or at least, it doesn't ring a bell. Is perl5-porters aware of this? Is this a known change in behaviour? Not that this invalidates your reasons for needing to stick to 5.6.x Nicholas Clark
Re: My Phalanx lightning talk
On Thu, Jun 10, 2004 at 09:27:09AM -0500, Andy Lester wrote: > So I'm giving an LT on Phalanx next week in Buffalo. Anything in > particular y'all think I should mention? > A brief discussion of Phalanx, a project devoted to increasing the > testing coverage of Perl and the key modules on the CPAN. I'll discuss > where we're at, and call for participation from interested parties. Well, how people acn help, particularly how they can help even if they don't think they have time to be *the* one true hoplite for a project. But you were probably going to tell them that already. I mentioned Phalanx as part of the ponie talk I did at les journées perl (the French workshop) but I wasn't sure of the best way people should go about approaching the project to help, and specifically how to lure people in by saying that they don't have to do very much. (ie writing a couple of tests is better than not, if you don't feel able to take on a whole module) Nicholas Clark
Re: C/C++ White-Box Unit Testing and Test::More
On Fri, Jun 25, 2004 at 02:18:49PM -0500, Andy Lester wrote: > Tests are all about quantity. I always thought that tests were about malice: "I bet the programmer didn't think of this..." "What happens if I just do this..." "Mmm, I wonder if it covers this corner case?" "Eat pathological data and die!" "Surrender feeble routine. You are no match for the weaponry of my tests!" Then again, I do like having lots of tests. And parameterising tests so that I can feed the same basic test lots of different permutations of the data. [To attempt to send the most variations of malice, in the hope that one of them knocks out the routine and finds a bug] Nicholas Clark
Re: CPANTS preview
On Thu, Jul 22, 2004 at 04:28:08PM +0200, Thomas Klausner wrote: > Hi! > > I ran CPANTS today, you can view results here: Oooh. Nice > http://cpants.dev.zsi.at/ > ( or http://test1.dev.zsi.at if DNS isn't updated..) > > in /metrics there is a yaml-file for each dist > cpants.db (or cpants.db.gz) is a SQLite DB file > > Oh, there might be a bug, because 100 dists didn't get any kwalitee when at > least some of them should... > > Max Kwalitee is 10, which is reached by 99 dists. Will it go up to eleven soon? :-) > is_prereq (is listed as PREREQ by at least 3 other dists) Is it possible to get the lists of prereqs for each module, so as to build a citation index? (without the need to do all the messy downloading or parsing - being lazy I'd like to leave that to you) Or should I download the generator and run it myself? Nicholas Clark
Re: Test::Builder versus Unicode
On Wed, Dec 22, 2004 at 10:26:02AM -0800, David Wheeler wrote: > 1. Perl gets smarter about duping file handles, so that the dupes get > the same i/o layer settings as the handles they dupe. Changing this going forwards doesn't change any of the installed perls out there in the wild. So whatever happens in the future, if we want things to work on existing installations, we need to work round the problem in some way. (I don't currently have time to look into or think about whether it's a bug. Or how to fix it if it is. Or who could fix it.) Nicholas Clark
Re: Test::Builder versus Unicode
On Wed, Dec 22, 2004 at 11:41:56AM -0800, Ovid wrote: > --- Nicholas Clark <[EMAIL PROTECTED]> wrote: > > > On Wed, Dec 22, 2004 at 10:26:02AM -0800, David Wheeler wrote: > > > 1. Perl gets smarter about duping file handles, so that the dupes > > get > > > the same i/o layer settings as the handles they dupe. > > > > Changing this going forwards doesn't change any of the installed > > perls out there in the wild. > > So far, given that this problem has only surfaced in relation to > Unicode, I can't say I'm overly concerned about fixing it on versions > of Perl where Unicode is already known to be broken. Of course, as Which you're sort of implying is all versions up to and including 5.8.6 :-) (Well, I can misread it as that. I don't think that you really are implying this). Personally I'd be quite happy using anything 5.8.3 or later with Unicode. The later the better, as more bugs have got fixed. But I feel that Unicode isn't more broken than any other existing part of perl by 5.8.1. And they are out there, and they aren't going away rapidly. Nicholas Clark
Re: Anomalous Difference in Output between HTML Files Created by
On Mon, Jan 31, 2005 at 10:12:16AM +0100, Paul Johnson wrote: > I suppose that's the price you pay for TIMTOWTDI. > > [ Is that a Python programmer I hear giggling in the background? ] Does Python have any equivalent tool to Devel::Cover? Nicholas Clark
Re: Test comments
On Mon, Feb 14, 2005 at 10:04:44AM -0800, Ovid wrote: > Is this not correct? Where is the TAP protocol documented? http://search.cpan.org/~petdance/Test-Harness-2.46/lib/Test/Harness/TAP.pod (Any Test-Harness distribution 2.46 or later, IIRC) Nicholas Clark
Re: CPAN modules coverage
On Mon, Mar 07, 2005 at 07:59:40PM +0100, Paul Johnson wrote: > To do it properly it would need to be on a machine somewhere which would > accept uploaded coverage databases from anyone who wanted to submit one. > > I discussed cover.perl.org or something with Andy and Robrt on irc a > while back, but I've not made progress on the sw side. Because your free time is already filled up with working on Devel::Cover itself? This sounds like the sort of project that anyone who felt particularly keen about could try experimenting with at home, to see if they can get it to work. Having a demonstrable experiment is a very good first step to convincing others to help you with it, and having something that is mostly finished is a good way to get someone else to host a fully working version. Actions speak louder than words. Nicholas Clark
Re: [RFC] Test::CPANTS
On Mon, Mar 14, 2005 at 04:05:36PM -0500, Christopher H. Laco wrote: > I have an itch. It just came to me while surfing PerlMonks and CPAN. > I noticed the new Test::Strict module which keeps me honest by making > sure I always 'use strict'. I'll be adding that to my modules this > evening. [I wish is did use warnings too]. Acme::use::strict::with::pride does warnings too :-) http://search.cpan.org/dist/Acme-use-strict-with-pride/pride.pm I'm not suggesting that you use it in production code though. Partly because it is an Acme module, but mainly because it stops Exporter working. Then again, OO purists might consider that a feature. Nicholas Clark
Re: like, cmp_ok and undef
On Fri, Jul 01, 2005 at 07:26:16PM -0700, Michael G Schwern wrote: > It seems pretty clear to me that most warnings should not be supressed. > But what about uninit warnings, specificly? Is there any utility to > something like: > > like( $foo, qr/.../ ); > > and: > > cmp_ok( $foo, 'eq', $bar ); > > warning about $foo being undefined? Especially since the test diagnostics > will let you know about what was undef. I'm not sure. For the specific example of cmp_ok with 'eq', then because the programmer has chosen that over is(), it seems fair that he/she made an explicit choice that $foo being undef was an acceptable match for $bar being an empty string. (Not so sure about the other 13 comparison operators) I think I'd prefer like to fail (distinct from warning) if $foo is undef, given that the expected regexp may well intentionally match an empty string. Nicholas Clark
Re: False Positives from Automated Testing at testers.cpan.org
On Tue, Jul 19, 2005 at 09:37:39PM -0700, Michael G Schwern wrote: > On Tue, Jul 19, 2005 at 10:49:12PM -0400, James E Keenan wrote: > > Am I correct in this inference and this judgment? Or is there something > > about the automated testing that I don't understand? > > If I understand correctly, the issue here is you failed to list a > dependency yet the tests failed to catch it because that module was already > installed. Its a false positive, but not testers.cpan.org's fault. It > would have to attempt to automatically figure out what modules your code > needs and check that you list them as dependencies. This is fraut with > peril. When I rule the world (and therefore have requisitioned everyone's tuits) there will be smoke testing that starts from a clean install each time. However, right now I don't have time to try to write this. Sorry. Nicholas Clark
Re: OSCON testing tutorial?
On Wed, Jul 20, 2005 at 02:46:43PM -0500, Andy Lester wrote: > On Wed, Jul 20, 2005 at 11:57:36AM -0700, Michael G Schwern ([EMAIL > PROTECTED]) wrote: > > Am I imagining things or was there supposed to be a testing tutorial at > > OSCON > > with Andy Lester, chromatic and the gang? Or am I thinking of YAPC? > > No, it's not a tutorial. It's 90 minutes of random testing stuff. > Think 16 testing lightning talks. A big grab bag of testing excitement. > It's me, Bill Odom, Ian Langworth and chromatic, the latter two pushing > their new "Perl Testing: A Developer's Notebook". Because Andy would be far too polite to push the book he and Richard Foley wrote: http://www.apress.com/book/bookDisplay.html?bID=399 Nicholas Clark
Re: OSCON testing tutorial?
On Wed, Jul 20, 2005 at 02:52:40PM -0500, Andy Lester wrote: > On Wed, Jul 20, 2005 at 08:49:11PM +0100, Nicholas Clark ([EMAIL PROTECTED]) > wrote: > > Because Andy would be far too polite to push the book he and Richard Foley > > wrote: http://www.apress.com/book/bookDisplay.html?bID=399 > > That's because I didn't write it. I'm more a uber-tech-editor and > helped shape some of the prose, but none of it is really my writing. > That's why it's "with Andy Lester." > > Apress asked me to sign books at the book signing at OSCON and I > declined. It's really Richard's book. Well, "Andy far too polite", so at least I was 50% correct. :-) Nicholas Clark
Re: kwalitee: drop Acme?
On Thu, Sep 08, 2005 at 11:20:33PM +0200, Philippe 'BooK' Bruhat wrote: > Le jeudi 08 septembre 2005 à 10:45, Ovid écrivait: > Take a less random example: > > Acme::MetaSyntactic You've been learning this new definition of "random" from cog, haven't you? :-) [eg "random person to wear fishnet"] > > We should at least throw the poor module author's a bone and leave > > Acme:: out of this. > > Hey! that would decrease *my* score! ;-) I've no idea what it would do to my score. Nicholas Clark
Re: kwalitee: drop Acme?
On Fri, Sep 09, 2005 at 10:37:01PM +1000, Adam Kennedy wrote: > Or make is_prereq SO easy to game that it's a nonissue? Why should a > module depended upon by another author be ranked any higher than one > that isn't. For CPANTS I see no reason. For search results quite the opposite. I'd really like if if the default way people got search results back for CPAN modules at least attempted to order at some level based on citations. (ie number of pre-requisites) For example, I believe that that way the 4 or so decent template modules would fall out immediately from the 50 others on CPAN. And that this is worth doing because the commonly used templating modules are almost certainly what most people who need to search would find more than good enough. Rather than letting them fight through bogosorted results and getting swamped by choice. But it seems that there is something amiss with the roundness of my stock of tuits. (or in other words there are things I find I care more about and deal with first) Nicholas Clark
Re: Testing module madness
On Sun, Sep 11, 2005 at 12:35:43PM -0500, Andy Lester wrote: > >Usually, Test::* modules are only used for the test phase. > > I really don't understand the idea of "only used for the test phase", > as if the tests don't matter, or if there are levels of failure. > Either they install OK on the target system, and you can use them > with confidence, and they've done their job, or you're going to > ignore the tests completely and then who needs 'em? > > It's like if I'm installing a washing machine, and I don't have a > level. I can say "Ah, I only need it for the installation, and it > looks pretty level, so I don't need the level", or I can say "I'm not > using this appliance until I've proven to myself that the machine is > level and won't cause me any problems in the future because of an > imbalance." This is a good analogy. It's correct. But the assumptions behind it cover only one case. It's as if the requirements for the washing machine say: To install and use this machine you will need: * a power supply * a water supply * drainage * a level which is valid if you're both the installer and the user. But if someone else helps you install the machine, then you don't actually need the level, if they bring theirs and use it for the install. I think that the build_requires/test_requires distinction *is* important, if it can be made, as it eases the lives of anyone wishing to package up modules, build them from source in one place, and then distribute their packages to many other machines, be they OS vendors or sysadmins. The tests are run and pass on the build machine, prior to packaging. But the automatic dependency system doesn't need to make installation of this module depend on installing Test::* onto the production machine. (for the general case) But it's only important if it's easy to make. And I'd much prefer time and effort to go into writing better modules, better tests, and better tools, than generating heat. Nicholas Clark
Re: ENV problems with testing
On Mon, Sep 19, 2005 at 09:42:52PM -0400, James E Keenan wrote: > (Secondary reason for tests needing 5.6: as you, Schwern, pointed out > several weeks back, doing without qr// is just too painful!) 5.005 introduced qr// $ ~/Reference/5.005_04/bin/perl5.00504-32 -le '$r = qr/(p...)/; $^X =~ $r; print $1' perl Nicholas Clark
Re: Flexible testing
On Wed, Dec 21, 2005 at 01:34:39PM -0800, Joe McMahon wrote: > print < 1..1 > not ok 1 ... failed once > not ok 1 ... failed twice > ok 1 ... worked > EOS > Obviously, you can wrap up the actual test in a retry loop/function, > but this doesn't match up with the simplicity of Test::More and related > testing methods. It seems like the only way to address this is to > subclass Test::Builder (or write a new class) that buffers up the test > output and only outputs it after tests are "committed" (i.e., "I've run > this N times and am sticking with the final result"). > > Or am I stretching TAP too far? Thoughts? To me the cleanest way to do this ought to be to subclass Test::Builder to give the behaviour that it only prints the last result line for a given test number when it sees the first result line for the next test number appears. That would mean that the body of your test script would still call all the regular ok/is/like etc functions without changing them, and your test script still outputs regular TAP where each numbered test reports exactly once. Nicholas Clark
Re: Kwalitee in your dependencies (was CPAN Upload: etc etc)
On Fri, Jan 27, 2006 at 11:50:49PM -0800, chromatic wrote:
> Let me save you the trouble of writing it to find the biggest problem right
> now: fairly broken automated testing systems that can't even *run* the
> Build.PL file *or* the compatibility Makefile.PL yet send FAIL reports
> anyway.
I believe that the following won't be a problem with PITA, but the general
smoke system really $expletive annoys me for sending FAIL reports when it
tries to build one of my XS modules on a system without a C compiler.
My opinion is that this is a BUG IN THE SMOKE SYSTEM. The default for perl
is source distribution, so to build perl it's inferred you need a C compiler.
MakeMaker and Module build already automatically handle converting any .xs or
.c files that I ship into installable modules. When creating my package, I
don't need to do anything - it *already just works*. It's already supported.
So my opinion is that it's not necessary for me to manually add to my metadata
saying "I need a C compiler". If my module ships with .c or .xs files, it's
BLOODY OBVIOUS.
So the smoke system should verify whether $Config::Config{cc} exists and can
run, and if it can't, N/A any package that isn't pure perl.
Yes, I really like valid smoke reports, pass or fail.
But FALSE NEGATIVES ARE PROCESS ERRORS.
Nicholas Clark
Re: Dependency trees was: CPAN Upload: D/DO/DOMM/Module-CPANTS-Analyse-0.5.tar.gz
On Sat, Jan 28, 2006 at 09:08:48PM +1100, Adam Kennedy wrote: > Graph::Easy installation failing here with YAML 0.50 (newer versions of > YAML seem to be uninstallable at the moment due to Class::Spiffy + > Spiffy + Test::Base install failures... > > Any suggestions? You're getting install failures even with the newest versions of Class::Spiffy etc? I had failures with YAML not liking an existing installed Spiffy, but upgrading everything seemed to resolve that. If that's not it, then I don't know, but Ingy is often around on IRC. Nicholas Clark
Re: Kwalitee in your dependencies (was CPAN Upload: etc etc)
On Fri, Feb 03, 2006 at 11:30:13AM +0100, Tels wrote: > Problaby just because the last guy running RISC OS has died 4 years ago. > SCNR :-) Well, the list is *slightly* more active than that: http://www.nntp.perl.org/group/perl.riscos Nicholas Clark
Re: New kwalitee metric - eg/ directory
On Tue, Mar 14, 2006 at 12:48:52PM -0500, David Golden wrote: > Adam Kennedy wrote: > >For all those component distributions I consider it a failure if it is > >so complex that you need something more than just three or four lines > >from the SYNOPSIS. > > Maybe there should be a Kwalitee metric for the length of the synopsis? I was thinking this too. Some synopses aren't. Nicholas Clark
Re: Module requirements (was: Module::Build and installing in non-standardlocations)
On Tue, Apr 04, 2006 at 11:27:07AM +0200, Sébastien Aperghis-Tramoni wrote: > I don't think that the problem of "core is too big" is a matter of disk > size, but more a matter of number of modules. P5Porters time is a scarce > ressource, and they already lack the time to do all the work they'd > like to do just on the interpreter. Making core modules dual-life is > a way to handle these to someone else who has spare time and who doesn't > need to have deep XS or Perl guts knowledge. At least I think that's > the reason, otherwise why was I accepted as the maintainer of two > such modules (XSLoader and Sys::Syslog)? Well, I think it's that reason too. :-) Yes, to me, "size" is maintainance liability, not disk space or bandwidth. Putting things in core is a pain. Keeping them there is a pain. I remember the "fun" of getting Storable sufficiently portable that it could go into the core. Trying to work around strange issues thrown up by certain AIX compilers in certain configurations... Nicholas Clark
TODO tests
Last time I checked the core has "6 TESTS UNEXPECTEDLY SUCCEEDED" What's the expected number of unexpected successes? Can it be made to be zero, even though we're testing the test modules? If so, I think that that would be useful, as it would mean that any (real) TODO test that unexpectedly started passing would be noticed. I bring this up because we seem to have inadvertently fixed really old regexp bugs that we didn't have a test case for, but I realise that right now adding TODO tests wouldn't actually have been *that* useful - if a TODO passes we don't notice. It would be good if we were in a position to notice. I'm not sure how much work that would be. Nicholas Clark
Re: TODO tests
On Tue, Apr 18, 2006 at 01:21:37PM -0500, Steve Peters wrote: > One of my unwritten TODOs is to go through the current Perlbug queue and > write test cases for all the currently testable problems. My hope is > that unexpected fixes would be caught much sooner in these cases. I've > made a bit of a start on this, and given tuits (or proper financial > motivation) I'm hoping to finish it sometime this year. Of course, The is an application form for motivation: http://www.perlfoundation.org/gc/grants/proposals.html Beware. Motivation is a double edged sword. It means everyone knows that you're going to do stuff. Nicholas Clark
Re: Test me please: P/PE/PETDANCE/Test-Harness-2.57_06.tar.gz
On Sun, Apr 23, 2006 at 01:02:00AM -0500, Andy Lester wrote:
> I'm approaching the end of this release cycle. I really want to get
> this released.
>
> I've removed the meaningless percentages of tests that have failed.
> If you rely on the output at the end, it's different now.
>
> xoa
> >
> > file: $CPAN/authors/id/P/PE/PETDANCE/Test-Harness-2.57_06.tar.gz
> > size: 69114 bytes
> > md5: 41efc0985146e4f7d678ec7cb9b59047
I have merged this in to blead with change 27940. (Because I want the speedup)
Specifically blead is now identical (Unless I messed up) which means that
this change got removed before I committed:
--- Test-Harness-2.57_06/lib/Test/Harness.pmSun Apr 23 06:53:11 2006
+++ perl/lib/Test/Harness.pmSun Apr 23 21:23:30 2006
@@ -38,7 +38,8 @@
=cut
-$VERSION = '2.57_06';
+$VERSION = "2.57_06";
+$VERSION = eval $VERSION;
# Backwards compatibility for exportable variable names.
*verbose = *Verbose;
And this change did get committed:
//depot/perl/lib/Test/Harness/t/version.t#2 -
/home/nick/p4perl/perl/lib/Test/Harness/t/version.t
--- /tmp/tmp.27775.0Sun Apr 23 21:25:55 2006
+++ /home/nick/p4perl/perl/lib/Test/Harness/t/version.t Sun Apr 23 21:23:28 2006
@@ -19,5 +19,5 @@ BEGIN {
}
my $ver = $ENV{HARNESS_VERSION} or die "HARNESS_VERSION not set";
-like( $ver, qr/^2.\d\d(_?\d\d)?$/, "Version is proper format" );
+ok( $ver =~ /^2.\d\d(_\d\d)?$/, "Version is proper format" );
is( $ver, $Test::Harness::VERSION );
Note that I removed the ? from the _ inside the regexp, which was added
in http://public.activestate.com/cgi-bin/perlbrowse?patch=27925
I felt it better to get the code perfectly in sync than to preserve minor
recent (possible) fixes and have to keep track of the fork.
Nicholas Clark
Re: Test::Harness wrangling
On Wed, Jun 28, 2006 at 11:13:02PM -0500, Andy Lester wrote: > Tomorrow, Adam Kennedy and I (and Schwern?) will be banging on > Test::Harness. > > Any bugs that we especially need to work on? They're not so much bugs as structural annoyances that it would be useful to relax to make subclassing Test::Harness more useful. (Well, most accurately, to allow me to write a parallel test runner easily) I hacked Test::Harness and Test::Harness::Straps around to get parallel tests running (version 1), but I made no effort to keep the conventional behaviour working cleanly, hence I'm not sure if sending you that code would be useful. However, based on that, things that it would be nice to refactor out (from memory): The code to do the printing of the test names, the ...s and the numbers could be refactored out into a proper helper class, to allow derivatives more flexibility in what the chose to display. This class would suck in the subroutines associated with &strap_callback as methods. In turn, it would stop the evil storing of callback state in the main T::H object. In turn, the globals $ML $Last_ML_Print can be eliminated and stored as state in this. Nicholas Clark
Re: TAP::Harness
On Sat, Jul 01, 2006 at 02:36:37PM -0500, Michael G Schwern wrote: > Right now the use cases I have in mind include things such as > parallelized test runs, fancy GUI and HTML outputs (for example, > Provide use cases, what would you want to do with Test::Harness if you > could? What are you doing with Straps? What features do other Use case: Parallel test runs I'd like to be able to specify parallel test runs. Specifically, I'd like to be able to send tests as a set of groups, with instructions on how the tests are to be parallelised. I already have a prototype, hacked from Test::Harness. 1: Send tests as groups 2: Tests within a group run in series (in traditional order) 3: Groups run in parallel 4: have sync points. All groups before a sync point must complete before any groups after a sync point can start. This let me run many of the core tests in parallel, but still allowed the tests that need to run in order to run in order. This level of control let me run the core tests as follows: t/base/*.t one by one (each test is its own group, sync point between each group) t/comp/*.t t/cmd/*.t t/run/*.t t/io/*.t t/op/*.t tests in each directory in sequence, but the 5 directories in parallel wait for all these to finish t/uni/*.t, all tests in ext and lib tests in a the same directory ran in series. All directories could run in parallel With this I can max out a multi CPU (or multi core) box. (Or at least stand a fighting chance, although tests in t/op/*.t in the core need some issues fixing before they can run in parallel with each other. I suspect that their names for temporary files are unimaginative and therefore clash) I've got the code for a select driven event loop that can run all this, if that's helpful. Nicholas Clark
Re: Proposal Suggestion - Test::Run
On Wed, Jul 05, 2006 at 11:57:42PM -0700, chromatic wrote: > On Wednesday 05 July 2006 23:02, Shlomi Fish n wrote: > > > I don't see using the X11 licence for my software as anti-social. Like I > > said, anyone can easily fork it as a software of a different licence. > > Supposing you actually find a mentee and TPF actually does fund this project > and this code is substantially better than Test::Harness such that it is an > obvious candidate for inclusion in the Perl core. > > It must be under the Perl license. For the core, what he says. Period. Nicholas Clark
Re: TAP Namespace Nonproliferation Treaty
On Sat, Jul 08, 2006 at 05:10:50AM -0700, Ovid wrote: > - Original Message > > From: Nik Clayton <[EMAIL PROTECTED]> > > TAPxParser > > > Thought about that, but immediately discarded it. TAPx::OVID::Parser doesn't > say anything about the parser other than authorship and the latter is > verified by glancing at the docs. My jumping on the TAPx::Parser namespace > is almost as bad as my taking TAP::Parser, so I'm just trying to figure out > how to truly distinguish my (currently incomplete) parser from others. I > suspect it will be as arbitrary as your idea, though :) Also naming modules with the original author's CPANID is going to become "fun" if maintainership is transferred, or the original author wants to start a clean second implementation. Nicholas Clark
Re: Time for a Revolution
On Sat, Jul 15, 2006 at 08:36:54AM +1000, Andrew Savige wrote: > --- Clayton O'Neill wrote: > > I think a core difference between your list and Chromatic's is that > > yours would be part of the standard library in a lot of languages, > > whereas Chromatic seems to be aiming more for things that would be > > part of the language. > > Who's Chromatic? And it wasn't even the start of a sentence. :-) [When doing the perl 6 summaries, Piers reconciled the forces of accuracy and traditional grammar by ensuring by always rephrasing to some sentence order that didn't start with "chromatic"] Nicholas Clark
Re: planning at the end
On Thu, Jul 20, 2006 at 04:01:00PM -0700, Michael G Schwern wrote: > On 7/20/06, Gabor Szabo <[EMAIL PROTECTED]> wrote: > >If I am not mistaken the problem with no_plan is that the test script > >might exit before actually running all the tests you wanted and Harness > >won't notice it. > > PS In all my years of testing I have never been bitten by this in the > real world which is why I've never bothered to code up a solution. > Anyone else? I've been bitten by library code calling exit() while I've been using no_plan. I can't remember more than that (well, apart from that library wasn't pleasant code to read or use). I would have found some sort of "done testing" call useful to have made the test script fail for this case. Nicholas Clark
unit tests or functional tests
This is sort of off-topic because it's more a general question about testing, rather than Perl specific, but the code in question happens to be written in Perl... There is this big hairball of under-tested code. (Nothing new here) So the question is, which to tackle first - unit tests, or functional tests. A colleague's view is that you can't have functional tests until you know that the individual units work, hence start with unit tests. (I believe that the assumption is that when they're mostly complete start on functional tests, but that wasn't stated). This seems the logical approach if you want to refactor things. My view is that because the actual output of the code isn't well specified (sadly nothing that new there either), if we write functional tests to verify that the behaviour we desire is present, then we're actually killing two birds with one stone - we have tests for the spec, and the tests are the spec. (Which isn't perfect as specs go, but it's a heck of a lot better than the current spec). Also, right now we really don't care about the specific output of the the individual units that make up the code - all that matters to the client is the final behaviour. Hence writing unit tests at fine detail for existing code could well be a (relative) waste of effort in that it's quite possible that the units they test are thrown away soon if the implementation is changed. Whereas functional requirements are much less likely to change on a release-by-release basis, so writing them is less likely to generate code that has a short lifetime. And having functional tests is likely to give us better coverage up front, so we're more likely to spot a change that unintentionally breaks behaviour, even if we can't use them to efficiently nail down which change was the culprit. Why am I wrong? Nicholas Clark
Re: designing a test suite for multiple implementations
On Sat, Aug 12, 2006 at 03:32:47PM +0800, Audrey Tang wrote: > > ?b 2006/8/12 ?W?? 3:01 ???Ajerry gay ?g???G > > >the question is, should it be moved into their own repository, or > >into the repo of > >the "official" perl6 implementation (if such a beast will indeed > >exist,) > > Currently the svn.perl.org repo is the most official-sounding one, by > the domain > name alone. (But I don't understand the motivation for putting the > tests with > "the true implementation" -- I thought the idea is to decouple the > tests with > any implementations.) So I think svn.perl.org is the right choice, > if the admins > are okay with a more relaxed commit bit policy there. There's nothing technical stopping the Perl 6 tests being on svn.perl.org, but in a different svn repository from the current repositories, is there? Setting a svn:external property in the right place on both Parrot and Pugs would mean that both could check out the same testsuite, and both could commit back to it. Nicholas Clark
Re: designing a test suite for multiple implementations
On Sat, Aug 12, 2006 at 09:44:38PM +0800, Audrey Tang wrote: > That's assuming that the new repo, say, http://svn.perl.org/ > perl6tests/, can > give out commit permissions to parrot and pugs committers, yes. > > But as Jerry's initial motivation was moving Perl 6 tests to a more > official > location, and that http://svn.perl.org/perl6/ is the official repo > for the > Perl 6 design documents, I wonder what's the advantage of hosting the > tests > in a separate repository. Can you elaborate on that? I thought that it might be easier to audit different committer bit policies if the repositories were different, instead of simply one being a subtree of another. That was all. Nicholas Clark
Re: designing a test suite for multiple implementations (tools thread)
On Mon, Aug 14, 2006 at 07:12:06PM -0700, chromatic wrote: > PS - sbk30, please don't send me any more automated followup messages. Fix > your mailing software. I've found that our resident neighbourhood BOFHs have been very helpful at forcibly un-subscribing anyone anti-social enough to be sending automated crap in response to mailing list messages. (Given the full headers of the offensive message. BOFH contact details being in the message headers of any message distributed by the list) Nicholas Clark
Re: post-YAPC::Europe CPANTS news
On Fri, Sep 08, 2006 at 09:15:47PM +1000, Adam Kennedy wrote: > One of the linux distro guys pinged me about Test::Object needing the > very latest (CPAN-only) version of Test::Builder, because it means they > can't package it properly for the distros without upgrading the main > Perl package. Some packaging systems can't handle having the same file > in more than one distro it seems. Said maintainers can work round the deficiencies of their packaging systems by breaking out all the dual life modules from the core Perl package, and making them into other packages on which the core Perl package depends. In my opinion. Clearly this might be a bit tricky if the packager of Test::Object isn't the same person as the packager of core Perl, because then the former has to persuade the latter of the need for such a change. Alternatively said maintainers can work round the deficiencies of their packaging system by having another directory tree in @INC ahead of the core library paths, into which they install newer versions of dual life modules. This way no files need to be overwritten on disk, or dual-owned, yet Perl will still pick up the correct (newer) version. Nicholas Clark
Re: New TAP Grammar
On Thu, Sep 14, 2006 at 09:28:19PM -0400, Michael G Schwern wrote: > Ovid wrote: > > Also, for purposes of backwards compatability, I'm concerned about qr// and > > $code->(). > > When were those first introduced in Perl? I don't want someone using > > really old versions to not be able to use this code. > > qr// was introduced in 5.6. $code->() has been in there as long as I know. > "use warnings" is 5.6. > > To give you an idea, the Test-Simple distribution has been broken on 5.5.3 > for a few releases now (Test::Builder::Tester uses qr//) and I've gotten a > grand total of one bug report about it. I think that it came in with 5.005_00. Certainly, it's in 5.005_03, which is the earliest version I can find precompiled: $ perl5.00503 -le '$a = qr/(.*)/; $] =~ $a; print $1' 5.00503 So whatever the actual cause of the bug leading to the report is, it wouldn't seem to be what the reporter thought it was. Nicholas Clark
Re: Perl::MinimumVersoin (was Re: New TAP Grammar)
On Sat, Sep 16, 2006 at 09:27:21AM -0400, Michael G Schwern wrote: > Seems to me this should be an config option to decide between checking the > point where a feature existed and when it stopped sucking. That sounds like a useful distinction. Hopefully for most features "stopped sucking" won't be vague (due to no clear single release where it became workable) Nicholas Clark
Re: Bad TAP in Perl core?
On Sun, Sep 17, 2006 at 08:39:12AM -0700, Ovid wrote: > So I now have well over 2,000 tests for the parser, but there's a corner case > that's killing me. Test::Harness handles it fine, so I'm thinking that the > TAP specification (and subsequent grammar) is wrong. > Because TAP forbids unescaped hash marks in descriptions and that's how my > parser works, it falls down and goes boom. Unless I hear differently, I'm > going to have to adjust my parser to allow them. Otherwise I can't even > parse the Perl core tests, much less many different test modules on the CPAN. > This is because many people are putting their descriptions after hash marks, > not before. We could just fix the core. Nicholas Clark
Re: A short rant on the purpose of the CPAN install chain.
On Fri, Sep 22, 2006 at 09:47:02PM +0100, Adrian Howard wrote: > Belated response. For me good tests suites: > > * are good code (i.e. DRY, well factored, intention revealing, etc.) > * encourage good design (ala TDD) > * fail for bugs, but not refactoring > * only fail for bugs in the thing they're testing Which includes never failing due to random causes. (Such as there being a known 5% chance of a test failing) > * don't depend on the order the tests are run in Which to me means both 1: No .t file relies on others running before them to set up 2: Any .t file can be re-run infinite times without starting to fail (before 1 is achieved, due to some sort of set up data being exhausted) Nicholas Clark
Re: AnnoCPAN Doc Patch Maker
On Sat, Oct 07, 2006 at 12:16:58AM -0400, Ivan Tubert-Brohman wrote:
> On 10/6/06, Ivan Tubert-Brohman <[EMAIL PROTECTED]> wrote:
> >I got something that can send you directly to rt.cpan.org and prefill
> >the textarea with the patch. The problem is that that textarea has
> >hard text wrapping which can cause the patch to be malformed.
> >Therefore, to make life easier for the module maintainer, the patch
> >should be attached as a file. The sad part for the patcher is that the
> >file can't be transmitted easily from one form to the other. The best
> >protocol I could come up with is the following:
> I'm thinking now that a convenient solution for the user would be to
> send the bug report directly by email (from annocpan.org to
> rt.perl.org, with From: being the user's email); that way I can
> include an attachment so the patch won't be messed up. Some drawbacks,
> if I understand RT correctly, are that the report would not be
> associated with an rt.cpan.org user account and the user won't be able
> to set some of the metadata (such as version numbers).
Given that {annocpan,rt}.perl.org both know who the user is, and both are
written in Perl, isn't there some web API available from RT that would let
an authenticated machine file a bug report with unmangled patch?
Nicholas Clark
Re: Failing test on Windows
On Tue, Oct 24, 2006 at 08:08:45PM -0400, Christopher H. Laco wrote: > With most modules, I agree. But with utility modules like > Module::Pluggable, File::Find::Recursive, etc, not working under taint I'd be surprised if the author of Module::Pluggable wasn't open to patches to fix this. Nicholas Clark
Re: Sparse Test Output
On Wed, Oct 25, 2006 at 03:53:58PM -0700, Michael G Schwern wrote: > I am totally unconvinced. I would love to see some profiling on this. I > would hope that the act of running 50,000 tests would swamp the cost of > parsing 50,000 tests. I don't have the exact numbers any more, but had experimented with multithreading the core tests (needs some refactoring of Test::Harness to do properly) I had 1 process parsing the TAP streams, with the others running the tests and generating the TAP. Above about 7 processes in parallel the TAP parser process was the bottleneck. This was on a 32 core Sun, so I had plenty of headroom to run more TAP generating threads. Nicholas Clark
Re: Thoughts about test harness summary
On Fri, Jan 05, 2007 at 11:11:25AM -0800, Ovid wrote: > Pursuant to some discussion with BrowserUK at > http://perlmonks.org/?node_id=593087, I'm looking at this and seeing > some problems. > > -- > > Failed Test Stat Wstat Total Fail List of Failed > - > t/bar.t4 1024134 2 6-8 > t/foo.t1 256101 5 > (2 subtests UNEXPECTEDLY SUCCEEDED). > Failed 2/3 test scripts. 5/33 subtests failed. > Files=3, Tests=33, 0 wallclock secs ( 0.10 cusr + 0.01 csys = 0.11 > CPU) > Failed 2/3 test programs. 5/33 subtests failed. > > -- > > How about this instead? > > -- > > Failed Test | Total | Fail | List of Failed | TODO Passed > +---+--++ > t/bar.t | 13| 4|2, 6-8 |3-4 > +---+--++ > t/foo.t | 10| 1|5 | I'd like Wstat, even if I don't have Stat. I like to know if tests coredumped. I may be in a minority here, but being able to optionally switch to that output is useful. I like the prominence of TODO passed I'm not sure if I like the lines making the table. I guess it's a bit of a bikeshed, but having horizontal lines between each will increase the amount of vertical space needed to convey the same information, which will mean fewer failures will be needed to exceed my screen's height. Nicholas Clark
Re: Using pip to get testing done better and faster...
On Tue, Jan 09, 2007 at 02:30:56PM +, David Cantrell wrote: > Sadly, I doubt it's really possible. Perl makes far too many Unixy > assumptions. What are these "directories" of which you speak? :-) although I doubt that that one is really the clincher, as Perl has been ported to some somewhat un-Unixy portable devices. IIRC DOS didn't get directories until 1.1 Nicholas Clark
Re: Parallelizing TAPx::Harness?
On Mon, Feb 05, 2007 at 03:24:37AM +, Andy Armstrong wrote:
> On 5 Feb 2007, at 01:08, Eric Wilhelm wrote:
> >I was just thinking that since my tests are nicely orthogonal and yet
> >there's very little cpu utilization during testing, that I could
> >probably get them done about 20x faster if they were parallelized.
>
> [snip]
>
> I can't say off the top of my head what problems might arise.
> Obviously the output displayed during test runs would have to change.
> Sounds like a good idea though - I've added it to the TODO list :)
I had an experiment with making Test::Harness run in parallel. The code
exists, but is (intentionally) very hacky - I wanted to see what needed
doing.
>From memory, and I don't know how TAPx::Harness differs, currently when you
run a test under Test::Harness it does roughly the following:
1: Print out the test name with the s
2: Start the test running
3: Suck TAP data from the running test and print out progress
4: When the test completes print out its summary
5: repeat
To run tests in parallel, what I was doing was starting a farm of tests,
and when a test completed, printing out its results and starting another
one, until there were no new tests left to start, at which point wait for
tests to finish. The farm was run via a select loop.
So the logic had to change to roughly (and lets see if I can get the syntax
right):
async {
1: Start a test running
2: Suck TAP data from the test and buffer it
3: When the test completes process it all
4: Print out a summary
5: repeat
}
Important differences were
1: you only print out the entire test results as an atomic line when it has
finished
2: You have no idea which test is going to finish next
3: You can't print running progress on tests
Ideally the same code can run in both modes, so that when you aren't running
tests in parallel (which includes the case of waiting-for-the-last-test)
you go back to printing out the name and processing TAP as it appears.
Something I discovered was that processing TAP is slow. Up to about 7 CPUs,
parallelising tests feeding TAP into the master "thread" was a win. Beyond
that, the process processing the TAP was the bottleneck, and adding more
CPUs (and more parallel tests) didn't help. I would assume that from that
point on it would be better to have each test run by slave harness processes,
which takes the TAP, parses it, and slings (say) Storable objects across
to the master. But right now this isn't an issue as the 32 core Sun isn't
powered up. :-(
For the actual running of tests I tweaked runtests() so that the list of
arguments was treated as follows
1: Plain file names were tests to run in parallel, in any order
2: Array references were groups of tests to run in series
(This let me run tests for each module in ext/ and lib/ in order)
3: There was a special sync object that meant "complete everything up to here"
which was sufficient to give a lot of flexibility in how tests were run
(and discover that the core is very unoriginal in its naming of temporary
files)
Nicholas Clark
Re: Test::Simple API incompatibility - cpan-testers FAIL results
On Fri, Feb 16, 2007 at 09:17:00AM +1100, Adam Kennedy wrote: > >This comes from the OMGYOUBROKECPAN! diagnostics formatting change in > >0.60_01. > > Can we use that as the official name for this sort of error? > > Because the other alternative would be "Doing a Schwern" and I I'm sure > you'd prefer to preserve that term for something positive :) Such as herding MakeMaker for years with no-one ever saying thanks. Thanks, Schwern. (although I'm feeling a bit false as I'm not sure how much of that is simply because "I'm glad that it wasn't me who had to do it") Nicholas Clark
Re: Paying for TAP 2.0
On Thu, Mar 08, 2007 at 11:05:31AM -0500, Christopher H. Laco wrote: > Paul Beckingham wrote: > > Because I need to retain the output of all tests, and those files get > > large, but mostly because of the sheer redundancy. > > > > > > > > But how do you know "23 ok" if you were never told that it ran ok? Surely one can post-process a regular TAP file to "sparse" output? And only do so if the TAP file is valid non-sparse output. This seems safer than generating it by default. Nicholas Clark
Re: [ANNOUNCE] Test::More/Simple/Builder 0.68
Erk. Fails big time on 5.005_03
(Sorry, only just had reason to want to build 5.005_03)
On Tue, Mar 13, 2007 at 05:34:48PM -0700, Michael G Schwern wrote:
> 0.68 Tue Mar 13 17:27:26 PDT 2007
> Bug fixes
> * If your code has a $SIG{__DIE__} handler in some cases functions like
> use_ok(), require_ok(), can_ok() and isa_ok() could trigger that
> handler. [rt.cpan.org 23509]
> - Minor improvement to TB's filehandle detection in the case of overridden
> isa(). [rt.cpan.org 20890]
> - Will now install as a core module in 5.6.2 which ships with Test::More.
> [rt.cpan.org 25163]
>
> New Features
> - Test::Builder->is_fh() provides a way to determine if a thing
> can be used as a filehandle.
>
> Documentation improvements
> - Improved the docs for $Test::Builder::Level showing the encouraged
> use (increment, don't set)
> - Documented the return value of Test::Builder's test methods
> - Split out TB's method documentation to differenciate between test
> methods (ok, is_eq...), methods useful in testing (skip, BAILOUT...)
> and methods useful for building your own tests (maybe_regex...).
>
> Test fixes
> - We required too old a version of Test::Pod::Coverage. Need 1.08 and not
> 1.00. [rt.cpan.org 25351]
Something somewhere in those changes makes it go very wrong, as 0.67 works
a treat. I've not started digging yet, but the first test to clearly go
wrong is:
$ ~/Reference/5.005_03/bin/perl5.00503 -w -Mblib t/Builder.t
Using /export/home/nwc10/Perl/Test-Simple-0.68/blib
ok 6 - current_test() set
And this might give the game away:
$ ~/Reference/5.005_03/bin/perl5.00503 -w -Mblib t/buffer.t
Using /export/home/nwc10/Perl/Test-Simple-0.68/blib
ok 2 - You're ok
ok 4 - You're ok
ok 6 - You're ok
ok 8 - You're ok
ok 10 - You're ok
ok 12 - You're ok
ok 14 - You're ok
ok 16 - You're ok
ok 18 - You're ok
ok 20 - You're ok
given that the output is supposed to be:
$ perl -Mblib t/buffer.t
1..20
ok 1 - I'm ok
ok 2 - You're ok
ok 3 - I'm ok
ok 4 - You're ok
ok 5 - I'm ok
ok 6 - You're ok
ok 7 - I'm ok
ok 8 - You're ok
ok 9 - I'm ok
ok 10 - You're ok
ok 11 - I'm ok
ok 12 - You're ok
ok 13 - I'm ok
ok 14 - You're ok
ok 15 - I'm ok
ok 16 - You're ok
ok 17 - I'm ok
ok 18 - You're ok
ok 19 - I'm ok
ok 20 - You're ok
Does anything spring to mind as to the cause?
Nicholas Clark
Re: Eliminating STDERR without any disruption.
On Sun, Mar 18, 2007 at 01:06:48AM +, Andy Armstrong wrote: > And in fact now I see how simple the proposal is I realise that even > mentioning how it might be implemented is completely unnecessary. > Sorry folks - it's been a long day :) Well then, you should spend less time working actually writing code, and more time relaxing and being sociable by replying to e-mail on lists. :-) Meetings - the practical alternative to work. Nicholas Clark
Re: TAP::Parser internationalization
On Sun, Mar 18, 2007 at 05:00:32AM -0700, Ovid wrote: > This will likely *not* be in Test::Harness 3.0 unless it turns out to > be easier/more important than I thought, but I think that > internationalization may be important. Right now, all text that it > output is hard-coded in TAP::Parser/TAP::Harness/etc. This has some > negative consequences. > I have three questions: > > 1. How important is this feature? I don't know. I've only worked in English speaking companies. But getting the framework in, even if you only have 2 languages, where one is CAPITALS (or some-such) for testing makes it a lot easier later, as you don't have to go through retrofitting. But on the other hand, doing it now might be over engineering. > 2. Is a YAML phrasebook a sensible approach? Surely that's an implementation detail, behind Locale::Maketext ? > 3. Where's my beer? Last seen here: http://london.pm.org/meetings/locations/antelope.html There will be more beer on 5th April, venue to be confirmed. Possibly with an emergency before then because Simon Cozens is going to Japan. Nicholas Clark
Re: Eliminating STDERR without any disruption.
On Sun, Mar 18, 2007 at 01:37:36AM +, Andy Armstrong wrote: > Are we expecting a YAML reader / writer to be core anytime soon? Not that I'm aware of. Nicholas Clark
Re: New TAP mailing list
On Mon, Mar 19, 2007 at 09:12:08AM -0500, Andy Lester wrote: > > On Mar 19, 2007, at 8:11 AM, Andy Lester wrote: > > > > > > >Begin forwarded message: > > > >>Alright, [EMAIL PROTECTED] should work within 30 minutes or so. > > I dunno, I just saw that Nicholas Clark had requested it, and I > forwarded the email. I don't know the story behind it. I didn't request it. The specific question I asked was "how does one go about doing it?" Ask assumed that it was wanted. Nicholas Clark
Re: New TAP mailing list
On Mon, Mar 19, 2007 at 02:14:23PM +, Nicholas Clark wrote: > On Mon, Mar 19, 2007 at 09:12:08AM -0500, Andy Lester wrote: > > > > On Mar 19, 2007, at 8:11 AM, Andy Lester wrote: > > > > > > > > > > >Begin forwarded message: > > > > > >>Alright, [EMAIL PROTECTED] should work within 30 minutes or so. > > > > I dunno, I just saw that Nicholas Clark had requested it, and I > > forwarded the email. I don't know the story behind it. > > I didn't request it. The specific question I asked was "how does one go > about doing it?" Ask assumed that it was wanted. Which might be my fault for not making it clear that I was only asking about the process, not making a request. Nicholas Clark
Re: Summarizing the pod tests thread
On Tue, Jul 31, 2007 at 04:43:08PM -0400, David Golden wrote: > * CPANTS -- eliminating or penalizing Kwalitee for pod/pod-coverage > tests would seem to be up to Thomas Klausner; I don't think I've seen > him weigh in on this recently. So people need to lobby him or fork > the project. He's somewhere in Outer Mongolia, and won't be back for about 2 weeks: http://use.perl.org/~domm/journal/33801 http://use.perl.org/~domm/journal/33878 He then might be rather busy until the end of YAPC::EU Don't be hasty with the f word. Nicholas Clark
Re: prove -j 9
On Mon, Sep 10, 2007 at 07:39:21PM +0200, demerphq wrote: > On 9/10/07, Eric Wilhelm <[EMAIL PROTECTED]> wrote: > > Note that your test suite may or may not play nicely with that. Have > > you ever run them in parallel before? ;-) Assuming common tempfiles > > and such really rains on the parade. Other than that, we're talking > > 40% less waiting. > > Will this make it easier to parallelize tests in the perl core? > > There the issue isnt so much running the tests in a single module > directory simultaneously but rather running the tests in different > test directories simultaneously. I had a hacked proof of concept for this, but it needed moderately large changes to Test::Harness to make it work. This was around the time that the TAPx::Parser work commenced, so it wasn't going to happen with Test::Harness 2. I still *have* the proof of concept code. The key thing I needed changed was that once you're running 2 or more tests you can't print any progress about *the* test currently running, because your assumption that there is just one is no longer valid. As to the core tests, they trip up like crazy, I think because they are not original in their choice of names for temporary files. Nicholas Clark
Re: prove -j 9
On Mon, Sep 10, 2007 at 11:37:00AM -0700, Eric Wilhelm wrote: > # from demerphq > # on Monday 10 September 2007 10:51 am: > > >> As to the core tests, they trip up like crazy, I think because they > >> are not original in their choice of names for temporary files. > > I solved that in dotReader's tests with some '. $$' action. > > I was playing with the 't/' directory from perl-5.8.8 last night and > having very little luck. For starters, I had to hack t/TEST to print > the filenames, but still came up with 995 files instead of 933, so I > think I did it wrong ;-) In the core use the files as listed in MANIFEST > I currently have no partitioning for what runs in parallel or "groups of > series" (ala `make -j`.) I supposed grouping by directories could be > an easy switch-activated feature. In general though, I'm thinking this > ventures into the realm of "test metadata", so I hesitate to ad-hoc too > much into it. In the core, I found that ensuring that all the test files from the same directory in order worked nicely. I'm sure that some tests can be run in arbitrary order (or literally in parallel) but there are enough of them to ensure that lots of CPUs can get fed. The problems I had were with trying to get t/op/*.t in parallel (which should work) and t/class/*.t (which are slow) Also, I was trying to feed 32 cores, not 2. I think that I documented this somewhere, but once you get to 7 tests in parallel, where you pass the tap stream back to a central parser/reporter, the parser in Test::Harness became the bottleneck. Nicholas Clark
Re: New proposed CPANTS metric: prereq_matches_use
On Mon, Nov 19, 2007 at 03:03:51PM -0800, chromatic wrote: > On Monday 19 November 2007 14:30:51 Jonathan Rockway wrote: > > > I think the real solution, though, is to agree that the perl interpreter > > without all of the core modules installed isn't Perl. (I'm not a big > > fan of core modules, but the concept is especially worthless if you > > can't depend on their existence.) To your own modules? If so, why on earth is anyone upset about this? > Any chance you could convince a few thousand ISPs of that? I'm game. And this is why. Nicholas Clark
Re: shouldn't "UNEXPECTEDLY SUCCEEDED" mean failure?
On Tue, Dec 04, 2007 at 08:25:10PM -0800, Michael G Schwern wrote: > Fergal Daly wrote: > > You have no idea what version of Foo they're using > > Well, you do with version dependency declarations so you control the range. > New versions are, of course, open to breakage but at some point you have to > trust something. I'm also missing why no-one has suggested local $TODO = "Frobulator low on pie" if Foo->VERSION < 3.14; which ought to mean that TODO tests stop being TODO if the user has a new enough version of Foo installed. (ie, you *do* know what version of Foo they're using, at least in the general case) Nicholas Clark
Re: Impotent disclaimers (was: Milton Keynes PM coding collaboration)
On Fri, Dec 14, 2007 at 09:09:12AM +0100, A. Pagaltzis wrote: > Hi Tony, > > * Edwardson, Tony <[EMAIL PROTECTED]> [2007-12-14 08:55]: > > This is a commercial communication from Commerzbank AG. > > > > [20 lines snipped] > is there any chance you could subscribe this list using an > address whose outgoing mail server does not feel the need to > append 25 lines of legally ineffectual lawyeristic mumbojumbo > to a one-line mail? Or encourage Morgan Stanley to take Commerzbank AG over, as the Morgan Stanley disclaimer manages to say more in only 2 lines. [some lawyers are more efficient than others. I wonder if lawyers can be upgraded] Nicholas Clark
Re: Auto: Your message 'FAIL IO-AIO-2.51 i386-freebsd-thread-multi 6.2-release' has NOT been received
On Wed, Dec 19, 2007 at 12:38:46PM +, Andy Armstrong wrote: > On 19 Dec 2007, at 12:32, Michael G Schwern wrote: > >>Obviously that's my (probably extremely unprofessional) impression of > >>his views. He did mention religion and ego though :) > > > >CPAN Testers does mug his modules pretty badly, just look at all > >that red. > >http://cpantesters.perl.org/author/MLEHMANN.html > > > >He does an awful lot of XS which is always going to be problematic. > > > From what he's said to me off-list I'd guess that if he used > Devel::Check* many of those failures would go away. > > I'm trying to persuade him to let me archive our exchange somewhere so > people can read it if they're interested. It would be interesting. He's a smart guy and usually right. So are you. Without seeing it, it makes me think of the exchanges between IlyaZ and TomC on p5p. Both were right, but both contradicted each other. The cause was that each based their reasoning on a different set of assumptions, and what one thought important the other thought was not, and vice versa. Nicholas Clark
Re: Testing print failures
On Sat, Jan 05, 2008 at 11:00:41PM +0100, nadim khemir wrote: > day an the answer was 'no'. Do you happend to know something about > replacing 'print' with XS code short of patching perl (which doesn't sound > like a good idea) Not tested, but, can you 1: grab the address of print's op from PL_ppaddr 2: store it somewhere useful 3: replace it in PL_ppaddr with your own function Your own function calls the original, and then before returning, checks the return value on the stack. If it indicates fatal, then check the calling context. If that's void, croak. Otherwise return normally. Nicholas Clark
Re: Testing print failures
On Mon, Jan 07, 2008 at 04:19:17PM +, David Cantrell wrote: > On Sat, Jan 05, 2008 at 10:24:47PM +0000, Nicholas Clark wrote: > > > Not tested, but, can you > > > > 1: grab the address of print's op from PL_ppaddr > > 2: store it somewhere useful > > 3: replace it in PL_ppaddr with your own function > > > > Your own function calls the original, and then before returning, checks > > the return value on the stack. If it indicates fatal, then check the calling > > context. If that's void, croak. > > > > Otherwise return normally. > > Please for the love of god don't do that. Global changes to perl's But I can for the love of Mammon? :-) [ http://www.miltonbrewery.co.uk/beers/mammon.html Not that I had any last night, because Christmas was still in bottled form: http://www.individualpubs.co.uk/pembury/drinks.html ] > behaviour propagating out of some obscure module several steps away in > your application's dependency tree are a Bad Idea. See UNIVERSAL::* for > examples of why. Good point. I think that it's already been suggested that it can be done lexically in 5.10 (and later) by inspecting $^H, and only croaking if it's void context and and a user defined lexical pragma has enabled it. Nicholas Clark
Re: Testing print failures
On Mon, Jan 07, 2008 at 03:59:18PM +, Adrian Howard wrote: > Actually, I've had some nasty bugs in the past with prints failing > with dodgy NFS mounts. Caused some important data to go away. Not my > code fortunately :-) In these circumstances print failed but the close on the file did not? I know that it happens the other way, because I clearly remember that a certain firm's compiler got caught with its pants down when a certain firm's NFS implementation went multithreaded in release 2.4 of the OS. With 2.4 write failures (here caused by hitting quotas) were not reported as errors on the writes, but instead only spotted at close time (when everything resynchronised). The compiler wasn't checking the return value of close. Then again, close time might be too late for the application you're describing, if it is holding files open for long periods. Nicholas Clark
