Re: Keeping tests for execution at a later point in time
On Thu, Apr 10, 2008 at 10:01 PM, David Golden <[EMAIL PROTECTED]> wrote: > On Thu, Apr 10, 2008 at 9:44 AM, Smylers <[EMAIL PROTECTED]> wrote: > > David Golden writes: > > > Seems like a lot of headache. Why not just save the tarballs > > > somewhere? > > > > Were there tarballs in the first place? > > Presumably at some point, for modules that live on CPAN. > > > > Suppose a Fedora or Ubuntu user is wanting to check that upgrading the > > OS hasn't caused any test regressions, it's likely that at least some of > > the existing Perl modules were installed as RPMs or Debian packages, not > > by CPAN.pm from tarballs. > > Perhaps the tarball could be included with the RPM then and installed > in some standardized place. > I think first we need to figure out how this should work when using CPAN.pm or CPANPLUS and then we can come with a recommendation for the case when the modules come as rpms or debs. For now I wrote on the Fedora list that IMHO the best is to have a separate "source" rpm that includes everything there was in the original tarball downloaded from CPAN. Gabor
Re: Keeping tests for execution at a later point in time
On Thu, Apr 10, 2008 at 9:44 AM, Smylers <[EMAIL PROTECTED]> wrote: > David Golden writes: > > Seems like a lot of headache. Why not just save the tarballs > > somewhere? > > Were there tarballs in the first place? Presumably at some point, for modules that live on CPAN. > Suppose a Fedora or Ubuntu user is wanting to check that upgrading the > OS hasn't caused any test regressions, it's likely that at least some of > the existing Perl modules were installed as RPMs or Debian packages, not > by CPAN.pm from tarballs. Perhaps the tarball could be included with the RPM then and installed in some standardized place. David
Re: Keeping tests for execution at a later point in time
David Golden writes: > On Thu, Apr 10, 2008 at 7:14 AM, Gabor Szabo <[EMAIL PROTECTED]> wrote: > > > So let's see what needs to be done in order to be able to keep > > the test files and run them later. > > > > There are two concerns I could immediately see. > > 1) Tests might assume a certain directory structure, they might say > > use lib 'lib'; > > use lib 't/lib'; > > or other things. > > 2) Tests might use other files outside the t/ directory. > > 3) What else do you think might be problematic? > > Seems like a lot of headache. Why not just save the tarballs > somewhere? Were there tarballs in the first place? > Then you're certain to recreate the environment that the distribution > expected for its tests when run by CPAN.pm. Suppose a Fedora or Ubuntu user is wanting to check that upgrading the OS hasn't caused any test regressions, it's likely that at least some of the existing Perl modules were installed as RPMs or Debian packages, not by CPAN.pm from tarballs. Smylers
Re: Keeping tests for execution at a later point in time
On Thursday 10 April 2008 04:14:15 Gabor Szabo wrote: > The issue was raised on the Oslo Hackathon that it would be cool > if we could keep the tests around so that they can be executed > later again making sure that even after one has upgraded other > parts of his system the previously installed modules still work as > expected. I use Test::Class and install the test modules with the other modules such that anyone who subclasses my code can subclass my tests for great justice. -- c
Test::TAP
Test::TAP is now available at http://search.cpan.org/dist/Test-TAP/. Currently the only tests exposed are those to validate whether or not a TAP stream passed: use Test::TAP; is_passing_tap $tap1, 'TAP tests passed'; is_failing_tap $tap2, 'TAP tests failed'; This will be expanded later, but it's intended for TAP developers who would like to have alternate tools for generating TAP. Currently we are dependent on Test::Builder, but it's limited. It's also only for Perl. Eventually this module should be expanded with the YAML TAP tests that were started at the Oslo hackathon. If anyone who worked on those would like to take over Test::TAP, that's fine. Just drop me a line and I'll transfer it to you. Also note that, following up on my earlier comments, the primary TAP parser for the above two functions is only 40 lines of code. This could easily be broken out into a separate module for embedding in Test::Builder. Cheers, Ovid -- Buy the book - http://www.oreilly.com/catalog/perlhks/ Perl and CGI - http://users.easystreet.com/ovid/cgi_course/ Personal blog - http://publius-ovidius.livejournal.com/ Tech blog - http://use.perl.org/~Ovid/journal/
Re: Keeping tests for execution at a later point in time
--- Gabor Szabo <[EMAIL PROTECTED]> wrote:
> The issue was raised on the Oslo Hackathon that it would be cool
> if we could keep the tests around so that they can be executed
> later again making sure that even after one has upgraded other
> parts of his system the previously installed modules still work as
> expected.
I've been thinking about this a lot and want a solution. The main
issue with "keeping the tests" is that you probably already have a lot
of modules installed and it's not always easy to tell which
distributions you have to get the tests for.
brian d foy tells me that in about a month he hopes to have software
which, given a particular file, will tell you which distribution (if
any) it's from. With that, what we can do is this:
my %files_in;
my @unknown_distributions;
for my $file (pm_files_in_inc()) {
if ( my $distribution = distribution_for($file) ) {
$files_in{$distribution} ||= [];
push $files_in{$distribution} => $file;
}
else {
push @unknown_distributions => $file;
}
}
for my $dist_name (keys $files_in) {
# with local caching, obviously
my $distribution = get_from_cpan_or_backpan($dist_name);
# should extract into dirs named after the dist
extract_tests($distribution, 'installation_tests/');
}
There are *all* sorts of caveats with this, but basically:
1. You can produce a report of packages from unknown distributions.
2. Run tests against your current setup.
3. CPAN hooks to see how the tests will pass with a new module.
There has never been a serious way to test your Perl installation.
Some tests won't be able to pass because they require environment
variables, access to the Web, ponies, etc. However, it could be a good
starting solution.
Cheers,
Ovid
--
Buy the book - http://www.oreilly.com/catalog/perlhks/
Perl and CGI - http://users.easystreet.com/ovid/cgi_course/
Personal blog - http://publius-ovidius.livejournal.com/
Tech blog - http://use.perl.org/~Ovid/journal/
Re: Keeping tests for execution at a later point in time
On Thu, Apr 10, 2008 at 7:14 AM, Gabor Szabo <[EMAIL PROTECTED]> wrote:
> So let's see what needs to be done in order to be able to keep
> the test files and run them later.
>
>
> There are two concerns I could immediately see.
> 1) Tests might assume a certain directory structure, they might say
> use lib 'lib';
> use lib 't/lib';
> or other things.
> 2) Tests might use other files outside the t/ directory.
> 3) What else do you think might be problematic?
Seems like a lot of headache. Why not just save the tarballs
somewhere? (Aside from disk issues, that is.) Then you're certain to
recreate the environment that the distribution expected for its tests
when run by CPAN.pm.
Given that, I would tackle it this way:
* unpack tarball (for all archive types)
* cd into directory
* locally set CPAN's prerequisites_policy to 'ignore', have CPAN test
the current directory:
perl -MCPAN -e 'local
$CPAN::Config->{prerequisites_policy}="ignore"; test(".")'
Then the challenge is just one of bookkeeping which distribution
tarballs have been installed.
And if disk is an issue, just pull the tarballs off BackPAN when needed.
David
Keeping tests for execution at a later point in time
The issue was raised on the Oslo Hackathon that it would be cool if we could keep the tests around so that they can be executed later again making sure that even after one has upgraded other parts of his system the previously installed modules still work as expected. AFAIK the issue did not get anywhere but as I have just seen on the Fedora packagers mailing list https://www.redhat.com/archives/fedora-perl-devel-list/2008-April/msg00095.html there too is some request for this. There they also point out the documentary value of the test files. So let's see what needs to be done in order to be able to keep the test files and run them later. There are two concerns I could immediately see. 1) Tests might assume a certain directory structure, they might say use lib 'lib'; use lib 't/lib'; or other things. 2) Tests might use other files outside the t/ directory. 3) What else do you think might be problematic? I wonder if we could put together some guidelines amending the "Testing Best Practices" that will allow the easy distribution and and later execution of the test files? I know many people create helper modules in t/lib/... some would call these helper packages My::Package::Test and then say use lib 't/lib' in the test suit while others - I saw in the code of AdamK - name their packages t::lib::Test so they don't have to change @INC. Both assume the current directory to be the parent of /t Testing examples: At least in one module I test the example scripts living in the eg/ directory. I might not be the only one. What should we do about this? For now I think Checking if all this can work: CPANTS cannot do that without actually executing the tests but the smokers could have a mode where - after unzipping the tarball - they move the whole t/ directory to some other place, move blib to another place and chdir to a 3rd place and run the tests then. If they did this for packages that have already passed their tests and then report the possible issues we could have an understanding how many of the distributions might actually be tested after installation? regards Gabor -- Gabor Szabo http://www.szabgab.com/
Nested TAP
One issue we covered at the Hackathon was how to do nested TAP. The suggestion that was adopted was not backwards-compatible, but it was felt this was OK because the TAP consumer specifically had to request this. I hated this decision, but was hard-pressed to argue against it because making things backwards-compatible seemed to involve putting a full-blown parser into the TAP producer. (In other words, sort of like embedding Test::Harness into Test::Builder). I *think* I've figured out how to make things backwards-compatible and not require a parser. The background is here: http://use.perl.org/~Ovid/journal/36120 Cheers, Ovid -- Buy the book - http://www.oreilly.com/catalog/perlhks/ Perl and CGI - http://users.easystreet.com/ovid/cgi_course/ Personal blog - http://publius-ovidius.livejournal.com/ Tech blog - http://use.perl.org/~Ovid/journal/
Re: User Supplied YAML Diagnostic Keys
--- Eric Wilhelm <[EMAIL PROTECTED]> wrote: > But what's wrong with throwing them the !~ m/^[a-z][a-z0-9_]*$/ bone > and > keeping the future reserved words within that constraint? Everything > else is "something else". The main issue here is actually convincing Schwern. He stated quite clearly at Oslo that he controls what Test::Builder emits and he'll happily be stubborn if he thinks we're doing the wrong thing. I believe he's in Edinburgh right now (or at the tail end of his Oslo trip) and may not be checking email for a bit. Cheers, Ovid -- Buy the book - http://www.oreilly.com/catalog/perlhks/ Perl and CGI - http://users.easystreet.com/ovid/cgi_course/ Personal blog - http://publius-ovidius.livejournal.com/ Tech blog - http://use.perl.org/~Ovid/journal/
Re: User Supplied YAML Diagnostic Keys
# from chromatic # on Wednesday 09 April 2008 10:52: >Even in the few cases where you need to shuttle TAP documents back and > forth, the plan is to include version information in the document, > correct? Isn't the point of including a version so that TAP > consumers will be able to consume the document correctly? Is this a discussion about the "X-" thing, or something else entirely? If we have a version number and a list of reserved words, we can append to the list of reserved words at a version bump. As far as conflicts during upgrading: I'm in the "deal with it" camp. But what's wrong with throwing them the !~ m/^[a-z][a-z0-9_]*$/ bone and keeping the future reserved words within that constraint? Everything else is "something else". (And, didn't I say that already? tap, tap... is this thing on?) --Eric -- Moving pianos is dangerous. Moving pianos are dangerous. Buffalo buffalo buffalo buffalo buffalo buffalo buffalo. --- http://scratchcomputing.com ---
