Re: Installing Tests
Adam Kennedy wrote: > I'm not suggesting we test against the post-make code, but if we keep > everything, we can be fairly certain we still have any data files or > other things that are needed to support the testing. > > Now we rerun the test suite, but without including blib. > > And you get a full test run against the currently installed code, but > with any data files or other stuff the tests need available. How are you going to exclude blib? Since the point of hanging onto the source tree is to account for customized installers, they will have customized test harnesses.
Re: Installing Tests
Adam Kennedy wrote: >> Hmm, that would waste a lot of disk. Maybe we could cache the >> tarballs somewhere. Maybe on the network or even Amazon S3. Some >> sort of big repository of modules. We can call it the Network of >> Archived Postinstall Crap, NAPC for short! > > This same argument applies to the tests. Sure does. :) > Further, you've missed the main point here, which is that I'm suggesting > saving the post-make state of the tarballs with all build assumptions > and such in place, not the pristine just-unrolled version. > > We're already "wasting" disk saving the tests, why not waste a bit more > and make sure we actually save enough to know the tests will work for sure. I just thought of something. The point is to test that what's *installed* is working, right? But you're talking about testing at the point of installation. This is different from what's actually installed. Consider some common ways the code you're running can fall out of sync with what was installed. * Admin A installs and archives the source properly. Admin B installs by hand and doesn't update the archive. Or the package manager installs a different version. Or any number of module shadowing and overlay scenarios like the above. * The installed module is edited in place. Either in an attempt to fix a bug, add a feature or alter a Config.pm. * Disk corruption. * The CPAN shell's "recompile" command is run (such as after an architecture change like when going from a PowerPC to Intel Mac like I just did). Testing the post-make source doesn't do anything for the above scenarios, and the first two are rather common. It gives you a false sense of security. To be useful and accurate you have to test against the actual code which is installed. Not what you think is installed.
Re: Installing Tests
On 13 Sep 2006, at 19:28, Michael G Schwern wrote: Adrian Howard wrote: On 13 Sep 2006, at 05:37, Michael G Schwern wrote: [snip] (For the sarcasm impaired, if you're just going to store the whole post-install source tarball you might as well just grab it from CPAN again) [snip] Isn't the post-install source tree potentially different in useful ways from the pre-install source tree? I'd wager that at least 90% of all modules neither modify their code before installation nor change the behavior of "make/Build test". There are two common exceptions. 1) Installing Perl scripts which only change the #! line and 2) Installing XS which generates new files. In both cases you have the altered files already installed to test against. They'll also be 3) Tests that rely on local configuration Archiving installed source dists might be useful, but I don't think its generally useful. Which is to say, write a tool to do it but don't build it into the existing installation tools. Sounds sane to me. Adrian
Re: Installing Tests
Adrian Howard wrote: > > On 13 Sep 2006, at 05:37, Michael G Schwern wrote: > [snip] >> (For the sarcasm impaired, if you're just going to store the whole >> post-install source tarball you might as well just grab it from CPAN >> again) > [snip] > > Isn't the post-install source tree potentially different in useful ways > from the pre-install source tree? I'd wager that at least 90% of all modules neither modify their code before installation nor change the behavior of "make/Build test". There are two common exceptions. 1) Installing Perl scripts which only change the #! line and 2) Installing XS which generates new files. In both cases you have the altered files already installed to test against. Archiving installed source dists might be useful, but I don't think its generally useful. Which is to say, write a tool to do it but don't build it into the existing installation tools.
Re: Installing Tests
On 13 Sep 2006, at 05:37, Michael G Schwern wrote: [snip] (For the sarcasm impaired, if you're just going to store the whole post-install source tarball you might as well just grab it from CPAN again) [snip] Isn't the post-install source tree potentially different in useful ways from the pre-install source tree? Adrian
Re: Installing Tests
On Tuesday 12 September 2006 21:37, Michael G Schwern wrote: > (For the sarcasm impaired, if you're just going to store the whole > post-install source tarball you might as well just grab it from CPAN again) I believe you meant to say BackPAN. -- c
Re: Installing Tests
Adam Kennedy wrote: > Frankly, I think anything you might write to try and pick and choose > what to install is going to have problems. > > Since we're wasting... ermm... using all this extra disk space for tests > anyway, why not just do the lot. > > Take the entire distribution post-make and just tarball that up and save > it somewhere. > > Then if needed, you can unroll it later, rerun the tests with including > blib, and there you go. > > It's a little on the brute force side, but it should work. Hmm, that would waste a lot of disk. Maybe we could cache the tarballs somewhere. Maybe on the network or even Amazon S3. Some sort of big repository of modules. We can call it the Network of Archived Postinstall Crap, NAPC for short! (For the sarcasm impaired, if you're just going to store the whole post-install source tarball you might as well just grab it from CPAN again)
Re: Installing Tests
On Sep 12, 2006, at 8:20 PM, Adam Kennedy wrote: Frankly, I think anything you might write to try and pick and choose what to install is going to have problems. Since we're wasting... ermm... using all this extra disk space for tests anyway, why not just do the lot. Take the entire distribution post-make and just tarball that up and save it somewhere. Then if needed, you can unroll it later, rerun the tests with including blib, and there you go. It's a little on the brute force side, but it should work. Adam K I agree with all this except the post-make idea. The "perl Makefile.PL/Build.PL" step often does runtime things, like check the version of Module::Build or insert boilerplate into Makefile. If your install tools are updated between initial install and later re- test, this could go haywire. Except for processor time, I assert that you're better off starting with a clean tarball as if it's a fresh install. The only reason I can think of for the post-make argument is to preserve choices made during Makefile.PL, like Module::Install optional prereqs. Chris -- Chris Dolan, Software Developer, Clotho Advanced Media Inc. 608-294-7900, fax 294-7025, 1435 E Main St, Madison WI 53703 vCard: http://www.chrisdolan.net/ChrisDolan.vcf Clotho Advanced Media, Inc. - Creators of MediaLandscape Software (http://www.media-landscape.com/) and partners in the revolutionary Croquet project (http://www.opencroquet.org/)
Re: Installing Tests
Frankly, I think anything you might write to try and pick and choose what to install is going to have problems. Since we're wasting... ermm... using all this extra disk space for tests anyway, why not just do the lot. Take the entire distribution post-make and just tarball that up and save it somewhere. Then if needed, you can unroll it later, rerun the tests with including blib, and there you go. It's a little on the brute force side, but it should work. Adam K Dave Rolsky wrote: On Mon, 11 Sep 2006, David Golden wrote: * test.pl vs t/*.t * Custom Makefile.PL or Build.PL that affects test runs * build_requires modules bundled in inc/ And don't forget that some tests may include tests data that is needed to run the tests. If we're lucky, it's under t/ -dave /*=== VegGuide.Orgwww.BookIRead.com Your guide to all that's veg. My book blog ===*/
Re: Installing Tests
On Mon, 11 Sep 2006, David Golden wrote: * test.pl vs t/*.t * Custom Makefile.PL or Build.PL that affects test runs * build_requires modules bundled in inc/ And don't forget that some tests may include tests data that is needed to run the tests. If we're lucky, it's under t/ -dave /*=== VegGuide.Orgwww.BookIRead.com Your guide to all that's veg. My book blog ===*/
Re: Installing Tests
- Original Message From: Ken Williams > Yes, I've been thinking about this for a long time. In fact, in the > most recent M::B beta I made some steps toward it, by adding a > 'retest' action that's just like 'test' except that it doesn't look > in blib/, just in @INC. Functionally that actually covers a lot of > the same ground you're after. Sweet! > 4) Many distributions, including many of the most crucial and well- > used ones, have some extra set-up steps in their build/install > sequences. Others make assumptions in their test suites about where > various files are located relative to the test code or relative to > the current working directory. It's quite possible that in order for > "installed" tests to work correctly it could take some serious > coöperation by modules' authors. In that case, ./Build install could be this (or something similar): ./Build install --with-tests Then, after the module is installed, the installed tests are run against the installed version and if there's a failure, the user could be given the option to uninstall them. > 5) Where should tests be installed? Where would any other > supporting materials be installed? Don't ask me :) > >> 1. How does one install tests for modules already installed? > For your #1 above, I'd say just perform a reinstall. This is often very difficult in a corporate environment, but that's the environment for which installed tests might be the most useful. > >> 2. If you install a module with already failing tests, you need > >> to track what the failures are so you can note different failures > For #2, maybe > just punt - is there much of a need for that? Yes! Handling this correctly is critical to its success. Consider that I install Foo::Bar and tests 4 and 8 fail. Subsequent runs should either skip those tests suites entirely, mark 4 and 8 as 'TODO' or have some other method of ensuring that running the full installed test suite doesn't report a bunch of failures. If it does report a bunch of failures, developers are going to learn to ignore failures and that makes the entire project useless. Cheers, Ovid -- Buy the book -- http://www.oreilly.com/catalog/perlhks/ Perl and CGI -- http://users.easystreet.com/ovid/cgi_course/
Re: Installing Tests
On Sep 11, 2006, at 7:08 PM, Randy W. Sims wrote: Ovid wrote: Last week I was at a testing conference with Acme and he came up with the idea of installing tests. He looked into hacking Module::Build and ExtUtils::MakeMaker. He also considered hacking CPAN.pm and CPANPLUS.pm. While I don't know if he plans to continue working on this idea, he said he didn't mind me posting his idea here for others to consider. Basically, installing tests would be good because then you can run your full test suite against *installed* modules. That would be nice because then you could install a module and rerun your tests for your entire installation and see what broke. I love this idea, but here are some issues that we spotted: 1. How does one install tests for modules already installed? 2. If you install a module with already failing tests, you need to track what the failures are so you can note different failures when you run the test suite in the future. 3. What's the best way to install them? Should a separate tool just for this be built? Anyone want to take a crack at this? I'm also going to post this to Perlmonks. IIRC, I think this is something Ken has had in mind for a long time for Module::Build. He might have some ideas about how it might be done. Yes, I've been thinking about this for a long time. In fact, in the most recent M::B beta I made some steps toward it, by adding a 'retest' action that's just like 'test' except that it doesn't look in blib/, just in @INC. Functionally that actually covers a lot of the same ground you're after. What I like about the 'retest' approach is that it's very easy and it's much more likely to work. It also makes it possible to run old tests against a new installation, or vice versa. What I don't like about it is that the user has to find the tarball again that they previously installed. I can imagine that in some situations that wouldn't be trivial. In other situations, when people can plan ahead, it's probably not a big deal. I think there are some larger issues than 1,2,3 above that you might have missed, too: 4) Many distributions, including many of the most crucial and well- used ones, have some extra set-up steps in their build/install sequences. Others make assumptions in their test suites about where various files are located relative to the test code or relative to the current working directory. It's quite possible that in order for "installed" tests to work correctly it could take some serious coöperation by modules' authors. 5) Where should tests be installed? Where would any other supporting materials be installed? For your #1 above, I'd say just perform a reinstall. For #2, maybe just punt - is there much of a need for that? For #3, I think we can work it into Module::Build as an action or flag(s) to the 'install' action. For EU::MM-based modules I'm not sure what the best approach would be, but probably I don't have to think about it. =) -Ken
Re: Installing Tests
Ovid wrote: Last week I was at a testing conference with Acme and he came up with the idea of installing tests. He looked into hacking Module::Build and ExtUtils::MakeMaker. He also considered hacking CPAN.pm and CPANPLUS.pm. While I don't know if he plans to continue working on this idea, he said he didn't mind me posting his idea here for others to consider. Basically, installing tests would be good because then you can run your full test suite against *installed* modules. That would be nice because then you could install a module and rerun your tests for your entire installation and see what broke. I love this idea, but here are some issues that we spotted: 1. How does one install tests for modules already installed? 2. If you install a module with already failing tests, you need to track what the failures are so you can note different failures when you run the test suite in the future. 3. What's the best way to install them? Should a separate tool just for this be built? Anyone want to take a crack at this? I'm also going to post this to Perlmonks. IIRC, I think this is something Ken has had in mind for a long time for Module::Build. He might have some ideas about how it might be done. Randy.
Re: Installing Tests
On Sep 11, 2006, at 8:02 AM, Ovid wrote: Last week I was at a testing conference with Acme and he came up with the idea of installing tests. He looked into hacking Module::Build and ExtUtils::MakeMaker. He also considered hacking CPAN.pm and CPANPLUS.pm. While I don't know if he plans to continue working on this idea, he said he didn't mind me posting his idea here for others to consider. Basically, installing tests would be good because then you can run your full test suite against *installed* modules. That would be nice because then you could install a module and rerun your tests for your entire installation and see what broke. I love this idea, but here are some issues that we spotted: 1. How does one install tests for modules already installed? 2. If you install a module with already failing tests, you need to track what the failures are so you can note different failures when you run the test suite in the future. 3. What's the best way to install them? Should a separate tool just for this be built? Anyone want to take a crack at this? I'm also going to post this to Perlmonks. Interesting. A setup like this would have solved a recent bug in Text-PDF-0.27 where installation failed silently due to a bogus pm_to_blib file. However, why install the tests? Why not just keep the latests tarballs for each installed module and periodically do the following for each of the tarballs: tar -xzvf Foo-1.00.tgz cd Foo-1.00 perl Makefile.PL make test cd .. rm -rf Foo-1.00 That seems significantly less fragile than creating a new infrastructure, and still exercises all of the non-Foo dependencies. The most significant drawback of that approach is that it doesn't exercise the installed copy of Foo itself. Perhaps that can be accomplished by simply deleting lib and blib in Foo-1.00 before running tests? Chris -- Chris Dolan, Software Developer, Clotho Advanced Media Inc. 608-294-7900, fax 294-7025, 1435 E Main St, Madison WI 53703 vCard: http://www.chrisdolan.net/ChrisDolan.vcf Clotho Advanced Media, Inc. - Creators of MediaLandscape Software (http://www.media-landscape.com/) and partners in the revolutionary Croquet project (http://www.opencroquet.org/)
Re: Installing Tests
Ovid wrote: I love this idea, but here are some issues that we spotted: 1. How does one install tests for modules already installed? 2. If you install a module with already failing tests, you need to track what the failures are so you can note different failures when you run the test suite in the future. 3. What's the best way to install them? Should a separate tool just for this be built? 4. What assumptions are we making about how tests are packaged? Scenarios: * test.pl vs t/*.t * Custom Makefile.PL or Build.PL that affects test runs * build_requires modules bundled in inc/ I'm not convinced that you can get this idea to work short of caching the full distribution directory or tarball at install-time and then iterating through those using the actual Makefile.PL or Build.PL files to prep and call tests. Regards, David Golden
Re: Installing Tests
On 11 Sep 2006, at 14:02, Ovid wrote: Last week I was at a testing conference with Acme and he came up with the idea of installing tests. He looked into hacking Module::Build and ExtUtils::MakeMaker. He also considered hacking CPAN.pm and CPANPLUS.pm. While I don't know if he plans to continue working on this idea, he said he didn't mind me posting his idea here for others to consider. [snip] I'm more of this way of thinking, and tend to install my Test::Class modules along with the classes their testing. See <http://www.perlmonks.org/index.pl?node_id=553653> for some comments. Adrian
Installing Tests
Last week I was at a testing conference with Acme and he came up with the idea of installing tests. He looked into hacking Module::Build and ExtUtils::MakeMaker. He also considered hacking CPAN.pm and CPANPLUS.pm. While I don't know if he plans to continue working on this idea, he said he didn't mind me posting his idea here for others to consider. Basically, installing tests would be good because then you can run your full test suite against *installed* modules. That would be nice because then you could install a module and rerun your tests for your entire installation and see what broke. I love this idea, but here are some issues that we spotted: 1. How does one install tests for modules already installed? 2. If you install a module with already failing tests, you need to track what the failures are so you can note different failures when you run the test suite in the future. 3. What's the best way to install them? Should a separate tool just for this be built? Anyone want to take a crack at this? I'm also going to post this to Perlmonks. Cheers, Ovid -- Buy the book -- http://www.oreilly.com/catalog/perlhks/ Perl and CGI -- http://users.easystreet.com/ovid/cgi_course/
