Re: TODO test paradox: better TODO test management?
On 1/31/06, chromatic <[EMAIL PROTECTED]> wrote: > On Tuesday 31 January 2006 11:44, A. Pagaltzis wrote: > > > * chromatic <[EMAIL PROTECTED]> [2006-01-31 19:40]: > > > >Write your own. perldoc Test::Harness::Straps or see the > > >examples in chapter 3 of the Perl Testing book: > > > > That's not a long-term answer though, is it? > > Why not? You know more about your application and your team and your > requirements than Test::Harness does. There's also always verbose mode and > grep, if the default summary is too simple. The application im talking about is perl. -- perl -Mre=debug -e "/just|another|perl|hacker/"
Re: TODO test paradox: better TODO test management?
On 1/31/06, chromatic <[EMAIL PROTECTED]> wrote: > On Tuesday 31 January 2006 12:22, A. Pagaltzis wrote: > Adding more information to the default Test::Harness summary doesn't make > sense to me. It's a user tool. It's important to list failures there, as > the code might not work right, but unexpected successes? I don't want to > have to explain those or the implications; I'm not sure it's useful > information for most users. I think that mischaracterization how harness is used. Its both a user and a developer tool. In fact most of the information presented by the summary is useless to a user except that it gives them something to mail to the author to say "see this stuff broke". From a users point of view a response like "all tests passed" or "not all tests passed" would be sufficient. All the other information is only useful to the maintainer. And I think you've conveniently sidestepped my main point which is that TODO tests passing are errors. Consider you have two TODO tests, both of which depend on a common set of functionality. Both should pass or both should fail. With the current model if one of them passes its supposed to be a good thing. But its probably not, its probably an indication of serious error. But the end user will never know because harness doesnt consider it an issue, and the author will never know because the user doesnt have any reason to inform the author. (This is assuming the spurious TODO passes didnt occur in the authors build enviornment). And this issue in particular affects perl. Which is why it should be changed. Writing custom harnesses is fine for special projects, but im talking about the mother of all them. Yves -- perl -Mre=debug -e "/just|another|perl|hacker/"
Re: TODO test paradox: better TODO test management?
On 1/31/06, chromatic <[EMAIL PROTECTED]> wrote: > On Tuesday 31 January 2006 13:31, A. Pagaltzis wrote: > > > Hmm. That's a good point. Maybe the way to approach this would be > > to include a default harness for use by developer tools, which > > would include more chattiness about passing TODO tests. > > My perfect developer tool would complain noisily about failing tests (and give > all of the diagnostic information for me to debug them), highlight bonus > tests (and give the diagnostic information for me to promote them), and > summarize everything else. If it passes, increment the counter but don't > tell me. > > How would that work for you? My only thought is that is that it should be possible to easily control whether unexpected TODO passes are treated as "bonuses" or as "failures". And IMO when building perl itself it should default to the latter. yves -- perl -Mre=debug -e "/just|another|perl|hacker/"
Re: Kwalitee in your dependencies (was CPAN Upload: etc etc)
Hi!
On Tue, Jan 31, 2006 at 10:36:47AM -0800, Tyler MacDonald wrote:
> OK, speaking of "Kwalitee", I saw cpants for the first time today.
> And saw that it claims to update every sunday, but there hasn't been an
> update since december 5th.
On the one hand I'm having problems with the server (which is ancient
and overloaded), on the other hand I quited my job at the company where
the server is located (it's our dev-box).
So until I find a new server (either at my new job or somewhere else) I
won't update cpants.perl.org.
Hmm, maybe I should put a note up there...
> I also saw this interesting .pm file that
> appeared to have an anonymous hash of every tarball in CPAN in it, all on
> one line. That nearly crashed my browser.
Well, that's really outdated. I think it's a leftover from Leon Brocard.
> :-) What's up with CPANTS? Is it a
> defunct project?
No. Last week I released Module::CPANTS::Analyse
http://use.perl.org/~domm/journal/28475
As soon as I find a server and some tuits, I will revive
cpants.perl.org
--
#!/usr/bin/perl http://domm.zsi.at
for(ref bless{},just'another'perl'hacker){s-:+-$"-g&&print$_.$/}
Re: TODO test paradox: better TODO test management?
On Wednesday 01 February 2006 00:26, demerphq wrote: > And I think you've conveniently sidestepped my main point which is > that TODO tests passing are errors. I didn't sidestep it. I just disagree. > Consider you have two TODO tests, > both of which depend on a common set of functionality. Both should > pass or both should fail. With the current model if one of them passes > its supposed to be a good thing. But its probably not, its probably an > indication of serious error. That's possible, but there's absolutely nothing in TAP to indicate such a thing. If you want this feature in your test, you have to code it yourself, using the return value of the test functions and a non-TODO test. I've never come across that situation. Having the test suite quit and say "Sorry, don't use this -- it works better than we expect" seems awfully silly. Again, there's already a way to catch your (rare) catastrophic bonus tests -- capture and check their return values. -- c
Re: [Module::Build] [RFC] author tests
Chris Dolan <[EMAIL PROTECTED]> wrote: > There is a class of tests that module authors perform that end users > are not expected to run. For example code coverage tests, spelling > tests, coding style tests, etc. These tests are either prohibitively > expensive or complicated or unpredictable for end users to run. I > call these "private tests" or author tests. I really like this idea. But as you pointed out, it's not just authors that need to worry about running these tests, it's packagers (ppm/deb/etc), automated testers (cpants/testers.cpan.org/etc), and hackers. I'd suggest we call these "exhaustive" tests. Spelling tests, code coverage tests, testing interaction with CPAN.pm, etc. are things that I think should be done by automated testers, and by people packaging your module for distribution inside another larger system (eg, PPMs or debian packages). So maybe it would make sense if there were certain tests that only run if AUTOMATED_TESTING is set, and/or if the "disttest" action is being run. That said, it would be easy to whip together a Test:: module that only allows these tests to run under these conditions, especially if Module::Build, say, set a "DIST_TEST" environment variable when the disttest action was being run. I already check for AUTOMATED_TESTING to manipulate my test environments to be more exhaustive. Both Module::Build, and perhaps future versions of ExtUtils::MakeMaker (if there will ever be a future version) could depend on this Test:: module (whether or not they use it themselves) so that it's already on everybody's system. It'd be nice and small so I don't see anybody complaining too loudly. Maybe we could call it "Test::Exhaustive"? It's mandate being to SKIP_ALL of a test if you havent somehow (AUTOMATED_TESTING, disttest, etc) made it clear that you wish to exhaustively test this module? > Roughly, I propose that we adopt a standard file extension for author > tests, like "t/*.ta", and add an ACTION_authortest to M::B that runs > both t/*.t and t/*.ta (which ones first??) If they were named differently (i don't think they have to be), I'd want to see them run in lexical order like the current tests, so it'd go "01foo.t", "02bar.ta", "03baz.t". Cheers, Tyler
Re: [Module::Build] [RFC] author tests
The trouble is, EVERYONE wants to add just one more little dependency (me included *cough*Params::Util*cough*). I'll make you a deal. Write this up. Then exhaustively test it on every single Perl platform (50ish?) and every Perl version back to 5.004, including a random collection similarly weird combinations (5.004 VMS, "that" 5.6.0 from RedHat 7, 5.6.1 on Windows 95). Then make sure the code is so clean and complete you'll never need add another lines of code or tweak the docs. Then lets think about adding a new compulsory near-core dependency. Adam K Tyler MacDonald wrote: Chris Dolan <[EMAIL PROTECTED]> wrote: There is a class of tests that module authors perform that end users are not expected to run. For example code coverage tests, spelling tests, coding style tests, etc. These tests are either prohibitively expensive or complicated or unpredictable for end users to run. I call these "private tests" or author tests. I really like this idea. But as you pointed out, it's not just authors that need to worry about running these tests, it's packagers (ppm/deb/etc), automated testers (cpants/testers.cpan.org/etc), and hackers. I'd suggest we call these "exhaustive" tests. Spelling tests, code coverage tests, testing interaction with CPAN.pm, etc. are things that I think should be done by automated testers, and by people packaging your module for distribution inside another larger system (eg, PPMs or debian packages). So maybe it would make sense if there were certain tests that only run if AUTOMATED_TESTING is set, and/or if the "disttest" action is being run. That said, it would be easy to whip together a Test:: module that only allows these tests to run under these conditions, especially if Module::Build, say, set a "DIST_TEST" environment variable when the disttest action was being run. I already check for AUTOMATED_TESTING to manipulate my test environments to be more exhaustive. Both Module::Build, and perhaps future versions of ExtUtils::MakeMaker (if there will ever be a future version) could depend on this Test:: module (whether or not they use it themselves) so that it's already on everybody's system. It'd be nice and small so I don't see anybody complaining too loudly. Maybe we could call it "Test::Exhaustive"? It's mandate being to SKIP_ALL of a test if you havent somehow (AUTOMATED_TESTING, disttest, etc) made it clear that you wish to exhaustively test this module? Roughly, I propose that we adopt a standard file extension for author tests, like "t/*.ta", and add an ACTION_authortest to M::B that runs both t/*.t and t/*.ta (which ones first??) If they were named differently (i don't think they have to be), I'd want to see them run in lexical order like the current tests, so it'd go "01foo.t", "02bar.ta", "03baz.t". Cheers, Tyler
