Re: The spewing problem.
--- chromatic [EMAIL PROTECTED] wrote: How is this simpler than 'bail on fail' or 'die on fail'? It doesn't conflate output TAP results from tests with interpret TAP results from tests. Neither does die on fail. Die on fail doesn't violate that in the slightest because it's the *tests* and Test::Builder which determines what TAP is output and how the tests behave. The TAP::Parser merely reads what they tell 'em. This solution was quick, easy (with the limitations imposed by Test::Builder) and more importantly, it's real, working code that is available on the CPAN. Now if I have a test suite which takes 45 minutes to run and I get a failure 3 minutes into that, it's not an *insane* idea to say hey, let's stop the test suite and not tie up my resources for another 42 minutes. Not only do different test suites and environments potentially require different test strategies, different tester's personalities also require different test strategies. This Is Not A Bad Thing! Side note: I'm thinking that all of those 't/00-load.t' tests are excellent candidates for 'bail on fail'. The various solutions people are talking about are more akin to MVC views. I think implementing more views would be an excellent idea and I really do hope that folks follow through on this. Cheers, Ovid -- Buy the book - http://www.oreilly.com/catalog/perlhks/ Perl and CGI - http://users.easystreet.com/ovid/cgi_course/ Personal blog - http://publius-ovidius.livejournal.com/ Tech blog - http://use.perl.org/~Ovid/journal/
Re: DTrace in Perl: What probes should we have?
On Fri, Jan 11, 2008 at 08:45:44PM +, Andy Armstrong wrote: Currently we have probes on subroutine entry and return. Next I'd like to solicit input about what other probes would be useful. I have Sven Dowideit's patch that adds probes on new__sv, del__sv (creation and deletion of values) and, main__enter, main__exit (interpreter lifecycle) and load__module (require, use etc). Where else might we usefully probe? What would people find useful? * start of loop; * loop iteration (either by the end-condition not being met or 'next'); * end of loop (either by the end-condition being met or 'last'); if that's practical. -- David Cantrell | A machine for turning tea into grumpiness
Re: DTrace in Perl: What probes should we have?
On 14 Jan 2008, at 12:12, David Cantrell wrote: Next I'd like to solicit input about what other probes would be useful. I have Sven Dowideit's patch that adds probes on new__sv, del__sv (creation and deletion of values) and, main__enter, main__exit (interpreter lifecycle) and load__module (require, use etc). Where else might we usefully probe? What would people find useful? * start of loop; * loop iteration (either by the end-condition not being met or 'next'); * end of loop (either by the end-condition being met or 'last'); if that's practical. I'll have to benchmark it - but if doesn't cost any performance I'm sure it's possible. -- Andy Armstrong, Hexten
Re: The spewing problem.
Matisse Enzer wrote: Ok, why do you want to stop it as fast as possible when a failure occurs? So I can more quickly focus on fixing that first test failure. I use make test 21 | less Works for individual tests too make perl -Mblib t/testname.t 21 | less I don't see how this stops running the test suite upon the first failure, can you explain please? Sure. What happens is that less doesn't read all of its input. As soon as the script emits more than 8k of output (or whatever the pipe buffer happens to be) it blocks on the next write() call that it makes and doesn't run any more tests. If you quit less, the test script gets a SIGPIPE and probably quits. Ok, that's not the first failure, but you'll always see the first failure on your screen with that. Or, back to the question Schwern posed, Ok, it's nice to want things, but why do you want it? Still curious - perhaps you can explain more about why you think this is useful thing. Sam.
Re: The spewing problem.
Because if my test script is startled, I want it to stop and fall over. Google for feinting goat. Adam K On 14/01/2008, Michael G Schwern [EMAIL PROTECTED] wrote: Adam Kennedy wrote: This shouldn't be any more complicated than -g (where g in my case stands for goat as in feinting goat) Ok, I'll bite. Why a goat and why is it feinting? -- ...they shared one last kiss that left a bitter yet sweet taste in her mouth--kind of like throwing up after eating a junior mint. -- Dishonorable Mention, 2005 Bulwer-Lytton Fiction Contest by Tami Farmer
Re: The spewing problem.
On Monday 14 January 2008 02:27:56 Ovid wrote: --- chromatic [EMAIL PROTECTED] wrote: How is this simpler than 'bail on fail' or 'die on fail'? It doesn't conflate output TAP results from tests with interpret TAP results from tests. Neither does die on fail. Die on fail doesn't violate that in the slightest because it's the *tests* and Test::Builder which determines what TAP is output and how the tests behave. Now if I have a test suite which takes 45 minutes to run and I get a failure 3 minutes into that, it's not an *insane* idea to say hey, let's stop the test suite and not tie up my resources for another 42 minutes. I agree, but Test::Builder doesn't run test *suites*. Test::Builder doesn't *run* anything. To solve organizational problems of a test suite, address it at the level of code that manages the entire suite. -- c
Re: The spewing problem.
Still curious - perhaps you can explain more about why you think this is
useful thing.
it's useful to me because I say it is. personally I don't feel the need
to defend something many people would like to see this like we're being
forced to here. schwern has a valid point in not wanting to lose
diagnostics upon implementing this feature, but outside of that it
wastes too many cycles going back and forth like this over what is a
pretty minimal feature.
as for the diagnostics bit, it seems like the root cause of this is that
the diagnostics functionality is separated from the comparison
functionality in T::B in each of the different comparison subroutines.
I think if ok() were to accept a diagnostic callback as a final argument
things might work as we'd like. that is, instead of
$self-ok($test, $name);
$self-_is_diag($got, 'eq', $expect) unless $test;
return $test;
we'd have
$self-ok($test, $name, sub { $self-_is_diag($got, 'eq', $expect) });
return $test;
and then the bailout logic could include diagnostics before quitting. at
least for the functionality I'd most like to see - ovid has something
similar but different in mind, so that would probably need a different
approach.
or somesuch :)
--Geoff
Re: The spewing problem.
Michael G Schwern wrote: Michael Peters wrote: make test || echo -e \a I keep digging away at this because I'm looking for a problem other than I want to see the first failure. And that's what I'm hearing from you and from Matisse and everyone else. Yours is a little different, it's I want to be alerted on first failure. Not just that I want to be alerted. I want to be able to work and fix things. A 45 minute test suite is also going to be machine intensive (and in my case have other services running that I'd need to stop/restart to be able to troubleshoot). So having the whole test suite come to a stop is actually important. Just knowing that a failure happened 3 minutes in isn't much help if I don't get to immediately start working on the fix. You see how this is distinct from halt on first failure? It gives me a lot more room for different solutions that don't involve just cutting off all the following information. I can see why sometimes and (some people) want all the information and I don't think you should remove that option. Just add the option to stop for those of us who don't. -- Michael Peters Developer Plus Three, LP
Re: What should it's name be?
Gabor Szabo wrote: I know I am a bit late to the party but what about Test::Anything ? Rapidly drifting towards Test::Anything::Protocol. -- But there's no sense crying over every mistake. You just keep on trying till you run out of cake. -- Jonathan Coulton, Still Alive
Re: The spewing problem.
Michael Peters wrote:
Michael G Schwern wrote:
Michael Peters wrote:
make test || echo -e \a
I keep digging away at this because I'm looking for a problem other than I
want to see the first failure. And that's what I'm hearing from you and from
Matisse and everyone else. Yours is a little different, it's I want to be
alerted on first failure.
Not just that I want to be alerted. I want to be able to work and fix things. A
45 minute test suite is also going to be machine intensive (and in my case have
other services running that I'd need to stop/restart to be able to
troubleshoot). So having the whole test suite come to a stop is actually
important. Just knowing that a failure happened 3 minutes in isn't much help if
I don't get to immediately start working on the fix.
You see how this is distinct from halt on first failure? It gives me a lot
more room for different solutions that don't involve just cutting off all the
following information.
I can see why sometimes and (some people) want all the information and I don't
think you should remove that option. Just add the option to stop for those of us
who don't.
If we're concerned about losing the diagnostics, then just have it fail
on the NEXT test...
Put the...
if ( any failures ) { bail out }
... at the beginning of the ok call, not at the end.
Or give us the damned rope and allow us to make it bail, knowing we
don't give one hoot about the diagnosics.
If we really care, we can just run that single test again with verbose
on anyway.
Adam K
Re: The spewing problem.
Test::Builder should just do what it's told. If the code that IS responsible for testing suites tells it to bail on fail, that's exactly what it should do. Adam K chromatic wrote: On Monday 14 January 2008 02:27:56 Ovid wrote: --- chromatic [EMAIL PROTECTED] wrote: How is this simpler than 'bail on fail' or 'die on fail'? It doesn't conflate output TAP results from tests with interpret TAP results from tests. Neither does die on fail. Die on fail doesn't violate that in the slightest because it's the *tests* and Test::Builder which determines what TAP is output and how the tests behave. Now if I have a test suite which takes 45 minutes to run and I get a failure 3 minutes into that, it's not an *insane* idea to say hey, let's stop the test suite and not tie up my resources for another 42 minutes. I agree, but Test::Builder doesn't run test *suites*. Test::Builder doesn't *run* anything. To solve organizational problems of a test suite, address it at the level of code that manages the entire suite. -- c
Re: The Star Trek: Generations problem.
On 15 Jan 2008, at 00:11, chromatic wrote: This is why making Test::Builder bail out the whole test suite is a Flies Through Space* solution. If you want to stop running the test suite, stop the thing that runs the test suite from running the test suite. So, back to my suggestion... :) # Don't launch any more tests after ten failed assertions have been seen $ prove --max-fail 10 -- Andy Armstrong, Hexten
Re: The Star Trek: Generations problem.
On Monday 14 January 2008 15:42:49 Adam Kennedy wrote: Test::Builder should just do what it's told. If the code that IS responsible for testing suites tells it to bail on fail, that's exactly what it should do. Let me rephrase this in an amusing way which demonstrates how silly this idea is. We have a program which runs suites of tests. It does this by spawning new processes which produce TAP somehow. Some of those processes produce TAP by using Perl modules which eventually use Test::Builder. Some do not. They don't have to, because there's a separation between producing TAP and consuming TAP, by design. This is why we have a program which runs suites of tests. Now some people say If anything which generates TAP generates a TAP failure, I don't want to run the rest of the test suite. They have plenty of potential reasons, and those reasons are reasonable reasons. We do have a program which runs suites of tests. Actually, we can have several, because one of the goals of writing TAP::Parser which is now much of the guts of Test::Harness 3.0 is to allow people to write programs which run suites of TAP producers and consume and aggregate the resulting TAP. The point is customized behavior. That's a digression though, because as I understand it the goal of putting this OH NOES SOMETHING FALLDED!!! logic in Test::Builder is so that the currently running test file (necessarily Perl, necessarily using Test::Builder, so see the caveats about neither necessarily being true for TAP generators) can stop running when it encounters a failure. Of course, that won't stop a whole test suite from continuing to run. Dot dot dot. ... unless you *could* figure out some way of writing a temporary file somewhere that Test::Builder could check to see if it's supposed to abort subsequent test files in the suite (caveats again applying about how they have to be Perl programs which somehow use Test::Builder) and just not run the tests. That's a little tricky though, if you use parallel test runs, which has become one of the other goals of TAP::Parser, and I'm all for that. I tried to think of a way to use environment variables set by one test process to communicate with another process, but then you've limited this to working only in certain shells on Windows for Perl programs which somehow use Test::Builder. At least you can get away from the nasty file locking race condition autoflush problem of using temporary files to communicate Hey I launched the process but ha ha just fooling not really why don't you die now thanks? to child processes. It's not as if THAT test failed. Some other test did. Or, you know, stop launching new processes to generate TAP from the test suite runner, which itself knows a few things about launching new processes and is customizable, thanks to one of the goals of TAP::Parser. This is why making Test::Builder bail out the whole test suite is a Flies Through Space* solution. If you want to stop running the test suite, stop the thing that runs the test suite from running the test suite. Bridge on the captain, -- c * http://www.mail-archive.com/perl-qa@perl.org/msg07433.html
Re: The Star Trek: Generations problem.
On Monday 14 January 2008 16:18:22 Andy Armstrong wrote: So, back to my suggestion... :) # Don't launch any more tests after ten failed assertions have been seen $ prove --max-fail 10 I can't see how anything else will work in a general sense, at least without severe limitations and hack piled upon hack. -- c
Re: The Star Trek: Generations problem.
On Jan 14, 2008, at 4:18 PM, Andy Armstrong wrote: # Don't launch any more tests after ten failed assertions have been seen $ prove --max-fail 10 That works for me, especially if it's a switch I can pass to MakeMaker and Module::Build MAX_FAIL=1 make test and ./Build test --max-fail 1 --- Matisse Enzer [EMAIL PROTECTED] http://www.matisse.net/ - http://www.eigenstate.net/
Re: The Star Trek: Generations problem.
chromatic wrote: On Monday 14 January 2008 15:42:49 Adam Kennedy wrote: Test::Builder should just do what it's told. If the code that IS responsible for testing suites tells it to bail on fail, that's exactly what it should do. Let me rephrase this in an amusing way which demonstrates how silly this idea is. We have a program which runs suites of tests. It does this by spawning new processes which produce TAP somehow. Some of those processes produce TAP by using Perl modules which eventually use Test::Builder. Some do not. They don't have to, because there's a separation between producing TAP and consuming TAP, by design. This is why we have a program which runs suites of tests. Now some people say If anything which generates TAP generates a TAP failure, I don't want to run the rest of the test suite. They have plenty of potential reasons, and those reasons are reasonable reasons. We do have a program which runs suites of tests. Actually, we can have several, because one of the goals of writing TAP::Parser which is now much of the guts of Test::Harness 3.0 is to allow people to write programs which run suites of TAP producers and consume and aggregate the resulting TAP. The point is customized behavior. That's a digression though, because as I understand it the goal of putting this OH NOES SOMETHING FALLDED!!! logic in Test::Builder is so that the currently running test file (necessarily Perl, necessarily using Test::Builder, so see the caveats about neither necessarily being true for TAP generators) can stop running when it encounters a failure. Of course, that won't stop a whole test suite from continuing to run. Dot dot dot. I'm confused here, but that's probably because I'm not intimate with T::B and T::H: =item BBAIL_OUT $Test-BAIL_OUT($reason); Indicates to the Test::Harness that things are going so badly all testing should terminate. This includes running any additional test scripts. this is what I'm trying to to - generate TAP that communicates with TAP consumer something along the lines of yikes, stop what you are doing as soon as you can. the patch I submitted to Test::Builder just calls BAIL_OUT() at $time. I think this is a legitimate thing to communicate between the TAP producer and TAP consumer, even across disparate languages (which I'm in the middle of regularly :) is this not reasonable? --Geoff
Re: The spewing problem.
Geoffrey Young wrote: schwern has a valid point in not wanting to lose diagnostics upon implementing this feature, but outside of that it wastes too many cycles going back and forth like this over what is a pretty minimal feature. Stop wasting cycles arguing, and start posting patches then. If it's that minimal to implement, show us the code, and it can be reviewed. Sam.
Re: The Star Trek: Generations problem.
On Monday 14 January 2008 18:14:54 Adam Kennedy wrote: I think we might be mentally going about this wrong. I'll say. Running tests or not running tests or ceasing to run tests as they're running is *not* T::B's problem. You cannot solve it reliably at the T::B level. You can't even guarantee that the TAP you get comes from T::B, or Perl 5, or any version of Perl or a programming language in the entire Perl metaverse, or even from something that runs code on the machine where you set the environment variable when you launch it and can pay attention to an environment variable at all. This is by design. This is a feature. This is why TAP exists. This is why and how the full Parrot test suite, languages and everything, has tests written in several different languages and people who run 'make test' do not have to care and people who write tests have to care only long enough to write (or copy) a two-line harness and a little shim to use the right testing library. Stop looking in T::B for the solution. It is not there. It does not belong there. If you want to stop running a long test suite when it produces 1 .. n failures, do it in the code that *runs* the test suite. That, at least, has the chance of being at least somewhat reliable, which is sort of a nice characteristic in software such as testing libraries and testing harnesses. I know this is Perl and specifically Perl 5, but that doesn't mean we have to pile up fragile hacks with enough corner cases to make MIT Building 38 look boring in our documented protocols too. -- c
Re: The Star Trek: Generations problem.
On Jan 14, 2008, at 6:36 PM, chromatic wrote: If you want to stop running a long test suite when it produces 1 .. n failures, do it in the code that *runs* the test suite. That seems quite reasonable. Is anyone here saying otherwise? --- Matisse Enzer [EMAIL PROTECTED] http://www.matisse.net/ - http://www.eigenstate.net/
Re: The spewing problem.
On Jan 13, 2008, at 8:22 PM, Adam Kennedy wrote: Because if my test script is startled, I want it to stop and fall over. Google for feinting goat. Adam K fainting = falling unconscious feinting = making a false attack to lure your opponent off guard before the real attack A feinting goat makes for an amusing mental image. Perhaps the test script emits ^G a whole lot of times while doing 'rm -rf /' in the background? Chris
Re: The Star Trek: Generations problem.
For the record, I don't like the approach of requiring the failure condition to have the ability to do math (even if it's just counting). It complicates things quite a bit, and means that it will require more than three characters to indicate you want fainting-mode. Adam K On 15/01/2008, Matisse Enzer [EMAIL PROTECTED] wrote: On Jan 14, 2008, at 4:18 PM, Andy Armstrong wrote: # Don't launch any more tests after ten failed assertions have been seen $ prove --max-fail 10 That works for me, especially if it's a switch I can pass to MakeMaker and Module::Build MAX_FAIL=1 make test and ./Build test --max-fail 1 --- Matisse Enzer [EMAIL PROTECTED] http://www.matisse.net/ - http://www.eigenstate.net/
Re: The Star Trek: Generations problem.
On 15/01/2008, chromatic [EMAIL PROTECTED] wrote: On Monday 14 January 2008 15:42:49 Adam Kennedy wrote: Test::Builder should just do what it's told. If the code that IS responsible for testing suites tells it to bail on fail, that's exactly what it should do. Let me rephrase this in an amusing way which demonstrates how silly this idea is. We have a program which runs suites of tests. It does this by spawning new processes which produce TAP somehow. Some of those processes produce TAP by using Perl modules which eventually use Test::Builder. Some do not. They don't have to, because there's a separation between producing TAP and consuming TAP, by design. This is why we have a program which runs suites of tests. Now some people say If anything which generates TAP generates a TAP failure, I don't want to run the rest of the test suite. They have plenty of potential reasons, and those reasons are reasonable reasons. We do have a program which runs suites of tests. Actually, we can have several, because one of the goals of writing TAP::Parser which is now much of the guts of Test::Harness 3.0 is to allow people to write programs which run suites of TAP producers and consume and aggregate the resulting TAP. The point is customized behavior. That's a digression though, because as I understand it the goal of putting this OH NOES SOMETHING FALLDED!!! logic in Test::Builder is so that the currently running test file (necessarily Perl, necessarily using Test::Builder, so see the caveats about neither necessarily being true for TAP generators) can stop running when it encounters a failure. Of course, that won't stop a whole test suite from continuing to run. Dot dot dot. ... unless you *could* figure out some way of writing a temporary file somewhere that Test::Builder could check to see if it's supposed to abort subsequent test files in the suite (caveats again applying about how they have to be Perl programs which somehow use Test::Builder) and just not run the tests. That's a little tricky though, if you use parallel test runs, which has become one of the other goals of TAP::Parser, and I'm all for that. I tried to think of a way to use environment variables set by one test process to communicate with another process, but then you've limited this to working only in certain shells on Windows for Perl programs which somehow use Test::Builder. At least you can get away from the nasty file locking race condition autoflush problem of using temporary files to communicate Hey I launched the process but ha ha just fooling not really why don't you die now thanks? to child processes. I think we might be mentally going about this wrong. A number of people thinking mostly about the practicalities of this seem to be thinking in terms of perfection, and really, perfection is not required. I'm increasingly thinking a common environment variable is the best communication mechanism. For the sake or argument lets call it PERL_TEST_FAINTING_GOAT Set the environment variable to true, and any components of the test process are instructed to abort and fall over as early as possible, with a methodology appropriate to that particular component. For Test::Builder, this is going to mean basically bail on fail, with either an immediate bail or waiting until the next test (so as to assure diagnostics are in place). For the test harness, that means not running any more tests once a failure has been spotted, and either stopping any currently-running tests or ignoring any further results (in parallel mode). Since the entire purpose of the testing architecture we have is to allow diversity, perfection is more or less impossible. That's no reason not to treat this as a do the best you can to abort situation and encourage situation-specific behavior. Adam K
Re: The spewing problem.
--- Adam Kennedy [EMAIL PROTECTED] wrote:
If we're concerned about losing the diagnostics, then just have it
fail
on the NEXT test...
Put the...
if ( any failures ) { bail out }
... at the beginning of the ok call, not at the end.
FYI: That was Aristotle's suggestion and that's exactly what
Test::Most does (well, that and a DESTROY block to keep identical
semantics if there is no next test).
Cheers,
Ovid
--
Buy the book - http://www.oreilly.com/catalog/perlhks/
Perl and CGI - http://users.easystreet.com/ovid/cgi_course/
Personal blog - http://publius-ovidius.livejournal.com/
Tech blog - http://use.perl.org/~Ovid/journal/
