Re: TODO tests for outstanding bugs?
On Fri, Jul 18, 2003 at 01:54:24PM -0700, chromatic wrote: > In general, the person who fixes the bug writes a regression test for > the bug. The pumpkings and other porters are really good about making > sure patches have tests. Yes, but I'm talking about outstanding bugs that don't have tests yet... > Putting them as TODO tests might just shift the problem -- too many > bugs, not enough people looking at RT to solve them -- into "not enough > people are looking at the TODO tests to solve them". But putting them in the code moves them closer to the pain. Then they're part of the language, rather than one stage removed in a database somewhere. We can also configure things like the smoke test to report on them. If we could mark their severity in some way too, then we could have a better idea of whether or not a new release can happen ... Plus all the usual Test-First reasons to have them before someone writes the patch ... > One idea is attaching a simple test case to every bug report that > doesn't have test code that's nearly right for the core. It's a lot > easier to touch up a test case than it is to write one, so we could do > a lot of good by turning bug reports into executable tests without > having to worry about where they'll eventually end up. I quite like the idea of just annotating the bugs with test cases where applicatable. But I'd prefer to turn them into *real* test cases - not just documented ones... Tony
Re: TODO tests for outstanding bugs?
On Fri, Jul 18, 2003 at 06:36:41PM +0100, Tony Bowden wrote:
> What's the current approach to turning perlbugs into tests?
>
> Should they be done as TODOs? Is there a distinct set of test files for
> them? etc? Can they use Test::More? etc. etc. etc.
>
> e.g. http://bugs6.perl.org/rt2/Ticket/Display.html?id=5430
>
> can have a fairly simple test like:
>
> package main;
> eval '::is(__PACKAGE__, "main", "Package is main inside eval")';
> package Foo;
> eval '::is(__PACKAGE__, "Foo", "Package is Foo inside eval")';
>
> But how we actually get that into the core somewhere?
Just stick it into the core as a normal TODO test. This way we at least
know when it gets fixed and can close the bug.
In this case, t/op/eval.t. Either Test::More can be used or t/test.pl, a
similar interface that uses less Perl core features. In the case of
eval.t, use t/test.pl since its a pretty basic op. There's some more
docs about this in perlhack.
Often, a test will not be written using Test::More or test.pl. In that
case you can introduce new tests at the bottom and update the counter
manually using curr_test() so the old and new styles live together rather
than having to rewrite the whole test.
--- t/op/eval.t 2003/07/19 04:13:11 1.1
+++ t/op/eval.t 2003/07/19 04:23:33
@@ -5,7 +5,8 @@
@INC = '../lib';
}
-print "1..91\n";
+require './test.pl';
+print "1..93\n";
eval 'print "ok 1\n";';
@@ -381,7 +382,7 @@
print DB::db5() == 3 ? 'ok' : 'not ok', " $test\n"; $test++;
print db6() == 4 ? 'ok' : 'not ok', " $test\n"; $test++;
}
-require './test.pl';
+
$NO_ENDING = 1;
# [perl #19022] used to end up with shared hash warnings
# The program should generate no output, so anything we see is on stderr
@@ -437,4 +438,15 @@
print "$r$c" eq 'SS' ? "ok " : "# '$r$c' ne 'SS'\nnot ok ", $test++, "\n";
eval $code;
print $c eq 'V' ? "ok " : "# '$c' ne 'V'\nnot ok ", $test++, "\n";
+}
+
+
+{
+ curr_test($test);
+ package main;
+ eval q{::is(__PACKAGE__, 'main', 'Package is main inside eval')};
+
+ local $TODO = 'Bug #5430';
+ package Foo;
+ eval q{::is(__PACKAGE__, 'Foo', 'Package is Foo inside eval')};
}
TODO tests work fine in the core, but a normal 'make test' doesn't currently
report them. 'make test_harness' will, but they're still a little hard to
track down because the harnesses don't have much support for that part
at the moment.
--
"A Masterpiece."
"Well, better than average, maybe."
Re: TODO tests for outstanding bugs?
chromatic wrote in perl.qa : > One idea is attaching a simple test case to every bug report that > doesn't have test code that's nearly right for the core. It's a lot > easier to touch up a test case than it is to write one, so we could do > a lot of good by turning bug reports into executable tests without > having to worry about where they'll eventually end up. "Write tests before fixes" :) This always help. For such standalone 1-bug test scripts, Test::More is (most of the time) ok. However there are places in the core where Test::More can't be used, simply because they're tests for core features used by Test::More.
Re: TODO tests for outstanding bugs?
On Friday, July 18, 2003, at 10:36 AM, Tony Bowden wrote: What's the current approach to turning perlbugs into tests? In general, the person who fixes the bug writes a regression test for the bug. The pumpkings and other porters are really good about making sure patches have tests. Should they be done as TODOs? Is there a distinct set of test files for them? etc? Can they use Test::More? etc. etc. etc. I like the idea and I can see how it would make some things easier. We might get a lot of benefit from a little simpler idea, though. Putting them as TODO tests might just shift the problem -- too many bugs, not enough people looking at RT to solve them -- into "not enough people are looking at the TODO tests to solve them". Library bugs can use Test::More. Language bugs may or may not, depending on how complex the bug seems to be. (A bug in print would likely need testing in t/op/print.t.) I could imagine a test case being massaged and mangled a couple of times before it's ready to be checked in. As a matter of policy, if it were up to me, I'd hesitate to go quite that far. e.g. http://bugs6.perl.org/rt2/Ticket/Display.html?id=5430 can have a fairly simple test like: package main; eval '::is(__PACKAGE__, "main", "Package is main inside eval")'; package Foo; eval '::is(__PACKAGE__, "Foo", "Package is Foo inside eval")'; But how we actually get that into the core somewhere? One idea is attaching a simple test case to every bug report that doesn't have test code that's nearly right for the core. It's a lot easier to touch up a test case than it is to write one, so we could do a lot of good by turning bug reports into executable tests without having to worry about where they'll eventually end up. -- c
TODO tests for outstanding bugs?
What's the current approach to turning perlbugs into tests? Should they be done as TODOs? Is there a distinct set of test files for them? etc? Can they use Test::More? etc. etc. etc. e.g. http://bugs6.perl.org/rt2/Ticket/Display.html?id=5430 can have a fairly simple test like: package main; eval '::is(__PACKAGE__, "main", "Package is main inside eval")'; package Foo; eval '::is(__PACKAGE__, "Foo", "Package is Foo inside eval")'; But how we actually get that into the core somewhere? Tony
