On 7/20/06, chromatic <[EMAIL PROTECTED]> wrote:
On Wednesday 19 July 2006 15:17, demerphq wrote:
> On 7/19/06, chromatic <[EMAIL PROTECTED]> wrote:
> No. I didnt imply anything. I spelled it out quite clearly.
Fine, then you said "quite clearly" that it was broken.
The whole reason this thread started was that i suggested that test
descriptions be mandatory, as they were in my opinion the best way to
resolve this problem.
When the subject of wrong line numbers came up I was writing in
context of replying to someone who felt that the line number reporting
of Test::Builder was sufficient to not require test descriptions. I
responded by pointing out that such information is not so useful as it
is often wrong.
The fact that we have digressed into discussing the myriad ways that
one can get wrong line number reports and whether they are bugs in the
test file or in the builder is a totally different discussion.
Now i consider "wrong" to be different from "broken" or "buggy". To me
broken and buggy mean that the reporting doesnt do what its supposed
to do. But since I know that what its supposed to do is report where a
given Test::Builder based test routine was called I dont consider it
broken when it does exactly that. I call it wrong because often
finding where the routine is called doesnt help you find the code
being tested. And thats what started this whole thing off.
> I said that the code to find the line number uses a heurisitic that
> gets things wrong, and that I didn't see any way to improve the
> heurisitc.
But there is one! Give it the explicit knowledge that you, the writer of the
test suite, has.
Right. Give it a damn name so i can find it in an editor with its
built in search function.
> And therefore that the line number information is an
> unreliable way to find what code actually failed the test, which in my
> experience is a problem properly solved by using test names.
If you don't give it a test name, you have the exact same problem, only worse,
because there's absolutely *no* information. Test::Builder has a heuristic
for giving the location of failures and it works *most* of the time.
Test::Builder has no heuristic for giving test names in the absence of test
names, and thus it works *none* of the time.
Thus make test names mandatory, and the problem is solved.
> > > use Test::More tests => 3;
> > >
> > > sub my_ok {
> > > ok($_[0],$_[1]);
> > > }
> >
> > I don't know why you'd expect this to report the right line numbers; this
> > code really *is* broken.
>
> No, its not broken, using subroutines is not broken.
No, using buggy, incomplete subroutines is. At least complaining that an
incomplete, buggy subroutine does the wrong thing is silly. Of course it
does the wrong thing. You wrote it to do the wrong thing.
Show me where this is documented in Test::More and we can discuss why
I didnt follw that documentation. Until its there you have right
calling my usage buggy.
Would you complain that my favorite chocolate chip cookie recipe is awful if
you deliberately left out the sugar?
If the sugar wasnt in the recipie, but was rather documented somewhere
else, perhaps under "cookie builder" then yes I would.
> And, even it were
> broken, its a common mistake. The code is basically a stripped down
> example of stuff i see in test files A LOT.
Yes, it's common. That doesn't mean it's not broken. I'm not in the habit of
voting whether to redefine wrong behavior as right because it's common.
If its so easy to use wrong then its probably deficient.
> And even if i were to concede that my_ok() is broken (which I don't)
> there is still Fergals example of a data driven tests in a loop:
>
> my @tests=(....);
>
> foreach my $test (@tests) {
> is($test->[0],$test->[1]);
> }
>
> how do you propose to get a useful line number out of that? Are you
> going to say that its broken as well?
Not at all. The line number is correct: there are no additional call frames
between the call to the underlying library and the call point of the test.
The line number reported there is the correct line number for the important
point of calling the test.
I didn't say it was useful. I said it was correct.
Correct in some restricted definition. It correctly tells you where
the test procedure was called. It often tells you the wrong thing
about what failed in the test file.
> Now if it was written
>
> foreach my $test (@tests) {
> is($test->[0],$test->[1],$test->[2]);
> }
>
> Then i can find the test _easily_.
For a value of "easily" defined as "by looking up the definition of the data
structure in @tests and then reading further in the code until I find the
point of the call", yes.
What? If that test fails i will see the value of $test->[2] which will
be somewhere in the test file. How hard is that?
I prefer to have two points of information for debugging this sort of test:
the diagnostic and the line number and source file of the call point.
If i have the description i need neither. I can find everything else
quickly from that.
> And speaking of
> $Test::Builder::Level at me let me ask a question, how many are going
> to read Test::Builder to get the line numbers from tests in Test::More
> right? Experience shows not very many people[1].
People who don't read the documentation and get bad results get approximately
zero sympathy from me.
We are talking about me debugging some other guys test file. Im the
one that suffers because test descriptions are optional and because he
didnt either find $Test::Builder::Level (which is not documented where
the routines he was using were documented) or didnt notice for the
reasons i mentioned earlier. Either way i suffer for somethign i
shouldnt have to, and that isnt my fault.
> Heck the variable isn't even mentioned in Test::More.
Then it should be, though maybe Test::Tutorial is a better place for it.
> And Test::Builder isn't mentioned in Test::Simple at all (presumably because
> it doesnt use it, in which case $Test::Builder::Level isnt going to help.)
That's quite an assumption. Write a test case.
Why, what does it matter? Its not mentioned in the docs of the module
so its not even relevent if it is or isnt in the context of this
discussion. You cant say "RTFM" when its not even in the FM.
D:\dev\perl>perldoc Test::Simple | grep "Test::Builder"
D:\dev\perl>
> [1] I might wager that a lot of test authors dont even notice the
> problem. I say this because in my own experience of writing my own
> test suites the line number is something I dont even look at. I know
> what test failed because I wrote the test file and the code its
> testing and I can find the relevent stuff in an instant. Finding the
> code responsible for a failing test is something that IMO is done more
> often by module consumers who for one reason or another are seeing
> things go wrong. The author on the other hand is unlikey to have seen
> the tests fail, and therefore might not even know the line numbers are
> wrong.
Lazy people often write buggy code. I sure do.
Everybody who writes code writes buggy code some of the time.
Yves
--
perl -Mre=debug -e "/just|another|perl|hacker/"
