On 1 Jan 2008, at 18:47, Eric Wilhelm wrote:
# from Ovid
# on Tuesday 01 January 2008 00:12:
[snip]
This is the sort of stuff that tests are designed to catch, but stuff
this bad *might* get missed with tight process boundaries.
...
(such as the time someone was parsing
Data::Dumper output without considering that I may have set
$Data::Dumper::Indent to a different value than the default).
What are the chances that tests aggregated in this way will *actually*
catch the $D::D::Indent issue? The bad assumptions might still work,
right?
Yup. However it's more likely to be visible this way. I think that's
all Ovid was saying.
Perhaps you're trying to address the "code makes global state
assumptions" issue? Well, I think that might be borrowing trouble.
I'm not directly trying to address it, but it's a side-benefit and my
real-world experience (as opposed to just sitting back and thinking
about it), tells me that I gain more than I lose.
Yeah, you caught me thinking again. Too much is left to accident (of
both order and omission.) You *might* find some of these sorts of
bugs, but you'll be looking at them through the same puzzling lens
(i.e. "what the? ... makes no sense!") and scratching your head
just as
much as you would if they were to manifest in a normal test.
I think the point was that:
a) Run them as separate tests scripts - bug invisible
b) Run them in single process - bug visible
I prefer (b). Even if I get a totally opaque error out of it.
Of course if I was perfect I wouldn't have written the bug in the
first place, and I would have a better designed system to make
interaction issues like this less likely, but alas...
If your mission is to speed up the tests, lumping them all into one
file
and running that accomplishes this -- but at the cost of any
distributed or parallel options. And there is a side-effect (whether
or not you claim it as a benefit.)
[snip]
Why is it an either-or situation? I don't think anybody is saying
that distributed/parallel options are a _bad_ idea, or that T::A
should be the one-true-way of running tests. I'm certainly not. I
want parallel/distributed tests. Test farms rock!
Just that the single-process aggregation method isn't the choice of
idiots :-)
Cheers,
Adrian
