# from Andy Armstrong
# on Thursday 01 November 2007 09:33:
>On 1 Nov 2007, at 15:44, A. Pagaltzis wrote:
>> * Andy Armstrong <[EMAIL PROTECTED]> [2007-11-01 15:50]:
>>> The config needs to be dynamic at test time - so it might as
>>> well be a script...
>> But it only needs to be dynamic in a minority of cases. So it
>> seems to me it should be the other way around – rather than run
>> a script to emit the config, have a config that has a directive
>> to run a script and include its output as part of the config.
This would be preferable for introspectability, yes.
>Well if it doesn't need to be dynamic it doesn't need to exist at all
>- you just have the tests you want to run.
But rather than say "not dynamic", let's say "not arbitrary". The
config is always dynamic -- e.g. a canned method require_gui() would
need to check whatever cross-platform checks and also maybe consult the
tester's "allowables" settings.
The distinction between arbitrary and dynamic is that arbitrary code
hinders introspection while dynamic can mean that it simply refers to a
pre-determined check routine (e.g. the Apache Ant 'exec' directive
introduces "arbitrary" into what is otherwise a 100% declarative
manifest.)
>> That makes it easier for other tools to analyse the test configs.
Introspection. Not only can we decide whether or not to run the gui
tests, but we know which ones require the gui. If you just source
`controller.pl` (arbitrary code) and get a valid list for the current
environment, you have no way to tell which of the selections are
affected by which bits of the environment. To determine the gui tests,
you have to turn-off the $DISPLAY (won't work on windows, and possibly
mac) run `controller.pl` again and then compare the new output. Note
the difficulty of handling "I have a gui, but I don't want the tests
messing with it" on platforms like windows.
With a declarative scheme, we have a tree of data and we can simply ask
which selections require_gui(). Those tests can then be run as their
own group, or even simply examined for some other interesting
information.
And even in the standard use-case, if the tester has opted to not have
her gui interrupted by the tests, they don't run. The declarative
scheme makes it easy to "do the right thing" because author education
is reduced to simply: "set require_gui for the tests that need a gui"
(as opposed to e.g. "check $DISPLAY and then check the preferences
file, etc.") It's a lot harder to get it wrong if it involves code
that you didn't write.
>>> So isn't that nearly TSP? :)
>> ... don’t think the harness should look at the
>> output of tests to check if it’s TSP rather than TAP. The list of
>> tests to run should be determined up front
>
>Well that's desirable certainly - but isn't it also valuable to be
>able to dynamically include include or exclude large numbers of tests
>when you need to? And to do so using a well defined protocol?
Possibly, but it shouldn't be part of the t/*.t files and I think we
would do well to make it declarative rather than a protocol emitted by
arbitrary code.
>> , and the test scripts
>> should only ever output TAP.
+1
>> ... YAML ...
>Oh sure - that'd be fine. The choice of representation is somewhat
>orthogonal to whether it's a good idea in the first place though.
I think we've decided that "selectable tests" are a good idea WRT "extra
testing." Declarative requirements are probably just as useful WRT
normal tests. Note though, that Martin mentions "xt/" in the bug
report ;-)
--Eric
--
To a database person, every nail looks like a thumb.
--Jamie Zawinski
---------------------------------------------------
http://scratchcomputing.com
---------------------------------------------------
