Re: [tap-l] Mad TAP proposal

[Andy Armstrong]

On 1 Nov 2007, at 00:11, Eric Wilhelm wrote:

But what about a more general mechanism? A TAP directive that means
'schedule these other tests'. So then you'd have a controller test
which was the only one directly visible to Test::Harness and that'd
decide which other tests to run.


It sounds like it would be re-creating a lot of the same functionality
needed for declarative extra testing and/or Test::Manifest.

 http://scratchcomputing.com/tmp/extra_testing.txt

I would rather it not be a TAP directive.  Yes, it abstracts
the "manifesting" into TAP, but it still requires some code to run to
determine the manifest, and will therefore be less introspectable.



Yeah - I slept on it. It should be TSP - Test Steering Protocol. If a  
test emits TAP it's a test, if it emits TSP it's a controller. TAP  
should stay pure.


--
Andy Armstrong, Hexten

Re: [tap-l] Mad TAP proposal

[Andy Armstrong]

On 1 Nov 2007, at 08:19, Andy Armstrong wrote:

Yeah - I slept on it. It should be TSP - Test Steering Protocol. If a
test emits TAP it's a test, if it emits TSP it's a controller. TAP
should stay pure.



http://testanything.org/wiki/index.php/TSP_Test_Steering_Protocol

--
Andy Armstrong, Hexten

Re: [tap-l] Mad TAP proposal

[Michael Peters]

Andy Armstrong wrote:

> Yeah - I slept on it. It should be TSP - Test Steering Protocol. If a
> test emits TAP it's a test, if it emits TSP it's a controller. TAP
> should stay pure.

Sounds to me like it should just be a custom test harness. If the user needs to
dynamically figure out which tests to run, then why not create a custom harness
which will do what they need. This way they can have control over not only which
files to run, but which routines (if they're using Test::Class), etc, etc.
Writing your own harness is trivial now. I'd say writing something to emit a new
protocol is no easier than writing a script that picks the test files/subs to 
run.

-- 
Michael Peters
Developer
Plus Three, LP

Re: [tap-l] Mad TAP proposal

[Andy Armstrong]

On 1 Nov 2007, at 14:12, Michael Peters wrote:


Andy Armstrong wrote:


Yeah - I slept on it. It should be TSP - Test Steering Protocol. If a
test emits TAP it's a test, if it emits TSP it's a controller. TAP
should stay pure.


Sounds to me like it should just be a custom test harness. If the  
user needs to
dynamically figure out which tests to run, then why not create a  
custom harness
which will do what they need. This way they can have control over  
not only which
files to run, but which routines (if they're using Test::Class),  
etc, etc.
Writing your own harness is trivial now. I'd say writing something  
to emit a new
protocol is no easier than writing a script that picks the test  
files/subs to run.



Yeah, but from the user's PoV this is pretty easy:

# t/controller.t
use Test::Steering;

include ('xt/frob') if frob_avail();
include ('xt/slow') if all_the_time_in_the_world();

And it's not too hard to support in the harness.

--
Andy Armstrong, Hexten

Re: Mad TAP proposal

[A. Pagaltzis]

* Michael Peters <[EMAIL PROTECTED]> [2007-11-01 15:15]:
> Sounds to me like it should just be a custom test harness. If
> the user needs to dynamically figure out which tests to run,
> then why not create a custom harness which will do what they
> need.

I was going to say the same. Then it occured to me that if you
want to make such a harness reusable and you want to make the
test plans introspectable, then such a harness needs to parse
some sort of config. Which is very nearly what TSP is.

I just don’t see the “protocol” part. I’d rather think of it in
terms of the Test Plan Configuration format, and that this be
something specific to TAP::Harness rather than documented widely
as something common to TAP implementations. It veers into
operational questions that I don’t think are applicable to every
scenario where TAP is in use. Because…

> This way they can have control over not only which files to
> run, but which routines (if they're using Test::Class), etc,
> etc.

… TAP encompasses scenarios where the notion of files and test
routines doesn’t even compute.

> Writing your own harness is trivial now. I'd say writing
> something to emit a new protocol is no easier than writing a
> script that picks the test files/subs to run.

Agree. Putting this sort of functionality in the default harness
*is* a good idea so people don’t need to reinvent a common wheel,
and so you can use a declarative language to define this sort of
thing. But just make this a config for TAP::Harness.

Regards,
-- 
Aristotle Pagaltzis // 

Re: [tap-l] Mad TAP proposal

[Michael Peters]

Andy Armstrong wrote:

> Yeah, but from the user's PoV this is pretty easy:
> 
> # t/controller.t
> use Test::Steering;
> 
> include ('xt/frob') if frob_avail();
> include ('xt/slow') if all_the_time_in_the_world();

But if they just do their own harness it becomes:

push('xt/frob', @tests)if frob_avail();
push('xt/slow) if all_the_time_in_the_world();
Test::Harness::runtests(@tests);

-- 
Michael Peters
Developer
Plus Three, LP

Re: Mad TAP proposal

[Andy Armstrong]

On 1 Nov 2007, at 14:31, A. Pagaltzis wrote:

This way they can have control over not only which files to
run, but which routines (if they're using Test::Class), etc,
etc.


… TAP encompasses scenarios where the notion of files and test
routines doesn’t even compute.


Yes - agree 100%. Hence it shouldn't be part of TAP.


Writing your own harness is trivial now. I'd say writing
something to emit a new protocol is no easier than writing a
script that picks the test files/subs to run.


Agree. Putting this sort of functionality in the default harness
*is* a good idea so people don’t need to reinvent a common wheel,
and so you can use a declarative language to define this sort of
thing. But just make this a config for TAP::Harness.



The config needs to be dynamic at test time - so it might as well be a  
script that runs and outputs a description of which tests to run,  
right? So isn't that nearly TSP? :)


--
Andy Armstrong, Hexten

Re: [tap-l] Mad TAP proposal

[Andy Armstrong]

On 1 Nov 2007, at 14:32, Michael Peters wrote:

But if they just do their own harness it becomes:

push('xt/frob', @tests)if frob_avail();
push('xt/slow) if all_the_time_in_the_world();
Test::Harness::runtests(@tests);



Yes, they could. But if you write your own harness you presumably  
lose, e.g.,  the advantages of all the different ways you can run the  
tests via prove. And presumably you couldn't use tk-prove or whatever  
it's called.


It's mixing test logic into harness logic. Isn't it better for the  
tests to be completely self describing and keep the harness generic?  
Then your tests will work with any Test::Harness based tool and any  
compatible TAP harness.


--
Andy Armstrong, Hexten

Re: [tap-l] Mad TAP proposal

[Michael Peters]

Andy Armstrong wrote:
> On 1 Nov 2007, at 14:32, Michael Peters wrote:
>> But if they just do their own harness it becomes:
>>
>> push('xt/frob', @tests)if frob_avail();
>> push('xt/slow) if all_the_time_in_the_world();
>> Test::Harness::runtests(@tests);
> 
> 
> Yes, they could. But if you write your own harness you presumably lose,
> e.g.,  the advantages of all the different ways you can run the tests
> via prove. And presumably you couldn't use tk-prove or whatever it's
> called.

Ok, that's true.

> It's mixing test logic into harness logic. Isn't it better for the tests
> to be completely self describing and keep the harness generic? Then your
> tests will work with any Test::Harness based tool and any compatible TAP
> harness.

Isn't this the problem that SKIP was designed to fix? If it needs to dynamically
determined if a test should run, isn't the test itself the best place to do
that? xt tests shouldn't be run unless explicity done so by the runner of the
tests (a developer) who presumably knows that they are doing.

Instead of "something else" telling the harness which tests to run, why not have
the harness run them all and the individual tests can decide if they don't want
to go through with it or not. If "something else" tells the harness which tests
to run, we loose the information about which tests didn't run and why.

Testing in Perl is really nice and simple and I'm not seeing the benefits or the
problems being solved with this complexity that can't already be solved using
something else.

-- 
Michael Peters
Developer
Plus Three, LP

Re: Mad TAP proposal

[A. Pagaltzis]

* Andy Armstrong <[EMAIL PROTECTED]> [2007-11-01 15:50]:
> The config needs to be dynamic at test time - so it might as
> well be a script that runs and outputs a description of which
> tests to run, right?

But it only needs to be dynamic in a minority of cases. So it
seems to me it should be the other way around – rather than run
a script to emit the config, have a config that has a directive
to run a script and include its output as part of the config.

That makes it easier for other tools to analyse the test configs.

> So isn't that nearly TSP? :)

In some ways. I just don’t think the harness should look at the
output of tests to check if it’s TSP rather than TAP. The list of
tests to run should be determined up front, and the test scripts
should only ever output TAP.

Mixing them with scripts that output TSP feels like a confusion
of concerns to me.

For the configs there are also a number of issues like dealing
well with multiple possible configurations and resolution of
relative paths (esp. in compound configs).

I think this needs a little bit more design. And since YAML is
making its way into TAP anyway, maybe it should be YAML-based
rather than a completely ad-hoc TAPish format?

Regards,
-- 
Aristotle Pagaltzis // 

Re: [tap-l] Mad TAP proposal

[Andy Armstrong]

On 1 Nov 2007, at 15:38, Michael Peters wrote:
Isn't this the problem that SKIP was designed to fix? If it needs to  
dynamically
determined if a test should run, isn't the test itself the best  
place to do
that? xt tests shouldn't be run unless explicity done so by the  
runner of the

tests (a developer) who presumably knows that they are doing.


The feature request that started all this relates to a large test  
suite most of which needs to be skipped if a graphical environment  
isn't available - so presumably it'd involve executing hundreds of  
tests just to find out that they skipped.


Instead of "something else" telling the harness which tests to run,  
why not have
the harness run them all and the individual tests can decide if they  
don't want
to go through with it or not. If "something else" tells the harness  
which tests

to run, we loose the information about which tests didn't run and why.


Having to run each test to find out it didn't want to run doesn't  
scale well. Having a mechanism to dynamically select subsets of tests  
can only encourage people to write more tests I'd have thought.


If we decide on a set of rules for which xt/* tests are executed under  
which circumstances it also gives us a language in which to express  
those rules - so instead of being hard wired into TAP::Harness we'll  
have a little TSP document that could be used with any TAP parser.


I've also seen a discussion of test ordering and the practice of  
numbering tests to guarantee a ordering recently; ordering and more  
complex dependencies could also be handled by a t/controller.t that  
emits TSP.


Testing in Perl is really nice and simple and I'm not seeing the  
benefits or the
problems being solved with this complexity that can't already be  
solved using

something else.



You could presumably have said the same about the idea of TAP based  
testing. You can after all just have a bunch of programs that pass or  
fail and run them from the shell.


I'm being slightly facetious there :)

I agree that parsimony should guide our decisions but I'm not finding  
it hard to think of use cases for TSP.


--
Andy Armstrong, Hexten

Re: [tap-l] Mad TAP proposal

[Andy Armstrong]

On 1 Nov 2007, at 15:44, A. Pagaltzis wrote:


* Andy Armstrong <[EMAIL PROTECTED]> [2007-11-01 15:50]:

The config needs to be dynamic at test time - so it might as
well be a script that runs and outputs a description of which
tests to run, right?


But it only needs to be dynamic in a minority of cases. So it
seems to me it should be the other way around – rather than run
a script to emit the config, have a config that has a directive
to run a script and include its output as part of the config.


Well if it doesn't need to be dynamic it doesn't need to exist at all  
- you just have the tests you want to run.



That makes it easier for other tools to analyse the test configs.


So isn't that nearly TSP? :)


In some ways. I just don’t think the harness should look at the
output of tests to check if it’s TSP rather than TAP. The list of
tests to run should be determined up front


Well that's desirable certainly - but isn't it also valuable to be  
able to dynamically include include or exclude large numbers of tests  
when you need to? And to do so using a well defined protocol?



, and the test scripts
should only ever output TAP.


Yeah, I have sympathy with that. I really don't want to complicate  
things gratuitously.



Mixing them with scripts that output TSP feels like a confusion
of concerns to me.


I can see that too. In that sense the existing "Bail Out" directive  
feels like an anomaly too.



For the configs there are also a number of issues like dealing
well with multiple possible configurations and resolution of
relative paths (esp. in compound configs).


Yup - not impossible though. You can use HTML-like semantics -  
relative paths are resolved relative to the config, abs paths relative  
to some well defined project root.



I think this needs a little bit more design. And since YAML is
making its way into TAP anyway, maybe it should be YAML-based
rather than a completely ad-hoc TAPish format?



Oh sure - that'd be fine. The choice of representation is somewhat  
orthogonal to whether it's a good idea in the first place though.


--
Andy Armstrong, Hexten

Re: [tap-l] Mad TAP proposal

[Andy Armstrong]

On 1 Nov 2007, at 16:53, Geoffrey Young wrote:

anyway, its not the best way to handle things necessarily (I didn't
write it, and it's over 6 years old now :) but it it does come in  
handy

every once in a while, so the idea in general is probably sound.



So that's another use case for TSP right? :)

--
Andy Armstrong, Hexten

Re: dynamic test selection

[Eric Wilhelm]

# from Andy Armstrong
# on Thursday 01 November 2007 09:33:

>On 1 Nov 2007, at 15:44, A. Pagaltzis wrote:
>> * Andy Armstrong <[EMAIL PROTECTED]> [2007-11-01 15:50]:
>>> The config needs to be dynamic at test time - so it might as
>>> well be a script...
>> But it only needs to be dynamic in a minority of cases. So it
>> seems to me it should be the other way around – rather than run
>> a script to emit the config, have a config that has a directive
>> to run a script and include its output as part of the config.

This would be preferable for introspectability, yes.

>Well if it doesn't need to be dynamic it doesn't need to exist at all
>- you just have the tests you want to run.

But rather than say "not dynamic", let's say "not arbitrary".  The 
config is always dynamic -- e.g. a canned method require_gui() would 
need to check whatever cross-platform checks and also maybe consult the 
tester's "allowables" settings.

The distinction between arbitrary and dynamic is that arbitrary code 
hinders introspection while dynamic can mean that it simply refers to a 
pre-determined check routine (e.g. the Apache Ant 'exec' directive 
introduces "arbitrary" into what is otherwise a 100% declarative 
manifest.)

>> That makes it easier for other tools to analyse the test configs.

Introspection.  Not only can we decide whether or not to run the gui 
tests, but we know which ones require the gui.  If you just source 
`controller.pl` (arbitrary code) and get a valid list for the current 
environment, you have no way to tell which of the selections are 
affected by which bits of the environment.  To determine the gui tests, 
you have to turn-off the $DISPLAY (won't work on windows, and possibly 
mac) run `controller.pl` again and then compare the new output.  Note 
the difficulty of handling "I have a gui, but I don't want the tests 
messing with it" on platforms like windows.

With a declarative scheme, we have a tree of data and we can simply ask 
which selections require_gui().  Those tests can then be run as their 
own group, or even simply examined for some other interesting 
information.

And even in the standard use-case, if the tester has opted to not have 
her gui interrupted by the tests, they don't run.  The declarative 
scheme makes it easy to "do the right thing" because author education 
is reduced to simply: "set require_gui for the tests that need a gui" 
(as opposed to e.g. "check $DISPLAY and then check the preferences 
file, etc.")  It's a lot harder to get it wrong if it involves code 
that you didn't write.

>>> So isn't that nearly TSP? :)

>> ... don’t think the harness should look at the
>> output of tests to check if it’s TSP rather than TAP. The list of
>> tests to run should be determined up front
>
>Well that's desirable certainly - but isn't it also valuable to be
>able to dynamically include include or exclude large numbers of tests
>when you need to? And to do so using a well defined protocol?

Possibly, but it shouldn't be part of the t/*.t files and I think we 
would do well to make it declarative rather than a protocol emitted by 
arbitrary code.

>> , and the test scripts
>> should only ever output TAP.

+1

>> ... YAML ...
>Oh sure - that'd be fine. The choice of representation is somewhat
>orthogonal to whether it's a good idea in the first place though.

I think we've decided that "selectable tests" are a good idea WRT "extra 
testing."  Declarative requirements are probably just as useful WRT 
normal tests.  Note though, that Martin mentions "xt/" in the bug 
report ;-)

--Eric
-- 
To a database person, every nail looks like a thumb.
--Jamie Zawinski
---
http://scratchcomputing.com
---

Re: Mad TAP proposal

[brian d foy]

In article <[EMAIL PROTECTED]>, Andy
Armstrong <[EMAIL PROTECTED]> wrote:

> We have this ticket in the Test::Harness RT queue:
> 
> http://rt.cpan.org/Ticket/Display.html?id=29633
> 
> Martin Thurn is asking for a SKIP_OUT directive that would skip all  
> remaining test files and return a PASS.

Well, if we have the Test Steering Protocol, wouldn't the best thing be
to group the tests-to-skip in an exclude directive?

   TSP version 1
   include "non-gui"
   exclude "gui"

I hadn't been paying too close attention, but it looks like TSP might
be able to do the stuff I am doing with Test::Manifest. If TSP become
core, that would be really nice. :)

Re: [tap-l] Mad TAP proposal

[brian d foy]

In article <[EMAIL PROTECTED]>, Eric Wilhelm
<[EMAIL PROTECTED]> wrote:

> # from Andy Armstrong
> # on Wednesday 31 October 2007 16:51:
> 
> >But what about a more general mechanism? A TAP directive that means  
> >'schedule these other tests'. So then you'd have a controller test  
> >which was the only one directly visible to Test::Harness and that'd  
> >decide which other tests to run.
> 
> It sounds like it would be re-creating a lot of the same functionality 
> needed for declarative extra testing and/or Test::Manifest.

I don't care if Test::Manifest gets folded into the new stuff, even if
re-implemented. If I can do the same thing with the core test modules I
don't need Test::Manifest anymore. :)

Test::Manifest can do this though. I list tests in t/test_manifest,
which doesn't have to exist until `make test` time. In t/test_manifest,
there is an ;include directive to include other files that have other
lists of tests, recursively. There's also a way to mark test levels, so
I can run all tests at level 1, or level 2, or whatever without
worrying about which diretories or names there are. All that stuff is
unconnected to the structure of t/.

Re: [tap-l] Mad TAP proposal

[brian d foy]

In article <[EMAIL PROTECTED]>, Andy
Armstrong <[EMAIL PROTECTED]> wrote:

> On 1 Nov 2007, at 14:12, Michael Peters wrote:
> 

> 
> Yeah, but from the user's PoV this is pretty easy:
> 
> # t/controller.t
> use Test::Steering;
> 
> include ('xt/frob') if frob_avail();
> include ('xt/slow') if all_the_time_in_the_world();


That would happen at `make test` time (or whatever M::B does)? When I
want this sort of thing with Test::Manifest, I create t/test_manifest
at
`make time`.

It might be interesting to reflect on the harness though, and write to
the test queue. I don't think I need this. I just wanted to say
"reflect" :)

   use Test::Harness;

   add_test_file( 'foo.t', AT_END ); # or NEXT, or whatever

Re: Mad TAP proposal

[Andy Armstrong]

On 1 Nov 2007, at 21:12, brian d foy wrote:

I hadn't been paying too close attention, but it looks like TSP might
be able to do the stuff I am doing with Test::Manifest. If TSP become
core, that would be really nice. :)



Well I'd like that - but it's not without controversy :)


--
Andy Armstrong, Hexten

Re: [tap-l] Mad TAP proposal

[Andy Armstrong]

On 1 Nov 2007, at 21:22, brian d foy wrote:

# t/controller.t
use Test::Steering;

include ('xt/frob') if frob_avail();
include ('xt/slow') if all_the_time_in_the_world();



That would happen at `make test` time (or whatever M::B does)?


That's what I envisaged, yes. The harness would run t/controller.t  
which would spit out some TSP and in response to that the harness  
would schedule additional tests.



When I
want this sort of thing with Test::Manifest, I create t/test_manifest
at
`make time`.

It might be interesting to reflect on the harness though, and write to
the test queue. I don't think I need this. I just wanted to say
"reflect" :)


Good work :)


  use Test::Harness;

  add_test_file( 'foo.t', AT_END ); # or NEXT, or whatever


Well that's pretty much what I have in mind. TSP isn't imperative - it  
doesn't cause a test to be run immediately - it causes tests to be  
added to - or perhaps removed from the harness's queue.


--
Andy Armstrong, Hexten

Build CPAN Modules with Universal Binaries on Mac OS X

[Matisse Enzer]

Trying to build CPAN modules, e.g. XML::Parser so that the XS  
extensions end up as universal binaries on Mac OS X.


So what do I need to put in makepl_arg and  mbuildpl_arg so that  
LDFLAGS, OTHERLDFLAGS, CFLAGS, and

all get

   -arch ppc -arch i386

and whatever else is needed?

And am I missing some obvious documentation on this?


---
Matisse Enzer <[EMAIL PROTECTED]>
http://www.matisse.net/  - http://www.eigenstate.net/