Re: META.yml feature for autotesters?

[Adam Kennedy]

Tyler MacDonald wrote:

How does everybody feel about making this a defined "feature" in
the META.yml spec? Something like:

optional_features:
  - automated_testing:
  description: Automated testing of all of this package's features
  requires:
DBD::SQLite2: 0

If an optional_feature field for packages useful for automated
testing was standardized, functionality could be added to CPAN::YACSmoke and
PITA to take advantage of it, and then there would be an easy way for an
author to define prerequisites for doing an exhaustive smoke test of their
distribution.

What do y'all think?


Problem number one is unlike debian packages or other single-platform 
packages the META.yml files are not inherently stable. They can only be 
indicative of dependencies, not authorative.


While a META.yml file provides a good description of what is required, 
the dependencies WILL change once the Metafile.PL runs.


Adding an automated_testing key only really serves to add logic to the 
META.yml, logic that is less accurate and less capable than the Perl 
code in the Makefile.PL or Build.PL.


I'm more than happy for the Makefile.PL to work out when it wants extra 
dependencies.


To give you some more data points, imagine the automated testing 
additions applied only on Win32. How would you then specify the deps?


Apart from optional features, which apply across platforms, and across 
all installation/testing modes, I think adding more conditional logic to 
the META.yml should be considered bad.


Adam K

Re: META.yml feature for autotesters?

[Adam Kennedy]

While a META.yml file provides a good description of what is required, 
the dependencies WILL change once the Metafile.PL runs.


Actually, that should read "will often change, 90% of the time not 
turning on the dynamic_config flag in the process".


So while we are on the subject of META.yml, I think the dynamic_config 
approach is horrible, because it defaults to an efficient error case and 
relies on the author to fix the error, rather than defaulting to the 
inefficient correct case, and giving the option to make it faster.


Where would be the best place to get this flag changed (or for a 
discussion on doing so) in the spec?


Adam K

Re: META.yml feature for autotesters?

[Tyler MacDonald]

Adam Kennedy <[EMAIL PROTECTED]> wrote:
> So while we are on the subject of META.yml, I think the dynamic_config 
> approach is horrible, because it defaults to an efficient error case and 
> relies on the author to fix the error, rather than defaulting to the 
> inefficient correct case, and giving the option to make it faster.
> 
> Where would be the best place to get this flag changed (or for a 
> discussion on doing so) in the spec?

The module-build project, maintained by Ken Williams, is currently
hosting the META.yml spec;

http://module-build.sourceforge.net/

I guess META.yml's "reccomends" feature accomplishes most of what
I'm after (if you can't install the module don't worry about it, but if you
can, it'll enhance things), but what I was looking for was a place to dump
stuff like that that doesn't affect the usability of the package, just the
testability. "build_recommends" was suggested on module-build-general, but
it appears to have disappeared from the Module::Build code and isn't on the
META.yml spec.

- Tyler

Re: META.yml feature for autotesters?

[Randy W. Sims]

Adam Kennedy wrote:
To give you some more data points, imagine the automated testing 
additions applied only on Win32. How would you then specify the deps?


#187 on the TODO list for M::B is to implement the dEx[1] (Dependency 
EXpression) language for inserting complicated requirements in META.yml.


1. 

I'm not completely convinced it's a good idea, but it's one possible way 
of handling more complex dependency specifications.


Apart from optional features, which apply across platforms, and across 
all installation/testing modes, I think adding more conditional logic to 
the META.yml should be considered bad.


I'm on the fence. Ideally, META.yml should be a simple declarative 
specification for a module, but with the myriad of runtime environments 
that modules (or libraries of any kind) are used in nowadays it seems 
necessary. The idea with dEx and its companion CPAN::Metadata is that 
they will be the de facto method of reading & writing META.yml, much 
like Test::Harness is for TAP.


But that is all theoretical at this point, and subject to much debate 
before becoming anything. Specifically, PAUSE/CPAN.pm/CPANPLUS would 
have accept it or fall-back on the current method of specification.


Randy.

Re: META.yml feature for autotesters?

[Adam Kennedy]

Randy W. Sims wrote:

Adam Kennedy wrote:
To give you some more data points, imagine the automated testing 
additions applied only on Win32. How would you then specify the deps?


#187 on the TODO list for M::B is to implement the dEx[1] (Dependency 
EXpression) language for inserting complicated requirements in META.yml.


1. 

I'm not completely convinced it's a good idea, but it's one possible way 
of handling more complex dependency specifications.


I'm not convinced either.

Just reading through dEx, it seems to be an example of a fairly common 
response to being limited only to data expressions.


It seems almost any time someone invents a data format, there's a desire 
to have it do more, which leads to the introduction of things that 
aren't data.


You start with branch logic, and fairly quickly you need to generalise 
that to operators, which requires precedence, and macros/subroutines to 
simply and reduce duplication, which means you probably want inclusion 
of external files, and then before you know it you have loop logic and 
you've just invented a new Turing-complete language.


If we are going to invent a language, I'd much rather just stick to Perl.

So you have a Makefile.PL, it runs on a platform and generates a 
META.yml that shows what the dependencies are for that platform.


When you build the package you end up shipping a META.yml that may or 
may not be accurate. So while the META.yml can provide a good indication 
it will never be able to be 100% useful.


The only exception would be if the META.yml had some form of 
"static_config" flag set in it to specifically indicate that the 
META.yml is going to be the same everywhere.


Any packaging or automated testing system is going to have to be 
required to execute the Makefile.PL, in order to localise the META.yml 
for that platform.


That keeps things nice and clear, and means we don't have to create 
additional infrastructure to compensate for a lack of sophistication in 
the META.yml format.


The alternative leads to things like



   If we repair the problem with the dynamic_config flag being backwards, I 
think what we have already can be made to work quite nicely (and sanely) 
without the need for additional infrastructure.


Adam K

Apart from optional features, which apply across platforms, and across 
all installation/testing modes, I think adding more conditional logic 
to the META.yml should be considered bad.


I'm on the fence. Ideally, META.yml should be a simple declarative 
specification for a module, but with the myriad of runtime environments 
that modules (or libraries of any kind) are used in nowadays it seems 
necessary. The idea with dEx and its companion CPAN::Metadata is that 
they will be the de facto method of reading & writing META.yml, much 
like Test::Harness is for TAP.


But that is all theoretical at this point, and subject to much debate 
before becoming anything. Specifically, PAUSE/CPAN.pm/CPANPLUS would 
have accept it or fall-back on the current method of specification.


Randy.

Re: META.yml feature for autotesters?

[Barbie]

On Fri, Feb 24, 2006 at 12:51:13PM -0800, Tyler MacDonald wrote:
> 
>   Sometimes it's beneficial for an automated tester to install
> additional packages (in software I'm releasing, Test::CPANpm and
> sqlite are perfect examples). It would be good to have a standard way
> to tell a smoke tester what these packages are.

I had some thoughts surrounding this for a Bundle::CPAN::YACSmoke
distribution, however, it also means that any distribution that doesn't
include them in the PREREQ correctly will still get them by default, and
may miss them in the FAIL report the author should get. 

Your suggestion seems reasonable, but currently CPAN::YACSmoke doesn't
parse the META.yml file. In fact it doesn't parse any part of the
distribution, just the name and the test results. It leaves all 
distribution parsing to CPANPLUS. It may be possible via a callback, but
it seems more sensible to leave that in the hands of CPANPLUS.

>   If an optional_feature field for packages useful for automated
> testing was standardized, functionality could be added to 
> CPAN::YACSmoke and PITA to take advantage of it, and then there would
> be an easy way for an author to define prerequisites for doing an
> exhaustive smoke test of their distribution.

I think it would be possible for CPANPLUS to be patched to add these to
the list of prerequisites if in AUTOMATED_TESTING mode, so it should be
possible. I think PITA should be able to do something similar.

Barbie

Re: [Module::Build] Re: META.yml feature for autotesters?

[Ken Williams]

On Feb 24, 2006, at 11:42 PM, Tyler MacDonald wrote:


Adam Kennedy <[EMAIL PROTECTED]> wrote:
So while we are on the subject of META.yml, I think the  
dynamic_config
approach is horrible, because it defaults to an efficient error  
case and

relies on the author to fix the error, rather than defaulting to the
inefficient correct case, and giving the option to make it faster.

Where would be the best place to get this flag changed (or for a
discussion on doing so) in the spec?


Weird, how did this discussion get into this thread?

In any case, I agree about dynamic_config.  It's actually documented  
oppositely in two different places, the latter of which is correct:


  http://search.cpan.org/~kwilliams/Module-Build-0.27_07/lib/Module/ 
Build/Authoring.pod
  http://module-build.sourceforge.net/META-spec- 
v1.2.html#dynamic_config


We'll need to change this in the M::B docs (there's nothing to do in  
the code), but my sense is that there's very little being done "in  
the wild" with this flag right now so I'd expect very little breakage  
with this doc change. =)


 -Ken