Re: Testing scripts with expected STDOUT and STDERR in external files

[David Golden]

On Sat, Feb 21, 2009 at 5:24 AM, Gabor Szabo  wrote:
>>> I wonder if there are modules out there that already do this?
>>> I could not find any that would fit my needs.

Test::Cmd?  It's not built on Test::Builder but could be
adapted/wrapped.  It uses shell redirection to capture output, which
should be OK as long as you don't need interactivity with the programs
being run.

I have something I wrote a long time ago for repetitive testing of a
command line program.  I've considered putting it on CPAN but never
got around to writing much documentation for it.  It works more or
less like this:

  my $program = Test::CLI->new($path_to_program);
  $program->stdin( $input );
  $program->runs_ok( @args );
  $program->stdout_like( qr/$expected/ );
  $stderr -> $program->stderr;  # for custom examination

Right now, it assumes the "program" is invoked via perl, but that
could be stripped out.  It also supports default arguments that are
added to every run. So you might use it like this:

  my $interp = Test::CLI->new($path_to_interpreter, @default_params);
  for my $c (@cases) {
$interp->stdin( slurp $c->{in_file} );
$interp->runs_ok( $c->{utility} );
$interp->stdout_like( $c->{stdout} );
$interp->stderr_is( $c->{stderr} );
  }

If you want, I'll stick it in some boilerplate and publish it to CPAN
and github and then you're welcome to document it or improve it.  Just
let me know if that would be helpful.

Or, if it would need a lot of adaptation for what you're looking to
do, you're welcome to browse it and take whatever is useful.  It's
t/CLI.pm in the Pod::WikiDoc repo:
http://github.com/dagolden/pod-wikidoc/

> Capture::Tiny is cute.

More than cute -- it works and works pretty universally for all
situations.  I have a few more edge cases to code (e.g. if STDOUT or
STDERR are closed before it starts capturing), but my goal is to have
it replace the 20 or so other capturing modules out there.  Then
things like Test::Output can be re-written around it and will become
much more powerful.

-- David

Re: Testing scripts with expected STDOUT and STDERR in external files

[Gabor Szabo]

On Fri, Feb 20, 2009 at 11:30 PM, David E. Wheeler  wrote:
> On Feb 20, 2009, at 1:23 PM, Gabor Szabo wrote:
>
>> I wonder if there are modules out there that already do this?
>> I could not find any that would fit my needs.
>
> Test::Output?
>
>  http://search.cpan.org/perldoc?Test::Output
>
> If it doesn't capture output from other programs, have a look at
> Capture::Tiny.

I looked at both Test::Output and Test::Trap but for both of them I
need to do lots of other things.
Capture::Tiny is cute.

Maybe I need to put together something using some of those though
I think I'll start just by putting what I have now in a module.

What should I call though ?
Test::Exec ?
Test::Execute ?
Test::External ?

Maybe I should extend Text::Cmd or Text::Commands.

Test::Commands ?

Gabor

Re: Testing scripts with expected STDOUT and STDERR in external files

[Eirik Berg Hanssen]

Gabor Szabo  writes:

> For this it is enough to run
>
> system "path/to/utitlity < in.txt > out.txt 2> err.txt"
>
> and then compare them to the expected.out and expected.err

  Not what you're asking for, but when I see testing stdout and stderr
from child processes, I cannot resist ...


use Test::More tests => 2;
use Test::Trap qw( :output(systemsafe) );

trap { system "path/to/utitlity < in.txt" };
$trap->stdout_like( qr/for real!/, 'Got a matching STDOUT' );
$trap->stderr_is( '', 'Got no STDERR' );


  :-)


  (Now, if I could just whip up some support for systemsafe input as
well ...)


Eirik
-- 
An honest newspaper publishes corrections on the front page.
There are no honest newspapers.
 - Dr. Newton, Dean of Journalism, NAU.

Re: Testing scripts with expected STDOUT and STDERR in external files

[David E. Wheeler]

On Feb 20, 2009, at 1:23 PM, Gabor Szabo wrote:


I wonder if there are modules out there that already do this?
I could not find any that would fit my needs.


Test::Output?

  http://search.cpan.org/perldoc?Test::Output

If it doesn't capture output from other programs, have a look at  
Capture::Tiny.


Best,

David

Testing scripts with expected STDOUT and STDERR in external files

[Gabor Szabo]

Lately I need to test many small utilities written in various languages.

For the most simple cases I just need to run the utilities with a set of
input
and check if the output is the same as expected.

For this it is enough to run

system "path/to/utitlity < in.txt > out.txt 2> err.txt"

and then compare them to the expected.out and expected.err
To facilitate this I generate the input and the expected output files
and put them next to the utility:


path/to/utility
path/to/utility.out
path/to/utility.err
path/to/utility.in

In case I need to test the same utility with various inputs I keep

path/to/utility
path/to/utility.2.out
path/to/utility.2.err
path/to/utility.2.in 


Sometimes these utilities need to be executed with
some other tool that even needs parameters. In such cases I use:

system "pat/to/interpreter param param  path/to/utitlity < in.txt > out.txt
2> err.txt"

In most cases all the utilities in one package need to be executed the same
way.


I wonder if there are modules out there that already do this?
I could not find any that would fit my needs.


Gabor