Return values from a test as a right value.
Hello Perl QA Wizards,
I apologize in advance if this is documented somewhere, because I must
have missed it.
Is it OK to have the return value from a test be something more than
just true when a test passes? Thus the test might be used as a right
value in the test script.
Background:
I am working on a test suite to support distributed functional
testing. I have a test 'like_body(, , )'. It
makes a request of another system and compares the result returned
from the other system with expected.
There may be information in the result that does not affect the
outcome of the test, but would be useful to have for a future test.
A specific example:
The function being tested is a web security application.
Alice is running a web server and the test script. Bob is set up to
attack Alice from random IP addresses. In the test script Alice asks
Bob to visit a url:
$bob->like_body('tget url http://alice/nasty_url resp=404 nat_ip',
qr(^ok), "Nasty_url not found");
If this test passes, we know that Bob got a HTTP 404, but not from
where. It could have been Alice or the security application. We need
another test to look in Alice's logs to see if the query came in, and
she needs to know the source IP address to make sure that we're
looking for the right query. The nat_ip parameter in the request
indicates to Bob to return the source IP address used with the
response.
Now, I know I can just make the request outside of a test and then
process the results within a test, but I lose some diagnostic elegance
that way.
I'm somewhat comfortable with the object syntax required to make
requests of different systems. It has either got to be
$bob->like_body(...) or like_body($bob,...) and I'm leaning toward the
former as being clearer.
However to make the test a right value does not seem in line with the
general expectations of how test scripts are written. $resp =
$bob->like_body(...) does not look like a test to me anymore. Perhaps
I am just being too particular or just haven't seen enough test
scripts, though. So I ask, what do you think?
Thanks for all the hard work in making Perl, TAP, etc. the excellent
foundation from which to build a tool to fulfill my testing need.
Regards,
--
Eric Hacker, CISSP
aptronym (AP-troh-NIM) noun
A name that is especially suited to the profession of its owner
I _can_ leave well enough alone, but my criteria for well enough is
pretty darn high.
Re: Return values from a test as a right value.
On 2/26/07, Michael G Schwern <[EMAIL PROTECTED]> wrote: Also you don't want it to always be true. You want it to reflect whether the test passed or failed. I presume you still want the extra value even if the test failed. Well, at first I didn't think that I'd want it, but after seeing your suggestion, I realized that perhaps in some cases someone would want it when the test fails, and there is no reason not to give it to them. Well, maybe backwards compatibility would be a reason, but I already have many dependencies on recent versions of Perl lurking about. I consider my project 'proof of concept' and that's one desire that dropped off the priority list. I appreciate all the feedback. I know what I'm going to try. I'll let thelist know how it works out.
Re: Managing Test Plans with Vim
Given all of this chat, what would folks think about an optional switch to 'runtests' (the TAPx::Parser equivalent to 'prove'), which would warn users about which test programs are being run without a plan? Brilliant idea. In functional testing, there are situations where one doesn't know how many tests will be run. Say I am testing a web site to make sure that the copyright is included on every page. I don't want a plan, I just want to know if any pages fail (and which ones). How is this handled with pod checking? I think making it possible for the lack of a plan to be flagged when it is important retains Perl's inherent flexibility while supporting correctness when necessary. -- Eric Hacker, CISSP aptronym (AP-troh-NIM) noun A name that is especially suited to the profession of its owner I _can_ leave well enough alone, but my criteria for well enough is pretty darn high.
Re: Managing Test Plans with Vim
On 3/5/07, Ovid <[EMAIL PROTECTED]> wrote: Good point. How about, if an .rc file is used, the output of runtests is something like: shared_cp $ runtests -lrq Using .rc file: /home/ovid/.runtestsrc t/cp_demo_lib.ok t/cp_lib..ok t/db_connection...ok t/dh_password.ok t/dh_server...ok ... Thoughts? I'm a testing newbie, and I apologize if this has been beaten to a pulp in the past. I'd try to use something like YAML for as much of the output as practical, especially the configuration information or warnings. Perl is great at parsing, but why make it difficult. Especially for the person I'll worship who'll write the Eclipse TAP plugin. Hey, we all have our vices, I like Eclipse. :) So I'd do something like. rc_file: /home/ovid/.runtestsrc Peace, -- Eric Hacker, CISSP aptronym (AP-troh-NIM) noun A name that is especially suited to the profession of its owner I _can_ leave well enough alone, but my criteria for well enough is pretty darn high.
Re: Should TAP capture exit codes
On 3/7/07, Andy Armstrong <[EMAIL PROTECTED]> wrote: Ovid's post about TAP::Tests has reminded me: would it be useful to have a TAP statement that conveys the exit code of a test script? At the moment in a hypothetical situation where there's some distance between the harness and the test script - like perhaps they're on different machines - the test script's exit status has to be sent out- of-band. If you add exit to the grammar then the TAP transcript becomes a complete log of the test. Exit code or Status code? In a distributed environment one often wants a bit more than the information that an Exit code provides depending one what's driving the test. I'm not sure what might be best, but it would be worth considering something like HTTP::Status, RFC 2616 and RFC 2518. I'm working on a distributed functional testing system where the individual tests are distributed, not an entire script. Here it is necessary to know the difference between a Command not found vs a failure, a successful create test vs the later successful pass in order to provide useful diagnostics. The RFC Status codes might not be a perfect fit for test status, but like the SIP protocol, they can be reworked to something compatible and generally recognizable. Everyone knows that a 404 is bad. Regards, -- Eric Hacker, CISSP aptronym (AP-troh-NIM) noun A name that is especially suited to the profession of its owner I _can_ leave well enough alone, but my criteria for well enough is pretty darn high.
Re: Should TAP capture exit codes
On 3/7/07, Andy Armstrong <[EMAIL PROTECTED]> wrote: On 7 Mar 2007, at 13:01, Eric Hacker wrote: > Exit code or Status code? Well let's generalise it and discuss the specifics: "any useful information that's available when the test script terminates" Ok > The RFC Status codes might not be a perfect fit for test status, but > like the SIP protocol, they can be reworked to something compatible > and generally recognizable. Everyone knows that a 404 is bad. That's an extra layer of complexity. What benefits does it bring? any useful information ... :) If the testing is distributed, who ordered the test? If the result collector ordered the test, then it would likely want to know the difference between the test failed from a network problem than from a code problem than from a permissions problem. Look at CPAN testers. The lack of an installed module vs all the prereqs are installed and it still won't work on this system is not distinguished. Granted, the module test suite should handle that better itself, so maybe this shouldn't come up, but it does. I think this extra layer of complexity is required for functional testing. It may not be for just testing Perl. I have way more experience testing and breaking functionality than developing. I'm sure if you ever get a look at my code, this will be clear. I'll leave it up to the pros to decide what is really necessary for Perl. I think it was Ovid who recently called it the Test Anything Protocol. If really what is desired, then some additional complexity is required. Meanwhile, I'll adapt HTTP::Status::is_success and is_error to formulate a correct test result from my test status codes, and if more is ever needed, I've already got it. Peace, Hacker
Re: Should TAP capture exit codes
On 3/7/07, Andy Armstrong <[EMAIL PROTECTED]> wrote: On 7 Mar 2007, at 13:48, Eric Hacker wrote: > I think it was Ovid who recently called it the Test Anything Protocol. > If really what is desired, then some additional complexity is > required. Sure - I'm completely in favour of being able to test anything and capture everything that might be useful. But if we want more nuanced exit information (did the test even run? did a connection fail?) we could still capture that without having to massage everything into HTTP style status codes. Not massage, leverage. :) I think this is a situation where you want a code rather than just using text. The conciseness eases automatic interpretation and assures clarity of what was said. If so, then how many codes? Probably not as many as hundreds. Best to group codes so that similar codes are easily identified. Best not to do this using some cool binary or hex code which pushes the calculation to people rather than the big calculator/space heater. So lets say that the codes will be 3 digits, so we'll have enough room to grow for unforseen needs. The first digit can be a grouping by success/failure. They'll be another status group for situations where the test didn't run at all, because of an operation issue. Might want a status group for keepalive info and whatnot on long tests. Sure the Harness might not ever get them, but why have some entirely different way of communication this type of status. I'm not /strongly/ opposed to your proposal - just trying to understand what it might do that we can't do more simply by other means. So then if I'm not too far off base with the above, then to use something different than HTTP::Status type codes would be reinventing. 1xx Info 2xx Success 3xx Redirect, probably not applicable to testing 4xx Failure 5xx Server/System Error Then, if I'm writing a test engine with an http interface for running tests, little or no translation is necessary. :)
Re: Should TAP capture exit codes
How gross would it be just to have a logical channel in TAP that could represent output to STDERR? That plus the exit status of the test script is pretty much all you have at the moment. Would that be so bad? Perhaps, for non-Unix testing, perhaps not. Here is what I have. There are bots that are running on various systems. One asks the bot to do something, and it responds. My bots are being set up to do testing things. Can you ping the server, can you open a TCP connection to port 80, can you get a URL over HTTP, can you run this script, etc. The idea is that without a traditional debugger, there should be a command line user interface that does these things as well as an RPC method for automated tests. When I send an RPC response, it has the human readable reply and the response code. Sure, I could have written a regex to figure out if the response is correct of failed, and I allow that. It is easier to process the codes though, and that suffices for most tests. Sure all of this could have been done in SSH, if everyone had account on and the machines and etc. There are other ways, this is what I'm trying to do and I may fail. Then you all can learn from my 409, 416 or 417. Now, I know you are thinking about exit status on test scripts and I'm thinking individual tests (of which running another test script might be an instance), but in the distributed functional testing space, one really can't rely on independent test scripts to do much. I need to set up a tail with regex on a log file on one bot, hit it from another bot, and know that both were successful real time. Then keep going. Now if all TAP can handle is Perl, or even just code output, then those of us pushing the envelope will eventually migrate to some other test output format. Functional testing is years behind, especially testing distributed applications like network security. I'm trying to leverage what I can to get something useful working. Perl and Test::Harness seem to be good tools to try.
Re: Should TAP capture exit codes
On 3/7/07, Michael G Schwern <[EMAIL PROTECTED]> wrote:
Any time you start writing a system that involves representing states as
numbers and doing bitmasks and math to add extra meaning, step back and
remind yourself that its 2007 and this is not C and you're not writing a
network protocol. You shouldn't have to memorize a table or do math in your
head to figure out the basics of what a message means.
I agree, except we are writing a network protocol. Once the test
results are being shipped around to different systems, a network
protocol is in the works.
And god forbid we had more than 100 failure types!
If you want to say "Temporary Redirect" don't say "307" say "Temporary
Redirect"!
How about "temporär adressieren Sie um" or "provisoire réorientez" or
"de tijdelijke werkkracht richt opnieuw". :)
A decent coding system simplifies the translation between machines and
humans and human languages. It is a means of data sanitizing. It
establishes a scheme of the important details within an area that are
agreed upon by the participants. This provides guidance to new
participants, who can see from the coding system what is relevant.
If an existing coding system that many of the user population is
already familiar with can be leveraged, all the better.
If you want to put lots of information into one value, like
categorization, use a hash! { type => "Redirect", permanent => 0 }
Ok. YAML or XML for serialization? :)
Re: Should TAP capture exit codes
On 3/7/07, Andy Armstrong <[EMAIL PROTECTED]> wrote:
It sounds as if you're doing monitoring rather than testing though.
Although they're related the requirements are quite different.
Poor explaining on my part then. Monitoring has similar needs, but us
usually much more shallow. Consider a web app test (simplified):
# Set tail before running test
$database->ok("tail test add $data");
$client->ok("submit $data to url http://www.example.com/form.cgi";);
Monitoring would check that the http server is up, and check that the
database server is up, but this is functional testing. Does the
application work, end to end.
Now stick some new web app security thingy in from of the web server,
and a database security tool in between the web app server and the
database. We need to run a regression test on the changes, but how.
This is not monitoring. This is scripted testing.
You
need real time capture of as much information as you can get; the
tests we're talking about aren't time dependent and it's quite
acceptable to run any test multiple times until the source of the
problem is found. The ability to capture more nuanced information
about the outcome of a particular test would burden the tests with
producing that information.
That's what Test::More does. Otherwise, everything would just be Ok.
In summary I think you're saying "don't just tell me it failed, tell
me /why/ it failed dammit" and we're saying pretty much the opposite:
"just tell me if it worked or not and leave it to me to find out why".
is_deeply and cmp_ok start to say why.
Now why is a matter of degree, and if a test could tell that the
developer meant && instead of || or that the new HTTP firewall was
dropping all POSTS with "hacker" in the data, then we'd be out of
work. I thought the spirit was to present as much info as practical.
> Functional testing is years behind, especially testing distributed
> applications like network security. I'm trying to leverage what I can
> to get something useful working. Perl and Test::Harness seem to be
> good tools to try.
I'd say not actually. Perl's certainly an excellent tool for your
application but I don't think Perl's testing framework is a good
match for what you're doing. Use the testing framework to prove that
your code works of course but don't try to make it do something it
wasn't intended to do and then blame it when it fights back :)
I really don't care too much, because I am already making what's there
work for me now. But if you are asking what to improve... :)
I took a look at http://www.testobsessed.com the other day. Check out
the functional test tools next generation dream. That makes status
codes look like Lego Duplos.
Re: You cannot predict what TAP will be used for (was Re: Should TAP capture exit codes)
On 3/8/07, Andy Armstrong <[EMAIL PROTECTED]> wrote: I propose that we prefix lines from STDERR with '! ' in the same way that '# ' is used for diagnostics. wstat and exit can just be wstat 256 exit 1 How about this? wstat: 256 exit: 1 YAML, YAML, do! ;)
Re: You cannot predict what TAP will be used for (was Re: Should TAP capture exit codes)
On 3/8/07, Adam Kennedy <[EMAIL PROTECTED]> wrote: Andy Armstrong wrote: >> Otherwise when dealing with TAP streams that don't have a concept of >> an exit code or a seperate error channel, the most common example >> being web testing, we're left high and dry. > > In which case you'd just omit them, no? I didn't imagine they'd be > mandatory. Web testing doesn't have an error channel? HTTP has lot's of channels. They are called Status Codes. :) There are a whole slew of them for errors. Way more than you'd ever need for testing, or so I'm told. Peace, Hacker
TAP.ng Collecting stdout/stderr and diagnostics with Test runs
Current TAP diagnostics appear after the the test results in the standard TAP producers. If one is capturing application output like stdout/stderr, the relevant output would appear before the test. I am often using the test for debugging and have some of my modules use Test::More's diag instead of print/log when running inside test scripts with verbose/debug turned on. That output always occurs before the test completes. Perhaps when a diagnostic message appears after a test, it could be numbered with the test it applies to if test numbering is on. And/or: Perhaps there needs to be more that just clean TAP, # diagnostics, maybe ! stderr for output classes. or Perhaps extreme programming is evil, and a test run should be just a test run. Failure assigns a bug to anyone remotely associated with the project, and they all have a big pow-wow to pass the hot potato of blame around. :) Disclaimer: I often operate outside of any boxes. I am often an edge case. In any given instance, I often don't know if I'm in or out, hump or tail. I offer ideas so that we can move the discussion from "Nobody would ever do that.", through "Why would you do that?" to "Well, I can't see how that is useful, but it's not too much trouble to allow it." That's what got me using Perl in the first place. -- Eric Hacker, CISSP aptronym (AP-troh-NIM) noun A name that is especially suited to the profession of its owner I _can_ leave well enough alone, but my criteria for well enough is pretty darn high.
Re: TAP.ng Collecting stdout/stderr and diagnostics with Test runs
On 3/9/07, Andy Armstrong <[EMAIL PROTECTED]> wrote: On 9 Mar 2007, at 13:37, Eric Hacker wrote: > Current TAP diagnostics appear after the the test results in the > standard TAP producers. D'you mean messages produced with diag() ? They appear in same order relative to the test results in the TAP as they do in the test. Yes, but associating a particular message to the right test result could be difficult for a TAP consumer. Time order does not provide real result association. Currently Test::More will say: not ok 14 - user1 no protocol # Failed test 'user1 no protocol' # in ./t/TCLI.User.t at line 56. So if a GUI TAP consumer wanted to present the associated diagnostics, it could figure it out pretty easily. Suppose though, I had some extra debug output in the application and was using diag to ensure TAP compatibility, as suggested in the docs. # 1:TCLI::User:not_authorized: for [EMAIL PROTECTED] returning 'This is not me ' not ok 17 - user2 not_auth against user1 exact # Failed test 'user2 not_auth against user1 exact' # in ./t/TCLI.User.t at line 149. # 'This is not me ' # doesn't match '(?-xism:Improper protocol)' # 1:TCLI::User:not_authorized: for [EMAIL PROTECTED] returning '' ok 18 - user1 not_auth against user1 with resource So which lines belong with which test result? I know. It probably isn't hard for a human to figure it out. A TAP consumer can only really make a best guess though.
Re: TAP.ng Collecting stdout/stderr and diagnostics with Test runs
On 3/9/07, Andy Armstrong <[EMAIL PROTECTED]> wrote: On 9 Mar 2007, at 16:15, Eric Hacker wrote: > I know. It probably isn't hard for a human to figure it out. A TAP > consumer can only really make a best guess though. It wouldn't be hard to information to the TAP to associate diagnostics with tests but where would that information come from? Current test scripts don't make that association. It wouldn't come from the test script, it would be in the Test::module. Perhaps there would be an 'info|debug|verbose' command in addition to diag. It should still be up to the programmer to follow best practices and use it. Or better yet, UNIVERSAL::print and UNIVERSAL::warn I think I'd better stay down for a while after that. :)
Re: Tests fail under load
On 3/27/07, Kirrily Robert <[EMAIL PROTECTED]> wrote: The problem is that all the developer sandbox websites run on one server that's groaning under the strain. It's in the process of being replaced but we're not there yet. The upshot of this is that on a good day, the web tests take ages to run, and on a bad day they time out. Any suggestions for how to work around this? Add a check of the response time of the sandbox server before starting the functional tests and skip if it is not good enough. Regards -- Eric Hacker, CISSP aptronym (AP-troh-NIM) noun A name that is especially suited to the profession of its owner I _can_ leave well enough alone, but my criteria for well enough is pretty darn high.
Re: Passing parameters to test scripts
On 4/4/07, Philippe Bruhat (BooK) <[EMAIL PROTECTED]> wrote: I've found myself passing parameters to some of my tests scripts. Usually because I want to run a specific subset of tests only. ... Thoughts, anyone? Wonderful idea. I'll take it off of my list of things to bring up someday. I often use my tests for debugging, especially where other modules are involved and I'm not sure where the fault is. Using modules like POE and Net::XMPP seems to cause the Eclipse Debugger to choke. Both distributions support turning on debugging output, which I turn on via a parameter to the test script. Sometimes it is just easier to debug through print than run the debugger anyway, but maybe I'm just missing something. In this case, it wouldn't be required to send it to all tests, but I don't see that as getting in the way. This would make prove way more useful for me. Somewhat related. I've been meaning to go back and figure out where 'prove' came from. I don't recall seeing any reference to it when I first went through the Test::Tutorial and other Test:: docs. It is very useful, but I found it by accident. Thanks -- Eric Hacker, CISSP aptronym (AP-troh-NIM) noun A name that is especially suited to the profession of its owner I _can_ leave well enough alone, but my criteria for well enough is pretty darn high.
