Re: UI Regression Testing
David Wheeler sent the following bits through the ether: > we can't really figure out a way to automate the testing of the UI Well, on our current project we're using using OpenFrame[1] and the Template Toolkit to completely seperate application logic from presentation logic. We have applications which deal with the business logic and all that the templates do is present the data. This way we can completely test the application using "make test". In fact, here is some sample output ;-): Files=4, Tests=205, 4 wallclock secs ( 3.80 cusr + 0.19 csys = 3.99 CPU) Of course, this doesn't stop us from creating templates that don't work and thus break the user path (so we have to test that seperately), but seeing as all the templates do is present information that is passed to them from the application, this testing gives us great faith in the application as a whole. HTH, Leon [1] http://openframe.fotango.com/ -- Leon Brocard.http://www.astray.com/ Nanoware...http://www.nanoware.org/ ... Beware of Geeks bearing gifs
Re: UI Regression Testing
> Have you considered talking about Testing at OSC this summer? Mischael > Schwern's talk was a great success last summer. Thanks for the suggestion. I'll think about it, and see what I can do. > Also writing things down as a doc explaining how things work, with some > light examples, to add to our knowledge base would be really cool! Absolutely. If there other Extreme Perl programmers out there, send me a private email. Rob
Re: UI Regression Testing
Rob Nagler wrote: > FWIW, we are very happy with our unit test structure. It has evolved > over many years, and many different languages. I've appended a simple > example, because it is quite different than most of the unit testing > frameworks out there. It uses the XP philosophy of once and only once > as well as test what is likely to break. Rob, it seems like you definitely have a lot of experience with these things. Have you considered talking about Testing at OSC this summer? Mischael Schwern's talk was a great success last summer. Also writing things down as a doc explaining how things work, with some light examples, to add to our knowledge base would be really cool! I'm sure other folks with similar experience will help working on perfecting the document. _ Stas Bekman JAm_pH -- Just Another mod_perl Hacker http://stason.org/ mod_perl Guide http://perl.apache.org/guide mailto:[EMAIL PROTECTED] http://ticketmaster.com http://apacheweek.com http://singlesheaven.com http://perl.apache.org http://perlmonth.com/
Re: UI Regression Testing
Gunther Birznieks writes: > From the description of your scenario, it sounds like you have a long > product life cycle etc. We release weekly. We release to test multiple times a day. We "code freeze" the test system over the weekend. We run all weekly jobs on test during the day on Sat, and then release to production Sat Night. The job testing change was introduced recently. On production, we have a large job which runs on Tues. It also ran on Tues on test. We changed something later in the week one release which broke the job, but it wasn't tested. Now, we get that extra assurance of having the weeklies run just before the release. Having an efficient release mechanism is critical. Also, we get paged when something goes wrong on production. With Perl, we can and do patch individual files midweek in critical emergencies. For example, our ec code broke soon after our site went to a pure subscription model. It was fun, because it broke from too many paying customers. $-) Needless to say, we patched the system asap! > I think your testing, especially regression testing and the amount of > effort you put into it makes a lot of sense because your software is a > long-term investment possibly even a product. Yes, that's an important point. We run the accounting for over 8,000 investment clubs. We have a responsibility to make sure the software is reliable. We released our Petshop demo without any tests. :) > To each his own I guess. Agreed. Rob
Re: UI Regression Testing
Perrin Harkins writes:
> But what about the actual data? In order to test my $product->name()
> method, I need to know what the product name is in the database. That's
> the hard part: writing the big test data script to run every time you
> want to run a test (and probably losing whatever data you had in that
> database at the time).
There are several issues here. I have answers for some but not all.
We don't do complex unit tests. We save those for the acceptance test
suite. The unit tests do simple things. I've attached a basic unit
test for our DBI abstraction layer. It runs on Oracle and Postgres.
Acceptance tests take over an hour to run. We have a program which
sets up some basic users and clubs. This is run once. It could be
run before each test suite run, but we don't. We have tests which
test creating users, entering subscription payments, twiddling files
and email. By far the biggest piece is testing our accounting. As I
said, we used student labor to write the tests. They aren't perfect,
but they catch lots of errors that we miss.
Have a look at:
http://petshop.bivio.biz/src?s=Bivio::PetShop::Util
This program populates the database for our petshop demo. It builds
the entire schema, too. The test suite for the petshop will assume
this data.
The amount of data need not be large. This isn't the point of
acceptance testing imo. What you want is enough data to exercise
features such as paging, form submits, etc. Our production database
is multi-GB.
We do have a particularly nasty problem of our quote database. We
update all of our quote databases nightly using the same software
which talks to our quote provider. This tests the software in
real-time on all systems. We run our acceptance test suite in the
morning after all the nightly stuff is done. It takes hours to
re-import our quote database.
You need a test system distinct from your production and development
systems. It should be as close in configuration to the production
system as possible. It can be very cheap. Our test system consists of
a refurb Compaq Presario and a Dell 1300 with 4 disks. We use
hardware RAID on production and software RAID on test. Differences
like these don't matter.
The database source needs to be configurable. Disk is cheap.
You can have multiple users (schemata) using the same database host.
Our database abstraction allows us to specify the target database
vendor, instance, user, and password. Our command line utility
software allows us to switch instances easily, and the config module
does, too.
I often test against my development database at the same time as I
compare the same results against the test database. I can do this,
e.g.
b-petshop -db test create_db
All utilities have a '-db' argument. Alternatively, I can specify the
user in long hand for the Connection test below:
perl -w Connection.t --Bivio::Ext::DBI.database=test
All config parameters can be specified this way, or in a dynamically
selectable file.
> This has been by far the biggest obstacle for me in testing, and from
> Gunther's post it sounds like I'm not alone. If you have any ideas
> about how to make this less painful, I'd be eager to hear them.
It isn't easy. We don't write a unit test per class. Indeed we're
far from this. OTOH, we reuse heavily. For example, we don't need
to test our product list:
http://petshop.bivio.biz/src?s=Bivio::PetShop::Model::ProductList
It contains no "code", only declarations. All the SQL is generated
by the object-relational mapping layer which handles paging,
column sorting, and so on. The view is as simple:
http://petshop.bivio.biz/src?s=View.products
Neither of these modules is likely to break, so we feel confident
about not writing unit tests for them.
Rob
--
#!/usr/bin/perl -w
use strict;
use Bivio::Test;
use Bivio::SQL::Connection;
my($_TABLE) = 't_connection_t';
Bivio::Test->unit([
Bivio::SQL::Connection->create => [
execute => [
# Drop the table first, we don't care about the result
["drop table $_TABLE"] => undef,
],
commit => undef,
{
method => 'execute',
result_ok => \&_expect_statement,
} => [
# We expect to get a statement back.
[<<"EOF"] => [],
create table $_TABLE (
f1 numeric(8),
f2 numeric(8),
unique(f1, f2)
)
EOF
["insert into $_TABLE (f1, f2) values (1, 1)"] => [],
],
commit => undef,
execute => [
["insert into $_TABLE (f1, f2) values (1, 1)"]
=> Bivio::DieCode->DB_CONSTRAINT,
],
{
method => 'execute',
result_ok => \&_expect_one_row,
} => [
["update $_TABLE set f2 = 13 where f2 = 1"] => [],
],
execute_one_row => [
[
Re: UI Regression Testing
Hi Craig, > Have you ever heard of the hw verification tool Specman Elite by Verisity > (www.verisity.com)? No, but it looks interesting. It would be good to have something like this for unit tests. I haven't had very good experience with automated acceptance testing, however. The software should be robust against garbage in, but the main problem we have is making sure the numbers add up, and that we generate the correct tax forms! It's pretty tricky stuff. FWIW, we are very happy with our unit test structure. It has evolved over many years, and many different languages. I've appended a simple example, because it is quite different than most of the unit testing frameworks out there. It uses the XP philosophy of once and only once as well as test what is likely to break. Rob -- #!perl -w # $Id: Integer.t,v 1.7 2001/11/24 04:30:19 nagler Exp $ # use strict; use Bivio::Test; use Bivio::Type::Integer; use Bivio::TypeError; Bivio::Test->unit([ 'Bivio::Type::Integer' => [ get_min => -9, get_max => 9, get_precision => 9, get_width => 10, get_decimals => 0, can_be_zero => 1, can_be_positive => 1, can_be_negative => 1, from_literal => [ ['9'] => [9], ['+9'] => [9], ['-9'] => [-9], ['x'] => [undef, Bivio::TypeError->INTEGER], [undef] => [undef], [''] => [undef], [' '] => [undef], ['-99'] => [undef, Bivio::TypeError->NUMBER_RANGE], ['-09'] => [-9], ['+09'] => [9], ['-9'] => [-9], ['+9'] => [9], ['+10'] => [undef, Bivio::TypeError->NUMBER_RANGE], ['-10'] => [undef, Bivio::TypeError->NUMBER_RANGE], ], ], Bivio::Type::Integer->new(1,10) => [ get_min => 1, get_max => 10, get_precision => 2, get_width => 2, get_decimals => 0, can_be_zero => 0, can_be_positive => 1, can_be_negative => 0, from_literal => [ ['1'] => [1], ['+1'] => [1], ['0'] => [undef, Bivio::TypeError->NUMBER_RANGE], ['11'] => [undef, Bivio::TypeError->NUMBER_RANGE], ['-1'] => [undef, Bivio::TypeError->NUMBER_RANGE], [undef] => [undef], ['-09'] => [undef, Bivio::TypeError->NUMBER_RANGE], ['+09'] => [9], ], ], ]);
Re: UI Regression Testing
From the description of your scenario, it sounds like you have a long product life cycle etc. I think your testing, especially regression testing and the amount of effort you put into it makes a lot of sense because your software is a long-term investment possibly even a product. I think you can "worry" about an API not having a test for it, and I do a bit, but I don't lose sleep over it. The reality is many APIs succeed OK without tests and maybe a few grumbles hear and there if and when things break. I've rarely seen a documented API break in such a significant way that either I programmed or used from someone else that I could not easily rectify it or go back to a previous version until it was resolved. ie major API changes usually have a way of being known anyway very quickly. Would API tests for every API in existance solve that? Could be, probably yes. But is it worth all the extra coding time for those tests? I don't think so. Not for all APIs. To each his own I guess. I agree with tests for some things, just not for all things including not all APIs. At 08:59 AM 1/26/2002, Rob Nagler wrote: >Gunther Birznieks writes: > > the database to perform a test suite, this can get time consuming and > > entails a lot of infrastructural overhead. > >We haven't found this to be the case. All our database operations are >programmed. We install the database software with an RPM, run a >program to build the database, and program all schema upgrades. We've >had 194 schema upgrades in about two years. > > > unit testing being done on the basis of writing a test class for every > > class you write. Ugh! That means that any time you refactor you throw away > > the 2x the coding you did. > >By definition, refactoring doesn't change observable behavior. You >validate refactorings with unit tests. See http://www.refactoring.com > > > To some degree, there should be intelligent rules of thumb as to which > > interfaces tests should be written to because the extreme of writing tests > > for everything is quite bad. > >Again, we haven't seen this. Every time I don't have unit tests, I >get nervous. How do I know if I broke something with my change? > > > Finally, unit tests do not guarantee an understanding of the specs because > > the business people generally do not read test code. So all the time spent > > writing the test AND then writing the program AND ONLY THEN showing it to > > the users, then you discover it wasn't what the user actually wanted. > So 2x > > the coding time has been invalidated when if the user was shown a > prototype > > BEFORE the testing coding commenced, then the user could have confirmed or > > denied the basic logic. > >Unit tests aren't about specs. They are about APIs. Acceptance tests >need to be written by the user or written so the user can understand >them. You need both kinds of testing. >See http://www.xprogramming.com/xpmag/Reliability.htm > >Rob __ Gunther Birznieks ([EMAIL PROTECTED]) eXtropia - The Open Web Technology Company http://www.eXtropia.com/
Re: UI Regression Testing
On Sat, 26 Jan 2002 00:23:40 -0500
"Perrin Harkins" <[EMAIL PROTECTED]> wrote:
> But what about the actual data? In order to test my $product->name()
> method, I need to know what the product name is in the database. That's
> the hard part: writing the big test data script to run every time you
> want to run a test (and probably losing whatever data you had in that
> database at the time).
>
> This has been by far the biggest obstacle for me in testing, and from
> Gunther's post it sounds like I'm not alone. If you have any ideas
> about how to make this less painful, I'd be eager to hear them.
You're not alone ;) here is my solution.
* All datasource are maintained with separate config file
* Generate config file for testing
* Create database and tables for testing (called test_foo)
* Insert dummy data into test_foo
* Test 'em
* Drop dummy data
Then my test script has both client side testing and server side
testing, like this.
use Test::More 'no_plan';
BEGIN { do 'db_setup.pl'; }
END { do 'db_teardown.pl'; }
# server-side
my $product = Product->create({ name => 'foo' });
# client-side
my $ua = LWP::UserAgent->new;
my $res = $ua->request(GET "/foo/bar");
like $res->content, qr/foo/;
my $form = HTML::Form->parse($res->content);
my $req2 = $form->click;
my $res2 = $ua->request($req);
like $res2->content, qr/blah/;
# server-side
my @p = Product->retrieve_all;
is @p, 2;
--
Tatsuhiko Miyagawa <[EMAIL PROTECTED]>
Re: UI Regression Testing
David Wheeler wrote: > Hi All, > > A big debate is raging on the Bricolage development list WRT CVS > configuration and application testing. > > >http://www.geocrawler.com/mail/thread.php3?subject=%5BBricolage-Devel%5D+More+on+Releases&list=15308 > > It leads me to a question about testing. Bricolage is a monster > application, and its UI is built entirely in HTML::Mason running on > Apache. Now, while we can and will do a lot more to improve the testing > of our Perl modules, we can't really figure out a way to automate the > testing of the UI. I'm aware of the the performance testing utilities > mentioned in the mod_perl guide -- > > http://perl.apache.org/guide/performance.html > > -- but they don't seem to be suited to testing applications. > > Is anyone familiar with how to go about setting up a test suite for a > web UI -- without spending an arm and a leg? (Remember, Bricolage is an > OSS effort!). You probably also need some good back-end engine for testing since you mostly likely need to test against a live Apache/mod_perl. If that's the case you should try to use the new Apache::Test framework that will be released with mod_perl 2.0. You can get it from here: http://cvs.apache.org/snapshots/modperl-2.0/ (or use cvs) Some docs are here: http://cvs.apache.org/viewcvs.cgi/modperl-docs/src/docs/2.0/devel/testing/testing.pod _ Stas Bekman JAm_pH -- Just Another mod_perl Hacker http://stason.org/ mod_perl Guide http://perl.apache.org/guide mailto:[EMAIL PROTECTED] http://ticketmaster.com http://apacheweek.com http://singlesheaven.com http://perl.apache.org http://perlmonth.com/
Re: UI Regression Testing
> Gunther Birznieks writes: > > the database to perform a test suite, this can get time consuming and > > entails a lot of infrastructural overhead. > > We haven't found this to be the case. All our database operations are > programmed. We install the database software with an RPM, run a > program to build the database, and program all schema upgrades. We've > had 194 schema upgrades in about two years. But what about the actual data? In order to test my $product->name() method, I need to know what the product name is in the database. That's the hard part: writing the big test data script to run every time you want to run a test (and probably losing whatever data you had in that database at the time). This has been by far the biggest obstacle for me in testing, and from Gunther's post it sounds like I'm not alone. If you have any ideas about how to make this less painful, I'd be eager to hear them. - Perrin
Re: UI Regression Testing
On Sat, 26 Jan 2002, Gunther Birznieks wrote: > I agree that testing is great, but I think it is quite hard in practice. > Also, I don't think programmers are good to be the main people to write > their own tests. It is "OK" for programmers to write their own tests but > frequently it is the user or a non-technical person who is best at doing > the unexpected things that are really were the bug lies. My experience is that the best testers come from technical support, although this is not to suggest that all technical support individuals are good at this; even among this group, it's rare. Users or other non-technical people may find a few more bugs, but frequently, the non-technical people don't have the ability to correctly convey how to reproduce the problems, or even what the problem was. "I clicked on the thingy, and it didn't work." This being said, users and tech support can't create unit tests; they're not in a position to. > Finally, unit tests do not guarantee an understanding of the specs because > the business people generally do not read test code. So all the time spent > writing the test AND then writing the program AND ONLY THEN showing it to > the users, then you discover it wasn't what the user actually wanted. So 2x > the coding time has been invalidated when if the user was shown a prototype > BEFORE the testing coding commenced, then the user could have confirmed or > denied the basic logic. For your understanding of the spec, you use functional tests. If your functional test suite uses test rules which the users can understand, you can get the users to double-check them. For example, at work, we use a suite which uses a rendered web page as its test output, and the input can be sent to a web page to populate a form; this can be read by most people who can use the application. Unit software is a means of satisfying a spec, but it doesn't satisfy the spec itself - if it did, you'd be talking about the entire package, and therefore refering to functional testing. (At least, this is the way I distinguish between them.) Admittedly, we are a bit lacking in our rules, last I checked. Ed
Re: UI Regression Testing
Gunther Birznieks writes: > the database to perform a test suite, this can get time consuming and > entails a lot of infrastructural overhead. We haven't found this to be the case. All our database operations are programmed. We install the database software with an RPM, run a program to build the database, and program all schema upgrades. We've had 194 schema upgrades in about two years. > unit testing being done on the basis of writing a test class for every > class you write. Ugh! That means that any time you refactor you throw away > the 2x the coding you did. By definition, refactoring doesn't change observable behavior. You validate refactorings with unit tests. See http://www.refactoring.com > To some degree, there should be intelligent rules of thumb as to which > interfaces tests should be written to because the extreme of writing tests > for everything is quite bad. Again, we haven't seen this. Every time I don't have unit tests, I get nervous. How do I know if I broke something with my change? > Finally, unit tests do not guarantee an understanding of the specs because > the business people generally do not read test code. So all the time spent > writing the test AND then writing the program AND ONLY THEN showing it to > the users, then you discover it wasn't what the user actually wanted. So 2x > the coding time has been invalidated when if the user was shown a prototype > BEFORE the testing coding commenced, then the user could have confirmed or > denied the basic logic. Unit tests aren't about specs. They are about APIs. Acceptance tests need to be written by the user or written so the user can understand them. You need both kinds of testing. See http://www.xprogramming.com/xpmag/Reliability.htm Rob
Re: UI Regression Testing
I suppose it depends on what you want out of testing. Frequently, unit testing is OK in simple applications. But in an application whose job it is to communicate with a mainframe or back-end databases, frequently the tests you might perform are based on some previous persistent state of the database. It takes a lot of effort in other words. For example, 2-3 programmers could easily get away (through normal testing) with sharing a database. But generally if you require each programmer to have their own database and their own system completely where they are constantly wiping the state of the database to perform a test suite, this can get time consuming and entails a lot of infrastructural overhead. I've seen some articles that demonstrate doing something with test XML backends to emulate database retrieval results, but these seem quite hard to set up also. I agree that testing is great, but I think it is quite hard in practice. Also, I don't think programmers are good to be the main people to write their own tests. It is "OK" for programmers to write their own tests but frequently it is the user or a non-technical person who is best at doing the unexpected things that are really were the bug lies. The other annoying thing about programmers writing tests is deciding where to stop. I believe the HTTP level for tests is really good. But I see much unit testing being done on the basis of writing a test class for every class you write. Ugh! That means that any time you refactor you throw away the 2x the coding you did. To some degree, there should be intelligent rules of thumb as to which interfaces tests should be written to because the extreme of writing tests for everything is quite bad. Finally, unit tests do not guarantee an understanding of the specs because the business people generally do not read test code. So all the time spent writing the test AND then writing the program AND ONLY THEN showing it to the users, then you discover it wasn't what the user actually wanted. So 2x the coding time has been invalidated when if the user was shown a prototype BEFORE the testing coding commenced, then the user could have confirmed or denied the basic logic. The same frequently and especially is true for UIs. Later, Gunther
Re: UI Regression Testing
> Have you tried webchat? You can find webchatpp on CPAN.
Just had a look. It appears to be a rehash of chat (expect) for the
web. Great stuff, which is really needed and demonstrates the power
of Perl for test scripting.
But...
This is a bit hard to explain. There are two types of XP testing:
unit and acceptance. Unit testing is pretty clear in Perl circles
(ok, I have a thing or two to say about it, but not now :-).
Acceptance testing (aka functional testing) is traditionally handled
by a third party testing organization. The test group writes scripts.
If they are testing GUIs, they click in scripts via a session
recorder. They don't "program" anything. There's almost no reuse,
and very little abstraction.
XP flips testing on its head. It says that the programmers are
responsible for testing, not some 3rd party org. The problem I have
found is that instead of programming the test suite, XPers script it,
using the same technology that a testing organization would use. With
the advent of the web, this is a real shame.
HTTP and HTML are middleware. You have full programmatic control to
test your application. You can't control the web browser, so you
still need to do some ad hoc "how does it look" testing, but this
isn't the hard part.
The acceptance test suite is testing the system from the user's point
of view. In XP, the user is the customer, and the customer writes
tests. In my opinion, this means the customer writes tests in a pair
with a programmer. The programmer's job is to create a language which
the user understands.
Here's an example from our test suite:
Accounting->setup_investment('AAPL');
The user knows what an investment is. She also knows that AAPL is a
stock ticker. This statement sets up the environment (using LWP to
the app) to execute tests such as entering dividends, buys, sells,
etc.
The test infrastructure must support the ability to create new
language elements with the ability to build elements using the other
elements. This requires modularization, and today this means classes
and instances. There's also a need for state management, just like
the request object in your web application.
Part of the packaging process we're going through is making it even
easier to create domain specific languages. You actually want to
create lots of dialects, e.g. in our case this means investments, cash
accounts, member accounts, and message boards. These dialects use
building blocks such as logging in, creating a club, and so on. At
the bottom you use LWP or webchat. However, the user doesn't care if
the interface is HTTP or Windows. You're job as a test suite
programmer is meeting her domain knowledge, and abstracting away
details like webchat's CLICK and EXPECT OK.
In the end, your test suite is a domain knowledge repository. It
contains hundreds of concise scenarios comprised of statements, or
facts, in knowledge base parlance. The execution of the test suite
asserts all the facts are true about your application. The more
concise the test language. The more easily the user-tester can verify
that she has encoded her expertise correctly.
Rob
Re: UI Regression Testing
On Fri, 2002-01-25 at 10:12, Perrin Harkins wrote: > Have you tried webchat? You can find webchatpp on CPAN. Looks interesting, although the documentation is rather sparse. Anyone know of more examples than come with it? Thanks, David -- David Wheeler AIM: dwTheory [EMAIL PROTECTED] ICQ: 15726394 Yahoo!: dew7e Jabber: [EMAIL PROTECTED]
Re: UI Regression Testing
> There are many web testers out there. To put it bluntly, they don't > let you write maintainable test suites. The key to maintainability is > being able to define your own domain specific language. Have you tried webchat? You can find webchatpp on CPAN.
Re: UI Regression Testing
> Is anyone familiar with how to go about setting up a test suite for a > web UI -- without spending an arm and a leg? (Remember, Bricolage is an > OSS effort!). Yes, it's very easy. We did this using student labor, because it is an excellent project for students and it's probably cheaper. It's very important. We run our test suite nightly. I'm an extreme programming (XP) advocate. Testing is one of the most important practices in XP. I'm working on packaging what we did so it is fit for public consumption. Expect something in a month or so. It'll come with a rudimentary test suite for our demo petshop app. There are many web testers out there. To put it bluntly, they don't let you write maintainable test suites. The key to maintainability is being able to define your own domain specific language. Just like writing maintainable code, you have to encapsulate commonality and behavior. The scripts should be short and only contain the details pertinent to the particular test. Perl is ideal for this, because you can easily create domain specific languages. Rob
