Here are my first thoughts on organizing automated test cases. To make it easy for people to contribute automated tests, I think we need:
• Test discussion - A way to get feedback when you are thinking about writing a test. Although a test might be only a few lines of code, it's affect on Webware is significant. Writing a test implies that all of Webware must now comply with this test and the "proper" result may not be obvious for some cases. So people who are new might need help to know what to test for, especially taking into account other platforms. Simply discussing on the development list is probably sufficient, but the issue should be mentioned in the documentation.
• Easy to run tests - When writing a test, I want to know that I am not breaking things on other platforms that I cannot test on. So there should be a way to get feedback quickly after checking in a new test. Conversely, all the people who are developing with Webware are reluctant to download new versions and run tests if they break things and disrupt their own development. So I think Webware needs to be arranged so you can have two separate versions on your machine. One that you develop with, and a second that you can download often just to run the tests. This implies that we will not use disutils to install Webware into the Python standard places, and that we have a way to control the python path to determine which version of Webware is being used.
• Testing Configuration - With an automated testing system, we need a way to specify what tests run on my computer, e.g. I am on a Mac so don't run COMKit tests, or I have only PostgreSQL so don't run MySQL or MSSQL tests, or I have Apache with mod_kit or I use the cgi adapter.
• Testing daemon - Eventually, we might setup a "continuous testing daemon" that people can run on their machine which will check for new versions in CVS, download them to a separate place, run the tests, and email any failures to a mailing list. I've heard of one for Ant, perhaps we could use that, or write a simple one in Python. That would allow developers to get feedback on new tests from all platforms within minutes of committing a new test.
So to accomplish this we need:
• Documentation describing the need to discuss a test on the webware-devel list, and describing how to add a test, run the tests, etc.
• alltests.py in each kit - We need a standard module that contains all the automated tests for that component. I have always called it alltests.py. It has a suite() function that returns the suite of unittest.TestCases for that component.
• Test Launcher - A script that will set the python path, choose the local machine's configuration, and then run all or just some tests.
How does that sound?
-winston
On Nov 29, 2004, at 6:06 PM, Mark Phillips wrote:
On Nov 29, 2004, at 12:13 PM, Winston Wolff wrote:_________________________________________
I have used WebUnit very successfully for high level tests. (http://mechanicalcat.net/tech/webunit)
I haven't been able to figure out how to unit-test pages since anything that extends Page requires the AppServer to be running. Perhaps part of the dependency can be eliminated with a mock HTTPTransaction, but I'm not sure.
I think it would be good to organize a working group of people willing to work on tests. I know several people including myself have volunteered to work on the release, but not to lead it themselves citing inexperience. However I have not been "called to service" yet. A difficulty with volunteering to do a release is to know if I've made a mistake in the release process. Hence the need for tests. Once we have a group together, we can pick a few priorities in the tests, and settle anything needed to let people get to work individually, such as how our tests will fit into the larger set of test cases.
-winston
Thanks for the reply, Winston. I will add your commentary and take a look at WebUnit when I get the chance.
I am open to discussing how a "release team" might be organized and operate. Feel free to send me a note off-list if you wish.
Mark Phillips
winston wolff - (646) 827-2242 - www.stratolab.com - learning by creating
