Hi,
Back in the days, when we were approaching a new release of Qt, we would use a
black team <http://www.t3y.com/tangledwebs/07/tw0706.html> which would be
responsible for driving the testing of a new package. Their main focus would be
to test every new package generated, both source and binary packages, and try
to break them with the most used configuration (every configuration is not
possible due to the vast number of configurable options, and system
configurations).
When doing this they would have a common list of issues to look out for, to
make sure that the reporting had some structure (same report fields for each
reporter/platform/package) and that they wouldn't skip something for one
package, etc. The list contained things like:
* General:
* Report date
* Testers name
* Package name
* Package type
* Build date
* Makespec used
* Source packages:
* Configuration line for main testing
* Configure asks about license when run with no license options
* Compiling the package
* Compiles with minimal options (e.g. -opensource -confirm-license)
* Compiles as a static build, where supported
* Compiles in namespace, where supported
* Compiles with shadow build
* Cross-compiles, where supported
* Installing the package works
* "make install" to system directory as root works
* "make install" to local prefix as regular user works
* "make install" distributes files correctly on Mac
* "make clean" and "make distclean" works
* Text files have the correct EOL for the type of package?
* Files/directories in the package have sane file permissions and
timestamps?
* Tags (%VERSION%, %SHORTVERSION%) have been replaced properly
* README has valid information about how to compile the package on the
platform tested
* Binary packages:
* Fresh install
* Installer is correctly signed, e.g. Windows shows correct
vendor/certificate data and no warnings about untrusted vendor.
* Installer displays appropriate graphics, strings, version numbers
* Installer offers the correct license(s)
* Installer offers sane default install directory (including version
number)
* Installer correctly installs to default directory
* installer correctly installs to non-default drive/directory
* Installer sanely reports progress and completion
* Installer installs only selected components
* Component selection works sanely
* Shortcuts from last page of installer work (e.g. shortcuts to
README, demos, etc)
* Installer correctly creates desktop shortcuts and they all work
* Installer correctly creates Start Menu shortcuts and they all work
* Environment settings are correct in Qt MSVC Command Prompt
* Package shows up in Control Panel/Package Manager with correct
description/version
* Any patching of files has been done correctly, e.g. patching of
library paths into .exe files and the paths hard-coded in the QtCore library.
* Upgrade install, if supported
- As above
* Parallel install, if supported
- As above
- Can't install over the top of an existing package without being warned
- Shortcuts point to the right package
- Previously installed packages still work
* Uninstall
- Removes all installed files,
- Removes all empty dirs,
- Removes registry keys,
- Reverses any other changes made by installer
- What to do with data files created by installed files, e.g. saved settings
and files created by demo apps?
- What to do with data shared between more than one package instance?
* Aborting installation, if supported
- Cancel button is available
- Installer cleans up, as for full uninstallation
* Failed installation
- insufficient disk space (can be faked by installing onto a small USB key/SD
card)
- network problems while using online installer
* Both package types:
* Verify the license
* Assistant works
* Designer works
* "Showcase" demo/example apps launch without crashing
* "Showcase" demo/example apps function acceptably.
* Demo/example apps can be rebuilt
* External Qt apps can be built against the package (e.g. Qt Creator)
* Did "DLL Swapping" on a Qt app (e.g. KDE, Google Earth, Qt Creator)
work?
* Click around various examples/demos for a while (GUI stress-testing)
* Audio/Video w/phonon
* Audio/Video w/QtMultimedia
* Raster paint engine works
* Image formats work
* Graphicsview
* OpenGL
* OpenVG
* Printing
* QML
* QtNetworking
* QtScript
* QtSql
* QtSvg
* WebKit works?
* Logging into a popular site (facebook, hotmail, gmail etc) using
the demo-browser
* Uploading an image to imgur.com using the demo-browser
* Watch a cat video on youtube using the demo-browser
* Xml
* qmlviewer (outside of qtdemo)
Thanks Jason for a thorough list from the Qt 4 days!
This testing should be distributed on as many people as possible to lower the
load on individuals, and to allow us to get as much feedback as quickly as
possible. This in turn allows for shorter test periods, and that way we can get
the Qt releases out according to our schedule(s).
The question is now, how do we organize this? We need a form, and a place to
collect them and allowing us to easily browse the reports and see the status
for any given release.
I don't think using SurveyMonkey would cut it with this scope. :)
How about maybe installing the "JIRA Issue
Collector<https://plugins.atlassian.com/plugins/com.atlassian.jira.collector.plugin.jira-issue-collector-plugin>"
plugin (You are not required to have a Jira account to use the issue
collector, as it can use a fall-back account for general reporting!), allowing
us to report against a "Fix Version", and perhaps use "Labels" to distinguish
between build dates? Maybe these reports should have their own types (currently
we only have Bug/Task), like "Test Report"? Then on a no-go report, Jira "Bug"
issues could be created and linked to the "Test Report"?
Opinions?
--
.marius
_______________________________________________
Development mailing list
Development@qt-project.org
http://lists.qt-project.org/mailman/listinfo/development
