Any opinions on this?
I fear the mail drowned in other mails on the list.
--
.marius
On May 1, 2012, at 3:33 PM, marius.storm-ol...@nokia.com
marius.storm-ol...@nokia.com wrote:
Hi,
Back in the days, when we were approaching a new release of Qt, we would use
a black team which would be responsible for driving the testing of a new
package. Their main focus would be to test every new package generated, both
source and binary packages, and try to break them with the most used
configuration (every configuration is not possible due to the vast number of
configurable options, and system configurations).
When doing this they would have a common list of issues to look out for, to
make sure that the reporting had some structure (same report fields for each
reporter/platform/package) and that they wouldn't skip something for one
package, etc. The list contained things like:
General:
Report date
Testers name
Package name
Package type
Build date
Makespec used
Source packages:
Configuration line for main testing
Configure asks about license when run with no license options
Compiling the package
Compiles with minimal options (e.g. -opensource -confirm-license)
Compiles as a static build, where supported
Compiles in namespace, where supported
Compiles with shadow build
Cross-compiles, where supported
Installing the package works
make install to system directory as root works
make install to local prefix as regular user works
make install distributes files correctly on Mac
make clean and make distclean works
Text files have the correct EOL for the type of package?
Files/directories in the package have sane file permissions and timestamps?
Tags (%VERSION%, %SHORTVERSION%) have been replaced properly
README has valid information about how to compile the package on the platform
tested
Binary packages:
Fresh install
Installer is correctly signed, e.g. Windows shows correct vendor/certificate
data and no warnings about untrusted vendor.
Installer displays appropriate graphics, strings, version numbers
Installer offers the correct license(s)
Installer offers sane default install directory (including version number)
Installer correctly installs to default directory
installer correctly installs to non-default drive/directory
Installer sanely reports progress and completion
Installer installs only selected components
Component selection works sanely
Shortcuts from last page of installer work (e.g. shortcuts to README, demos,
etc)
Installer correctly creates desktop shortcuts and they all work
Installer correctly creates Start Menu shortcuts and they all work
Environment settings are correct in Qt MSVC Command Prompt
Package shows up in Control Panel/Package Manager with correct
description/version
Any patching of files has been done correctly, e.g. patching of library paths
into .exe files and the paths hard-coded in the QtCore library.
Upgrade install, if supported
- As above
Parallel install, if supported
- As above
- Can't install over the top of an existing package without being warned
- Shortcuts point to the right package
- Previously installed packages still work
Uninstall
- Removes all installed files,
- Removes all empty dirs,
- Removes registry keys,
- Reverses any other changes made by installer
- What to do with data files created by installed files, e.g. saved
settings and files created by demo apps?
- What to do with data shared between more than one package instance?
Aborting installation, if supported
- Cancel button is available
- Installer cleans up, as for full uninstallation
Failed installation
- insufficient disk space (can be faked by installing onto a small USB
key/SD card)
- network problems while using online installer
Both package types:
Verify the license
Assistant works
Designer works
Showcase demo/example apps launch without crashing
Showcase demo/example apps function acceptably.
Demo/example apps can be rebuilt
External Qt apps can be built against the package (e.g. Qt Creator)
Did DLL Swapping on a Qt app (e.g. KDE, Google Earth, Qt Creator) work?
Click around various examples/demos for a while (GUI stress-testing)
Audio/Video w/phonon
Audio/Video w/QtMultimedia
Raster paint engine works
Image formats work
Graphicsview
OpenGL
OpenVG
Printing
QML
QtNetworking
QtScript
QtSql
QtSvg
WebKit works?
Logging into a popular site (facebook, hotmail, gmail etc) using the
demo-browser
Uploading an image to imgur.com using the demo-browser
Watch a cat video on youtube using the demo-browser
Xml
qmlviewer (outside of qtdemo)
Thanks Jason for a thorough list from the Qt 4 days!
This testing should be distributed on as many people as possible to lower the
load on individuals, and to allow us to get as much feedback as quickly as
possible. This in turn allows for shorter test periods, and that way we can
get the Qt releases out according to
