Thought I would chime in with my thoughts on freezes, snapshot testing,
and QA in general.
Freezes:
When I was testing the Arm images, there were times when I would go for
weeks without an image. And if I remember correctly, there was a period
during Natty where there were no images between milestones due to pool
skew. And there were image specific changes being made at the time
(jasper-initramfs, Ubiquity/oem-config, etc). When those changes can't
be tested without an image, failures become critical at milestone,
causing a lot of needless stress.
Manual QA:
Having been in a QA role for most of my career in computers (+25 years),
I can site areas where automated QA can and will fail. Sure, repetitive
tests can be automated, especially server stacks and background
libraries. But how do you automate testing a GUI for usability
(Evolution STILL has option windows that flow past the bottom of my
netbook screen). Plus there are tests that find the oddball combination
that most developers don't think users will select (enabling autologin &
encrypted home directory during install was found this way). When
developing automation scripts for tests, they really should be designed
to test on all platforms, not just simulated/virtualized environments.
Otherwise others have to constantly reinvent the wheel to run the same
test on systems that don't support that test environment.
Snapshot retention:
I set up my own internal mirror so that I could keep daily images
between milestones. This way if I missed something during daily testing
and we needed to determine how far back it went, I had a complete image
history to go back to (Mono/Banshee in Oneiric). This also helped to
determine if one package or a combination of package updates caused
issues.
As to testing 15 arm images during milestones, my secret was to take a
daily image for one platform (panda) and focus on it over a period of
2-3 days, testing everything. Most bugs in applications there were the
same on all arm platforms. That left hardware specific testing on other
platforms to be done only when testing kernel/xorg updates, or (in the
case of Mono not supporting SMP on arm) reproducing a bug found on the
main test environment. I would also try to reproduce the bug on
x86/amd64 to determine if the bug was arch specific. Often times, an
application bug found on the panda would also be found on other
architectures.
While I no longer have the time to test Ubuntu images, I am saddened by
the failures I do see on a daily basis on the LTS release (anyone try to
run a remote desktop lately using rdesktop?). These issues should have
been found prior to release.
--
--
Tobin Davis
The "cutting edge" is getting rather dull.
-- Andy Purshottam
--
ubuntu-devel mailing list
ubuntu-devel@lists.ubuntu.com
Modify settings or unsubscribe at:
https://lists.ubuntu.com/mailman/listinfo/ubuntu-devel
