Just a comment on this...
I used to work for a pretty big Unix OS vendor in the operating systems
development group. 90% of the bug fixes I applied were never found by
the QA group (otherwise they would have been fixed long before I ever
worked there :-). Where they really found problems were cross-platform
problems. E.g. I committed a change, tested it on platform X & Y and
it worked just fine. It failed to build on legacy platform Z. Building
and testing on platform Z was near impossible, so as a developer, unless
you were actually fixing a problem for platform Z, you never did it.
After being bit a couple of times, I automated a script that did the
cross-build, and never got bit again, but most people didn't do that.
Heck, a major part of the QA group was dedicated to making sure the
documentation was correct (not a bad thing, but it didn't help
the actualy state of the software).
Our -current works pretty much like the QA we had at this company.
Most machines ran what we call -stable. All development machines (and
there were a lot) ran -current, and a few ran a couple of releases back,
just in case we had to fix something in that release.
Knowing how many people actually run -current, I would say we have
better testing than the OS company I worked for. Even though we don't
have a real "QA" dept.
-Mike
On Thu, Dec 21, 2000 at 09:48:44AM -0800, SteveB wrote:
> Here's the thing about open software that still concerns me. My
> background is with the major software development tools companies, so
> that is my point of reference. It is great that code is available and
> fixes are made and pushed out, but who is doing real testing of these
> fixes. Sure the obvious problem is fixed, but what other problems has
> it uncovered, what side effect has it created, and how about
> compatibility with other software or drivers in this case.
>
> With commercial software (well at least the places I worked) nothing
> could go out the door without a complete QA cycle performed on it.
> Even the smallest of bug fixes couldn't be released without a QA
> cycle. A full QA cycle was time consuming and expensive, so fixes sat
> until there was a stack of them to QA'd as a group or they had to wait
> until next upgrade. That way we knew state of the product. Yes, the
> state of the product would include known bugs. The key was a known bug
> and a known documented bug was as valuable as a fix. Sure a bug is
> bad, but if it is documented you don't waste trying to make something
> work that is known to be broke.
>
> So who is testing these fixes in open source world? Just seeing if
> the problem at hand is gone isn't real testing, even claiming
> thousands of people are now using it isn't enough. There can still be
> lurking potentially data destroying bugs lurking. In the open source
> world is there a official QA process or group. Is there a FreeBSD
> test suite that releases go through. QA is unglamorous work, but
> needs to be done.
>
> Steve B.
--
Mike Pritchard
[EMAIL PROTECTED] or [EMAIL PROTECTED]
To Unsubscribe: send mail to [EMAIL PROTECTED]
with "unsubscribe freebsd-hackers" in the body of the message
