On 22 Mar 2007, at 18:29, Richard Gaskin wrote:
Dave continued:
I suppose my point is that in order to have a beta test, you need
to have gone thru the steps to get there. It's no good producing
mountains of code, doing little or no testing at the development
phase and then throwing that out for beta.
With 20/20 hindsight it's easy to suggest that stress testing would
have found this leak, and indeed it might well have. But I don't
think it would be a cost-effective practice for RunRev to adopt
across this product.
In the ten years I've been working with this engine this is the
first verified leak I've seen. Let's be generous and say that
maybe one or two others might have been discovered in that time.
Even then, over a decade that's really quite good -- and
accomplished without automated stress testing.
There were three problems, the leak was just one of them.
The export command appears to work well when run once or even a
dozen times. Unit testing should always be done, and in this case
would yield a good result. Only a sustained test with a great many
iterations will expose this specific problem, and only in the Rev
IDE. The leak doesn't exist in a standalone or in other IDEs, and
since some issues may be specific to standalones it would be
necessary to run any soak tests in at least those two environments.
Not really if you were to write files 1 to 300 you would hit it at
288 and I had it happen earlier than that to start with. In fact the
memory leak would be visible straight away, all you have to do is run
once it and look at the memory allocations.
And because Rev supports multiple operating systems, each with
differing APIs and idiosyncrasies, each of those two tests would
need to be run on all supported OSes. Here we already have a
combinatorial explosion of test scenarios: 2 engines X Win 98,
Win2K, WinME, WinXP, Vista, Mac OS 9, OS X 10.2, OS X 10.3, OS X
10.4, at least one Linux, and for anything that involves QT
multiply 9 of those by the number of supported QT versions. That's
20 tests without QT, and just for one command.
Agreed. To test it all those platforms would be hard. But in this
case it just needed to be tested on the development machine of the
person doing the coding and stressed for a lot of operations. I do
this automatically without even thinking about it.
I agree that you'd have to run it under the IDE and as a Standalone.
To start with you just run it in the IDE, then at some point when you
feel the time is right you build a standalone. To build a standalone
for Mac and Windows takes literally seconds and in the case of the
export snapshot the tests take 2 minutes to run.
Consider all the effort to design, build, and run these tests,
release after release, year after year, and after a decade of that
we might turn up a leak or two and maybe a relative handful of
other errors of the sort which can be exposed through that specific
form of testing.
Took me 10 minutes to build the test for the export snapshot command
and 2 minutes to run it. On the first part I was working on (last
week) it took me 30 minutes to build the tests and about the same to
run the tests. I then ran it at least once a day (after I'd added/
changed things) to make sure I hadn't broken something.
Another problem here is that people may have different ideas on
what "Beta" means and I haven't seen it defined in terms of
RunRev. One company I worked for defined it as meaning "Feature
Complete, No Known Crashing Bugs".
That's the ideal, but I've known no company that ships every Beta
in a state that meets that definition.
Well, I've beta tested Photoshop and I AFAIK there were no known
crashing bugs and AFAIR it was feature complete.
I've participated in tests for Adobe, Microsoft, Apple, Oracle, and
others who have shipped Betas while still developing new features.
"Feature Complete" was just the way that company did it, I've also
seen that beta versions that still being developed. I was trying to
find out what "beta" meant in the wonderful world of RunRev.
That you're able to run stress tests on all features and identify
100% of requirements to complete satisfaction before your first
Beta is quite an accomplishment,
Could you tell me where I said that? I run Stress Tests while
developing my software as for requirements and when Beta testing is
performed is up to the company I am working for in this case. In the
past I have worked where this was the case though.
However stress testing is not an accomplishment at all it's really
easy, that's why I really can't see why you are going on about it, I
do it without even thinking about it! In fact I was absolutely
gobsmacked that you were surprised that this was my way of working.
and we look forward to URLs of the products you ship so we can all
learn how to improve the quality of our own work.
Now you're just being silly! How will looking at the products I have
worked on help you learn how to improve the quality of your own work?
Besides which unless you just happen to have £50,000+ worth of video
equipment lying around it wouldn't do you much good!
And Apart from all that - I've told you how to do it already!
Take Care and All the Best
Dave
_______________________________________________
use-revolution mailing list
use-revolution@lists.runrev.com
Please visit this url to subscribe, unsubscribe and manage your subscription
preferences:
http://lists.runrev.com/mailman/listinfo/use-revolution
