I think people are making this unnecessarily complicated. Variable does not
mean random. Jobs don't arbitrarily consume two or three times the
resources they normally do for "identical" runs, so the question is simply
one of how to account for normal variation.
There is no getting around the need for multiple runs to get a sense of how
much variation occurs under similar loads. Running benchmarks to evaluate
the effects of "light" versus "heavy" loads are also necessary. But more
importantly I think we need to dispense with the silly notion that we have
any concept of what "identical" is.
Given the large numbers of events occurring in modern computer systems, this
is somewhat analogous to the notion of individual water droplets from a lawn
sprinkler. Ultimately we can agonize over the paths of individual water
particles, and discuss all the elements that can impede or improve the range
a droplet travels but in the end, we only want to get our lawn watered.
I don't want to offend anyone, but if you're worried about CPU microseconds
and coding in high-level languages, I would suggest there is a fundamental
disconnect and it makes me think you're not really serious.
In addition, if you don't have access to performance data, then regardless
of the "demands" made by management, you're going to find it difficult to
evaluate performance results. Regardless of the politics or frustrations
involved, performance cannot be derived psychically, but rather it involves
simply crunching the numbers. So you will need them .. audit requirements
or not.
If variability in the measurement tool hides the improvement, is it
worth doing? Maybe it would be in the production environment, but the
variability in the test environment makes it impossible to prove in
advance.
Adam
----------------------------------------------------------------------
For IBM-MAIN subscribe / signoff / archive access instructions,
send email to [EMAIL PROTECTED] with the message: GET IBM-MAIN INFO
Search the archives at http://bama.ua.edu/archives/ibm-main.html
