Knut Anders Hatlen wrote:
"Daniel John Debrunner (JIRA)" <[EMAIL PROTECTED]> writes:
Daniel John Debrunner commented on DERBY-2118:
----------------------------------------------
The tests pass for the clearLimit() changes and I performance
changes for the select test ranging from -0.70% to +2.44%.
Sometimes I seem to have trouble getting consistent numbers out of
the test from DERBY-1961, even with an idle machine.
I am also seeing huge variations with the test from DERBY-1961. In
fact, I always see huge variations when I'm running performance tests
on Derby. My best guess so far is that the native code produced by the
hotspot compiler varies between runs (it uses statistics to decide
what to optimize and how to optimize it, so it might not be 100%
predictable). To eliminate the variation, I usually look at the
average of multiple runs.
True, but when the variations are multiples of the percent improvement
I'm expecting for the change (ie. 5-10% for a 1% change) I have little
confidence I can detect if the code is providing an improvement or not.
It would be good to convert this test to the harness Sunitha provided in
DERBY-1970, then multiple runs within the same VM could be performance,
these tend to become more stable as time the JIT settles down. Of course
some of it could be due to garbage collection.
Also DERBY-1961 test has a minor problem in its TPS reporting though it
does not seem to be significant cause in the inconsistent numbers. The
number of operations is counted (N) while the main thread is sleeping
for R seconds and then TPS = N / R. The issue is that the thread is not
guaranteed to sleep for R seconds, thus the collection time is really (R
+ delta), so it should be TPS = N / (R + delta). I modified the code to
measure the actual sleep time and it seems with these tests delta was
less than 0.01 seconds. I can imagine with some multi-client tests that
delta could become significant. Moving to the DERBY-1970 harness would
at least ensure we had a single place where we could work on consistent
good reporting numbers.
Thanks,
Dan.
