On Thu, Jul 5, 2012 at 11:11 PM, zhangjf <zhan...@apache.org> wrote: > The right url is > http://wiki.services.openoffice.org/wiki/Performance/AOO3.4.1_PVT_Introduction >
Great. Thanks! A few questions: 1) Are the scripts and test documents checked into SVN? 2) How long does it take for a complete run of the tests? 3) What is the order of the tests? For example, are you doing: a) document 1 run 1, document 1 run 2, document 1 run 3...document 1 run 8, document 2 run 1...document 2 run 8, etc. or b) document 1 run 1, document 2 run 1, document 3 run 1...document N run 1, then document 1 run 2, document 2 run 2, etc. 4) Within the test do you restart OpenOffice? If so, do you restart after every document? Or every measurement? If it is at all possible to take more measurements, I think we would get more high quality results. Right now, you take 8 measurements and throw away 3 of them (first run, highest time and lowest time). That throws away information and biases the results because the first run is probably also slower, so you toss out the two slowest runs but only toss out the single fastest run. But the fastest run is probably also the most accurate one, since there are many things in a test that can accidentally slow things down, but almost nothing can happen to make a test run faster. (assuming the test logic is accurate). In general, in an experiment, keep all the data you have, and get more accurate results by doing more repetitions. For example, what if we did 100 iterations of each test? How long would that take? If we did that, it would have some benefits: 1) We wouldn't need to worry about tossing out high and low values. Our error bounds would be good because of the number of runs we have. The impact of any one anomalous measurement will be much smaller. 2) We could at the same time look at the trend of the measurement over the test run. For example, compare the average of the first 10% of the runs with the average of the last 10%. Is there a difference? If a test slows down over time that might indicate a memory leak or other problem. You will never find this with only 8 measurements. 3) It would tell us the distribution of timings, as well as the average. (Another trick. If you are going to do N load measurements of the same document, maybe start the test run by creating N identical copies of the same document on disk. Then load each copy only once. That helps even out the disk cache and I/O environment compared to loading the exact same file N times) -Rob > On Fri, Jul 6, 2012 at 11:07 AM, Yi Xuan Liu <liuyixuan....@gmail.com> wrote: >> Hi, all: >> >> I wrote a wiki >> http://wiki.services.openoffice.org/wiki/Performance/AOO3.4.1_PVT_Introductionabout >> PVT project in AOO 3.4.1. >> Any comment is welcomed!
