[dev] Re: [tools-perf] Re: [dev] Benchmarking multiple versions of OOo
On Mon, 2008-04-07 at 14:40 -0600, Andrew Z wrote: > > What I really want is something like Michael's > > http://live.gnome.org/iogrind but that just says "your app burned up > > 110,000 bogoios and 90,000,000 bogocpus" and every time you run it it > > says "110,000 bogoios and 90,000,000 bogocups". It doesn't even matter > > too much if it the ratio is wildly different to the real world as long > > as it's consistent between runs and reducing measurable bogoios reduces > > real world work by some amount. > > I am not a performance guru, but I think not all bogoios are worth the > same in practice. For example, see slides 15-16 here So - bogoio's are the right approach; the problem is less getting an accurate simulation, but getting a repeatable simulation :-) of course, accuracy is nice if you can be repeatable; but ... Unfortunately, iogrind doesn't work wonderfully well for threaded applications; but it can be run on OO.o to profile cold-start; and it might even give some useful numbers - particularly now there is a 'warming' feature (so you can simulate eg. gedit first to warm the gtk + / glibc stack (etc.)). The console mode will give you a single "12.35 bogoseconds" type number. HTH, Michael. -- [EMAIL PROTECTED] <><, Pseudo Engineer, itinerant idiot - To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]
[dev] benchmarking multiple versions of OOo
Hello, I benchmarked OpenOffice.org 1.1.5 through 3.0.0 m3 to identify trends, and I'm quasi-privately offering the benchmark system and the results to you, the experts, before I publish anything. **Method 1. I simulate a cold start (this marks the beginning of a "pass"). 2. I use Python to start OpenOffice.org, open a document, scroll through it, and save it. 3. Each iteration captures 5 durations (start, open, scroll, export, and close). 4. I repeat for 5 iterations (each pass has 5 iterations). 5. I repeat this process (a "pass") a total of 10 times per OpenOffice.org version. 6. I repeat this process (10 "passes") for the next OpenOffice.org version **Results 1. The latest versions are not the fastest---especially for cold starts. 2. Warm application startup is fairly consistent. See the spreadsheet for more. **Thoughts 1. The cold start simulator is not so good. The first pass of the first iteration is generally the slower than the first iteration of the second pass. 2. So I don't hurt performance, I sleep for 0.10 seconds while waiting for OpenOffice.org to start accepting UNO connections, but 0.10 seconds may be too high because of the small differences in warm startup. 3. I want to add OOo 1.1.0 using a conversion of the .odt test doc to .sxw. 4. OOo 1.1.5 imports the .odt test document, but you could argue it should import .sxw? Also, OOo of course saves to .sxw. **System Hardware made about three years ago AMD Athlon XP 3000+ (32-bit single core), ~750MB RAM, PATA disks, Fedora 7, Linux 2.6.23 http://www.smolts.org/client/show?uuid=pub_c71602de-1592-48e5-8dec-7f5265f4c5c5 **Downloads: Results and code You can download the results (spreadsheets with charts and numbers), the source code, and the test document at the following address. The system is a bit messy, so the code is not officially released. :) The interesting parts of the results are in >in the sheets cold_start and warm_start. If you dare run the program, please read the README.txt first. http://katana.oooninja.com/f/ref/ODF_text_reference_v1.odt <-- reference document http://katana.oooninja.com/f/tmp/benchmark-ooo-2008-04-07-08-25.tar.bz2 <-- results and code Andrew - To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]
Re: [dev] Benchmarking multiple versions of OOo
On Mon, 07 Apr 2008 20:10:25 +0100, "Caolan McNamara" <[EMAIL PROTECTED]> said: > On Mon, 2008-04-07 at 12:39 -0600, Andrew Z wrote: > > > **Thoughts > > 1. The cold start simulator is not perfect. The first pass of the > > first iteration is generally the slower than the first iteration > > of the second pass. The difference varies from -0.93s to +5.92s. > > I ran some tests at one stage on exactly the same version of OOo for 20+ > runs running the coldstart reset between each run, and then 20+ > warmstart runs. But the deviation between runs was too large for me to > accept :-( And the deviation from un-installing a version, and > re-installing the same version swings wildly too, so I retired in > defeat. I just couldn't trust the results sufficiently to reliably > determine if a change X made OOo faster or slower by comparing one set > of results against the other from two different install sets :-( Hmm. My warm starts are consistent, but I'll keep looking into cold starts. > What I really want is something like Michael's > http://live.gnome.org/iogrind but that just says "your app burned up > 110,000 bogoios and 90,000,000 bogocpus" and every time you run it it > says "110,000 bogoios and 90,000,000 bogocups". It doesn't even matter > too much if it the ratio is wildly different to the real world as long > as it's consistent between runs and reducing measurable bogoios reduces > real world work by some amount. I am not a performance guru, but I think not all bogoios are worth the same in practice. For example, see slides 15-16 here http://marketing.openoffice.org/ooocon2005/presentations/thursday_d5.pdf > > 2. So I don't hurt performance, I sleep for 0.10 seconds while > > waiting for OpenOffice.org to start accepting UNO connections, but > > 0.10 seconds may be too high because of the small differences in > > warm startup. > > How about launching OOo with a document that had a starbasic macro > hooked to onload (or whatever it was) to avoid the connect, fail, retry > cycle of connecting to the uno port, and have it do a big "killall -9 > soffice.bin" at the end. That's sort of what I tried out. Wouldn't that have its own overhead to load and execute the macro? Andrew -- http://www.fastmail.fm - Access all of your messages and folders wherever you are - To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]
Re: [dev] Benchmarking multiple versions of OOo
On Mon, 2008-04-07 at 12:39 -0600, Andrew Z wrote: > **Thoughts > 1. The cold start simulator is not perfect. The first pass of the > first iteration is generally the slower than the first iteration > of the second pass. The difference varies from -0.93s to +5.92s. I ran some tests at one stage on exactly the same version of OOo for 20+ runs running the coldstart reset between each run, and then 20+ warmstart runs. But the deviation between runs was too large for me to accept :-( And the deviation from un-installing a version, and re-installing the same version swings wildly too, so I retired in defeat. I just couldn't trust the results sufficiently to reliably determine if a change X made OOo faster or slower by comparing one set of results against the other from two different install sets :-( What I really want is something like Michael's http://live.gnome.org/iogrind but that just says "your app burned up 110,000 bogoios and 90,000,000 bogocpus" and every time you run it it says "110,000 bogoios and 90,000,000 bogocups". It doesn't even matter too much if it the ratio is wildly different to the real world as long as it's consistent between runs and reducing measurable bogoios reduces real world work by some amount. > 2. So I don't hurt performance, I sleep for 0.10 seconds while > waiting for OpenOffice.org to start accepting UNO connections, but > 0.10 seconds may be too high because of the small differences in > warm startup. How about launching OOo with a document that had a starbasic macro hooked to onload (or whatever it was) to avoid the connect, fail, retry cycle of connecting to the uno port, and have it do a big "killall -9 soffice.bin" at the end. That's sort of what I tried out. C. - To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]
[dev] Benchmarking multiple versions of OOo
Hello, I benchmarked OpenOffice.org 1.1.5 through 3.0.0 m3 to identify trends, and I'm quasi-privately offering the benchmark system and the results to you, the experts, before I publish what I plan as the first article in a performance series. **Method 1. I simulate a cold start (this marks the beginning of a "pass"). 2. I use Python to start OpenOffice.org, open a document, scroll through it, and save it. 3. Each iteration captures 5 durations (start, open, scroll, export, and close). 4. I repeat for 5 iterations (each pass has 5 iterations). 5. I repeat this process (a "pass") a total of 10 times per OpenOffice.org version. 6. I repeat this process (10 "passes") for the next OpenOffice.org version **Results The most remarkable result is the latest versions are not the fastest---especially for cold starts and for exporting. See the spreadsheet for all the results. **Thoughts 1. The cold start simulator is not perfect. The first pass of the first iteration is generally the slower than the first iteration of the second pass. The difference varies from -0.93s to +5.92s. 2. So I don't hurt performance, I sleep for 0.10 seconds while waiting for OpenOffice.org to start accepting UNO connections, but 0.10 seconds may be too high because of the small differences in warm startup. 3. I want to test OOo 1.1.0 using a conversion of the .odt test doc to .sxw. 4. OOo 1.1.5 imports the .odt test document, but you could argue it should import .sxw? Also, OOo of course saves to .sxw. **System Hardware made about three years ago AMD Athlon XP 3000+ (32-bit single core), ~750MB RAM, PATA disks, Fedora 7, Linux 2.6.23 http://www.smolts.org/client/show?uuid=pub_c71602de-1592-48e5-8dec-7f5265f4c5c5 **Downloads: Results and code You can download the results (spreadsheets with charts and numbers), the source code, and the test document at the following address. The interesting parts of the results are in in the sheets cold_start and warm_start. If you dare run the program, please read the README.txt first. It's a bit messy, so the code is not officially released. :) http://katana.oooninja.com/f/ref/ODF_text_reference_v1.odt <-- reference document http://katana.oooninja.com/f/tmp/benchmark-ooo-2008-04-07-08-25.tar.bz2 <-- results and code Andrew P.S. Apologies if this comes across twice: I sent hours ago from gmail, but it hasn't appeared. -- http://www.fastmail.fm - Faster than the air-speed velocity of an unladen european swallow - To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]
