William Voorsluys wrote:
Hello,
I was able to run the Olio benchmark successfully, but there are still
some unresolved issues that prevent to collect results.
1) Loading the database for 100 users takes aprox. 20 minutes. I
measured this time by observing the logs. I also tried to run the
dbloader.sh script manually, but it takes the same amount of time. Is
this a acceptable time for such operation? Is there a possibility of
some misconfiguration in the database?
This is a common misconfiguration of the database that causes such long
load times. Did you use the my.cnf provided in the Olio etc directory?
Please copy that and try out.
2) I'm unable to pass the pass/fail result status, for a variety of
concurrent users: 25, 50, 100, 500. Some miscellaneous statistics
always fail, especially those related to images.
Average images references on Home Page 9.10 10 0.5
FAILED
Please load the database for 100 concurrent users or more. This one is
just because the scale of the DB is too small. The results should be
exactly 10.
Average images loaded per Home Page 3.27 >= 3
PASSED
Average image bytes received per Home Page 8108.02 >= 15000
FAILED
Average images on Tag Search Results 3.22 >= 7
FAILED
This is an issue we already fixed. But the target is still about 3.6. 7
was way too high.
Average images on Event Detail 0.92 >= 1
FAILED
Please make sure to reload the file store. Seems like some images are
missing.
Especially, "Average images on Tag Search Results" always fails.
Yes, that's a bug. We'll generate a new kit soon after we get this round
of latest bugs resolved.
------------------------------------------------------------------------
Web20Bench 0.4
Summary Report
Web20Bench metric: 20.193 ops/sec
Benchmark Start: Fri Jan 16 17:11:35 EST 2009
Benchmark End: Fri Jan 16 17:17:35 EST 2009
Run ID: Web20Driver.3C
Pass/Fail: FAILED
Active Drivers: UIDriver <#UIDriver>
------------------------------------------------------------------------
UIDriver
UIDriver metric: 20.193 ops/sec
Driver start: Fri Jan 16 17:11:35 EST 2009
Driver end: Fri Jan 16 17:17:35 EST 2009
Total number of operations: 6058
Pass/Fail: FAILED
Operation Mix
Type Success
Count Failure
Count Mix Required Mix
(2% deviation allowed) Pass/Fail
HomePage 1593 0 26.3% 26.15% PASSED
Login 621 0 10.25% 10.22% PASSED
TagSearch 1978 0 32.65% 33.45% PASSED
EventDetail 1535 0 25.34% 24.68% PASSED
PersonDetail 151 0 2.49% 2.61% PASSED
AddPerson 58 0 .96% .84% PASSED
AddEvent 122 0 2.01% 2.04% PASSED
Response Times (seconds)
Type Avg Max 90th% Reqd. 90th% Pass/Fail
HomePage 0.016 0.737 0.020 1.000 PASSED
Login 0.022 0.810 0.040 1.000 PASSED
TagSearch 0.020 0.844 0.040 2.000 PASSED
EventDetail 0.023 1.004 0.020 2.000 PASSED
PersonDetail 0.046 0.661 0.100 2.000 PASSED
AddPerson 0.249 0.793 0.360 3.000 PASSED
AddEvent 0.029 0.557 0.040 4.000 PASSED
Cycle/Think Times (seconds)
Type Targeted Avg Actual Avg Min Max Pass/Fail
HomePage 4.891 4.886 0.000 25.189 PASSED
Login 4.909 4.908 0.000 25.003 PASSED
TagSearch 4.727 4.723 0.000 25.011 PASSED
EventDetail 4.895 4.889 0.000 25.135 PASSED
PersonDetail 5.352 5.348 0.060 22.921 PASSED
AddPerson 5.124 5.119 0.080 20.726 PASSED
AddEvent 4.299 4.295 0.026 24.997 PASSED
Miscellaneous Statistics
Description Results Targeted
Results Allowed
Deviation Pass/Fail
% EventDetail views where attendee added 8.21 >= 6 PASSED
EventDetail count where attendee can be added 1204
Average images references on Home Page 9.10 10 0.5 FAILED
Average images loaded per Home Page 3.27 >= 3 PASSED
Average image bytes received per Home Page 8108.02 >= 15000
FAILED
Average images on Tag Search Results 3.22 >= 7 FAILED
Average images on Event Detail 0.92 >= 1 FAILED
Total successful AddEvent calls 140
Total successful AddPerson calls 69
Little's Law Verification
Number of users = 100
Sum of Avg. RT * TPS for all Tx Types = 96.5430
------------------------------------------------------------------------
Your report looks very good, except for the misc stats. You can
certainly scale much higher than 100 concurrent users.
Regards,
-Akara
