Hi folks,
I've been doing some rudimentary load testing of a Spectra app we're
working on, and the results are less than I would have liked. I'm hoping
they can be explained away. :) Here's the deal:
Platform: dual-processer Pentium III 800Mhz with 1GB of RAM running
Win2000, IIS, Spectra 1.0.1, CF 4.5 sp1, SQL Server 7.0.
Target Template: Very basic. All it does is use cfa_contentObject to call
the display method of an object (the ObjectID is passed to the
template). The object has 20 properties or so, but they are all basic text
or numeric data - no arrays, structures, or embedded objects.
Test Script: A Perl script that uses the ParallelUserAgent module to fake
concurrent users. These virtual users are quite a bit more relentless than
real users would be, as they immediately make a new request as soon as they
receive the first packet from the response of their last request.
Here are my test results:
--- 1 USER / 30 REQUESTS ---
Total Time: 8.46 secs
Throughput: 3.55 Requests/sec
Latency: 0.28 secs/Request
--- 2 USERS / 60 REQUESTS ---
Total Time: 8.54 secs
Throughput: 7.02 Requests/sec
Latency: 0.28 secs/Request
--- 4 USERS / 120 REQUESTS ---
Total Time: 19.55 secs
Throughput: 6.14 Requests/sec
Latency: 0.64 secs/Request
--- 8 USERS / 240 REQUESTS ---
Total Time: 58.99 secs
Throughput: 4.07 Requests/sec
Latency: 1.94 secs/Request
--- 16 USERS / 240 REQUESTS ---
Total Time: 60.37 secs
Throughput: 3.98 Requests/sec
Latency: 3.87 secs/Request
--- 32 USERS / 480 REQUESTS ---
Total Time: 119.31 secs
Throughput: 4.02 Requests/sec
Latency: 7.65 secs/Request
--- 64 USERS / 640 REQUESTS ---
Total Time: 164.98 secs
Throughput: 3.88 Requests/sec
Latency: 15.43 secs/Request
--- 100 USERS / 1000 REQUESTS ---
Total Time: 255.64 secs
Throughput: 3.91 Requests/sec
Latency: 24.00 secs/Request
These numbers didn't look too good to me (24 seconds per request!), so of
course I had to wonder if the cause was poor programming on our part, or
Spectra itself. The page we are requesting couldn't be simpler from a
programming perspective. All it does it invoke the display method of an
object. There is no way to make the programming simpler. So I figure the
overhead is either coming from Spectra or from CF server itself. To test
that, I copied the target page, but removed the Spectra tags and instead
did a straight DB query to return objectdata from the Objects table without
deserializing it. Here are the results (Spectra baseline in parens):
--- 16 USERS / 240 REQUESTS ---
Total Time: 36.30 secs (60.37)
Throughput: 6.61 Requests/sec (3.98)
Latency: 2.32 secs/Request (3.87)
--- 32 USERS / 480 REQUESTS ---
Total Time: 71.07 secs (119.31)
Throughput: 6.75 Requests/sec (4.02)
Latency: 4.52 secs/Request (7.65)
--- 64 USERS / 640 REQUESTS ---
Total Time: 98.30 secs (164.98)
Throughput: 6.51 Requests/sec (3.88)
Latency: 9.10 secs/Request (15.43)
--- 100 USERS / 1000 REQUESTS ---
Total Time: 149.91 secs (255.64)
Throughput: 6.67 Requests/sec (3.91)
Latency: 13.95 secs/Request (24)
Much better, but I was still hoping it would scale better to 100 users. So
then I wondered if all that stuff we're doing on the application scope was
bogging things down. So I took my CF test above, and put it in it's own
directory with it's own stripped application.cfm file. The results (this
time with the straight CF and Spectra results in parens):
--- 16 USERS / 240 REQUESTS ---
Total Time: 30.33 secs (CF: 36.30 / Spectra: 60.37)
Throughput: 7.91 Requests/sec (CF: 6.61 / Spectra: 3.98)
Latency: 1.96 secs/Request (CF: 2.32 / Spectra: 3.87)
--- 32 USERS / 480 REQUESTS ---
Total Time: 56.47 secs (CF: 71.07 / Spectra: 119.31)
Throughput: 8.50 Requests/sec (CF: 6.75 / Spectra: 4.02)
Latency: 3.64 secs/Request (CF: 4.52 / Spectra: 7.65)
--- 64 USERS / 640 REQUESTS ---
Total Time: 72.78 secs (CF: 98.30 / Spectra: 164.98)
Throughput: 8.79 Requests/sec (CF: 6.51 / Spectra: 3.88)
Latency: 6.99 secs/Request (CF: 9.10 / Spectra: 15.43)
--- 100 USERS / 1000 REQUESTS ---
Total Time: 114.85 secs (CF: 149.91 / Spectra: 255.64)
Throughput: 8.71 Requests/sec (CF: 6.67 / Spectra: 3.91)
Latency: 11.03 secs/Request (CF: 13.95 / Spectra: 24.00)
Better, but not as dramatic a change. So for 100 users, I'm down at 11
seconds per request in a bare-bones CF template that just pulls one record
from a properly-indexed SQL Server table.
Just for kicks, I tried running the load test against a
mod_perl/MySQL/Apache app I built. The results (along with the best-worst
CF-Spectra range in parens):
--- 16 USERS / 240 REQUESTS ---
Total Time: 12.83 secs (30.33 to 60.37)
Throughput: 18.71 Requests/sec (7.91 to 3.98)
Latency: 0.82 secs/Request (1.96 to 3.87)
--- 32 USERS / 480 REQUESTS ---
Total Time: 24.49 secs (56.49 to 119.31)
Throughput: 19.60 Requests/sec (8.5 to 4.02)
Latency: 1.57 secs/Request (3.64 to 7.65)
--- 64 USERS / 640 REQUESTS ---
Total Time: 33.09 secs (72.78 to 164.98)
Throughput: 19.34 Requests/sec (8.79 to 3.91)
Latency: 3.16 secs/Request (6.99 to 15.43)
--- 100 USERS / 1000 REQUESTS ---
Total Time: 53.24 secs (114.85 to 255.64)
Throughput: 18.78 Requests/sec (8.71 to 3.91)
Latency: 5.06 secs/Request (11.03 to 24.00)
Since I *MUCH* prefer writing CF code to Perl code, I was dismayed to find
that Perl, doing heavier lifting (running two queries, one of which returns
multiple rows and some involved display vs. one query returning one row
with minimal display) on a vastly inferior machine (Pentium 2 350
single-processor w/ 128MB RAM vs. Pentium 3 800 dual-processor with 1GB
RAM) seemed more than twice as fast as Cold Fusion at it's best.
Can this be right? I know statistics and benchmarks can be twisted to say
just about anything, but I *want* CF to be faster. And this really makes
it seem like if you expect even moderate traffic you should be running
Spectra on a cluster from day one. Is that an accurate assessment? Or
should I assume that my load test script is behaving like more than 100
"actual real-live" users.
Thanks!
Jim
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Structure your ColdFusion code with Fusebox. Get the official book at
http://www.fusionauthority.com/bkinfo.cfm
------------------------------------------------------------------------------
To Unsubscribe visit
http://www.houseoffusion.com/index.cfm?sidebar=lists&body=lists/spectra_talk or send a
message to [EMAIL PROTECTED] with 'unsubscribe' in the body.
