On Mon, 23 Jan 2006 01:03:42 +0100, Andrea Arcangeli <[EMAIL PROTECTED]> wrote:
Hello,
With Cheetah I have full control on the serialization or parallelism of
my queries and thanks to a real dbms I can execute very expensive
queries in parallel on multiple cpus with just a small change:
[snip big diff]
I've an old (slow) 2-way smp server, and by sending (readonly) queries
to postgresql in parallel I just reduce the time it takes to render the
usual klive pages from 2.9sec to 1.49sec (2 cpus working at 100%
executes the queries in parallel and so the results now arrives in
_exactly_ half the time, postgresql is really scaling perfectly in this
test). This new smp improvement comes after the huge boost I had after
moving from Nevow to Cheetah for the rendering of the html (from >5.9sec
to 2.9sec). On my 4-way dualcore opteron desktop the rendering time
changes from ~1sec to ~370msec.
So the overall since I started migrating to Cheetah, the time it takes
to render the klive homepage went down from 5.9sec (old
nevow+twisted.web code) to 1.5sec (new cheetah+twisted.web2 code). A
~300% performance improvement in rendering pages isn't bad. The code is
almost unchanged, just the html rendering model has changed.
This is just to share my real life experience and perhaps this could be
useful to others.
Now the 1.5sec are almost all spent waiting the sql queries and anyway
both cpus are fully utilized (Cheetah renders the first #blocks while
the other cpu calculates the results for the next block in parallel), so
there's probably not much left to optimize (nothing as easy as the
changes I did in the last days to move from 5.9sec to 1.5sec at least).
That's great, Andrea.
Parallel expensive queries to the db, is something that the axiom model
will never be able to achieve (as long as the api isn't changed to
return deferreds to allow real dbms instead of sqllite, which means all
code written today for the current axiom API will have to changed
significantly). Having to schedule to another context while waiting the
deferred to fire, may open a window for for race conditions too,
and have fun writing unit-test verifying race conditions.
While you can scale the network load by running multiple twisted
servers, you can't scale the db queries coming from different twisted
servers in parallel with the axiom API. This ignoring the fact nevow
doesn't contemplates the possibility of sending all queries in parallel.
I don't understand why you insist on continuing to harp on this point, though. Even
if it were accurate (which I must stress, it /isn't/, and if you care to discuss the
technical merits of Axiom, I'd be happy to do so on <[EMAIL PROTECTED]> but
/not/ here), no one cares. You've been asked repeatedly to let it go, and by
refusing to do so you are being very rude to the people who wrote the software you
continue to use. Please, you do your thing, let us do ours.
Jean-Paul
_______________________________________________
Twisted-web mailing list
[email protected]
http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-web
