Graham Dumpleton wrote: >....... > In that blog, Guido says: > > """Concurrency: It seems we're now happily out exploring here. I'm > looking forward to benchmarks showing that PP or similar (or > dissimilar!) solutions actually provide a performance gain. Another > route I'd like to see explored is integrating one such solution into > an existing web framework (or perhaps as WSGI middleware) so that web > applications have an easy way out without redesigning their > architecture.""" .......
Well I have been trying out a system very like PP. For various reasons I prefer to write my own scheduler based on more traditional code, but it's very easy to get SSH based workers going. Agreed it's a noddy system, but I find the following timings for the pp callback demo with start=1, end=200000000. All the workers are similar to a pp local worker, but some are controlled via ssh and hence have to be initialised by an argument like -c"exec input()" which downloads some bootstrap code; I find that an easy way to utilize a server which already has ssh. The local workers are clearly dominant, but even though app1,app3,app4 are not in the same country they can still be used to provide useful work. I suspect most people won't want to use either ssh or socket based workers when 64 cpu machines arrive. The worker farm topology is also a bit wrong for massively parallel stuff. ################################################################ Starting scheduler with 1 workers(1 local) Partial sum is 0.69314718306 | diff = -2.50006693125e-009 Job execution statistics: job count | % of all jobs | job time sum | time per job | worker 128 | 100.00 | 59.7190 | 0.466555 | local:3676 128 | 100.00 | 59.7190 | 0.466555 | all local 128 | 100.00 | 59.7190 | 0.466555 | total Time elapsed since scheduler creation 59.7820000648 Real average time 0.467046875507 Starting scheduler with 2 workers(2 local) Partial sum is 0.69314718306 | diff = -2.50006693125e-009 Job execution statistics: job count | % of all jobs | job time sum | time per job | worker 64 | 50.00 | 31.5300 | 0.492656 | local:324 64 | 50.00 | 31.2190 | 0.487797 | local:3684 128 | 100.00 | 62.7490 | 0.490227 | all local 128 | 100.00 | 62.7490 | 0.490227 | total Time elapsed since scheduler creation 31.5460000038 Real average time 0.24645312503 Starting scheduler with 3 workers(2 local) Partial sum is 0.69314718306 | diff = -2.50006704228e-009 Job execution statistics: job count | % of all jobs | job time sum | time per job | worker 53 | 41.41 | 25.6580 | 0.484113 | local:2392 52 | 40.63 | 25.5150 | 0.490673 | local:416 23 | 17.97 | 24.2500 | 1.054348 | ssh:RLUK_robin:20457 105 | 82.03 | 51.1730 | 0.487362 | all local 128 | 100.00 | 75.4230 | 0.589242 | total Time elapsed since scheduler creation 25.7349998951 Real average time 0.20105468668 Starting scheduler with 4 workers(2 local) Partial sum is 0.69314718306 | diff = -2.50006704228e-009 Job execution statistics: job count | % of all jobs | job time sum | time per job | worker 46 | 35.94 | 22.2200 | 0.483043 | local:2328 46 | 35.94 | 22.4670 | 0.488413 | local:3580 21 | 16.41 | 22.3150 | 1.062619 | ssh:RLUK_robin:20500 15 | 11.72 | 21.9730 | 1.464867 | ssh:app1:3549 92 | 71.88 | 44.6870 | 0.485728 | all local 128 | 100.00 | 88.9750 | 0.695117 | total Time elapsed since scheduler creation 23.0320000648 Real average time 0.179937500507 Starting scheduler with 5 workers(2 local) Partial sum is 0.69314718306 | diff = -2.50006704228e-009 Job execution statistics: job count | % of all jobs | job time sum | time per job | worker 40 | 31.25 | 19.3600 | 0.484000 | local:1376 41 | 32.03 | 19.7030 | 0.480561 | local:3608 18 | 14.06 | 19.1090 | 1.061611 | ssh:RLUK_robin:20524 14 | 10.94 | 19.1110 | 1.365071 | ssh:app1:3596 15 | 11.72 | 19.0950 | 1.273000 | ssh:app3:44316 81 | 63.28 | 39.0630 | 0.482259 | all local 128 | 100.00 | 96.3780 | 0.752953 | total Time elapsed since server creation 19.9850001335 Real average time 0.156132813543 Starting scheduler with 7 workers(2 local) Partial sum is 0.69314718306 | diff = -2.50006693125e-009 Job execution statistics: job count | % of all jobs | job time sum | time per job | worker 31 | 24.22 | 14.9540 | 0.482387 | local:2588 31 | 24.22 | 15.0630 | 0.485903 | local:472 13 | 10.16 | 14.0790 | 1.083000 | ssh:RLUK_robin:20548 9 | 7.03 | 14.7680 | 1.640889 | ssh:app1:3626 11 | 8.59 | 14.3260 | 1.302364 | ssh:app3:44344 15 | 11.72 | 14.2990 | 0.953267 | ssh:app4:61396 18 | 14.06 | 14.4050 | 0.800278 | ssh:app5:1258 62 | 48.44 | 30.0170 | 0.484145 | all local 128 | 100.00 | 101.8940 | 0.796047 | total Time elapsed since scheduler creation 15.2030000687 Real average time 0.118773438036 ################################################################ -- Robin Becker -- http://mail.python.org/mailman/listinfo/python-list