Simon Clewer wrote
Hi,I can suggest to create a frontend proxy with mod_proxy or mod_accel (a better aproach) -
We're using ithreads with modperl to run some complicated robots concurrently ( in a commercial environment ) - there is however an issue.
Huge memory usage ... each ithread uses about 10M of ram ( image of Apache,
image of mod perl and image of our deep-link robot ), and as we use 5
ithreads plus the original thread that means that each Apache is using 60 M
and because we trade on being the best and the fastest at what we do we need
to keep plenty of Apaches ready and waiting ( about 20 ) - so we're using
heaps of memory.
You can keep 2-3 mod_perl processes on a backend server, and 20+ on the frontend,
each apache+mod_proxy process will be around 1,5Mb. With 2 Gigs on board you can run
20+ mod_perls and 200+ frontent server processes.
Also, i would like to suggest not to fork robots directly from apache,
but make them running as another preforked server. Your apache then
should display a refreshing page ("please wait") until a result is fetched
into cache by some of the robots.
That also means you should intensively cache robot's results where possible.Surely, I can be completely wrong since I don't know all the details. This is just a suggestion :)
Sincerely, Alex
-- Reporting bugs: http://perl.apache.org/bugs/ Mail list info: http://perl.apache.org/maillist/modperl.html
