On Wed, Dec 05, 2001 at 09:31:46 +0000, Wiliam Stephens wrote:
> I've written the below sub to check all URLs in my database and return a
> error message for each individual URL if the status code returned is not
> 200.
> Does anyone know of a (much) faster way of doing this? Is LWP::UserAgent
> the way to go, or is the way I'm accessing my MySQL slow?
Have you benchmarked your code?
I'm betting that the internet connection/webserver part is
slow here, i.e. it's not your code nor the DB.
Having been faced with a similar problem (and potential very
long timeouts) I used Parallel::ForkManager and LWP::Simple
(I need to download them as well) with good success.
Be aware however that any END block will be executed if any of
your children thus finishes (that includes automatic cleanup
(can be disabled) in File::Temp, etc). A working defense is
END {
exit unless ( $::PARENTS_PID and $$ == $::PARENTS_PID );
[your code here]
}
and setting $::PARENTS_PID as required. (No, you cannot easily
assume that $::PARENTS_PID is set, look up 'perl -c'.)
Also be aware that you will be starting the defined number of
processes -- runnin in parallel you can get slumped. I use
max ((load - maxload) * factor, 1)
as a limiter, executed ever so often -- and twiddling the factor
(automatically) for added performance. :-)
-Wolfgang
