Hi,
albeit *not* as a daemon, we've successfully developed a Crawler in PHP
within our company. It can run for hours without a leak, if I remember
correctly it's peak memory consumption is below 64MB. However we're
crawling only a small amount of URLs, just around 10.000 .
As Brian mentioned: fre
First, this is not an internals question. But, I would be not many
people on the general list could help you.
I wrote some daemon scripts for a web crawler in PHP. Now I spent one
day to work around the growing memory consumtion of these scripts.
I have PHP daemons running for weeks that don'
Hi,
I wrote some daemon scripts for a web crawler in PHP. Now I spent one
day to work around the growing memory consumtion of these scripts.
Since I use a MySQL connection, Syslog and many classes, I wanted to let
the script run a while before restarting. So the scripts had lines like:
$i = 1000