You may want to use 'wget' spider function, then parse the files with PHP if necessary.
http://wget.sunsite.dk/ bvr. On Thu, 24 Jan 2002 14:52:54 +0100, Mattias Andersson wrote: >Hello! > >I was making a spider for a simple searchengine and all was well until I >started testing it on larger sites. >The problem isn't that it doesn't work, it does, but it is very very slow, >just handling one connection at the time. >Basically, it just downloads a page, extract all links from it, then >downloads those pages, extract their links, until all pages on a site has >been downloaded and archived. >Is there any way to do it in several threads so that it handles several >connections simultaneously? >If it would dynamically allocate threads that would be even better. > >Regards, >Mattias Andersson > >Software Developer, humany AB >Direct: 08-4540701 >Mobile: 0704-526685 > > -- PHP General Mailing List (http://www.php.net/) To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED] To contact the list administrators, e-mail: [EMAIL PROTECTED]
