RE: Re: Re: Web Crawler

[ricarDo oliveiRa]

I can't answer that, since I wasn't working on this project when it started
some time ago. But it must have been version beta-beta-beta-beta :))

But there are lot of open source crawlers avaiable. The Google isn't open
source anymore.

Take a look at http://www.searchtools.com/tools/tools.html

Cheers
./Ricardo Oliveira


--Original Message--
From: "Miguel A.L. Paraz" [EMAIL PROTECTED]
To: ricarDo oliveiRa[EMAIL PROTECTED]
Sent: November 23, 1999 1:31:45 PM GMT
Subject: Re: Re: Web Crawler


On Tue, Nov 23, 1999 at 07:00:27AM -0500, ricarDo oliveiRa wrote:
 - I'm not writing a web crawler from scratch. the initial source code was
 based on Google's, adapted to serve the specific needs of this web site

where did you get source code of Google!?

cheers,
---m

--
Miguel "Migs" A.L. Paraz IPhil Communications Network, Inc.
http://www.iphil.net   Business Development/Training
Group
5/F 116 Herrera St., Legaspi Village, Makati City, Philippines 
+63-2-750-2288

__
FREE Email for ALL! Sign up at http://www.mail.com

Web Crawler

[ricarDo oliveiRa]

Hi there,

thanks in advance for any further help.
I have a web crawler running in a solaris+apache+mod_perl web server, and
for some reason, when I try go get the contents of a certain page, it hangs
and gives no timeout whatsoever.

use LWP::Simple;

...

$page_text = get($thisURL);

...

besides this problem, the crawler is monolithic. can I do a fork to speed
things up? Any suggestions?


Thanks!
./Ricardo Oliveira

__
FREE Email for ALL! Sign up at http://www.mail.com