Hello everyone ,
How do you guys create your web crawler in such a way that it would step over bot bait pages like WSPosion? Do you simply include them in a list of urls to avoid ? or do you keep track of web sites with unusually large amounts of web page such as a web site with about 200 pages before abandoning or sending an alert ? stephen __________________________________________________ Do You Yahoo!? Yahoo! Sports - live college hoops coverage http://sports.yahoo.com/ -- This message was sent by the Internet robots and spiders discussion list ([EMAIL PROTECTED]). For list server commands, send "help" in the body of a message to "[EMAIL PROTECTED]".
