Hello,
I have gone through the document, http://www.research.compaq.com/SRC/mercator/papers/Java99/final.html and two other documents related to mercator project are also found helpful. My major problem is set timeout, and that is solved by using the API provided by Ronald Tschaldr's HTTPClient Version 0.3-3 . I think the implementation works well. I also had gone though the following documents for the Robot Exclusion Standards, http://www.kollar.com/robots.html http://www.w3.org/TR/REC-html40/appendix/notes.html#h-B.4.1 reference for the robots meta tag http://www.robotstxt.org/wc/meta-user.html Thanks for the help in providing the following documents. Cheers Mohan __________________________________________________ Do You Yahoo!? Get personalized email addresses from Yahoo! Mail - only $35 a year! http://personal.mail.yahoo.com/ -- This message was sent by the Internet robots and spiders discussion list ([EMAIL PROTECTED]). For list server commands, send "help" in the body of a message to "[EMAIL PROTECTED]".
