brian 98/01/25 23:12:59
Modified: htdocs/manual/misc howto.html Log: tsk tsk, randy. Can't find this on covalent.net either. Revision Changes Path 1.4 +8 -1 apachen/htdocs/manual/misc/howto.html Index: howto.html =================================================================== RCS file: /export/home/cvs/apachen/htdocs/manual/misc/howto.html,v retrieving revision 1.3 retrieving revision 1.4 diff -u -r1.3 -r1.4 --- howto.html 1997/06/04 11:42:57 1.3 +++ howto.html 1998/01/26 07:12:58 1.4 @@ -130,7 +130,14 @@ <P><code>robots.txt</code> provides a means to request that robots limit their activities at the site, or more often than not, to leave the site alone.</P> -<P>When the first robots were developed, they had a bad reputation for sending hundreds/thousands of requests to each site, often resulting in the site being overloaded. Things have improved dramatically since then, thanks to <A HREF="http://info.webcrawler.com/mak/projects/robots/guidelines.html";> Guidelines for Robot Writers</A>, but even so, some robots may <A HREF="http://www.zyzzyva.com/robots/alert/";>exhibit unfriendly behavior</A> which the webmaster isn't willing to tolerate, and will want to stop.</P> +<P>When the first robots were developed, they had a bad reputation for +sending hundreds/thousands of requests to each site, often resulting +in the site being overloaded. Things have improved dramatically since +then, thanks to <A +HREF="http://info.webcrawler.com/mak/projects/robots/guidelines.html";> +Guidelines for Robot Writers</A>, but even so, some robots may exhibit +unfriendly behavior which the webmaster isn't willing to tolerate, and +will want to stop.</P> <P>Another reason some webmasters want to block access to robots, is to stop them indexing dynamic information. Many search engines will use the
