On Wed, 2011-01-19 at 09:32 -0500, William L. Thomson Jr. wrote: > > 1. http://www.robotstxt.org/orig.html > 2. > http://en.wikipedia.org/wiki/Robots_exclusion_standard#About_the_standard >
By the way anyone notice that the document regarding robots.txt was authored back in 1994, and nada since... I am just not that confident on loosely defined stuff like that with no RFC or standard, and the document that covers the common known "standard" is ancient. Think about the web and crawlers back in 1994 as compared to 2011, any changes? :) -- William L. Thomson Jr. Systems Administrator Jacksonville Linux Users Group --------------------------------------------------------------------- Archive http://marc.info/?l=jaxlug-list&r=1&w=2 RSS Feed http://www.mail-archive.com/[email protected]/maillist.xml Unsubscribe [email protected]
