On Wed, 2011-01-19 at 10:58 -0500, Mike Rathburn wrote: > You can speed up and control in more granular detail the Google stuff > by signing up an account with Google Webmaster Tools. >
Also with regard to the Crawler Access portion of the Google Webmasters tools. You are not provided with anything to speed up, much less control in a granular way crawler access. Its a lame tool for generating robots.txt files, IMHO. Here is an example I just generated, but via hand would have been much faster. User-agent: * Allow: / User-agent: Mediapartners-Google Disallow: / User-agent: Googlebot Disallow: / User-agent: Adsbot-Google Disallow: / User-agent: Googlebot-Image Disallow: / User-agent: Googlebot-Mobile Disallow: / However there is no mention if robots.txt files are parsed in order. The first entry might negate all others, and the rest be ignored. If it was like the a firewall, you would do allow after deny. But gain this is straight from Googles own tool in their Webmasters tool. Who really needs a robots.txt file generator. The one Google provides does nothing to help out wrt to the problems on the wiki. That has to be done that manually. What Googles tool does most anyone can do by hand just the same in less time. Now it was useful for the following Line 3: Crawl-delay: 15 Rule ignored by Googlebot -- William L. Thomson Jr. Obsidian-Studios, Inc. http://www.obsidian-studios.com --------------------------------------------------------------------- Archive http://marc.info/?l=jaxlug-list&r=1&w=2 RSS Feed http://www.mail-archive.com/[email protected]/maillist.xml Unsubscribe [email protected]
