On Wed, 2011-01-19 at 11:17 -0500, William L. Thomson Jr. wrote: > On Wed, 2011-01-19 at 10:58 -0500, Mike Rathburn wrote: > > You can speed up and control in more granular detail the Google stuff > > by signing up an account with Google Webmaster Tools. > > > > Also with regard to the Crawler Access portion of the Google Webmasters > tools. You are not provided with anything to speed up
Correction in another area you can control the crawl rate. However if Google respected Crawl-delay: or others directives in robots.txt. One would not have to create an account in Google Webmasters, and go set a custom crawl rate specific to just Google. Again identifying problems with the robots.txt approach, as that alone is not enough. -- William L. Thomson Jr. Obsidian-Studios, Inc. http://www.obsidian-studios.com --------------------------------------------------------------------- Archive http://marc.info/?l=jaxlug-list&r=1&w=2 RSS Feed http://www.mail-archive.com/[email protected]/maillist.xml Unsubscribe [email protected]
