On Mon, 2011-01-17 at 09:05 -0500, Chad Bailey wrote: > Google's crawler obey's /robots.txt, that's where i'd start.
Yes and I was looking into that, but its not so easy or straight forward when dealing with a wiki. For example http://en.wikipedia.org/wiki/MediaWiki:Robots.txt http://meta.wikimedia.org/wiki/MediaWiki:Robots.txt http://commons.wikimedia.org/wiki/MediaWiki:Robots.txt Really not sure where to even begin. Oh and Google is still hitting the wiki. Its been going on for days now..... -- William L. Thomson Jr. Obsidian-Studios, Inc. http://www.obsidian-studios.com --------------------------------------------------------------------- Archive http://marc.info/?l=jaxlug-list&r=1&w=2 RSS Feed http://www.mail-archive.com/[email protected]/maillist.xml Unsubscribe [email protected]
