On 05/04/2011 09:09 PM, Ethan Grammatikidis wrote:
On 4 May 2011, at 11:40 am, Balwinder S Dheeman wrote:
On 04/26/11 12:03, Ethan Grammatikidis wrote:
On 24 Apr 2011, at 9:16 am, hiro wrote:
In http://plan9.bell-labs.com/robots.txt you will find:
User-agent: *
Disallow: /
*facepalm* I wondered if this was the case; didn't think to check.
Anyone have any idea why this is there?
Very simple, since the webmaster have already allowed some bots and
disallowed everyone else ;)
You need to read/analyze the whole robots.txt indeed.
Now I've read it I can't understand why Google can't find anything under
/wiki. Even if it did, that robots.txt isn't all that pleasant, blindly
disallowing everyone who isn't google or msn, more or less. O.o
May be either the robots.txt is incorrect or Google and MSN/Bing are
interpreting it in wrong manner.
I maintain anu.homelinux.net and werc.homelinux.net sites, mine
robots.txt files are quite simple and I never ever facing these servers
being chocked by any such bot.
--
Balwinder S "bdheeman" Dheeman
(http://werc.homelinux.net/contact/)
