On 04/26/11 12:03, Ethan Grammatikidis wrote: > > On 24 Apr 2011, at 9:16 am, hiro wrote: > >> In http://plan9.bell-labs.com/robots.txt you will find: >> >> User-agent: * >> Disallow: / > > *facepalm* I wondered if this was the case; didn't think to check. > Anyone have any idea why this is there?
Very simple, since the webmaster have already allowed some bots and disallowed everyone else ;) You need to read/analyze the whole robots.txt indeed. -- Balwinder S "bdheeman" Dheeman (http://werc.homelinux.net/contact/)
