dexen deVries <dexen.devr...@gmail.com> writes: > I believe we need an ``Allow: /'' below the long list of `Disallows' in the > User-agent: Googlebot, User-agent: msnbot section. Otherwise, only the final > ``Disallow: /'' matches, and in effecet, every robot is cut off.
I rather like mine own robots.txt (http://www.icebubble.org/robots.txt). As members of the "alternative OS" crowd, you may appreciate its design. Incidentally, the wiki employs a very good anti-spam technology: edits to it can only be made over 9P. (The HTTP edit mechanism isn't enabled.) The wiki's lack of spidering also means it wouldn't be a very valuable target even IF spambots could edit it over HTTP. -- +---------------------------------------------------------------+ |E-Mail: smi...@zenzebra.mv.com PGP key ID: BC549F8B| |Fingerprint: 9329 DB4A 30F5 6EDA D2BA 3489 DAB7 555A BC54 9F8B| +---------------------------------------------------------------+
