> [http://www.hastingsresearch.com/net/06-anti-thesaurus.shtml] is a proposal > for a meta-tag to tell search engines to ignore certain > words on a page when scoring relevancy. Among other things, it mentions > robots.txt as problematic:
This is a bit ill-thought-out. It's not the words so much as the phrases that are the problem. And the "problem phrases" will only come out after the page has been indexed, only on some search engines, and maybe not for many months after indexing has taken place. If you could predict the problem phrases in advance, you could also re-arrange the page so that they would not be problem phrases! On an agent theme, has anybody defined a null agent - i.e. a known agent name to describe an agent about which you know nothing. It would be useful to be able to ask a server "What would you give an agent you knew nothing about?" without having to make up an agent name. Something like NullBot or FooBot would be a good generic name. And, just to squeeze another one in, is anybody doing any work on defining agent capabilities in the HTTP request header? Alan Perkins, Chief Technology Officer e-Brand Management Limited http://www.ebrandmanagement.com -- This message was sent by the Internet robots and spiders discussion list ([EMAIL PROTECTED]). For list server commands, send "help" in the body of a message to "[EMAIL PROTECTED]".
