[EMAIL PROTECTED] wrote:
Another 59115 hits from the MSN bot contain the name "luggle.com"
in the URL.
This usually happens if people put broken links in their pages on their
site. It can be due to just plain getting the URLs wrong or the more
usual making some broken relative link instead of an intended absolute
link, or half assed link obfuscation that ends up wrong. You are
unlikely to be able to fix the rest of the Internet so it is best to
expect broken links to the SQLite site.
So the wiki server ignores it all and serves up the wiki home
page.
BTW that is absolutely the wrong behaviour to have. You should either
return a 404 as the page really doesn't exist with helpful content as
part of the 404 page, or if you really want to show the home page then
redirect to it using status 301 Moved Permanently.
I consider the behavior of the MSN bot to be abusive. I'm
sorely tempted to ban the MSN bot from the entire sqlite.org
website.
Robots.txt should take care of it. You could add the luggle.com prefix
to get any following to not happen. The MSN bot does look for
robots.txt on my site.
> If SQLite ceased to be indexed by MSN, would that
seriously inconvenience any users?
I don't think people use MSN much, or if they do I don't think they find
anything :-) On my own site this month where most visitors are coming
for information about a particular cell phone, I have had 1203 referrals
from google, 84 from yahoo and 13 from MSN. Looking at the logs of MSN
referrals, my site is completely inappropriate for about half of them.
(ie topic not covered, words not even on my site)
Roger
-----------------------------------------------------------------------------
To unsubscribe, send email to [EMAIL PROTECTED]
-----------------------------------------------------------------------------
