I think you should disalow MSN bot... nobody use that... if there is top in google when you search "Embedded SQL database", what else could sqlite needs?... MSN is not used (at least here in south America and the rest of the world)
Another question... your robots.txt is this.... User-agent: * Disallow: /cvstrac/attach_add Disallow: /cvstrac/attach_get Disallow: /cvstrac/chngedit Disallow: /cvstrac/chngview Disallow: /cvstrac/dir Disallow: /cvstrac/filediff Disallow: /cvstrac/getfile Disallow: /cvstrac/msedit Disallow: /cvstrac/msnew Disallow: /cvstrac/rlog Disallow: /cvstrac/rptedit Disallow: /cvstrac/rptnew Disallow: /cvstrac/rptsql Disallow: /cvstrac/timeline Disallow: /cvstrac/tktedit Disallow: /cvstrac/tktview Disallow: /cvstrac/wikiedit Disallow: /cvstrac/honeypot Disallow: /cvstrac/wiki/attach_get Disallow: /contrib/download wont be better if you change for this???: User-agent: * Disallow: /cvstrac/attach_add* Disallow: /cvstrac/attach_get* Disallow: /cvstrac/chngedit* Disallow: /cvstrac/chngview* Disallow: /cvstrac/dir* Disallow: /cvstrac/filediff* Disallow: /cvstrac/getfile* Disallow: /cvstrac/msedit* Disallow: /cvstrac/msnew* Disallow: /cvstrac/rlog* Disallow: /cvstrac/rptedit* Disallow: /cvstrac/rptnew* Disallow: /cvstrac/rptsql* Disallow: /cvstrac/timeline* Disallow: /cvstrac/tktedit* Disallow: /cvstrac/tktview* Disallow: /cvstrac/wikiedit* Disallow: /cvstrac/honeypot* Disallow: /cvstrac/wiki/attach_get* Disallow: /contrib/download* Is just a question On 16/03/07, Roger Binns <[EMAIL PROTECTED]> wrote:
[EMAIL PROTECTED] wrote: > Another 59115 hits from the MSN bot contain the name "luggle.com" > in the URL. This usually happens if people put broken links in their pages on their site. It can be due to just plain getting the URLs wrong or the more usual making some broken relative link instead of an intended absolute link, or half assed link obfuscation that ends up wrong. You are unlikely to be able to fix the rest of the Internet so it is best to expect broken links to the SQLite site. > So the wiki server ignores it all and serves up the wiki home > page. BTW that is absolutely the wrong behaviour to have. You should either return a 404 as the page really doesn't exist with helpful content as part of the 404 page, or if you really want to show the home page then redirect to it using status 301 Moved Permanently. > I consider the behavior of the MSN bot to be abusive. I'm > sorely tempted to ban the MSN bot from the entire sqlite.org > website. Robots.txt should take care of it. You could add the luggle.com prefix to get any following to not happen. The MSN bot does look for robots.txt on my site. > If SQLite ceased to be indexed by MSN, would that > seriously inconvenience any users? I don't think people use MSN much, or if they do I don't think they find anything :-) On my own site this month where most visitors are coming for information about a particular cell phone, I have had 1203 referrals from google, 84 from yahoo and 13 from MSN. Looking at the logs of MSN referrals, my site is completely inappropriate for about half of them. (ie topic not covered, words not even on my site) Roger ----------------------------------------------------------------------------- To unsubscribe, send email to [EMAIL PROTECTED] -----------------------------------------------------------------------------
-- Cesar Rodas http://www.cesarodas.com/ Mobile Phone: 595 961 974165 Phone: 595 21 645590 [EMAIL PROTECTED] [EMAIL PROTECTED]
