--- Ken Schweigert <[EMAIL PROTECTED]> wrote: > On Fri, May 21, 2004 at 03:55:57PM +0200, Robert > Brenstein wrote: > > > > > >Not a virus, just microsoft releasing a beta > version of their spider on > > >the web. We had it bring a site to a crawl. > It's hitting a java-based > > >app that created a new java session every time > msnbot hit it. Fortunately > > >the resources freed themselves quickly, but > having a hundred instances > > >running was not good. I added msnbot to the > robots.txt file and it > > >stopped spidering the site within a few minutes. > Once the spider is out > > >of beta I'll remove them from robots.txt. > > > > > > > How did you identify it in the robots.txt file ? > > User-agent: msnbot > Disallow: / > > > -ken > > +------------------------------------------------------------------------ > | TO UNSUBSCRIBE from this list: > | > http://lists.isite.net/listgate/analog-help/unsubscribe.html > | > | Digest version: > http://lists.isite.net/listgate/analog-help-digest/ > | Usenet version: > news://news.gmane.org/gmane.comp.web.analog.general > | List archives: > http://www.analog.cx/docs/mailing.html#listarchives > +------------------------------------------------------------------------
I was looking at the file here, http://wadsack-allen.com/products/robot-list.html and noticed the msnbot wasn't listed. Am I missing something, or is the file? Thanks. __________________________________ Do you Yahoo!? New and Improved Yahoo! Mail - 100MB free storage! http://promotions.yahoo.com/new_mail +------------------------------------------------------------------------ | TO UNSUBSCRIBE from this list: | http://lists.isite.net/listgate/analog-help/unsubscribe.html | | Digest version: http://lists.isite.net/listgate/analog-help-digest/ | Usenet version: news://news.gmane.org/gmane.comp.web.analog.general | List archives: http://www.analog.cx/docs/mailing.html#listarchives +------------------------------------------------------------------------
