On Mon, 16 May 2011, Ian Lance Taylor wrote: > Richard Guenther <richard.guent...@gmail.com> writes: > > > On Fri, May 13, 2011 at 7:14 PM, Ian Lance Taylor <i...@google.com> wrote: > >> I noticed that buglist.cgi was taking quite a bit of CPU time. I looked > >> at some of the long running instances, and they were coming from > >> searchbots. I can't think of a good reason for this, so I have > >> committed this patch to the gcc.gnu.org robots.txt file to not let > >> searchbots search through lists of bugs. I plan to make a similar > >> change on the sourceware.org and cygwin.com sides. Please let me know > >> if this seems like a mistake. > >> > >> Does anybody have any experience with > >> http://code.google.com/p/bugzilla-sitemap/ ? That might be a slightly > >> better approach. > > > > Shouldn't we keep searchbots way from bugzilla completely? Searchbots > > can crawl the gcc-bugs mailinglist archives. > > I don't see anything wrong with crawling bugzilla, though, and the > resulting links should be better.
Indeed. I think the individual bugs, and the GCC-specific help texts (such as describekeywords.cgi and describecomponents.cgi), should be indexed. -- Joseph S. Myers jos...@codesourcery.com
