Just create an additional backend and assign the bots to it. You can set queues and max connections there as needed.
Also an additional tip might be to adjust the robots.txt file as some bots can be slowed down. http://www.google.com/support/webmasters/bin/answer.py?answer=48620 Check if the bots that are crawling have some real use for you, otherwise just adjust your robots.txt or block them. Some stuff for basic mysql + mediawiki might be to check if the mysql querycache is working. Karsten Am Sonntag, 15. November 2009 schrieben Sie: > Hi there, > > I was wondering if HAProxy helps in the following situation: > > - We have a wiki site which is quite slow > - Regular users don't have many problems > - We also get crawled by a search bot, which creates many concurrent > connections, more than the hardware can handle - Therefore, service is > degraded and users usually have their browsers time out on them > > Given that we can't make the wiki faster, I was thinking that we could > solve this by having a per-source-IP queue, which made sure that a given > source IP cannot have more than e.g. 3 requests active at the same time. > Requests beyond that would get queued. > > Is this possible? > > Thanks, > > Wout. > -- Mit freundlichen Grüßen Karsten Elfenbein Entwicklung und Systemadministration erento - Der Online-Marktplatz für Mietartikel. erento GmbH Friedenstrasse 91 D-10249 Berlin Tel: +49 (30) 2000 42064 Fax: +49 (30) 2000 8499 eMail: karsten.elfenb...@erento.com - - - - - - - - - - - - - - - - - - - - - - - - - - Hotline: 01805 - 373 686 (14 ct/min.) Firmensitz der erento GmbH ist Berlin Geschäftsführer: Chris Möller & Oliver Weyergraf Handelsregister Berlin Charlottenburg, HRB 101206B - - - - - - - - - - - - - - - - - - - - - - - - - - http://www.erento.com - alles online mieten.
