Hello,I'm fetching about 150 web servers in Belgium. My total bandwith used is around 2 Mbits. Today I had a big problem, a phone call from Belgian gouvernment saying i'm breaking down their web server. I'm crawling with unpolite parameters like (fetcher.server.delay = 0.5 and threads.per.host=15 and http.max.delay=1500).
To have a polite crawler, what are the best parameters with threads.per.host =1 ?
Thank you very much for your answer. Christophe Noel
