On Fri, 23 Oct 2020, John Wiseman via curl-library wrote:
It's the number of max concurrent dns queries. Without such limit, my crawler will easyily blow up the dns server of my home network with almost 1 thousand threads doing dns query on just several names.
Ouch. Yes, we clearly need a way to ratelimit that. It could probably be made to work similarly to how we limit the total number of concurrent connections and connections to the same host name. Maybe set the maximum number of name resolves per second to allow or something.
I've set connection limits on multi handle, but seems it won't limit these dns queries.
It should do it somewhat, and only indirectly. If it reaches the maximum number of connections, in total or per host name, it will not resolve the new host name until that connection can be allowed to get setup. But if you have short-lived connections or HTTP requests doing redirects etc, that might not be a very affective rate limit.
-- / daniel.haxx.se | Commercial curl support up to 24x7 is available! | Private help, bug fixes, support, ports, new features | https://www.wolfssl.com/contact/ ------------------------------------------------------------------- Unsubscribe: https://cool.haxx.se/list/listinfo/curl-library Etiquette: https://curl.haxx.se/mail/etiquette.html
