I have a simple C program where I use the CURL multi interface to download a
bunch of files. My code looks like this:
#include <stdlib.h>
#include <stdio.h>
#include <curl/curl.h>
int main(void) {
int code = 0;
size_t index = 0;
int running = 1;
CURLMsg* msg = NULL;
int left = 0;
CURL* curl = NULL;
CURLM* curl_multi = NULL;
curl = curl_easy_init();
curl_multi = curl_multi_init();
curl_easy_setopt(curl, CURLOPT_URL, "https://example.com/";);
curl_easy_setopt(curl, CURLOPT_HTTP_VERSION, CURL_HTTP_VERSION_1_1);
curl_multi_setopt(curl_multi, CURLMOPT_MAX_HOST_CONNECTIONS, 5L);
for (index = 0; index < 15; index++) {
CURL* handle = curl_easy_duphandle(curl);
curl_multi_add_handle(curl_multi, handle);
}
while (running) {
curl_multi_perform(curl_multi, &running);
if (running) {
curl_multi_poll(curl_multi, NULL, 0, 1000, NULL);
}
while ((msg = curl_multi_info_read(curl_multi, &left))) {
if (msg->msg != CURLMSG_DONE) {
continue;
}
curl_multi_remove_handle(curl_multi, msg->easy_handle);
printf("result: %i\n", (int) msg->data.result);
}
}
return 0;
}
I firstly create an easy handle and then duplicate that handle with
curl_easy_duphandle() for usage with the multi interface. For each handle, I
set the CURLOPT_URL to the URL of the file I want to download.
I set a maximum number of host connections here (CURLMOPT_MAX_HOST_CONNECTIONS)
because I don't want to cause overhead on the server.
This code does what I intended: download up to 5 files simultaneously.
However, things get a little weird if I set a CURLOPT_DOH_URL on the main easy
handle:
...
curl_easy_setopt(curl, CURLOPT_DOH_URL, "https://dns.google/dns-query";);
...
This causes the multi interface to hang indefinitely. I tried setting
CURLOPT_VERBOSE to discover what's going on, and here are the logs:
* Found bundle for host: 0xb4000073436429d0 [serially]
* Found bundle for host: 0xb4000073436429d0 [serially]
* Found bundle for host: 0xb4000073436429d0 [serially]
* Found bundle for host: 0xb4000073436429d0 [serially]
* Found bundle for host: 0xb4000073436429d0 [serially]
* No more connections allowed to host: 5
* No connections available.
* Found bundle for host: 0xb4000073436429d0 [serially]
* No more connections allowed to host: 5
* No connections available.
* Found bundle for host: 0xb4000073436429d0 [serially]
* No more connections allowed to host: 5
* No connections available.
* Found bundle for host: 0xb4000073436429d0 [serially]
* No more connections allowed to host: 5
* No connections available.
* Found bundle for host: 0xb4000073436429d0 [serially]
* No more connections allowed to host: 5
* No connections available.
* Found bundle for host: 0xb4000073436429d0 [serially]
* No more connections allowed to host: 5
* No connections available.
* Found bundle for host: 0xb4000073436429d0 [serially]
* No more connections allowed to host: 5
* No connections available.
* Found bundle for host: 0xb4000073436429d0 [serially]
* No more connections allowed to host: 5
* No connections available.
* Found bundle for host: 0xb4000073436429d0 [serially]
* No more connections allowed to host: 5
* No connections available.
* Found bundle for host: 0xb4000073436429d0 [serially]
* No more connections allowed to host: 5
* No connections available.
* Found bundle for host: 0xb4000073436429d0 [serially]
* No more connections allowed to host: 5
* No connections available.
* Found bundle for host: 0xb4000073436429d0 [serially]
* No more connections allowed to host: 5
* No connections available.
* Found bundle for host: 0xb4000073436429d0 [serially]
* No more connections allowed to host: 5
* No connections available.
* Found bundle for host: 0xb4000073436429d0 [serially]
* No more connections allowed to host: 5
* No connections available.
* Found bundle for host: 0xb4000073436429d0 [serially]
* No more connections allowed to host: 5
* No connections available.
* Found bundle for host: 0xb4000073436429d0 [serially]
* No more connections allowed to host: 5
* No connections available.
It then stays like this. No requests are performed.
After some tests, I found that setting CURLMOPT_MAX_HOST_CONNECTIONS to
anything higher than 10 resolves this issue:
```
curl_multi_setopt(curl_multi, CURLMOPT_MAX_HOST_CONNECTIONS, 10L);
```
However, this setting causes the multi interface to make 10 requests
simultaneously instead of just 5, which is undesired for my specific use case.
Here is the output of curl_version():
libcurl/8.7.1 OpenSSL/1.0.2k-fips zlib/1.2.7 brotli/1.0.9 libidn2/2.3.7
libpsl/0.7.0 (+libicu/50.1.2) libssh2/1.10.0 nghttp2/1.33.0 OpenLDAP/2.4.44
I'm on CentOS 7.
--
Unsubscribe: https://lists.haxx.se/mailman/listinfo/curl-library
Etiquette: https://curl.se/mail/etiquette.html
