Weird behavior when using DoH with the multi interface
I have a simple C program where I use the CURL multi interface to download a
bunch of files. My code looks like this:
#include
#include
#include
int main(void) {
int code = 0;
size_t index = 0;
int running = 1;
CURLMsg* msg = NULL;
int left = 0;
CURL* curl = NULL;
CURLM* curl_multi = NULL;
curl = curl_easy_init();
curl_multi = curl_multi_init();
curl_easy_setopt(curl, CURLOPT_URL, "https://example.com/";);
curl_easy_setopt(curl, CURLOPT_HTTP_VERSION, CURL_HTTP_VERSION_1_1);
curl_multi_setopt(curl_multi, CURLMOPT_MAX_HOST_CONNECTIONS, 5L);
for (index = 0; index < 15; index++) {
CURL* handle = curl_easy_duphandle(curl);
curl_multi_add_handle(curl_multi, handle);
}
while (running) {
curl_multi_perform(curl_multi, &running);
if (running) {
curl_multi_poll(curl_multi, NULL, 0, 1000, NULL);
}
while ((msg = curl_multi_info_read(curl_multi, &left))) {
if (msg->msg != CURLMSG_DONE) {
continue;
}
curl_multi_remove_handle(curl_multi, msg->easy_handle);
printf("result: %i\n", (int) msg->data.result);
}
}
return 0;
}
I firstly create an easy handle and then duplicate that handle with
curl_easy_duphandle() for usage with the multi interface. For each handle, I
set the CURLOPT_URL to the URL of the file I want to download.
I set a maximum number of host connections here (CURLMOPT_MAX_HOST_CONNECTIONS)
because I don't want to cause overhead on the server.
This code does what I intended: download up to 5 files simultaneously.
However, things get a little weird if I set a CURLOPT_DOH_URL on the main easy
handle:
...
curl_easy_setopt(curl, CURLOPT_DOH_URL, "https://dns.google/dns-query";);
...
This causes the multi interface to hang indefinitely. I tried setting
CURLOPT_VERBOSE to discover what's going on, and here are the logs:
* Found bundle for host: 0xb473436429d0 [serially]
* Found bundle for host: 0xb473436429d0 [serially]
* Found bundle for host: 0xb473436429d0 [serially]
* Found bundle for host: 0xb473436429d0 [serially]
* Found bundle for host: 0xb473436429d0 [serially]
* No more connections allowed to host: 5
* No connections available.
* Found bundle for host: 0xb473436429d0 [serially]
* No more connections allowed to host: 5
* No connections available.
* Found bundle for host: 0xb473436429d0 [serially]
* No more connections allowed to host: 5
* No connections available.
* Found bundle for host: 0xb473436429d0 [serially]
* No more connections allowed to host: 5
* No connections available.
* Found bundle for host: 0xb473436429d0 [serially]
* No more connections allowed to host: 5
* No connections available.
* Found bundle for host: 0xb473436429d0 [serially]
* No more connections allowed to host: 5
* No connections available.
* Found bundle for host: 0xb473436429d0 [serially]
* No more connections allowed to host: 5
* No connections available.
* Found bundle for host: 0xb473436429d0 [serially]
* No more connections allowed to host: 5
* No connections available.
* Found bundle for host: 0xb473436429d0 [serially]
* No more connections allowed to host: 5
* No connections available.
* Found bundle for host: 0xb473436429d0 [serially]
* No more connections allowed to host: 5
* No connections available.
* Found bundle for host: 0xb473436429d0 [serially]
* No more connections allowed to host: 5
* No connections available.
* Found bundle for host: 0xb473436429d0 [serially]
* No more connections allowed to host: 5
* No connections available.
* Found bundle for host: 0xb473436429d0 [serially]
* No more connections allowed to host: 5
* No connections available.
* Found bundle for host: 0xb473436429d0 [serially]
* No more connections allowed to host: 5
* No connections available.
* Found bundle for host: 0xb473436429d0 [serially]
* No more connections allowed to host: 5
* No connections available.
* Found bundle for host: 0xb473436429d0 [serially]
* No more connections allowed to host: 5
* No connections available.
It then stays like this. No requests are performed.
After some tests, I found that setting CURLMOPT_MAX_HOST_CONNECTIONS to
anything higher than 10 resolves this issue:
```
curl_multi_setopt(curl_multi, CURLMOPT_MAX_HOST_CONNECTIONS, 10L);
```
However, this setting causes the multi interface to make 10 requests
simultaneously instead of just 5, which is undesired for my specific use case.
Here is the output of curl_version():
libcurl/8.7.1 OpenSSL/1.0.2k-fips zlib/1.2.7 brotli/1.0.9 libidn2/2.3.7
libpsl/0.7.0 (+libicu/50.1.2) libssh2/1.10.0 nghttp2/1.33.0 OpenLDAP/2.4.44
I'm on CentOS 7.
--
Unsubscribe: https://lists.haxx.se/mailman/listinfo/curl-library
Etiquette: https://curl.se/mail/etiquette.html
Re: Weird behavior when using DoH with the multi interface
On Tue, 4 Jun 2024, Daniel Stenberg via curl-library wrote: > Interesting case. I think we should make DoH requests exempt from that > check/restriciton, as otherwise you can end up in this kind of catch-22 > situation. kartatz, how about a patch like this? Thank you for the patch! I checked out the latest master from GitHub and did a minimal curl build with CMake (-DHTTP_ONLY=ON) using OpenSSL as SSL backend. Now the multi interface can resolve using DoH, but the program crashes as soon as it attempts to connect to the resolved address: * Found bundle for host: 0x119dc60 [serially] * Found bundle for host: 0x119dc60 [serially] * Found bundle for host: 0x119dc60 [serially] * Found bundle for host: 0x119dc60 [serially] * Found bundle for host: 0x119dc60 [serially] * No more connections allowed to host: 5 * No connections available. * Found bundle for host: 0x119dc60 [serially] * No more connections allowed to host: 5 * No connections available. * Found bundle for host: 0x119dc60 [serially] * No more connections allowed to host: 5 * No connections available. * Found bundle for host: 0x119dc60 [serially] * No more connections allowed to host: 5 * No connections available. * Found bundle for host: 0x119dc60 [serially] * No more connections allowed to host: 5 * No connections available. * Found bundle for host: 0x119dc60 [serially] * No more connections allowed to host: 5 * No connections available. * Found bundle for host: 0x119dc60 [serially] * No more connections allowed to host: 5 * No connections available. * Found bundle for host: 0x119dc60 [serially] * No more connections allowed to host: 5 * No connections available. * Found bundle for host: 0x119dc60 [serially] * No more connections allowed to host: 5 * No connections available. * Found bundle for host: 0x119dc60 [serially] * No more connections allowed to host: 5 * No connections available. * Found bundle for host: 0x119dc60 [serially] * No more connections allowed to host: 5 * No connections available. * Found bundle for host: 0x119dc60 [serially] * No more connections allowed to host: 5 * No connections available. * Found bundle for host: 0x119dc60 [serially] * No more connections allowed to host: 5 * No connections available. * Found bundle for host: 0x119dc60 [serially] * No more connections allowed to host: 5 * No connections available. * Found bundle for host: 0x119dc60 [serially] * No more connections allowed to host: 5 * No connections available. * Found bundle for host: 0x119dc60 [serially] * No more connections allowed to host: 5 * No connections available. * Found bundle for host: 0x119dc60 [serially] * No more connections allowed to host: 5 * No connections available. * Found bundle for host: 0x119dc60 [serially] * No more connections allowed to host: 5 * No connections available. * Found bundle for host: 0x119dc60 [serially] * No more connections allowed to host: 5 * No connections available. * Found bundle for host: 0x119dc60 [serially] * No more connections allowed to host: 5 * No connections available. * Host example.com:443 was resolved. * IPv6: 2606:2800:21f:cb07:6820:80da:af6b:8b2c * IPv4: 93.184.215.14 * Trying 93.184.215.14:443... * Hostname 'example.com' was found in DNS cache [1] 15690 segmentation fault ./main I attached the gdb logs. Starting program: /home/kartz/main Missing separate debuginfos, use: debuginfo-install glibc-2.17-326.el7_9.x86_64 [Thread debugging using libthread_db enabled] Using host libthread_db library "/lib64/libthread_db.so.1". Program received signal SIGSEGV, Segmentation fault. [34m0x77f5bb19[m in [33mstart_connect[m ([36mcf[m=0x457b50, [m [m[36mdata[m=0x42a9d0, [36mremotehost[m=0x0)[m [m at [32m/home/kartz/curl/lib/connect.c[m:780 780 ai_family0 [31m=[m remotehost[31m->[maddr[31m?[m Missing separate debuginfos, use: debuginfo-install keyutils-libs-1.5.8-3.el7.x86_64 krb5-libs-1.15.1-55.el7_9.x86_64 libcom_err-1.42.9-19.el7.x86_64 libselinux-2.5-15.el7.x86_64 openssl-libs-1.0.2k-26.el7_9.x86_64 pcre-8.32-17.el7.x86_64 zlib-1.2.7-21.el7_9.x86_64 #0 [34m0x77f5bb19[m in [33mstart_connect[m ([36mcf[m=0x457b50, [m [m[36mdata[m=0x42a9d0, [36mremotehost[m=0x0)[m [m at [32m/home/kartz/curl/lib/connect.c[m:780 [36mctx[m = [34m0x453320[m [36mconn[m = [34m0x43f770[m [36mresult[m = CURLE_COULDNT_CONNECT [36mai_family0[m = 0 [36mai_family1[m = 0 [36mtimeout_ms[m = 294872 [36maddr0[m = [34m0x3004012f0[m [36maddr1[m = [34m0x77f5c79b[m <[33mcf_happy_eyeballs_create[m+149> #1 [34m0x77f5c000[m in [33mcf_he_connect[m ([36mcf[m=0x457b50, [m [m[36mdata[m=0x42a9d0, [36mblocking[m=false, [36mdone[m=0x7fffdb8f)[m [m at [32m/home/kartz/curl/lib/connect.c[m:898 [36mctx[m = [34m0x453320[m [36mresult[m = CURLE_OK #2 [34m0x77f57f79[m in [33mCurl_conn_cf_connect[
Re: Weird behavior when using DoH with the multi interface
I don't think my patch causes that. The PR ran 159 successful CI jobs. Yes, you are right. It appears that something from my previous builds was messing with things. I deleted the build directory and did a clean build. This time, everything worked as expected. I apologize for not checking properly. -- Unsubscribe: https://lists.haxx.se/mailman/listinfo/curl-library Etiquette: https://curl.se/mail/etiquette.html
