On Wed, Feb 22, 2023 at 05:01:00PM +0100, Daniel Stenberg wrote: > On Tue, 21 Feb 2023, Richard W.M. Jones via curl-library wrote: > > >Firstly, I don't understand if the multi interface would actually > >help us here. Because nbdkit gives us lots of threads and expects > >an NBD request to be processed synchronously on that thread, using > >the easy interface is a natural .. easy(!) .. fit. > > Doing multiple transfers using the multi interface instead of > separate threads using the easy interface has some pros and some > cons. > > Pros: > > - shared pools and caches > - you can multiplex transfers with h2/h3 > > Cons: > > - multiple threads running on multiple cores can perform faster in case you > are CPU bound > > >A second thing I'm unclear about with multi is whether the > >individual easy handles which are added are related in any way -- > >eg. if they all share the same TCP connection to the web server? > > The *can* share the same conncetion with HTTP/2 and HTTP/3 multiplexing, yes.
Does this sharing happen without us doing anything, or do we need to use the curl share interface? I only discovered share after sending the original message yesterday. I then patched nbdkit to use it: https://listman.redhat.com/archives/libguestfs/2023-February/030839.html but it doesn't seem to make any noticable difference in my performance tests. At the same time I thought I'd actually set up an HTTP/2 webserver (just a local Apache 2.4) so I could see how curl works against these. But I get very strange problems (with or without the above patch) where the server appears to report 403 errors at random, and curl itself can deadlock. I'm sorry for not being very specific on this as I'm still investigating, but should we expect curl + HTTP/2 to work well, or is it still experimental? If it's expected to work then I'll try to investigate further, use upstream curl, and see if I can make a proper reproducer. > >Reading the page makes me think this is not the case, the multi > >interface is just a way to group easy handles for the purposes of > >using a select/poll or event-driven API, and apart from that there > >is no relationship. > > They also share connection pool, DNS cache, CA cert cache etc. > > >The third and main concern is whether we are using curl most > >efficiently. In particular, whether we are using HTTP/2 (and in > >future HTTP/3) as efficiently as we could be (eg. exploiting > >multiplexing). > > For multiplexing you *must* use the multi interface and add multiple > transfers to it. The share interface (instead of multi) _won't_ give us HTTP/2 multiplexing? > >I notice that HTTP/1.1-style pipelining was removed from curl, and > >I suppose HTTP/2 multiplexing is meant to replace this. However > >since we are using the easy interface and doing everything > >synchronously, it's my understanding that we are not exploiting > >multiplexing, unless curl itself does something clever internally. > > It can't do anything clever when for synchronous serial transfers. > Then you can't do multiplexing. If we need to use the multi interface then I can try coding that up later. Thanks, Rich. -- Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones Read my programming and virtualization blog: http://rwmj.wordpress.com virt-builder quickly builds VMs from scratch http://libguestfs.org/virt-builder.1.html -- Unsubscribe: https://lists.haxx.se/mailman/listinfo/curl-library Etiquette: https://curl.se/mail/etiquette.html
