libcurl version: 7.52.1-5+deb9u6
My applications goal: Use pause/unpause in libcurl to ensure a slow client can
push back to a fast server so that it slows down while also keeping buffered
data in curl for the transfer to a minimum. I want to use the
culr_multi_interface so that my application can decide when to transfer data
and work with many transfers at a time.
Operations performed:
My application calls curl_multi_perform on a multi-handle that has a single
easy-handle ‘add’ed to it. The WRITEFUNCTION gets called to consume data being
transferred for a HTTPS GET request. The write function returns
CURL_WRITEFUNC_PAUSE to request pause. As data is consumed, in another thread,
internal data structures are updated and the curl processing thread calls
curl_easy_unpause to unpause the transfer on an easy handle that is already
added to the only multi handle being used by the thread. The curl processing
thread also calls curl_multi_perform immediately afterwards. Both
curl_easy_pause and curl_multi_perform transfer data.
Behavior observed:
Libcurl stops calling the WRITEFUNCTION after pause is requested. However,
libcurl keeps receiving data on the connection and buffers it. At some point,
when transferring 1GB objects, memory allocation fails and libcurl returns
CURLE_OUT_OF_MEMORY.
The above behavior is seen only when using HTTP2.
When using HTTP1.1 to talk to the server, the data is received as transfer is
resumed after pauses.
My guess is that the buffered data can get quite large and memory allocation
fails. It also defeats the attempt to control flow of data and resource
consumption at the client.
Questions:
1. The code in lib/transfer.c:Curl_readwrite does seem to avoid reading from
connection when KEEP_RECV_PAUSE is set. Is the intent of the pause
functionality to stop reading from socket when WRITING is paused?
2. Is pausing/un-pausing the right way for applications to control the flow of
data between libcurl and http server? Is there any other mechanism?
3. My understanding is that curl_multi_perform ultimately calls
lib/transfer.c:readwrite_data to read data off the connection from http server
and delivers to WRITEFUNCTION using Curl_client_write. When WRITEFUNC requests
pause, this state is set in the easy-handle but readwrite_data does not check
this state and continues to read data from socket but cannot deliver to the
WRITEFUNC. If this is indeed a bug, should readwrite_data check the
KEEP_RECV_PAUSE flag in the easy handle and stop reading?
4. Curl_easy_pause may call the writefunc hander synchronously. This goes
against the intent of using multi-interface where curl_multi_perform is used to
move all data. When using multi interface, is curl_easy_unpause the right way
to un-pause paused transfers?
5. Why does maxloops=100 in readwrite_data not prevent it from stopping at
reading 100 times from the connection. I sometimes see more than a 1000
invocations of debugfunction that indicate there were 1000 reads of data from
the connection. Does the 100 reads somehow transform to 1000+ invocation of
debug function? Note that the content I am reading is not compressed.
6. Does this have something to do with the way curl interfaces with nghttp2 for
http2?
------------
A change(lib/transfer.c:readwrite_data(…)) to stop reading from a connection
when the WRITEFUNCTION has requested pause is:
diff --git a/lib/transfer.c b/lib/transfer.c
index b73f94d9e..d9ef1e394 100644
--- a/lib/transfer.c
+++ b/lib/transfer.c
@@ -844,6 +844,11 @@ static CURLcode readwrite_data(struct Curl_easy *data,
k->keepon &= ~KEEP_RECV;
}
+ if(k->keepon & KEEP_RECV_PAUSE) {
+ /* this is a paused transfer */
+ break;
+ }
+
} while(data_pending(conn) && maxloops--);
if(maxloops <= 0) {
------------------------------------------
I understand this is against the limit of maxloops=100.
Are there other gotchas with this approach or is this against the principle of
what is being done in this code path?
-Dheeraj Sangamkar
-------------------------------------------------------------------
Unsubscribe: https://cool.haxx.se/list/listinfo/curl-library
Etiquette: https://curl.haxx.se/mail/etiquette.html
