Hi Daniel, Thank you for your response. I have done some additional debugging.
> On Tue, 16 Dec 2014, Alex Lyakas wrote: > >> I always specify one "extra_fd", which most of the time is not readable, >> and a timeout of 30,000 msec. >> >> What I see is that curl_multi_wait appears to always change this value to 0 >> ms, and as a result it's poll() call returns immediately. This results in a >> busy loop in the calling thread. Is this something expected? > > No. That sounds like a possible bug. > >> As an experiment, I tried to specify timeout=-1 to the curl_multi_wait >> API. In this case, the timeout value is not overridden, and everything >> works as expected. However, I am bit worried to use timeout=-1, > > You should be. It is not documented to work. Maybe it should be, but you're > relying on undocumented behaviors right now. > >> because as I learned from email thread "curl_multi_wait returns immediately >> when using bandwidth limit" in >> http://curl.haxx.se/mail/lib-2014-05/0224.html, in some cases, the code may >> decide that the state of the SessionHandle is not appropriate for returning >> any socket. In that case, it may happen that the only socket that libcurl >> will wait on, will be my extra socket. > > Not only during that situation, so yes it can certainly happen. > >> Am I missing something? What the caller is expected to do if curl_multi_wait >> returns immediately? > > I've been thinking about either adding a curl_multi_wait() replacement, or an > additional helper function to provide more information to the application in > the case when 'numfds' is zero on return. Did it then return due to timeout or > due to there being no file desciptors to wait for? Currently you basically > need to time the function to figure that out and I find that a bit disturbing. What happens is this: curl_multi_wait() calls multi_timeout(), which sets its output parameter to 0. As a result, curl_multi_wait does: /* If the internally desired timeout is actually shorter than requested from the outside, then use the shorter time! But only if the internal timer is actually larger than -1! */ (void)multi_timeout(multi, &timeout_internal); if((timeout_internal >= 0) && (timeout_internal < (long)timeout_ms)) timeout_ms = (int)timeout_internal; So timeout_ms ends up being 0. Next, we call Curl_poll() with timeout_ms=0. This ends up in Linux system call poll(), which returns immediately due to zero timeout. This happens again and again, and leads to busy loop. As for the descriptors: even if curl_multi_wait() would not have found any curl descriptors to wait for, I always pass an additional descriptor, so we always have to wait for at least one descriptor. But in my tests, curl_multi_wait() always finds some curl descriptor(s) to wait for as well. Debugging further, I found that curl_multi_wait() calls Curl_splaycomparekeys() and it always returns -1. As a result, multi_timeout does: *timeout_ms = 0; I added prints to see why Curl_splaycomparekeys() returns -1: #define Curl_splaycomparekeys(i,j) ( ((i.tv_sec) < (j.tv_sec)) ? -1 : \ ( ((i.tv_sec) > (j.tv_sec)) ? 1 : \ ( ((i.tv_usec) < (j.tv_usec)) ? -1 : \ ( ((i.tv_usec) > (j.tv_usec)) ? 1 : 0 )))) What happens is that i.tv_sec==j.tv_sec, but i.tv_usec<j.tv_usec. Please see[1] for values that I got in my test. This was a very simple test, performing 3 small PUTs onto AWS S3 in parallel. > > If there are no file descriptors to wait for, the application better either > add file descriptors or wait a little while before it loops back and waits > again as otherwise it'll just busy-loop until a descriptor shows up. > >> I see that in curl_easy_perform, there is some code which sleeps if that >> happens (depending on the number of times that it happened). Is this the >> recommended approach? > > The curl_multi_fdset() has some language for this and it says: "How long to > wait? We suggest 100 milliseconds at least, but you may want to test it out in > your own particular conditions to find a suitable value. " The behavior that would be ideal for me (after fixing the timeout=0 issue): # if curl_multi_wait() finds at least one curl descriptor to wait for, wait for all the descriptors (including the extra_fds) for the proper timeout. # otherwise: wait for extra_fd descriptors for a very short timeout, like 100ms and reutrn. This way, user has a chance to poll every 100ms in case no curl descriptors to wait are available. But I realize this behavior is very custom to my use-case. Thanks, Alex. [1] The values in brackes are: tv.sec:tv.usec : multi_timeout: return 0, multi->timetree->key=[485:670697] now=[485:690777] : multi_timeout: return 0, multi->timetree->key=[485:890191] now=[485:907776] : multi_timeout: return 0, multi->timetree->key=[486:429936] now=[486:430346] : multi_timeout: return 0, multi->timetree->key=[685:656845] now=[685:676601] : multi_timeout: return 0, multi->timetree->key=[685:656845] now=[685:676704] : multi_timeout: return 0, multi->timetree->key=[685:656845] now=[685:676721] : multi_timeout: return 0, multi->timetree->key=[685:656845] now=[685:676752] : multi_timeout: return 0, multi->timetree->key=[685:656845] now=[685:676766] : multi_timeout: return 0, multi->timetree->key=[685:656845] now=[685:676782] : multi_timeout: return 0, multi->timetree->key=[685:656845] now=[685:676796] : multi_timeout: return 0, multi->timetree->key=[685:656845] now=[685:676823] : multi_timeout: return 0, multi->timetree->key=[685:656845] now=[685:676837] : multi_timeout: return 0, multi->timetree->key=[685:656845] now=[685:676865] : multi_timeout: return 0, multi->timetree->key=[685:656845] now=[685:676878] : multi_timeout: return 0, multi->timetree->key=[685:656845] now=[685:676892] : multi_timeout: return 0, multi->timetree->key=[685:656845] now=[685:676906] : multi_timeout: return 0, multi->timetree->key=[685:656845] now=[685:676933] : multi_timeout: return 0, multi->timetree->key=[685:656845] now=[685:676947] : multi_timeout: return 0, multi->timetree->key=[685:656845] now=[685:676961] : multi_timeout: return 0, multi->timetree->key=[685:656845] now=[685:676975] : multi_timeout: return 0, multi->timetree->key=[685:656845] now=[685:677002] : multi_timeout: return 0, multi->timetree->key=[685:656845] now=[685:677016] : multi_timeout: return 0, multi->timetree->key=[685:656845] now=[685:677043] : multi_timeout: return 0, multi->timetree->key=[685:656845] now=[685:677057] : multi_timeout: return 0, multi->timetree->key=[685:656845] now=[685:677073] : multi_timeout: return 0, multi->timetree->key=[685:656845] now=[685:677087] : multi_timeout: return 0, multi->timetree->key=[685:656845] now=[685:677114] : multi_timeout: return 0, multi->timetree->key=[685:656845] now=[685:677128] : multi_timeout: return 0, multi->timetree->key=[685:656845] now=[685:677142] : multi_timeout: return 0, multi->timetree->key=[685:656845] now=[685:677156] : multi_timeout: return 0, multi->timetree->key=[685:656845] now=[685:677696] : multi_timeout: return 0, multi->timetree->key=[685:656845] now=[685:677711] : multi_timeout: return 0, multi->timetree->key=[685:656845] now=[685:679534] : multi_timeout: return 0, multi->timetree->key=[685:656845] now=[685:679548] : multi_timeout: return 0, multi->timetree->key=[685:656845] now=[685:679563] : multi_timeout: return 0, multi->timetree->key=[685:656845] now=[685:679577] : multi_timeout: return 0, multi->timetree->key=[685:656845] now=[685:679673] : multi_timeout: return 0, multi->timetree->key=[685:656845] now=[685:679687] : multi_timeout: return 0, multi->timetree->key=[685:656845] now=[685:679702] : multi_timeout: return 0, multi->timetree->key=[685:656845] now=[685:679716] : multi_timeout: return 0, multi->timetree->key=[685:656845] now=[685:679745] : multi_timeout: return 0, multi->timetree->key=[685:656845] now=[685:680697] : multi_timeout: return 0, multi->timetree->key=[685:656845] now=[685:680727] : multi_timeout: return 0, multi->timetree->key=[685:656845] now=[685:680741] : multi_timeout: return 0, multi->timetree->key=[685:656845] now=[685:680823] : multi_timeout: return 0, multi->timetree->key=[685:656845] now=[685:680837] : multi_timeout: return 0, multi->timetree->key=[685:656845] now=[685:680865] : multi_timeout: return 0, multi->timetree->key=[685:656845] now=[685:680879] : multi_timeout: return 0, multi->timetree->key=[685:656845] now=[685:680907] : multi_timeout: return 0, multi->timetree->key=[685:656845] now=[685:680934] : multi_timeout: return 0, multi->timetree->key=[685:656845] now=[685:680948] : multi_timeout: return 0, multi->timetree->key=[685:656845] now=[685:680976] : multi_timeout: return 0, multi->timetree->key=[685:656845] now=[685:680989] : multi_timeout: return 0, multi->timetree->key=[685:656845] now=[685:681003] : multi_timeout: return 0, multi->timetree->key=[685:656845] now=[685:681016] : multi_timeout: return 0, multi->timetree->key=[685:656845] now=[685:681044] : multi_timeout: return 0, multi->timetree->key=[685:656845] now=[685:681058] : multi_timeout: return 0, multi->timetree->key=[685:656845] now=[685:681085] : multi_timeout: return 0, multi->timetree->key=[685:656845] now=[685:682134] : multi_timeout: return 0, multi->timetree->key=[685:656845] now=[685:682149] : multi_timeout: return 0, multi->timetree->key=[685:656845] now=[685:682177] : multi_timeout: return 0, multi->timetree->key=[685:656845] now=[685:682191] : multi_timeout: return 0, multi->timetree->key=[685:656845] now=[685:682209] : multi_timeout: return 0, multi->timetree->key=[685:656845] now=[685:682223] : multi_timeout: return 0, multi->timetree->key=[685:656845] now=[685:682251] : multi_timeout: return 0, multi->timetree->key=[685:656845] now=[685:682265] : multi_timeout: return 0, multi->timetree->key=[685:656845] now=[685:682292] : multi_timeout: return 0, multi->timetree->key=[685:656845] now=[685:682320] : multi_timeout: return 0, multi->timetree->key=[685:656845] now=[685:682334] : multi_timeout: return 0, multi->timetree->key=[685:656845] now=[685:682361] : multi_timeout: return 0, multi->timetree->key=[685:656845] now=[685:682375] : multi_timeout: return 0, multi->timetree->key=[685:656845] now=[685:682388] : multi_timeout: return 0, multi->timetree->key=[685:656845] now=[685:682402] : multi_timeout: return 0, multi->timetree->key=[685:656845] now=[685:682430] : multi_timeout: return 0, multi->timetree->key=[685:656845] now=[685:682443] : multi_timeout: return 0, multi->timetree->key=[685:656845] now=[685:682471] : multi_timeout: return 0, multi->timetree->key=[685:656845] now=[685:682499] : multi_timeout: return 0, multi->timetree->key=[685:656845] now=[685:682512] : multi_timeout: return 0, multi->timetree->key=[685:656845] now=[685:682540] : multi_timeout: return 0, multi->timetree->key=[685:656845] now=[685:682553] : multi_timeout: return 0, multi->timetree->key=[685:656845] now=[685:682567] : multi_timeout: return 0, multi->timetree->key=[685:656845] now=[685:682581] : multi_timeout: return 0, multi->timetree->key=[685:656845] now=[685:682609] : multi_timeout: return 0, multi->timetree->key=[685:656845] now=[685:682622] : multi_timeout: return 0, multi->timetree->key=[685:656845] now=[685:682650] : multi_timeout: return 0, multi->timetree->key=[685:656845] now=[685:682677] : multi_timeout: return 0, multi->timetree->key=[685:656845] now=[685:682691] : multi_timeout: return 0, multi->timetree->key=[685:656845] now=[685:682719] : multi_timeout: return 0, multi->timetree->key=[685:656845] now=[685:682732] : multi_timeout: return 0, multi->timetree->key=[685:656845] now=[685:682746] : multi_timeout: return 0, multi->timetree->key=[685:656845] now=[685:682760] : multi_timeout: return 0, multi->timetree->key=[685:656845] now=[685:682788] : multi_timeout: return 0, multi->timetree->key=[685:656845] now=[685:682801] : multi_timeout: return 0, multi->timetree->key=[685:656845] now=[685:682829] : multi_timeout: return 0, multi->timetree->key=[685:656845] now=[685:684299] : multi_timeout: return 0, multi->timetree->key=[685:656845] now=[685:684314] : multi_timeout: return 0, multi->timetree->key=[685:656845] now=[685:684343] : multi_timeout: return 0, multi->timetree->key=[685:656845] now=[685:684356] : multi_timeout: return 0, multi->timetree->key=[685:656845] now=[685:684373] : multi_timeout: return 0, multi->timetree->key=[685:656845] now=[685:684387] : multi_timeout: return 0, multi->timetree->key=[685:656845] now=[685:684416] : multi_timeout: return 0, multi->timetree->key=[685:656845] now=[685:684429] : multi_timeout: return 0, multi->timetree->key=[685:656845] now=[685:684457] : multi_timeout: return 0, multi->timetree->key=[685:656845] now=[685:684485] : multi_timeout: return 0, multi->timetree->key=[685:656845] now=[685:684499] : multi_timeout: return 0, multi->timetree->key=[685:656845] now=[685:684526] : multi_timeout: return 0, multi->timetree->key=[685:656845] now=[685:684540] : multi_timeout: return 0, multi->timetree->key=[685:656845] now=[685:684554] : multi_timeout: return 0, multi->timetree->key=[685:656845] now=[685:684568] ... : multi_timeout: return 0, multi->timetree->key=[936:243969] now=[936:317176] : multi_timeout: return 0, multi->timetree->key=[936:243969] now=[936:317197] : multi_timeout: return 0, multi->timetree->key=[936:243969] now=[936:317227] : multi_timeout: return 0, multi->timetree->key=[936:243969] now=[936:317248] : multi_timeout: return 0, multi->timetree->key=[936:243969] now=[936:317249] : multi_timeout: return 0, multi->timetree->key=[936:243969] now=[936:317279] : multi_timeout: return 0, multi->timetree->key=[936:243969] now=[936:317295] : multi_timeout: return 0, multi->timetree->key=[936:243969] now=[936:317305] : multi_timeout: return 0, multi->timetree->key=[936:243969] now=[936:349265] : multi_timeout: return 0, multi->timetree->key=[936:243969] now=[936:349312] : multi_timeout: return 0, multi->timetree->key=[936:243969] now=[936:349312] : multi_timeout: return 0, multi->timetree->key=[936:243969] now=[936:349375] : multi_timeout: return 0, multi->timetree->key=[936:243969] now=[936:349375] : multi_timeout: return 0, multi->timetree->key=[936:243969] now=[936:349375] : multi_timeout: return 0, multi->timetree->key=[936:243969] now=[936:349398] : multi_timeout: return 0, multi->timetree->key=[936:243969] now=[936:349419] : multi_timeout: return 0, multi->timetree->key=[936:243969] now=[936:349422] : multi_timeout: return 0, multi->timetree->key=[936:243969] now=[936:349482] : multi_timeout: return 0, multi->timetree->key=[936:243969] now=[936:349482] : multi_timeout: return 0, multi->timetree->key=[936:243969] now=[936:349508] : multi_timeout: return 0, multi->timetree->key=[936:243969] now=[936:349530] : multi_timeout: return 0, multi->timetree->key=[936:243969] now=[936:349563] : multi_timeout: return 0, multi->timetree->key=[936:243969] now=[936:349563] : multi_timeout: return 0, multi->timetree->key=[936:243969] now=[936:349581] : multi_timeout: return 0, multi->timetree->key=[936:243969] now=[936:349620] : multi_timeout: return 0, multi->timetree->key=[936:243969] now=[936:349627] : multi_timeout: return 0, multi->timetree->key=[936:243969] now=[936:370686] : multi_timeout: return 0, multi->timetree->key=[936:243969] now=[936:370715] : multi_timeout: return 0, multi->timetree->key=[936:243969] now=[936:370745] : multi_timeout: return 0, multi->timetree->key=[936:243969] now=[936:370774] : multi_timeout: return 0, multi->timetree->key=[936:243969] now=[936:370806] : multi_timeout: return 0, multi->timetree->key=[936:243969] now=[936:370806] : multi_timeout: return 0, multi->timetree->key=[936:243969] now=[936:370836] : multi_timeout: return 0, multi->timetree->key=[936:243969] now=[936:370836] : multi_timeout: return 0, multi->timetree->key=[936:243969] now=[936:370866] : multi_timeout: return 0, multi->timetree->key=[936:243969] now=[936:370894] : multi_timeout: return 0, multi->timetree->key=[936:243969] now=[936:370924] : multi_timeout: return 0, multi->timetree->key=[936:243969] now=[936:370953] : multi_timeout: return 0, multi->timetree->key=[936:243969] now=[936:370953] : multi_timeout: return 0, multi->timetree->key=[936:243969] now=[936:370982] : multi_timeout: return 0, multi->timetree->key=[936:243969] now=[936:370982] : multi_timeout: return 0, multi->timetree->key=[936:243969] now=[936:371013] : multi_timeout: return 0, multi->timetree->key=[936:243969] now=[936:371013] : multi_timeout: return 0, multi->timetree->key=[936:243969] now=[936:371045] : multi_timeout: return 0, multi->timetree->key=[936:243969] now=[936:394992] : multi_timeout: return 0, multi->timetree->key=[936:243969] now=[936:394992] : multi_timeout: return 0, multi->timetree->key=[936:243969] now=[936:395045] : multi_timeout: return 0, multi->timetree->key=[936:243969] now=[936:395057] : multi_timeout: return 0, multi->timetree->key=[936:243969] now=[936:395095] : multi_timeout: return 0, multi->timetree->key=[936:243969] now=[936:395104] : multi_timeout: return 0, multi->timetree->key=[936:243969] now=[936:395116] : multi_timeout: return 0, multi->timetree->key=[936:243969] now=[936:395116] : multi_timeout: return 0, multi->timetree->key=[936:243969] now=[936:395162] : multi_timeout: return 0, multi->timetree->key=[936:243969] now=[936:395173] : multi_timeout: return 0, multi->timetree->key=[936:243969] now=[936:395210] : multi_timeout: return 0, multi->timetree->key=[936:243969] now=[936:395231] : multi_timeout: return 0, multi->timetree->key=[936:243969] now=[936:395231] : multi_timeout: return 0, multi->timetree->key=[936:243969] now=[936:395276] : multi_timeout: return 0, multi->timetree->key=[936:243969] now=[936:395288] : multi_timeout: return 0, multi->timetree->key=[936:243969] now=[936:395288] : multi_timeout: return 0, multi->timetree->key=[936:243969] now=[936:395288] : multi_timeout: return 0, multi->timetree->key=[936:243969] now=[936:395345] > >> Or should I drop the curl_multi_wait approach and try to use the >> curl_multi_socket_action API? > > I can't really tell since it depends on many factors. Sure you can but you > don't _have to_. > > -- > > / daniel.haxx.se > > > ------------------------------ > > Subject: Digest Footer > > _______________________________________________ > curl-library mailing list > curl-library@cool.haxx.se > http://cool.haxx.se/cgi-bin/mailman/listinfo/curl-library > > > ------------------------------ > > End of curl-library Digest, Vol 112, Issue 35 > ********************************************* ------------------------------------------------------------------- List admin: http://cool.haxx.se/list/listinfo/curl-library Etiquette: http://curl.haxx.se/mail/etiquette.html
