Dimitry, having let this stew around in my mind the last couple of days, I came up with a tweak that hopefully addresses your issue.
Could you throw https://github.com/curl/curl/pull/20228 into your statistics grinder? I would be interested to hear how it turns out. Thanks, Stefan > Am 07.01.2026 um 21:16 schrieb Dmitry Karpov via curl-library > <[email protected]>: > > > My point is that just the same statement: "there are cases that look like > > as the rate limit hasn’t applied at all" - is also true for 8.6.0. The only > > difference is that with 8.6.0 you have to go to smaller transfer sizes to > > see the effect. > > No, this is not true. As I mentioned in my message, in 8.6.0, the smaller > transfer sizes decreased accuracy, but it was always applied unless the > transfers were extremely small. > But in 8.18.0 we have cases when no rate limit is applied to the transfers > depending on run-time conditions. > > > So yes, at a given transfer size the behavior may be different, because the > > rate-limiting "resolution" has changed, but claiming that this is a radical > > change in behavior seems wrong. > > It is a radical change, because we now have cases in 8.18.0, when the rate > limit is either not applied at all or there is a huge gap between the rate > limit and the measured transfer speed. > And it may increase the operational costs significantly for libcurl users in > such cases. > > Let’s consider an example from my test where the rate limit wasn’t applied: > Url=… file_1M.bin, max_speed=20000000 bps > time=18 ms, dnld=1048576 B, speed=457518843 bps, spd_diff=437518843 bps, > pct=2187.6 % > > If I do multiple downloads like this back-to-back (i.e. 100+ times) then CPU > utilization for these transfers will be as high as without the rate limit > both for client and the server. > But in 8.6.0 – where the rate limit for these conditions was applied > correctly, the total CPU utilization (both for client and server) was a few > times lower. > > So, while 8.18.0 rate limiting mechanism decreased CPU utilization compared > to 8.6.0 for the cases where it works, it increased CPU utilization on order > of magnitude both for client and server > for run-time conditions where it doesn’t work. And this may have a large cost > effect for applications with large population using cloud services. > > And what makes things worse is that it is now very difficult to predict where > the new mechanism will “not work”. > > In the same test example, the rate limit of 16 Mbps was applied correctly: > Url=… file_1M.bin, max_speed=16000000 bps > time=510 ms, dnld=1048576 B, speed=16423165 bps, spd_diff=423165 bps, pct=2.6 > % > > This means that in 8.18.0 some client app doing multiple back-to-back > transfers, may suddenly step on cases where the rate limiting doesn’t work as > before and thus incur much higher operational costs. > And this kind of defeats the purpose of optimizing the rate limiting > mechanism to use less CPU cycles. > > I am trying to avoid calling it “regression”, but it is a radical behavior > change, which potentially can cost much more to libcurl users than previous > implementation. > > Thanks, > Dmitry > From: David Pfitzner <[email protected]> > Sent: Tuesday, January 6, 2026 7:36 PM > To: Dmitry Karpov <[email protected]> > Cc: libcurl development <[email protected]> > Subject: Re: [EXTERNAL] Re: Rate limit regressions in libcurl 8.18.0 vs 8.17.0 > On Wed, Jan 7, 2026 at 1:02 PM Dmitry Karpov <[email protected]> wrote: > In the 8.6.0, the rate limiter mechanism was kind of embedded into the run > loop, so it was always engaged and showed very good precision (< 5% deviation > of the measured speed from the rate limit) even for relatively small transfer > sizes (~300 KB) and super-fast network speeds. > > It worked properly even when client and server were on the same host machine > where there were no network delays. > Here is the test example for 390 KB download throttling test on 8.6.0 when > server and client are on the same machine: > (8.6.0) Speed limit test [server on the same host as client], iterations=5 > Url=http://192.168.1.3/Data/DSC_391kB.jpg, max_speed=8000000 bps > time=399 ms, dnld=400096 B, speed=8020969 bps, spd_diff=20969 bps, pct=0.3 % > time=405 ms, dnld=400096 B, speed=7884597 bps, spd_diff=-115403 bps, pct=-1.4 > % > time=406 ms, dnld=400096 B, speed=7882480 bps, spd_diff=-117520 bps, pct=-1.5 > % > time=404 ms, dnld=400096 B, speed=7910005 bps, spd_diff=-89995 bps, pct=-1.1 % > time=397 ms, dnld=400096 B, speed=8056036 bps, spd_diff=56036 bps, pct=0.7 % > -------------------------------------------------------------------- > avg_deviation=-0.6 %, max_deviation=-1.5 % > Url=http://192.168.1.3/Data/DSC_391kB.jpg, max_speed=16000000 bps > time=209 ms, dnld=400096 B, speed=15286666 bps, spd_diff=-713334 bps, > pct=-4.5 % > time=197 ms, dnld=400096 B, speed=16231980 bps, spd_diff=231980 bps, pct=1.4 % > time=197 ms, dnld=400096 B, speed=16202236 bps, spd_diff=202236 bps, pct=1.3 % > time=198 ms, dnld=400096 B, speed=16110410 bps, spd_diff=110410 bps, pct=0.7 % > time=195 ms, dnld=400096 B, speed=16394185 bps, spd_diff=394185 bps, pct=2.5 % > -------------------------------------------------------------------- > avg_deviation=0.3 %, max_deviation=-4.5 % > Url=http://192.168.1.3/Data/DSC_391kB.jpg, max_speed=20000000 bps > time=162 ms, dnld=400096 B, speed=19697882 bps, spd_diff=-302118 bps, > pct=-1.5 % > time=161 ms, dnld=400096 B, speed=19856004 bps, spd_diff=-143996 bps, > pct=-0.7 % > time=166 ms, dnld=400096 B, speed=19249034 bps, spd_diff=-750966 bps, > pct=-3.8 % > time=162 ms, dnld=400096 B, speed=19721549 bps, spd_diff=-278451 bps, > pct=-1.4 % > time=159 ms, dnld=400096 B, speed=20035479 bps, spd_diff=35479 bps, pct=0.2 % > -------------------------------------------------------------------- > avg_deviation=-1.4 %, max_deviation=-3.8 % > Url=http://192.168.1.3/Data/DSC_391kB.jpg, max_speed=32000000 bps > time=97 ms, dnld=400096 B, speed=32785684 bps, spd_diff=785684 bps, pct=2.5 % > time=99 ms, dnld=400096 B, speed=32143324 bps, spd_diff=143324 bps, pct=0.4 % > time=99 ms, dnld=400096 B, speed=32290545 bps, spd_diff=290545 bps, pct=0.9 % > time=98 ms, dnld=400096 B, speed=32509298 bps, spd_diff=509298 bps, pct=1.6 % > time=98 ms, dnld=400096 B, speed=32478290 bps, spd_diff=478290 bps, pct=1.5 % > -------------------------------------------------------------------- > avg_deviation=1.4 %, max_deviation=2.5 % > > The precision started deteriorating (> 15%) only for the downloads < 250 KB, > but the measured transfer speeds were always less than the rate limit, > so, the back-to-back transfers never created high load issues on the server > side. > > So, to be on the safe side, my statement about high precision of rate > limiting in 8.6.0 should be corrected as "for any transfer sizes > 1MB and > any network conditions". > > Unfortunately, the price for such a precision in 8.6.0 was higher CPU > utilization as more cycles were needed to calculate and apply delays. > > So, the intent to minimize CPU usage while losing some precision in 8.18 is > totally understandable, but the loss of precision due to lesser points of > delay calculations should be somehow compensated at the end of the transfer > to avoid regressions for applications that expect more or less reliable > correlation between applied rate limit and measured transfer speeds. > > Unfortunately, in 8.18.0 there are cases that look like as the rate limit > hasn’t applied at all, and it creates issues for logic making decisions based > on transfer speed measurements, > and unexpected higher load on the server side. > My point is that just the same statement: "there are cases that look like as > the rate limit hasn’t applied at all" - is also true for 8.6.0. The only > difference is that with 8.6.0 you have to go to smaller transfer sizes to see > the effect. So it depends on what transfer sizes you are interested in (and > other parameters). So yes, at a given transfer size the behavior may be > different, because the rate-limiting "resolution" has changed, but claiming > that this is a radical change in behavior seems wrong. > -- David > > > From: curl-library <[email protected]> On Behalf Of David > Pfitzner via curl-library > Sent: Tuesday, January 6, 2026 5:17 PM > To: libcurl development <[email protected]> > Cc: David Pfitzner <[email protected]> > Subject: [EXTERNAL] Re: Rate limit regressions in libcurl 8.18.0 vs 8.17.0 > On Wed, Jan 7, 2026 at 10:05 AM Dmitry Karpov via curl-library > <[email protected]> wrote: > > The 8.6.0 worked very well for rate limiting as it provided stable and > predictable measured transfer download speeds for any transfer sizes and > network conditions. > That "for any transfer sizes and network conditions" is not true. The rate > limiting may have done what you wanted for the parameters you care about, or > the parameters that you happened to test, but even for 8.6.0 there are > possible cases where rate limiting is specified but has no effect. (That is, > because the transfer finishes before any rate limiting delay is imposed.). > -- David > -- > Unsubscribe: https://lists.haxx.se/mailman/listinfo/curl-library > Etiquette: https://curl.se/mail/etiquette.html -- Unsubscribe: https://lists.haxx.se/mailman/listinfo/curl-library Etiquette: https://curl.se/mail/etiquette.html
