Hi, On 2023-11-08 19:29:38 +0100, Tomas Vondra wrote: > >>> I haven't checked, but I'd assume that 100bytes back and forth should > >>> easily > >>> fit a new message to update LSNs and the existing feedback response. Even > >>> just > >>> the difference between sending 100 bytes and sending 10k (a bit more than > >>> a > >>> single WAL page) is pretty significant on a 1gbit network. > >>> > >> > >> I'm on decaf so I may be a bit slow, but it's not very clear to me what > >> conclusion to draw from these numbers. What is the takeaway? > >> > >> My understanding is that in both cases the latency is initially fairly > >> stable, independent of the request size. This applies to request up to > >> ~1000B. And then the latency starts increasing fairly quickly, even > >> though it shouldn't hit the bandwidth (except maybe the 1MB requests). > > > > Except for the smallest end, these are bandwidth related, I think. > > Converting > > 1gbit/s to bytes/us is 125 bytes / us - before tcp/ip overhead. Even leaving > > the overhead aside, 10kB/100kB outstanding take ~80us/800us to send on > > 1gbit. If you subtract the minmum latency of about 130us, that's nearly all > > of > > the latency. > > > > Maybe I don't understand what you mean "bandwidth related" but surely > the smaller requests are not limited by bandwidth. I mean, 100B and 1kB > (and even 10kB) requests have almost the same transaction rate, yet > there's an order of magnitude difference in bandwidth (sure, there's > overhead, but this much magnitude?).
What I mean is that bandwidth is the biggest factor determining latency in the numbers I showed (due to decent sized packet and it being a local network). At line rate it takes ~80us to send 10kB over 1gbit ethernet. So a roundtrip cannot be faster than 80us, even if everything else added zero latency. That's why my numbers show such a lower latency for the 10gbit network - it's simply faster to put even small-ish amounts of data onto the wire. That does not mean that the link is fully utilized over time - because we wait for the other side to receive the data, wake up a user space process, send back 100 bytes, wait for the data be transmitted, and then wake up a process, there are periods where the link in one direction is largely idle. But in case of a 10kB packet on the 1gbit network, yes, we are bandwidth limited for ~80us (or perhaps more interestingly, we are bandwidth limited for 0.8ms when sending 100kB). Greetings, Andres Freund