On Wed, Sep 22, 2021 at 9:57 AM Kévin Corre <[email protected]> wrote:
> @Tom Jones <[email protected]> > > Have you seen draft-banks-quic-performance? I don't see a reference to > > it in your document. This protocol has implementations using a number of > > quic different libraries. > @Ian Swett <[email protected]> > > I would suggest looking at the ACK frequency draf > <https://nam12.safelinks.protection.outlook.com/?url=https%3A%2F%2Fdatatracker.ietf.org%2Fdoc%2Fdraft-ietf-quic-ack-frequency%2F&data=04%7C01%7Ckevin.corre%40exfo.com%7Ce1895191b0f44507a63f08d97db8ea9e%7C1c75be0f25694bcc95f73ad9d904f42a%7C0%7C0%7C637679055450418551%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&sdata=SymOgANn9Tt737DFff%2FSXqpeqpHHpDpSoZq2AkDMOkk%3D&reserved=0>t, > or some similar approaches. Otherwise it can be very difficult to achieve > comparable speeds as TCP. > > Hi Tom, Ian, > Thanks for your comments. > > I've read draft Banks before starting my work. Actually, just to cite > myself discussing that with Lucas Pardue: " Regarding RFC6349 and > draft-banks, it's true that there is a similarity in the fact they stress a > QUIC implementation and work similarly with clients opening multiple > connections to a server. The major difference between the two is that their > intent and metrics differ. > > Banks states in section 3 that "Generally, the goal of all these > performance tests is to measure the maximum load that can be achieved with > the given QUIC implementation and hardware configuration. To that end, the > network is not expected to be the bottleneck in any of these tests. To > achieve that, the appropriate network hardware must be used so as to not > limit throughput". Conversely, the "primary focus of [RFC6349] methodology > is managed business-class IP networks, e.g. those Ethernet-terminated > services for wich organizations are provided an SLA from the network > provider. Because of the SLA, the expectation is that the TCP Throughput > should achieve the guaranteed bandwidth". The RFC6349 methodology is to 1. > measure the path MTU, 2. baseline RTT and bottleneck bandwidth, 3. do a TCP > Throughput test "to baseline network performance". > The metrics captured also show this difference in intent. RFC6349 measures > transfer time ratio over an ideal transfer time depending on the network, > TCP efficiency depending on the loss rate of the network, and buffer delay > which depends on the baselined RTT. Whereas Banks performance scenarios > measure metrics related to the server performance (e.g. requests per > second)." > > Now that you both mention performances, I can see how a QUIC Throughput > test software intended for testing network should be first validated using > recommendations in draft-banks and other recommended optimizations in the > goal of demonstrating performance rather than interoperability. My draft > states that parameters are specified to optimize the QUIC Throughput. At > the moment it's more a declaration of intent taken from RFC6349, but I > guess I could be more specific and more reading/experimentation is probably > required to get a better picture of the impact parameters have on actual > network performances. > > @Ian Swett <[email protected]> > Thanks for your suggestions. I came across fastly's post but now better > realise I should take a look at it again 😉. > While reading the ACK frequency draft, I noted a difference in exponents > limiting min_ack_delay and max_ack_delay. > min_ack_delay (0xff02de1a): > > A variable-length integer representing the minimum amount of time in > microseconds by which the endpoint can delay an acknowledgement. *Values > of 2^24 or greater are invalid,* > *An endpoint's min_ack_delay MUST NOT be greater than its max_ack_delay. * > At the same time, RFC9000 states that > > - max_ack_delay (0x0b): [...] *Values of 214* or greater are invalid > > > Thanks for noticing. That was issue 44 <https://github.com/quicwg/ack-frequency/issues/44> which has been fixed in the editors copy, but we haven't published an updated version yet. We'll push an update quite soon I expect, ideally in the next week. > Regards, > Kevin > > > ------------------------------ > *From:* Ian Swett <[email protected]> > *Sent:* Wednesday, September 22, 2021 1:05 PM > *To:* Tom Jones <[email protected]> > *Cc:* Kévin Corre <[email protected]>; [email protected] <[email protected]> > *Subject:* [E!] : Re: Draft: QUIC Throughput Testing > > I would suggest looking at the ACK frequency draf > <https://nam12.safelinks.protection.outlook.com/?url=https%3A%2F%2Fdatatracker.ietf.org%2Fdoc%2Fdraft-ietf-quic-ack-frequency%2F&data=04%7C01%7Ckevin.corre%40exfo.com%7Ce1895191b0f44507a63f08d97db8ea9e%7C1c75be0f25694bcc95f73ad9d904f42a%7C0%7C0%7C637679055450418551%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&sdata=SymOgANn9Tt737DFff%2FSXqpeqpHHpDpSoZq2AkDMOkk%3D&reserved=0>t, > or some similar approaches. Otherwise it can be very difficult to achieve > comparable speeds as TCP. > > See Fastly's nice post > <https://nam12.safelinks.protection.outlook.com/?url=https%3A%2F%2Fwww.fastly.com%2Fblog%2Fmeasuring-quic-vs-tcp-computational-efficiency&data=04%7C01%7Ckevin.corre%40exfo.com%7Ce1895191b0f44507a63f08d97db8ea9e%7C1c75be0f25694bcc95f73ad9d904f42a%7C0%7C0%7C637679055450418551%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&sdata=bMAieBa9ZkZFyJbobiCFRyy54TiUTcXqeCKPCO279uA%3D&reserved=0> > about what it took to get comparable performance in QUIC as TCP. > > On Wed, Sep 22, 2021 at 6:06 AM Tom Jones <[email protected]> wrote: > > On Wed, Sep 22, 2021 at 09:15:58AM +0000, Kévin Corre wrote: > > Hi everyone, > > > > Our team at EXFO is working on adapting the TCP Throughput Testing > methodology described in RFC6349 to QUIC. We just published a draft that > can be found at > https://datatracker.ietf.org/doc/html/draft-corre-quic-throughput-testing > <https://nam12.safelinks.protection.outlook.com/?url=https%3A%2F%2Fdatatracker.ietf.org%2Fdoc%2Fhtml%2Fdraft-corre-quic-throughput-testing&data=04%7C01%7Ckevin.corre%40exfo.com%7Ce1895191b0f44507a63f08d97db8ea9e%7C1c75be0f25694bcc95f73ad9d904f42a%7C0%7C0%7C637679055450428506%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&sdata=5aJ79pllQs2zckYUPyvj1ccNNg5kQE8tsSSSTwpluX8%3D&reserved=0> > > The draft follows the same organization as the RFC6349 and discuss what > should be changed to support QUIC Throughput Testing. > > The methodology is interested in measuring the achievable QUIC > throughput and efficiency of a managed network when the connection is at an > equilibrium. Although, it could also be used for un-managed networks. > > > > We are particularly looking for comments on the way we use the protocol > to measure the network's QUIC Throughput but also feedback from potential > users of such a test. > > For instance, our draft closely follows the RFC6349 methodology, but > since QUIC introduces new features, it would be interesting to gauge > interest in extending the methodology to test features such as 0-RTT effect. > > > > Comments would be gladly appreciated, so let us know what you think. > > > > Hi Kévin, > > Have you seen draft-banks-quic-performance? I don't see a reference to > it in your document. This protocol has implementations using a number of > quic different libraries. > > There is also a perf channel on the quic slack, please ask off list if > you require an invite. > > https://datatracker.ietf.org/doc/html/draft-banks-quic-performance-00.txt > <https://nam12.safelinks.protection.outlook.com/?url=https%3A%2F%2Fdatatracker.ietf.org%2Fdoc%2Fhtml%2Fdraft-banks-quic-performance-00.txt&data=04%7C01%7Ckevin.corre%40exfo.com%7Ce1895191b0f44507a63f08d97db8ea9e%7C1c75be0f25694bcc95f73ad9d904f42a%7C0%7C0%7C637679055450428506%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&sdata=%2BXdEofehdUsCMCixoA%2BD1QmAnrhfizVsCrnrk1o4mkY%3D&reserved=0> > > - Tom > >
