Hi Yongyi, I think the design choice of one stream per data block or multiple block is about the tradeoff between the flexibility of treating each block independently vs the complexity/overhead it brings. Please see my detailed comments inline.
Best, Hang Shi From: QUIC <[email protected]> On Behalf Of Yongyi Yu Sent: Friday, September 30, 2022 3:22 PM To: Konstantin Tsoy <[email protected]> Cc: Lubashev, Igor <[email protected]>; Tommy Pauly <[email protected]>; Ryan Hamilton <[email protected]>; [email protected]; 陈鉴平 <[email protected]>; 景君羡 <[email protected]>; 刘天一 <[email protected]> Subject: Re: [External] New Version Notification for draft-chen-quic-quicu-01.txt Hi Konstantin, Thanks for your reply. We think there are still some scenarios using HTTP to deliver streaming data, such as delivering the uncompleted media segment in low latency DASH, where one segment would be split into multiple chunks. Would it be helpful if we try to come up with some mechanisms to enable partial reliability feature for those scenarios without heavy impact on caching infrastructure? Back to the discussion about how to achieve partial reliability, we still have some doubts about using one stream per data block. 1. Additional information and application logic are needed to manage order and to achieve multiplexing. This do increase the complexity of the application, and introduce unnecessary overheads. [HS] Increase complexity, yes. Unnecessary, not exactly. By using one stream per block, you get the flexibility to drop/cancel any block without causing head of line blocking. And it reuse the QUIC stream management frame. No need to invent a new type of CANCEL-TILL-THIS-POINT frame. 2. When transmitting data in order, like media data, an application may want to first retransmit the lost data that were sent most early. Since all the streams are independent from each other, the transport layer may not correctly decide whose lost data should be retransmitted first. Although this problem might be resolved by indicating priority for each stream, the behavior of transport layer is still implementation-dependent. [HS] By using one stream for multiple blocks, you basically leverage the FIFO scheduling *inside* the stream. While using one stream per data block, you can rely on FIFO scheduler *between* streams. AFAIK, FIFO scheduler is widely implemented as the default scheduler for many QUIC implementations. What do you mean by “the behavior of transport layer is still implementation-dependent?” Depending on different QUIC implementations? Then FIFO scheduler is usually the default behavior. If an application want to do more optimizations, they are free to change the scheduler both at the application layer *and* QUIC layer. I think this is the main benefit of QUIC running in the user space, right? The approach to implement the scheduling (Application gives hint to QUIC or directly modify the QUIC scheduler) can be discussed. 3. Creating, terminating and monitoring too many streams may cause performance issues, depending on the implementation. [HS] I would argue that multiplexing without HoL blocking(one type of performance impact) is the core advantage of the QUIC comparing to TCP. And QUIC stream id space is quite large(62bit) which indicates the confidence of low overhead to support many concurrent streams. For the media transport use case(especially live streaming which need the partial reliability most), the concurrency is actually small because the video frame is generated/encoded one by one in real time. And to reduce the latency, a video frame is sent immediately right after it is generated. 4. In order to properly cancel the streams which are expired, the application may need to create many timers. It might be more graceful to track the deadlines in the transport layer instead, since there are already some timers for sending packets, retransmitting and so on. [HS] That is exactly how our proposal(DTP: https://datatracker.ietf.org/doc/draft-shi-quic-dtp/ ) works! DTP will decide stream expiration when the scheduler is running, no additional timer needed. As for cancelling implemented at the application layer, maybe it is not necessary to use per block timer by leveraging the application layer event such as sending frame every 1/25 seconds(suppose video FPS = 25)? Looking forward to discuss the approach further. The responsibility splitting between different layer is constantly debatable:) Since our proposal aims at low latency transmission with partial reliability, it should be compatible with other drafts which share the similar target, like RUSH. With the features introduced in our proposal, we believe an application will be able to achieve low latency transmission while avoiding boosting its complexity too much. The stream group approach you mentioned seems great, since we do realize the importance for the application to combine multiple streams. And that is exactly the reason we introduce the Correlation Frame. Looking forward to discussing this approach further. [HS] Agree. Stream group is a useful abstraction. In addition to simply gather multiple streams together, the relationship between streams inside the same group can be expressed. For example, in video use case P frame depends on I frame and they belong to the same Group of Pictures(GOP). The dependency can be expressed as an attribute inside the same stream group and help the scheduler algorithm. Thanks, Yongyi. From: "Konstantin Tsoy"<[email protected]<mailto:[email protected]>> Date: Wed, Sep 28, 2022, 03:56 Subject: Re: [External] New Version Notification for draft-chen-quic-quicu-01.txt To: "Lubashev, Igor"<[email protected]<mailto:[email protected]>> Cc: "Yongyi Yu"<[email protected]<mailto:[email protected]>>, "Tommy Pauly"<[email protected]<mailto:[email protected]>>, "Ryan Hamilton"<[email protected]<mailto:[email protected]>>, "[email protected]<mailto:[email protected]>"<[email protected]<mailto:[email protected]>>, "陈鉴平"<[email protected]<mailto:[email protected]>>, "景君羡"<[email protected]<mailto:[email protected]>>, "刘天一"<[email protected]<mailto:[email protected]>> Yongyi, As Matt described it below, we have experimented with a similar approach in the past at Meta but did not ship it to production given the complexity it brought. If you are using HTTP to deliver streaming data (I presume you do), you are probably using some kind of caching mechanism(s) to not melt your infrastructure. And this approach, while doable, presents certain challenges for caching infrastructure as well - just to keep that in mind. We’re experimenting with something called stream groups to achieve partial reliability for variable latency streaming in a different way. Basically we’re trying to use a QUIC stream abstraction for what you describe as blocks - e.g. one block goes into separate stream. It could be a video frame, video segment or some metadata. QUIC streams are a great generic abstraction and a lot of partial reliability concepts could be laid on top of it, instead of within - at least this is out current thinking after having experimented with a few approaches. You mentioned that ordering is what you lose with using this approach which is true, however IMO it should be trivial to instrument the application itself to manage order using either QUIC stream ids or by adding a few bytes at the beginning of each stream for application to parse and manage order (e.g. some kind of app space ever incrementing id defining each “block” order). Stream groups would allow application to bundle logical streams of data together, and one could enforce stream order within a group if we make stream id space scoped down to a group. We did not officially propose stream groups to the community yet, but planning on doing so going forward. Konstantin Tsoy On Sep 27, 2022, at 9:10 AM, Lubashev, Igor <[email protected]<mailto:[email protected]>> wrote: This Message Is From an External Sender Yongyi, Without going into the question of whether partially reliable streams are necessary (as Matt said, Meta implemented them but is not using that implementation in production), here are a few comments I have. 1. Have you thought about the interaction of you draft with stream and connection flow control? 95% of the time I spent on my design was spent on thinking of ways to augment flow control to avoid extra round trips just to open a flow control window, while still ensuring that the resulting protocol is resilient against an adversarial endpoint forcing the peer to commit an unreasonable amount of resources (memory) to the connection. * 1. Our proposal splits a QUIC stream into many data blocks by Boundary Frame. The receiver would provide data to the application layer only when a data block is completely received. Therefore, the application layer needn't be aware of the gap at the expiration point or discard data already read. * 2. Because the stream is split into blocks, the sender could send or expire data by data blocks. Then the transport layer could track the message boundaries instead of the application layer. 1. Section 4.4 Data Receiving of your draft is saying the same. It seems wrong to have the transport protocol specify the behavior of the QUIC library APIs so precisely. It is ok to try to enable a particular library API, but the transport protocol’s specification ought to limit itself to the semantics of bytes on the wire. Very best, * Igor From: Yongyi Yu <[email protected]<mailto:[email protected]>> Sent: Tuesday, September 27, 2022 11:04 AM Hi Igor, Thanks for your replying. Our proposal indeed shares similar ideas with yours, but there are also some difference. We'd like to update our draft to clarify the difference in detail, and to briefly summary here, there are three major points. 1. Our proposal splits a QUIC stream into many data blocks by Boundary Frame. The receiver would provide data to the application layer only when a data block is completely received. Therefore, the application layer needn't be aware of the gap at the expiration point or discard data already read. 2. Because the stream is split into blocks, the sender could send or expire data by data blocks. Then the transport layer could track the message boundaries instead of the application layer. 3. With the introduction of Correlation Frame, we can combine multiple QUIC streams into single data stream while avoiding head of line blocking. This feature could be quite useful in many scenarios, e.g., transmitting both video and audio data through single HTTP response. Please do let me know if you have any further advice, and please do not hesitate to point out if I make any mistakes. Thanks, Yongyi. From: "Lubashev, Igor"<[email protected]<mailto:[email protected]>> Date: Mon, Sep 26, 2022, 00:29 Subject: RE: [External] New Version Notification for draft-chen-quic-quicu-01.txt To: "Yongyi Yu"<[email protected]<mailto:[email protected]>>, "Tommy Pauly"<[email protected]<mailto:[email protected]>> Cc: "Ryan Hamilton"<[email protected]<mailto:[email protected]>>, "[email protected]<mailto:[email protected]>"<[email protected]<mailto:[email protected]>>, "陈鉴平"<[email protected]<mailto:[email protected]>>, "景君羡"<[email protected]<mailto:[email protected]>>, "刘天一"<[email protected]<mailto:[email protected]>> Are you trying to implement something like https://datatracker.ietf.org/doc/html/draft-lubashev-quic-partial-reliability-02 or https://datatracker.ietf.org/doc/html/draft-lubashev-quic-partial-reliability-03? (The -02 and -3 versions have somewhat different properties, and some people prefer -02 version.) * Igor From: Yongyi Yu <[email protected]<mailto:[email protected]>> Sent: Friday, September 23, 2022 1:11 AM To: Tommy Pauly <[email protected]<mailto:[email protected]>> Cc: Ryan Hamilton <[email protected]<mailto:[email protected]>>; [email protected]<mailto:[email protected]>; 陈鉴平<[email protected]<mailto:[email protected]>>; 景君羡 <[email protected]<mailto:[email protected]>>; 刘天一<[email protected]<mailto:[email protected]>> Subject: Re: [External] New Version Notification for draft-chen-quic-quicu-01.txt Thanks for your replying. I agree "partial reliability" fits our draft better, since we do retransmit lost data and provide in-order transmission. And, our proposal could be compatible with H3, should we clarify the behavior in the draft? Please do let me know if you have any further advice. Thanks, Yongyi. From: "Tommy Pauly"<[email protected]<mailto:[email protected]>> Date: Thu, Sep 22, 2022, 11:32 Subject: Re: [External] New Version Notification for draft-chen-quic-quicu-01.txt To: "Ryan Hamilton"<[email protected]<mailto:[email protected]>> Cc: "于涌溢"<[email protected]<mailto:[email protected]>>, "[email protected]<mailto:[email protected]>"<[email protected]<mailto:[email protected]>>, "陈鉴平"<[email protected]<mailto:[email protected]>>, "景君羡"<[email protected]<mailto:[email protected]>>, "刘天一"<[email protected]<mailto:[email protected]>> I noticed that this draft does reference RFC 9221, and DATAGRAM frames. If I’m understanding correctly, it seems like this is trying to define an approach for what’s more commonly referred to as “partial reliability”, which was something DATAGRAM explicitly didn’t try to include. If this is the case, I think it would be useful to reframe the document in terms of that, since the heavy use of “unreliable” is quite confusing. It also isn’t clear to me how this proposal would work with specific application protocols on top of QUIC — is this something that’s meant to be compatible with H3, or is it for entirely separate use cases? Best, Tommy On Sep 21, 2022, at 3:17 PM, Ryan Hamilton <[email protected]<mailto:[email protected]>> wrote: QUIC has support for unreliable data via the DATAGRAM frame, RFC 9221<https://www.rfc-editor.org/info/rfc9221>. It seems that this new proposal attempts to add unreliable data support not to QUIC, but to QUIC *streams*. QUIC streams, of course, offer a reliable, in-order, sequence of bytes. Did you consider starting with DATAGRAMs, and building from there instead? Cheers, Ryan On Wed, Sep 21, 2022 at 1:14 AM 于涌溢 <[email protected]<mailto:[email protected]>> wrote: Deal all, We would like to introduce an extension of the QUIC protocol for unreliable data transmission called QUIC Unreliable (QUICU) . The following draft is submitted for your consideration and comments. We would also like to present this at IETF 115 to discuss further. To summarize on this draft: it describes an extension of the QUIC protocol for unreliable data transmission called QUIC Unreliable (QUICU), which mainly through the definition and use of three new types of frames, namely the Expire Offset Frame, Boundary Frame, and Correlation Frame. The main purpose of this extension is to reduce data delivery delay. Through controlling the timing and scope of packet losses, QUICU reduces useless transmission to improve transmission efficiency, reduces head-of-line blocking to improve transmission timeliness, which are beneficial to delay-sensitive applications, especially audio and video applications. This document describes the motivation, the operations, and the wire formats of the three new types of frames. Link to html: https://datatracker.ietf.org/doc/html/draft-chen-quic-quicu-01 Please feel free to reach us for any comments or questions on this. Thanks, Yongyi. ---------- Forwarded message --------- From: [email protected]<mailto:[email protected]> Date: Mon, Sep 19, 2022, 20:54 Subject: [External] New Version Notification for draft-chen-quic-quicu-01.txt To: "Jianping Chen"<[email protected]<mailto:[email protected]>>, "Junxian Jing"<[email protected]<mailto:[email protected]>>, "Tianyi Liu"<[email protected]<mailto:[email protected]>>, "Yongyi Yu"<[email protected]<mailto:[email protected]>> A new version of I-D, draft-chen-quic-quicu-01.txt has been successfully submitted by Yongyi Yu and posted to the IETF repository. Name: draft-chen-quic-quicu Revision: 01 Title: An Unreliable Extension to QUIC Document date: 2022-09-19 Group: Individual Submission Pages: 10 URL: https://www.ietf.org/archive/id/draft-chen-quic-quicu-01.txt Status: https://datatracker.ietf.org/doc/draft-chen-quic-quicu/ Htmlized: https://datatracker.ietf.org/doc/html/draft-chen-quic-quicu Diff: https://www.ietf.org/rfcdiff?url2=draft-chen-quic-quicu-01 Abstract: QUIC Unreliable (QUICU) is an extension of the QUIC protocol for unreliable data transmission, mainly through the definition and use of three new types of frames, namely the Expire Offset Frame, Boundary Frame, and Correlation Frame. The main purpose of this extension is to reduce data delivery delay. Through controlling the timing and scope of packet losses, QUICU reduces useless transmission to improve transmission efficiency, reduces head-of-line blocking to improve transmission timeliness, which are beneficial to delay- sensitive applications, especially audio and video applications. This document describes the motivation, the operations, and the wire formats of the three new types of frames. The IETF Secretariat
