And I forgot to mention the real way this is handled in sophisticated wireless systems.
There shouldn’t be any idle time! If there are less bits to be transmitted then the modulation order is reduced increasing the margin and reducing the probability of error! Even for NRZ Ethernet transmitting random frames that happen to the next in a cyclic buffer is in general not the best option. Not only does it not guarantee retransmission of a bad packet (unless is really very little to transmit), it wastes a lot of energy on transmitting packets that have already been received. If there is so little to transmit you should be powering down your transmitter (802.3az). Y(J)S From: Jakob Heitz (jheitz) <jhe...@cisco.com> Sent: 26/02/2021 07:56 To: Yaakov Stein <yaako...@rad.com>; Pascal Thubert (pthubert) <pthubert=40cisco....@dmarc.ietf.org> Cc: det...@ietf.org; Tianran Zhou <zhoutian...@huawei.com>; pce@ietf.org; spr...@ietf.org Subject: RE: new draft on segment routing approach to TSN Lots of ways. Another: Use a circular buffer. As you transmit a packet put it into the buffer. As it is acknowledged, remove it. Just keep transmitting the buffer circularly. If one is NACKed, retransmit it immediately. The receiver can keep track of received sequence numbers. They should be in order. Ack the latest sequence whenever there is a chance. If the receiver sees a hole in the received sequence, NACK it. Regards, Jakob. From: Yaakov Stein <yaako...@rad.com<mailto:yaako...@rad.com>> Sent: Thursday, February 25, 2021 9:48 PM To: Jakob Heitz (jheitz) <jhe...@cisco.com<mailto:jhe...@cisco.com>>; Pascal Thubert (pthubert) <pthubert=40cisco....@dmarc.ietf.org<mailto:pthubert=40cisco....@dmarc.ietf.org>> Cc: det...@ietf.org<mailto:det...@ietf.org>; Tianran Zhou <zhoutian...@huawei.com<mailto:zhoutian...@huawei.com>>; pce@ietf.org<mailto:pce@ietf.org>; spr...@ietf.org<mailto:spr...@ietf.org> Subject: RE: new draft on segment routing approach to TSN Jakob I forgot to address your first sentence. Retransmission without NACK is an interesting idea, but I am not sure it really helps. If there is nothing to be transmitted then you could go into a loop and retransmit packets over and over, but I assume that this usually won’t happen. So assume that you have a short idle window – which packet are you going to retransmit? Without knowing which packet was not received correctly you have no idea. So let’s assume that you reserve a full 50% of your BW and retransmit every packet. Some packets received twice will differ – which one is correct? You can’t know without a FEC. And if you are using a FEC it is more efficient to use a strong one then retransmit the whole packet! So, let’s assume that you don’t use a FEC (maybe you don’t want the hassle of implementing one in hardware). Then you need to send each packet at least 3 times to be able to do a majority decision. So, you have reduced your BW efficiency to 33%. I’m not saying that this is never justified. Lots of messages are sent 3 times for redundancy. Going back to cellular, triple redundancy is used for critical control messages (in addition to other error correction means)/ But only for them. It is really suboptimal for everything else. Y(J)S From: Yaakov Stein Sent: 26/02/2021 07:38 To: Jakob Heitz (jheitz) <jhe...@cisco.com<mailto:jhe...@cisco.com>>; Pascal Thubert (pthubert) <pthubert=40cisco....@dmarc.ietf.org<mailto:pthubert=40cisco....@dmarc.ietf.org>> Cc: det...@ietf.org<mailto:det...@ietf.org>; Tianran Zhou <zhoutian...@huawei.com<mailto:zhoutian...@huawei.com>>; pce@ietf.org<mailto:pce@ietf.org>; spr...@ietf.org<mailto:spr...@ietf.org> Subject: RE: new draft on segment routing approach to TSN My point regarding HARQ was that this kind of retransmission can be more efficiently handled at layers lower than where you see packets. 4G/5G has actually two mechanisms precisely of the type you suggest at two different layers, but the magic happens on the physical signals (HARQ) or the SDUs (PDCP retransmission) and is completely transparent to the user packets. True, high speed Ethernet links are no longer physical broadcast domains and one could insert sequence numbers into the frames and perform local retransmission. This has been done in the past (I personally have a patent on such a method, although it also gives an alternative to numbering). But if you are already numbering it might be even better to go all the way and use RFER. (Or maybe even combine ACK/NACK retransmission and replication.) Y(J)S From: Jakob Heitz (jheitz) <jhe...@cisco.com<mailto:jhe...@cisco.com>> Sent: 26/02/2021 07:00 To: Yaakov Stein <yaako...@rad.com<mailto:yaako...@rad.com>>; Pascal Thubert (pthubert) <pthubert=40cisco....@dmarc.ietf.org<mailto:pthubert=40cisco....@dmarc.ietf.org>> Cc: det...@ietf.org<mailto:det...@ietf.org>; Tianran Zhou <zhoutian...@huawei.com<mailto:zhoutian...@huawei.com>>; pce@ietf.org<mailto:pce@ietf.org>; spr...@ietf.org<mailto:spr...@ietf.org> Subject: RE: new draft on segment routing approach to TSN Suppose you're sending packets out an interface and all the queues are empty. You got nothing to send. What are you going to do with the dead air? Send idle patterns? No. Retransmit your previous queues. Just in case something got lost the preemptive retransmission will fix it quicker. Then put a link layer sequence number and ACK and NACK numbers so you can manage what you retransmit. This will only work on point-to-point. Ethernet's not broadcast on the physical layer anymore, is it? Regards, Jakob. From: Yaakov Stein <yaako...@rad.com<mailto:yaako...@rad.com>> Sent: Thursday, February 25, 2021 8:56 PM To: Jakob Heitz (jheitz) <jhe...@cisco.com<mailto:jhe...@cisco.com>>; Pascal Thubert (pthubert) <pthubert=40cisco....@dmarc.ietf.org<mailto:pthubert=40cisco....@dmarc.ietf.org>> Cc: det...@ietf.org<mailto:det...@ietf.org>; Tianran Zhou <zhoutian...@huawei.com<mailto:zhoutian...@huawei.com>>; pce@ietf.org<mailto:pce@ietf.org>; spr...@ietf.org<mailto:spr...@ietf.org> Subject: RE: new draft on segment routing approach to TSN Of course they are common in wireless. Cellular HARQ works just like that – if the FEC fixes the problem then fine, if not then the information is retransmitted. And HARQ not only causes timing problems when there are errors, the HARQ SAW (Stop and Wait) process adds delay to all information transmitted. However, I was mostly thinking about wireline cases. Pascal pointed to the work being done in RAW, which is where this question needs to be addressed. Y(J)S From: Jakob Heitz (jheitz) <jhe...@cisco.com<mailto:jhe...@cisco.com>> Sent: 26/02/2021 06:47 To: Yaakov Stein <yaako...@rad.com<mailto:yaako...@rad.com>>; Pascal Thubert (pthubert) <pthubert=40cisco....@dmarc.ietf.org<mailto:pthubert=40cisco....@dmarc.ietf.org>> Cc: det...@ietf.org<mailto:det...@ietf.org>; Tianran Zhou <zhoutian...@huawei.com<mailto:zhoutian...@huawei.com>>; pce@ietf.org<mailto:pce@ietf.org>; spr...@ietf.org<mailto:spr...@ietf.org> Subject: RE: new draft on segment routing approach to TSN Are they rare in wireless transmission? I believe some wireless protocols have lower layers that retransmit to provide reliability, but I don't know much about them. My point is if you are going to provide a guarantee of time for delivery, you need to consider all factors that influence that time. Also, what happens if you can't meet the deadline? Do you drop the packet or deliver with best effort? Regards, Jakob. From: Yaakov Stein <yaako...@rad.com<mailto:yaako...@rad.com>> Sent: Thursday, February 25, 2021 8:40 PM To: Jakob Heitz (jheitz) <jhe...@cisco.com<mailto:jhe...@cisco.com>>; Pascal Thubert (pthubert) <pthubert=40cisco....@dmarc.ietf.org<mailto:pthubert=40cisco....@dmarc.ietf.org>> Cc: det...@ietf.org<mailto:det...@ietf.org>; Tianran Zhou <zhoutian...@huawei.com<mailto:zhoutian...@huawei.com>>; pce@ietf.org<mailto:pce@ietf.org>; spr...@ietf.org<mailto:spr...@ietf.org> Subject: RE: new draft on segment routing approach to TSN Jakob Yes, one always needs to consider bit errors (even if they are very rare in wireline nowadays). I’m just not sure where you want to hang this on the present discussion. A local retransmission mechanism such as the one you suggest would obviously thwart any calendaring mechanism that attempts to coordinate scheduling based on traffic timing (unless the detection, signaling, and retransmission takes place much faster than an opening time). LIS could be made to work by retransmitting with the original timestamp, and EDF would work just fine. Y(J)S From: Jakob Heitz (jheitz) <jhe...@cisco.com<mailto:jhe...@cisco.com>> Sent: 26/02/2021 01:36 To: Pascal Thubert (pthubert) <pthubert=40cisco....@dmarc.ietf.org<mailto:pthubert=40cisco....@dmarc.ietf.org>>; Yaakov Stein <yaako...@rad.com<mailto:yaako...@rad.com>> Cc: det...@ietf.org<mailto:det...@ietf.org>; Tianran Zhou <zhoutian...@huawei.com<mailto:zhoutian...@huawei.com>>; pce@ietf.org<mailto:pce@ietf.org>; spr...@ietf.org<mailto:spr...@ietf.org> Subject: RE: new draft on segment routing approach to TSN Another thing to consider is packets dropped due to bit errors. Possibilities: Link level acknowledgement and preemptive retransmission if the link would otherwise be idle. I.e. "just in case" retransmission. Regards, Jakob. From: spring <spring-boun...@ietf.org<mailto:spring-boun...@ietf.org>> On Behalf Of Pascal Thubert (pthubert) Sent: Thursday, February 25, 2021 1:27 PM To: Yaakov Stein <yaako...@rad.com<mailto:yaako...@rad.com>> Cc: det...@ietf.org<mailto:det...@ietf.org>; Tianran Zhou <zhoutian...@huawei.com<mailto:zhoutian...@huawei.com>>; pce@ietf.org<mailto:pce@ietf.org>; spr...@ietf.org<mailto:spr...@ietf.org> Subject: Re: [spring] new draft on segment routing approach to TSN Hello Yaakov Please see below: Le 25 févr. 2021 à 21:33, Yaakov Stein <yaako...@rad.com<mailto:yaako...@rad.com>> a écrit : Pascal, Wow, there is a lot to respond to. Thanks for the pointers. I was not aware of this work. However, I believe this proposal is both different and can be complementary to the ones you mention. Indeed; the questions that this raises are: - which WG, so I added RAW to the candidate list in this thread - how to package the complementary proposals in how many draft Your excellent work focuses on SR signaling in the forwarding plane with strict intermediate latencies at each stage. It would be grand to neatly insert this in a SRv6 together with other information that the RAW PSE processes in its forwarding/scheduling decision. There are multiple dimensions in which the scope of your work could be enlarged; one of them would be to use a similar signaling but with a best effort objective, which is what I discussed. This could easily be combined if we decide to, so the same signaling serves both use cases. And maybe more, to be discussed. I was going to add a section on how to reproduce Qbv behavior using a stack of deadlines (the argument is a bit tricky, but straightforward). I don’t necessarily think that is a good idea, but it should be there to counter claims that SRTSN only reduces the expectation but can’t provide an absolute upper bound on latency (when based on EDF). Very cool; I like to use the term latency, but sometimes use delay to avoid overusing it. Your decision, at least as long as it is personal submission ; but having a clear terminology is useful. Literature value is of lesser importance here I guess. Latency is what we use in DetNet and RAW. Delay often refers to extra, possibly unwanted time, like a queueing delay. At least you’d make my reader life easier by defining your terminology and sticking to it quite strictly : ) I thought that “green wave” would be understood by those who drive cars in cities where traffic light coordination is common. Yes, and I think you were clear enough; but then only a few deterministic flows really operate like that. Another flow that use the same outgoing port will have to suffer some delay in a time-triggered queue till the port is free, the delay being taken from the end to end latency budget. My suggestion was to reconsider the text to avoid giving the impression that all flows benefit from that green wave. I attempted to describe a data structure for EDF to be used instead of a queue, Neat but never implied that “queueing” (meaning packets waiting to be transmitted) doesn’t occur. If that were the case there would not be a need for any data structure at all. Cool (Another thing I didn’t get into was the trade-offs among the different options, from worst case complexity and difficulty to implement in hardware PoVs.) I’d love to read that too I took a quick look at US 9602420 and will study it further. It’s not as deep as an academic paper but a good hint at how you can turn schedule into QoS or another technique to reorder the set of packets that compete for the port. However the purpose of this draft is not to mandate any specific scheduling mechanism, rather to show how such mechanisms can be integrated and hopefully more readily distributed. Usually you’re expected to provide the meaning of the signaling and the visible behavior of the node; the way the node implements the does not need to be spec but some hints on how that can be achieved are welcome, possibly in annex. Incidentally I am working on a joint path+deadline optimization algorithm which only requires partial information. I’d love to chat in this; missing the face to face meetings! Could it be that we do SF in summer? Proofs of absolute upper bounds on latency are nice, but generally come at the price of either This is exactly why we did what we did; it also comes at the cost of gathering very specific info on the flows and not all flows can do that, think CBR. 1. significantly reduced bandwidth efficiency (like TDM, or equivalently time gating as in Qbv or FlexE), 2. unscalable computation 3. relatively loose deadlines to being with (like Andrews Zhang) I need to read it, sorry I was not aware enough of that line of work. 1. 2. assumptions that aren’t necessarily obeyed in real life (like a lot of the network calculus proofs). I would be happy to discuss your variant to see in which class it falls. Same here! Pascal Y(J)S From: Pascal Thubert (pthubert) <pthub...@cisco.com<mailto:pthub...@cisco.com>> Sent: 25/02/2021 19:40 To: Tianran Zhou <zhoutian...@huawei.com<mailto:zhoutian...@huawei.com>>; Yaakov Stein <yaako...@rad.com<mailto:yaako...@rad.com>>; det...@ietf.org<mailto:det...@ietf.org>; spr...@ietf.org<mailto:spr...@ietf.org>; pce@ietf.org<mailto:pce@ietf.org> Subject: RE: new draft on segment routing approach to TSN CAUTION: External sender. Do not click links or open attachments unless you know the content is safe. Hi Yaakov and all: Whereever Yaakov decides to place it I’ll be there supporting the work. The draft itself is incredibly well-written and information-rich. Note that there’s also work in RAW that mentions SR operation DetNet related operations (draft-pthubert-raw-architecture<https://eur01.safelinks.protection.outlook.com/?url=https%3A%2F%2Ftools.ietf.org%2Fhtml%2Fdraft-pthubert-raw-architecture-05&data=04%7C01%7Cyaakov_s%40rad.com%7C7c35c6cd3c294bfa0c3a08d8da1b359a%7Cf9047108cc2c4e4897a343fad1b3bf9d%7C1%7C0%7C637499157699129811%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&sdata=SdSb0fY0Sz9L303hegarZn8vsWQTHl4btRH8OH8TvIs%3D&reserved=0>). RAW has vested interest in intelligent forwarding decision, that would be the trademark vs. DetNet. With this draft, the forwarding is not based on Qbv schedule but the forwarder has some latitude as long as it matches the hop deadline. So RAW may be a good place. And then there’s draft-chen-detnet-sr-based-bounded-latency<https://eur01.safelinks.protection.outlook.com/?url=https%3A%2F%2Ftools.ietf.org%2Fhtml%2Fdraft-chen-detnet-sr-based-bounded-latency-01&data=04%7C01%7Cyaakov_s%40rad.com%7C7c35c6cd3c294bfa0c3a08d8da1b359a%7Cf9047108cc2c4e4897a343fad1b3bf9d%7C1%7C0%7C637499157699139798%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&sdata=VH2cFU4arYdrqyJ0THSOl2CSmdqorFRrOk0aSc%2FL6vg%3D&reserved=0>. Ideally all these related items would progress in the same room. Also a few notes on the draft itself: - maybe use latency instead of delay; it would be nice to maybe define delay as something else, e.g., the delay representing the time the packet spends queued in one hop vs. the latency that is end to end? - not sure the term green wave is well understood by the public here; the draft gives the impression that the TSN path is faster than the best effort and involves no queueing. For the most part that is untrue; the latency is bounded but for most flows it is longer than best effort. Best effort can be really fast with passthrough in an empty network. The problem is the long tail and possibly congestion loss. For TSN, there can be very special flows that will traverse the city with all the lights green, but usually there’ll be queuing. The difference is that the queueing latency is constant and the overall latency is withing bounds. - Time triggered is not the only TSN operation. I wonder what the draft would become with asynchronous shaper in mind. We designed (and as I must announce, patented as US9602420<https://eur01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fpatents.google.com%2Fpatent%2FUS9602420&data=04%7C01%7Cyaakov_s%40rad.com%7C7c35c6cd3c294bfa0c3a08d8da1b359a%7Cf9047108cc2c4e4897a343fad1b3bf9d%7C1%7C0%7C637499157699149797%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&sdata=6JZDeqWc3chpJRaH5PTTdKWZvyeuDOFVhhGx%2BVA3Mw0%3D&reserved=0>) a system very similar to the one proposed in the draft, but that is designed to adapt QoS depending on whether the packet is early or late vs. its schedule, and not tagging the schedule in the since the latency is considered end to end not hop by hop. The use case is slightly different since we apply this without a global controller and a provable guarantees all flows will meet the deadline – so not really detnet-, but more like a best effort that all flows meet their deadline in a stochastic environment. If Yaakov is interested, we can contribute on that aspect. Good luck with the draft, Pascal From: detnet <detnet-boun...@ietf.org<mailto:detnet-boun...@ietf.org>> On Behalf Of Tianran Zhou Sent: jeudi 25 février 2021 9:14 To: Yaakov Stein <yaako...@rad.com<mailto:yaako...@rad.com>>; det...@ietf.org<mailto:det...@ietf.org>; spr...@ietf.org<mailto:spr...@ietf.org>; pce@ietf.org<mailto:pce@ietf.org> Subject: Re: [Detnet] new draft on segment routing approach to TSN Hi Yaakov, This is an interesting topic. After a quick review, there are several questions as follows: 1. It’s clear to me to have a deadline for each packet. So that router can schedule the packet based on the urgency. But what’s the motivation to split the end to end deadline to several local ones? 2. How to divide an end to end deadline into several local deadlines? Is there any example algorithm that could be used by the controller? 3. As far as I know, most devices do not support edf. I am not sure whether your proposal based on edf could really be useful. Cheers, Tianran From: Pce [mailto:pce-boun...@ietf.org] On Behalf Of Yaakov Stein Sent: Tuesday, February 23, 2021 9:14 PM To: det...@ietf.org<mailto:det...@ietf.org>; spr...@ietf.org<mailto:spr...@ietf.org>; pce@ietf.org<mailto:pce@ietf.org> Subject: [Pce] new draft on segment routing approach to TSN All, I would like to call your attention to a new ID https://www.ietf.org/archive/id/draft-stein-srtsn-00.txt<https://eur01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fwww.ietf.org%2Farchive%2Fid%2Fdraft-stein-srtsn-00.txt&data=04%7C01%7Cyaakov_s%40rad.com%7C7c35c6cd3c294bfa0c3a08d8da1b359a%7Cf9047108cc2c4e4897a343fad1b3bf9d%7C1%7C0%7C637499157699149797%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&sdata=mE2eo7Ic3Iv%2BYuiFIgl2o32vgV0nLMkQedGLv%2BOk2n0%3D&reserved=0> which describes using a stack-based approach (similar to segment routing) to time sensitive networking. It furthermore proposes combining segment routing with this approach to TSN resulting in a unified approach to forwarding and scheduling. The draft is information at this point, since it discusses the concepts and does not yet pin down the precise formats. Apologies for simultaneously sending to 3 lists, but I am not sure which WG is the most appropriate for discussions of this topic. * DetNet is most relevant since the whole point is to control end-to-end latency of a time-sensitive flow. * Spring is also directly relevant due to the use of a stack in the header and the combined approach just mentioned. * PCE is relevant to the case of a central server jointly computing an optimal path and local deadline stack. I’ll let the chairs decide where discussions should be held. Y(J)S
_______________________________________________ Pce mailing list Pce@ietf.org https://www.ietf.org/mailman/listinfo/pce
