Re: [RFC net-next 0/5] TSN: Add qdisc-based config interfaces for traffic shapers
Hi, On 10/19/2017 01:39 PM, Richard Cochran wrote: > On Wed, Oct 18, 2017 at 03:37:35PM -0700, Jesus Sanchez-Palencia wrote: >> I also did some tests with when you don't set valid launch times, but here >> using >> your idea from above, so with the driver calculating a valid launch time >> (i.e. >> current NIC time + X ns, varying X across tests) for packets that didn't >> have it >> set by the user, and I wasn't too happy with its reliability. It could >> definitely be improved, but it has left me wondering: instead, what about >> documenting that if you enable TXTIME, then you *must* provide a valid Launch >> time for all packets on traffic classes that are affected? > > If txtime is enabled, then CBS is pointless because the txtime already > specifies the bandwidth implicitly. Assuming there is no "interfering" traffic on that traffic class, yes. Otherwise, CBS could be configured just to avoid that outbound traffic ever goes beyond the reserved bandwidth. > > The problem is when one program uses txtime and another uses CBS, then > the CBS user will experience really wrong performance. Good point. We'll need to adjust the launch time for controllers that behave like the i210 then, imo. Thanks, Jesus > > Thanks, > Richard >
Re: [RFC net-next 0/5] TSN: Add qdisc-based config interfaces for traffic shapers
On Wed, Oct 18, 2017 at 03:37:35PM -0700, Jesus Sanchez-Palencia wrote: > I also did some tests with when you don't set valid launch times, but here > using > your idea from above, so with the driver calculating a valid launch time (i.e. > current NIC time + X ns, varying X across tests) for packets that didn't have > it > set by the user, and I wasn't too happy with its reliability. It could > definitely be improved, but it has left me wondering: instead, what about > documenting that if you enable TXTIME, then you *must* provide a valid Launch > time for all packets on traffic classes that are affected? If txtime is enabled, then CBS is pointless because the txtime already specifies the bandwidth implicitly. The problem is when one program uses txtime and another uses CBS, then the CBS user will experience really wrong performance. Thanks, Richard
Re: [RFC net-next 0/5] TSN: Add qdisc-based config interfaces for traffic shapers
Hi Richard, On 09/19/2017 10:25 PM, Richard Cochran wrote: (...) > >> I have a question, what about a controller that doesn't provide a way to >> set a per-packet transmission time, but it supports Qbv/Qbu. What would >> be your proposal to configure it? > > SO_TXTIME will have a generic SW fallback. > > BTW, regarding the i210, there is no sensible way to configure both > CBS and time based transmission at the same time. The card performs a > logical AND to make the launch decision. The effect of this is that > each and every packet needs a LaunchTime, and the driver would be > forced to guess the time for a packet before entering it into its > queue. > > So if we end up merging CBS and SO_TXTIME, then we'll have to make > them exclusive of each other (in the case of the i210) and manage the > i210 queue configurations correctly. > I've ran some quick tests here having launch time enabled on i210 + our cbs patchset. When valid Launch times are set on each packet you still get the expected behavior, so I'm not sure we should just make them exclusive of each other. I also did some tests with when you don't set valid launch times, but here using your idea from above, so with the driver calculating a valid launch time (i.e. current NIC time + X ns, varying X across tests) for packets that didn't have it set by the user, and I wasn't too happy with its reliability. It could definitely be improved, but it has left me wondering: instead, what about documenting that if you enable TXTIME, then you *must* provide a valid Launch time for all packets on traffic classes that are affected? With the SO_TXTIME qdisc idea in place, that could even be enforced before packets were enqueued into the netdevice. Regards, Jesus
Re: [RFC net-next 0/5] TSN: Add qdisc-based config interfaces for traffic shapers
Hi all, On Mon, 2017-10-02 at 12:45 -0600, Levi Pearson wrote: > Hi Rodney, > > Some archives seem to have threaded it, but I have CC'd the > participants I saw in the original discussion thread since they may > not otherwise notice it amongst the normal traffic. > > On Fri, Sep 29, 2017 at 2:44 PM, Rodney Cummings > wrote: [...] > > 1. Question: From an 802.1 perspective, is this RFC intended to support > > end-station (e.g. NIC in host), bridges (i.e. DSA), or both? > > > > This is very important to clarify, because the usage of this interface > > will be very different for one or the other. > > > > For a bridge, the user code typically represents a remote management > > protocol (e.g. SNMP, NETCONF, RESTCONF), and this interface is > > expected to align with the specifications of 802.1Q clause 12, > > which serves as the information model for management. Historically, > > a standard kernel interface for management hasn't been viewed as > > essential, but I suppose it wouldn't hurt. > > I don't think the proposal was meant to cover the case of non-local > switch hardware, but in addition to dsa and switchdev switch ICs > managed by embedded Linux-running SoCs, there are SoCs with embedded > small port count switches or even plain multiple NICs with software > bridging. Many of these embedded small port count switches have FQTSS > hardware that could potentially be configured by the proposed cbs > qdisc. This blurs the line somewhat between what is a "bridge" and > what is an "end-station" in 802.1Q terminology, but nevertheless these > devices exist, sometimes acting as an endpoint + a real bridge and > sometimes as just a system with multiple network interfaces. During the development of this proposal, we were most focused on end-station use-cases. We considered some bridge use-cases as well just to verify that the proposed design wouldn't be an issue if someone else goes for it. We agree that the line between end-station and bridge can be a bit blurred (in this case). Even though we designed this interface with end-station use-cases in mind, if the proposed infrastructure could be used as is in bridge use- cases, good. > > For an end station, the user code can be an implementation of SRP > > (802.1Q clause 35), or it can be an application-specific > > protocol (e.g. industrial fieldbus) that exchanges data according > > to P802.1Qcc clause 46. Either way, the top-level user interface > > is designed for individual streams, not queues and shapers. That > > implies some translation code between that top-level interface > > and this sort of kernel interface. Yes, you're right. Our understanding is that the top-level interfaces should be implemented at user space as well as any stream management functionality. The idea here is to keep the kernel-side as simple as possible. The kernel handles hardware configuration (via Traffic Control interface) while the user space handles TSN streams i.e. the kernel provides the mechanism and the user space provides the policy. > > As a specific end-station example, for CBS, 802.1Q-2014 subclause > > 34.6.1 requires "per-stream queues" in the Talker end-station. > > I don't see 34.6.1 represented in the proposed RFC, but that's > > okay... maybe per-stream queues are implemented in user code. > > Nevertheless, if that is the assumption, I think we need to > > clarify, especially in examples. > > You're correct that the FQTSS credit-based shaping algorithm requires > per-stream shaping by Talker endpoints as well, but this is in > addition to the per-class shaping provided by most hardware shaping > implementations that I'm aware of in endpoint network hardware. I > agree that we need to document the need to provide this, but it can > definitely be built on top of the current proposal. > > I believe the per-stream shaping could be managed either by a user > space application that manages all use of a streaming traffic class, > or through an additional qdisc module that performs per-stream > management on top of the proposed cbs qdisc, ensuring that the > frames-per-observation interval aspect of each stream's reservation is > obeyed. This becomes a fairly simple qdisc to implement on top of a > per-traffic class shaper, and could even be implemented with the help > of the timestamp that the SO_TXTIME proposal adds to skbuffs, but I > think keeping the layers separate provides more flexibility to > implementations and keeps management of various kinds of hardware > offload support simpler as well. Indeed, 'per-stream queue' is not covered in this RFC. For now, we expect it to be implemented in user code. We believe the proposed CBS qdisc could be extended to support a full software-based implementation which would be used to implement 'per-stream queue' support. This functionality should be addressed by a separated series. Anyways, we're about to send the v3 patchset implementing this proposal and we'll make it clear. > > 2. Suggestion: Do not assume tha
RE: [RFC net-next 0/5] TSN: Add qdisc-based config interfaces for traffic shapers
Thanks Levi, Great discussion. It sounds like we're aligned. Qbv is essential in switches. My concern was the end-station side only. You're absolutely right that i210 LaunchTime doesn't do everything that is needed for scheduling. It still requires quite a bit of frame-juggling to ensure that transmit descriptors are ordered by time, and that the control system's cyclic traffic is repeated as expected. Like you, I'm not sure where that code is best located in the long run. For a hardware design that connects an FPGA to one of the internal ports of a Qbv-capable switch, the preferable scheduling design can be done there. The FPGA IP essentially implements a cyclic schedule of transmit descriptors, each with its own buffer that can be written atomically by application code. For a control system (i.e. industrial field-level or automotive control), each buffer (stream) can hold a single frame. If you take a look at the "Network Interface" specs for FlexRay in automotive, they went so far as to mandate that design for a NIC. The IP for that sort of thing doesn't take alot of gates or memory, so I can see why FlexRay went that way. That sort of design wasn't destined to happen as part of Qbv, because the switch vendors are worried about things like 64-port switches. Also, if we treat Qbv as applying to switches only, FlexRay-style IP isn't needed. For the switch, we only need it to "get out of the way", and Qbv is fine for that. I may be naive and idealistic, but in the long run, I'm still hopeful that an Ethernet NIC vendor will implement Flexray-style scheduling for end-station. We need a NIC trend-setter to do it as a differentiator, and the rest will probably follow along. That completely eliminates code in Linux for the frame-juggling. We'll need to provide a way to open a socket for a specific stream, but that doesn't seem too challenging. Rodney > -Original Message- > From: Levi Pearson [mailto:levipear...@gmail.com] > Sent: Monday, October 2, 2017 4:49 PM > To: Rodney Cummings > Cc: Linux Kernel Network Developers ; Vinicius > Costa Gomes ; Henrik Austad ; > richardcoch...@gmail.com; jesus.sanchez-palen...@intel.com; > andre.gue...@intel.com > Subject: Re: [RFC net-next 0/5] TSN: Add qdisc-based config interfaces for > traffic shapers > > Hi Rodney, > > On Mon, Oct 2, 2017 at 1:40 PM, Rodney Cummings > wrote: > > > It's a shame that someone built such hardware. Speaking as a > manufacturer > > of daisy-chainable products for industrial/automotive applications, I > > wouldn't use that hardware in my products. The whole point of scheduling > > is to obtain close-to-optimal network determinism. If the hardware > > doesn't schedule properly, the product is probably better off using CBS. > > I should note that the hardware I'm using was not designed to be a TSN > endpoint. It just happens that the hardware, designed for other > reasons, happens to have hardware support for Qbv and thus makes it an > interesting and inexpensive platform on which to do Qbv experiments. > Nevertheless, I believe the right combination of driver support for > the hardware shaping and appropriate software in the network stack > would allow it to schedule things fairly precisely in the > application-centric manner you've described. I will ultimately defer > to your judgement on that, though, and spend some time with the Qcc > draft to see where I may be wrong. > > > It would be great if all of the user-level application code was > scheduled > > as tightly as the network, but the reality is that we aren't there yet. > > Also, even if the end-station has a separately timed RT Linux thread for > > each stream, the order of those threads can vary. Therefore, when the > > end-station (SoC) has more than one talker, the frames for each talker > > can be written into the Qbv queue at any time, in any order. > > There is no scheduling of frames (i.e. streams)... only the queue. > > Frames are presented to *the kernel* in any order. The qdisc > infrastructure provides mechanisms by which other scheduling policy > can be added between presentation to the kernel and presentation to > hardware queues. > > Even with i210-style LaunchTime functionality, this must also be > provided. The hardware cannot re-order time-scheduled frames, and > reviewers of the SO_TXTIME proposal rejected the idea of having the > driver try to re-order scheduled frames already submitted to the > hardware-visible descriptor chains, and rightly so I think. > > So far, I have thought of 3 places this ordering constraint might be > provided: > > 1. Built into each driver with offload support for SO_TXTIME > 2. A qdis
Re: [RFC net-next 0/5] TSN: Add qdisc-based config interfaces for traffic shapers
Hi Rodney, On Mon, Oct 2, 2017 at 1:40 PM, Rodney Cummings wrote: > It's a shame that someone built such hardware. Speaking as a manufacturer > of daisy-chainable products for industrial/automotive applications, I > wouldn't use that hardware in my products. The whole point of scheduling > is to obtain close-to-optimal network determinism. If the hardware > doesn't schedule properly, the product is probably better off using CBS. I should note that the hardware I'm using was not designed to be a TSN endpoint. It just happens that the hardware, designed for other reasons, happens to have hardware support for Qbv and thus makes it an interesting and inexpensive platform on which to do Qbv experiments. Nevertheless, I believe the right combination of driver support for the hardware shaping and appropriate software in the network stack would allow it to schedule things fairly precisely in the application-centric manner you've described. I will ultimately defer to your judgement on that, though, and spend some time with the Qcc draft to see where I may be wrong. > It would be great if all of the user-level application code was scheduled > as tightly as the network, but the reality is that we aren't there yet. > Also, even if the end-station has a separately timed RT Linux thread for > each stream, the order of those threads can vary. Therefore, when the > end-station (SoC) has more than one talker, the frames for each talker > can be written into the Qbv queue at any time, in any order. > There is no scheduling of frames (i.e. streams)... only the queue. Frames are presented to *the kernel* in any order. The qdisc infrastructure provides mechanisms by which other scheduling policy can be added between presentation to the kernel and presentation to hardware queues. Even with i210-style LaunchTime functionality, this must also be provided. The hardware cannot re-order time-scheduled frames, and reviewers of the SO_TXTIME proposal rejected the idea of having the driver try to re-order scheduled frames already submitted to the hardware-visible descriptor chains, and rightly so I think. So far, I have thought of 3 places this ordering constraint might be provided: 1. Built into each driver with offload support for SO_TXTIME 2. A qdisc module that will re-order scheduled skbuffs within a certain window 3. Left to userspace to coordinate I think 1 is particularly bad due to duplication of code. 3 is the easy default from the point of view of the kernel, but not attractive from the point of view of precise scheduling and low latency to egress. Unless someone has a better idea, I think 2 would be the most appropriate choice. I can think of a number of ways one might be designed, but I have not thought them through fully yet. > From the perspective of the CNC, that means that a Qbv-only end-station is > sloppy. In contrast, an end-station using per-stream scheduling > (e.g. i210 with LaunchTime) is precise, because each frame has a known time. > It's true that P802.1Qcc supports both, so the CNC might support both. > Nevertheless, the CNC is likely to compute a network-wide schedule > that optimizes for the per-stream end-stations, and locates the windows > for the Qbv-only end-stations in a slower interval. Because out-of-order submission of frames to LaunchTime hardware would result in the frame that is enqueued later but scheduled earlier only transmitting once the later-scheduled frame completes, enqueuing frames immediately with LaunchTime information results in even *worse* sloppiness if two applications don't always submit their frames in the correct order to the kernel during a cycle. If your queue is simply gated at the time of the first scheduled launch, the frame that should have been first will only be late by the length of the queued-first frame. If they both have precise launch times, the frame that should have been first will have to wait until the normal launch time of the later-scheduled frame plus the time it takes for it to egress! In either case, re-ordering is required if it is not controlled for in userspace. In other words, I am not disagreeing with you about the insufficiency of a Qbv shaper for an endpoint! I am just pointing out that neither hardware LaunchTime support nor hardware Qbv window support are sufficient by themselves. Nor, for that case, is the CBS offload support sufficient to meet multi-stream requirements from SRP. All of these provide a low-layer mechanism by which appropriate scheduling of a multi-stream or multi-application endpoint can be enhanced, but none of them provide the complete story by themselves. > I realize that we are only doing CBS for now. I guess my main point is that > if/when we add scheduling, it cannot be limited to Qbv. Per-stream scheduling > is essential, because that is the assumption for most CNC designs that > I'm aware of. I understand and completely agree. I have believed that per-stream shaping/scheduling in some form is essential from
RE: [RFC net-next 0/5] TSN: Add qdisc-based config interfaces for traffic shapers
Thanks Levi, > One particular device I am working with now provides all network > access through a DSA switch chip with hardware Qbv support in addtion > to hardware Qav support. The SoC attached to it has no hardware timed > launch (SO_TXTIME) support. In this case, although the proposed > interface for Qbv is not *sufficient* to make a working time-aware end > station, it does provide a usable building block to provide one. As > with the credit-based shaping system, Talkers must provide an > additional level of per-stream shaping as well, but this is largely > (absent the jitter calculations, which are sort of a middle-level > concern) independent of what sort of hardware offload of the > scheduling is provided. It's a shame that someone built such hardware. Speaking as a manufacturer of daisy-chainable products for industrial/automotive applications, I wouldn't use that hardware in my products. The whole point of scheduling is to obtain close-to-optimal network determinism. If the hardware doesn't schedule properly, the product is probably better off using CBS. It would be great if all of the user-level application code was scheduled as tightly as the network, but the reality is that we aren't there yet. Also, even if the end-station has a separately timed RT Linux thread for each stream, the order of those threads can vary. Therefore, when the end-station (SoC) has more than one talker, the frames for each talker can be written into the Qbv queue at any time, in any order. There is no scheduling of frames (i.e. streams)... only the queue. From the perspective of the CNC, that means that a Qbv-only end-station is sloppy. In contrast, an end-station using per-stream scheduling (e.g. i210 with LaunchTime) is precise, because each frame has a known time. It's true that P802.1Qcc supports both, so the CNC might support both. Nevertheless, the CNC is likely to compute a network-wide schedule that optimizes for the per-stream end-stations, and locates the windows for the Qbv-only end-stations in a slower interval. I realize that we are only doing CBS for now. I guess my main point is that if/when we add scheduling, it cannot be limited to Qbv. Per-stream scheduling is essential, because that is the assumption for most CNC designs that I'm aware of. Rodney > -Original Message- > From: Levi Pearson [mailto:levipear...@gmail.com] > Sent: Monday, October 2, 2017 1:46 PM > To: Rodney Cummings > Cc: Linux Kernel Network Developers ; Vinicius > Costa Gomes ; Henrik Austad ; > richardcoch...@gmail.com; jesus.sanchez-palen...@intel.com; > andre.gue...@intel.com > Subject: Re: [RFC net-next 0/5] TSN: Add qdisc-based config interfaces for > traffic shapers > > Hi Rodney, > > Some archives seem to have threaded it, but I have CC'd the > participants I saw in the original discussion thread since they may > not otherwise notice it amongst the normal traffic. > > On Fri, Sep 29, 2017 at 2:44 PM, Rodney Cummings > wrote: > > Hi, > > > > I am posting my reply to this thread after subscribing, so I apologize > > if the archive happens to attach it to the wrong thread. > > > > First, I'd like to say that I strongly support this RFC. > > We need Linux interfaces for IEEE 802.1 TSN features. > > > > Although I haven't looked in detail, the proposal for CBS looks good. > > My questions/concerns are more related to future work, such for 802.1Qbv > > (scheduled traffic). > > > > 1. Question: From an 802.1 perspective, is this RFC intended to support > > end-station (e.g. NIC in host), bridges (i.e. DSA), or both? > > > > This is very important to clarify, because the usage of this interface > > will be very different for one or the other. > > > > For a bridge, the user code typically represents a remote management > > protocol (e.g. SNMP, NETCONF, RESTCONF), and this interface is > > expected to align with the specifications of 802.1Q clause 12, > > which serves as the information model for management. Historically, > > a standard kernel interface for management hasn't been viewed as > > essential, but I suppose it wouldn't hurt. > > I don't think the proposal was meant to cover the case of non-local > switch hardware, but in addition to dsa and switchdev switch ICs > managed by embedded Linux-running SoCs, there are SoCs with embedded > small port count switches or even plain multiple NICs with software > bridging. Many of these embedded small port count switches have FQTSS > hardware that could potentially be configured by the proposed cbs > qdisc. This blurs the line somewhat between what is a "bridge" and > what is an "end-station" in 802.1Q terminology, but nev
Re: [RFC net-next 0/5] TSN: Add qdisc-based config interfaces for traffic shapers
Hi Rodney, Some archives seem to have threaded it, but I have CC'd the participants I saw in the original discussion thread since they may not otherwise notice it amongst the normal traffic. On Fri, Sep 29, 2017 at 2:44 PM, Rodney Cummings wrote: > Hi, > > I am posting my reply to this thread after subscribing, so I apologize > if the archive happens to attach it to the wrong thread. > > First, I'd like to say that I strongly support this RFC. > We need Linux interfaces for IEEE 802.1 TSN features. > > Although I haven't looked in detail, the proposal for CBS looks good. > My questions/concerns are more related to future work, such for 802.1Qbv > (scheduled traffic). > > 1. Question: From an 802.1 perspective, is this RFC intended to support > end-station (e.g. NIC in host), bridges (i.e. DSA), or both? > > This is very important to clarify, because the usage of this interface > will be very different for one or the other. > > For a bridge, the user code typically represents a remote management > protocol (e.g. SNMP, NETCONF, RESTCONF), and this interface is > expected to align with the specifications of 802.1Q clause 12, > which serves as the information model for management. Historically, > a standard kernel interface for management hasn't been viewed as > essential, but I suppose it wouldn't hurt. I don't think the proposal was meant to cover the case of non-local switch hardware, but in addition to dsa and switchdev switch ICs managed by embedded Linux-running SoCs, there are SoCs with embedded small port count switches or even plain multiple NICs with software bridging. Many of these embedded small port count switches have FQTSS hardware that could potentially be configured by the proposed cbs qdisc. This blurs the line somewhat between what is a "bridge" and what is an "end-station" in 802.1Q terminology, but nevertheless these devices exist, sometimes acting as an endpoint + a real bridge and sometimes as just a system with multiple network interfaces. > For an end station, the user code can be an implementation of SRP > (802.1Q clause 35), or it can be an application-specific > protocol (e.g. industrial fieldbus) that exchanges data according > to P802.1Qcc clause 46. Either way, the top-level user interface > is designed for individual streams, not queues and shapers. That > implies some translation code between that top-level interface > and this sort of kernel interface. > > As a specific end-station example, for CBS, 802.1Q-2014 subclause > 34.6.1 requires "per-stream queues" in the Talker end-station. > I don't see 34.6.1 represented in the proposed RFC, but that's > okay... maybe per-stream queues are implemented in user code. > Nevertheless, if that is the assumption, I think we need to > clarify, especially in examples. You're correct that the FQTSS credit-based shaping algorithm requires per-stream shaping by Talker endpoints as well, but this is in addition to the per-class shaping provided by most hardware shaping implementations that I'm aware of in endpoint network hardware. I agree that we need to document the need to provide this, but it can definitely be built on top of the current proposal. I believe the per-stream shaping could be managed either by a user space application that manages all use of a streaming traffic class, or through an additional qdisc module that performs per-stream management on top of the proposed cbs qdisc, ensuring that the frames-per-observation interval aspect of each stream's reservation is obeyed. This becomes a fairly simple qdisc to implement on top of a per-traffic class shaper, and could even be implemented with the help of the timestamp that the SO_TXTIME proposal adds to skbuffs, but I think keeping the layers separate provides more flexibility to implementations and keeps management of various kinds of hardware offload support simpler as well. > 2. Suggestion: Do not assume that a time-aware (i.e. scheduled) > end-station will always use 802.1Qbv. > > For those who are subscribed to the 802.1 mailing list, > I'd suggest a read of draft P802.1Qcc/D1.6, subclause U.1 > of Annex U. Subclause U.1 assumes that bridges in the network use > 802.1Qbv, and then it poses the question of what an end-station > Talker should do. If the end-station also uses 802.1Qbv, > and that end-station transmits multiple streams, 802.1Qbv is > a bad implementation. The reason is that the scheduling > (i.e. order in time) of each stream cannot be controlled, which > in turn means that the CNC (network manager) cannot optimize > the 802.1Qbv schedules in bridges. The preferred technique > is to use "per-stream scheduling" in each Talker, so that > the CNC can create an optimal schedules (i.e. best determinism). > > I'm aware of a small number of proprietary CNC implementations for > 802.1Qbv in bridges, and they are generally assuming per-stream > scheduling in end-stations (Talkers). > > The i210 NIC's LaunchTime can be used to implement per-stream > scheduling. I h
Re: [RFC net-next 0/5] TSN: Add qdisc-based config interfaces for traffic shapers
Hi, I am posting my reply to this thread after subscribing, so I apologize if the archive happens to attach it to the wrong thread. First, I'd like to say that I strongly support this RFC. We need Linux interfaces for IEEE 802.1 TSN features. Although I haven't looked in detail, the proposal for CBS looks good. My questions/concerns are more related to future work, such for 802.1Qbv (scheduled traffic). 1. Question: From an 802.1 perspective, is this RFC intended to support end-station (e.g. NIC in host), bridges (i.e. DSA), or both? This is very important to clarify, because the usage of this interface will be very different for one or the other. For a bridge, the user code typically represents a remote management protocol (e.g. SNMP, NETCONF, RESTCONF), and this interface is expected to align with the specifications of 802.1Q clause 12, which serves as the information model for management. Historically, a standard kernel interface for management hasn't been viewed as essential, but I suppose it wouldn't hurt. For an end station, the user code can be an implementation of SRP (802.1Q clause 35), or it can be an application-specific protocol (e.g. industrial fieldbus) that exchanges data according to P802.1Qcc clause 46. Either way, the top-level user interface is designed for individual streams, not queues and shapers. That implies some translation code between that top-level interface and this sort of kernel interface. As a specific end-station example, for CBS, 802.1Q-2014 subclause 34.6.1 requires "per-stream queues" in the Talker end-station. I don't see 34.6.1 represented in the proposed RFC, but that's okay... maybe per-stream queues are implemented in user code. Nevertheless, if that is the assumption, I think we need to clarify, especially in examples. 2. Suggestion: Do not assume that a time-aware (i.e. scheduled) end-station will always use 802.1Qbv. For those who are subscribed to the 802.1 mailing list, I'd suggest a read of draft P802.1Qcc/D1.6, subclause U.1 of Annex U. Subclause U.1 assumes that bridges in the network use 802.1Qbv, and then it poses the question of what an end-station Talker should do. If the end-station also uses 802.1Qbv, and that end-station transmits multiple streams, 802.1Qbv is a bad implementation. The reason is that the scheduling (i.e. order in time) of each stream cannot be controlled, which in turn means that the CNC (network manager) cannot optimize the 802.1Qbv schedules in bridges. The preferred technique is to use "per-stream scheduling" in each Talker, so that the CNC can create an optimal schedules (i.e. best determinism). I'm aware of a small number of proprietary CNC implementations for 802.1Qbv in bridges, and they are generally assuming per-stream scheduling in end-stations (Talkers). The i210 NIC's LaunchTime can be used to implement per-stream scheduling. I haven't looked at SO_TXTIME in detail, but it sounds like per-stream scheduling. If so, then we already have the fundamental building blocks for a complete implementation of a time-aware end-station. If we answer the preceding question #1 as "end-station only", I would recommend avoiding 802.1Qbv in this interface. There isn't really anything wrong with it per-se, but it would lead developers down the wrong path. Rodney Cummings (National Instruments) Editor, IEEE P802.1Qcc --- Hi, This patchset is an RFC on a proposal of how the Traffic Control subsystem can be used to offload the configuration of traffic shapers into network devices that provide support for them in HW. Our goal here is to start upstreaming support for features related to the Time-Sensitive Networking (TSN) set of standards into the kernel. As part of this work, we've assessed previous public discussions related to TSN enabling: patches from Henrik Austad (Cisco), the presentation from Eric Mann at Linux Plumbers 2012, patches from Gangfeng Huang (National Instruments) and the current state of the OpenAVNU project (https://github.com/AVnu/OpenAvnu/). Please note that the patches provided as part of this RFC are implementing what is needed only for 802.1Qav (FQTSS) only, but we'd like to take advantage of this discussion and share our WIP ideas for the 802.1Qbv and 802.1Qbu interfaces as well. The current patches are only providing support for HW offload of the configs. Overview Time-sensitive Networking (TSN) is a set of standards that aim to address resources availability for providing bandwidth reservation and bounded latency on Ethernet based LANs. The proposal described here aims to cover mainly what is needed to enable the following standards: 802.1Qat, 802.1Qav, 802.1Qbv and 802.1Qbu. The initial target of this work is the Intel i210 NIC, but other controllers' datasheet were also taken into account, like the Renesas RZ/A1H RZ/A1M group and the Synopsis DesignWare Ethernet QoS controller. Proposal Feature-wise, what is covered here are configuration interfaces for H
Re: TSN Scorecard, was Re: [RFC net-next 0/5] TSN: Add qdisc-based config interfaces for traffic shapers
Hi, On 09/19/2017 10:49 PM, Richard Cochran wrote: (...) > > No, that is not what I meant. We need some minimal additional kernel > support in order to fully implement the TSN family of standards. Of > course, the bulk will have to be done in user space. It would be a > mistake to cram the stuff that belongs in userland into the kernel. > > Looking at the table, and reading your descriptions of the state of > OpenAVB, I remained convinced that the kernel needs only three > additions: > > 1. SO_TXTIME > 2. CBS Qdisc > 3. ALSA support for DAC clock control (but that is another story) We'll be posting the CBS v1 series for review soon. The current SO_TXTIME RFC for the purpose of Launchtime looks great, and we are looking forward for the v1 + its companion qdisc so we can test / review and provide feedback. We are still under the impression that a config interface for HW offload of Qbv / Qbu config will be needed, but we'll be deferring the 'taprio' proposal until there are NICs (end stations) that support these standards available. We can revisit it if that ever happens, and if it's still needed, but then taking into account SO_TXTIME (and its related qdisc). Thanks everyone for all the feedback so far. Regards, Jesus
Re: [RFC net-next 0/5] TSN: Add qdisc-based config interfaces for traffic shapers
On Tue, Sep 19, 2017 at 05:19:18PM -0700, Vinicius Costa Gomes wrote: > (I think LaunchTime is something specific to the i210, right?) Levi just told us: Recent SoCs from NXP (the i.MX 6 SoloX, and all the i.MX 7 and 8 parts) support Qav shaping as well as scheduled launch functionality; Thanks, Richard
Re: [RFC net-next 0/5] TSN: Add qdisc-based config interfaces for traffic shapers
On Tue, Sep 19, 2017 at 07:59:11PM -0600, levipear...@gmail.com wrote: > If some endpoint device shows up with direct Qbv support, this interface would > probably work well there too, although a talker would need to be able to > schedule its transmits pretty precisely to achieve the lowest possible > latency. This is an argument for SO_TXTIME. > One concern here is calling the base-time parameter an interval; it's really > an absolute time with respect to the PTP timescale. Good documentation will > be important to this one, since the specification discusses some subtleties > regarding the impact of different time values chosen here. > > The format for specifying the actual intervals such as cycle-time could prove > to be an important detail as well; Qbv specifies cycle-time as a ratio of two > integers expressed in seconds, while extension-time is specified as an integer > number of nanoseconds. > > Precision with the cycle-time is especially important, since base-time can be > almost arbitrarily far in the past or future, and any given cycle start should > be calculable from the base-time plus/minus some integer multiple of cycle- > time. The above three points also. Thanks, Richard
Re: TSN Scorecard, was Re: [RFC net-next 0/5] TSN: Add qdisc-based config interfaces for traffic shapers
On Tue, Sep 19, 2017 at 11:17:54PM -0600, levipear...@gmail.com wrote: > In addition to OpenAvnu, Renesas has a number of github repositories with > what looks like a fairly > complete media streaming system: Is it a generic stack or a set of hacks for their HW? > Although your SO_TXTIME proposal could certainly form the basis of an > endpoint's implementation of Qbv, I > think it is a stretch to consider it a Qbv implementation in itself, if > that's what you mean by this. No, that is not what I meant. We need some minimal additional kernel support in order to fully implement the TSN family of standards. Of course, the bulk will have to be done in user space. It would be a mistake to cram the stuff that belongs in userland into the kernel. Looking at the table, and reading your descriptions of the state of OpenAVB, I remained convinced that the kernel needs only three additions: 1. SO_TXTIME 2. CBS Qdisc 3. ALSA support for DAC clock control (but that is another story) > The proper interfaces for the Qbv configuration and managing of switch-level > PTP timestamps are not yet > in place, so there's nothing even at RFC stage to present yet, but > Qbv-capable Linux-managed switch > hardware is available and we hope to get some reusable code published even if > it's not yet ready to be > integrated in the kernel. Right, configuring Qbv in an attached DSA switch needs its own interface. Regarding PHC support for DSA switches, I have something in the works to be published soon. > A bit of progress has been made since that was written, although it is true > that it's still not > quite complete and certainly not turnkey. So OpenAVB is neither complete nor turnkey. That was my impression, too. > Things are maybe a bit farther along than they seemed, but there is still > important kernel work to be > done to reduce the need for out-of-tree drivers and to get everyone on the > same interfaces. I plan > to be an active participant going forward. You mentioned a couple of different kernel things you implemented. I would encourage you to post the work already done. Thanks, Richard
Re: [RFC net-next 0/5] TSN: Add qdisc-based config interfaces for traffic shapers
On Tue, Sep 19, 2017 at 05:19:18PM -0700, Vinicius Costa Gomes wrote: > One of the problems with OpenAVNU is that it's too coupled with the i210 > NIC. One of the things we want is to decouple OpenAVNU from the > controller. Yes, I want that, too. > The way we thought best was to propose interfaces (that > would work along side to the Linux networking stack) as close as > possible to what the current standards define, that means the IEEE > 802.1Q family of specifications, in the hope that network controller > vendors would also look at the specifications when designing their > controllers. These standard define the *behavior*, not the programming APIs. Our task as kernel developers is to invent the best interfaces for supporting 802.1Q and other standards, the hardware capabilities, and the widest range of applications (not jut AVB). > Our objective with the Qdiscs we are proposing (both cbs and taprio) is > to provide a sane way to configure controllers that support TSN features > (we were looking specifically at the IEEE specs). I can see how your proposed Qdiscs are inspired by the IEEE standards. However, in the case of time based transmission, I think there is a better way to do it, namely with SO_TXTIME (which BTW was originally proposed by Eric Mann). > After we have some rough consensus on the interfaces to use, then we can > start working on OpenAVNU. Did you see my table in the other mail? Any comments? > (Sorry if I am being annoying here, but the idea of an opaque schedule > is not ours, that comes from the people who wrote the Qbv specification) The schedule is easy to implement using SO_TXTIME. > I have a question, what about a controller that doesn't provide a way to > set a per-packet transmission time, but it supports Qbv/Qbu. What would > be your proposal to configure it? SO_TXTIME will have a generic SW fallback. BTW, regarding the i210, there is no sensible way to configure both CBS and time based transmission at the same time. The card performs a logical AND to make the launch decision. The effect of this is that each and every packet needs a LaunchTime, and the driver would be forced to guess the time for a packet before entering it into its queue. So if we end up merging CBS and SO_TXTIME, then we'll have to make them exclusive of each other (in the case of the i210) and manage the i210 queue configurations correctly. > (I think LaunchTime is something specific to the i210, right?) To my knowledge yes. However, if TSN does take hold, then other MAC vendors will copy it. Thanks, Richard
TSN Scorecard, was Re: [RFC net-next 0/5] TSN: Add qdisc-based config interfaces for traffic shapers
On Mon, Sep 18, 2017, Richard Cochran wrote: > Just for the record, here is my score card showing the current status > of TSN support in Linux. Comments and corrections are more welcome. > > Thanks, > Richard > > > | FEATURE| STANDARD| > STATUS | > > |+-+--| > | Synchronization| 802.1AS-2011| > Implemented in | > || | - > Linux kernel PHC subsystem | > || | - > linuxptp (userspace) | > > |+-+--| An alternate implementation of the userspace portion of gPTP is also available at [1] > | Forwarding and Queuing Enhancements| 802.1Q-2014 sec. 34 | RFC > posted (this thread) | > | for Time-Sensitive Streams (FQTSS) | | > | > > |+-+--| > | Stream Reservation Protocol (SRP) | 802.1Q-2014 sec. 35 | in > Open-AVB [1] | > > |+-+--| > | Audio Video Transport Protocol (AVTP) | IEEE 1722-2011 | DNE > | > > |+-+--| > | Audio/Video Device Discovery, Enumeration, | IEEE 1722.1-2013| > jdksavdecc-c [2] | > | Connection Management and Control (AVDECC) | | > | > | AVDECC Connection Management Protocol (ACMP) | | > | > | AVDECC Enumeration and Control Protocol (AECP) | | > | > | MAC Address Acquisition Protocol (MAAP)| | in > Open-AVB | > > |+-+--| All of the above are available to some degree in the AVTP Pipeline part of [1], specifically at this location: https://github.com/AVnu/OpenAvnu/tree/master/lib/avtp_pipeline The code is very modular and configurable, although some parts are in better shape than others. The AVTP portion can use the custom userspace driver for the i210, which can be configured to use launch scheduling, or it can use standard kernel interfaces via sendmsg or PACKET_MMAP. It runs as-is when configured for standard interfaces with any network hardware that supports gPTP. I previously implemented a CMSG-based launch time scheduling mechanism like the one you have proposed, and I have a socket backend for it that could easily be ported to your proposal. It is not part of the repository yet since there's no kernel support for it outside of my prototype and your RFC. It is currently tied to the OpenAvnu gPTP daemon rather than linuxptp, as it uses a shared memory interface to get the current rate-ratio and offset information between the various clocks. There may be better ways to do this, but that's how the initial port of the codebase was done. It would be nice to get it working with linuxptp's userspace tools at some point as well, though. The libraries under avtp_pipeline are designed to be used separately, but a simple integrated application is provided and is built by the CI system. In addition to OpenAvnu, Renesas has a number of github repositories with what looks like a fairly complete media streaming system: https://github.com/renesas-rcar/avb-mse https://github.com/renesas-rcar/avb-streaming https://github.com/renesas-rcar/avb-applications I haven't examined them in great detail yet, though. > | Frame Preemption | P802.1Qbu | DNE > | > | Scheduled Traffic | P802.1Qbv | RFC > posted (SO_TXTIME) | > | SRP Enhancements and Performance Improvements | P802.1Qcc | DNE > | > > DNE = Does Not Exist (to my knowledge) Although your SO_TXTIME proposal could certainly form the basis of an endpoint's implementation of Qbv, I think it is a stretch to consider it a Qbv implementation in itself, if that's what you mean by this. I have been working with colleagues on some experiments relating to a Linux-controlled DSN switch (a Marvell Topaz) that are a part of this effort in TSN: http://ieee802.org/1/files/public/docs2017/tsn-cgunther-802-3cg-multidrop-0917-v01.pdf The proper interfaces for the Qbv configurat
Re: [RFC net-next 0/5] TSN: Add qdisc-based config interfaces for traffic shapers
On Thu, Aug 31, 2017 at 06:26:20PM -0700, Vinicius Costa Gomes wrote: > Hi, > > This patchset is an RFC on a proposal of how the Traffic Control subsystem can > be used to offload the configuration of traffic shapers into network devices > that provide support for them in HW. Our goal here is to start upstreaming > support for features related to the Time-Sensitive Networking (TSN) set of > standards into the kernel. I'm very excited to see these features moving into the kernel! I am one of the maintainers of the OpenAvnu project and I've been involved in building AVB/TSN systems and working on the standards for around 10 years, so the support that's been slowly making it into more silicon and now Linux drivers is very encouraging. My team at Harman is working on endpoint code based on what's in the OpenAvnu project and a few Linux-based platforms. The Qav interface you've proposed will fit nicely with our traffic shaper management daemon, which already uses mqprio as a base but uses the htb shaper to approximate the Qav credit-based shaper on platforms where launch time scheduling isn't available. I've applied your patches and plan on testing them in conjunction with our shaper manager to see if we run into any hitches, but I don't expect any problems. > As part of this work, we've assessed previous public discussions related to > TSN > enabling: patches from Henrik Austad (Cisco), the presentation from Eric Mann > at Linux Plumbers 2012, patches from Gangfeng Huang (National Instruments) and > the current state of the OpenAVNU project (https://github.com/AVnu/OpenAvnu/). > > Please note that the patches provided as part of this RFC are implementing > what > is needed only for 802.1Qav (FQTSS) only, but we'd like to take advantage of > this discussion and share our WIP ideas for the 802.1Qbv and 802.1Qbu > interfaces > as well. The current patches are only providing support for HW offload of the > configs. > > > Overview > > > Time-sensitive Networking (TSN) is a set of standards that aim to address > resources availability for providing bandwidth reservation and bounded latency > on Ethernet based LANs. The proposal described here aims to cover mainly what > is > needed to enable the following standards: 802.1Qat, 802.1Qav, 802.1Qbv and > 802.1Qbu. > > The initial target of this work is the Intel i210 NIC, but other controllers' > datasheet were also taken into account, like the Renesas RZ/A1H RZ/A1M group > and > the Synopsis DesignWare Ethernet QoS controller. Recent SoCs from NXP (the i.MX 6 SoloX, and all the i.MX 7 and 8 parts) support Qav shaping as well as scheduled launch functionality; these are the parts I have been mostly working with. Marvell silicon (some subset of Armada processors and Link Street DSA switches) generally supports traffic shaping as well. I think a lack of an interface like this has probably slowed upstream driver support for this functionality where it exists; most vendors have an out-of- tree version of their driver with TSN functionality enabled via non-standard interfaces. Hopefully making it available will encourage vendors to upstream their driver support! > Proposal > > > Feature-wise, what is covered here are configuration interfaces for HW > implementations of the Credit-Based shaper (CBS, 802.1Qav), Time-Aware shaper > (802.1Qbv) and Frame Preemption (802.1Qbu). CBS is a per-queue shaper, while > Qbv and Qbu must be configured per port, with the configuration covering all > queues. Given that these features are related to traffic shaping, and that the > traffic control subsystem already provides a queueing discipline that offloads > config into the device driver (i.e. mqprio), designing new qdiscs for the > specific purpose of offloading the config for each shaper seemed like a good > fit. This makes sense to me too. The 802.1Q standards are all based on the sort of mappings between priority, traffic class, and hardware queues that the existing tc infrastructure seems to be modeling. I believe the mqprio module's mapping scheme is flexible enough to meet any TSN needs in conjunction with the other parts of the kernel qdisc system. > For steering traffic into the correct queues, we use the socket option > SO_PRIORITY and then a mechanism to map priority to traffic classes / > Txqueues. > The qdisc mqprio is currently used in our tests. > > As for the shapers config interface: > > * CBS (802.1Qav) > >This patchset is proposing a new qdisc called 'cbs'. Its 'tc' cmd line is: >$ tc qdisc add dev IFACE parent ID cbs locredit N hicredit M sendslope S \ > idleslope I > >Note that the parameters for this qdisc are the ones defined by the >802.1Q-2014 spec, so no hardware specific functionality is exposed here. These parameters look good to me as a baseline; some additional optional parameters may be useful for software-based implementations--such as setting an interval at which to recalculate queues--but
Re: [RFC net-next 0/5] TSN: Add qdisc-based config interfaces for traffic shapers
Hi Richard, Richard Cochran writes: > On Mon, Sep 18, 2017 at 04:06:28PM -0700, Vinicius Costa Gomes wrote: >> That's the point, the application does not need to know that, and asking >> that would be stupid. > > On the contrary, this information is essential to the application. > Probably you have never seen an actual Ethernet field bus in > operation? In any case, you are missing the point. > >> (And that's another nice point of how 802.1Qbv works, applications do >> not need to be changed to use it, and I think we should work to achieve >> this on the Linux side) > > Once you start to care about real time performance, then you need to > consider the applications. This is industrial control, not streaming > your tunes from your ipod. > >> That being said, that only works for kinds of traffic that maps well to >> this configuration in advance model, which is the model that the IEEE >> (see 802.1Qcc) and the AVNU Alliance[1] are pushing for. > > Again, you are missing the point of what they aiming for. I have > looked at a number of production systems, and in each case the > developers want total control over the transmission, in order to > reduce latency to an absolute minimum. Typically the data to be sent > are available only microseconds before the transmission deadline. > > Consider OpenAVB on github that people are already using. Take a look > at simple_talker.c and explain how "applications do not need to be > changed to use it." Just let me use the mention of OpenAVNU as a hook to explain what we (the team I am part of) are working to do, perhaps it will make our choices and designs clearer. One of the problems with OpenAVNU is that it's too coupled with the i210 NIC. One of the things we want is to decouple OpenAVNU from the controller. The way we thought best was to propose interfaces (that would work along side to the Linux networking stack) as close as possible to what the current standards define, that means the IEEE 802.1Q family of specifications, in the hope that network controller vendors would also look at the specifications when designing their controllers. Our objective with the Qdiscs we are proposing (both cbs and taprio) is to provide a sane way to configure controllers that support TSN features (we were looking specifically at the IEEE specs). After we have some rough consensus on the interfaces to use, then we can start working on OpenAVNU. > >> [1] >> http://avnu.org/theory-of-operation-for-tsn-enabled-industrial-systems/ > > Did you even read this? > > [page 24] > > As described in section 2, some industrial control systems require > predictable, very low latency and cycle-to-cycle variation to meet > hard real-time application requirements. In these systems, > multiple distributed controllers commonly synchronize their > sensor/actuator operations with other controllers by scheduling > these operations in time, typically using a repeating control > cycle. > ... > The gate control mechanism is itself a time-aware PTP application > operating within a bridge or end station port. > > It is an application, not a "god box." > >> In short, I see a per-packet transmission time and a per-queue schedule >> as solutions to different problems. > > Well, I can agree with that. For some non real-time applications, > bandwidth shaping is enough, and your Qdisc idea is sufficient. For > the really challenging TSN targets (industrial control, automotive), > your idea of an opaque schedule file won't fly. (Sorry if I am being annoying here, but the idea of an opaque schedule is not ours, that comes from the people who wrote the Qbv specification) I have a question, what about a controller that doesn't provide a way to set a per-packet transmission time, but it supports Qbv/Qbu. What would be your proposal to configure it? (I think LaunchTime is something specific to the i210, right?) Cheers, -- Vinicius
Re: [RFC net-next 0/5] TSN: Add qdisc-based config interfaces for traffic shapers
Hi all, On Tue, Sep 19, 2017 at 07:22:44AM +0200, Richard Cochran wrote: > On Mon, Sep 18, 2017 at 04:06:28PM -0700, Vinicius Costa Gomes wrote: > > That's the point, the application does not need to know that, and asking > > that would be stupid. > > On the contrary, this information is essential to the application. > Probably you have never seen an actual Ethernet field bus in > operation? In any case, you are missing the point. > > > (And that's another nice point of how 802.1Qbv works, applications do > > not need to be changed to use it, and I think we should work to achieve > > this on the Linux side) > > Once you start to care about real time performance, then you need to > consider the applications. This is industrial control, not streaming > your tunes from your ipod. Do not underestimate the need for media over TSN. I fully see your point of real-time systems, but they are not the only valid use-cases for TSN. > > That being said, that only works for kinds of traffic that maps well to > > this configuration in advance model, which is the model that the IEEE > > (see 802.1Qcc) and the AVNU Alliance[1] are pushing for. > > Again, you are missing the point of what they aiming for. I have > looked at a number of production systems, and in each case the > developers want total control over the transmission, in order to > reduce latency to an absolute minimum. Typically the data to be sent > are available only microseconds before the transmission deadline. > > Consider OpenAVB on github that people are already using. Take a look > at simple_talker.c and explain how "applications do not need to be > changed to use it." I do not think simple-talker was everintended to be how users of AVB should be implemented, but as a demonstration of what the protocol could do. ALSA/V4L2 should supply some interface to this so that you can attach media-applications to it without the application itself having to be "TSN aware". > > [1] > > http://avnu.org/theory-of-operation-for-tsn-enabled-industrial-systems/ > > Did you even read this? > > [page 24] > > As described in section 2, some industrial control systems require > predictable, very low latency and cycle-to-cycle variation to meet > hard real-time application requirements. In these systems, > multiple distributed controllers commonly synchronize their > sensor/actuator operations with other controllers by scheduling > these operations in time, typically using a repeating control > cycle. > ... > The gate control mechanism is itself a time-aware PTP application > operating within a bridge or end station port. > > It is an application, not a "god box." > > > In short, I see a per-packet transmission time and a per-queue schedule > > as solutions to different problems. > > Well, I can agree with that. For some non real-time applications, > bandwidth shaping is enough, and your Qdisc idea is sufficient. For > the really challenging TSN targets (industrial control, automotive), > your idea of an opaque schedule file won't fly. Would it make sense to adapt the proposed Qdisc here as well as the back-o-the-napkin idea in the other thread to to a per-socket queue for each priority and then sort those sockets based on SO_TXTIME? TSN operates on a per-StreamID basis, and that should map fairly well to a per-socket approach I think (let us just assume that an application that sends TSN traffic will open up a separate socket for each stream. This should allow a userspace application that is _very_ aware of its timing constraints to send frames exactly when it needs to as you have SO_TXTIME available. It would also let applications that basically want a fine-grained rate control (audio and video comes to mind) to use the same qdisc. For those sockets that do not support SO_TXTIME, but still map to a priority handled by sch_cbs (or whatever it'll end up being called) you can set the transmit-time to be the time of the last skb in the queue + an delta which will give you the correct rate (TSN operates on observation intervals which you can specify via tc when you create the queues). Then you can have, as you propose in your other series, a hrtimer that is being called when the next SO_TXTIME enters, grab a skb and move it to the hw-queue. This should also allow you to keep a sorted per-socket queue should an application send frames in the wrong order, without having to rearrange descriptors for the DMA machinery. If this makes sense, I am more than happy to give it a stab and see how it goes. -Henrik signature.asc Description: PGP signature
Re: [RFC net-next 0/5] TSN: Add qdisc-based config interfaces for traffic shapers
On Mon, Sep 18, 2017 at 04:06:28PM -0700, Vinicius Costa Gomes wrote: > That's the point, the application does not need to know that, and asking > that would be stupid. On the contrary, this information is essential to the application. Probably you have never seen an actual Ethernet field bus in operation? In any case, you are missing the point. > (And that's another nice point of how 802.1Qbv works, applications do > not need to be changed to use it, and I think we should work to achieve > this on the Linux side) Once you start to care about real time performance, then you need to consider the applications. This is industrial control, not streaming your tunes from your ipod. > That being said, that only works for kinds of traffic that maps well to > this configuration in advance model, which is the model that the IEEE > (see 802.1Qcc) and the AVNU Alliance[1] are pushing for. Again, you are missing the point of what they aiming for. I have looked at a number of production systems, and in each case the developers want total control over the transmission, in order to reduce latency to an absolute minimum. Typically the data to be sent are available only microseconds before the transmission deadline. Consider OpenAVB on github that people are already using. Take a look at simple_talker.c and explain how "applications do not need to be changed to use it." > [1] > http://avnu.org/theory-of-operation-for-tsn-enabled-industrial-systems/ Did you even read this? [page 24] As described in section 2, some industrial control systems require predictable, very low latency and cycle-to-cycle variation to meet hard real-time application requirements. In these systems, multiple distributed controllers commonly synchronize their sensor/actuator operations with other controllers by scheduling these operations in time, typically using a repeating control cycle. ... The gate control mechanism is itself a time-aware PTP application operating within a bridge or end station port. It is an application, not a "god box." > In short, I see a per-packet transmission time and a per-queue schedule > as solutions to different problems. Well, I can agree with that. For some non real-time applications, bandwidth shaping is enough, and your Qdisc idea is sufficient. For the really challenging TSN targets (industrial control, automotive), your idea of an opaque schedule file won't fly. Thanks, Richard
Re: [RFC net-next 0/5] TSN: Add qdisc-based config interfaces for traffic shapers
Hi Richard, Richard Cochran writes: > On Thu, Aug 31, 2017 at 06:26:20PM -0700, Vinicius Costa Gomes wrote: >> * Time-aware shaper (802.1Qbv): > > I just posted a working alternative showing how to handle 802.1Qbv and > many other Ethernet field buses. > >>The idea we are currently exploring is to add a "time-aware", priority >> based >>qdisc, that also exposes the Tx queues available and provides a mechanism >> for >>mapping priority <-> traffic class <-> Tx queues in a similar fashion as >>mqprio. We are calling this qdisc 'taprio', and its 'tc' cmd line would >> be: >> >>$ $ tc qdisc add dev ens4 parent root handle 100 taprio num_tc 4\ >> map 2 2 1 0 3 3 3 3 3 3 3 3 3 3 3 3 \ >> queues 0 1 2 3 \ >> sched-file gates.sched [base-time ] \ >>[cycle-time ] [extension-time ] >> >> is multi-line, with each line being of the following format: >> >> >>Qbv only defines one : "S" for 'SetGates' >> >>For example: >> >>S 0x01 300 >>S 0x03 500 >> >>This means that there are two intervals, the first will have the gate >>for traffic class 0 open for 300 nanoseconds, the second will have >>both traffic classes open for 500 nanoseconds. > > The idea of the schedule file will not work in practice. Consider the > fact that the application wants to deliver time critical data in a > particular slot. How can it find out a) what the time slots are and > b) when the next slot is scheduled? With this Qdisc, it cannot do > this, AFAICT. The admin might delete the file after configuring the > Qdisc! That's the point, the application does not need to know that, and asking that would be stupid. From the point of view of the Qbv specification, applications only need to care about its basic bandwidth requirements: its interval, frame size, frames per interval (using the terms of the SRP section of 802.1Q). The traffic schedule is provided (off band) by a "god box" which knows all the requirements of all applications in all the nodes and how they are connected. (And that's another nice point of how 802.1Qbv works, applications do not need to be changed to use it, and I think we should work to achieve this on the Linux side) That being said, that only works for kinds of traffic that maps well to this configuration in advance model, which is the model that the IEEE (see 802.1Qcc) and the AVNU Alliance[1] are pushing for. In the real world, I can see multiple types of applications, some using something like TXTIME, and some configured in advance. > > Using the SO_TXTIME option, the application has total control over the > scheduling. The great advantages of this approach is that we can > support any possible combination of periodic or aperiodic scheduling > and we can support any priority scheme user space dreams up. It has the disavantage of that the scheduling information has to be in-band with the data. I *really* think that for scheduled traffic, there should be a clear separation, we should not mix the dataflow with scheduling. In short, an application in the network don't need to have all the information necessary to schedule its own traffic well. I have two points here: 1. I see both "solutions" (taprio and SO_TXTIME) as being ortoghonal and useful, both; 2. trying to make one do the job of the other, however, looks like "If all I have is a hammer, everything looks like a nail". In short, I see a per-packet transmission time and a per-queue schedule as solutions to different problems. > > For example, one can imaging running two or more loops that only > occasionally collide. When they do collide, which packet should be > sent first? Just let user space decide. > > Thanks, > Richard Cheers, -- Vinicius [1] http://avnu.org/theory-of-operation-for-tsn-enabled-industrial-systems/
Re: [RFC net-next 0/5] TSN: Add qdisc-based config interfaces for traffic shapers
On Mon, Sep 18, 2017 at 10:02:14AM +0200, Richard Cochran wrote: > On Thu, Aug 31, 2017 at 06:26:20PM -0700, Vinicius Costa Gomes wrote: > > * Time-aware shaper (802.1Qbv): > > I just posted a working alternative showing how to handle 802.1Qbv and > many other Ethernet field buses. Yes, I saw them, grabbing them for testing now - thanks! > >The idea we are currently exploring is to add a "time-aware", priority > > based > >qdisc, that also exposes the Tx queues available and provides a > > mechanism for > >mapping priority <-> traffic class <-> Tx queues in a similar fashion as > >mqprio. We are calling this qdisc 'taprio', and its 'tc' cmd line would > > be: > > > >$ $ tc qdisc add dev ens4 parent root handle 100 taprio num_tc 4\ > >map 2 2 1 0 3 3 3 3 3 3 3 3 3 3 3 3 \ > >queues 0 1 2 3 \ > >sched-file gates.sched [base-time ] \ > >[cycle-time ] [extension-time ] > > > > is multi-line, with each line being of the following format: > > > > > >Qbv only defines one : "S" for 'SetGates' > > > >For example: > > > >S 0x01 300 > >S 0x03 500 > > > >This means that there are two intervals, the first will have the gate > >for traffic class 0 open for 300 nanoseconds, the second will have > >both traffic classes open for 500 nanoseconds. > > The idea of the schedule file will not work in practice. Consider the > fact that the application wants to deliver time critical data in a > particular slot. How can it find out a) what the time slots are and > b) when the next slot is scheduled? With this Qdisc, it cannot do > this, AFAICT. The admin might delete the file after configuring the > Qdisc! > > Using the SO_TXTIME option, the application has total control over the > scheduling. The great advantages of this approach is that we can > support any possible combination of periodic or aperiodic scheduling > and we can support any priority scheme user space dreams up. Using SO_TXTIME makes a lot of sense. TSN has a presentation_time, which you can use to deduce the time it should be transmitted (Class A has a 2ms latency guarantee, B has 50), but given how TSN uses the timestamp, it will wrap every 4.3 seconds, using SO_TXTIME allows you to schedule transmission at a much later time. It should also lessen the dependency on a specific protocol, which is also good. > For example, one can imaging running two or more loops that only > occasionally collide. When they do collide, which packet should be > sent first? Just let user space decide. If 2 userspace apps send to the same Tx-queue with the same priority, would it not make sense to just do FIFO? For all practical purposes, they have the same importance (same SO_PRIORITY, same SO_TXTIME). If the priority differs, then they would be directed to different queues, where one queue will take presedence anyway. How far into the future would it make sense to schedule packets anyway? I'll have a look at the other series you just posted! -- Henrik Austad signature.asc Description: PGP signature
Re: [RFC net-next 0/5] TSN: Add qdisc-based config interfaces for traffic shapers
On Thu, Aug 31, 2017 at 06:26:20PM -0700, Vinicius Costa Gomes wrote: > This patchset is an RFC on a proposal of how the Traffic Control subsystem can > be used to offload the configuration of traffic shapers into network devices > that provide support for them in HW. Our goal here is to start upstreaming > support for features related to the Time-Sensitive Networking (TSN) set of > standards into the kernel. Just for the record, here is my score card showing the current status of TSN support in Linux. Comments and corrections are more welcome. Thanks, Richard | FEATURE| STANDARD| STATUS | |+-+--| | Synchronization| 802.1AS-2011| Implemented in | || | - Linux kernel PHC subsystem | || | - linuxptp (userspace) | |+-+--| | Forwarding and Queuing Enhancements| 802.1Q-2014 sec. 34 | RFC posted (this thread) | | for Time-Sensitive Streams (FQTSS) | | | |+-+--| | Stream Reservation Protocol (SRP) | 802.1Q-2014 sec. 35 | in Open-AVB [1] | |+-+--| | Audio Video Transport Protocol (AVTP) | IEEE 1722-2011 | DNE | |+-+--| | Audio/Video Device Discovery, Enumeration, | IEEE 1722.1-2013| jdksavdecc-c [2] | | Connection Management and Control (AVDECC) | | | | AVDECC Connection Management Protocol (ACMP) | | | | AVDECC Enumeration and Control Protocol (AECP) | | | | MAC Address Acquisition Protocol (MAAP)| | in Open-AVB | |+-+--| | Frame Preemption | P802.1Qbu | DNE | | Scheduled Traffic | P802.1Qbv | RFC posted (SO_TXTIME) | | SRP Enhancements and Performance Improvements | P802.1Qcc | DNE | DNE = Does Not Exist (to my knowledge) 1. https://github.com/Avnu/OpenAvnu (DISCLAIMER from the website:) It is planned to eventually include the various packet encapsulation types, protocol discovery daemons, libraries to convert media clocks to AVB clocks and vice versa, and drivers. This repository does not include all components required to build a full production AVB/TSN system (e.g. a turnkey solution to stream stored or live audio or video content). Some simple example applications are provided which illustrate the flow - but a professional Audio/Video system requires a full media stack - including audio and video inputs and outputs, media processing elements, and various graphical user interfaces. Various companies provide such integrated solutions. 2. https://github.com/jdkoftinoff/jdksavdecc-c
Re: [RFC net-next 0/5] TSN: Add qdisc-based config interfaces for traffic shapers
On Thu, Aug 31, 2017 at 06:26:20PM -0700, Vinicius Costa Gomes wrote: > * Time-aware shaper (802.1Qbv): I just posted a working alternative showing how to handle 802.1Qbv and many other Ethernet field buses. >The idea we are currently exploring is to add a "time-aware", priority > based >qdisc, that also exposes the Tx queues available and provides a mechanism > for >mapping priority <-> traffic class <-> Tx queues in a similar fashion as >mqprio. We are calling this qdisc 'taprio', and its 'tc' cmd line would be: > >$ $ tc qdisc add dev ens4 parent root handle 100 taprio num_tc 4\ > map 2 2 1 0 3 3 3 3 3 3 3 3 3 3 3 3 \ > queues 0 1 2 3 \ > sched-file gates.sched [base-time ] \ >[cycle-time ] [extension-time ] > > is multi-line, with each line being of the following format: > > >Qbv only defines one : "S" for 'SetGates' > >For example: > >S 0x01 300 >S 0x03 500 > >This means that there are two intervals, the first will have the gate >for traffic class 0 open for 300 nanoseconds, the second will have >both traffic classes open for 500 nanoseconds. The idea of the schedule file will not work in practice. Consider the fact that the application wants to deliver time critical data in a particular slot. How can it find out a) what the time slots are and b) when the next slot is scheduled? With this Qdisc, it cannot do this, AFAICT. The admin might delete the file after configuring the Qdisc! Using the SO_TXTIME option, the application has total control over the scheduling. The great advantages of this approach is that we can support any possible combination of periodic or aperiodic scheduling and we can support any priority scheme user space dreams up. For example, one can imaging running two or more loops that only occasionally collide. When they do collide, which packet should be sent first? Just let user space decide. Thanks, Richard
Re: [RFC net-next 0/5] TSN: Add qdisc-based config interfaces for traffic shapers
On Thu, Sep 07, 2017 at 06:29:00PM -0700, Vinicius Costa Gomes wrote: > >> * Time-aware shaper (802.1Qbv): > >> > >>The idea we are currently exploring is to add a "time-aware", priority > >> based > >>qdisc, that also exposes the Tx queues available and provides a > >> mechanism for > >>mapping priority <-> traffic class <-> Tx queues in a similar fashion as > >>mqprio. We are calling this qdisc 'taprio', and its 'tc' cmd line would > >> be: > > > > As far as I know, this is not supported by i210, and if time-aware shaping > > is enabled in the network - you'll be queued on a bridge until the window > > opens as time-aware shaping is enforced on the tx-port and not on rx. Is > > this required in this driver? > > Yeah, i210 doesn't support the time-aware shaper. I think the second > part of your question doesn't really apply, then. Actually, you can implement 802.1Qbv (as an end station) quite easily using the i210. I'll show how by posting a series after net-next opens up again. Thanks, Richard
Re: [RFC net-next 0/5] TSN: Add qdisc-based config interfaces for traffic shapers
On Thu, Sep 07, 2017 at 07:58:53PM +, Guedes, Andre wrote: > Hi Henrik, > > Thanks for your feedback! I'll address some of your comments below. > > On Thu, 2017-09-07 at 07:34 +0200, Henrik Austad wrote: > > > As for the shapers config interface: > > > > > > * CBS (802.1Qav) > > > > > > This patchset is proposing a new qdisc called 'cbs'. Its 'tc' cmd line > > > is: > > > $ tc qdisc add dev IFACE parent ID cbs locredit N hicredit M sendslope > > > S > > > \ > > > idleslope I > > > > So this confuses me a bit, why specify sendSlope? > > > > sendSlope = portTransmitRate - idleSlope > > > > and portTransmitRate is the speed of the MAC (which you get from the > > driver). Adding sendSlope here is just redundant I think. > > Yes, this was something we've spent quite a few time discussing before this > RFC > series. After reading the Annex L from 802.1Q-2014 (operation of CBS > algorithm) > so many times, we've came up with the rationale explained below. > > The rationale here is that sendSlope is just another parameter from CBS > algorithm like idleSlope, hiCredit and loCredit. As such, its calculation > should be done at the same "layer" as the others parameters (in this case, > user > space) in order to keep consistency. Moreover, in this design, the driver > layer > is dead simple: all the device driver has to do is applying CBS parameters to > hardware. Having any CBS parameter calculation in the driver layer means all > device drivers must implement that calculation. Ok, that actually makes a lot of sense, and anything that keeps this kind of arithmetic outside the kernel is a good thing! Thanks for the clarification! > > Also, does this mean that when you create the qdisc, you have locked the > > bandwidth for the scheduler? Meaning, if I later want to add another > > stream that requires more bandwidth, I have to close all active streams, > > reconfigure the qdisc and then restart? > > If we want to reserve more bandwidth to "accommodate" a new stream, we don't > need to close all active streams. All we have to do is changing the CBS qdisc > and pass the new CBS parameters. Here is what the command-line would look > like: > > $ tc qdisc change dev enp0s4 parent 8001:5 cbs locredit -1470 hicredit 30 > sendslope -98 idleslope 2 > > No application/stream is interrupted while new CBS parameters are applied. Ah, good. > > > Note that the parameters for this qdisc are the ones defined by the > > > 802.1Q-2014 spec, so no hardware specific functionality is exposed > > > here. > > > > You do need to know if the link is brought up as 100 or 1000 though - which > > the driver already knows. > > User space knows that information via ethtool or /sys. Fair point. > > > Testing this RFC > > > > > > > > > For testing the patches of this RFC only, you can refer to the samples and > > > helper script being added to samples/tsn/ and the use the 'mqprio' qdisc > > > to > > > setup the priorities to Tx queues mapping, together with the 'cbs' qdisc > > > to > > > configure the HW shaper of the i210 controller: > > > > I will test it, feedback will be provided soon! :) > > That's great! Please let us know if you find any issue and thanks for you > support. > > > > 8) You can also run a Talker for class B (prio 2 here) > > > $ ./talker -i enp3s0 -p 2 > > > > > > * The bandwidth displayed on the listener output now should increase to > > > very > > > close to the one configured for class A + class B. > > > > Because you grab both class A *and* B, or because B will eat what A does > > not use? > > Because the listener application grabs both class A and B traffic. Right, got it. Thanks for the feedback, I'm getting really excited about this! :D -- Henrik Austad signature.asc Description: PGP signature
Re: [RFC net-next 0/5] TSN: Add qdisc-based config interfaces for traffic shapers
Henrik Austad writes: > On Thu, Aug 31, 2017 at 06:26:20PM -0700, Vinicius Costa Gomes wrote: >> Hi, >> >> This patchset is an RFC on a proposal of how the Traffic Control subsystem >> can >> be used to offload the configuration of traffic shapers into network devices >> that provide support for them in HW. Our goal here is to start upstreaming >> support for features related to the Time-Sensitive Networking (TSN) set of >> standards into the kernel. > > Nice to see that others are working on this as well! :) > > A short disclaimer; I'm pretty much anchored in the view "linux is the > end-station in a TSN domain", is this your approach as well, or are you > looking at this driver to be used in bridges as well? (because that will > affect the comments on time-aware shaper and frame preemption) > > Yet another disclaimer; I am not a linux networking subsystem expert. Not > by a long shot! There are black magic happening in the internals of the > networking subsystem that I am not even aware of. So if something I say or > ask does not make sense _at_all_, that's probably why.. > > I do know a tiny bit about TSN though, and I have been messing around > with it for a little while, hence my comments below > >> As part of this work, we've assessed previous public discussions related to >> TSN >> enabling: patches from Henrik Austad (Cisco), the presentation from Eric Mann >> at Linux Plumbers 2012, patches from Gangfeng Huang (National Instruments) >> and >> the current state of the OpenAVNU project >> (https://github.com/AVnu/OpenAvnu/). > > /me eyes Cc ;p > >> Overview >> >> >> Time-sensitive Networking (TSN) is a set of standards that aim to address >> resources availability for providing bandwidth reservation and bounded >> latency >> on Ethernet based LANs. The proposal described here aims to cover mainly >> what is >> needed to enable the following standards: 802.1Qat, 802.1Qav, 802.1Qbv and >> 802.1Qbu. >> >> The initial target of this work is the Intel i210 NIC, but other controllers' >> datasheet were also taken into account, like the Renesas RZ/A1H RZ/A1M group >> and >> the Synopsis DesignWare Ethernet QoS controller. > > NXP has a TSN aware chip on the i.MX7 sabre board as well Cool. Will take a look. > >> Proposal >> >> >> Feature-wise, what is covered here are configuration interfaces for HW >> implementations of the Credit-Based shaper (CBS, 802.1Qav), Time-Aware shaper >> (802.1Qbv) and Frame Preemption (802.1Qbu). CBS is a per-queue shaper, while >> Qbv and Qbu must be configured per port, with the configuration covering all >> queues. Given that these features are related to traffic shaping, and that >> the >> traffic control subsystem already provides a queueing discipline that >> offloads >> config into the device driver (i.e. mqprio), designing new qdiscs for the >> specific purpose of offloading the config for each shaper seemed like a good >> fit. > > just to be clear, you register sch_cbs as a subclass to mqprio, not as a > root class? That's right. > >> For steering traffic into the correct queues, we use the socket option >> SO_PRIORITY and then a mechanism to map priority to traffic classes / Tx >> queues. >> The qdisc mqprio is currently used in our tests. > > Right, fair enough, I'd prefer the TSN qdisc to be the root-device and > rather have mqprio for high priority traffic and another for 'everything > else'', but this would work too. This is not that relevant at this stage I > guess :) That's a scenario I haven't considered, will give it some thought. > >> As for the shapers config interface: >> >> * CBS (802.1Qav) >> >>This patchset is proposing a new qdisc called 'cbs'. Its 'tc' cmd line is: >>$ tc qdisc add dev IFACE parent ID cbs locredit N hicredit M sendslope S \ >> idleslope I > > So this confuses me a bit, why specify sendSlope? > > sendSlope = portTransmitRate - idleSlope > > and portTransmitRate is the speed of the MAC (which you get from the > driver). Adding sendSlope here is just redundant I think. > > Also, does this mean that when you create the qdisc, you have locked the > bandwidth for the scheduler? Meaning, if I later want to add another > stream that requires more bandwidth, I have to close all active streams, > reconfigure the qdisc and then restart? > >>Note that the parameters for this qdisc are the ones defined by the >>802.1Q-2014 spec, so no hardware specific functionality is exposed here. > > You do need to know if the link is brought up as 100 or 1000 though - which > the driver already knows. > >> * Time-aware shaper (802.1Qbv): >> >>The idea we are currently exploring is to add a "time-aware", priority >> based >>qdisc, that also exposes the Tx queues available and provides a mechanism >> for >>mapping priority <-> traffic class <-> Tx queues in a similar fashion as >>mqprio. We are calling this qdisc 'taprio', and its 'tc' cmd line would >> be: > > As far as I know, thi
Re: [RFC net-next 0/5] TSN: Add qdisc-based config interfaces for traffic shapers
On Thu, 2017-09-07 at 18:18 +0200, Henrik Austad wrote: > On Thu, Sep 07, 2017 at 05:53:15PM +0200, Richard Cochran wrote: > > On Thu, Sep 07, 2017 at 05:27:51PM +0200, Henrik Austad wrote: > > > On Thu, Sep 07, 2017 at 02:40:18PM +0200, Richard Cochran wrote: > > > And if you want to this driver to act as a bridge, how do you accomodate > > > change in network requirements? (i.e. how does this work with switchdev?) > > > > To my understanding, this Qdisc idea provides QoS for the host's > > transmitted traffic, and nothing more. > > Ok, then we're on the same page. > > > > - Or am I overthinking this? > > > > Being able to configure the external ports of a switchdev is probably > > a nice feature, but that is another story. (But maybe I misunderstood > > the authors' intent!) > > ok, chalk that one up for later perhaps Just to clarify, we've been most focused on end-station use-cases. We've considered some bridge use-cases as well just to verify that the proposed design won't be an issue if someone else goes for it. - Andre smime.p7s Description: S/MIME cryptographic signature
Re: [RFC net-next 0/5] TSN: Add qdisc-based config interfaces for traffic shapers
Hi Henrik, Thanks for your feedback! I'll address some of your comments below. On Thu, 2017-09-07 at 07:34 +0200, Henrik Austad wrote: > > As for the shapers config interface: > > > > * CBS (802.1Qav) > > > > This patchset is proposing a new qdisc called 'cbs'. Its 'tc' cmd line > > is: > > $ tc qdisc add dev IFACE parent ID cbs locredit N hicredit M sendslope S > > \ > > idleslope I > > So this confuses me a bit, why specify sendSlope? > > sendSlope = portTransmitRate - idleSlope > > and portTransmitRate is the speed of the MAC (which you get from the > driver). Adding sendSlope here is just redundant I think. Yes, this was something we've spent quite a few time discussing before this RFC series. After reading the Annex L from 802.1Q-2014 (operation of CBS algorithm) so many times, we've came up with the rationale explained below. The rationale here is that sendSlope is just another parameter from CBS algorithm like idleSlope, hiCredit and loCredit. As such, its calculation should be done at the same "layer" as the others parameters (in this case, user space) in order to keep consistency. Moreover, in this design, the driver layer is dead simple: all the device driver has to do is applying CBS parameters to hardware. Having any CBS parameter calculation in the driver layer means all device drivers must implement that calculation. > Also, does this mean that when you create the qdisc, you have locked the > bandwidth for the scheduler? Meaning, if I later want to add another > stream that requires more bandwidth, I have to close all active streams, > reconfigure the qdisc and then restart? If we want to reserve more bandwidth to "accommodate" a new stream, we don't need to close all active streams. All we have to do is changing the CBS qdisc and pass the new CBS parameters. Here is what the command-line would look like: $ tc qdisc change dev enp0s4 parent 8001:5 cbs locredit -1470 hicredit 30 sendslope -98 idleslope 2 No application/stream is interrupted while new CBS parameters are applied. > > Note that the parameters for this qdisc are the ones defined by the > > 802.1Q-2014 spec, so no hardware specific functionality is exposed here. > > You do need to know if the link is brought up as 100 or 1000 though - which > the driver already knows. User space knows that information via ethtool or /sys. > > Testing this RFC > > > > > > For testing the patches of this RFC only, you can refer to the samples and > > helper script being added to samples/tsn/ and the use the 'mqprio' qdisc to > > setup the priorities to Tx queues mapping, together with the 'cbs' qdisc to > > configure the HW shaper of the i210 controller: > > I will test it, feedback will be provided soon! :) That's great! Please let us know if you find any issue and thanks for you support. > > 8) You can also run a Talker for class B (prio 2 here) > > $ ./talker -i enp3s0 -p 2 > > > > * The bandwidth displayed on the listener output now should increase to > > very > > close to the one configured for class A + class B. > > Because you grab both class A *and* B, or because B will eat what A does > not use? Because the listener application grabs both class A and B traffic. Regards, Andre smime.p7s Description: S/MIME cryptographic signature
Re: [RFC net-next 0/5] TSN: Add qdisc-based config interfaces for traffic shapers
On Thu, Sep 07, 2017 at 05:53:15PM +0200, Richard Cochran wrote: > On Thu, Sep 07, 2017 at 05:27:51PM +0200, Henrik Austad wrote: > > On Thu, Sep 07, 2017 at 02:40:18PM +0200, Richard Cochran wrote: > > And if you want to this driver to act as a bridge, how do you accomodate > > change in network requirements? (i.e. how does this work with switchdev?) > > To my understanding, this Qdisc idea provides QoS for the host's > transmitted traffic, and nothing more. Ok, then we're on the same page. > > - Or am I overthinking this? > > Being able to configure the external ports of a switchdev is probably > a nice feature, but that is another story. (But maybe I misunderstood > the authors' intent!) ok, chalk that one up for later perhaps > > If you have more than 1 application in userspace that wants to send data > > using this scheduler, how do you ensure fair transmission of frames? (both > > how much bandwidth they use, > > There are many ways to handle this, and we shouldn't put any of that > policy into the kernel. For example, there might be a monolithic > application with configurable threads, or an allocation server that > grants bandwidth to applications via IPC, or a multiplexing stream > server like jack, pulse, etc, and so on... true > > but also ordering of frames from each application) > > Not sure what you mean by this. Fair enough, I'm not that good at making myself clear :) Let's see if I can make a better attempt: If you have 2 separate applications that have their own streams going to different endpoints - but both are in the same class, then they will share the qdisc bandwidth. So application - A sends frame A1, A2, A3, .. An - B sends B1, B2, .. Bn What I was trying to describe was: if application A send 2 frames, and B sends 2 frames at the same time, then you would hope that the order would be A1, B1, A2, B2, and not A1, A2, B1, B2. None of this would be a problem if you expect a *single* user, like the allocation server you described above. Again, I think this is just me overthinking the problem right now :) > > Do you expect all of this to be handled in userspace? > > Yes, I do. ok, fair enough Thanks for answering my questions! -- Henrik Austad signature.asc Description: PGP signature
Re: [RFC net-next 0/5] TSN: Add qdisc-based config interfaces for traffic shapers
On Thu, Sep 07, 2017 at 05:27:51PM +0200, Henrik Austad wrote: > On Thu, Sep 07, 2017 at 02:40:18PM +0200, Richard Cochran wrote: > And if you want to this driver to act as a bridge, how do you accomodate > change in network requirements? (i.e. how does this work with switchdev?) To my understanding, this Qdisc idea provides QoS for the host's transmitted traffic, and nothing more. > - Or am I overthinking this? Being able to configure the external ports of a switchdev is probably a nice feature, but that is another story. (But maybe I misunderstood the authors' intent!) > If you have more than 1 application in userspace that wants to send data > using this scheduler, how do you ensure fair transmission of frames? (both > how much bandwidth they use, There are many ways to handle this, and we shouldn't put any of that policy into the kernel. For example, there might be a monolithic application with configurable threads, or an allocation server that grants bandwidth to applications via IPC, or a multiplexing stream server like jack, pulse, etc, and so on... > but also ordering of frames from each application) Not sure what you mean by this. > Do you expect all of this to be handled in userspace? Yes, I do. Thanks, Richard
Re: [RFC net-next 0/5] TSN: Add qdisc-based config interfaces for traffic shapers
On Thu, Sep 07, 2017 at 02:40:18PM +0200, Richard Cochran wrote: > On Thu, Sep 07, 2017 at 07:34:11AM +0200, Henrik Austad wrote: > > Also, does this mean that when you create the qdisc, you have locked the > > bandwidth for the scheduler? Meaning, if I later want to add another > > stream that requires more bandwidth, I have to close all active streams, > > reconfigure the qdisc and then restart? > > No, just allocate enough bandwidth to accomodate all of the expected > streams. The streams can start and stop at will. Sure, that'll work. And if you want to this driver to act as a bridge, how do you accomodate change in network requirements? (i.e. how does this work with switchdev?) - Or am I overthinking this? > > So my understanding of all of this is that you configure the *total* > > bandwith for each class when you load the qdisc and then let userspace > > handle the rest. Is this correct? > > Nothing wrong with that. Didn't mean to say it was wrong, just making sure I've understood the concept. > > In my view, it would be nice if the qdisc had some notion about streams so > > that you could create a stream, feed frames to it and let the driver pace > > them out. (The fewer you queue, the shorter the delay). This will also > > allow you to enforce per-stream bandwidth restrictions. I don't see how you > > can do this here unless you want to do this in userspace. > > > > Do you have any plans for adding support for multiplexing streams? If you > > have multiple streams, how do you enforce that one stream does not eat into > > the bandwidth of another stream? AFAIK, this is something the network must > > enforce, but I see no option of doing som here. > > Please, lets keep this simple. Simple is always good > Today we have exactly zero user space > applications using this kind of bandwidth reservation. The case of > wanting the kernel to police individual stream usage does not exist, > and probably never will. That we have *zero* userspace applications today is probably related to the fact that we have exacatly *zero* drivers in the kernel that talks TSN :) To rephrase a bit, what I'm worried about: If you have more than 1 application in userspace that wants to send data using this scheduler, how do you ensure fair transmission of frames? (both how much bandwidth they use, but also ordering of frames from each application) Do you expect all of this to be handled in userspace? > For serious TSN use cases, the bandwidth needed by each system and > indeed the entire network will be engineered, and we can reasonably > expect applications to cooperate in this regard. yes.. that'll happen ;) > Thanks, > Richard Don't get me wrong, I think it is great that others are working on this! I'm just trying to fully understand the thought that have gone into this and how it is inteded to be used. I'll get busy testing the code and wrapping my head around the different parameters. -- Henrik Austad signature.asc Description: PGP signature
Re: [RFC net-next 0/5] TSN: Add qdisc-based config interfaces for traffic shapers
On Thu, Sep 07, 2017 at 07:34:11AM +0200, Henrik Austad wrote: > Also, does this mean that when you create the qdisc, you have locked the > bandwidth for the scheduler? Meaning, if I later want to add another > stream that requires more bandwidth, I have to close all active streams, > reconfigure the qdisc and then restart? No, just allocate enough bandwidth to accomodate all of the expected streams. The streams can start and stop at will. > So my understanding of all of this is that you configure the *total* > bandwith for each class when you load the qdisc and then let userspace > handle the rest. Is this correct? Nothing wrong with that. > In my view, it would be nice if the qdisc had some notion about streams so > that you could create a stream, feed frames to it and let the driver pace > them out. (The fewer you queue, the shorter the delay). This will also > allow you to enforce per-stream bandwidth restrictions. I don't see how you > can do this here unless you want to do this in userspace. > > Do you have any plans for adding support for multiplexing streams? If you > have multiple streams, how do you enforce that one stream does not eat into > the bandwidth of another stream? AFAIK, this is something the network must > enforce, but I see no option of doing som here. Please, lets keep this simple. Today we have exactly zero user space applications using this kind of bandwidth reservation. The case of wanting the kernel to police individual stream usage does not exist, and probably never will. For serious TSN use cases, the bandwidth needed by each system and indeed the entire network will be engineered, and we can reasonably expect applications to cooperate in this regard. Thanks, Richard
Re: [RFC net-next 0/5] TSN: Add qdisc-based config interfaces for traffic shapers
On Thu, Aug 31, 2017 at 06:26:20PM -0700, Vinicius Costa Gomes wrote: > Hi, > > This patchset is an RFC on a proposal of how the Traffic Control subsystem can > be used to offload the configuration of traffic shapers into network devices > that provide support for them in HW. Our goal here is to start upstreaming > support for features related to the Time-Sensitive Networking (TSN) set of > standards into the kernel. Nice to see that others are working on this as well! :) A short disclaimer; I'm pretty much anchored in the view "linux is the end-station in a TSN domain", is this your approach as well, or are you looking at this driver to be used in bridges as well? (because that will affect the comments on time-aware shaper and frame preemption) Yet another disclaimer; I am not a linux networking subsystem expert. Not by a long shot! There are black magic happening in the internals of the networking subsystem that I am not even aware of. So if something I say or ask does not make sense _at_all_, that's probably why.. I do know a tiny bit about TSN though, and I have been messing around with it for a little while, hence my comments below > As part of this work, we've assessed previous public discussions related to > TSN > enabling: patches from Henrik Austad (Cisco), the presentation from Eric Mann > at Linux Plumbers 2012, patches from Gangfeng Huang (National Instruments) and > the current state of the OpenAVNU project (https://github.com/AVnu/OpenAvnu/). /me eyes Cc ;p > Overview > > > Time-sensitive Networking (TSN) is a set of standards that aim to address > resources availability for providing bandwidth reservation and bounded latency > on Ethernet based LANs. The proposal described here aims to cover mainly what > is > needed to enable the following standards: 802.1Qat, 802.1Qav, 802.1Qbv and > 802.1Qbu. > > The initial target of this work is the Intel i210 NIC, but other controllers' > datasheet were also taken into account, like the Renesas RZ/A1H RZ/A1M group > and > the Synopsis DesignWare Ethernet QoS controller. NXP has a TSN aware chip on the i.MX7 sabre board as well > Proposal > > > Feature-wise, what is covered here are configuration interfaces for HW > implementations of the Credit-Based shaper (CBS, 802.1Qav), Time-Aware shaper > (802.1Qbv) and Frame Preemption (802.1Qbu). CBS is a per-queue shaper, while > Qbv and Qbu must be configured per port, with the configuration covering all > queues. Given that these features are related to traffic shaping, and that the > traffic control subsystem already provides a queueing discipline that offloads > config into the device driver (i.e. mqprio), designing new qdiscs for the > specific purpose of offloading the config for each shaper seemed like a good > fit. just to be clear, you register sch_cbs as a subclass to mqprio, not as a root class? > For steering traffic into the correct queues, we use the socket option > SO_PRIORITY and then a mechanism to map priority to traffic classes / Tx > queues. > The qdisc mqprio is currently used in our tests. Right, fair enough, I'd prefer the TSN qdisc to be the root-device and rather have mqprio for high priority traffic and another for 'everything else'', but this would work too. This is not that relevant at this stage I guess :) > As for the shapers config interface: > > * CBS (802.1Qav) > >This patchset is proposing a new qdisc called 'cbs'. Its 'tc' cmd line is: >$ tc qdisc add dev IFACE parent ID cbs locredit N hicredit M sendslope S \ > idleslope I So this confuses me a bit, why specify sendSlope? sendSlope = portTransmitRate - idleSlope and portTransmitRate is the speed of the MAC (which you get from the driver). Adding sendSlope here is just redundant I think. Also, does this mean that when you create the qdisc, you have locked the bandwidth for the scheduler? Meaning, if I later want to add another stream that requires more bandwidth, I have to close all active streams, reconfigure the qdisc and then restart? >Note that the parameters for this qdisc are the ones defined by the >802.1Q-2014 spec, so no hardware specific functionality is exposed here. You do need to know if the link is brought up as 100 or 1000 though - which the driver already knows. > * Time-aware shaper (802.1Qbv): > >The idea we are currently exploring is to add a "time-aware", priority > based >qdisc, that also exposes the Tx queues available and provides a mechanism > for >mapping priority <-> traffic class <-> Tx queues in a similar fashion as >mqprio. We are calling this qdisc 'taprio', and its 'tc' cmd line would be: As far as I know, this is not supported by i210, and if time-aware shaping is enabled in the network - you'll be queued on a bridge until the window opens as time-aware shaping is enforced on the tx-port and not on rx. Is this required in this driver? >$ $ tc qdisc add dev ens4 parent root hand
Re: [RFC net-next 0/5] TSN: Add qdisc-based config interfaces for traffic shapers
On Fri, Sep 01, 2017 at 09:12:17AM -0700, Jesus Sanchez-Palencia wrote: > On 09/01/2017 06:03 AM, Richard Cochran wrote: > > The timing of this RFC is good, as I am just finishing up an RFC that > > implements time-based transmit using the i210. I'll try and get that > > out ASAP. I have an RFC series ready for net-next, but the the merge window just started. I'll post it when the window closes again... Thanks, Richard
Re: [RFC net-next 0/5] TSN: Add qdisc-based config interfaces for traffic shapers
On Fri, Sep 01, 2017 at 09:12:17AM -0700, Jesus Sanchez-Palencia wrote: > Is it correct to assume you are referring to an interface for Launchtime here? Yes. Thanks, Richard
Re: [RFC net-next 0/5] TSN: Add qdisc-based config interfaces for traffic shapers
Hi Richard, On 09/01/2017 06:03 AM, Richard Cochran wrote: > > I happy to see this posted. At first glance, it seems like a step in > the right direction. > > On Thu, Aug 31, 2017 at 06:26:20PM -0700, Vinicius Costa Gomes wrote: >> * Time-aware shaper (802.1Qbv): > ... >>S 0x01 300 >>S 0x03 500 >> >>This means that there are two intervals, the first will have the gate >>for traffic class 0 open for 300 nanoseconds, the second will have >>both traffic classes open for 500 nanoseconds. > > The i210 doesn't support this in HW, or does it? No, it does not. i210 only provides support for a per-packet feature called LaunchTime that can be used control both the fetch and the transmission time of packets. > >> * Frame Preemption (802.1Qbu): >> >>To control even further the latency, it may prove useful to signal which >>traffic classes are marked as preemptable. For that, 'taprio' provides the >>preemption command so you set each traffic class as preemptable or not: >> >>$ tc qdisc (...) \ >> preemption 0 1 1 1 > > Neither can the i210 preempt frames, or what am I missing? No, it does not. But when we started working on the shapers we decided to look ahead and try to come up with interfaces that could cover beyond 802.1Qav. These are just some ideas we've been prototyping here together with the 'cbs' qdisc. > > The timing of this RFC is good, as I am just finishing up an RFC that > implements time-based transmit using the i210. I'll try and get that > out ASAP. Is it correct to assume you are referring to an interface for Launchtime here? Thanks, Jesus
Re: [RFC net-next 0/5] TSN: Add qdisc-based config interfaces for traffic shapers
I happy to see this posted. At first glance, it seems like a step in the right direction. On Thu, Aug 31, 2017 at 06:26:20PM -0700, Vinicius Costa Gomes wrote: > * Time-aware shaper (802.1Qbv): ... >S 0x01 300 >S 0x03 500 > >This means that there are two intervals, the first will have the gate >for traffic class 0 open for 300 nanoseconds, the second will have >both traffic classes open for 500 nanoseconds. The i210 doesn't support this in HW, or does it? > * Frame Preemption (802.1Qbu): > >To control even further the latency, it may prove useful to signal which >traffic classes are marked as preemptable. For that, 'taprio' provides the >preemption command so you set each traffic class as preemptable or not: > >$ tc qdisc (...) \ > preemption 0 1 1 1 Neither can the i210 preempt frames, or what am I missing? The timing of this RFC is good, as I am just finishing up an RFC that implements time-based transmit using the i210. I'll try and get that out ASAP. Thanks, Richard
[RFC net-next 0/5] TSN: Add qdisc-based config interfaces for traffic shapers
Hi, This patchset is an RFC on a proposal of how the Traffic Control subsystem can be used to offload the configuration of traffic shapers into network devices that provide support for them in HW. Our goal here is to start upstreaming support for features related to the Time-Sensitive Networking (TSN) set of standards into the kernel. As part of this work, we've assessed previous public discussions related to TSN enabling: patches from Henrik Austad (Cisco), the presentation from Eric Mann at Linux Plumbers 2012, patches from Gangfeng Huang (National Instruments) and the current state of the OpenAVNU project (https://github.com/AVnu/OpenAvnu/). Please note that the patches provided as part of this RFC are implementing what is needed only for 802.1Qav (FQTSS) only, but we'd like to take advantage of this discussion and share our WIP ideas for the 802.1Qbv and 802.1Qbu interfaces as well. The current patches are only providing support for HW offload of the configs. Overview Time-sensitive Networking (TSN) is a set of standards that aim to address resources availability for providing bandwidth reservation and bounded latency on Ethernet based LANs. The proposal described here aims to cover mainly what is needed to enable the following standards: 802.1Qat, 802.1Qav, 802.1Qbv and 802.1Qbu. The initial target of this work is the Intel i210 NIC, but other controllers' datasheet were also taken into account, like the Renesas RZ/A1H RZ/A1M group and the Synopsis DesignWare Ethernet QoS controller. Proposal Feature-wise, what is covered here are configuration interfaces for HW implementations of the Credit-Based shaper (CBS, 802.1Qav), Time-Aware shaper (802.1Qbv) and Frame Preemption (802.1Qbu). CBS is a per-queue shaper, while Qbv and Qbu must be configured per port, with the configuration covering all queues. Given that these features are related to traffic shaping, and that the traffic control subsystem already provides a queueing discipline that offloads config into the device driver (i.e. mqprio), designing new qdiscs for the specific purpose of offloading the config for each shaper seemed like a good fit. For steering traffic into the correct queues, we use the socket option SO_PRIORITY and then a mechanism to map priority to traffic classes / Tx queues. The qdisc mqprio is currently used in our tests. As for the shapers config interface: * CBS (802.1Qav) This patchset is proposing a new qdisc called 'cbs'. Its 'tc' cmd line is: $ tc qdisc add dev IFACE parent ID cbs locredit N hicredit M sendslope S \ idleslope I Note that the parameters for this qdisc are the ones defined by the 802.1Q-2014 spec, so no hardware specific functionality is exposed here. * Time-aware shaper (802.1Qbv): The idea we are currently exploring is to add a "time-aware", priority based qdisc, that also exposes the Tx queues available and provides a mechanism for mapping priority <-> traffic class <-> Tx queues in a similar fashion as mqprio. We are calling this qdisc 'taprio', and its 'tc' cmd line would be: $ $ tc qdisc add dev ens4 parent root handle 100 taprio num_tc 4\ map 2 2 1 0 3 3 3 3 3 3 3 3 3 3 3 3 \ queues 0 1 2 3 \ sched-file gates.sched [base-time ] \ [cycle-time ] [extension-time ] is multi-line, with each line being of the following format: Qbv only defines one : "S" for 'SetGates' For example: S 0x01 300 S 0x03 500 This means that there are two intervals, the first will have the gate for traffic class 0 open for 300 nanoseconds, the second will have both traffic classes open for 500 nanoseconds. Additionally, an option to set just one entry of the gate control list will also be provided by 'taprio': $ tc qdisc (...) \ sched-row \ [base-time ] [cycle-time ] \ [extension-time ] * Frame Preemption (802.1Qbu): To control even further the latency, it may prove useful to signal which traffic classes are marked as preemptable. For that, 'taprio' provides the preemption command so you set each traffic class as preemptable or not: $ tc qdisc (...) \ preemption 0 1 1 1 * Time-aware shaper + Preemption: As an example of how Qbv and Qbu can be used together, we may specify both the schedule and the preempt-mask, and this way we may also specify the Set-Gates-and-Hold and Set-Gates-and-Release commands as specified in the Qbu spec: $ tc qdisc add dev ens4 parent root handle 100 taprio num_tc 4 \ map 2 2 1 0 3 3 3 3 3 3 3 3 3 3 3 3\ queues 0 1 2 3 \ preemption 0 1 1 1 \ sched-file preempt_gates.sched is multi-line, with each line being of the following format: F
