Re: [PATCH RFC V1 net-next 0/6] Time based packet transmission
> This series is an early RFC that introduces a new socket option > allowing time based transmission of packets. This option will be > useful in implementing various real time protocols over Ethernet, > including but not limited to P802.1Qbv, which is currently finding > its way into 802.1Q. > > * Open questions about SO_TXTIME semantics > > - What should the kernel do if the dialed Tx time is in the past? > Should the packet be sent ASAP, or should we throw an error? Based on the i210 and latest NXP/Freescale FEC launch time behavior, the hardware timestamps work over 1-second windows corresponding to the time elapsed since the last PTP second began. When considering the head-of-queue frame, the launch time is compared to the elapsed time counter and if the elapsed time is between exactly the launch time and half a second after the launch time, it is launched. If you enqueue a frame with a scheduled launch time that ends up more than half a second late, it is considered by the hardware to be scheduled *in the future* at the offset belonging to the next second after the 1-second window wraps around. So *slightly* late (<<.5sec late) frames could be scheduled as normal, but approaching .5sec late frames would have to either be dropped or have their schedule changed to avoid blocking the queue for a large fraction of a second. I don't like the idea of changing the scheduled time, and anything that is close to half a second late is most likely useless. But it is also reasonable to let barely-late frames go out ASAP--in the case of a Qav- shaped stream, the bunching would get smoothed out downstream. A timed launch schedule need not be used as an exact time, but a "don't send before time X" flag. Both are useful in different circumstances. A configurable parameter for allowable lateness, with the upper bound set by the driver based on the hardware capabilities, seems ideal. Barring that, I would suggest dropping frames with already-missed launch times. > > - Should the kernel inform the user if it detects a missed deadline, > via the error queue for example? I think some sort of counter for mis-scheduled/late-delivered frames would be in keeping with the general 802.1 error handling strategy. > > - What should the timescale be for the dialed Tx time? Should the > kernel select UTC when using the SW Qdisc and the HW time > otherwise? Or should the socket option include a clockid_t? When I implemented something like this, I left it relative to the HW time for the sake of simplicity, but I don't have a strong opinion. > > * Things todo > > - Design a Qdisc for purpose of configuring SO_TXTIME. There should > be one option to dial HW offloading or SW best effort. You seem focused on Qbv, but there is another aspect of the endpoint requirements for Qav that this would provide a perfect use case for. A bridge can treat all traffic in a Qav-shaped class equally, but an endpoint must essentially run one credit-based shaper per active stream feeding into the class--this is because a stream must adhere to its frames-per-interval promise in its t-spec, and when the observation interval is not an even multiple of the sample rate, it will occasionally have an observation interval with no frame. This leaves extra bandwidth in the class reservation, but it cannot be used by any other stream if it would cause more than one frame per interval to be sent! Even if a stream is not explicitly scheduled in userspace, a per-stream Qdisc could apply a rough launch time that the class Qdisc (or hardware shaping) would use to ensure the frames-per-interval aspect of the reservation for the stream is adhered to. For example, each observation interval could be assigned a launch time, and all streams would get a number of frames corresponding to their frames-per-interval reservation assigned that same launch time before being put into the class queue. The i210's shaper would then only consider the current interval's set of frames ready to launch, and spread them evenly with its hardware credit-based shaping. For industrial and automotive control applications, a Qbv Qdisc based on SO_TXTIME would be very interesting, but pro and automotive media uses will most likely continue to use SRP + Qav, and these are becoming increasingly common uses as you can see by the growing support for Qav in automotive chips. > - Implement the SW best effort variant. Here is my back of the > napkin sketch. Each interface has its own timerqueue keeping the > TXTIME packets in order and a FIFO for all other traffic. A guard > window starts at the earliest deadline minus the maximum MTU minus > a configurable fudge factor. The Qdisc uses a hrtimer to transmit > the next packet in the timerqueue. During the guard window, all > other traffic is defered unless the next packet can be transmitted > before the guard window expires. This sounds plausible to me. > > * Current
TSN Scorecard, was Re: [RFC net-next 0/5] TSN: Add qdisc-based config interfaces for traffic shapers
On Mon, Sep 18, 2017, Richard Cochran wrote: > Just for the record, here is my score card showing the current status > of TSN support in Linux. Comments and corrections are more welcome. > > Thanks, > Richard > > > | FEATURE| STANDARD| > STATUS | > > |+-+--| > | Synchronization| 802.1AS-2011| > Implemented in | > || | - > Linux kernel PHC subsystem | > || | - > linuxptp (userspace) | > > |+-+--| An alternate implementation of the userspace portion of gPTP is also available at [1] > | Forwarding and Queuing Enhancements| 802.1Q-2014 sec. 34 | RFC > posted (this thread) | > | for Time-Sensitive Streams (FQTSS) | | > | > > |+-+--| > | Stream Reservation Protocol (SRP) | 802.1Q-2014 sec. 35 | in > Open-AVB [1] | > > |+-+--| > | Audio Video Transport Protocol (AVTP) | IEEE 1722-2011 | DNE > | > > |+-+--| > | Audio/Video Device Discovery, Enumeration, | IEEE 1722.1-2013| > jdksavdecc-c [2] | > | Connection Management and Control (AVDECC) | | > | > | AVDECC Connection Management Protocol (ACMP) | | > | > | AVDECC Enumeration and Control Protocol (AECP) | | > | > | MAC Address Acquisition Protocol (MAAP)| | in > Open-AVB | > > |+-+--| All of the above are available to some degree in the AVTP Pipeline part of [1], specifically at this location: https://github.com/AVnu/OpenAvnu/tree/master/lib/avtp_pipeline The code is very modular and configurable, although some parts are in better shape than others. The AVTP portion can use the custom userspace driver for the i210, which can be configured to use launch scheduling, or it can use standard kernel interfaces via sendmsg or PACKET_MMAP. It runs as-is when configured for standard interfaces with any network hardware that supports gPTP. I previously implemented a CMSG-based launch time scheduling mechanism like the one you have proposed, and I have a socket backend for it that could easily be ported to your proposal. It is not part of the repository yet since there's no kernel support for it outside of my prototype and your RFC. It is currently tied to the OpenAvnu gPTP daemon rather than linuxptp, as it uses a shared memory interface to get the current rate-ratio and offset information between the various clocks. There may be better ways to do this, but that's how the initial port of the codebase was done. It would be nice to get it working with linuxptp's userspace tools at some point as well, though. The libraries under avtp_pipeline are designed to be used separately, but a simple integrated application is provided and is built by the CI system. In addition to OpenAvnu, Renesas has a number of github repositories with what looks like a fairly complete media streaming system: https://github.com/renesas-rcar/avb-mse https://github.com/renesas-rcar/avb-streaming https://github.com/renesas-rcar/avb-applications I haven't examined them in great detail yet, though. > | Frame Preemption | P802.1Qbu | DNE > | > | Scheduled Traffic | P802.1Qbv | RFC > posted (SO_TXTIME) | > | SRP Enhancements and Performance Improvements | P802.1Qcc | DNE > | > > DNE = Does Not Exist (to my knowledge) Although your SO_TXTIME proposal could certainly form the basis of an endpoint's implementation of Qbv, I think it is a stretch to consider it a Qbv implementation in itself, if that's what you mean by this. I have been working with colleagues on some experiments relating to a Linux-controlled DSN switch (a Marvell Topaz) that are a part of this effort in TSN: http://ieee802.org/1/files/public/docs2017/tsn-cgunther-802-3cg-multidrop-0917-v01.pdf The proper interfaces for the Qbv
Re: [RFC net-next 0/5] TSN: Add qdisc-based config interfaces for traffic shapers
On Thu, Aug 31, 2017 at 06:26:20PM -0700, Vinicius Costa Gomes wrote: > Hi, > > This patchset is an RFC on a proposal of how the Traffic Control subsystem can > be used to offload the configuration of traffic shapers into network devices > that provide support for them in HW. Our goal here is to start upstreaming > support for features related to the Time-Sensitive Networking (TSN) set of > standards into the kernel. I'm very excited to see these features moving into the kernel! I am one of the maintainers of the OpenAvnu project and I've been involved in building AVB/TSN systems and working on the standards for around 10 years, so the support that's been slowly making it into more silicon and now Linux drivers is very encouraging. My team at Harman is working on endpoint code based on what's in the OpenAvnu project and a few Linux-based platforms. The Qav interface you've proposed will fit nicely with our traffic shaper management daemon, which already uses mqprio as a base but uses the htb shaper to approximate the Qav credit-based shaper on platforms where launch time scheduling isn't available. I've applied your patches and plan on testing them in conjunction with our shaper manager to see if we run into any hitches, but I don't expect any problems. > As part of this work, we've assessed previous public discussions related to > TSN > enabling: patches from Henrik Austad (Cisco), the presentation from Eric Mann > at Linux Plumbers 2012, patches from Gangfeng Huang (National Instruments) and > the current state of the OpenAVNU project (https://github.com/AVnu/OpenAvnu/). > > Please note that the patches provided as part of this RFC are implementing > what > is needed only for 802.1Qav (FQTSS) only, but we'd like to take advantage of > this discussion and share our WIP ideas for the 802.1Qbv and 802.1Qbu > interfaces > as well. The current patches are only providing support for HW offload of the > configs. > > > Overview > > > Time-sensitive Networking (TSN) is a set of standards that aim to address > resources availability for providing bandwidth reservation and bounded latency > on Ethernet based LANs. The proposal described here aims to cover mainly what > is > needed to enable the following standards: 802.1Qat, 802.1Qav, 802.1Qbv and > 802.1Qbu. > > The initial target of this work is the Intel i210 NIC, but other controllers' > datasheet were also taken into account, like the Renesas RZ/A1H RZ/A1M group > and > the Synopsis DesignWare Ethernet QoS controller. Recent SoCs from NXP (the i.MX 6 SoloX, and all the i.MX 7 and 8 parts) support Qav shaping as well as scheduled launch functionality; these are the parts I have been mostly working with. Marvell silicon (some subset of Armada processors and Link Street DSA switches) generally supports traffic shaping as well. I think a lack of an interface like this has probably slowed upstream driver support for this functionality where it exists; most vendors have an out-of- tree version of their driver with TSN functionality enabled via non-standard interfaces. Hopefully making it available will encourage vendors to upstream their driver support! > Proposal > > > Feature-wise, what is covered here are configuration interfaces for HW > implementations of the Credit-Based shaper (CBS, 802.1Qav), Time-Aware shaper > (802.1Qbv) and Frame Preemption (802.1Qbu). CBS is a per-queue shaper, while > Qbv and Qbu must be configured per port, with the configuration covering all > queues. Given that these features are related to traffic shaping, and that the > traffic control subsystem already provides a queueing discipline that offloads > config into the device driver (i.e. mqprio), designing new qdiscs for the > specific purpose of offloading the config for each shaper seemed like a good > fit. This makes sense to me too. The 802.1Q standards are all based on the sort of mappings between priority, traffic class, and hardware queues that the existing tc infrastructure seems to be modeling. I believe the mqprio module's mapping scheme is flexible enough to meet any TSN needs in conjunction with the other parts of the kernel qdisc system. > For steering traffic into the correct queues, we use the socket option > SO_PRIORITY and then a mechanism to map priority to traffic classes / > Txqueues. > The qdisc mqprio is currently used in our tests. > > As for the shapers config interface: > > * CBS (802.1Qav) > >This patchset is proposing a new qdisc called 'cbs'. Its 'tc' cmd line is: >$ tc qdisc add dev IFACE parent ID cbs locredit N hicredit M sendslope S \ > idleslope I > >Note that the parameters for this qdisc are the ones defined by the >802.1Q-2014 spec, so no hardware specific functionality is exposed here. These parameters look good to me as a baseline; some additional optional parameters may be useful for software-based implementations--such as setting an interval at which to recalculate queues--but
